You are obviously right, in fact I think most people would really accept this as true. The problem is that people love to conflate being a pedophile with molesting children, which results in people calling for all pedophiles to be jailed, which is obviously bizarre as they have not by definition committed a crime.
The solution is to emphasise the distinction between rapists, specifically child rapists, and pedophiles, who are presumably on the whole harmless. I would in fact posit that the way we treat pedophiles now aggravates the situation because it forces them to turn to other experienced pedophiles, who may be child rapists, for comfort, which could obviously drag them in to molesting children.