The wall has certain bonds in it which require energy to break, not force. As you tap it lightly there is insufficient energy to break any of the bonds so the energy dissipates. When you get over the threshold you rightly identified, there is enough energy to start breaking bonds, it's nothing to do with the force really. The extra force your applying is more a byproduct of giving the hammer extra energy. It also takes a while to fall down because you're only breaking a certain number of bonds with each strike, you're going to need to seperate whole sections of the wall before they start to topple.