It's basically electrical potential energy. Electrons repel one another due to the Coulomb force so, if you have a high electron density at one end of the battery, they will try to push their way out along the path of least resistance. In a circuit that path is normally along the wire.
As for how batteries work, it involves the oxidization and reduction of two different ionic solutions separated by a salt bridge, and I don't fully understand it because I never did chemistry beyond GCSE. But the end result is that they manage to maintain a constant electric potential difference, which forces electrons to keep flowing from one solution to the other via a circuit.