Well banks used to observe something like 33 different bank holidays, mostly being religious festivals or saint' days.
The main reason they're called bank holidays is because the Bank of England decided them, not the state (aka the state is what makes it a public holiday). Many employers gradually started giving workers time off these days as a mixture of religiousness sentiment and the fact banks would be closed anyway.
In effect this resulted in Bank holidays being the Days that Banks and businesses closed, eventually made law, such as in the "Banking and Financial Dealings Act 1971". These days are set in the legal groundwork.
Public Holidays however are not legally bank holidays, however are completely the same in reality, and as such are Common Law Holidays, Good friday and Christmas (yes christmas) are such examples of public holidays that aren't bank holidays.
'However' there's no technical difference...... the Law makes Christmas Day and Good Friday identical to bank holidays, but keeps them separate 'just cos english law'.