I mean technically to listen to the heart properly, the doctor should lift the breast out of the way with the back of the hand - the doctor perhaps thought that would be considered even more inappropriate?
I imagine the most likely explanation is that the doctor was listening for heart sounds/breath sounds, but obviously it is difficult to comment accurately without being there.