You are not alone here, there will be many many thousands of people who cannot reverse in a straight line, or even reverse.
The inability to recognise an access road, even if it is an unmade dirt track firstly is training. It is also an attitudinal thing. You always need to consider who else needs to use the same location you are on, and why you exclusively have the right to block a road or others. Some would say selfish which is why the other car driver got so cross. What if it had been an ambulance?
Your driving anxiety will not go away until you get the skills and confidence to be able to reverse and move into tight spaces and gaps. Just get some more training when you can (afford it) and you will ace it.