I understand that it bothers you. I haven't been in the same situation myself, so don't know exactly how I'd feel about it. But for me - when a relationship is over, it is over. It doesn't really matter if I'm dating someone new yet, I still get rid of old stuff, simply to move on. The only reason you'd keep old letters, texts, pictures and so on is basically to have the possibility to sit and think about the time you spent together, which is highly inappropriate with a new partner and will get you nowhere.
Yes sometimes it's painful to delete sweet words your love wrote to you, but I get rid of it no matter how hard. It's just stuff, and particularly in this situation - if it bothers you, he should throw it out.