It should always matter as to what you wear. When you first walk into the room, it's telling the interviewer who you are and whether you want to make an effort. Looking smart is always a good way to show that you can be professional - rather than turning up in a top and jeans.