I think it's important to remember the cost of war, but I don't personally feel that the poppy is the best means to do that. Wearing the poppy has become somewhat self-congratulory. It's like updating your Facebook status to let people know you've gone out and voted. It turns something that's supposed to be about others into something about yourself. It's also a way for people to feel superior to others (patriotism, poppy shaming).
The poppy as a symbol is taken from World War 1, which I believe should remind us of the horror and senseless tragedy of war, but today that message has been muddied. The poppy today is associated with themes of militarism and nationalism, and of glorifying the dead for having various motives that we project onto them (e.g. to protect the right to vote). If not wearing a poppy is disrespectful to their sacrifice, then what is the manipulation of their memory to support our personal agendas?
We all know that the poppy isn't supposed to glorify war, but when we talk about the noble sacrifices made by our ancestors for going to war (and in the case of WW1, not necessarily for the right reasons), it sure feels like we're glorifying war - glorifying the idea of making the ultimate sacrifice in the name of one's country.
If the goal is to remember the dead, and to empathise with the situation they were placed under, then in my opinion Remembrance Day is appropriate. The two minute silence is solemn and introspective. The poppy, on the other hand, is loud and is an outward statement of one's views. It would be nice if it could just be a simple mark of respect, but I don't think that it is.