If nobody's watching TV there then there's no problem, even if someone is foolish enough to let an inspector in.
Depending on your tenancy contract and what room in which anybody is watching TV (if they actually are and you just aren't around to know), you could be liable for part of it (well, jointly liable for it all) should your flatmates be doing that.
Lastly, why are you paying rent year round if you only use the flat at weekends during the holidays? If you have that much cash to burn, just buy the TV license for the flat as a Christmas present and get peace of mind.