Rubbish, there are shades of gray in life not just black and white. It is debatable whether fish fill pain. Most fish are caught from the ocean, not reared in cages. Fish are certainly less intelligent than farm animals. There is nothing contradictory about taking the view that catching fish is fine, whereas slaughtering a very intelligent animal like a pig is not.
There is a sliding scale of animal rights. Different people take different account of animal welfare and draw the line in a different place. Similarly, eating meat doesn't mean you have no problem with battery-farming and it doesn't mean you are fine with people who abuse animals.