The purpose of calorimetry is to measure the amount of heat involved in a chemical reaction, generally the amount of heat given during the reaction. We measure it by measuring the temperature change of the water. But if heat energy is instead escaping to the environment, we won't be able to measure it (effectively out temperature reading will be lower than it should be), so our calculations won't be accurate.