What endothermic and exothermic describe is the flow of heat transfer.
If something is exothermic, it means that heat is transfering from the inside to the outside.
That's why something will feel hot if you touch it, because all that heat is being applied to your receptors on your hand, or whatever instrument you use.
If something is endothermic, it means the heat is transfering from the outside to the inside.
That's why something will feel cold if you touch it, because it's drawing the heat away from your hand, or instrument.
The actual substance itself is going to be cold (exothermic) or hot (endothermic).
You're right that if a substance is absorbing energy, it should feel hotter, but the temperature measured is the transfer of heat energy from the medium we use to measure, whether it'd be our hand or a temperature.