I agree.
Work on central and local Government and regulator panels, industry panels etc. is usually done for non-negotiable fixed rates, but those rates are, as Jamie mentions, usually higher than legal aid rates, although not as high as individually negotiated market rates.
Clerks do most of the fee negotiations but, as you become more senior, it's not unusual to have more involvement in the negotiating yourself. I am quite senior and I do most of my own clerking through direct contact with solicitors, and I mainly negotiate my own fees. It used to be thought ungentlemanly or unladylike to do this, but times have changed. Almost every other self employed professional negotiates his or her own fees, where these are not set by Government (NHS GP fees, for example), and there is no reason for barristers not to do so.
Barristers in Dublin don't have clerks or operate in chambers, and do all the business stuff themselves. Scots Advocates operate in groups called "stables", and some but not all have clerks, IIRC. Barrister business practices vary in Australia NZ, Canada etc.