As others have said, it isn't utterly mad at all. You've been given examples of prominent barristers who changed careers 'late' (if you really think that there is such a thing; as I get older I tend to think that there isn't), but there are those who secure pupillage every year that have had previous careers. Obviously it's a minority of pupils because only a minority of applicants are career changers, but it's a perfectly realistic path to take for the right candidates. To give a more recent anecdote, one of the current pupils in my set is a career changer, and is in fact older than their pupil master.
The points that have been made about the relevant factors to take into consideration are good ones. It is obviously important not just to weigh up whether or not pupillage is a realistic aspiration for you, but whether it is the right thing for you personally. The frame of reference for that decision is very different for you in your late 30s compared to someone in their early 20s, and you will have both advantages and disadvantages over other younger candidates in that regard. You may well have the money to self fund, for example, and you equally may have a career to fall back on as well as significant experience that can give you an edge over other candidates, both on paper and in person. At the same time, family and other obligations may not only make the prospect of reduced earnings for a number of years a difficult one, but may also make you less flexible in terms of where you can accept pupillage. There is also always the reality that some sets, often inadvertently, will have scoring frameworks and approaches to recruitment that favour younger candidates.
Those are all things to weigh up. But in principle, a career change to the Bar at this stage isn't inherently unreasonable or even unrealistic, let alone mad.