Chris, it's never too late to take up a sport, especially for recreation.
Sports are for fun, so you can take part at any age. Jimber's right, it's better to join a group of players with similar ability, you'll enjoy yourself more and it'll build your confidence.