I read a novel that addressed this once.
They used the other senses to describe the colours
eg,
green is the sound of tree's leaves, smells like freshly cut grass, tastes like cucumber, and feels like moss.
I started thinking about this.
Brown; tastes like chocolate, feels like the bark on a tree, is the sound of a sparrow's call, and smells like... what's brown and smells very definite?