As a brown person with Indian ancestry, I wouldn’t necessarily mind being called “a coloured man”, “a person of colour” or simply “a black man”. Often it’s the context that matters. Or the tone used by the speaker. There are special derogatory words which are clearly meant to in insult, like wog. The worst thing that happened to me was being spoken to in a mock Indian accent.