By dressing in a special way, you announce that you belong to, and feel affinity to, a certain group. This gives companionship to other people in that group, but there is also the risk of rejection from people who do not belong to this group.
It is true that people can be nasty and condemn others, just because of the way they dress. This is not good. But you have a choice, - either dress in a neutral way in some situations, and thus avoid getting adverse reactions, or dress in the special way you like, and then be ready for these negative reactions from some people you meet.
If you want to be left alone, then dress neutrally. If you want to cause people to react, then dress provocatively. The choice is yours!