Teen Gothic

My (much) older colleague at work asked me that question a few days ago, causing an akward situation somehow. As we are far away from being friends I was really surprised, that's him, who was asking. Actually nobody ever asked me that (in the past most of the time I didn't wear black), so I was a little bit shocked what to say. "No" would have been a lie (obviously, as I wore an all black outfit with some pentagram jewelry at that day), but a simple "yes" felt wrong, as well. As I didn't know what to say else, finally I said yes and the scene ended as strange as it began, by him replying "Aha" and returning to work.

Does that sound familiar for anyone? Can anyone advise me how to handle such situations more elegant?

Edit: Thank you so much for your numerous and precious comments, to ALL OF YOU. You really helped me alot, to handle those situations more confident the next time.