Probably right. Most people don't pay attention anyway.
You would be surprised at how often people notice what's on a t-shirt. When away from work, I often wear t-shirts that represent various non-firearms interests, and I frequently get asked what the shirt is about. Just happened yesterday.
And I never wear anything firearms-related or political.
People DO notice how you dress - how you choose to play into that is up to you, but don't complain if you don't like the result.