Downtown is surprisingly safe. I have worked in the downtown for over 20 years and have seen it do nothing but improve. Trouble begins when you start to head southeast. Get out of the historic district and head towards Hwy 41 and things can get dicey.
Interesting, there is not now, nor has there ever been, a Denny's downtown.
A 2nd on the Denny's comment. I can't think of where he could have been to think he was a Denny's and Downtown.