I'd say it's a good idea to begin again if you have stopped. All of these home invasions, armed robberies and shootings seem to have the same skin color in common. If you live at 160th and whatever and slovenly black male in black glasses, hoodie and beanie cap rings your bell... You should probably assume bad intentions.
I think you can probably do that with most slovenly people wearing glasses, a hoodie, and beanie. But simply be aware. As long as he not committing a crime, he has as much right to be there as anyone else.