I used to be bigger than I am. I'm 6', used to weigh 225 (down to 185 right now). I was attacked once by a guy who weighed no more than 140 and was about a foot shorter than me.
Don't look at people's size and assume that just because someone is smaller, that he's reasonable and won't pick a fight with someone bigger.
I saw a tiny Guamanian kid try to pick a fight on my 6ft 3 200lb roommie at a bowling alley!? I saw a few of the kid's "friends" watching and suggested to the that they should talk their buddy from fighting my friend since he fought golden gloves in St. Louis and was one of Michael Spinks former sparring partners. The kid was either dumb or high or possibly both, I saved him from getting a beatdown.