My family came over on the Mayflower. To my knowledge, none of them even traveled south of the Mason Dixon line, so I know they never had slaves.
Somehow I doubt that factoid will help much if someone asks me to "check my privilege."
I have family that did come from SC & NC, but they were Quakers.