Even a non-voter is affected by the government, just as representatives today are distributed by total population, not registered voters. Similarly, the English Parliament was held to represent the entire empire although only those in Britain could vote. You could properly argue that this was one of the critical issues leading up to the revolution (taxation without representation) but then again, slaves didn't pay taxes. It was definitely not the solution that would be accepted today, but was better than it could have been.
This, exactly.