The assumption that those who vote for third party candidates would all choose your pick over the other guy does not hold true in reality - demonstrated by exit polling after many elections in the past.
A vote for X is a vote for X, not Y....
I would say Johnson is closer on the issues to Romney than Obama and that right there would be the difference.