isn't that the point of the whole Libertarian movement? "Don't tread on me" and so forth? Perhaps if we abolish the Fed, EPA, FDA, TSA, Homeland Security, Dept of Education (just to name a few) we wouldn't need to be heavily taxed and the government could go back to protecting my individual rights instead of telling me how, what or why I should live my life. If I want to smoke pot, then I'll smoke pot. If I don't then I won't. But I don't need the government telling me that I have to pay them to allow me to ingest something. That's my business.
You can already grow your own tobacco and you can brew your own beer - what makes you think you wouldn't be allowed to grow your own weed?
Also, weed could be taxed a lot and still be a lot cheaper than it currently is.