Realistically, we're not going to lose the IRS. Changing their area(s) of influence would do wonders, though. (Someone still has to collect it, even if it's a flat tax.
My preference would be no direct federal taxes, period. Assess the states by some reasonable method and let them figure out how to raise it. No federal intrusion into individual lives. No federal mega-law enforcement agencies. If a state chooses to enforce a federal law, fine, if not, fine.