Pretty much nailed it. All of my guns are not loaded. I treat each of them like they are loaded to avoid an AD/ND. I see no problem at all with handling a firearm while it is loaded, after all I am treating unloaded guns as if they were loaded so why would I treat my loaded guns any differently.
Of course that is exactly the point of the "treat all guns as if they are loaded" rule. If you always handle all guns with the same caution you will never have a mental lapse and have an "accident".