One simple thing that will change this in a big way, is to always have IoT devices act a a client instead of a server when being accessed remotely. They talk to a broker/server and then that server manages access control. Most of the latest IoT devices that I have seen adhere to this architectural model. Yes, things become a bit difficult for devices with multimedia, but that should also be possible to accomplish.
No mucking around, opening ports on your gateway/firewall.

Kedar.