I was an iPhone early adopter and then upgraded to the 3GS.
Those are incredible devices, and personally I consider their size to be one of their greatest assets. They are truly everywhere devices, you put em in your pocket and the work where ever, when ever you need them. Their limitations are that the web is not fully functional, the exchange support is less than ideal, you can't tether the device, you can't multitask (every app must be started from scratch every time you load it), the iPod app has some serious deficiencies in regards to how it creates/syncs playlists, the podcast support is poor, that it requires syncing with iTunes on a computer is annoying and there are several preloaded apps which are just clutter, and this list is far from conclusive. All of which is to say that there are a lot of little problems, and some great big ones as well... most of which are forgivable because it is an everywhere device. I don't remember to have to pack something extra, just grab my phone and I have damn near everything I need. The iPad has ALL of the deficiencies of the iPhone/Touch but "solves" only one problem, screen size, which in my mind means it solved a benefit and replaced it with very little else.
AppleTV is actually a pretty cool product, FWIW. That said, that it has been a failure in the market is not really disputable. The only reason why I am more hopeful for the iPad is that the physical hardware is portable, and therefor has higher "look at me" appeal. That said, the only way it differentiates itself from the Touch is that it requires you to bring a case with you if you are going anywhere.
I'm curious, why did you find it necessary to purchase both an iPhone and a Touch? Do you use them in different place and for different reasons? Owning both seems redundant to me.