I believe you are more-or-less correct theoretically. As a practical matter, there are undoubtedly limits. From what I understand (this is not my area of expertise) it seems highly unlikely that light from the early years of the universe "survived". By that I mean to see back to (or really near) the beginning of the universe, you would need to observe a fair amount of light that was emitted by an object from that time, and then traveled undisturbed ever since. At some point early on, I imagine the universe was just too dense and there was too much going on for that to happen. In fact, according to Wikipedia, there is a so-called "surface of last scattering" when the universe was about a few hundred thousand years old.

Here's a link that might be of interest: map.gsfc.nasa.gov/m_uni/uni...test3.html

There are some issues with inflation in there, too. If the original inflation started out such that distances grew slowly enough, oddly enough, the light from that period would have already passed us by. It wouldn't be until later that objects would be distant enough for light to have to play catch up. This depends a lot on how inflation happened, which I don't know much about. To put that another way, if the universe hadn't "grown" but was just it's current size, we would not actually be seeing very far at all. From what I've read, current estimates suggest the universe is somewhere in the many tens of billions of light years across, much larger than the 12-14 billion years age estimate. If the universe had always been at its present size, we would only be able to see a fairly small fraction of it (assuming the current age estimates remained correct).