I have 4yo twins, and we've done some simple trips with them-Hawaii with my parents and brother when they were 10 months old (having family help to lug the stuff helped enormously--packing for a week long trip with two babies sometimes feels like packing for a Peace Corps stint in Sierra Leone), lots of road trips and domestic flights, and only one international trip so far--a corny but ultimately surprisingly enjoyable trip to a Club Med resort with a daycare center.
We are fortunate that they're pretty good travelers--easy to amuse, slow to tears, and nowadays, easy to bribe/distract with iPad videos. And when they were babies, they were good sleepers.
It's fun, but it definitely changes what you do and how you do it. More playgrounds, parks, and markets, fewer museums or historical sites. Fewer hotels, more rental houses/apartments. Etc. And you don't get much of a break from childcare unless you bring help or go somewhere with childcare services (lots of resorts have this, which is why we did Club Med--we had never considered an all-inclusive before).
Talking to a lot of other parents, we arrived at the near consensus that international travel with five year olds is easier and more fun than with four year olds or younger. So we're spending six weeks in Italy next summer, then expatriating to Switzerland next winter.
But the trips when they were lap babies also had some simplicity (and savings--we are now buying four tickets everywhere). And if you're just strapping the kid on and walking somewhere, you still get pretty flexible options when they're still tiny.