Yeah, they are really tough issues. Vegan ethics are not always clear cut.
The basic thing I learned from Ellie is that animals are not meant to be ridden upon (in the case of pony rides). They aren't born with saddles on their backs, after all. Even if an animal is well taken care of by its owner, it's still not how the animal would prefer to live. I think she also mentioned how bored those animals are going around in a circle all the time, and that it damages their gait to walk in a circle so much.
I'm not the expert on this type of issue, though, so ...
As for zoos, the animals are basically prisoners. None of them would choose to be there. Animals in captivity live fewer years than they would if left alone in their own habitats. I've read some stuff from Peta.com about animals in a zoo.
Here's a link to what they have to say about the issue: