Men, particularly, seem to be very concerned about the amount of plant estrogen, called isoflavones, that are contained in soy, and therefor use it as an excuse to say that a veg diet is not healthy (despite the fact that everything now has soy in it; don't believe me? Start checking some labels!). Look up "soy estrogen" on Google and the first result you will see is "Soy is making kids 'gay'", which really just blows my mind at how people can be so insane.
Now.. I'm not going to go into the underlying misogyny in the fear that soy is turning our little boys into little girls...
But I will say that this whole thing seems to be blown WAY out of proportion.
As far as what science says, the jury is still out. Nobody really knows what to think. Every report and study I came across seemed to contradict the last. Some say soy estrogen helps fight breast and prostate cancer, some say it weakens your bones (which is strange to me, considering how low the levels of osteoporosis are among vegetarians/vegans and the great quantities of soy most of us consume), some say it helps repair brain damage, it helps lower cholesterol, fermented soy is better for you than non-fermented, etc., etc.... However, this H.H.S. Healthcare Research and Quality Agency meta-analysis of all the studies done on soy estrogen has found that there is no substantial evidence to suggest soy estrogen has any ill effect on human health, but that more long-term research is needed.
I have to wonder, though, if soy caused all these health problems people say they do and turn men into little sissy girls and homosexuals, why have we seen no evidence of this from parts of the world we originally got the idea of eating soy from? Asians have been eating soy in one form or another (and often many) for literally thousands of years. In fact, it's been a staple food item in Southeast Asia for so long that they've come up with hundreds of different ways to prepare it, from tofu to tempeh to miso to some plain ol' edamame. Even today, most people in the area eat at least one meal containing soy every day of the week. It's also the part of the world where people tended to live longer, healthier lives, until modernization and western money started pouring into the region and now more of them are eating meat more frequently and (surprise) now suddenly having to deal with skyrocketing levels of cancer, diabetes and obesity.
If you really want to worry about how much estrogen is in your food, nothing is going to effect you more than the hormones that are fed to livestock. There's a reason why most of Europe has banned the use of growth hormones and will not allow US beef into the EU. Livestock growth hormones has been linked to earlier onset of puberty in girls (which makes them more susceptible to breast cancer) and the hormone residue in manure that washes off into surrounding waterways has been shown to effect the gender and reproductive capacity of fish. Growth hormones are most certainly not healthy for the animals they're given to either; dairy cows have a higher frequency of mastitis (basically booby-rot) and chickens have been known to suffer broken legs from the crushing weight of their own bodies.
Still, if you're concerned about soy, you don't have to eat it! There are plenty of soy-free foods out there. Try almond, rice or hemp milk as an alternative to soy milk. Seitan (aka: wheat meat) is tasty and easy to cook with and is made entirely of wheat protein. Either way, stop making excuses for why you can't give up meat.
For further reading, I suggest this article: What About Soy?