Why are colleges so underrepresented with most minorities?

With the end of my junior year I have researched and visited many universities and I can't help to notice the fact that most schools, especially the 'best' ones, are almost all White, with Asians taking almost all of the population that wasn't White. With all the schools' claim that they see the importance of diversity and with Affirmative Action(which many complain about, but really isn't as relevant as advertised) I expected to see, maybe not a large population, but large enough for Hispanics and Blacks to really have a presence in these schools. Instead I found that they were virtually nonexistent. In my opinion this is a damn shame and I personally, as a black male, feel massively disappointed. Thanks to all answerers!!!