We put people into racial categories for various reasons, some of which are quite sensible from an evolutionary perspective. In a tribal society that describes its members as human and outsiders as something less, you have to know whom to include in the human embrace, whom to trust, whom not to waste resources on. Where homicide between tribes is among the main causes of death, categorization is even more crucial to survival. One need only read the Bible to realize how much of history is concerned with the violent movement of tribes, be they Israelite, Egyptian, Assyrian, Babylonian, Persian, Median, or Greek.
Emotionally, people need to make sense of tribes that, whether recently or historically, inflicted death, rape, and devastation on the tribe they identify with. The easiest solution is to conclude that one’s own tribe is good and the other tribe is evil. An even more difficult problem that also leads to the derogation of other tribes is making sense of one’s own tribe’s violent behavior. What that happens, it is overwhelmingly tempting to justify it by trashing the worthiness or humanity of the victims. On a personal rather than societal level, my experience has been that the people who really despise me are those who have mistreated me, not those that I have mistreated—the latter as a rule accept my apology and move on.
In America, there are still places where tribalization is useful for survival, especially as a reaction to other people’s tribal conduct. A black man in Boston must learn that it is dangerous to venture into Charlestown, even in broad daylight, just as an Israelite from Ephraim had to learn not to get caught among Israelites from Gilead in Jephthah’s reign. Many prisons become tribal, and inmates find they must gang up if they are to survive. When tribal hatred was between Israelites, those from Gilead had to find a way to distinguish themselves from those from Ephraim, and they made the latter say the word, shibboleth, because they believed, apparently accurately, that the hated Ephraimites could not pronounce the “sh” sound. Skin color is an easier marker for categorization than a phoneme, and is often relied upon, but Hutus murdering Tutsis in Rwanda had, like the Israelites, to find more subtle distinctions to know whom to kill.
In most parts of America, tribalization is used for financial advantage rather than for killing, but in a country like ours (compared to Nazi Germany, say, or slavery-based America), you don’t lose your money and opportunity and feel grateful for just being alive, so even though survival is a relatively rare concern, people understandably fume over physical abuse, lack of safety, derogation, and lost money or opportunity. Economic disadvantages to being perceived as Protestant or Catholic in parts of Utah, to being perceived as white in certain sports and occupations, and to being perceived as black nearly everywhere breed less, but similar, avoidance and hatred than is bred by concern about survival. It’s only sensible, knowing that another group hates the category you are placed in, whether the hatred is based on the way they have treated us or on the way we have treated them, to teach one’s children that the other group is best met with suspicion and caution.
Race is in the eye of the beholder, but racism is not. I refuse to take responsibility for harms inflicted on people by members of my race or sex, because I don’t consider myself to be a member of my race or sex just because someone else puts me in those categories. Plus, as I’ve said, to the extent that I do at times identify as a man, I think it is part of being a man to feel guilty about other people’s suffering only if I caused it. But it would be stupid not to take notice of how one’s physical presence alarms some people and doesn’t alarm others, or not to take notice of the harms done by such alarms, or not to take notice of how much easier it is for me to avoid alarming people in the vast majority of American scenarios than for someone perceived as black. One of the few times it’s useful to think in categories is when you’re trying to decode what has happened to people because of the categories they have been placed in. But in so doing, we must be very careful not to get literal about the category or the individual’s membership in it.
Michael Karson, Ph.D. 
I have found that I can feel guilty about other peoples’ suffering even though I am not the one causing it, if I didn’t do anything to attempt to stop it. Think of being a bystander while someone is being bullied, or not speaking out when one’s government takes needless, aggressive action.
I don’t devalue such feelings; I just don’t share them. Some people who feel that kind of guilt (like you, I suspect) act on it, or try to; others, in my experience, react to that kind of guilt by condemning those who don’t feel it. If guilt motivates some people to do good, they sometimes try to motivate others to do good by inducing guilt. That’s been my experience in a lot of diversity conversations.