I've had similar attitudes with my husband and son. It's usually by people who THINK they're well-meaning and/or being "considerate" of your not-white companions. And, because they believe they're "one of the good guys," you're "too sensitive" or a terrible person if you point out the ingrained bigotry of their suppositions.