The Guardian reports that within the context of the disaster caused by hurricane Katrina, the term "refugee" was condemned by civil rights groups: "I think it’s an offensive term," said Bruce Gordon, the president of the National Association for the Advancement of Colored People. "These people are fellow Americans. Using the word refugees makes it sound like they are not of us."
Refugee comes from Latin refugium, from refugere, to run away: re- + fugere, to flee. So in its original sense, the word refugee, which designates someone who has to flee, for whatever reason, has no negative connotations and couldn’t possibly be seen as offensive. I was quite surprised by his understanding of the word, and at first, I thought that he reacted in this way because refugees are normally seen in Third World countries rife with civil war and political unrest; I thought that what he really meant was: "We Americans can’t possibly be associated with the refugees we normally see on our TV screens fleeing from some poverty-stricken foreign land."
Then I had a look at what the OED says about the word, and one of the meanings it gives (along with "a displaced person", which is the meaning which applies best to the situation in the US at the moment, I think) is the following: "One who, owing to religious persecution or political trouble, seeks refuge in a foreign country". This must have been the meaning Bruce Gordon had in mind when he said that the word refugee seems to imply some kind of "otherness".