I, too, read the story as a child growing up in the West Indies, and was not made aware of its apparent racial connotation until I moved to the U.S. As others have said, this particular phrase (as do others), derives its meaning based solely on the context in which it's used.

__________________
I'll drink your poison if you fill the cup. You make me crazy, baby, don't let up.
Break Free, DMB 2006