Here's the picture that in my mind best illustrates the opposition to unrestrained sympathy. Suppose we all felt great sympathy for those in New Orleans and Biloxi, but someone had another idea in mind for how we should spend our resources. They might spend time telling us -- and showing us -- how those folks didn't deserve our charity. They might say "charity begins at home" and criticize us for spending time and energy on folks who couldn't be civilized anyway, folks who perhaps couldn't adequately appreciate the subtleties of religion or art.
Unbelievable as it may sound today, this line of argument was used in the nineteenth century. Then, it was argued that it was a waste of time to send resources to Africans with whom people like Mrs. Jellyby in Dickens' Bleak House might sympathize. Mrs. Jellyby is the object of Dickens' criticism. In the picture above, the cover of Bleak House, she's got two African children in her care. The nearby sign -- Exeter Hall -- is the center of the Anti Slavery movement. (For more on Dickens and illustrators, see this.)