For Smith, impartiality is the means by which we come to know and do what's right. Since we all wish to be praiseworthy, we'll at least try for impartiality and so come to do what deserves praise. Here's a key passage on this
The man who is conscious to himself that he has exactly observed those measures of conduct which experience informs him are generally agreeable, reflects with satisfaction on the propriety of his own behaviour. When he views it in the light in which the impartial spectator would view it, he thoroughly enters into all the motives which influenced it. He looks back upon every part of it with pleasure and approbation, and though mankind should never be acquainted with what he has done, he regards himself, not so much according to the light in which they actually regard him, as according to that in which they would regard him if they were better informed. He anticipates the applause and admiration which in this case would be bestowed upon him, and he applauds and admires himself by sympathy with sentiments, which do not indeed actually take place, but which the ignorance of the public alone hinders from taking place, which he knows are the natural and ordinary effects of such conduct, which his imagination strongly connects with it, and which he has acquired a habit of conceiving as something that naturally and in propriety ought to follow from it.
How does it go wrong? Suppose we come to believe we are exceptional especially in our moral judgment. We see ourselves as somehow superior to the rest, deserving of praise for that reason; the impartial spectator no longer summarizes what's "generally agreeable"; we make exceptions; we break rules and norms that we, mistakenly, believe are made for the rest but not for us. Smith's impartial spectator requires that we see ourselves as no more -- or less -- than the rest of humanity.