What would you consider to be the classiest U.S. city?

I've noticed that I have only been to one or two cities here in the United States that I would really consider to be classy and elegant. Many places in New York City would be wonderful if it weren't for all the tourists. For instance, I love Greenwich Village, Soho, and the Upper East/West Sides. Small college towns I also find very nice (Lexington, Virginia and Carlisle, Pennsylvania, for example). What do you think?