As an Atlanta resident since January 1996, I have to agree with Roux's comments regarding MARTA and the Peach Drop! Your best bet would be to scan the local free press (www.creativeloafing.com) and see if you can find a good hotel party or something along those lines to attend New Years Eve. The Peach Drop is just too boring to waste your whole night - trust Roux's advice on that!
Rent a car if at all possible - 90% of Atlanta is inaccessible via MARTA and what you can see will be wasted because you spent 2 hours on a bus or waiting for the bus in the friggin' cold..... rent the car, driving in this city is not THAT bad especially this week when so many are on vacation.
My advice for Olympics "history" - go to the permanant exhibit at the History Center and do a walking tour of Centennial Olympic Park. Aside from that, most of the venues are either not here any longer or have been so altered that no "legacy" remains from 1996.
If you're looking for some great neighborhoods, I'd suggest Midtown, Buckhead (for shopping), Little 5 Points, Virginia-Highlands, Decatur - these areas are where I spend 99% of my free time - believe me, the rest of Atlanta isn't worth seeing. Check out Creative Loafing though and see their "best of" listings for some hot spots to check out.
I hope you have a good time here in the ATL - there are some nice little gems around if you can get past the tourist traps!