Athens 1996, obviously. Therefore, I'd say 2004 should've gone to Cape Town.
The 60s were perfect, Detroit anywhere there might have been interesting but never as defining as any of those cities. Actually, Detroit being the biggest Olympic loser ever is sort of an interesting omen.
Rio 1936 would've been incredible in terms of architecture with Niemeyer, Costa and Burle Marx emerging, so they would have been probably involved. Le Corbusier could've been invited.
And 2012 for booming Istanbul. Paris and NYC would've been great, too.
Havana 1920. Antwerp, WTF.