So is there scope to make most sports open air maybe? It would slash costs and add a new experience for visitors. I think that for events like the olympics, people are there for sports tourism rather than tourism sport so they probably wouldn't mind a stripped back setting - they are there for the spectacle. Plus it would be a new and interesting way of hosting.
In my studies, I have been looking at something called the Dutch Approach. It is a proposal from the Netherlands aimed at showing how the legacy of a mega sporting event can be leveraged. They apply it to the context of bidding for the 2028 olympic games and put emphasis early planning. Preparations for it started in 2006! So yes, Rio may have only had 3 years, but if it was something they knew they wanted then they should've started as soon as - before even bidding. If you want to, you can look up the Dutch Approach article: The future of mega sport events: examining the "Dutch Approach" to legacy planning - Stefan Hartman and Tjeerd Zandberg.
Modular venues do sound like a great idea. This would especially help the argument that money should be spent on things like hospitals and schools rather than these big events.