I think the abscence of French territories at the games has to do with the idea of French unity. Under the tricouleur, everyone is French. It's not like the British commonwealth where the Commonwealth itself is made of many parts, instead many parts make the whole.
New Caledonia and French Polynesia have a little more autonomy, and are allowed to compete in the South Pacific Games. Noumea, the capital of New Caledonia, is even set to host the next Games in 2011. However, if any athletes win qualifying events, they do not advance to the Olympics. Other departments have less autonomy and therefore don't even get to compete in regional games. Although, I believe some of these departments have FIFA soccer teams. Tahiti (French Polynesia) recently participated in the U-20 World Cup, qualifying by defeating New Caledonia in a regional tournament. If I'm no mistaken, Martinique has a national soccer team too.
Long story short, it's a complicated political issue in French society.