How/when are you calculating your FPS? It is highly possible that you are seeing these due to the increased graphics loads throughout the game.

Running the computer below, when playing TrackMania Nations, I normally see about 48fps, but occasionally it drops down to ~20fps because of the graphics at that point in the game.