Not something I've ever noticed. I don't think that happens on Linux, probably because of the software rendering.
Maybe that would help to explain the discrepancy, at least. Perhaps VLC is affecting DirectX or some other system state in such a way that the game runs faster than normal, but also makes it run a lot slower when the FPS display is enabled. Or... maybe the FPS counter is just wrong? It depends on Allergo calling fps_callback() at the right times, and I don't know how that works internally.