To record in Game mode you have to start the game and then press F9; depending on where the overlay is you should see the Action overlay displaying your FPS and a blue circle (indicating not recording). Press F9 and the circle turns red.

I hope this helps?

Jedi