Probably you are using "selected application" feature and you trying to record application you haven't selected.
Use "Games and Applications" mode to record games and DirectX applications.
Go to Active Screen mode and change the selected application or turn off this feature.
What is your display's resolution and what is the game's resolution?
Open display settings and set DPI to 100%, restart PC, run the game in display's native resolution and see if that helps.