This workaround is helpful for anyone having trouble recording games through normal Game Mode. Using Screen Mode with fullscreen windowed playback sounds like a practical way to avoid popup issues and keep the capture smoother.
This is a common issue with screen recorders, especially when games run in exclusive fullscreen mode. Trying borderless windowed mode and checking overlay permissions is a good first step before reinstalling the software.