Installation

Activate the nvenc USE flag to make use of Nvidia hardware encoder. You also need to activate the uvm USE flag on nvidia-drivers for NVENC to work.

Usage

Not really clear why, but even with nvenc hardware encoding and good hardware, some frames can be dropped by OBS Studio. It's important to check on console logs, it will tell you the % of frames dropped. Anything above 1% indicates potential serious problems (choppy/lagging video).
Recording a video from Firefox seems to be problematic in particular, with about 40% dropped frames even with nvenc. The same exact video (and source) in Chromium does not produce lagged frames.