Since I recently got a RTX4090, I ran a little comparison AV1 encoding in Handbrake between software (SVT) and NVENC.

The same system was used for both runs, Ryzen 9 5900X, 64GB RAM, RTX4090, latest nvidia drivers as of yesterday 2/9/24, latest Handbrake as of same date, Win10x64.

Input file was the same, my standard video that I use for testing: a 3h 1920x1080 25fps h.264 recording of a concert. Input file size 5.2GB.

Software SVT AV1 encoding using standard preset took 1h58min. File size was reduced to 3GB. Visual quality subjectively indistinguishable from the input file on a 4k monitor.

NVENC AV1 encoding using the exact same settings otherwise, took 9min 56s. File size increased to 10.1GB. Visual quality same.

Isn't the whole point of AV1 to preserve visual quality while reducing file size? The NVENC AV1 does not do that. Software encoder does, at the cost of 12x longer encoding time (on this particular system I use).

At this time, I don't see the point of someone using NVENC for AV1 encoding in its current state. Unless they were in a hurry and didn't care for file size, which in my opinion is an unlikely scenario.

Am I missing something here?

I have been disappointed with the compression achieved when encoding with H.265 and NVENC compared with straight software.

How does software versus NVENC encoding compare when using AV1?

I've been doing some playing around with the new Nvenc AV1 10bit encoder in Handbrake (as well as Rigaya's tool) and unfortunately regular NVEnc h265 still offers more detailed quality, especially under 1000kbit.. It seems like NVenc AV1 is over-smoothing everything.. anyone else experiencing the same thing? Maybe there's a certain point of bitrate where AV1 takes over? I was hoping for that up to 30% more gains bit for bit but i'm definitely not seeing it (using various side-by-side video comparison tools and the like), nor am I seeing motion handling any better with a static bitrate.

I've often heard that NVENC offers great speed but lower quality or larger file sizes, so I ran some benchmarks to see for myself. I compared file sizes, compression time, and quality using the FFMetrics tool on my RTX 4070 GPU and AMD 5800X CPU.

I used a 1.8GB H.264 720p video as the source and aimed to compress it to 550MB with a target VMAF score of 90 (a quality measure comparing the original to the compressed version).

Note: These results are based on a single test run, and outcomes may vary depending on your hardware and file types. My goal was fast compression with acceptable quality, prioritizing smaller file sizes.

Results:

1. CPU (H.265, fast, quality 25):

   • 592MB, VMAF: 87.98, took 20 minutes.

2. H.265 NVENC (slow, quality 32):

   • 556MB, VMAF: 87.66, 778 avg fps (97 seconds).

3. AV1 NVENC (medium, quality 38):

   • 558MB, VMAF: 89.62, 1051 avg fps (72 seconds).

AV1 NVENC was the clear winner with faster compression speeds, comparable file size, and better quality.

Conclusion:

If you have an Nvidia GPU, use the AV1 NVENC encoder for fast compression and good quality. I recommend setting quality to 38 with the medium speed preset in Handbrake for optimal results.

In Handbrake you can take the preset that is on Hardware section and just adjust the quality and dimensions you want.