Since I recently got a RTX4090, I ran a little comparison AV1 encoding in Handbrake between software (SVT) and NVENC.

The same system was used for both runs, Ryzen 9 5900X, 64GB RAM, RTX4090, latest nvidia drivers as of yesterday 2/9/24, latest Handbrake as of same date, Win10x64.

Input file was the same, my standard video that I use for testing: a 3h 1920x1080 25fps h.264 recording of a concert. Input file size 5.2GB.

Software SVT AV1 encoding using standard preset took 1h58min. File size was reduced to 3GB. Visual quality subjectively indistinguishable from the input file on a 4k monitor.

NVENC AV1 encoding using the exact same settings otherwise, took 9min 56s. File size increased to 10.1GB. Visual quality same.

Isn't the whole point of AV1 to preserve visual quality while reducing file size? The NVENC AV1 does not do that. Software encoder does, at the cost of 12x longer encoding time (on this particular system I use).

At this time, I don't see the point of someone using NVENC for AV1 encoding in its current state. Unless they were in a hurry and didn't care for file size, which in my opinion is an unlikely scenario.

Am I missing something here?

I've been doing some playing around with the new Nvenc AV1 10bit encoder in Handbrake (as well as Rigaya's tool) and unfortunately regular NVEnc h265 still offers more detailed quality, especially under 1000kbit.. It seems like NVenc AV1 is over-smoothing everything.. anyone else experiencing the same thing? Maybe there's a certain point of bitrate where AV1 takes over? I was hoping for that up to 30% more gains bit for bit but i'm definitely not seeing it (using various side-by-side video comparison tools and the like), nor am I seeing motion handling any better with a static bitrate.

I've often heard that NVENC offers great speed but lower quality or larger file sizes, so I ran some benchmarks to see for myself. I compared file sizes, compression time, and quality using the FFMetrics tool on my RTX 4070 GPU and AMD 5800X CPU.

I used a 1.8GB H.264 720p video as the source and aimed to compress it to 550MB with a target VMAF score of 90 (a quality measure comparing the original to the compressed version).

Note: These results are based on a single test run, and outcomes may vary depending on your hardware and file types. My goal was fast compression with acceptable quality, prioritizing smaller file sizes.

Results:

1. CPU (H.265, fast, quality 25):

   • 592MB, VMAF: 87.98, took 20 minutes.

2. H.265 NVENC (slow, quality 32):

   • 556MB, VMAF: 87.66, 778 avg fps (97 seconds).

3. AV1 NVENC (medium, quality 38):

   • 558MB, VMAF: 89.62, 1051 avg fps (72 seconds).

AV1 NVENC was the clear winner with faster compression speeds, comparable file size, and better quality.

Conclusion:

If you have an Nvidia GPU, use the AV1 NVENC encoder for fast compression and good quality. I recommend setting quality to 38 with the medium speed preset in Handbrake for optimal results.

In Handbrake you can take the preset that is on Hardware section and just adjust the quality and dimensions you want.

I have been disappointed with the compression achieved when encoding with H.265 and NVENC compared with straight software.

How does software versus NVENC encoding compare when using AV1?

Hi All

I have an AMD 7950x and Nvidia 4070 GPU and have a few questions. I'm using the nightly build of handbrake to encode Nvenc Av1 (10 bit)

• I believe the 4070 has dual Nvenc encoders, does handbrake use both automatically? Taskmanager says Nvenc is 100%, but i'm not sure it's using both.

• I'm transcoding from AVC to AV1 and Nvdec remains unused, while my 7950x is at 50% usage during encoding. Nvdec is ticked in options.

• It seems the 8th gen Nvenc is much better than previous versions and Nvenc AV1 can produce a similar result to SVT-AV1 preset 7/8 at the same bitrate (based on VMAF). Does that sound right?

• Handbrake only has only opions from FAST -> SLOWEST. Should it not be using P1-P6 and having options like b-frames etc?

Thanks!