Since I recently got a RTX4090, I ran a little comparison AV1 encoding in Handbrake between software (SVT) and NVENC.

The same system was used for both runs, Ryzen 9 5900X, 64GB RAM, RTX4090, latest nvidia drivers as of yesterday 2/9/24, latest Handbrake as of same date, Win10x64.

Input file was the same, my standard video that I use for testing: a 3h 1920x1080 25fps h.264 recording of a concert. Input file size 5.2GB.

Software SVT AV1 encoding using standard preset took 1h58min. File size was reduced to 3GB. Visual quality subjectively indistinguishable from the input file on a 4k monitor.

NVENC AV1 encoding using the exact same settings otherwise, took 9min 56s. File size increased to 10.1GB. Visual quality same.

Isn't the whole point of AV1 to preserve visual quality while reducing file size? The NVENC AV1 does not do that. Software encoder does, at the cost of 12x longer encoding time (on this particular system I use).

At this time, I don't see the point of someone using NVENC for AV1 encoding in its current state. Unless they were in a hurry and didn't care for file size, which in my opinion is an unlikely scenario.

Am I missing something here?

I have been disappointed with the compression achieved when encoding with H.265 and NVENC compared with straight software.

How does software versus NVENC encoding compare when using AV1?

I've been doing some playing around with the new Nvenc AV1 10bit encoder in Handbrake (as well as Rigaya's tool) and unfortunately regular NVEnc h265 still offers more detailed quality, especially under 1000kbit.. It seems like NVenc AV1 is over-smoothing everything.. anyone else experiencing the same thing? Maybe there's a certain point of bitrate where AV1 takes over? I was hoping for that up to 30% more gains bit for bit but i'm definitely not seeing it (using various side-by-side video comparison tools and the like), nor am I seeing motion handling any better with a static bitrate.

I've often heard that NVENC offers great speed but lower quality or larger file sizes, so I ran some benchmarks to see for myself. I compared file sizes, compression time, and quality using the FFMetrics tool on my RTX 4070 GPU and AMD 5800X CPU.

I used a 1.8GB H.264 720p video as the source and aimed to compress it to 550MB with a target VMAF score of 90 (a quality measure comparing the original to the compressed version).

Note: These results are based on a single test run, and outcomes may vary depending on your hardware and file types. My goal was fast compression with acceptable quality, prioritizing smaller file sizes.

Results:

1. CPU (H.265, fast, quality 25):

   • 592MB, VMAF: 87.98, took 20 minutes.

2. H.265 NVENC (slow, quality 32):

   • 556MB, VMAF: 87.66, 778 avg fps (97 seconds).

3. AV1 NVENC (medium, quality 38):

   • 558MB, VMAF: 89.62, 1051 avg fps (72 seconds).

AV1 NVENC was the clear winner with faster compression speeds, comparable file size, and better quality.

Conclusion:

If you have an Nvidia GPU, use the AV1 NVENC encoder for fast compression and good quality. I recommend setting quality to 38 with the medium speed preset in Handbrake for optimal results.

In Handbrake you can take the preset that is on Hardware section and just adjust the quality and dimensions you want.

I’m looking to convert my library to all AV1. I’ve been reading about how AV1 encoding is creating better quality and smaller file sizes than h.265. All my devices stream from Plex and I have a few AV1 files that playback fine so I’d like to capture the advantages of AV1 for my library. My goal is to retain quality while maximizing shrinking the file size. There is no speed constraints(and solar panels so free power) as I have a spare PC that will do nothing but run handbrake and these files will maybe only get played once every few years and largely sit in an archive on my Plex Server. My library is all h.264 and h.265.

1. My first question is on the video tab under the video encoder dropdown. I see 4 AV1 options. AV1 (SVT), AV1 10-bit (SVT), AV1 (NVEnc), and AV1 10-bit(NVEnc). What’s the difference between those 4 and which is generally the best for my objective of maximum file size shrinkage with no-minimal file quality loss?

2. On the same video tab under encoder options there is a slider named Encoder preset that goes from 13 to -1. It says this trades off compression efficiency vs compression speed. For my use case I believe I want maximum efficiency and minimal speed(please correct me if I’m wrong). It doesn’t say what the numbers represent. Is 13 max efficiency and min speed or is -1 max efficiency and min speed?

3. Same video tab under quality is there an ideal number for my objective? It suggests 20-23 for HD sources which my content is all 1080p or 4k. It says 0 would give me a higher file size(which I don’t want, but would going to 63 ruin the quality). Given speed is of no concern for me are there any suggestions on what to go for?

Thank you in advance for any guidance.