HandBrake and Video Transcoding: Your Media Library Deserves Better Compression

Your 40GB MKV File Is Not a Personality

Look, I get it. You downloaded the “best quality” encode, the 40GB Blu-ray remux with six audio tracks and subtitles for languages you’ve never heard of. Storage was cheap. The NAS had room. It felt responsible.

Now you’ve got a 20TB drive that’s 80% full, Jellyfin is struggling to transcode for your phone on-the-go, and you’ve started eyeing another hard drive. Stop. HandBrake exists. Let’s use it properly.

What Transcoding Actually Does (and When Not To)

Transcoding re-encodes your video from one codec to another. You’re trading some encode time for a smaller file. Done right, the quality difference is invisible to human eyes. Done wrong, you get blocky garbage.

When to transcode:

Converting 40GB remuxes to 8-10GB H.265 encodes you can’t tell apart at normal viewing distance
Making content compatible with devices that can’t decode certain codecs
Reducing file sizes before archiving or moving to slower storage

When NOT to transcode:

If your media server (Plex, Jellyfin) is already direct-playing the file fine
If you only have one copy and no backup (transcode FROM a backup, not the original)
For content you watch once and delete — not worth the CPU time
4K HDR content destined for an HDR display — re-encoding HDR is a rabbit hole, tread carefully

HandBrakeCLI: The Actually Useful Part

The GUI is fine for one-off jobs. For a library, you want the CLI:

# Debian/Ubuntu
sudo apt install handbrake-cli

# Or via Flatpak
flatpak install flathub fr.handbrake.ghb

Basic encode, H.265 with sane defaults:

HandBrakeCLI \
  -i "/media/Movies/SomeMovie.mkv" \
  -o "/media/Encoded/SomeMovie-x265.mkv" \
  --preset "H.265 MKV 1080p30" \
  --quality 22

The --quality flag sets Constant Rate Factor (CRF). Lower number = better quality, bigger file. For H.265:

CRF 18-20: Visually lossless, large files
CRF 22-24: Excellent quality, great compression (sweet spot)
CRF 28+: Smaller files, visible artifacts on fast motion

Codec Trade-offs: H.264 vs H.265 vs AV1

Let’s settle this quickly:

H.264 (x264)

Universally compatible — everything plays it
Older compression: larger files than H.265 at same quality
Fast to encode on CPU
Use when: you need max compatibility or serving to very old devices

H.265 (x265)

~40-50% smaller than H.264 at equivalent quality
Excellent hardware support now (anything made after 2016)
Slower CPU encode than H.264, but GPU encoding handles it fine
Use when: this is your main codec for everything

AV1

~30% smaller than H.265 at equivalent quality
Open, royalty-free (Netflix, YouTube use it)
CPU encoding is SLOW — we’re talking 10x slower than H.265
GPU encoding (NVENC AV1 on RTX 4000+, AV1 on Intel Arc) makes it viable
Playback support is good on modern hardware but verify your devices
Use when: you have a capable GPU and want maximum compression

My recommendation for a home library: H.265 CRF 22. You’ll store more stuff, everything plays it, and you can do it on any hardware. Switch to AV1 when your GPU supports it and you want to squeeze more.

GPU Encoding: Stop Wasting Your GPU

CPU encoding is fine. GPU encoding is faster. Much faster. Here’s how to enable it:

NVIDIA (NVENC)

# H.265 via NVENC
HandBrakeCLI \
  -i input.mkv \
  -o output.mkv \
  --encoder nvenc_h265 \
  --quality 28 \
  --encopts "spatial-aq=1:temporal-aq=1:rc-lookahead=32"

NVENC uses a fixed-quality mode slightly differently — quality 28 with NVENC is roughly equivalent to CRF 22 in x265. Adjust to taste.

Intel QuickSync

HandBrakeCLI \
  -i input.mkv \
  -o output.mkv \
  --encoder qsv_h265 \
  --quality 25

QuickSync is excellent on 11th gen+ Intel. If you’re running a Proxmox NAS with a modern Intel chip, you’re leaving performance on the table if you’re not using this.

AMD VCE

HandBrakeCLI \
  -i input.mkv \
  -o output.mkv \
  --encoder vce_h265 \
  --quality 22

Note: GPU encoding generally produces slightly larger files than CPU at the same “quality” setting because the hardware encoder is optimizing for speed, not compression efficiency. The difference is typically 10-20%. For a library of movies, that tradeoff is almost always worth it.

Batch Transcoding Script

Here’s a script that processes a directory, skips already-encoded files, and logs what it’s doing:

#!/bin/bash

INPUT_DIR="/media/Movies"
OUTPUT_DIR="/media/Encoded"
LOG_FILE="/var/log/transcode.log"
QUALITY=22
ENCODER="nvenc_h265"  # Change to x265 for CPU

mkdir -p "$OUTPUT_DIR"

find "$INPUT_DIR" -type f \( -name "*.mkv" -o -name "*.mp4" -o -name "*.avi" \) | while read -r input_file; do
    filename=$(basename "$input_file")
    output_file="$OUTPUT_DIR/${filename%.*}-encoded.mkv"

    # Skip if already encoded
    if [[ -f "$output_file" ]]; then
        echo "[SKIP] $filename already encoded" | tee -a "$LOG_FILE"
        continue
    fi

    echo "[START] $(date): $filename" | tee -a "$LOG_FILE"

    HandBrakeCLI \
        -i "$input_file" \
        -o "$output_file" \
        --encoder "$ENCODER" \
        --quality "$QUALITY" \
        --audio "1,2" \
        --aencoder copy \
        --subtitle "scan" \
        --subtitle-forced \
        2>> "$LOG_FILE"

    if [[ $? -eq 0 ]]; then
        echo "[DONE] $(date): $filename" | tee -a "$LOG_FILE"
    else
        echo "[FAIL] $(date): $filename" | tee -a "$LOG_FILE"
        rm -f "$output_file"  # Don't leave partial files
    fi
done

Run this in a tmux session overnight. Come back to a smaller library.

Audio and Subtitle Handling

Don’t re-encode audio unless you have to. Copy it:

--aencoder copy      # Copy all audio tracks as-is
--audio "1,2"        # Only keep first two audio tracks (ditch the commentary tracks nobody watches)

For subtitles, the scan mode is smart — it keeps forced subtitles (the ones that show when characters speak a foreign language) without burning in full subtitles:

--subtitle "scan" \
--subtitle-forced

Want specific subtitle tracks? List them:

--subtitle "1,3" \
--srt-file "/path/to/extra.srt"

HandBrake in Docker

Running on a server without a desktop? Docker keeps it clean:

# Dockerfile
FROM jlesage/handbrake:latest

Or just run the official image for CLI use:

docker run --rm \
  -v /media:/media \
  --gpus all \  # For NVENC, needs nvidia-container-toolkit
  jlesage/handbrake:latest \
  HandBrakeCLI -i /media/input.mkv -o /media/output.mkv --encoder nvenc_h265 --quality 28

For a persistent watch folder setup (auto-transcode anything dropped in a folder), jlesage/handbrake has that built in via the AUTOMATED_CONVERSION environment variables. Genuinely useful.

Jellyfin/Plex Integration

After encoding, update your media library path. The big win: if you’ve encoded everything to H.265 in a container your clients support, Jellyfin/Plex can direct play instead of transcoding on the fly. Direct play means:

Zero server CPU usage for playback
No quality loss from double-transcoding
Instant seek, no buffering lag

Check Jellyfin’s playback logs to confirm direct play is happening. If you see “Transcode” in the logs, either the codec isn’t supported by the client or the container format is wrong — MKV and MP4 both work broadly; MKV is more flexible for multiple tracks.

The rule: transcode your library once, good, offline. Never let your server transcode live if you can avoid it. Your server (and your electricity bill) will thank you.