Emby's ffmpeg only runs from one folder

While adding transcode dispatch to kube-arr, I swap each media server's transcoder for a small dispatcher. Real transcodes become Kubernetes pods; quick calls like ffmpeg -version still run the original binary, which the dispatcher keeps alongside itself as ffmpeg.real.

Plex and Jellyfin passed their end-to-end tests first time. Emby failed with this:

[kube-arr-transcoder] cannot run /app/emby/bin/ffmpeg.real locally: no such file or directory

The odd part: the step just before it had run test -x /app/emby/bin/ffmpeg.real, and that passed. The file was there and executable, yet execve said it didn't exist.

When "no such file" means a different file#

ENOENT from execve on a file that exists almost always means the kernel couldn't find something the binary asked for. For a dynamically linked ELF, the first thing it asks for is its interpreter: the dynamic loader named in the program header. file shows it:

$ file emby-server/bin/ffmpeg
emby-server/bin/ffmpeg: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter lib/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped

Or more directly:

$ readelf -l emby-server/bin/ffmpeg | grep interpreter
      [Requesting program interpreter: lib/ld-linux-x86-64.so.2]

Almost every binary on your system says /lib64/ld-linux-x86-64.so.2 there. Emby's says lib/ld-linux-x86-64.so.2, with no leading slash. The kernel resolves that relative to the process's current directory. Emby ships its own loader and libraries in lib/ so its ffmpeg doesn't depend on the host's glibc, and it only works if you start it from the install folder.

That's easy to reproduce with the Emby 4.10.1.0 package:

$ ./emby-server/bin/ffmpeg -version
bash: ./emby-server/bin/ffmpeg: cannot execute: required file not found
 
$ cd emby-server && LD_LIBRARY_PATH=lib:extra/lib ./bin/ffmpeg -version
ffmpeg version 5.1-emby_2023_06_25_p6 Copyright (c) 2000-2022 the FFmpeg developers and softworkz for Emby LLC

Newer shells word the error better ("required file not found"), but Go's syscall.Exec just hands you the raw ENOENT.

Why Emby itself never trips over it#

LinuxServer.io's Emby image starts the server from that folder:

exec \
    s6-notifyoncheck -d -n 300 -w 1000 -c "nc -z localhost 8096" \
        cd /app/emby env LD_LIBRARY_PATH="${APP_DIR}"/lib:"${APP_DIR}"/extra/lib s6-setuidgid abc /app/emby/system/EmbyServer \
        -programdata /config \
        -ffdetect /app/emby/bin/ffdetect \
        -ffmpeg /app/emby/bin/ffmpeg \
        ...

Every ffmpeg that Emby launches inherits /app/emby as its working directory and the library path in its environment. ffprobe and ffdetect are built the same way.

My test called ffmpeg through kubectl exec, which starts in /. The dispatcher was fine: it already passes its working directory and environment on to the transcode pod, so pods launched by Emby start in /app/emby too. The test was the thing that didn't behave like Emby. Running it from /app/emby with the same LD_LIBRARY_PATH fixed it, and Emby now runs a real transcode through a pod in CI as well.

What I took from it#

• If exec says a file doesn't exist and it clearly does, check the interpreter with file or readelf -l before anything else.
• A relative interpreter makes the working directory part of the binary's contract. Anything that wraps or relaunches such a binary has to preserve it.
• Test the way the real caller calls. My test ran ffmpeg the way I would; it needed to run it the way Emby does.