Stop bundling libva in the AppImage so VAAPI works on modern hosts

The AppImage bundles libva from the build environment (Ubuntu 22.04,
VA-API 1.20). VA-API driver modules (e.g. Mesa's radeonsi_drv_video.so)
are dlopen()ed by libva and export an entrypoint named after the libva
minor version they were built against (e.g. __vaDriverInit_1_23). An
older libva cannot load newer drivers, so on any host more recent than
the build container va_openDriver() fails:

    Trying .../radeonsi_drv_video.so
    libva error: has no function __vaDriverInit_1_0
    va_openDriver() returns -1

and Moonlight silently falls back to software decoding (reproduced on
Fedora 44 / Mesa 26.1 / RX 6500 XT; overriding the bundled libva with
LD_PRELOAD of the host libva immediately yields HW decode). Refreshing
the bundled libva at each release only postpones the breakage: the host
drivers always eventually outrun it (see #1398).

Exclude libva and its backends from linuxdeploy so Moonlight and the
bundled FFmpeg resolve them through the loader's normal search path and
pick up the host libva, which the distro's package manager keeps in step
with the host drivers. This mirrors how Mesa/GL/EGL are already
host-provided.

Hosts without a usable libva (e.g. bare NVIDIA systems) must still not
fail to launch, since libva.so.2 is a direct DT_NEEDED of Moonlight and
the bundled FFmpeg. Stage the build-environment libva in
opt/libva-fallback (outside every loader search path) together with
libva-probe, a tiny program whose only job is to carry the same
DT_NEEDED entries and VA_API_* versioned symbol references as the
binaries shipped in the AppImage. The generated AppRun runs the probe
first: if the dynamic loader can start it against the host, the host
libva is used (probe passing is, by construction, exactly Moonlight
being loadable); if not, it prepends the fallback directory and launches
with the bundled libva, matching pre-existing behavior. The build script
verifies that the probe references every version node required by the
shipped binaries, so the two cannot silently drift apart.

app/deploy/linux/libva-probe.c

@@ -0,0 +1,35 @@
+/*
+ * libva-probe: loadability oracle for the AppImage AppRun.
+ *
+ * Moonlight and the bundled FFmpeg import libva through symbols carrying
+ * VA_API_* ELF version nodes. Whether the libva installed on the host can
+ * satisfy those requirements cannot be inferred portably from file names or
+ * loader caches, so this program embeds the same DT_NEEDED entries and
+ * versioned symbol references as those binaries without ever calling into
+ * them: either the dynamic loader resolves everything and main() runs (the
+ * host libva is usable), or the exec itself fails and AppRun must fall back
+ * to the build-environment libva staged next to this binary.
+ *
+ * scripts/build-appimage.sh verifies at build time that this file references
+ * every VA_API_* node required by the binaries shipped in the AppImage, so
+ * it stays automatically in sync with the libva usage there.
+ */
+
+#include <stddef.h>
+
+#include <va/va.h>
+#include <va/va_x11.h>
+
+/* Address references are enough: they produce relocations against the
+ * versioned symbols, which is what makes the loader check the host's
+ * VA_API_* version nodes. The functions must never actually be called.
+ * volatile keeps the table (and its relocations) alive through -O2. */
+static void *volatile requirements[] = {
+    (void *)vaCreateSurfaces, /* libva.so.2, VA_API_0.33.0 */
+    (void *)vaGetDisplay,     /* libva-x11.so.2 */
+};
+
+int main(void)
+{
+    return requirements[0] == NULL;
+}

scripts/build-appimage.sh

@@ -62,11 +62,80 @@ popd
 export QML_SOURCES_PATHS=$SOURCE_ROOT/app/gui
 export QMAKE=qmake6
 
+# Stage the build-environment libva in opt/libva-fallback (outside usr/, so linuxdeploy
+# does not scan it, and outside every loader search path), together with a small probe
+# binary that carries the same libva ELF requirements as Moonlight. The AppRun below
+# uses the probe to decide, via the dynamic loader itself, whether the host libva can
+# satisfy Moonlight; when it cannot, the fallback copy is made visible. opt/ is a
+# standard linuxdeploy location for application data that must not be processed.
+LIBVA_FALLBACK_DIR=$DEPLOY_FOLDER/opt/libva-fallback
+SYSTEM_LIBVA=$(ldconfig -p 2>/dev/null | awk '/libva\.so\.2/{print $NF; exit}')
+mkdir -p $LIBVA_FALLBACK_DIR
+if [ -n "$SYSTEM_LIBVA" ]; then
+  cp -a "$(dirname "$SYSTEM_LIBVA")"/libva*.so* $LIBVA_FALLBACK_DIR/ || fail "Unable to stage libva fallback copy!"
+
+  echo Compiling libva-probe
+  cc -O2 -L"$(dirname "$SYSTEM_LIBVA")" -Wl,--no-as-needed -o $LIBVA_FALLBACK_DIR/libva-probe \
+    $SOURCE_ROOT/app/deploy/linux/libva-probe.c -lva -lva-x11 || fail "Unable to compile libva-probe!"
+
+  # Keep the probe honest: it must reference every VA_API_* version node that the
+  # binaries shipped in the AppImage require, otherwise a host libva could pass the
+  # probe and still fail to load Moonlight or the bundled FFmpeg.
+  va_nodes() { LC_ALL=C readelf -V "$1" 2>/dev/null | grep -oE 'VA_API_[0-9]+\.[0-9]+\.[0-9]+' | sort -u; }
+  NEEDED_NODES=$(for b in $DEPLOY_FOLDER/usr/bin/moonlight \
+                          /usr/local/lib*/libav*.so* /usr/local/lib*/libsw*.so* \
+                          /usr/lib/x86_64-linux-gnu/libav*.so* /usr/lib/x86_64-linux-gnu/libsw*.so*; do
+                   [ -f "$b" ] && va_nodes "$b"; done | sort -u)
+  PROBE_NODES=$(va_nodes $LIBVA_FALLBACK_DIR/libva-probe)
+  [ -z "$(comm -13 <(echo "$PROBE_NODES") <(echo "$NEEDED_NODES"))" ] || \
+    fail "libva-probe is missing version node(s): $(comm -13 <(echo "$PROBE_NODES") <(echo "$NEEDED_NODES")) - update app/deploy/linux/libva-probe.c!"
+fi
+
+APP_RUN=$BUILD_ROOT/AppRun-libva
+cat > $APP_RUN <<'APPRUN_EOF'
+#!/bin/bash
+# AppRun: prefer the host libva; use the staged copy only if the host cannot run us.
+#
+# VA-API driver modules on the host are loaded by libva and export an entrypoint
+# named after the libva version they were built against (__vaDriverInit_1_XX).
+# A libva can only load drivers that are not newer than itself, so bundling our
+# own (older) libva silently breaks hardware decoding on up-to-date distros.
+# Distros keep host libva and host drivers in step, so whenever the host libva
+# can satisfy Moonlight's own ELF version requirements it is the right choice.
+#
+# "Can satisfy" is answered here by libva-probe, a tiny program linked against
+# the same versioned libva symbols as Moonlight and the bundled FFmpeg: if the
+# dynamic loader can start it, the host libva works; if not (no libva installed,
+# or one too old to link), we make the staged build-environment copy visible
+# via LD_LIBRARY_PATH, matching the pre-existing bundled behavior.
+APPDIR="${APPDIR:-$(dirname "$(dirname "$(readlink -f "$0")")")}"
+LIBVA_FALLBACK="$APPDIR/opt/libva-fallback"
+
+if [ -d "$LIBVA_FALLBACK" ] && ! "$LIBVA_FALLBACK/libva-probe" 2>/dev/null; then
+    export LD_LIBRARY_PATH="$LIBVA_FALLBACK${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+fi
+
+exec "$APPDIR/usr/bin/moonlight" "$@"
+APPRUN_EOF
+chmod +x $APP_RUN
+
 echo Creating AppImage
 pushd $INSTALLER_FOLDER
+# Don't bundle libva: the bundled build-environment version (jammy: VA-API 1.20/1.22)
+# cannot dlopen GPU drivers compiled against newer libva on the host (which export
+# __vaDriverInit_1_23+), so va_openDriver() always fails and VAAPI falls back to
+# software decoding. The host always provides libva on systems where VA-API is
+# usable, so link against it at runtime instead (the AppRun shim above keeps a
+# bundled last-resort copy for hosts without libva).
 VERSION=$VERSION $LINUXDEPLOY --appdir $DEPLOY_FOLDER \
   --library=/usr/local/lib/libSDL3.so.0 \
-  --plugin qt --output appimage || fail "linuxdeploy failed!"
+  --plugin qt \
+  --custom-apprun $APP_RUN \
+  --exclude-library=libva.so* \
+  --exclude-library=libva-drm.so* \
+  --exclude-library=libva-wayland.so* \
+  --exclude-library=libva-x11.so* \
+  --output appimage || fail "linuxdeploy failed!"
 popd
 
 echo Build successful