Hacking dlopen to statically link Pipewire

Note: this blog post is cross-posted from Mason’s personal blog

I recently helped Tuple solve an interesting problem. They’re working on a Linux version of their pair programming tool, and they want to make it a fully static executable so that they can ship the same build everywhere.

Alright no big deal, just link any dependencies statically and use musl. Easy, right?

Unfortunately, it’s not quite that easy. Tuple needs to talk to PipeWire for screen capture, and PipeWire makes heavy use of dlopen, so statically linking it isn’t an option…or is it?

I’ve open sourced a build.zig configuration that generates a static build of PipeWire here. The rest of this post elaborates on why this is challenging, why solving it is possible, and how my solution works.

Quick Disclaimer: the build works, and you can use it today! However, depending on your use case, you may need to add more plugins to the library table. See the README for more info.

The Challenge

To talk to your system’s PipeWire daemon, you’re expected to speak PipeWire’s protocol. The most straightforward way to do this is to link with their shared library which implements the protocol.

We don’t want to do this, because that introduces a dependency on the system’s dynamic linker path, and the dynamic linker path can vary across distros.

There’s nothing fundamental about the PipeWire client library’s goals that require dynamic linking, but unfortunately for us, it was implemented assuming a dynamic linker would always be present. In particular, its implementation includes two separate plugin systems that implement much of the core functionality:

• PipeWire Modules
• Spa Plugins

Both of these plugin systems use dlopen to dynamically load shared libraries at runtime. All of the plugins are provided by PipeWire itself, so they don’t need to be in separate shared libraries, but they are, and the implementation relies on this in various ways and there’s quite a bit of code here so we don’t want to try to rework it manually.

Running the video-play example requires loading two Spa Plugins, and six PipeWire Modules. The audio-src example contributed by jmstevers (thanks!) is similar. In practice these are a small fraction of the plugins and modules available, and as you use more of PipeWire’s functionality you’ll need more of the plugins.

This leaves us in a difficult situation. We could sidestep all of this by writing our own PipeWire client—and we may do that eventually—but in the short term, is there any way to leverage the existing code to get a quicker solution in the hands of users?

Well, you probably guessed from the title, there is.

How I Built PipeWire Statically

Build System

Let’s raise the stakes a little higher. Not only do we want to build PipeWire statically. We want to build PipeWire statically without editing the upstream code. If we can pull this off, it’ll be much easier to maintain, since we won’t have to maintain a fork—just some configuration.

To do this, we’re going to use Zig’s build system. Zig’s build system is well suited to building C code. Our build will produce both a Zig module, and a .a paired with some headers in case you want to use the result from a non-Zig based application.

Since we aren’t modifying the upstream source, we won’t copy it into our source tree. We’ll just fetch it on first build by adding it to our build.zig.zon:

.dependencies = .{
    .upstream = .{
        .url = "git+https://gitlab.freedesktop.org/pipewire/pipewire.git#1.5.81",
        .hash = "N-V-__8AAKYw2AD301ZQsWszbYSWZQF5y-q4WXJif0UGRvFh",
    },
}

This has the added benefits of making our build easy to audit, and making it easy to upstream any patches we make to Pipewire without having to first remove our build config.

High Level Approach

At the end of the day, PipeWire can only do anything dynamic by talking to the system it’s running on through a well defined API. We don’t need to change the internal guts of PipeWire, we just need to give it an alternate implementation of that API that never calls into the dynamic linker.

In particular, we need to replace the following functions provided by dlfcn.h:

• dlopen
• dlclose
• dlsym
• dlerror
• dlinfo

If we can reimplement these functions in a way that doesn’t require a dynamic linker, then nothing should be able to stop us from building PipeWire statically!

Building A Fake Dynamic Linker

No more of this!

Here’s our strategy. For example’s sake, assume we have a C library that tries to load “mylib.so” and “otherlib.so” at runtime. Once it loads them, it’ll try to access the symbols “foo”, “bar” and “baz.”

First, let’s define our own fake dynamic library type that maps symbol names to opaque pointers:

pub const Lib = struct {
    symbols: std.StaticStringMap(?*anyopaque),
};

Next, let’s create a table of available libraries and fill it with our data:

pub const libs: std.StaticStringMap(Lib) = .initComptime(.{
    .{
        "mylib.so",
        Lib{
            .symbols = .initComptime(.{
                .{ "foo", @ptrCast(@constCast(&foo)) },
                .{ "bar", @ptrCast(@constCast(&bar)) },
            }),
        },
    },
    .{
        "otherlib.so",
        Lib{
            .symbols = .initComptime(.{
                .{ "baz", @ptrCast(@constCast(&baz)) },
            }),
        },
    },
};

Next let’s implement dlopen. dlopen takes a path, and returns either a pointer sized opaque handle to the library at that path, or null if it doesn’t exist.

There are some other details we’re ignoring—we’ll come back and make our implementation more conformant once the basics work.

Since our opaque handle could be anything, let’s make it a pointer to our library object:

fn dlopen(path: [*:0]const u8, mode: std.c.RTLD) ?*anyopaque {
    const span = std.mem.span(path);
    const lib = if (libs.getIndex(span)) |index|
        &libs.kvs.values[index] else null;
    return @ptrCast(@constCast(lib));
}

Now that dlopen does something potentially useful, let’s try to implement dlsym. dlsym is supposed to take a library handle and symbol name, and return a pointer to the given symbol. Again, we’ll do a pass later to make this more conformant, but here’s the basic version:

fn dlsym(
    noalias handle: ?*anyopaque,
    noalias name: [*:0]u8,
) ?*anyopaque {
    const lib: *const Lib = @ptrCast(@alignCast(handle.?));
    const span = std.mem.span(name);
    return lib.symbols.get(span);
}

dlclose is allowed to do nothing, and in our case, there’s nothing to do. Easy. We’re going to ignore dlerror and dlinfo for now.

This should be good enough for a first pass, but how do we get the C library to actually call our methods instead of those provided by libc?

First, let’s rename our methods, specify a calling convention, and export them:

pub export fn __wrap_dlopen(
    path: [*:0]const u8,
    mode: std.c.RTLD,
) callconv(.c) ?*anyopaque {
    // ...
}

pub export fn __wrap_dlsym(
    noalias handle: ?*anyopaque,
    noalias name: [*:0]u8,
) callconv(.c) ?*anyopaque {
    // ...
}

// ... others follow ...

Next, we’re going to use every C programmer’s favorite trick: the preprocessor. We just redfine every reference to these methods to point to our wrappers.

const flags: []const []const u8 = &.{
    "-Ddlopen=__wrap_dlopen",
    "-Ddlclose=__wrap_dlclose",
    "-Ddlsym=__wrap_dlsym",
    "-Ddlerror=__wrap_dlerror",
    "-Ddlinfo=__wrap_dlinfo",
};

And that’s it, now our C library will call into our fake dlopen API instead of the real one!

In the future, this logic may be moved to a linker script.

Making Our Fake Linker More Conformant

This all might feel like a bit of a hack, but remember, there’s no way for Pipewire to know whether or not our linker is “real”. As long as we follow the spec for the methods we’re replacing, we’re good.

However, we’ve ignored some things in the spec up until this point. Let’s fix that up now. If you don’t care about the details of dlopen/dlsym, skip this section, the main ideas don’t change.

First off, you’re allowed to pass a null path to dlopen. When you do this, you’re supposed to get a handle to the main program. We’ll handle this case by creating a constant Lib.main_program_name and then adding a new library of that name to our symbol table. Then, in dlopen, we return that library when the path is null. We’ll also add some debug logs for good measure.

We’ll continue to ignore the mode flags, none of these matter to us.

pub export fn __wrap_dlopen(
    path: ?[*:0]const u8,
    mode: std.c.RTLD,
) callconv(.c) ?*anyopaque {
    const span = if (path) |p| std.mem.span(p) else Lib.main_program_name;
    const lib = if (libs.getIndex(span)) |index|
        &libs.kvs.values[index]
    else
        null;
    log.debug("dlopen(\"{f}\", {f}) -> {?f}", .{
        std.zig.fmtString(span),
        fmtFlags(mode),
        lib,
    });
    return @ptrCast(@constCast(lib));
}

Next, dlsym is allowed to accept two special handles:

• RTLD_DEFAULT
• RTLD_NEXT
• RTLD_SELF

Pipewire doesn’t use RTLD_DEFAULT/RTLD_SELF so we’re just going to panic if they’re passed in, but it does make use of RTLD_NEXT. You can read the spec for what this feature does here, we’re going to just add another library to our table for it.

Lastly, null is a valid value for a symbol, so we need to also set an error flag when we fail to find a symbol so that it’s possible to disambiguate these cases.

Here’s our improved dlsym:

pub export fn __wrap_dlsym(
    noalias handle: ?*anyopaque,
    noalias name: [*:0]u8,
) callconv(.c) ?*anyopaque {
    const lib: *const Lib = if (handle == c.RTLD_DEFAULT)
        @panic("unimplemented")
    else if (@hasDecl(c, "RTLD_SELF") and handle == c.RTLD_SELF)
        @panic("unimplemented")
    else if (handle == c.RTLD_NEXT)
        &libs.get(Lib.rtld_next_name).?
    else
        @ptrCast(@alignCast(handle.?));
    const span = std.mem.span(name);
    var msg: ?[:0]const u8 = null;
    const symbol = lib.symbols.get(span) orelse b: {
        msg = "symbol not found";
        break :b null;
    };
    log.debug("dlsym({f}, \"{f}\") -> 0x{x} ({s})", .{
        lib,
        std.zig.fmtString(span),
        @intFromPtr(symbol),
        if (msg) |m| m else "success",
    });
    if (msg) |m| err = m;
    return symbol;
}

I also wired up dlinfo to panic since PipeWire doesn’t use this feature. You can see the full implementation at the time of writing here.

Wiring Up Our Fake Linker To PipeWire

Now that we have a fake dynamic linker, we just need to wire it up to PipeWire by adding the real PipeWire symbols to our table!

Unfortunately, we immediately hit an issue. Every Spa Plugin defines spa_handle_factory_enum and spa_log_topic_enum, and every PipeWire module defines pipewire__module_init. This is gonna result in a lot of duplicate symbol errors if we try to statically link all of these implementations.

No worries, we’ll just use the preprocessor again, this time to namespace these symbols. For example, libpipewire-module-protocol-native.so’s pipewire__module_init becomes pipewire_module_protocol_native__pipewire__module_init.

If we do this, we’ll find that PipeWire is still failing to load its modules and plugins. What gives?

Well, it turns out that PipeWire is checking for the existance of the shared libraries with stat before calling dlopen. I’m not sure why it’s doing this, this check seems redundant, but we can easily work around it by wrapping stat the same way we wrapped dlopen.

Our wrapper will look for the given path in our symbol table. If it finds it, it will write to statbuf indicating that the file exists. Otherwise it will forward the call to the real stat:

pub export fn __wrap_stat(
    noalias pathname_c: [*:0]const u8,
    noalias statbuf: *std.c.Stat,
) callconv(.c) c_int {
    const pathname = std.mem.span(pathname_c);
    const result, const strategy = b: {
        if (dlfcn.libs.get(pathname) != null) {
            statbuf.* = std.mem.zeroInit(std.c.Stat, .{
                .mode = std.c.S.IFREG,
            });
            break :b .{ 0, "faked" };
        } else {
            break :b .{ std.c.stat(pathname_c, statbuf), "real" };
        }
    };
    log.debug("stat(\"{f}\", {*}) -> {} (statbuf.* == {f}) ({s})", .{
        std.zig.fmtString(pathname),
        statbuf,
        result,
        fmtFlags(statbuf.*),
        strategy,
    });
    return result;
}

While we’re here, it’s not strictly necessary, but we can also wrap fstat mmap and mmunmap to trick PipeWire into reading the client config from an embedded file instead of searching around on the filesystem for it.

And that’s it! We now have a working fully statically linked build of Pipewire.

> ldd video-play
not a dynamic executable

Closing Thoughts

The power of interfaces is that they can be replaced.

According to cloc PipeWire is 500k+ lines of code. I’m confident none of those lines were written with our use case in mind. However, we were still able to satisfy our requirements, because at the end of the day PipeWire—like almost all other software—has to eventually call into a well defined API.

Don’t underestimate the power of interfaces!