Static compilations with musl standard library – great alternative to glibc.

I’ve been following a musl project for some time already. It looks like there’s a real chance for musl to take the lead one day.

 

comparision selected

 

For now, musl is the clear winner compared to known alternatives. Bloat comparison[2], dealing with resource exhaustion, security, and more. Musl is doing it right but isn’t yet popular enough, fingers crossed!

 

 

Building fully static binaries is another feature musl is good with and that’s the part I’ll explain below.

First of – the difference – executables:

  • Dynamic – binaries that require external libraries to work, smaller size.
  • Static – binaries with all libraries built-in no dependencies needed to run it, heavy.

 

Under normal circumstances there’s no reason to use static programs, however, in some cases, it’s the only way, for example:

  •  compromised system where shared libraries can’t be trusted;
  •  target system is too old;
  •  damaged system recovery;
  •  building a tool that will run on many different systems;

Static compilation can be done with glibc as well but often it’s pain to deal with, in some cases, it will keep failing and when it’s going smoothly – the resulting static executable is really big.


1. Building musl cross compiler.

Note: instructions below were executed on fresh Debian 10 minimal install with testing repositories.

Actually, before we start with a cross compiler, musl dynamic linker must be present, which is shipped with musl-gcc wrapper:

apt -y install wget gcc g++ make patch unzip
cd /usr/src ; wget https://musl.libc.org/releases/musl-1.2.0.tar.gz
tar -zxf musl-1.2.0.tar.gz ; cd musl-1.2.0
./configure ; make ; make install

 

now the dynamic linker is present at /lib/ld-musl-x86_64.so.1 so cross compiler can be built:

cd /usr/src
wget https://github.com/richfelker/musl-cross-make/archive/master.zip
unzip master.zip ; cd musl-cross*

 

It’s recommended to adjust make’s number of concurrent jobs (-j), as it’s going to take some time on older hardware. Optimal value = num_cores+1

time make -j7 TARGET=x86_64-linux-musl OUTPUT=/usr/local/musl-cross-compiler install

(optional step) activate musl ‘ldd’ tool via symlink to musl dynamic linker:

ln -s /lib/ld-musl-x86_64.so.1 /usr/bin/musl-ldd
/usr/bin/musl-ldd /usr/local/musl-cross-compiler/bin/x86_64-linux-musl-ld
        /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f6b9106b000)

 

 

[2]. Testing on sample code.

musl dynamic linker and cross compiler is now installed – let’s compile some sample code using glibc and musl to check the differences.

#include <stdio.h>
int main(void) {
    fprintf(stdout, "plonk\n");
    return 0;
}

Save the above to test.c file.

Compilation – gcc uses glibc, second is musl:

gcc -static -o glibc-test-static test.c
/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-gcc -static -o musl-test-static test.c

Verify it’s static:

echo $(ldd ./glibc-test-static ./musl-test-static)
./glibc-test-static: not a dynamic executable ./musl-test-static: not a dynamic executable

Compare sizes:

du -b glibc-test-static musl-test-static
768168  glibc-test-static
10112   musl-test-static	(98.6% smaller)

Removing symbols via strip to shrink it further returns even better results:

strip glibc-test-static musl-test-static
694824  glibc-test-static
5240    musl-test-static	(99.2% smaller)

Detailed view – comparison of ELF and program headers between glibc and musl generated executables:

glibc-c ELF Header:
 Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00
 Class:                             ELF64
 Data:                              2's complement, little endian
 Version:                           1 (current)
 OS/ABI:                            UNIX - GNU
 ABI Version:                       0
 Type:                              EXEC (Executable file)
 Machine:                           Advanced Micro Devices X86-64
 Version:                           0x1
 Entry point address:               0x401ac0
 Start of program headers:          64 (bytes into file)
 Start of section headers:          693096 (bytes into file)
 Flags:                             0x0
 Size of this header:               64 (bytes)
 Size of program headers:           56 (bytes)
 Number of program headers:         8
 Size of section headers:           64 (bytes)
 Number of section headers:         27
 Section header string table index: 26
musl-c ELF Header:
 Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
 Class:                             ELF64
 Data:                              2's complement, little endian
 Version:                           1 (current)
 OS/ABI:                            UNIX - System V
 ABI Version:                       0
 Type:                              EXEC (Executable file)
 Machine:                           Advanced Micro Devices X86-64
 Version:                           0x1
 Entry point address:               0x400152
 Start of program headers:          64 (bytes into file)
 Start of section headers:          4472 (bytes into file)
 Flags:                             0x0
 Size of this header:               64 (bytes)
 Size of program headers:           56 (bytes)
 Number of program headers:         4
 Size of section headers:           64 (bytes)
 Number of section headers:         12
 Section header string table index: 11
glibc-c:     file format elf64-x86-64
Program Header:
   LOAD off    0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**12
        filesz 0x0000000000000488 memsz 0x0000000000000488 flags r--
   LOAD off    0x0000000000001000 vaddr 0x0000000000401000 paddr 0x0000000000401000 align 2**12
        filesz 0x000000000007d601 memsz 0x000000000007d601 flags r-x
   LOAD off    0x000000000007f000 vaddr 0x000000000047f000 paddr 0x000000000047f000 align 2**12
        filesz 0x0000000000024c1c memsz 0x0000000000024c1c flags r--
   LOAD off    0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**12
        filesz 0x0000000000005110 memsz 0x00000000000068a0 flags rw-
   NOTE off    0x0000000000000200 vaddr 0x0000000000400200 paddr 0x0000000000400200 align 2**2
        filesz 0x0000000000000044 memsz 0x0000000000000044 flags r--
    TLS off    0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**3
        filesz 0x0000000000000020 memsz 0x0000000000000060 flags r--
  STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
        filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
  RELRO off    0x00000000000a4120 vaddr 0x00000000004a5120 paddr 0x00000000004a5120 align 2**0
        filesz 0x0000000000002ee0 memsz 0x0000000000002ee0 flags r--
musl-c:     file format elf64-x86-64
Program Header:
   LOAD off    0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**21
        filesz 0x0000000000000e1c memsz 0x0000000000000e1c flags r-x
   LOAD off    0x0000000000000fe0 vaddr 0x0000000000600fe0 paddr 0x0000000000600fe0 align 2**21
        filesz 0x0000000000000130 memsz 0x0000000000000838 flags rw-
  STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**4
        filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
  RELRO off    0x0000000000000fe0 vaddr 0x0000000000600fe0 paddr 0x0000000000600fe0 align 2**0
        filesz 0x0000000000000020 memsz 0x0000000000000020 flags r--

The LOAD segments define parts of the executable meant to be opened in the memory on runtime – long story short – more segments means more mmap() operations the kernel deals with (see ELF specification for details).

However, the difference in the size of the executables is plain crazy – musl needed 1% of what glibc allocated, no point to compare further.

 

3. Practical examples – static compilation with musl – using two popular projects.

Note: For Nginx I’ve used ‘configure’ script to define musl compiler and cflag options. Then with util-linux I’ve set musl details via environmental variables (CC, (CPP/CXX), CFLAGS). Generally I’m using environmental variables unless the software I build provided equivalent options to use.

A) Nginx webserver with all components

apt -y install perl-modules
groupadd nginx ; useradd -g nginx nginx ; passwd -l nginx
cd /usr/src
# download needed deps
wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz \
    https://www.openssl.org/source/openssl-1.1.1g.tar.gz \
    http://www.zlib.net/zlib-1.2.11.tar.gz \
    https://nginx.org/download/nginx-1.18.0.tar.gz

# unpack:
cat pcre-*.tar.gz zlib-*.tar.gz openssl-*.tar.gz nginx-*.tar.gz|tar -zxif -
cd nginx-1.18.0

NOTE: at the time of writing this there’s an issue with the OpenSSL part. Nginx build will fail while processing OpenSSL, the error is:
crypto/blake2/m_blake2s.c:53:1: error: missing initializer for field ‘md_ctrl’ of ‘EVP_MD’

It might be confusing since the error is real, OpenSSL could ignore missing initializers and successfully finish compiling but Nginx isn’t passing needed flag so it needs to be manually added. Simply pass ‘-Wno-missing-field-initializers‘ flag to Nginx via ‘–with-cc-opt‘ option.

./configure --prefix=/usr/local/nginx-1.18.0-static \
            --with-cc=/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-gcc \
            --with-ld-opt="-static" --user=nginx  --group=nginx \
            --with-cc-opt="-static -static-libgcc -Wno-missing-field-initializers" \
            --with-cpp=/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-g++ \
            --with-pcre=/usr/src/pcre-8.44  --with-zlib=/usr/src/zlib-1.2.11 \
            --with-openssl=/usr/src/openssl-1.1.1g --with-file-aio --with-mail \
            --with-poll_module --with-select_module --with-stream \
            --with-select_module --with-poll_module --with-http_ssl_module \
            --with-http_realip_module --with-http_sub_module \
            --with-http_addition_module --with-http_dav_module --with-http_flv_module \
            --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module \
            --with-http_auth_request_module --with-http_random_index_module --with-http_secure_link_module \
            --with-http_degradation_module --with-http_stub_status_module --with-http_v2_module
make -j7
make install

Done, Nginx is now ready to use.

 

B) Toolset – latest util-linux

cd /usr/src
wget https://mirrors.edge.kernel.org/pub/linux/utils/util-linux/v2.35/util-linux-2.35.tar.xz
tar xJf util-linux-2.35.tar.xz  ;cd util-linux-2.35
./configure CC="/usr/local/musl-cross-compiler/bin/x86_64-linux-musl-cc" \
        CFLAGS="-static --static" --prefix=/usr/local/util-linux-2.35-static
make
make install

 

That’s it. Both Nginx and util-linux tools are now fully static:

/usr/local/nginx-1.18.0-static/sbin/nginx: not a dynamic executable
/usr/local/util-linux-2.35-static/sbin/fdisk: not a dynamic executable

du -h /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk
12M     /usr/local/nginx-1.18.0-static/sbin/nginx
812K    /usr/local/util-linux-2.35-static/sbin/fdisk

strip /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk
du -h /usr/local/nginx-1.18.0-static/sbin/nginx /usr/local/util-linux-2.35-static/sbin/fdisk
3.9M    /usr/local/nginx-1.18.0-static/sbin/nginx
688K    /usr/local/util-linux-2.35-static/sbin/fdisk

 

Leave a Reply

Your email address will not be published. Required fields are marked *