honeybadger

cosmopolitan libc

your build-once run-anywhere c library

Compiling Actually Portable Executables on Windows

This tutorial explains how to compile simple C programs as αcτµαlly pδrταblε εxεcµταblεs on Windows, which you can then distribute to users on Linux, Mac OS X, FreeBSD, and OpenBSD too.

Background

On the Index Page we explained this wasn't possible, and that you have to use Linux to compile Windows programs. That wasn't entirely accurate. It's mostly just said to calibrate expectations, because GCC toolchains on non-Linux platforms (e.g. Cygwin, MinGW, Homebrew) lack the features we need to generate portable binaries.

One of the most important principles behind Cosmopolitan is that we should only need to support 1-2 compilers, rather than 𝑛³ toolchains. Since we reconfigured stock Linux GCC so that it generalizes to all platforms, we figured: why not run it on those other platforms too?

Introducing x86_64-linux-gnu for Windows NT

Here's a statically-linked build of GNU GCC 9.x and Binutils:

Tutorial

To get started, first download and extract both cross9.zip and cosmopolitan.zip

Second, create a simple program called hello.c which contains the following:

/* no #include lines needed! */
main() {
  printf("Hello World!\n");
}

You can then build hello.c as an Actually Portable Executable by running the Linux compiler you extracted earlier from cross9.zip.

# unix style compile commands (for mintty users)
bin/x86_64-pc-linux-gnu-gcc.exe -g -O -o hello.com.dbg hello.c                       \
  -static -fno-pie -no-pie -mno-red-zone -fno-omit-frame-pointer -nostdlib -nostdinc \
  -Wl,--gc-sections -Wl,-z,max-page-size=0x1000 -fuse-ld=bfd                         \
  -Wl,-T,ape.lds -include cosmopolitan.h crt.o ape.o cosmopolitan.a
bin/x86_64-pc-linux-gnu-objcopy.exe -SO binary hello.com.dbg hello.com
./hello.com

The toolchain doesn't need mintty or msys2. You can run it on Windows' stock vanilla command prompt too.

REM microsoft style compile commands (for cmd.exe users)
bin\x86_64-pc-linux-gnu-gcc.exe -g -O -o hello.com.dbg hello.c^
  -static -fno-pie -no-pie -mno-red-zone -fno-omit-frame-pointer -nostdlib -nostdinc^
  -Wl,--gc-sections -Wl,-z,max-page-size=0x1000 -fuse-ld=bfd^
  -Wl,-T,ape.lds -include cosmopolitan.h crt.o ape.o cosmopolitan.a
bin\x86_64-pc-linux-gnu-objcopy.exe -SO binary hello.com.dbg hello.com
hello.com

Here's the best part. The HELLO.COM binary you built on Windows, will actually run fine on Linux, Mac, FreeBSD, and OpenBSD too. So you can reach a broader audience from the comfort of your Windows PC without needing to fiddle around with things like virtual machines, Docker, or CI systems. What we've done instead is liberated the one true GCC toolchain from Linux so that it can run on Windows too.

Debugging

Screenshot of GDB debugging an APE binary on Windows 7

You may encounter the need to debug your .com binaries. Our Windows Debugging Tutorial explains how to do that. There's also a Documentation page which serves as good reference material.

Another useful command worth noting is you can view the disassembly of your binaries as follows:

bin/x86_64-pc-linux-gnu-objdump -xd hello.com      # view pe exe structure
bin/x86_64-pc-linux-gnu-objdump -d hello.com.dbg   # view disassembly
bin/x86_64-pc-linux-gnu-objdump -Sd hello.com.dbg  # view disassembly w/ source

Go Deeper

Cosmopolitan uses an LP64 data model consistently across operating systems. Agner Fog wrote an excellent doc called Calling Conventions where he describes the all pain that lack of consensus surrounding binary interfaces has caused in the past. Cosmpolitan grants us the luxury of only having to care about one ABI: the System V ABI.

What this means is that long with Cosmopolitan is actually long (i.e. 64-bits) unlike MSVC x64 which defines it as 32-bits. So if you use Cosmopolitan instead, you can safely assume a sane data model for all platforms without any #ifdef typedef toil.

While this is mostly an implementation detail, the Cosmopolitan C Library actually exports most of the surface area of the entire WIN32 API using thunks that turn fix the Microsoft x64 convention into System V. Normally if you want to open a file you'd just call open() and Cosmpolitan will do the right thing on Windows. However if you wanted to call CreateFile instead, it would delegate through the following generated function:

	.text.windows
CreateFile:
	push	%rbp
	mov	%rsp,%rbp
	.profilable
	mov	__imp_CreateFileW(%rip),%rax
	jmp	__sysv2nt8
	.endfn	CreateFile,globl

All that's needed to generate a thunk is to code the function arity in libc/nt/master.sh. Those thunks are very fast and allows us to declare normal looking prototypes. However, if you want to save a couple cycles by skipping the indirection, the GCC 9.x Linux toolchain above has got you covered. It supports an attribute that lets you declare a function pointer that plugs into Cosmopolitan's assembly linkage. For example:

extern __attribute__((__ms_abi__))
int64_t (*const __imp_CreateFileW)(
    const char16_t *lpFileName,
    uint32_t dwDesiredAccess,
    uint32_t dwShareMode,
    struct NtSecurityAttributes *opt_lpSecurityAttributes,
    int dwCreationDisposition,
    uint32_t dwFlagsAndAttributes,
    int64_t opt_hTemplateFile);

The variable above can be called as a normal function, and GCC will do the right thing in terms of making sure the first parameter is passed as RCX rather than RDI, etc. Cosmopolitan actually uses this technique for a couple files such as libc/runtime/winmain.greg.c where having the tiniest code size possible matters.

So in other words, calling WIN32 functions which use a completely foreign ABI is almost effortless when you have a really good compiler. There is however one important gotcha: callbacks. Here's how we handle cases where we need to pass a function pointer to WIN32 libraries, which need to call back into our application.

static int64_t WindowProc(int64_t hwnd, uint32_t uMsg, uint64_t wParam, int64_t lParam) {
  // ...
}
int main() {
  // ...
  windowClass.lpfnWndProc = NT2SYSV(WindowProc);
  // ...
}

The NT2SYSV macro uses inline assembly magic to generate a static trampoline for your function pointer expression, which indirects it through the __nt2sysv thunk.

#define NT2SYSV(FUNCTION) TRAMPOLINE(FUNCTION, __nt2sysv)
#define TRAMPOLINE(FUNCTION, THUNK)   \
  ({                                  \
    typeof(FUNCTION) *Tramp;          \
    asm(".section .text.trampoline\n" \
        "183:\n\t"                    \
        "mov\t%1,%%eax\n\t"           \
        "jmp\t" #THUNK "\n\t"         \
        ".previous\n\t"               \
        "mov\t$183b,%k0"              \
        : "=r"(Tramp)                 \
        : "i"(FUNCTION));             \
    Tramp;                            \
  })

The above assembly trick was learned by reading the Linux Kernel source code, and it's a great example of a GCC feature that's only possible to do when you're using the canonical Linux compiler.

It's also worth noting that, while System V -> WIN32 ABI function calls are very cheap, the reverse ABI translation used for callbacks (WIN32 ABI -> System V) is comparatively expensive, since we need to use pushf and popf which is slower.

What this means is thet Microsoft designed their binary interface so that the incentives are geared towards migrating away from it. The more code in your codebase you have that uses the System V ABI, the more your performance will improve. Cosmpolitan just makes it easy.

Since Cosmopolitan has a bias towards the System V ABI, we don't need popf as often than alternatives like Cygwin, which use it even for performance sensitive routines such as longjmp. Another strength of Cosmopolitan is that, unlike MinGW, we never link Microsoft's CRT due to concerns surrounding its performance and track record of secretly wrapping main() with telemetry.

Lastly, since these are portable binaries that bake in support for all major operating systems, you may be wondering, "what happens if I call the WIN32 API on Linux, Mac, or BSD"? The answer is you can, but it'll just return an error code. Cosmopolitan only polyfills UNIX system calls so that they work on WIN32, but not the other way around, since that task is better left to WINE.

Links

If you want to learn more about how Cosmpolitan supports Windows, check out these files: