Go without the Operating System

12 July 2016

Without the Operating System?

Conventional deployment

There's some physical hardware in a data-center (or closet).

We want to run multiple user applications on that hardware.

The operating system allows our applications to safely interface with and share the hardware.

Cloud Deployment

Works like conventional deployment, except multi-tenant.

A hypervisor allows multiple operating systems to safely interface with and share hardware.

The operating system duplicates work, allowing user applications to share hardware that's already shared (or even virtualized).

Operating systems do a lot

Operating systems do a lot more than manage hardware.

They provide a general purpose computing environment. Shell scripting, Cron jobs, terminal sessions, POSIX.

They are built on established, understood tools and have years (or decades) of development and production time.

Operating systems are large projects

Typical server deployments have can 500+ packages built from 100,000,000+ lines of source code.

The Linux Kernel alone is 15,000,000+ lines of code.

These strongly impact:

• Boot time
• Footprint
• Performance
• Security (see Heartbleed, Shellshock, ImageTragick)

Without the Operating System

We can remove the operating system if we run user applications that interface directly with the hypervisor.

These hypervisor-aware applications are called unikernels.

Where operating systems are general purpose, unikernels are highly specialized.

They take a number of forms. Some are for OS research. Others target particular language runtimes, like Haskell and Erlang. And, of course, Go!

AtmanOS

AtmanOS

It allows you to cross-compile Go programs, just like for Linux and Windows, to run directly on the Xen hypervisor.

The images can be deployed on Amazon EC2, Linode, Rackspace, and IBM SoftLayer.

There are a few components:

• a microkernel implemented in Go (and a little assembly)
• a suite of packages for communicating with Xen, reading and writing network packets, etc.
• a port of Go's runtime and standard library to the microkernel and system packages

It's currently around 3,000 lines of Go and 150 lines of Assembly.

Microkernel

Provides memory management, lightweight threads and scheduler, event handlers, and timekeeping.

Implemented within the runtime package. Normal (if low-level) Go code, with some caveats:

• stub methods implemented in Assembly
• unsafe for a number of components
• uses //go:nosplit and //gc:nowritebarrier annotations
• cannot allocate memory on heap

System packages

Provides user-space access to microkernel and Xen:

• atman/mm - allocate physical pages and map machine frames
• atman/xen - interface to core Xen features

Also peripherals needed to implement the standard library.

• atman/xenstore - implementation of XenStore protocol for configuration and device drivers
• atman/net - network driver for Xen
• atman/net/ip - TCP/IP implementation

Porting Go

Implementation of Go's platform-agnostic functions and types.

In the runtime package, that's adapter functions for memory management, time, processes, and locks.

For the standard library, it's syscall functions, network types, etc.

Demo: low-level networking

Accessing the network device

atamn/net implements Xen's network device interface, leveraging other AtmanOS packages like atman/mm and atman/xen.

Enables user-space programs to talk directly to the network device, for example, to read ethernet packets directly from the device.

github.com/atmanos/example/net is a program which does exactly that.

Building the example program

AtmanOS provides an atman command which wraps go with the appropriate flags for building our program:

> atman build github.com/atmanos/example/net

The build artifact is an ELF executable, with a few special notes so Xen knows how to load it.

> file net
net: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

> greadelf --sections net | grep Xen
  [12] .note.Xen.loader  NOTE             0000000000400fb0  00000fb0
  [13] .note.Xen.version NOTE             0000000000400f98  00000f98
  [14] .note.Xen.hyperca NOTE             0000000000400f80  00000f80

Run the example

The net executable can be run from a Xen host using xl, or AtmanOS's helper script startvm.

Start the net program with a bridge network device configured and connect to the console:

> startvm net -c 'vif = ["bridge = xenbr0"]'
Network device initialized:
  Mac address: 00:16:3e:4d:4e:04
  IP address:  ENOENT

If we ping the MAC accress with `arping -i xenbr0 00:16:3e:4d:4e:04`, we see some packets printed:

rx: packet from="08:00:27:a2:0c:15" to="00:16:3e:0a:bc:a1" type=0x0800 (IPV4)
rx: packet from="08:00:27:a2:0c:15" to="00:16:3e:0a:bc:a1" type=0x0800 (IPV4)

Project status

• Single-processor, cooperatively scheduled threads
• Non-paging memory manager
• High-precision timing
• Debug console (write)
• Basic infrastructure for networking
• Basic infrastructure for out-of-tree Xen packages

What's next?

• Preemptive scheduler
• Extract microkernel into runtime/internal packages
• Test harness
• Document deployment to EC2, Linode, etc.
• Finish TCP/IP stack and net

Future:

• pprof to enable profiling app and OS together
• Block storage, filesystems, os.File, ...
• Multi-processor scheduler

Thank you