tiny-firecracker

a firecracker deep-dive (kvm-userspace-monitor)
firecracker - amazon tool for microvm's - for lambda functions and light-weight containers

arch and features:

main VMM cycle is async but
every VCPU has to be in its own sync blocking thread
initializing some channels in main.rs - vmm<->api bidirectional channels, vcpu_to_vmm to pass statuses
also api_event_fd eventfd for process_api_requests
jailer starting with root privileges, creating chroot env, using pivot_root for isolation, creating /dev/kvm link with mknod, then starting static vmm binary with empty privileges (set nobody uid/gid) have a basic axum's rest api through unix socket (simpler than tcp) to interact with vm
have mmio (VcpuExit::MmioRead, VcpuExit::MmioWrite) and a VirtIO module with partially implemented disk(blk) and console devices

wanted to go on and implement a bit more things:

seccomp filters for ioctl
virtio-net
snapshotting and live migration

some differences with firecracker and naive/study potential pitfalls:

1 using EventManager pattern with epoll - VMM cycle is async in general, vcpu's on blocking system thread but api requests go through AsyncFd and eventfd
2 instant 64-bit booting without bios (QEMU diligently goes from 16-bit to 32 and 64 to emulate every os)
3 bpf filter for ioctl (seccompiler?) instead of allowing all requests
4 for jailer - have to use musl static compiler cuz we need to put binary in a custom empty root (it wont find libc.so)

not using vm-memory virtio-queue vm-virtio vm-device, everything recreated from scratch expect for 2 util crates:kvm_ioctls and vmm_sys_util

known limitations

this project reached some final conceptual milestone
its possible to launch linux kernel with this but im sure there are a ton of bugs and it needs to be go through some debugging
and after refactoring to make everything async i met with some tokio panics and found out some architectural problems
so decided to put it off

makefile commands

make run-dev (simple vmm launch)
make build-jailed (static binary)
make run-jailed ARGS="--id vm1 --uid 1000 --gid 1000" (run static binary with args)

will panic now
curl --unix-socket /tmp/tiny-firecracker.sock -X PUT -H "Content-Type: application/json" -d '{"action_type": "InstanceStart"}' http://localhost/actions