Isolated Network Namespaces with IP Aliasing & NAT for QEMU VMs

[klogg]

Overview

We need a robust, scriptable networking architecture to support multiple isolated QEMU VM environments running as systemd services. The architecture must allow multiple identical environments (with overlapping internal IP subnets, e.g., 10.0.0.0/24) to run simultaneously on the same host without routing conflicts.

The host must provide outbound internet access to the VMs (via NAT) and allow direct, transparent inbound access from the host to individual VMs using standard ports (e.g., standard SSH on port 22) and hostnames, entirely avoiding client-side workarounds like ~/.ssh/config or custom port mapping.

Requirements

1. Isolation: VMs must be grouped into distinct Linux network namespaces.
2. Overlapping Subnets: Namespaces must support identical internal subnets (e.g., 10.0.0.0/24) without host routing table collisions.
3. Local Services: Each namespace must run its own dnsmasq instance attached to a bridge to serve DHCP/DNS to its local VMs.
4. Systemd Native: VMs and dnsmasq must run as standard systemd services trapped within their respective namespaces.
5. Outbound Internet: VMs must have internet access routed through the host.
6. Transparent Inbound Access: The host must be able to resolve VM hostnames natively (e.g., ssh root@main.aos-unit) and connect on default ports without manual port forwarding rules on the client.

Architecture Blueprint

To solve the routing conflicts of overlapping subnets and the limitations of L4 port forwarding, we will use a Transit Network combined with IP Aliasing and 1:1 DNAT via nftables.

Network Topology

NOTE: all specific network addresses are just for reference

• Host Transit Network: 192.168.100.0/24
• Namespace Internal Subnets: 10.0.0.0/24 (Identical across all namespaces)

Example: Namespace 1 (ns1)

• Host Veth (veth-host1): Unnumbered or attached to a host-side transit bridge.
• Namespace Uplink (veth-ns1):
  • Primary IP (NAT GW): 192.168.100.10
  • Alias IP for VM1: 192.168.100.11
  • Alias IP for VM2: 192.168.100.12
• Namespace Bridge (br0): 10.0.0.254 (runs local dnsmasq)
• VM Tap Interfaces (tap0, tap1): Attached to br0

Key Benefits

1. Configuration Simplification

DNS Port Standardization:

• Eliminates DNS_ALT_PORT: Each namespace runs dnsmasq on standard port 53 without conflicts
• No systemd-resolved conflicts: Namespace isolation prevents port collisions with host DNS services
• Standard DNS queries: All DNS tools work without custom port configuration

Network Configuration Auto-Derivation:

• No manual DHCP_CIDR configuration: Automatically derived from main node IP/mask
• Only virtual endpoints CIDR: Transit network pool also used for Aliasing

# /etc/aos-unit/unit-name.yml - Main node IP drives everything
unit:
  node_configs:
    - name: main
      ip: 10.0.0.100           # Optional, drives DHCP_CIDR derivation, default DHCP_CIDR is 10.200.1.0/24
      netmask: 255.255.255.0   # Optional, default /24
      cpu: 4
      mem: 8G

Auto-derived values from main node IP:

Input: main.ip = 10.0.0.100, main.netmask = 255.255.255.0
  ↓
DHCP_CIDR = "10.0.0.0/24"      (calculated: IP & netmask)
BRIDGE_IP = "10.0.0.1"         (network + 1)
DHCP_START = "10.0.0.128"      (second half of range)
DHCP_END = "10.0.0.254"        (last usable IP)

2. Standard Port Access

No Port Mapping Required:

• SSH on port 22: ssh root@vm1.unit.aos-unit (not ssh -p 2222)
• HTTP on port 80: curl http://vm1.unit.aos-unit (not curl http://unit:8080)
• PostgreSQL on port 5432: psql -h vm1.unit.aos-unit (standard connection)
• All standard tools work: No -p flags, no custom port configuration needed

Client-side simplicity:

• No ~/.ssh/config port overrides required
• No custom port mapping documentation needed
• Ansible, monitoring tools, database clients work without modification
• Port numbers match production environments (dev/test parity)

3. Namespace Isolation Benefits

Network Stack Isolation:

• Independent routing tables: Each namespace has its own routing table
• Independent firewall rules: nftables/iptables rules scoped per namespace
• Independent network interfaces: Bridge names can be identical (e.g., br0 in all namespaces)
• No host routing conflicts: Overlapping internal subnets (10.0.0.0/24) cause no issues

Service Isolation:

• Per-namespace dnsmasq: Each runs on port 53 without conflicts
• Independent DHCP scopes: Identical IP ranges in different namespaces
• Isolated broadcast domains: ARP, DHCP broadcasts contained within namespace
• Process isolation: Services in one namespace cannot interfere with another

4. Operational Benefits

Simplified Debugging:

• Clear namespace boundaries: ip netns exec ns1 <command> for namespace-specific operations
• Isolated packet capture: ip netns exec ns1 tcpdump -i br0 shows only namespace traffic
• Per-namespace logs: dnsmasq, firewall logs naturally separated
• Standard tools work: No custom wrappers needed for diagnostics
• Easy identification: Namespace name can be equal to unit name

Resource Efficiency:

• Minimal overhead: Namespaces add negligible CPU/memory overhead vs full VMs
• Efficient IP allocation: 1 IP per VM from transit pool (not /24 per unit)
• Scalable design: Supports hundreds of units with /16 transit network

Maintainability:

• Familiar model: Mimics cloud VM networking (each VM has IP, standard ports)
• Standard Linux networking: No exotic technologies, well-documented kernel features
• Easy to reason about: Clear separation of transit (192.168.x.x) and internal (10.0.0.x) networks

5. Security Improvements

Attack Surface Reduction:

• No exposed non-standard ports: Eliminates port scanning target confusion
• Clear security boundaries: Namespace = trust boundary
• Explicit NAT rules: 1:1 DNAT is transparent and auditable

Isolation Guarantees:

• Units cannot interfere: Network namespaces provide kernel-level isolation
• Contained compromise: Breach in one unit doesn't expose others' networks
• Independent firewall policies: Different security postures per unit possible

Implementation Steps

1. Transit & Namespace Setup

For each namespace, establish the network boundary using standard iproute2 commands:

# Create namespace and veth pair
ip netns add ns1
ip link add veth-host1 type veth peer name veth-ns1
ip link set veth-ns1 netns ns1

# Configure Namespace Uplink (IP Aliasing)
ip -n ns1 addr add 192.168.100.10/24 dev veth-ns1  # Primary GW
ip -n ns1 addr add 192.168.100.11/32 dev veth-ns1  # VM1 Alias
ip -n ns1 addr add 192.168.100.12/32 dev veth-ns1  # VM2 Alias
ip -n ns1 link set veth-ns1 up

# Host-side veth configuration
ip link set veth-host1 up
ip addr add 192.168.100.1/24 dev veth-host1  # Transit gateway

2. Internal Bridge & Tap Interfaces

Inside the namespace, create the Layer 2 bridge for the VMs:

ip -n ns1 link add br0 type bridge
ip -n ns1 addr add 10.0.0.254/24 dev br0
ip -n ns1 link set br0 up

# Routing: default via transit network
ip -n ns1 route add default via 192.168.100.1 dev veth-ns1

# Enable forwarding
ip netns exec ns1 sysctl -w net.ipv4.ip_forward=1

# Tap interfaces will be pre-created or created by systemd/QEMU and enslaved to br0
ip -n ns1 link add tap0 type tap
ip -n ns1 link set tap0 master br0
ip -n ns1 link set tap0 up

3. Traffic Routing & nftables Configuration

Inside each namespace, apply the following nftables ruleset to handle outbound Masquerading and inbound 1:1 DNAT based on the Alias IPs:

#!/usr/sbin/nft -f
# /etc/nftables/ns1-nat.nft

table ip nat {
    # Inbound: Map transit Alias IPs directly to internal VM IPs
    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
        ip daddr 192.168.100.11 dnat to 10.0.0.5   # VM1
        ip daddr 192.168.100.12 dnat to 10.0.0.6   # VM2
    }
    
    # Outbound: Masquerade internal traffic out through the transit network
    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;
        oifname "veth-ns1" masquerade
    }
}

Apply inside namespace:

ip netns exec ns1 nft -f /etc/nftables/ns1-nat.nft

4. DNS Configuration

Inside Namespace (dnsmasq):

ip netns exec ns1 dnsmasq \
    --interface=br0 \
    --bind-interfaces \
    --port=53 \                          # Standard port! (no DNS_ALT_PORT needed)
    --domain=unit1.aos-unit \
    --dhcp-range=10.0.0.128,10.0.0.254,12h \
    --dhcp-option=option:router,10.0.0.254 \
    --dhcp-option=option:dns-server,10.0.0.254 \
    --dhcp-host=52:54:00:12:34:56,10.0.0.5,vm1 \  # VM1 internal IP
    --dhcp-host=52:54:00:12:34:57,10.0.0.6,vm2    # VM2 internal IP

On Host (/etc/hosts or systemd-resolved):

# Map VM hostnames to transit Alias IPs
192.168.100.11  vm1.unit1.aos-unit
192.168.100.12  vm2.unit1.aos-unit

# Or use systemd-resolved DNS configuration
resolvectl domain veth-host1 '~unit1.aos-unit'
resolvectl dns veth-host1 10.0.0.254

5. Systemd Integration

Service units for dnsmasq and qemu-system-x86_64 will utilize systemd's native namespace isolation.

dnsmasq service:

[Unit]
Description=dnsmasq for namespace ns1
After=network.target

[Service]
Type=forking
NetworkNamespacePath=/var/run/netns/ns1
ExecStart=/usr/sbin/dnsmasq --conf-file=/etc/dnsmasq.d/ns1.conf
PIDFile=/run/dnsmasq-ns1.pid

[Install]
WantedBy=multi-user.target

VM service:

[Unit]
Description=QEMU VM1 in namespace ns1
After=dnsmasq-ns1.service
Requires=dnsmasq-ns1.service

[Service]
Type=simple
NetworkNamespacePath=/var/run/netns/ns1
ExecStart=/usr/bin/qemu-system-x86_64 \
    -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
    -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:56 \
    -drive file=/var/lib/vms/vm1.qcow2,if=virtio \
    -m 4G -smp 2 -enable-kvm

[Install]
WantedBy=multi-user.target