Update the client_init.sh and client_sidecar.sh scripts

[scatat]

Details

Describe the solution you'd like:

Where do I start? I started to use the vpn-pod-gateway because I thought it was much more efficient to use a single pod for VPN connections than running a VPN sidecar on every single pod ( yuck ). I found this - the pod-gateway mechanism and learned a huge amount about admission controllers and mutating webhooks in order to fix things to work on my k3s setup.

I forked this repo: https://github.com/scatat/pod-gateway and have made a no of changes, namely,

• changing the config dir to /pod-gw-config as it was clashing with the /config dir in the container I use ( the linux server images which use /config. For this reason, I can't really push a PR to you as I'm sure this is too "custom" for people's tastes
• had to change the way vxlan was added ( not sure how anyone else's was workign tbh as vxlan simply didn't come up until I added the multicast group 239.1.1.1 t to the creation of the vxlan interface
• disabled one of the ping test to the gw ip ( k3s gw ip simply does not respond to pings - again, otherw probably wont' like this so no pr )
• add and iptables rule to allow DHCP - again not sure how anyone else's implementation worked without this iptables -I INPUT 1 -p udp --dport 67 -j ACCEPT

I then noticed that I kept on having to nuke the pod-gateway constantly as it kept on failing to give out DHCP addresses - the client pods would just fail.

When I looked at the pod gw - the dnsmasq which dished out DHCP leases had run out of leases - I only run 5 pods so this sould not happen

I then looked at my client pods and in the gateway sidecar, there are hundreds of dhcp clients running - all hanging.

Basically, the client pod is DOsing the dnsmasq server - this is really bad management of DHCP and bad process management in general.

I also decided to upgarde the Alpine container to 3.21.3 and this is where the fun really started - the dhcp client used by pod gateway has been removed.

With this in mind, I thought I could not just leave things be any more - the whole vpn gw is a mess and needed some actual work rather than a few hacks.

Basically, I've completely reworked the client_init.sh and client_sidecar.sh scripts. This is because they both violate various K8 concepts and are generally not well formed or written ( a bit of a hack )

The current client set up runs a daemon which periodically runs the init script arbitrarily - the init script renews using the dhclient. There is no proper resource management going on here.. I see someone has introduced some hack to just kill old dhcp processes but this is merely a hack.

What shoudl happen:

Init Container: Runs and must complete successfully before app containers start
Sidecar Container: Runs alongside app containers for the pod's lifetime

I have re-written the client_init.sh and client_sidecar.sh scripts like so:

https://github.com/scatat/pod-gateway/blob/main/bin/client_init.sh

or:

set -e

# Load settings
. /default_config/settings.sh
. /pod-gw-config/settings.sh

# Setup variables
K8S_DNS_IP="$(cut -d ' ' -f 1 <<< "$K8S_DNS_IPS")"
GATEWAY_IP="$(dig +short "$GATEWAY_NAME" "@${K8S_DNS_IP}")"
VXLAN_GATEWAY_IP="${VXLAN_IP_NETWORK}.1"

echo "Setting up vxlan interface and routes"

# Setup routing rules for local traffic
K8S_GW_IP=$(/sbin/ip route | awk '/default/ { print $3 }')
for local_cidr in $NOT_ROUTED_TO_GATEWAY_CIDRS; do
    ip route add "$local_cidr" via "$K8S_GW_IP" || true
done

# Delete default routes
ip route del 0/0 || true
ip route -6 del default || true

# Add gateway route
ip route add "$GATEWAY_IP" via "$K8S_GW_IP" || true

# Create and configure vxlan interface
ip link add vxlan0 type vxlan id "$VXLAN_ID" group 239.1.1.1 dev eth0 dstport 0 || true
bridge fdb append to 00:00:00:00:00:00 dst "$GATEWAY_IP" dev vxlan0
ip link set up dev vxlan0

# Set MTU if needed
if [[ -n "$VPN_INTERFACE_MTU" ]]; then
    ETH0_INTERFACE_MTU=$(cat /sys/class/net/eth0/mtu)
    VXLAN0_INTERFACE_MAX_MTU=$((ETH0_INTERFACE_MTU-50))
    if [ ${VPN_INTERFACE_MTU} -ge ${VXLAN0_INTERFACE_MAX_MTU} ]; then
        ip link set mtu "${VXLAN0_INTERFACE_MAX_MTU}" dev vxlan0
    else
        ip link set mtu "${VPN_INTERFACE_MTU}" dev vxlan0
    fi
fi

# Get initial IP configuration
NAT_ENTRY="$(grep "^$(hostname) " /config/nat.conf || true)"
if [[ -z "$NAT_ENTRY" ]]; then
    echo "Setting up dynamic IP via DHCP (one-time)"
    # Get initial IP address using foreground mode (exit after lease)
    if ! udhcpc -i vxlan0 -n -q -f; then
        echo "ERROR: Failed to obtain initial IP lease" >&2
        exit 1
    fi
else
    # Static IP configuration
    IP=$(cut -d' ' -f2 <<< "$NAT_ENTRY")
    VXLAN_IP="${VXLAN_IP_NETWORK}.${IP}"
    echo "Using static IP $VXLAN_IP"
    ip addr add "${VXLAN_IP}/24" dev vxlan0 || true
    route add default gw "$VXLAN_GATEWAY_IP" || true
fi

# Test connectivity before allowing pod to start
echo "Testing gateway connectivity"
if ! ping -c "${CONNECTION_RETRY_COUNT}" "$VXLAN_GATEWAY_IP"; then
    echo "ERROR: Gateway not reachable" >&2
    exit 1
fi

echo "Network setup complete - pod can start"
exit 0

and https://github.com/scatat/pod-gateway/blob/main/bin/client_sidecar.sh

#!/bin/bash
set -e

# Load settings
. /default_config/settings.sh
. /pod-gw-config/settings.sh

VXLAN_GATEWAY_IP="${VXLAN_IP_NETWORK}.1"
DHCP_PID_FILE="/var/run/udhcpc.vxlan0.pid"
NAT_ENTRY="$(grep "^$(hostname) " /config/nat.conf || true)"

# Function to check gateway connectivity
check_gateway() {
    ping -c "${CONNECTION_RETRY_COUNT}" "$VXLAN_GATEWAY_IP" > /dev/null 2>&1
    return $?
}

# Function to verify and start DHCP client if needed
ensure_dhcp_client() {
    if [[ -n "$NAT_ENTRY" ]]; then
        # Using static IP, no DHCP needed
        return 0
    fi

    # Check if DHCP client is running
    if [ -f "$DHCP_PID_FILE" ] && kill -0 $(cat "$DHCP_PID_FILE") 2>/dev/null; then
        return 0
    fi

    echo "Starting DHCP client daemon"
    # Use background mode (-b) which handles renewals automatically
    udhcpc -b -i vxlan0 -p "$DHCP_PID_FILE" -O subnet -O router

    # Verify we got an IP
    if ! ip addr show dev vxlan0 | grep -q "inet ${VXLAN_IP_NETWORK}"; then
        echo "ERROR: Failed to get IP address for vxlan0" >&2
        return 1
    fi
    return 0
}

# Initial startup check
echo "Sidecar starting - verifying network configuration"
ensure_dhcp_client

# Main monitoring loop
while true; do
    # Check interface exists with IP
    if ! ip link show vxlan0 &>/dev/null; then
        echo "ERROR: vxlan0 interface missing, attempting recovery" >&2
        ip link add vxlan0 type vxlan id "$VXLAN_ID" group 239.1.1.1 dev eth0 dstport 0 || true
        ip link set up dev vxlan0
    fi

    if ! ip addr show dev vxlan0 | grep -q "inet ${VXLAN_IP_NETWORK}"; then
        echo "ERROR: vxlan0 has no valid IP address" >&2
        ensure_dhcp_client
    fi

    # Check gateway connectivity
    if ! check_gateway; then
        echo "ERROR: Gateway connectivity lost" >&2

        # Check DHCP client status for dynamic IPs
        if [[ -z "$NAT_ENTRY" ]]; then
            if ! ensure_dhcp_client; then
                echo "Failed to restart DHCP client"
            fi
        else
            # For static IPs, just ensure the route exists
            echo "Ensuring route for static IP"
            route add default gw "$VXLAN_GATEWAY_IP" || true
        fi
    else
        echo "Gateway connection OK - $(date)"
    fi

    sleep 10
done

This approach:

Uses udhcpc's built-in daemon mode (-b) which manages lease renewals automatically
Only runs a single udhcpc process per interface
Stores lease information in a simple file for the sidecar to monitor
Has clear error handling and diagnostics
Is more event-driven (reacts to lease issues and connectivity problems)
Has minimal process overhead (just the sidecar and udhcpc daemon)
This follows K8s best practices for sidecar containers, which continuously monitor and attempt to fix issues rather than exiting. The error messages will appear in the pod logs for diagnostics.

I strongly suggest that you integrate these changes into your pod gateway.

My updated pod gateway now:

• Runs latest versions of Alpine
• uses the inbuild busybox udhcpc client which won't just be nuked by Alpine
• updates the client scripts to give decent resource managment and error handling

I think I've done a fair bit of work and just trying to give back - I may try to improve the rest of the setup but this is a fair chunk of work and I have a day job, haha.

Thanks to Angelnu by the way for keeping this alive. I've found this code quite useful on the whole.

Cherrio.

Anything else you would like to add:

Nothing

Additional Information:

[radi-ka]

Hi @scatat,

thanks for putting work in. The code is looking much cleaner,

but i can't wrap my head around how you got udhcpc to update the resolv.conf.

For me udhcpc prints "mv: cannot move '/etc/resolv.conf.30' to '/etc/resolv.conf': Resource busy".
I am wondering if this might be the cause for my persistent DNS issues or if I need to look elsewhere.

Do you see the same logs?

[EDIT]: nvm. The client did not have the dnsPolicy: None flag set... Had it set for the gw, somehow forgot for the client -.-

regards