From SWIM Paper to eBPF Kernel Hooks: Building a Service Mesh Sidecar

Distributed systems infrastructure is typically treated as a black box. Developers use Consul for service discovery, etcd for consensus, and Istio for traffic management — but rarely understand what is actually happening beneath those abstractions. This project is an attempt to remove that black box entirely.

Over the course of this project, I built two foundational pieces of distributed infrastructure from first principles in Go: a service mesh sidecar and a Raft consensus layer. No existing mesh framework. No consensus library. Every component implemented directly from the relevant academic paper or protocol specification.

The service mesh sidecar combines three distinct layers. The first is a gossip-based cluster membership system implementing the SWIM protocol — each node independently tracks which peers are alive, suspects failures through a cascade of direct and indirect probes, and propagates state changes across the cluster by piggybacking updates on routine protocol messages. The second is an L7 HTTP reverse proxy that reads from the live membership table on every request, routing traffic exclusively to healthy nodes and rerouting automatically within seconds of a failure. The third is a transparent L4 TCP redirector implemented as an eBPF program running inside the Linux kernel — intercepting outgoing connection attempts at the syscall level and rewriting their destinations before the TCP handshake occurs, making the entire mesh completely invisible to the applications running on top of it.

The Raft consensus layer implements leader election, log replication, and a key-value state machine on top of a gRPC transport. A cluster of nodes elects a single leader through a randomized timeout mechanism, replicates every write to a majority of nodes before committing, and guarantees that all nodes apply the same operations in the same order — maintaining consistent distributed state even through leader failures and re-elections.

Together, these two systems mirror the architecture of production tools like HashiCorp's Consul: SWIM-based membership feeding into a Raft consensus layer, with a data plane handling traffic routing transparently. The difference is that every line here was written from scratch, against the original papers, with the explicit goal of understanding rather than shipping.

This post documents what was built, how it works, the significant bugs encountered along the way, and what a production implementation would do differently.

Why I Built This

While working on a distributed load testing platform, I used Apache Kafka for workers queue. Basically every piece of communication between master and workers went through kafka. There for masters to learn which workers are alive, the workers needed to send a heartbeat every 5 sec to the master to let it know that the worker is alive and working or else the master would assign the work to someone else.

While working on that heartbeat process it didnt felt so techie using that type of algorithm. That's when I came across Scalable Weakly Consistent Infectious style Membership Protocol or SWIM Protocol.

Every peice of implementation in this project comes from the original paper of the protocol.

Part 1: Cluster Membership with SWIM Gossip

1.1 — The problem: how does a node know who's alive?

This whole paper revolves around how to inform other nodes about the state of a specific node.

Previously we used (and still do when necessary) Heartbeat type member aliveness protocol where every node sent every other node a heartbeat in a fixed interval of time. But now lets take an example.

Imagine there are N members in a distributed system. Every member periodically says: "I'm alive!" And sends that heartbeat to every other member.

So if there are 100 machines:

Machine A sends to 99 machines

Machine B sends to 99 machines

Machine 100 sends to 99 machines That's roughly N × (N−1) messages every heartbeat interval. So as the group gets bigger, the traffic grows quadratically: O(N²).

Why not just send heartbeats less frequently?

Because you have a requirement like:

"I need to detect a failed machine within 1 second."

If you make heartbeats less frequent, you reduce network traffic, but failure detection becomes slower.

So you're stuck between:

Frequent heartbeats

→ fast failure detection

→ huge network traffic

Infrequent heartbeats

→ lower network traffic

→ slow failure detection

That's where the infectious style membership protocol comes to help.

1.2 — What the SWIM paper actually says

From above SWIM fuses the two principal functions of the membership problem specification:

1) A Failure Detector Component, that detects failures of members

2) A Dissemination Component, that disseminates information about members that have recently either joined or left the group, or failed.

Failure Detector Component

The SWIM protocol operates on a fixed protocol period a configurable time interval during which each node executes exactly one failure detection cycle. At the start of each period, the node selects a target uniformly at random from its local membership list and transmits a ping message to it, then waits for an ack within a predefined timeout derived from the expected message round-trip time.

If the ack is received within the timeout, the target is considered reachable and the protocol period ends. If no ack arrives, the node does not immediately declare the target as failed. Network asymmetry, transient congestion, or momentary process scheduling delays can cause a single probe to fail even when the target is fully operational. Declaring failure on a single missed ack would produce an unacceptably high false positive rate.

Instead, the node initiates an indirect probing phase. It selects a fixed number of other members — referred to as the indirect probers — and transmits a ping-req(target) message to each of them. Each indirect prober, upon receiving this request, independently issues its own direct ping to the target and forwards any resulting ack back to the originating node. The indirect probers take different network paths to the target than the originating node did, which is the key property that makes this mechanism useful. A failed direct probe could reflect a localized network partition between two specific nodes rather than an actual process failure. If any indirect prober can successfully reach the target, that constitutes evidence that the target is alive and the originating node's direct path to it is simply impaired.

At the end of the protocol period, the originating node evaluates all received acks — both direct and those forwarded by indirect probers. If at least one ack has arrived through any path, the target is considered reachable. If no ack has arrived through any path, the target is marked suspect in the local membership list and the update is handed off to the dissemination component to be propagated across the cluster.

This design has a critical scalability property worth noting explicitly. Unlike naive heartbeating schemes — where each node periodically broadcasts to all other nodes, producing O(n²) message complexity as the cluster grows — SWIM maintains constant per-node message load. Each node generates exactly one probe per protocol period regardless of cluster size. The indirect probing phase generates a bounded number of additional messages only when a direct probe fails, which under normal operating conditions is rare. This makes the protocol suitable for large clusters where broadcast-based failure detection would generate prohibitive traffic volume.

In this implementation, the protocol period is one second, the direct ping timeout is 500 milliseconds, the indirect probe timeout is an additional 500 milliseconds, and the number of indirect probers is fixed at three — consistent with the parameters discussed in the original paper for small to medium cluster sizes.

The figure above shows the working of the protocol for an arbitrary member M_i

How this works is for a fixed interval of time a random member is picked from M_i membership list (say M_j) and a PING is sent to it.

M_i then wait for a ACK from M_j. If it doesnt recieve the ACK in a certain time limit then M_i marks the member as suspecious

Code implementation

Starting with the node structure

But before that let's get familiar with some technical term defined for a node.

Starting with the structure of the node.

type Node struct {
    ID              string
    BindAddr        *net.UDPAddr
    Members         map[string]*MemberState
    mu              sync.RWMutex
    conn            *net.UDPConn
    pendingPings    map[uint64]chan struct{}
    pendingMu       sync.Mutex
    counter         uint64
    pendingJoin     map[uint64]chan struct{}
    pendingJoinMu   sync.Mutex
    gossipQueue     []string
    queuePos        int
    relayRequests   map[uint64]*net.UDPAddr
    relayMu         sync.Mutex
    pendingUpdates  []Update
    updatesMu       sync.Mutex
    Incarnation     int
}

This struct contains necessary fields along with their mutex implemented using the sync library in order to main many read and one write policy.

This whole implemetation of SWIM basically run 2 go routines. First is the gossip loop and second is the recieve loop.

func (n *Node) Start() {
    go n.receiveLoop()
    go n.gossipLoop()
}

What this gossip loop does is that for every 1 second it picks a member from the nodes member's list using the algorithm below.

func (n *Node) pickNextMember() *MemberState {
    n.mu.Lock()
    defer n.mu.Unlock()

    if n.queuePos >= len(n.gossipQueue) {
        n.rebuildQueue()
        n.queuePos = 0
    }

    if len(n.gossipQueue) == 0 {
        return nil
    }

    id := n.gossipQueue[n.queuePos]
    n.queuePos++
    return n.Members[id]
}

For this picking next member we use Round Robin Algorithm.

The gossip loop then sends a ping to that random member use pingMember()

func (n *Node) pingMember(target *MemberState) {
    seq := n.nextCounter()

    waitCh := make(chan struct{})
    n.pendingMu.Lock()
    n.pendingPings[seq] = waitCh
    n.pendingMu.Unlock()

    defer func() {
        n.pendingMu.Lock()
        delete(n.pendingPings, seq)
        n.pendingMu.Unlock()
    }()

    msg := Message{
        MessageType: "PING",
        From:        n.BindAddr.String(),
        Counter:     seq,
        Updates:     n.drainUpdates(),
    }

    mess, err := json.Marshal(msg)
    if err != nil {
        log.Println("marshal ping failed:", err)
        return
    }

    if _, err := n.conn.WriteToUDP(mess, target.Addr); err != nil {
        log.Println("send ping failed:", err)
        return
    }

    select {
    case <-waitCh:
        n.mu.Lock()
        target.Status = StatusAlive
        target.LastSeen = time.Now()
        n.mu.Unlock()

    case <-time.After(500 * time.Millisecond):
        n.indirectPing(target, seq)

        select {
        case <-waitCh:
            n.mu.Lock()
            target.Status = StatusAlive
            target.LastSeen = time.Now()
            n.mu.Unlock()

        case <-time.After(500 * time.Millisecond):
            n.mu.Lock()
            target.Status = StatusSuspect
            n.buildUpdates(target.Status, target.ID, target.Incarnation)
            n.mu.Unlock()

            log.Printf(
                "indirect ping also failed, marking %s suspect (seq %d)\n",
                target.Addr, seq,
            )

            select {
            case <-waitCh:
                n.mu.Lock()
                target.Status = StatusAlive
                target.LastSeen = time.Now()
                n.buildUpdates(target.Status, target.ID, target.Incarnation)
                n.mu.Unlock()

            case <-time.After(3 * time.Second):
                n.mu.Lock()
                target.Status = StatusConfirm
                n.buildUpdates(target.Status, target.ID, target.Incarnation)
                n.mu.Unlock()

                log.Printf(
                    "indirect ping also failed, marking %s suspect (seq %d)\n",
                    target.Addr, seq,
                )
            }
        }
    }
}

What this code does basically does is that it crease a go channel waitCh and stores it in the pending pings queue with a unique key seq which is generated using nextCounter()

It then sends a ping message containing that seq counter and then waits for the channel to get triggered (happens when the target recieves the ping).

The waiting period is divided into 3 parts

First if the ACK request comes back under 500 ms then the target is marked Alive by the node.

But if 500 ms is passed then we dont mark it as suspect but instead according to the SWIM protocol we pick 3 more random member and ask them to ping the target instead just in case the target's network is flooded. This subprotocol is called Indirect Probing

We use the function indirectPing for that

func (n *Node) indirectPing(target *MemberState, seq uint64) {
    relays := n.pickRelays(target, 3)
    if len(relays) == 0 {
        return
    }

    for _, relay := range relays {
        msg := Message{
            MessageType:    "PING-REQ",
            From:           n.BindAddr.String(),
            Counter:        seq,
            IndirectTarget: target.Addr.String(),
            Updates:        n.drainUpdates(),
        }

        mess, err := json.Marshal(msg)
        if err != nil {
            continue
        }

        go n.conn.WriteToUDP(mess, relay.Addr)
    }
}

Here we create a slice of 3 random member using the function pickRelays which excludes the target and the node itself.

Then we send a PING-REQ to the target through 3 different nodes using the same seq number increasing the probability of recieving an ACK.

Now if recieve the ACK under 500 ms we mark it as alive and if not then we mark it as suspect.

If the target doesnt respond under 3 seconds of being marked as suspect, the node marks it as Confirm i.e Dead.

func (n *Node) receiveLoop() {
    buf := make([]byte, 65535) // max theoretical UDP payload size

    for {
        b, senderAddr, err := n.conn.ReadFromUDP(buf)
        if err != nil {
            log.Println("read failed:", err)
            continue
        }

        n.handleMessage(buf[:b], senderAddr)
    }
}

The 2nd go routine recieveLoop loads the incoming UDP packets in a buffer and calls handleMessage on those bytes.

func (n *Node) handleMessage(buf []byte, addr *net.UDPAddr) {
    var msg Message

    if err := json.Unmarshal(buf, &msg); err != nil {
        log.Println("unmarshal failed:", err)
        return
    }

    n.mergeUpdates(msg.Updates)

    switch msg.MessageType {
    case "PING":
        n.handlePing(msg, addr)

    case "ACK":
        n.handleAck(msg, addr)

    case "PING-REQ":
        n.handlePingReq(msg, addr)

    case "JOIN":
        n.handleJoin(msg)

    case "JOIN-ACK":
        n.handleJoinAck(msg)
    }
}

Handle message divides the incoming packets based on their message type and applies appropriate functions based on that.

func (n *Node) handlePing(msg Message, addr *net.UDPAddr) {
    reply := Message{
        MessageType: "ACK",
        From:        n.BindAddr.String(),
        Counter:     msg.Counter,
    }

    mess, err := json.Marshal(reply)
    if err != nil {
        log.Println("marshal ack failed:", err)
        return
    }

    if _, err := n.conn.WriteToUDP(mess, addr); err != nil {
        log.Println("send ack failed:", err)
        return
    }
}

This handlePing triggers on every Ping request and sends back ACK to the senders with the message counter.

func (n *Node) handleAck(msg Message, addr *net.UDPAddr) {
    // case 1: this is a relay job -- forward the ack to whoever asked us
    n.relayMu.Lock()
    requesterAddr, isRelay := n.relayRequests[msg.Counter]
    if isRelay {
        delete(n.relayRequests, msg.Counter)
    }
    n.relayMu.Unlock()

    if isRelay {
        mess, err := json.Marshal(msg) // forward as-is, same seq
        if err != nil {
            return
        }

        n.conn.WriteToUDP(mess, requesterAddr)
        return
    }

    // case 2: this is our own ping -- existing logic
    n.pendingMu.Lock()
    waitCh, ok := n.pendingPings[msg.Counter]
    n.pendingMu.Unlock()

    if !ok {
        return
    }

    close(waitCh)
}

handleAck processes every ACK reply from the node.

There are two types of ACK recieved from a target

One from the relay where we randomly select 3 members and ask them to ping the target. In this we send the message as is to the desired node which asked for relay in the first place.

Second is from the typical node asking directly, in that case we close the waitCh channel created in pingMember function.

func (n *Node) handlePingReq(msg Message, requesterAddr *net.UDPAddr) {
    targetAddr, err := net.ResolveUDPAddr("udp", msg.IndirectTarget)
    if err != nil {
        log.Println("bad target address:", err)
        return
    }

    n.relayMu.Lock()
    n.relayRequests[msg.Counter] = requesterAddr
    n.relayMu.Unlock()

    reply := Message{
        MessageType: "PING",
        From:        n.BindAddr.String(),
        Counter:     msg.Counter,
        Updates:     n.drainUpdates(),
    }

    mess, err := json.Marshal(reply)
    if err != nil {
        log.Println("marshal ping failed:", err)
        return
    }

    if _, err := n.conn.WriteToUDP(mess, targetAddr); err != nil {
        log.Println("send ping failed:", err)
        return
    }
}

The handlePingRequest function handles the relay requests and send a PING to the desired target.

1.3 — Merging Incoming Updates

Every message in the SWIM protocol — ping, ack, ping-req carries a slice of piggybacked membership updates in its payload. When a node receives any of these messages, before it does anything else, it processes those updates against its own local membership list. This is the dissemination component doing its work: membership changes spreading through the cluster not through dedicated broadcast messages, but riding along on traffic that was already being sent for failure detection purposes.

The merging logic is where several correctness properties of the protocol are enforced simultaneously, and it is worth understanding each rule precisely.

We have seen in multiple function the use of buildUpdates, what it does is that it takes the members status and pushes an update on the nodes pending update.

func (n *Node) buildUpdates(status string, memberId string, incarnation int) {
    n.updatesMu.Lock()
    defer n.updatesMu.Unlock()

    n.pendingUpdates = append(n.pendingUpdates, Update{
        MemberID:    memberId,
        Status:      status,
        Incarnation: incarnation,
    })
}

In the above request handlers we have seen the use of drainUpdates, what it does is that it takes the pending updates slice and sends it whole to the other node to keep them updated about other nodes which are suspecious or dead

func (n *Node) drainUpdates() []Update {
    n.updatesMu.Lock()
    defer n.updatesMu.Unlock()

    out := n.pendingUpdates[:]

    return out
}

When handling messages we use mergeUpdates, it merges all the updates based on certain rules.

Stale updates are ignored. Each update carries an incarnation number alongside the member ID and status. If an incoming update's incarnation number is lower than what the receiving node already holds for that member, the update is silently dropped. This prevents old information from overwriting newer state — for example, a node that was previously suspected but has since refuted itself should not be re-suspected by a stale update still propagating through the cluster.

Suspect updates are accepted conservatively. An incoming Suspect update is only applied if its incarnation number is greater than or equal to the receiver's current incarnation number for that member. Equal incarnation numbers are accepted here because suspicion is a downgrade in status — if two nodes independently observed the same member failing its probes in the same incarnation, both should converge on Suspect.

Confirm triggers immediate deletion. Unlike Suspect, which sets a status field and waits, Confirm causes the member to be deleted from the membership map entirely. There is no further processing — a continue statement ensures the loop moves on without attempting to update fields on a struct that no longer exists in the map.

Self-refutation handles false suspicion. If an incoming update names this node itself as Suspect or Confirm, the node responds immediately by incrementing its own incarnation number and enqueuing an Alive update about itself with the new incarnation number. When this Alive update propagates to other nodes, they will accept it over the stale Suspect — because the incarnation number is strictly higher — and the false suspicion is overwritten. This mechanism allows a node that was temporarily unreachable to defend its own liveness once it recovers connectivity, without any central coordinator involved.

The result of these four rules applied together is that membership state across the cluster converges correctly even when updates arrive out of order, are duplicated by multiple propagation paths, or are delayed by network conditions. No coordination is required — each node independently applies the same deterministic merge rules to whatever updates it receives, and the cluster converges to a consistent view.

func (n *Node) mergeUpdates(updates []Update) {
    n.mu.Lock()
    defer n.mu.Unlock()

    for _, u := range updates {
        if u.MemberID == n.ID &&
            (u.Status == StatusSuspect || u.Status == StatusConfirm) {

            n.Incarnation = u.Incarnation + 1
            n.buildUpdates(StatusAlive, n.ID, n.Incarnation)
        }

        existing, known := n.Members[u.MemberID]
        if !known {
            continue
        }

        if u.Status == StatusConfirm {
            delete(n.Members, u.MemberID)
            continue
        }

        if u.Status == StatusSuspect && u.Incarnation >= existing.Incarnation {
            existing.Status = u.Status
            existing.Incarnation = u.Incarnation
            continue
        }

        if u.Incarnation < existing.Incarnation && u.Status != StatusConfirm {
            continue
        }

        if u.Incarnation == existing.Incarnation &&
            u.Status == existing.Status {
            continue
        }

        existing.Status = u.Status
        existing.Incarnation = u.Incarnation
    }
}

That was all for the failure detection part, now we move on to dissemination component.

1.4 — Dissemination Component

The SWIM paper doesnt really mention this in deep rather it focuses more on the failure detection part but a cluster will be incomplete without the dissemination component

How this works is that we give a newly joined node the address of any alive node in the cluster, and the newbie sends a Join message to that node. This pretty much works like the pingMember function we made in Failure detection part.

func (n *Node) Join(peerAddr *net.UDPAddr) error {

    seq := n.nextCounter()

    waitCh := make(chan struct{})
    n.pendingJoinMu.Lock()
    n.pendingJoin[seq] = waitCh
    n.pendingJoinMu.Unlock()

    defer func() {
        n.pendingJoinMu.Lock()
        delete(n.pendingJoin, seq)
        n.pendingJoinMu.Unlock()
    }()

    reply := Message{
        MessageType: "JOIN",
        From:        n.BindAddr.String(),
        JoinerID:    n.ID,
        Counter:     seq,
    }

    mess, err := json.Marshal(reply)
    if err != nil {
        log.Println("marshal ack failed:", err)
        return err
    }

    if _, err := n.conn.WriteToUDP(mess, peerAddr); err != nil {
        log.Println("send ack failed:", err)
        return err
    }

    select {
    case <-waitCh:
        return nil

    case <-time.After(3 * time.Second):
        return errors.New("timeout")
    }
}

Just like the failure detection, we have handleJoin and handleJoinAck function to properly handle the request.

func (n *Node) handleJoin(msg Message) {
    JoinerAddr, err := net.ResolveUDPAddr("udp", msg.From)
    if err != nil {
        log.Println("handle join failed")
        return
    }

    state := MemberState{
        ID:          msg.JoinerID,
        Addr:        JoinerAddr,
        Incarnation: 1,
        Status:      StatusAlive,
    }

    n.mu.Lock()
    defer n.mu.Unlock()

    n.Members[msg.JoinerID] = &state

    var member MemberInfo
    var members []MemberInfo

    for _, value := range n.Members {
        member = MemberInfo{
            ID:          value.ID,
            Status:      value.Status,
            Incarnation: value.Incarnation,
            Addr:        value.Addr.String(),
        }

        members = append(members, member)
    }

    members = append(members, MemberInfo{
        ID:          n.ID,
        Status:      StatusAlive,
        Incarnation: n.Incarnation,
        Addr:        n.BindAddr.String(),
    })

    reply := &Message{
        MessageType: "JOIN-ACK",
        From:        n.BindAddr.String(),
        Members:     members,
        JoinerID:    n.ID,
        Counter:     msg.Counter,
    }

    value, err := json.Marshal(reply)
    if err != nil {
        log.Println(err)
    }

    if _, err := n.conn.WriteToUDP(value, JoinerAddr); err != nil {
        log.Println(err)
    }

    update := Update{
        MemberID:    msg.JoinerID,
        Status:      StatusAlive,
        Incarnation: 0,
    }

    n.updatesMu.Lock()
    defer n.updatesMu.Unlock()

    n.pendingUpdates = append(n.pendingUpdates, update)
}

func (n *Node) handleJoinAck(msg Message) {
    n.mu.Lock()
    defer n.mu.Unlock()

    for _, value := range msg.Members {
        addr, err := net.ResolveUDPAddr("udp", value.Addr)
        if err != nil {
            log.Println(err)
        }

        n.Members[value.ID] = &MemberState{
            ID:          value.ID,
            Addr:        addr,
            Status:      value.Status,
            Incarnation: value.Incarnation,
        }
    }

    n.pendingJoinMu.Lock()
    defer n.pendingJoinMu.Unlock()

    waitCh, ok := n.pendingJoin[msg.Counter]
    if !ok {
        return
    }

    close(waitCh)
}

Part 2: L7 HTTP Proxy with Live Membership Routing

The gossip layer built in Part 1 answers one question reliably: which nodes in the cluster are currently alive. But knowing who is alive is only useful if something acts on that information. A membership table sitting in memory, never consulted, changes nothing about how traffic flows through the system.

Part 2 adds that action layer. Each node runs an HTTP reverse proxy alongside its application backend. On every incoming request, the proxy reads the current live membership table, selects a healthy target, and forwards the request there. The proxy does not maintain its own health checks, does not run separate probes, and does not keep a separate list of backends. It reads directly from the same membership state that the gossip layer maintains — which means the moment a node is marked Suspect or Confirm by the failure detection protocol, the proxy stops sending traffic to it, automatically, without any configuration change or restart.

This section covers what an L7 proxy is and why membership data is the right source of truth for routing decisions, how Go's standard library httputil.ReverseProxy is used to forward requests with dynamic target selection rather than a fixed backend, how the health-aware routing logic filters the membership snapshot to exclude non-Alive nodes, and what the end-to-end behavior looks like when a node dies mid-test and traffic reroutes within seconds.

2.1 — What an L7 proxy is and why it needs membership data

The OSI model is a standard which enables different system to communicate using different protocols. You can read more about from here.

For our purpose we will mainly focus of L4 and L7 of the OSI model which is the transport and application layer.

At Layer 4, the transport layer, a proxy sees raw TCP or UDP segments. It knows the source IP address, the destination IP address, the source port, and the destination port. That is the complete picture. The actual content of the communication — what application protocol is being spoken, what operation is being requested, what data is being transferred — is opaque at this layer. An L4 proxy can forward connections based on destination address and port, and it can perform network address translation, but it cannot inspect or route based on anything inside the payload.

At Layer 7, the application layer, a proxy operates at the level of the application protocol itself. For HTTP, this means the proxy has full visibility into the request method (GET, POST, PUT), the URL path, the headers, the query parameters, and the response status code. It receives a complete, parsed http.Request object rather than a stream of raw bytes. This additional visibility is what makes meaningful routing decisions possible — routing based on path prefixes, routing based on request headers, retrying failed requests, and in this case, routing based on which backend nodes are currently healthy.

The proxy built in this project operates at Layer 7 using Go's httputil.ReverseProxy. When a client sends an HTTP request, the proxy receives it as a fully parsed object, consults the live membership table to select a healthy backend, rewrites the request's destination, and issues a new outgoing HTTP request to the chosen node. The original client's connection and the proxy's outgoing connection to the backend are two separate TCP connections — the proxy sits between them, reading the response from the backend and streaming it back to the client. This is fundamentally different from a Layer 4 proxy, which would forward the raw bytes of the original connection without ever parsing them.

Why membership data is the correct source of truth for routing

Every routing decision the proxy makes reduces to one question: which nodes are currently healthy enough to receive traffic? The answer to that question already exists — the gossip layer computes it continuously, updating the membership table as nodes are probed, suspected, and confirmed dead.

The proxy reads directly from that membership table via node.Snapshot() on every incoming request. It does not run its own health checks. It does not maintain a separate list of backends with their own independent liveness tracking. This is a deliberate design decision, and the reason is straightforward: if the proxy maintained its own health checks alongside the gossip layer's failure detection, you would have two separate systems independently deciding whether a node is alive, and they could disagree.

Consider what that disagreement looks like in practice. The gossip layer, using the SWIM protocol's direct and indirect probing mechanism, has marked node B as Suspect after observing that it failed probes from multiple different nodes across two consecutive protocol periods. But the proxy's own health check — perhaps a simple TCP connection attempt to port 9001 — still succeeds because the port is technically accepting connections even though the process is degraded. The proxy continues routing traffic to B. Some of those requests fail or time out. The client sees errors that the gossip layer's membership state, correctly consulted, would have prevented.

Maintaining a single authoritative source of membership state and having all components read from it eliminates this class of disagreement entirely. The gossip layer is responsible for knowing who is alive. The proxy is responsible for routing to whoever the gossip layer says is alive. The separation of concerns is clean, the failure modes are predictable, and the system behaves as a coherent whole rather than a collection of independently operating subsystems that happen to share a process.

2.2 — Dynamic target selection with httputil.ReverseProxy

What traditional proxy do is that for every request they get, they redirect it to a certain fixed url and for that we use the function httputil.NewSingleHostReverseProxy.

func NewProxy(target string) (http.Handler, error) {
    targetURL, err := url.Parse(target)
    if err != nil {
        return nil, err
    }

    return httputil.NewSingleHostReverseProxy(targetURL), nil
}

It comes with its own Director function.

But for our implementation we need a dynamic target selection to redirect to and that is why we create our own director function and use it directly in our httputil.ReverseProxy.

func NewProxy(node *members.Node) (http.Handler, error) {

    proxy := &httputil.ReverseProxy{
        Director: func(r *http.Request) {
            for _, value := range node.SnapShot() {
                if value.Status == "Alive" {
                    host, _, err := net.SplitHostPort(value.Addr.String())
                    if err != nil {
                        continue
                    }

                    httpPort := value.Addr.Port + 1000
                    r.URL.Host = fmt.Sprintf("%s:%d", host, httpPort)
                    r.URL.Scheme = "http"
                    return
                }
            }

            r.URL.Host = "" // deliberately invalid
            return
        },
    }

    proxy.ErrorHandler = func(
        w http.ResponseWriter,
        r *http.Request,
        err error,
    ) {
        http.Error(
            w,
            "no healthy backend available",
            http.StatusServiceUnavailable,
        )
    }

    return proxy, nil
}

In this function we use the SnapShot function which gives us the member list of the node and then we filter out the first members which is Alive.

In the director function we you'll notice we split the host and make the http port to listen on the nodes udp port + 1000 so if the node is listening at 8001 port then its http port will be 9001, it a common logic to seperate the ports.

Part 3: Transparent L4 Redirection with eBPF

The L7 proxy built in Part 2 solves the routing problem, but it requires cooperation from the client. A client that wants to benefit from health-aware routing must deliberately connect to the proxy port rather than the real service port. The application has to know the mesh exists.

Part 3 removes that requirement entirely. By intercepting TCP connections at the kernel level using eBPF, the mesh becomes completely invisible to the applications running on top of it. A client dials what it believes is the real service address. The kernel silently rewrites the destination before the TCP handshake occurs. The connection arrives at a different node. The client never knew the switch happened.

This section covers why transparency is the fundamental motivation for operating at Layer 4, what the cgroup/connect4 hook point is and how it fits into the kernel's network stack, how the BPF program is written in C and what it does at the instruction level, how cilium/ebpf and bpf2go are used to compile and load that program from Go, how the BPF hash map serves as the shared data structure between the kernel program and userspace membership sync, and the extended byte order debugging journey that occupied most of the implementation time for this phase.

3.1 — L4 vs L7: why transparency matters

The L7 proxy operates at the HTTP level — it parses incoming requests, makes routing decisions, and issues new outgoing HTTP connections to the selected backend. This works correctly, but it introduces a coupling that limits its applicability. The client must be configured to dial the proxy port. A legacy application, a third-party library, or any service that hardcodes its destination address bypasses the proxy entirely. The mesh only works for traffic that is explicitly directed at it.

Layer 4 interception breaks this coupling. At L4 the proxy does not parse application-layer data at all — it operates on TCP connections before any application bytes are exchanged. When a process calls connect() to establish a TCP connection, that syscall passes through the kernel's networking stack before the connection is actually established. Hooking into that path means the destination can be rewritten before the connection leaves the machine, without the connecting process having any knowledge of or involvement in the rewrite.

The practical consequence is significant. Any TCP traffic — HTTP, gRPC, database connections, raw socket communication — gets transparently redirected regardless of what port it is aimed at or what application is sending it. No configuration change is required on the client side. No awareness of the mesh is required from the application. The redirection is enforced at the kernel level, below everything the application can observe.

3.2 — What cgroup/connect4 actually does

eBPF programs do not run in isolation — they are attached to specific hook points in the kernel where they intercept and optionally modify kernel operations. The hook point used in this project is cgroup/connect4, which fires whenever a process within a specified cgroup calls the connect() syscall to establish an IPv4 TCP connection.

A cgroup — control group — is a Linux kernel mechanism for organizing processes into hierarchical groups for resource management and policy enforcement. Every process on the system belongs to a cgroup. When a cgroup/connect4 program is attached to a specific cgroup path, it intercepts connect() calls from every process in that cgroup and its descendants.

The hook receives a bpf_sock_addr context structure that exposes the destination address and port the process is trying to connect to — user_ip4 for the destination IPv4 address and user_port for the destination port. Crucially, these fields are writable. The BPF program can modify them in place before the kernel proceeds with establishing the connection. The connecting process calls connect() with one destination, the BPF program rewrites it to another, and the kernel establishes the connection to the rewritten destination. From the process's perspective the connect() call succeeded normally.

The program returns 1 to allow the connection to proceed — either with the original destination if no redirect rule matched, or with the rewritten destination if one did.

3.3 — Writing the BPF program in C

eBPF programs are written in a restricted subset of C and compiled to eBPF bytecode using clang with a BPF target. The restriction is meaningful — no unbounded loops, no arbitrary memory access, no function calls outside a predefined set of BPF helper functions. Every program is statically verified by the kernel's BPF verifier before it is allowed to load, which rejects any program that could potentially crash the kernel or access memory it should not.

//go:build ignore

#include <linux/bpf.h>
#include <linux/in.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

struct redirect_target {
    __u32 ip;
    __u32 port;
};

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 256);
    __type(key, __u32);
    __type(value, struct redirect_target);
} redirect_map SEC(".maps");

SEC("cgroup/connect4")
int redirect_connect(struct bpf_sock_addr *ctx) {

    __u32 port = bpf_ntohs((__u16)ctx->user_port);

    if (ctx->user_ip4 != 0x0100007f) {
        return 1;
    }

    struct redirect_target *target =
        bpf_map_lookup_elem(&redirect_map, &port);

    if (!target) {
        return 1;
    }

    ctx->user_ip4  = target->ip;
    ctx->user_port = target->port;

    return 1;
}

char _license[] SEC("license") = "GPL";

The program used in this project is deliberately minimal. It defines a BPF hash map keyed by destination port number and valued by a redirect target struct containing the new destination IP address and port. On every connect() call, it extracts the destination port from ctx->user_port, looks it up in the hash map, and if a matching entry exists, overwrites ctx->user_ip4 and ctx->user_port with the redirect target's values before returning.

The most significant implementation detail is the byte ordering of ctx->user_port. The field stores the port number in network byte order — big-endian — in the upper 16 bits of a 32-bit integer. Extracting the plain port number requires shifting right by 16 bits and then converting from network to host byte order using the bpf_ntohs helper. Getting this wrong — and this project got it wrong several times before getting it right — results in map lookups that silently miss every entry, because the key computed from the incoming connection does not match the key written by the Go userspace code.

3.4 — The Go Bridge: Loading and Managing the BPF Program

The redirect.go file is the userspace side of the eBPF layer — the Go code responsible for loading the compiled BPF program into the kernel, attaching it to the right cgroup, and maintaining the redirect map that tells the kernel where to forward connections.

Loading and attaching

func NewRedirector(cgroupPath string) (*Redirector, error) {
    os.Remove(pinnedMapPath)

    objs := RedirectObjects{}
    if err := LoadRedirectObjects(&objs, nil); err != nil {
        return nil, fmt.Errorf(
            "loading BPF objects: %w", err,
        )
    }

    if err := objs.RedirectMap.Pin(pinnedMapPath); err != nil {
        objs.Close()
        return nil, fmt.Errorf(
            "pinning map: %w", err,
        )
    }

    l, err := link.AttachCgroup(link.CgroupOptions{
        Path:    cgroupPath,
        Attach:  ciliumebpf.AttachCGroupInet4Connect,
        Program: objs.RedirectConnect,
    })

    if err != nil {
        os.Remove(pinnedMapPath)
        objs.Close()
        return nil, fmt.Errorf(
            "attaching cgroup program: %w", err,
        )
    }

    return &Redirector{
        objs: objs,
        link: l,
    }, nil
}

NewRedirector is the entry point. Before doing anything else, it removes any stale pinned map left over from a previous run — if the process crashed without cleaning up, the old map file at /sys/fs/bpf/mesh_redirect_map would prevent a fresh pin from succeeding. It then calls LoadRedirectObjects, a function generated by bpf2go that reads the BPF bytecode embedded in the binary, submits it to the kernel via the bpf() syscall, and returns typed Go handles to the resulting in-kernel objects — the program and the map.

The map is immediately pinned to the BPF filesystem. Pinning creates a persistent file-like reference to the in-kernel map object that other processes can open by path, which is how nodes B and C — which do not own the BPF program — are able to write their own redirect entries into the same shared map.

The program is then attached to the specified cgroup path using link.AttachCgroup with the AttachCGroupInet4Connect attachment type. This registers the BPF program as the handler for every connect() syscall made by any process in that cgroup hierarchy. If the attachment fails — wrong cgroup path, insufficient permissions, unsupported kernel version — the map pin is cleaned up before returning the error so the next startup attempt starts from a clean state.

Writing redirect rules

func (r *Redirector) SetTarget(
    origPort uint16,
    destIP net.IP,
    destPort uint16,
) error {

    ip4 := destIP.To4()
    if ip4 == nil {
        return fmt.Errorf("only IPv4 supported")
    }

    key := uint32(origPort)

    target := RedirectRedirectTarget{
        Ip:   binary.LittleEndian.Uint32(ip4),
        Port: uint32(htons(destPort)),
    }

    return r.objs.RedirectMap.Put(key, target)
}

SetTarget writes a single entry into the BPF hash map. The key is the original destination port as a plain uint32 in host byte order, matching what the C program extracts from ctx->user_port after its shift and bpf_ntohs conversion. The value is a RedirectRedirectTarget struct — the Go type generated by bpf2go to mirror the C struct — containing the destination IP and port.

The IP address is encoded using binary.LittleEndian.Uint32 rather than the more intuitive binary.BigEndian. This is the byte order that bpf_sock_addr->user_ip4 expects on a little-endian x86 machine — 127.0.0.1 becomes 0x0100007f rather than 0x7f000001. Using big-endian here silently redirects connections to 1.0.0.127, a nonexistent address, which was one of the more difficult bugs to diagnose in this project.

The port is converted using htons — a helper that converts a uint16 from host to network byte order, matching the format bpf_sock_addr->user_port expects. RemoveTarget is the inverse operation — it deletes the map entry for a given port, causing the kernel to stop redirecting connections aimed at that port and let them through to their original destination.

Finding the right cgroup path

DefaultCgroupPath attempts to locate the cgroup v2 root automatically rather than hardcoding a path. It checks two candidate locations in order and uses isCgroupV2Root to verify each one. The verification is done via syscall.Statfs — checking the filesystem magic number 0x63677270 (the CGROUP2_SUPER_MAGIC constant defined in the Linux kernel headers) rather than trusting the path string alone. This is a more reliable check than os.Stat because it confirms the mount type, not just the existence of the directory.

The two candidates cover the two configurations observed across different Linux and WSL2 setups: a pure cgroup v2 system where /sys/fs/cgroup is itself a cgroup2 mount, and a hybrid system where cgroup v1 controllers occupy /sys/fs/cgroup and cgroup v2 is mounted separately at /sys/fs/cgroup/unified. If neither path is a valid cgroup v2 root, the error message includes a concrete remediation step specific to WSL2 — adding kernelCommandLine=cgroup_no_v1=all to ~/.wslconfig to force a pure cgroup v2 configuration.

3.5 — The shared BPF map: syncing membership into the kernel

The BPF hash map is the data structure that connects the kernel-level redirect program to the userspace membership table maintained by the gossip layer. The kernel program reads from it on every connect() call. The Go userspace code writes to it every time the membership table changes.

The map is pinned to the BPF filesystem at /sys/fs/bpf/mesh_redirect_map by the node that owns the BPF program attachment. Pinning creates a persistent reference to the in-kernel map object that survives beyond the lifetime of the process that created it and can be reopened by path. This is the mechanism used to share the map across processes — the owning node creates and pins the map, and other nodes open it by path to write their own redirect entries.

A goroutine runs every second, reads the current live membership via node.Snapshot(), builds the sorted round-robin rotation, and writes one entry per node into the BPF map — each node is responsible for writing only its own entry, mapping its own app port to the next alive node in the rotation. When a node is confirmed dead by gossip, its entry is removed from the map via RemoveTarget, and the kernel immediately stops redirecting connections to it.

3.6 — The byte order debugging journey

This section deserves its own honest account because it took longer to debug than any other part of the project, and the lesson it produced is worth documenting.

The symptom was consistent: the BPF program loaded without errors, the cgroup attachment succeeded, the map was being written from Go, but curl to any redirected port always returned the response from the original destination as if no redirect rule existed. The hook was firing — confirmed by an array map that counted connect() interceptions — but every map lookup was returning nothing.

The root cause was byte order, applied incorrectly in three different places across two different boundaries.

The first boundary is between the C program and the kernel's bpf_sock_addr context. ctx->user_port stores the port in network byte order in the upper 16 bits of a 32-bit field. Extracting the port for a map lookup requires bpf_ntohs((__u16)(ctx->user_port >> 16)) — shift first to get the upper 16 bits into the lower position, then convert from network to host byte order. Doing these in the wrong order, or omitting either step, produces a number that looks plausible but does not match any key in the map.

The second boundary is between Go and the kernel for the IP address field. ctx->user_ip4 expects the address in network byte order as interpreted on a little-endian machine — for 127.0.0.1, the correct uint32 value is 0x0100007f, not 0x7f000001. Go's binary.BigEndian.Uint32 produces the latter. binary.LittleEndian.Uint32 produces the former. Using the wrong one results in connections being redirected to 1.0.0.127 — a nonexistent address — which produces immediate connection failures rather than the silent wrong-destination behavior of the map lookup miss.

The third issue was the cgroup attachment path. The BPF program was attaching to /sys/fs/cgroup/unified — confirmed as a valid cgroup v2 mount — but the nsdelegate mount option on that path prevents attachment from within a user namespace, which is how WSL2 runs all processes. The attach call succeeded without error, but the hook never fired for any connection because the processes making those connections were not in the cgroup hierarchy the program was attached to. The correct path on this WSL2 configuration is /sys/fs/cgroup/init.scope, which is where all interactive WSL2 processes actually live.

Each of these bugs produced a different observable symptom — silent map miss, connection refused to wrong host, hook not firing at all — which made them appear to be three separate problems when they were all manifestations of the same underlying category: incorrect understanding of how the kernel represents network addresses and port numbers at the boundary between userspace and kernel space. The lesson is that debugging at the kernel boundary requires reading the kernel documentation for each field precisely, not inferring byte order from what seems logical.

Part 4: Raft Consensus and Distributed Key-Value Store

The gossip layer and the eBPF proxy built in Parts 1 through 3 solve the problem of routing traffic reliably across a cluster of nodes that can fail at any time. But they do not solve a different, harder problem: getting those nodes to agree on anything.

Consider a scenario where multiple clients simultaneously write different values to the same key in a distributed store. Without coordination, different nodes might apply those writes in different orders and arrive at different final states. The cluster has diverged — there is no single correct answer, and no way to know which node to trust. This is the consistency problem, and gossip-based membership alone cannot solve it. Gossip is designed to be eventually consistent, meaning different nodes can temporarily disagree. For shared mutable state, temporary disagreement is not acceptable.

Raft is a consensus algorithm designed specifically to solve this problem. It guarantees that a cluster of nodes agrees on an ordered sequence of operations — a log — even when some nodes crash, restart, or temporarily lose network connectivity. Every write goes through a single elected leader, which replicates the operation to a majority of nodes before considering it committed. As long as a majority of nodes are alive, the cluster makes progress. Committed entries are never lost, even across leader failures.

This section covers how Raft's leader election mechanism works, how log replication achieves majority-based commitment, and how a key-value state machine sits on top of the committed log to produce consistent distributed state across all nodes.

4.1 — Leader Election

Every Raft node starts as a follower. Followers are passive — they receive messages from the leader and respond, but they do not initiate anything. The leader sends periodic heartbeat messages to all followers to assert its authority and suppress new elections. As long as a follower keeps receiving heartbeats, it stays a follower.

Each follower maintains an election timeout — a duration after which, if no heartbeat has been received, it concludes the leader is dead and starts an election. The timeout is randomized within a fixed range, 150 to 300 milliseconds in this implementation. The randomization is critical: if all nodes had the same timeout they would all start elections simultaneously, split the votes among themselves, and never elect a leader. With randomized timeouts, one node's timer fires first, it starts an election before the others, and in most cases wins before they even begin.

func (n *Node) startElection() {
    n.mu.Lock()
    n.state = Candidate
    n.currentTerm++
    n.votedFor = n.id
    n.resetElectionTimeout()

    term := n.currentTerm
    lastLogIndex, lastLogTerm := n.lastLogInfo()
    peers := n.peers
    n.mu.Unlock()

    log.Printf(
        "[%s] starting election for term %d",
        n.id, term,
    )

    votes := 1
    voteMu := sync.Mutex{}

    for _, peer := range peers {
        go func(peer string) {
            granted := n.callRequestVote(
                peer, term, lastLogIndex, lastLogTerm,
            )

            if !granted {
                return
            }

            voteMu.Lock()
            votes++
            currentVotes := votes
            voteMu.Unlock()

            majority := (len(peers)+1)/2 + 1
            if currentVotes >= majority {
                n.becomeLeader(term)
            }
        }(peer)
    }
}

To start an election, a node transitions to the Candidate state, increments its current term — a monotonically increasing logical clock that represents the cluster's current generation — votes for itself, and sends RequestVote RPCs to all peers simultaneously. A peer grants its vote if it has not already voted in this term and the candidate's log is at least as up-to-date as its own. A candidate that receives votes from a majority of nodes — including itself — becomes the new leader for that term.

If no candidate wins a majority in a given term, perhaps because two candidates split the votes evenly, the election times out and a new term begins. The randomized timeouts make it unlikely that two nodes start elections at exactly the same time repeatedly, so the cluster converges on a leader quickly in practice.

Once elected, the leader immediately sends heartbeats to all followers to suppress new elections and assert its authority for the new term. If any node ever receives a message with a term higher than its own, it immediately steps down to follower — this single rule prevents stale leaders from causing split-brain, because a node from a previous term can never override decisions made in a later term.

func (n *Node) electionLoop() {
    for {
        select {
        case <-n.done:
            return
        default:
        }

        n.mu.Lock()
        state := n.state
        elapsed := time.Since(n.lastHeartbeat)
        timeout := n.electionTimeout
        n.mu.Unlock()

        switch state {
        case Follower, Candidate:
            if elapsed >= timeout {
                n.startElection()
            }

        case Leader:
            n.sendHeartbeats()
            time.Sleep(50 * time.Millisecond)
            continue
        }

        time.Sleep(10 * time.Millisecond)
    }
}

4.2 — Log Replication

Once a leader is elected, all writes go through it. When a client submits a command, the leader appends it to its own log as a new entry tagged with the current term, then sends AppendEntries RPCs to all followers in parallel, carrying the new entry along with metadata about the preceding log entry — the prevLogIndex and prevLogTerm.

func (n *Node) Submit(command string) (uint64, bool) {
    n.mu.Lock()
    if n.state != Leader {
        n.mu.Unlock()
        return 0, false
    }

    entry := LogEntry{
        Term:    n.currentTerm,
        Command: command,
    }

    n.log = append(n.log, entry)
    index := uint64(len(n.log))
    term := n.currentTerm
    peers := n.peers
    commitIndex := n.commitIndex

    log.Printf(
        "[%s] submitted command %q at index %d term %d",
        n.id, command, index, term,
    )
    n.mu.Unlock()

    // confirmCh receives one message per peer that confirms replication
    confirmCh := make(chan bool, len(peers))

    for _, peer := range peers {
        go func(peer string) {
            success := n.callAppendEntries(
                peer,
                term,
                commitIndex,
                []LogEntry{entry},
            )

            if success {
                n.mu.Lock()
                if index > n.matchIndex[peer] {
                    n.matchIndex[peer] = index
                    n.nextIndex[peer] = index + 1
                }
                n.mu.Unlock()
            }

            confirmCh <- success
        }(peer)
    }

    // wait for majority -- we already have 1 (ourselves)
    majority := (len(peers)+1)/2 + 1
    confirmed := 1
    responded := 0

    timeout := time.After(2 * time.Second)

    for confirmed < majority && responded < len(peers) {
        select {
        case ok := <-confirmCh:
            responded++
            if ok {
                confirmed++
            }

        case <-timeout:
            log.Printf(
                "[%s] Submit timed out waiting for majority",
                n.id,
            )
            return 0, false
        }
    }

    if confirmed < majority {
        return 0, false
    }

    n.mu.Lock()
    if n.currentTerm == term && index > n.commitIndex {
        n.commitIndex = index
        go n.applyCommitted()
    }
    n.mu.Unlock()

    log.Printf(
        "[%s] committed command %q at index %d",
        n.id, command, index,
    )

    return index, true
}

A follower accepts the entry only if its own log contains an entry at prevLogIndex with the matching prevLogTerm. This consistency check is what keeps all logs identical — a follower will not append an entry unless it can verify that everything preceding it already matches the leader's log. If the check fails, the follower rejects the entry and the leader retries with earlier entries until it finds the point where the logs diverge and repairs from there.

The leader counts confirmations as responses arrive. Once a majority of nodes — including the leader itself — have written the entry to their logs, the entry is considered committed. The leader advances its commitIndex to reflect the new committed position and notifies followers of the new commitIndex on the next heartbeat, allowing them to advance their own commit positions and apply the committed entries to their state machines.

This majority requirement is what makes Raft fault-tolerant. A cluster of three nodes can lose one node and still commit entries, because two nodes constitute a majority. A cluster of five nodes can lose two. As long as a majority are alive and can communicate, the cluster makes progress. And because any two majorities share at least one node in common, it is impossible for two different values to both achieve majority confirmation for the same log index — the cluster can never commit two conflicting entries.

4.3 — The Key-Value State Machine

The Raft log is a means to an end. What matters is not the log itself but what is derived from it — a state machine that every node drives from the same sequence of committed entries, producing identical state across the cluster.

func (kv *KVStore) apply(command string) {
    parts := strings.Fields(command)
    if len(parts) == 0 {
        return
    }

    kv.mu.Lock()
    defer kv.mu.Unlock()

    switch strings.ToUpper(parts[0]) {
    case "SET":
        if len(parts) < 3 {
            return
        }
        kv.data[parts[1]] = parts[2]

    case "DEL":
        if len(parts) < 2 {
            return
        }
        delete(kv.data, parts[1])
    }
}

The state machine in this project is a simple key-value store: an in-memory map[string]string that supports SET key value and DEL key operations. Every committed log entry carries one of these commands as a string. As entries are committed by Raft, they are sent through the ApplyCh channel to the KVStore, which parses the command and applies it to the local map.

Because every node applies the same entries from the same log in the same order, all nodes converge to identical map state. There is no coordination required at the read or write level of the key-value store itself — consistency is guaranteed by the Raft log that drives it.

Writes go through the leader via kv.Set or kv.Delete, which call node.Submit internally. Submit appends the command to the log, replicates it to a majority via AppendEntries, waits for majority confirmation, advances commitIndex, and applies the entry through ApplyCh. The call blocks until the entry is committed, so by the time it returns, a majority of nodes have the entry durably in their logs.

Reads are served locally from the node's own map without any coordination. This is a deliberate simplification — a follower's map may be slightly behind the leader's if recent entries have not yet propagated. In a production system, reads would be routed through the leader or use a read index mechanism to guarantee linearizability. For a learning implementation, local reads are sufficient to demonstrate that all nodes converge to the same state after a brief propagation delay.

4.4 — Testing: All Nodes Converging to Identical State

The test scenario runs three Raft nodes in the same process, each with its own gRPC server on a different localhost port. After election, the leader receives three sequential writes — SET x 1, SET y hello, SET x 2 — followed by DEL y. Each write blocks until committed by a majority.

After each batch of writes, all three nodes print their full map state. The expected output demonstrates two properties simultaneously: that all nodes show exactly the same key-value pairs after the writes propagate, and that the final state reflects the correct order of operations — x ends up as 2 rather than 1 because the second SET x was applied after the first, as guaranteed by the log ordering.

The leader failure test extends this scenario. After the initial writes commit, the leader is stopped. Within 150 to 300 milliseconds — the election timeout window — one of the remaining nodes detects the missing heartbeats, starts an election in the next term, and wins. The cluster continues without data loss. The key-value state on the surviving nodes is identical to what it was before the leader failed, because only committed entries — those already replicated to a majority — were applied to the state machine, and committed entries are never lost across leader changes.

This end-to-end test validates leader election, log replication, majority-based commitment, state machine application, and leader failure recovery in a single observable sequence. The output is deterministic enough to verify correctness by inspection — every node prints the same map, in the same state, after every operation.

Part 5: How SWIM and Raft Fit Together

At this point the project has two independent systems running side by side. The SWIM gossip layer tracks cluster membership — it knows which nodes are alive, which are suspected, and which have been confirmed dead. The Raft consensus layer maintains a consistent distributed log — it elects a leader, replicates entries to a majority, and drives a key-value state machine. Both systems work correctly in isolation. But they have been built separately, and they do not yet talk to each other.

This section explains why that separation is not a design flaw, where the natural integration points between the two systems are, and what connecting them would look like in a production-grade implementation.

5.1 — Why the Separation is Deliberate

The instinct when building two systems that both deal with cluster nodes is to merge them — to have one unified component that handles both membership and consensus. This instinct is worth resisting, because the two systems have fundamentally different consistency requirements that make them poor candidates for unification.

SWIM is designed to be eventually consistent. Different nodes can temporarily disagree about whether a peer is alive, and the protocol tolerates that disagreement. The suspicion mechanism introduces deliberate delay before declaring a node dead precisely because premature agreement on a false failure is worse than temporary disagreement. SWIM optimizes for low message overhead and scalability — each node generates a constant number of messages per protocol period regardless of cluster size.

Raft requires strong consistency by design. Every node must agree on the same log entries in the same order, with no temporary disagreement permitted. A committed entry is committed everywhere, immediately, by definition — if it were not, the safety guarantee that makes Raft useful would not hold. Raft tolerates network partitions and node failures through its majority requirement, not through eventual convergence.

Conflating these two consistency models into a single system would force one of them to compromise. Either the membership system becomes unnecessarily heavyweight — running consensus-grade coordination for information that does not require it — or the consensus system becomes unreliable by depending on eventually consistent membership data for decisions that require certainty. Keeping them separate preserves the design properties of each.

This is not a theoretical concern. HashiCorp's Consul, one of the most widely deployed service discovery and consensus tools in production infrastructure, makes exactly this separation. Serf handles gossip-based membership using a SWIM variant. Raft handles consensus over configuration and service catalog data. They run as distinct subsystems within the same agent, communicating through well-defined interfaces rather than sharing internal state.

5.2 — Where the Integration Points Are

Although the two systems are kept separate, they are not independent. Raft needs information from SWIM in two specific places, and understanding those integration points is what makes the architecture coherent rather than just two unrelated programs running in the same process.

Peer set initialization. When a Raft node starts, it needs to know the addresses of its peers — the other nodes it will send RequestVote and AppendEntries RPCs to. In this implementation that list is static, passed in at startup. In a system that integrates SWIM and Raft, the initial peer set would come from the SWIM membership table. A new node joins the cluster via the SWIM JOIN handler, learns the full current membership from the seed node, and uses that membership list to initialize its Raft peer configuration. There is no separate bootstrap step for Raft — the gossip layer already handled discovery.

Dynamic peer set updates. More significantly, as the cluster evolves — nodes joining, nodes failing and being confirmed dead — the Raft peer set needs to reflect those changes. If a node is confirmed dead by SWIM and removed from the membership table, Raft should stop sending it AppendEntries RPCs and stop waiting for its vote. Continuing to include a dead node in the Raft peer set does not cause incorrectness — the majority calculation still works because the dead node simply never responds — but it does cause unnecessary latency, as the leader waits for RPC timeouts on every round before proceeding. More importantly, if enough nodes die that the remaining live nodes no longer constitute a majority of the original peer set, the cluster stalls even though a majority of currently alive nodes could still make progress if the peer set were updated.

Handling dynamic peer set changes in Raft correctly is non-trivial. The Raft paper describes a joint consensus approach where the cluster transitions through an intermediate state in which both the old and new peer configurations must independently achieve majority, preventing any window during which two different leaders could be elected under two different majority definitions. This is one of the more complex parts of the Raft protocol and is one of the known simplifications in this implementation — the peer set is static.

Routing integration. The eBPF redirect layer and L7 proxy already read from the SWIM membership table to make routing decisions. If Raft were integrated, the same membership table could inform Raft client routing — directing write requests to whichever node the current Raft leader is, rather than requiring clients to discover the leader through trial and error. The leader identity could be stored in the SWIM membership state as an additional field, propagated through the same piggybacking mechanism as Alive and Suspect updates, and read by the proxy layer to route write traffic to the leader automatically.

5.3 — What a Full Integration Would Look Like

Connecting SWIM and Raft into a cohesive system would involve three concrete changes to this codebase.

The first is initializing the Raft peer list from the SWIM membership table after JOIN completes. Rather than passing a static peers []string to raft.NewNode, the joining node would call node.Snapshot() after the JOIN-ACK is processed and derive the initial peer list from the returned membership. The Raft node would start with the same view of the cluster that the gossip layer already established.

The second is registering a membership change callback in the SWIM layer that notifies the Raft node when a peer is confirmed dead. The SWIM mergeUpdates function already processes Confirm events — adding a hook there that calls a method on the Raft node to remove the dead peer from its active peer set would be a small, localized change. The Raft node would update its nextIndex and matchIndex maps and exclude the dead peer from future RPC rounds and majority calculations.

The third is propagating leader identity through the gossip layer. When a Raft node wins an election and transitions to Leader state, it would enqueue a membership update marking itself as the current leader — a new status field alongside Alive, Suspect, and Confirm. Other nodes would receive this update through the normal piggybacking mechanism and the proxy layer would read it from the membership snapshot to route client write requests directly to the leader without needing to probe multiple nodes.

These three changes would produce the same architecture that Consul implements internally — a gossip layer that handles membership and failure detection, a consensus layer that handles strong consistency, and a well-defined interface between them through which membership changes flow into the consensus configuration and leader identity flows back out into the routing layer.

5.4 — What This Project Taught About Distributed Systems Design

Building both systems from scratch, separately and then thinking through how they connect, produced a specific understanding that reading existing implementations does not.

The first lesson is that consistency is not a single dial. Different parts of a distributed system have different consistency requirements, and the right tool for each part depends on what guarantees that part actually needs. The membership layer needs to be scalable and available — it should keep working even when some nodes are unreachable, and it should not generate prohibitive message traffic as the cluster grows. The consensus layer needs to be correct — it should never allow two different values to both be considered committed. These requirements pull in opposite directions, and trying to satisfy both with a single mechanism produces a system that does neither well.

The second lesson is that the integration points between systems matter as much as the systems themselves. SWIM and Raft are both well-designed protocols independently. What makes them useful together is the specific interface through which they communicate — peer set initialization, membership change notification, and leader identity propagation. Getting those interfaces right requires understanding what each system needs from the other, which requires understanding each system deeply enough to know what it assumes about its environment.

The third lesson is about debugging at system boundaries. The most difficult bugs in this project were not bugs in the gossip logic or bugs in the Raft state machine — they were bugs at the boundaries between systems. The byte order issue in the eBPF layer was a boundary bug between the C kernel program and the Go userspace code. The cgroup path issue was a boundary bug between the Linux kernel's cgroup hierarchy and the WSL2 virtualization layer. Boundary bugs are harder to diagnose because the symptoms appear in one system while the cause lives in another, and the documentation for each side of the boundary assumes knowledge of the other side that you may not have.

Both SWIM and Raft are foundational protocols that appear, in some form, in nearly every large distributed system built today. Understanding them at the implementation level — not just as black boxes with documented APIs — changes how you reason about the systems built on top of them.