Strengthening Deployment Safety with eBPF: A GitHub-Inspired Guide

Overview

GitHub’s infrastructure faces a unique circular dependency: the very platform that hosts millions of repositories is itself hosted on github.com. A failure could lock teams out of the code needed to fix the outage. While a mirror mitigates this, deeper circular dependencies remain hidden in deployment scripts. For example, a script might pull a tool from GitHub (direct dependency), a tool might check for updates (hidden dependency), or call an internal service that in turn contacts GitHub (transient dependency). To solve this, GitHub turned to eBPF (extended Berkeley Packet Filter)—a Linux kernel technology that can safely monitor and block specific system calls during deployments. This guide walks you through the problem, the eBPF approach, and how to implement a similar solution.

Prerequisites

• A Linux system with kernel 4.4+ (eBPF support).
• Basic knowledge of Linux system calls and process execution.
• Familiarity with bpftrace or libbpf for writing eBPF programs (we’ll use C-style examples).
• Root or sudo access to load eBPF programs.
• Understanding of circular dependency types (covered in this section).

Step-by-Step Guide

1. Understand the Types of Circular Dependencies

Before writing eBPF programs, you must identify the patterns you want to block. GitHub identifies three types:

• Direct: A deployment script directly contacts an external service (e.g., downloading a binary from GitHub).
• Hidden: An existing binary on the host checks for updates during execution.
• Transient: A script calls an internal API that in turn contacts an external service.

eBPF can intercept these at the kernel level, regardless of the application’s logic.

2. Select the System Calls to Monitor

Common system calls for network access include connect, sendto, sendmsg, and open for local file access. For GitHub’s use case, monitoring connect to external domains (like github.com) is key. You can also monitor execve to track binary execution and open for config files.

3. Write an eBPF Program to Intercept System Calls

Below is a simplified eBPF program (C with libbpf) that blocks connect calls to specific IP addresses. In practice, GitHub would integrate this into their deployment tooling.

// block_connect.bpf.c
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <linux/ip.h>

#define DENY_IP "192.168.1.1" // Example

SEC("kprobe/tcp_v4_connect")
int block_connect(struct pt_regs *ctx) {
    struct sock *sk;
    // Parse sockaddr_in from arguments
    // Compare with DENY_IP
    if (match) {
        return -1; // Block
    }
    return 0;
}

char _license[] SEC("license") = "GPL";

Compile with clang -O2 -target bpf -c block_connect.bpf.c -o block_connect.o.

4. Attach the Program to Tracepoints

Use bpftool or bpf syscall to load and attach. For a deployment script, you might attach only to the script’s process by filtering with pids.

# Load program
bpftool prog load block_connect.o /sys/fs/bpf/block_connect
# Attach to kprobe (example - adjust path)
bpftool prog attach pinned /sys/fs/bpf/block_connect kprobe tcp_v4_connect

5. Filter and Block Based on Context

To avoid blocking legitimate traffic, filter by:

• Process ID: Only apply to the deployment script and its children.
• Destination: Block only IPs belonging to critical services (e.g., github.com).
• Syscall type: Possibly allow DNS lookups but block actual connects.

GitHub uses eBPF maps to maintain dynamic allow/deny lists that can be updated without recompiling.

6. Test and Validate

Simulate a deployment scenario: run a script that attempts to curl github.com. The eBPF program should block the connection and log the event. Verify via bpftool prog tracelog or custom maps. GitHub’s approach ensures that even if deployment scripts change, the eBPF layer provides a last-resort safety net.

Common Mistakes

• Overblocking: Blocking all network calls can break legitimate internal communication. Always use per-process or per-script filtering.
• Ignoring Hidden Dependencies: A tool might already be on disk but still check for updates. Monitor execve and open to catch these.
• Not Handling Transient Dependencies: A script’s dependency chain can be deep. eBPF must track child processes via clone and fork events.
• Performance Overhead: Attaching to high-frequency syscalls like connect can impact latency. Use efficient filtering (e.g., pre-calculated maps) and limit scope.
• Kernel Version Mismatch: eBPF features evolve. Ensure your program only uses stable helpers compatible with your kernel.

Summary

Circular dependencies in deployment are a serious reliability risk. GitHub demonstrated that eBPF offers a surgical, kernel-level approach to block problematic system calls without modifying application code. By intercepting connect to external services, filtering by process context, and covering hidden/transient patterns, you can harden your own deployment pipelines. Start with the steps above, adapt the filtering to your environment, and test rigorously. eBPF is a powerful tool for deployment safety, but like any kernel instrumentation, it requires care. Use it wisely to keep your systems resilient.