# An Applied Introduction to eBPF with Go One of our engineering leaders discusses how to develop eBPF programs with Go, and how we use eBPF at Edge Delta. Ozan Sazak Software Engineer Mar 12, 2024•11 minutes Most of the time when we are developing software or even using software, we are playing within the safe boundaries of the operating system. We don’t even know how that IP packet was welcomed from the network interface, or those inodes were handled by the filesystem when we save a file. That boundary is called the **`user space`**, and it’s where we write our applications, libraries, and tools. But there’s another world, which is the **`kernel space`**. It’s where the operating system’s kernel resides, and it’s responsible for managing the system’s resources, such as memory, CPU, and I/O devices. ## The Need to Go Deeper Let’s say you want to go down the stack and somehow insert your custom code into the kernel to profile the application, or to trace the system calls, or to monitor the network packets. How would you do that? Traditionally you have two options. ### Option 1: Edit the Kernel Source Code If you want to change the Linux kernel source code and then ship the same kernel to your customer’s machine, you will need to convince the Linux kernel community that the change is required. Then, you will need to wait for several years for the new kernel version to be adopted by the Linux distributions. This is not a practical approach for most of the cases, and it’s also a little much for just profiling an application, or monitoring the network packets. ### Option 2: Write a Kernel Module You can write a kernel module, which is a piece of code that can be loaded into the kernel and executed. This is a more practical approach, but it has its own risks and downsides. First, you need to write a kernel module, which is not an easy task. Then, you need to maintain it regularly, because the kernel is a living thing, and it changes over time. If you don’t maintain your kernel module, it will be outdated and won’t work with the new kernel versions. Second, you are risking corrupting your Linux kernel, because kernel modules don’t have security boundaries. If you write a kernel module that has a bug, it can crash the whole system. ## Enter eBPF [eBPF](https://ebpf.io/) (Extended Berkeley Packet Filter) is a revolutionary technology that allows you to **reprogram the Linux kernel within minutes, without even rebooting the system**. eBPF allows you to trace system calls, user space functions, library functions, network packets, and much more. It’s a powerful tool for systems performance, monitoring, security, and much more. ### But how? eBPF is a system consisting of several components: - eBPF programs - eBPF hooks - BPF maps - eBPF verifier - The eBPF virtual machine > _**Note that I have used the term “BPF” and “eBPF” interchangeably. eBPF stands for “Extended Berkeley Packet Filter”. BPF was originally introduced to Linux to filter network packets, but eBPF extends the original BPF to allow it to be used for other purposes. Today it’s not related to Berkeley, and it’s not only for filtering packets.**_ An eBPF program is attached to a specific code path in the kernel, such as a system call. These code paths are called **`"hooks"`**. When the hook is triggered, the eBPF program is executed and now it performs the custom logic you have written. This way we can run our custom code in the kernel space. ## Hello World with eBPF Before moving on to the details, let’s write a simple eBPF program to trace the **`execve`** system call. We will write the program in C, the user space program in Go, and then run the user space program. This will load the eBPF program into the kernel, and poll the custom events we will emit from the eBPF program, right before the actual **`execve`** system call is executed. ### Writing the eBPF Program Let’s start writing the eBPF program first. I will write part by part to explain the details better, but you can find the whole program in my GitHub repo: [ozansz/intro-ebpf-with-go](https://github.com/ozansz/intro-ebpf-with-go/tree/main/0x01-helloworld). ```c #include "vmlinux.h" #include struct event { u32 pid; u8 comm[100]; }; struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 1000); } events SEC(".maps"); ``` Here we import the **`vmlinux.h`** header file, which contains the kernel’s data structures and function prototypes. Then we include the **`bpf_helpers.h`** header file, which contains helper functions for eBPF programs. Then we define a **`struct`** to hold the event data, and then we define a [BPF map](https://docs.kernel.org/bpf/maps.html) to store the events. We will use this map to communicate the events between the eBPF program, which will run in kernel space, and the user space program. ### Writing the User Space Program Before starting to write the user space program, let me briefly explain what the program needs to do in user space. We need a user space program to load the eBPF program into the kernel, create the BPF map, attach to the BPF map, and then read the events from the BPF map. To perform these operations, we need to use a specific system call. This system call is called **`bpf()`**, and it’s used to perform several eBPF-related operations, such as reading the contents of a BPF map. ```go //go:generate go run github.com/cilium/ebpf/cmd/bpf2go -type event ebpf hello_ebpf.c func main() { stopper := make(chan os.Signal, 1) signal.Notify(stopper, os.Interrupt, syscall.SIGTERM) // Allow the current process to lock memory for eBPF resources. if err := rlimit.RemoveMemlock(); err != nil { log.Fatal(err) } objs := ebpfObjects{} if err := loadEbpfObjects(&objs, nil); err != nil { log.Fatalf("loading objects: %v", err) } defer objs.Close() kp, err := link.Kprobe(kprobeFunc, objs.HelloExecve, nil) if err != nil { log.Fatalf("opening kprobe: %s", err) } defer kp.Close() rd, err := ringbuf.NewReader(objs.Events) if err != nil { log.Fatalf("opening ringbuf reader: %s", err) } defer rd.Close() ... } ``` ### Running the Program Now we are ready to run the program. First, we need to compile the eBPF program, and then run the user space program. ```bash $ go generate $ go build -o hello_ebpf ``` Now, let's run it: ```bash $ sudo ./hello_ebpf ``` ### eBPF Hooks and Maps So, what actually happened in the previous example? We attached the eBPF program to the **`sys_execve`** system call using the **`kprobe`** hook, to run the **`hello_execve`** function whenever the **`sys_execve`** system call is called, right before the original system call code is executed. ### Automate Alert Triage with AI Edge Delta's out-of-the-box AI agent for SRE automatically surfaces important signals from alert noise and analyzes real-time telemetry data to pinpoint the root cause.