Introduction to XDP
eXpress Data Path (XDP) is a feature in the Linux kernel which allows you to execute a user-supplied eBPF program when a packet is received on a network interface (NIC). This program is executed in the NIC’s device driver, right after the interrupt processing and before SKB memory allocation (which is an expensive operation). This results in lots of flexibility as the program is able to modify the packet before anything else processes it and high performance since the eBPF program is executed after very little (minimum) processing on the packet. The goal of XDP is to offer comparable performance to kernel bypass solutions while working with the existing kernel networking stack.
XDP has 2 modes:
- Generic Mode – The NIC driver does not need to support XDP, but the kernel will fake it instead. The XDP program is executed after SKB allocation so there is no performance benefit. Advantage of this mode is that you do not need a special network driver that supports XDP.
- Native Mode – The NIC driver supports XDP and the XDP program is executed before SKB allocation.
When an XDP program is finished processing a packet, it should return one of the following XDP actions:
- XDP_PASS: let the packet continue through the network stack
- XDP_DROP: silently drop the packet
- XDP_ABORTED: drop the packet with trace point exception
- XDP_TX: bounce the packet back to the same NIC it arrived on
- XDP_REDIRECT: redirect the packet to another NIC or user space socket via the AF_XDP address family
Requirements to use XDP
I was able to successfully use XDP of Amazon Linux 2, Fedora 31, CentOS 7 and RHEL 8. The following are the requirements that must be met in order to use XDP:
1. The kernel must support XDP.
XDP support was introduced in the Linux Kernel version 4.80. XDP support is configured by setting the CONFIG_XDP_SOCKETS parameter in the kernel’s .config so you can run the following command to check if your current kernel supports XDP:
$ grep -i CONFIG_XDP_SOCKETS /boot/config-$(uname -r)
CONFIG_XDP_SOCKETS=y
Currently, the default kernel in Amazon Linux 2 kernel is version 4.14 so you will have to first update the kernel to a later version to get XDP support and the instructions are laid out later in this article.
2. The network driver must support XDP.
XDP Support was introduced to the ENA driver in version 2.2.0 [1]. If the NIC driver does not support XDP, you can only use XDP in generic mode.
3. The maximum MTU supported by XDP is 3818.
The default MTU in EC2 is 9001 so you will have to reduce this MTU to use XDP which you can do using the following command:
$ ip link set dev eth0 mtu 3818
4. The number of channels available for the Rx/Tx queues must be at least half of the maximum available channels on the NIC.
You can use the following command to see how the channels are allocated on your NIC and what the maximum available channel count is:
$ ethtool -l eth0
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 8
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 8
In this example (c5.18xlarge instance), there are a maximum of 8 available channels which are all allocated for multi-purpose channels ( “combined” )… so I need to free up at least 4 channels. I can do this by setting “combined” = 1 to free up 7 channels using the following command:
$ ethtool -L eth0 combined 1
5. The ENA driver in EC2 can only be used if your instance is using Enhanced Networking.
It is possible to have the ENA module installed but if the instance is not using Enhanced Networking, the network interface will not be using the ENA driver. You can confirm that your NIC is using the correct ENA driver using the following command:
$ ethtool -i eth0
If your NIC is not using the ENA driver (if it is using a version older than v2.2.0), follow this guide to install/update your ENA driver [2].
Using XDP With The ENA Driver on Amazon Linux 2
1. Launch an Amazon Linux 2 instance using a Nitro instance type (T3, M5, C5, etc…). Nitro instances have Enhanced Networking which will allow you to use the ENA driver which has XDP support.
2. Attach a second ENI on your instance which we will use for experimenting with XDP. You can use your eth0 but your XDP program might disconnect you from your machine.
3. Verify that eth1 is working as expected:
$ ping -I eth1 1.1.1.1
4. Update your kernel to a later version which supports XDP and reboot the system:
$ sudo -i
$ amazon-linux-extras enable kernel-ng
$ yum clean metadata
$ yum install -y kernel
$ uname -r
$ init 6
$ modinfo ena
5. After updating the kernel, the ENA driver should already be a version greater than v2.2.0 which you can confirm using the following command:
$ ethtool -i eth0
If this command returns an older version of the ENA driver, follow this guide to update your ENA driver [2].
6. Install a C compiler along with BPF libraries and then write and compile a simple eBPF program to drop all packets [3]:
$ yum install -y clang llvm kernel-headers bpftool
$ cat << EOF >> xdp-example.c
#include <linux/bpf.h>
#define SEC(NAME) __attribute__((section(NAME), used))
SEC("prog")
int xdp_prog_simple(struct xdp_md *ctx)
{
return XDP_DROP;
}
char _license[] SEC("license") = "GPL";
EOF
$ clang -O2 -Wall -target bpf -g -c xdp-example.c -o xdp-example.o
7. Load the XDP program using XDP Native Mode on eth1 (make sure you use eth1 otherwise you will drop your SSH connection):
$ ip link set dev eth1 xdpdrv object xdp-example.o
8. Test eth1 again to see if packets are being dropped as expected:
$ ping -I eth1 1.1.1.1
9. You can disable the XDP program using the following command:
$ ip link set dev eth1 xdp off
Further Reading
https://pantheon.tech/what-is-af_xdp/
https://www.redhat.com/en/blog/using-express-data-path-xdp-red-hat-enterprise-linux-8
https://www.systutorials.com/docs/linux/man/8-ip-link/
https://blogs.igalia.com/dpino/2019/01/02/build-a-kernel/
https://github.com/xdp-project/xdp-tutorial/tree/master/testenv
Well written and useful article, thanks!