A quick search for “How to set MTU for Amazon Linux” would lead you straight here and you might think to yourself “This is easy! No problemo!” but upon following the instructions, you’ll be shocked to learn your MTU is still 9001 and asking yourself why do machines not listen to the instructions we give them?

It turns out despite the best efforts of the machines, a brain worm everyone loves immensely called ‘systemd’ has made this an exciting learning experience.

For this guide, our target MTU is 1440 bytes and the NIC is ens5.

What Amazon tells you to do

The short summary is:

1. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and append MTU=1440
2. Add request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers; to /etc/dhcp/dhclient.conf
3. Reboot and bask in your success

Wait a minute!

~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah

How did this happen?

Inspect the logs

Thankfully there are logs and I hope you’re a savvy journalctl user as you’ll need it for troubleshooting issues like this. Start by querying the logs for systemd-networkd (excellent name, Poettering)

journalctl -u systemd-networkd

EC2 instances use DHCP for grabbing their IP configuration, so the log will be spammed with countless DHCP events, but the line which caught my eye was this:

systemd-networkd[pid]: ens5: Configured with /usr/lib/systemd/network/80-ec2.network

My love of systemd grows stronger with every passing day.

What is this file?

The ArchWiki team has an excellent article on systemd-networkd that explains how these configuration files work, what order they’re loaded in and where they should be located. View it here

Open this file up in nano (accept no substitutes) and you’ll see

[Link]
MTUBytes=9001

Unbelievable! My first instinct here was to avoid editing the system file as the ArchWiki explains that files in /etc/systemd/network/ take precedence. So therefore we should drop our modified .network files in /etc/ and be on our our way, right?

Overriding systemd-networkd

Well this is what I tried and after a reboot I saw this:

systemd-networkd[pid]: ens5: Configured with /etc/systemd/network/80-ec2.network

Awesome!

~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah

Noo! Back to the logs

systemd-networkd[pid]: ens5: Reconfiguring with /run/systemd/network/70-ens5.network

When I dump out the volatile config, I see it’s configured for jumbo frames

[Link]
MTUBytes=9001

Back to the drawing board.

The desperate search begins

Now we’ve established there’s a ghost in the machine, let’s hunt for it.

journalctl | grep ens5

Something caught my eye

ec2net[pid]: Starting configuration for ens5

What the heck is ec2net? A quick search of the filesystem revealed it’s a helpful little script located at /usr/share/amazon-ec2-net-utils/lib.sh and searching for that revealed it’s open source. GitHub

The information on GitHub is enlightening:

The version 1.x branch of the amazon-ec2-net-utils package was used in Amazon Linux 2 and earlier releases. It has a long history and is tightly coupled to ISC dhclient and initscripts network configuration. Both of these components are deprecated and will not make up the primary network configuration framework in future releases of Amazon Linux or other distributions. The 2.x branch (released from the main branch in git) represents a complete rewrite targeting a more modern network management framework. The rest of this document describes the 2.x branch.

So the reason we’re in this situation is the documentation is relevant to just Amazon Linux 2 and 2023 does things differently.

This script is the bridge between IMDS and systemd-networkd. It creates the volatile configuration we saw above and sources its defaults from /usr/lib/systemd/network/80-ec2.network.

In conclusion

Edit /usr/lib/systemd/network/80-ec2.network, change the MTU value from 9001 to your desired value and make sure to document your changes as it’ll probably come to haunt you next time amazon-ec2-net-utils updates.

You may undo any edits you made to /etc/dhcp/dhclient.conf and /etc/sysconfig/network-scripts/ifcfg-eth0 as these have no effect.

p.s. This article was updated to correct references to AL2 as I didn’t realise that AL2023 is an entirely different version from AL2.