Now staring at the INACCESSIBLE_BOOT_DEVICE BSOD, your options are abort or press ahead. In this article, we charge forward and overcome our lack of preparedness!

Enter Windows Recovery

Reset the VM a few times while at the Windows loading screen and it’ll dump you into the recovery environment. From here you can perform an advanced repair and summon a command prompt. It typically takes 2 failed boots to trigger recovery.

Mount an ISO Containing Drivers

In this example, I’ve neglected to install the VirtIO SCSI driver so I’ll mount the VirtIO ISO for Windows. You can grab it from here

Switch to your disc in the Command Prompt using its drive letter (usually D:)

Load the SCSI Driver

Find the driver you wish to load. (vioscsi.inf in my case)

Load the driver by running drvload vioscsi.inf

List Disks

Use diskpart to list your disks to confirm they’re now visible

Use list volume to see what drive letter your Windows drive got assigned

My Windows partition was now visible on drive letter E:

Add the Driver to Windows

Use dism to insert the driver into your Windows install

dism /Image:E:\ /Add-Driver:D:\vioscsi\2k19\amd64\vioscsi.inf

Success!

Reboot into Windows and enjoy your new SCSI driver.

Google Chrome and its variants also do this but there’s no workaround for them as they’re browsers that can charitably described as garbage.

You can fix this with Firefox by setting the following preference to false:

browser.urlbar.resultMenu.keyboardAccessible

No restart required, takes effect immediately.

It turns out despite the best efforts of the machines, a brain worm everyone loves immensely called ‘systemd’ has made this an exciting learning experience.

For this guide, our target MTU is 1440 bytes and the NIC is ens5.

What Amazon tells you to do

The short summary is:

1. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and append MTU=1440
2. Add request subnet-mask, broadcast-address, time-offset, routers, domain-name, domain-search, domain-name-servers, host-name, nis-domain, nis-servers, ntp-servers; to /etc/dhcp/dhclient.conf
3. Reboot and bask in your success

Wait a minute!

~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah

How did this happen?

Inspect the logs

Thankfully there are logs and I hope you’re a savvy journalctl user as you’ll need it for troubleshooting issues like this. Start by querying the logs for systemd-networkd (excellent name, Poettering)

journalctl -u systemd-networkd

EC2 instances use DHCP for grabbing their IP configuration, so the log will be spammed with countless DHCP events, but the line which caught my eye was this:

systemd-networkd[pid]: ens5: Configured with /usr/lib/systemd/network/80-ec2.network

My love of systemd grows stronger with every passing day.

What is this file?

The ArchWiki team has an excellent article on systemd-networkd that explains how these configuration files work, what order they’re loaded in and where they should be located. View it here

Open this file up in nano (accept no substitutes) and you’ll see

[Link]
MTUBytes=9001

Unbelievable! My first instinct here was to avoid editing the system file as the ArchWiki explains that files in /etc/systemd/network/ take precedence. So therefore we should drop our modified .network files in /etc/ and be on our our way, right?

Overriding systemd-networkd

Well this is what I tried and after a reboot I saw this:

systemd-networkd[pid]: ens5: Configured with /etc/systemd/network/80-ec2.network

Awesome!

~ ip link
2: ens5 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 blahblah

Noo! Back to the logs

systemd-networkd[pid]: ens5: Reconfiguring with /run/systemd/network/70-ens5.network

When I dump out the volatile config, I see it’s configured for jumbo frames

[Link]
MTUBytes=9001

Back to the drawing board.

The desperate search begins

Now we’ve established there’s a ghost in the machine, let’s hunt for it.

journalctl | grep ens5

Something caught my eye

ec2net[pid]: Starting configuration for ens5

What the heck is ec2net? A quick search of the filesystem revealed it’s a helpful little script located at /usr/share/amazon-ec2-net-utils/lib.sh and searching for that revealed it’s open source. GitHub

The information on GitHub is enlightening:

The version 1.x branch of the amazon-ec2-net-utils package was used in Amazon Linux 2 and earlier releases. It has a long history and is tightly coupled to ISC dhclient and initscripts network configuration. Both of these components are deprecated and will not make up the primary network configuration framework in future releases of Amazon Linux or other distributions. The 2.x branch (released from the main branch in git) represents a complete rewrite targeting a more modern network management framework. The rest of this document describes the 2.x branch.

So the reason we’re in this situation is the documentation is relevant to just Amazon Linux 2 and 2023 does things differently.

This script is the bridge between IMDS and systemd-networkd. It creates the volatile configuration we saw above and sources its defaults from /usr/lib/systemd/network/80-ec2.network.

In conclusion

Edit /usr/lib/systemd/network/80-ec2.network, change the MTU value from 9001 to your desired value and make sure to document your changes as it’ll probably come to haunt you next time amazon-ec2-net-utils updates.

You may undo any edits you made to /etc/dhcp/dhclient.conf and /etc/sysconfig/network-scripts/ifcfg-eth0 as these have no effect.

p.s. This article was updated to correct references to AL2 as I didn’t realise that AL2023 is an entirely different version from AL2.

This appears to be a firmware bug with the Segate devices, but Seagate does a poor job of advertising where to get firmware. If you’re affected, update to firwmare version SU6SM005 by doing the following:

1. Download SeaTools from here
2. Click on the “Advanced” tab once launched, it’ll give you the serial number
3. Return to the SeaTools download page, there’s a Firmware Downloads widget on the side. Insert your serial number into that
4. Download the firmware update ZIP and extract it
5. Select the affected drive in SeaTools and then click “Firmware Update”
6. Browse to the new .bin file and hit Ok
7. The firmware update should complete within a few seconds

Subsequent to this I formatted the drive and reinstalled Windows onto another drive since I wasn’t sure whether this would fix the issue. After formatting, I 80% filled the drive with large files and left them there for months.

After letting them age on the drive, I copied them to another SSD and the read performance was extremely fast. Therefore it seems the firmware update has resolved the issue and I would suggest this to anyone having issues.

This post will cover how to replicate common vSphere networking setups in Proxmox. Please note, I’ll only cover Linux networking, not OVS or the newish SDN feature in Proxmox.

Why do I have to “Apply Configuration”?

Proxmox is fundamentally based on Debian Linux whose networking is built upon the ifupdown package. Proxmox builds on this by using ifupdown2, an improved version that makes life easier for developers, but the fundamentals for end users are the same.

When a configuration change is made, you’re editing /etc/network/interfaces to include the new configuration. Proxmox won’t apply the changes straight away as you may be making multiple changes which are dependant on each other and reloading the configuration does have the potential to briefly disrupt the network.

Creating vSwitches in Proxmox

Proxmox’s Linux networking can act like a standard vSwitch within vSphere, but the terminology is a little different. Of the options presented, the one you’re interested in is “Linux Bridge”.

Once you click this, you’ll be presented with this.

This is one aspect of Proxmox you’ll have to get used to: Endless forms with not a lot of hints, though the Help button generally works well.

Here are the options relevant for vSwitches

VLAN Aware

This allows the bridge to pass around dot1q tagged traffic. If you leave this unchecked, it is still possible to use VLANs on a guest’s NIC, but Proxmox will create VLAN NICs bridged to your Linux bridge in the background to achieve this.

If you are using VLANs, I strongly recommend you enable VLAN awareness.

Bridge Ports

This is the equivalent of assigned physical adapters in VMware and is essentially what interfaces the bridge is actually connected to. Unlike VMware, this doesn’t need to be a real physical adapter. I’ll cover bridging to bonded interfaces in the link failover section.

IP Addresses?

One thing that may seem confusing is the form asks for IP networking configuration. This is because Linux bridges combine two separate VMware concepts, that of a VMkernel adapter and its associated network.

Configuring an IP address is optional, guests can still use the bridge without it.

Networking Within the Hypervisor Only

With vSphere, it’s possible to create a vSwitch with no physical ports that just passes traffic between connected guests. This is useful if you use a software router such as OPNsense to handle layer 3 routing, NAT, etc. for your guests as you won’t want to connect them to an outside network.

This is extremely simple with Proxmox. It’s literally as simple as leaving the “Bridge ports” field empty.

Link Failover Configuration

VMware vSphere standard switches have a direct relationship with the physical network adapters they’re assigned to (if any). In VMware, active-active, active-passive, etc. link failover is achieved by just setting the interface order for the vSwitch.

Proxmox approaches this a little differently, failover is achieved by creating a new interface called a Linux Bond. Below is an example configuration that mimics the active-passive configuration above.

In this instance, eno3 and enp65s0f1 are actual physical network interfaces. As enp65s0f1 is the primary, it’ll always use that whenever a link is available and only use eno3 as a last resort.

As with bridges, IP address configuration is optional. You cannot plug a guest’s NIC directly into a bond, so an IP address configuration is likely only useful for creating a management interface that has link redundancy.

A Linux bridge may make use of a bonded interface by simply bridging to it. Specifying the Linux bond’s interface name as the bridged port is all you need to do.

Port Groups

In the VMware world, a port group is used for connecting guests to a vSwitch. It is intended to be analogous to a switchport on a real world network switch.

Proxmox approaches this a little differently, there are no port groups and the VLAN configuration is held by the guest’s NIC. This is akin to if you could attach a VMware guest directly to a vSwitch and just specify the VLAN directly on the guest interface.

VLAN Tag

The VLAN Tag field is self-explanatory, guest traffic passing through the bridge will be tagged with the value you specify here. However the “no VLAN” placeholder when nothing is inserted is somewhat misleading, as in the case of a VLAN Aware bridge, it acts like a port group tagged 4095. It’ll pass all tagged and untagged traffic.

If you just want untagged traffic, you may “tag” the port as VLAN 1, which acts like the “None” option in VMware, and disables VLANing for the NIC.

VMkernel NICs on Specific VLANs

In vSphere, it’s possible to configure the management interface to sit on a specific VLAN by configuring a VLAN ID for the VMkernel NIC’s port group. Proxmox approaches this a little differently, since any interface is a candidate for carrying management traffic, we instead just create a Linux VLAN interface and configure an IP on it.

In this example, I’ve used a common naming convention of Device.VLAN, which Proxmox picks up on and auto fills to the correct device and tag. Any guest or physical network plugged into vmbr1 can reach Proxmox on VLAN 601 with this configuration, and no IP configuration is required on the bridge.

The errors I experienced were when running --ff-only merges

fatal: refusing to merge unrelated histories

If you supply --allow-unrelated-histories, then git errors out with

fatal: Not possible to fast-forward, aborting.

To resolve this, I set fetchDepth to 0 in the checkout step which ensured all commits were fetched and git could perform the merge operation.

The relevant documentation on shallow fetching is available at https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines#shallow-fetch

Except it’s wrong

To be fair to Scripting Guy, he was correct when he wrote that article, but the change in this behaviour is one of the lesser known exciting features of Windows Vista.

So, how does it work now?

Through the magic of procmon, we can figure out what changes are taking place when we change the script host

Et voilà!

Here’s a PowerShell snippet for querying this value

1(Get-ItemProperty registry::HKEY_CLASSES_ROOT\VBSFile\Shell).'(Default)'

Open means WScript, Open2 means CScript.

NetBox 3.1 introduced support for SSO providers via python-social-auth which is fantastic, but integrating this into Authentik was an absolute pain so here’s how I got it working to save you the trouble.

Create your OIDC Provider

1. Click on Applications -> Providers in the Authentik admin UI
2. Click create and select OAuth2/OpenID Provider
3. Give it a meaningful name, I would usually name something like this “NetBox OIDC”
4. Select an authorization flow, if you’re relatively new and are using the out-of-the-box flows, the implicit consent flow is likely what you want
5. Leave Client type set to Confidential
6. Note down the client ID and secret
7. (Optional) I would recommend raising the access code and token validity in the Advanced settings as the defaults are rather aggressive

Create the application in Authentik

1. Click on Applications -> Applications in the Authentik admin UI
2. Click on Create and name your application
3. Note down the slug you use or at least make it something simple (like “netbox”)
4. Associate it with the provider you created just before
5. (Optional) You can grab an SVG of the NetBox logo here

Configure NetBox

For the sake of adding some background, the python-social-auth library is dependant on a component called social-core which is where the actual social backends live. There is no proper implementation of the Authentik backend but it does offer a generic OIDC backend (source) that we can take advantage of.

Caveat emptor: The SSO link will appear as “oidc” on the NetBox login page because the generic OIDC backend has this baked in as its name. Aside from changing the source code or extending the class to create a proper Authentik implementation, I cannot see a way to change this.

1. Open your NetBox configuration.py file in your editor of choice
2. Add the following lines

1REMOTE_AUTH_BACKEND = 'social_core.backends.open_id_connect.OpenIdConnectAuth'
2SOCIAL_AUTH_OIDC_OIDC_ENDPOINT = "https://authentik.example.com/application/o/<slug>"
3SOCIAL_AUTH_OIDC_KEY = '<client ID>'
4SOCIAL_AUTH_OIDC_SECRET = '<secret>'
5
6SOCIAL_AUTH_PROTECTED_USER_FIELDS = ['groups'] # Workaround for an issue where social-auth would die with an error when signing in due to a bug. You will have to assign users to groups because of this.
7SOCIAL_AUTH_REDIRECT_IS_HTTPS = True # Forces HTTPS for redirect URIs. Handy if you're behind a proxy and the schema is wrong.

3. Add python-jose to your local_requirements.txt file
4. Run upgrade.sh to ensure python-jose is installed
5. Restart NetBox

Associating existing NetBox users with social users

When a social user logs in, an association is created in the database that ties their UID to the Django user. You can update these associations at https://netbox.example.com/admin/social_django/usersocialauth/ for the purposes of connecting existing NetBox users with their OIDC sign in.

These SSDs have a variety of different firmwares, you cannot obtain new firmware from Samsung and as a result, you may have issues running the latest builds of ESXi 7.

How can I workaround this?

1. Consult the VMware HCL to figure out what version of ESXi your drive will work on
2. Grab the NVMe VIB from here for your supported version
3. Install the VIB over the top of the existing NVMe VIB
4. Reboot for the changes to take effect

But I’m booting from the NVMe drive!

It’s possible to make a custom ESXi installer with your own VIBs loaded or alternatively you could install onto a USB (unsupported in 8), load a working VIB then migrate your install to the NVMe device using dd.

Recently I successfully bid on a laptop that was recycled by one of the large mining companies, normally these come wiped clean and this one was no exception. I went to install Windows 10 LTSC 2021 and as I progressed through the OOBE, something strange happened.

There is no option to setup offline

Normally you can yank the network cord and setup the machine completely offline with a local account. This one absolutely refused to let me do this in a manner I had never seen before. Its setup was reminiscent of the absolute hell that is the Windows 11 OOBE.

Why?

It turns out this machine was previously enrolled in Intune and had some flag set (probably in the ACPI tables) that changed the OOBE’s behaviour to require an Internet connection to proceed.

After connecting the Internet, it ran through Autopilot whereupon it became apparent that Rio Tinto had not removed the machine from their directory and suddenly I was staring at their logon page.

How’d you get around it?

The trick here is to try and get past the OOBE without connecting to the Internet, otherwise Autopilot will run and you’ll be forced to reinstall.

The easiest way to achieve this is to setup an unattend.xml file that skips the OOBE altogether. In this case, I let Rufus create an unattend.xml file (it’ll prompt you when creating the USB stick, I told it to create a local account) and added some extra properties to the <OOBE> section.

You’ll find the unattend.xml file in \sources\$OEM$\$$\Panther

1<HideEULAPage>true</HideEULAPage>
2<HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
3<HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
4<HideOnlineAccountScreens>true</HideOnlineAccountScreens>
5<SkipMachineOOBE>true</SkipMachineOOBE>