With the weekend maintenance window agreed, begrudgingly, by the internal customers at home I set about replacing the EdgeRouter we last visited on https://itskimpossible.blog/farewell-my-little-edgerouter/ with my frankenrouter Futro S920 OPNsense.
First of all, before pulling the connection down, I booted up the S920 with the power consumption monitor to see how things looked. Initially shocked at 30W being drawn, it promptly settled down to 16W once OPNsense had loaded up. It went down a little further with PowerNowD but still a good deal higher than the mighty EdgeRouter at ~8W.
I was expecting an increase, driving the 2x10GbE ports on the PCI-E riser card was going to be heavy so I made sure to disable the onboard audio, onboard NIC, and anything else I could find in the BIOS that didn’t seem necessary.
I could live with the 16W because I happened to read that Unbound DNS as part of OPNsense could effectively replace the pi-hole VM that runs 24/7 so maybe I’d save a few pennies there.
Several headless tests with no monitor or keyboard connected to ensure a smooth boot up we were off the races…!
I fed the power adapter cabling through the necessary journey and began my cutover, with an ISP modem reboot thrown in because of the change of mac address behind it (had no reason to try and spoof my mac address, had visions of confusion later if ever the EdgeRouter was brought back!) it would need a reboot to issue a new DHCP IP.
Within a few minutes we were back online, and then I proceeded to play with the settings in OPNsense marking the environment ‘at risk’ for a while longer. I went through setting up Unbound DNS, enabling a few blocklists I recognised then happened upon one I didn’t recognise https://oisd.nl/ – a quick de-tour for a read there and decided I quite liked the look of it so enabled that and moved on.
As I was nearing the end of my configuration I realised I hadn’t carried out the obligatory speed tests…. and then disaster struck, a very very slow speedtest. Initially I put it down to a bad mirror, nope, next mirror had the same problem – urgh. We’d gone from a happy enough 900Mbit/s+ download to 80Mbit/s (so close to 100Mbit/s but it didn’t register at the time – spoiler alert, it was relevant!)
I went around and confirmed various settings, flicking them on and off with a ‘short disruption’ to service each time:
- Poor performance from power saving by PowerNowD? Was on ‘adaptive’ which should be fairly balanced on performance vs power draw ramping up and down as necessary. Turned it off, no difference.
- Had a dig around the OPNsense UI remembering I’d seen some disabled hardware offload settings.
I flicked them off and back one by one, no difference. Getting desperate now.. the internal customers are demanding a swift return.
I started looking into the network card interface configuration to look for crc errors, drops, re-tries etc – nothing.
Hmmm… then it dawned on me, how are we looking at the physical layer… ah yes, it would appear we have found our problem…
The connection between the frankenrouter and the ISP modem was stubborn and negotiated 100mbit/s. It was starting to make sense why everything was indeed running at around 100mbit/s because it genuinely was.
Eureka I cried (much more expletive words were uttered instead), maybe I’d damaged the cables when moving them over. A quick look in the spares box and a brand new CAT6 cable was fished out, and plugged in…. still… 100mbit/s. Some different but equally frustrating words were used at this time.
Quickly plugged the EdgeRouter back in to the ISP modem and 1Gbps was negotiated, the little green LED mocking me.
Back to the frankenrouter I decided to force 1Gbps without negotiation to see if that would help. Nope, now the link would just stay down on both sides. At least the green LED wasn’t there anymore.
Off I went in search of why it seemed OPNsense or the Futro S920+10GbE network cards were conspiring against me.
I quickly found some very specific to the exact Intel card driver I was using ‘ix’ and PfSense having a negotiation problem – https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#intel-ix-4-cards This sounded like what I was seeing, except despite trying the various sysctl commands to restrict/change the advertised speeds no success. But at least we had found something that was close to our experience.
Further on I found I was not alone in my strange experience when I happened upon a forum post with someone having a similar experience. https://forum.netgate.com/topic/173897/pfsense-22-05-ix-driver-vs-intel-ix-kmod ‘ the card, at least one port, ix0, connected to ISP LAN port became unstable — sometimes port set it speed to 100mbit instead of 1000, and sometimes it just blinked like can’t detect link speed in endless loop‘
This clever chap had found a solution that worked for him – a new driver! https://www.freshports.org/net/intel-ix-kmod/ I was almost giddy by this point at the prospect of this annoying setback being behind me. Attempts to follow the commands on the website however wasn’t working properly because OPNsense doesn’t include ports or kernel source by default – it is an appliance and I absolutely understand why they’d want to keep as clean a build as possible.
A quick detour to the OPNsense documentation https://docs.opnsense.org/manual/software_included.html#the-ports-tree where two magic commands were found:
opnsense-code ports
opnsense-code src
(this took a frustratingly long time (a small number of minutes) over the 100mbit/s connection!)
Once in place, the commands from the website worked!
cd /usr/ports/net/intel-ix-kmod/ && make install clean
We now have our driver, the last step in the instructions was to add if_ix_updated_load=”YES” into /boot/loader.conf – did that and waited the longest minute for a headless reboot. Once booted back up I could see in sysctl that whilst the driver version was indeed intel-ix-kmod-3.3.35_2 we were unfortunately still connecting at 100mbit/s. I found a forum post that recommended against using /boot/loader.conf with OPNsense as it does not persist upgrades, so I added the same into the Tuneables section to no success either.
Beaten at this point I retreated, to re-group and have another look when the family is more understanding…
The EdgeRouter lives on!