Awesome

"Configuring NIC NVRAM" and "Modifying the NIC driver" steps are needed if 2500BASE-X link between the NIC and SFP ONT is desired instead of 1000BASE-T link. "Rooting the SFP ONT" and "Modifying the SFP ONT serial" steps are mandatory because Telia authenticates ONT by its serial number.

CPU: Intel Core i7-4790K @ 4.00GHz (Cooler Master Hyper 212 EVO cooler)
MB: ASRock H97 Anniversary
RAM: Crucial 8 GiB CT102464BA160B.C16FER
SSD: Corsair Force LS 2.5" 60 GB SATA3
NIC: Dell N20KJ; Broadcom BCM57810 (wan0 and lan0; driver bnx2x; firmware /usr/lib/firmware/bnx2x/bnx2x-e2-7.13.1.0.fw; NF-A4x10 fan)
NIC: integrated; Realtek RTL8111/8168/8411 (lan1; driver r8169; firmware /usr/lib/firmware/rtl_nic/rtl8168g-2.fw)
NIC: TP-LINK TG-3468; Realtek RTL8111/8168/8411 (lan2; driver r8169; firmware /usr/lib/firmware/rtl_nic/rtl8168e-2.fw)
NIC: TP-LINK TG-3468; Realtek RTL8111/8168/8411 (lan3; driver r8169; firmware /usr/lib/firmware/rtl_nic/rtl8168e-2.fw)
WNIC: Asus PCE-AC88; Broadcom BCM4366(v4) (wlan0; driver brcmfmac; firmware /usr/lib/firmware/brcm/brcmfmac4366c-pcie.bin)
LTE modem: Huawei E3372s-153 (hw ver: CL1E3372SM Ver.A; firmware ver: 22.286.03.02.07)
SFP ONT: Huawei MA5671A
PSU: Chieftec CTG-500-80P
case: Chieftec 1E0-500A-CT04

Both the <a href="https://github.com/tonusoo/koduinternet-cpe/blob/main/docs/hw_415752.pdf">Huawei MA5671A SFP ONT</a> and <code>Dell N20KJ</code> NIC using the <code>Broadcom BCM57810</code> chipset support the 2500BASE-X mode. According to <a href="https://github.com/tonusoo/koduinternet-cpe/blob/main/docs/netxtreme_ii.pdf">Broadcom NetXtreme II Network Adapter User Guide</a>, the 2500BASE-X is a term used by Broadcom to describe 2.5 Gbit/s operation, where electricals are leveraged from IEEE 802.3ae-2002 (XAUI).

By default, the Dell N20KJ branded NIC supports 1GigE and 10GigE modes. One needs to configure the NIC's NVRAM with QLogic BCM577xx/BCM578xx diagnostics utility named eDiag in order to enable the 2500BASE-X mode. For example, the NIC can be passed through to a VM where ediag_x64.efi UEFI binary in engineering mode(-b10eng) is executed:

Command for creating the VM in the upper tmux pane was:

# PCIe passthrough requires IOMMU support.
# "OVMF.fd" UEFI is from the "ovmf" package.
qemu-system-x86_64 \
    -name eDiag-VM \
    --enable-kvm \
    -m 1G \
    -bios /usr/share/ovmf/OVMF.fd \
    -serial telnet:127.0.0.1:5016,server,nowait \
    -nographic \
    -drive file=fat:rw:UEFI/uefi_ediag/x64/,format=raw \
    -device vfio-pci,host=01:00.0,rombar=0 \
    -device vfio-pci,host=01:00.1,rombar=0

As seen on the lower tmux pane, the first ports is set to 2.5G mode:

2500BASE-X mode was enabled and set as a default for the first port with eDiag CLI commands below:

device 1
nvm cfg
7
35=70
36=70
56=6
59=6
save
exit

Correct link settings for the first port can be seen below:

bnx2x driver was patched in order to support the 2500BASE-X mode and ignore the Tx fault:

Without ignoring the Tx fault, the Ethernet link would drop down if the fiber to MA5671A SFP ONT is not connected and ONT is not successfully registered by the OLT.

Example of taking the patched driver into use:

For troubleshooting purposes, the driver can be loaded in debug mode with modprobe -v bnx2x debug=0x4102034.

2500BASE-X link between the MA5671A SFP ONT and Dell N20KJ NIC observed from the Linux router:

The same link mode seen from the non-rooted MA5671A SFP ONT minishell with the lanpsg(LAN port status get) command where link_status value 5 means 2500BASE-X:

Link mode checked from the rooted MA5671A SFP ONT:

In case the 2500BASE-X mode is not desired, then Huawei MA5671A SFP ONT links also in 1000BASE-T mode:

Once the link is established, the SFP is accessible at 192.168.1.10 via SSH. User is root and password is admin123.

One option to root the Huawei MA5671A SFP ONT is to short the GND(pin 4) and Serial Data Input(pin 5) of the flash memory and power on the SFP. The SFP will boot into bootrom CLI and one can transfer a modified boot loader(1224ABORT.bin) to flash chip with XMODEM file transfer protocol.

Huawei MA5671A SFP ONT with wires soldered to GND and SI pins:

Huawei MA5671A SFP ONT connected to Reveltronics adapter board:

As seen above, the RXD, TXD and GND of the USB to TTL (3.3V) adapter are connected to transceivers adapter board.

Minicom(115200 8N1) screenshot of the Huawei MA5671A SFP ONT bootup where GND and SI pins of the flash were shorted:

Typing the 7 in the ROM: prompt started the XMODEM file transfer mode and Ctrl + c aborted the autoboot. Setting the variables below will keep the bootloader open after the SFP module restart:

FALCON => setenv bootdelay 5
FALCON => setenv asc 0
FALCON => setenv preboot "gpio input 105;gpio input 106;gpio input 107;gpio input 108;gpio set 3;gpio set 109;gpio set 110;gpio clear 423;gpio clear 422;gpio clear 325;gpio clear 402;gpio clear 424"
FALCON => saveenv

After rebooting the module, the minishell was replaced with ash shell with sed -i "s|/opt/lantiq/bin/minishell|/bin/ash|g" /etc/passwd.

1224ABORT.bin(MD5 10e94a4b4acdc82dec20c7904b69e5c0) is a slightly modified U-Boot version 1.22.4 binary image for MA5671A. Hex dumps diff between the 1224ABORT.bin and unmodified U-boot for this module(Huawei MA5671A is based on Alcatel-Lucent G-010S-P) can be seen below:

Boot messages over serial of the rooted MA5671A SFP ONT can be seen here.

GPON OLT does not register the new SFP ONT if its serial number differs from the serial number of the Huawei HG8010H ONT installed by Telia. In other words, the serial number of the ONT is used for authentication. Here is a screenshot of the OLT CLI from Huawei forum showing the status of the ONT:

The serial of the Huawei HG8010H provided by Telia can be seen from the web interface:

Default username is root and password is admin. One could also log in as a privileged user named telecomadmin with password admintelecom. For example, this allows one to enable telnet access to Huawei HG8010H ONT by downloading its configuration file from the web interface, modifying the TELNETLanEnable parameter and uploading the configuration. The same serial from Huawei HG8010H CLI:

The hex representation of the serial is also printed on the label under the Huawei HG8010H chassis and additionally on its cardboard box.

MA5671A SFP ONT stores the base64 encoded serial number in U-Boot environment(/dev/mtd1 flash partition named uboot_env) variable called "sfp_a2_info". The "sfp_a2_info" variable contains multiple base64 encoded fields/lines which are separated with @ character. For example, the MAC address is on the ninth field and GPON ONT password is on the fifth field. However, only the serial needs to be changed and this is on the sixth, 45 bytes long base64 encoded field:

As seen above, the output of the fw_printenv sfp_a2_info was transferred to r1 Debian machine where the sixth base64 field was decoded to binary and dumped to hex. In order to change the serial number for example from HWTCa1b1c1d1 to HWTCa2b2c2d2, it's necessary to prepare a new value of "sfp_a2_info" variable:

As a final step, the modified "sfp_a2_info" variable needs to be written to /dev/mtd1 with fw_setenv and SFP ONT has to be rebooted so U-Boot reads the updated variable:

As mentioned before, the default password for root user is admin123. It's recommended to back up the /dev/mtd1 and output of fw_printenv before the changes.

MA5671A SFP ONT registration status can be checked with the onu ploamsg command. Value of the curr_state has to be 5 which means O5 or operation-state:

The same O5 value was seen on the screenshot of the Huawei HG8010H ONT web interface. On lower tmux pane the optical Tx and Rx power reported by the MA5671A SFP ONT(plugged into wan0 port) can be seen.

Configuration leans towards traditional or legacy way of doing things, e.g iptables over nftables, udev rules in /etc/udev/rules.d/ for renaming network interfaces instead of /etc/systemd/network/*.link files, etc. This is an intentional design choice.

List of packages installed using apt:

firmware-bnx2x
firmware-realtek
firmware-brcm80211
usb-modeswitch
bridge-utils
radvd
hostapd
libpam-yubico
libpam-google-authenticator

Huawei E3372s-153 USB LTE modem has to be mode switched from 157d(CD/DVD drive) to modem mode which is 14dc for this particular device.

This is taken care of by following udev rule in /usr/lib/udev/rules.d/40-usb_modeswitch.rules file installed by usb-modeswitch-data package:

ATTRS{idVendor}=="12d1", ATTRS{manufacturer}!="Android", ATTR{bInterfaceNumber}=="00", ATTR{bInterfaceClass}=="08", RUN+="usb_modeswitch '/%k'"

The rule above executes the /usr/lib/udev/usb_modeswitch script with a "kernel name" for the device as an argument, e.g /usr/lib/udev/usb_modeswitch 3-2:1.0. The script starts an instance of the /lib/systemd/system/usb_modeswitch@.service, which in turn executes the usb_modeswitch_dispatcher binary with a --switch-mode option for a specified device. usb_modeswitch_dispatcher reads the configuration from the /usr/share/usb_modeswitch/configPack.tar.gz tarball:

root@r1:~# tar -xOzf /usr/share/usb_modeswitch/configPack.tar.gz 12d1:157d
# Huawei E3331, E3372
TargetVendor=0x12d1
TargetProductList="14db,14dc"
HuaweiNewMode=1
root@r1:~#

.. and calls the usb_modeswitch with the content of the appropriate configuration file as a value of the --long-config argument. However, the default configuration does not work if the cdc_mbim driver is loaded. As a workaround, /etc/usb_modeswitch.d/12d1:157d file is created with NoMBIMCheck=1 configuration option added:

root@r1:~# cat /etc/usb_modeswitch.d/12d1:157d
# Huawei E3331, E3372
TargetVendor=0x12d1
TargetProductList="14db,14dc"
HuaweiNewMode=1
NoMBIMCheck=1
root@r1:~#

With the configuration above the usb_modeswitch called by usb_modeswitch_dispatcher makes the switch from 157d to 14dc:

root@r1:~# lsusb -d 12d1:
Bus 003 Device 042: ID 12d1:14dc Huawei Technologies Co., Ltd. E33372 LTE/UMTS/GSM HiLink Modem/Networkcard
root@r1:~#

udev rule named 70-persistent-net.rules ensures the wan*, wwan*, lan* and wlan* interfaces naming convention:

Configuration files: /etc/usb_modeswitch.d/12d1:157d, /etc/udev/rules.d/70-persistent-net.rules

Telia's service works in a way that the Internet traffic is untagged and IPTV traffic is in the VLAN(IEEE 802.1q) 4. Router has a wan0 interface for untagged Internet traffic and a VLAN interface named wan0.4 for IPTV. Both interfaces receive the IPv4 address and netmask via DHCP. wan0 will get a publicly routable IPv4 address and IPTV interface will get a private IPv4 address. In addition, several v4 routes are installed by DHCP. Few /24 publicly routable networks and the 10/8 are routed via the IPTV interface. Default route is via the Internet interface wan0. IPv6 part works in a way that Telia delegates a /56 prefix for LAN hosts from their /32 allocation using a DHCPv6. Internet facing interface will not get an IPv6 address and routing works using the IPv6 link local addresses:

root@r1:~# # v6 default route via VRRP virtual router
root@r1:~# ip -6 r sh default
default via fe80::200:5eff:fe00:1 dev wan0 proto ra metric 1024 expires 3286sec hoplimit 64 pref medium
root@r1:~#

Additionally, the wan0 interface has a manually configured address of 192.168.1.200/24 for accessing the Huawei MA5671A SFP ONT. dhclient enter hook script named get-static-ipv4-addrs and exit hook script named restore-static-ipv4-addrs ensure that this address does not get flushed by dhclient.

lan* and wlan* interfaces are part of the Linux bridge named br0:

root@r1:~# bridge link
2: lan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
3: lan3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 4
4: lan1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 100
6: lan0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br0 state disabled priority 32 cost 100
47: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
root@r1:~#

The br0 interface has a manually configured IPv4 address 192.168.0.1/24. IPv6 address on br0 is managed by dhclient exit hook script.

In addition, there is a wwan0(USB CDC Ethernet device) interface facing the Telia's mobile broadband network with static address of 192.168.8.200/24 and corresponding floating default route:

root@r1:~# ip r sh default dev wwan0
default via 192.168.8.1 metric 100
root@r1:~#

Switching the IPv4 connectivity to mobile network and back to fiber by adjusting the default routes metric is managed by isp-switch systemd service which calls the isp-switch script. IPv6 is not supported by Telia for the mobile broadband service.

LAN hosts get the IPv4 address of the DNS server(dnsmasq running in the router) from DHCPv4 server(also dnsmasq). By default, the domain name server sent to DHCPv4 clients is set to the address of the interface running dnsmasq which is 192.168.0.1:

root@r1:~# ip -4 --brief a sh dev br0
br0              UP             192.168.0.1/24
root@r1:~#

IPv6 address of the DNS server(dnsmasq running in the router) is propagated by NDP "Router Advertisement" messages sent by radvd. RDNSS(Recursive DNS Server) statement in radvd configuration file /etc/radvd.conf is set automatically by the dhclient exit hook script ipv6-pd-br0. This script will use the IPv6 link-local address(found by ip -6 a or mac_to_ip6_ll_addr script) of the br0 interface as a value for the RDNSS statement for radvd:

root@r1:~# ip --brief -6 addr show dev br0 scope link
br0              UP             fe80::a7:29ff:fea6:ec61/64
root@r1:~#
root@r1:~# grep RDNSS /etc/radvd.conf
    RDNSS fe80::a7:29ff:fea6:ec61 { };
root@r1:~#

nodnsupdate dhclient enter hook ensures that local /etc/resolv.conf is not overwritten by DHCP client. As a safety net, modifications of the /etc/resolv.conf were disabled with chattr +i /etc/resolv.conf:

root@r1:~# lsattr /etc/resolv.conf
----i---------e---- /etc/resolv.conf
root@r1:~#

Configuration files and scripts: /etc/network/interfaces, /etc/systemd/system/networking.service.d/override.conf, /etc/dhcp/dhclient.conf, /etc/dhcp/dhclient-exit-hooks.d/ipv6-pd-br0, /usr/local/bin/mac_to_ip6_ll_addr, /etc/dhcp/dhclient-enter-hooks.d/get-static-ipv4-addrs, /etc/dhcp/dhclient-exit-hooks.d/restore-static-ipv4-addrs, /etc/dhcp/dhclient-enter-hooks.d/nodnsupdate, /etc/dhcp/dhclient-enter-hooks.d/unset-dhcp-options, /etc/resolv.conf, /etc/modprobe.d/bnx2x.conf, /etc/sysctl.conf, /etc/systemd/system/isp-switch.service, /usr/local/sbin/isp-switch

Notes on iptables and ip6tables rules:

• dhclient in DHCPv4 mode uses raw sockets(fallback UDP socket for sending unicast packets is also opened) and thus no firewall rule is needed
• important ICMP messages like "frag needed" or "TTL exceeded" are accepted thanks to conntrack "RELATED" state match
• certain ICMP messages sent by the router including "Echo Reply" or "Destination Unreachable" are rate limited by adjusting the kernel parameters in /etc/sysctl.conf
• recent(used in SSH chain) module internals are seen in the /proc/net/xt_recent/SSH file
• while igmpproxy is using raw sockets on the downstream interface, then in regard to upstream interface the igmpproxy simply acts as a normal multicast client(calls setsockopt() with IP_ADD_MEMBERSHIP) and thus there is a need to accept IGMP membership query messages sent by ISP. This also means that the IPTV UDP datagrams are sent towards the router application layer and dropped in the filter table INPUT chain.
• IGMP messages can not be tracked by the conntrack module. At least without an helper module. IGMP messages are sent to IPv4 multicast address and thus the conntrack module expects a reply sourced from a multicast address in order to move from UNREPLIED state to ESTABLISHED state. However, multicast address is never used as a src IP.
• igmpproxy subscribes to 224.0.0.2(IGMP "Leave group" messages) on a downstream interface and packets sent to this address are subject to INPUT chain rules. That's the reason for -A INPUT -d 224.0.0.2/32 -i br0 -p igmp -j ACCEPT rule. Details can be seen in netfilter user mailinglist message.
• dhclient in DHCPv6 mode is able to use ordinary UDP sockets thanks to link-local addresses and does not need to use raw sockets. This means that a firewall rule for DHCPv6 traffic is needed.
• ICMP6 "echo request" messages are rate limited by the limit module. Newer kernel versions have the net.ipv6.icmp.ratemask which would allow to rate limit the replies to "echo request" messages by adjusting the net.ipv6.icmp.ratelimit. ICMP6 "destination unreachable" messages are rate limited according to net.ipv6.icmp.ratelimit.
• important ICMP6 messages like "packet too big" or "time exceeded" or "destination unreachable" are accepted thanks to conntrack "RELATED" state match
• RA messages sent by radvd to ff02::1 multicast addr via LAN-facing interface are looped back by the IP layer for local delivery. This is a default behavior and can be controlled by IPV6_MULTICAST_LOOP(man 7 ipv6). Those messages are dropped.

Rules are loaded by iptables-restore and ip6tables-restore utilities before bringing the br0 bridge interface up and after taking the br0 interface down. This is configred in /etc/network/interfaces file.

Configuration files: /usr/local/etc/IPv4_fw_rules, /usr/local/etc/IPv4_default_fw_rules, /usr/local/etc/IPv6_fw_rules, /usr/local/etc/IPv6_default_fw_rules

As dnsmasq is configured with interface=br0 and bind-interfaces, then setsockopt() with SO_BINDTODEVICE option is called by dnsmasq when started and socket is bound to a br0 interface. Under the hood, the socket is bound to an ifindex of br0 interface. As restarting the networking.service deletes and recreates the network bridge br0 and the new br0 has an incremented ifindex, then dnsmasq.service has to be restarted if the networking.service is restarted:

root@r1:~# systemctl show dnsmasq | grep PartOf=
PartOf=networking.service
root@r1:~#
root@r1:~# systemctl show networking | grep ConsistsOf=
ConsistsOf=hostapd.service igmpproxy.service dnsmasq.service
root@r1:~#

dnsmasq is configured not to read /etc/resolv.conf and use the upstream DNS servers specified in its configuration file:

root@r1:~# grep ^server /etc/dnsmasq.conf
server=2620:fe::fe
server=2620:fe::9
server=9.9.9.9
server=149.112.112.112
root@r1:~#

Configuration files: /etc/dnsmasq.conf, /etc/systemd/system/dnsmasq.service.d/override.conf

radvd configuration and necessary reloads for activating the configuration is automatically handled by dhclient exit hook script and expired prefixes removal script executed by cron at every minute.

The general idea is that it's needed to keep advertising the prefixes with no valid or preferred lifetime left towards LAN at least as long as the last non-zero valid lifetime of the prefix in order to ensure that all the hosts in LAN pick up the prefix deprecation. Such setup ensures that the prefix deprecation is seen even by devices which were for example in suspended to RAM state at the time of the delegated prefix change.

For each "prefix" statement in radvd.conf file the script finds a timestamp in the future when this prefix could be removed from the radvd.conf file in order to avoid stale entries.

Example of /etc/radvd.conf:

Scripts: /etc/dhcp/dhclient-exit-hooks.d/ipv6-pd-br0, /usr/local/sbin/rm-expired-prefixes

Patched(issue #95) version of igmpproxy is installed. igmpproxy_0.2.1-1+relaxed-timeout1_amd64.deb package was built with commands below:

1. apt source igmpproxy
2. cd igmpproxy-0.2.1/
3. patch --verbose src/igmpproxy.c ~/koduinternet-cpe/patches/igmpproxy.patch
4. dch --local +relaxed-timeout with "relaxed pselect() timeout in main loop" note for the changelog
5. cd . && dpkg-buildpackage -b

Package can be installed with dpkg -i igmpproxy_0.2.1-1+relaxed-timeout1_amd64.deb and apt-mark hold igmpproxy prevents the package from being automatically upgraded.

Restarting the networking.service will call the /lib/bridge-utils/ifupdown.sh script which deletes(brctl delbr ..) and recreates(brctl addbr ..) the network bridge br0. However, the new br0 interface will have an incremented ifindex(cat /sys/class/net/br0/ifindex or ip l sh dev br0) and thus the setsockopt() calls by igmpproxy will fail. Example where interface with ifindex 17(21 in octal) no longer existst:

setsockopt(3, SOL_IP, IP_MULTICAST_IF, "\217U\0\0\300\250\0\1\21\0\0\0", 12) = -1 EADDRNOTAVAIL (Cannot assign requested address)

That's the reason why igmpproxy.service is configured to be restarted if networking.service is restarted:

root@r1:~# systemctl show igmpproxy | grep PartOf=
PartOf=networking.service
root@r1:~#
root@r1:~# systemctl show networking | grep ConsistsOf=
ConsistsOf=hostapd.service igmpproxy.service dnsmasq.service
root@r1:~#

As an example, here is the multicast routing table(populated by igmpproxy) and switch multicast group membership table(populated by IGMP snooping) at the time when host connected to lan0 is running mplayer -vf yadif -volume 80 -fs -ao pulse udp://@239.3.1.106:1234:

root@r1:~# ip mroute
(10.0.2.28,239.3.1.106)          Iif: wan0.4     Oifs: br0  State: resolved
root@r1:~#
root@r1:~# bridge -s mdb
33: br0  lan0  239.3.1.106  temp   251.57
root@r1:~#

IGMP snooping is enabled by default:

root@r1:~# cat /sys/devices/virtual/net/br0/bridge/multicast_snooping
1
root@r1:~#

force_igmp_version for wan0.4 is not touched, i.e it's in IGMPv3 mode and falls back to IGMPv1/v2 mode if needed.

Configuration files: /etc/igmpproxy.conf, /etc/systemd/system/igmpproxy.service.d/override.conf

By default, the dhclient /etc/dhcp/dhclient-exit-hooks.d/timesyncd script builds the /run/systemd/timesyncd.conf.d/01-dhclient.conf configuration file for systemd-timesyncd SNTP client and instructs it to use the NTP servers provided by Telia's DHCP server. Instead, the default Debian NTP pool compiled to systemd-timesyncd is used by creating a /etc/systemd/timesyncd.conf.d/01-dhclient.conf symlink pointing to /dev/null. Output of timedatectl:

Asus PCE-AC88(Broadcom BCM4366/4 SoC) WNIC is configured to work as IEEE 802.11a/n/ac AP. For the IEEE 802.11ac mode, the channel width is up to 80 MHz(5170 - 5250 MHz). hostapd adds the WNIC named wlan0 to bridge br0. Connectivity between the Wi-Fi clients is allowed. Output of hostapd_cli status and get_config commands can be seen below:

Firmware blob loaded by brcmfmac driver is brcmfmac4366c-pcie.bin:

root@r1:~# md5sum /lib/firmware/brcm/brcmfmac4366c-pcie.bin
e3bb4457082aa54769253e9168db2269  /lib/firmware/brcm/brcmfmac4366c-pcie.bin
root@r1:~#
root@r1:~# strings /lib/firmware/brcm/brcmfmac4366c-pcie.bin | tail -1
4366c0-roml/pcie-ag-splitrx-fdap-mbss-mfp-wnm-osen-wl11k-wl11u-txbf-pktctx-amsdutx-ampduretry-chkd2hdma-proptxstatus-11nprop-obss-dbwsw-ringer-dmaindex16-bgdfs-murx-wwdfs Version: 10.28.2 (r769115) CRC: 6924533a Date: Mon 2018-11-05 03:22:36 PST Ucode Ver: 1128.17924 FWID: 01-d2cbb8fd
root@r1:~#

This is provided by firmware-brcm80211 package, but the same firmware version can also be downloaded from Linux kernel git repo:

root@r1:~# curl -s https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain/brcm/brcmfmac4366c-pcie.bin | md5sum
e3bb4457082aa54769253e9168db2269  -
root@r1:~#

For firmware related troubleshooting purposes, the brcmfmac driver can be loaded with modprobe -v brcmfmac debug=0x1416 when built with debugging support:

Configuration files: /etc/hostapd/hostapd.conf, /etc/systemd/system/hostapd.service.d/override.conf

Diff with Debian 10 default Linux PAM common-auth enabling 2FA(user password and OTP):

root@r1:~# diff -u /etc/pam.d/common-auth{~,}
--- /etc/pam.d/common-auth~     2023-06-01 21:22:43.894184345 +0300
+++ /etc/pam.d/common-auth      2023-06-01 21:48:15.777645409 +0300
@@ -13,8 +13,16 @@
 # pam-auth-update to manage selection of other modules.  See
 # pam-auth-update(8) for details.

+# If pam_yubico.so returns PAM_SUCCESS, then pam_google_authenticator.so is skipped.
+# All other return codes are ignored and pam_google_authenticator.so is used.
+# pam_yubico.so has "forward_pass"(pam_set_item(pamh, PAM_AUTHTOK, onlypasswd)) enabled by default.
+auth   [success=1 default=ignore]      pam_yubico.so id=12345 authfile=/etc/authorized_yubikeys urllist=https://api.yubico.com/wsapi/2.0/verify
+
+# pam_google_authenticator.so is able to detect if user password is entered before TOTP.
+auth   required        pam_google_authenticator.so forward_pass
+
 # here are the per-package modules (the "Primary" block)
-auth   [success=1 default=ignore]      pam_unix.so nullok_secure
+auth   [success=1 default=ignore]      pam_unix.so nullok_secure try_first_pass
 # here's the fallback if no module succeeds
 auth   requisite                       pam_deny.so
 # prime the stack with a positive return value if there isn't one already;
root@r1:~#

YubiKeys token IDs are mapped to username in /etc/authorized_yubikeys file. $HOME/.google_authenticator is created by google-authenticator utility and it includes the secret key, emergency scratch codes, rate-limit configuration, etc. SSH keyboard interactive authentication is enabled in /etc/ssh/sshd_config by changing the value of ChallengeResponseAuthentication to yes.

Configuration files: /etc/pam.d/common-auth

Statistics of wan0 when host connected to lan2 was running a TCP bidirectional test against iperf3 server in another ISP network:

Statistics of all the interfaces when hosts connected to lan2 and lan3 were running a TCP bidirectional test against iperf3 server in another ISP network and smartphone connected to Wi-Fi was running Ookla's Speedtest:

Speedtest.net results when executed from a host connected to lan2:

Speedtest.net results when executed from a smartphone connected to r1 Wi-Fi network:

Stock ADDA AD0412HB-K96 fan was replaced with silent, Noctua NF-A4x10 fan:

Noctua NF-A4x10 is connected to motherboard chassis fan connector where it is picked up by lm-sensors:

Dell N20KJ fan fault detection was disabled in eDiag engineering mode(ediag_x64.efi -b10eng) with commands below:

nvm cfg
option 4 (board i/o)
83=1 (disabled)
save
exit

Users upnatom and JAMESMTL in DSLReports forum "Bypassing the HH3K up to 2.5Gbps using a BCM57810S NIC" thread. User anon23891239 and others in OpenWRT forum "Support MA5671A SFP GPON" thread. README is inspired by "Human-Activity-Recognition" repo.

MIT License