Linksys E8450: Difference between revisions

From Leo's Notes
This page was last edited on 30 September 2022, at 05:28.
No edit summary
No edit summary
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
The Linksys E8450 is a Wi-Fi 6 (802.11ax) router. This router is based on MediaTek MT7622 SOC and a MT7531 switch. As of 2022, it is supported by a snapshot release of OpenWRT.
The Linksys E8450 is a Wi-Fi 6 (802.11ax) router. This router is based on MediaTek MT7622 SOC and a MT7531 switch. As of 2022, it is supported by [[OpenWRT]] 22.03.


I purchased this switch on sale from Amazon for $109 CAD in mid 2022.
I purchased this switch on sale from Amazon for $109 and later $90 CAD in mid 2022.


== Installing OpenWRT ==
== OpenWRT ==
If you are installing OpenWRT from the stock Linksys firmware, please refer to the instructions outlined at: https://github.com/dangowrt/owrt-ubi-installer


=== Installing OpenWRT ===
If you are installing [[OpenWRT]] from the stock Linksys firmware, please refer to the instructions outlined by Daniel at https://github.com/dangowrt/owrt-ubi-installer.
[[File:Linksys E8450 Wizard.png|alt=Linksys E8450 Wizard|thumb|217x217px|Linksys E8450 Wizard]]
==== Login to stock firmware ====
The steps that I used to install OpenWRT are as follows:
The steps that I used to install OpenWRT are as follows:


# Power on the device, navigate to http://192.168.1.1.
# Power on the device, navigate to http://192.168.1.1.
# Decline the terms and conditions and quit the setup wizard
# Click on the radio button and click 'Cancel' to quit the setup wizard.
# Login as 'admin' / 'admin'
#Login using 'admin' as the password.
# Go to Administration -> Firmware Upgrade
[[File:Linksys E8450 Flash Firmware.png|alt=Linksys E8450 Flash Firmware|thumb|220x220px|Linksys E8450 Flash Firmware]]
# Upload openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery.itb (Not the installer)
 
# Once the device reboots, you should be able to see OpenWRT at 192.168.1.1
==== Backup flash data ====
# SSH to OpenWRT and back up the flash.
#Ensure that you are using version 1.0 of the Linksys firmware. You might want to consider downgrading if you are on a newer version.
#Go to Configuration -> Administration -> Firmware Upgrade
# Upload [https://downloads.openwrt.org/releases/22.03.0/targets/mediatek/mt7622/openwrt-22.03.0-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery.itb linksys_e8450-ubi-initramfs-recovery.itb] (Note: '''Not the installer''', we want the initramfs one to do a flash backup only.)
# The firmware update page will reload once the device reboots. You should be able to see OpenWRT at 192.168.1.1
# Backup the flash contents in case you need to restore to the stock firmware. (Note: You'll likely need serial access to restore, which requires soldering wires onto the board). SSH to root@192.168.1.1 and dump the flash:
{{Highlight
{{Highlight
| code = # cd /dev
| code = # cd /dev
# for part in mtd[0123] ; do dd if=$part of=/tmp/$part; done
# for part in mtd[0123] ; do dd if=$part of=/tmp/$part; done
| lang = terminal
| lang = terminal
}} Copy the resulting mtd files to your computer in case you ever need it.
}} Copy the resulting mtd files to your computer. There should be 4 files each being 524288, 1310720, 1048576, and 131072000 bytes large.
# Reboot the router. I had to power cycle the device a few times before it stopped going into OpenWRT.
 
# Back on the original Linksys firmware, upload the UBI based installer openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb.  
If you didn't do this step, a copy of the first 3 and a partial of the last mtd block are saved on the <code>ubi0_3</code> device by the OpenWRT installer / conversion script (in the next step below). You can get these by mounting <code>ubi0_3</code> with: <code>mount -t ubifs -o ro /dev/ubi0_3 /mnt/</code>.  
# Once the device reboots, you should be able to see OpenWRT at 192.168.1.1
 
# Upgrade the OpenWRT firmware using the UBI sysupgrade file. Go to System -> Flash Firmware, under "Flash new firmware image", upload openwrt-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb.
==== Installing OpenWRT for real ====
# Once the device reboots, you should see OpenWRT again.
# Reboot the router 3 times until it boots back into the original Linksys firmware. You may need to power cycle the device multiple times. Do not try anything else other than rebooting / powercycling.
# On the original Linksys firmware, go to Configuration -> Administration -> Firmware Upgrade
#Upload the UBI based installer [https://github.com/dangowrt/owrt-ubi-installer/releases/download/v1.0.0/openwrt-22.03.0-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb]. This installer will set up the UBI based filesystem by executing a [https://github.com/dangowrt/owrt-ubi-installer/blob/main/files/installer/install.sh conversion script].
# The firmware update page will reload once the device reboots. You should be able to see OpenWRT at 192.168.1.1
# Upgrade the OpenWRT firmware using the UBI sysupgrade file. Go to System -> Flash Firmware, under "Flash new firmware image", upload [https://github.com/dangowrt/owrt-ubi-installer/releases/download/v1.0.0/openwrt-22.03.0-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb openwrt-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb]. No need to save any configs or package lists.
# Once the device reboots, you should see OpenWRT again. This time, when you log in, it should not say it is running in initramfs / recovery mode.
 
=== Configuring OpenWRT ===
Log in to OpenWRT and navigate to System -> Startup -> Local Startup. Edit <code>/etc/rc.local</code> with the following contents. These changes are to:
 
# Prevent the system from locking up when the iwinfo library tries to read the now invalid <code>/dev/mtdblock2</code> device
# To make sure the device doesn't hang on boot due to the CPU clock being too low (defaults to 300MHz), resulting in "the CPU supply voltage [set] to 0.9V which causes DDR RAM calibration in ARM Trusted Firmware <code>bl2</code> to never complete and hang forever". <ref>https://forum.openwrt.org/t/belkin-rt3200-linksys-e8450-wifi-ax-discussion/94302/1486</ref> 437.5MHz apparently is what you need.
# Change the CPU governor to either ondemand or schedutil to allow CPU to clock down when idle (to slightly reduce power usage)
{{Highlight
| code = # Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
 
# libiwinfo will try to read /dev/mtdblock*, but mtdblock2 is invalid and can hang up everything.
mv /dev/mtdblock2 /dev/mtdblock2a
 
# Defaults to 300000
echo 437500 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
# Defaults to userspace
echo schedutil > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
 
exit 0
| lang = terminal
}}


== Upgrading OpenWRT ==
=== Upgrading OpenWRT ===
If you installed OpenWRT using the steps above (ie. OpenWRT with the UBI filesystem converted), you'll upgrade the system via OpenWRT using a UBI-enabled sysupgrade file. For example: <code>openwrt-22.03-snapshot-r19451-2726c8c315-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb</code>.
After you have installed OpenWRT with the steps above, you can upgrade to newer versions of OpenWRT by uploading a UBI-compatible sysupgrade file from:
 
* https://github.com/dangowrt/owrt-ubi-installer/releases
* https://downloads.openwrt.org/releases/22.03.0/targets/mediatek/mt7622/
 
Make sure you select the <code>ubi-squashfs-upgrade.itb</code> file. Eg. <code>openwrt-22.03-snapshot-r19451-2726c8c315-mediatek-mt7622-linksys_e8450-'''ubi-squashfs-sysupgrade'''.itb</code>.


Note that you'll lose any installed packages once you do this upgrade. If you have any specific kernel modules or programs installed, you'll have to reinstall them again. The configs however should remain persistent.
Note that you'll lose any installed packages once you do this upgrade. If you have any specific kernel modules or programs installed, you'll have to reinstall them again. The configs however should remain persistent.
== Troubleshooting ==
=== blk_update_request I/O errors on mtdblock2 ===
The Linksys E8450 running OpenWRT started stalling out and reporting lots of blk_update_request I/O errors on <code>/dev/mtdblock2</code>. This started to happen recently, perhaps after adding more devices to the network. When the stall happens, WiFi connections just stop working entirely on both 2.4 and 5GHz. Luci, the web control panel, also doesn't respond.
'''Solution / Update''': Move the /dev/mtdblock2 device to /dev/mtdblock2a (or some other name) so that the iwinfo library won't try reading it and hang up the system.
My original troubleshooting log follows below..
Checking on the kernel dmesg logs show that some sectors are failing to write:
{{Highlight
| code = [355545.032552] blk_update_request: I/O error, dev mtdblock2, sector 256 op 0x0:(READ) flags 0x84700 phys_seg 32 prio class 0
[355545.045135] blk_update_request: I/O error, dev mtdblock2, sector 264 op 0x0:(READ) flags 0x84700 phys_seg 31 prio class 0
[355545.056711] blk_update_request: I/O error, dev mtdblock2, sector 272 op 0x0:(READ) flags 0x84700 phys_seg 30 prio class 0
[355545.068156] blk_update_request: I/O error, dev mtdblock2, sector 280 op 0x0:(READ) flags 0x84700 phys_seg 29 prio class 0
[355545.079640] blk_update_request: I/O error, dev mtdblock2, sector 288 op 0x0:(READ) flags 0x84700 phys_seg 28 prio class 0
[355545.091038] blk_update_request: I/O error, dev mtdblock2, sector 296 op 0x0:(READ) flags 0x84700 phys_seg 27 prio class 0
[355545.102418] blk_update_request: I/O error, dev mtdblock2, sector 304 op 0x0:(READ) flags 0x84700 phys_seg 26 prio class 0
[355545.113821] blk_update_request: I/O error, dev mtdblock2, sector 312 op 0x0:(READ) flags 0x84700 phys_seg 25 prio class 0
[355545.125211] blk_update_request: I/O error, dev mtdblock2, sector 320 op 0x0:(READ) flags 0x84700 phys_seg 24 prio class 0
[355545.136592] blk_update_request: I/O error, dev mtdblock2, sector 328 op 0x0:(READ) flags 0x84700 phys_seg 23 prio class 0
[355545.155734] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.164051] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.172324] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.180307] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.188286] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.196255] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.204253] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.213419] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.221699] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.231766] Buffer I/O error on dev mtdblock2, logical block 32, async page read
| lang = text
}}And the kernel logs just continue erroring with the Buffer I/O error message.
For some reason, <code>/sbin/rpcd -s /var/run/ubus/ubus.sock -t 30</code> was using 25% CPU. Killing it brought back Luci?
<code>/dev/mtdblock2</code> really can't be read:
{{Highlight
| code = root@linksys:/# dd if=/dev/mtdblock2  of=/tmp/x conv=noerror
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
...
root@linksys:/# dmesg
[149324.472780] print_req_error: 3434 callbacks suppressed
[149324.472788] blk_update_request: I/O error, dev mtdblock2, sector 256 op 0x0:(READ) flags 0x84700 phys_seg 26 prio class 0
[149324.489773] blk_update_request: I/O error, dev mtdblock2, sector 264 op 0x0:(READ) flags 0x84700 phys_seg 25 prio class 0
[149324.501309] blk_update_request: I/O error, dev mtdblock2, sector 272 op 0x0:(READ) flags 0x84700 phys_seg 24 prio class 0
[149324.513110] blk_update_request: I/O error, dev mtdblock2, sector 280 op 0x0:(READ) flags 0x84700 phys_seg 23 prio class 0
[149324.524642] blk_update_request: I/O error, dev mtdblock2, sector 288 op 0x0:(READ) flags 0x84700 phys_seg 22 prio class 0
[149324.536136] blk_update_request: I/O error, dev mtdblock2, sector 296 op 0x0:(READ) flags 0x84700 phys_seg 21 prio class 0
[149324.547439] blk_update_request: I/O error, dev mtdblock2, sector 304 op 0x0:(READ) flags 0x84700 phys_seg 20 prio class 0
[149324.558870] blk_update_request: I/O error, dev mtdblock2, sector 312 op 0x0:(READ) flags 0x84700 phys_seg 19 prio class 0
[149324.570155] blk_update_request: I/O error, dev mtdblock2, sector 320 op 0x0:(READ) flags 0x84700 phys_seg 18 prio class 0
[149324.581596] blk_update_request: I/O error, dev mtdblock2, sector 328 op 0x0:(READ) flags 0x84700 phys_seg 17 prio class 0
[149324.604720] buffer_io_error: 3425 callbacks suppressed
[149324.604728] Buffer I/O error on dev mtdblock2, logical block 32, async page read
| lang = terminal
}}
Updated from OpenWRT 22.03-SNAPSHOT to 22.03.0 but still can't read <code>mtdblock2</code>. No USB devices are mounted, in case that's relevant.
Looking through this very unwieldy discussion https://forum.openwrt.org/t/belkin-rt3200-linksys-e8450-wifi-ax-discussion/94302, the developer (daniel) says "This error can show when trying to access areas in the flash with broken OOB/ECC data." and that mtdblock2 is the factory partition containing calibration data.
Restarting the 5GHz WiFi will trigger this error, despite me trying to make <code>/dev/mtdblock2</code> chmod <code>000</code>.
<code>lsof</code> shows the <code>rpcd</code> process having <code>/dev/mtdblock2</code> open:
{{Highlight
| code = root@linksys:~# lsof -p 3422
COMMAND  PID USER  FD      TYPE            DEVICE SIZE/OFF                NODE NAME
rpcd    3422 root  cwd      DIR              0,18      352                  67 /
rpcd    3422 root  rtd      DIR              0,18      352                  67 /
rpcd    3422 root  txt      REG              0,18    66139 9223372036854776300 /sbin/rpcd
rpcd    3422 root  mem      REG              259,1                          492 /sbin/rpcd (path dev=0,18, inode=9223372036854776300)
rpcd    3422 root  mem      BLK              31,8                          54 /dev/mtdblock2
rpcd    3422 root  mem      REG              259,1                        1122 /usr/lib/rpcd/rrdns.so (path dev=0,18, inode=9223372036854776930)
rpcd    3422 root  mem      REG              259,1                        1121 /usr/lib/rpcd/luci.so (path dev=0,18, inode=9223372036854776929)
rpcd    3422 root  mem      REG              259,1                          599 /usr/lib/libnl-tiny.so (path dev=0,18, inode=9223372036854776407)
rpcd    3422 root  mem      REG              259,1                          583 /usr/lib/libiwinfo.so.20210430 (path dev=0,18, inode=9223372036854776391)
rpcd    3422 root  mem      REG              259,1                        1120 /usr/lib/rpcd/iwinfo.so (path dev=0,18, inode=9223372036854776928)
rpcd    3422 root  mem      REG              259,1                        1119 /usr/lib/rpcd/file.so (path dev=0,18, inode=9223372036854776927)
rpcd    3422 root  mem      REG              259,1                          318 /lib/libgcc_s.so.1 (path dev=0,18, inode=9223372036854776126)
rpcd    3422 root  mem      REG              259,1                          588 /usr/lib/libjson-c.so.5.1.0 (path dev=0,18, inode=9223372036854776396)
rpcd    3422 root  mem      REG              259,1                          315 /lib/libblobmsg_json.so.20220515 (path dev=0,18, inode=9223372036854776123)
rpcd    3422 root  mem      REG              259,1                          325 /lib/libuci.so (path dev=0,18, inode=9223372036854776133)
rpcd    3422 root  mem      REG              259,1                          324 /lib/libubus.so.20220601 (path dev=0,18, inode=9223372036854776132)
rpcd    3422 root  mem      REG              259,1                          323 /lib/libubox.so.20220515 (path dev=0,18, inode=9223372036854776131)
rpcd    3422 root  mem      REG              259,1                          316 /lib/libc.so (path dev=0,18, inode=9223372036854776124)
rpcd    3422 root    0r      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    1w      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    2w      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    3u  a_inode                0,8        0                  74 [eventpoll:4,6]
rpcd    3422 root    4r    FIFO                0,7      0t0              11701 pipe
rpcd    3422 root    5w    FIFO                0,7      0t0              11701 pipe
rpcd    3422 root    6u    unix 0x000000000aec3471      0t0              11702 type=STREAM
rpcd    3422 root    7u  netlink                        0t0              15712 GENERIC
rpcd    3422 root    8r      BLK              31,8      0t0                  54 /dev/mtdblock2
| lang = terminal
}}
This is likely what's causing things to hang since rpcd is used for handling other things like WiFi connections or luci.
Searching through the entire filesystem for <code>mtdblock2</code> returned nothing. The discussion mentioned that iwinfo might be reading <code>mtdblock2</code>. There is a string '<code>mtdblock%d</code>' in the <code>libiwinfo.so.20210430</code> library. The source for this does indeed show it try to read <code>/dev/mtdblockN</code> as seen here: https://git.openwrt.org/?p=project/iwinfo.git;a=blob_plain;f=iwinfo_utils.c;hb=3d38d1d17eba37b0985dd2857b4539410824a3e6 in order to determine the hardware information.
Perhaps a simple fix here would be to rename <code>/dev/mtdblock2</code> to <code>/dev/mtdblock2a</code> so it can't try reading from the device? Basically, <code>mv /dev/mtdblock2 /dev/mtdblock2a</code>


== See also ==
== See also ==


* https://openwrt.org/toh/linksys/e8450
* https://openwrt.org/toh/linksys/e8450

Latest revision as of 05:28, 30 September 2022

The Linksys E8450 is a Wi-Fi 6 (802.11ax) router. This router is based on MediaTek MT7622 SOC and a MT7531 switch. As of 2022, it is supported by OpenWRT 22.03.

I purchased this switch on sale from Amazon for $109 and later $90 CAD in mid 2022.

OpenWRT

Installing OpenWRT

If you are installing OpenWRT from the stock Linksys firmware, please refer to the instructions outlined by Daniel at https://github.com/dangowrt/owrt-ubi-installer.

Linksys E8450 Wizard
Linksys E8450 Wizard

Login to stock firmware

The steps that I used to install OpenWRT are as follows:

  1. Power on the device, navigate to http://192.168.1.1.
  2. Click on the radio button and click 'Cancel' to quit the setup wizard.
  3. Login using 'admin' as the password.
Linksys E8450 Flash Firmware
Linksys E8450 Flash Firmware

Backup flash data

  1. Ensure that you are using version 1.0 of the Linksys firmware. You might want to consider downgrading if you are on a newer version.
  2. Go to Configuration -> Administration -> Firmware Upgrade
  3. Upload linksys_e8450-ubi-initramfs-recovery.itb (Note: Not the installer, we want the initramfs one to do a flash backup only.)
  4. The firmware update page will reload once the device reboots. You should be able to see OpenWRT at 192.168.1.1
  5. Backup the flash contents in case you need to restore to the stock firmware. (Note: You'll likely need serial access to restore, which requires soldering wires onto the board). SSH to root@192.168.1.1 and dump the flash:
# cd /dev
# for part in mtd[0123] ; do dd if=$part of=/tmp/$part; done

Copy the resulting mtd files to your computer. There should be 4 files each being 524288, 1310720, 1048576, and 131072000 bytes large.

If you didn't do this step, a copy of the first 3 and a partial of the last mtd block are saved on the ubi0_3 device by the OpenWRT installer / conversion script (in the next step below). You can get these by mounting ubi0_3 with: mount -t ubifs -o ro /dev/ubi0_3 /mnt/.

Installing OpenWRT for real

  1. Reboot the router 3 times until it boots back into the original Linksys firmware. You may need to power cycle the device multiple times. Do not try anything else other than rebooting / powercycling.
  2. On the original Linksys firmware, go to Configuration -> Administration -> Firmware Upgrade
  3. Upload the UBI based installer openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb. This installer will set up the UBI based filesystem by executing a conversion script.
  4. The firmware update page will reload once the device reboots. You should be able to see OpenWRT at 192.168.1.1
  5. Upgrade the OpenWRT firmware using the UBI sysupgrade file. Go to System -> Flash Firmware, under "Flash new firmware image", upload openwrt-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb. No need to save any configs or package lists.
  6. Once the device reboots, you should see OpenWRT again. This time, when you log in, it should not say it is running in initramfs / recovery mode.

Configuring OpenWRT

Log in to OpenWRT and navigate to System -> Startup -> Local Startup. Edit /etc/rc.local with the following contents. These changes are to:

  1. Prevent the system from locking up when the iwinfo library tries to read the now invalid /dev/mtdblock2 device
  2. To make sure the device doesn't hang on boot due to the CPU clock being too low (defaults to 300MHz), resulting in "the CPU supply voltage [set] to 0.9V which causes DDR RAM calibration in ARM Trusted Firmware bl2 to never complete and hang forever". [1] 437.5MHz apparently is what you need.
  3. Change the CPU governor to either ondemand or schedutil to allow CPU to clock down when idle (to slightly reduce power usage)
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

# libiwinfo will try to read /dev/mtdblock*, but mtdblock2 is invalid and can hang up everything.
mv /dev/mtdblock2 /dev/mtdblock2a

# Defaults to 300000
echo 437500 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
# Defaults to userspace
echo schedutil > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor

exit 0

Upgrading OpenWRT

After you have installed OpenWRT with the steps above, you can upgrade to newer versions of OpenWRT by uploading a UBI-compatible sysupgrade file from:

Make sure you select the ubi-squashfs-upgrade.itb file. Eg. openwrt-22.03-snapshot-r19451-2726c8c315-mediatek-mt7622-linksys_e8450-ubi-squashfs-sysupgrade.itb.

Note that you'll lose any installed packages once you do this upgrade. If you have any specific kernel modules or programs installed, you'll have to reinstall them again. The configs however should remain persistent.

Troubleshooting

blk_update_request I/O errors on mtdblock2

The Linksys E8450 running OpenWRT started stalling out and reporting lots of blk_update_request I/O errors on /dev/mtdblock2. This started to happen recently, perhaps after adding more devices to the network. When the stall happens, WiFi connections just stop working entirely on both 2.4 and 5GHz. Luci, the web control panel, also doesn't respond.

Solution / Update: Move the /dev/mtdblock2 device to /dev/mtdblock2a (or some other name) so that the iwinfo library won't try reading it and hang up the system.

My original troubleshooting log follows below..

Checking on the kernel dmesg logs show that some sectors are failing to write:

[355545.032552] blk_update_request: I/O error, dev mtdblock2, sector 256 op 0x0:(READ) flags 0x84700 phys_seg 32 prio class 0
[355545.045135] blk_update_request: I/O error, dev mtdblock2, sector 264 op 0x0:(READ) flags 0x84700 phys_seg 31 prio class 0
[355545.056711] blk_update_request: I/O error, dev mtdblock2, sector 272 op 0x0:(READ) flags 0x84700 phys_seg 30 prio class 0
[355545.068156] blk_update_request: I/O error, dev mtdblock2, sector 280 op 0x0:(READ) flags 0x84700 phys_seg 29 prio class 0
[355545.079640] blk_update_request: I/O error, dev mtdblock2, sector 288 op 0x0:(READ) flags 0x84700 phys_seg 28 prio class 0
[355545.091038] blk_update_request: I/O error, dev mtdblock2, sector 296 op 0x0:(READ) flags 0x84700 phys_seg 27 prio class 0
[355545.102418] blk_update_request: I/O error, dev mtdblock2, sector 304 op 0x0:(READ) flags 0x84700 phys_seg 26 prio class 0
[355545.113821] blk_update_request: I/O error, dev mtdblock2, sector 312 op 0x0:(READ) flags 0x84700 phys_seg 25 prio class 0
[355545.125211] blk_update_request: I/O error, dev mtdblock2, sector 320 op 0x0:(READ) flags 0x84700 phys_seg 24 prio class 0
[355545.136592] blk_update_request: I/O error, dev mtdblock2, sector 328 op 0x0:(READ) flags 0x84700 phys_seg 23 prio class 0
[355545.155734] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.164051] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.172324] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.180307] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.188286] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.196255] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.204253] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.213419] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.221699] Buffer I/O error on dev mtdblock2, logical block 32, async page read
[355545.231766] Buffer I/O error on dev mtdblock2, logical block 32, async page read

And the kernel logs just continue erroring with the Buffer I/O error message.

For some reason, /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30 was using 25% CPU. Killing it brought back Luci?

/dev/mtdblock2 really can't be read:

root@linksys:/# dd if=/dev/mtdblock2  of=/tmp/x conv=noerror
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
dd: /dev/mtdblock2: I/O error
...

root@linksys:/# dmesg
[149324.472780] print_req_error: 3434 callbacks suppressed
[149324.472788] blk_update_request: I/O error, dev mtdblock2, sector 256 op 0x0:(READ) flags 0x84700 phys_seg 26 prio class 0
[149324.489773] blk_update_request: I/O error, dev mtdblock2, sector 264 op 0x0:(READ) flags 0x84700 phys_seg 25 prio class 0
[149324.501309] blk_update_request: I/O error, dev mtdblock2, sector 272 op 0x0:(READ) flags 0x84700 phys_seg 24 prio class 0
[149324.513110] blk_update_request: I/O error, dev mtdblock2, sector 280 op 0x0:(READ) flags 0x84700 phys_seg 23 prio class 0
[149324.524642] blk_update_request: I/O error, dev mtdblock2, sector 288 op 0x0:(READ) flags 0x84700 phys_seg 22 prio class 0
[149324.536136] blk_update_request: I/O error, dev mtdblock2, sector 296 op 0x0:(READ) flags 0x84700 phys_seg 21 prio class 0
[149324.547439] blk_update_request: I/O error, dev mtdblock2, sector 304 op 0x0:(READ) flags 0x84700 phys_seg 20 prio class 0
[149324.558870] blk_update_request: I/O error, dev mtdblock2, sector 312 op 0x0:(READ) flags 0x84700 phys_seg 19 prio class 0
[149324.570155] blk_update_request: I/O error, dev mtdblock2, sector 320 op 0x0:(READ) flags 0x84700 phys_seg 18 prio class 0
[149324.581596] blk_update_request: I/O error, dev mtdblock2, sector 328 op 0x0:(READ) flags 0x84700 phys_seg 17 prio class 0
[149324.604720] buffer_io_error: 3425 callbacks suppressed
[149324.604728] Buffer I/O error on dev mtdblock2, logical block 32, async page read

Updated from OpenWRT 22.03-SNAPSHOT to 22.03.0 but still can't read mtdblock2. No USB devices are mounted, in case that's relevant.

Looking through this very unwieldy discussion https://forum.openwrt.org/t/belkin-rt3200-linksys-e8450-wifi-ax-discussion/94302, the developer (daniel) says "This error can show when trying to access areas in the flash with broken OOB/ECC data." and that mtdblock2 is the factory partition containing calibration data.

Restarting the 5GHz WiFi will trigger this error, despite me trying to make /dev/mtdblock2 chmod 000.

lsof shows the rpcd process having /dev/mtdblock2 open:

root@linksys:~# lsof -p 3422
COMMAND  PID USER   FD      TYPE             DEVICE SIZE/OFF                NODE NAME
rpcd    3422 root  cwd       DIR               0,18      352                  67 /
rpcd    3422 root  rtd       DIR               0,18      352                  67 /
rpcd    3422 root  txt       REG               0,18    66139 9223372036854776300 /sbin/rpcd
rpcd    3422 root  mem       REG              259,1                          492 /sbin/rpcd (path dev=0,18, inode=9223372036854776300)
rpcd    3422 root  mem       BLK               31,8                           54 /dev/mtdblock2
rpcd    3422 root  mem       REG              259,1                         1122 /usr/lib/rpcd/rrdns.so (path dev=0,18, inode=9223372036854776930)
rpcd    3422 root  mem       REG              259,1                         1121 /usr/lib/rpcd/luci.so (path dev=0,18, inode=9223372036854776929)
rpcd    3422 root  mem       REG              259,1                          599 /usr/lib/libnl-tiny.so (path dev=0,18, inode=9223372036854776407)
rpcd    3422 root  mem       REG              259,1                          583 /usr/lib/libiwinfo.so.20210430 (path dev=0,18, inode=9223372036854776391)
rpcd    3422 root  mem       REG              259,1                         1120 /usr/lib/rpcd/iwinfo.so (path dev=0,18, inode=9223372036854776928)
rpcd    3422 root  mem       REG              259,1                         1119 /usr/lib/rpcd/file.so (path dev=0,18, inode=9223372036854776927)
rpcd    3422 root  mem       REG              259,1                          318 /lib/libgcc_s.so.1 (path dev=0,18, inode=9223372036854776126)
rpcd    3422 root  mem       REG              259,1                          588 /usr/lib/libjson-c.so.5.1.0 (path dev=0,18, inode=9223372036854776396)
rpcd    3422 root  mem       REG              259,1                          315 /lib/libblobmsg_json.so.20220515 (path dev=0,18, inode=9223372036854776123)
rpcd    3422 root  mem       REG              259,1                          325 /lib/libuci.so (path dev=0,18, inode=9223372036854776133)
rpcd    3422 root  mem       REG              259,1                          324 /lib/libubus.so.20220601 (path dev=0,18, inode=9223372036854776132)
rpcd    3422 root  mem       REG              259,1                          323 /lib/libubox.so.20220515 (path dev=0,18, inode=9223372036854776131)
rpcd    3422 root  mem       REG              259,1                          316 /lib/libc.so (path dev=0,18, inode=9223372036854776124)
rpcd    3422 root    0r      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    1w      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    2w      CHR                1,3      0t0                  26 /dev/null
rpcd    3422 root    3u  a_inode                0,8        0                  74 [eventpoll:4,6]
rpcd    3422 root    4r     FIFO                0,7      0t0               11701 pipe
rpcd    3422 root    5w     FIFO                0,7      0t0               11701 pipe
rpcd    3422 root    6u     unix 0x000000000aec3471      0t0               11702 type=STREAM
rpcd    3422 root    7u  netlink                         0t0               15712 GENERIC
rpcd    3422 root    8r      BLK               31,8      0t0                  54 /dev/mtdblock2

This is likely what's causing things to hang since rpcd is used for handling other things like WiFi connections or luci.

Searching through the entire filesystem for mtdblock2 returned nothing. The discussion mentioned that iwinfo might be reading mtdblock2. There is a string 'mtdblock%d' in the libiwinfo.so.20210430 library. The source for this does indeed show it try to read /dev/mtdblockN as seen here: https://git.openwrt.org/?p=project/iwinfo.git;a=blob_plain;f=iwinfo_utils.c;hb=3d38d1d17eba37b0985dd2857b4539410824a3e6 in order to determine the hardware information.

Perhaps a simple fix here would be to rename /dev/mtdblock2 to /dev/mtdblock2a so it can't try reading from the device? Basically, mv /dev/mtdblock2 /dev/mtdblock2a

See also