PI3: 4.8.2 breaks networking - it bricks PI based gw!

[g6094199]

• The issue is present in the latest release.
• I have searched the issues of this repository and believe that this is not a duplicate.

What happened?

i upgaded my FULL installation on raspberry PI 3 from 4.7.1 to 4.8.2.

both are openwrt 24.11.x based so no problems exprected.
the upgrade finished successfully. but afte the reboot no ethernet connection was established. i found the br0 inferface NOT coming up automaticly.

a short dig shows that simply the /etc/config/network file is missing!

THIS BUILD BRICKS PI BASED (at least bcm2709) GATEWAYS AND SHOULD BE PULLED FROM THE SERVERS!

[g6094199]

why is the testing outsourced to users and their live equipment? this is a very basic bug and should be discovered before doing ANY release! why is there no (public) beta phase where these bug get discovered?!?

[brocaar]

We're you able to reproduce the issue by flashing the SD card to the old version and then perform the upgrade again?

…

On Wed, Sep 3, 2025, 13:07 g6094199 ***@***.***> wrote: *g6094199* left a comment (chirpstack/chirpstack-gateway-os#139) <#139 (comment)> why is the testing outsourced to users and their live equipment? this is a very basic bug and should be discovered before doing ANY release! why is there no (public) beta phase where these bug get discovered?!? — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABIM6K4HF7FC5EX7NVMDOT3Q3KXDAVCNFSM6AAAAACFQLYR3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENBYHE3TGNZVGI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[g6094199]

yep, did exaclty that. same behavior!

[brocaar]

So far, I can not reproduce the issue:

• I have flashed the v4.7.1 full version on a test Raspberry Pi
• I have configured the wifi using the ChirpStackAP hotspot
• After the RPi connected to my wifi, I have upgraded to v4.8.2 (using the web-interface)
• After the upgrade to v4.8.2, /etc/config/network is still present
• After the upgrade to v4.8.2, the RPi connects fine to ethernet

I will re-test using ethernet only

[brocaar]

Hmmm... it looks like with wifi interface disabled and then doing the update, I am able to reproduce the issue.

[brocaar]

@g6094199 I have been able to reproduce the issue, but after testing it several times, I also had test-runs where everything worked just fine. It looks like in this case the overlay partition can't be mounted. You can see this by executing:

dmesg | grep F2FS

Any help on debugging this issue is welcome.

[brocaar]

In my case, I was able to recover using firstboot and then reboot. This should have reset the configuration partition completely, but as the overlay was not mounted it returned an error. However, after a reboot, the configuration partition (overlay) did mount correctly.

Still not sure what is causing this.

[g6094199]

ok. while is a partially solution for test installation this is no fix for live devices.

...will have another look at it.

[brocaar]

Maybe some information that might be helpful for you or others while debugging the root-cause:

OpenWrt mounts a /dev/loop0 device and uses this for the overlay filesystem (containing config changes). This /dev/loop0 is written into the remaining space of the root partition, with an offset equal to the size of the read-only data in the root partition (e.g. if the partition is 4GB, of which 1GB is consumed by the root FS, then the offset of /dev/loop0 will be 1GB and it will write into the remaining 3GB).

On upgrade, OpenWrt asks you if you want to keep the settings, and if yes, it will create a tarball of the files that must be retained. Then it writes the new image to flash, re-creates the /dev/loop0 device and restores the files into.

My assumption is that in that last step somethings goes wrong. It can not mount /dev/loop0 since the offset might not be correct, and thus it does not mount it and does not restore the tarball containing the configuration.

When I run firstboot and then reboot, I see a line equal / similar to "Restoring configuration", after which all seems to work again normally. I agree this is only a solution for testing, but it might help to narrow-down the root-cause.

[g6094199]

have you tried to backup config. upgrade without restore and put the config back in?

[g6094199]

no, resetting to default on upgrade doesnt work either.

but this is interessting

[brocaar]

@g6094199 I think I have found the issue + a possible fix:

From my earlier comment

My assumption is that in that last step somethings goes wrong. It can not mount /dev/loop0 since the offset might not be correct, and thus it does not mount it and does not restore the tarball containing the configuration.

I think this might indeed where it fails. I noticed the mount_root in my logs too, and I think that is where things start to break. This is in summary what happens. The rootfs partition size is fixed-size and consists of the read-only squashfs + read/write overlay loopback device, that starts where the sqashfs filesystem ends (defined by an offset). As each OpenWrt version (or GW OS version) has a different sqashfs size, the offset of this loopback device changes. In general this is fine, as OpenWrt restores it on every upgrade.

I tried to understand how OpenWrt does this, and I believe it tries to detect the loopback device and if found, it mounts it. If it fails it will create it and formats it. What I believe happens is that OpenWrt thinks it has found the loopback device base on the first N bytes starting at the offset, but then when mounting it, it fails (failed to mount ..... in your logs) because it is not actually the start of the loopback device.

A similar issue can be found here, which also contains a solution:
openwrt/openwrt#9113 (comment)

Instead of just writing the squashfs partition, it would also zero-out the remaining bytes of the rootfs partition to make sure OpenWrt would never falsely detect the loopback device.

---

why is the testing outsourced to users and their live equipment? this is a very basic bug and should be discovered before doing ANY release! why is there no (public) beta phase where these bug get discovered?!?

I would like to ask you to be a bit more constructive when reporting a bug next time. As it turns out, reproducing the bug depends on the content of the loopback device data. It seems this issue is not basic after all as it touched quite some internals about how OpenWrt handles a sysupgrade and the persistence of (configuration) files. Nevertheless, thanks for reporting this issue, as I do agree it must be fixed sooner than later.

[g6094199]

why is the testing outsourced to users and their live equipment? this is a very basic bug and should be discovered before doing ANY release! why is there no (public) beta phase where these bug get discovered?!?

I would like to ask you to be a bit more constructive when reporting a bug next time. As it turns out, reproducing the bug depends on the content of the loopback device data. It seems this issue is not basic after all as it touched quite some internals about how OpenWrt handles a sysupgrade and the persistence of (configuration) files. Nevertheless, thanks for reporting this issue, as I do agree it must be fixed sooner than later.

just FYI. this is the 2nd build in a row (or was it in 4.8.0?) where i discovered showstopper bugs (the last one broke udp forwarding). both on the really NON exotic hw of raspberry pi. so the question is quiet legit. it seems there is obviously (automated) testing missing in the whole release process. i personally like to have less releases but fairly tested and stable ones. so just start a beta release or RC. whatever u will call it. so everyone who's installing it will know that it might break things! this is nothing that is expected from a release and should happen only in rare cases. and 2 in a row is not a rare case.

[brocaar]

FYI, I have created a PR to fix this in OpenWrt: openwrt/openwrt#19997.