Wi-SUN causing file corruption and assert erros on LittleFS

[felipeLeast]

Description

• What target does this relate to?
  This issue is related to Wi-SUN and LittleFS

• What toolchain are you using?
  I'm using GCC_ARM as toolchain

• What tools are you using?
  Mbed-os 5.12.4, mbed compiler and visual studio code as IDE.

• What is the SHA of Mbed OS?
  73f0963

• Steps to reproduce

Activate Wi-SUN and littleFS, create some files on littleFS and after a while littleFS files will dissapear or get corrupted.

I'm trying to use littleFS on Wi-SUN project but every time I try to use both together LittleFS get's eventually lost. When I comment the Wi-SUN part of the project it goes smooth. I want to know if the Wi-SUN messes with filesystem at some point, or if the queue event from mbed os could cause problems to the littleFS filesystem.

I've tried to protect my files with a mutex and tried to disable irq on the thread that I write on files (which is not possible to do when writing files).

Besides files corruption sometimes littleFS throws an exception regarding block count, lfs type or pcache block.

Issue request type

[  ] Question
[  ] Enhancement
[x] Bug

[0xc0170]

@felipeLeast is there a code snippet to reproduce the issue? Without seeing more in detail we can just guess what goes on in the app.

@ARMmbed/mbed-os-ipcore Any suggestions?

[teetak01]

Which board is this?

Would this be related to #9845

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1612

[felipeLeast]

Which board is this?

Would this be related to #9845

It seems to be exaclty this problem, I've notice that the problem usually occur rigth after a PAN notification (when the device tries to write on S2LP SPI) which is a different SPI from the on the Flash.

But I'm using nrf52840, and when I saw the fix from #9845 I don't know how to apply it on NRF52840, because any pin could be anything, so que get_peripheral_name won't work, and it's not needed, or at least shouldn't be.

Any tips on how to debug and fix this?

[felipeLeast]

@felipeLeast is there a code snippet to reproduce the issue? Without seeing more in detail we can just guess what goes on in the app.

@ARMmbed/mbed-os-ipcore Any suggestions?

I'm sorry for not being specific, but it's very strange error, it doesn't seem to follow a specific sequence or timeline, it's sporadic and happens on both LittleFS and FatFS, with the difference that LittleFS throws a assert error from time to time.

Seems to be related to the SPI, I just couldn't thing of a way to debug it yet. I've just updated my mbed-os to 5.13.3 to see if there's any fix, but still get errors eventually.

When I comment the Wi-SUN initialization routine the FS works for hours without errors. That's why I think they are related. My Wi-SUN config uses the S2LP in a differente SPI than my SPIFBlockDevice, on the same SPI as SPIF I have, RTC and a Energy Meter, but I don't believe that they are the cause of the problems.

I don't have a code snippet because is a custom board, so you wouldn't be able to test it. But It's would be simple to reproduce, just start nanostack border router and place by the end a thread start which appends 50 bytes into a file every 150ms, when this file reaches 1024 bytes of more start a new file and keep the old one. Eventually when you list you directory entry some files will be gone or with a different name, or missing something.

I'l try to reproduce on some demo board and past the code here.

Follows a print from my current code log so you can have an idea:

[INFO][meas]: ========================================
[DBG ][conf]: Current index: 24, Types: 13
[DBG ][rtc ]: Tue Nov 30 00:00:00 1999
[DBG ][meas]: Measurement control: (Configurations: 13) Runs: 168 Current minutes: 62068708
Opening the root directory... OK
root directory:
    .                             0
    ..                            0
    boot_count                    2
    configuration                 9
    measurement_config            70
    location                      8
    measurement_0                 1056
    measurement_1                 1048
    measurement_2                 1056
    measurement_3                 1048
    measurement_4                 1048
    measurement_5                 1056
    measurement_6                 1048
    measurement_7                 1056
    measurement_8                 1048
    measurement_9                 1056
    measurement_10                1048
    measurement_11                1056
    measurement_12                1048
    measurement_13                1056
    measurement_14                1048
    measurement_15                1056
    measurement_16                1048
    measurement_17                1056
    measurement_18                1048
    measurement_19                1056
    measurement_20                1048
    measurement_21                1056
    measurement_22                1048
    measurement_23                1056
    measurement_24                948
Closing the root directory... OK
[INFO][meas]: File /fs/measurement_24 currently has 948 bytes
[INFO][meas]: File /fs/measurement_24 has 1002 bytes
[INFO][meas]: ========================================
[DBG ][conf]: Current index: 24, Types: 13
[DBG ][rtc ]: Tue Nov 30 00:00:00 1999
[DBG ][meas]: Measurement control: (Configurations: 13) Runs: 169 Current minutes: 62068708
Opening the root directory... OK
root directory:
    .                             0
    ..                            0
    boot_count                    2
    configuration                 9
    measurement_config            70
    location                      8
    measurement_0                 1056
    measurement_                  1
Closing the root directory... OK
[INFO][meas]: File /fs/measurement_24 currently has 4 bytes
[INFO][meas]: File /fs/measurement_24 has 48 bytes

[kjbracey]

But I'm using nrf52840, and when I saw the fix from #9845 I don't know how to apply it on NRF52840, because any pin could be anything, so que get_peripheral_name won't work, and it's not needed, or at least shouldn't be.

The SPI class uses mutexes to serialise SPI accesses to make sure only one SPI slave is being accessed at once on each SPI bus/peripheral.

If the HAL does not provide spi_get_peripheral_name, then SPI will assume worst-case, and that your Wi-Sun and SPIF are potentially on the same bus or peripheral, and mutex them against each other.

(Note that simply being on different MOSI/MISO/CLK pins isn't sufficient to know they're independent - the hardware might actually be using one SPI master peripheral block muxed onto different pins, which is why we need to ask the HAL).

Contending for a mutex could be fatal for Wi-Sun - critical radio timings broken by waiting for slow file accesses. I'm less clear why the file accesses should be bothered by contending for a mutex with the radio.

I believe you can implement #9845 for NRF52 by just having it return the result of pin_instance_spi, as used in NRF52's spi_init.

That would stop the two SPI buses from mutexing against each other.

[TuomoHautamaki]

Not sure which radio you are using but there's at least one PIN conflict issue on X-Nucleo-S2868A1.
Default SPI SCLK pin configuration is D13. On X-Nucleo-S2868A1, resistor R6 must be soldered instead of R11. For more information, see user manual UM2405. https://www.st.com/resource/en/user_manual/dm00498153.pdf

[VeijoPesonen]

@kjbracey-arm are this and #11732 related?

[kjbracey]

It could be related. That SPIFBlockDevice locking bug would offer a mechanism for the radio to mess storage up.

But if that is the issue, that means the WiSUN device and the storage are on the same bus. That's never going to work well. When the storage is in use, it will be hogging the bus, and the radio won't get access to it fast enough for timeslicing things.

[kjbracey]

Ah, a comment above says that the radio isn't on the same bus, but two other devices are. It's possible the drivers for the other 2 devices could be breaking storage because of 11732.

[felipeLeast]

@VeijoPesonen and @kjbracey-arm I´ve seen that issue a couple of days ago, I'll try to apply that fix and come back here. If it fixes I'll close this thread.