lemp10/darp7/galp5 not powering off after suspend (5.11)

[jacobgkau]

The lemp10/darp7 are sometimes not powering off after shutting down (this happens ~4/5 times, but not every time.) When this happens, journalctl indicates the shutdown process is finishing:

Mar 30 14:31:53 pop-os systemd[1]: Stopped Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Mar 30 14:31:53 pop-os systemd[1]: Reached target Shutdown.
Mar 30 14:31:53 pop-os systemd[1]: Reached target Final Step.
Mar 30 14:31:53 pop-os systemd[1]: systemd-poweroff.service: Succeeded.
Mar 30 14:31:53 pop-os systemd[1]: Finished Power-Off.
Mar 30 14:31:53 pop-os systemd[1]: Reached target Power-Off.
Mar 30 14:31:53 pop-os systemd[1]: Shutting down.
Mar 30 14:31:53 pop-os systemd-shutdown[1]: Syncing filesystems and block devices.
Mar 30 14:31:53 pop-os systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Mar 30 14:31:53 pop-os systemd-journald[513]: Journal stopped
-- Reboot --
Mar 30 14:41:50 pop-os kernel: Linux version 5.11.0-7612-generic (buildd@lgw01-amd64-052) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU Binutils for Ubuntu) 2.35.1) #13~1616168001~20.10~cf74746-Ubuntu SMP Mon Mar 29 17:37:00 UTC  (Ubuntu 5.11.0-7612.13~1616168001~20.10~cf74746-generic 5.11.7)
Mar 30 14:41:50 pop-os kernel: Command line: initrd=\EFI\Pop_OS-47ff7a17-79d7-421f-91ee-67f6bd9b0973\initrd.img root=UUID=47ff7a17-79d7-421f-91ee-67f6bd9b0973 ro quiet loglevel=0 systemd.show_status=false splash

The LCD turns off, but the power LED remains solid green, and pushing the power button doesn't turn the machine back on. I'm able to recover from this by holding down the power button until the power LED turns off.

It seems like an EC reset is needed to power on again when this state is reached; I have to wait a second after powering off before it will power on again. I had one case where I had the system plugged in, and it did power off eventually, turning the power LED to orange; but the system wouldn't power on again until I unplugged for a few seconds. If I try to power on without an EC reset, the power LED turns green for a second, then goes back to its previous state.

This seems like it might involve the EC, but I'm starting the issue here because if I downgrade to our previous 5.8 kernel, once I'm booted from that kernel, the system powers off as expected again. Also, the last lemp10 firmware update was nearly two months ago, and I know the issue hasn't been occurring for that long; it just started in the past day or two as we released new kernels (and unfortunately took some time to notice and then narrow down.) This is not happening on the galp5.

[jacobgkau]

I installed TLP on lemp10 to test something unrelated yesterday, and the issue didn't seem to be happening this morning until I removed it again.

The issue still occurs with the latest kernel in the Ubuntu mainline PPA, 5.12-rc5.

The issue does not seem to occur on kernel version 5.10.0 from the mainline PPA.

The issue doesn't occur on 5.10.27, but it does occur on version 5.11.0-rc2 (rc1 didn't build in Ubuntu's PPA.)

[jackpot51]

I see the same thing on my galp5. I'll look into it

[jacobgkau]

I built our kernel with the following six commits reverted (the last six commits to reboot.c, between 5.10 and 5.11):

40247e5#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42
1a9d079#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42
0c5c017#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42
2c622ed#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42
f9a9050#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42
42b4ca0#diff-d8ab18c7fe4aecdebc357cc3821e8f92cd7a82669f8e1091013e2ab751fe9c42

I got two successful power-offs, but then it hung again (but did finish powering off after no more than a couple of minutes.) I'll step back from this since you're looking at it now.

[jacobgkau]

Not fixed, still happens after suspend/resume.

[arbitrary-dev]

Same issue here, applied the latest system/firmware updates for my lemp10 just 15 minutes ago, and now shutdowns doesn't work for me anymore. Just blank screen and the temperature slowly rises at the bottom case.

Hard shutdown with holding the power button helps.

[jackpot51]

I am close to finding the kernel commit that introduced this

[jackpot51]

I think this patch might have introduced the hang: https://www.spinics.net/lists/intel-gfx/msg249449.html