panic: kernel BUG at net/core/skbuff.c:109!

[melver]

Using a recent syzkaller config, we get the below panic on very recent Clang (0b90a08f7722980f6074c6eada8022242408cdb4). This issue does not exist in Clang 11 (no bisection attempted yet).

.config: bad.config.txt
steps to reproduce: 1) boot kernel; 2) try to ssh into VM or any other network-related activity.

------------[ cut here ]------------
kernel BUG at net/core/skbuff.c:109!
invalid opcode: 0000 [#1] PREEMPT SMP
CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.8.0+ #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014
RIP: 0010:skb_panic+0xc4/0xd0 net/core/skbuff.c:105
Code: 48 8b 74 24 08 48 8b 54 24 10 44 89 e9 44 8b 44 24 04 49 89 e9 b8 00 00 00 00 53 41 54 41 57 41 56 e8 7e ba 17 fc 48 83 c4 20 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 55 41 57 41 56 41 55 41 54 53
RSP: 0018:ffffc900000e8450 EFLAGS: 00010286
RAX: 0000000000000098 RBX: ffffffff8739c99c RCX: 9f928d0f82712900
RDX: 0000000000000301 RSI: 0000000000000301 RDI: 0000000000000000
RBP: ffff8888136be800 R08: ffffffff813ca1fc R09: 0000ffff875f7d1f
R10: 0000ffffffffffff R11: 0000000000000000 R12: 00000000000002c0
R13: 00000000f19e81cc R14: ffff888721cd6774 R15: 0000000000000140
FS:  0000000000000000(0000) GS:ffff88881fa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3a3c1efab4 CR3: 000000081177a006 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
 <IRQ>
 skb_under_panic+0xc/0x10 net/core/skbuff.c:119
 skb_push+0x96/0xa0 net/core/skbuff.c:1884
 tcp_make_synack+0x439/0x800 net/ipv4/tcp_output.c:3414
 tcp_v4_send_synack+0x71/0x3a0 net/ipv4/tcp_ipv4.c:979
 tcp_conn_request+0x1348/0x1640 net/ipv4/tcp_input.c:6771
 tcp_v4_conn_request+0x10b/0x130 net/ipv4/tcp_ipv4.c:1474
 tcp_rcv_state_process+0x74e/0x17a0 net/ipv4/tcp_input.c:6246
 tcp_v4_do_rcv+0x401/0x4a0 net/ipv4/tcp_ipv4.c:1664
 tcp_v4_rcv+0x1fdd/0x2830 net/ipv4/tcp_ipv4.c:2012
 ip_protocol_deliver_rcu+0x2f2/0x540 net/ipv4/ip_input.c:204
 ip_local_deliver_finish net/ipv4/ip_input.c:231 [inline]
 NF_HOOK include/linux/netfilter.h:301 [inline]
 ip_local_deliver+0x26d/0x310 net/ipv4/ip_input.c:252
 dst_input include/net/dst.h:449 [inline]
 ip_sublist_rcv_finish net/ipv4/ip_input.c:550 [inline]
 ip_list_rcv_finish net/ipv4/ip_input.c:600 [inline]
 ip_sublist_rcv+0x61f/0x670 net/ipv4/ip_input.c:608
 ip_list_rcv+0x262/0x290 net/ipv4/ip_input.c:643
 __netif_receive_skb_list_ptype net/core/dev.c:5329 [inline]
 __netif_receive_skb_list_core+0x34b/0x450 net/core/dev.c:5377
 __netif_receive_skb_list+0x262/0x2e0 net/core/dev.c:5429
 netif_receive_skb_list_internal+0x16c/0x440 net/core/dev.c:5534
 gro_normal_list net/core/dev.c:5645 [inline]
 napi_complete_done+0x1a1/0x3a0 net/core/dev.c:6370
 virtqueue_napi_complete+0x28/0x80 drivers/net/virtio_net.c:329
 virtnet_poll+0x64f/0x780 drivers/net/virtio_net.c:1455
 napi_poll net/core/dev.c:6687 [inline]
 net_rx_action+0x317/0x8f0 net/core/dev.c:6757
 __do_softirq+0x1b6/0x30e kernel/softirq.c:298
 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:706
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline]
 do_softirq_own_stack+0x71/0x90 arch/x86/kernel/irq_64.c:77
 invoke_softirq kernel/softirq.c:393 [inline]
 __irq_exit_rcu+0x10c/0x120 kernel/softirq.c:423
 irq_exit_rcu+0x5/0x10 kernel/softirq.c:435
 common_interrupt+0x1e6/0x240 arch/x86/kernel/irq.c:239
 asm_common_interrupt+0x1e/0x40 arch/x86/include/asm/idtentry.h:572
RIP: 0010:native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:61
Code: ff ff e9 0d ff ff ff e8 10 d7 24 fb e9 72 ff ff ff cc cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 84 2b 5c 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 74 2b 5c 00 f4 c3 cc cc 65 48 8b 04
RSP: 0018:ffffc90000083ef0 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff88881c74c340
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffff88881c74c340 R08: ffffffff814f5c10 R09: 000088881c74c367
R10: 0000ffffffffffff R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffc90000083f07 R15: 0000000000000000
 arch_safe_halt arch/x86/include/asm/paravirt.h:150 [inline]
 default_idle+0x1b/0x30 arch/x86/kernel/process.c:688
 default_idle_call kernel/sched/idle.c:94 [inline]
 cpuidle_idle_call kernel/sched/idle.c:163 [inline]
 do_idle+0xf7/0x2c0 kernel/sched/idle.c:276
 cpu_startup_entry+0x15/0x20 kernel/sched/idle.c:372
 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243
Modules linked in:
---[ end trace e6cbb0171868af05 ]---

[melver]

Likely related, but I've recently encountered other strange memory corruptions on Clang 12 (and no other compiler).

Clang 12 @ 43d239d0fadb1f8ea297580ca39dfbee96c913c1
Kernel @ next-20200930

Sample trace:

------------[ cut here ]------------
list_del corruption, ffffcb01d14e6408->next is LIST_POISON1 (dead000000000100)
WARNING: CPU: 0 PID: 138 at lib/list_debug.c:47 __list_del_entry_valid+0x46/0xa0 lib/list_debug.c:45
Modules linked in:
CPU: 0 PID: 138 Comm: systemd-journal Tainted: G        W         5.9.0-rc7-next-20200930 #3
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014
RIP: 0010:__list_del_entry_valid+0x46/0xa0 lib/list_debug.c:45
Code: c2 22 48 39 d1 74 29 48 39 31 75 3c b3 01 48 39 70 08 75 4f 89 d8 5b c3 90 31 db 48 c7 c7 10 7f a2 a9 31 c0 e8 7b f8 ab ff 90 <0f> 0b 90 90 eb e4 90 31 db 48 c7 c7 46 7f a2 a9 31 c0 e8 63 f8 ab
RSP: 0018:ffffa26801413b48 EFLAGS: 00010046
RAX: 38c708c80cfa2900 RBX: 0000000000000000 RCX: ffff9b20bd3c0040
RDX: 0000000000000000 RSI: ffffffffa99fd122 RDI: 00000000ffffffff
RBP: 0000000000000000 R08: 0000000362362d81 R09: 0000000000000002
R10: 000000002d2d2d2d R11: ffffffffa82689b0 R12: 0000000000190018
R13: 0000000000190018 R14: ffffcb01d14e6400 R15: 0000000000190019
FS:  00007f10a432c8c0(0000) GS:ffff9b221fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f10a1252000 CR3: 00000006ba9e2002 CR4: 0000000000770ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 __list_del_entry include/linux/list.h:132 [inline]
 list_del include/linux/list.h:146 [inline]
 remove_full mm/slub.c:1064 [inline]
 __slab_free+0x29f/0x440 mm/slub.c:3055
 do_slab_free mm/slub.c:3134 [inline]
 slab_free mm/slub.c:3147 [inline]
 kmem_cache_free+0x1d0/0x230 mm/slub.c:3162
 skb_free_datagram+0x15/0x60 net/core/datagram.c:325
 unix_dgram_recvmsg+0x417/0x4e0 net/unix/af_unix.c:2181
 ____sys_recvmsg+0x22a/0x250 net/socket.c:885
 ___sys_recvmsg net/socket.c:2627 [inline]
 __sys_recvmsg+0x138/0x2c0 net/socket.c:2663
 do_syscall_64+0x34/0x50 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f10a38bddc7
Code: 89 01 b8 ff ff ff ff eb d8 66 2e 0f 1f 84 00 00 00 00 00 8b 05 0a b6 20 00 85 c0 75 2e 48 63 ff 48 63 d2 b8 2f 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 b1 71 20 00 f7 d8 64 89 02 48
RSP: 002b:00007ffda5d549e8 EFLAGS: 00000246 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 00007ffda5d54f60 RCX: 00007f10a38bddc7
RDX: 0000000040000040 RSI: 00007ffda5d54a40 RDI: 0000000000000006
RBP: 00007ffda5d54a40 R08: 0000000000000008 R09: 0000000000000070
R10: 000000000000000e R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000006 R14: 0000563c323ac958 R15: 0005b0894d3619db
CPU: 0 PID: 138 Comm: systemd-journal Tainted: G        W         5.9.0-rc7-next-20200930 #3
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0xdb/0x10e lib/dump_stack.c:118
 __warn+0xdd/0x1a0 kernel/panic.c:608
 report_bug+0x1bc/0x260 lib/bug.c:198
 handle_bug+0x43/0x80 arch/x86/kernel/traps.c:235
 exc_invalid_op+0x18/0xb0 arch/x86/kernel/traps.c:255
 asm_exc_invalid_op+0x12/0x20 arch/x86/include/asm/idtentry.h:581
RIP: 0010:__list_del_entry_valid+0x46/0xa0 lib/list_debug.c:45
Code: c2 22 48 39 d1 74 29 48 39 31 75 3c b3 01 48 39 70 08 75 4f 89 d8 5b c3 90 31 db 48 c7 c7 10 7f a2 a9 31 c0 e8 7b f8 ab ff 90 <0f> 0b 90 90 eb e4 90 31 db 48 c7 c7 46 7f a2 a9 31 c0 e8 63 f8 ab
RSP: 0018:ffffa26801413b48 EFLAGS: 00010046
RAX: 38c708c80cfa2900 RBX: 0000000000000000 RCX: ffff9b20bd3c0040
RDX: 0000000000000000 RSI: ffffffffa99fd122 RDI: 00000000ffffffff
RBP: 0000000000000000 R08: 0000000362362d81 R09: 0000000000000002
R10: 000000002d2d2d2d R11: ffffffffa82689b0 R12: 0000000000190018
R13: 0000000000190018 R14: ffffcb01d14e6400 R15: 0000000000190019
 __list_del_entry include/linux/list.h:132 [inline]
 list_del include/linux/list.h:146 [inline]
 remove_full mm/slub.c:1064 [inline]
 __slab_free+0x29f/0x440 mm/slub.c:3055
 do_slab_free mm/slub.c:3134 [inline]
 slab_free mm/slub.c:3147 [inline]
 kmem_cache_free+0x1d0/0x230 mm/slub.c:3162
 skb_free_datagram+0x15/0x60 net/core/datagram.c:325
 unix_dgram_recvmsg+0x417/0x4e0 net/unix/af_unix.c:2181
 ____sys_recvmsg+0x22a/0x250 net/socket.c:885
 ___sys_recvmsg net/socket.c:2627 [inline]
 __sys_recvmsg+0x138/0x2c0 net/socket.c:2663
 do_syscall_64+0x34/0x50 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f10a38bddc7
Code: 89 01 b8 ff ff ff ff eb d8 66 2e 0f 1f 84 00 00 00 00 00 8b 05 0a b6 20 00 85 c0 75 2e 48 63 ff 48 63 d2 b8 2f 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 b1 71 20 00 f7 d8 64 89 02 48
RSP: 002b:00007ffda5d549e8 EFLAGS: 00000246 ORIG_RAX: 000000000000002f
RAX: ffffffffffffffda RBX: 00007ffda5d54f60 RCX: 00007f10a38bddc7
RDX: 0000000040000040 RSI: 00007ffda5d54a40 RDI: 0000000000000006
RBP: 00007ffda5d54a40 R08: 0000000000000008 R09: 0000000000000070
R10: 000000000000000e R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000000006 R14: 0000563c323ac958 R15: 0005b0894d3619db
---[ end trace e7c592efbbc987ba ]---

Config: clang12-corruption.config.txt

[nickdesaulniers]

Is this reproducible? Sounds like yes, based on ssh? If you have a ramdisk, we can probably boot test this quickly in QEMU.

Are you able to bisect LLVM commits to pinpoint the first bad commit?

[melver]

Is this reproducible? Sounds like yes, based on ssh? If you have a ramdisk, we can probably boot test this quickly in QEMU.

I'd suggest to reproduce the 2nd newer report -- no user interaction required, just booting the kernel to user space is sufficient.

Are you able to bisect LLVM commits to pinpoint the first bad commit?

Am bisecting, but this will take a while. Maybe I have something by tomorrow.

[melver]

This is what I get:

4b0aa5724feaa89a9538dcab97e018110b0e4bc3 is the first bad commit
commit 4b0aa5724feaa89a9538dcab97e018110b0e4bc3
Author: James Y Knight <jyknight@google.com>
Date:   Fri May 15 23:43:30 2020 -0400
    
    Change the INLINEASM_BR MachineInstr to be a non-terminating instruction.
    
    Before this instruction supported output values, it fit fairly
    naturally as a terminator. However, being a terminator while also
    supporting outputs causes some trouble, as the physreg->vreg COPY
    operations cannot be in the same block.
    
    Modeling it as a non-terminator allows it to be handled the same way
    as invoke is handled already.
    
    Most of the changes here were created by auditing all the existing
    users of MachineBasicBlock::isEHPad() and
    MachineBasicBlock::hasEHPadSuccessor(), and adding calls to
    isInlineAsmBrIndirectTarget or mayHaveInlineAsmBr, as appropriate.

    Reviewed By: nickdesaulniers, void

    Differential Revision: https://reviews.llvm.org/D79794

$ git bisect log
git bisect start
# bad: [7ab7b979d29e1e43701cf690f5cf1903740f50e3] Bump the trunk major version to 12
git bisect bad 7ab7b979d29e1e43701cf690f5cf1903740f50e3
# good: [ca2dcbd030eadbf0aa9b660efe864ff08af6e18b] [SafeStack,NFC] Make StackColoring read-only
git bisect good ca2dcbd030eadbf0aa9b660efe864ff08af6e18b
# good: [f7a14514ee63dc2ab9558c50254efb8ac2ad7cc6] [darwin][driver] isMacosxVersionLT should check against the minimum supported OS version
git bisect good f7a14514ee63dc2ab9558c50254efb8ac2ad7cc6
# bad: [021d56abb9ee3028cb88895144d71365e566c32f] [SVE] Make Constant::getSplatValue work for scalable vector splats
git bisect bad 021d56abb9ee3028cb88895144d71365e566c32f
# bad: [0059f6ffe84241b9728e48c1eabdaf1a6abbef66] [NewPM] Add -basic-aa to pr33196.ll
git bisect bad 0059f6ffe84241b9728e48c1eabdaf1a6abbef66
# good: [3ee580d0176f69a9f724469660f1d1805e0b6a06] [ARM][LowOverheadLoops] Handle reductions
git bisect good 3ee580d0176f69a9f724469660f1d1805e0b6a06
# bad: [a59dc55c2a11c1b125d1a356a90b0b2bf72b16fb] [InstSimplify] Move assume icmp test (NFC)
git bisect bad a59dc55c2a11c1b125d1a356a90b0b2bf72b16fb
# good: [05a20a9e9aba301a828bcbd72b0ed724755752d1] [RISCV] Temporarily move riscv-expand-pseudo pass to PreEmitPass2
git bisect good 05a20a9e9aba301a828bcbd72b0ed724755752d1
# bad: [f11305780f08969488add6c84439fc91d18692dc] [CodeGen] Fix warnings in DAGCombiner::visitSCALAR_TO_VECTOR
git bisect bad f11305780f08969488add6c84439fc91d18692dc
# good: [6bd1db08e7ccd61996d3867d22ff8eb1979f8621] [InstCombine] Don't let an alignment assume prevent new/delete removals.
git bisect good 6bd1db08e7ccd61996d3867d22ff8eb1979f8621
# bad: [3eacfdc72f1aa3ac53eb300116f194d560053ec7] [BPF] Fix a BTF gen bug related to a pointer struct member
git bisect bad 3eacfdc72f1aa3ac53eb300116f194d560053ec7
# good: [0f6afd946d25a2e83288339934f8fa384e38eea3] [CVP] Use different number in test (NFC)
git bisect good 0f6afd946d25a2e83288339934f8fa384e38eea3
# bad: [4b0aa5724feaa89a9538dcab97e018110b0e4bc3] Change the INLINEASM_BR MachineInstr to be a non-terminating instruction.
git bisect bad 4b0aa5724feaa89a9538dcab97e018110b0e4bc3
# good: [78c69a00a4cff786e0ef13c895d0db309d6b3f42] [NFC] Clean up uses of MachineModuleInfoWrapperPass
git bisect good 78c69a00a4cff786e0ef13c895d0db309d6b3f42
# first bad commit: [4b0aa5724feaa89a9538dcab97e018110b0e4bc3] Change the INLINEASM_BR MachineInstr to be a non-terminating instruction.

Also sadly it seems this found its way into Clang 11:

$ git branch -a --contains 4b0aa5724feaa89a9538dcab97e018110b0e4bc3
  ...
  remotes/origin/master
  remotes/origin/release/11.x
  ...

[nickdesaulniers]

ok, next thing is to isolate the configs.
tools/testing/ktest/config-bisect.pl can be rerun first with a good and bad config file, then repeatedly with the same arguments plus good or bad and can help bisect configs. I assume we don't observe this for defconfig?

[nickdesaulniers]

with the above config on mainline, I observe the following warnings from objtool:

  OBJTOOL vmlinux.o
vmlinux.o: warning: objtool: __do_fast_syscall_32()+0x49: call to syscall_enter_from_user_mode_work() leaves .noinstr.text section
vmlinux.o: warning: objtool: do_machine_check()+0x49: call to mce_rdmsrl() leaves .noinstr.text section
vmlinux.o: warning: objtool: rcu_nmi_enter()+0x11e: call to preempt_schedule_notrace_thunk() leaves .noinstr.text section

but the kernel boots just fine for me. I see now you mention linux-next. I get the same warnings from objtool when testing -next. Boots fine. It looks like your panic is in recvmsg; so maybe your test setup is listing on a socket on boot and mine is not?

There were also two critical fixes to 4b0aa5724feaa89a9538dcab97e018110b0e4bc3 which will complicate a bisection. If you test after that landed but before the fixes, then it will seem bad, when it may be a separate issue already fixed. The fixes were:

1. 60433c63acb71935111304d71e41b7ee982398f8
2. f7a53d82c0902147909f28a9295a9d00b4b27d38

I also have a pending fix that should be tested:
https://reviews.llvm.org/D88438

Another thing to test; enabling assertions in LLVM then rebuild and see if anything trips. I'll try to isolate those objtool warnings, since they may be our smoking gun.

cc @jyknight @gwelymernans

[nickdesaulniers]

(the objtool warnings I observe exist regardless of 4b0aa5724feaa89a9538dcab97e018110b0e4bc3. Forked #1169 to track those).