Crash in ObjWriter on ARM during publish

[kant2002]

After #1387 during publish ILC crashed with following stack trace.

Thread 1 "ilc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x76c36510 (LWP 26921)]
0x268d53f8 in ?? ()
(gdb) backtrace
#0  0x268d53f8 in ?? ()
#1  0x460addb4 in llvm::MCExpr::evaluateAsAbsolute(long long&, llvm::MCAssembler const&) const () from /home/pi/runtimelab/samples/HelloWorld/pkg/runtime.linux-arm.microsoft.dotnet.ilcompiler/6.0.0-dev/tools/libobjwriter.so
#2  0x453a3e40 in ObjectWriter::EmitSymbolRef(char const*, RelocType, int) () from /home/pi/runtimelab/samples/HelloWorld/pkg/runtime.linux-arm.microsoft.dotnet.ilcompiler/6.0.0-dev/tools/libobjwriter.so
#3  0x453a36b0 in EmitSymbolRef () from /home/pi/runtimelab/samples/HelloWorld/pkg/runtime.linux-arm.microsoft.dotnet.ilcompiler/6.0.0-dev/tools/libobjwriter.so
#4  0x52a19706 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I try to look at locals, but without luck

(gdb) frame 2
#2  0x453a3e40 in ObjectWriter::EmitSymbolRef(char const*, RelocType, int) () from /home/pi/runtimelab/samples/HelloWorld/pkg/runtime.linux-arm.microsoft.dotnet.ilcompiler/6.0.0-dev/tools/libobjwriter.so
(gdb) info locals
No symbol table info available.

Even if I build LLVM and ObjWriter on this Raspberry using ./build.sh nativeaot.objwriter -rc Debug -lc Release. It will take a bit of time to figure out what's going on.

[kant2002]

When I try run ILC on Windows, On the ObjWriter stage probably I receive LLVM ERROR: unsupported relocation on symbol

That's seems to be from one of these places
https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/llvm/lib/Target/ARM/MCTargetDesc/ARMELFObjectWriter.cpp#L90
or
https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/llvm/lib/Target/ARM/MCTargetDesc/ARMELFObjectWriter.cpp#L162

I understand that this maybe invalid case and @MichalStrehovsky advice to run on Windows does not applied.

[MichalStrehovsky]

Which reloc is that? (It should be visible which data blob the reloc is coming from on the managed side, in ObjectWriter.cs).

Some work was done to support the various ARM32 ELF relocs, but maybe we regressed somewhere (e.g. dotnet/corert#4899).

[kant2002]

I actually not sure where this happens. since this is not a crash, just "unsupported" scenario which exit ILC. I will try to find what reloc is that. Also thanks for the hint, I will take a look and try to understand what can go wrong.

[kant2002]

Okay. This is some sort of crash. At least application exits at call to FinishObjWriter, and since I run it on Windows it's from Nuget package, which miss PDB. If I compile ObjWriter.dll on Windows, can I expect that it would work for ARM ELF generation?

[kant2002]

Error happens when MCFixup.getKind() == FK_Data_8 (3), IsPCRel == false at this line
https://github.com/llvm/llvm-project/blob/d480f968ad8b56d3ee4a6b6df5532d485b0ad01e/llvm/lib/Target/ARM/MCTargetDesc/ARMELFObjectWriter.cpp#L162

Stack trace.

objwriter.dll!`anonymous namespace'::ARMELFObjectWriter::GetRelocTypeInner(const llvm::MCValue & Target, const llvm::MCFixup & Fixup, bool IsPCRel, llvm::MCContext & Ctx) Line 167
	at runtimelab\artifacts\llvm\lib\Target\ARM\MCTargetDesc\ARMELFObjectWriter.cpp(167)
objwriter.dll!`anonymous namespace'::ARMELFObjectWriter::getRelocType(llvm::MCContext & Ctx, const llvm::MCValue & Target, const llvm::MCFixup & Fixup, bool IsPCRel) Line 73
	at runtimelab\artifacts\llvm\lib\Target\ARM\MCTargetDesc\ARMELFObjectWriter.cpp(73)
[Inline Frame] objwriter.dll!`anonymous-namespace'::ELFObjectWriter::getRelocType(llvm::MCContext &) Line 155
	at runtimelab\artifacts\llvm\lib\MC\ELFObjectWriter.cpp(155)
objwriter.dll!`anonymous namespace'::ELFObjectWriter::recordRelocation(llvm::MCAssembler & Asm, const llvm::MCAsmLayout & Layout, const llvm::MCFragment * Fragment, const llvm::MCFixup & Fixup, llvm::MCValue Target, unsigned __int64 & FixedValue) Line 694
	at runtimelab\artifacts\llvm\lib\MC\ELFObjectWriter.cpp(694)
objwriter.dll!llvm::MCAssembler::handleFixup(const llvm::MCAsmLayout & Layout, llvm::MCFragment & F, const llvm::MCFixup & Fixup) Line 661
	at runtimelab\artifacts\llvm\lib\MC\MCAssembler.cpp(661)
objwriter.dll!llvm::MCAssembler::layout(llvm::MCAsmLayout & Layout) Line 743
	at runtimelab\artifacts\llvm\lib\MC\MCAssembler.cpp(743)
objwriter.dll!llvm::MCAssembler::Finish() Line 755
	at runtimelab\artifacts\llvm\lib\MC\MCAssembler.cpp(755)
[Inline Frame] objwriter.dll!ObjectWriter::Finish() Line 188
	at runtimelab\src\coreclr\tools\aot\ObjWriter\objwriter.cpp(188)
objwriter.dll!FinishObjWriter(ObjectWriter * OW) Line 208
	at runtimelab\src\coreclr\tools\aot\ObjWriter\objwriter.h(208)
[Managed to Native Transition]
ILCompiler.Compiler.dll!ILCompiler.DependencyAnalysis.ObjectWriter.Dispose(bool bDisposing) Line 891
	at runtimelab\src\coreclr\tools\aot\ILCompiler.Compiler\Compiler\DependencyAnalysis\ObjectWriter.cs(891)
ILCompiler.Compiler.dll!ILCompiler.DependencyAnalysis.ObjectWriter.Dispose() Line 881
	at runtimelab\src\coreclr\tools\aot\ILCompiler.Compiler\Compiler\DependencyAnalysis\ObjectWriter.cs(881)
ILCompiler.Compiler.dll!ILCompiler.DependencyAnalysis.ObjectWriter.EmitObject(string objectFilePath, System.Collections.Generic.IEnumerable<ILCompiler.DependencyAnalysisFramework.DependencyNode> nodes, ILCompiler.DependencyAnalysis.NodeFactory factory, ILCompiler.DependencyAnalysis.ObjectWritingOptions options, ILCompiler.DependencyAnalysis.IObjectDumper dumper) Line 1157
	at runtimelab\src\coreclr\tools\aot\ILCompiler.Compiler\Compiler\DependencyAnalysis\ObjectWriter.cs(1157)
ILCompiler.RyuJit.dll!ILCompiler.RyuJitCompilation.CompileInternal(string outputFile, ILCompiler.ObjectDumper dumper) Line 75
	at runtimelab\src\coreclr\tools\aot\ILCompiler.RyuJit\Compiler\RyuJitCompilation.cs(75)
ILCompiler.Compiler.dll!ILCompiler.Compilation.ILCompiler.ICompilation.Compile(string outputFile, ILCompiler.ObjectDumper dumper) Line 503
	at runtimelab\src\coreclr\tools\aot\ILCompiler.Compiler\Compiler\Compilation.cs(503)
ilc.dll!ILCompiler.Program.Run(string[] args) Line 757
	at runtimelab\src\coreclr\tools\aot\ILCompiler\Program.cs(757)
ilc.dll!ILCompiler.Program.Main(string[] args) Line 928
	at runtimelab\src\coreclr\tools\aot\ILCompiler\Program.cs(928)

So far, I will try to get reloc from that stack trace. But not sure what I should look for and where to continue search for now. Will continue, but gladly ask for any help or advice.

[kant2002]

From what I understand.
The issue happens in first Section (index 0) and 3rd Fragment of that section (index 2) on first Fixup (index 0).
Section 0 is "__managed_code". Fixup has Offset == 0 and seems to be point to Loc == NULL.
And now I'm stuck. My next steps would be understanding who specifically wrote that Section, Fragment and who detect or write Fixups.

libObjwriter seems to be producing optimized code, so debugging of iterators a bit problematic. Is it possible to build it in Debug mode. I was using build.cmd nativeaot.objwriter -rc Debug -lc Release.

[MichalStrehovsky]

ILCompiler.csproj is hardcoded to pick up the Release version of ObjWriter. Look for ObjWriterArtifactPath - this needs to be set to Debug to pick up your debug build of it. I think it's also hardcoded to release in objwriter.proj - you need to set that to Debug as well to build the Debug version in the first place.

All the relocs you see should go through ObjectWriter::EmitSymbolRef (src\coreclr\tools\aot\ObjWriter\objwriter.cpp) so maybe there's something wrong in how the reloc is translated to LLVM terms there. I don't see anything obvious that would be producing 8 byte non-PC-relative relocs on ARM there, unless we're somehow ending up with a IMAGE_REL_BASED_DIR64 reloc (might want to set breakpoint on that just to be sure).

[kant2002]

Debug version was fruitful.
I hit assertion for Symbol _ZTV6Object at
https://github.com/llvm/llvm-project/blob/bbea64250f65480d787e1c5ff45c4de3ec2dcda8/llvm/include/llvm/MC/MCFixup.h#L111
where Size == 0
Which is passed from ObjWriter at following location

runtimelab/src/coreclr/tools/aot/ObjWriter/objwriter.cpp

Line 473 in 4ffe2eb

const MCExpr *TargetExpr = GenTargetExpr(SymbolName, Kind, Delta, IsPCRel, Size);

From my understanding this is caused that C# pass RelocType = IMAGE_REL_BASED_THUMB_MOV32_PCREL which is not present in C++ side and thus not handled.

[kant2002]

So far I look at the NativeAOT branch to check where IMAGE_REL_BASED_THUMB_MOV32 was processed and where processing of IMAGE_REL_BASED_REL_THUMB_MOV32_REL was missing. I understand that both should not go together, but in some places they have identical processing so I decide to at least record these sessions. Some places seems a bit fishy to me.

Loc 1

runtimelab/src/coreclr/debug/daccess/nidump.cpp

Lines 5343 to 5350 in 4ffe2eb

	const NativeImageDumper::EnumMnemonics s_RelocType[] =
	{
	#define REL_ENTRY(x) NativeImageDumper::EnumMnemonics( x, 0, W(#x))
	REL_ENTRY(IMAGE_REL_BASED_ABSOLUTE),
	REL_ENTRY(IMAGE_REL_BASED_HIGHLOW),
	REL_ENTRY(IMAGE_REL_BASED_DIR64),
	REL_ENTRY(IMAGE_REL_BASED_THUMB_MOV32),
	#undef REL_ENTRY

If this is Windows only that make sense, or at least maybe this is Win ARM only.

Loc 2

runtimelab/src/coreclr/tools/aot/ILCompiler.Compiler/Compiler/DependencyAnalysis/ObjectWriter.cs

Lines 1069 to 1076 in 4ffe2eb

	{
	case RelocType.IMAGE_REL_BASED_THUMB_MOV32:
	case RelocType.IMAGE_REL_BASED_THUMB_BRANCH24:
	case RelocType.IMAGE_REL_BASED_ARM64_BRANCH26:
	case RelocType.IMAGE_REL_BASED_ARM64_PAGEBASE_REL21:
	case RelocType.IMAGE_REL_BASED_ARM64_PAGEOFFSET_12A:
	case RelocType.IMAGE_REL_BASED_ARM64_PAGEOFFSET_12L:
	unsafe

Not sure that this location may benefit from adding one more case.

Loc 3

runtimelab/src/coreclr/tools/aot/ObjWriter/objwriter.cpp

Line 440 in 4ffe2eb

case RelocType::IMAGE_REL_BASED_THUMB_MOV32: {

I think I would like add one case here, but maybe I should copy something like this

runtimelab/src/coreclr/tools/aot/ObjWriter/objwriter.cpp

Lines 435 to 439 in 4ffe2eb

	case RelocType::IMAGE_REL_BASED_RELPTR32:
	Size = 4;
	IsPCRel = true;
	Delta += 4;
	break;

Loc 4

runtimelab/src/coreclr/tools/Common/Compiler/DependencyAnalysis/ObjectDataBuilder.cs

Line 302 in 4ffe2eb

case RelocType.IMAGE_REL_BASED_THUMB_MOV32:

First impression is to add one more case here, but with my limited knowledge it can be and case above:

runtimelab/src/coreclr/tools/Common/Compiler/DependencyAnalysis/ObjectDataBuilder.cs

Lines 294 to 295 in 4ffe2eb

	case RelocType.IMAGE_REL_SYMBOL_SIZE:
	EmitInt(delta);

Loc 5

runtimelab/src/coreclr/tools/Common/Compiler/DependencyAnalysis/Relocation.cs

Line 388 in 4ffe2eb

case RelocType.IMAGE_REL_BASED_THUMB_MOV32:

This is most likely proper handling, but include this for completeness.

Loc 6

runtimelab/src/coreclr/vm/peimagelayout.cpp

Line 270 in 4ffe2eb

case IMAGE_REL_BASED_THUMB_MOV32:

If I understand properly this is for Win AMR32 and I should ignore this.

This seems to be location which trigger creation of reloc which crash ObjWriter.

runtimelab/src/coreclr/jit/emit.cpp

Lines 8772 to 8779 in 4ffe2eb

	if (emitComp->opts.jitFlags->IsSet(JitFlags::JIT_FLAG_RELATIVE_CODE_RELOCS))
	{
	emitRecordRelocation(location, target, IMAGE_REL_BASED_REL_THUMB_MOV32_PCREL);
	}
	else
	{
	emitRecordRelocation(location, target, IMAGE_REL_BASED_THUMB_MOV32);
	}

[MichalStrehovsky]

I think the relocation is generated by RyuJIT in "Loc 6" you identified above because we set the JIT_FLAG_RELATIVE_CODE_RELOCS here:

runtimelab/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs

Lines 3755 to 3756 in 4ffe2eb

	if (targetArchitecture == TargetArchitecture.ARM && !_compilation.TypeSystemContext.Target.IsWindows)
	flags.Set(CorJitFlag.CORJIT_FLAG_RELATIVE_CODE_RELOCS);

This was added when Samsung was fixing crossgen2 for Linux ARM32 here: dotnet/runtime#33153

I'm not sure we need it for NativeAOT. Samsung was running CoreRT on ARM32 in the past. So it might be worth trying to put those two lines under #if READYTORUN.

The other option (if ifdeffing causes issues at runtime) is to pipe the new relocation through to the object writer. It will go somewhere along the lines of dotnet/corert#3433.

[jkotas]

CORJIT_FLAG_RELATIVE_CODE_RELOCS should not be set for NativeAOT ARM32.

[kant2002]

Now I have another problem and code fails with assertion here.

Expression: I != M+Size && I->FromReg == RegNum && "Invalid RegNum"

This from here
https://github.com/llvm/llvm-project/blame/2c99246624b1a6f6673a613fe5a5548e5f2b7a87/llvm/lib/MC/MCRegisterInfo.cpp#L89
But unfortunately this code is not the latest now with changes introduced in llvm/llvm-project@aaff1a6
Should I try apply this commit locally, and if that works modify llvm.patch in objwriter?