Update docker images to latest tags

[am11]

Revives #3337
Supersedes #3492

[mikem8361]

I'm not sure about 2 of the CI failures (SOS.TaskNestedException/SOS.LineNums). It looks like newer lldb is segfaulting when loading the help python script (lldbhelper.py).

The other EEStack on 3.1 failure looks like a bug in the DAC GetJumpThunkTarget API that for some reason is being hit more with the new lldb. It can be disabled for 3.1 in the StackAndOtherTests script. DumpStack/EEStack are kind of unreliable on Linux/MacOS because there isn't any SEH to catch the segfaults like on Windows.

[am11]

Not sure how they are related. Last commit has no CentOS related change, and all tests were passing in previous commit. Maybe they are flaky tests?

[am11]

flaky tests

Resetting to 12fe666 revealed that it is indeed the case of flaky test. It's failing now while it was passing in previous attempt.

The other EEStack on 3.1 failure looks like a bug in the DAC GetJumpThunkTarget API that for some reason is being hit more with the new lldb.

netcoreapp3.1 and net5.0 are out of support, not sure how valuable it is to make them work with new lldb.

[mikem8361]

Yes, NET Core 3.1 is going out of support (I think this month) but we haven't had a chance to remove testing against (and adding net 8.0). These DumpStack/EEStack command tests are already disabled for 7.0 and above. It is the commands themselves that make testing them unreliable. Maybe they should be disabled for Linux/MacOS under lldb for all runtime versions.

The other failures (SOS.TaskNestedException/SOS.LineNums) seem to be related to the new lldb (or at least the new lldb's python script support). I really don't know what to do about them.

[mikem8361]

I'm not sure what to do here as far as the python script crash on lldb startup. Maybe @hoyosjs has some ideas.

[am11]

the python script crash on lldb startup.

The python script runs for every test, but not every test is failing. The python script looks related but it isn't.

If I run the container locally with a test script like this:

eng/docker-build.sh --docker-image mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7 --source-directory $(pwd) --container-name diagnostics-100832 sh test.sh

where test.sh is:

/usr/local/bin/lldb --no-lldbinit \
  -o "settings set target.disable-aslr false" \
  -o "settings set interpreter.prompt-on-quit false" \
  -o "command script import /home/am11/projects/diagnostics/src/SOS/SOS.UnitTests/Scripts/lldbhelper.py" \
  -o version \
  $(pwd)/.dotnet-test/dotnet

it succeed, but if I remove $(pwd)/ from the last line:

/usr/local/bin/lldb --no-lldbinit \
  -o "settings set target.disable-aslr false" \
  -o "settings set interpreter.prompt-on-quit false" \
  -o "command script import /home/am11/projects/diagnostics/src/SOS/SOS.UnitTests/Scripts/lldbhelper.py" \
  -o version \
  .dotnet-test/dotnet

(which we are seeing in the CI build but I haven't found where in the code we are transforming those paths...) it throws sigsegv:

Initialize Docker Container
Docker version 20.10.18, build b40c2f6
user name: am11
user id: 1000
centos-7: Pulling from dotnet-buildtools/prereqs
Digest: sha256:55fb89b6c779a7e4afb3125d12de6b5130e574fb5b0342e3d7a7f76500bd2cd6
Status: Image is up to date for mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7
mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7
89c95a5a96f08388ca29dffc54ce8a833eef24e60620bf114f963c8f60fa875d
721552fad213bcbb1775f873eca64b7fce9254c4a55be73af5ecca2f1d08e975
container user name: vsts_am11
Creating mailbox file: Not a directory
Execute  sh test.sh
(lldb) target create ".dotnet-test/dotnet"
Current executable set to '.dotnet-test/dotnet' (x86_64).
(lldb) settings set target.disable-aslr false
(lldb) settings set interpreter.prompt-on-quit false
(lldb) command script import /home/am11/projects/diagnostics/src/SOS/SOS.UnitTests/Scripts/lldbhelper.py
 #0 0x00000000004214aa llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/local/bin/lldb+0x4214aa)
 #1 0x000000000041fa64 llvm::sys::RunSignalHandlers() (/usr/local/bin/lldb+0x41fa64)
 #2 0x000000000041fbc5 SignalHandler(int) (/usr/local/bin/lldb+0x41fbc5)
 #3 0x00007fee54458630 __restore_rt (/lib64/libpthread.so.0+0xf630)
 #4 0x00007fee4c669582 wcscmp (/lib64/libc.so.6+0xa7582)
 #5 0x00007fee4bfcca64 parse_cmd (/lib64/libedit.so.0+0x13a64)
 #6 0x00007fee4bfcba0d map_bind (/lib64/libedit.so.0+0x12a0d)
 #7 0x00007fee4bfdddc4 el_set (/lib64/libedit.so.0+0x24dc4)
 #8 0x00007fee4bfdd237 rl_variable_bind (/lib64/libedit.so.0+0x24237)
 #9 0x00007fee420ddbe3 initreadline (/usr/lib64/python2.7/lib-dynload/readline.so+0x3be3)
#10 0x00007fee4c2f0379 _PyImport_LoadDynamicModule (/lib64/libpython2.7.so.1.0+0xfa379)
#11 0x00007fee4c2ee451 (/lib64/libpython2.7.so.1.0+0xf8451)
#12 0x00007fee4c2ee736 (/lib64/libpython2.7.so.1.0+0xf8736)
#13 0x00007fee4c2ef07e PyImport_ImportModuleLevel (/lib64/libpython2.7.so.1.0+0xf907e)
#14 0x00007fee4c2d232f (/lib64/libpython2.7.so.1.0+0xdc32f)
#15 0x00007fee4c242073 PyObject_Call (/lib64/libpython2.7.so.1.0+0x4c073)
#16 0x00007fee4c2d3f07 PyEval_CallObjectWithKeywords (/lib64/libpython2.7.so.1.0+0xddf07)
#17 0x00007fee4c2d8bc5 PyEval_EvalFrameEx (/lib64/libpython2.7.so.1.0+0xe2bc5)
#18 0x00007fee4c2dd64d PyEval_EvalCodeEx (/lib64/libpython2.7.so.1.0+0xe764d)
#19 0x00007fee4c2dd752 PyEval_EvalCode (/lib64/libpython2.7.so.1.0+0xe7752)
#20 0x00007fee4c2ed53c PyImport_ExecCodeModuleEx (/lib64/libpython2.7.so.1.0+0xf753c)
#21 0x00007fee4c2ed7b8 (/lib64/libpython2.7.so.1.0+0xf77b8)
#22 0x00007fee4c2ee451 (/lib64/libpython2.7.so.1.0+0xf8451)
#23 0x00007fee4c2ee69d (/lib64/libpython2.7.so.1.0+0xf869d)
#24 0x00007fee4c2ef0b8 PyImport_ImportModuleLevel (/lib64/libpython2.7.so.1.0+0xf90b8)
#25 0x00007fee4c2d232f (/lib64/libpython2.7.so.1.0+0xdc32f)
#26 0x00007fee4c242073 PyObject_Call (/lib64/libpython2.7.so.1.0+0x4c073)
#27 0x00007fee4c2d3f07 PyEval_CallObjectWithKeywords (/lib64/libpython2.7.so.1.0+0xddf07)
#28 0x00007fee4c2d8bc5 PyEval_EvalFrameEx (/lib64/libpython2.7.so.1.0+0xe2bc5)
#29 0x00007fee4c2dd64d PyEval_EvalCodeEx (/lib64/libpython2.7.so.1.0+0xe764d)
#30 0x00007fee4c2dd752 PyEval_EvalCode (/lib64/libpython2.7.so.1.0+0xe7752)
#31 0x00007fee4c2f6b8f (/lib64/libpython2.7.so.1.0+0x100b8f)
#32 0x00007fee4c2f7a05 PyRun_StringFlags (/lib64/libpython2.7.so.1.0+0x101a05)
#33 0x00007fee4c2f93ab PyRun_SimpleStringFlags (/lib64/libpython2.7.so.1.0+0x1033ab)
#34 0x00007fee4ee37a6d lldb_private::ScriptInterpreterPythonImpl::InitializePrivate() (/usr/local/bin/../lib64/liblldb.so.9+0x143ba6d)
#35 0x00007fee4ee3b364 lldb_private::ScriptInterpreterPythonImpl::ScriptInterpreterPythonImpl(lldb_private::Debugger&) (/usr/local/bin/../lib64/liblldb.so.9+0x143f364)
#36 0x00007fee4ee3b5c9 lldb_private::ScriptInterpreterPythonImpl::CreateInstance(lldb_private::Debugger&) (/usr/local/bin/../lib64/liblldb.so.9+0x143f5c9)
#37 0x00007fee4e927eb9 lldb_private::PluginManager::GetScriptInterpreterForLanguage(lldb::ScriptLanguage, lldb_private::Debugger&) (/usr/local/bin/../lib64/liblldb.so.9+0xf2beb9)
#38 0x00007fee4e8dcaaa lldb_private::Debugger::GetScriptInterpreter(bool) (/usr/local/bin/../lib64/liblldb.so.9+0xee0aaa)
#39 0x00007fee50279076 CommandObjectCommandsScriptImport::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) (/usr/local/bin/../lib64/liblldb.so.9+0x287d076)
#40 0x00007fee4e9ddcb1 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) (/usr/local/bin/../lib64/liblldb.so.9+0xfe1cb1)
#41 0x00007fee4e9d035f lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&, lldb_private::ExecutionContext*, bool, bool) (/usr/local/bin/../lib64/liblldb.so.9+0xfd435f)
#42 0x00007fee4e9d1c32 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::string&) (/usr/local/bin/../lib64/liblldb.so.9+0xfd5c32)
#43 0x00007fee4e90f2f1 lldb_private::IOHandlerEditline::Run() (/usr/local/bin/../lib64/liblldb.so.9+0xf132f1)
#44 0x00007fee4e8df9b4 lldb_private::Debugger::ExecuteIOHandlers() (/usr/local/bin/../lib64/liblldb.so.9+0xee39b4)
#45 0x00007fee4e9c8933 lldb_private::CommandInterpreter::RunCommandInterpreter(bool, bool, lldb_private::CommandInterpreterRunOptions&) (/usr/local/bin/../lib64/liblldb.so.9+0xfcc933)
#46 0x00007fee4e5a63f1 lldb::SBDebugger::RunCommandInterpreter(bool, bool, lldb::SBCommandInterpreterRunOptions&, int&, bool&, bool&) (/usr/local/bin/../lib64/liblldb.so.9+0xbaa3f1)
#47 0x000000000040ae1e Driver::MainLoop() (/usr/local/bin/lldb+0x40ae1e)
#48 0x000000000040916f main (/usr/local/bin/lldb+0x40916f)
#49 0x00007fee4c5e4555 __libc_start_main (/lib64/libc.so.6+0x22555)
#50 0x000000000040a105 _start (/usr/local/bin/lldb+0x40a105)
Stack dump:
0.	Program arguments: /usr/local/bin/lldb --no-lldbinit -o settings set target.disable-aslr false -o settings set interpreter.prompt-on-quit false -o command script import /home/am11/projects/diagnostics/src/SOS/SOS.UnitTests/Scripts/lldbhelper.py -o version .dotnet-test/dotnet 
test.sh: line 6:    59 Segmentation fault      (core dumped) /usr/local/bin/lldb --no-lldbinit -o "settings set target.disable-aslr false" -o "settings set interpreter.prompt-on-quit false" -o "command script import /home/am11/projects/diagnostics/src/SOS/SOS.UnitTests/Scripts/lldbhelper.py" -o version .dotnet-test/dotnet

Current theory is that this is happening with dot files.

[mikem8361]

It looks like your recent change made the tests fail with the segfault in python even more.

[am11]

I added "Print pwd" temporarily to check the current path and compare it to the one in the error message. It has revealed nothing new to the experiment in my previous comment, so it can be deleted.

[am11]

After merge from main, only SOS.TaskNestedException test is failing.

I think these are intermittent failures, re-running the CI build multiple times against the same commit will give different results. It could be the problem with this specific lldb and python 2.7 installation.

[mikem8361]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mikem8361]

This PR isn't done yet: 2 of the new docker images don't have lldb installed:

Debian_Bullseye         mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-amd64
Ubuntu_18_04            mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04

I need to figure out how to add lldb to these images.