Interpreter to JIT/AOT calls

[janvorli]

This change adds support for making calls from the interpreter to JIT/AOT generated code. For each target method, it parses the signature and creates a list of hand written asm routines that transfer the arguments from the interpreter stack to the CPU registers / stack based on the native calling convention, call the target method and then places the return value to the interpreter stack. This list is cached in the MethodDescData so that for repeated calls to the same method, it doesn't need to be re-generated.

For example, let's say that interpreter needs to call a JIT/AOT generated method with the following signature on Windows x64:

long M(int a, int b, long c, double d)

The interpreter stack stores these arguments aligned to 8 byte slots like this:

Slot 0: a
Slot 1: b
Slot 2: c
Slot 3: d

The CallStubGenerator::GenerateCallStub would then generate the following list of routines:

Load_RCX_RDX_R8
Load_XMM3
<target method address>

• The interpreter calls the CallJittedMethodRetI8 function, because the return type is long. The arguments passed to this function are the list of routines above, pointer to slot 0 mentioned above, pointer to the return value slot on the interpreter stack and the size of the stack arguments the target method uses (0 in this case, as all arguments are passed in registers).
• The CallJittedMethodRetI8 calls the first routine in the list
• The Load_RCX_RDX_R8 loads RCX from slot 0, RDX from slot 1, R8 from slot 2 and then jumps to the next routine, which is the Load_XMM3
• The Load_XMM3 loads XMM3 from the slot 3 and jumps to the next routine, which is the target method
• After the target method returns, the control flow gets back to the CallJittedMethodRetI8. That function stores the RAX that contains the result to the return value slot on the interpreter stack and returns back to the interpreter.

[Copilot]

Pull Request Overview

This PR implements support for making calls from the interpreter to JIT/AOT generated code by generating and caching call stubs for target methods. The key changes include:

• Adding new fields and methods in MethodDesc to store and manage call stub headers.
• Implementing a new routine in interpexec.cpp to invoke compiled methods via generated call stubs.
• Adding new assembly routines in AsmHelpers.asm and updating CMakeLists.txt to include the call stub generator sources.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/method.hpp	Added call stub header field and new API for call stub management
src/coreclr/vm/method.cpp	Implemented SetCallStubHeader and GetCallStubHeader methods
src/coreclr/vm/interpexec.cpp	Added InvokeCompiledMethod and updated interpreter branch to handle JIT/AOT calls
src/coreclr/vm/callstubgenerator.h	Introduced the definition for call stub generation
src/coreclr/vm/amd64/AsmHelpers.asm	Added multiple assembly routines to move arguments and perform calls
src/coreclr/vm/CMakeLists.txt	Updated build configuration to include call stub generator sources

[Copilot]

Using printf and assert directly for error handling may not be robust in production. Consider replacing with structured error logging or exception handling to better manage execution in release builds.

[dotnet-policy-service]

Tagging subscribers to this area: @BrzVlad, @janvorli, @kg
See info in area-owners.md if you want to be subscribed.

[kg]

This was very easy to adapt for p/invoke, I like how you engineered it.

[janvorli]

The PR is now ready for review.

[jkotas]

Suggested change

	if (!codeInfo.IsValid())
	{
	EEPOLICY_HANDLE_FATAL_ERROR_WITH_MESSAGE(COR_E_EXECUTIONENGINE, W("Attempted to execute native code from interpreter"));
	}
	else if (codeInfo.GetCodeManager() != ExecutionManager::GetInterpreterCodeManager())
	if (!codeInfo.IsValid() || codeInfo.GetCodeManager() != ExecutionManager::GetInterpreterCodeManager())

I do not see why we need to block native code (FCalls?) here. Should this be like this?

(I agree with TODO that this needs to be faster.)

[janvorli]

This was initially a temporary check to make sure something unexpected doesn't leak in here. As the comment mentions, I want to get rid of the codeInfo construction here soon and move to tagged pointer for the interpreter code so that we don't need to burn time looking up the code ranges. Then this will go away.
As for FCalls, I have made a local change in a testing branch yesterday to make them work, but they need to be handled explicitly, as their GetNativeCode returns NULL, so I have used the TryGetMultiCallableAddrOfCode. Maybe we can use that function instead of the GetNativeCode for all calls, but I need to refresh my memory on the differences.

[jkotas]

LGTM. Somebody from the interpreter v-team should sign-off as well.

[BrzVlad]

I think it would be great if we didn't have to work with MethodDesc at all for the fast invocation path. Also if these transition thunks would be shared per signature and not owned by each method independently, meaning the compiled targetIp would be passed explicitly. So a call pseudocode could look like:

obtain targetIp;
if (targetIp is not interp)
    callStubInvoke = check call site cache
    if (!callStubInvoke)
         build CallStubGenerator
         set callStubInvoke and write cache
    callStubInvoke->Invoke(targetIp, ...)
else
    interp call 

[janvorli]

I have considered generating the thunk per (normalized) signature, but decided to leave it as a possible future optimization. I don't have a good idea yet on how to make normalized signature comparison and generate cache key for a signature that would be significantly faster than generating it per method. And the size of the thunk is very small, so I am not sure how much we would save space-wise. Anyways, it is still worth looking into at some point.

[BrzVlad]

Nice