ignore '**/java/lang/invoke/**' from runtimeClassPath

[kenyee]

We're hitting transform differences in cache artifacts:

Only in /tmp/extracted/java.base: NOTICES
Binary files extracted/java.base/java/lang/invoke/DelegatingMethodHandle$Holder.class and /tmp/extracted/java.base/java/lang/invoke/DelegatingMethodHandle$Holder.class differ
Binary files extracted/java.base/java/lang/invoke/DirectMethodHandle$Holder.class and /tmp/extracted/java.base/java/lang/invoke/DirectMethodHandle$Holder.class differ

I was asking about it in this thread: https://gradle-community.slack.com/archives/CAHSN3LDN/p1661952779052469

And someone mentioned, they added this to work around this issue:

allProjects {
  normalization {
    runtimeClasspath {
      ignore '**/module-descriptor.txt' // don't need if you are up to date on AGP cache plug in fix
      ignore '**/java/lang/invoke/**'
    }
}

It looks like this plugin has a JdkImageWorkaround class that does this normalization as well but doesn't cover this case:
https://github.com/gradle/android-cache-fix-gradle-plugin/blob/d7f486e318e3698afa58834917b781cb0ef151c8/src/main/groovy/org/gradle/android/workarounds/JdkImageWorkaround.groovy

Would be nice if it were done in this plugin as well so we don't have to and this would improve cache hits for everyone.

[ghale]

Since this looks like differences in the compiled class files, I would assume that the JDKs being used are slightly different versions (e.g. 11.0.12 vs 11.0.13). The goal of JdkImageWorkaround is to fix the problems with this command line provider when the JDK version is the same, but the distribution is different (i.e. linux vs mac). Now, it may work between slightly different versions, but there are sometimes differences and it's up to the user to decide if those differences are important or not. You can then further normalize the runtime classpath to exclude those differences if you choose to, but I think it would be presumptive for the workaround to make that decision for you.

On the other hand, if this is somehow the same JDK version that actually has different class files between OS, then that would be surprising. If that's the case, it would be interesting to know what specific distro and version you're seeing this with.

[kenyee]

It's an internal Twitter OpenJDK11 build unfortunately (our folks tweak it for server usage and then we all use it) and the version #s match exactly. Our CI servers are x86 linux based while our laptops are ARM macbooks.
I'll check w/ our VM folks to confirm it's based on the same source code and version.

[kenyee]

FWIW, our VM folks confirmed same version.

That $Holder class is apparently generated code: https://github.com/openjdk/jdk11u/blob/73eef16128417f4a489c4dde47383bb4a00f39d4/src/jdk.jlink/share/classes/jdk/tools/jlink/internal/plugins/GenerateJLIClassesPlugin.java#L433
https://github.com/openjdk/jdk11u/blob/73eef16128417f4a489c4dde47383bb4a00f39d4/src/java.base/share/classes/java/lang/invoke/GenerateJLIClassesHelper.java#L105

And the code varies slightly between x86/ARM architectures despite it generating JVM bytecode. That's pretty gnarly code...never knew the JVM did this... 😸
I suspect other folks will hit this as well or have hit it but don't realize it.
Will report back if I have any other info...

[ghale]

Wow. This is news to me, too. It's probably worth considering adding this then.

[kenyee]

Our VM folks took a deeper dig and decompiled the two class files...apparently the ARM one added two extra methods:

  static java.lang.Object reinvoke_L(java.lang.Object, int, java.lang.Object);
    descriptor: (Ljava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object;
    flags: (0x0008) ACC_STATIC
    Code:
      stack=3, locals=4, args_size=3
         0: aload_0
         1: checkcast     #14                 // class java/lang/invoke/BoundMethodHandle$Species_L
         4: getfield      #18                 // Field java/lang/invoke/BoundMethodHandle$Species_L.argL0:Ljava/lang/Object;
         7: astore_3
         8: aload_3
         9: checkcast     #20                 // class java/lang/invoke/MethodHandle
        12: iload_1
        13: aload_2
        14: invokevirtual #253                // Method java/lang/invoke/MethodHandle.invokeBasic:(ILjava/lang/Object;)Ljava/lang/Object;
        17: areturn
    RuntimeVisibleAnnotations:
      0: #8()
        java.lang.invoke.LambdaForm$Hidden
      1: #9()
        java.lang.invoke.LambdaForm$Compiled
      2: #10()
        jdk.internal.vm.annotation.ForceInline

  static java.lang.Object delegate(java.lang.Object, int, java.lang.Object);
    descriptor: (Ljava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object;
    flags: (0x0008) ACC_STATIC
    Code:
      stack=3, locals=4, args_size=3
         0: aload_0
         1: checkcast     #27                 // class java/lang/invoke/DelegatingMethodHandle
         4: invokevirtual #31                 // Method java/lang/invoke/DelegatingMethodHandle.getTarget:()Ljava/lang/invoke/MethodHandle;
         7: astore_3
         8: aload_3
         9: checkcast     #20                 // class java/lang/invoke/MethodHandle
        12: iload_1
        13: aload_2
        14: invokevirtual #253                // Method java/lang/invoke/MethodHandle.invokeBasic:(ILjava/lang/Object;)Ljava/lang/Object;
        17: areturn
    RuntimeVisibleAnnotations:
      0: #8()
        java.lang.invoke.LambdaForm$Hidden
      1: #9()
        java.lang.invoke.LambdaForm$Compiled
      2: #10()
        jdk.internal.vm.annotation.ForceInline

And their interpretation of what's going on in that bytecode generator:

"that class is a bucket of ahead of time compiled delegating method handles based on what's passed to jlink. My guess would be that there's a platform specific class that uses different delegating method handles between arm and x86, and when jlink creates that class, those precreated delegated handles differ"

[kenyee]

FYI, the person in that gradle slack thread mentioned this:
"Liberica arm64 for macs with M1 chips. Open jdk 11 for Mac books with intel.
All CI machines are m1. Engineers may have intel or m1"

[dannybduval]

🖐️ - we put the exclude in the normalization after looking at the diffs . They only appeared under '**/java/lang/invoke/**', so we limited the scope to that.

Hopefully we all end up on M1 machines + a consistent JDK and that part can be removed.

[runningcode]

I understand that this issue that there are differences in the java.lang.invoke packages between different JDK images happens quite often. I think it could make sense to ignore this using runtime classpath normalization.

However, because runtime classpath normalization can only be applied on a project basis and it cannot be made task-specific or project specific I would favor not implicitly ignoring these for all tasks and inputs in the projects.
I think we should add a sample or link to a github gist with the code to apply this runtime classpath normalization but it should not be added as code to the Android Cache Fix plugin itself.