ILCompiler no longer works with paths containing spaces

[eerhardt]

Description

Installing the .NET SDK to a path with a space in it (i.e. C:\Program Files\dotnet) no longer works with PublishAot=true.

Reproduction Steps

1. Install the latest 8.0 SDK from https://github.com/dotnet/installer#installers-and-binaries to a path with a space - C:\dot net.
2. Put that path on your $env:PATH
3. Add <add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" /> to your nuget.config.
4. dotnet new console
5. Change <TargetFramework>net7.0</TargetFramework> => <TargetFramework>net8.0</TargetFramework>
6. dotnet publish -c Release -p:PublishAot=true -r win-x64 /bl

Expected behavior

The publish should succeed.

Actual behavior

It fails with error:

C:\dot net\sdk\8.0.100-alpha.1.22477.3\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(257,5): message NETSDK1057: You are using a
preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\DotNetTest\HelloWorld\HelloWorld.csproj]
  HelloWorld -> C:\DotNetTest\HelloWorld\bin\Release\net8.0\win-x64\HelloWorld.dll
  Generating native code
  Unhandled exception. System.CommandLine.CommandLineException: No files matching net\sdk\8.0.100-alpha.1.22477.3\Sdks\Microsoft.DotNet.ILCompiler\build\WindowsAPI
  s.txt
     at System.CommandLine.Helpers.AppendExpandedPaths(Dictionary`2, String, Boolean) in /_/src/coreclr/tools/Common/CommandLineHelpers.cs:line 282
     at System.CommandLine.Helpers.BuildPathDictionay(IReadOnlyList`1, Boolean) in /_/src/coreclr/tools/Common/CommandLineHelpers.cs:line 33
     at System.CommandLine.Argument`1.<>c__DisplayClass5_0.<.ctor>b__1(ArgumentResult, Object& )
     at System.CommandLine.Parsing.ArgumentResult.Convert(Argument)
     at System.CommandLine.Parsing.ArgumentResult.GetArgumentConversionResult()
     at System.CommandLine.Parsing.ParseResultVisitor.ValidateAndConvertArgumentResult(ArgumentResult)
     at System.CommandLine.Parsing.ParseResultVisitor.ValidateAndConvertArgumentResults(IReadOnlyList`1, Int32)
     at System.CommandLine.Parsing.ParseResultVisitor.Stop()
     at System.CommandLine.Parsing.Parser.Parse(IReadOnlyList`1, String )
     at ILCompiler.Program.Main(String[]) in /_/src/coreclr/tools/aot/ILCompiler/Program.cs:line 763
C:\dot net\sdk\8.0.100-alpha.1.22477.3\Sdks\Microsoft.DotNet.ILCompiler\build\Microsoft.NETCore.Native.targets(278,5): error MSB3073: The command ""c:\dotnet\packa
ges\runtime.win-x64.microsoft.dotnet.ilcompiler\8.0.0-alpha.1.22475.4\tools\\ilc" @"obj\Release\net8.0\win-x64\native\HelloWorld.ilc.rsp"" exited with code -532462
766. [C:\DotNetTest\HelloWorld\HelloWorld.csproj]

Regression?

Yes

Known Workarounds

Don't install to a path with a space - which means you can't use the default installer on Windows.

Configuration

No response

Other information

No response

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jonsequitur]

Here's an example showing how to restore the old response file parsing behavior, so that each line is treated as a single token and doesn't need to be wrapped in quotes if it contains spaces:

        [Fact]
        public void Legacy_line_delimited_response_file_parsing_can_be_enabled_using_token_replacer()
        {
            var option = new Option<int>("--opt");
            var argument = new Argument<string>();

            var responseFile = CreateResponseFile(" --opt ", "123", "one two three");

            var parser = new CommandLineBuilder(new RootCommand { option, argument })

// Add this line to replace the new response file behavior:
                         .UseTokenReplacer(TryReadResponseFile)
// --------------------------------------------------------                         

                         .Build();

            var result = parser.Parse($"@{responseFile}");

            result.GetValueForArgument(argument)
                  .Should()
                  .Be("one two three");
        }

        private bool TryReadResponseFile(
            string filePath,
            out IReadOnlyList<string> newTokens,
            out string error)
        {
            try
            {
                newTokens = ExpandResponseFile(filePath).ToArray();
                error = null;
                return true;
            }
            catch (FileNotFoundException)
            {
                error = $"Response file not found: '{filePath}'";
            }
            catch (IOException e)
            {
                error = $"Error reading response file '{filePath}': {e}";
            }

            newTokens = null;
            return false;

            static IEnumerable<string> ExpandResponseFile(string filePath)
            {
                var lines = File.ReadAllLines(filePath);

                for (var i = 0; i < lines.Length; i++)
                {
                    var line = lines[i];

#if RescursiveResponseFileSupportIsNeeded
                    foreach (var p in SplitLine(line))
                    {
                        if (GetReplaceableTokenValue(p) is { } path)
                        {
                            foreach (var q in ExpandResponseFile(path))
                            {
                                yield return q;
                            }
                        }
                        else
                        {
                            yield return p;
                        }
                    }
#else // otherwise
                    foreach (var p in SplitLine(line))
                    {
                        yield return p;
                    }
#endif
                }
            }

            static string GetReplaceableTokenValue(string arg) =>
                arg.Length > 1 && arg[0] == '@'
                    ? arg.Substring(1)
                    : null;

            static IEnumerable<string> SplitLine(string line)
            {
                var arg = line.Trim();

                if (arg.Length == 0 || arg[0] == '#')
                {
                    yield break;
                }

                yield return arg;
            }
        }
    }

[eerhardt]

@am11 @MichalStrehovsky @jkotas - can we use the above to fix the issue? Instead of reverting the System.CommandLine change completely?