How should Proc::Async.new, run and shell call cmd.exe?

[patrickbkr]

On Windows it is not possible to call cmd.exe with complex arguments. Calling .bat files is affected as well. Rakudo bug r#2005 also reports this.

Short problem description

On Windows the API to start programs (CreateProcess) does not take an array, but a single string. There is a convention nearly all programs adhere to of how to turn an array of arguments into a single string (a well defined quoting). This convention is what rakudo currently implements (via libuv, which does the actual quoting).
Not all programs adhere to this convention. Most prominently cmd.exe and as a result .bat scripts.
When trying to call a .bat file rakudo applies the usual quoting to the commandline arguments and as a result makes it impossible to pass some arguments.

Perl 6 functionality affected: Proc::Async, shell and run.

How could the API look that in addition to the current behaviour also allows to call cmd.exe and any other program not adhering to the commandline quoting convention?

[patrickbkr]

How command line processing on Windows works

Do read this awesome writeup of how commandline processing works on Windows.

Also don't miss this bugreport that already has a proposal of how to handle this in Perl 6.

Windows does not have the concept of an ARGV array. At the API level there is only one single string that is passed. There is a convention (CommandLineToArgv() implements that) of how a string array is converted to a single string and back. With rare exceptions all programs adhere to this convention. The most prominent exception is cmd.exe. cmd.exe just directly processes its argument string as if you'd have pasted it into a cmd.exe window. When calling a .bat file, Windows implicitly wraps the call in cmd.exe /C <your.bat stuff>. (Not entirely sure what exact command Windows actually puts around the command, it's difficult to find out.) Thus calling any .bat file is subject to cmd.exe processing. One usually doesn't want that.

When calling an .exe file through cmd.exe, then cmd.exe will process the argument string first and the output is afterwards dequoted using CommandLineToArgv(). When calling a .bat file with cmd.exe (irrespective of implicitly or explicitly), then CommandLineToArgv() will not be called. The arguments will arrive in the .bat file as cmd.exe processed them.

The CommandLineToArgv() quoting (CLTA)

If the argument contains \n\t\v" it needs quotation as follows. Put it in "" and put a \ before every " the argument contains. If a contained " is preceded by one or more \ , duplicate all the \ . Also duplicate all \ at the end of the argument. One must not duplicate any other \ . All the so quoted and unquoted arguments are joined with a single space in between.

The cmd quoting

• Either: Prefix all cmd.exe metacharacters (those are ( ) % ! ^ " < > & |) with the escape character ^. It's not possible to escape spaces using ^.
• Or: Wrap arguments in " and prefix all contained " with ^

[patrickbkr]

Five possible usage scenarios

In decreasing order of commonness.

1. I want to run a exe file
   • Then I don't use cmd.exe
   • I want CLTA quoting
2. I want to run a bat file
   • I don't want any cmd.exe side effects
   • I thus need cmd quoting, but no CLTA quoting
3. I want to do shell stuff (or use one of the other programs that have different argument parsing, e.g. nmake)
   • Irrespective whether I want to start a .exe or .bat or something else altogether: I want the side effects cmd.exe has
   • I want no quoting at all. Otherwise I'd circumvent the functionality of cmd.exe
4. I want to do something bare metal. I know what i do:
   • I want no quoting at all
5. I want to run a exe file via cmd.exe, but don't want any cmd.exe side effects
   • I'd want CLTA quoting first, then cmd quoting on the result
   • That's what the Microsoft blog recommends.
   • Why would one want to do such a thing and not just call the exe file directly?

Number 4. and 5. won't happen in any normal usage scenarios.

What others do

libuv quotes arguments compatible to CommandLineToArgv(). When passing the UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS argument this quoting is disabled and the arguments are just joined with a space.

node exposes a shell flag which wraps the call in cmd.exe /d /s /c "<command>" by hand and enables UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS.

javas ProcessBuilder has a very sophisticated and complex processing of arguments. e.g. it detects whether the user called a .bat file or .cmd file and acts on that.

[patrickbkr]

Initial proposal

A good API should:

• Work in the common case (CLTA quoting) without the user having to know about the ugly details.
• Give enough control to the user to call any program the way he wants.
• Make sure the changes don't make the API more complex for the use cases not affected by this.

Proc::Async idea 1

Add a new enum WindowsProcQuoting and add an additional argument to Proc::Async to select the quoting to use.

enum WindowsProcQuoting < normal, cmd, off >;
multi method new(:$path, :@args,  :$w, :$enc, :$translate-nl, WindowsProcQuoting :$win-quote = normal --> Proc::Async:D)

Proc::Async idea 2

Add a new flag to the Proc::Async constructor to turn quoting off. Add quoting subs.

multi method new(:$path, :@args,  :$w, :$enc, :$translate-nl, :$win-quote-args = True --> Proc::Async:D)
sub win-quote-normal(@args --> List)
sub win-quote-cmd(@args --> List)

run idea

run should be changed analogous to Proc::Async.

shell idea

shell should never quote its arguments.

[patrickbkr]

ping @jnthn, @ugexe
Can I encourage you to give your opinion?
I'm willing to work on this, but I'd like to have consensus first on how the API should look first.

[ugexe]

I like the java way of doing it (+ a flag to disable it if they don't do that) because it doesn't require the programmer to have as much of an understanding of e.g. Windows to write platform-independent code. Not thrilled about the idea of speccing some potentially complex and heuristic-based logic though.

The :windows-specific-flag alternative is acceptable, but I do feel like exposing the platform-specificness is a bit weird.

[raiph]

(Edited to simplify.)

This comment is my 2c on the API / paintwork, either reflecting my confusion or, if I'm not confused, reflecting my concerns and attempts to reduce confusion:

enum arg-processing < default CommandLineArgvW cmd-exe WINDOWS_VERBATIM >;
:arg-processing = ...

I named the overall enum "arg-processing" rather than something Windows specific to establish an overall context, so someone encountering it gets that that's the overall context. And then:

• default (could be auto) instead of normal more strongly suggests Rakudo has a default enum if you don't specify a choice.

• CommandLineArgvW is named so that the first page listed when googling for it says "Parses a Unicode command line string and returns an array of pointers to the command line arguments, along with a count of such arguments, in a way that is similar to the standard C run-time argv and argc values." I consider it useful that a google will immediately produce a result that is both exactly the right direction and both obviously and authoritatively exactly the right direction.

• WINDOWS_VERBATIM is so that a google for "WINDOWS_VERBATIM" provides good results.

Putting these together, our doc might say something like:

On unix default does what the standard C run-time does. On Windows it does the nearest Windows equivalent ("CommandLineArgvW") most of the time, cmd-exe when that makes sense, and WINDOWS_VERBATIM (which corresponds to libuv's option) if you say so.

[patrickbkr]

@raiph

enum arg-processing < default CommandLineArgvW cmd-exe WINDOWS_VERBATIM >; :arg-processing = ...
As I understand it you either get the default, and live with it, or use one of three Windows specific options. Is that right?
If I understand correctly the above is your proposed alternative for the following I proposed:
enum WindowsProcQuoting < normal, cmd, off >;

Here is how I intended the respective parameters to influence processing:

• normal: Does CommandLineArgvW quoting.
• cmd: Escapes the string so cmd.exe doesn't transform it.
• off: Concatenates the arguments with a single space, nothing else. (The flag to trigger this mode of quoting in libuv is called UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS).

Now on to your proposal.
I'm unsure what default should do on Windows. Multiple strings need to be concatenated in some way. There is no "we just don't do anything to the parameters" option.
If I understand correctly the other three options you list are different names for the options I listed:

• CommandLineArgvW corresponds to normal
• cmd-exe corresponds to cmd
• WINDOWS_VERBATIM corresponds to off
  I do like the idea of picking names with a high "googleability". The names you chose are definitely better than the ones I chose in that respect.
  CommandLineArgvW does not apply on other platforms though (in contrast to normal). We might be able to solve this by simply providing different methods on different OSes. Only the one on Windows would accept the enum parameter.

[patrickbkr]

@ugexe I fully agree with all of your points. Sadly I haven't been able to come up with a solution that fits all bills. I'm open for other suggestions!

[raiph]

Some links I found during my research of this issue that might be useful.

A key comment in a libuv issue: libuv/libuv#2627 (comment)

Two SOs that may only be useful because they're so monumental and scary that it emphasizes the need for a "Give enough control to the user to call any program the way he wants." escape hatch: https://stackoverflow.com/questions/6828751/batch-character-escaping and https://stackoverflow.com/questions/4094699/how-does-the-windows-command-interpreter-cmd-exe-parse-scripts

A link to fellow PL folk (Julia) in the process of figuring out related issues: JuliaLang/julia#34111

[raiph]

@patrickbkr You have it about right for my paintjob. To sort the one point of confusion, my intent was that default and CommandLineArgvW are aliases and are just a rename of your normal option. Now I'm thinking it might make sense to rename normal to portable or unix-conventions or just unix.

Is your cmd option in essence rakudo-escapes-and-or-quotes-user's-command-line-so-that-when-cmd-exe-processes-it-it-turns-the-command-line-back-into-what-it-was-before-rakudo-escaped-and-or-quoted-it?

[patrickbkr]

Is your cmd option in essence rakudo-escapes-and-or-quotes-user's-command-line-so-that-when-cmd-exe-processes-it-it-turns-the-command-line-back-into-what-it-was-before-rakudo-escaped-and-or-quoted-it?

Exactly.