Attachments connecting to toil/cwl_runner. Add util to compose post. Test cleanup.

[DailyDreaming]

• Added an option to use workflow_attachments from the commandline with the client.
• cwltool and toil now both run the subprocess command in the tempdir created for uploaded files, the same as arvados.
• Updated the readme.
• Removed auto-attempting to upload all files contained in the directory from wherever you happen to run the command.
• Removed a return if not CWL in the client.
• Also adds a util function to compose the post request body from the workflow filepath, json filepath, and any attachments.
• Cleaned up the tests a bit.
• Added a wdl test to toil (edit: this is only passing locally for me, and no idea why it isn't passing on travis; added a manual flag to run this until I can figure out what travis is doing).
• Updated the toil version.
• Changed a couple of leftover references of workflow_id in the client to run_id.
• Added the rest of the swagger utility call functions to wes_client.util based on: http://ga4gh.github.io/workflow-execution-service-schemas/ . These all work, and I've mainly added them as functions for James Eddy's orchestrator to be able to import and use, though I wouldn't mind replacing the ones in the client (which aren't working for me) with the ones in wes_client.util. Though I'm not sure if the current ones work for Arvados or not.

[tetron]

I don't understand the motivation for making symlinks. The runner invocation should specify an empty outdir. Instead of making symlinks, the command line should reference the tmpdir by full path.

[DailyDreaming]

I think I misunderstood the intention of outdir since it is the cwd for the cwl_runner backend. I see arvados is using the tempdir as the cwd. It makes more sense now. I'm only testing on the cwltool and toil end, and the attachments weren't being read.

[DailyDreaming]

@tetron I've removed the symlinking. I do os.link() the cwl/wdl and json files in both the tmpdir and the workflow dir. I think it might be a good idea to keep them stashed after the tmpdir is potentially deleted (though cwltool and toil don't delete tmpdir yet).

This should allow the cwltool and toil backends to use any number of attachments. There is still a buffering issue where the size of files sent is limited by the sending computer's RAM.

Also got rid of the portion where it tried to upload all files from whichever cwd it was run from and made a user option to specify attachments instead.

[tetron]

• Suggest that --attachments takes a glob pattern, also support directories (so that attaching a directory means attaching all the files it contains)

• I'm fine with getting rid of Bravado on the client, it doesn't add a huge amount of value

• I'm going to test this a little bit but for the most part it looks good

[DailyDreaming]

@tetron Thank you! Really appreciate the review. I'll implement support for glob patterns and replace the bravado client with the direct calls (and some unittests for them).

One side note is that Toil status has some edge cases. So we're working on a better implementation on the toil-side that should store status in the jobstore when it changes so that the jobstore can be queried directly. Once that's done, I'll likely make some modifications to that function.

[tetron]

If you can fix the failing Travis test, I'm fine with merging this branch and continuing work in another pull request. I'd like to make another release (release early, release often).

[DailyDreaming]

@tetron Sounds good to me. I'll take a look at the failing test.

[DailyDreaming]

@tetron So... not only do all 7 tests pass on my computer locally, but making an attempt to debug on my personal fork also has them all passing on a different travis: DailyDreaming#1

I think this might be something odd with this travis's server? Or dockstore was down? I'll push another change to this branch and see if it doesn't pass this time.

[DailyDreaming]

@tetron I used a magic docstring that has fixed everything.

[tetron]

Awesome thanks!