[RRFC] New subcommand “npm clone”

[cactysman]

Motivation ("The Why")

Most publicly available NPM packages are usually also open source and provide a link to their code repository through the repository field in their package.json file.

However, starting to work on the code for a package is not exactly a quick thing to do . You first have to find it (it doesn't necessarily have to live on GitHub) and then manually install the dependencies before you may begin.

I came up with a much simpler solution: npm clone

How

Cloning from a source

npm clone <packageName>

would go look into the repository configuration of packageName on the NPM registry (default, can be overridden with --registry as usual) and clones it from there with whatever program can be found that might make sense (git, svn, hg?).

It would look at the repository.type set in the package.json.

• "type": "git" -> use git
• "type": "svn" -> use something like TortoiseSVN.

npm clone <URL>

would clone from the given URL and, if it's an NPM package (package.json or package-lock.json was found), then do the same as above.

After cloning

Unless an option like --no-install is given, npm clone could then install the dependencies for that package or run a script!

Options

Option	Parameters	Default	Meaning
-d / --dest	<path>	The current directory	Where to clone the code to
-w / --work-dir	<path>	The root directory of the repository	Where to run the npm commands (such as npm install) inside the cloned worktree
-s / --script	<script>		A script to run after cloning
-n / --no-install			No automatic install will be run

Note: These options names are mere suggestions. If you can come up with something that makes more sense, please let me know.

Example

Command	Result
npm clone express -s test	Clones the source of the express package and runs tests.
npm clone typescript -n	Clones TypeScript from GitHub and then does nothing.
npm clone https://github.com/badges/shields -w gh-badges	Clones the shields repository from GitHub and installs the dependencies of the gh-badges package in the gh-badges directory.

Current Behaviour

You have to do these things:

1. Find out where the code lives.
2. Clone it from there.
3. Install the dependencies.

Desired Behaviour

Run npm clone <package> with whatever options suit your case and you're good to go.

References

• Original post featuring this idea in the archived NPM forums.

[isaacs]

I like this, and it would let me do away with my npmclone bash shortcut ;)

The config options suggested are a bit challenging, since they all have a current meaning. Maybe we just follow git's pattern here and specify the dir name as a second positional arg?

So instead of:

npm clone express -w express-checkout

You'd do:

npm clone express express-checkout

Then make the following changes:

• -d --dest would always be the local project prefix (ie, either the nearest parent dir with package.json or node_modules, or an explicit --prefix cli option, or the cwd.)
• -s --script --> -c --command
• not sure what to do with the -n --no-install option? It's a bit confusing to add a --install (which --no-install would imply). I agree that it's useful to install the package deps, especially if you do something like -c 'npm test', but I'm not sure the best way to spell the option that suppresses that behavior.

[cactysman]

The config options suggested are a bit challenging, since they all have a current meaning. Maybe we just follow git's pattern here and specify the dir name as a second positional arg?

So instead of:

npm clone express -w express-checkout

You'd do:

npm clone express express-checkout

Oh I 100% agree to this, sticking to more conventional patterns sounds good.

---

-s --script --> -c --command

Maybe I made this a bit confusing.
With script I actually meant an NPM script (like npm run foo),

Being able to specify a whole command here sounds better though, especially if something is not part of an NPM script.

I guess that command should run in the target directory then for it to make sense? Otherwise you could just chain commands like npm clone foo && ls.

---

not sure what to do with the -n --no-install option? It's a bit confusing to add a --install (which --no-install would imply). I agree that it's useful to install the package deps, especially if you do something like -c 'npm test', but I'm not sure the best way to spell the option that suppresses that behavior.

I thought of that to be a switch to really only clone and only using npm clone to resolve the repository URL and do the cloning, whereas omitting --no-install would actually also prepare the package by installing the dependencies.

[ruyadorno]

@metaa it's in the agenda for the OpenRFC call today, starting in 15min 😊 in case you're able to attend (or follow youtube stream), here are the links: #212

[cactysman]

Maybe not automatically installing dependencies and instead requiring a -i or --install to do that would make more sense, so I'd get a command like npm clone -i express.

Thoughts?

Also please let me know if I should update the original post or post an updated version later on once some stuff has been decided on.

[cactysman]

So in the #214 meeting I noticed that it doesn't seem to be clear what the original idea was 😦
Someone brought up a possibly similar working unpack subcommand that would basically download a package from the registry and unpack it, but that's not what this is supposed to be. Also someone chimed in and stated that that would probably not help much in most cases, as the content of the package usually is not what made the published content (makefiles and similar stuff).

So what is it?

It's more like what @isaacs called it.

A more comfortable way around

git clone $(npm show . repository.url)

but not only for git.

npm clone is meant to be what its name says; it clones the repository linked in package.json of the uploaded package. The intention is to primarily make contributing to packages more straightforward and faster.

npm clone would delegate the cloning of the repository to the user's installed client and run the correct shell command.

Thoughts:
Should the client be discovered automatically? (Well known paths or something)
The user could set a path to a preferred application via npm config.

If there is no suitable client installed, it could suggest some:

You don't have software installed to handle repositories of type git.
Check out some of these:

Let's look at examples from the link above and turn that into use-cases.

Examples

"type": "git" (or service aliases)

"repository": {
  "type" : "git",
  "url" : "https://github.com/npm/cli.git"
}
"repository": "npm/npm"
"repository": "github:user/repo"
"repository": "gist:11081aaa281"
"repository": "bitbucket:user/repo"
"repository": "gitlab:user/repo"

This package references a git repository.

All of the currently available aliases for popular git services host git repositories too. Maybe if more were to be added in the future, they should have some type metadata internally to denote what kind of repository that is.

npm clone npm would, without any options, run something like this:

git clone https://github.com/npm/cli.git

Based on the options passed (@isaacs brought up maybe being able to clone a specific version?), the command might look different of course.

"type": "svn"

"repository": {
  "type" : "svn",
  "url" : "https://v8.googlecode.com/svn/trunk/"
}

This package references a Subversion repository.

Just like above, in the git section, npm clone would build an SVN command (rely on community packages?) to clone the repository, that would either use TortoiseSVN, Mercurial or whatever else there is out there that handles SVN (I got no clue).

"type": "something else"

Yeah, uh, maybe here npm clone could try to be modular. What if it looked for globally installed modules following a name pattern like npm-clone-whatever or (in an interactive shell) showed quick search results from the registry for whatever might do the job?

Maybe, but that might go to far, there could also be a pool of approved "official" modules or it just filters by keywords or something.

What the user experience could look like

npm clone foobar
# as long as I got whatever client suitable for handling the repository link installed, this works as intended
cd foobar

That's all it takes to start contributing to that repository.

(Of course, stuff like making a fork of that repository or whatever will still have to be done manually, but that would have to be done anyways unless this goes even further and supports the GitHub CLI and related stuff or other NPM packages that do that)

---

I hope this made the idea behind it clearer 😀

I'll gladly await feedback.

[ruyadorno]

So in the #214 meeting I noticed that it doesn't seem to be clear what the original idea was 😦
Someone brought up a possibly similar working unpack subcommand that would basically download a package from the registry and unpack it, but that's not what this is supposed to be.

sorry @metaa that someone was me 😅 don't worry, I was fully aware that unpack had nothing to do with the idea here, I was just trying to extrapolate a bit more and hear diff thoughts on it - that said, I believe that in the actual meeting (maybe the notes are confusing) it was quite clear what is the npm clone idea and the team really like it and seems to be a reasonable feature to maintain in the cli.

What we then mentioned as next steps are taking all these ideas (this last comment is also great!) and open a formal RFC Pull Request so that we can better follow up with the discussion and gather more feedback from the community.

[isaacs]

Hm, supporting not-git would be harder, and may not be worth the effort, as >99% of the projects on npm use git anyway. For a first pass, we could just have npm clone fail if we see a repository.type set to something other than 'git', and then add support for other repo types in the future. We already have utils and facilities for doing git checkouts programmatically for installing from git, so I don't think it's unreasonable to keep making it more of a first-class citizen than other VCSes.

The work to be done at this point is to specify in detail exactly what is cloned in each combination of cases:

Assuming npm clone <specifier> [<folder>] where <folder> defaults to the name on the specifier.

• command npm clone foobar vs npm clone foobar@1.2.3 vs npm clone foobar@1.x - should we look for a version tag if the user specifies a version or range?
• what happens if you do npm clone https://x.com/some-tarball.tgz? Should we unpack it and look for a name and repository? (Not hard to do this, we do the same for a bunch of use cases already.)
• any way to specify a branch or tag name other than a semver version?
• cloning over SSH vs HTTPS? Should we use the canonical SSH endpoint for known hosts, or HTTPS, or go based on what the repository field says?
• how should repository.directory be handled? do we check out just that folder, or the whole repo, and let the user figure it out? (not super hard to do either, and some UX trade-offs to be analyzed there.)

So, yes, it's time to write a complete RFC for this :)

[ljharb]

fwiw, if you ever use https for me, it will prompt me for a username and password, whereas ssh will use my key, so i'd prefer a way to force it to always use ssh.

[cactysman]

@ljharb

fwiw, if you ever use https for me, it will prompt me for a username and password, whereas ssh will use my key, so i'd prefer a way to force it to always use ssh.

I could think of something like

npm config set clone-force-protocol ssh