yarn no longer saves git/github packages in offline mirror

[agibralter]

Do you want to request a feature or report a bug?

bug

What is the current behavior?

Given our ./package.json like this:

{
  "dependencies": {
    "@kadira/react-storybook-addon-info": "^3.3.0",
    "@kadira/storybook-ui": "^3.11.0",
    "internal-pkg": "git+ssh://git@github.com/me/internal-pkg.git#abcdef0",
    ...
  }
}

And ./.yarnrc:

yarn-offline-mirror "./vendor/npm_cache"
yarn-offline-mirror-pruning true

Running yarn no longer seems to save the dependencies from github/git in our npm_cache.

If the current behavior is a bug, please provide the steps to reproduce.

What is the expected behavior?

Running ls on the ./vendor/npm_cache should include non-npm-hosted packages.

Please mention your node.js, yarn and operating system version.

macOS 10.12.4:

$ which node
/Users/me/.nvm/versions/node/v6.9.4/bin/node

$ node -v
v6.9.4

$ which yarn
/usr/local/bin/yarn

$ yarn --version
0.23.2

(Node is installed via nvm and yarn via brew)

[tribou]

I'm getting the same issue when yarn tries to resolve console-polyfill inside the rollbar-browser package:

yarn install v0.23.2
warning You don't appear to have an internet connection. Try the --offline flag to use the cache for registry queries.
[1/4] 🔍  Resolving packages...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
warning There appears to be trouble with your network connection. Retrying...
error An unexpected error occurred: "https://github.com/rollbar/console-polyfill.git: getaddrinfo ENOTFOUND github.com github.com:443".
info If you think this is a bug, please open a bug report with the information provided in "/Users/tribou/dev/react-template/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

> cat yarn.lock | grep -A 3 console-polyfill
console-polyfill@rollbar/console-polyfill#eab6ff9d2b7597fc2f259baa18100556bdb94dbe:
  version "0.2.3"
  resolved "https://codeload.github.com/rollbar/console-polyfill/tar.gz/eab6ff9d2b7597fc2f259baa18100556bdb94dbe"

    console-polyfill rollbar/console-polyfill#eab6ff9d2b7597fc2f259baa18100556bdb94dbe
    error-stack-parser "1.3.3"
    extend "3.0.0"

yarn why v0.23.2
[1/4] 🤔  Why do we have the module "console-polyfill"...?
[2/4] 🚚  Initialising dependency graph...
[3/4] 🔍  Finding dependency...
[4/4] 🚡  Calculating file sizes...
info This module exists because "rollbar-browser" depends on it.
info Disk size without dependencies: "24kB"
info Disk size with unique dependencies: "24kB"
info Disk size with transitive dependencies: "24kB"
info Amount of shared dependencies: 0
✨  Done in 2.79s.

From my current node_modules/rollbar-browser/package.json:

"dependencies": {
    "console-polyfill": "rollbar/console-polyfill#eab6ff9d2b7597fc2f259baa18100556bdb94dbe",

• OS X El Capitan
  10.11.6 (15G1421)
• node v6.10.1
• yarn v0.23.2

[agibralter]

Note I get the correct behavior when using: npm install -g yarn@v0.21.3 + yarn.

Also the entires in yarn.lock are not correctly displaying resolved:

v0.23.2: resolved "https://registry.yarnpkg.com/mime/-/mime-1.3.4.tgz#115f9e3b6b3daf2959983cb38f149a2d40eb5d53"

vs

v0.21.3: resolved mime-1.3.4.tgz#115f9e3b6b3daf2959983cb38f149a2d40eb5d53

[arcanis]

Is it still an issue, @agibralter ? The resolved field has changed starting from 0.23, so it's completely expected that you now have full urls even when using an offline mirror.

[tribou]

@arcanis please see my comment above as I was using yarn v0.23 which couldn't get console-polyfill offline.

[agibralter]

@arcanis I just tested with 0.23.4. I have a package in package.json as I described above ("internal-pkg": "git+ssh://git@github.com/me/internal-pkg.git#abcdef0"). When I start fresh (e.g. rm -rf node_modules; rm -rf vendor/npm_cache; rm -rf yarn.lock) and run yarn, my internal-pkg does not get added to vendor/npm_cache.

[agibralter]

$ rm -rf node_modules/

$ yarn cache clean
yarn cache v0.23.4
success Cleared cache.
✨  Done in 6.52s.

$ yarn --offline
yarn install v0.23.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error Can't make a request in offline mode
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

I don't see how anyone has offline mirror working in 0.23.x now... Can someone please confirm that offline mirror works at all anymore?

[dizlexik]

I've been keeping my eye on this issue hoping for it to get resolved.

I've been using v0.21.3 for a while now because offline mirror functionality seems to be completely broken in newer releases, and I was running into the same issues as you, @agibralter.

Among other reasons, I too rely on offline mirror to cache copies of my internal git dependencies so I don't have to massively overcomplicate my CI process. I'd really love to see this fixed sooner than later so I'm not stuck on this old version of yarn.

[agibralter]

For what it's worth, I've had some success with 0.22.0. It seems to have slightly inconsistent behavior about when it outputs a URL to yarn.lock vs. when it outputs a filename...

[arcanis]

The offline mirror works fine, at least for regular packages. I'm not sure we have a unit test for git repositories, which might explain why something could have broke here :/ Definitely something we should add (PR accepted, if you're faster than us :).

I've tried to reproduce your issues, and:

• I haven't been able to bring Yarn to not add the tarball to the offline mirror, even when it's from a git repository (I've tried with arcanis/private-pkg). If you still have this particular issue, I would really appreciate a repro script that I can run.

• I, however, have been experiencing a problem with the offline mirror where the tarball hashes cannot match in offline mode, because we're trying to compare the tarball sha1 hash with the commit hash. That shouldn't happen, I'll look into fixing that.

I'm sorry for the problems you've experienced, we'll do our best to fix it in the next tagged release.