Yarn writes registry.yarnpkg.com to lockfile when private scoped packages is from registry.npmjs.org

[stieg]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
I have something like the following in my ~/.npmrc file:

@mycompany:registry=https://registry.npmjs.org/
//registry.npmjs.org/:_authToken=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

This is necessary so I can login to my private repo and get the packages I need to build locally.

When I add a new package like @mycompany/foo I would expect an entry in the yarn.lock file to look something like this:

"@mycompany/foo@^1.1.0":
  version "1.1.0"
  resolved "https://registry.npmjs.org/@mycompany/foo/-/foo-1.1.0.tgz#103034ba565c288eb781b1fd1065c9f05dfb90f9"

Yeah... not so much. Instead it looks like this:

"@mycompany/foo@^1.1.0":
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/@mycompany/foo/-/foo-1.1.0.tgz#103034ba565c288eb781b1fd1065c9f05dfb90f9"

Uncool yarn. This causes my CI to break because it will try and dig up the packages at registry.yarnpkg.com. Since this is a private registry the login will fail (since the registry is hosted on registry.npmjs.org) and thus it will cause other systems to fail to install packages.

Here is how to reproduce:

1. Setup a private repo on npm with some namespace
2. Set your .npmrc file to something like the one I provided above (except you replace the scope mycompany with your own scope, and set a valid auth token of course), and run yarn add <your scoped package>. You can do this by doing npm login --registry=https://registry.npmjs.org --scope=@<your scope name>. I would use yarn login but that doesn't work :(
3. Add a package from your scope using yarn add
4. Push a trivial update to that package in your scope.
5. Tell yarn to upgrade the package.

You will see that the URL gets changed in the lock file.

Please mention your node.js, yarn and operating system version.
node: 8.2.1
yarn: 0.28.4
OS: Fedora 26

[stieg]

Bump Anyone else seeing this behavior? No reply from anyone so far :(

[Haroenv]

Should be fixable by implementing #3330

[stieg]

@Haroenv Yep... should be fixed by #3330 but that is going to break backwards compatibility and based on discussion in that thread that won't be resolved for some time (need RFC, general agreement, roll out of a major release).

A simpler fix here AFAICT would involve disabling yarns predisposition to replace registry.npmjs.org with registry.yarnpkg.com. I'm not sure why Yarn is so very keen on doing this as it seems to have created plenty of issues (I should cite them here... but I don't want to spend the time to dig them up so will not do so but can if needed). By removing that silly behavior this problem here goes away.

[karlringrose]

+1 I am finding this issue as well. have you gotten any work around to effectively work for you @stieg ?

[stieg]

Honestly we have moved back to npm for now since it works. My work around was a script that would modify the lock file with the correct path for those packages.

On Sun, Sep 10, 2017, 18:16 Karl Ringrose ***@***.***> wrote: I am finding this issue as well. have you gotten any work around to effectively work for you @stieg <https://github.com/stieg> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4157 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIozzeWZwvPAnbbh5r5GRugYWc6voD3ks5shLQJgaJpZM4O1Ps2> .

-- - Andrew S.

[karlringrose]

I was about to do this, though my work around now is to just delete the yarn.lock file, and have hard versions set within pkg for build. We will most likely migrate to npm@5 if this issue is not resolved.

[vinicius73]

This bug has been causing a lot of inconvenience for a long time.
Unfortunately we will have to go back to NPM 😢

[BYK]

I'm sorry that you are having issues with this. I'm trying to see exactly what is wrong since Yarn's registry is just a CDN that needs to bypass everything so it should work with scoped packages properly. In fact, I can locally use scoped packages such as @yarnpkg/lockfile with my npm credentials.

Can anyone provide me some step-by-step instructions to replicate the issue locally and then we can start working on a solution? This may simply be a bug when you say your default registry should be NPM and Yarn incorrectly replaces it with its default registry.

[stieg]

@BYK I would start by re-reading the first comment on this thread to get an understanding of the behavioural problem. The maybe not so clear subtext is that since Yarn is re-writing the lock file paths to registry.yarnpkg.com instead of registry.npmjs.org, yarn is trying to fetch packages from a private repository without any authentication (or the authentication is not going through the CDN; which ever one it is). This causes the failure described in step 1 (or at least I believe it to be the case). I believe the root of the problem is that Yarn has decided they need to automatically rewrite the registry.npmjs.org to registry.yarnpkg.com. This is no problem with public packages that require no auth but is a major one with those that do require auth. I hope that makes things a bit clearer.

[BYK]

@stieg - I've read the initial comment and have understood the issue. I'm just trying to understand how it happens since it doesn't happen to scoped packages for instance which require authentication.

I'm also investigating why the rewriting happens and if it is something that should happen in the presence of an explicit registry override.

That's why I was curious about new and precise step-by-step instructions. Also we've made a bunch of improvements about detecting when to send aithentication headers and when not to on the latest master. Would you be willing to test one of the latest nightlies and see if they already solve the issue (not the rewrite but the actual, fetching the package issue). Nightlies can be found here: https://yarnpkg.com/en/docs/nightly

[stieg]

I see @BYK; you need a way to reproduce this (as you mentioned in your first comment). When I first encountered this issue I was running yarn v0.28.4 (as noted in the first comment). It seems that I can only hit this issue with that version of yarn and lower. On the current 1.0.1 release things are broken in a different way for me (described below). Alas here are the requested steps to reproduce:

With yarn v0.28.4

1. Setup a private repo on npmjs.org as some scope. I am going to use the scope "foo" for this example.
2. Upload some bunk package. I am going to call this package "bar". When uploaded I refer to it as @foo/bar
3. Create a new project locally. I'm going to call this "baz".
4. Login to npmjs using npm login --scope=@foo command. I do this because yarn login does not work for private packages.
5. Add package bar as a dependency of baz. So yarn add @foo/bar. This should add the dependency to the yarn.lock file.
6. Blow away your node_modules directory. So rm -rf node_modules.
7. Nuke the yarn cache. yarn cache clean.
8. Try running yarn install. You will see that it fails with a message like this:

$ yarn install
yarn install v0.28.4
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@foo/bar/-/bar-0.0.1.tgz: Request failed \"404 Not Found\"".

That is the failure.
9. Run the following command to fix the lockfile path:

sed -ie 's,registry.yarnpkg.com/@foo,registry.npmjs.org/@foo,' yarn.lock

10. Run yarn install again. Observe that it works this time.

With yarn v1.0.1
Using the same steps from above I get stuck at step 5 with the following message:

$ yarn add @foo/bar
yarn add v1.0.1
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@foo/bar/-/bar-0.0.1.tgz: Request failed \"404 Not Found\"".

I can work around the problem by doing the following:

yarn config set registry https://registry.npmjs.org

This will cause the package to install successfully. But then I will get stuck with a diffent problem when I hit step 10 (yarn doesn't seem to be authenticating using the data from .npmrc). So I'm dead in the water :(. This behavior is the same with both 1.0.1 and the latest nightly.

Perhaps I am just doing something so very wrong that yarn is confused and thus not working? Tips appreciated :).