build tools: Yarn 2, yarn 3 and NPM 7+ for Monorepo Solution

[awentzel]

Perform an investigation to verify if Yarn 2 and yarn 3 or NPM7, can meet our mono-repository needs as features could have changed since the last time we investigated and we should understand the implications of choosing one of the other and if we can achieve our usual development process without Lerna.

Yarn 2

Version 2 of the popular JavaScript package manager Yarn2 has been here for a year now. Yet try to find information on upgrading from 1.X or how it compares to current versions of npm (especially the 7.x dev release), and you’ll get plenty of well-written but outdated blog posts mainly bemoaning the new direction the Yarn team has taken. On its initial release, taking a wait-and-see approach with Yarn 2 may have been the correct call, but it’s time to reconsider that decision.

Here are three reasons you might have waited to make the switch — and why those reasons are out of date in 2021

1. Backward Compatibility with node_modules

• Yarn 2 uses a new Plug’n’Play (PnP) architecture that is a huge departure from how npm projects have always worked. PnP does away with the node_modules/ folder where packages get installed. Unfortunately, this is incompatible with certain JavaScript-based projects that rely on that directory to find packages, namely React Native, Flow, and VSCode Extension packages.

• However, Yarn 2 now offers an option that copies packages to the node_modules/ folder just like Yarn 1, providing backward compatibility for these projects. It literally requires adding a single line to your new .yarnrc.yml configuration file to turn on this feature.

2. Difficulty Migrating from Yarn 1.x

• The PnP incompatibilities that plagued the initial Yarn 2 release — and the seismic shift it created in getting rid of that hefty node_modules/ folder — also meant migration could be a big pain for pre-existing projects. But with node_modules/ support added back in, Yarn 2 now provides a clean migration path. There are a not-insignificant number of steps here, especially if you’re moving to PnP support, but the documentation is clear and easy to follow. It shouldn’t actually be that difficult to undertake.

3. But My Yarn 1.X Setup Isn’t Broken!

• As the Yarn 2 documentation mentions:
  Even if you don’t use Plug’n’Play nor plan to use it, your project will still benefit from more stable node_modules layouts, improved performances, improved user experience, active development, etc.
  Sooner or later, you still have to migrate.

• Yarn 2 migration and documentation: Migration | Yarn - Package Manager

Step by step migration to Yarn 2

Yarn 2 Documentation for Migration

1. Run npm install -g yarn to update the global yarn version to latest v1
2. Go into your project directory
3. Run yarn set version berry to enable v2 (cf Install for more details)
4. If you used .npmrc or .yarnrc, you'll need to turn them into the new format (see also 1, 2)
5. Add nodeLinker: node-modules in your .yarnrc.yml file
6. Commit the changes so far (yarn-X.Y.Z.js, .yarnrc.yml, ...)
7. Runyarn install to migrate the lockfile
8. Take a look at this article to see what should be gitignored
9. Commit everything remaining

Some optional features are available via external plugins:

1. Run yarn plugin import interactive-tools if you want upgrade-interactive

2. Run yarn plugin list to see what other official plugins exist and might be useful

3. Commit the yarn plugins

Yarn 3.0.0 Release

Breaking Changes

1. Node 10 isn't supported anymore.
2. Plugins can't access yup anymore (we migrated to Typanion as part of Clipanion v3).
3. To upgrade workspace-tools, remove it from your .yarnrc.yml, upgrade, then import it back.
4. The enableImmutableInstalls will now default to true on CI (we still recommend to explicitly use --immutable on the CLI).
5. You can re-allow mutations by adding YARN_ENABLE_IMMUTABLE_INSTALLS=false in your environment variables.
6. The initVersion and initLicense configuration options have been removed. initFields should be used instead.
   6.Yarn will now generate .pnp.cjs files (instead of .pnp.js) when using PnP, regardless of what the type field inside the manifest is set to.
   7.The virtual folder (used to disambiguate peer dependencies) got renamed from $$virtual into virtual.
7. The -a alias flag of yarn workspaces foreach got removed; use -A,--all instead, which is strictly the same.
8. The old PnPify SDK folder (.vscode/pnpify) won't be cleaned up anymore.
9. The --skip-builds flag from yarn install got renamed into --mode=skip-build.
10. The bstatePath configuration option has been removed. The build state (.yarn/build-state.yml) has been moved into the install state (.yarn/install-state.gz)
11. The cache files need to be regenerated. We had to change their timestamps in order to account for a flaw in the zip spec that was causing problems with some third-party tools.
12. @yarnpkg/pnpify has been refactored into 3 packages:
13. @yarnpkg/sdks now contains the Editor SDKs
14. @yarnpkg/pnpify now contains the PnPify CLI compatibility tool that creates in-memory node_modules
15. @yarnpkg/nm now contains the node_modules tree builder and hoister
16. @yarnpkg/plugin-node-modules has been renamed to @yarnpkg/plugin-nm
17. The --clipanion=definitions commands supported by our CLIs will now expose the definitions on the entry point (rather than on .command)

NPM 7

• With version 7 of npm they've reduced their dependencies by roughly 54%, while increasing the code test coverage by about 17%. It should also include a performance boost in multiple areas according to their own benchmarks.

• Npm 7 is now the latest version in the npm registry and therefore default. To install the new version of npm, you'll run the following in your command line interpreter of choice: npm install --global npm@latest

Couple new features from NPM 7

1. Version 2 of package-lock

• With the new package-lock.json file we'll unlock the ability to do deterministically reproducible builds. It should now include everything npm needs to install the packages needed. Before npm 7 yarn.lock was ignored by npm, but this is no longer the case. It can now use it to keep itself up to date with the package tree.

• The new lock file should be backward compatible with users of npm 6. Though, when you run npm install in a project with a version 1 lock file it will replace that file with the new structure. This can be avoided by running npm install --no-save when installing.

2. Workspaces

• This is one of the new features that I'm most excited about. It includes a set of features that will make the management of multiple packages a lot better. It lets you handle packages from a singular top-level root package. This has already been possible to do with for example yarn, Lerna, or Pnpm.

• In order to make npm aware that the current project is a workspace you have to add workspaces key in your package.json. This can be done by adding every single sub-folder or by using a glob, like in the example below.

• More on workspace: rfcs/0026-workspaces.md at latest ,

• more on workspace on npm(workspaces | npm Docs )

Automatically installing peer dependencies

• In versions before npm 7 developers had to install the peer dependencies. Now npm will use a new algorithm to ensure the dependencies are installed properly. If a peer dependency, that is not compatible with the specified one, is installed npm 7 will now block the installation.

Breaking changes

• Since the new version is considered a major version it'll come with a couple of breaking changes. Here are some:
  You can no longer use require() npm's internal modules. Npm now uses the package. exports field.
  The team has completely rewritten npx to internally use npm exec, the npx CLI will still be available. Some functionality changes are to expect. One is that you'll now be prompted if you try to run a module that is not installed yet.

• The changes mentioned above regarding peer dependencies might disturb some workflows.
  npm audit has a new output.. npm 6 showed all packages by default when running npm ls. With npm 7 it will only show the top-level packages. Run npm ls --all to mimic the behavior from npm 6.

Node Version

**Node Version 14 recommended**

There are many things to look forward to with this major release. Some of the new features include:

• The once experimental feature, Diagnostic Reporting, is now stable!
• V8 has been upgraded to V8 8.1
• Node Streams have been updated
• Removal of Experimental Modules Warning

Node version 16 current

• As is standard in Node.js releases, this version will not be promoted to long-term support (LTS) until October. We need the greater ecosystem to try it out and give the community feedback. This will allow us to address any issues in advance and make sure both the release, the ecosystem, and our customers are ready when it’s promoted.

Here are some of the new features and ongoing work in the 16 release, which include:

  1. Updated Platform support 

  2. V8 JavaScript Engine Version 9 

  3.  N-API Version 8 

  4. New Promises APIs 

  5. Async Local Storage APIs 

Summary

• Before npm7, it was not possible to handle a large monorepo project with npm alone. That why developers switched to yarn. With NPM7, this is no longer an issue, NPM can do exactly what yarn does. npm v7 arrives with a newer version for the package-lock.json format - allowing to reduce the need to read package.json files and to have enough information to reliably describes the full and precise package tree all by itself. More than that, the resulting package tree using the new lockfile is flattened, and this is crucial to boost the performance.
  Support. We have a partnership with JavaScript Cloud Advocates v-Team at Microsoft, who provide information on what happening on NPM 7, Node Js, and GitHub. If we start using npm 7, we will be getting support from them.
  As GitHub owns NPM 7, the most recent information is posted on GitHub.
  npm 7 is now generally available! | The GitHub Blog

• Yarn has been a leading Monorepo solution sometimes in combination with Lerna. The early release of Yarn 2 has some incompatibility with Certains JavaScript library like react. Sounds they have provided some fix for it. Here is the official link to Yarn: 2 - Installation | Yarn - Package Manager
  The community of Yarn has stated that Yarn 2 is a little slower than Yarn 1. This could be due to changes in Yarn infrastructure and added functionalities. The release of Yarn 3.0.0 has taken the performance into consideration and reduce configuration during the setup process or migration from the old version of Yarn.

• Node.js. We will use the recommended version 14. This is to prevent a breaking change in the customer application. If we use the latest version 16. Most customers may not be up to date with their node version.

Features	Yarn	npm
Installation process	To install Yarn, npm has to be installed Installation Docs	npm gets installed with the Node.JS solutions automatically Installation Docs
Lock file	In order to get consistent installs across machines, Yarn needs more information than the dependencies you configure in your package.json. Yarn needs to store exactly which versions of each dependency were installed. Yarn generates a yarn-lock file and it is auto-generated and should be handled entirely by Yarn. As you add/upgrade/remove dependencies with the Yarn CLI, it will automatically update your yarn.lock file. Do not edit this file directly as it is easy to break something.	npm generates a package-lock.json file that is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates. This file is intended to be committed into source repositories, and serves various purposes: Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies. Provide a facility for users to "time-travel" to previous states of node_modules without having to commit the directory itself. To facilitate greater visibility of tree changes through readable source control diffs. And optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
Output log	Output logs are clean, brief, and visually distinct	Output logs are voluminous
Global dependencies	Has a “why” functionality built-in	Does not have a “why” functionality built-in as alternative use npm list -g
License checker	Can restrict licenses of installed modules and means for supplying license information Running yarn licenses list command will list, in alphabetical order all of the packages that were installed by yarn or yarn install, and give you the license (and URL to the source code) associated with each package.	Running npm install license-checker, npx license-checker This will give you a printout of all the licensing details of packages used in your project. Other cool features of the project: Print a summary of licenses used by npx license-checker --summary. This can also be Include in your CI/CD pipeline by providing it with a whitelist or a blacklist of licenses
Fetching packages	Yarn stores dependencies locally and fetches them from the disk, instead of sending an HTTP request	npm fetches dependencies from the npm registry. however, you can always use npx to fetch dependencies locally if they exist else from the npm registry, which also reduces HTTP call.

[KingOfTac]

From my own grueling experience over the last three weeks, yarn 2 has some issues with running scripts across all packages in a monorepo, and lerna still seems better suited for running scripts. I am using yarn 2 workspaces with pnp and their no install methodology and it works great. It is just running scripts that has been my biggest issue. I'm sure it is something that can be worked out with the right package config, I just haven't found it yet.

[awentzel]

Thanks @KingOfTac for your insight.

[awentzel]

@Ibrahimmaga thanks for the detailed analysis, could you please expand on the following for each option (NPM, Lerna, Node, Yarn)

• Why NPM 7 may not be the best fit for a large enterprise project? Could you also describe performance implications between Node 14 vs 16 as far as this project goes?
• What is the impact on Code Coverage?
• What is the impact on Beachball used for publishing?
• What is the impact on performance?

Thanks,

[bennypowers]

I just got finished moving from yarn1 to npm7 there were some hiccups but thankfully it's all looking good now. I even got to drop wsrun since npm has npm run --if-present built in.

npm7 + changesets is looking pretty good if you can pull it off

[KingOfTac]

I'm still having issues with typescript not finding package imports when using yarn2 with pnp. I haven't tried npm7 yet to see if it is any better.

[Ibrahimmaga]

@Ibrahimmaga thanks for the detailed analysis, could you please expand on the following for each option (NPM, Lerna, Node, Yarn)

• Why NPM 7 may not be the best fit for a large enterprise project? Could you also describe performance implications between Node 14 vs 16 as far as this project goes?
• What is the impact on Code Coverage?
• What is the impact on Beachball used for publishing?
• What is the impact on performance?

Thanks,

@awentzel

1. I was saying is all version of NPM before NPM 7 was not the best choice for large monorepo project because they did not have workspace and package.lock.json file format for rebuilding your package tree from scratch. That's why people were using yarn.
2. Looking back on node releases, they always have improved in performance. However node version 16 will ship with the v9.0 engine, and still be written in C++ with new features, and will also remove experimental API. This could actually be a win using node version 16 for the creator project since customers do not have to download a library. We should consult with @janechu if customers will be impacted.
3. code coverage, no impact.
4. beachball, no impact