Bundling sharp doesn't work anymore since updating to 4.5.0

[pkeuter]

Since updating to version 4.5.0 it is not possible to bundle sharp anymore. Previously this was fairly simple, by just copying the node_modules/sharp folder next to the build output. But this doesn't work anymore giving the following error:

Error:
Something went wrong installing the "sharp" module

File '/**/test/node_modules/sharp/build/Release/libvips-cpp.so.42' was not included into executable at compilation stage. Please recompile adding it as asset or script.

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/install
- Search for this error at https://github.com/lovell/sharp/issues

    at Object.<anonymous> (/snapshot/test/node_modules/sharp/lib/constructor.js:34:9)
    at Module._compile (pkg/prelude/bootstrap.js:1329:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1218:10)
    at Module.load (internal/modules/cjs/loader.js:1047:32)
    at Function.Module._load (internal/modules/cjs/loader.js:935:14)
    at Module.require (internal/modules/cjs/loader.js:1087:19)
    at Module.require (pkg/prelude/bootstrap.js:1234:31)
    at require (internal/modules/cjs/helpers.js:73:18)
    at Object.<anonymous> (/snapshot/test/node_modules/sharp/lib/index.js:3:15)
    at Module._compile (pkg/prelude/bootstrap.js:1329:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1218:10)
    at Module.load (internal/modules/cjs/loader.js:1047:32)
    at Function.Module._load (internal/modules/cjs/loader.js:935:14)
    at Module.require (internal/modules/cjs/loader.js:1087:19)
    at Module.require (pkg/prelude/bootstrap.js:1234:31)
    at require (internal/modules/cjs/helpers.js:73:18)

I'm fairly convinced this has to do with #837 or #1066 or maybe a combination of both.

Looking at the below error message running in CI, it seems that prebuild-install is not correctly installed or referenced, this has been added in #1066:

> pkg@4.5.0
> Fetching base Node.js binaries to PKG_CACHE_PATH
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
/bin/sh: 1: /builds/test/node_modules/pkg/node_modules/.bin/prebuild-install: not found
/bin/sh: 1: /builds/test/node_modules/pkg/node_modules/.bin/prebuild-install: not found

I'm mentioning both @david-mohr as well as @geekuillaume hoping they have a suggestion on how to fix this.

On a sidenote; this seems to me a breaking change so 5.0.0 would have been a more logical version for this release I suppose.

[david-mohr]

have you run yarn to ensure that prebuild-install is actually installed?

[jesec]

If the mentioned changes are indeed breaking, I think we would have to revert them immediately and release a new version.

First, I would wait for a while for responses from the related parties, and see if there is a misconfiguration or if the changes can be tweaked to be non-breaking.

@pkeuter mentioned just copying the node_modules/sharp folder next to the build output. Is this "load from outside" way previously supported?

Any thoughts? @leerob @robertsLando

[leerob]

Before doing any reverting, would need to verify and confirm it is indeed a breaking change. I'm not convinced those two PRs are related yet but would love to see the investigation here.

[david-mohr]

prebuild-install is definitely in the package.json. @pkeuter how did you upgrade and does running 'yarn' or npm i fix the issue?

[pkeuter]

I see it is in the package.json and it is also in the package-lock.json (I'm using npm):

"pkg": {
      "version": "4.5.0",
      "resolved": "https://registry.npmjs.org/pkg/-/pkg-4.5.0.tgz",
      "integrity": "sha512-TEAQAbrd4PKyEydg28OLzg61L/NsGN1NcCqBFzY9TxzpPAds/Nogd+9bc2r/nZxBQhHLL1l+LldVNRFoQhdPWg==",
      "dev": true,
      "requires": {
        "@babel/parser": "^7.9.4",
        "@babel/runtime": "^7.9.2",
        "chalk": "^3.0.0",
        "escodegen": "^1.14.1",
        "fs-extra": "^8.1.0",
        "globby": "^11.0.0",
        "into-stream": "^5.1.1",
        "minimist": "^1.2.5",
        "multistream": "^2.1.1",
        "pkg-fetch": "^2.6.9",
        "prebuild-install": "6.0.1",
        "progress": "^2.0.3",
        "resolve": "^1.15.1",
        "stream-meter": "^1.0.4"
      },
...

and this is top-level the prebuild-install:

    "prebuild-install": {
      "version": "6.0.1",
      "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-6.0.1.tgz",
      "integrity": "sha512-7GOJrLuow8yeiyv75rmvZyeMGzl8mdEX5gY69d6a6bHWmiPevwqFw+tQavhK0EYMaSg3/KD24cWqeQv1EWsqDQ==",
      "requires": {
        "detect-libc": "^1.0.3",
        "expand-template": "^2.0.3",
        "github-from-package": "0.0.0",
        "minimist": "^1.2.3",
        "mkdirp-classic": "^0.5.3",
        "napi-build-utils": "^1.0.1",
        "node-abi": "^2.7.0",
        "noop-logger": "^0.1.1",
        "npmlog": "^4.0.1",
        "pump": "^3.0.0",
        "rc": "^1.2.7",
        "simple-get": "^3.0.3",
        "tar-fs": "^2.0.0",
        "tunnel-agent": "^0.6.0",
        "which-pm-runs": "^1.0.0"
      },
      "dependencies": {
        "simple-get": {
          "version": "3.1.0",
          "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-3.1.0.tgz",
          "integrity": "sha512-bCR6cP+aTdScaQCnQKbPKtJOKDp/hj9EDLJo3Nw4y1QksqaovlW/bnptB6/c1e+qmNIDHRK+oXFDdEqBT8WzUA==",
          "requires": {
            "decompress-response": "^4.2.0",
            "once": "^1.3.1",
            "simple-concat": "^1.0.0"
          }
        }
      }
    },

This was running in CI so there was no node_modules folder. No cache or something either.

When I look in the folder referenced /home/peter/test/node_modules/pkg/node_modules/.bin/ I don't see the binary either. I do see it a bit higher in /home/peter/test/node_modules/.bin. This probably is because of the flattening procedure that npm uses (see https://docs.npmjs.com/cli/v7/configuring-npm/folders). I am not sure if this differs from yarn but this seems to be a logical structure.

[david-mohr]

@pkeuter ah yes, I see what you mean now. Here's the target reference https://github.com/vercel/pkg/blob/master/lib/producer.js#L100. As you say, yarn handles this differently which is why it worked fine during testing. I also have my pkg install globally.

If you know how to change the path to handle any combination of situations (local/global, yarn/npm) that would be great.

[pkeuter]

@david-mohr I do not know a solution right away, but the fact is that this is, indeed, a breaking change so it might be useful to revert the change in a new release until this has been resolved. Because pkg is currently broken for npm users that use native modules.

[david-mohr]

@pkeuter I think this can be patched to work with both npm and yarn, but if it needs to be backed out, that's fine too.

@leerob guess it's your call. To fix this, the function can use existsSync to hunt around for the path. Otherwise just back it out, it doesn't seem that other users are interested in cross-platform support, so it's not really a helpful patch.

[leerob]

I'm sorry you're having this issue @pkeuter ☹️ It currently appears it's isolated to one report. Given that, I'd say we patch forward. Cross-platform support seems like something we would want to support.

[JeffJassky]

The prebuild-install problem:

The path to the prebuild script is being built incorrectly. The error is in producer.js

pkg/lib/producer.js

Line 135 in f08d083

const prebuild = path.join(

A Workaround:

I got it to resolve the correct path by changing that line to: const prebuild = _path.default.join(process.cwd(), './node_modules/.bin/prebuild-install');. After that, the prebuild-install script path was correct - and it was being executed successfully.

However... The prebuild script shows a warning prebuild-install WARN install No prebuilt binaries found (target=v12.2.0 runtime=node arch=x64 libc= platform=win32)

Manually Installing cross-platform prebuilt binaries:

So I installed my native module (sqlite3) manually using npm install sqlite3 --arch=x64 --platform=win32 which resulted in a very happy looking Success: "/Users/jeffjassky/Projects/StoryFolderDesktop/node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node" is installed via remote

Unfortunately:

When running pkg I still got the same error: prebuild-install WARN install No prebuilt binaries found (target=v12.2.0 runtime=node arch=x64 libc= platform=win32) as if it doesn't recognize or include the newly installed module.

My guess is that the target version was incorrect - but it's hard to know for sure. When I tried adding the --target=v12.2.0 flag I got a nasty Unsupported target version: v12.2.0 error.

So I tried manually added the module as an asset in package.json

  "pkg": {
    "assets": [
      "./node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node"
    ]
  }

When I run my packaged .exe on Windows I can see that the file indeed gets included, but I get this error: Caught exception: Error: The specified module could not be found. C:\snapshot\StoryFolderDesktop\node_modules\sqlite3\lib\binding\napi-v3-win32-x64\node_sqlite3.node

So, lastly, I copied the node_sqlite3.node file into path-to-executable/node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node along side the executable and that finally seems to work, as inelegant as it is.

So, for some reason, it can't locate the module even though it's right there in the snapshot - but it works when the module is placed alongside it.

[robertsLando]

@JeffJassky Usually this kind of problems are fixed by dictionaries. You need to edit this https://github.com/vercel/pkg/blob/master/dictionary/sqlite3.js see an example here: https://github.com/vercel/pkg/blob/master/dictionary/puppeteer.js

[JeffJassky]

@robertsLando I sort-of understand. These are package-specific configs for pkg - though I don't precisely know what I should put in there - 'cause I don't quite understand what the problem is, to begin with.

[robertsLando]

I'm not the author of this package I have just take a general lookup of the code and from what I have understand that's a collection of package-specific fixes. With that you can specify patches (replace/add/remove code from files) and or additional assets/scripts to add for a specific package like:

'use strict';

module.exports = {
  pkg: {
    scripts: ['lib/middleware/*.js'],
    assets: [
      'lib/public/**/*', // for connect@2.3
    ],
  },
};

In your case I think a possible fix could be adding the assets needed here in the sqlite3 dictionary, something like:

'use strict';

module.exports = {
  pkg: {
    assets: [
      'lib/binding/**/*',
    ],
  },
};

Give this a try and let me know :)

[JeffJassky]

@robertsLando After looking into it further, the .node module is actually getting included in the snapshot - in the exact correct location. Hmm... the plot thickens.

Seems to be related directly to #1075, possibly #894, and #687

[robertsLando]

We might check where the lookup fails so. Based on what I understand native node modules behaves different then normal assets, them are copied in a temporary folder iin order to be used, maybe the lookup or the copy fails somewhere. That feature has been added in #837. Did you alo try to put the node addon in the same root dir of the application to see if that works?

The magic happens here: https://github.com/vercel/pkg/pull/837/files#diff-0c3f42fd6825d8d5b4d0422402a89ff11dc1d1cfb52ee5901f4774e503783c27R1598