Error running with webpack: ENOENT: no such file or directory, open 'iam_service.json'

[hgardneriv]

Environment details

• OS: Linux & MacOS
• Node.js version: v14.15.0
• npm version: 6.14.8
• @google-cloud/pubsub version: 2.10.0

Steps to reproduce

1. When running a pubsub consumer or producer with webpack, the following error occurs:

"err":{"message":"ENOENT: no such file or directory, open 'iam_service.json'"," stack":"Error: ENOENT: no such file or directory, open 'iam_service.json'\n at Object.openSync (fs.js:476:3)\n at Object.readFileSync (fs.js:377:35)\n at fetch (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403345:34) at Root.load (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403379:13) at Root.loadSync (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:403420:17) at Object.loadSync (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:105027:29) at GrpcClient.loadFromProto (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245393:48) at GrpcClient.loadProto (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245434:21) at new IamClient (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245684:38) at new SubscriberClient (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:86374:26)"

NOTE: This issue only happens when running with webpack, it works expected when webpack is not in use.

2. This problem can be worked around by creating directory ../../protos and place the following files in it:

$ ls ../../protos
iam_service.json
protos.json

Unfortunately ../../protos falls one level up of our install directory, so it's not a valid work around.

3. Using the {fallback: true} option in the PubSub constructor, as suggested in this post: Webpack and other bundler support google-cloud-node#2933, results in another error:

`"fetch is not a function","stack":"TypeError: fetch is not a function
at Service.newServiceStub. [as testIamPermissions] (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244522:17)
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:245730:29
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:246661:16
at OngoingCall.call (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244030:27)
at NormalApiCaller.call (/Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:246452:19)
at /Users/harrygardner/Documents/code/cribl/dist/sluice/bin/cribl.bundle.js:244163:30\n at processTicksAndRejections (internal/process/task_queues.js:93:5)

Is there a work around for this issue?

[feywind]

@hgardneriv Thanks for the report! I think for now our take on webpack / rollup / et al is that we don't officially support it, but there is some talk about moving toward making it work at some point. We're putting the (badly named) web label on issues that have to do with that.

[amiracle]

@feywind is there any way to escalate this issue? We are running into this same issue would like to know if there is a path forward from GCP / Google. Thanks again!

[feywind]

@amiracle The quickest way I know of to start some sort of escalation is through a support case, but I can say that there are actually some higher level discussions going on about the webpack support right now, so that might not be necessary. I can't promise anything, but it's more and more on the radar lately.

[feywind]

@alexander-fenster is taking a look at this now, he's our gax maintainer, so hopefully he'll have something for you.

[alexander-fenster]

Hi @hgardneriv,

Judging by your stack, I'm assuming you run webpack with target: "node", is this assumption correct?

In this case, don't use the fallback option (which is for browsers and other environments with no gRPC support).

The problem is caused by this snippet in google-gax:

    const nodejsProtoPath = path.join(
      __dirname,
      '..',
      '..',
      'protos',
      'iam_service.json'
    );

The default behavior of webpack changes how __dirname works (you can check it by doing console.log(__dirname) in your script and comparing the result with and without webpack), so this logic does not work properly when webpacked. The workaround is to add the following into your webpack config:

module.exports = {
  mode: 'production',
  target: 'node',
  node: {
    __dirname: true,  // this makes all the difference
  },
};

Please let us know if this workaround is acceptable for you.

[alexander-fenster]

To add to what I just said - since this code actually uses fs.readFileSync to read the JSON file, the webpack bundle produced won't be self contained: it will still need to be able to read node_modules/@google-cloud/pubsub/build/protos/protos.json and node_modules/google-gax/protos/iam_service.json from the filesystem so you will need to put at least these two files in your distribution along with the bundle.

[alexander-fenster]

Adding one more thing :) This is my test project that worked for me: https://gist.github.com/alexander-fenster/33c0142f0d62c2d78d0fa3e76559e26b

[hgardneriv]

Hello @alexander-fenster thank you very much for the info. Yes, we are running webpack with 'target: "node"'.

Unfortunately enabling __dirname will not work in this case, we use the following:

      node: {
        __dirname: false,
        __filename: false,
      },

Something that would be nice is the ability to pass in the config directly vs. reading from the filesystem. With this option we could easily work around the issue. Given our webpack config, do you see any other viable work arounds?