With webpack 5 and once deployed, local files are not readable from APIs anymore

[QuentinRoy]

What version of Next.js are you using?

10.2.0

What version of Node.js are you using?

14.0.0

What browser are you using?

Chrome,Firefox,Safari

What operating system are you using?

macOS

How are you deploying your application?

Vercel

Describe the Bug

Since updating to 10.2.0, local files are not accessible from apis anymore once deployed on Vercel. I do not know if it was on purpose, but it used to work fine before upgrading.

In both case, an internal server error is returned. After closer inspection of the logs, the error message is as follows:

[Error: ENOENT: no such file or directory, open '/app/data/metadata.yml'] {

errno: -2,

code: 'ENOENT',

syscall: 'open',

path: '/app/data/metadata.yml'

}

It went back to normal after downgrading to next 10.1.3.

EDIT: It seems the problem comes from webpack 5, it is working fine if webpack5 is disabled in next.config.js.

EDIT: It does not fail if there is only one API trying to read only one local file. It is possible the problem occurs when multiple files have to be available for reading.

EDIT: According to @jacksonhardaker, this also fails when reading files from getServerSideProps.

Expected Behavior

Local files should be readable from apis. If these are not embedded by default anymore, maybe an option could be used to indicate what files should be available?

To Reproduce

Here is an example showing APIs failing: https://github.com/QuentinRoy/vercel-local-min-example deployed at https://vercel-local-min-example.vercel.app.

Here is another example from @jacksonhardaker showing getServerSideProps failing: https://github.com/jacksonhardaker/next-webpack-5-fs-bug deployed at https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/.

[QuentinRoy]

Following @mmazzarolo suggestion in #8251, I can attest the problem actually comes with webpack 5. Disabling webpack 5 after updating fixes it:

next.config.js

module.exports = {
  future: {
    webpack5: false,
  },
};

EDIT:
With newer Next versions, switching back to webpack 4 is:

module.exports = {
  webpack5: false,
};

[joe-bell]

I manually opted into webpack5 and can confirm I'm also seeing this behaviour inside API routes

Example

API Route

const CONTENT_DIR = path.join(process.cwd(), "src", "content");

export const getContentByDir = async (
  dir: string,
  options?: IGetContentOptions
) => {
  const srcDir = path.join(CONTENT_DIR, dir);

  if (!fs.lstatSync(srcDir)?.isDirectory()) {
    // Error thrown to Vercel
    throw Error(`${srcDir} is not a valid directory`);
  }
  // other unrelated code
};

Vercel Logs

Error: ENOENT: no such file or directory, lstat '/var/task/src/content/blog'
    at Object.lstatSync (fs.js:1077:3)
    at getContentByDir (/var/task/.next/server/pages/api/sitemap.js:283:49)
    at getBlogPaths (/var/task/.next/server/pages/api/sitemap.js:292:23)
    at getAllPaths (/var/task/.next/server/pages/api/sitemap.js:313:67)
    at getSitemapPaths (/var/task/.next/server/pages/api/sitemap.js:314:43)
    at exports.modules.3692.__webpack_exports__.default (/var/task/.next/server/pages/api/sitemap.js:331:96)
    at apiResolver (/var/task/node_modules/next/dist/next-server/server/api-utils.js:8:7)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/next-server/server/next-server.js:66:462)
    at async Object.fn (/var/task/node_modules/next/dist/next-server/server/next-server.js:58:580) {
  errno: -2,
  syscall: 'lstat',
  code: 'ENOENT',
  path: '/var/task/src/content/blog'
}

Additional Context

I tried adding this to my next.config.js, but had no luck:

  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }

    return config;
  },

This behaviour doesn't occur when I opt-out of webpack5

[ijjk]

Hi, can someone who's run into this provide a repo with a minimal reproduction, we aren't able to reproduce this with the above provided information

[QuentinRoy]

Hi @ijjk, thank you for looking into this.
I investigated the problem a bit more. It seems it is working fine when there is only one api reading one local file. However, if there are two apis reading two local files, it breaks.
Here is a minimal reproduction repository: https://github.com/QuentinRoy/vercel-local-min-example.
This is deployed here: https://vercel-local-min-example.vercel.app

[jacksonhardaker]

Good tip @QuentinRoy I can confirm that the addition of a second read appears to reveal the issue.
Repo: https://github.com/jacksonhardaker/next-webpack-5-fs-bug
Deployment: https://next-webpack-5-fs-bug-mfyukza5m-jacksonhardaker1.vercel.app/ https://next-webpack-5-fs-bug-jwegk8q1o-jacksonhardaker1.vercel.app/

edit: updated the deployment url

[wworrall]

This is a major issue causing Vercel deployments to break silently (build completing successfully). Rolling back to webpack 4 for now - please fix asap, Vercel 😍. Also, perhaps add something to your tests as to which projects are automatically compatible with webpack5 https://nextjs.org/docs/messages/webpack5