Version 1.0 problems in manifest.json when importing from composer vendor folder

[minimit]

Hello I'm trying to update sylius webpack encore to 1.0.3 but the manifest.json is all wrong, maybe it's because sylius entry point is a javascript that imports files from vendor?

This is the entry point. sylius is inside vendor.

import 'sylius/bundle/AdminBundle/Resources/private/entry';

and the content of manifest.json is this

{
  "build/shop/shop-entry.css": "/build/shop/shop-entry.css",
  "build/shop/shop-entry.js": "/build/shop/shop-entry.js",
  "build/shop/shop-entry.eot": "/build/shop/fonts/outline-icons.752905fa.eot",
  "build/shop/shop-entry.svg": "/build/shop/images/outline-icons.9c4845b4.svg",
  "build/shop/shop-entry.ttf": "/build/shop/fonts/outline-icons.53671035.ttf",
  "build/shop/shop-entry.woff": "/build/shop/fonts/outline-icons.ddae9b1b.woff",
  "build/shop/shop-entry.woff2": "/build/shop/fonts/outline-icons.687a4990.woff2",
  "build/shop/shop-entry.png": "/build/shop/images/logo.72a9465e.png",
  "build/shop/shop-entry.gif": "/build/shop/images/loading.f657825a.gif",
  "build/shop/shop-entry.jpg": "/build/shop/images/homepage-banner.8d8f072a.jpg",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/brand-icons.svg": "/build/shop/images/brand-icons.6729d297.svg",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/icons.svg": "/build/shop/images/icons.62d9dae4.svg",
  "build/shop/vendor/sylius/sylius/src/Sylius/Bundle/ShopBundle/Resources/private/img/homepage-banner.jpg": "/build/shop/images/homepage-banner.8d8f072a.jpg",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/outline-icons.svg": "/build/shop/images/outline-icons.9c4845b4.svg",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/icons.eot": "/build/shop/fonts/icons.a01e3f2d.eot",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/icons.ttf": "/build/shop/fonts/icons.c656b8ca.ttf",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/brand-icons.eot": "/build/shop/fonts/brand-icons.d68fa3e6.eot",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/brand-icons.ttf": "/build/shop/fonts/brand-icons.65a2fb6d.ttf",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/brand-icons.woff": "/build/shop/fonts/brand-icons.cac87dc0.woff",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/brand-icons.woff2": "/build/shop/fonts/brand-icons.278156e4.woff2",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/icons.woff": "/build/shop/fonts/icons.425399f8.woff",
  "build/shop/vendor/sylius/sylius/src/Sylius/Bundle/ShopBundle/Resources/private/img/sylius-plus-banner.png": "/build/shop/images/sylius-plus-banner.fff4fe72.png",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/icons.woff2": "/build/shop/fonts/icons.38c6d8ba.woff2",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/outline-icons.eot": "/build/shop/fonts/outline-icons.752905fa.eot",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/outline-icons.ttf": "/build/shop/fonts/outline-icons.53671035.ttf",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/images/flags.png": "/build/shop/images/flags.99f63ae7.png",
  "build/shop/vendor/sylius/sylius/src/Sylius/Bundle/UiBundle/Resources/private/img/logo.png": "/build/shop/images/logo.72a9465e.png",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/outline-icons.woff": "/build/shop/fonts/outline-icons.ddae9b1b.woff",
  "build/shop/node_modules/semantic-ui-css/themes/default/assets/fonts/outline-icons.woff2": "/build/shop/fonts/outline-icons.687a4990.woff2",
  "build/shop/node_modules/lightbox2/dist/images/loading.gif": "/build/shop/images/loading.f657825a.gif",
  "build/shop/vendor/sylius/sylius/src/Sylius/Bundle/UiBundle/Resources/private/img/avatar.png": "/build/shop/images/avatar.fa261429.png",
  "build/shop/node_modules/lightbox2/dist/images/prev.png": "/build/shop/images/prev.0edc57cc.png",
  "build/shop/node_modules/lightbox2/dist/images/next.png": "/build/shop/images/next.62074ac7.png",
  "build/shop/node_modules/lightbox2/dist/images/close.png": "/build/shop/images/close.0cfd6489.png"
}

Should be this

  "build/shop/shop-entry.css": "/build/shop/shop-entry.css",
  "build/shop/shop-entry.js": "/build/shop/shop-entry.js",
  "build/shop/fonts/brand-icons.woff2": "/build/shop/fonts/brand-icons.278156e4.woff2",
  "build/shop/fonts/brand-icons.ttf": "/build/shop/fonts/brand-icons.65a2fb6d.ttf",
  "build/shop/fonts/brand-icons.woff": "/build/shop/fonts/brand-icons.cac87dc0.woff",
  "build/shop/fonts/brand-icons.eot": "/build/shop/fonts/brand-icons.d68fa3e6.eot",
  "build/shop/fonts/icons.woff2": "/build/shop/fonts/icons.38c6d8ba.woff2",
  "build/shop/fonts/icons.woff": "/build/shop/fonts/icons.425399f8.woff",
  "build/shop/fonts/icons.eot": "/build/shop/fonts/icons.a01e3f2d.eot",
  "build/shop/fonts/icons.ttf": "/build/shop/fonts/icons.c656b8ca.ttf",
  "build/shop/fonts/outline-icons.ttf": "/build/shop/fonts/outline-icons.53671035.ttf",
  "build/shop/fonts/outline-icons.woff2": "/build/shop/fonts/outline-icons.687a4990.woff2",
  "build/shop/fonts/outline-icons.eot": "/build/shop/fonts/outline-icons.752905fa.eot",
  "build/shop/fonts/outline-icons.woff": "/build/shop/fonts/outline-icons.ddae9b1b.woff",
  "build/shop/images/avatar.png": "/build/shop/images/avatar.fa261429.png",
  "build/shop/images/brand-icons.svg": "/build/shop/images/brand-icons.6729d297.svg",
  "build/shop/images/close.png": "/build/shop/images/close.0cfd6489.png",
  "build/shop/images/flags.png": "/build/shop/images/flags.99f63ae7.png",
  "build/shop/images/homepage-banner.jpg": "/build/shop/images/homepage-banner.8d8f072a.jpg",
  "build/shop/images/icons.svg": "/build/shop/images/icons.62d9dae4.svg",
  "build/shop/images/loading.gif": "/build/shop/images/loading.f657825a.gif",
  "build/shop/images/logo.png": "/build/shop/images/logo.72a9465e.png",
  "build/shop/images/next.png": "/build/shop/images/next.62074ac7.png",
  "build/shop/images/outline-icons.svg": "/build/shop/images/outline-icons.9c4845b4.svg",
  "build/shop/images/prev.png": "/build/shop/images/prev.0edc57cc.png",
  "build/shop/images/sylius-plus-banner.png": "/build/shop/images/sylius-plus-banner.fff4fe72.png"
}

[ureimers]

This all seems to be related with the webpack-manifest-plugin in connection with Webpack 5.0. The currently open tickets that are related to this are:
Manifest keys contain source path: shellscape/webpack-manifest-plugin#238
Manifest populates main entry point with unrelated files with webpack 5: shellscape/webpack-manifest-plugin#237

There are other issues too that might be related (see current list of issues in https://github.com/shellscape/webpack-manifest-plugin/issues).

I fear that the webpack-manifest-plugin currently isn't working with Webpack 5.0 and that we have to move back to webpack-encore prior to v1.0 which is a bummer since I spent half the day migrating and fixing our webpack.config.js but our manifest.json is in such a bad shape and even assets that are correctly mapped in it are then prefixed with the build path twice when using Symfonys assets function (bundle/build/icons/spritemap.svg becomes /bundle/build/bundle/build/spritemap.svg in the rendered HTML). We'll try to figure it out piece by piece...

[weaverryan]

Hi everyone!

Thanks for the issue and the detailed update @ureimers - that's very helpful!

I need to dig deeper, but since we need to get this fixed, I see 2 options:

1. Find the "bug" in webpack-manifest-plugin, embed that plugin temporarily into Encore, then fix it. We have done this in the past - it's not ideal, but it gets the job done. Then, ideally, we could get that patch merged upstream and drop our copy of the plugin.

2. Try using assets-webpack-plugin instead. We already have this package in Encore - it generates the entrypoints.json file. I do not know if it suffers from the same bug or not. It actually has way less downloads than webpack-manifest-plugin but it seems to be more active.

If anyone wants to play with assets-webpack-plugin to see if it can be used for the manifest.json file (with the same format as we have now), that would be awesome and help us get this fixed faster.

Cheers!

[ureimers]

Hi @weaverryan,

glad I could help kicking this off.

1. I'm no JS expert, but I could still try to find out what's causing the problem with webpack-manifest-plugin. It's more a matter of having enough free time to do so. The package itself was taken over by shellscape end of october last year (Manifest keys contain hashes in production build when using file-loader shellscape/webpack-manifest-plugin#209 (comment)). He's seems to be quite busy at the moment and it's hard for me to see how active all the other contributors are. Still, embedding the plugin into Encore and then fixing it seems to be the best option here, because:

2. The problem, I see with the alterantive assets-webpack-plugin, is, that it doesn't produce manifest.json files like Webpack Encore needs them (or is used to have). Its documentation describes the format like this (https://github.com/ztoben/assets-webpack-plugin#example-output):

The output is a JSON object in the form:

{
   "bundle_name": {
       "asset_kind": "/public/path/to/asset"
   }
}

Where:

"bundle_name" is the name of the bundle (the key of the entry object in your webpack config, or "main" if your entry is an array).
"asset_kind" is the camel-cased file extension of the asset

I'm not sure if I get the naming right here, but every single mapped asset would need to be a bundle name, and the result would have a lot of superfluous "asset_kind" keys. For example:

// Before
{
  "build/form.js": "/relaunch2020/build/form.cd8233e8.js",
  "build/spritemap.svg": "/build/spritemap.2970a35c.svg",
  "build/fonts/28EB38_0_0.woff": "/build/fonts/28EB38_0_0.3c310b9b.woff"
}

// After
{
  "build/form.js": {
    "js": "/relaunch2020/build/form.cd8233e8.js"
  },
  "build/spritemap.svg": {
    "svg": "/build/spritemap.2970a35c.svg"
  },
  "build/fonts/28EB38_0_0.woff": {
    "woff": "/build/fonts/28EB38_0_0.3c310b9b.woff"
  }
}

It's possible, but it doesn't quite seem to be the right format. What do you think?

Regards,

Ulf

[weaverryan]

If you're having this problem, could you please test #921 and tell me if it works for you or doesn't work for you?

Thanks!

[minimit]

If you're having this problem, could you please test #921 and tell me if it works for you or doesn't work for you?

Thanks!

It works great the output is the same as version 0.33.0 with the right manifest!

[ureimers]

If you're having this problem, could you please test #921 and tell me if it works for you or doesn't work for you?

Thanks!

@weaverryan Looks a lot better now, thank you!

There are three issues left:

1. Spritemaps (fixed)
We're also using svg-spritemap-webpack-plugin like this:

Encore.addPlugin(new SVGSpriteMapPlugin('./public/bundles/my-bundle/icons/**/*.svg', {
        output: {
            filename: Encore.isProduction()
                ? 'spritemap.[contenthash].svg'
                : 'spritemap.svg',
        // ...
}

This worked in Webpack Encore 0.24 (we hadn't updated in a while) but with the new Webpack Encore 1.0 / webpack-manifest-plugin combination the manifest's source filename would include the [contenthash] too. Like:

"build/spritemap.9a7fc4ab.js":"build/spritemap.9a7fc4ab.js"

I fixed this by using and modifying the new removeHashKey method from webpack-manifest-plugin like I described here: #808 (comment)

2. Stylus assets (no fix needed?)
Also all of our inlined font files (using Encore.enableStylusLoader() and .styl files) are now in the map, which they weren't before. But I would say that this new behaviour is better since the fonts might be updated in the future and thus, having them use/support cash busting is helpful.

3. Numerically named files (conditional imports?)
There are now three numerically named files in the manifest.json:

  "build/217.js": "build/217.9ec7e8c9.js",
  "build/181.js": "build/181.9120a97c.js",
  "build/54.css": "build/54.4acca938.css",

They weren't there before (Encore 0.24) and are only created in a production build. As they are minified it's not that easy to find their source files but I was able to locate the .css file: It's conditionally imported inside a .vue file which itself is imported in a .js file which is added as an entry to Encore (Encore.addEntry() -> .js file -> import -> .vue file -> if() import -> .css file).

I'll try to investigate this further.

[stof]

@ureimers if fonts are referenced from the CSS, they don't need to be in the map (as the map is meant for usages in Symfony's asset() function in Twig). Cache busting of fonts is unrelated to them being in the map or no.

The fact that they are in the map is related to a change in webpack-manifest-plugin that is being reverted (see the associated PR).
The fact that chunks for conditional imports are there too might be related to the same issue in the plugin.

[ureimers]

@stof You're absolutely right: As the hashed font names are already written into all built assets they aren't needed in the manifest.

Which associated PR did you mean? shellscape/webpack-manifest-plugin#249 doesn't seem to revert a change to webpack-manifest-plugin but that's the only "hot" PR I can see (and am watching) right now.

Btw. @weaverryan Great work so far Sherlock! Following your track of comments and findings through the different packages (webpack-manifest-plugin, as well as webpack itself) is like reading an investigative story :-) Sorry to hear, that npm on Windows almost killed you ;-)

[weaverryan]

Hey @ureimers!

Following your track of comments and findings through the different packages (webpack-manifest-plugin, as well as webpack itself) is like reading an investigative story :-) Sorry to hear, that npm on Windows almost killed you ;-)

Ha! I'm super happy to know that someone else followed that rabbit hole of insanity! Thanks for the nice words :)

I fixed this by using and modifying the new removeHashKey method from webpack-manifest-plugin like I described here: #808 (comment)

I think this might, unfortunately, be the "correct" solution. That removeHashKey was added to fix a bug with CopyWebpackPlugin where the same things was happening. And now that I know the code super well, it doesn't surprise me: probably webpack-manifest-plugin is simply only aware of the final filename. It may be possible to fix that in webpack-manifest-plugin, I'm not sure. There is an issue (you probably saw) on Webpack itself to make this "manifest" thing simpler - it's pretty crazy complex at the moment.

There are now three numerically named files in the manifest.json:

If you build in "dev" mode, these should have super long names - like node_modules_something_else_foo.js. You mentioned that the CSS is "conditionally" imported - I assume you mean via the import() function? Yes, that's the type of thing that causes Webpack to split into multiple files (but not the only reason). Anyways, Webpack does a lot of file splitting, and in prod mode, it gives those files numbers like this :).

Which associated PR did you mean? shellscape/webpack-manifest-plugin#249 doesn't seem to revert a change to webpack-manifest-plugin but that's the only "hot" PR I can see (and am watching) right now.

My PR has rapidly changed in the past 36 hours :)

Cheers!