Webpack Additional Compilation Pass

February 20, 2020 (Last updated February 27, 2020)

Recently, I was working on a server-side rendering application, and encounter a scenario that I think it requires "double compilation" with webpack.

The problem

I am not entirely sure I am doing it in the right approach, feel free to suggest or discuss it with me. The following will be about the problem I faced and how I worked on it.

The server-side rendering application that I worked on, has an endpoint that takes in request and respond with a partial HTML content and CSS files required for styling:

{
  "css": ["http://cdn/assets/style.xxx.css"],
  "html": "<div class="container_xyz">Hello world</div>"
}
/* filename: http://cdn/assets/style.xxx.css */
.container_xyz {
  font-family: 'Comic Sans';
}

The application code itself uses Express and React:

import express from 'express';
import React from 'react';
import { renderToStaticMarkup } from 'react-dom/server';
import styles from './app.scss';

const app = express();
app.get('/', (req, res) => {
  const app = <div className={styles.container}>Hello world</div>;
  const htmlContent = renderToStaticMarkup(app);

  res.json({
    css: [],
    html: htmlContent,
  });
});
app.listen(process.env.PORT);

Now, the problem is, how do I get the list of CSS files?

The list of CSS files produced by the build is only available after I compile the application, but I need the information to be part of compiled code.

The compiled code being part of the compilation, needs to contain information of the compilation.

The 1st approach

A naive solution at first is to use Webpack Manifest Plugin to get the compilation manifest, and in the code, import the manifest as json and consumes it:

// ...
import webpackManifest from './dist/webpack-manifest.json';
const cssFiles = filterCssFiles(webpackManifest);
// ...
app.get('/', (req, res) => {
  res.json({
    css: cssFiles,
    html: htmlContent,
  });
});
// ...

Yet, the ./dist/webpack-manifest.json is not available in the first place, before compiling the code.

Since the ./dist/webpack-manifest.json can only be available after build, maybe we can import it during runtime, using non_webpack_require. The difference between require and __non_webpack_require__ is that the latter is webpack specific, which tells webpack to transform it to just pure require() expression, without bundling the required module:

// ...
const webpackManifest = __non_webpack_require__('./dist/webpack-manifest.json');
const cssFiles = filterCssFiles(webpackManifest);
// ...
app.get('/', (req, res) => {
  res.json({
    css: cssFiles,
    html: htmlContent,
  });
});
// ...

If you scrutinize the code, you may wonder whether ./dist/webpack-manifest.json is the correct relative path from the compiled code?

Probably ./webpack-manifest.json would be more accurate, if our output folder looks like this:

dist
├── webpack-manifest.json
└── bundle.js  // <-- the main output bundle

One can safely argue that, the approach above works and let's move on the next task. But, curiosity drives me to seek deeper for a more "elegant" solution, where one don't need require('webpack-manifest.json') in runtime, but that information is compiled into the code.

[Updated Feb 27, 2020]

Thanks to @wSokra's suggestion, instead of using __non_webpack_require__(), you can use a normal import and declaring the manifest file as an external:

// ...
import webpackManifest from 'webpack-manifest';
const cssFiles = filterCssFiles(webpackManifest);
// ...
webpack.config.js
module.exports = {
  externals: {
    'webpack-manifest': "commonjs2 ./webpack-manifest.json"
  }
}

What this output is something similar to the following:

bundle.js
const webpackManifest = require('./webpack-manifest.json');
const cssFiles = filterCssFiles(webpackManifest);
// ...

The reason we are using the relative path ./webpack-manifest.json is that we are assuming the output folder looks like this:

dist
├── bundle.js  // <-- the main output bundle
└── webpack-manifest.json // <-- relative to bundle.js

You can read more about webpack externals from the webpack documentation.

The 2nd approach

So, the next "intuitive" approach is to write a custom template plugin, that adds the webpack manifest on top of the main bundle, an example of the output:

bundle.js
// added by template plugin
const CSS_FILES = ['http://cdn/assets/style.xxx.css'];
// END added by template plugin
// ...the main bundle
app.get('/', (req, res) => {
  // ...
  res.json({
    css: CSS_FILES,
    html: htmlContent,
  });
});

In the source code, I will use the global variable CSS_FILES, and hopefully it will get defined by webpack, by adding const CSS_FILES = ... at the very top of the file.

And to be extra careful, I have to make sure also that there's no variable CSS_FILES declared between the global scope and the current scope the variable is being used.

const ManifestPlugin = require('webpack-manifest-plugin');

class MyWebpackPlugin {
  apply(compiler) {
    new ManifestPlugin(manifestOptions).apply(compiler);
    // get manifest from `webpack-manifest-plugin`
    ManifestPlugin.getCompilerHooks(compiler).afterEmit.tap(
      'MyWebpackPlugin',
      manifest => {
        this.manifest = manifest;
      }
    );

    // see https://lihautan.com/webpack-plugin-main-template
    // on writing template plugin
    compiler.hooks.thisCompilation.tap('MyWebpackPlugin', compilation => {
      // ...
      hooks.render.tap('MyWebpackPlugin', (source, { chunk, chunkGraph }) => {
        // ...
        const prefix = `const CSS_FILES = ${JSON.stringify(this.manifest)};`;
        return new ConcatSource(prefix, source);
      });
    });
  }
}

Apparently, this does not work at all. The compiled output shows:

bundle.js
const CSS_FILES = undefined;
// ...continue with bundle.js

After tracing through the code, I realised that I was ignorant of the sequence of execution of the compiler hooks.

In the docs for compiler hooks, each hooks is executed in sequence:

  • ...
  • run
  • ...
  • thisCompilation
  • ...
  • emit
  • afterEmit
  • ...

The webpack manifest plugin executes mainly during the emit phase, right before webpack writes all the assets into the output directory. And, we are modifying the template source in the thisCompilation phase, which is way before the emit phase. That's why this.manifest property is still undefined at the time of execution.

- thisCompliation (this.manifest == undefined;)
- // ...
- emit (this.manifest = manifest) // too late!

Upon reading the code fot he webpack-manifest-plugin, I realised that during the emit phase, I can access to the compilation.assets, and so, I could modifying the source for the assets during that time!

const ManifestPlugin = require('webpack-manifest-plugin');

class MyWebpackPlugin {
  apply(compiler) {
    new ManifestPlugin(manifestOptions).apply(compiler);
    // get manifest from `webpack-manifest-plugin`
    ManifestPlugin.getCompilerHooks(compiler).afterEmit.tap(
      'MyWebpackPlugin',
      manifest => {
        this.manifest = manifest;
      }
    );
    compiler.hooks.emit.tap('MyWebpackPlugin', compilation => {
      const prefix = `const CSS_FILES = ${JSON.stringify(this.manifest)};`;

      for (const file of Object.keys(compilation.assets)) {
        if (!file.endsWith('.js')) continue;
        compilation.assets[file] = new ConcatSource(
          prefix,
          compilation.assets[file]
        );
      }
    });
  }
}

Apparently that works, but I wonder whether is it a good practice to modifying the source of an asset during the emit phase? 🤔

And, if you noticed, I need to append the const CSS_FILES = [...] to every file, that's because I have no idea in which file CSS_FILES is referenced. And because I declared it using const, it only exists within the file's scope, so I have to redeclare it all the other files.

[Updated Feb 27, 2020]

According to @evilebottnawi that this is not appropriate

The 3rd approach

I was still not convinced that this is the best I could do, so I continued looking around webpack's doc. I found a particular compilation hooks, needAdditionalPass, which seems useful. It says, "Called to determine if an asset needs to be processed further after being emitted.".

So, if I return true in the needAdditionalPass, webpack will recompile the asset again:

class MyWebpackPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap('MyWebpackPlugin', compilation => {
      compilation.hooks.needAdditionalPass.tap('MyWebpackPlugin', () => {
        return true;
        // if it is always true, will lead to infinite loop!
      });
    });
  }
}
- thisCompliation (this.manifest == undefined;)
- // ...
- emit (this.manifest = manifest) // too late!
- // ...
- needAddtionalPass (return true) // to start the compilation again
- // ...
- thisCompilation (this.manifest == manifest) // now `this.manifest` is available
- // ... will continue run through every stages again
- emit (this.manifest = manifest)
- // ...

Note that using needAdditionalPass will cause the build time to roughly doubled!

You may argue that why do we need to rerun the compilation process again, isn't the end result can be equally achieved by modifying the assets source in the emit phase?

Well, that's because, I realised I could make use some of the code from the DefinePlugin, which could replace the usage of CSS_FILES throughout the code. That way, I don't have to prefix every file with const CSS_FILES = ....

DefinePlugin uses something called JavaScriptParser Hooks, which you can rename a variable through canRename and identifier hooks or replace an expression through the expression hook:

class MyWebpackPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap(
      'MyWebpackPlugin',
      (compilation, { normalModuleFactory }) => {
        normalModuleFactory.hooks.parser
          .for('javascript/auto')
          .tap('MyWebpackPlugin', parser => {
            parser.hooks.expression
              .for('CSS_FILES')
              .tap('MyWebpackPlugin', expr => {
                return ParserHelpers.toConstantDependency(
                  parser,
                  JSON.stringify(this.manifest)
                )(expr);
              });
          });
      }
    );
  }
}

The complete code can be found in this gist.

An example of the compiled output:

bundle.js
// ...
app.get('/', (req, res) => {
  // ...
  res.json({
    // replaced via parser hooks
    css: ['http://cdn/assets/style.xxx.css'],
    html: htmlContent,
  });
});

Closing Notes

The compile output for the 3rd approach seemed to be better (more precise?) than the other, yet I am not entirely sure using a needAdditionalPass is the right way of going about it.

So, let me know if you have any thoughts or suggestions, yea?

[Updated Feb 27, 2020]

You can read the discussions that's happening on Twitter: