Parsing error when calling generic function with type arguments

April 23, 2019 (Last updated April 27, 2019)

Background

Mentioned in the previous post, we have upgraded babel 7 to support the new Flow syntax.

foobar<Type>(x) is now call foobar with x, and type parameter Type rather than (foobar < Type) > x) is the result of foobar smaller than Type, greater than x?.

Everything is great, until a weird runtime error caught us off guard.

ReferenceError: XType is not defined

export function foobar<XType>(baz);
                       ^ ReferenceError: XType is not defined

The first time I saw this error, my first impression was that I forgot to import XType, so I scrolled to the top of the document. But, alas, I did import XType.

So what is going on? 🤷‍

I clicked into the error,

export function foobar < XType > baz;

looked at me innocently. I knew something was wrong. XType wasn't stripeed by babel!

Shameless plug: If you read my eslint for flow syntax, you should be able to come to the same conclusion! 😅

So I checked the @babel/plugin strip-flow-types, I realise there's a all option that I had missed out, basically it says,

only parse Flow-specific features if a @flow pragma is present atop the file

It seems that in my file,

/** @module foobar */
// ...
export function foobar<XType>(baz);

I had a innocent looking /** @module */ comment above // @flow that breaks my babel plugin!

So the quick fix is to either:

  • Move // @flow comment above /** @module foobar */, or
  • Set all: true in @babel/plugin-transform-flow-strip-types.

Either way, it solves the issue.

However, one thing bothers me:

My Flow works perfectly fine with an extra comment on top // @flow, it still typechecks and provides auto-suggestions.

So, the logic for @babel/plugin-transform-flow-strip-types and Flow to determine whether a file is a Flow file or not is different!

And as a frequent user of Open Source libraries, this is something I think I can should fix, for the betterment of the JavaScript Open Source world 😅. I always imagine there's another innocent front-end developer across the world like me stumbled upon a perplexing bug, if only me let the bug go with a workaround/patch.

There's so much to achieve if we, not just consume the effort of others from the Open Source, but to also contribute into it.

Game Plan

So, to fix this bug, one simply has to:

  • Read Flow's source code and understand the logic
  • Read @babel/plugin-transform-strip-flow-type's source code and understand the logic
  • Make changes to babel code
  • Send a MR and brag about it 😎

Whether this is achieveable at my current level, that's a different story. But one has nothing to lose to try and fail.

Flow

I've read a bit of Flow source code previously, mainly to fix a bad developer experience I had with flowconfig previously. I had to learn OCaml, which was a fad a while ago because of ReasonML, to understand Flow source code.

So, this time around, I am much more comfortable to dig the code to find out the information I want.

I searched for the term "@flow", which ended me up with this function, extract_docblock which returns me the information of whether @flow is present in the file. And I dug further, I ended up with the annonymous function that extract_docblck returns.

Allow me to loosely translate the logic into some pseudo JavaScript:

const extract_docblock = ({ maxTokens, filename, content }) => {
  const file = read(filename);
  for (let i = maxTokens; i > 0; i--) {
    const token = file.nextToken();
    switch (token.type) {
      case 'string':
      case 'semicolon':
        continue;
      case 'comment':
        if (isFlowComment(token)) {
          return flowPragmaType(token);
        }
        break;
      default:
        return null;
    }
  }
};

In human language:

Flow will read maxTokens number of tokens, look for comments that matches @flow, if it encounters any order tokens, it will bail out early, with the exception of string and semicolon.

So,

'use strict';
foobar();

and

/** @module */
foobar();

is considered as a valid Flow file.

But

foobar();

or

foobar();

is not.

Babel

At first, I thought that the logic would be in @babel/transform-strip-flow-types, but apparently, its not.

I discovered that by realising that the source code of @babel/transform-strip-flow-types did not include anything about the all options, and this plugin extends the @babel/plugin-syntax-flow, which I knew fairly well that syntax plugins in babel does nothing but to enable syntax switch of the @babel/parser. The bulk of the logic lies within the @babel/parser's flow plugin.

That was all because I contributed to @babel/parser before.

And here we are in babel-parser, and the line that caught my attention is this:

addComment(comment: N.Comment): void {
  if (this.flowPragma === undefined) {
    // Try to parse a flow pragma.
    const matches = FLOW_PRAGMA_REGEX.exec(comment.value);
    if (!matches) {
      this.flowPragma = null;
    } else if (matches[1] === "flow") {
      this.flowPragma = "flow";
    } else if (matches[1] === "noflow") {
      this.flowPragma = "noflow";
    } else {
      throw new Error("Unexpected flow pragma");
    }
  }
  return super.addComment(comment);
}

So, the babel's logic of getting a Flow pragma is that as soon as the first comment encountered, we parse the comment and we turn on the Flow syntax switch.

This is the reason why if we have a comment before // @flow, we will not treat the file as a valid Flow file.

Interesting enough, this means that if we write

foobar < XType > 1;
foobar<XType>(1);

the first half of the code before // @flow was parsed as a normal JS code, and the second half after // @flow was parsed as a Flow code.

You can see this clearly with my recently build ASTExplorer clone for babel.

(I built it with React + Hooks over a long weekend, which I will share about how did it in the future.).

You can see that the first expression is a BinaryExpression but the second expression is a CallExpression;

Make changes to the babel code

Now step 3, make changes to babel code. So I decided to open an issue and started fixing the code. Surprisingly, someone else had reported the issue a few months ago, and the issue was still opened.

So I explained what I had discovered, and tried to propose a solution. Well, after some struggle, I realised I am still a bit behind from being able to fix this code.

So how?

I submitted a PR with a big WIP, because I didn't know how to look ahead n tokens and determine the flowPragma flag before babel starts parsing the code. I explored around the babel-parser source code, uncover new concepts that I never knew before. It took me a day to contemplate and fiddle around, until something sparked me.

I realised I do not have to follow exactly Flow's logic in order to achieve similar behaviour. That's when I submitted another PR and closed the previous one. (You can check it out if you are curious about it).

And finally, the fix has merged into babel v7.4.4! 🎉🎉

And I can't wait to try all the edge cases that I have fixed in babel repl:

Closing Remark

Well, I am sorry that I am going to stop here, because the issue is still opened, but I hoped you enjoy the detective journey along the way of hunting this bug.

If you encountered similar issues, you can patch it first with the solution I mentioned earlier. And do follow the Github issue, I will do my best to fix this.

If you encountered similar issues, please upgrade babel to v7.4.4.

The best thing about open source is that the source code is open. As part of the JS community, we should not just reap the efforts of the community when we are building our next billion dollar idea, we should also contribute back so that the community as a whole can grow and improve together.

As usual, here are the list of references for this article: