How to create a Gatsby.js transformer plugin

This article is a guide on how to create a Gatsby.js transformer plugin, even more specifically a plugin for a plugin (mainly gatsby-transformer-remark, but it works with gatsby-plugin-mdx, too). You can inspect the code of the finished plugin from this tutorial here.

What can you do with a Gatsby.js plugin?

Before jumping into the code, let's investigate what you can do with a Gatsby.js plugin. For this, I want to remind you what Gatsby.js is: A static website builder, similar to Jekyll or Hugo.

The magic happens when it fetches your desired content dynamically and integrates it into static files based on your configuration. You can set up the layout to your liking and wrapper components with React upfront and then plug in the content.

As simple as that. But then also, plugins can do things to your content. For example, if your content happens to be in Markdown, you want to transform it to HTML to output it on your website.

That's pretty much the whole process (with more details and mechanisms put in place along the process, but this is already a very light yet accurate description).

So there are mostly two main tasks Gatsby does for you, and so you can hook into each of those:

1. Fetching content from somewhere in a specific format.
2. Alter the content.

That gives you the possibility to create two types of plugins. Their concept and API are distinct, so practical experience from writing the one type of plugin does not strictly transfer to the other.

1. Source plugins

A source plugin has only one job: To connect your Gatsby.js build process to data, which usually is external, but can sometimes be internal, too.

Compared to the second type of plugins, source plugins are quite complex code-wise, or at least much more comprehensive by default.

Here is the official guide on creating source plugins.

2. Transformer plugins

A transformer plugin takes an input (usually some content data) and transforms it wholly or in parts.

Here is the official guide on creating transformer plugins.

Plugin example: gatsby-remark-color-highlight

I have created a plugin that searches your remark content for color hex codes and wraps them into an element to make the color visually appear.

// Takes this
<p>Lorem ipsum #abcdef and foo bar.</p>

// and transforms into this
<p>Lorem ipsum <span style="background-color: #abcdef">#abcdef<span> and foo bar.</p>

In this case, it's a plugin for another transformer plugin (gatsby-transformer-remark). That means, that after gatsby-transformer-remark has hooked into the delivered content and transformed a Markdown string into an AST (Abstract Syntax Tree), we are going to write a middleware basically, that takes this AST and further transforms it as we need it.

This fact again makes the plugin and tutorial not exactly transferrable to source or transformer plugins (although it's closer to the latter). The docs page on creating a Remark plugin is immensely helpful to our operation.

Let's start already:

$ yarn init # and fill out the questionnaire

Because we want to develop the plugin live with a Gatsby site, let's connect both. Change into the directory of your Gatsby project and adjust your gatsby-config.js. You can set the path to your plugin locally for now.

module.exports = {
  plugins: [
    {
      resolve: require.resolve(`../local/path/to/gatsby-remark-color-highlight`),
    },
  ],
};

Add some dev packages to make development pleasant and a few more utility libraries to juggle the data and colors.

$ yarn add --dev babel-preset-gatsby-package @babel/core cross-env
$ yarn add unist-util-visit color

The following is the signature of the function that we are going to export from the src/index.js.

module.exports = (
  { markdownAST }, // AST tree
  pluginOptions,
) => {
  // further code goes here

  return markdownAST;
});

unist-util-visit is a function, that visits each node in the AST tree (which is what we are working with here) with a custom callback. In the callback, you can mutate data to your liking. The end goal of the plugin is to return the mutated markdownAST.

On a related note, here is a list of more, possibly useful AST tree utility packages.

Before going into the visit callback, we can test each node and only visit the nodes that meet our conditions. For that, you can pass an unist-util-is compatible test function to visit. We can use a regular expression to check for the existence of color hex codes in the node value.

For this tutorial, we are going to check on text nodes only, because any node is going to boil down to its text content in any event. Take `foobar`, which is an inline code tag in Markdown. In the AST, this results in a node with typeinlineTagand a single child in thenode.childrenproperty, which is atext node on its own.

The only exceptions to this rule are HTML nodes. But those are a whole different story, and we are not going to include them, because you can't use a regex to check (think <i style="color: #ffffff">text</i>, which has no readable color code when rendered in the browser).

const visit = require("unist-util-visit");

module.exports = ({ markdownAST }, pluginOptions) => {
  // thanks to https://stackoverflow.com/a/1636354/744230
  const hexCodeRegex = /#(?:[0-9a-fA-F]{3}){1,2}/gim;

  const test = (node, n) => node.type === "text" && hexCodeRegex.test(node.value);

  visit(markdownAST, test, (node, index, parent) => {
    // `node` can only be only a node with a color hex code inside
  }

  return markdownAST;
}

Before we start mutating the AST, let's add a few helper functions to build a new color AST node and generate the HTML for its value. Also, let's create plugin options so that the user can affect the output to some degree:

const visit = require("unist-util-visit");
const Color = require("color");

module.exports = ({ markdownAST }, {
  wrapperElement = "code",
  className = "color-highlight",
}) => {
  // thanks to https://stackoverflow.com/a/1636354/744230
  const hexCodeRegex = /#(?:[0-9a-fA-F]{3}){1,2}/gim;

  const test = (node, n) => node.type === "text" && hexCodeRegex.test(node.value);

  const buildNodeHtml = color =>
    `<${wrapperElement} class="${className}" style="background-color: ${color}; color: ${
      Color(color).isLight() ? "#000" : "#fff"
    }">${color}</${wrapperElement}>`;

  const buildColorNode = color => ({
    type: "html",
    children: [],
    value: buildNodeHtml(color),
  });

  visit(markdownAST, test, (node, index, parent) => {
    // `node` can only be only a node with a color hex code inside
  }

  return markdownAST;
}

In the next step, we are going to deal with the actual replacement of hex code colors. This part is mostly autonomous, vanilla JavaScript. The only thing to be aware of is the data structure of AST nodes, as we split up nodes at the hex color, create a new node for whatever came before the color code, add a newly constructed color node and finish up with whatever came after the hex color.

const visit = require("unist-util-visit");
const Color = require("color");

module.exports = ({ markdownAST }, {
  wrapperElement = "code",
  className = "color-highlight",
}) => {
  // Thanks to https://stackoverflow.com/a/1636354/744230
  const hexCodeRegex = /#(?:[0-9a-fA-F]{3}){1,2}/gim;

  const test = (node, n) => node.type === "text" && hexCodeRegex.test(node.value);

  // helper function, returns an HTML string with the color code wrapped in a tag
  // with background color
  const buildNodeHtml = color =>
    `<${wrapperElement} class="${className}" style="background-color: ${color}; color: ${
      Color(color).isLight() ? "#000" : "#fff"
    }">${color}</${wrapperElement}>`;

  // helper function, creates an object with a structure that resembles an AST node
  const buildColorNode = color => ({
    type: "html",
    children: [],
    value: buildNodeHtml(color),
  });

  // `node` can be only a node with a color hex code in the value because of `test`
  visit(markdownAST, test, (node, index, parent) => {
    // Split the node value at any occurrence of a color hex code. Each element of `parts`
    // is a substring of the original node value. No element can include color codes,
    // because those are the separator
    parts = text.split(hexCodeRegex);

    // Get all the occurrences of hex codes. Note, that the length of this array
    // is always `parts.length - 1` (unless the node value doesn't have a
    // hex color anywhere, which we omit with `test`)
    matches = text.match(hexCodeRegex);

   // This is an array with all the nodes, that, joined together, results in the
    // desired output. Essentially, we want to fill it alternating with
    // non-hex-code-content nodes and our newly created hex-code nodes alternating
    const replacementSiblings = [];

    // Loop through all the parts.
    for (let i = 0; i < parts.length; i++) {
      // Clone the node, because we want to use the same for the split content nodes
      // around the hex codes
      const nodeCopy = JSON.parse(JSON.stringify(node));

      // Assign the first node of the split
      nodeCopy.value = parts[i];

      // Clean up properties not required properties, which are not transferrable from
      // the original node, therefore polluting the object
      delete nodeCopy.position;
      delete nodeCopy.indent;

      // This statement is purely optional. The plugin would work just as well without.
      // It's just to keep the AST clean, e.g. if we split the string
      // "hex code at the end #123456" with `hexCodeRegex` as a separator, it results
      // in `["hex code at the end ", ""]`. But why fill the AST with useless nodes, that
      // do nothing?
      if (nodeCopy.value !== "") {
        replacementSiblings.push(nodeCopy);
      }

      // Make sure matches[i] exists. You could check with `if (matches[i])` just as well.
      // I am just utilizing the knowledge of `matches.length === parts.length - 1`
      if (i < matches.length) {
        replacementSiblings.push(buildColorNode(matches[i]));
      }
    }

    // `splice` removes the node at the current index (which is the node we are
    // visiting), and replace it with all the items we have assembled in
    // `replacementSiblings`
    parent.children.splice(index, 1, ...replacementSiblings);

    // This is a crucial line. The callback can override the internal index of `visit`,
    // thereby overriding what node it is going to visit next. Because we just
    // replaced 1 node with 1 + x (where x === replacementSiblings.length) nodes, if
    // we don't skip those newly injected sibling nodes, `visit` is going to run into an
    // infinite loop by visiting the sibling node that you just created, create new
    // siblings again, visit those, and so on.
    return index + replacementSiblings.length - 1;
  }

  // We return the mutated `markdownAST` here.
  return markdownAST;
}

And that's it. You've successfully written a plugin for Gatsby.js. You can inspect and study the actual code, which also considers HTML nodes, on GitHub.

Further tips

Helpful resources

I've found it incredibly helpful to study the source code of other plugins to learn and understand how to write my Gatsby.js plugin. It might not tell you explicitly what to do, and blind copy-pasting likely won't work, but it's the fastest and most effective of learning.

It also requires the ability to read and digest uncommented code by yourself, which can be easy or hard depending on how complex the code is.

Debugging and logging

I noticed it is quite tedious to debug your plugin because the build process takes up some time (about 20 seconds for each run in my case). And you are required to restart Gatsby.js each time you want to see the output because the building process runs only once upfront.

On that note, I noticed that for some reason, Gatsby cache silently swallows my console.logs on the second and subsequent runs, so I have to prepend each run with gatsby clean

$ gatsby clean && gatsby develop
# instead of just
$ gatsby develop

Why order of plugins matters (sometimes)

This tip is usually only important for transformer plugins, including the one we've created here.

Because such plugins change the content (or in this case the node tree of your content) and run in the same order as defined in the gatsby-config.js, one plugin may not receive the same markdownAST as another. For example, if you would like to transform hex codes in code snippets produced by gatsby-remark-vscode, you need to put gatsby-remark-color-highlight after.

That is because you don't want to suggest to gatsby-remark-vscode that your added wrapper element is user content and, therefore, should be escaped in the snippet. Instead, you want to wrap the content in the HTML produced by gatsby-remark-vscode.

On other occasions, plugins, which are a prerequisite for another plugin, need to be put to the front.

Conclusion

When you want to learn more about plugin development for Gatsby.js, the best way is to have a look at their plugin authoring guide and related docs, as well as on the source code of existing plugins, like:

• gatsby-remark-autolink-headers (transformer plugin)
• gatsby-source-wordpress (source plugin)