Writing a jq engine in TypeScript

Prettify needed jq in the browser. Not jq-flavored filtering, not a subset with the good parts: actual jq, the query language people already know, running on JSON that never leaves the tab.

There were two ways to get it, and both were wrong for this.

Why not the obvious options

Compile the real thing to WebAssembly. jq-web does exactly this, and it works. It also ships around a megabyte of wasm before you have queried anything, and that megabyte has to download, compile, and instantiate before the first keystroke does anything. For a tool whose entire pitch is "paste JSON, see results now", a megabyte of startup is the product getting worse.

Shell out to the binary. node-jq does this. It is the right answer on a server and no answer at all in a browser tab, which is where Prettify runs.

So: write it. The engine is about 3,600 lines of TypeScript across six files, plus 1,300 lines of tests: 122 builtins and 220 tests. Most of that is the boring part. The interesting part is one idea.

The one idea: jq is a stream language

The thing that makes jq feel strange when you come from JavaScript is that every expression produces a stream of values, not a value.

Not "sometimes an array". A stream. .foo produces exactly one output. .[] produces one output per element. empty produces zero. 1, 2, 3 produces three.

This is why jq can do things that read like magic:

$ echo '{"a": [1,2,3]}' | jq '.a[] | . * 2'
2
4
6

There is no map there. .a[] emits three values, and everything downstream of the pipe runs once per value. The pipe is not function composition. It is "for each thing on the left, run the right".

Once you see that, most of jq stops being special cases:

• , is stream concatenation
• empty is the empty stream
• select(f) emits its input or emits nothing
• first(f) takes one value off a stream and stops

Every one of those is trivial if your evaluator produces streams. Every one is a nightmare if it produces values and you are trying to fake streams with arrays.

Generators are that idea, in the language

JavaScript already has a stream primitive that is lazy, composable, and built into the syntax. It is the generator.

So the evaluator is one recursive generator function:

export function* evaluate(
  node: AstNode,
  input: JsonValue,
  env: Environment
): Generator<JsonValue> {
  switch (node.type) {
    case 'identity':
      yield input
      break
    // ...
  }
}

Generator<JsonValue> is the whole design. Every AST node evaluates to a stream of JSON values. Nothing accumulates unless something asks it to.

Here is what that buys you.

Pipe is four lines

case 'pipe': {
  for (const leftResult of evaluate(node.left, input, env)) {
    yield* evaluate(node.right, leftResult, env)
  }
  break
}

Read it in English: for every value the left side produces, run the right side with that value as input, and pass along everything it produces.

That is jq's pipe. Completely. The nested iteration that makes .users[] | .name | ascii_downcase work across three levels falls out of yield* and recursion.

I did not have to write a scheduler, a trampoline, or a "collect then flatten" pass. The semantics of the language and the semantics of generators are the same shape, so the implementation is a transcription.

Comma is two lines

case 'comma': {
  yield* evaluate(node.left, input, env)
  yield* evaluate(node.right, input, env)
  break
}

Both sides get the same input, and their outputs concatenate. That is why {name: .name, id: .id} works, and why .a, .b gives you two results rather than an array of two.

The optional operator is a try/catch that yields nothing

jq's ? suppresses errors. .foo? on a number does not blow up; it produces no output at all.

case 'optional': {
  try {
    for (const val of evaluate(node.expr, input, env)) {
      yield val
    }
  } catch {
    // Suppress errors, yield nothing
  }
  break
}

An expression that throws becomes an expression that contributed nothing to the stream. In a value-based evaluator you would need a sentinel, an option type, or an out-of-band error channel. In a stream-based one, "no output" is already a thing you can express.

The parser is the boring part, and that is good

Tokenizer, then precedence-climbing parser, then evaluate. Nothing clever.

The tokenizer is about 550 lines and mostly exists to handle the pieces of jq that are not JSON: $variables, format strings like @base64 and @csv, .. for recursive descent, and the keyword set (if, then, elif, reduce, foreach, try, catch, as).

The parser is a standard precedence climb. One function answers "how tightly does this operator bind":

function getInfixPrecedence(token: Token): number | null

and the rest is recursion. | binds loosest, then ,, then //, then comparisons, then arithmetic. If you have written a calculator parser you have written this parser.

I want to be clear that this is the right kind of boring. The interesting decision was the evaluator. Spending novelty on the parser would have bought nothing.

122 builtins is just a map

export const builtins: Record<string, BuiltinFn> = {}

length, keys, map, select, group_by, to_entries, sub, gsub, test, splits, paths, getpath, setpath, del, limit, until, while, @csv, @tsv, @base64, and about a hundred more.

There is no trick here. It is a long afternoon per twenty builtins, and the tests are the actual work. Most of them are also generators, because most of them can produce more than one output.

The ones worth calling out are path() and del(), which do not evaluate an expression for its value but for the path it walked to get there. Those need a second interpreter over the same AST that yields paths instead of values. It is the one place the design needed a real second pass.

The wrinkle nobody warns you about: numbers

JSON.parse corrupts JSON. Quietly, and more often than you would like.

JSON.parse('{"id": 9007199254740993}').id
// 9007199254740992   ← off by one

JSON.parse('{"id": 1071004833782874112}').id
// 1071004833782874100  ← last two digits invented

Prettify preserves those numbers exactly, which means the values flowing through the jq engine are not always JavaScript numbers. Some of them are wrappers holding the original digit string.

That forces a policy, because you cannot both preserve a number exactly and do arithmetic on it in a language whose only number type is a float. The engine splits it:

• Pass-through preserves. .id on a 19-digit integer emits the exact digits. Identity, field access, and indexing never touch the value.
• Computing coerces. Arithmetic, aggregation, and comparison flatten to a float, because they have to.

And then the part I would argue matters most: the engine reports when it happened.

const didCoerce = watchLosslessCoercions()
// ... run the query ...
return { result, coercedLossless: didCoerce() }

If your query flattened a preserved number, the panel says so. A wrong answer you are told about is a different thing from a wrong answer you are not, and JSON.parse never tells you.

I wrote about that problem on its own, because it is not really a jq problem. It is everyone's problem.

What it does not do

Being honest about the edges, since "I wrote jq" invites the assumption of completeness:

• No def. User-defined functions are not implemented. The tokenizer does not have the keyword. This is the biggest gap.
• No input / inputs. They throw. Both assume a stream of documents from stdin, which does not exist in a browser tab.
• No label / break. Throws. Rarely used, and the generator design makes non-local exit genuinely awkward.

Everything else in normal use works: paths, slices, iteration, all the operators, string interpolation, reduce, foreach, try/catch, as bindings, the format strings, and the 122 builtins.

An aside on checking your own claims

An earlier version of this section was longer. It also listed slices, foreach, and $variables inside string interpolation as missing, because they were.

I found all three the same way: by running every example in this post and its companion tutorial through the engine rather than trusting that they worked. That is a cheap habit and it caught three things I would otherwise have published as true.

The fixes were smaller than the bugs looked. foreach had a token, a keyword-table entry, and nothing else: no AST node, no parser case, no evaluator case, so the token was produced and never consumed. Interpolation was a one-line mismatch, where the main tokenizer emitted $x and the interpolation-only tokenizer emitted x, against an environment keyed on the first.

Slicing was the interesting one, because implementing it broke something else. del and path walk the AST through a second interpreter that yields paths instead of values, and its fall-through branch returns the parent path for any node it does not recognise. While slices did not parse, that branch was unreachable. Once they did, del(.a[1:3]) resolved to the path ["a"] and deleted the entire array rather than two elements from it.

The fix was to give a slice a real path representation. jq already has one: a path component can be an object, {"start": 1, "end": 3}, with null for an open bound. Teaching getpath, setpath, and deletePath to recognise that shape makes del(.a[1:3]) splice out the range, and setpath assign into it, exactly as jq does.

The lesson is not "write more tests". It is that a fall-through branch which quietly returns something plausible is a trap waiting for the next feature. It had been correct for exactly as long as nothing reached it, and it failed the moment something did, in the most expensive direction available: deleting data rather than refusing.

The part worth stealing

If you take one thing from this, it is not "write your own jq". It is this:

When a language's core semantics match a primitive your host language already has, the implementation collapses.

jq's pipe is four lines because generators are streams. Had I started with "evaluate returns a value" and tried to bolt streaming on afterward, every one of those four-line cases would have been forty lines of array plumbing, and ? and empty and first would each have been a special case with its own bug.

The hard part was not writing the evaluator. It was noticing, before writing anything, that jq is a stream language and JavaScript already ships streams.

---

The engine powers the jq panel in Prettify, which is a Pro feature. If you want to use jq rather than build one, start with Learn jq by example, which covers the same stream idea from the other side, or open the free JSON editor. Either way, nothing you paste leaves your device.