Building Custom JS String Parsers: Full Tutorial

Learn how to build custom string parsers in JavaScript from scratch. Tokenize input, handle grammar rules, and turn raw text into structured data without external libraries.

RuneHub Team
RuneHub Team
9 min read

A parser turns raw text into structured data that your code can work with. Every programming language, every template engine, every query language, every config file format runs through a parser.

Building one yourself demystifies how compilers, linters, formatters, and syntax highlighters work. You do not need a library for small, custom grammars. A hand-written parser is often clearer and faster than configuring a parser generator for a tiny language.

What You Will Build

This tutorial builds a parser for a tiny math expression language that handles addition, subtraction, multiplication, division, parentheses, and numbers. Here is what valid input looks like:

texttext
3 + 4 * 2
(1 + 2) * 3
10 / (5 - 3)

The parser will turn "3 + 4 * 2" into a tree structure that your code can evaluate step by step.

Step 1: The Tokenizer

The tokenizer (or lexer) scans the raw string character by character and produces a list of tokens. Each token has a type and a value:

javascriptjavascript
function tokenize(input) {
  const tokens = [];
  let pos = 0;
 
  while (pos < input.length) {
    const char = input[pos];
 
    // Skip whitespace
    if (char === " " || char === "\t" || char === "\n") {
      pos += 1;
      continue;
    }
 
    // Numbers (integers and decimals)
    if (char >= "0" && char <= "9") {
      let num = "";
      while (pos < input.length && input[pos] >= "0" && input[pos] <= "9") {
        num += input[pos];
        pos += 1;
      }
      // Handle decimal part
      if (input[pos] === ".") {
        num += ".";
        pos += 1;
        while (pos < input.length && input[pos] >= "0" && input[pos] <= "9") {
          num += input[pos];
          pos += 1;
        }
      }
      tokens.push({ type: "number", value: Number(num) });
      continue;
    }
 
    // Operators and parentheses (single-character tokens)
    const singles = {
      "+": "plus",
      "-": "minus",
      "*": "star",
      "/": "slash",
      "(": "lparen",
      ")": "rparen"
    };
 
    if (char in singles) {
      tokens.push({ type: singles[char], value: char });
      pos += 1;
      continue;
    }
 
    throw new Error(`Unexpected character '${char}' at position ${pos}`);
  }
 
  tokens.push({ type: "eof", value: null });
  return tokens;
}

Test it:

javascriptjavascript
console.log(tokenize("3 + 4 * 2"));
texttext
[
  { type: "number", value: 3 },
  { type: "plus", value: "+" },
  { type: "number", value: 4 },
  { type: "star", value: "*" },
  { type: "number", value: 2 },
  { type: "eof", value: null }
]

The raw string "3 + 4 * 2" is now a flat list of typed tokens. The tokenizer has done its job: whitespace is gone, multi-character numbers are grouped, and every piece has a clear type.

Step 2: The Parser

The parser consumes the token list and builds a tree. It uses recursive descent: each grammar rule becomes a function, and rules call each other to handle precedence.

Our expression grammar:

plaintextplaintext
expression  -> term (("+" | "-") term)*
term        -> factor (("*" | "/") factor)*
factor      -> number | "(" expression ")"

In plain English: an expression is a term, optionally followed by + or - and another term. A term is a factor, optionally followed by * or / and another factor. A factor is a number or a parenthesized expression.

Recursive descent parser grammar rules calling each other

parseExpression calls parseTerm. parseTerm calls parseFactor. parseFactor handles numbers or recurses back into parseExpression when it sees parentheses. This mutual recursion is what handles operator precedence naturally.

Here is the implementation:

javascriptjavascript
function parse(tokens) {
  let pos = 0;
 
  function current() {
    return tokens[pos];
  }
 
  function consume(expectedType) {
    const token = current();
    if (token.type !== expectedType) {
      throw new Error(
        `Expected ${expectedType} but got ${token.type} at position ${pos}`
      );
    }
    pos += 1;
    return token;
  }
 
  // factor -> number | "(" expression ")"
  function parseFactor() {
    const token = current();
 
    if (token.type === "number") {
      consume("number");
      return { type: "number", value: token.value };
    }
 
    if (token.type === "lparen") {
      consume("lparen");
      const node = parseExpression();
      consume("rparen");
      return node;
    }
 
    throw new Error(`Unexpected token '${token.type}' at position ${pos}`);
  }
 
  // term -> factor (("*" | "/") factor)*
  function parseTerm() {
    let left = parseFactor();
 
    while (current().type === "star" || current().type === "slash") {
      const op = current().type;
      pos += 1;
      const right = parseFactor();
      left = { type: "binary", operator: op, left, right };
    }
 
    return left;
  }
 
  // expression -> term (("+" | "-") term)*
  function parseExpression() {
    let left = parseTerm();
 
    while (current().type === "plus" || current().type === "minus") {
      const op = current().type;
      pos += 1;
      const right = parseTerm();
      left = { type: "binary", operator: op, left, right };
    }
 
    return left;
  }
 
  const ast = parseExpression();
  consume("eof");
  return ast;
}

Step 3: Seeing the Tree

Parse an expression and inspect the tree:

javascriptjavascript
const tokens = tokenize("3 + 4 * 2");
const ast = parse(tokens);
console.log(JSON.stringify(ast, null, 2));
jsonjson
{
  "type": "binary",
  "operator": "plus",
  "left": { "type": "number", "value": 3 },
  "right": {
    "type": "binary",
    "operator": "star",
    "left": { "type": "number", "value": 4 },
    "right": { "type": "number", "value": 2 }
  }
}

The tree respects operator precedence. Multiplication (4 * 2) is nested deeper than addition. The evaluator will compute 4 * 2 = 8 first, then 3 + 8 = 11.

Step 4: Evaluating the Tree

Now walk the tree and compute the result:

javascriptjavascript
function evaluate(node) {
  if (node.type === "number") {
    return node.value;
  }
 
  if (node.type === "binary") {
    const left = evaluate(node.left);
    const right = evaluate(node.right);
 
    switch (node.operator) {
      case "plus":  return left + right;
      case "minus": return left - right;
      case "star":  return left * right;
      case "slash": return left / right;
    }
  }
 
  throw new Error(`Unknown node type: ${node.type}`);
}

Full pipeline:

javascriptjavascript
function calculate(input) {
  const tokens = tokenize(input);
  const ast = parse(tokens);
  return evaluate(ast);
}
 
console.log(calculate("3 + 4 * 2"));       // 11
console.log(calculate("(1 + 2) * 3"));     // 9
console.log(calculate("10 / (5 - 3)"));    // 5

Adding Variables and Assignment

Extend the parser to support variables and assignment. Add a token type for identifiers:

javascriptjavascript
// Inside tokenize(), after the singles check:
if ((char >= "a" && char <= "z") || (char >= "A" && char <= "Z")) {
  let name = "";
  while (
    pos < input.length &&
    ((input[pos] >= "a" && input[pos] <= "z") ||
     (input[pos] >= "A" && input[pos] <= "Z"))
  ) {
    name += input[pos];
    pos += 1;
  }
  tokens.push({ type: "identifier", value: name });
  continue;
}

Add assignment to the grammar:

javascriptjavascript
// In parseExpression, check for assignment before parsing:
function parseExpression() {
  // Check for assignment: identifier "=" expression
  if (current().type === "identifier") {
    const name = current().value;
    pos += 1;
    if (current().type === "equals") {
      pos += 1;
      return { type: "assign", name, value: parseExpression() };
    }
    // Not an assignment, backtrack
    pos -= 1;
  }
 
  let left = parseTerm();
  while (current().type === "plus" || current().type === "minus") {
    const op = current().type;
    pos += 1;
    const right = parseTerm();
    left = { type: "binary", operator: op, left, right };
  }
  return left;
}

Extend the evaluator with a variable environment:

javascriptjavascript
function createEvaluator() {
  const env = {};
 
  function evaluate(node) {
    if (node.type === "number") return node.value;
 
    if (node.type === "identifier") {
      if (!(node.value in env)) {
        throw new Error(`Undefined variable: ${node.value}`);
      }
      return env[node.value];
    }
 
    if (node.type === "assign") {
      const val = evaluate(node.value);
      env[node.name] = val;
      return val;
    }
 
    if (node.type === "binary") {
      const left = evaluate(node.left);
      const right = evaluate(node.right);
      switch (node.operator) {
        case "plus":  return left + right;
        case "minus": return left - right;
        case "star":  return left * right;
        case "slash": return left / right;
      }
    }
 
    throw new Error(`Unknown node type: ${node.type}`);
  }
 
  return { evaluate };
}
 
const evaluator = createEvaluator();
console.log(evaluator.evaluate(parse(tokenize("x = 10"))));     // 10
console.log(evaluator.evaluate(parse(tokenize("y = x + 5"))));  // 15
console.log(evaluator.evaluate(parse(tokenize("y * 2"))));      // 30

Variables persist in the environment across evaluations. The design pattern here is the strategy pattern: each AST node type has its own evaluation strategy, and the evaluator dispatches based on node.type.

Parser Architecture Overview

The full flow from text to result:

Parser pipeline: text to tokens to tree to result

Each stage is independent. You can swap the tokenizer, add new grammar rules to the parser, or change the evaluator to produce different output (code generation, formatting, linting) without touching the other stages.

Common Pitfalls

Left recursion. A rule like expression -> expression "+" term causes infinite recursion. Always restructure to expression -> term ("+" term)* so the parser consumes a token before recursing.

Forgetting to skip whitespace. The tokenizer must handle spaces, tabs, and newlines. If it does not, every space becomes an error.

No error recovery. The parser above stops at the first error. Production parsers try to recover and report multiple errors. For small languages, a single clear error with position is enough.

For more on tree structures and traversal, see the JavaScript AST guide.

Rune AI

Rune AI

Key Insights

  • A tokenizer (lexer) breaks raw text into a list of typed tokens.
  • A parser consumes tokens and builds a tree structure according to grammar rules.
  • Recursive descent parsing is the most intuitive approach for hand-written parsers.
  • Each grammar rule becomes a function. Rules call each other recursively.
  • Error handling in parsers should produce clear messages with position information.
RunePowered by Rune AI

Frequently Asked Questions

When should I build a custom parser instead of using a library?

Build a custom parser when your grammar is small and specific (a custom query language, a config file format, a template engine, or a math expression evaluator). Use a parser generator like PEG.js or ANTLR for complex grammars.

What is the difference between tokenizing and parsing?

Tokenizing (lexing) breaks raw text into tokens (words, numbers, symbols). Parsing takes the token stream and builds a structure (usually a tree) according to grammar rules. Tokenizing answers 'what are the pieces?' Parsing answers 'how do the pieces relate?'

Is this related to ASTs?

Yes. Most parsers produce an Abstract Syntax Tree (AST). See the AST guide for a deeper dive into tree structures and traversal.

Conclusion

Building a parser teaches you how programming languages work under the hood. Tokenize first to break text into pieces. Parse next to build structure from those pieces. The recursive descent approach is the most readable way to write a parser by hand.A parser has two stages: tokenizing and parsing. The tokenizer turns raw text into a flat list of typed tokens. The parser turns that list into a tree by following grammar rules, one function per rule. The recursive descent approach is the most readable way to write a parser by hand. Once you have the tree, you can do anything with it: evaluate expressions, generate code, format text, check for errors, or highlight syntax. The parser is the doorway. The tree is the tool.

Previous
Vanilla JS State Management for Advanced Apps
8 min read
Next
Rate Limiting in JavaScript: Complete Tutorial
7 min read

More in this topic

Using Reflect and Proxy Together in JavaScript

Proxy traps intercept operations, but Reflect is what makes them behave correctly. Learn why Reflect belongs inside every trap and what breaks when you skip it.

Top JS Array Methods Interview Questions to Know

The array method questions that come up most often in JavaScript interviews, answered directly with short examples: map vs forEach, mutating vs non-mutating methods, reduce, and more.

JavaScript Reflect API: Advanced Architecture

The Reflect object exposes JavaScript's own internal operations as plain functions. Learn every Reflect method, what it returns, and why it exists as its own API.