class SimpleLexer

A very basic lexer that simply just takes string and RegExp patterns and spits out the associated tokens with no advanced bells and whistles.

Examples

Example 1

const lexer = new SimpleLexer({
    tokens: {
        inlineWhitespace: /[ \t]+/u,
        lineEnding: /\r\n?|[\n\f\v\u2028\u2029]/u,
        word: /[^ \t\r\n\f\v\u2028\u2029]+/u,
    }
});

const tokens = lexer.tokenize(input);

for (const token of tokens) {
    // ...
}

Constructors

new
SimpleLexer(definition: D)

Type Parameters

Methods

match(
input: string,
offset: number,
): SimpleLexerToken<_ExtractType<D["tokens"]>> | null

Matches a token at the given offset in input.

Returns a copy of the internal match pattern.

tokenize(
input: string,
offset?: number,
): Generator<SimpleLexerToken<_ExtractType<D["tokens"]>>, void>

Tokenizes the input starting at the given offset.

Usage

import { SimpleLexer } from ".";