method SimpleLexer.prototype.tokenize
SimpleLexer.prototype.tokenize(
input: string,
offset?: number,
): Generator<SimpleLexerToken<_ExtractType<D["tokens"]>>, void>

Tokenizes the input starting at the given offset.

Parameters

input: string

The input string.

optional
offset: number = 0

The offset to start parsing the input string. Default: 0

Return Type

Generator<SimpleLexerToken<_ExtractType<D["tokens"]>>, void>

Usage

import { SimpleLexer } from ".";