Tokenizer

Tokenizer — parse any string, slice or infinite buffer to any tokens.

Main features:

• High performance.
• No regexp.
• Provides simple API.
• By default, support integer and float numbers.
• Support quoted string or other "framed" strings.
• Support token injection in quoted or "framed" strings.
• Support unicode.
• Customization of tokens.
• Autodetect white space symbols.
• Parse any data syntax (xml, json, yaml), any programming language.
• Single pass through the data.
• Parsing infinite data.

Use cases:

• Parsing html, xml, json, yaml and other text formats.
• Parsing any programming language.
• Parsing string templates.
• Parsing formulas.

Example:

// define custom tokens keys
const ( 
	TEquality = 1
	TDot      = 2
	TMath     = 3
)

// configure tokenizer
parser := tokenizer.New()
parser.DefineTokens(TEquality, []string{"<", "<=", "==", ">=", ">", "!="})
parser.DefineTokens(TDot, []string{"."})
parser.DefineTokens(TMath, []string{"+", "-", "/", "*", "%"})
parser.DefineStringToken(`"`, `"`).SetEscapeSymbol(tokenizer.BackSlash)

// parse data
stream := parser.ParseString(`user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`)
defer stream.Close()

for stream.Valid() {
	if stream.CurrentToken().Is(tokenizer.TokenKeyword) {
		field := stream.CurrentToken().ValueString()
		// ... 
	}
	stream.Next()
}

parsing details:

string:  user_id  =  119  and  modified  >  "2020-01-01 00:00:00"  or  amount  >=  122.34
tokens: |user_id| =| 119| and| modified| >| "2020-01-01 00:00:00"| or| amount| >=| 122.34|
        |   0   | 1|  2 |  3 |    4    | 5|            6         | 7 |    8  | 9 |    10 |

0:  {key: TokenKeyword, value: "user_id"}                token.Value()          == "user_id"
1:  {key: TEquality, value: "="}                         token.Value()          == "="
2:  {key: TokenInteger, value: "119"}                    token.ValueInt()       == 119
3:  {key: TokenKeyword, value: "and"}                    token.Value()          == "and"
4:  {key: TokenKeyword, value: "modified"}               token.Value()          == "modified"
5:  {key: TEquality, value: ">"}                         token.Value()          == ">"
6:  {key: TokenString, value: "\"2020-01-01 00:00:00\""} token.ValueUnescaped() == "2020-01-01 00:00:00"
7:  {key: TokenKeyword, value: "or"}                     token.Value()          == "and"
8:  {key: TokenKeyword, value: "amount"}                 token.Value()          == "amount"
9:  {key: TEquality, value: ">="}                        token.Value()          == ">="
10: {key: TokenFloat, value: "122.34"}                   token.ValueFloat()     == 122.34

More examples:

• JSON parser

Embedded tokens

• tokenizer.TokenUnknown — unspecified token key.
• tokenizer.TokenKeyword — keyword, any combination of letters, including unicode letters.
• tokenizer.TokenInteger — integer value
• tokenizer.TokenFloat — float/double value
• tokenizer.TokenString — quoted string
• tokenizer.TokenStringFragment — fragment framed (quoted) string

Unknown token — tokenizer.TokenUnknown

A token marks as TokenUnknown if the parser detects an unknown token:

parser.ParseString(`one!`)

{
    {
        Key: tokenizer.TokenKeyword
        Value: "One"
    },
    {
        Key: tokenizer.TokenUnknown
        Value: "!"
    }
}

By default, TokenUnknown tokens are added to the stream. To exclude them from the stream, use the tokenizer.StopOnUndefinedToken() method

{
    {
        Key: tokenizer.TokenKeyword
        Value: "one"
    }
}

Please note that if the tokenizer.StopOnUndefinedToken setting is enabled, then the string may not be fully parsed. To find out that the string was not fully parsed, check the length of the parsed string stream.GetParsedLength() and the length of the original string.

Keywords

Any word that is not a custom token is stored in a single token as tokenizer.TokenKeyword.

The word can contain unicode characters, numbers (see tokenizer.AllowNumbersInKeyword ()) and underscore (see tokenizer.AllowKeywordUnderscore ()).

parser.ParseString(`one two четыре`)

tokens: {
    {
        Key: tokenizer.TokenKeyword
        Value: "one"
    },
    {
        Key: tokenizer.TokenKeyword
        Value: "two"
    },
    {
        Key: tokenizer.TokenKeyword
        Value: "четыре"
    }
}

Integer number

Any integer is stored in one token tokenizer.Token Integer.

parser.ParseString(`223 999`)

tokens: {
    {
        Key: tokenizer.TokenInteger
        Value: "223"
    },
    {
        Key: tokenizer.TokenInteger
        Value: "999"
    },
}

To get int64 from the token value use stream.GetInt():

stream := tokenizer.ParseString("123")
fmt.Print("Token is %d", stream.GetInt())  // Token is 123

Float number

Any float number is stored in one token tokenizer.TokenFloat. Float number may

• have point, for example 1.2
• have exponent, for example 1e6
• have lower e or upper E letter in the exponent, for example 1E6, 1e6
• have sign in the exponent, for example 1e-6, 1e6, 1e+6

tokenizer.ParseString(`1.3e-8`):
{
    {
        Key: tokenizer.TokenFloat
        Value: "1.3e-8"
    },
}

To get float64 from the token value use token.GetFloat():

stream := tokenizer.ParseString("1.3e2")
fmt.Print("Token is %d", stream.CurrentToken().GetFloat())  // Token is 130

Framed string

Strings that are framed with tokens are called framed strings. An obvious example is quoted a string like " one two ". Where quotes are edge tokens.

You can create and customize framed string through tokenizer.AddQuote():

const TokenDoubleQuotedString = 10
tokenizer.DefineStringToken(TokenDoubleQuotedString, `"`, `"`).SetEscapeSymbol('\\')
stream := tokenizer.ParseString(`"two \"three"`)

{
    {
        Key: tokenizer.TokenString
        Value: "\"two \\"three\""
    },
}

To get a framed string without edge tokens and special characters, use the stream.ValueUnescape() method:

v := stream.CurrentToken().ValueUnescape() // result: two "three

The method token.StringKey() will be return token string key defined in the DefineStringToken:

stream.CurrentToken().StringKey() == TokenDoubleQuotedString // true

Injection in framed string

Strings can contain expression substitutions that can be parsed into tokens. For example "one {{two}} three". Fragments of strings before, between and after substitutions will be stored in tokens as tokenizer.TokenStringFragment.

const (
    TokenOpenInjection = 1
    TokenCloseInjection = 2
    TokenQuotedString = 3
)

parser := tokenizer.New()
parser.DefineTokens(TokenOpenInjection, []string{"{{"})
parser.DefineTokens(TokenCloseInjection, []string{"}}"})
parser.DefineStringToken(TokenQuotedString, `"`, `"`).AddInjection(TokenOpenInjection, TokenCloseInjection)

parser.ParseString(`"one {{ two }} three"`)

Tokens:

{
    {
        Key: tokenizer.TokenStringFragment,
        Value: "one"
    },
    {
        Key: TokenOpenInjection,
        Value: "{{"
    },
    {
        Key: tokenizer.TokenKeyword,
        Value: "two"
    },
    {
        Key: TokenCloseInjection,
        Value: "}}"
    },
    {
        Key: tokenizer.TokenStringFragment,
        Value: "three"
    },
}

Use cases:

• parse templates
• parse placeholders

User defined tokens

The new token can be defined via the DefineTokens method:

const (
    TokenCurlyOpen    = 1
    TokenCurlyClose   = 2
    TokenSquareOpen   = 3
    TokenSquareClose  = 4
    TokenColon        = 5
    TokenComma        = 6
	TokenDoubleQuoted = 7
)

// json parser
parser := tokenizer.New()
parser.
		DefineTokens(TokenCurlyOpen, []string{"{"}).
		DefineTokens(TokenCurlyClose, []string{"}"}).
		DefineTokens(TokenSquareOpen, []string{"["}).
		DefineTokens(TokenSquareClose, []string{"]"}).
		DefineTokens(TokenColon, []string{":"}).
		DefineTokens(TokenComma, []string{","}).
        DefineStringToken(TokenDoubleQuoted, `"`, `"`).SetSpecialSymbols(tokenizer.DefaultStringEscapes)

stream := parser.ParseString(`{"key": [1]}`)

Parsing string

There is two ways to parse string/slice

• parser.ParseString(str)
• parser.ParseBytes(slice)

Parse buffer

The package allows to parse an endless stream of data into tokens. For parsing, you need to pass io.Reader, from which data will be read (chunk-by-chunk):

fp, err := os.Open("data.json") // huge JSON file
// check fs, configure tokenizer ...

stream := parser.ParseStream(fp, 4096).SetHistorySize(10)
for stream.IsValid() {
	// ...
	stream.Next()
}

Known issues

• zero-byte \0 will be ignored in the source string.

Benchmark

Parse string/bytes

pkg: tokenizer
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkParseBytes
    stream_test.go:251: Speed: 70 bytes string with 19.689µs: 3555284 byte/sec
    stream_test.go:251: Speed: 7000 bytes string with 848.163µs: 8253130 byte/sec
    stream_test.go:251: Speed: 700000 bytes string with 75.685945ms: 9248744 byte/sec
    stream_test.go:251: Speed: 11093670 bytes string with 1.16611538s: 9513355 byte/sec
BenchmarkParseBytes-8   	  158481	      7358 ns/op

Parse infinite stream

pkg: tokenizer
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkParseInfStream
    stream_test.go:226: Speed: 70 bytes at 33.826µs: 2069414 byte/sec
    stream_test.go:226: Speed: 7000 bytes at 627.357µs: 11157921 byte/sec
    stream_test.go:226: Speed: 700000 bytes at 27.675799ms: 25292856 byte/sec
    stream_test.go:226: Speed: 30316440 bytes at 1.18061702s: 25678471 byte/sec
BenchmarkParseInfStream-8   	  433092	      2726 ns/op
PASS