smalto/token

Token types for syntax-highlighted source code.

Each token wraps a string value representing the matched source text. Well-known semantic variants cover common cases, while Custom handles language-specific token categories.

Types

AnsiToken

</>

A token type identifier without the associated source text.

Used as a key in AnsiTheme to map token categories to styling functions.

pub type AnsiToken {
  AnsiKeyword
  AnsiString
  AnsiNumber
  AnsiComment
  AnsiFunction
  AnsiOperator
  AnsiPunctuation
  AnsiType
  AnsiModule
  AnsiVariable
  AnsiConstant
  AnsiBuiltin
  AnsiTag
  AnsiAttribute
  AnsiSelector
  AnsiProperty
  AnsiRegex
  AnsiWhitespace
  AnsiOther
  AnsiCustom(name: String)
}

Constructors

```
AnsiKeyword
```
```
AnsiString
```
```
AnsiNumber
```
```
AnsiComment
```
```
AnsiFunction
```
```
AnsiOperator
```
```
AnsiPunctuation
```
```
AnsiType
```
```
AnsiModule
```
```
AnsiVariable
```
```
AnsiConstant
```
```
AnsiBuiltin
```
```
AnsiTag
```
```
AnsiAttribute
```
```
AnsiSelector
```
```
AnsiProperty
```
```
AnsiRegex
```
```
AnsiWhitespace
```
```
AnsiOther
```
```
AnsiCustom(name: String)
```

Token

</>

A single token produced by the syntax highlighting engine.

pub type Token {
  Keyword(String)
  String(String)
  Number(String)
  Comment(String)
  Function(String)
  Operator(String)
  Punctuation(String)
  Type(String)
  Module(String)
  Variable(String)
  Constant(String)
  Builtin(String)
  Tag(String)
  Attribute(String)
  Selector(String)
  Property(String)
  Regex(String)
  Whitespace(String)
  Other(String)
  Custom(name: String, value: String)
}

Constructors

```
Keyword(String)
```
```
String(String)
```
```
Number(String)
```
```
Comment(String)
```
```
Function(String)
```
```
Operator(String)
```
```
Punctuation(String)
```
```
Type(String)
```
```
Module(String)
```
```
Variable(String)
```
```
Constant(String)
```
```
Builtin(String)
```
```
Tag(String)
```
```
Attribute(String)
```
```
Selector(String)
```
```
Property(String)
```
```
Regex(String)
```
```
Whitespace(String)
```
```
Other(String)
```
```
Custom(name: String, value: String)
```

Values

name

</>

pub fn name(token: Token) -> String

Returns the token category name as a lowercase string.

For built-in variants, this is the variant name in lowercase (e.g., Keyword(_) returns "keyword"). For Custom, this returns the user-provided name.

to_ansi_token

</>

pub fn to_ansi_token(token: Token) -> AnsiToken

Convert a token to its corresponding AnsiToken key.

value

</>

pub fn value(token: Token) -> String

Extracts the source text content from a token.