80 lines
2.7 KiB
Markdown
80 lines
2.7 KiB
Markdown
Remap Tokens During Parsing
|
|
===========================
|
|
|
|
{{#include ../links.md}}
|
|
|
|
[`Engine::on_parse_token`]: https://docs.rs/rhai/{{version}}/rhai/struct.Engine.html#method.on_parse_token
|
|
[`Token`]: https://docs.rs/rhai/{{version}}/rhai/enum.Token.html
|
|
|
|
|
|
The Rhai [`Engine`] first parses a script into a stream of _tokens_.
|
|
|
|
Tokens have the type [`Token`] which is only exported under [`internals`].
|
|
|
|
The function [`Engine::on_parse_token`], available only under [`internals`], allows registration of a
|
|
_mapper function_ that converts (remaps) a [`Token`] into another.
|
|
|
|
|
|
```admonish tip.small "Hot Tips: Use as safety checks"
|
|
|
|
Since it is called for _every_ token parsed from the script, this token mapper function
|
|
can also be used to implement _safety checks_ against, say, stack-overflow or out-of-memory
|
|
situations during parsing.
|
|
|
|
See [here][memory] for more details.
|
|
```
|
|
|
|
|
|
Function Signature
|
|
------------------
|
|
|
|
```admonish tip.side "Tip: Raising errors"
|
|
|
|
Raise a parse error by returning [`Token::LexError`](https://docs.rs/rhai/{{version}}/rhai/enum.Token.html#variant.LexError)
|
|
as the mapped token.
|
|
```
|
|
|
|
The function signature passed to [`Engine::on_parse_token`] takes the following form.
|
|
|
|
> ```rust
|
|
> Fn(token: Token, pos: Position, state: &TokenizeState) -> Token
|
|
> ```
|
|
|
|
where:
|
|
|
|
| Parameter | Type | Description |
|
|
| --------- | :---------------------------------------------------------------------------------: | -------------------------------- |
|
|
| `token` | [`Token`] | the next symbol parsed |
|
|
| `pos` | `Position` | location of the [token][`Token`] |
|
|
| `state` | [`&TokenizeState`](https://docs.rs/rhai/{{version}}/rhai/struct.TokenizeState.html) | current state of the tokenizer |
|
|
|
|
|
|
Example
|
|
-------
|
|
|
|
```rust
|
|
use rhai::{Engine, FLOAT, Token};
|
|
|
|
let mut engine = Engine::new();
|
|
|
|
// Register a token mapper function.
|
|
engine.on_parse_token(|token, pos, state| {
|
|
match token {
|
|
// Change 'begin' ... 'end' to '{' ... '}'
|
|
Token::Identifier(s) if &s == "begin" => Token::LeftBrace,
|
|
Token::Identifier(s) if &s == "end" => Token::RightBrace,
|
|
|
|
// Change all integer literals to floating-point
|
|
Token::IntegerConstant(n) => Token::FloatConstant((n as FLOAT).into()),
|
|
|
|
// Disallow '()'
|
|
Token::Unit => Token::LexError(
|
|
LexError::ImproperSymbol("()".to_string(), "".to_string()).into()
|
|
),
|
|
|
|
// Pass through all other tokens unchanged
|
|
_ => token
|
|
}
|
|
});
|
|
```
|