4.9 KiB
module scanner
Contents
Constants
const digit_extras = [`_`, `.`, `x`, `o`, `b`, `e`, `E`]
const end_of_text = u32(~0)
new_scanner
fn new_scanner(config Config) !&Scanner
new_scanner returns a new heap allocated Scanner instance, based on the file in config.input.file_path, or based on the text in config.input.text .
new_simple
fn new_simple(config Config) !Scanner
new_simple returns a new stack allocated Scanner instance.
new_simple_file
fn new_simple_file(path string) !Scanner
new_simple_file returns a new stack allocated Scanner instance ready for parsing TOML in file read from path.
new_simple_text
fn new_simple_text(text string) !Scanner
new_simple_text returns a new stack allocated Scanner instance ready for parsing TOML in text.
Config
struct Config {
pub:
input input.Config
tokenize_formatting bool = true // if true, generate tokens for `\n`, ` `, `\t`, `\r` etc.
}
Config is used to configure a Scanner instance. Only one of the fields text and file_path is allowed to be set at time of configuration.
Scanner
struct Scanner {
pub:
config Config
text string // the input TOML text
mut:
col int // current column number (x coordinate)
line_nr int = 1 // current line number (y coordinate)
pos int // current flat/index position in the `text` field
header_len int // Length, how many bytes of header was found
// Quirks
is_left_of_assign bool = true // indicates if the scanner is on the *left* side of an assignment
}
Scanner contains the necessary fields for the state of the scan process. the task the scanner does is also referred to as "lexing" or "tokenizing". The Scanner methods are based on much of the work in vlib/strings/textscanner.
scan
fn (mut s Scanner) scan() !token.Token
scan returns the next token from the input.
free
fn (mut s Scanner) free()
free frees all allocated resources.
remaining
fn (s &Scanner) remaining() int
remaining returns how many characters remain in the text input.
next
fn (mut s Scanner) next() u32
next returns the next character code from the input text. next returns end_of_text if it can't reach the next character.
skip
fn (mut s Scanner) skip()
skip skips one character ahead.
skip_n
fn (mut s Scanner) skip_n(n int)
skip_n skips ahead n characters. If the skip goes out of bounds from the length of Scanner.text, the scanner position will be sat to the last character possible.
at
fn (s &Scanner) at() u32
at returns the current character code from the input text. at returns end_of_text if it can't get the current character. unlike next(), at() does not change the state of the scanner.
peek
fn (s &Scanner) peek(n int) u32
peek returns the character code from the input text at position + n. peek returns end_of_text if it can't peek n characters ahead.
reset
fn (mut s Scanner) reset()
reset resets the internal state of the scanner.
excerpt
fn (s &Scanner) excerpt(pos int, margin int) string
excerpt returns a string excerpt of the input text centered at pos. The margin argument defines how many chacters on each side of pos is returned
state
fn (s &Scanner) state() State
state returns a read-only view of the scanner's internal state.
State
struct State {
pub:
col int // current column number (x coordinate)
line_nr int = 1 // current line number (y coordinate)
pos int // current flat/index position in the `text` field
}
State is a read-only copy of the scanner's internal state. See also Scanner.state().