package mantra.compiler
Lexer
CLASS
public class
Indentation-aware token stream scanner that synthesizes INDENT and DEDENT tokens based on 4-space indentation rules.
🏗️ Constructor Summary
Constructor Signature
def __init__(self, source: String)⚙️ Method Summary
| Modifier & Return Type | Method Signature | Description |
|---|---|---|
List |
tokenize() |
Scans raw source string into token vector. |
🔍 Detailed Method Specifications
pub def tokenize() -> List
Scans raw source string into token vector.
💡 Example Usage
lexer_example.mantra
let lexer = Lexer("class App:\n pass")
let tokens = lexer.tokenize()