The Compilation Process

Source Code

• Parsing step (scanned, split up, groped) aka. lexical (isolation from the sentence containing it) analysis. (grammar or meaning of text not taken into context. Just the meaning of the words themselves)

Lexical Analysis phase

1 Scans code

• Source text is considered as a chunk of string text.
• scanner reads the text one character at a time.
• for each char. it marks the line and position of where the character was found in source text.

2 Evaluates (lexing/tokenization)

• the lexer/tokenizer determines what type of token it has found

Example of how compiler lexes a phrase

token

• represented as a pair consisting of a token name and some (optional) value

lexemes

• words of a program
• substring of source code.
• grouping of smallest sequence of characters.

출처

https://medium.com/basecs/reading-code-right-with-some-help-from-the-lexer-63d0be3d21d