Token Data Structure

Converting lexical items into tokens simplifies later processing by reducing the input to a few standard kinds of tokens: reserved words, identifiers, numbers, operators, strings, delimiters. The tokens serve as the terminal symbols of the parser grammar.

A token will contain:

1. token type (identifier, operator, etc.)

2. basic data type basicdt: a numeric code indicating integer, real, pointer

3. pointers to the symbol table

4. pointers for making trees from tokens

5. value of the token (identifier name, number value, numeric code indicating which operator).