Gymbo

Classes | Namespaces | Enumerations | Functions
tokenizer.h File Reference

Implementation of tokenizer. More...

`#include <ctype.h>`
`#include <stdarg.h>`
`#include <stdbool.h>`
`#include <stdio.h>`
`#include <stdlib.h>`
`#include <string.h>`
`#include <cstdlib>`
`#include <cstring>`
`#include <string>`
`#include <unordered_map>`
Include dependency graph for tokenizer.h:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

## Classes
struct | gymbo::Token
| Structure representing a token. More...

## Namespaces
| gymbo

## Enumerations
enum | gymbo::TokenKind { gymbo::TOKEN_RESERVED , gymbo::TOKEN_RETURN , gymbo::TOKEN_IF , gymbo::TOKEN_ELSE , gymbo::TOKEN_FOR , gymbo::TOKEN_IDENT , gymbo::TOKEN_NUM , gymbo::TOKEN_EOF }
| Enumeration representing different token kinds. More...

## Functions
bool | gymbo::is_alpha (char c)
| Checks if a character is an alphabetical character or underscore. More...
bool | gymbo::is_alnum (char c)
| Checks if a character is an alphanumeric character. More...
void | gymbo::error (char *fmt,...)
| Reports an error and exits the program. More...
void | gymbo::error_at (char *user_input, char *loc, char *fmt,...)
| Reports an error location and exits the program. More...
bool | gymbo::consume (Token *&token, char *op)
| Consumes the current token if it matches `op`. More...
bool | gymbo::consume_tok (Token *&token, TokenKind tok)
| Consumes the current token if it matches `tok`. More...
Token * | gymbo::consume_ident (Token *&token)
| Consumes the current token if it is an identifier. More...
void | gymbo::expect (Token *&token, char *user_input, char *op)
| Ensures that the current token matches `op`. More...
float | gymbo::expect_number (Token *&token, char *user_input)
| Ensures that the current token is a number. More...
bool | gymbo::at_eof (Token *token)
| Checks if the current token is at the end of the program. More...
Token * | gymbo::new_token (TokenKind kind, Token *cur, char *str, int len)
| Creates a new token and adds it as the next token of `cur`. More...
bool | gymbo::startswith (char *p, char *q)
| Checks if the string `p` starts with the string `q`. More...
Token * | gymbo::tokenize (char *user_input, std::unordered_map< std::string, int > &var_counter)
| Tokenizes a given string and returns a linked list of tokens. More...

## Detailed Description

Implementation of tokenizer.

Author
  Hideaki Takahashi