Oblivion Mod:Oblivion Token Types

The CTokenType class holds a single custom token definition as loaded from a text file. These definitions will be used to create tokens by the tokenizer for use in the parser.

CTokenType

This class is very simple and is mainly composed of three members:

  class CTokenType {
     dword   m_ID;       /* Unique ID for this token */
     string  m_Name;     /* Token name */
     dword   m_Flags;    /* Token flags */
  };

The flags member is a bit field with the possible values:

• Keep: Store token value when tokenizing (used for strings, identifiers, etc...).
• Simple: Do not store the token value (most tokens).
• Ignore: Ignore token (not saved to the token stream, like comments).
• TrimFirst: Remove the first character in the token value (for quoted strings).
• TrimLast: Remove the last character in the token value (for quoted strings).
• EndToken: Identifies the token that ends a token sequence (stops parsing).

Data File Format

Custom tokens are defined from a text file with the following format:

  Tokens
     [TokenName] = [ID], [Option1], ... [OptionN]
     ...
  End

The [TokenName] is a unique string identifying the token. [ID] is a unique integer value which can also be used to identify the token. Multiple options can be specified after the ID and can be any of:

• keep
• simple
• ignore
• trimlast
• trimfirst
• endtoken

Oblivion Token File

The following token definitions can be used for Oblivion scripts:

  Tokens
     Unknown      = 0, keep
     LBracket     = 1, simple
     RBracket     = 2, simple
     AddOp        = 3, keep
     MultOp       = 4, keep
     RelOp        = 5, keep
     BoolOp       = 6, keep
     Comma        = 7, ignore
     String       = 16, keep, trimfirst, trimlast
     Identifier   = 17, keep
     If           = 18, simple
     Elseif       = 19, simple
     Else         = 20, simple
     Endif        = 21, simple
     Set          = 22, simple
     Scriptname   = 23, simple
     Begin        = 24, simple
     End          = 25, simple
     Comment      = 26, ignore
     EndLine      = 27, simple  
     Integer      = 28, keep
     Float        = 29, keep
     EndofProgram = 30, simple, endtoken
  End