Oblivion Mod:Oblivion Token Tables

The input character steam will be tokenized using a table based rules as defined in text files.

Type Definitions

The basic type definitions for all the classes needed for a table based tokenizer are given below:

  tokentableresult {
     int        Action  { ignore moveto return error continue jmpreturn jumpto }     //Type of action to take
     string     TableName            
     table*     Table                 //Destination for a table move/jump type
     string     TokenName            
     tokentype* TokenType             //Token to return for a return type
     string     Message               //User message for an error type
  }
 
  tokentablerow {
     int               MatchType { none charclass char string }    //Type of match to perform
     string            CharClassName  
     charclass*        CharClass      //Class to match for a class type row
     char              Char           //Character to match for a character type row
     string            StringValue    //String to match for a string type row
     tokentableresult  Result         //Action to take when the row matches
  }
 
  tokentable {
     string         Name              //Identifies the table
     tokentablerow  Rows[]            //Array of all rows in the table
     tokentablerow* DefaultRow        //Default row to be used when no matches are found
  }
 
  tokentablearray {
     tokentable   Tables[]            //Array of all tables
     tokentable*  StartTable          //The table to start tokenizing with
  }

Data File Format

The token tables are defined in a text file with the following format:

  [TableName]
     [RowMatch] = [Action]
     ...
  End
  ...

[RowMatch] can have the following values:

• CharacterClass -- A character class name as defined in the current class file.
• "[string]" -- A string surrounded by double quotes.
• [Character] -- A single character.
• Default -- The default row which is used when no match in the table is found.

The [Action] for a row can have the following values:

• ignore -- Ignore the current character and restart parsing the current token table.
• moveto [TableName] -- Add the current character to the token and start parsing the given token table.
• jumpto [TableName] -- Start parsing the given token table (does not add the current character to the token).
• return [TokenType] -- Add the current character to the token and return the given token type. Clear the current token value.
• jmpreturn [TokenType] -- Return the given token type (does not add the current character to the token). Clear the current token value.
• continue -- Add the current character to the token and restart parsing the current table.
• error [Message] -- Add the current character to the token and report an error using the supplied string. Clear the current token value.

Oblivion Token Tables

The following token tables (along with the matching Character Classes and Token Types) are capable of successfully tokenizing all of the 1687 scripts found in Oblivion.

  Begin
     Comment    = moveto Comment_Table
     Digit      = moveto Digit_Table
     Quote      = moveto String_Table
     IDStart    = moveto ID_Table
     (          = return LBracket
     )          = return RBracket
     AddOp      = return AddOp
     MultOp     = return MultOp
     &          = moveto And_Table
     |          = moveto Or_Table
     Equal      = moveto EQ_Table
     >          = moveto GT_Table
     <          = moveto LT_Table
     !          = moveto NEQ_Table
     EndLine    = return endline
     WhiteSpace = ignore
     Default    = return EndofProgram
  End
 
  And_Table
     &       = return BoolOp
     Default = jmpreturn Unknown
  End
 
  Or_Table
     |       = return BoolOp
     Default = jmpreturn Unknown
  End

  Digit_Table
     Digit   = continue
     Decimal = moveto Decimal_Table
     Default = jmpreturn Integer
  End
   
  Decimal_Table
     Digit   = continue
     Default = jmpreturn Float
  End
   
  String_Table
     Quote     = return String
     EndString = moveto String_Table
     Default   = error Unterminated string found! Ensure all strings end with a quote character.
  End
  
  ID_Table
     ID      = continue
     Default = jumpto CheckKeyWord_Table
  End

  CheckKeyWord_Table
     "if"         = jmpreturn if
     "endif"      = jmpreturn endif
     "set"        = jmpreturn set
     "elseif"     = jmpreturn elseif
     "else"        = jmpreturn else
     "endif"      = jmpreturn endif
     "begin"      = jmpreturn begin
     "end"        = jmpreturn end
     "scriptname" = jmpreturn scriptname
     "scn"        = jmpreturn scriptname
     Default      = jmpreturn Identifier
  End
  
  Comment_Table
     EndLine = jmpreturn Comment
     Default = continue
  End

  EQ_Table
     Equal   = return RelOp
     Default = error Unexpected = found! Valid comparison operators are: <, >, <=, >=, ==, and !=
  End
  
  GT_Table
     Equal   = return RelOp 
     Default = jmpreturn RelOp
  End

  LT_Table      Equal   = return RelOp
     Default = jmpreturn RelOp
  End

  NEQ_Table
     Equal   = return RelOp
     Default = error Unexpected ! found! Valid comparison operators are: <, >, <=, >=, ==, and !=
  End