Bison

The GNU Project yacc replacement

1. These notes provide a summary of Bison. The full reference manual is located at http://www.gnu.org/software/bison/manual/bison.html.
2. Normally Bison is run in batch (offline) mode. In these notes, I am presenting an expression interpreter that needs to run in interactive mode, because an expression should be evaluated every time a new line character is entered. If you are running Bison in interactive mode, add the directive

   %option interactive
   

   to the top of your flex file. If you fail to do so, then flex will wait to read your last token until it sees more input. As this last token is the newline character, my parser will fail to reduce the last expression until the next expression is entered.

A bison input file consists of 3 sections; definitions, rules, and user subroutines. These sections are separated by two percent signs. The first 2 sections are required although one may be empty. The third section, which is the user subroutine section, and its preceeding %% are optional.

Definition Section

%start

The %start direction tells bison what is the start non-terminal for your grammar. Here the start non-terminal is program:

%start program

%union

%union {
        float num;
        char *id;
        };

%token

%token <num> NUMBER
%token <id> ID
%token NEWLINE
%token EQUALS PRINT

%type

%type <num> exp 
%type <num> stmt

%left, %right, and %nonassoc

• %left -- specifies tokens which are left associative
• %right -- specifies tokens which are rigth associative
• %nonassoc -- tokens which are non-associative

The associativity of an operator op determines how repeated uses of the operator nest: whether `x op y op z' is parsed by grouping x with y first or by grouping y with z first. %left specifies left-associativity (grouping x with y first) and %right specifies right-associativity (grouping y with z first). %nonassoc specifies no associativity, which means that `x op y op z' is considered a syntax error.

The precedence of an operator determines how it nests with other operators. All the tokens declared in a single precedence declaration have ` equal precedence and nest together according to their associativity. When two tokens declared in different precedence declarations associate, the one declared later has the higher precedence and is grouped first.

In the sample expression parser, there are four associativity declarations:

%left PLUS MINUS 
%left TIMES DIVIDE 
%left LPAREN RPAREN
%nonassoc UMINUS

%{ and %} for user-defined code

Finally the definitions section is the place to put user-defined code that you plan to use in either your parser or your scanner. As in a lex/flex specification, you place user-defined code between %{ and %} delimiters:

%{
#include <iostream>
#include <string>
#include <stdlib.h>
#include <map>

using namespace std;

// our hash table for variable names
  map<string, float> idTable;

// for keeping track of line numbers in the program we are parsing
  int line_num = 1;  

// we need the yylex and yyerror prototypes because bison uses them
// in its generated code, but does not declare them for us automatically.
int yylex();
void yyerror(char * s);
%}

Rules Section

stmt: ID EQUALS exp
    | PRINT ID 

exp: MINUS exp %prec UMINUS
    | exp PLUS exp
    | exp MINUS exp
    | NUMBER

%prec

Converting Extended BNF Form to BNF Form

The style in which you have seen grammars written in class is called Extended Backus Naur Form (EBNF) after the two creators of this style. Bison expects your grammar to be written in standard BNF form, which means without the shorthand notation for repetition (* for 0 or more, and + for 1 or more) and without the shorthand notation for optional items (?). Here is how to convert each of these three notations into a standard BNF form:

1. *: A star represents a list of 0 or more elements. For example, I might write:

   exp -> stmt*
   

   This type of list can be written in standard BNF as:

   exp -> stmtList
   stmtList -> stmtList stmt
            |
   

   Notice that the second production for stmtList derives the empty string, which is what allows us to have an empty list.

2. +: A plus represents a list of 1 or more elements. For example, I might write:

   exp -> stmt+
   

   This type of list can be written in standard BNF as:

   exp -> stmtList
   stmtList -> stmtList stmt
            | stmt
   

   Notice that the second production for stmtList derives a single stmt, which forces us to have at least a one element list.

3. ?: A question mark represents an optional element. For example, I might write:

   edge -> label ([thickness = NUM])? ([color = STRING])?
   
   

   An optional element can be written in two different forms in standard BNF:

   Style 1:
     edge -> label
         |   label [thickness = NUM]
         |   label [color = STRING]
         |   label [thickness = NUM] [color = STRING]
   
   Style 2:
     edge -> label optionalThickness optionalColor
     optionalThickness -> [thickness = NUM]
          |
     optionalColor -> [color = STRING]
          |
   

   Style 1 writes out all possible variations of the production, while Style 2 removes each optional component to a separate set of 2 productions, one which derives the optional component and one which derives an empty string. Either one is fine. Style 1 is often more readable for productions that contain 1 or 2 optional elements, but it blows up exponentially with the number of optional elements. Hence for 3 or more optional elements, it is often more readable to use Style 2.

Left versus Right Recursion

List: List Exp

and right recursion is the opposite

List: Exp List

The reason to prefer left recursion over right recursion is that you use less stack space wth left recursion. Right recursion forces the parser to shift all the recursive elements recognized by the rule onto the stack, which can make the stack arbitrarily deep.

Additionally, you always make sure that every recursive rule has at least one non-recursive alternative to avoid the possiblity of infinite recursion.

Rule Actions

stmt: ID EQUALS exp { 
   idTable[$1] = $3;       // update the value of the id in the hash table
   cout << $3 << endl;     // print the value of the expression to the console
   cout << ">>> ";
   $$ = $3;                // make the value of stmt equal to the value of expression
}

1. $$ represents the value assigned to the LHS non-terminal.
2. $1, $2, ..., $n represents the values that have already been assigned to the RHS symbols. If $i is a non-terminal, then the value was assigned by the action of a previously recognized production. If $i is a terminal, then the value was assigned by the scanner. Bison has already assigned the correct value to the terminal because the %token statement told it what field in yylval contained the value of this token.

User Subroutines

Errors

This is probably not the behavior you want. Instead you would like to resume parsing and find any additional errors in the input. To do so, you can judiciously place the reserved word error in one or more grammar rules. For example, in exp.yacc I have included the production:

stmtlist : stmtlist error NEWLINE { cout << ">>> "; yyclearin; } 

The yyclearin statement is a bison macro that tells the parser to consume the last token. If you do not use yyclearin, then the parser will attempt to re-use the last token when it resumes parsing. If the last token was an error token, then you will get two error messages, which can be confusing to the user.

Another potential point of confusion is that bison will recognize an error-free prefix of a statement, before generating an error. Hence if the user types:

>>> a = 20 ? 30

error token: ? on line 1
20
line 1: syntax error, unexpected NUMBER, expecting NEWLINE
>>> >>>

The message about the syntax error is generated by the function yyerror. Even though yyerror is not called by the action, it is still called by Bison. yyerror is discussed more in the next section.

Usually Bison recovers properly after recognizing an error production, but sometimes your action must execute the macro yyerrok to indicate that error recovery is complete. For example:

stmtlist: stmtlist error NEWLINE { cout << ">>> "; yyclearin; yyerrok; }

To prevent a lot of error messages, the parser will output no error message for another syntax error that happens shortly after the first; only after three consecutive input tokens have been successfully shifted will error messages resume. Rules which accept the error token may have actions, just as any other rules can.

yyerror()

yyerror (char *s)
{
  cerr << s << endl;
}

%error-verbose

>>> a ? 10
syntax error, unexpected NUMBER, expecting EQUALS

1. In the definitions section of bison, define a line number variable. For example:

   %{
     int line_num = 1;
   %}
   

2. In the definitions section of flex, define this line number variable as an external variable:

   %{ 
     extern int line_num;
   %}
   

3. Place a rule in flex that recognizes the newline character (\n) and increments line_num by 1.
4. Place rules in flex for recognizing error tokens, and print them out, prefixed with the line number on which they occurred.
5. Use the line_num variable in yyerror to report the line number on which an error occurred.

Compiling and Debugging

• -d -- this causes the output file which will contain the union and token definitions to be created.
• -y -- this causes the output files to be named in the yacc manner, that is y.tab.c and y.tab.h.
• -t -- this causes the debugging information to be output in the y.tab.c file.

The debugging information used by bison consists of printing to stderr the states and transitions used to parse a given input. In order to make use of this, you must generate the y.tab.c file with the -t option AND put in the statement 'yydebug=1;' before yyparse() is called (see the commented out yydebug statement in exp.yacc). It doesn't matter what value is used so long as it is greater than 0. This output will be quit large even for a small input. See the attached listing. Hopefully this debugging information will help you fix your rules so the grammar is correctly parsed.

Files

• flex input file exp.lex
• bison input file exp.yacc
• Makefile
• y.tab.h bison header file, generated by bison with -d option
• test input file
• debugging output listing

References