Initial Commit

[packages] / xemacs-packages / semantic / doc / semantic-langdev.info

This is semantic-langdev.info, produced by makeinfo version 5.2 from
lang-support-guide.texi.

This manual documents Application Development with Semantic.

Copyright (C) 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2007 Eric M.
Ludlam Copyright (C) 2001, 2002, 2003, 2004 David Ponce Copyright (C)
2002, 2003 Richard Y. Kim

     Permission is granted to copy, distribute and/or modify this
     document under the terms of the GNU Free Documentation License,
     Version 1.1 or any later version published by the Free Software
     Foundation; with the Invariant Sections being list their titles,
     with the Front-Cover Texts being list, and with the Back-Cover
     Texts being list.  A copy of the license is included in the section
     entitled "GNU Free Documentation License".
INFO-DIR-SECTION Emacs
START-INFO-DIR-ENTRY
* Semantic Language Writer's guide: (semantic-langdev).
END-INFO-DIR-ENTRY

   This file documents Language Support Development with Semantic.
_Infrastructure for parser based text analysis in Emacs_

   Copyright (C) 1999, 2000, 2001, 2002, 2003, 2004 Eric M. Ludlam,
David Ponce, and Richard Y. Kim

\1f
File: semantic-langdev.info,  Node: Top,  Next: Tag Structure,  Up: (dir)

Language Support Developer's Guide
**********************************

Semantic is bundled with support for several languages such as C, C++,
Java, Python, etc.  However one of the primary gols of semantic is to
provide a framework in which anyone can add support for other languages
easily.  In order to support a new lanaugage, one typically has to
provide a lexer and a parser along with appropriate semantic actions
that produce the end result of the parser - the semantic tags.

This chapter first discusses the semantic tag data structure to
familiarize the reader to the goal.  Then all the components necessary
for supporting a lanaugage is discussed starting with the writing lexer,
writing the parser, writing semantic rules, etc.  Finally several
parsers bundled with semantic are discussed as case studies.

* Menu:

* Tag Structure::               
* Language Support Overview::   
* Writing Lexers::              
* Writing Parsers::             
* Parsing a language file::     
* Debugging::                   
* Parser Error Handling::       
* GNU Free Documentation License::  
* Index::                       

\1f
File: semantic-langdev.info,  Node: Tag Structure,  Next: Language Support Overview,  Prev: Top,  Up: Top

1 Tag Structure
***************

The end result of the parser for a buffer is a list of tags.  Currently
each tag is a list with up to five elements:
     ("NAME" CLASS ATTRIBUTES PROPERTIES OVERLAY)

CLASS represents what kind of tag this is.  Common CLASS values include
'variable', 'function', or 'type'.  *note (semantic-appdev.info)Tag
Basics::.

ATTRIBUTES is a slot filled with langauge specific options for the tag.
Function arguments, return type, and other flags all are stored in
attributes.  A language author fills in the ATTRIBUTES with the tag
constructor, which is parser style dependant.

PROPERTIES is a slot generated by the semantic parser harness, and need
not be provided by a language author.  Programmatically access tag
properties with 'semantic--tag-put-property',
'semantic--tag-put-property-no-side-effect' and
'semantic--tag-get-property'.

OVERLAY represents positional information for this tag.  It is
automatically generated by the semantic parser harness, and need not be
provided by the language author, unless they provide a tag expansion
function via 'semantic-tag-expand-function'.

The OVERLAY property is accessed via several functions returning the
beginning, end, and buffer of a token.  Use these functions unless the
overlay is really needed (see *note (app-dev-guide)Tag Query::).
Depending on the overlay in a program can be dangerous because sometimes
the overlay is replaced with an integer pair
     [ START END ]
when the buffer the tag belongs to is not in memory.  This happens when
a user has activated the Semantic Database *note
(semantic-appdev)semanticdb::.

To create tags for a functional or object oriented language, you can use
s series of tag creation functions.  *note (semantic-appdev)Creating
Tags::

\1f
File: semantic-langdev.info,  Node: Language Support Overview,  Next: Writing Lexers,  Prev: Tag Structure,  Up: Top

2 Language Support Overview
***************************

Starting with version 2.0, semantic provides many ways to add support
for a language into the semantic framework.

The primary means to customize how semantic works is to implement
language specific versions of overloadable functions.  Semantic has a
specialized mode bound way to do this.  *note Semantic Overload
Mechanism::.

The parser has several parts which are all also overloadable.  The
primary entry point into the parser is 'semantic-fetch-tags' which calls
'semantic-parse-region' which returns a list of semantic tags which get
set to 'semantic--buffer-cache'.

'semantic-parse-region' is the first "overloadable" function.  The
default behavior of this is to simply call 'semantic-lex', then pass the
lexical token list to 'semantic-repeat-parse-whole-stream'.  At each
stage, another more focused layer provides a means of overloading.

The parser is not the only layer that provides overloadable methods.
Application APIs *note (semantic-appdev)top:: provide many overload
functions as well.

* Menu:

* Semantic Overload Mechanism::  
* Semantic Parser Structure::   
* Application API Structure::   

\1f
File: semantic-langdev.info,  Node: Semantic Overload Mechanism,  Next: Semantic Parser Structure,  Prev: Language Support Overview,  Up: Language Support Overview

2.1 Semantic Overload Mechanism
===============================

one of semantic's goals is to provide a framework for supporting a wide
range of languages.  writing parsers for some languages are very simple,
e.g., any dialect of lisp family such as emacs-lisp and scheme.  parsers
for many languages can be written in context free grammars such as c,
java, python, etc.  on the other hand, it is impossible to specify
context free grammars for other languages such as texinfo.  Yet semantic
already provides parsers for all these languages.

In order to support such wide range of languages, a mechanism for
customizing the parser engine at many levels was needed to maximize the
code reuse yet give each programmer the flexibility of customizing the
parser engine at many levels of granularity.  The solution that semantic
provides is the function overloading mechanism which allows one to
intercept and customize the behavior of many of the functions in the
parser engine.  First the parser engine breaks down the task of parsing
a language into several steps.  Each step is represented by an
Emacs-Lisp function.  Some of these are 'semantic-parse-region',
'semantic-lex', 'semantic-parse-stream', 'semantic-parse-changes', etc.

Many built-in semantic functions are declared as being over-loadable
functions, i.e., functions that do reasonable things for most languages,
but can be customized to suit the particular needs of a given language.
All over-loadable functions then can easily be over-ridden if necessary.
The rest of this section provides details on this overloading mechanism.

Over-loadable functions are created by defining functions with the
'define-overload' macro rather than the usual 'defun'.
'define-overload' is a thin wrapper around 'defun' that sets up the
function so that it can be overloaded.  An over-loadable function then
can be over-ridden in one of two ways:
'define-mode-overload-implementation' and
'semantic-install-function-overrides'.

Let's look at a couple of examples.  'semantic-parse-region' is one of
the top level functions in the parser engine defined via
'define-overload':

     (define-overload semantic-parse-region
       (start end &optional nonterminal depth returnonerror)
       "Parse the area between START and END, and return any tokens found.

     ...

     tokens.")

The document string was truncated in the middle above since it is not
relevant here.  The macro invocation above defines the
'semantic-parse-region' Emacs-Lisp function that checks first if there
is an overloaded implementation.  If one is found, then that is called.
If a mode specific implementation is not found, then the default
implementation is called which in this case is to call
'semantic-parse-region-default', i.e., a function with the same name but
with the tailing -default.  That function needs to be written separately
and take the same arguments as the entry created with 'define-overload'.

One way to overload 'semantic-parse-region' is via
'semantic-install-function-overrides'.  An example from
'semantic-texi.el' file is shown below:

     (defun semantic-default-texi-setup ()
       "Set up a buffer for parsing of Texinfo files."
       ;; This will use our parser.
       (semantic-install-function-overrides
        '((parse-region . semantic-texi-parse-region)
          (parse-changes . semantic-texi-parse-changes)))
       ...
       )

     (add-hook 'texinfo-mode-hook 'semantic-default-texi-setup)

Above function is called whenever a buffer is setup as texinfo mode.
'semantic-install-function-overrides' above indicates that
'semantic-texi-parse-region' is to over-ride the default implementation
of 'semantic-parse-region'.  Note the use of 'parse-region' symbol which
is 'semantic-parse-region' without the leading semantic- prefix.

Another way to over-ride a built-in semantic function is via
'define-mode-overload-implementation'.  An example from
'wisent-python.el' file is shown below.

     (define-mode-overload-implementation
       semantic-parse-region python-mode
       (start end &optional nonterminal depth returnonerror)
       "Over-ride in order to initialize some variables."
       (let ((wisent-python-lexer-indent-stack '(0))
             (wisent-python-explicit-line-continuation nil))
         (semantic-parse-region-default
          start end nonterminal depth returnonerror)))

Above over-rides 'semantic-parse-region' so that for buffers whose major
mode is 'python-mode', the code specified above is executed rather than
the default implementation.

2.1.1 Why not to use advice
---------------------------

One may wonder why semantic defines an overload mechanism when Emacs
already has advice.  *Note (elisp)Advising Functions::.

Advising is generally considered a mechanism of last resort when
modifying or hooking into an existing package without modifying that
sourse file.  Overload files advertise that they should be overloaded,
and define syntactic sugar to do so.

\1f
File: semantic-langdev.info,  Node: Semantic Parser Structure,  Next: Application API Structure,  Prev: Semantic Overload Mechanism,  Up: Language Support Overview

2.2 Semantic Parser Structure
=============================

NOTE: describe the functions that do parsing, and how to overload each.





\1f
File: semantic-langdev.info,  Node: Application API Structure,  Prev: Semantic Parser Structure,  Up: Language Support Overview

2.3 Application API Structure
=============================

NOTE: improve this:

How to program with the Application programming API into the data
structures created by semantic are in the Application development guide.
Read that guide to get a feel for the specifics of what you can
customize.  *note (semantic-appdev)top::

Here are a list of applications, and the specific APIs that you will
need to overload to make them work properly with your language.

'imenu'
'speedbar'
'ecb'
     These tools requires that the 'semantic-format' methods create
     correct strings.  *note (semantic-addpev)Format Tag::
'semantic-analyze'
     The analysis tool requires that the 'semanticdb' tool is active,
     and that the searching methods are overloaded.  In addition,
     'semanticdb' system database could be written to provide symbols
     from the global environment of your langauge.  *note
     (semantic-appdev)System Databases::

     In addition, the analyzer requires that the 'semantic-ctxt' methods
     are overloaded.  These methods allow the analyzer to look at the
     context of the cursor in your language, and predict the type of
     location of the cursor.  *note (semantic-appdev)Derived Context::.
'semantic-idle-summary-mode'
'semantic-idle-completions-mode'
     These tools use the semantic analysis tool.  *note Context
     Analysis. . semantic-appdev::

* Menu:

* Semantic Analyzer Support::   

\1f
File: semantic-langdev.info,  Node: Semantic Analyzer Support,  Prev: Application API Structure,  Up: Application API Structure

2.3.1 Semantic Analyzer Support
-------------------------------

\1f
File: semantic-langdev.info,  Node: Writing Lexers,  Next: Writing Parsers,  Prev: Language Support Overview,  Up: Top

3 Writing Lexers
****************

In order to reduce a source file into a tag table, it must first be
converted into a token stream.  Tokens are syntactic elements such as
whitespace, symbols, strings, lists, and punctuation.

The lexer uses the major-mode's syntax table for conversion.  *Note
(elisp)Syntax Tables::.  As long as that is set up correctly (along with
the important 'comment-start' and 'comment-start-skip' variable) the
lexer should already work for your language.

The primary entry point of the lexer is the "semantic-lex" function
shown below.  Normally, you do not need to call this function.  It is
usually called by _semantic-fetch-tags_ for you.

 -- Function: semantic-lex start end &optional depth length
     Lexically analyze text in the current buffer between START and END.
     Optional argument DEPTH indicates at what level to scan over entire
     lists.  The last argument, LENGTH specifies that "semantic-lex"
     should only return LENGTH tokens.  The return value is a token
     stream.  Each element is a list, such of the form (symbol
     start-expression .  end-expression) where SYMBOL denotes the token
     type.  See 'semantic-lex-tokens' variable for details on token
     types.  END does not mark the end of the text scanned, only the end
     of the beginning of text scanned.  Thus, if a string extends past
     END, the end of the return token will be larger than END.  To truly
     restrict scanning, use "narrow-to-region".

* Menu:

* Lexer Overview::              What is a Lexer?
* Lexer Output::                Output of a Lexical Analyzer
* Lexer Construction::          Constructing your own lexer
* Lexer Built In Analyzers::    Built in analyzers you can use
* Lexer Analyzer Construction::  Constructing your own anlyzers
* Keywords::                    Specialized lexical tokens.
* Keyword Properties::          

\1f
File: semantic-langdev.info,  Node: Lexer Overview,  Next: Lexer Output,  Prev: Writing Lexers,  Up: Writing Lexers

3.1 Lexer Overview
==================

semantic lexer breaks up the content of an Emacs buffer into a stream of
tokens.  This process is based mostly on regular expressions which in
turn depend on the syntax table of the buffer's major mode being setup
properly.  *Note (emacs)Major Modes::.  *Note (elisp)Syntax Tables::.
*Note (emacs)Regexps::.

The top level lexical function "semantic-lex", calls the function stored
in "semantic-lex-analyzer".  The default value is the function
"semantic-flex" from version 1.4 of Semantic.  This will eventually be
depricated.

In the default lexer, the following regular expressions which rely on
syntax tables are used:

'\\s-'
     whitespace characters
'\\sw'
     word constituent
'\\s_'
     symbol constituent
'\\s.'
     punctuation character
'\\s<'
     comment starter
'\\s>'
     comment ender
'\\s\\'
     escape character
'\\s)'
     close parenthesis character
'\\s$'
     paired delimiter
'\\s\"'
     string quote
'\\s\''
     expression prefix

In addition, Emacs' built-in features such as 'comment-start-skip',
'forward-comment', 'forward-list', and 'forward-sexp' are employed.

\1f
File: semantic-langdev.info,  Node: Lexer Output,  Next: Lexer Construction,  Prev: Lexer Overview,  Up: Writing Lexers

3.2 Lexer Output
================

The lexer, *note semantic-lex::, scans the content of a buffer and
returns a token list.  Let's illustrate this using this simple example.

     00: /*
     01:  * Simple program to demonstrate semantic.
     02:  */
     03:
     04: #include <stdio.h>
     05:
     06: int i_1;
     07:
     08: int
     09: main(int argc, char** argv)
     10: {
     11:     printf("Hello world.\n");
     12: }

Evaluating '(semantic-lex (point-min) (point-max))' within the buffer
with the code above returns the following token list.  The input line
and string that produced each token is shown after each semi-colon.

     ((punctuation     52 .  53)     ; 04: #
      (INCLUDE         53 .  60)     ; 04: include
      (punctuation     61 .  62)     ; 04: <
      (symbol          62 .  67)     ; 04: stdio
      (punctuation     67 .  68)     ; 04: .
      (symbol          68 .  69)     ; 04: h
      (punctuation     69 .  70)     ; 04: >
      (INT             72 .  75)     ; 06: int
      (symbol          76 .  79)     ; 06: i_1
      (punctuation     79 .  80)     ; 06: ;
      (INT             82 .  85)     ; 08: int
      (symbol          86 .  90)     ; 08: main
      (semantic-list   90 . 113)     ; 08: (int argc, char** argv)
      (semantic-list  114 . 147)     ; 09-12: body of main function
      )

As shown above, the token list is a list of "tokens".  Each token in
turn is a list of the form

     (TOKEN-TYPE BEGINNING-POSITION . ENDING-POSITION)

where TOKEN-TYPE is a symbol, and the other two are integers indicating
the buffer position that delimit the token such that

     (buffer-substring BEGINNING-POSITION ENDING-POSITION)

would return the string form of the token.

Note that one line (line 4 above) can produce seven tokens while the
whole body of the function produces a single token.  This is because the
DEPTH parameter of 'semantic-lex' was not specified.  Let's see the
output when DEPTH is set to 1.  Evaluate '(semantic-lex (point-min)
(point-max) 1)' in the same buffer.  Note the third argument of '1'.

     ((punctuation    52 .  53)     ; 04: #
      (INCLUDE        53 .  60)     ; 04: include
      (punctuation    61 .  62)     ; 04: <
      (symbol         62 .  67)     ; 04: stdio
      (punctuation    67 .  68)     ; 04: .
      (symbol         68 .  69)     ; 04: h
      (punctuation    69 .  70)     ; 04: >
      (INT            72 .  75)     ; 06: int
      (symbol         76 .  79)     ; 06: i_1
      (punctuation    79 .  80)     ; 06: ;
      (INT            82 .  85)     ; 08: int
      (symbol         86 .  90)     ; 08: main

      (open-paren     90 .  91)     ; 08: (
      (INT            91 .  94)     ; 08: int
      (symbol         95 .  99)     ; 08: argc
      (punctuation    99 . 100)     ; 08: ,
      (CHAR          101 . 105)     ; 08: char
      (punctuation   105 . 106)     ; 08: *
      (punctuation   106 . 107)     ; 08: *
      (symbol        108 . 112)     ; 08: argv
      (close-paren   112 . 113)     ; 08: )

      (open-paren    114 . 115)     ; 10: {
      (symbol        120 . 126)     ; 11: printf
      (semantic-list 126 . 144)     ; 11: ("Hello world.\n")
      (punctuation   144 . 145)     ; 11: ;
      (close-paren   146 . 147)     ; 12: }
      )

The DEPTH parameter "peeled away" one more level of "list" delimited by
matching parenthesis or braces.  The depth parameter can be specified to
be any number.  However, the parser needs to be able to handle the extra
tokens.

This is an interesting benefit of the lexer having the full resources of
Emacs at its disposal.  Skipping over matched parenthesis is achieved by
simply calling the built-in functions 'forward-list' and 'forward-sexp'.

\1f
File: semantic-langdev.info,  Node: Lexer Construction,  Next: Lexer Built In Analyzers,  Prev: Lexer Output,  Up: Writing Lexers

3.3 Lexer Construction
======================

While using the default lexer is certainly an option, particularly for
grammars written in semantic 1.4 style, it is usually more efficient to
create a custom lexer for your language.

You can create a new lexer with "define-lex".

 -- Function: define-lex name doc &rest analyzers
     Create a new lexical analyzer with NAME.  DOC is a documentation
     string describing this analyzer.  ANALYZERS are small code snippets
     of analyzers to use when building the new NAMED analyzer.  Only use
     analyzers which are written to be used in "define-lex".  Each
     analyzer should be an analyzer created with "define-lex-analyzer".
     Note: The order in which analyzers are listed is important.  If two
     analyzers can match the same text, it is important to order the
     analyzers so that the one you want to match first occurs first.
     For example, it is good to put a numbe analyzer in front of a
     symbol analyzer which might mistake a number for as a symbol.

The list of ANALYZERS, needed here can consist of one of several built
in analyzers, or one of your own construction.  The built in analyzers
are:

\1f
File: semantic-langdev.info,  Node: Lexer Built In Analyzers,  Next: Lexer Analyzer Construction,  Prev: Lexer Construction,  Up: Writing Lexers

3.4 Lexer Built In Analyzers
============================

 -- Special Form: semantic-lex-default-action
     The default action when no other lexical actions match text.  This
     action will just throw an error.

 -- Special Form: semantic-lex-beginning-of-line
     Detect and create a beginning of line token (BOL).

 -- Special Form: semantic-lex-newline
     Detect and create newline tokens.

 -- Special Form: semantic-lex-newline-as-whitespace
     Detect and create newline tokens.  Use this ONLY if newlines are
     not whitespace characters (such as when they are comment end
     characters) AND when you want whitespace tokens.

 -- Special Form: semantic-lex-ignore-newline
     Detect and create newline tokens.  Use this ONLY if newlines are
     not whitespace characters (such as when they are comment end
     characters).

 -- Special Form: semantic-lex-whitespace
     Detect and create whitespace tokens.

 -- Special Form: semantic-lex-ignore-whitespace
     Detect and skip over whitespace tokens.

 -- Special Form: semantic-lex-number
     Detect and create number tokens.  Number tokens are matched via
     this variable:

      -- Variable: semantic-lex-number-expression
          Regular expression for matching a number.  If this value is
          'nil', no number extraction is done during lex.  This
          expression tries to match C and Java like numbers.

               DECIMAL_LITERAL:
                   [1-9][0-9]*
                 ;
               HEX_LITERAL:
                   0[xX][0-9a-fA-F]+
                 ;
               OCTAL_LITERAL:
                   0[0-7]*
                 ;
               INTEGER_LITERAL:
                   <DECIMAL_LITERAL>[lL]?
                 | <HEX_LITERAL>[lL]?
                 | <OCTAL_LITERAL>[lL]?
                 ;
               EXPONENT:
                   [eE][+-]?[09]+
                 ;
               FLOATING_POINT_LITERAL:
                   [0-9]+[.][0-9]*<EXPONENT>?[fFdD]?
                 | [.][0-9]+<EXPONENT>?[fFdD]?
                 | [0-9]+<EXPONENT>[fFdD]?
                 | [0-9]+<EXPONENT>?[fFdD]
                 ;

 -- Special Form: semantic-lex-symbol-or-keyword
     Detect and create symbol and keyword tokens.

 -- Special Form: semantic-lex-charquote
     Detect and create charquote tokens.

 -- Special Form: semantic-lex-punctuation
     Detect and create punctuation tokens.

 -- Special Form: semantic-lex-punctuation-type
     Detect and create a punctuation type token.  Recognized
     punctuations are defined in the current table of lexical types, as
     the value of the 'punctuation' token type.

 -- Special Form: semantic-lex-paren-or-list
     Detect open parenthesis.  Return either a paren token or a semantic
     list token depending on 'semantic-lex-current-depth'.

 -- Special Form: semantic-lex-open-paren
     Detect and create an open parenthisis token.

 -- Special Form: semantic-lex-close-paren
     Detect and create a close paren token.

 -- Special Form: semantic-lex-string
     Detect and create a string token.

 -- Special Form: semantic-lex-comments
     Detect and create a comment token.

 -- Special Form: semantic-lex-comments-as-whitespace
     Detect comments and create a whitespace token.

 -- Special Form: semantic-lex-ignore-comments
     Detect and create a comment token.

\1f
File: semantic-langdev.info,  Node: Lexer Analyzer Construction,  Next: Keywords,  Prev: Lexer Built In Analyzers,  Up: Writing Lexers

3.5 Lexer Analyzer Construction
===============================

Each of the previous built in analyzers are constructed using a set of
analyzer construction macros.  The root construction macro is:

 -- Function: define-lex-analyzer name doc condition &rest forms
     Create a single lexical analyzer NAME with DOC.  When an analyzer
     is called, the current buffer and point are positioned in a buffer
     at the location to be analyzed.  CONDITION is an expression which
     returns 't' if FORMS should be run.  Within the bounds of CONDITION
     and FORMS, the use of backquote can be used to evaluate expressions
     at compile time.  While forms are running, the following variables
     will be locally bound: 'semantic-lex-analysis-bounds' - The bounds
     of the current analysis.  of the form (START .  END)
     'semantic-lex-maximum-depth' - The maximum depth of semantic-list
     for the current analysis.  'semantic-lex-current-depth' - The
     current depth of 'semantic-list' that has been decended.
     'semantic-lex-end-point' - End Point after match.  Analyzers should
     set this to a buffer location if their match string does not
     represent the end of the matched text.  'semantic-lex-token-stream'
     - The token list being collected.  Add new lexical tokens to this
     list.  Proper action in FORMS is to move the value of
     'semantic-lex-end-point' to after the location of the analyzed
     entry, and to add any discovered tokens at the beginning of
     'semantic-lex-token-stream'.  This can be done by using
     "semantic-lex-push-token".

Additionally, a simple regular expression based analyzer can be built
with:

 -- Function: define-lex-regex-analyzer name doc regexp &rest forms
     Create a lexical analyzer with NAME and DOC that will match REGEXP.
     FORMS are evaluated upon a successful match.  See
     "define-lex-analyzer" for more about analyzers.

 -- Function: define-lex-simple-regex-analyzer name doc regexp toksym
          &optional index &rest forms
     Create a lexical analyzer with NAME and DOC that match REGEXP.
     TOKSYM is the symbol to use when creating a semantic lexical token.
     INDEX is the index into the match that defines the bounds of the
     token.  Index should be a plain integer, and not specified in the
     macro as an expression.  FORMS are evaluated upon a successful
     match BEFORE the new token is created.  It is valid to ignore
     FORMS.  See "define-lex-analyzer" for more about analyzers.

Regular expression analyzers are the simplest to create and manage.
Often, a majority of your lexer can be built this way.  The analyzer for
matching punctuation looks like this:

     (define-lex-simple-regex-analyzer semantic-lex-punctuation
       "Detect and create punctuation tokens."
       "\\(\\s.\\|\\s$\\|\\s'\\)" 'punctuation)

More complex analyzers for matching larger units of text to optimize the
speed of parsing and analysis is done by matching blocks.

 -- Function: define-lex-block-analyzer name doc spec1 &rest specs
     Create a lexical analyzer NAME for paired delimiters blocks.  It
     detects a paired delimiters block or the corresponding open or
     close delimiter depending on the value of the variable
     'semantic-lex-current-depth'.  DOC is the documentation string of
     the lexical analyzer.  SPEC1 and SPECS specify the token symbols
     and open, close delimiters used.  Each SPEC has the form:

     (BLOCK-SYM (OPEN-DELIM OPEN-SYM) (CLOSE-DELIM CLOSE-SYM))

     where BLOCK-SYM is the symbol returned in a block token.
     OPEN-DELIM and CLOSE-DELIM are respectively the open and close
     delimiters identifying a block.  OPEN-SYM and CLOSE-SYM are
     respectively the symbols returned in open and close tokens.

These blocks is what makes the speed of semantic's Emacs Lisp based
parsers fast.  For exmaple, by defining all text inside { braces } as a
block the parser does not need to know the contents of those braces
while parsing, and can skip them all together.

\1f
File: semantic-langdev.info,  Node: Keywords,  Next: Keyword Properties,  Prev: Lexer Analyzer Construction,  Up: Writing Lexers

3.6 Keywords
============

Another important piece of the lexer is the keyword table (see *note
Writing Parsers::).  You language will want to set up a keyword table
for fast conversion of symbol strings to language terminals.

The keywords table can also be used to store additional information
about those keywords.  The following programming functions can be useful
when examining text in a language buffer.

 -- Function: semantic-lex-keyword-p name
     Return non-'nil' if a keyword with NAME exists in the keyword
     table.  Return 'nil' otherwise.

 -- Function: semantic-lex-keyword-put name property value
     For keyword with NAME, set its PROPERTY to VALUE.

 -- Function: semantic-lex-keyword-get name property
     For keyword with NAME, return its PROPERTY value.

 -- Function: semantic-lex-map-keywords fun &optional property
     Call function FUN on every semantic keyword.  If optional PROPERTY
     is non-'nil', call FUN only on every keyword which as a PROPERTY
     value.  FUN receives a semantic keyword as argument.

 -- Function: semantic-lex-keywords &optional property
     Return a list of semantic keywords.  If optional PROPERTY is
     non-'nil', return only keywords which have a PROPERTY set.

Keyword properties can be set up in a grammar file for ease of
maintenance.  While examining the text in a language buffer, this can
provide an easy and quick way of storing details about text in the
buffer.

\1f
File: semantic-langdev.info,  Node: Keyword Properties,  Prev: Keywords,  Up: Writing Lexers

3.7 Standard Keyword Properties
===============================

Keywords in a language can have multiple properties.  These properties
can be used to associate the string that is the keyword with additional
information.

Currently available properties are:

summary
     The summary property is used by semantic-summary-mode as a help
     string for the keyword specified.

Notes:

Possible future properties.  This is just me musing:

face
     Face used for highlighting this keyword, differentiating it from
     the keyword face.
template
skeleton
     Some sort of tempo/skel template for inserting the programatic
     structure associated with this keyword.
abbrev
     As with template.
action
menu
     Perhaps the keyword is clickable and some action would be useful.

\1f
File: semantic-langdev.info,  Node: Writing Parsers,  Next: Parsing a language file,  Prev: Writing Lexers,  Up: Top

4 Writing Parsers
*****************

When converting a source file into a tag table it is important to
specify rules to accomplish this.  The rules are stored in the buffer
local variable 'semantic--buffer-cache'.

While it is certainly possible to write this table yourself, it is most
likely you will want to use the *note Grammar Programming Environment::.

There are three choices for parsing your language.

Bovine Parser
     The "bovine" parser is the original semantic parser, and is an
     implementation of an LL parser.  For more information, *note the
     Bovine Parser Manual: (bovine)top.

Wisent Parser
     The "wisent" parser is a port of the GNU Compiler Compiler Bison to
     Emacs Lisp.  Wisent includes the iterative error handler of the
     bovine parser, and has the same error correction as traditional
     LALR parsers.  For more information, *note the Wisent Parser
     Manual: (wisent)top.

External Parser
     External parsers, such as the texinfo parser can be implemented
     using any means.  This allows the use of a regular expression
     parser for non-regular languages, or external programs for speed.

* Menu:

* External Parsers::            Writing an external parser
* Grammar Programming Environment::  Using the grammar writing environemt
* Parser Backend Support::             Lisp needed to support a grammar.

\1f
File: semantic-langdev.info,  Node: External Parsers,  Next: Grammar Programming Environment,  Prev: Writing Parsers,  Up: Writing Parsers

4.1 External Parsers
====================

The texinfo parser in 'semantic-texi.el' is an example of an external
parser.  To make your parser work, you need to have a setup function.

Note: Finish this.

\1f
File: semantic-langdev.info,  Node: Grammar Programming Environment,  Next: Parser Backend Support,  Prev: External Parsers,  Up: Writing Parsers

4.2 Grammar Programming Environment
===================================

Semantic grammar files in '.by' or '.wy' format have their own
programming mode.  This mode provides indentation and coloring services
in those languages.  In addition, the grammar languages are also
supported by semantic so tagging information is available to tools such
as imenu or speedbar.

For more information, *note the Grammar Framework Manual:
(grammar-fw)top.

\1f
File: semantic-langdev.info,  Node: Parsing a language file,  Next: Debugging,  Prev: Writing Parsers,  Up: Top

5 Parsing a language file
*************************

The best way to call the parser from programs is via
'semantic-fetch-tags'.  This, in turn, uses other internal API functions
which plug-in parsers can take advantage of.

 -- Function: semantic-fetch-tags
     Fetch semantic tags from the current buffer.  If the buffer cache
     is up to date, return that.  If the buffer cache is out of date,
     attempt an incremental reparse.  If the buffer has not been parsed
     before, or if the incremental reparse fails, then parse the entire
     buffer.  If a lexcial error had been previously discovered and the
     buffer was marked unparseable, then do nothing, and return the
     cache.

Another approach is to let Emacs call the parser on idle time, when
needed, then use 'semantic-fetch-available-tags' to retrieve and process
only the available tags, provided that the 'semantic-after-*-hook' hooks
have been setup to synchronize with new tags when they become available.

 -- Function: semantic-fetch-available-tags
     Fetch available semantic tags from the current buffer.  That is,
     return tags currently in the cache without parsing the current
     buffer.

     Parse operations happen asynchronously when needed on Emacs idle
     time.  Use the 'semantic-after-toplevel-cache-change-hook' and
     'semantic-after-partial-cache-change-hook' hooks to synchronize
     with new tags when they become available.

 -- Command: semantic-clear-toplevel-cache
     Clear the toplevel tag cache for the current buffer.  Clearing the
     cache will force a complete reparse next time a token stream is
     requested.

\1f
File: semantic-langdev.info,  Node: Parser Backend Support,  Prev: Grammar Programming Environment,  Up: Writing Parsers

5.1 Parser Backend Support
==========================

Once you have written a grammar file that has been compiled into Emacs
Lisp code, additional glue needs to be written to finish connecting the
generated parser into the Emacs framework.

Large portions of this glue is automatically generated, but will
probably need additional modification to get things to work properly.

Typically, a grammar file 'foo.wy' will create the file 'foo-wy.el'.  It
is then useful to also create a file 'wisent-foo.el' (or
'sematnic-foo.el') to contain the parser back end, or the glue that
completes the semantic support for the language.

* Menu:

* Example Backend File::
* Tag Expansion::

\1f
File: semantic-langdev.info,  Node: Example Backend File,  Next: Tag Expansion,  Prev: Parser Backend Support,  Up: Parser Backend Support

5.1.1 Example Backend File
--------------------------

Typical structure for this file is:

     ;;; semantic-foo.el -- parser support for FOO.

     ;;; Your copyright Notice

     (require 'foo-wy.el)  ;; The parser
     (require 'foo) ;; major mode definition for FOO

     ;;; Code:

     ;;; Lexical Analyzer
     ;;
     ;; OPTIONAL
     ;; It is possible to define your lexical analyzer completely in your
     ;; grammar file.

     (define-lex foo-lexical-analyzer
       "Create a lexical analyzer."
       ...)

     ;;; Expand Function
     ;;
     ;; OPTIONAL
     ;; Not all langauges are so complex as to need this function.
     ;; See `semantic-tag-expand-function' for more details.
     (defun foo-tag-expand-function (tag)
       "Expand TAG into multiple tags if needed."
       ...)

     ;;; Parser Support
     ;;
     ;; OPTIONAL
     ;; If you need some specialty routines inside your grammar file
     ;; you can add some here.   The process may be to take diverse info
     ;; and reorganize it.
     ;;
     ;; It is also appropriate to write these functions in the prologue
     ;; of the grammar function.
     (defun foo-do-something-hard (...)
       "...")

     ;;; Overload methods
     ;;
     ;; OPTIONAL
     ;; To allow your langauge to be fully supported by all the
     ;; applications that use semantic, it is important, but not necessary
     ;; to create implementations of overload methods.
     (define-mode-overload-implementation some-semantic-function foo-mode (tag)
       "Implement some-semantic-function for FOO."
       )

     ;;;###autoload
     (defun semantic-default-foo-setup ()
       "Set up a buffer for semantic parsing of the FOO language."
       (semantic-foo-by--install-parser)
       (setq semantic-tag-expand-function foo-tag-expand-function
             ;; Many other language specific settings can be done here
             ;; as well.
             )
       ;; This may be optional
       (setq semantic-lex-analyzer #'foo-lexical-analyzer)
       )

     ;;;###autoload
     (add-hook 'foo-mode-hook 'semantic-default-foo-setup)

     (provide 'semantic-c)

     ;;; semantic-foo.el ends here

\1f
File: semantic-langdev.info,  Node: Tag Expansion,  Prev: Example Backend File,  Up: Parser Backend Support

5.1.2 Tag Expansion
-------------------

In any language with compound tag types, you will need to implement an
_expand function_.  Once written, assign it to this variable.

 -- Variable: semantic-tag-expand-function
     Function used to expand a tag.  It is passed each tag production,
     and must return a list of tags derived from it, or 'nil' if it does
     not need to be expanded.

     Languages with compound definitions should use this function to
     expand from one compound symbol into several.  For example, in C or
     Java the following definition is easily parsed into one tag:

     int a, b;

     This function should take this compound tag and turn it into two
     tags, one for A, and the other for B.

Additionally, you can use the expand function in conjuntion with your
language for other types of compound statements.  For example, in Common
Lisp Object System, you can have a definition:

     (defclass classname nil
       (slots ...) ...)

This will create both the datatype 'classname' and the functional
constructor 'classname'.  Each slot may have a ':accessor' method as
well.

You can create a special compounded tag in your rule, for example:

     classdef: LPAREN DEFCLASS name semantic-list semantic-list RPAREN
               (TAG "custom" 'compound-class
                    :value (list
                             (TYPE-TAG $3 "class" ...)
                             (FUNCTION-TAG $3 ...)
                             ))
             ;

and in your expand function, you would write:

     (defun my-tag-expand (tag)
       "Expand tags for my langauge."
       (when (semantic-tag-of-class-p tag 'compound-class)
          (remq nil
             (semantic-tag-get-attribute tag :value))))

This will cause the custom tag to be replaced by the tags created in the
:value attribute of the specially constructed tag.

\1f
File: semantic-langdev.info,  Node: Debugging,  Next: Parser Error Handling,  Prev: Parsing a language file,  Up: Top

6 Debugging
***********

Grammars can be tricky things to debug.  There are several types of
tools for debugging in Semantic, and the type of problem you have
requires different types of tools.

* Menu:

* Lexical Debugging::           
* Parser Output tools::         
* Bovine Parser Debugging::     
* Wisent Parser Debugging::     
* Overlay Debugging::           
* Incremental Parser Debugging::  
* Debugging Analysis::          
* Semantic 1.4 Doc::            

\1f
File: semantic-langdev.info,  Node: Lexical Debugging,  Next: Parser Output tools,  Prev: Debugging,  Up: Debugging

6.1 Lexical Debugging
=====================

The first major problem you may encounter is with lexical analysis.  If
the text is not transformed into the expected token stream, no parser
will understand it.

You can step through the lexical analyzer with the following command:

 -- Command: semantic-lex-debug arg
     Debug the semantic lexer in the current buffer.  Argument ARG
     specifies of the analyze the whole buffer, or start at point.
     While engaged, each token identified by the lexer will be
     highlighted in the target buffer A description of the current token
     will be displayed in the minibuffer.  Press 'SPC' to move to the
     next lexical token.

For an example of what the output of the 'semantic-lex' function should
return, see *note Lexer Output::.

\1f
File: semantic-langdev.info,  Node: Parser Output tools,  Next: Bovine Parser Debugging,  Prev: Lexical Debugging,  Up: Debugging

6.2 Parser Output tools
=======================

There are several tools which can be used to see what the parser output
is.  These will work for any type of parser, including the Bovine
parser, Wisent parser.

The first and easiest is a minor mode which highlights text the parser
did not understand.

 -- Command: semantic-show-unmatched-syntax-mode &optional arg
     Minor mode to highlight unmatched lexical syntax tokens.  When a
     parser executes, some elements in the buffer may not match any
     parser rules.  These text characters are considered unmatched
     syntax.  Often time, the display of unmatched syntax can expose
     coding problems before the compiler is run.

     With prefix argument ARG, turn on if positive, otherwise off.  The
     minor mode can be turned on only if semantic feature is available
     and the current buffer was set up for parsing.  Return non-'nil' if
     the minor mode is enabled.

     'key'
          binding
     'C-c ,'
          Prefix Command
     'C-c , `'
          semantic-show-unmatched-syntax-next

Another interesting mode will display a line between all the tags in the
current buffer to make it more obvious where boundaries lie.  You can
enable this as a minor mode.

 -- Command: semantic-show-tag-boundaries-mode &optional arg
     Minor mode to display a boundary in front of tags.  The boundary is
     displayed using an overline in Emacs 21.  With prefix argument ARG,
     turn on if positive, otherwise off.  The minor mode can be turned
     on only if semantic feature is available and the current buffer was
     set up for parsing.  Return non-'nil' if the minor mode is enabled.

Another interesting mode helps if you are worred about specific
attributes, you can se this minor mode to highlight different tokens in
different ways based on the attributes you are most concerned with.

 -- Command: semantic-highlight-by-attribute-mode &optional arg
     Minor mode to highlight tags based on some attribute.  By default,
     the protection of a tag will give it a different background color.

     With prefix argument ARG, turn on if positive, otherwise off.  The
     minor mode can be turned on only if semantic feature is available
     and the current buffer was set up for parsing.  Return non-'nil' if
     the minor mode is enabled.

Another tool that can be used is a dump of the current list of tags.
This shows the actual Lisp representation of the tags generated in a
rather bland dump.  This can be useful if text was successfully parsed,
and you want to be sure that the correct information was captured.

 -- Command: bovinate &optional clear
     Bovinate the current buffer.  Show output in a temp buffer.
     Optional argument CLEAR will clear the cache before bovinating.  If
     CLEAR is negative, it will do a full reparse, and also not display
     the output buffer.

\1f
File: semantic-langdev.info,  Node: Bovine Parser Debugging,  Next: Wisent Parser Debugging,  Prev: Parser Output tools,  Up: Debugging

6.3 Bovine Parser Debugging
===========================

The bovine parser is described in *note (bovine)top::.

Asside using a traditional Emacs Lisp debugger on functions you provide
for token expansion, there is one other means of debugging which
interactively steps over the rules in your grammar file.

 -- Command: semantic-debug
     Parse the current buffer and run in debug mode.

Once the parser is activated in this mode, the current tag cache is
flushed, and the parser started.  At each stage in the LALR parser, the
current rule, and match step is highlighted in your parser source
buffer.  In a second window, the text being parsed is shown, and the
lexical token found is highlighted.  A clue of the current stack of
saved data is displayed in the minibuffer.

There is a wide range of keybindings that can be used to execute code in
your buffer.  (Not all are implemented.)

'n'
'SPC'
     Next.
's'
     Step.
'u'
     Up.  (Not implemented yet.)
'd'
     Down.  (Not implemented yet.)
'f'
     Fail Match.  Pretend the current match element and the token in the
     buffer is a failed match, even if it is not.
'h'
     Print information about the current parser state.
's'
     Jump to to the source buffer.
'p'
     Jump to the parser buffer.
'q'
     Quit.  Exits this debug session and the parser.
'a'
     Abort.  Aborts one level of the parser, possibly exiting the
     debugger.
'g'
     Go.  Stop debugging, and just start parsing.
'b'
     Set Breakpoint.  (Not implemented yet.)
'e'
     'eval-expression'.  Lets you execute some random Emacs Lisp
     command.

Note: While the core of 'semantic-debug' is a generic debugger interface
for rule based grammars, only the bovine parser has a specific backend
implementation.  If someone wants to implement a debugger backend for
wisent, that would be spiff.

\1f
File: semantic-langdev.info,  Node: Wisent Parser Debugging,  Next: Overlay Debugging,  Prev: Bovine Parser Debugging,  Up: Debugging

6.4 Wisent Parser Debugging
===========================

Wisent does not implement a backend for 'semantic-debug', it does have
some debugging commands the rule actions.  You can read about them in
the wisent manual.

*note (wisent)Grammar Debugging::

\1f
File: semantic-langdev.info,  Node: Overlay Debugging,  Next: Incremental Parser Debugging,  Prev: Wisent Parser Debugging,  Up: Debugging

6.5 Overlay Debugging
=====================

Once a buffer has been parsed into a tag table, the next most important
step is getting those tags activated for a buffer, and storable in a
'semanticdb' backend.  *note (semantic-appdev)semanticdb::.

These two activities depend on the ability of every tag in the table to
be linked and unlinked to the current buffer with an overlay.  *note
(Tag Overlay)semantic-appdev:: *note (Tag Hooks)semantic-appdev::

In this case, the most important function that must be written is:

 -- Function: semantic-tag-components-with-overlays tag
     Return the list of top level components belonging to TAG.  Children
     are any sub-tags which contain overlays.

     Default behavior is to get "semantic-tag-components" in addition to
     the components of an anonymous types (if applicable.)

     Note for language authors: If a mode defines a language tag that
     has tags in it with overlays you should still return them with this
     function.  Ignoring this step will prevent several features from
     working correctly.  This function can be overriden in semantic
     using the symbol 'tag-components-with-overlays'.

If your are successfully building a tag table, and errors occur saving
or restoring tags from semanticdb, this is the most likely cause of the
problem.

\1f
File: semantic-langdev.info,  Node: Incremental Parser Debugging,  Next: Debugging Analysis,  Prev: Overlay Debugging,  Up: Debugging

6.6 Incremental Parser Debugging
================================

The incremental parser is a highly complex engine for quickly refreshing
the tag table of a buffer after some set of changes have been made to
that buffer by a user.

There is no debugger or interface to the incremental parser, however
there are a few minor modes which can help you identify issues if you
think there are problems while incrementally parsing a buffer.

The first stage of the incremental parser is in tracking the changes the
user makes to a buffer.  You can visibly track these changes too.

 -- Command: semantic-highlight-edits-mode &optional arg
     Minor mode for highlighting changes made in a buffer.  Changes are
     tracked by semantic so that the incremental parser can work
     properly.  This mode will highlight those changes as they are made,
     and clear them when the incremental parser accounts for those
     edits.  With prefix argument ARG, turn on if positive, otherwise
     off.  The minor mode can be turned on only if semantic feature is
     available and the current buffer was set up for parsing.  Return
     non-'nil' if the minor mode is enabled.

Another important aspect of the incremental parser involves tracking the
current parser state of the buffer.  You can track this state also.

 -- Command: semantic-show-parser-state-mode &optional arg
     Minor mode for displaying parser cache state in the modeline.  The
     cache can be in one of three states.  They are Up to date, Partial
     reprase needed, and Full reparse needed.  The state is indicated in
     the modeline with the following characters:
     '-'
          The cache is up to date.
     '!'
          The cache requires a full update.
     '^'
          The cache needs to be incrementally parsed.
     '%'
          The cache is not currently parseable.
     '@'
          Auto-parse in progress (not set here.)

     With prefix argument ARG, turn on if positive, otherwise off.  The
     minor mode can be turned on only if semantic feature is available
     and the current buffer was set up for parsing.  Return non-'nil' if
     the minor mode is enabled.

When the incremental parser starts updating the tags buffer, you can
also enable a set of messages to help identify how the incremental
parser is merging changes with the main buffer.

 -- Variable: semantic-edits-verbose-flag
     Non-'nil' means the incremental perser is verbose.  If 'nil',
     errors are still displayed, but informative messages are not.

\1f
File: semantic-langdev.info,  Node: Debugging Analysis,  Next: Semantic 1.4 Doc,  Prev: Incremental Parser Debugging,  Up: Debugging

6.7 Debugging Analysis
======================

The semantic analyzer is a at the top of the food chain when it comes to
semantic service functionality.  The semantic support for a language
must be absolute before analysis will work property.

A good way to test analysis is by placing the cursor in different
places, and requesting a dump of the context.

 -- Command: semantic-analyze-current-context position
     Analyze the current context at POSITION.  If called interactively,
     display interesting information about POSITION in a separate
     buffer.  Returns an object based on symbol
     "semantic-analyze-context".

*note Semantic Analyzer Support:: *note (semantic-user)Analyzer::

\1f
File: semantic-langdev.info,  Node: Semantic 1.4 Doc,  Prev: Debugging Analysis,  Up: Debugging

6.8 Semantic 1.4 Doc
====================

In semantic 1.4 the following documentation was written for debugging.
I'm leaving in here until better doc for 2.0 is done.

Writing language files using BY is significantly easier than writing
then using regular expressions in a functional manner.  Debugging them,
however, can still prove challenging.

There are two ways to debug a language definition if it is not behaving
as expected.  One way is to debug against the source '.by' file.

If your language definition was written in BNF notation, debugging is
quite easy.  The command 'semantic-debug' will start you off.

 -- Command: semantic-debug
     Reprase the current buffer and run in parser debug mode.

While debugging, two windows are visible.  One window shows the file
being parsed, and the syntactic token being tested is highlighted.  The
second window shows the table being used (in the BY source) with the
current rule highlighted.  The cursor will sit on the specific match
rule being tested against.

In the minibuffer, a brief summary of the current situation is listed.
The first element is the syntactic token which is a list of the form:

     (TYPE START . END)

The rest of the display is a list of all strings collected for the
currently tested rule.  Each time a new rule is entered, the list is
restarted.  Upon returning from a rule into a previous match list, the
previous match list is restored, with the production of the dependent
rule in the list.

Use 'C-g' to stop debugging.  There are no commands for any fancier
types of debugging.

NOTE: Semantic 2.0 has more debugging commands.  Use: 'C-h m
semantic-debug-mode' to view.

\1f
File: semantic-langdev.info,  Node: Parser Error Handling,  Next: GNU Free Documentation License,  Prev: Debugging,  Up: Top

7 Parser Error Handling
***********************

NOTE: Write Me

\1f
File: semantic-langdev.info,  Node: GNU Free Documentation License,  Next: Index,  Prev: Parser Error Handling,  Up: Top

Appendix A GNU Free Documentation License
*****************************************

                        Version 1.1, March 2000

     Copyright (C) 2000  Free Software Foundation, Inc.
     51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA

     Everyone is permitted to copy and distribute verbatim copies
     of this license document, but changing it is not allowed.

  0. PREAMBLE

     The purpose of this License is to make a manual, textbook, or other
     written document "free" in the sense of freedom: to assure everyone
     the effective freedom to copy and redistribute it, with or without
     modifying it, either commercially or noncommercially.  Secondarily,
     this License preserves for the author and publisher a way to get
     credit for their work, while not being considered responsible for
     modifications made by others.

     This License is a kind of "copyleft", which means that derivative
     works of the document must themselves be free in the same sense.
     It complements the GNU General Public License, which is a copyleft
     license designed for free software.

     We have designed this License in order to use it for manuals for
     free software, because free software needs free documentation: a
     free program should come with manuals providing the same freedoms
     that the software does.  But this License is not limited to
     software manuals; it can be used for any textual work, regardless
     of subject matter or whether it is published as a printed book.  We
     recommend this License principally for works whose purpose is
     instruction or reference.


  1. APPLICABILITY AND DEFINITIONS

     This License applies to any manual or other work that contains a
     notice placed by the copyright holder saying it can be distributed
     under the terms of this License.  The "Document", below, refers to
     any such manual or work.  Any member of the public is a licensee,
     and is addressed as "you".

     A "Modified Version" of the Document means any work containing the
     Document or a portion of it, either copied verbatim, or with
     modifications and/or translated into another language.

     A "Secondary Section" is a named appendix or a front-matter section
     of the Document that deals exclusively with the relationship of the
     publishers or authors of the Document to the Document's overall
     subject (or to related matters) and contains nothing that could
     fall directly within that overall subject.  (For example, if the
     Document is in part a textbook of mathematics, a Secondary Section
     may not explain any mathematics.)  The relationship could be a
     matter of historical connection with the subject or with related
     matters, or of legal, commercial, philosophical, ethical or
     political position regarding them.

     The "Invariant Sections" are certain Secondary Sections whose
     titles are designated, as being those of Invariant Sections, in the
     notice that says that the Document is released under this License.

     The "Cover Texts" are certain short passages of text that are
     listed, as Front-Cover Texts or Back-Cover Texts, in the notice
     that says that the Document is released under this License.

     A "Transparent" copy of the Document means a machine-readable copy,
     represented in a format whose specification is available to the
     general public, whose contents can be viewed and edited directly
     and straightforwardly with generic text editors or (for images
     composed of pixels) generic paint programs or (for drawings) some
     widely available drawing editor, and that is suitable for input to
     text formatters or for automatic translation to a variety of
     formats suitable for input to text formatters.  A copy made in an
     otherwise Transparent file format whose markup has been designed to
     thwart or discourage subsequent modification by readers is not
     Transparent.  A copy that is not "Transparent" is called "Opaque".

     Examples of suitable formats for Transparent copies include plain
     ASCII without markup, Texinfo input format, LaTeX input format,
     SGML or XML using a publicly available DTD, and standard-conforming
     simple HTML designed for human modification.  Opaque formats
     include PostScript, PDF, proprietary formats that can be read and
     edited only by proprietary word processors, SGML or XML for which
     the DTD and/or processing tools are not generally available, and
     the machine-generated HTML produced by some word processors for
     output purposes only.

     The "Title Page" means, for a printed book, the title page itself,
     plus such following pages as are needed to hold, legibly, the
     material this License requires to appear in the title page.  For
     works in formats which do not have any title page as such, "Title
     Page" means the text near the most prominent appearance of the
     work's title, preceding the beginning of the body of the text.

  2. VERBATIM COPYING

     You may copy and distribute the Document in any medium, either
     commercially or noncommercially, provided that this License, the
     copyright notices, and the license notice saying this License
     applies to the Document are reproduced in all copies, and that you
     add no other conditions whatsoever to those of this License.  You
     may not use technical measures to obstruct or control the reading
     or further copying of the copies you make or distribute.  However,
     you may accept compensation in exchange for copies.  If you
     distribute a large enough number of copies you must also follow the
     conditions in section 3.

     You may also lend copies, under the same conditions stated above,
     and you may publicly display copies.

  3. COPYING IN QUANTITY

     If you publish printed copies of the Document numbering more than
     100, and the Document's license notice requires Cover Texts, you
     must enclose the copies in covers that carry, clearly and legibly,
     all these Cover Texts: Front-Cover Texts on the front cover, and
     Back-Cover Texts on the back cover.  Both covers must also clearly
     and legibly identify you as the publisher of these copies.  The
     front cover must present the full title with all words of the title
     equally prominent and visible.  You may add other material on the
     covers in addition.  Copying with changes limited to the covers, as
     long as they preserve the title of the Document and satisfy these
     conditions, can be treated as verbatim copying in other respects.

     If the required texts for either cover are too voluminous to fit
     legibly, you should put the first ones listed (as many as fit
     reasonably) on the actual cover, and continue the rest onto
     adjacent pages.

     If you publish or distribute Opaque copies of the Document
     numbering more than 100, you must either include a machine-readable
     Transparent copy along with each Opaque copy, or state in or with
     each Opaque copy a publicly-accessible computer-network location
     containing a complete Transparent copy of the Document, free of
     added material, which the general network-using public has access
     to download anonymously at no charge using public-standard network
     protocols.  If you use the latter option, you must take reasonably
     prudent steps, when you begin distribution of Opaque copies in
     quantity, to ensure that this Transparent copy will remain thus
     accessible at the stated location until at least one year after the
     last time you distribute an Opaque copy (directly or through your
     agents or retailers) of that edition to the public.

     It is requested, but not required, that you contact the authors of
     the Document well before redistributing any large number of copies,
     to give them a chance to provide you with an updated version of the
     Document.

  4. MODIFICATIONS

     You may copy and distribute a Modified Version of the Document
     under the conditions of sections 2 and 3 above, provided that you
     release the Modified Version under precisely this License, with the
     Modified Version filling the role of the Document, thus licensing
     distribution and modification of the Modified Version to whoever
     possesses a copy of it.  In addition, you must do these things in
     the Modified Version:

     A. Use in the Title Page (and on the covers, if any) a title
     distinct from that of the Document, and from those of previous
     versions (which should, if there were any, be listed in the History
     section of the Document).  You may use the same title as a previous
     version if the original publisher of that version gives permission.
     B. List on the Title Page, as authors, one or more persons or
     entities responsible for authorship of the modifications in the
     Modified Version, together with at least five of the principal
     authors of the Document (all of its principal authors, if it has
     less than five).
     C. State on the Title page the name of the publisher of the
     Modified Version, as the publisher.
     D. Preserve all the copyright notices of the Document.
     E. Add an appropriate copyright notice for your modifications
     adjacent to the other copyright notices.
     F. Include, immediately after the copyright notices, a license
     notice giving the public permission to use the Modified Version
     under the terms of this License, in the form shown in the Addendum
     below.
     G. Preserve in that license notice the full lists of Invariant
     Sections and required Cover Texts given in the Document's license
     notice.
     H. Include an unaltered copy of this License.
     I. Preserve the section entitled "History", and its title, and add
     to it an item stating at least the title, year, new authors, and
     publisher of the Modified Version as given on the Title Page.  If
     there is no section entitled "History" in the Document, create one
     stating the title, year, authors, and publisher of the Document as
     given on its Title Page, then add an item describing the Modified
     Version as stated in the previous sentence.
     J. Preserve the network location, if any, given in the Document for
     public access to a Transparent copy of the Document, and likewise
     the network locations given in the Document for previous versions
     it was based on.  These may be placed in the "History" section.
     You may omit a network location for a work that was published at
     least four years before the Document itself, or if the original
     publisher of the version it refers to gives permission.
     K. In any section entitled "Acknowledgements" or "Dedications",
     preserve the section's title, and preserve in the section all the
     substance and tone of each of the contributor acknowledgements
     and/or dedications given therein.
     L. Preserve all the Invariant Sections of the Document, unaltered
     in their text and in their titles.  Section numbers or the
     equivalent are not considered part of the section titles.
     M. Delete any section entitled "Endorsements".  Such a section may
     not be included in the Modified Version.
     N. Do not retitle any existing section as "Endorsements" or to
     conflict in title with any Invariant Section.

     If the Modified Version includes new front-matter sections or
     appendices that qualify as Secondary Sections and contain no
     material copied from the Document, you may at your option designate
     some or all of these sections as invariant.  To do this, add their
     titles to the list of Invariant Sections in the Modified Version's
     license notice.  These titles must be distinct from any other
     section titles.

     You may add a section entitled "Endorsements", provided it contains
     nothing but endorsements of your Modified Version by various
     parties-for example, statements of peer review or that the text has
     been approved by an organization as the authoritative definition of
     a standard.

     You may add a passage of up to five words as a Front-Cover Text,
     and a passage of up to 25 words as a Back-Cover Text, to the end of
     the list of Cover Texts in the Modified Version.  Only one passage
     of Front-Cover Text and one of Back-Cover Text may be added by (or
     through arrangements made by) any one entity.  If the Document
     already includes a cover text for the same cover, previously added
     by you or by arrangement made by the same entity you are acting on
     behalf of, you may not add another; but you may replace the old
     one, on explicit permission from the previous publisher that added
     the old one.

     The author(s) and publisher(s) of the Document do not by this
     License give permission to use their names for publicity for or to
     assert or imply endorsement of any Modified Version.

  5. COMBINING DOCUMENTS

     You may combine the Document with other documents released under
     this License, under the terms defined in section 4 above for
     modified versions, provided that you include in the combination all
     of the Invariant Sections of all of the original documents,
     unmodified, and list them all as Invariant Sections of your
     combined work in its license notice.

     The combined work need only contain one copy of this License, and
     multiple identical Invariant Sections may be replaced with a single
     copy.  If there are multiple Invariant Sections with the same name
     but different contents, make the title of each such section unique
     by adding at the end of it, in parentheses, the name of the
     original author or publisher of that section if known, or else a
     unique number.  Make the same adjustment to the section titles in
     the list of Invariant Sections in the license notice of the
     combined work.

     In the combination, you must combine any sections entitled
     "History" in the various original documents, forming one section
     entitled "History"; likewise combine any sections entitled
     "Acknowledgements", and any sections entitled "Dedications".  You
     must delete all sections entitled "Endorsements."

  6. COLLECTIONS OF DOCUMENTS

     You may make a collection consisting of the Document and other
     documents released under this License, and replace the individual
     copies of this License in the various documents with a single copy
     that is included in the collection, provided that you follow the
     rules of this License for verbatim copying of each of the documents
     in all other respects.

     You may extract a single document from such a collection, and
     distribute it individually under this License, provided you insert
     a copy of this License into the extracted document, and follow this
     License in all other respects regarding verbatim copying of that
     document.

  7. AGGREGATION WITH INDEPENDENT WORKS

     A compilation of the Document or its derivatives with other
     separate and independent documents or works, in or on a volume of a
     storage or distribution medium, does not as a whole count as a
     Modified Version of the Document, provided no compilation copyright
     is claimed for the compilation.  Such a compilation is called an
     "aggregate", and this License does not apply to the other
     self-contained works thus compiled with the Document, on account of
     their being thus compiled, if they are not themselves derivative
     works of the Document.

     If the Cover Text requirement of section 3 is applicable to these
     copies of the Document, then if the Document is less than one
     quarter of the entire aggregate, the Document's Cover Texts may be
     placed on covers that surround only the Document within the
     aggregate.  Otherwise they must appear on covers around the whole
     aggregate.

  8. TRANSLATION

     Translation is considered a kind of modification, so you may
     distribute translations of the Document under the terms of section
     4.  Replacing Invariant Sections with translations requires special
     permission from their copyright holders, but you may include
     translations of some or all Invariant Sections in addition to the
     original versions of these Invariant Sections.  You may include a
     translation of this License provided that you also include the
     original English version of this License.  In case of a
     disagreement between the translation and the original English
     version of this License, the original English version will prevail.

  9. TERMINATION

     You may not copy, modify, sublicense, or distribute the Document
     except as expressly provided for under this License.  Any other
     attempt to copy, modify, sublicense or distribute the Document is
     void, and will automatically terminate your rights under this
     License.  However, parties who have received copies, or rights,
     from you under this License will not have their licenses terminated
     so long as such parties remain in full compliance.

  10. FUTURE REVISIONS OF THIS LICENSE

     The Free Software Foundation may publish new, revised versions of
     the GNU Free Documentation License from time to time.  Such new
     versions will be similar in spirit to the present version, but may
     differ in detail to address new problems or concerns.  See
     http://www.gnu.org/copyleft/.

     Each version of the License is given a distinguishing version
     number.  If the Document specifies that a particular numbered
     version of this License "or any later version" applies to it, you
     have the option of following the terms and conditions either of
     that specified version or of any later version that has been
     published (not as a draft) by the Free Software Foundation.  If the
     Document does not specify a version number of this License, you may
     choose any version ever published (not as a draft) by the Free
     Software Foundation.

ADDENDUM: How to use this License for your documents
====================================================

To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and license
notices just after the title page:


       Copyright (C)  YEAR  YOUR NAME.
       Permission is granted to copy, distribute and/or modify this document
       under the terms of the GNU Free Documentation License, Version 1.1
       or any later version published by the Free Software Foundation;
       with the Invariant Sections being LIST THEIR TITLES, with the
       Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
       A copy of the license is included in the section entitled ``GNU
       Free Documentation License''.
If you have no Invariant Sections, write "with no Invariant Sections"
instead of saying which ones are invariant.  If you have no Front-Cover
Texts, write "no Front-Cover Texts" instead of "Front-Cover Texts being
LIST"; likewise for Back-Cover Texts.

If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of free
software license, such as the GNU General Public License, to permit
their use in free software.

\1f
File: semantic-langdev.info,  Node: Index,  Prev: GNU Free Documentation License,  Up: Top

Index
*****

\0\b[index\0\b]
* Menu:

* bovinate:                              Parser Output tools.  (line 61)
* define-lex:                            Lexer Construction.   (line 12)
* define-lex-analyzer:                   Lexer Analyzer Construction.
                                                               (line  9)
* define-lex-block-analyzer:             Lexer Analyzer Construction.
                                                               (line 60)
* define-lex-regex-analyzer:             Lexer Analyzer Construction.
                                                               (line 34)
* define-lex-simple-regex-analyzer:      Lexer Analyzer Construction.
                                                               (line 39)
* function overloading:                  Semantic Overload Mechanism.
                                                               (line 17)
* Language Support Overview:             Language Support Overview.
                                                               (line  6)
* overloading, function:                 Semantic Overload Mechanism.
                                                               (line 17)
* Parser Error Handling:                 Parser Error Handling.
                                                               (line  6)
* semantic-analyze-current-context:      Debugging Analysis.   (line 13)
* semantic-clear-toplevel-cache:         Parsing a language file.
                                                               (line 34)
* semantic-debug:                        Bovine Parser Debugging.
                                                               (line 12)
* semantic-debug <1>:                    Semantic 1.4 Doc.     (line 19)
* semantic-edits-verbose-flag:           Incremental Parser Debugging.
                                                               (line 55)
* semantic-fetch-available-tags:         Parsing a language file.
                                                               (line 24)
* semantic-fetch-tags:                   Parsing a language file.
                                                               (line 10)
* semantic-highlight-by-attribute-mode:  Parser Output tools.  (line 47)
* semantic-highlight-edits-mode:         Incremental Parser Debugging.
                                                               (line 17)
* semantic-lex:                          Writing Lexers.       (line 19)
* semantic-lex-beginning-of-line:        Lexer Built In Analyzers.
                                                               (line 10)
* semantic-lex-charquote:                Lexer Built In Analyzers.
                                                               (line 68)
* semantic-lex-close-paren:              Lexer Built In Analyzers.
                                                               (line 86)
* semantic-lex-comments:                 Lexer Built In Analyzers.
                                                               (line 92)
* semantic-lex-comments-as-whitespace:   Lexer Built In Analyzers.
                                                               (line 95)
* semantic-lex-debug:                    Lexical Debugging.    (line 12)
* semantic-lex-default-action:           Lexer Built In Analyzers.
                                                               (line  6)
* semantic-lex-ignore-comments:          Lexer Built In Analyzers.
                                                               (line 98)
* semantic-lex-ignore-newline:           Lexer Built In Analyzers.
                                                               (line 21)
* semantic-lex-ignore-whitespace:        Lexer Built In Analyzers.
                                                               (line 29)
* semantic-lex-keyword-get:              Keywords.             (line 21)
* semantic-lex-keyword-p:                Keywords.             (line 14)
* semantic-lex-keyword-put:              Keywords.             (line 18)
* semantic-lex-keywords:                 Keywords.             (line 29)
* semantic-lex-map-keywords:             Keywords.             (line 24)
* semantic-lex-newline:                  Lexer Built In Analyzers.
                                                               (line 13)
* semantic-lex-newline-as-whitespace:    Lexer Built In Analyzers.
                                                               (line 16)
* semantic-lex-number:                   Lexer Built In Analyzers.
                                                               (line 32)
* semantic-lex-number-expression:        Lexer Built In Analyzers.
                                                               (line 36)
* semantic-lex-open-paren:               Lexer Built In Analyzers.
                                                               (line 83)
* semantic-lex-paren-or-list:            Lexer Built In Analyzers.
                                                               (line 79)
* semantic-lex-punctuation:              Lexer Built In Analyzers.
                                                               (line 71)
* semantic-lex-punctuation-type:         Lexer Built In Analyzers.
                                                               (line 74)
* semantic-lex-string:                   Lexer Built In Analyzers.
                                                               (line 89)
* semantic-lex-symbol-or-keyword:        Lexer Built In Analyzers.
                                                               (line 65)
* semantic-lex-whitespace:               Lexer Built In Analyzers.
                                                               (line 26)
* semantic-show-parser-state-mode:       Incremental Parser Debugging.
                                                               (line 30)
* semantic-show-tag-boundaries-mode:     Parser Output tools.  (line 36)
* semantic-show-unmatched-syntax-mode:   Parser Output tools.  (line 13)
* semantic-tag-components-with-overlays: Overlay Debugging.    (line 16)
* semantic-tag-expand-function:          Tag Expansion.        (line  9)
* Tag Structure:                         Tag Structure.        (line  6)
* Writing Lexers:                        Writing Lexers.       (line  6)
* Writing Parsers:                       Writing Parsers.      (line  6)


\1f
Tag Table:
Node: Top\7f1132
Node: Tag Structure\7f2340
Node: Language Support Overview\7f4214
Node: Semantic Overload Mechanism\7f5510
Node: Semantic Parser Structure\7f10651
Node: Application API Structure\7f10956
Node: Semantic Analyzer Support\7f12514
Node: Writing Lexers\7f12710
Ref: semantic-lex\7f13504
Node: Lexer Overview\7f14716
Node: Lexer Output\7f15974
Node: Lexer Construction\7f19831
Ref: define-lex\7f20293
Node: Lexer Built In Analyzers\7f21142
Node: Lexer Analyzer Construction\7f24639
Node: Keywords\7f28814
Node: Keyword Properties\7f30388
Node: Writing Parsers\7f31265
Node: External Parsers\7f32749
Node: Grammar Programming Environment\7f33095
Node: Parsing a language file\7f33688
Ref: semantic-fetch-tags\7f34062
Ref: semantic-fetch-available-tags\7f34832
Ref: semantic-clear-toplevel-cache\7f35282
Node: Parser Backend Support\7f35440
Node: Example Backend File\7f36243
Node: Tag Expansion\7f38554
Ref: semantic-tag-expand-function\7f38883
Node: Debugging\7f40533
Node: Lexical Debugging\7f41125
Ref: semantic-lex-debug\7f41559
Node: Parser Output tools\7f42032
Ref: semantic-show-unmatched-syntax-mode\7f42531
Ref: semantic-show-tag-boundaries-mode\7f43467
Ref: semantic-highlight-by-attribute-mode\7f44100
Ref: bovinate\7f44810
Node: Bovine Parser Debugging\7f45045
Ref: semantic-debug\7f45520
Node: Wisent Parser Debugging\7f47022
Node: Overlay Debugging\7f47412
Ref: semantic-tag-components-with-overlays\7f48133
Node: Incremental Parser Debugging\7f48875
Ref: semantic-highlight-edits-mode\7f49646
Ref: semantic-show-parser-state-mode\7f50381
Ref: semantic-edits-verbose-flag\7f51399
Node: Debugging Analysis\7f51534
Ref: semantic-analyze-current-context\7f52081
Node: Semantic 1.4 Doc\7f52368
Node: Parser Error Handling\7f54129
Node: GNU Free Documentation License\7f54322
Node: Index\7f73980
\1f
End Tag Table