This file documents the emodule loading technology of SXEmacs.

Copyright © 1998 J. Kean Johnston. Copyright © 2007 Sebastian Freundt. Copyright © 2008 Steve Youngs.

Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that this permission notice may be stated in a translation approved by the Foundation.

Permission is granted to copy and distribute modified versions of this manual under the conditions for verbatim copying, provided also that the section entitled ``GNU General Public License'' is included exactly as in the original, and provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modified versions, except that the section entitled ``GNU General Public License'' may be included in a translation approved by the Free Software Foundation instead of in the original English.

This Info file contains v3.0 of the SXEmacs dynamic loadable emodule support documentation.

Introduction
************

SXEmacs is a powerful, extensible editor.  The traditional way of extending the functionality of SXEmacs is to use its built-in Lisp language (called Emacs Lisp, or elisp for short).  However, while elisp is a full programming language and capable of extending SXEmacs in more ways than you can imagine, it does have its short-comings.

Firstly, elisp is an interpreted language, and this has serious speed implications.  Like all other interpreted languages (like Java), elisp is often suitable only for certain types of application or extension. So although elisp is a general purpose language, and very high level, there are times when it is desirable to descend to a lower level compiled language for speed purposes.

Secondly, elisp (or Lisp in general) is not a very common language any more, except for certain circles in the computer industry.  C is a far more commonly known language, and because it is compiled, more suited to a wider range of applications, especially those that require low level access to a system or need to be as quick as possible.

This manual describes a way of extending SXEmacs using dynamic sharable objects (DSOs aka dynamic loadable modules aka dynamically loadable libraries, or just simply shared objects), which can be written in C or C++ (or more generally in anything that can be compiled to native object code on platforms that support shared objects).  After the build, they can be loaded into SXEmacs at any time.

SXEmacs emodule support is detected and enabled during configure time by default on all systems that support loading of shared objects, provided libtool and libltdl is properly installed.  From a users perspective, the internals of SXEmacs emodules are irrelevant. All a user will ever need to know about shared objects is the name of the shared object when they want to load a given emodule.  From a developers perspective though, a lot more is provided.

The rest of this document will discuss the actual mechanics of SXEmacs emodules and work through several of the samples.  Please be sure that you have read the SXEmacs Internals Manual and understand everything in it.  The concepts there apply to all emodules.  This document may have some overlap, but it is the internals manual which should be considered the final authority.  It will also help a great deal to look at the actual SXEmacs source code to see how things are done.

Anatomy of a Emodule
********************

A dynamically loadable SXEmacs extension (hereafter referred to as a emodule) can carry certain special pieces of information and functions which are then used in a predefined special way.  This chapter describes the basic layout of such a emodule, and provides a very simple sample.

Special Header Files
====================

Every emodule better includes the file emodules-ng.h.  This will primarily set up certain vital macros.  If you want to stay close to the rest of the SXEmacs binary, for example provide lisp bindings of your stuff or implement functionality depending on whether or not SXEmacs was configured with a certain feature, you should have a thorough glance at sxemacs.h.

Most emodules will probably require some pre-processor conditionals based on constants defined in config.h which is included automatically upon inclusion of sxemacs.h.  This file is automatically generated during the configure phase of SXEmacs, its prototype is config.h.in which in turn is autogenerated from various macros in the m4/ directory and configure.ac.

Depending on what your emodule will be doing at last, you will probably need to include other header files as well.

Recognised Functions
====================

A typical emodule will implement a fancy feature (or hopefully many thereof) and thence be willing to provide a lisp binding in the surface, e.g. a function, or anything similar to make use of the feature.  Sounds simple, is simple: SXEmacs just asks its inbuilt crystal ball to determine which of your functions was meant to appear in the lisp engine.

Albeit tempting with respect to simplicity and comfort this behaviour is technically an effort to realise.  SXEmacs therefore treats some symbols of your emodule in a special way.

 -- Function: init
     Code which is run after opening the emodule and all its requisite modules.

     Usually this carries all the bindings you want to export to the lisp surface, such as macros, functions, variables and (lisp) symbols. Moreover, depending on your code this can be used to initialise private setups, such as private tables or memory blocks.

 -- Function: reinit
     Code which is run upon a reload of the emodule.

     Normally this is just a deinit(); call followed by another init(); call.  The reason to have a special entry point for reinitialisation is that, roughly, reinit() is not exactly just a deinit(); init(); sequence.  For instance you might want to avoid to initialise certain private memory blocks again because their setup was a very expensive operation.

 -- Function: deinit
     Code which is run after closing all of a emodule's requisites upon a unload-module request.

     Usually you would unleash all your privately held resources and unbind any exported variables, functions, macros, and so forth.

 -- Function: docs
     A special function called to incorporate documentation strings for your lisp bindings.

     Normally this entry point is generated automatically using the make-docfile utility.

All these function are truly optional.  Their signature is actually

     void(*)(emodng_t)

but you can also make them

     void(*)(void)

Quick notes re emodule unloading.  It seems quite natural to be able to unload stuff, at least in the perfect world.  While this is still true if your emodule provides truly complementary functionality (whatever that is) it becomes a critical issue once you truly intervene or even replace internal functionality.  We are NOT barring you from unloading your emodule nor are we saying you should avoid that feature but you should be aware of its dramatical consequences when done sloppily.

The best comparison we can come up with is the concept of emodules for the linux kernel.  Imagine you were a emodule providing ultra fast access to, say, scsi drives and therefore had to oust the kernel's normal handlers for those drives.  Now after unloading your emodule you would reestablish the old, superseded handlers, would you not?  As mentioned before, today's system are usually not equipped with a superior intelligence being able to completely understand your concept and your mode of operation in order to automatically withdraw, or simply revert, your actions just by looking at the emodule's code.

Ergo YOU are responsible to restore the state of the system!

It does not necessarily mean that you have to unwind completely to the exact state before you entered the scene, no, you are `only' encouraged to leave a working system behind, whatever that means in the context of your emodule.

  * Carefully consider if you want to unbind exported lisp variables.

     If they are not filled with special lisp objects only your emodule knows about it is definitely wise to leave them in the environment.

  * Carefully consider if you want to unbind exported macros.

     After all a macro is just a piece of code which produces another piece of code under the influence of some variables passed to it, so why not simply keep that?

  * Carefully consider if you want to unbind exported functions.

     This is a bit more delicate.  The lisp language does not distinguish between code and data and your function could have ended up literally anywhere.  Simply unbinding your functions, using UNDEFSUBR or Ffmakunbound, will possibly turn any reference to them into an error generator.  This is especially annoying when your emodule's functions were used in so frequented hooks like pre-gc-hook.

     Solution: Do not unbind your functions.  Instead rebind them to something which has the same input signature but, for instance, produces an unmistakable message.  Be prolific!

  * Any other scenario not covered here is basically either a real auxiliary, read optionally, thing where unloading does not harm at all, or it really affects the guts of SXEmacs in which case we take it for granted that you know what you are doing anyway.

Recognised Variables
====================

Basically there is only one specially-treated variable.

 -- Variable: dependencies
     A const char * array which lists the names of all requisite modules and is terminated with a NULL entry.  These are opened and initialised before the actual emodule is initialised.

Like the specially-treated functions above, this variable is also optional.  Moreover, the same naming policies apply, and most importantly there is a convenience macro which hides the raw C work you had to do.

 -- macro: REQUIRE name &rest names
     Proclaim the requisite emodules of NAME.  Set up and fill the dependencies variable from NAMES as discussed above.

Loading other Emodules
======================

Sometimes it is necessary to use functions, or generally symbols, defined in a foreign emodule you depend on.  In general it is impossible to open (as in dlopen()) your emodule to extract the dependencies variable when you refer to one of these external symbols in your code.  The corresponding error message goes along the lines of

     Cannot open modules foo: Undefined symbol bar.

So what happened?

The emodule itself can contain dynamic references to other libraries, or, most likely, symbols from the SXEmacs binary.  However, they have to be resolvable, and in fact are tried to be resolved, when opening a module.  Now if you refer to symbols defined in another emodule (This was not possible with the old emodule system) you, the user, have to load that other emodule before.

So what's the rant about that DEPENDENCIES thingiedingie?

As mentioned above the new load-module function will exactly relieve you, the user, of this task provided the emodule proclaimed the list of its dependencies.  Now this seems to be a charade, in order to open the emodule to read its dependencies you, the user, have to load all of its dependencies because naively opening the emodule will yield nothing but the error above when foreign symbols are used.

The solution is to provide a, sort of, low-quality draught version of the foreign functions you, the emodule author, refer to.  The missing link here are the so-called weak symbols.  They usually do not carry any real information (variable) or functionality (function) and are meant to be overwritten by the ``real'' symbol as soon as possible.  When properly coded their life-time spans just about the time from opening the emodule to opening the requisite module which, hopefully, defines real symbol and thence replaces the weak one.

Imagine module-A is doing all the hard work and provides the function

     /* inside module-A.c */
     bool solve_complex_problem(void)
     {
             ...
             return true;
     }

Now module-B contains a function void compute_lotto_numbers(void) which goes:

     /* inside module-B.c */
     void compute_lotto_numbers(void)
     {
             if (solve_complex_problem()) {
                     /* lotto numbers found, hooray! print them */
                     ...
             }
             ...
     }

Leaving it like that will effectively bar your users from computing the lotto numbers and I personally would hate you, the emodule developer, for the rest of my life.  However, to finish this example using weak symbols you would additionally put into module-B.c:

     bool solve_complex_problem(void) __attribute__((weak));

     bool solve_complex_problem(void)
     {
             return false;
     }

Some practical hints:

  * If you have included sxemacs.h already you can use its WEAK and WEAK_EXTERN macros to hide the compiler specific attribute declaration.

  * This magic spell can also be applied to C variables.

  * Do not simply make your weak version of a function a no-op, rather make it notify your users or yourself that the weak version is still in effect.  This can save you a lot of trouble when things go wrong and the weak version is stronger than you thought or never overridden or the like.

Building your Emodule
*********************

In contrast to former emodule concepts, the current <wordhere> is much more flexible in all respects.  This is a drama however since there is no THE way to do it and hence this section is only a showcase. Nonetheless, we try to give a few more examples, clues even, in the More Ideas section below, obviously in the hope that they turn out to be useful.

Configuring your emodule
========================

This step really depends on the purpose of your emodule.  So we cut it really short here.  Consider this passage an assortment of useful tips.

  * SXEmacs by default installs its configuration file config.h in its header directory, $prefix/include/sxemacs/VERSION/ by default, so you can conditionalise on whatever has been configured by the user.

  * SXEmacs installs all its configure tests (written in m4) into aclocal's macro directory, $prefix/share/aclocal/ by default, so you can basically repeat every single test which facilitates the scenario where your emodule is configured differently than SXEmacs.

Compiling and linking your emodule
==================================

Basically using autotools and libtool your emodule is compiled with almost a one liner:

     module_LTLIBRARIES = my-module
     my_module_SOURCES = source1.c source2.c ...
     my_module_LDFLAGS = -module

     Makefile.am assuming MODULEDIR is properly defined.

It becomes a wee bit more complex than that if you define lisp bindings for which documentation strings are supposed to be exported.

     module_LTLIBRARIES = my-module.la
     my_module_la_SOURCES = source1.c source2.c ...
     nodist_my_module_la_SOURCES = source.doc.c
     my_module_la_LDFLAGS = -module
     BUILT_SOURCES = source.doc.c

     SUFFIXES = .doc.c
     source.doc.c: $(my_module_la_SOURCES)
     	$(make_docfile) --modname $* -E $ $^

     Makefile.am for emodule my-module with built
     documentation strings

Note: In both examples we assume that you defined a (make) variable moduledir.  The final emodule will be installed there.  In the second example we also assume that the variable MAKE_DOCFILE points to the make-docfile utility.

More Ideas
==========

The sole purpose of the above examples is to show off.  Making emodules has become as easy as making children, but basically you are not restricted to the above way.  Hence an assortment of useful clues:

  * The basic manual compilation command for C sources goes along the lines of:

          gcc -g -O2 -shared -o my-module.so.0 my-source.c

  * For fortran:

          f77 -g -O2 -shared -o my-module.so.0 my-source.f

  * Emodules themselves can be linked to a whole world of shared libraries, they are automatically loaded.  Linking emodules to other emodules is possible on some platforms but in general not portable.

  * The make-docfile utility can always be very easily found when you know the location of an installed SXEmacs binary.  Instead of calling make-docfile directly you would call sxemacs --make-docfile and pass all arguments you would normally pass thereafter.

     The target of the example in the previous section then reads:

          make_docfile = sxemacs --make-docfile

          SUFFIXES = .doc.c
          source.doc.c: $(my_module_la_SOURCES)
          	$(make_docfile) --modname $* -E $ $^

  * Basically the suffix .doc.c is not compulsory.  In fact, you can choose whatever you like there.  In our eyes this is yet convenient since the created file is indeed a C file and hence there will be no trouble for any .c.o rules you might have to cope with the compilation.

     On the other hand, having its suffix be .doc.c facilitates the use of a pattern-based rule in your Makefile when your emodule consists of many single source files which otherwise means that you had to create a FILE.doc.c rule for each file.

     We ourselves use the following pattern-based rule:

          make_docfile = $(SXEMACS) --make-docfile

          SUFFIXES = .doc.c
          .c.doc.c:
          	$(make_docfile) --modname $* -E $ $<

     where the SXEMACS variable points to the just built sxemacs binary.

     For all make-agnostics the above rule means that given a C file foo.c the rule to create a file foo.doc.c with the documentation strings in it goes:

          $(SXEMACS) --make-docfile --modname foo -E foo.doc.c foo.c

     when the command is maximally expanded.

  * The make-docfile utility itself does not (yet) come with a sensible --help output.  Here in short the explanation you may long for:

          make-docfile [--modname <name>] -E <outfile> <infile> [<infile> ...]

Defining Functions
******************

One of the main reasons you would ever write a emodule is to provide one or more functions for the user or the editor to use. The term function is a bit overloaded here, as it refers to both a C function and the way it appears to Lisp, which is a subroutine, or simply a subr.

A Lisp subr is also known as a Lisp primitive, but that term applies less to dynamic emodules.  See Writing Lisp Primitives (SXEmacs Internals Manual), for details on how to declare functions.

Normal Lisp primitives document the functions they define by including the documentation as a C comment.  During the build process, a program called make-docfile is run, which will extract all of these comments, build up a single large documentation file, and will store pointers to the start of each documentation entry in the dumped SXEmacs.

Using DEFUN
===========

Although the full syntax of a function declaration is discussed in the SXEmacs internals manual in greater depth, what follows is a brief description of how to define and implement a new Lisp primitive in a module.  This is done using the DEFUN macro.  Here is a small example:

     DEFUN("my-function", Fmy_function, 1, 1, "FFile name: ", /*
     Sample Emacs primitive function.

     The specified FILE is frobnicated before it is fnozzled.
     */
         (file))
     {
             char *filename;

             if (NILP(file))
                     return Qnil;

             filename = (char *)XSTRING_DATA(file);
             frob(filename);
             return Qt;
     }

The first argument is the name of the function as it will appear to the Lisp reader.  This must be provided as a string.  The second argument is the name of the actual C function that will be created.  This is typically the Lisp function name with a preceding capital F, with hyphens converted to underscores.  This must be a valid C function name.  Next come the minimum and maximum number of arguments, respectively.  This is used to ensure that the correct number of arguments are passed to the function.  Next is the interactive definition.  If this function is meant to be run by a user interactively, then you need to specify the argument types and prompts in this string.  Please consult the SXEmacs Lisp manual for more details.  Next comes a C comment that is the documentation for this function.  Last comes the list of function argument names, if any.

Declaring Functions
===================

Simply writing the code for a function is not enough to make it available to the Lisp reader.  You should use the specially treated init function in your emodule to let the lisp reader know about all the great subroutines you have coded, see Recognised Functions.

This is done by calling DEFSUBR with the name of the function (its C name, ya know?) as its only argument.  Using the example function above, your body of the init function would look like:

     void init(void)
     {
             DEFSUBR(Fmy_function);
     }

This call will instruct SXEmacs to make the function visible to the Lisp reader and will prepare for the insertion of the documentation into the right place.  Once this is done, the user can call the Lisp function my-function.

Basically, that is all you need to define and proclaim new functional lisp bindings.  The rules for what goes inside the functions, and how to write good emodules, is beyond the scope of this document.  Please consult the SXEmacs internals manual for more details.

Defining Variables
******************

Rarely will you write a emodule that only contains functions.  It is common to also provide variables which can be used to control the behaviour of functions or store a certain state or simply the results of a function being executed.  The actual C variable types are the same for emodules and internal SXEmacs primitives, and the declaration of the variables is identical.

See Adding Global Lisp Variables (SXEmacs Internals Manual), for more information on variables and naming conventions.

Once your variables are defined and initialised properly, you need to make the Lisp reader aware of them.  This is done in the specially treated init function of your emodule using special SXEmacs macros such as DEFVAR_LISP, DEFVAR_BOOL, DEFVAR_INT etc.  The best way to see how to use these macros is to look at existing source code, or read the internals manual.

Below is a small example which declares and properly proclaims two variables.  You will note that this code takes into account the fact that this emodule may very well be compiled into SXEmacs itself.  This is a prudent thing to do.

     Lisp_Object Vsample_string;
     int sample_boolean;

     void init(void)
     {
             DEFVAR_LISP("sample-string", &Vsample_string /*
     This is a sample string, declared in a module.

     Nothing magical about it.
     */);

             DEFVAR_BOOL("sample-boolean", &sample_boolean /*
     *Sample user-settable boolean.
     */);

             sample_boolean = 0;
             Vsample_string = build_string("My string");
     }

Finding & Loading Emodules
**************************

Loading a SXEmacs Emodules is usually simply a matter of...

     (require 'emodname)

Emodules can also be loaded interactively with...

     M-x load-module

#'load-module supports completion, making it a bit easier for the user.

The default search path for Emodules (also known as module-load-path) is...

     ~/.sxemacs/$machinetriplet/modules
     $prefix/lib/sxemacs/$machinetriplet/site-modules
     $prefix/lib/sxemacs-$version/$machinetriplet/modules

Searching recurses down one level below these directories.  The module-load-path is also part of the standard load-path. This means that if an emodule shares the same feature symbol name as a elisp library in the load-path, that one that is loaded from a #'require call will be the one that is higher up in the load-path.

It is possible to set the module-load-path at SXEmacs configure time using the --with-module-path=PATH configure option.  But you normally would not need to do this, as SXEmacs' build chain is smart enough to set sane defaults.

Locating where a particular emodule is installed on disc can be done with the function, #'locate-module.

To see which emodules are loaded in the current SXEmacs session, use #'list-modules.  It returns a list of emodule names (strings), or if called interactively, it displays the list of emodules in the echo area.  The names are the emodules' internal.

          (list-modules)
              => ("cl" "cl-loop" "ase" "ase_set")