30. Writing Pike Modules

This chapter will discuss how to extend Pike by writing modules. There are two major ways to write modules, either they can be written in Pike, or they can be written in C. Generally, modules can be seen as a collection of pike programs and functions. This is, obviously, handy for grouping related programs and functions.

A pike module is actually a pike program which is cloned by the pike compiler during compilation of programs. This means that all lfuns that can be used in a pike program also can be used in a module. This is, for instance, useful for overloading the operators of a module to obtain a certain behaviour. Bear in mind that variables defined on a module-wide bases are shared among all clones of programs in the module. FIXME: Explain difference between .pmod and .pike

Pike searches for modules in the module path as defined during the compilation of a pike program. The module-path defaults to contain the directory where all standard pike modules are installed. This can be altered using /master.CompatResolver()->add_module_path() in a program or by letting the environment variable PIKE_MODULE_PATH contain a colon-separated list of directories to be searched for modules before looking at the default location. Note that overlapping directory modules are merged.

30.1. Writing Modules in Pike

Writing modules in pike is by far the easiest way to extend pike. They are also useful for structuring a larger programming project into different source files.

There are two ways of create a pike module written in pike. Either create a file named as the module will be called with the extension .pmod and place all program and function definitions in it. The other way, which usually is more flexible, is to create a directory named as the module with the extension .pmod and place all program definitions (.pike-files) within this directory. If a file called module.pmod is placed in the directory the function and program definitions within it will be merged with the programs found in the directory. This file could, as an example, be used to specify functions residing in the module, while programs in the module are placed in .pike-files.

To refer to another module or program beside the current source file, prefix its name with a dot. The compiler resolves this name relative to the source file's directory, rather than the process's working directory or the global module search path. For example, with Helpers.pmod and Base.pike beside the source:

import .Helpers;
inherit .Base;

int answer()
{
  return .Helpers.add(base_value(), 22);
}

Here Helpers.pmod supplies add() and Base.pike supplies base_value(). import .Helpers makes the helper module's members available without qualification; .Helpers.add() explicitly selects a member from that source-relative module. Prefer this lookup syntax for module and program dependencies instead of constructing filenames from __FILE__. It uses the master's module resolution and supports symbolic module paths in relocatable installations. For data files, use the file operations described below.

Relocatable dumped modules are enabled only when Pike's source is configured with --with-relocatable-dumped-modules, which enables PIKE_MODULE_RELOC. In such installations, __FILE__ and __DIR__ can contain a symbolic module path beginning with /${PIKE_MODULE_PATH}/, rather than a native filesystem path. This allows precompiled modules to be moved between installation directories. This configuration option is used by several binary packages of Pike.

Use the master's file operations when reading resources relative to a module, since they resolve symbolic paths through the module search path. For example:

string|zero data = master()->master_read_file(
  combine_path(__DIR__, "data.txt"));

Direct filesystem operations and external programs do not perform this translation. Before passing a module-relative path to them, resolve it with master()->relocate_module() when that function is available. Changing __FILE__ to __DIR__ alone does not resolve the symbolic prefix.

30.2. Writing Modules in C

A C module exposes native functions, constants and classes as members of a Pike module. Use it when an existing C library needs a Pike interface, or when work cannot conveniently be done in Pike. The interface should still use Pike values and exceptions, so callers need not know how it is implemented.

There are two ways to write the interface. In a plain .c file, register the members and handle arguments and return values yourself. In a .cmod file, the Pike precompiler generates much of this code from declarations such as PIKEFUN and PIKECLASS. Both approaches use the same interpreter API. The examples below implement a module called Example; they are alternative implementations, not files to link together.

30.2.1. A minimal C module

Include global.h first, before system or library headers. It supplies Pike's platform definitions. The other headers below declare module registration, the value stack, type descriptions, and argument decoding. Save this example as example.c.

#include "global.h"
#include "module.h"
#include "interpret.h"
#include "pike_types.h"
#include "module_support.h"

static void f_is_positive(INT32 args)
{
  INT_TYPE value;

  if (args != 1)
    Pike_error("Example.is_positive(): Expected one argument.\n");

  get_all_args("Example.is_positive", args, "%i", &value);
  pop_n_elems(args);
  push_int(value > 0);
}

PIKE_MODULE_INIT
{
  ADD_FUNCTION("is_positive", f_is_positive, tFunc(tInt, tInt), 0);
  ADD_INT_CONSTANT("API_VERSION", 1, 0);
}

PIKE_MODULE_EXIT
{
  /* No module-wide resources to release. */
}

PIKE_MODULE_INIT defines the entry point invoked when the module is initialized. The loader has already started the module's program, so register its members here without calling start_new_program() for the module itself. ADD_FUNCTION takes the Pike name, C implementation, type description and identifier flags. Here tFunc(tInt, tInt) describes a function taking one integer and returning an integer. Use the type macros rather than constructing encoded type strings by hand. PIKE_MODULE_EXIT releases resources retained by the module when it is shut down; it is not an object destructor.

Once installed as Example, the members are used like those of a Pike-written module:

Example.is_positive(7);   // Returns 1.
Example.is_positive(-2);  // Returns 0.
Example.API_VERSION;      // Equals 1.

30.2.2. Arguments and the value stack

A plain C implementation has the signature void function(INT32 args). The arguments are on the Pike value stack, and Pike_sp points just beyond its top. The first argument is Pike_sp[-args] and the last is Pike_sp[-1]. Check the argument count before indexing the stack. Check TYPEOF(value) before reading a union member such as value.u.integer or value.u.string.

get_all_args() decodes arguments and reports missing or incorrectly typed values as Pike exceptions. Its %i conversion stores a native integer in an INT_TYPE. It does not reject surplus arguments, which is why the example checks args separately. The registered function type is useful to the compiler, but does not replace runtime validation.

On a normal return from the example, the arguments have been removed with pop_n_elems(args) and the result has been pushed with push_int(). Other result helpers include push_float(), push_string(), push_array() and push_mapping(). Returning from the C function does not by itself push a Pike result. Do not pop the arguments a second time, or leave temporary stack values above the result.

Pike integers can be larger than INT_TYPE. The minimal example deliberately accepts only native integers. An interface that promises arbitrary-size integers must also handle bignums; do not truncate them to a C integer. Likewise, a Pike string has an explicit length and may contain NUL characters or wide characters. Do not treat every struct pike_string * as a NUL-terminated byte string. Check its size_shift and use its len when passing data to a byte-oriented C API.

30.2.3. Using the cmod precompiler

The following example.cmod implements the same function. The precompiler reads the Pike-style declaration, generates argument checks and local C variables, and registers the function. The body is C code. RETURN evaluates the result, removes the arguments and pushes the result with the appropriate type.

#include "global.h"
#include "module.h"
#include "interpret.h"
#include "pike_types.h"
#include "module_support.h"

DECLARATIONS

/*! @module Example */

/*! @decl int is_positive(int value)
 *! Returns 1 if the native integer value is positive, otherwise 0.
 */
PIKEFUN int is_positive(int value)
{
  RETURN value > 0;
}

/*! @endmodule */

PIKE_MODULE_INIT
{
  INIT
  ADD_INT_CONSTANT("API_VERSION", 1, 0);
}

PIKE_MODULE_EXIT
{
  EXIT
}

DECLARATIONS is the insertion point for generated declarations. At module scope, INIT and EXIT insert generated registration and cleanup code into the entry points. They are precompiler constructs, not C functions. Do not edit the generated .c file; change the .cmod source and regenerate it instead.

Simple argument types are checked by generated code. For a union of different runtime types, such as int|string, the precompiler supplies an svalue and the body must inspect its type. Optional and variable-length arguments also need the appropriate declarations and handling. Consult bin/precompile.pike --help and existing modules for these cases. Automatic decoding does not remove the need to validate ranges and other constraints of the underlying C library.

30.2.4. Reference ownership and error handling

Strings, arrays, mappings, multisets, objects and programs are reference-counted. Pointers obtained from arguments are borrowed: they remain valid while the corresponding stack values retain them. Removing the arguments can free those values. Acquire an additional reference with add_ref() before retaining a value beyond that lifetime, and release it with the matching free_string(), free_array() or other type-specific function when it is no longer needed.

push_string(), push_array() and similar helpers transfer an existing reference to the stack; they do not add a reference. Use ref_push_string(), ref_push_array() and their counterparts when the stack needs its own reference to a value still owned elsewhere. In a .cmod function, RETURN also transfers a reference for reference-counted results, while REF_RETURN acquires a reference before removing the arguments. This matters when returning an argument. Do not copy an svalue containing a reference-counted value with a plain C assignment; use the appropriate assign_svalue() helper and initialize its destination.

Report failures with Pike_error(), or use helpers such as SIMPLE_ARG_TYPE_ERROR for argument errors. Pike exceptions unwind C calls rather than returning through their normal cleanup paths. Protect temporary native resources with an ONERROR handler registered by SET_ONERROR before calling code that can throw. Remove the handler with UNSET_ONERROR after normal cleanup or after transferring ownership. Pike stack unwinding releases stack-owned values, but does not close a native handle or free a C buffer that only your code knows about.

30.2.5. Classes and per-object storage

Use PIKECLASS to define a class in a .cmod file, CVAR for C storage belonging to each object, and THIS to access the current object's storage. For example, this class can be added inside the Example module in the preceding .cmod source:

/*! @class Flag */
PIKECLASS Flag
{
  CVAR int enabled;

  INIT
  {
    THIS->enabled = 0;
  }

  /*! @decl int set(int enabled)
   *! Sets the flag and returns its new value, either 0 or 1.
   */
  PIKEFUN int set(int enabled)
  {
    THIS->enabled = enabled != 0;
    RETURN THIS->enabled;
  }

  /*! @decl int get()
   *! Returns the current value of the flag.
   */
  PIKEFUN int get()
  {
    RETURN THIS->enabled;
  }
}
/*! @endclass */

Example.Flag() creates an object with its own enabled field. Class-level INIT runs for each object, unlike the module-level registration step. Use class-level EXIT to release per-object native resources. Initialize handles and pointers to a state that cleanup can safely handle even if later initialization throws.

A CVAR containing a pointer to a Pike value is not automatically visible to the garbage collector. Supply the appropriate GC_CHECK and GC_RECURSE handlers for owned references, as well as releasing them in EXIT. Where possible, use PIKEVAR for Pike-valued fields instead of manually storing references in C storage. The Flag example stores only a C integer and therefore needs no such handlers.

The plain C equivalents are start_new_program(), ADD_STORAGE, registration of methods and callbacks, end_program() and add_program_constant(). Callbacks include set_init_callback(), set_exit_callback(), set_gc_check_callback() and set_gc_recurse_callback(). Keep the returned program's reference only if your module needs it, otherwise release it after adding the program constant. See the Math module for examples of several classes registered from one C module.

30.2.6. Blocking operations and threads

Native code normally runs with the interpreter lock held. Around a blocking library call, use THREADS_ALLOW() and THREADS_DISALLOW(), declared in pike_threadlib.h, so other Pike threads can run. Prepare the native arguments before releasing the lock and convert the result or throw a Pike exception only after reacquiring it.

While the lock is released, do not access the Pike stack, allocate Pike values, change shared Pike values or their reference counts, or call back into Pike. Keep required data alive and protect native state against concurrent use. Read the rules in src/pike_threadlib.h before adding an unlocked region; the macros release the interpreter lock, not your library's own locks.

30.2.7. Practical details

For an in-tree module, place the source in src/modules/Example/. Use a neighboring module as a build template. A small Makefile.in for example.cmod can look like this:

@make_variables@
VPATH=@srcdir@
OBJS=example.o
MODULE_LDFLAGS=@LDFLAGS@ @LIBS@
CONFIG_HEADERS=@CONFIG_HEADERS@

@dynamic_module_makefile@

example.o: $(SRCDIR)/example.c

@dependencies@

The common module rules run the precompiler for .cmod sources. If using example.c directly, omit the explicit generated-source dependency. Add all object files to OBJS and library link options to MODULE_LDFLAGS. Use @static_module_makefile@ for a module that must be linked into the interpreter instead of loaded dynamically. Native modules must be built with headers and configuration matching the interpreter that will load them.

The corresponding configure.in must call AC_MODULE_INIT() and check the headers, libraries and functions the module actually needs. For this example, no external library is required:

AC_INIT(example.cmod)
AC_CONFIG_HEADER(config.h)
AC_MODULE_INIT()
AC_OUTPUT(Makefile)

Generate configuration scripts using Pike's src/run_autoconfig wrapper, rather than invoking an arbitrary autoconf version directly. After adding a new module, run make force_autoconfig, make force_configure and make from the source tree's top directory. Configuration and compilation take place in the selected build directory. See README-GIT for build options.

30.2.8. Documentation and tests

Document the public Pike interface in AutoDoc comments beside its implementation. Use @module, @class and @decl to describe the Pike namespace and signatures, not the C implementation names. Describe accepted values, return values, exceptions and ownership of resources exposed to callers. Document platform or library requirements when a feature is conditional. Keep a Pike wrapper in module.pmod.in when part of the interface is better expressed in Pike; existing modules show how to combine a wrapper with native code.

Add a testsuite.in file to the module directory. A first test suite for either implementation above is:

START_MARKER

cond_resolv(Example, [[
  test_eq(Example.API_VERSION, 1)
  test_eq(Example.is_positive(7), 1)
  test_eq(Example.is_positive(0), 0)
  test_eq(Example.is_positive(-2), 0)
  test_eval_error([[function f = Example.is_positive; f(1, 2);]])
  test_eval_error([[function f = Example.is_positive; f();]])
  test_eval_error([[function f = Example.is_positive; f("not an integer");]])
]])

END_MARKER

Run make verify from the top directory to build a test installation and execute the test suites. Also test extra arguments, integer limits, repeated calls and cleanup after errors. For classes, test that separate objects have independent state and that destruction releases native resources. Optional modules should guard tests with cond_resolv so the suite also runs on systems where the module cannot be built.

30.3. Special Module Variables and functions

30.3.1. _module_value

If _module_value is non-zero it will be used as the value of the module. _module_value has to be of a type which is indicable, ie. an object, mapping or multiset.

30.3.2. The indexing operator

If a lfun::`[] is defined in a module it will be called when the module is indexed using the .-operator.