Title

FTL (Function Template Library)

Author

Sergei Egorov <segorov@malgil.com>

Abstract

The purpose of this SRFI is to propose a comprehensive and easy-to-remember set of high-order procedures to be used as buiding blocks for a variety of ad-hoc sequence libraries. It is designed around the conventional notion of a sequence as an ordered set of elements represented by some enumerating algorithm or list- or vector-like data structure.

Issues

Generators can be extended to fold multiple sequences of the same type; algorithms can be extended to make use of this. Is such an extension necessary? Can it conflict with other extensions?

Half-baked FTL-Syntax SRFI: just a bizarre idea?

Rationale

No rationale yet

Specification

All proposed high-order procedures take arguments implementing one of a few abstract "interfaces" to data that serve as parameters for the corresponding generalized algorithm; its specialized version is returned as the result. Each abstract interface provides access to opaque data via a small set of conventional operations; for example, mutable vector-like objects are characterized by a 4-tuple <length, ref, set!, make>. In order for an algorithm parameterized by these four operations to work as expected, all operations should conform to a certain set of requirements, explicitly specified for each interface. In all cases, these requirements are only as strict as needed by the present set of algorithms. As a result, both purely functional (eager and lazy) and side-effect-based implementations of interfaces can be used with most of the algorithms.

To make the resulting collection of high-order procedures easier to remember and use, their names follow the naming convention established by traditional low-order sequence libraries (R5RS, SRFI-1, CommonLisp, ...) The result of run-time parameterization of a high-order procedure can be predicted by "substituting" names of parameters for their "placeholders" in the high-order procedure's name, and guessing at procedure's behavior by the resulting "low-order" name (which may be used to name the result). As a simple example, one can construct a procedure converting strings to vectors as follows:

(define string->vector (%v->%mv v=string mv=vector))

Here, %v->%mv is the name of a high-order procedure and v=string and mv=vector are names of two interfaces of vector-like objects (read-only strings and mutable vectors). Performing a "mental printf" on the right-hand-side names allows one to check that the left-hand-side name is not misleading. In more complex examples, more than one such "printf" may be needed to understand the result:

(%g-remove-%t->%o g=string (t=not-%t t=char-ci) o=list)

Here, "mental printf" gives t=not-char-ci for the inner high-order application and string-remove-not-char-ci->list for the whole thing; the resulting peculiar procedure converts a string to a list after filtering out every character that is not char-ci=? to an explicit argument. If you ever need such a procedure, you'll probably name it something like string-filter-ci->list and use as follows:

(string-filter-ci->list #\m "Metaprogramming")
  => (#\M #\m #\m)

More realistic example on the same theme is SRFI-1's filter that can be constructed as list-remove-if-not->list:

(define filter (%g-remove-%t->%o g=list t=if-not o=list))

[The regularity of the proposed naming convention can be exploited in interesting ways; FTL-Syntax SRFI is just one example]

Note, that arguments to our high-order procedures should be given in the exact order suggested by the order of %<interface> placeholders and should have matching <interface>=... names; Scheme knows nothing of these conventions and may not provide enough error-checking to spot invalid procedural arguments before actual use of the result.

Entry format

The body of the specification is organized into entries. Each entry describes one constant or procedure or a group of related constants or procedures. An entry begins with one or more header lines of the form

If category is constant, template gives the name of the constant (global identifier, bound to an interface object). If category is procedure, template gives a template for a call to a procedure, followed, by ⇒ and the description of the returned value. We follow the RⁿRS convention that if an argument or result name is also the name of the type, than that argument must be of the named type. Additional conventions used in this SRFI are:

Interfaces

Although there are dozens of ways to represent sequences and iterative computation, we decided to select only those closely matching traditional list/vector processing algorithms with some support for more esoteric cases involving hidden state and side effects. This SRFI does not allow for such things as backtracking and does not provide an elegant solution to the fringe comparison problem. The following 11 interfaces represent common abstractions generic enough to build most (but not all) finctions defined in SRFI-1, SRFI-13, and SRFI-43 plus thousands more by combination.

1. Equality (e)
2. Order and Equality (oe)
3. Transformation (x)
4. Test (t)
5. Generator (g)
6. Output (o)
7. Accumulator (a)
8. Input (i)
9. Lookahead Input (li)
10. Vector (v)
11. Mutable Vector (mv)

Interfaces have one- or two-letter abbreviated names used to form names of high-order procedures and interface implementations. Two-letter interfaces (oe, mv, li) are extensions of the corresponding one-letter ones (e, v, i). Extensions conform to all requirements of the "base" interfaces plus some extra requirements of their own.

Equality (e)

Equality interface is the simplest one in our set - it consists of a single two-argument procedure implementing an equivalence relation; that is, it must be symmetric, reflexive, and transitive. In addition, calls to the the procedure are expected to be referentially transparent (i.e. return the same result given the same arguments) and have no side effects. The rationale for this requirement is to allow algorithms to calculate equality as many times as needed and expect consistent results.

Equality interfaces can be constructed and de-constructed as follows:

Returns a new e interface defined by eq predicate.

Returns the equality predicate component of e.

Common e interfaces:

Order and Equality (oe)

Order and Equality interface is an extension of the Equality interface; in addition to the equivalence procedure, it includes a two-argument procedure implementing a partial ordering relation, compatible with the equivalence relation. This means that the ordering relation must be transitive, irreflexive, and obey the trichotomy law with regard to the equivalence relation. Calls to both procedures are expected to be referentially transparent.

Returns a new oe interface defined by eq and less predicates.

Returns the equality predicate component of oe.

Returns the ordering predicate component of oe.

These procedures return ordering predicates derived from primitive components of oe.

Converts oe interface into e via (e-interface (%oe=? oe)). If an implementation supports "backward compatibility" between oe and e interfaces, this function is an identity.

Common oe interfaces:

Transformation (x)

The Transformation interface is a wrapper for a one-argument procedure transforming a value to another value. Transformations are used to build interfaces from interfaces by fusion (mapping over series of produced or consumed values) in cases when the transformation is common enough to be hard-wired into an algorithm instead of being supplied by the caller. Fused transformations can play a role of CommonLisp's :key modifiers. Applications of the transformation procedure are expected to be referentially transparent.

Returns a new x interface defined by f.

Returns the transformation function component of t.

Common x interfaces:

Test (t)

Test interface is a generalization of testing methods used by search and compare algorithms such as R5RS memq and SRFI-1's find-tail and plays the role of CommonLisp's -if, -if-not, :test, and :test-not conventions.

Test interface consists of a single two-argument procedure implementing some sort of test of its first "variable" argument (usually coming from a sequence) with respect to its second "fixed" argument (an argument to a sequence algorithm). Calls of the test procedure are expected to be referentially transparent.

Returns a new t interface defined by p (a predicate). Example:

(define t=memq (t-interface memq)) ;memq matches the requirements for p

Returns the test predicate component of t.

Converts e interface into t via (t-interface (%e=? e))

Returns a logical complement of t. t=not-%t can be defined as follows:

(define (t=not-%t t)
  (let ((t? (%t? t))) 
    (t-interface (lambda (v f) (not (t? v f))))))

Returns a new t that adds an "input transformation" specified by x to its variable argument. t=%x&%t can be defined as follows:

(define (t=%x&%t x t)
  (let ((fn (%x x)) (t? (%t? t))) 
    (t-interface (lambda (v f) (t? (fn v) f)))))

t=if expects that the "fixed" argument is a predicate and applies it to the "variable" argument, returning the result of the application (or its logical complement, in case of t=if-not). They can be defined as follows:

(define t=if (t-interface (lambda (v f) (f v))))
(define t=if-not (t=not-%t t=if))

Other common t interfaces:

Generator (g)

The Generator interface captures the common notion of an algorithm producing a finite series of elements based on some initial "source". A source can be a container (in which case the generator enumerates its content), a value encapsulating initial state for an algorithm that produces alternative solutions (moves in a chess game), or any other object that can be mapped onto a sequence of values.

Generators adhere to the "push" model of output: the internal state of the process of generation is implicit and need not to be reified in a first class form by rewriting of the original algorithm or via call/cc. In this SRFI, algorithms that consume the entire input sequence and fit naturally into the passive filtering/accumulation model are defined as high-order procedures over generators (%g algorithms). Formal criterion for what constitutes a "natural fit" is given in [1] (algorithm should be closed under cons).

The Generator interface consists of a single three-argument procedure patterned after the traditional fold family of functions (e.g.: SRFI-1's fold and fold-right):

(fold kons knil src) ⇒ klist

The generator takes its third argument, src, as its source and "folds" it by feeding each element it produces to its first argument, a cons-like binary procedure kons. This procedure is applied to each subsequent element and the result of a previous such application, if it existed, or the second generator argument, knil. The algorithm behind the process of generation need not to be referentially transparent, but it should not expect that applications of its second argument are referentially transparent, and it should satisfy the fusion law:

For all f, v, x, y, prime, and f', v' such that

prime(v) = v', and
prime(f (x, y)) = f'(x, prime(y)), the following should hold for fold:

prime(fold(f, v)) = fold(f', v')

In practice, these requirements mean that the generator should treat its first two arguments as opaque and is only allowed to apply its second argument once to the same intermediate value of klist. Restricting generators to single-pass mode make them compatible with non-functional ways to consume generated values, such as writing to a file.

Returns a new g interface defined by fold.

Returns the fold component of e.

Generators can be built from other, more specific interfaces:

These procedures return a new g interface based on functionality available through i and v interfaces. Reverse variants are produced differently: g=reverse-%i is based on recursive (right) fold, while g=reverse-%v iterates through indices in reverse order.

Returns a new g interface defined by mapping x over values produced by g. This operation is known as fusion.

Common g interfaces with sources, description of their output, and more specific interfaces they can be built from:

Returns a "subrange" object, distinguishable from strings, vectors and other standard data types except procedures. Subrange objects store the arguments used to construct them and are accepted as valid arguments to vector- and string-based inputs and generators.

Output (o)

The Output interface is complementary to the Generator interface - it provides the exact functionality needed to consume values produced by a generator. The initial output object is created by calling Output's constructor procedure with no arguments or an appropriate "destination" argument (copied verbatim from the invokation of a sequence algorithm). As new elements appear, they are fed to the output's write procedure, patterned after cons. When all elements are consumed, output's result procedure is called to convert the output to the appropriate final form.

Algorithms, using the Output interface, should not expect that applications of the the output's write procedure are referentially transparent, so write should not be called more than once with the same output object; the out argument to write should come from the constructor or previous call to write. Restricting the use of outputs to single-pass makes it possible to model side-effect-based output processes such as writing to a file.

Outputs are "passive" in a sense that the internal state of the consumption process exists in a first-class form (an output object, a.k.a. out) and there is no mechanism for the output to stop the generation process before it is over (i.e. to signal the "full" condition). In this SRFI, algorithms that need direct control over one or more data consumers are defined as high-order procedures over outputs (%o algorithms).

Returns a new g interface defined by component procedures.

These procedures return the respective components of o.

Common o interfaces:

Accumulator (a)

The Accumulator interface captures the common notion of an algorithm consuming a possibly infinite series of elements "unfolded" from some initial value and calculating its result value based on an initial state (created from an optional "destination") and this series. Accumulators may create and populate containers, fill existing containers, calculate sums, products and averages etc.

Accumulators adhere to the "pull" model of output: the internal state of the process of accumulation is implicit and need not to be reified in a first class form by rewriting of the original algorithm or via call/cc. In this SRFI, algorithms that produce a possibly infinite input sequence and fit naturally into the passive scanning/selection model are defined as high-order procedures over accumulators (%a algorithms). Formal criterion for what constitutes a "natural fit" is given in [1] (algorithm should be able to produce tail of every value it produces).

The Accumulator interface consists of a single two-or-three-argument procedure similar to the traditional unfold family of functions (cf.: SRFI-1's unfold and unfold-right):

(unfold dekons klist [dst]) ⇒ result

The accumulator is initialized by optional third argument, dst, taken verbatim from the invokation of a sequence algorithm (usually, it is also the last optional argument there). It continues by unfolding its second argument klist with the help of its first argument dekons, an unary procedure returning zero or two values. Dekons is first applied to the initial value of klist; if it returns two values, first of them is a new element and second is a new value for klist. When there are no (more) values to get, dekons returns no values.

The algorithm behind the process of accumulation need not to be referentially transparent, but it should not expect that applications of its first argument are referentially transparent, meaning that it cannot apply dekons more than once to the same value of klist. Restricting accumulators to single-pass mode make them compatible with non-functional ways to produce values, such as reading from a file. This is also the rationale behind the choice of dekons over more traditional <knull?, kar, kdr> triple: dekons is able to model no-lookahead producers such as read.

Returns a new a interface defined by unfold.

Return the unfold component of a.

"Active" Accumulators can be easily built from "passive" Output interfaces. As the outputs they are buit from, such accumulators rely on the algorithm caller to guarantee that the input series is finite.

Convert o interface into a.

Accumulators can be built from mutable vector interfaces:

These procedures return a new a interface based on functionality available through the mv interface. Non-destructive accumulators, built by a=%mv and a=reverse-%mv, do not support dst arguments; they build new vectors of the appropriate type from scratch. Destructive (!) variants require the caller to supply the dst argument of the appropriate type (vec) to be filled with incoming elements while there is space and elements are coming. Reverse variants iterate through indices in reverse order.

Returns a new a interface defined by mapping x over values consumed by a. This operation is known as fusion.

Prospective a interfaces [special interest?]:

Input (i)

The Input interface is complementary to the Accumulator interface - it provides the exact functionality needed to produce values consumed by an accumulator. Inputs are created explicitly by providing a first-class input object (a.k.a. in) as an argument to a sequence algorithm. Elements are extracted by calling the input's read procedure compatible with Accumulator's dekons (receives an input object and returns zero values or two values: element and new input object). Input ends when the invokation of input's read procedure returns zero values.

Inputs are "passive" in a sense that the internal state of the production process exists in a first-class form (an input object). Inputs need not to be read to the end; many algorithms consume input elements until some condition is satisfied, and return the "rest" input object to the caller. In this SRFI, algorithms that need direct control over one or more data producers are defined as high-order procedures over inputs (%i algorithms).

Returns a new i interface defined by read.

Return the read component of i.

Inputs can be built from vector interfaces:

These procedures return a new i interface based on functionality available through the v interface. Reverse variant iterates through indices in reverse order.

Common i interfaces:

Lookahead Input (li)

The Lookahead Input interface is an extension of the Input interface. It provides two additional procedures - empty? and peek. The empty? procedure should be a predicate returning non-#f when next call to read is guaranteed to return an element, and returning #f when read is guaranteed to return no values. The peek procedure should accept an input object and return the first element without actually removing it from the input. Peek guarantees that the same element (in eqv? sense) will be returned by the next call to read. It is possible to test a lookahead stream for emptyness and peek more than once with no intervening read, and the result is guaranteed to be the same. The effect of calling peek on an empty input is undefined.

Lookahead Inputs provide functionality needed by many stop-at-element search algorithms (memq is a good example). An ability to look one element ahead is required by many parsing algoritms working on character or token streams, but not all inputs can provide it; for example, Scheme's standard S-expression parser (read) does not support checking for emptyness and one-element lookahead, while character-oriented input provides both via peek-char.

Note, that although this SRFI allows li interface objects to be "backward compatible" with requirements of i interface and used as-is wherever i interface object is expected, it does not require such compatibility and defines the i=%li procedure for explicit conversion.

Returns a new li interface defined by component procedures.

These procedures return the respective components of li.

Converts li interface into i via (i-interface (%li-read li)). If an implementation supports "backward compatibility" between li and i interfaces, this function is an identity.

Lookahead Inputs can be built from vector interfaces:

These procedures return a new li interface based on functionality available through the v interface. Reverse variant iterates through indices in reverse order.

Common li interfaces:

Vector (v)

The Vector interface generalizes read-only finite sequences supporting random access to elements. Obvious candidates for such generalization are vectors and strings, but other possibilities like random-access files and bit-vectors exist. We will make a distinction between Vectors (implementations of v interface), vectors (all lower case, standard Scheme vectors), and vecs (sequence objects, manipulated through the appropriate Vectors).

Vectors are defined by providing length and ref procedures. The length procedure accepts one argument (a sequence of the appropriate type) and should return a nonnegative exact integer, specifying the upper bound for indexing operations (valid indices go from zero to one less than upper bound). The ref operation should accept two arguments - a sequence and an exact integer in bounds, defined by length, and return an element located at that index. Vectors guarantee that if vecs (sequence objects) are accessed only through v interface, repeated refs to the same location will return the same object (in eqv? sense). This guarantee supports algotrithms that need to access the same location multiple times.

Vectors provide functionality needed by search algorithms requiring indexed access to the sequence (for example, binary-search). Although it is easy to build g, i, and li interfaces from an instance of v interface (and there are procedures for that), Vectors are not considered extensions of Generator or Input/Lookahead Input interfaces, because there are many ways to build "weaker" interfaces from a Vector; this SRFI specifies only one of them: enumeration in ascending order of indices.

Returns a new v interface defined by component procedures.

These procedures return the respective components of v.

Common v interfaces:

Mutable Vector (mv)

The Mutable Vector interface is an extension of the Vector interface. It provides two additional procedures - set! and make. The set! operation should accept three arguments - a sequence, an exact integer in bounds, defined by length, and a new value to store at that index. The return value of set! is unspecified. The make procedure accepts a nonnegative exact integer and an optional initial value and returns a newly allocated sequence of the given length, filled with the initial value. If no initial value were given, the contents of the sequence is not specified.

In addition to Vector's guarantees, Mutable Vectors guarantee that if mvecs (mutable sequence objects) are accessed only through mv interface, a ref to a location will return an object, placed there by the most recent application of set! to that location, or initial value, if no set! calls to that location were made.

Note, that although this SRFI allows mv interface objects to be "backward compatible" with requirements of v interface and used as-is wherever v interface object is expected, it does not require such compatibility and defines the v=%mv procedure for explicit conversion.

Returns a new mv interface defined by component procedures.

These procedures return the respective components of mv.

Converts mv interface into v via (v-interface (%mv-length mv) (%mv-ref mv). If an implementation supports "backward compatibility" between mv and v interfaces, this function is an identity.

Common mv interfaces:

Algorithms

Returns a procedure that drops the first element from in and returns in containing the remaining elements. The effect of calling next on an empty input is undefined. %i-next can be defined as follows:

(define (%i-next i)
  (let ((i-read (%i-read i))) 
    (lambda (in)
      (call-with-values 
        (lambda () (i-read in))
        (lambda (e in1) in1))))) ;2 values expected

Implementation

Reference implementation (incomplete draft). TODO: explanation of how it meets the reference implementation requirement.

Copyright

Copyright © Sergei Egorov (2004). All Rights Reserved.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Scheme Request For Implementation process or editors, except as needed for the purpose of developing SRFIs in which case the procedures for copyrights defined in the SRFI process must be followed, or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by the authors or their successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and THE AUTHOR AND THE SRFI EDITORS DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Last modified: Sat Oct 9 17:43:25 EDT 1999