14 - Speed Tables API Examples

To demonstrate how speed tables, the speed table transfer protocol, and stapi works, let's build a table based on something that should be familiar to most UNIX programmers... the UNIX /etc/passwd file.

The file consists of lines looking something like this:

fred:*:1616:1616:Fred Smith:/home/fred:/bin/tcsh
To represent these in a speed table, we'll create a table like this:
speedtables U_passwd {
  table u_passwd {
    varstring username unique 1 indexed 1 notnull 1
    varstring passwd
    int       uid      indexed 1 notnull 1
    int       gid      notnull 1
    varstring fullname
    varstring home     notnull 1
    varstring shell
  }
}
This creates a Speed Tables package named "U_passwd", containing one table named "u_passwd". The package name must always start with an uppercase letter.

The table u_passwd contains 7 fields: username, passwd, uid, gid, fullname (we know the GCOS field is not just a full name, this is an example), home directory, and shell. The passwd field in most modern UNIX systems is not used directly, and we won't reference it further.

Fields have types: varstring is a variable length string, and int is a 32-bit integer. Fields have parameters, a name-value list, so to enable then you have to set them to a non-zero value.

This table has an anonymous key, that can be referenced as "_key", because we are going to make it a shared memory table later and shared memory tables can only be searched by skiplist-indexed fields.

In a full scale application we'd have a variety of methods to load and edit this table, but we'll load it from /etc/passwd using read_tabsep:

proc load_pwfile {tab file} {
    set fp [open $file r]
    $tab read_tabsep $fp -tab ":" -skip "#*" -nokeys
    close $fp
}
The read_tabsep method is a convenient mechanism for reading a variety of data sources quickly and efficiently. Here we're setting the tab string to ":" and skipping lines that begin with a "#". It is also used internally in client-server tables, and for quickly reading data from PostgreSQl databases.

Once the file is loaded into a table we will search for users in the table with a little search procedure that does a callback for each :

proc search_passwd {tab id proc} {
    if [string is integer $id] {
        set field uid
    } else {
        set field username
    }

    return [
        $tab search \
            -compare [list [list = $field $id]] \
            -array_get_with_nulls row \
            -code {$proc $row}
    ]
}

The search operation is the most powerful tool in speed tables. It can be used to search the table using any combination of fields and values, using a lisp-like prefix syntax. Each element in the comparison list is an expression in the form {operator field value...}. Operators include basic comparison operations such as "=", "<", and ">", as well as ranges and sets.

For example let's say you reserve user IDs below 1000 for "role" accounts like webmaster, and you want to check for users over 1000 that are really "role" accounts. You can search for accounts with a home directory outside "/home" but a user ID over 999, and so use search -compare {{>= uid 1000} {notmatch "/home/*" $home}}. You can also search on a set of names with search -compare {{in username root daemon operator}}.

When a user is found this procedure uses a simple callback routine. The direct 1:1 mapping between name-value lists, Tcl arrays, and speed table rows is convenient here.

proc show_user {list} {
    array set entry $list

    puts "User name: $entry(username)"
    puts "       ID: $entry(uid)"
    puts "    Group: $entry(gid)"
    puts "     Name: $entry(fullname)"
    puts "     Home: $entry(home)"
    puts "    Shell: $entry(shell)"
}
To use the speed table and these helper routines in a standalone program, you would do something like this:
package require speedtable

source passwd_table.tcl
source show_user.tcl

set pwtab [u_passwd create #auto]

load_pwfile $pwtab /etc/passwd

foreach id $argv {
    if {[search_passwd $pwtab $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

$pwtab destroy
Speed tables follow a class/object design (though they are written in C, not C++), with the class of a table being the creator table (eg, u_passwd) and the create method creating a new Tcl command that is a instance of the class. The subcommands for the table object are methods, and that's how we refer to them.

Client-server Speedtables

of course loading the password file every time you want to look up a user is rather inefficient, so we can create a speed table server. Turning this program into a server is easy, we load the speed table server package, and then instead of looking up a user name and display it we go to sleep waiting on requests:
package require speedtable
package require ctable_server

source passwd_table.tcl

u_passwd create passwd

load_pwfile passwd /etc/passwd

::ctable_server::register sttp://*:3100/passwd passwd

::ctable_server::serverwait

passwd destroy
This is basically the same program as the first example, with the changed lines hilighted. Most of the examples are going to be like this.

The client code is similarly simple. Instead of building our own speed table we connect to the server and use it via the speed table transfer protocol, STTP:

package require ctable_client

source show_user.tcl

remote_ctable sttp://localhost:3100/passwd passwd

if {[llength $argv] == 0} {
   puts stderr "Usage: $argv0 user \[user...]"
   exit 2
}

foreach id $argv {
    if {[search_passwd passwd $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

passwd destroy

Speed Table API

None of the code in show_user.tcl needs to know that it's operating on a remote table instead of a local one. This can be generalised even further, producing the speed table API (STAPI). To use STAPI, you use ::stapi::connect URI, which returns a proc that behaves like a speed table. So, first, let's modify the STTP client to use STAPI:

package require st_client

source show_user.tcl

set pwtab [::stapi::connect sttp://localhost:3100/passwd]

if {[llength $argv] == 0} {
   puts stderr "Usage: $argv0 user \[user...]"
   exit 2
}

foreach id $argv {
    if {[search_passwd $pwtab $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

$pwtab destroy
The package "st_client" pulls in the STAPI connection code and registers the sttp: method with it. Each method has been defined as a separate package so you don't have to pull in connection code for methods you're not using.

SQL Access Method

This doesn't seem to be much of a change, and it isn't. But now let's say we need to connect to an SQL server that's using this SQL table:
CREATE TABLE passwd (
    username    varchar PRIMARY KEY,
    passwd      varchar,
    uid         integer NOT NULL,
    gid         integer NOT NULL,
    fullname    varchar,
    home        varchar NOT NULL,
    shell       varchar
);
To use this table in PostgreSQL instead of the speed table server, we load up the sql: method with package require st_client_pgtcl and connect to an sql: table.
package require st_client_pgtcl

source show_user.tcl

::stapi::set_conn [pg_connect -conninfo $login_info]

set pwtab [::stapi::connect sql:///passwd?_key=username]

if {[llength $argv] == 0} {
   puts stderr "Usage: $argv0 user \[user...]"
   exit 2
}

foreach id $argv {
    if {[search_passwd $pwtab $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

$pwtab destroy
The sql: connection method in STAPI reads the database to find the structure of the table and converts the Speed Table search term into an SQL query and performs the same callbacks as the original search.

To populate the password table in SQL, for example when converting from flat files to an SQL database... we should be able to use the speed table loading routine from the first example:

package require st_client_pgtcl

::stapi::set_conn [pg_connect -conninfo $login_info]

set pwtab [::stapi::connect sql:///passwd?_key=username]

load_pwfile $pwtab /etc/passwd

$pwtab destroy
Unfortunately, we have not yet implemented read_tabsep for sql: tables, so you have to load up a local speed table and write_tabsep it to SQL like the example from Chapter 13 of the manual.
package require speedtable
package require Pgtcl

source passwd_table.tcl

set pwtab [u_passwd create #auto]

load_pwfile $pwtab /etc/passwd

set conn [pg_connect -conninfo $login_info]

set r [pg_exec $conn "COPY passwd FROM stdin WITH DELIMITER AS '\t' NULL AS '\\\\N';"]
if {"[pg_result $r -status]" == "PGRES_COPY_IN"} {
  $pwtab search -write_tabsep $conn -nokeys 1
  puts $conn "\\."
} else {
  puts stderr [pg_result $r -error]
}
pg_result $r -clear

$pwtab destroy
To use other database back ends instead of postgresql would be a matter of copying and modifying client/pgtcl.c. There should not be major changes necessary for any SQL back end, other than the code that queries the database to deduce the table structure.

Speed Tables Display

STAPI also includes a web interface to speedtables, based on the DIODisplay package from Rivet. STDisplay currently uses the Rivet form and CGI procedures so requires Rivet to run. The simplest example of STDisplay is based on the first simple Speed Table example:
<?
package require speedtables
package require st_display

# Simple demo

source passwd_table.tcl

set pwtab [u_passwd create #auto]

load_pwfile $pwtab /etc/passwd

set display [::STDisplay #auto -table $pwtab -readonly 1 -details 0]

$display show

$pwtab destroy
?>
This produces a straightforward table view:

  Select:

38 records; page: 1  2 

Key Username Passwd Uid Gid Fullname Home Shell
37 mailcatcher * 2525 2525 SC Mail Catcher /usr/home/mailcatcher /bin/bash
36 postfix * 125 125 Postfix Mail System /var/spool/postfix /sbin/nologin
35 vmail * 11184 2110 VMail Owner for Virtual User Accounts /var/qmail/maildirs /bin/true
34 ldap * 90 90 the openldap server /nonexistent /bin/bash
33 cyrus * 60 60 the cyrus mail server /nonexistent /sbin/nologin
...
13 mailnull * 26 26 Sendmail Default User /var/spool/mqueue /sbin/nologin

STDisplay will work with any STAPI object that provides a minimal set of STAPI methods. You could use any of the standard STAPI front ends -- speed tables, STTP, SQL, or shared memory speed tables -- or create your own using, for example, iTcl. The methods that need to be implemented are:

The search options required are:

Shared Memory Speed Tables

The other major use of STAPI is for shared memory speed tables. The shared memory interface requires that the client get a connection token from the master process, and it can only perform "search" operations on the table. The server changes from the previous example are minimal:
package require speedtable
package require ctable_server

source passwd_table.tcl

u_passwd create passwd master file passwd.dat size 20M

load_pwfile passwd /etc/passwd

::ctable_server::register sttp://*:3100/passwd passwd

::ctable_server::serverwait

passwd destroy
The story on the client side is quite different, since you have to connect to the server and then attach a slave "reader" table to the shared memory segment for shared-memory search access. The "attach" method hides most of the details, but you still need
package require ctable_client
package require U_passwd

remote_ctable sttp://localhost:3100/passwd master
u_passwd create slave reader [master attach [pid]]

if {[llength $argv] == 0} {
   puts stderr "Usage: $argv0 user \[user...]"
   exit 2
}

foreach id $argv {
    if {[search_passwd slave $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

slave destroy
master destroy
And then remembering to use the STTP conection to the master table (master) or the shared memory reader table (slave) depending on whether you're using search or not. For our example program this would not be all that significant, since it only uses search, but any other speed table methods would fail on the reader. For example, you could not use Speed Tables Display directly on a shared memory table.

The "st_shared" package hides this detail:

package require st_shared

source show_user.tcl

set pwtab [::stapi::connect shared://localhost:3100/passwd]

if {[llength $argv] == 0} {
   puts stderr "Usage: $argv0 user \[user...]"
   exit 2
}

foreach id $argv {
    if {[search_passwd $pwtab $id show_user] == 0} {
        puts stderr "$id: not found"
    }
}

$pwtab destroy
STAPI keeps track of both tables and calls the appropriate one for each method. So the STDisplay example on a shared memory speed table would become:
<?
package require st_shared
package require st_display

set pwtab [::stapi::connect shared://localhost:3100/passwd]

set display [::STDisplay #auto -table $pwtab -readonly 1 -details 0]

$display show

$pwtab destroy
?>