Updated 2014-04-13 23:52:02 by RLE ▲

Distribution mechanisms for Tcl source and binaries

KJN I have started this page because I am beginning to get confused by the number of possible distribution mechanisms. (My thanks to LV for clearing up a number of points, including the difference between extension mechanisms and delivery systems.)

Delivery Systems

One of several delivery systems -

  • Teacup - a tool for managing packages in a local ("installation") repository, including fetching packages from a remote ("archive") repository

(one can also use ftp or http or mount and other cd-rom, etc. access tools to access source or binary code and use cp, ginstall, etc. to manage the packages. And of course some OSes (for instance, Debian, MacOSX) etc. have package management tools as well).

Extension Mechanisms

An extension mechanism is a way that binary or Tcl code can be organised into an extension, so that when a script running on a Tcl installation requires that extension's functionality, the script can access the extension either automatically or by issuing a simple command.

  • Tcl Modules - a recent innovation - a single file that contains a package and can be sourced
  • starkit - a single file that contains a VFS with source and binary
  • package require - command for loading packaged code that is usually installed in Tcl's lib directory or in a local repository
  • load - load a shared library (DLL or SO file)
  • auto_load - load a command using the tclIndex files in the auto-load path
  • source - command to load a Tcl source file

Neil Madden pointed out [1] that package require delegates actual loading of packages to either source (for Tcl scripts) or load (for binary extensions).

LV I believe auto_load uses source.

KJN Yes, that is correct.

LV As far as I am aware, starkits have no built in method of accessing - it requires access to a couple of external extensions I think to be able to load extensions and even some development code I think.

As for Tcl modules - I'm not certain exactly what is needed there.

KJN Tcl modules can be loaded by the source command. Also, the package command has been modified to use packages in that are in module form - a new mechanism has been added for locating modules, and the loading of module files is delegated to source.

package require has two ways to locate packages. The "traditional" way is to search the directories listed in auto_path, and their immediate children, for files named pkgIndex.tcl. Each of these files contains a short Tcl script that gives the name and version number of each available package, together with instructions for how to load each one.

Since Tcl Modules can be loaded by the source command, it would be possible for package require to manage modules using the mechanism above, if appropriate paths were added to auto_path, and appropriate files pkgIndex.tcl were created.

Instead, Tcl provides a second mechanism for managing modules with the package command, and this mechanism is the one most frequently used. A second list of directories (the "Module path") is managed by the commands ::tcl::tm::path and ::tcl::tm::roots. package require searches these directories for modules, which are files named "*.tm". Files pkgIndex.tcl are not required: the package name and version are defined by the file name, and separate loading instructions for each module are unnecessary, because any module can be loaded by source.

Local repository access by package -

  • An "installation repository" is a local repository generated and managed by the teacup command. It can contain (binary or Tcl) packages either in "traditional" format, or in the newer single-file "module" format.
  • When a package is in module format, package require fetches it from the repository using the module path + module filename mechanism. If Tcl is correctly configured to use the repository, the directories where modules are found will have been added to the Module path, whose value is returned by the command [::tcl::tm::path list].
  • When a package is not in module format, package require fetches it from the repository using the auto_path + pkgIndex.tcl mechanism. If Tcl is correctly configured to use the repository, the directory where each pkgIndex.tcl file is found (or its parent directory) will have been added to auto_path.

LV This information is a bit confusing. I have read how to use teacup to manage a local directory where files are downloaded from a teapot server. However, I've not read much, if anything, about building extensions and installing them into that directory structure so that the package command can find them. Where can I read more about that?

KJN Location and loading of modules by the package command is in the manual for the command tm at http://www.tcl.tk/man/tcl8.5/TclCmd/tm.htm

To rewrite a pure-Tcl package in module form, the manuals for tm and teacup imply three levels of sophistication -

  1. Write the package as a single script file - this file, regardless of its name or location, can be loaded by source.
  2. If in addition you name the file according to the rules given in the manual for tm, and place it in an appropriate folder according to your shell's value of [::tcl::tm::path list], then you will be able to load the module by using the package command. By default the appropriate folder will be one of the folders contained in your Tcl installation's lib/tcl8/ folder.
  3. If in addition you provide metadata in Tcl comments in the header and footer of the file, then it will be in suitable format for distribution in teapot repositories.

To build a module from a package that is not pure Tcl -

auto_load searches for individual commands, by searching the directories listed in auto_path for files named tclIndex.

auto_load appears to be an older mechanism, but it is still used by the Tcl and Tk cores and a few libraries. It is not marked in the manual as deprecated. Should it be?

Please discuss.

LV Good question. I would think that some of the criteria for deprecating a command would be:

  • Does the command provide any functionality that is unique - that cannot be replicated by the other packages?
  • Is the command used by any other language extending mechanisms?
  • Is the command used by any of the major language extensions?

KJN Should starkits be in the main list? They are not quite an extension mechanism, but a complete application that can be run by a basekit.

LV Take a look at kitten - where a starkit can contain extensions. One could, in theory, use starkits as tcl modules, I suspect. I don't know whether it has been done yet, though.

KJN You're right, TIP 190 [2] and the teapot manual discuss the use of a starkit as a tcl module.

See also:

Earlier discussions

KJN Some of the conclusions have been integrated into the text above.

LV I'm confused by what you mean by distribution mechanism. Do you mean formats in which code can be distributed to users or do you mean mechanisms by which users can locate new extensions or do you mean means by which a program can invoke other code or what?

The reason I ask is because Teacup seems different than the other items on the above list - and there are certainly other repositories other than teacup.

Also, I don't see listed in the above list things like "zip files", "starkits", "tar files", "opening a file and reading in code", "exec", "opening a pipe", etc.

Perhaps after we know better what you are after, more information can be added.

KJN Sorry for the lack of clarity - I followed Tcl Modules which had been placed in Category Distribution.

I'm interested in the ways that binary or Tcl code can be organized into an extension, so that when a script running on a stock Tcl installation requires an extension's functionality, the script can access the extension either automatically or by issuing a simple command. So I suppose I'm looking for means by which a program can locate and invoke extensions that will run in the same interpreter in the same process. The user who needs to locate a missing extension, and the program that invokes another process (exec) or accesses another process by IPC, are different problems.

The simplest way to do this when organizing one's own Tcl code is to use source; but when a piece of code has well-defined functionality, and is intended for re-use in different projects, it is convenient to organize it as a package: this is how most libraries are provided at present.

Tcl Modules and Teacup are innovations, and I would be glad of any comments on how these fit in with the older commands. Teacup appears to differ from package require by fetching modules from a repository, rather than looking in the system's Tcl installation.

LV Actually, teacup sets up a teapot environment whereby one can manage a couple of variants of local repositories, or access a variety of remote repositories. HOWEVER, the teapot repository framework is a delivery mechanism. The other items on your list are language file formats and processes. There is really a fundamental difference, in my mind, between the two. Teacup is a program which manages how stuff gets into a directory. Why not list cp, unzip, gtar, etc. as they are parallels to teacup?

The other things on your list are the various file formats that people have used to create packages (a.k.a modules or extensions) that, when loaded, provide new tcl commands.

KJN It would be feasible to modify tclPkgUnknown so that a running script would automatically use teacup to try to fetch missing packages from the archive repository.

Clearly a network connection would be needed, and user feedback might be requested; but if the alternative is that the script fails because of a missing package, perhaps these hardships can be tolerated.

LV For that matter, it would be possible to modify the unknown proc to go out to sf.net or some other site, ftp down code, unzip/untar it, etc.

There would be a variety of social issues that one should consider before trying to extend teapot - for instance, how safe is it to download an extension for use in a running program, without any idea what license it uses, what permissions it requires, etc. Then there is the situation that, in a commercial setting, security and system admins generally don't want all users to have write permissions in these types of directories ... it makes security audit trails, etc. more difficult.

KJN OK, I see the distinction you are making - it is generally a bad idea to autoload packages from a teapot (remote) archive repository, and therefore teapot is a distribution mechanism and not a language extending mechanism.

LV Actually, there is a teacup option that sets up things so a non-activetcl tclsh will look in the appropriate local repository. Check that page for a hint, though you will unfortunately need to read more on the activetcl mailing list for details, until someone gathers that info together and updates the wiki page.

I believe the details are simple enough that other tcl uses could derive a similar concept as well.

Category Distribution