Tcl Core - Is All

Philip Quaife 17 Jan 05 - Freeing the core for all

There is a lot of discussion on what should and shouldn't be in the core of tcl/tk.

I propose a simple way to end this endless and mostly pointless debate whilst enabling unbounded testing of new features. The aim is to put the word extension back into tcl.

There are two principle areas to address:

Central Repository.
Replacable Core API.

Central Repository

While the TCT provides a focal point at tcl.sf.net for getting the core interpreters, there is no central repository for extensions.

No group of users or Company has committed to providing a long term storage area for either source or binary extensions.

This means that new users have to go to conciderable effort to locate an extension (once they are even aware of it), then they must most likely compile it themselves only to find that it needs some tweeking to be compatible with the latest version of tcl/tk.

I don't want to debate the case for or against a central repository, but it cannot be denied that the lack of one provides an impediment to new users getting the maximum benefit out of tcl/tk.

Once there is a central repository (this does not preclude the repository from being implemented as distributed system), we have the ability to add features such as:

Automated installation (ie no user understanding required)
Peer reviews (rating for both usability as well as security)
Open source base for new developers.
The core can considered the sum of the repository.

Why is being in the core so important?

Partly because of the above. Without being in the core you don't get exposure for your code. Most importantly , if you want other people to try out your code and it has features that are not in the generic core , you require the other users to update their core system.

This automatically cuts down the exposure to those users that have both the time and the skill to do so.

While most core feature additions are made with dynamically loadable extensions, some of them require modifications to the core interpreter. All new ideas for TCL will require core changes.

Replacable Core API

When the extension of TCL requires modifications to the tcl/tk base code it requires a formal process to have the change implemented.

This is because it affects all users. Once the change has been made, we have made that the default feature for all users.

Contrast this with extensions, these can be loaded or not at the application writers discretion. A new extension does not affect existing users.

Quite often the change wanted is only wanted by a small number of tcl users or just as an accademic exercise by one person seeking input from the programming community.

In these cases it is not applicable to make a permanent change to the base functionality.

The Solution To merge the two disparate needs we can simply do the following:

Make Stubs Extendable (VStubs, think of it as VFS for stubs)
Make all internal API Calls through the VStubs table.
Provide a structure for stacking and unstacking Stubs calls.
Change the Tcl_Obj structure to allow it to be variable length.
Provide API for registering extensions to the Tcl_Obj format.
Optional: Make all internal structures based on Tcl_Obj.

Before we start posting objections to why this can't be done , won't work, and other unfounded remarks. Take a pause and have a deep breath.

I am not going into the technical details of how this would be implemented, but I have spent enough time analysing the core to know that it is quite feasible, amenable to change by automated scripts, and something that the tcl community could do collectively in a short space of time.

The last change proposed alieviates the need to make internal structures extendible. While not necessary it does provide a unified structure that would alleviate some work on the part of extension writers. And of course there will be someone out there that will want that feature in the future.

Summary of the above

The above could be done under the TIP system.
We open the core by removing the difference between core and extension.
We remove the requirement of the TCT to focus on issues outside of the base tcl/tk code.
We bring TCL into the forefont of scripted development.
Without the central repository and the ability to have the application automatically fetch extensions, the core changes are less effective.

CMcC 20050117.

I think a repository for binary extensions is a very sound idea and worthwhile goal. I also think the analysis of what's needed misses some critical QA and maintenance functions of the TCT.

PWQ CMcC, The TCT has no required involvement in anything outside of tcl/tk as the base core that they see fit to evolve. The repository provides the user with the notion of every extension being part of the core.

If binary extensions are considered part of tcl, the same high engineering standards must apply to them because binary extensions are able to cause deep problems.

The gatekeepers of a binary extension repository would have to require adherence to engineering standards as a prerequisite to inclusion. For each new tcl version each extension would have to be re-qualified. These overheads will subtract from the maintenance and development manpower available for revision and development of tcl.

XX While I agree with these above sentiments, I would make several points "against" them:

QA is not an all or nothing proposition. The key, however, is to be upfront and clear about the level of QA applied to each offered extension. In other words, it would still be a service to provide abandoned, unsupported extensions to users provided it is clearly marked as such. "You want to do that, well, someone started the work, here it is - but don't you go telling people that TCL sucks when it crashes your installation" would be the basic idea. There might even be a requirment that such extensions be hosted in a distributed fashion. But users and developers would still be able to assess "at a glance" what is available in a particular area of need and be sure that they are not reinventing the wheel. It also becomes more likely that the code will be "picked up" and developed further.
I think there is a differences between what should be done with these binary extensions and what is required of the core. The TCT should not handle nor be held responsible for what happens here. New blood should be vetted and handled the ball on this one. No reason to draw from the resources of the TCT. Much of the QA addressed through engineering guidelines and the requirement of test suite inclusion.

CMcC 20050117: I see the open questions as more political-economic than technical:

Where is the skilled manpower and political will coming from to QA and maintain extensions?
What happens with abandonware in the extension repository?
If code makes itself into the extension repository and bounces in and out due to QA issues, abandonment and similar, aren't the same problems we now have with extensions going to recur in a new form?

At the moment I observe a group of people overlapping with, and working closely with, the TCT, Activestate and some other corporate, government and educational entities. I observe this affiliation group providing core engineering of a very high standard. Given the effort and expense of the engineering, and the number of single points of potential failure in this organisation, I wonder about the economics of this binary repository proposal (however desireable it may be.)

PWQ XX, I don't see any need for the repository to be maintained. If an application depends (requests) a particular extension then what is important is that 10 years later when a tcl-newbie finds the application and trys to run it, that there is a process that will assist the user to locate the extension and make the application work. Anything more that that is useful but not critical.

CMcC Extension locations for newbies could be much more easily implemented via the wiki.

Lars H, 17 Jan 2005: A few questions.

Is that point about stubs not being extendable true? There is such a thing as Extension Stubs Tables.
Why should the Tcl_Obj structure be variable length? Anyone can allocate another piece of memory and have the internal representation be a pointer to that.
As for registering extensions to Tcl_Obj format... Is there any room left between what Tcl_RegisterObjType [1] provides and (the very nondynamic language) C permits?

DKF: The potential advantage to having a variable length Tcl_Obj format is that it allows for more than one internal representation at once. The down-side to that is that it becomes much more complex to work such instances, and they are also more costly to allocate.

PWQ Lars, When I talk of extending TCL, I am not limiting myself to adding new features. Just think back to when image transparancy was not in the core. Not one extension to TCL could make transparancy with images work until the core was modified.

What I propose is a system that would have allowed people to write extensions that change the way the existing core works. This would have meant that when transparancy was first discussed, an example extension could have been written that demonstrated with the current version of tcl/tk how it would look and work. Everyone could have tried the extension without having to patch their tcl interpreter.

Some time later the TCT could take the best ideas from the work and put them directly into the core so that overriding and duplicating existing core functions would not be necessary.

A new function may want to stack itself (hook) into a number of subsystems, some of these may not even be loaded yet. The current stubs mechanism (which is optional by the way) is restricted to compile time linking, so it does not lend itself in it's present form to the facilities that I have outlined.

Lars H: The entire purpose of stubs is to provide run-time linking, is it not? Certainly the prototypes for the stubs table entries have to be known at compile time, but that's a restriction coming from C and is nothing a redesign of the Tcl library can do much about. Your main complaint seems to be that the stubs don't give you enough control over the existing core, not that they aren't extendable.

PWQ When it comes to the Tcl_Obj structure, I am not talking about creating new data types. What I am talking about is adding possibilities (like paul duffin's feather). What if you wanted to add multiple representations to a object. The present structure only allows one at any one time. By making the structure variable length we allow these additions without requiring any changes to the existing system.

Lars H: Except that you need to modify all old code so that it supports multiple representations. That's not just something you could add as an extension, even if the core had supported variable-length Tcl_Objs and stackable stubs, it would also require that everything accessing Tcl_Objs directly interpret the variable length this way.

Of course, one consistent interpretation of your remarks is that you really dislike static typing, and would like to see it exorcised from the Tcl library. I can sympathise with that (I too dislike static typing), but you can't get away from it when programming in C. Making a Tcl library with a fully dynamic API would be equivalent to reimplementing it in some typeless semi-language, and then this reduces to an exercise in emulation. It's probably better to do that in Tcl than trying to do it in C.

PWQ Consider the use of tags in python. What if some user wanted to tag arbitary objects with meta data for some good purpose. They can't change the objects type as that would stop the application working or cause endless shimmering (losing the tagged information). Rather than having to have say a private hash table and registering each object it sees (and handling object deletion manually), they could register a place to store the information directly in the object, that then gets passed around with the object until the object is deleted.

An alternative would be to add a pointer that could be used like the aux-data pointer for the interp structure. This would be much less efficient and less flexible.

PWQ Dkf, I dont believe it would be more expensive to allocate.

DKF: It's not whether or not you believe it, it'd still be true. :^) Tcl_Obj structure instances are the most commonly allocated thing in Tcl by at least one order of magnitude (you can find this out for yourself by analysing the output from the memory command) and the Tcl core goes to considerable lengths to make sure that this is as fast an operation as possible. In particular, it takes advantage of the fact that all objects are the same size. If they are turned into variable-size entities (something which the core uses extensively elsewhere, so we are familiar with the techniques just as we are also familiar with Feather) they have to be allocated using the lower-level allocation subsystem instead, and that is considerably more expensive to use. Hence switching to a variable-sized Tcl_Obj structure would make every object more expensive to allocate. QED.

Now explain just how many objects really need to be multi-ported. (Note that in Tcl, the metadata would be attached to variables and not values due to other concerns relating to transparency of object representation and convertability of values. I ought to write more on these topics sometime.)

Small Core == Good

Category Internals

Category Discussion