Updated 2010-05-17 22:08:43 by AMG

JR: This is a simple critcl function to read an entire file a line at a time and return it as a list.
    package require critcl 0.33

    critcl::ccode {
      #include <stdio.h>
    }

    critcl::cproc loadf {char* fn} Tcl_Obj* {
        FILE *fd;
        Tcl_Obj *r,*s;
        char line[1024];
        r=Tcl_NewListObj(0,0);
        Tcl_IncrRefCount(r);
        s=Tcl_NewStringObj("",0);
        fd=fopen(fn,"r");
        if (!fd) {
            return r;
        }
        while(!feof(fd)) {
            int l;
            if (!fgets(line,1024,fd)) {
                break;
            }
            l=strlen(line);
            if (line[l-1]=='\n') {
                line[--l]='\0';
                Tcl_AppendToObj(s,line,l);
                Tcl_ListObjAppendElement(NULL,r,s);
                s=Tcl_NewStringObj("",0);
            } else {
                Tcl_AppendToObj(s,line,l);
            }
        }
        fclose(fd);
        if (Tcl_GetCharLength(s) > 0) {
            Tcl_ListObjAppendElement(NULL,r,s);
        }
        return r;
    }

    # tcl version for comparison
    proc loadf_tcl {file} {
        set f [open $file]
        fconfigure $f -translation binary
        set d [read -nonewline $f]
        close $f
        split $d \n
    }

    # use
    set lines [loadf something]

The critcl version is as much as 3 times faster than the tcl version on a large file.

I just noticed that on files with lots of short lines (e.g., /usr/share/dict/words) the critcl version is actually slower than the tcl version. This bears looking into.

AMG: Maybe your fgets() is buggy. I suggest using strace to see if it's making unreasonable numbers of read syscalls.

With the conventional wisdom that [read [open $file] [file size $file]] is faster than [read [open $file]], I wonder if the critcl function is still 3x faster.

Wow. I've never ever used critcl before. Now I'm convinced. I didn't even know I had critcl on my system, but the above cut'n'pasted and Just Worked. Nifty. And yes, the critcl version is still faster (after the initial compile)

AMG: Conventional wisdom? Why would it be faster to tell read how many bytes to get?