Reading and writing to a piped command See Also edit
- open
- pipe
- gzip
- unbuffer
- How can I run data through an external filter?
Description edit
The methods for accomplishing some task involving writing to and then reading from a piped command depend primarily on whether the communication will be interactive or non-interactive. The non-interactive case is fairly straight-forward and robust. The interactive case can be problematic, since external programs may buffer incoming and outgoing data in arbitrary ways.
Expect is the best all-purpose tool for interactive communication with another program.
Example: Non-Interactive: gzip edit
proc gzip {buf} {
set chan [open "|gzip -c" r+]
fconfigure $chan -translation binary -encoding binary
puts $chan $buf
flush $chan
chan close $chan write
set buf [read $chan]
close $chan
return $buf
}The
write side of
$chan must be closed so that gzip reads any remaining data in the buffers processes it, and sends the result to its stdtout.
For versions of Tcl that don't have
[chan close ?direction?
], try the following workaround:
proc gzip {buf} {
return [exec gzip -c << $buf]
} Example: Interactive Conversation with another Program edit
#! /bin/env tclsh
proc communicate {chan msg pump respond eof } {
fileevent $chan readable [list apply [list {chan respond eof} {
set res [gets $chan]
if {$res ne {}} {
fileevent $chan readable {}
{*}$respond $res
}
if {[eof $chan]} {
fileevent $chan readable {}
{*}$eof
}
}] $chan $respond $eof]
fileevent $chan writable [list apply [list {chan msg pump} {
catch {puts $chan $msg}
#something like this may be needed if the child program can be configured
#with reasonable output buffering
#fileevent $chan writable [list apply [list {chan pump} {
# puts $chan $pump
#}] $chan $pump]
fileevent $chan writable {}
}] $chan $msg $pump]
}
proc process {chan count args} {
puts "sed output: $args"
if {$count < 2} {
communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
[namespace code [list closed $chan]]
} else {
catch {close $chan}
set ::done 0
}
}
proc closed {chan} {
#flush first to catch any pipe error, getting it out of the way in order to grab
#the exist status with [close]
if {[catch {flush $chan} eres eopts]} {
set status [catch {close $chan} eres eopts]
set ::done $status
}
set ::done 0
}
#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]
set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none
communicate $chan hello \n [namespace code [list process $chan 1]] \
[namespace code [list closed $chan]]
vwait ::done
return -code $::doneIt's up to the individual program when to print out its results. The
-l option to
BSD sed configures sed to print output whenever at least one line of output is ready.
-u accomplishes the same for some other versions of sed. The
pump feature of this example can be used pump some program-specific value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task.
Expect is the fully-featured tool for this type of task. Another approach would be to
fork the current process into a producer and a consumer.
Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is
ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing
LaTeX markup. There are a number of issues to deal with:
- Keeping track of the position of the word in the widget.
- Filtering out useless (blank) lines from ispell
- Filtering out the version info that my version of ispell dumps out.
- When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
- Careful handling of blocking.
The example does not use [
fileevent], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
set wordnum 1
foreach word [split $line] {
puts $pipe $word ;# Feed word to ispell
while 1 {
set len [gets $pipe res]
if {$len > 0} {
# A valid result
# do stuff
continue
} else {
if {[fblocked $pipe]} {
# No output
break
} elseif {[eof $pipe]} {
# Pipe closed
catch {close $pipe}
return
}
# A blank line - skip
}
}
incr wordnum
}
incr linenum
}Thanks to
Kevin Kenny for helping me figure this out.
Arjen Markus: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from
Android :-).
I think it is worth a page of its own, but here is a summary:
- Open the pipeline and make sure buffering is minimal via
set inout [open |[list myprog] r+]
fconfigure $inout -buffering line
Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close.
- Set up [fileevent] handlers for reading from the process and reading from stdin.
- Make sure the process ("myprog" above) does not buffer its output to stdout!
