semantic pipes (london perl workshop 2009)
DESCRIPTION
Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline. But the technique of piping a *textual* stream of data, and having to extract data out of it is looking a bit long in the tooth. Microsoft (not historically an innovator in its shell environment :-) has stolen a march with its Powershell. Can we do better in Perl? With composable streams of objects? Written in a modern OO framework (Moose)? You bet we can! This talk was given at the London Perl Workshop 2009, http://conferences.yapceurope.org/lpw2009/talk/2456TRANSCRIPT
![Page 1: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/1.jpg)
Semantic Pipes
osfameron
London Perl Workshop 5 Dec 2009
![Page 2: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/2.jpg)
Everyone likes pipes
![Page 3: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/3.jpg)
![Page 4: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/4.jpg)
Unix philosophy
“This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams,
because that is a universal interface.”– Doug McIlroy
http://www.faqs.org/docs/artu/ch01s06.html
![Page 5: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/5.jpg)
Pipes
Foo foo fooBar bar barBaz baz baz
foo bar baz
generate
map
![Page 6: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/6.jpg)
Pipes
Foo foo fooBar bar barBaz baz baz
Bar bar bar
generate
grep
![Page 7: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/7.jpg)
Pipes
Foo foo fooBar bar barBaz baz baz
Bar bar barBaz baz bazFoo foo foo
generate
sort
![Page 8: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/8.jpg)
![Page 9: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/9.jpg)
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
![Page 10: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/10.jpg)
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
grep
![Page 11: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/11.jpg)
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
sort
![Page 12: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/12.jpg)
ls
SYNOPSIS ls [OPTION]... [FILE]...
OPTIONS
--format=WORD across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN do not list implied entries matching shell PATTERN
-r, --reverse reverse order while sorting
--sort=WORD none -U, extension -X, size -S, time -t, version -v
map/format
![Page 13: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/13.jpg)
ls | … ?
ls | grep
ls | wc -l
![Page 14: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/14.jpg)
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
![Page 15: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/15.jpg)
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
using cut
![Page 16: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/16.jpg)
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 READMEdrwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
cut … --bytes? or --delimiter?
![Page 17: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/17.jpg)
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽
![Page 18: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/18.jpg)
ls -ltr | … ?
total 3272-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe˽˽˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README˽˽˽˽˽˽˽˽˽drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list˽˽˽˽˽˽˽˽˽-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp˽˽
![Page 19: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/19.jpg)
OK, find...
● rather baroque● separate command● “consistent interface” ?
![Page 20: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/20.jpg)
So who does do piping properly?
![Page 21: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/21.jpg)
So who does do piping properly?
![Page 22: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/22.jpg)
Windows Powershell
● originally called Monad Shell● (this is important)
● streams of .Net objects● flexible and consistent filter/map/sort
![Page 23: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/23.jpg)
Disclaimer
● I've never used Powershell...
![Page 24: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/24.jpg)
Disclaimer
● I've never used Powershell...● but the idea is genuinely exciting
![Page 25: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/25.jpg)
![Page 26: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/26.jpg)
Pipe.pm
package Pipe;use Moose;extends 'MooseX::App::Cmd'; 1;
![Page 27: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/27.jpg)
pipe
#!/usr/bin/perluse lib 'lib';use Pipe;Pipe->run;
![Page 28: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/28.jpg)
Pipe.pm
![Page 29: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/29.jpg)
Pipe::Command::find
use Moose;extends 'Pipe::Command';use MooseX::Types::Path::Class qw(File to_File);use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } });}
![Page 30: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/30.jpg)
Pipe::Command::grephas where => ( isa => 'Str', is => 'rw', documentation => 'string to match', );has where_sub => ( … );
sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; };}
![Page 31: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/31.jpg)
![Page 32: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/32.jpg)
Pipe in action
$ ./pipe find lib
List of files found!
lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
![Page 33: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/33.jpg)
Pipe in action
$ ./pipe find lib | ./pipe grep '$_->stat->size > 1000'
List of files found!
lib/Pipe/Command.pm
![Page 34: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/34.jpg)
Pipe::Command::find
sub pretty_print { my ($self, $session) = @_; say "List of files found!\n\n"; my $it = $session->iterator; while (my @files = $it->()) { say join "\n", @files; }}
![Page 35: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/35.jpg)
Pipe to an external tool
$ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~
![Page 36: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/36.jpg)
Pipe to an external tool
$ ./pipe find lib –-pretty | cat
lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
![Page 37: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/37.jpg)
![Page 38: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/38.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
generate
Foo foo fooBar bar barBaz baz baz
![Page 39: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/39.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
generate
Foo foo fooBar bar barBaz baz bazI'm using YAML,
but could just as well be XML etc.
![Page 40: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/40.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
foo bar baz
![Page 41: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/41.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
![Page 42: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/42.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
●nothing left to pipe to (! -p)●or --pretty
![Page 43: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/43.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
bar baz foo
●nothing left to pipe to (! -p)●or --pretty
![Page 44: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/44.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
pretty_print
![Page 45: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/45.jpg)
Database
● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz
![Page 46: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/46.jpg)
Database
● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz
![Page 47: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/47.jpg)
“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
![Page 48: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/48.jpg)
“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
![Page 49: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/49.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
![Page 50: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/50.jpg)
Pipesapp: !!Pipestack: generate blah blah blah
app: !!Pipestack: map generate
blah blah blah
generate
map
app: !!Pipestack: sort map generate
blah blah blah
sort
![Page 51: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/51.jpg)
Pipes Monadsapp: !!Pipestack: generate
app: !!Pipestack: map generate
generate
map
app: !!Pipestack: sort map generate
sort
generate>> map>> sort>> pretty_print (first [sort, map, generate])
![Page 52: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/52.jpg)
![Page 53: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/53.jpg)
![Page 54: Semantic Pipes (London Perl Workshop 2009)](https://reader033.vdocuments.us/reader033/viewer/2022051817/548e2ff5b4795956138b4913/html5/thumbnails/54.jpg)
Thank you● Details/links
● Early version of this talk given at NorthWestEngland.pm, 1st July 2009● email: [email protected]● http://github.com/osfameron/pipe/● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example)
● Images● CC-licensed, via Flickr (thanks for making your photos free to use!!)
– http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon– http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad– http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell– http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto!– http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill– http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-...– http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden
● Classic images, via Google– Einstein with pipe– I'm a PC, from the Mac ads– Ceci n'est pas une pipe– Einstein