f ranz i nc. an in-depth look at simple streams by duane rettig october, 2002
TRANSCRIPT
FRANZ INC.
An In-Depth Look at Simple Streams
By Duane Rettig
October, 2002
FRANZ INC.
What is a Lisp Stream?
Lisp
stream
stream
externaldevice
internal device
FRANZ INC.
An In-Depth Look at Simple Streams
• Simple Streams Design Goals• What is a Simple Stream?• Simple-stream Concepts
– Opening, closing, blocking, reading, writing
• Character Strategies• Common Windows Rewrite• Further changes in next version (references
distributed throughout presentation)
FRANZ INC.
Design Goals
• CL compliance• Bivalence• Thin Strategy layer that is uniform• Extensibility via specializations/mixins and
encapsulations• Minimal decision points in critical strategy code• Minimal calls to generic-functions• External-format ready for 8 and 16 bit lisps
FRANZ INC.
Gray vs. Simple Stream
Object functionality
buffer
API call
Strategy
Gray Stream
Simple Stream
Device interface
FRANZ INC.
Dual-channel stream
• Two octet buffers• External-format character translation
Ext. format
Device methods
Out buffer
API
In buffer
External device
FRANZ INC.
Single-channel stream
• One octet buffer• External-format character translation
Ext. format
Device methods
buffer
API
direction
Internal or External device
FRANZ INC.
String stream
• Zero, one, or two string buffers• No external-format translations• Plugable character/buffer strategies
Device methods
Out buffer
API
In buffer
Internal device
FRANZ INC.
Opening a Stream
• Creates or reuses a stream object
• Makes or retains a connection to a device
• Initializes character strategies
• Marks the stream as open
FRANZ INC.
Open
open make-instance (gf) ... shared-initialize (gf) shared-initialize :after [Method] device-open [Method]
FRANZ INC.
(Open): Simple encapsulation
(setq bun (open "sesame"))
(setq big-mac (make-instance 'all-beef :input-handle bun ...))
bunbig-mac
all-beef sesame
FRANZ INC.
(Open): shared-initialize
(defmethod shared-initialize :after ((stream simple-stream) slots
&rest initargs) (declare (ignore slots)) (unless (device-open stream initargs) (device-close stream t)) stream)
FRANZ INC.
(Open): device-open
• At device-open time:– All instance slots have been initialized– connection has been made or will be made by
device-open
• When device-open returns:– If non-nil, the stream is ready for appropriate
actions – If nil, then the open has failed.
FRANZ INC.
(Open): device-open
• device-open must:– Ensure a connection before returning– Ensure buffers are in place– Initialize/reinitialize pointers– Return nil for failure, or non-nil after success
• device-open must not:– Assume that the stream was closed– close the stream
FRANZ INC.
(Open): subclass example
(defclass file-with-header (file-simple-stream) ((header :initform nil :accessor file-header-info)))
(defmethod device-open ((stream file-with-header) options) (declare (ignore options)) (let ((success (call-next-method))) (when success (setf (file-header-info stream) (read-file-header stream)) t)))
FRANZ INC.
(Open): device-open :before methods
• In future releases, for string-simple-streams:– A catch-all primary method returns true for an
open stream.– Strategy installation functions are directional and
do not override.• install-string-character-strategy deprecated, replaced
by install-string-{input,output}-character-strategy
– All :before methods fire according to CPL, and may shadow default actions.
FRANZ INC.
(Open): device-open: string-input (future)
(defmethod device-open :before ((stream string-input-simple-stream) options) (with-stream-class (string-input-simple-stream stream) (let ((string (getf options :string))) (when (and string (null (sm buffer stream))) (let ((start (getf options :start)) (end (or (getf options :end) (length string)))) (setf (sm buffer stream) string (sm buffpos stream) start (sm buffer-ptr stream) end))))) (install-string-input-character-strategy stream) (add-stream-instance-flags stream :string :input :simple))
FRANZ INC.
(Open): device-open: string-output (future)(defmethod device-open :before ((stream string-output-simple-stream) options) (with-stream-class (string-output-simple-stream stream) (unless (sm out-buffer stream) (let ((string (getf options :string))) (if string (setf (sm out-buffer stream) string (sm max-out-pos stream) (length string)) (let ((buflen (max (device-buffer-length stream) 16))) (setf (sm out-buffer stream) (make-string buflen) (sm max-out-pos stream) buflen))))) (unless (sm control-out stream) (setf (sm control-out stream) *std-control-out-table*))) (install-string-output-character-strategy stream) (add-stream-instance-flags stream :string :output :simple))
FRANZ INC.
Closing a Stream
• Flushes output if any.• Breaks connection to device.• Secures against accidental future operations.• Does not change-class.
(defmethod close ((stream simple-stream) &key abort) (device-close stream abort))
FRANZ INC.
Close(defmethod device-close :around ((stream simple-stream) abort) (let (res) (when (pseudo::open-stream-p stream) (unwind-protect (progn (when (output-stream-p stream) (ignore-errors (if abort (clear-output stream) (force-output stream)))) (setq res (call-next-method))) (without-interrupts (pseudo::unset-open-flags stream) (setf (stream-input-handle stream) nil (stream-output-handle stream) nil)) (setf (stream-external-format stream) (find-external-format :void)) res))))
FRANZ INC.
(Close): device-close
• device-close should:– flush all data (unless aborting)– disconnect handles of any encapsulated streams– call lower-levels to close as necessary
• device-close should not:– operate on a closed stream– close an encapsulated stream
FRANZ INC.
Blocking
• Issues with listen– read direction only– covers character availability and not blocking– assumes character as the basic data unit
• stream-listen (carried over from Gray streams)
• read-no-hang-p and write-no-hang-p
FRANZ INC.
Blocking styles
• Non-blocking
• Blocking
• B/NB (blocking, then non-blocking)
Character: One element
Octet: One element
FRANZ INC.
Reading and Writing
• Mostly symmetrical
• Device methods obey B/NB discipline
FRANZ INC.
Basic read strategy
(block read (when (>= buffpos buffer-ptr) (let ((res (device-read stream nil 0 buffpos blocking))) (when (< res 0) (pseudo::do-error-handling)) (when (= res 0) (if blocking (pseudo::do-eof-handling) (return-from read nil))) (setq buffer-ptr res buffpos 0))) (prog1 (aref buffer buffpos) (incf buffpos)))))
FRANZ INC.
Basic write strategy
(block write (when (>= out-pos max-out-pos) (when (> out-pos 0) (erroring-device-write-whole stream nil 0 out-pos t)) (setq out-pos 0)) (setf (aref buffer out-pos) value) (incf out-pos))
FRANZ INC.
Implementing force-output, finish-output
• Both are implemented using device-write
• blocking argument determines whether force (nil) or finish (t)
• device-write buffer argument is :flush (future)
FRANZ INC.
Reading and Writing Sequences
• read-sequence and write-sequence– are width sensitive and by-element– require blocking semantics– do not return a count
• read-vector and write-vector– are octet-based– employ B/NB semantics– return a count, 0, or error code
FRANZ INC.
Deprecated generic function: device-extend
• History and relationship to device-read and device-write– structural differences– purity of reference
• Problems– one generic-function for two directions– inconsistent interface
FRANZ INC.
Replacement of device-extend
• Actions :input and :input-check become blocking argument to device-read
• Actions :output and :output-check become blocking argument to device-write
• Extra actions come through via device-read and device-write buffer argument
FRANZ INC.
Record orientation
• Via device-finish-record gf– dual-channel input– string output
FRANZ INC.
Other generic-functions
• device-buffer-length
• device-clear-input
• device-clear-output
• device-file-length
• device-file-position
FRANZ INC.
Strategies
• What is a strategy’s purpose?– To satisfy the high-level requirements of a
specified behavior in a uniform manner.
• Octet strategies are non-programmable.– read-byte, write-byte are cast in concrete.
• Character strategies are programmable...
FRANZ INC.
Character strategies
• Programmable, replaceable.
• External-formats use template functions for dual-channel and single-channel streams.
• String-streams use simple strategy functions.
• (future): Subclassed strategy functions can shadow more general ones.
• Occupy “joint” slots in the stream.
FRANZ INC.
Example strategy set for *terminal-io*
cl-user(5): :iterminal-simple-stream @ #x711e057a = #<terminal-simple-stream [initial terminal io] fd 0/1 @ #x711e057a> 0 Class --------> #<standard-class terminal-simple-stream> 1 j-unread-char -> #<Function dual-channel-unread-char> 2 j-write-chars -> #<Function (:efft dc-write-chars :latin1-base)> 3 j-write-char -> #<Function (:efft dc-write-char :latin1-base)> 4 j-read-chars -> #<Function (:efft dc-read-chars :latin1-base)> 5 j-read-char --> #<Function (:efft dc-read-char :latin1-base)> 6 j-listen -----> #<Function (:efft dc-listen :latin1-base)> ... 34 src-position-table -> The symbol nilcl-user(6):
FRANZ INC.
Example strategy set for string-output
cl-user(9): :istring-output-simple-stream @ #x719cb5e2 = #<string-output-simple-stream "" pos 0 @ #x719cb5e2> 0 Class --------> #<standard-class string-output-simple-stream> 1 j-unread-char -> The symbol nil 2 j-write-chars -> #<Function string-output-write-chars> 3 j-write-char -> #<Function string-output-write-char> 4 j-read-chars -> The symbol nil 5 j-read-char --> The symbol nil 6 j-listen -----> The symbol nil ... 33 out-buffer ---> A simple-string (4096) that starts "xxxxxxxxxxxxxxxx”cl-user(10):
FRANZ INC.
Example strategy set for string-input
cl-user(14): :istring-input-simple-stream @ #x719e74aa = #<string-input-simple-stream "abc" pos 0 @ #x719e74aa> 0 Class --------> #<standard-class string-input-simple-stream> 1 j-unread-char -> #<Function string-input-unread-char> 2 j-write-chars -> The symbol nil 3 j-write-char -> The symbol nil 4 j-read-chars -> #<Function string-input-read-chars> 5 j-read-char --> #<Function string-input-read-char> 6 j-listen -----> #<Function string-listen> ... 31 src-position-table -> The symbol nilcl-user(15):
FRANZ INC.
j-read-char
• Args: stream eof-error-p eof-value blocking
• Implements read-char and read-char-no-hang functionality directly (after argument resolution)
• blocking argument is nil/true dichotomy
FRANZ INC.
j-write-char
• Args: character stream
• Implements write-char directly (after argument resolution)
• Writing nil as the character argument will flush an external-format’s output state.
FRANZ INC.
j-read-chars
• Args: stream string search start end blocking
• Implements read-sequence, read-line
• blocking argument is nil, :bnb, or true
FRANZ INC.
j-write-chars
• Args: stream string start end
• Implements write-sequence, write-string
• Always blocks (for now)
FRANZ INC.
j-listen
• Args: stream
• Implements stream-listen directly
• Must have a complete character or an error condition in order to return true.
FRANZ INC.
j-unread-char
• Args: stream relaxed
• May need to unread multiple characters as part of a composing or encapsulating character.
• relaxed allows non-error unreads beyond the first unread.
FRANZ INC.
Taking strategies beyond ANS
• Other strategies can be created for operations that don’t fit into CL standard functionalities.
• Example: one-buffer ring fifo queue
FRANZ INC.
Common Windows low-level rewrite
• Source code comparison between Gray and simple streams
• Demo