## FLEXI-STREAMS - Flexible bivalent streams for Common Lisp

> ### Abstract

> FLEXI-STREAMS implements "virtual" bivalent streams that can be layered atop real binary or bivalent streams and that can be used to read and write character data in various single- or multi-octet encodings which can be changed on the fly. It also supplies in-memory binary streams which are similar to string streams.

> The library needs a Common Lisp implementation that supports Gray streams and relies on David Lichteblau's trivial-gray-streams to offer portability between different Lisps.

> The code comes with a BSD-style license so you can basically do with it whatever you want.

> Download shortcut: https://github.com/edicl/flexi-streams/releases/latest

### Contents

1. Example usage
2. Download and installation
3. Support
4. The FLEXI-STREAMS dictionary
   1. External formats
      1. `make-external-format`
      2. `external-format-name`
      3. `external-format-eol-style`
      4. `external-format-little-endian`
      5. `external-format-id`
      6. `external-format-equal`
      7. `*default-eol-style*`
      8. `*default-little-endian*`
      9. `external-format-condition`
      10. `external-format-condition-external-format`
      11. `external-format-error`
      12. `external-format-encoding-error`
      13. `*substitution-char*`
      14. `accept-overlong-sequence`
   2. Flexi streams
      1. `flexi-stream`
      2. `flexi-input-stream`
      3. `flexi-output-stream`
      4. `flexi-io-stream`
      5. `make-flexi-stream`
      6. `flexi-stream-external-format`
      7. `flexi-stream-element-type`
      8. `flexi-stream-column`
      9. `flexi-stream-position`
      10. `flexi-stream-bound`
      11. `flexi-stream-stream`
      12. `unread-byte`
      13. `peek-byte`
      14. `octet`
      15. `flexi-stream-error`
      16. `flexi-stream-out-of-sync-error`
      17. `flexi-stream-element-type-error`
      18. `flexi-stream-element-type-error-element-type`
   3. In-memory streams
      1. `in-memory-stream`
      2. `in-memory-input-stream`
      3. `in-memory-output-stream`
      4. `list-stream`
      5. `vector-stream`
      6. `make-in-memory-input-stream`
      7. `make-in-memory-output-stream`
      8. `get-output-stream-sequence`
      9. `output-stream-sequence-length`
      10. `with-input-from-sequence`
      11. `with-output-to-sequence`
      12. `in-memory-stream-error`
      13. `in-memory-stream-closed-error`
      14. `in-memory-stream-position-spec-error`
      15. `in-memory-stream-position-spec-error-position-spec`
   4. Strings
      1. `string-to-octets`
      2. `octets-to-string`
      3. `octet-length`
      4. `char-length`
5. File positions
6. Acknowledgements

### Example usage

The examples were created with LispWorks 4.4.6 pro on Windows. The following two functions create the same file:

    (defun foo (pathspec)
      "With standard LispWorks streams."
      (with-open-file (out pathspec
                           :direction :output
                           :if-exists :supersede
                           :external-format '(:utf-8 :eol-style :crlf))
        (write-line "ÄÖÜ1" out))
      (with-open-file (out pathspec
                           :direction :output
                           :if-exists :append
                           :external-format '(:latin-1 :eol-style :lf))
        (write-line "ÄÖÜ2" out))
      (with-open-file (out pathspec
                           :direction :output
                           :if-exists :append
                           :element-type 'octet)
        (write-byte #xeb out)
        (write-sequence #(#xa3 #xa4 #xa5) out))
      (with-open-file (out pathspec
                           :direction :output
                           :if-exists :append
                           :external-format '(:unicode :little-endian nil :eol-style :crlf))
        (write-line "ÄÖÜ3" out)))

    (defun bar (pathspec)
      "With a flexi stream."
      (with-open-file (out pathspec
                           :direction :output
                           :if-exists :supersede
                           :external-format '(:latin-1 :eol-style :lf))
        (setq out (make-flexi-stream out :external-format :utf-8))
        (write-line "ÄÖÜ1" out)
        (setf (flexi-stream-external-format out) '(:latin-1 :eol-style :lf))
        (write-line "ÄÖÜ2" out)
        (write-byte #xeb out)
        (write-sequence #(#xa3 #xa4 #xa5) out)
        (setf (flexi-stream-external-format out) :ucs-2be)
        (write-line "ÄÖÜ3" out)))

And applying this function

    (defun baz (pathspec)
      (let (result)
        (with-open-file (in pathspec :element-type 'octet)
          (setq in (make-flexi-stream in :external-format :utf-8))
          (push (read-line in) result)
          (push (read-byte in) result)
          (setf (flexi-stream-external-format in) '(:latin-1 :eol-style :lf))
          (push (read-line in) result)
          (setf (flexi-stream-external-format in) :greek)
          (push (read-char in) result)
          (setf (flexi-stream-external-format in) :latin0)
          (let ((string (make-string 3 :element-type 'character)))
            (read-sequence string in)
            (push string result))
          (let ((octets (make-array 2 :element-type 'octet)))
            (read-sequence octets in)
            (push octets result))
          (setf (flexi-stream-external-format in) :ucs-2be)
          (push (read-line in) result))
        (nreverse result)))

to the file created above will yield the list

    ("ÄÖÜ1" 196 "ÖÜ2" #\λ "£€¥" #(0 196) "ÖÜ3")

For more examples see the source code of CL-RFC2047, Drakma, Chunga, or CL-WBXML.

### Download and installation

Before you try to install FLEXI-STREAMS, first check that in your Lisp each character's character code is equal to its Unicode code point and that `(CHAR-CODE #\Newline)` and `(CHAR-CODE #\Linefeed)` have the same value (10). (This is the case for all relevant CL implementations which were in use when this library was written. It is not mandated by the ANSI standard, though.)

FLEXI-STREAMS together with this documentation can be downloaded from https://github.com/edicl/flexi-streams/releases/latest.

Before you install FLEXI-STREAMS you first need to install the trivial-gray-streams library unless you already have it.

FLEXI-STREAMS comes with a system definition for ASDF so you can install the library with

    (asdf:oos 'asdf:load-op :flexi-streams)

if you've unpacked it in a place where ASDF can find it. Installation via asdf-install should also be possible, and there's a port to Gentoo Lisp thanks to Matthew Kennedy.

You can run a test suite which tests some (but not all) aspects of the library with

    (asdf:oos 'asdf:test-op :flexi-streams)

This might take a while...

The current development version of FLEXI-STREAMS can be found at http://bknr.net/trac/browser/trunk/thirdparty. This is the one to send patches against. Use at your own risk.

Luís Oliveira maintains a darcs repository of FLEXI-STREAMS at http://common-lisp.net/~loliveira/ediware/.

A Mercurial repository of older versions is available at http://arcanes.fr.eu.org/~pierre/2007/02/weitz/ thanks to Pierre Thierry.

### Support

The development version of flexi-streams can be found on github. Please use the github issue tracking system to submit bug reports. Patches are welcome, please use GitHub pull requests. If you want to make a change, please read this first.

### The FLEXI-STREAMS dictionary

#### External formats

`EXTERNAL-FORMAT` objects are used to denote the external formats of flexi streams. These objects are created using the `MAKE-EXTERNAL-FORMAT` function, and there are various readers to query their attributes. Once such an object is created it can't be changed.

An external format consists of a basic encoding (like ISO 8859-1 or UTF-8), a definition how line endings are denoted - by a carriage return character (ASCII 13), by a line feed character (ASCII 10), or by both of these characters in a row -, and optionally (for encodings that use units larger than 8 bits) information about the endianess of the encoding.

The following encodings are currently supported by FLEXI-STREAMS:

- UTF-8 (denoted by the keyword `:UTF-8`),
- UTF-16 (denoted by the keyword `:UTF-16`),
- UTF-32 (denoted by the keyword `:UTF-32`),
- all ISO 8859 character sets (denoted by keywords like `:ISO-8859-15`),
- KOI8-R (denoted by the keyword `:KOI8-R`),
- a couple of Windows code pages (denoted by the keyword `:CODE-PAGE` and an obligatory `:ID` argument), and
- US-ASCII.

A couple of alternative names are allowed that are listed below:

`:UTF-8` | `:UTF8`
`:UTF-16` | `:UTF16`
`:UCS-2`
`:UCS2`
`:UNICODE`
`:UTF-32` | `:UTF32`
`:UCS-4`
`:UCS4`
`:ISO-8859-1` | `:LATIN-1`
`:LATIN1`
`:ISO-8859-2` | `:LATIN-2`
`:LATIN2`
`:ISO-8859-3` | `:LATIN-3`
`:LATIN3`
`:ISO-8859-4` | `:LATIN-4`
`:LATIN4`
`:ISO-8859-5` | `:CYRILLIC`
`:ISO-8859-6` | `:ARABIC`
`:ISO-8859-7` | `:GREEK`
`:ISO-8859-8` | `:HEBREW`
`:ISO-8859-9` | `:LATIN-5`
`:LATIN5`
`:ISO-8859-10` | `:LATIN-6`
`:LATIN6`
`:ISO-8859-11` | `:THAI`
`:ISO-8859-13` | `:LATIN-7`
`:LATIN7`
`:ISO-8859-14` | `:LATIN-8`
`:LATIN8`
`:ISO-8859-15` | `:LATIN-9`
`:LATIN9`
`:LATIN-0`
`:LATIN0`
`:ISO-8859-16` | `:LATIN-10`
`:LATIN10`
`:CODE-PAGE` | `:CODEPAGE`
`WIN32:CODE-PAGE (only on LWW)`
`:KOI8-R` | `:KOI8R`
`:US-ASCII` | `:ASCII`

(Note that we treat UCS-2 exactly like UTF-16 although there are subtle differences. Also note that even though we support encodings like UTF-32 some Lisps only supports characters contained within the Basic Multilingual Plane (like LispWorks) or even less (like CMUCL), so if other characters are read from a flexi stream, `READ-CHAR` will try to be helpful and return the corresponding Unicode code point - an integer - instead. This might lead to an error if you're using functions like `READ-LINE`, though.)

Whenever a FLEXI-STREAMS function accepts an external format as one of its arguments, you can provide either an `EXTERNAL-FORMAT` object or a shortcut which can be a list or a symbol. The list shortcuts have a syntax similar to the one used by LispWorks - the cars are the names of and encoding and the cdrs of these lists correspond to the keyword arguments to `MAKE-EXTERNAL-FORMAT`, so for example

    (:latin-1 :eol-style :crlf)

is equivalent to

    (make-external-format :latin-1 :eol-style :crlf)

The symbol shortcuts are equivalent to calling `MAKE-EXTERNAL-FORMAT` without keyword arguments, i.e.

    :thai

behaves like

    (make-external-format :thai)

Finally, the following expansions are available:

`:UCS-2LE` | `(:UCS-2 :LITTLE-ENDIAN T)`
`:UCS-2BE` | `(:UCS-2 :LITTLE-ENDIAN NIL)`
`:UCS-4LE` | `(:UCS-4 :LITTLE-ENDIAN T)`
`:UCS-4BE` | `(:UCS-4 :LITTLE-ENDIAN NIL)`
`:UTF-16LE` | `(:UTF-16 :LITTLE-ENDIAN T)`
`:UTF-16BE` | `(:UTF-16 :LITTLE-ENDIAN NIL)`
`:UTF-32LE` | `(:UTF-32 :LITTLE-ENDIAN T)`
`:UTF-32BE` | `(:UTF-32 :LITTLE-ENDIAN NIL)`
`:IBM437` | `(:CODE-PAGE :ID 437)`
`:IBM850` | `(:CODE-PAGE :ID 850)`
`:IBM852` | `(:CODE-PAGE :ID 852)`
`:IBM855` | `(:CODE-PAGE :ID 855)`
`:IBM857` | `(:CODE-PAGE :ID 857)`
`:IBM860` | `(:CODE-PAGE :ID 860)`
`:IBM861` | `(:CODE-PAGE :ID 861)`
`:IBM862` | `(:CODE-PAGE :ID 862)`
`:IBM863` | `(:CODE-PAGE :ID 863)`
`:IBM864` | `(:CODE-PAGE :ID 864)`
`:IBM865` | `(:CODE-PAGE :ID 865)`
`:IBM866` | `(:CODE-PAGE :ID 866)`
`:IBM869` | `(:CODE-PAGE :ID 869)`
`:WINDOWS-1250` | `(:CODE-PAGE :ID 1250)`
`:WINDOWS-1251` | `(:CODE-PAGE :ID 1251)`
`:WINDOWS-1252` | `(:CODE-PAGE :ID 1252)`
`:WINDOWS-1253` | `(:CODE-PAGE :ID 1253)`
`:WINDOWS-1254` | `(:CODE-PAGE :ID 1254)`
`:WINDOWS-1255` | `(:CODE-PAGE :ID 1255)`
`:WINDOWS-1256` | `(:CODE-PAGE :ID 1256)`
`:WINDOWS-1257` | `(:CODE-PAGE :ID 1257)`
`:WINDOWS-1258` | `(:CODE-PAGE :ID 1258)`

Note that if you provide a shortcut, it will be converted to an `EXTERNAL-FORMAT` object first. So, if you're concerned about efficiency, create these objects once and re-use them.

[Function]
make-external-format name `&key` eol-style little-endian id => external-format

> Creates and returns an `EXTERNAL-FORMAT` object. `name` is a symbol, `eol-style` is one of the keywords `:CR`, `:LF`, or `:CRLF`, and `little-endian` is a generalized boolean. The default value for `eol-style` is the value of `*DEFAULT-EOL-STYLE*` except for Windows code pages where it is `:CRLF`. The default value for `little-endian` is the value of `*DEFAULT-LITTLE-ENDIAN*` - this value is ignored unless `name` denotes one of UTF-16 or UTF-32. `id` must be an integer denoting a Windows code page known by FLEXI-STREAMS if `name` is `:CODE-PAGE` or `WIN32:CODE-PAGE`, otherwise the value is ignored. See the section about external formats for more info.

> Examples (run on Windows):

>     CL-USER 1 > (make-external-format :latin-1)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :CRLF) 2067DA84>

>     CL-USER 2 > (make-external-format :latin-1 :eol-style :lf)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) 2068B4D4>

>     CL-USER 3 > (make-external-format :ibm437)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2069B33C>

>     CL-USER 4 > (make-external-format :ucs-2)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 206B4F4C>

>     CL-USER 5 > (make-external-format :ucs-2be)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CRLF :LITTLE-ENDIAN NIL) 2067DBE4>

>     CL-USER 6 > (make-external-format :ucs-2be :eol-style :cr)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :CR :LITTLE-ENDIAN NIL) 206B54AC>

[Readers]
external-format-name external-format => name
external-format-eol-style external-format => eol-style
external-format-little-endian external-format => little-endian
external-format-id external-format => id

> These methods can be used to query an `EXTERNAL-FORMAT` object for its attributes.

[Functions]
external-format-equal external-format-1 external-format-2 => generalized-boolean

> Checks whether the two external formats `external-format-1` and `external-format-2` are equivalent with respect to their effects on flexi streams.

> Examples (run on Windows):

>     CL-USER 1 > (make-external-format :ucs-4le)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-32 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 2067FB74>

>     CL-USER 2 > (external-format-equal * (make-external-format :utf32 :little-endian t))
>     T

>     CL-USER 3 > (make-external-format :code-page :id 437)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2069428C>

>     CL-USER 4 > (external-format-equal * (make-external-format :ibm437))
>     T

[Special variable]
*default-eol-style*

> The default value for the `eol-style` keyword argument of `MAKE-EXTERNAL-FORMAT`. Its initial value is `:CRLF` on Windows and `:LF` on other operating systems.

[Special variable]
*default-little-endian*

> The default value for the `little-endian` keyword argument of `MAKE-EXTERNAL-FORMAT`. Its initial value corresponds to the endianess of the platform FLEXI-STREAMS is used on as revealed by the `:LITTLE-ENDIAN` feature.

[Condition]
external-format-condition

> All conditions related to external formats are of this type. There's a slot for the external format which can be accessed with `EXTERNAL-FORMAT-CONDITION-EXTERNAL-FORMAT`.

[Reader]
external-format-condition-external-format condition => external-format

> If `condition` is of type `EXTERNAL-FORMAT-CONDITION`, this function will return the associated external format. Note that there are situation which happen during the creation of external formats where this method returns `NIL`.

[Condition]
external-format-error

> All errors related to external formats are of this type. This is a subtype of `EXTERNAL-FORMAT-CONDITION`.

[Condition]
external-format-encoding-error

> All errors related to encoding problems with external formats are of this type. (This includes situation where an end of file is encountered in the middle of a multi-octet character.) When this condition is signalled during reading, `USE-VALUE` restart is provided. See also `*SUBSTITUTION-CHAR*` and the example for it. `EXTERNAL-FORMAT-ENCODING-ERROR` is a subtype of `EXTERNAL-FORMAT-ERROR`.

[Special variable]
*substitution-char*

> If this value is not NIL, it should be a character which is used (as if by a `USE-VALUE` restart) whenever during reading an error of type `EXTERNAL-FORMAT-ENCODING-ERROR` would have been signalled otherwise.

>     CL-USER 1 > (defun foo ()
>                   ;; not a valid UTF-8 sequence
>                   (with-input-from-sequence (in '(#xe4 #xf6 #xfc))
>                     (setq in (make-flexi-stream in :external-format :utf8))
>                     (read-line in)))
>     FOO

>     CL-USER 2 > (foo)

>     Error: Unexpected value #xF6 in UTF-8 sequence.
>       1 (continue) Specify a character to be used instead.
>       2 (abort) Return to level 0.
>       3 Return to top loop level 0.

>     Type :b for backtrace, :c <option number> to proceed,  or :? for other options

>     CL-USER 3 : 1 > :c
>     Type a character: x

>     Error: End of file while in UTF-8 sequence.
>       1 (continue) Specify a character to be used instead.
>       2 (abort) Return to level 0.
>       3 Return to top loop level 0.

>     Type :b for backtrace, :c <option number> to proceed,  or :? for other options

>     CL-USER 4 : 1 > :c
>     Type a character: y
>     "xy"
>     T

>     CL-USER 5 > (handler-bind ((external-format-encoding-error (lambda (condition)
>                                                               (use-value #\-))))
>                   (foo))
>     "--"
>     T

>     CL-USER 6 > (let ((*substitution-char* #\?))
>                   (foo))
>     "??"
>     T

[Restart]
accept-overlong-sequence

> This is a restart which is established whenever a UTF-8 "overlong" sequence is encountered. If you invoke this restart, the corresponding code point will be accepted although it was encoded in an illegal way.

#### Flexi streams

Flexi streams are the core of the FLEXI-STREAMS library. You create them using the function `MAKE-FLEXI-STREAM` which takes an open binary stream (called the underlying stream) as its only required argument. A binary stream in this context means that if it's an input stream, you can read from it with `READ-BYTE` (or, as a workaround for LispWorks, you can at least apply `READ-SEQUENCE` to it where the sequence is an array of element type `OCTET`), and similarly for `WRITE-BYTE` (`WRITE-SEQUENCE` for LispWorks) and output streams. (Note that this specifically holds for bivalent streams like socket streams.)

A flexi stream behaves like an ordinary Lisp stream. It is an input stream if the underlying binary stream is an input stream, and it is an output stream when the underlying binary stream is an output stream. You can write characters as well as octets to an output flexi stream and similarly you can read characters and octets from an input flexi stream.

A flexi stream always has an external format associated with it which is deployed whenever you read characters from the stream or write characters to it. You can change the external format while you use the stream.

Once you're using a flexi stream you should not read from or write to the underlying stream directly anymore.

If you close a flexi stream, the underlying stream will also be closed. However, it also suffices to close the underlying stream directly should you not want to use the flexi stream anymore. So, the following usage (where `IN` is implicitly closed at the end) is OK:

    (with-open-file (in "/foo/bar/baz.txt")
      (let ((flexi (make-flexi-stream in :external-format :hebrew)))
        (read-line flexi)))

Output flexi streams will try to keep track of the column they're in but you can also set the column directly. This value will be incremented by one for each character written to the stream and it will be set to `0` if you send a `#\Newline` character. The column will be set to `NIL` if an `OCTET` is sent to the stream. Once the column is `NIL` it'll stay like that unless it is explicitly set to another value.

Input flexi streams keep track of their position within the stream. This value is incremented by one for each `OCTET` read from the stream, and it is incremented by the number of octets actually read for each character read from the stream. So, if the encoding is UTF-8, reading the character `#\ä` (a-umlaut) will advance the position by two. If the encoding is UTF-32 and the end-of-line style is `:CRLF`, reading a `#\Newline` will advance the position by eight.

You can also set the bound of an input flexi stream. Initially it is `NIL`, but when it's an integer and the stream's position has gone beyond this bound, the stream will behave as if no more input is available.

Caveat: You can only unread a character from a flexi stream if you haven't changed the external format after you read it.

Caveat: The underlying stream should either be a binary stream (i.e. have an element type that is a subtype of integer) or it should explicitly use an external format with `:LF` as its end-of-line style. Otherwise it might perform unwanted conversion of line endings on its own. (LispWorks does this even if you write binary data to the stream using `WRITE-SEQUENCE`.)

[Standard class]
flexi-stream

> Every flexi stream returned by `MAKE-FLEXI-STREAM` is of this type which is a subtype of `STREAM`.

[Standard class]
flexi-input-stream

> A flexi stream is of this type if its underlying stream is an input stream. This is a subtype of `FLEXI-STREAM`.

[Standard class]
flexi-output-stream

> A flexi stream is of this type if its underlying stream is an output stream. This is a subtype of `FLEXI-STREAM`.

[Standard class]
flexi-io-stream

> A flexi stream is of this type if it is both a `FLEXI-INPUT-STREAM` as well as a `FLEXI-OUTPUT-STREAM`.

[Function]
make-flexi-stream stream `&key` external-format element-type column position bound => flexi-stream

> Creates and returns a flexi stream, i.e. an object of type `FLEXI-STREAM`. `stream` is the underlying Lisp stream. `external-format` is the initial external format to be used by the stream, the default is the value of evaluating `(MAKE-EXTERNAL-FORMAT :LATIN1)`. `element-type` is the initial element type of the flexi stream the default of which is `LW:SIMPLE-CHAR` for LispWorks and `CHARACTER` otherwise. `column` is the initial column of the stream and should only be provided for output streams, the default is `0`. `position` is the initial octet position of the stream and must only be provided for input streams, the default is `0`. `bound` should be `NIL` (the default) or an integer and must only be provided for input streams. If the octet position of the stream has gone beyond this bound, the stream will behave as if no more input is available. See the section about flexi streams for more information.

[Accessors]
flexi-stream-external-format flexi-stream => external-format
`(setf (`flexi-stream-external-format flexi-stream) external-format`)`
flexi-stream-element-type flexi-stream => element-type
`(setf (`flexi-stream-element-type flexi-stream) element-type`)`
flexi-stream-column flexi-output-stream => column
`(setf (`flexi-stream-column flexi-output-stream) column`)`
flexi-stream-position flexi-input-stream => position
`(setf (`flexi-stream-position flexi-input-stream) position`)`
flexi-stream-bound flexi-input-stream => bound
`(setf (`flexi-stream-bound flexi-input-stream) bound`)`

> These methods can be used to get and set the corresponding attributes of a flexi stream.

> `(SETF FLEXI-STREAM-EXTERNAL-FORMAT)` accepts keyword symbols (names of external formats), lists (which should be valid lists of parameters to `MAKE-EXTERNAL-FORMAT`), or `EXTERNAL-FORMAT` objects:

>     CL-USER 1 > (setf (flexi-stream-external-format *my-stream*) :ucs-4le)
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-32 :EOL-STYLE :CRLF :LITTLE-ENDIAN T) 206920DC>

>     CL-USER 2 > (setf (flexi-stream-external-format *my-stream*) '(:ucs-2be :eol-style :br))
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:UTF-16 :EOL-STYLE :BR :LITTLE-ENDIAN NIL) 20696934>

>     CL-USER 3 > (setf (flexi-stream-external-format *my-stream*) (make-external-format :ibm437))
>     #<FLEXI-STREAMS::EXTERNAL-FORMAT (:CODE-PAGE :ID 437 :EOL-STYLE :CRLF) 2068716C>

[Reader]
flexi-stream-stream flexi-stream => stream

> This method returns the underlying stream of a flexi stream.

[Generic function]
unread-byte byte stream => nil

> Similar to `UNREAD-CHAR` in that it "unreads" the last octet from `stream` which must be a flexi stream. Note that you can only call `UNREAD-BYTE` after a corresponding `READ-BYTE`, not after `READ-CHAR`.

[Generic function]
peek-byte stream `&optional` peek-type eof-error-p eof-value => byte

> `PEEK-BYTE` is like `PEEK-CHAR`, i.e. it returns an octet from `stream` (which must be a flexi stream) without actually removing it. If `peek-type` is `NIL`, the next octet is returned, if `peek-type` is `T`, the next octet which is not `0` is returned, if `peek-type` is an octet, the next octet which equals `peek-type` is returned. `eof-error-p` and `eof-value` are interpreted as usual.

> Note that the parameters aren't in the same order as with `PEEK-CHAR` because it doesn't make much sense to make `stream` an optional argument.

[Type]
octet

> Just a shortcut for `(UNSIGNED-BYTE 8)`.

[Condition]
flexi-stream-error

> All errors related to flexi streams are of this type. This is a subtype of `STREAM-ERROR`.

[Condition]
flexi-stream-out-of-sync-error

> This can happen if you're trying to write to an IO stream which had prior to that "looked ahead" while reading and now can't "rewind" to the octet where you should be.

[Condition]
flexi-stream-element-type-error

> All errors related to problems with the element type of flexi streams are of this type. This is a subtype of `FLEXI-STREAM-ERROR` and has an additional slot for the element type which can be accessed with `FLEXI-STREAM-ELEMENT-TYPE-ERROR-ELEMENT-TYPE`.

[Reader]
flexi-stream-element-type-error-element-type condition => element-type

> If `condition` is of type `FLEXI-STREAM-ELEMENT-TYPE-ERROR`, this function will return the offending element type.

#### In-memory streams

The library also provides in-memory binary streams which are modeled after string streams and behave very similar only that they deal with octets instead of characters and the underlying data structure is not a string but either a list or a vector. These streams can obviously be used as the underlying streams for flexi streams.

[Standard class]
in-memory-stream

> Every in-memory stream returned by `MAKE-IN-MEMORY-INPUT-STREAM` or `MAKE-IN-MEMORY-OUTPUT-STREAM` is of this type which is a subtype of `STREAM`.

[Standard class]
in-memory-input-stream

> Every in-memory stream returned by `MAKE-IN-MEMORY-INPUT-STREAM` is of this type which is a subtype of `IN-MEMORY-STREAM`.

[Standard class]
in-memory-output-stream

> Every in-memory stream returned by `MAKE-IN-MEMORY-OUTPUT-STREAM` is of this type which is a subtype of `IN-MEMORY-STREAM`.

[Standard class]
list-stream

> Every in-memory input stream is of this type if it reads from a list.

[Standard class]
vector-stream

> Every in-memory stream is of this type if it reads from or writes to a vector.

[Generic function]
make-in-memory-input-stream sequence `&key` start end transformer => in-memory-input-stream

> Returns a binary input stream (of type `IN-MEMORY-INPUT-STREAM`) which will supply, in order, the octets in the subsequence of `sequence` bounded by `start` (the default is `0`) and `end` (the default is the length of `sequence`). `sequence` must either be a list or a vector of octets. Each octet returned will be transformed in turn by the optional `transformer` function.

[Function]
make-in-memory-output-stream `&key` element-type transformer => in-memory-output-stream

> Returns a binary output stream (of type `IN-MEMORY-OUTPUT-STREAM`) which accepts objects of type `element-type` (a subtype of `OCTET`) and makes available a sequence (see `GET-OUTPUT-STREAM-SEQUENCE`) that contains the octets that were actually output. The octets stored will each be transformed by the optional `transformer` function.

[Generic function]
get-output-stream-sequence stream `&key` as-list => sequence

> Returns a vector containing, in order, all the octets that have been output to the in-memory output stream `stream`. This operation clears any octets on `stream`, so the vector contains only those octets which have been output since the last call to `GET-OUTPUT-STREAM-SEQUENCE` or since the creation of the stream, whichever occurred most recently. If `as-list` is true the return value is coerced to a list.

[Generic function]
output-stream-sequence-length stream => length

> Returns the current length of the underlying vector of the in-memory output stream `stream`, i.e. this is the length of the sequence that `GET-OUTPUT-STREAM-SEQUENCE` would return if called at this very moment.

[Macro]
with-input-from-sequence (var sequence `&key` start end transformer) statement* => result*

> Creates an in-memory input stream from the sequence `sequence` using the parameters `start` and `end` (see `MAKE-IN-MEMORY-INPUT-STREAM`), binds `var` to this stream and then executes the `statement*` forms. A function `transformer` may optionally be specified to transform the returned octets. The stream is automatically closed on exit from `WITH-OUTPUT-TO-SEQUENCE`, no matter whether the exit is normal or abnormal. The return value of this macro is the return value of the last statement of `statement*`.

[Macro]
with-output-to-sequence (var `&key` as-list element-type transformer) statement* => sequence

> Creates an in-memory output stream, binds `var` to this stream and then executes the `statement*` forms. The stream stores data of type `element-type` (a subtype of `OCTET`) which is (optionally) transformed by the function `transformer` prior to storage. The stream is automatically closed on exit from `WITH-OUTPUT-TO-SEQUENCE`, no matter whether the exit is normal or abnormal. The return value of this macro is a vector (or a list if `as-list` is true) containing the octets that were sent to the stream within the body of the macro.

[Condition]
in-memory-stream-error

> All errors related to in-memory streams are of this type. This is a subtype of `STREAM-ERROR`.

[Condition]
in-memory-stream-closed-error

> An error of this type is signalled if one tries to read from or write to an in-memory stream which had already been closed. This is a subtype of `IN-MEMORY-STREAM-ERROR`.

[Condition]
in-memory-stream-position-spec-error

> Errors of this type are signalled if an erroneous position spec is used in conjunction with `FILE-POSITION`. This is a subtype of `IN-MEMORY-STREAM-ERROR` and has an additional slot for the position spec which can be accessed with `IN-MEMORY-STREAM-POSITION-SPEC-ERROR-POSITION-SPEC`.

[Reader]
in-memory-stream-position-spec-error-position-spec condition => position-spec

> If `condition` is of type `IN-MEMORY-STREAM-POSITION-SPEC-ERROR`, this function will return the offending position spec.

#### Strings

This section collects a few convenience functions for strings conversions.

[Function]
string-to-octets string `&key` external-format start end => vector

> Converts the Lisp string `string` from `start` to `end` to an array of octets corresponding to the external format designated by `external-format`. The defaults for `start` and `end` are `0` and the length of the string. The default for `external-format` is `:LATIN1`.

> In spite of the name, `string` can be any sequence of characters, but the function is optimized for strings.

[Function]
octets-to-string sequence `&key` external-format start end => string

> Converts the Lisp sequence `sequence` of octets from `start` to `end` to a string using the external format designated by `external-format`. The defaults for `start` and `end` are `0` and the length of the sequence. The default for `external-format` is `:LATIN1`.

> This function is optimized for the case of `sequence` being a vector. Don't use lists if you are in hurry.

[Function]
octet-length string `&key` external-format start end => length

> Returns the length of the subsequence of `string` from `start` to `end` in octets if encoded using the external format designated by `external-format`. The defaults for `start` and `end` are `0` and the length of `string`. The default for `external-format` is `:LATIN1`.

> In spite of the name, `string` can be any sequence of characters, but the function is optimized for strings.

[Function]
char-length sequence `&key` external-format start end => length

> Kind of the inverse of `OCTET-LENGTH`. Returns the length of the subsequence (of octets) of `sequence` from `start` to `end` in characters if decoded using the external format designated by `external-format`. The defaults for `start` and `end` are `0` and the length of the sequence. The default for `external-format` is `:LATIN1`. Note that this function doesn't check for the validity of the data in `sequence`.

> This function is optimized for the case of `sequence` being a vector. Don't use lists if you are in hurry.

### File positions

For flexi streams as well as for in-memory streams, FILE-POSITION will usually return `NIL` and do nothing when a second argument is supplied. This is correct w.r.t. the ANSI standard, but not very helpful. However, even with Gray streams there is no portable way to implement a better behaviour.

For LispWorks and CLISP, FILE-POSITION for flexi streams will work as if the function had been applied to the underlying stream, and for in-memory streams it will try to do something sensible if the underlying data structure is a vector (i.e. not a list). Patches for other Common Lisp implementations should be sent to the trivial-gray-streams maintainers.

### Acknowledgements

Thanks to David Lichteblau for numerous portability patches. Thanks to Igor Plekhov for the KOI8-R code. Thanks to Anton Vodonosov for numerous patches and additions. Thanks to Hans Hübner for his work on making FLEXI-STREAMS faster.