Next: , Previous: , Up: Miscellaneous additional functionality   [Contents][Index]


11.2 Controlling source file character encoding

Starting with ASDF 2.21, components accept a :encoding option so authors may specify which character encoding should be used to read and evaluate their source code. When left unspecified, the encoding is inherited from the parent module or system; if no encoding is specified at any point, or if nil is explicitly specified, an extensible protocol described below is followed, that ultimately defaults to :utf-8 since ASDF 3.

The protocol to determine the encoding is to call the function detect-encoding, which itself, if provided a valid file, calls the function specified by *encoding-detection-hook*, or else defaults to the *default-encoding*. The *encoding-detection-hook* is by default bound to function always-default-encoding, that always returns the contents of *default-encoding*. *default-encoding* is bound to :utf-8 by default (before ASDF 3, the default was :default).

Whichever encoding is returned must be a portable keyword, that will be translated to an implementation-specific external-format designator by function encoding-external-format, which itself simply calls the function specified *encoding-external-format-hook*; that function by default is default-encoding-external-format, that only recognizes :utf-8 and :default, and translates the former to the implementation-dependent *utf-8-external-format*, and the latter to itself (that itself is portable but has an implementation-dependent meaning).

In other words, there now are plenty of extension hooks, but by default ASDF enforces the previous de facto standard behavior of using :utf-8, independently from whatever configuration the user may be using. Thus, system authors can now rely on :utf-8 being used while compiling their files, even if the user is currently using :koi8-r or :euc-jp as their interactive encoding. (Before ASDF 3, there was no such guarantee, :default was used, and only plain ASCII was safe to include in source code.)

Some legacy implementations only support 8-bit characters, and some implementations provide 8-bit only variants. On these implementations, the *utf-8-external-format* gracefully falls back to :default, and Unicode characters will be read as multi-character mojibake. To detect such situations, UIOP will push the :asdf-unicode feature on implementations that support Unicode, and you can use reader-conditionalization to protect any :encoding encoding statement, as in #+asdf-unicode :encoding #+asdf-unicode :utf-8. We recommend that you avoid using unprotected :encoding specifications until after ASDF 2.21 or later becomes widespread (in April 2014, only LispWorks lags with ASDF 2.019, and is scheduled to be updated later this year).

While it offers plenty of hooks for extension, and one such extension is available (see asdf-encodings below), ASDF itself only recognizes one encoding beside :default, and that is :utf-8, which is the de facto standard, already used by the vast majority of libraries that use more than ASCII. On implementations that do not support unicode, the feature :asdf-unicode is absent, and the :default external-format is used to read even source files declared as :utf-8. On these implementations, non-ASCII characters intended to be read as one CL character may thus end up being read as multiple CL characters. In most cases, this shouldn’t affect the software’s semantics: comments will be skipped just the same, strings with be read and printed with slightly different lengths, symbol names will be accordingly longer, but none of it should matter. But a few systems that actually depend on unicode characters may fail to work properly, or may work in a subtly different way. See for instance lambda-reader.

We invite you to embrace UTF-8 as the encoding for non-ASCII characters starting today, even without any explicit specification in your .asd files. Indeed, on some implementations and configurations, UTF-8 is already the :default, and loading your code may cause errors if it is encoded in anything but UTF-8. Therefore, even with the legacy behavior, non-UTF-8 is guaranteed to break for some users, whereas UTF-8 is pretty much guaranteed not to break anywhere (provided you do not use a BOM), although it might be read incorrectly on some implementations. :utf-8 has been the default value of *default-encoding* since ASDF 3.

If you need non-standard character encodings for your source code, use the extension system asdf-encodings, by specifying :defsystem-depends-on (:asdf-encodings) in your defsystem. This extension system will register support for more encodings using the *encoding-external-format-hook* facility, so you can explicitly specify :encoding :latin1 in your .asd file. Using the *encoding-detection-hook* it will also eventually implement some autodetection of a file’s encoding from an emacs-style -*- mode: lisp ; coding: latin1 -*- declaration, or otherwise based on an analysis of octet patterns in the file. At this point, asdf-encoding only supports the encodings that are supported as part of your implementation. Since the list varies depending on implementations, we still recommend you use :utf-8 everywhere, which is the most portable (next to it is :latin1).

Recent versions of Quicklisp include asdf-encodings; if you’re not using it, you may get this extension using git: git clone git://common-lisp.net/projects/asdf/asdf-encodings.git or git clone ssh://common-lisp.net/project/asdf/git/asdf-encodings.git. You can also browse the repository on http://common-lisp.net/gitweb?p=projects/asdf/asdf-encodings.git.

When you use asdf-encodings, any .asd file loaded will use the autodetection algorithm to determine its encoding. If you depend on this detection happening, you should explicitly load asdf-encodings early in your build. Note that :defsystem-depends-on cannot be used here: by the time the :defsystem-depends-on is loaded, the enclosing defsystem form has already been read.

In practice, this means that the *default-encoding* is usually used for .asd files. Currently, this defaults to :utf-8, and you should be safe using Unicode characters in those files. This might matter, for instance, in meta-data about author’s names. Otherwise, the main data in these files is component (path)names, and we don’t recommend using non-ASCII characters for these, for the result probably isn’t very portable.

11.3 Miscellaneous Functions

These functions are exported by ASDF for your convenience.

Function: system-relative-pathname system name &key type

It’s often handy to locate a file relative to some system. The system-relative-pathname function meets this need.

It takes two mandatory arguments system and name and a keyword argument type: system is name of a system, whereas name and optionally type specify a relative pathname, interpreted like a component pathname specifier by coerce-pathname. See Pathname specifiers.

It returns a pathname built from the location of the system’s source directory and the relative pathname. For example:

> (asdf:system-relative-pathname 'cl-ppcre "regex.data")
#P"/repository/other/cl-ppcre/regex.data"
Function: system-source-directory system-designator

ASDF does not provide a turnkey solution for locating data (or other miscellaneous) files that are distributed together with the source code of a system. Programmers can use system-source-directory to find such files. Returns a pathname object. The system-designator may be a string, symbol, or ASDF system object.

Function: clear-system system-designator

It is sometimes useful to force recompilation of a previously loaded system. For these cases, (asdf:clear-system :foo) will remove the system from the table of currently loaded systems: the next time the system foo or one that depends on it is re-loaded, foo will be loaded again.14

Note that this does not and cannot undo the previous loading of the system. Common Lisp has no provision for such an operation, and its reliance on irreversible side-effects to global data structures makes such a thing impossible in the general case. If the software being re-loaded is not conceived with hot upgrade in mind, re-loading may cause many errors, warnings or subtle silent problems, as packages, generic function signatures, structures, types, macros, constants, etc. are being redefined incompatibly. It is up to the user to make sure that reloading is possible and has the desired effect. In some cases, extreme measures such as recursively deleting packages, unregistering symbols, defining methods on update-instance-for-redefined-class and much more are necessary for reloading to happen smoothly. ASDF itself goes to extensive effort to make a hot upgrade possible with respect to its own code. If you want, you can reuse some of its utilities such as uiop:define-package and uiop:with-upgradability, and get inspiration (or disinspiration) from what it does in header.lisp and upgrade.lisp.

Function: register-preloaded-system name &rest keys

A system with name name, created by make-instance with extra keys keys (e.g. :version), is registered as preloaded. That is, its code has already been loaded into the current image, and if at some point some other system :depends-on it yet no source code is found, it is considered as already provided, and ASDF will not raise a missing-component error.

This function is particularly useful if you distribute your code as fasls with either compile-bundle-op or monolithic-compile-bundle-op, and want to register systems so that dependencies will work uniformly whether you’re using your software from source or from fasl.

Function: run-shell-command control-string &rest args

This function is obsolete and present only for the sake of backwards-compatibility: “If it’s not backwards, it’s not compatible”. We strongly discourage its use. Its current behavior is only well-defined on Unix platforms (which include MacOS X and cygwin). On Windows, anything goes. The following documentation is only for the purpose of your migrating away from it in a way that preserves semantics.

Instead we recommend the use run-program, described in the next section, and available as part of ASDF since ASDF 3.

run-shell-command takes as arguments a format control-string and arguments to be passed to format after this control-string to produce a string. This string is a command that will be evaluated with a POSIX shell if possible; yet, on Windows, some implementations will use CMD.EXE, while others (like SBCL) will make an attempt at invoking a POSIX shell (and fail if it is not present).


Footnotes

(14)

Alternatively, you could touch foo.asd or remove the corresponding fasls from the output file cache.


Next: , Previous: , Up: Miscellaneous additional functionality   [Contents][Index]