Daniel Janus – Lisp

Making of “Clojure as a dependency”

2020-05-08T00:00:00Z

In my previous post, “Clojure as a dependency”, I’ve presented the results of some toy research on Clojure version numbers seen in the wild. I’m a big believer in reproducible research, so I’m making available a Git repo that contains code you can run yourself to reproduce these results. This post is an experience report from writing that code.

There are two main components to this project: acquisition and analysis of data (implemented in the namespaces versions.scrape and versions.analyze, respectively). Let’s look at each of these in turn.

Data acquisition

This step uses the GitHub API v3 to:

retrieve the 1000 most popular Clojure repositories (using the Search repositories endpoint and going through all pages of the paginated result);
for each of these repositories, look at its file list (in the master branch) and pick up any files named project.clj or deps.edn in the root directory, using the Contents endpoint);
parse each of these files and extract the list of dependencies.

As hinted by the namespace, I’ve opted to use Skyscraper to orchestrate the process. It would arguably have been simpler to use GitHub’s GraphQL v4 API, but I wanted to showcase Skyscraper’s custom parsing facilities.

There’s no actual HTML scraping going on (all processors use either JSON or Clojure parsers), but Skyscraper is still able to “restructure” the result – traverse the graph endpoint in a manner similar to that of GraphQL – with very little effort. It would have been possible with any other RESTful API. Plus, we get goodies like caching or tree pruning for free.

Most of the code is straightforward, but parsing of project.clj merits some explanation. Some of my initial assumptions proved incorrect, and it’s fun to see how. I initially tried to use clojure.edn, but Leiningen project definitions are not actually EDN – they are Clojure code, which is a superset of EDN. So I had to resort to read-string from core – with *read-eval* bound to nil (otherwise the code would have a Clojure injection vulnerability – think Bobby Tables). Needless to say, some project.cljs turned out to depend on read-eval.

Some projects (I’m looking at you, Closh, Babashka and sci) keep the version number outside of project.clj, in a text file (typically in resources/), and slurp it back into project.clj with a read-eval’d expression:

(defproject closh-sci
  #=(clojure.string/trim
     #=(slurp "resources/CLOSH_VERSION"))
  …)

A trick employed by one project, Metabase, is to dynamically generate JVM options containing a port number at parse time, so that test suites running at the same time don’t clash with each other:

#=(eval (format "-Dmb.jetty.port=%d" (+ 3001 (rand-int 500))))

Finally, it turned out that defproject is not always a first form in project.clj. Some projects, like bridge, only contain a placeholder project.clj with no forms; others, like aleph, first define some constants, and then refer to them in a defproject form. If those constants contain parts of the dependencies list, then those dependencies won’t be processed correctly. Fortunately, not a lot of projects do this, so it doesn’t skew the results much.

Anyway, the end result of the acquisition phase is a sequence of maps describing project definitions. They look like this:

{:name "clojure-koans",
 :full-name "functional-koans/clojure-koans",
 :deps-type :leiningen,
 :page 1,
 :deps {org.clojure/clojure #:mvn{:version "1.10.0"},
        koan-engine #:mvn{:version "0.2.5"}}},
 :profile-deps {:dev {lein-koan #:mvn{:version "0.1.5"}}}

Homogeneity is important: every dependency description has been converted to the cli-tools format, even if it comes from a project.clj.

Data analysis

I’ve long been searching for a way to do exploratory programming in Clojure without turning the code to a tangled mess, portable only along with my computer.

Exploratory (or research) programming is very different from “normal” programming. In the latter, most of the time you typically focus on a coherent project – a program or a library. In contrast, in the former, you spend a lot of time in the REPL, trying all sorts of different things and defing new values derived from already computed ones.

This is very convenient, but it’s extremely easy to get carried away in the REPL and get lost in a sea of defs. If you want to redo your computations from scratch, just about your only option is to take your REPL transcript and re-evaluate the expressions one by one, in the correct order. Cleaning up the code (e.g. deglobalizing) as you go is very difficult.

I’ve found an answer: Plumatic Graph, part of the plumbing library. There are a plethora of uses for it: for example, at Fy, my current workplace, we’re using it to define our test fixtures. But as it turns out, it makes exploratory programming enjoyable.

The bulk of code in versions.analyze consists of a big definition of a graph, with nodes representing computations – things that I’d normally have def’d in a REPL. Consequently, most of these definitions are short and to the point. I also gave the nodes verbose, descriptive, explicit names. Name and conquer. raw-repos is the output from data acquisition, repos is an all-important node containing those raw-repos that were successfully parsed, and most other things depend on it.

It also doesn’t obstruct much the normal REPL research flow. My normal workflow with REPL and Graph is something along the lines of:

(def result (main))
evaluate something using inputs from result
nah, it leads nowhere
evaluate something else
hey, that’s interesting!
add a new node to the graph definition
GOTO 1

Thanks to Graph’s lazy compiler, I can re-evaluate anything at need and have it evaluate only the things needed, and nothing else. Also, because the graph is explicit, it’s fairly easy to visualize it. (Click the image to open it in full-size in another tab.)

Because it’s lazy, it doesn’t hurt to put extra things in there just in case, even when you’re not going to report them. For example, I was curious what things besides a version number people put in dependencies. :exclusions, for sure, but what else? This is the :what-other-things-besides-versions node.

Imagine my surprise when I found :exlusions (sic) in there, which turned out to be a typo in shadow-cljs’ project.clj! I submitted a PR, and Thomas Heller merged it a few days after.

My only gripe with Graph is that it runs somewhat contrary to the current trends in the Clojure community: for example, it doesn’t support namespaced keywords (although there’s an open ticket for that). But on the whole, I’m sold. I’ll definitely be using it in the next piece of research in Clojure, and I’m on a lookout for something similar in pure R. If you know something, do tell me!

Some words on plotting

The plot from previous post has been generated in pure R, using ggplot2 (an extremely versatile API). Clojure generates a CSV with munged data, and then R reads that CSV as a data frame and generates the plot in a few lines.

I’ve briefly played around with clojisr, a bridge between Clojure and R. It was an enlightening experiment, and it would let me avoid the intermediate CSV, but I decided to ditch it for a few reasons:

It pulls in quite a few dependencies (I wanted to keep them down to a minimum), and requires some previous setup on the R side.
I’d much rather write my R as R, since I’m comfortable with it, rather than spend time wondering how it maps to Clojure. This is similar to the SQL story: these days I prefer HugSQL over Korma, unless I have good reasons to choose otherwise.
clojisr opens up a child R process just by requireing a namespace. I’m not a fan of that.

But it’s definitely very promising! I applaud the effort and I’ll keep a close eye on it.

Key takeaways

Skyscraper makes data acquisition bearable, if not fun.
Plumatic Graph makes writing research code in Clojure fun.
ggplot makes plotting data fun.
Clojure makes programming fun. (But you knew that already.)

Clojure as a dependency

2020-05-02T00:00:00Z

I have a shameful confession to make: I have long neglected an open-source library that I maintain, clj-tagsoup.

This would have been less of an issue, but this is my second-most-starred project on GitHub. Granted, I don’t feel a need for it anymore, but apparently people do. I wish I had spent some time reviewing and merging the incoming PRs.

Anyway, I’ve recently been prompted to revive it, and I’m preparing a new release. While on it, I’ve been updating dependencies to their latest versions, and upon seeing a dependency on [org.clojure/clojure "1.2.0"] in project.clj (yes, it’s been neglected for that long), I started wondering: which Clojure to depend on? Actually, should Clojure itself be a dependency at all?

I’ve googled around for best practices, but with no conclusive answer. So I set out to do some research.

TLDR: with Leiningen, add it with :scope "provided"; with cli-tools, you don’t have to, unless you want to be explicit.

Is it possible for a Clojure project to declare no dependency on Clojure at all?

Quite possible, as it turns out. But the details depend on the build tool.

Obviously, this only makes sense for libraries. Or, more broadly, for projects that are not meant to be used standalone, but rather included in other projects (which will have a Clojure dependency of their own).

Leiningen

If you try to create a Leiningen project that has no dependencies:

(defproject foo "0.1.0"
  :dependencies [])

then Leiningen (as of version 2.9.3, but I’d guess older versions behave similarly) won’t allow you to launch a REPL:

$ lein repl
Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main
Subprocess failed (exit code: 1)

But all is not lost: lein jar works just fine (as long as you don’t AOT-compile any namespaces), as does lein install. The resulting library will happily function as a dependency of other projects.

The upside of depending on no particular Clojure version is that you don’t impose it on your consumers. If a library depends on Clojure 1.9.0, but a project that uses it depends on Clojure 1.10.1, then Leiningen will fetch 1.9.0’s pom.xml (it’s smart enough to figure out that the jar itself won’t be needed, as the conflict will always be resolved in favour of the direct dependency), and lein deps :tree will report “possibly confusing dependencies”.

It’s not very useful to have a library that you can’t launch a REPL against, though. So what some people do is declare a dependency on Clojure not in the main :dependencies, but in a profile.

(defproject foo "0.1.0"
  :dependencies []
  :profiles {:dev {:dependencies [[org.clojure/clojure "1.10.1"]]}})

This avoids conflicts and brings back the possibility to launch a REPL. Sometimes, people create multiple profiles for different Clojure versions; Leiningen’s documentation mentions this possibility.

Unfortunately, with this approach it’s still not possible to AOT-compile things or create uberjars with Leiningen. (Putting Clojure in the :provided profile causes building the uberjar to succeed, but the resulting -standalone jar doesn’t actually contain Clojure).

Another option is to add Clojure to the main :dependencies, but with :scope "provided". Per the Maven documentation, this means:

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

The key are the last words: “not transitive.” If project A depends on a library B that declares a “provided” dependency C, then C won’t be automatically put in A’s dependencies, and A is expected to explicitly declare its own C.

This means that it’s adequate for both libraries and standalone projects when it comes to declaring a Clojure dependency. It doesn’t break anything, doesn’t cause any ephemeral conflicts, and can be combined with the profiles approach when multiple configurations are called for.

cli-tools

cli-tools will accept a deps.edn as simple as {}. Even passing -Srepro to clojure or clj (which excludes the Clojure dependency that you probably have in your ~/.clojure/deps.edn) doesn’t break anything: cli-tools will just use 1.10.1 (at least as of version 1.10.1.536).

With cli-tools, as a library author you probably don’t have to declare a Clojure dependency at all. But things are less uniform in this land than they are in Leiningen (for example, there are quite a few uberjarrers to choose from), so it’s reasonable to check with your tooling first.

Boot

I’m no longer a Boot user, so I can’t tell. But from what I know, it uses Aether just like Leiningen and Maven do, so I’d wager a guess the same caveats apply as for Leiningen. Haven’t checked, though.

So what do the existing projects do?

I figured it would be a fun piece of research to examine how the popular projects depend (or don’t depend) on Clojure. I queried GitHub’s API for the 1000 most starred Clojure projects, fetched and parsed their project.cljs and/or deps.edns, and tallied things up.

I’ll write a separate “making of” post, because it turned out to be an even more fun weekend project than I had anticipated. But for now, let me share the conclusions.

I ended up with 968 project definition files that I was able to successfully parse: 140 deps.edns and 828 project.cljs. Here’s a breakdown of Clojure version declared as a “main” dependency (i.e., not in a profile or alias):

N/A means that there’s no dependency on Clojure declared, and “other” is an umbrella for the zoo of alphas, betas and snapshots.

As expected, not depending on Clojure is comparatively more popular in the cli-tools land: almost half (48.6%) of cli-tools projects don’t declare a Clojure dependency, versus 21.5% (174 projects) for Leiningen.

That Leiningen number still seemed quite high to me, so I dug a little deeper. Out of those 174 projects, 100 have Clojure somewhere in their :profiles. The remaining 74 are somewhat of outliers:

some, like Ring or Pedestal, are umbrella projects composed of sub-projects (with the lein-sub plugin) that have actual dependencies themselves;
some, like Klipse or Reagent, are essentially ClojureScript-only;
some, like Overtone, use the lein-tools-deps plugin to store their dependencies in deps.edn while using Leiningen for other tasks.

Finally, the popularity of :scope "provided" is much lower. Only 68 Leiningen projects specify it (8.9% of those that declare any dependencies), and only two deps.edn files do so (re-frame and fulcro – note that re-frame actually has both a project.clj and a deps.edn).

Indenting cond forms

2020-02-10T00:00:00Z

Indentation matters when reading Clojure code. It is the primary visual cue that helps the reader discern the code structure. Most Clojure code seen in the wild conforms to either the community style guide or the proposed simplified rules; the existing editors make it easy to reformat code to match them.

I find both these rulesets to be helpful when reading code. But there’s one corner-case that’s been irking me: cond forms.

cond takes an even number of arguments: alternating test-expression pairs. They are commonly put next to each other, two forms per line.

(cond
  test expr-1
  another-test expr-2
  :else expr-3)

Sometimes, people align the expressions under one another, in a tabular fashion:

(cond
  test         expr-1
  another-test expr-2
  :else        expr-3)

But things get out of hand when either tests or exprs get longer and call for multiple lines themselves. There are several options here, all of them less than ideal.

Tests and expressions next to each other

In other words, keep the above rule. Because we’ll have multiple lines in a form, this tends to make the resulting code axe-shaped:

(cond
  (= (some-function something) expected-value) (do
                                                 (do-this)
                                                 (and-also-do-that))
  (another-predicate something-else) (try
                                       (do-another-thing)
                                       (catch Exception _
                                         (println "Whoops!"))))

This yields code that is indented abnormally far to the right, forcing the reader’s eyeballs to move in two dimensions – even more so if the tabular feel is desired. If both the test and the expression is multi-lined, it just looks plain weird.

Stack all forms vertically, no extra spacing

(cond
  (= (some-function something) expected-value)
  (do
    (do-this)
    (and-also-do-that))
  (another-predicate something-else)
  (try
    (do-another-thing)
    (catch Exception _
      (println "Whoops!"))))

This gets rid of the long lines, but introduces another problem: it’s hard to tell at a glance

where a given test or expression starts or ends;
which tests are paired with which expression;
whether a given line corresponds to a test or an expression, and which one.

Stack all forms vertically, blank lines between test/expr pairs

(cond
  (= (some-function something) expected-value)
  (do
    (do-this)
    (and-also-do-that))

  (another-predicate something-else)
  (try
    (do-another-thing)
    (catch Exception _
      (println "Whoops!"))))

The Style Guide says that this is an “ok-ish” thing to do.

But with the added blank lines, logical structure of the code is much more apparent. However, it breaks another assumption that I make when reading the code: functions contain no blank lines. The Style Guide even mentions it, saying that cond forms are an acceptable exception.

It is now harder to tell at a glance where the enclosing function starts or ends. And once this assumption is broken once, the brain expects it to be broken again, causing reading disruption across the entire file.

Forms one under another, extra indentation for expressions only

(cond
  (= (some-function something) expected-value)
    (do
      (do-this)
      (and-also-do-that))
  (another-predicate something-else)
    (try
      (do-another-thing)
      (catch Exception _
        (println "Whoops!"))))

I resorted to this several times. The lines are not too long; the visual cues are there; it’s obvious what is the condition, what is the test, and what goes with what.

Except… it’s against the rules. List items stacked vertically should be aligned one under the other. I have to actively fight my Emacs to enforce this formatting, and it will be lost next time I press C-M-q on this form. No good.

Forms one under another, expressions prefixed by `#_=>`

(cond
  (= (some-function something) expected-value)
  #_=> (do
         (do-this)
         (and-also-do-that))
  (another-predicate something-else)
  #_=> (try
         (do-another-thing)
         (catch Exception _
           (println "Whoops!"))))

This one is my own invention: I haven’t seen it anywhere else. But I think it manages to avoid most problems.

#_ is a reader macro that causes the next form to be elided and not seen by the compiler. => is a valid form. Thus, #_=> is effectively whitespace as far as the compiler is concerned, and the indentation rules treat it as yet another symbol (although it technically isn’t one). No tooling is broken, no assumptions are broken, and the #_=> tends to be syntax-highlighted unintrusively so it doesn’t stand out. I tend to read it aloud as “then.”

Meanwhile, in another galaxy

Other Lisps (Scheme and CL) wrap each test/expression pair in an extra pair of parens, thereby avoiding the blending of conditions and expressions when indented one under the other. But I’m still happy Clojure went with fewer parens. As I say, this is a corner case where additional pair of parens would somewhat help, but most of the time I find them less aesthetic and a visual clutter.

You already use Lisp syntax

2014-05-20T00:00:00Z

Unix Developer: I’m not going to touch Lisp. It’s horrible!

Me: Why so?

UD: The syntax! This illegible prefix-RPN syntax that nobody else uses. And just look at all these parens!

Me: Well, many people find it perfectly legible, although most agree that it takes some time to get accustomed to. But I think you’re mistaken. Lots of people are using Lisp syntax on a daily basis…

UD: I happen to know no one doing this.

Me: …without actually realizing this. In fact, I think you yourself are using it.

UD: Wait, what?!

Me: And the particular variant of Lisp syntax you’re using is called Bourne shell.

UD: Now I don’t understand. What on earth does the shell have to do with Lisp?

Me: Just look: in the shell, you put the name of the program first, followed by the arguments, separated by spaces. In Lisp it’s exactly the same, except that you put an opening paren at the beginning and a closing paren at the end.

Shell: run-something arg1 arg2 arg3

Lisp: (run-something arg1 arg2 arg3)

UD: I still don’t get the analogy.

Me: Then you need a mechanism for expression composition — putting the output of one expression as an input to another. In Lisp, you just nest the lists. And in the shell?

UD: Backticks.

Me: That’s right. Or $(), which has the advantage of being more easily nestable. Let’s try arithmetic. How do you do arithmetic in the shell?

UD: expr. Or the Bash builtin let. For example,

$ let x='2*((10+4)/7)'; echo $x
4

Me: Now wouldn’t it be in line with the spirit of Unix — to have programs do just one thing — if we had one program to do addition, and another to do subtraction, and yet another to do multiplication and division?

It’s trivial to write it in C:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, char **argv) {
  int mode = -1, cnt = argc - 1, val, i;
  char **args = argv + 1;
  switch (argv[0][strlen(argv[0]) - 1]) {
    case '+': mode = 0; break;
    case '-': mode = 1; break;
    case 'x': mode = 2; break;
    case 'd': mode = 3; break;
  }
  if (mode == -1) {
    fprintf(stderr, "invalid math operation\n");
    return 1;
  }
  if ((mode == 1 || mode == 3) && !cnt) {
    fprintf(stderr, "%s requires at least one arg\n", argv[0]);
    return 1;
  }
  switch (mode) {
    case 0: val = 0; break;
    case 2: val = 1; break;
    default: val = atoi(*args++); cnt--; break;
  }
  while (cnt--) {
    switch (mode) {
      case 0: val += atoi(*args++); break;
      case 1: val -= atoi(*args++); break;
      case 2: val *= atoi(*args++); break;
      case 3: val /= atoi(*args++); break;
    }
  }
  printf("%d\n", val);
  return 0;
}

This dispatches on the last character of its name, so it can be symlinked to +, -, x and d (I picked unusual names for multiplication and division to make them legal and avoid escaping).

Now behold:

$ x 2 $(d $(+ 10 4) 7)
4

UD: Wow, this sure looks a lot like Lisp!

Me: And yet it’s the shell. Our two basic rules — program-name-first and $()-for-composition — allowed us to explicitly specify the order of evaluation, so there was no need to do any fancy parsing beyond what the shell already provides.

UD: So is the shell a Lisp?

Me: Not really. The shell is stringly typed: a program takes textual parameters and produces textual output. To qualify as a Lisp, it would have to have a composite type: a list or a cons cell to build lists on top of. Then, you’d be able to represent code as this data structure, and write programs to transform code to other code.

But the Tao of Lisp lingers in the shell syntax.

I know I’ve glossed over many details here, like the shell syntax for redirection, globbing, subprocesses, the fact that programs have standard input in addition to command-line arguments, pipes, etc. — all these make the analogy rather weak. But I think it’s an interesting way to teach Lisp syntax to people.

Combining virtual sequences
or, Sequential Fun with Macros
or, How to Implement Clojure-Like Pseudo-Sequences with Poor Man’s Laziness in a Predominantly Imperative Language

2011-12-09T00:00:00Z

Sequences and iteration

There are a number of motivations for this post. One stems from my extensive exposure to Clojure over the past few years: this was, and still is, my primary programming language for everyday work. Soon, I realized that much of the power of Clojure comes from a sequence abstraction being one of its central concepts, and a standard library that contains many sequence-manipulating functions. It turns out that by combining them it is possible to solve a wide range of problems in a concise, high-level way. In contrast, it pays to think in terms of whole sequences, rather than individual elements.

Another motivation comes from a classical piece of functional programming humour, [The Evolution of a Haskell Programmer][1]. If you don’t know it, go check it out: it consists of several Haskell implementations of factorial, starting out from a straightforward recursive definition, passing through absolutely hilarious versions involving category-theoretical concepts, and finally arriving at this simple version that is considered most idiomatic:

fac n = product [1..n]

This is very Clojure-like in that it involves a sequence (a list comprehension). In Clojure, this could be implemented as

(defn fac [n]
  (reduce * 1 (range 1 (inc n)))

Now, I thought to myself, how would I write factorial in an imperative language? Say, Pascal?

function fac(n : integer) : integer;
var
  i, res : integer;
begin
  res := 1;
  for i := 1 to n do
    res := res * i;
  fac := res;
end;

This is very different from the functional version that works with sequences. It is much more elaborate, introducing an explicit loop. On the other hand, it’s memory efficient: it’s clear that its memory requirements are O(1), whereas a naïve implementation of a sequence would need O(n) to construct it all in memory and then reduce it down to a single value.

Or is it really that different? Think of the changing values of i in that loop. On first iteration it is 1, on second iteration it’s 2, and so on up to n. Therefore, one can really think of a for loop as a sequence! I call it a “virtual” sequence, since it is not an actual data structure; it’s just a snippet of code.

To rephrase it as a definition: a virtual sequence is a snippet of code that (presumably repeatedly) yields the member values.

Let’s write some code!

To illustrate it, throughout the remainder of this article I will be using Common Lisp, for the following reasons:

It allows for imperative style, including GOTO-like statements. This will enable us to generate very low-level code.
Thanks to macros, we will be able to obtain interesting transformations.

Okay, so let’s have a look at how to generate a one-element sequence. Simple enough:

(defmacro vsingle (x)
 `(yield ,x))

The name VSINGLE stands for “Virtual sequence that just yields a SINGLE element”. (In general, I will try to define virtual sequences named and performing similarly to their Clojure counterparts here; whenever there is a name clash with an already existing CL function, the name will be prefixed with V.) We will not concern ourselves with the actual definition of YIELD at the moment; for debugging, we can define it just as printing the value to the standard output.

(defun yield (x)
  (format t "~A~%" x))

We can also convert a Lisp list to a virtual sequence which just yields each element of the list in turn:

(defmacro vseq (list)
  `(loop for x in ,list do (yield x)))

(defmacro vlist (&rest elems)
  `(vseq (list ,@elems)))

Now let’s try to define RANGE. We could use loop, but for the sake of example, let’s pretend that it doesn’t exist and write a macro that expands to low-level GOTO-ridden code. For those of you who are not familiar with Common Lisp, GO is like GOTO, except it takes a label that should be established within a TAGBODY container.

(defmacro range (start &optional end (step 1))
  (unless end
    (setf end start start 0))
  (let ((fv (gensym)))
    `(let ((,fv ,start))
       (tagbody
        loop
          (when (>= ,fv ,end)
            (go out))
          (yield ,fv)
          (incf ,fv ,step)
          (go loop)
       out))))

Infinite virtual sequences are also possible. After all, there’s nothing preventing us from considering a snippet of code that loops infinitely, executing YIELD, as a virtual sequence! We will define the equivalent of Clojure’s iterate: given a function fun and initial value val, it will repeatedly generate val, (fun val), (fun (fun val)), etc.

(defmacro iterate (fun val)
  (let ((fv (gensym)))
    `(let ((,fv ,val))
       (tagbody loop
          (yield ,fv)
          (setf ,fv (funcall ,fun ,fv))
          (go loop)))))

So far, we have defined a number of ways to create virtual sequences. Now let’s ask ourselves: is there a way, given code for a virtual sequence, to yield only the elements from the original that satisfy a certain predicate? In other words, can we define a filter for virtual sequences? Sure enough. Just replace every occurrence of yield with code that checks whether the yielded value satisfies the predicate, and only if it does invokes yield.

First we write a simple code walker that applies some transformation to every yield occurrence in a given snippet:

(defun replace-yield (tree replace)
  (if (consp tree)
      (if (eql (car tree) 'yield)
          (funcall replace (cadr tree))
          (loop for x in tree collect (replace-yield x replace)))
      tree))

We can now write filter like this:

(defmacro filter (pred vseq &environment env)
  (replace-yield (macroexpand vseq env)
                 (lambda (x) `(when (funcall ,pred ,x) (yield ,x)))))

It is important to point out that since filter is a macro, the arguments are passed to it unevaluated, so if vseq is a virtual sequence definition like (range 10), we need to macroexpand it before replacing yield.

We can now verify that (filter #'evenp (range 10)) works. It macroexpands to something similar to

(LET ((#:G70192 0))
  (TAGBODY
    LOOP (IF (>= #:G70192 10)
           (PROGN (GO OUT)))
         (IF (FUNCALL #'EVENP #:G70192)
           (PROGN (YIELD #:G70192)))
         (SETQ #:G70192 (+ #:G70192 1))
         (GO LOOP)
    OUT))

concat is extremely simple. To produce all elements of vseq1 followed by all elements of vseq2, just execute code corresponding to vseq1 and then code corresponding to vseq2. Or, for multiple sequences:

(defmacro concat (&rest vseqs)
  `(progn ,@vseqs))

To define take, we’ll need to wrap the original code in a block that can be escaped from by means of return-from (which is just another form of goto). We’ll add a counter that will start from n and keep decreasing on each yield; once it reaches zero, we escape the block:

(defmacro take (n vseq &environment env)
  (let ((x (gensym))
        (b (gensym)))
    `(let ((,x ,n))
       (block ,b
         ,(replace-yield (macroexpand vseq env)
                         (lambda (y) `(progn (yield ,y)
                                             (decf ,x)
                                             (when (zerop ,x)
                                               (return-from ,b)))))))))

rest (or, rather, vrest, as that name is taken) can be defined similarly:

(defmacro vrest (vseq &environment env)
  (let ((skipped (gensym)))
    (replace-yield
     `(let ((,skipped nil)) ,(macroexpand vseq env))
     (lambda (x) `(if ,skipped (yield ,x) (setf ,skipped t))))))

vfirst is another matter. It should return a value instead of producing a virtual sequence, so we need to actually execute the code — but with yield bound to something else. We want to establish a block as with take, but our yield will immediately return from the block once the first value is yielded:

(defmacro vfirst (vseq)
  (let ((block-name (gensym)))
   `(block ,block-name
      (flet ((yield (x) (return-from ,block-name x)))
        ,vseq))))

Note that so far we’ve seen three classes of macros:

macros that create virtual sequences;
macros that transform virtual sequences to another virtual sequences;
and finally, vfirst is our first example of a macro that produces a result out of a virtual sequence.

Our next logical step is vreduce. Again, we’ll produce code that rebinds yield: this time to a function that replaces the value of a variable (the accumulator) by result of calling a function on the accumulator’s old value and the value being yielded.

(defmacro vreduce (f val vseq)
  `(let ((accu ,val))
     (flet ((yield (x) (setf accu (funcall ,f accu x))))
       ,vseq
       accu)))

We can now build a constructs that executes a virtual sequence and wraps the results up as a Lisp list, in terms of vreduce.

(defun conj (x y)
  (cons y x))

(defmacro realize (vseq)
 `(nreverse (vreduce #'conj nil ,vseq)))

Let’s verify that it works:

CL-USER> (realize (range 10))
(0 1 2 3 4 5 6 7 8 9)

CL-USER> (realize (take 5 (filter #'oddp (iterate #'1+ 0))))
(1 3 5 7 9)

Hey! Did we just manipulate an infinite sequence and got the result in a finite amount of time? And that without explicit support for laziness in our language? How cool is that?!

Anyway, let’s finally define our factorial:

(defun fac (n)
  (vreduce #'* 1 (range 1 (1+ n))))

Benchmarking

Factorials grow too fast, so for the purpose of benchmarking let’s write a function that adds numbers from 0 below n, in sequence-y style. First using Common Lisp builtins:

(defun sum-below (n)
  (reduce #'+ (loop for i from 0 below n collect i) :initial-value 0))

And now with our virtual sequences:

(defun sum-below-2 (n)
  (vreduce #'+ 0 (range n)))

Let’s try to time the two versions. On my Mac running Clozure CL 1.7, this gives:

CL-USER> (time (sum-below 10000000))
(SUM-BELOW 10000000) took 8,545,512 microseconds (8.545512 seconds) to run
                    with 2 available CPU cores.
During that period, 2,367,207 microseconds (2.367207 seconds) were spent in user mode
                    270,481 microseconds (0.270481 seconds) were spent in system mode
5,906,274 microseconds (5.906274 seconds) was spent in GC.
 160,000,016 bytes of memory allocated.
 39,479 minor page faults, 1,359 major page faults, 0 swaps.
49999995000000

CL-USER> (time (sum-below-2 10000000))
(SUM-BELOW-2 10000000) took 123,081 microseconds (0.123081 seconds) to run
                    with 2 available CPU cores.
During that period, 127,632 microseconds (0.127632 seconds) were spent in user mode
                    666 microseconds (0.000666 seconds) were spent in system mode
 4 minor page faults, 0 major page faults, 0 swaps.
49999995000000

As expected, SUM-BELOW-2 is much faster, causes less page faults and presumably conses less. (Critics will be quick to point out that we could idiomatically write it using LOOP’s SUM/SUMMING clause, which would probably be yet faster, and I agree; yet if we were reducing by something other than + — something that LOOP has not built in as a clause — this would not be an option.)

Conclusion

We have seen how snippets of code can be viewed as sequences and how to combine them to produce other virtual sequences. As we are nearing the end of this article, it is perhaps fitting to ask: what are the limitations and drawbacks of this approach?

Clearly, this kind of sequences is less powerful than “ordinary” sequences such as Clojure’s. The fact that we’ve built them on macros means that once we escape the world of code transformation by invoking some macro of the third class, we can’t manipulate them anymore. In Clojure world, first and rest are very similar; in virtual sequences, they are altogether different: they belong to different worlds. The same goes for map (had we defined one) and reduce.

But imagine that instead of having just one programming language, we have a high-level language A in which we are writing macros that expand to code in a low-level language B. It is important to point out that the generated code is very low-level. It could almost be assembly: in fact, most of the macros we’ve written don’t even require language B to have composite data-types beyond the type of elements of collections (which could be simple integers)!

Is there a practical side to this? I don’t know: to me it just seems to be something with hack value. Time will tell if I can put it to good use.

Keyword arguments

2010-05-04T00:00:00Z

There’s been an ongoing debate about how to pass optional named arguments to Clojure functions. One way to do this is the defnk macro from clojure.contrib.def; I hesitate to call it canonical, since apparently not everyone uses it, but I’ve found it useful a number of times. Here’s a sample:

user> (use 'clojure.contrib.def)
nil
user> (defnk f [:b 43] (inc b))
#'user/f
user> (f)
44
user> (f :b 100)
101

This is an example of keyword arguments in action. Keyword arguments are a core feature of some languages, notably Common Lisp and Objective Caml. Clojure doesn’t have them, but it’s pretty easy to emulate their basic usage with macros, as defnk does.

But there’s more to Common Lisp’s keyword arguments than defnk provides. In CL, the default value of a keyword argument can be an expression referring to other arguments of the same function. For example:

CL-USER> (defun f (&key (a 1) (b a))
           (+ a b))
F
CL-USER> (f)
2
CL-USER> (f :a 45)
90
CL-USER> (f :b 101)
102

I wish defnk had this feature. Or is there some better way that I don’t know of?

Who said Common Lisp programs cannot be small?

2008-08-09T00:00:00Z

So, how much disk space does your average CL image eat up? A hundred megs? Fifty? Twenty? Five, perhaps, if you’re using LispWorks with a tree-shaker? Well then, how about this?

[nathell@chamsin salza2-2.0.4]$ ./cl-gzip closures.lisp test.gz
[nathell@chamsin salza2-2.0.4]$ gunzip test
[nathell@chamsin salza2-2.0.4]$ diff closures.lisp test
[nathell@chamsin salza2-2.0.4]$ ls -l cl-gzip
-rwxr-xr-x 1 nathell nathell 386356 2008-08-09 11:08 cl-gzip

That’s right. A standalone executable of a mini-gzip, written in Common Lisp, taking up under 400K! And it only depends on glibc and GMP, which are available by default on pretty much every Linux installation. (This is on a 32-bit x86 machine, by the way).

I used the most recent version of ECL for compiling this tiny example. The key to the size was configuring ECL with --disable-shared --enable-static CFLAGS="-Os -ffunction-sections -fdata-sections" LDFLAGS="-Wl,-gc-sections". This essentially gives you a poor man’s tree shaker for free at a linker level. And ECL in itself produces comparatively tiny code.

I built this example from Salza2’s source by loading the following code snippet:

(defvar salza
  '("package" "reset" "specials"
    "types" "checksum" "adler32" "crc32" "chains"
    "bitstream" "matches" "compress" "huffman"
    "closures" "compressor" "utilities" "zlib"
    "gzip" "user"))

(defvar salza2
  (mapcar (lambda (x) (format nil "~A.lisp" x))
          salza))

(defvar salza3
  (mapcar (lambda (x) (format nil "~A.o" x))
          salza))

(defun build-cl-gzip ()
  (dolist (x salza2)
          (load x)
          (compile-file x :system-p t))
  (c:build-program
   "cl-gzip"
   :lisp-files salza3
   :epilogue-code
     '(progn
       (in-package :salza2)
       (gzip-file (second (si::command-args))
                  (third (si::command-args))))))

(build-cl-gzip)

(Sadly enough, there’s no ASDF in here. I have yet to figure out how to leverage ASDF to build small binaries in this constrained environment.)

This gave me a standalone executable 1.2 meg in size. I then proceeded to compress it with UPX (with arguments --best --crp-ms=999999) and got the final result. How cool is that?

I am actively looking for a new job. If you happen to like my writings and think I might be just the right man for the team you’re building up, please feel free to consult my résumé or pass it on.

Update 2010-Jan-17: the above paragraph is no longer valid.

cl-morfeusz: A ninety minutes’ hack

2008-06-23T00:00:00Z

Here’s what I came up with today, after no more than 90 minutes of coding (complete with comments and all):

MORFEUSZ> (morfeusz-analyse "zażółć gęślą jaźń")
((0 1 "zażółć" "zażółcić" "impt:sg:sec:perf")
 (1 2 "gęślą" "gęśl" "subst:sg:inst:f")
 (2 3 "jaźń" "jaźń" "subst:sg:nom.acc:f"))

This is cl-morfeusz in action, a Common Lisp interface to Morfeusz, the morphological analyser for Polish.

It’s a single Lisp file, so there’s no ASDF system definition or asdf-installability for now. I’m not putting it under version control, either. Or, should I say, not yet. When I get around to it, I plan to write a simple parser and write a Polish-language version of the text adventure that started it all.

Meanwhile, you may use cl-morfeusz for anything you wish (of course, as long as you comply with Morfeusz’s license). Have fun!

Update 2010-Jan-17: With the advent of UTF-8 support in CFFI, the ugly workarounds in the code are probably no longer necessary; I don’t have time to check it right now, though.

cl-netstrings

2008-04-30T00:00:00Z

I’ve just packaged up the Common Lisp netstring handling code that I wrote a week ago into a neat library. Unsurprisingly enough, it is called cl-netstrings and has its own home on the Web. It’s even asdf-installable! I wonder whether this one turns out to be useful for anybody besides me…

The other thing I’ve been working on is a new build system for Poliqarp. But that’s the story for another post — most probably I will write about it when it gets out of a state of constant flux.

Update 2010-Jan-17: cl-netstrings is now hosted on GitHub; I’ve updated the link.

Hacking away with JSON-RPC

2008-04-24T00:00:00Z

Let’s try:

(let ((s (socket-stream
          (socket-connect "localhost" 10081
                          :element-type '(unsigned-byte 8)))))
  (write-netstring "{\"method\":\"ping\",\"params\":[],\"id\":1}" s)
  (finish-output s)
  (princ (read-netstring s))
  (close s))
; { "result": "pong" }
; --> T

Yay! This is Common Lisp talking to a JSON-RPC server written in C. This means that I have now the foundations for rewriting Poliqarp on top of JSON-RPC (according to the protocol spec I have recently posted) up and running, and all that remains is to fill the remainder.

Well, to be honest, this is not exactly JSON-RPC. First off, as you might have noticed, the above snippet of code sends JSON-RPC requests as netstrings. This is actually intentional, and the reasons for adopting this encoding have been described in detail in the spec (it basically boils down to the fact that it greatly simplifies reading from and writing to network, especially in C). I wrote some crude code to handle netstrings in CL — now it occurred to me that it might actually be worthwhile to polish it up a little, write some documentation and put on CLiki as an asdf-installable library. I’ll probably get on to this quite soon.

Second, the resulting JSON object does not have all the necessary stuff. It contains the result, but not the error or id (as mandated by the JSON-RPC spec). This is actually a deficiency of the JSON-RPC C library I’m currently using. It places the burden of constructing objects that are proper JSON-RPC responses on the programmer, instead of doing that itself. This will be easy to sort out, however, because the library is small and adheres to the KISS principle. More of a problem is that the licensing of that library is unclear; I emailed the maintainers to explain the status.

ECLM 2008

2008-04-22T00:00:00Z

What is there left for me to do in this life?
Did I achieve what I had set in my sights?
Am I a happy man, or is this sinking sand?
Was it all worth it?—was it all worth it?
— Queen

Gusts of moderate wind are blowing at my face through the open window of a train from Amsterdam to Warsaw as I write these words. The last buildings of Amsterdam have vanished a while ago, giving ground to the damp, low countryside of the Netherlands — not quite fascinating sight to be watching — so I decided to fire up my laptop and write down some impressions while they are sharp and vivid — impressions from the European Common Lisp Meeting that was held in Amsterdam yesterday.

I was there with Maciek and Richard. Amsterdam did not receive us warmly, pouring some mild yet cold rain on us, but our hosts — Lispniks from the Hague, Gabriele and Victor from Streamtech — turned out to be really nice guys. I’m not going to go into a very detailed description of the social aspects of our trip, instead focusing on the conference itself. And that is definitely a topic worth talking about for a long time.

The first man to speak was Jeremy Jones of Clozure Associates, talking about InspireData and how they did it in Lisp. Although they also seem to be the people behind Clozure CL the implementation of Common Lisp, InspireData, the product their presentation was about, seems to have been written in LispWorks. It is a quite interesting application for browsing datasets in many interesting ways and draw conclusions for them. Jeremy started off with a demonstration presenting the key ideas of InspireData and what it can do, and this almost instantly hooked the attention of most of the gathered Lispers; mine, at least, definitely. First off, it seems to be quite a nice success story of a real-world application of Lisp, well worth learning about and mentioning where it deserves a mention. Second, one of its great features shown by the demo is that one can copy HTML tables from a Web browser and paste them as InspireData datasets. Given that Poliqarp now has statistical features and can export its data to HTML, I wonder whether it is possible to couple it with InspireData to interactively explore linguistic material in an absorbing way. That’s certainly a topic worthy of further research.

And last but not least, Jeremy outlined the points they did wrong and those they got right. Among those latter were two letters that now constitute a huge part of my professional life: QA. He just couldn’t emphasize enough how crucial the fact that they had a serious quality assurance process from the very beginning proved to yield the final quality of the product. That’s the lesson I’m now quickly learning. When I learned that InspireData was mostly tested by hand by a skilled QA team, I felt somewhat proud of being able to automate large parts of the process at Sentivision. I’m very curious where this path will lead me to. Let’s hope for the best!

The next speaker was Nicolas Neuss of University of Karlsruhe, talking about Femlisp, a framework for solving partial differential equations in Common Lisp. I have little to say about this one, since I lack the mathematical background needed to fully comprehend and appreciate the topic; it’s just not my kettle of fish. Undoubtedly, though, Femlisp seems to be filling its niche in a neat way, as the demonstrations showed.

After a coffee break, Stefan Richter came up with the one presentation that I’ve been looking forward to the most; that of using Common Lisp for large, scalable Internet systems. After all the talks were over, Maciek dubbed it a “very nice anti-FUD presentation” and I could not agree more. I didn’t learn many new things from it, but the author clearly knows how to attempt to convince non-Lispers to try out Lisp. The talk started off with outlining the typical designs of Web apps and portals, starting with simple one-server scenarios that don’t scale well and progressing in the direction of more scalable and extensible ones. Stefan then pointed out that in some mainstream languages like Java there exists a mature and proven infrastructure for employing such designs. And then came the key point — that this is the case also for Common Lisp! All the necessary tools are there, ready to use Right Now and free for the most part; they’re just not as mature in some cases. This is not much of a problem, though, given the incremental nature of Lisp development: any problem at hand is typically fixable much faster than in case of other languages. The only weird thing was that the author advocated using continuation-based Web frameworks (such as Weblocks or UnCommon Web) just a couple of minutes after discouraging using sticky sessions.

Next came Killian Sprotte with a speech about PWGL, the program for computer-aided music composing. I have very mixed feelings about it. Notice that I didn’t use the word “presentation” — there was no presentation at all. Yes, that’s right. The speaker was just talking and showing off various things in the musical engine. Now, having no presentation accompanying the talk is not necessarily a bad thing in itself; but without one, it’s a little harder to draw attention of the audience and a whole lot easier to deliver a chaotic talk instead of a cleanly-structured and well-organized one. Such was the case with this speech. Some features were shown, but with a fair amount of obscurity and boredom thrown in, leaving me with a rather low overall impression.

As for PWGL itself, some of the ideas employed in it seem a bit peculiar (for want of a better word) to me. As befits a Lisp program, it is an extensible utility that actually allows users to program music just as one programs software. But the way that programming is done… well, think of a graphical editor for Lisp programs. An editor in which to write, say, a factorial function, you right-click an initially blank sheet, select defun from a pop-up with a complicated set of menus and submenus… and kaboom! up comes a box divided into several sub-boxes. They correspond to — what else they could? — the name of the function, list of arguments, and a body. You can draw boxes representing computations, drag them around and link them with arrows — this is supposed to build complicated expressions out of simpler ones. And there is a huge library of musical tools, all available for the convenience of a programmer. Or, should I say, a composer.

Sounds cool? Maybe — for a newbie. I can’t really say. As someone who has high experience with Lisp and programming in general, I can only speak for myself. And for me all this click-and-drag-programming seems to be an unnecessarily tedious, obscure and error-prone way of doing things. Stuff like score editors, chord editors or various transformations is admittedly cool, but for lower-level matters the kind of visualization PWGL offers (and it obviously has its rough edges) seems to get in the way rather than staying out of it. But perhaps that’s just me?

By the time the fourth talk ended, most Lispers were already hungry, so a lunch break followed. I talked to some guy (I don’t remember his name, alas) who’s working on porting Clozure CL to 64-bit Windows. This is great news — when the port’s complete, it has high chances of becoming the free Common Lisp implementation of choice for many Windows Lisp hackers.

Juan José García-Ripoll then talked about ECL, another CL implementation that is characterized by a fairly small memory and disk footprint, while still managing to achieve decent performance (via compilation to C) and good standard compliance. It was good to see that ECL is still quite alive and getting better and better with each release. Just for the heck of it, I attempted in the evening to reproduce the problem I had with ECL a while ago on a fresh CVS checkout. I managed to reproduce it (for the curious, it was an issue with ECL failing to build after having been configured with the option --disable-shared). So I reported the bug to Juan, and he promised to look into it within the next days. And I must say that reporting bugs IRL to open source projects’ maintainers is a very nice experience. :-)

And then came a really big surprise, and I mean a nice surprise. It took the form of Kristofer Kvello of Selvaag, a Norwegian-based house-building company, and his presentation on House Designer, a Common Lisp program for aiding in designing residences, as the name suggests. Yet another example of a success story in an area CL can really excel at. Basically, what House Designer can do is that you give it a sketch, containing a rough description of the shape of a flat or residence and layout of rooms, and out comes a very detailed project with all sorts of bells and whistles: the program automatically figures out what the number of windows should be and where they should be located, the number and location of electric outlets, the optimal types of walls, layout of water installation and what not. It’s transfixing when you think of the sheer amount of tedious labour it automates, taking into account all of the professional knowledge about designing houses accumulated over years, some parts of which a human can easily omit. And it’s been Lisp all the lifetime of this project, and it’s Lisp all the way down (except for the GUI in Java)! Very, very impressive!

Marc Battyani’s talk about programming FPGAs in Lisp probably should not have been stacked so late in the programme. I mean, the topic seems to be quite interesting (though a bit low-level for my interests), but there was something about Marc’s way of talking and showing things that sent me off dozing almost instantaneously. I’d been a bit tired after the many hours of sitting and listening to speeches, especially after having woken up at six o’clock, and so I somewhat regret missing large parts of the talk. It’s nice to know, though, that it is possible to do such things with Lisp. Seems to have a high hack value, as in: “Why do it this way? Because we can!”

And what better end of a conference could one ask for than a rant by Kenny Tilton? If you have only encountered Kenny on the Usenet (some of the crème de la crème of his postings is meticulously collected by Maciek) and think he’s one heck of a freak, you definitely should listen to him live. Here was another talk without slides — just talking and demonstrating stuff — but this time, it was a totally different thing. Kenny sure knows how to attract the attention of the audience and how not to let it loose throughout an hour’s worth of talking. And he changes topics with mastery, using digressions to a great effect to avoid the boredom slipping in, caused by bragging about one thing all the time. There was Cells in that talk, there was teaching of algebra, and there was high-speed driving through the streets of New York. I only hope someone has recorded that to put it online.

So, this was it. There was much talk afterwards, there was much beer, there was much socializing, there was much rejoicing. I saw a real XO-1 and played with it for a while, and boy, isn’t it cute! And then we all came back. And here I am, sitting at my desk in Warsaw (it’s the next day already; I really wish my laptop had a better battery), finishing up this longish blog entry and asking myself: was this 50 euro well spent?

Yes, it was a worthwhile experience, hahhahahahahhaaaa!
(evil chuckle à la Kenny)
It was worth it!