Prologue

I’ll be releasing version 0.3 of Skyscraper, my Clojure framework for scraping entire sites, in a few days.

More than three years have passed since its last release. During that time, I’ve made a number of attempts at redesigning it to be more robust, more usable, and faster; the last one, resulting in an almost complete rewrite, is now almost ready for public use as I’m ironing out the rough edges, documenting it, and adding tests.

This story begins on August 9, 2017, when a friend messaged me on Facebook: “Hey, I’m going to be on a TV talent show this weekend. They’ll be giving me this kind of problems. Any ideas how to prepare?”

Intro

“But, you know, many explorers liked to go to places that are unusual. And, it’s only for the fun of it.” – Richard P. Feynman

A couple of nights ago, I hacked together a small Clojure program.

All it does is displays a terminal window with a red rectangle in it. You can use your cursor keys to move it around the window, and space bar to change its colour. It’s fun, but it doesn’t sound very useful, does it?

Unix Developer: I’m not going to touch Lisp. It’s horrible!

Me: Why so?

UD: The syntax! This illegible prefix-RPN syntax that nobody else uses. And just look at all these parens!

Me: Well, many people find it perfectly legible, although most agree that it takes some time to get accustomed to. But I think you’re mistaken. Lots of people are using Lisp syntax on a daily basis…

Remember Lithium? The x86 assembler written in Clojure, and a simple stripes effect written in it? Well, here’s another take on that effect:

And here is the source code:

Ah, the golden days of childhood’s hackage. Don’t you have fond memories of them?

I got my first PC when I was 10. It was a 486DX2/66 with 4 megs of RAM and a 170 meg HDD; it ran DOS and had lots of things installed on it, notably Turbo Pascal 6. I hacked a lot in it. These were pre-internet days when knowledge was hard to come by, especially for someone living in a small town in Poland; my main sources were the software I had (TP’s online help was of excellent quality), a couple of books, and a popular computing magazine that published articles on programming. From the latter, I learned how to program the VGA: how to enter mode 13h, draw pixels on screen, wait for vertical retrace, manipulate the palette and how to combine these things into neat effects. One of the very first thing I discovered was when you plot every pixel using sum of its coordinates modulo 40 as color, you get a nice-looking diagonal stripes effect. Because of the initially incomprehensible inline assembly snippets appearing all over the place, I eventually learned x86 assembly, too.

tl;dr: Don’t do it. If you really have to, use (#'other-library/private-function args).

A private function in Clojure is one that has been defined using the defn- macro, or equivalently by setting the metadata key :private to true on the var that holds the function. It is normally not allowed in Clojure to call such functions from outside of the namespace where they have been defined. Trying to do so results in an IllegalStateException stating that the var is not public.

I’ve moved to London last September. Like many new Londoners, I have changed accommodation fairly quickly, being already after one removal and with another looming in a couple of months; my current flat was largely unfurnished when I moved in, so I had to buy some basic homeware. I didn’t want to invest much in it, since it’d be only for a few months. Luckily, it is not hard to do that cheaply: many people are moving out and getting rid of their stuff, so quite often you can search for the desired item on Gumtree and find there’s a cheap one a short bike ride away.

Sequences and iteration

There are a number of motivations for this post. One stems from my extensive exposure to Clojure over the past few years: this was, and still is, my primary programming language for everyday work. Soon, I realized that much of the power of Clojure comes from a sequence abstraction being one of its central concepts, and a standard library that contains many sequence-manipulating functions. It turns out that by combining them it is possible to solve a wide range of problems in a concise, high-level way. In contrast, it pays to think in terms of whole sequences, rather than individual elements.

This is a slightly edited translation of an article I first published on my Polish blog on January 19, 2011. It is meant to target newcomers to Clojure and show how to use Clojure to solve a simple real-life problems.