Archives

• 2008 (21)
  • April (12)
  • May (2)
  • June (4)
  • August (1)
  • September (1)
  • December (1)
• 2009 (1)
  • January (1)
• 2010 (8)
  • January (1)
  • February (2)
  • March (1)
  • April (3)
  • May (1)

Wed, 30 Apr 2008

I’ve just packaged up the Common Lisp netstring handling code that I wrote a week ago into a neat library. Unsurprisingly enough, it is called cl-netstrings and has its own home on the Web. It’s even asdf-installable! I wonder whether this one turns out to be useful for anybody besides me…

The other thing I’ve been working on is a new build system for Poliqarp. But that’s the story for another post — most probably I will write about it when it gets out of a state of constant flux.

Update 2010-Jan-17: cl-netstrings is now hosted on GitHub; I’ve updated the link.

permanent link | comments

Fri, 25 Apr 2008

You see, there’s this Linux kernel module for kernels 2.6.17 and up (that’s what the phrase fairly recent in the previous paragraph macroexpands to), called Compcache. It works by slicing out a contiguous chunk of your RAM (25% by default, but it’s settable, of course) and setting it up as a swap space with highmost priority. The trick is that pages that are swapped out to this area are compressed using the LZO algorithm, which provides very fast compression/decompression while maintaining a decent compression ratio. In this way, more unused pages can fit in memory, and less of them are swapped out to disk, which can considerably cut down disk swap usage. I’ve enabled it in my system and it doesn’t seem to cause any problems, while providing a visible efficiency boost. Here’s how I did it on a freshly-installed Ubuntu Hardy:

• I installed the Ubuntu package build-essential, then downloaded Compcache from its site, extracted it, entered its directory and compiled it by saying make. So far, so easy.

• Unfortunately, one cannot say make install — creating a flexible cross-distro install target is admittedly hard. So I installed it by hand, ensuring that my system enables it automatically on boot-up.

• I created a directory /lib/modules/2.6.24-16-generic/ubuntu/compcache/ and copied the four kernel modules (compcache.ko, lzo1x_compress.ko, lzo1x_decompress.ko, and tlsf.ko) created by the compilation to that directory.

• Next, I ran depmod -a to make the modules loadable by modprobe.

• I edited the file /etc/modules and added a line at the end, containing the single word compcache.

• I copied the shell scripts use_compcache.sh and unuse_compcache.sh that come with compcache to /usr/local/bin.

• I created an executable script /etc/init.d/compcache with the following contents:

  #!/bin/sh
  case "$1" in
    start)
      /usr/local/bin/use_compcache.sh ;;
    stop)
      /usr/local/bin/unuse_compcache.sh ;;
  esac
  

• The last step was to create a symlink /etc/rc2.d/S02compcache pointing to that script.

I then rebooted the system and verified that the new swapspace is in use:

nathell@chamsin:~$ cat /proc/swaps
Filename        Type        Size    Used    Priority
/dev/sdb2       partition   996020  0       -1
/dev/ramzswap0  partition   128896  111396  100

With the final release of Hardy installed on my main box and compcache optimizing its memory usage, I do not hesitate to call this combo the best OS I have ever had installed.

And no, I don’t own a Mac. :-/

permanent link | comments

Thu, 24 Apr 2008

(let ((s (socket-stream 
          (socket-connect "localhost" 10081 
                          :element-type '(unsigned-byte 8)))))
  (write-netstring "{\"method\":\"ping\",\"params\":[],\"id\":1}" s)
  (finish-output s)
  (princ (read-netstring s))
  (close s))
{ "result": "pong" }
--> T

Yay! This is Common Lisp talking to a JSON-RPC server written in C. This means that I have now the foundations for rewriting Poliqarp on top of JSON-RPC (according to the protocol spec I have recently posted) up and running, and all that remains is to fill the remainder.

Well, to be honest, this is not exactly JSON-RPC. First off, as you might have noticed, the above snippet of code sends JSON-RPC requests as netstrings. This is actually intentional, and the reasons for adopting this encoding have been described in detail in the spec (it basically boils down to the fact that it greatly simplifies reading from and writing to network, especially in C). I wrote some crude code to handle netstrings in CL — now it occurred to me that it might actually be worthwhile to polish it up a little, write some documentation and put on CLiki as an asdf-installable library. I’ll probably get on to this quite soon.

Second, the resulting JSON object does not have all the necessary stuff. It contains the result, but not the error or id (as mandated by the JSON-RPC spec). This is actually a deficiency of the JSON-RPC C library I’m currently using. It places the burden of constructing objects that are proper JSON-RPC responses on the programmer, instead of doing that itself. This will be easy to sort out, however, because the library is small and adheres to the KISS principle. More of a problem is that the licensing of that library is unclear; I emailed the maintainers to explain the status.

permanent link | comments

permanent link | comments

Tue, 22 Apr 2008

Gusts of moderate wind are blowing at my face through the open window of a train from Amsterdam to Warsaw as I write these words. The last buildings of Amsterdam have vanished a while ago, giving ground to the damp, low countryside of the Netherlands — not quite fascinating sight to be watching — so I decided to fire up my laptop and write down some impressions while they are sharp and vivid — impressions from the European Common Lisp Meeting that was held in Amsterdam yesterday.

I was there with Maciek and Richard. Amsterdam did not receive us warmly, pouring some mild yet cold rain on us, but our hosts — Lispniks from the Hague, Gabriele and Victor from Streamtech — turned out to be really nice guys. I’m not going to go into a very detailed description of the social aspects of our trip, instead focusing on the conference itself. And that is definitely a topic worth talking about for a long time.

The first man to speak was Jeremy Jones of Clozure Associates, talking about InspireData and how they did it in Lisp. Although they also seem to be the people behind Clozure CL the implementation of Common Lisp, InspireData, the product their presentation was about, seems to have been written in LispWorks. It is a quite interesting application for browsing datasets in many interesting ways and draw conclusions for them. Jeremy started off with a demonstration presenting the key ideas of InspireData and what it can do, and this almost instantly hooked the attention of most of the gathered Lispers; mine, at least, definitely. First off, it seems to be quite a nice success story of a real-world application of Lisp, well worth learning about and mentioning where it deserves a mention. Second, one of its great features shown by the demo is that one can copy HTML tables from a Web browser and paste them as InspireData datasets. Given that Poliqarp now has statistical features and can export its data to HTML, I wonder whether it is possible to couple it with InspireData to interactively explore linguistic material in an absorbing way. That’s certainly a topic worthy of further research.

And last but not least, Jeremy outlined the points they did wrong and those they got right. Among those latter were two letters that now constitute a huge part of my professional life: QA. He just couldn’t emphasize enough how crucial the fact that they had a serious quality assurance process from the very beginning proved to yield the final quality of the product. That’s the lesson I’m now quickly learning. When I learned that InspireData was mostly tested by hand by a skilled QA team, I felt somewhat proud of being able to automate large parts of the process at Sentivision. I’m very curious where this path will lead me to. Let’s hope for the best!

The next speaker was Nicolas Neuss of University of Karlsruhe, talking about Femlisp, a framework for solving partial differential equations in Common Lisp. I have little to say about this one, since I lack the mathematical background needed to fully comprehend and appreciate the topic; it’s just not my kettle of fish. Undoubtedly, though, Femlisp seems to be filling its niche in a neat way, as the demonstrations showed.

After a coffee break, Stefan Richter came up with the one presentation that I’ve been looking forward to the most; that of using Common Lisp for large, scalable Internet systems. After all the talks were over, Maciek dubbed it a “very nice anti-FUD presentation” and I could not agree more. I didn’t learn many new things from it, but the author clearly knows how to attempt to convince non-Lispers to try out Lisp. The talk started off with outlining the typical designs of Web apps and portals, starting with simple one-server scenarios that don’t scale well and progressing in the direction of more scalable and extensible ones. Stefan then pointed out that in some mainstream languages like Java there exists a mature and proven infrastructure for employing such designs. And then came the key point — that this is the case also for Common Lisp! All the necessary tools are there, ready to use Right Now and free for the most part; they’re just not as mature in some cases. This is not much of a problem, though, given the incremental nature of Lisp development: any problem at hand is typically fixable much faster than in case of other languages. The only weird thing was that the author advocated using continuation-based Web frameworks (such as Weblocks or UnCommon Web) just a couple of minutes after discouraging using sticky sessions.

Next came Killian Sprotte with a speech about PWGL, the program for computer-aided music composing. I have very mixed feelings about it. Notice that I didn’t use the word “presentation” — there was no presentation at all. Yes, that’s right. The speaker was just talking and showing off various things in the musical engine. Now, having no presentation accompanying the talk is not necessarily a bad thing in itself; but without one, it’s a little harder to draw attention of the audience and a whole lot easier to deliver a chaotic talk instead of a cleanly-structured and well-organized one. Such was the case with this speech. Some features were shown, but with a fair amount of obscurity and boredom thrown in, leaving me with a rather low overall impression.

As for PWGL itself, some of the ideas employed in it seem a bit peculiar (for want of a better word) to me. As befits a Lisp program, it is an extensible utility that actually allows users to program music just as one programs software. But the way that programming is done… well, think of a graphical editor for Lisp programs. An editor in which to write, say, a factorial function, you right-click an initially blank sheet, select defun from a pop-up with a complicated set of menus and submenus… and kaboom! up comes a box divided into several sub-boxes. They correspond to — what else they could? — the name of the function, list of arguments, and a body. You can draw boxes representing computations, drag them around and link them with arrows — this is supposed to build complicated expressions out of simpler ones. And there is a huge library of musical tools, all available for the convenience of a programmer. Or, should I say, a composer.

Sounds cool? Maybe — for a newbie. I can’t really say. As someone who has high experience with Lisp and programming in general, I can only speak for myself. And for me all this click-and-drag-programming seems to be an unnecessarily tedious, obscure and error-prone way of doing things. Stuff like score editors, chord editors or various transformations is admittedly cool, but for lower-level matters the kind of visualization PWGL offers (and it obviously has its rough edges) seems to get in the way rather than staying out of it. But perhaps that’s just me?

By the time the fourth talk ended, most Lispers were already hungry, so a lunch break followed. I talked to some guy (I don’t remember his name, alas) who’s working on porting Clozure CL to 64-bit Windows. This is great news — when the port’s complete, it has high chances of becoming the free Common Lisp implementation of choice for many Windows Lisp hackers.

Juan José García-Ripoll then talked about ECL, another CL implementation that is characterized by a fairly small memory and disk footprint, while still managing to achieve decent performance (via compilation to C) and good standard compliance. It was good to see that ECL is still quite alive and getting better and better with each release. Just for the heck of it, I attempted in the evening to reproduce the problem I had with ECL a while ago on a fresh CVS checkout. I managed to reproduce it (for the curious, it was an issue with ECL failing to build after having been configured with the option --disable-shared). So I reported the bug to Juan, and he promised to look into it within the next days. And I must say that reporting bugs IRL to open source projects’ maintainers is a very nice experience. :-)

And then came a really big surprise, and I mean a nice surprise. It took the form of Kristofer Kvello of Selvaag, a Norwegian-based house-building company, and his presentation on House Designer, a Common Lisp program for aiding in designing residences, as the name suggests. Yet another example of a success story in an area CL can really excel at. Basically, what House Designer can do is that you give it a sketch, containing a rough description of the shape of a flat or residence and layout of rooms, and out comes a very detailed project with all sorts of bells and whistles: the program automatically figures out what the number of windows should be and where they should be located, the number and location of electric outlets, the optimal types of walls, layout of water installation and what not. It’s transfixing when you think of the sheer amount of tedious labour it automates, taking into account all of the professional knowledge about designing houses accumulated over years, some parts of which a human can easily omit. And it’s been Lisp all the lifetime of this project, and it’s Lisp all the way down (except for the GUI in Java)! Very, very impressive!

Marc Battyani’s talk about programming FPGAs in Lisp probably should not have been stacked so late in the venue. I mean, the topic seems to be quite interesting (though a bit low-level for my interests), but there was something about Marc’s way of talking and showing things that sent me off dozing almost instantaneously. I’d been a bit tired after the many hours of sitting and listening to speeches, especially after having woken up at six o’clock, and so I somewhat regret missing large parts of the talk. It’s nice to know, though, that it is possible to do such things with Lisp. Seems to have a high hack value, as in: “Why do it this way? Because we can!”

And what better end of a conference could one ask for than a rant by Kenny Tilton? If you have only encountered Kenny on the Usenet (some of the crème de la crème of his postings is meticulously collected by Maciek) and think he’s one heck of a freak, you definitely should listen to him live. Here was another talk without slides — just talking and demonstrating stuff — but this time, it was a totally different thing. Kenny sure knows how to attract the attention of the audience and how not to let it loose throughout an hour’s worth of talking. And he changes topics with mastery, using digressions to a great effect to avoid the boredom slipping in, caused by bragging about one thing all the time. There was Cells in that talk, there was teaching of algebra, and there was high-speed driving through the streets of New York. I only hope someone has recorded that to put it online.

So, this was it. There was much talk afterwards, there was much beer, there was much socializing, there was much rejoicing. I saw a real XO-1 and played with it for a while, and boy, isn’t it cute! And then we all came back. And here I am, sitting at my desk in Warsaw (it’s the next day already; I really wish my laptop had a better battery), finishing up this longish blog entry and asking myself: was this 50 euro well spent?

Yes, it was a worthwhile experience, hahhahahahahhaaaa!
(evil chuckle à la Kenny)
It was worth it!

permanent link | comments

Wed, 16 Apr 2008

The first version of the document I’ve been writing about a couple of days ago is now ready for public review. I’ll be making an initial attempt at the implementation once I return from the European Common Lisp Meeting ‘08 and write a report.

permanent link | comments

Mon, 14 Apr 2008

1. Get a final draft of the C++09 standard when it’s ready and acquaint myself with it as closely as possible. I strongly dislike C++ (and I’m not alone in this — see the Frequently Questioned Answers about C++ for very detailed criticisms); however, I’ve long wanted to learn that language better just to know all the strengths and weaknesses of the enemy. The ideal moment for this will be when the new standard is out; this will give me the advantage of not having to unlearn the things changed by the standard, while staying on a cutting and competitive edge.

2. Get a copy of Federico García Lorca’s poems translated into Polish by Jerzy Ficowski. I have only a very vague knowledge of Lorca (just his Romance of the Spanish Civil Guard (Romance de la Guardia Civil Española)), but I very much like what little I know.

permanent link | comments

Thu, 10 Apr 2008

LaTeX + Beamer (for typesetting the presentation in a visually pleasant, clean, simple and consistent way) + KeyJNote (for presenting it stylishly to the audience) = a recipe for success. In particular, KeyJNote, which I found only yesterday, seems to be a fine and tremendously useful piece of software, despite being very young. The only annoyance I have found in it is that it doesn’t respond to Alt-Tab when in fullscreen mode. On the typographical side, I used the progressbar Beamer theme and the Torunian Antiqua font, both to great effect.

While I’m at this topic, Jan Rychter has recently posted a great guide to giving presentations, especially short ones. I heartily recommend it to those of you who speak Polish (is there actually any non-Polish-speaking person reading this?)

Update 2010-Jan-17: KeyJNote is now called Impressive.

permanent link | comments

Tue, 08 Apr 2008

Yesterday, my level of frustration with my old operating system at work exceeded a critical point, and I installed a fresh daily build of the not-yet-released Ubuntu 8.04 in place of it. Then, in addition to usual post-installation chores like setting up mail, hardware, etc., I performed a couple of steps to make the system more pleasurable to use. Here’s what I did, just in case someone finds this useful.

1. First, I tweaked the font rendering. This was one area that has long been a PITA for Linux users (at least for me, since 2000 or so), but as far as Ubuntu is concerned, they introduced a change to Freetype somewhere along the way between Feisty and Gutsy which, when set up properly, makes the font rendering on LCD displays far superior for me to that of, say, Windows XP, in particular at small font sizes. The way to enable it is to enable sub-pixel rendering, and set the hinting level to “slight.” This results in an rendering very close to what the author of Texts Rasterization Exposures managed to achieve.

2. I installed the package msttcorefonts to get Microsoft’s free-as-in-beer set of core TrueType fonts, including Times New Roman, Arial, Georgia, etc. There are very many sites out there on the Web that were designed with these fonts in mind, and this is one of the few areas Microsoft doesn’t completely suck at.

3. Next I enabled bitmap fonts. The way to do this is to become root, cd to /etc/fonts/conf.d, remove the symlink named 70-no-bitmaps.conf, and make a symlink pointing to /etc/fonts/conf.avail/70-yes-bitmaps.conf instead. This would come in handy in the next step.

4. Which was installing my favourite console font. Unfortunately, it doesn’t come preinstalled with the Gnome-based Ubuntu, but it was no big deal. The font is named console8x16 and it comes with Kubuntu’s (and KDE’s) default terminal emulator, Konsole. So I downloaded an appropriate package (manually, without the help of APT, because all I wanted was the font, not the package itself). I then installed Midnight Commander (which I use a lot, if only for its great vfs feature, which allows to access, inter alia, Debian/Ubuntu packages as if they were directories), grabbed the file console8x16.pcf.gz, installed it in /usr/share/fonts/X11/misc, changed to that directory, ran mkfontdir and mkfontscale, logged out and restarted the X server.

5. The last step was to use this font for Emacs, too. So I installed Emacs, created the file ~/.Xdefaults containing the single line

   Emacs*font: -misc-console-medium-r-normal--16-160-72-72-c-80-iso10646-1
   

   and ran xrdb ~/.Xdefaults.

Then I got round to configuring Emacs itself. But that’s a story for another post.

permanent link | comments

Mon, 07 Apr 2008

Yay! Now that I have finished ADOM for the first time ever, after something like six years of trying, I can finally get back to work with peace of mind. :-)

permanent link | comments

Sun, 06 Apr 2008

After a longish while of inactivity, I finally got around to finishing the draft spec of a next-generation protocol for Poliqarp, the be-all-end-all corpus concordance tool that I maintain. The spec is being written in LaTeX, and it has a number of subsections that describe particular methods of the protocol. Each one of those is further divided into sub-subsections that describe the method’s signature, purpose, syntax of request, syntax of response, and an optional example. I thought to write a couple of macros to help me separate the document’s logic from details of formatting, so that I could say:

\synopsis/() -> {version : int; extensions : string*}/

and have it expanded into:

\paragraph{Synopsis}
\verb/{version : int; extensions : string*}/

Being a casual LaTeX user who hardly ever writes his own macros, I first thought to use LaTeX’s command-defining commands, \newcommand and \renewcommand. However, I quickly ran into the limitation that the argument of commands defined in such a way can only be delimited by curly braces, which I could not use because they might appear in the argument itself.

I googled around and found that this limitation can be overcome by using \def instead, which is not a LaTeX macro but rather an incantation of plain TeX, and allows to use arbitrary syntax for delimiting arguments. Having found that, my first shot was:

\def\synopsis/#1/{\paragraph{Synopsis}\verb/#1/}

which, obviously enough, turned out not to work, producing errors about \verb ended by an end-of-line.

“What the heck?” I thought, and resorted to Google again, this time searching for tex macros expanding to verb. This yielded an entry from some TeX FAQ, which basically states that the \verb is a “fragile” command, and as such it cannot appear in bodies of macros. Ook. So it can’t be done?

“But,” I thought, “TeX is such a flexible and powerful tool, there must be some way around this!” And, as it would turn out, there is. Yet more googling led me to this thread on comp.text.tex, where someone gives the following answer for a similar question:

\def\term#{% 
   \afterassignment\Term \let\TErm= }% 

\edef\Term{\noexpand\verb \string}}

Now this is overkill. Why in the world am I forced to stuff such incomprehensible hackery into my document just to perform a seemingly simple task?! Easy things should be easy — that’s one of the principles of good design.

Reluctantly, I copied it over, and attempted to adjust it to my needs. After a number of initial failed attempts, I thought that I might actually attempt to understand what all these \afterassignment’s, \noexpand’s and \edef’s are for, so I downloaded the TeXbook and dived straight in.

I spent another fifteen minutes or so reading bits of it and trying to understand tokens, macros, when they are expanded and when merely carried over, etc. But a sparkle of thought made me replace the whole complicated thingy with a simple snippet that actually worked.

\def\synopsis{\paragraph{Synopsis}\verb}

That’s right. This superficially resembles a C preprocessor macro, and works because I was lucky enough to have \verb appear last in the definition, thus allowing the “arguments” of \synopsis to be specified just like arguments to \verb and fit at exactly right place. I’m almost certain that it does not always work this way, but for now it’ll suffice.

Oh well. TeX is undoubtedly a fine piece of software that provides splendid results if used right. But I can’t get over the impressions that there are a great deal more idiosyncracies like this in it than in, say, Common Lisp, even though the latter’s heritage tracks back to as early as 1958 and is a whopping twenty years longer than TeX’s. (On the side note, as it turns out, someone has already written a Lisp-based preprocessor for TeX macros. Gotta check it out someday.)

As for the TeXbook itself: it is a fine piece of documentation that I will definitely have to add to my must-read list, though it admittedly has a math-textbookish feel to it. First, however, I want to finish “Shaman’s Crossing” by Robin Hobb (which I will probably brag about in a separate post once I’m finished with it) and tackle Christian Queinnec’s “Lisp in Small Pieces”.

permanent link | comments

Fri, 04 Apr 2008

So, there. Inspired by the newly-started blogs of some of my acquaintances, I had this thought that I might actually have a word or two on a number of subjects, and that it might even be worth sharing. And here I am, typing this introductory entry in my Emacs.

I guess the first thing one usually writes about in a blog is introducing himself to the public, so for those of you who have arrived here through some links on the Web and don’t know me, here goes. I am a 23-year-old (soon to be 24, though) programmer geek, living in Warsaw, Poland. In 2006, I graduated from Warsaw University, where I majored in computer science, and am now working full-time as a senior software engineer at Sentivision. I have a homepage (currently in Polish only, though I probably will translate it to English some day). These days, I tend to use Common Lisp for most of my programming work, though I also occassionally use Python, Ruby, C, Perl, OCaml, Haskell, Java, and a handful of other languages.

This is going to be a blog about my personal interests. This means mostly programming, with a fair share of posts about books, poetry, Scrabble, music, biking, cats, and a bunch of other topics thrown in. The new entries will most likely be added irregularly, whenever I feel like sharing a thought. My mother tongue is Polish, but I will try to maintain this blog in English just to polish up my English writing skills (no pun intended) and for greater worldwide understandability.

As you might have noticed, there is no possibility of leaving comments. This is a side-effect of the fact that this blog is, technically, just a bunch of static HTML pages (automatically generated with Blosxom), and is in line with my idea of blog comments: I view them as a way of providing direct feedback to the author, not as a publically available message board with discussions having a heavy tendency to drift off topic. So, should you like to comment on some post, feel free to drop me an email; I’ll be happy to respond to interesting mails in the blog. I can be contacted at dj at danieljanus dot pl.

Update 2010-Jan-17: Several things have changed — in particular, Sentivision doesn’t exist anymore — but I haven’t done any editing in this post other than updating the links.

permanent link | comments