building the command-line reliably from user provided input

The shell offers many opportunities for zen-like unlightenment, by throwing spitballs at every attempt to make sense of user-generated input handling. By user-generated I mean anything originating from an unreliable entity (a.k.a. a user), and therefore without any guarantees towards being well-formed for its intended purpose. Users are notoriously good at this. You ask them to type in a number, they say 'twelve'. You ask them for their surname, they type 'Mozes Kriebel'. The last case is problematic because most programmers are just like ordinary people, and ordinary people assume surnames are just single words. Our habit is reasoning about the common case, instead of considering all possible alternatives, and therefore we live our lives believing such unbearable nonsense as 'all crocodiles are green', 'all people have ten fingers' and 'it never rains in the Sahara.' Each of us has his own set of ridiculous believes; I feel safe in the claim that there isn't a fact so wrong or it has a true believer. But maybe that's just one of my beliefs.
Back to the programmer who made the wrong assumption that surnames don't contain spaces. Our friend is trying to process a dataset of names by calling a utility for each one and he's using shell scripts. His first attempt didn't go so well.
name=`get_next_surname`
process_surname $name
The process utility expects just one argument--a surname. But when get_next_surname returned Mozes Kriebel, the shell's word splitting turned this into two arguments: "Mozes" and "Kriebel".
The programmer would soon overcome this problem by adding double quotes to prevent word splitting:
name=`get_next_surname`
process_surname "$name"
That will do for today's lesson. But wait, there is more!
Let's assume that the process_surname utility is actually richer in functionality, and may accept some options with arguments.
if [ "$name" != "$origname" ]; then
option="--nee $origname"
fi
process_surname $option "$name"
See what happened? If not, here's a short run-down. The first line checks if the name is identical to the original name (assume for the sake of the example that a name change could happen through marriage). Here, the double quotes are required as in the original script, to defend against shell script errors. If $name was unquoted, the test would read 'Mozes Kriebel != ...' which is incorrect syntax. The second line sets the --nee option, to pass to the processing function later on. This is also a quoted string.
The catch is in the final line. Observant readers spot the lack of quotes around $option. This is not a mistake! If $option were quoted, it would be passed as $1 to process_surname in its entirety, i.e. including the space following the '--nee' and the original surname after that. If this utility scans it's arguments looking for an exact match of '--nee', it won't find it. So we need the shell's word-splitting to separate '-nee' from what comes after it.
The problem is now clear. If $origname happens to be 'Jemig de Pemig' there seems to be no way to preserve the spaces on passing it as an argument to --nee.
I won't dwell on my journey along the Path of Many Misconceptions About the Shell, but I will show you just about the simplest way to do this generally.
set --
if [ "$name" != "$origname" ]; then
set -- --nee "$origname"
fi
process_surname "$@" "$name"
This is one of the few times I found a use for setting the positional parameters. The magic bit is in the use of "$@", which expands to the positional parameters, with quotes around each individual parameter. There is no other construct in the shell that does this. The set on line 3 made $1 equal to "--nee" and $2 equal to "Jemig de Pemig". The last line is then equivalent to
process_surname "--nee" "Jemig de Pemig" "Mozes Kriebel"
which is exactly what we need.

CREAM did it, using bugs in path length constraints, in OpenSSL/Globus

Well, here goes. My first addition to the blog after many years of having the right to do so. As I've got a nice piece of software witchcraft to uncover.

This regards GGUS ticket #67040, which can be viewed by the happy few at the URL: https://ggus.eu/ws/ticket_info.php?ticket=67040

In short (inspired by the game Cluedo):

CREAM did it, using bugs in path length constraints, in OpenSSL/Globus

And now the slightly more elaborate explanation about the problem, how we analyzed it, interpreted the information and implemented a reliable workaround. It also shows that the CREAM CE itself is not directly the cause, but a trigger of the bug. This problem can occur in a lot of other places too and is a pain to analyse. One added motivation on why its such a pain to analyse is that I'm seeing known effects and problems occur along the analyses steering me in mildly the right direction, while I'm already mind-programming a workaround.

Reproducing the problem was hard:

The effects observed by users is a failure in job submission to any gLite 3.2 CREAM CE, when its submitted through a WMS. Probably also on all EMI-1 CREAM CE too. The error message returned from the CREAM CE indicates a failure in gLExec's LCMAPS plugin that verifies a proxy certificate chain.

Prerequisites (all of this must be true aka logical AND) to trigger the faulty situation:

- Use the Terena eScience Personal TCS, which has a pathlen = 0 set on the final CA.

- Use old style proxies (GT2), note: they don't feature a path length constraint field.

- Use a CREAM CE on gLite 3.2 (uses Globus GT4 from VDT)

- Access the CREAM CE through a WMS to use sufficient delegations or MyProxy

Change any of the above parameters and it will work. Meaning, the problem did NOT occure when the following was used:

- Direct job submission (only ONE proxy delegation may be used)

- Direct gLExec test on the shell, which just works.

Unverified situations:

- The effects when using RFC 3820 proxies

- Using EMI-1's CREAM CE

Hypothesis:

Tests have shown that the certificate chain is constructed properly. The hypothesis is that the GT4 from the VDT is interfering with OpenSSL sequences that we rely on in LCMAPS.

Cause(s) of the problem and analyses so far:

The gLExec in the CREAM CE uses LCMAPS to perform the account mapping in gLite 3.2. LCMAPS is dynamically linked to Globus to support its direct Globus based interfaces. The LCMAPS framework launched several plugins, of which the verify-proxy is the first, from the lcmaps-plugins-verify-proxy package.

The verify-proxy fails with an error in the log file, originating from OpenSSL, that the path length of the certificate chain exceeded the constraint bound from the certificate chain itself. Analyses of the chain has shown that both the RFC5280 path length constraint and the RFC3820 path length constraint did not apply here. The Terena eScence TERENA eScience Personal CA has a critical basic constraint set to indicate a path length is 0 (=zero). This means that no other CA certificate can follow this CA certificate in a chain. The RFC 3820 path length constraint doesn't apply on old-style (i.e. GT2) proxy certificates.

Despite the installation and the certificate chain involved; OpenSSL triggers an X509_V_ERR_PROXY_PATH_LENGTH_EXCEEDED error code, indicating the path length exceeded in the proxy certificates. Given the research on the certificate chain we will assume that this is a false-positive (or true-negative).

See wiki for details: http://www.nikhef.nl/pub/projects/grid/gridwiki/index.php/How_to_handle_OpenSSL_and_not_get_hurt_and_what_does_that_library_call_really_do%3F#Path_length_constraints

The interesting details here is that the Terena eScience Personal CA, Terena eScience SSL CA and the FNAL SLCS are the only CAs using a Path Length Constraint of 0 (=zero) in the IGTF. This gives a motivation to search in this direction as similar certificate chains are not affected at all.

On both our EMI and gLite 3.2 test nodes running gLExec we couldn't reproduce the problem. We tried a gLite 3.2 CREAM CE and could reproduce the failure when we introduced a few extra delegations to the certificate chain before we submitted a test job.

After looking at the libraries used on the CREAM CE, being GT4 from the VDT, and knowing that the OpenSSL interaction is significantly different made us put the blame on the GT4 libraries. They are known to have changed parts of OpenSSL itself and their own callbacks. This might cause the weird effect in the verification stage. We've experienced race condition in library loading where the order of dynamic library resolvement and loading was significant for the observed failures. This problem has characteristics of it as the problem seemed to be specifc to the machine. We would need to investigate the GT4 OpenSSL interacting code to be certain about it. This is not an easy task and might be too expensive, while a work around is possible.

We looked at the CREAM CE interaction some more, installed a new CREAM CE from scratch and were interested to reproduce the problem in gLExec. Somehow we couldn't reproduce it when we ran gLExec standalone on the CREAM CE. This should not happen. It should have failed. We tried another proxy chain (mine this time) created from my OSX build of voms-proxy-init version 1.8.8. Again, the problem didn't occure. I hacked the gLExec script that was executing on the failing CREAM CE, which I tested using the glite-ce-job-submit tool, to copy the proxy certificate before deleting itself. We used this chain in the bare gLExec run and then it failed. This certificate chain was examined, turned out to be OK, but is different as it had CA certificates in it.

This seemed to be the root cause of the problem. The CREAM CE (or perhaps its delegation service) is writing the proxy certificate chain from the SSL contect in the Tomcat instance from the user's interaction. This certificate chain was writing including all the CA certificates up to the root CA.

We tested the gLExec with the output of voms-proxy-init/grid-proxy-init which do *not* include the CA certificates in the certificate chain. As this is not added, the CA certificates will be added to the verification sequences in a different way by the OpenSSL routines. This is required to verify the full chain. There is a use case for adding your own (intermediate) CA to the client/host certificate chain, but this doesn't count in the Grid world with the IGTF. As the CA certificates are added in a different way later and treated differently, OpenSSL will verify the certificate chain differently. Either the Globus OpenSSL or the OpenSSL 0.9.8a is to blame that certificate chains with old-style proxies have the path length constraint field, used exlusively for RFC 3820 proxies, set to 0 (=zero) instead of -1 (=minus one) aka uninitialized. This nullification is most probably triggered by the path length constraint value in the Terena sub-CA certificate added to the normal certificate chain evaluation sequences, instead of kept aside in the list of used CA certificates for a certificate chain in an SSL context.

Workaround:

Build a DIY (=Do It Yourself) Path Length Constraint a la RFC 5280 and RFC 3820 in the verify proxy LCMAPS plugin. This will work around any potential library loading issue that could possibly happen. It also works around odd implementations of the verification sequences and it can work around the bug of wrong initialization values for path length constraint. Another possible workaround would be to alter the certificate chain before it hits the verification stage. This could work, but needs research in the right code-wise location in OpenSSL to let this work reliably. We're also going to introduce a duplication of the certificate chain to not tamper with the original input and pragmatically we need to work with two different certificate chains. The first option is significantly less work and straight forward.

To consider for other tools:

OpenSSL and possibly GT5 needs double checking if the support for RFC proxies is capable of handling edge-case input, demonstrated by the CREAM CE (or a component thereof). The CREAM CE should not add the CA certificates to the gLExec input. We should be tolerant on the gLExec side, but regardless the CREAM CE should not have done this and should have followed the same approach with gLExec as to setting up an SSL context. This means that you do not send CA certificates over the wire unless you are absolutely sure that this is really needed.

Output:

lcmaps-plugins-verify version 1.4.11 is to be certified featuring a function to catch the X509_V_ERR_PROXY_PATH_LENGTH_EXCEEDED error and check the certificate chain for its RFC 5280 and RFC 3820 compliance regarding path length constraints.

notes from a dirty system installation

Normal system installations involve boot media, such as a CD-ROM, USB, or even a floppy. In our case it's PXE boot (netboot), which is a little more involved to set up initially because you need a network plan with DNS, DHCP, tftp and probably HTTP, but it is definitely worth the effort if you have to manage a couple of hundred systems. Some new ways to do installations have arrived with the introduction of virtual machines, and this is very easy as you only need to provide the disk or CD-ROM images as files on the host system.
But Sven asked me to transfer a system from a virtual machine to a physical box, for some reason that I won't mention here now. He would provide me with the (small) disk image of the virtual machine, and the physical box was something old that had already seen some use and was hooked up with the network.
I quickly realised that this was going to be an interesting exercise. I would need to write the image to disk, which meant that I had to boot into a ramdisk of sorts. The first problem that presented itself was that the box would only do PXE boot, and as the network was not under my control I would have to involve other system administrators.
The box had a previous installation on it (Backtrack, which is Debian based), and I figured I might as well try to do everything from within this installation.
After adding my ssh key to /root/.ssh/authorized_keys (and turning off the firewall) I could get out of the noise of the machine room and work from the peaceful quiet of my office. By inspecting /proc/partitions I found out the machine had 2 disks, and Sven agreed that we should set up a (software) RAID1 mirror set.

Now the system was running from /dev/hda1, and I couldn't mess with that disk live. (You should try this some day if you feel in a particular evil mood; run dd if=/dev/zero of=/dev/sda in the background while you continue to work. Observe how the system develops amnesia, dementia and finally something close to mad cow disease.) I decided to do something dirty: I would create a RAID1 set with just one disk. The mdadm program thinks this is a bad idea, so you have to --force the issue. The command-line was mdadm --create --level=1 -n 1 --force /dev/md0 /dev/hdb1 or something. Of course I first repartitioned /dev/hdb to have just a single partition of type raid autodetect. Next step: losetup -f vmdisk.img to treat the disk image as a block device, and kpartx -a /dev/loop0 to have mapped devices for each of the partitions inside. Now a simple dd if=/dev/mapper/loop0p1 of=/dev/md0 was all I needed to write the image.
The next step was to boot into the newly written system (a Debian 5). This involved some grubbery, after mount /dev/md0 /mnt and chroot /mnt I could navigate the system as if it was already there. I had to edit /boot/grub/menu.lst to set the root device to (hd1,0) and root=/dev/md0, and I had to install grub on the first bootable disk, which was (hd0,0). After cloning /etc/network/interfaces from the present system and setting up ssh keys again to ensure acces, I rebooted with fingers crossed.
Call it luck, but it worked. I was now running the cloned system from a RAID1 root device with only one disk. I did a resize2fs /dev/md0 because the image I originally wrote to it was really small compared to the disk. Now it was /dev/hda's turn to be added to the RAID set. After repartitioning to have the same size as its counterpart (the two disks weren't the same size), I added it with mdadm --manage --add /dev/md0 /dev/hda1 which unfortunately didn't work as expected, as the new addition just became a spare.

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hda1[1](S) hdb1[0]
120053632 blocks [1/1] [U]

Notice the (S) which indicates that hda1 is a spare. It won't be used until another disk fails, but as this set unfortunately only has a single disk, a single disk failure means game over.
The final command to activate the spare was mdadm --grow --raid-devices=2 /dev/md0. This enlarges the raid set, and the spare will now be activated. Indeed, the system started to recover immediately:

# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 hda1[2] hdb1[0]
120053632 blocks [2/1] [U_]
[>....................] recovery = 0.0% (84608/120053632) finish=70.8min speed=28202K/sec


It took a while, but eventually I had a mirrored two-disk raid set!

Moving to Ubuntu 10.04 (Lucid) on my Macbook 2,1

I'm the happy owner/user of a Macbook (a 2,1 to be exact) since a few years. Actually it is company property, but since I'm the only user they let me run whatever I like. Which happens to be Ubuntu.

It was really Willem who set me on to this; I don't think I would have had the guts to do this to my Mac if he hadn't been adamant that it would work. And it does! It did require quite a few tweaks, though, as the Ubuntu community page testifies.

But overall I was happily running 9.10 (Karmic); I learned to live with the few quirks, like having to log off every time I came into work and attached the external monitor (it would crash the video driver if I didn't). And suspend/resume was just really slow.

Now that Lucid is just around the corner, and after my previous success with installing it on my wife's new desktop, I got feisty and tried it on the Mac. First, I used the live CD and that worked so splendidly that I just had to do the actual upgrade. It lasted all night, but in the end it worked like a charm.

So it's worth at this point to make a few notes.

• The new theme did not appear the first time I booted. This was to be expected because I did an upgrade, not a fresh install. Most things should preferably stay as they were.
• The ssh-agent environment variables were gone from my terminal shells. Why? I don't know, but a somewhat related bug report suggests the use of the keychain package.
• Thunderbird 3.0 is chugging away on indexing all my mail. I could turn it off but I think I'll let it run for a while and check out the new search capability.
• Sound didn't immediately work on the live CD but this was resolved by killing the pulseaudio daemon. Sound does work after the upgrade.
• The volume and brightness buttons work too. Very nice.
• Attaching the external monitor turns off the desktop effects. This may be related to the crash bug I mentioned earlier. But you can turn it back on right after.
  

That's it for now, more news later!

My new inspiron 560

(Sorry, this is not really about software but still something I'd like to share)

I decided to go all-in and finally buy myself (and my wife) a new desktop machine. It replaces a machine that I got for a token fee, an old box that my company was going to recycle. At the time I thought it would be good enough for running Linux, even though it was slow, didn't have a big disk and little memory; Linux is frugal, right? Well, the Linux desktop has grown up considerably in the last decade and the combined memory footprint of Evolution, Gnome and Firefox combined rivals that of mainstream systems.

So the new machine which now proudly hums away on my desk (more on the humming later) is an all-black, all-shiny, DELL Inspiron 560. The cheapest DELL has to offer through their website, for just over 400 euros including delivery to my home. Even though it is cheap, it is a step up from what I had: it has a dual-core 2.7 GHz CPU, 4 GB memory, 320 GB disk. Funny detail: black was the cheapest color. Reminds me of the Ford model T days.

My wife is the principal user of the desktop, since I do most of my work on a laptop that I carry everywhere. To save myself from a support nightmare, I switched to Ubuntu LTS releases some time ago. Just mid 2009, we made the switch to Ubuntu 8.04. So Ubuntu 8.04 was the first and most logical choice for the new box; the smoothest possible transition I could imagine.

After popping in a fresh Ubuntu 8.04.4 CD and starting up, most things looked alright. Except there was no network. This puzzled me somewhat as a device clearly showed up in the output of ifconfig; there were just no packets coming through at all. A quick search for the precise hardware spec revealed a known issue with the driver, and the workaround to download and use the Realtek provided (open-source) driver. I was worried that this problem would just keep coming back with every kernel update, but it fixed the immediate issue.

Meanwhile, my attention was drawn to another, quite severe problem. The machine was making quite a bit more noise than I expected, to the point of being irritating. There was clearly a fan spinning loud and hard in there. My first suspicion went to the case fan near the rear of the box, but this turned out to be wrong. To make matters worse, the loud fan had a bearing problem and started to make horrible rumbling sounds.

Since I had already done away with Windows completely I had no chance to verify that this was caused by some software defect. It started as soon as the computer was turned on, so in any case it wasn't due to something Linux did. It was such a depressing conclusion that I bought a lemon. Just the thought of having to spend time and energy getting this fixed (imagine explaining a support person over the phone that you run Linux, not Windows...) caused agony.

Luckily DELL shipped a bootable diagnostics CD with the computer, and it allows you to run several tests to verify the correct workings of the machine. Two tests were especially interesting: a CPU fan test and a case fan test. Both tests drive up the fans to a high RPM, and then down again. I should explain that while I was running from the diagnostics disc, the terrible noise persisted.

The CPU fan test revealed that the CPU does indeed have a fan (or maybe more than one) and that it can be heard, but only at high RPM. At low settings (normally, if the CPU is not under load) it can hardly be made out (1700 RPM or so). The case fan was even lower; it's a big one so it runs only at 500 RPM nominally, but can be stepped up to 1500 or so if the case runs hot. Bot tests produced different sounds, and the noisy fan stayed noisy. It could only mean one thing: the video card fan. The card is an Nvidia GeForce 310.

The Ubuntu 8.04 system worked, but not very smoothly. The network driver was a kludge, I couldn't get anything but the VESA driver working for video (or I didn't try hard enough) and the noise made the whole thing just unworkable. I decided that maybe, just maybe, the upcoming LTS release, Ubuntu 10.4, would be a better option. This proved to be sheer lucidity.

Ubuntu 10.4 is not even out, but you can get beta 2 already and this is actually encouraged as the experience of more people trying the system at this stage will help iron out the remaining wrinkles. From the moment I popped in the CD and started, I was amazed by the ground they've covered since the last release (I'm running 9.10 on my laptop so I'm close to cutting-edge there). The installation was smooth, very few questions asked and in no time at all I had a new OS running plus network. (Still, noise.) I logged on, clicked around appreciatively, and then I selected 'maximum visual effects'. This triggered the system to prompt me whether I wanted to install the proprietary Nvidia drivers (of course, you silly!). After the installation there was some hiccup about not being able to switch or load drivers (somekind of fb kernel driver got in the way? I couldn't tell), but a reboot set things straight. And how! As soon as the Nvidia driver loaded, the computer became silent (relatively speaking). And this makes all the difference. First I had regretted my choice for this particular desktop machine, and now I find it very good value for money. I should probably still chase up DELL about the resonance in the video card fan, but it no longer prevents me from enjoying the new computer. It's funny that during startup and shutdown the noise can still be heard, that is before loading and after unloading the Nvidia driver.

And Ubuntu 10.4 is just fine, even at beta 2. I was so confident that I replaced my old desktop with the new one just the day I had to leave for four days to visit the last EGEE User Forum, after I rsync'd all user data and tested that my wife could still read her e-mail.

Chicken and egg: install rpm using rpm

What do you do when a collegue has deleted the rpm and yum packages from a CentOS system (by mocking around with the sqlite package)? Reinstall them of course. Hmm but how, when rpm is absent?

Setting up a local rpm installation

The solution is to copy rpm from another computer with (approximately) the same operating system. The following files are required (substitute lib64 for lib when you're on a 32-bit system), put them in a temporary directory on the target system:

• /bin/rpm --> bin/
• /usr/lib64/librpm*.so --> lib/
• /usr/lib64/libsqlite*.so --> lib/
• /usr/lib/rpm/macros --> lib/rpm

Some configuration files are expected to be present, though, and rpm needs to be told to look for them in the correct location. This is done with a little wrapper script (named rpm.sh) like this:

#!/bin/sh
export LD_LIBRARY_PATH=`dirname $0`/lib
mv ~/.rpmmacros ~/.rpmmacros.orig
cp `dirname $0`/lib/rpm/macros ~/.rpmmacros
`dirname $0`/bin/rpm --rcfile `dirname $0`/lib/rpmrc --define "_rpmlock_path /var/lock/rpm" "$@"
[ -e ~/.rpmmacros.orig ] && mv ~/.rpmmacros.orig ~/.rpmmacros

You can then use the temporary rpm installation by going to the temporary directory and running ./rpm.sh. It will still work on the system's package database.

Installing rpm's RPMs

This is easy now. First download rpm and required packages from a CentOS mirror. You need the packages for rpm, rpm-libs and sqlite (make sure you choose the right platform, i386 or x86_64). Then do a ./rpm.sh -i *.rpm so all these packages are installed at once. Now you can run the system's rpm again, phew!

Installing Yum

You may still need to get yum back. This is done similarly, by downloading the packages yum, yum-fastestmirror, yum-metadata-parser, rpm-python and python-sqlite. Then do a ./rpm.sh -i *.rpm for these and you've can install packages easily again.

User SSH configuration for virtualised remote hosts

As part of the agile testbed here at Nikhef's grid group, there are a couple of Xen hosts. Thanks to Dennis' mktestbed scripts it's quite easy to manage the guests' lifecycles. The guests are on their own network and one can use ssh forwarding to access them from the desktop. So you would login to the Xen host, and from there login to the guests.
But sometimes it is convenient to use graphical tools available on the desktop to do something on the remote guest machine. This would be possible with a direct SSH connection. The straightforward solution would be to use SSH port forwarding.
There is a more convenient way to get the remote guests appear as ordinary hosts from the desktop via ssh (without resorting to a VPN or so): using the ssh configuration file located in ~/.ssh/ssh_config:

Host coolhost.testdomain coolhost
 Hostname coolhost.testdomain
 Protocol 2
 User root
 # avoid often changing host fingerprint prompt
 CheckHostIP no
 StrictHostKeyChecking no
 UserKnownHostsFile /dev/null
 ForwardX11 yes
 # route through the Xen host
 ProxyCommand ssh -q -A xenhost.domain nc %h %p 2>/dev/null

The key line here is the last one: this opens an ssh connection to the Xen host, and uses netcat to open a connection to the guest's ssh socket.
The configuration above also removes the hostkey check present in SSH. Usually one would really want this, but as I'm generating and destroying machines all the time and the connection to the xenhost is verified already, it doesn't really bring much additional security.

• Using ssh ProxyCommand to tunnel through two hosts
  
• How to let SSH ignore host key
  

Involuntary Grid Computing

It dawned on me how much alike botnets and grid computing really are.

• Botnet operators control hundreds of thousands heterogeneous machines.
• These machines have unpredictable uptimes. On this scale, there is always a sufficiently large pool of available machines.
• The diversity of systems requires a middleware layer that is portable and 'works everywhere'.
• The load on the machines should be kept 'under the radar' of the owner, otherwise the owner will become suspicious and reinstall the box.
• The botnet operators rent out their powers to various applications, such as DDOS attacks, spamming, scamming and hosting dubious content.
• A bot is deployed to a potentially hostile environment. The more advanced nets use PKI techniques, such as SSL and X509 certificates to protect their transfers, and to secure their control channel. This way they prevent being fingerprinted and counter-attacked.

I think botnets have matured enough to be called something else. Hence I coin the term 'involuntary grid computing'.

Ranting, raving madness of rpmbuild

We use rpmbuild to create nice, shining RPMs for a bundle of mostly open-source scientific software called the VL-e Proof-of-Concept. As our experience with building RPMs is growing, we automated a couple of things and have time to spare for more fun stuff.

This rant is about a problem I just discovered, and that I can only file under 'madness'.

First, the prologue. It started on a gray, drizzly morning, ... no wait; that's no good. 'A dark and stormy night.' Yeah, that's it. It started on a dark and stormy night...

One (1) of the files in a certain RPM (let's call it exhibit X) was missing an MD5 checksum. This was odd, because all files in an RPM must have a checksum. With the help of rpm2cpio I could extract the files and lo and behold: the file was there. Even better: the file was in good shape.

This particular RPM is a binary repackaging from a tarball distributed by a bunch of raving lunatics who think it's fun to ship a 1.5GB tarball, mostly consisting of irrelevant, experimental data. And to have a source tree that is impossible to build correctly, because all of the algorithms are somewhat numerically unstable. What else can you expect from brain surgeons?

So the binary tarball is used to create an RPM. And this one file I am talking about is a shared library, nothing strange about it. But rpmbuild tries to be clever...it knows about a phenomenon called prelinking.

Now, a disclaimer goes here: I know nothing about prelinking. Or rather, I wish I didn't. What I do know is that all your system's libraries are 'prelinked', which means that they are changed to behave better, or something. Apparently the change is reversible and system-dependent, because you would typically run the prelink script from a daily cron job.

Here comes the part where rpmbuild tries to be clever: it knows that RPM can verify the installed files by comparing their checksums to their original values in the RPM database. Prelinking defeats such a check, so RPM will try a just-in-time reversal of the change, save it to a temporary file, and compare the checksum of the original. If you don't believe me, look at /etc/rpm/macros.prelink.

The same reversal/md5sum thing is done on all libraries you package with rpmbuild, and on this one file, it went wrong.

prelink: libqt-mt.so.3.3.3: at least one of file's dependencies has changed since prelinking

So it decided to forego the calculation of an md5sum altogether. That's a bug, alright?

The 'solution' I came up with is editing my SPEC file to redefine the 'prelink undo' calculation.

%define __prelink_undo_cmd /bin/cat prelink library

Compare this to the original definition, and read the manual page of prelink to see why it works like this.

LVM bad hair day

Logical Volume Management (LVM) for Linux is nice, but the snapshot feature kills your hard disk. That was my conclusion after messing about with snapshots as a cheap-n-easy chroot solution, a la the Debian schroot package.

I set up two snapshots and started writing a 5 GB file to each. 6 hours later (!) the job was done, in the meantime the box was as good as dead.

This spurred me on to investigate the LVM snapshot performance, and boy is it bad!

Windows versus Mac OS

I just set up a virtual machine with Windows XP on my Apple MacBook. I've been using Linux since the mid-90s (and more recently, Mac OS X), and although I have used Windows at work in the past, I never actually had to go through the chores of installing and managing a Windows box myself.
The installation was a snap, but the trouble began when Windows really booted for the first time. So that is roughly the same point where so many normal consumers are when they take their shiny new computer out of the box, plug it in, and turn it on.
I was immediately warned by a yellow balloon in the lower right corner that my system may be at risk. The system insisted that I should turn on automatic updates, and install some anti-virus software. A colleague pointed me to avg, which was helpful, but before I settled down to do what I meant to do when I decided I needed windows (visiting a web site with Internet Explorer), I spent several hours boiler plating my Windows virtual machine, all the while feeling scared, like prey being hunted, as if the predatory dangers of malware and viruses could infect me any time until I secured everything completely.

So this led to the following epiphany:

The first thing you do when you use a new Mac, is get to work.
The first thing you do when you use Windows, is get to worry.

Grid software maturity

Wow, it's been two months since last post to this blog. Maybe this blogging thing is not for me. Anyway, blame it on the vacations.

Just a short post in the category of random thoughts. I'm working in grid computing, a software field that is trying to take cluster computing to a higher level. It dawned on me that grid software is rapidly evolving and that it's hard to tell end users what they should use now, because that may be the wrong advice in six months.

Then I figured that the evolution process was really a process of maturing. So in a sense, grid software is in puberty. It makes complete sense:

• it is going through rapid, sudden changes;
• some things are clearly getting better, but
• some other things are (temporarily) getting much worse
• it is getting bigger, growing more features, and
• it's getting hairy (sorry, couldn't resist that last one)

Douse richly with a sause of anxiety, unease, insecurity and rebelliousness and you'll get the typical adolescent, misunderstood, obnoxious little monsters that our beautiful little children have to turn into before they suddenly become sensible and respectable grown-ups.

So it's a phase that grid software has to pass through if it is going to come of age. But like with children in puberty, you feel somewhat embarrassed to introduce them to your friends and relatives.

To err is human, to really foul up you need a computer

Humans are the weakest link in any automated system, whether it is the humans at the end of the chain, sometimes derogatorily referred to as users, or the humans at the other end who did the best they could to design and build the system. Incidentally, that's why I believe the future envisioned in the Matrix is a reality; when the machines take over they will take humans out of the equation. Only they won't keep us around as organic batteries: we're not even good for that (bio fuel, maybe). But let's get back on topic.
In modern (semi-)automated systems, it's funny to see how little direct inter-human communication remains. Many of the emails and paper letters I receive are computer-generated, and most of my daily chores require little or no human interaction. To my amusement (or should I be worried?) many of the humans that are still needed around have just shut down their brains to let the thinking to the machines. The numbed look on the faces of the girls operating the supermarket checkout counters shows they know it: they are sort of lingering on the temporary state of being affordable robots, moving groceries past a bar code reader.

In a way, the computer has become a very complex, distorting communication channel between the humans at the opposing ends of the chain. Any sizable software project is like a modern tower of Babel; the software architects can't understand the network engineers, the database designer doesn't have a clue about the user's needs and the programmers are left in the dark with a cryptic technical design document. The resulting system doesn't do what the user expected, covers the wrong 90% of the technical requirements and oversteps both the deadline and the budget. The only one who thinks it's good is god.

The software industry has often been compared to other industries, often to its demerit. But it's not fair; software is extremely complex. It just doesn't stand a comparison to a car, or a coat hanger. A large program has many thousands of lines of code, and by simple statistics a very good programmer will make one mistake in a thousand lines. How far reaching a bug is can't easily be predicted.

Bug tracking systems are a great help (even though they are also semi-automated systems that humans communicate through). We are inclined to repair our faults as quickly as we may, in the hope of ever achieving the ultimate goal of having a bug-free program. But what I'm interested in has to do with software forensics. More of a biologists approach to bugs: instead of putting your heel on it to squash it, inspect it up closely to see where it came from, what it's habitat is, what it feeds on. A software bug is some form of programmer error; what was the cause? Programmer laziness? Fatigue? A misunderstanding of the requirements? An oversight? A subconscious TO DO item that was later forgotten? Hastiness? I have no hope of reducing the number of bugs in software until we start to dig in to these questions and find the key factors that influence the number of programming mistakes.

So let henceforth every bug tracker be outfitted with one more field: the blame field, indicating what was to blame for this bug. Otherwise, I have no hope.

Success: eToken PRO on Debian

Finally, after several weeks of kicking this stubborn bucket of bits, I've managed to produce a working .deb for the Aladdin eToken PRO (TM). The eToken looks like a small USB stick, but it is a smartcard in disguise. They're about $30 and we've purchased a hundred or so for putting grid certificates on.

Many thanks to Jan Just Keijser, who pioneered the effort of getting these (poorly supported) tokens to work under Linux. He built the software as an RPM, which works under Red Hat EL 4 (and derivatives), Fedora Core 6 and Suse 10. Unfortunately, we can't distribute the RPMs due to licensing constraints (the Aladdin drivers are proprietary, bleh).

Building debs has been an interesting exercise in finding the subtle differences between RHEL4 and Ubuntu; following the Debian Policy (as well as I could); building binary debs on the fly and doing it the proper way (with sources); setting up an apt repository for said debs and, most of all, stumbling over the impossibilities with autotools and libtool (see yesterdays post).

As soon as I find a proper place to put op my debs I'll post the location. Contrary to the rpms, the deb is freely redistributable as it contains only the free (libre) software. Provided that you find the Aladdin drivers elsewhere, you may get your eToken to work on your Ubuntu laptop!

And oh, by the way: this has been tested on Ubuntu 6.06 and 7.04, and Debian 4 ("Etch").

On building systems of interdependent components

These ideas come forth from my difficulties with libtool. After having spend some 48 hours pontificating how to deal with the following situation.

We have to build a software system consisting of several interdependent components (no circular dependencies, that would just be weird). The idea is to build and package the components in some form for distribution and installation, such as rpms or debs. So there are at least two different machines to consider: the build system where the software is compiled, and the runtime system where the software is installed and run. In chronological order, here's what happens to the software:
(build time)

1. compilation of source code into object code
2. linking together of object code and dynamic shared libraries into runnable programs and libraries
   

(run time)

3. linking of programs with dynamic shared libraries
4. loading of the program into memory and executing.
   

The problem is in the dynamic shared libraries (.so files), which play a role both in steps 2 (build time) and 3 (run time). Somehow, this dual nature of these objects make them confusing.
Imagine you have three components, A, B and C, where B depends on A and C depends on B (and therefore indirectly on A). First, you compile and link A. That's the easy part. Then you want to build component B, but a problem starts to present itself. Component B depends on A, which is compiled but not yet installed. The GNU build system (autoconf, automake, make and libtool) is not attuned to the case where the shared objects of step 2 (needed at build time) are in another location than the shared objects of step 3 (needed at run time). Usually, if you needed a certain package, you would first install it and then build against it.
So what is against installing component A prior to building B? It turns out there are two fundamental reasons.
The first reason is that installing a component requires root privileges. Remember, we are building to package software as rpms, not ad hoc. The final destination of A is in /usr, or /opt/someplace, and a typical user has no rights there.
The other reason is that there may be a huge difference between the system you build on and the system you build for. It just makes no sense to install a component for the wrong system. And I'm not even talking about cross-compiling or anything, just imaginge that A is the C library and B is the compiler. It would be a really bad idea to replace the C library with a new, experimental version just so you could build the new compiler against it and at the same time using that new, experimental C library to do the actual build, now, wouldn't it?
But even in simpler cases the horror of having to install and test and use multiple versions of a component on your build system should be enough to want to hide in a bomb shelter armoured with regression test suites and stocked with a lifetime supply of valium.

To get back to the original discussion: where do we put A? The answer is simple, but has complications. We have to put A in a non-privileged location which will be staging for the actual run-time location. The packaging tools for rpms and debs know how to deal with that, so that liba.so ends up in

/opt/happyspot/lib/liba.so (runtime)

when it is staged in

/tmp/staging/opt/happyspot/lib/liba.so (build time).

So now we can tell B where to find A, and after building B it ends up in (or on) the stage as well.
The complications only just got started. B knows that it should look for A in staging at build-time, but once both are packaged and installed, that location is no longer valid. So we have to build B so that it can find A in the run-time location. But when we want to build C and point to B, the run-time location is not yet valid. There we have it. The build-time nature of B is inherently different from the run-time nature of B.

Do we really need two different versions of B? I would think not. There is no real difference other than the resolving of dynamic linking, which happens at build-time and run-time. But libtool is a pain: it insists on inserting the path of the staging area into the libraries and binaries in so-called RPATH entries, and the same path ends up in the libtool libraries (.la files). Although both the staging and the real path end up in RPATH and .la file, it is no good: the staging path has no place in the run-time files. It could lead to really ugly situations: someone could insert a trojan horse by recreating the staging location and installing a malicious library of the same name as the right one. Another real-life example is having the staging in /home/builduser, while the run-time system has /home automounted. Everytime a library was loaded, the NFS server would be hit with the request to serve up a non-existent directory.

While I don't have the ultimate solution for this problem, I would like to point out one more thing: having an RPATH in your executable sounds cool, but it has its drawbacks. After all, the system administrator may not like the location you picked to put the software. Relocating the software defeats the purpose and effect of your RPATH (and the same issues arise as were mentioned above).