Involuntary Grid Computing

It dawned on me how much alike botnets and grid computing really are.

• Botnet operators control hundreds of thousands heterogeneous machines.
• These machines have unpredictable uptimes. On this scale, there is always a sufficiently large pool of available machines.
• The diversity of systems requires a middleware layer that is portable and 'works everywhere'.
• The load on the machines should be kept 'under the radar' of the owner, otherwise the owner will become suspicious and reinstall the box.
• The botnet operators rent out their powers to various applications, such as DDOS attacks, spamming, scamming and hosting dubious content.
• A bot is deployed to a potentially hostile environment. The more advanced nets use PKI techniques, such as SSL and X509 certificates to protect their transfers, and to secure their control channel. This way they prevent being fingerprinted and counter-attacked.

I think botnets have matured enough to be called something else. Hence I coin the term 'involuntary grid computing'.

Ranting, raving madness of rpmbuild

We use rpmbuild to create nice, shining RPMs for a bundle of mostly open-source scientific software called the VL-e Proof-of-Concept. As our experience with building RPMs is growing, we automated a couple of things and have time to spare for more fun stuff.

This rant is about a problem I just discovered, and that I can only file under 'madness'.

First, the prologue. It started on a gray, drizzly morning, ... no wait; that's no good. 'A dark and stormy night.' Yeah, that's it. It started on a dark and stormy night...

One (1) of the files in a certain RPM (let's call it exhibit X) was missing an MD5 checksum. This was odd, because all files in an RPM must have a checksum. With the help of rpm2cpio I could extract the files and lo and behold: the file was there. Even better: the file was in good shape.

This particular RPM is a binary repackaging from a tarball distributed by a bunch of raving lunatics who think it's fun to ship a 1.5GB tarball, mostly consisting of irrelevant, experimental data. And to have a source tree that is impossible to build correctly, because all of the algorithms are somewhat numerically unstable. What else can you expect from brain surgeons?

So the binary tarball is used to create an RPM. And this one file I am talking about is a shared library, nothing strange about it. But rpmbuild tries to be clever...it knows about a phenomenon called prelinking.

Now, a disclaimer goes here: I know nothing about prelinking. Or rather, I wish I didn't. What I do know is that all your system's libraries are 'prelinked', which means that they are changed to behave better, or something. Apparently the change is reversible and system-dependent, because you would typically run the prelink script from a daily cron job.

Here comes the part where rpmbuild tries to be clever: it knows that RPM can verify the installed files by comparing their checksums to their original values in the RPM database. Prelinking defeats such a check, so RPM will try a just-in-time reversal of the change, save it to a temporary file, and compare the checksum of the original. If you don't believe me, look at /etc/rpm/macros.prelink.

The same reversal/md5sum thing is done on all libraries you package with rpmbuild, and on this one file, it went wrong.

prelink: libqt-mt.so.3.3.3: at least one of file's dependencies has changed since prelinking

So it decided to forego the calculation of an md5sum altogether. That's a bug, alright?

The 'solution' I came up with is editing my SPEC file to redefine the 'prelink undo' calculation.

%define __prelink_undo_cmd /bin/cat prelink library

Compare this to the original definition, and read the manual page of prelink to see why it works like this.

LVM bad hair day

Logical Volume Management (LVM) for Linux is nice, but the snapshot feature kills your hard disk. That was my conclusion after messing about with snapshots as a cheap-n-easy chroot solution, a la the Debian schroot package.

I set up two snapshots and started writing a 5 GB file to each. 6 hours later (!) the job was done, in the meantime the box was as good as dead.

This spurred me on to investigate the LVM snapshot performance, and boy is it bad!

Windows versus Mac OS

I just set up a virtual machine with Windows XP on my Apple MacBook. I've been using Linux since the mid-90s (and more recently, Mac OS X), and although I have used Windows at work in the past, I never actually had to go through the chores of installing and managing a Windows box myself.
The installation was a snap, but the trouble began when Windows really booted for the first time. So that is roughly the same point where so many normal consumers are when they take their shiny new computer out of the box, plug it in, and turn it on.
I was immediately warned by a yellow balloon in the lower right corner that my system may be at risk. The system insisted that I should turn on automatic updates, and install some anti-virus software. A colleague pointed me to avg, which was helpful, but before I settled down to do what I meant to do when I decided I needed windows (visiting a web site with Internet Explorer), I spent several hours boiler plating my Windows virtual machine, all the while feeling scared, like prey being hunted, as if the predatory dangers of malware and viruses could infect me any time until I secured everything completely.

So this led to the following epiphany:

The first thing you do when you use a new Mac, is get to work.
The first thing you do when you use Windows, is get to worry.

Grid software maturity

Wow, it's been two months since last post to this blog. Maybe this blogging thing is not for me. Anyway, blame it on the vacations.

Just a short post in the category of random thoughts. I'm working in grid computing, a software field that is trying to take cluster computing to a higher level. It dawned on me that grid software is rapidly evolving and that it's hard to tell end users what they should use now, because that may be the wrong advice in six months.

Then I figured that the evolution process was really a process of maturing. So in a sense, grid software is in puberty. It makes complete sense:

• it is going through rapid, sudden changes;
• some things are clearly getting better, but
• some other things are (temporarily) getting much worse
• it is getting bigger, growing more features, and
• it's getting hairy (sorry, couldn't resist that last one)

Douse richly with a sause of anxiety, unease, insecurity and rebelliousness and you'll get the typical adolescent, misunderstood, obnoxious little monsters that our beautiful little children have to turn into before they suddenly become sensible and respectable grown-ups.

So it's a phase that grid software has to pass through if it is going to come of age. But like with children in puberty, you feel somewhat embarrassed to introduce them to your friends and relatives.

To err is human, to really foul up you need a computer

Humans are the weakest link in any automated system, whether it is the humans at the end of the chain, sometimes derogatorily referred to as users, or the humans at the other end who did the best they could to design and build the system. Incidentally, that's why I believe the future envisioned in the Matrix is a reality; when the machines take over they will take humans out of the equation. Only they won't keep us around as organic batteries: we're not even good for that (bio fuel, maybe). But let's get back on topic.
In modern (semi-)automated systems, it's funny to see how little direct inter-human communication remains. Many of the emails and paper letters I receive are computer-generated, and most of my daily chores require little or no human interaction. To my amusement (or should I be worried?) many of the humans that are still needed around have just shut down their brains to let the thinking to the machines. The numbed look on the faces of the girls operating the supermarket checkout counters shows they know it: they are sort of lingering on the temporary state of being affordable robots, moving groceries past a bar code reader.

In a way, the computer has become a very complex, distorting communication channel between the humans at the opposing ends of the chain. Any sizable software project is like a modern tower of Babel; the software architects can't understand the network engineers, the database designer doesn't have a clue about the user's needs and the programmers are left in the dark with a cryptic technical design document. The resulting system doesn't do what the user expected, covers the wrong 90% of the technical requirements and oversteps both the deadline and the budget. The only one who thinks it's good is god.

The software industry has often been compared to other industries, often to its demerit. But it's not fair; software is extremely complex. It just doesn't stand a comparison to a car, or a coat hanger. A large program has many thousands of lines of code, and by simple statistics a very good programmer will make one mistake in a thousand lines. How far reaching a bug is can't easily be predicted.

Bug tracking systems are a great help (even though they are also semi-automated systems that humans communicate through). We are inclined to repair our faults as quickly as we may, in the hope of ever achieving the ultimate goal of having a bug-free program. But what I'm interested in has to do with software forensics. More of a biologists approach to bugs: instead of putting your heel on it to squash it, inspect it up closely to see where it came from, what it's habitat is, what it feeds on. A software bug is some form of programmer error; what was the cause? Programmer laziness? Fatigue? A misunderstanding of the requirements? An oversight? A subconscious TO DO item that was later forgotten? Hastiness? I have no hope of reducing the number of bugs in software until we start to dig in to these questions and find the key factors that influence the number of programming mistakes.

So let henceforth every bug tracker be outfitted with one more field: the blame field, indicating what was to blame for this bug. Otherwise, I have no hope.

Success: eToken PRO on Debian

Finally, after several weeks of kicking this stubborn bucket of bits, I've managed to produce a working .deb for the Aladdin eToken PRO (TM). The eToken looks like a small USB stick, but it is a smartcard in disguise. They're about $30 and we've purchased a hundred or so for putting grid certificates on.

Many thanks to Jan Just Keijser, who pioneered the effort of getting these (poorly supported) tokens to work under Linux. He built the software as an RPM, which works under Red Hat EL 4 (and derivatives), Fedora Core 6 and Suse 10. Unfortunately, we can't distribute the RPMs due to licensing constraints (the Aladdin drivers are proprietary, bleh).

Building debs has been an interesting exercise in finding the subtle differences between RHEL4 and Ubuntu; following the Debian Policy (as well as I could); building binary debs on the fly and doing it the proper way (with sources); setting up an apt repository for said debs and, most of all, stumbling over the impossibilities with autotools and libtool (see yesterdays post).

As soon as I find a proper place to put op my debs I'll post the location. Contrary to the rpms, the deb is freely redistributable as it contains only the free (libre) software. Provided that you find the Aladdin drivers elsewhere, you may get your eToken to work on your Ubuntu laptop!

And oh, by the way: this has been tested on Ubuntu 6.06 and 7.04, and Debian 4 ("Etch").