Chicken and egg: install rpm using rpm

What do you do when a collegue has deleted the rpm and yum packages from a CentOS system (by mocking around with the sqlite package)? Reinstall them of course. Hmm but how, when rpm is absent?

Setting up a local rpm installation

The solution is to copy rpm from another computer with (approximately) the same operating system. The following files are required (substitute lib64 for lib when you're on a 32-bit system), put them in a temporary directory on the target system:

• /bin/rpm --> bin/
• /usr/lib64/librpm*.so --> lib/
• /usr/lib64/libsqlite*.so --> lib/
• /usr/lib/rpm/macros --> lib/rpm

Some configuration files are expected to be present, though, and rpm needs to be told to look for them in the correct location. This is done with a little wrapper script (named rpm.sh) like this:

#!/bin/sh
export LD_LIBRARY_PATH=`dirname $0`/lib
mv ~/.rpmmacros ~/.rpmmacros.orig
cp `dirname $0`/lib/rpm/macros ~/.rpmmacros
`dirname $0`/bin/rpm --rcfile `dirname $0`/lib/rpmrc --define "_rpmlock_path /var/lock/rpm" "$@"
[ -e ~/.rpmmacros.orig ] && mv ~/.rpmmacros.orig ~/.rpmmacros

You can then use the temporary rpm installation by going to the temporary directory and running ./rpm.sh. It will still work on the system's package database.

Installing rpm's RPMs

This is easy now. First download rpm and required packages from a CentOS mirror. You need the packages for rpm, rpm-libs and sqlite (make sure you choose the right platform, i386 or x86_64). Then do a ./rpm.sh -i *.rpm so all these packages are installed at once. Now you can run the system's rpm again, phew!

Installing Yum

You may still need to get yum back. This is done similarly, by downloading the packages yum, yum-fastestmirror, yum-metadata-parser, rpm-python and python-sqlite. Then do a ./rpm.sh -i *.rpm for these and you've can install packages easily again.

User SSH configuration for virtualised remote hosts

As part of the agile testbed here at Nikhef's grid group, there are a couple of Xen hosts. Thanks to Dennis' mktestbed scripts it's quite easy to manage the guests' lifecycles. The guests are on their own network and one can use ssh forwarding to access them from the desktop. So you would login to the Xen host, and from there login to the guests.
But sometimes it is convenient to use graphical tools available on the desktop to do something on the remote guest machine. This would be possible with a direct SSH connection. The straightforward solution would be to use SSH port forwarding.
There is a more convenient way to get the remote guests appear as ordinary hosts from the desktop via ssh (without resorting to a VPN or so): using the ssh configuration file located in ~/.ssh/ssh_config:

Host coolhost.testdomain coolhost
 Hostname coolhost.testdomain
 Protocol 2
 User root
 # avoid often changing host fingerprint prompt
 CheckHostIP no
 StrictHostKeyChecking no
 UserKnownHostsFile /dev/null
 ForwardX11 yes
 # route through the Xen host
 ProxyCommand ssh -q -A xenhost.domain nc %h %p 2>/dev/null

The key line here is the last one: this opens an ssh connection to the Xen host, and uses netcat to open a connection to the guest's ssh socket.
The configuration above also removes the hostkey check present in SSH. Usually one would really want this, but as I'm generating and destroying machines all the time and the connection to the xenhost is verified already, it doesn't really bring much additional security.

• Using ssh ProxyCommand to tunnel through two hosts
  
• How to let SSH ignore host key
  

Involuntary Grid Computing

It dawned on me how much alike botnets and grid computing really are.

• Botnet operators control hundreds of thousands heterogeneous machines.
• These machines have unpredictable uptimes. On this scale, there is always a sufficiently large pool of available machines.
• The diversity of systems requires a middleware layer that is portable and 'works everywhere'.
• The load on the machines should be kept 'under the radar' of the owner, otherwise the owner will become suspicious and reinstall the box.
• The botnet operators rent out their powers to various applications, such as DDOS attacks, spamming, scamming and hosting dubious content.
• A bot is deployed to a potentially hostile environment. The more advanced nets use PKI techniques, such as SSL and X509 certificates to protect their transfers, and to secure their control channel. This way they prevent being fingerprinted and counter-attacked.

I think botnets have matured enough to be called something else. Hence I coin the term 'involuntary grid computing'.

Ranting, raving madness of rpmbuild

We use rpmbuild to create nice, shining RPMs for a bundle of mostly open-source scientific software called the VL-e Proof-of-Concept. As our experience with building RPMs is growing, we automated a couple of things and have time to spare for more fun stuff.

This rant is about a problem I just discovered, and that I can only file under 'madness'.

First, the prologue. It started on a gray, drizzly morning, ... no wait; that's no good. 'A dark and stormy night.' Yeah, that's it. It started on a dark and stormy night...

One (1) of the files in a certain RPM (let's call it exhibit X) was missing an MD5 checksum. This was odd, because all files in an RPM must have a checksum. With the help of rpm2cpio I could extract the files and lo and behold: the file was there. Even better: the file was in good shape.

This particular RPM is a binary repackaging from a tarball distributed by a bunch of raving lunatics who think it's fun to ship a 1.5GB tarball, mostly consisting of irrelevant, experimental data. And to have a source tree that is impossible to build correctly, because all of the algorithms are somewhat numerically unstable. What else can you expect from brain surgeons?

So the binary tarball is used to create an RPM. And this one file I am talking about is a shared library, nothing strange about it. But rpmbuild tries to be clever...it knows about a phenomenon called prelinking.

Now, a disclaimer goes here: I know nothing about prelinking. Or rather, I wish I didn't. What I do know is that all your system's libraries are 'prelinked', which means that they are changed to behave better, or something. Apparently the change is reversible and system-dependent, because you would typically run the prelink script from a daily cron job.

Here comes the part where rpmbuild tries to be clever: it knows that RPM can verify the installed files by comparing their checksums to their original values in the RPM database. Prelinking defeats such a check, so RPM will try a just-in-time reversal of the change, save it to a temporary file, and compare the checksum of the original. If you don't believe me, look at /etc/rpm/macros.prelink.

The same reversal/md5sum thing is done on all libraries you package with rpmbuild, and on this one file, it went wrong.

prelink: libqt-mt.so.3.3.3: at least one of file's dependencies has changed since prelinking

So it decided to forego the calculation of an md5sum altogether. That's a bug, alright?

The 'solution' I came up with is editing my SPEC file to redefine the 'prelink undo' calculation.

%define __prelink_undo_cmd /bin/cat prelink library

Compare this to the original definition, and read the manual page of prelink to see why it works like this.

LVM bad hair day

Logical Volume Management (LVM) for Linux is nice, but the snapshot feature kills your hard disk. That was my conclusion after messing about with snapshots as a cheap-n-easy chroot solution, a la the Debian schroot package.

I set up two snapshots and started writing a 5 GB file to each. 6 hours later (!) the job was done, in the meantime the box was as good as dead.

This spurred me on to investigate the LVM snapshot performance, and boy is it bad!

Windows versus Mac OS

I just set up a virtual machine with Windows XP on my Apple MacBook. I've been using Linux since the mid-90s (and more recently, Mac OS X), and although I have used Windows at work in the past, I never actually had to go through the chores of installing and managing a Windows box myself.
The installation was a snap, but the trouble began when Windows really booted for the first time. So that is roughly the same point where so many normal consumers are when they take their shiny new computer out of the box, plug it in, and turn it on.
I was immediately warned by a yellow balloon in the lower right corner that my system may be at risk. The system insisted that I should turn on automatic updates, and install some anti-virus software. A colleague pointed me to avg, which was helpful, but before I settled down to do what I meant to do when I decided I needed windows (visiting a web site with Internet Explorer), I spent several hours boiler plating my Windows virtual machine, all the while feeling scared, like prey being hunted, as if the predatory dangers of malware and viruses could infect me any time until I secured everything completely.

So this led to the following epiphany:

The first thing you do when you use a new Mac, is get to work.
The first thing you do when you use Windows, is get to worry.

Grid software maturity

Wow, it's been two months since last post to this blog. Maybe this blogging thing is not for me. Anyway, blame it on the vacations.

Just a short post in the category of random thoughts. I'm working in grid computing, a software field that is trying to take cluster computing to a higher level. It dawned on me that grid software is rapidly evolving and that it's hard to tell end users what they should use now, because that may be the wrong advice in six months.

Then I figured that the evolution process was really a process of maturing. So in a sense, grid software is in puberty. It makes complete sense:

• it is going through rapid, sudden changes;
• some things are clearly getting better, but
• some other things are (temporarily) getting much worse
• it is getting bigger, growing more features, and
• it's getting hairy (sorry, couldn't resist that last one)

Douse richly with a sause of anxiety, unease, insecurity and rebelliousness and you'll get the typical adolescent, misunderstood, obnoxious little monsters that our beautiful little children have to turn into before they suddenly become sensible and respectable grown-ups.

So it's a phase that grid software has to pass through if it is going to come of age. But like with children in puberty, you feel somewhat embarrassed to introduce them to your friends and relatives.