Yenya's World

Wed, 15 Nov 2006

Hunting Ghosts

Today I worked on synchronizing filesystems on some of our high-availability systems. We use custom-made rsync-based setup for checking for differences between filesystems in a cluster. One of the hosts in a H-A pair has been down for a while because of a faulty hardware, so I had to manually check whether the changes on the active system can be propagated to the backup as well. I have synchronized the filesystems, and switched the load to the newly plugged-in host (because it is faster than the other one). Just to be sure, I re-ran the checks again, and was surprised: some files have been different on the new host now.

What was worse, the set of files which were different was a bit suspicious: bash, login, tcpdump, some other utils and libraries, including those which are run every time system boots (such as heartbeat and its libraries). I ran "rpm -V", just to be sure the files are different than in the RPM database, but it displayed that all files are OK and well matching the database. I took the clean RPMs from the FTP file repository, and the files in question were shorter in the package than on my filesystem. I thought: are current rootkits so smart that they modify the RPM database, and so stupid that "ls -l" still can tell the difference?

"rpm -qlv bash|grep /bin/bash" displayed that there was a different size in the RPM database than in the file itself, yet "rpm -V bash" said the package was perfectly OK. Strange. So I suspected the rpm program has been modified as well (even though it did not show up in the list of modified files). To prove this, I used strace. On a clean system its output was shorter, and the difference was that on a modified system rpm spawned some more threads/processes. "strace -f" then showed the quilty party - the rpm command executed prelink on each modified binary.

So I have been hunting ghosts all the time: the files in question have only not been prelinked yet, or the prelinking info has been overwritten (or not overwritten, I don't know) by my synchronization scripts. After running "/etc/cron.daily/prelink" on a "modified" system both filesystems look the same. Problem solved.

For a long time I wondered how prelinking can be done without modifying the binary (and thus breaking the packaging system). The answer for rpm appears to be: the package manager needs to know about prelinking as well. I have to find some time to read Jakub's prelink paper (PDF). Back to a serious work now.

Section: /computers (RSS feed) | Permanent link | 3 writebacks

3 replies for this story:

Vasek Stodulka wrote:

Yes, prelink. It shoud be written at the top of rpm man page (maybe also bold and red) that prelink modifies binaries and RPM knows about it. Most people find differences and then hunt ghosts - just like you. I only wonder how a guru like you do not know this. :-)

Yenya wrote: Too much knowledge

I think hunting ghosts is not bad per se, provided that I find the right answer after all. It is probably that I know about many things that can go wrong, so it takes time to find the right one. Last week one of my students asked me to find out whether he cannot login via KDM any more after installing some completely unrelated package. I have traced the X startup scripts and so on, and it took me at least quarter an hour before I ran "df /" and discovered that his root filesystem is full...

Peter Kruty wrote: knowledge

Right, there is so much knowledge about the live Linux system, that sometimes tooks time to find out right source of the problems.

Reply to this story:

URL/Email: [http://... or mailto:you@wherever] (optional)
Title: (optional)
Key image: key image (valid for an hour only)
Key value: (to verify you are not a bot)


Yenya's World: Linux and beyond - Yenya's blog.


RSS feed

Jan "Yenya" Kasprzak

The main page of this blog



Blog roll:

alphabetically :-)