Yenya's World

Wed, 15 Nov 2006

Hunting Ghosts

Today I worked on synchronizing filesystems on some of our high-availability systems. We use custom-made rsync-based setup for checking for differences between filesystems in a cluster. One of the hosts in a H-A pair has been down for a while because of a faulty hardware, so I had to manually check whether the changes on the active system can be propagated to the backup as well. I have synchronized the filesystems, and switched the load to the newly plugged-in host (because it is faster than the other one). Just to be sure, I re-ran the checks again, and was surprised: some files have been different on the new host now.

What was worse, the set of files which were different was a bit suspicious: bash, login, tcpdump, some other utils and libraries, including those which are run every time system boots (such as heartbeat and its libraries). I ran "rpm -V", just to be sure the files are different than in the RPM database, but it displayed that all files are OK and well matching the database. I took the clean RPMs from the FTP file repository, and the files in question were shorter in the package than on my filesystem. I thought: are current rootkits so smart that they modify the RPM database, and so stupid that "ls -l" still can tell the difference?

"rpm -qlv bash|grep /bin/bash" displayed that there was a different size in the RPM database than in the file itself, yet "rpm -V bash" said the package was perfectly OK. Strange. So I suspected the rpm program has been modified as well (even though it did not show up in the list of modified files). To prove this, I used strace. On a clean system its output was shorter, and the difference was that on a modified system rpm spawned some more threads/processes. "strace -f" then showed the quilty party - the rpm command executed prelink on each modified binary.

So I have been hunting ghosts all the time: the files in question have only not been prelinked yet, or the prelinking info has been overwritten (or not overwritten, I don't know) by my synchronization scripts. After running "/etc/cron.daily/prelink" on a "modified" system both filesystems look the same. Problem solved.

For a long time I wondered how prelinking can be done without modifying the binary (and thus breaking the packaging system). The answer for rpm appears to be: the package manager needs to know about prelinking as well. I have to find some time to read Jakub's prelink paper (PDF). Back to a serious work now.

Section: /computers (RSS feed) | Permanent link | 3 writebacks


Yenya's World: Linux and beyond - Yenya's blog.


RSS feed

Jan "Yenya" Kasprzak

The main page of this blog



Blog roll:

alphabetically :-)