[WBEL-users] ROOTKIT checking software

[Kirby C. Bohling]

On Thu, Nov 11, 2004 at 07:15:34PM -0000, Denis Croombs wrote:
> I have found some of my customers systems have had the root password changed
> remotely by an ex-employee, I know I can go do a rescue CD boot and change
> the root password to something I know BUT I have a few questions.
> 

	Take the machine offline, duplicate the disk, run an MD5SUM of
the original write it down for documentation purposes.  Put the new
drive back in, do your forensics off the new duplicate copy.  If you
want to start over on the forensics just copy the original disk
again.  Don't even mount the drive, just use "dd" to copy the
individual partitions if need be.

	In the US, I believe that's the policy of the FBI and police to
avoid data integrity accusations in court.  The defendant in the
case claims he's being framed, they can't prove that someone didn't
setup a harddrive to look just like he did it.  So they keep an
absolutely original copy w/ md5sum from the moment they touched the
drive.  Thus the forensics can be duplicated both independent
experts.

	Now because the drive has been in your hands, it might be
unnecessary for the md5sum.  However, I'd want an original copy I
could always duplicate to start over with to double check that I
hadn't modified something.  I'd always mount the disk read only so I
didn't accidently change a filesystem timestamp or contents by
accident.

> 1) What files record when root last loged in and from what IP address ?

Normally the command "last", will tell you the last sets of login's.
Last parses a file named utmp or wtmp.  I forget it's location, but
it's not a human readable file.  Just use "last".

If they logged in via ssh, I believe that /var/log/messages has the
timestamps (not sure about IP's).

$ grep "sshd" /var/log/messages

It appears to give me the reverse lookup, and fall back to the IP,
the reverse name could be faked, but that's a lot of work for the
attacker to go to.  (Generally storing IP's is easier, faster, and
more reliable.  I wish more people did that, or at least stored both
in logs).

If they didn't log into the machine as root, they probably su'ed,
for that do this:

$ grep "su" /var/log/messages

If they logged into the console this should bring up the appropriate
lines:

$ grep "login" /var/log/messages 

You'll probably have to look closer at it.  The archive logs might
have rolled, so you might have to look at files with the same
prefix.  The person who did this is a fool if they didn't wipe the
logs however.

> 2) What file record any other activity by the same person ?

If it is anywhere, it'll probably be in this directory:

/var/log

It might be in the .bash_history of the various users you believe
they are logged in as (save this file as soon as possible).  It
generally only holds 1000 commands, so if you do too much logged in
as root, it'll erase what history you have.


> 3) What log files should be kept for handing over to the police ?
> (we have informed them and they are sending someone tomorrow)

Keep the original disk if at all possible.  If they are serious
about forensics, they'll immediatly duplicate it as described above.

I'm doubtful they'll take you terribly seriously unless you can
demonstrate serious financial damage.  However, it never hurts to
assume they will take it seriously.

> 
> 4) What else should I be doing ?
> 
> All help would be very helpful 
> (I think I know the answer to most of the above must I MUST do a sanity
> check ! ) I just do not want to miss anything.
> 

I'd take the machine offline and try the rootkits the other people
have mentioned.  I'd examine everything:

/var/spool/cron/
/etc/cron.*
/etc/cron

That's where crontabs are stored.

Those are the places people stick files to do nasty things in a
week.  I forget where "at" stores it's commands.  You probably want
to check that too.

I've had one machine broken into, we setup a new machine to replace
the existing one.  We burned the old one to the ground once we
figured out what they had done.  Made sure we had patched for the
attack they used and got current with all the other patches we could
for security purposes.

I can understand your reluctance if it's a customer machine, or if
you have to do this for 20 machines in a short period.

There's a ton of things to do, I really don't want to ramble on,
it's not an area I have great deal of knowledge or expertise on.
I've read about this before, but never had to deal with it
personally much.  You really should find a forensics expertise list.

I've really just listed the things I've done to track down what
happened to a machine when I can't get in touch with the admin.
Looking in /var/log/messages, and examining .bash_history of various
users is an easy way of seeing what someone has been doing on your
machine.  However, a compentent malicious user won't leave those
behind.

	Thanks,
		Kirby