[WBEL-users] ROOTKIT checking software

[Graham Waring]

G'day all,

Great advice here from Kirby.  My first port of call in situations like this 
is always the wtmp file, just try utmpdump /var/log/wtmp which is quite 
helpful.

Good luck
Graham

>From: "Kirby C. Bohling" <kbohling@birddog.com>
>To: Denis Croombs <denis@just-servers.co.uk>
>CC: whitebox-users@beau.org
>Subject: Re: [WBEL-users] ROOTKIT checking software
>Date: Thu, 11 Nov 2004 14:04:38 -0600
>
>On Thu, Nov 11, 2004 at 07:15:34PM -0000, Denis Croombs wrote:
> > I have found some of my customers systems have had the root password 
>changed
> > remotely by an ex-employee, I know I can go do a rescue CD boot and 
>change
> > the root password to something I know BUT I have a few questions.
> >
>
>	Take the machine offline, duplicate the disk, run an MD5SUM of
>the original write it down for documentation purposes.  Put the new
>drive back in, do your forensics off the new duplicate copy.  If you
>want to start over on the forensics just copy the original disk
>again.  Don't even mount the drive, just use "dd" to copy the
>individual partitions if need be.
>
>	In the US, I believe that's the policy of the FBI and police to
>avoid data integrity accusations in court.  The defendant in the
>case claims he's being framed, they can't prove that someone didn't
>setup a harddrive to look just like he did it.  So they keep an
>absolutely original copy w/ md5sum from the moment they touched the
>drive.  Thus the forensics can be duplicated both independent
>experts.
>
>	Now because the drive has been in your hands, it might be
>unnecessary for the md5sum.  However, I'd want an original copy I
>could always duplicate to start over with to double check that I
>hadn't modified something.  I'd always mount the disk read only so I
>didn't accidently change a filesystem timestamp or contents by
>accident.
>
> > 1) What files record when root last loged in and from what IP address ?
>
>Normally the command "last", will tell you the last sets of login's.
>Last parses a file named utmp or wtmp.  I forget it's location, but
>it's not a human readable file.  Just use "last".
>
>If they logged in via ssh, I believe that /var/log/messages has the
>timestamps (not sure about IP's).
>
>$ grep "sshd" /var/log/messages
>
>It appears to give me the reverse lookup, and fall back to the IP,
>the reverse name could be faked, but that's a lot of work for the
>attacker to go to.  (Generally storing IP's is easier, faster, and
>more reliable.  I wish more people did that, or at least stored both
>in logs).
>
>If they didn't log into the machine as root, they probably su'ed,
>for that do this:
>
>$ grep "su" /var/log/messages
>
>If they logged into the console this should bring up the appropriate
>lines:
>
>$ grep "login" /var/log/messages
>
>You'll probably have to look closer at it.  The archive logs might
>have rolled, so you might have to look at files with the same
>prefix.  The person who did this is a fool if they didn't wipe the
>logs however.
>
> > 2) What file record any other activity by the same person ?
>
>If it is anywhere, it'll probably be in this directory:
>
>/var/log
>
>It might be in the .bash_history of the various users you believe
>they are logged in as (save this file as soon as possible).  It
>generally only holds 1000 commands, so if you do too much logged in
>as root, it'll erase what history you have.
>
>
> > 3) What log files should be kept for handing over to the police ?
> > (we have informed them and they are sending someone tomorrow)
>
>Keep the original disk if at all possible.  If they are serious
>about forensics, they'll immediatly duplicate it as described above.
>
>I'm doubtful they'll take you terribly seriously unless you can
>demonstrate serious financial damage.  However, it never hurts to
>assume they will take it seriously.
>
> >
> > 4) What else should I be doing ?
> >
> > All help would be very helpful
> > (I think I know the answer to most of the above must I MUST do a sanity
> > check ! ) I just do not want to miss anything.
> >
>
>I'd take the machine offline and try the rootkits the other people
>have mentioned.  I'd examine everything:
>
>/var/spool/cron/
>/etc/cron.*
>/etc/cron
>
>That's where crontabs are stored.
>
>Those are the places people stick files to do nasty things in a
>week.  I forget where "at" stores it's commands.  You probably want
>to check that too.
>
>I've had one machine broken into, we setup a new machine to replace
>the existing one.  We burned the old one to the ground once we
>figured out what they had done.  Made sure we had patched for the
>attack they used and got current with all the other patches we could
>for security purposes.
>
>I can understand your reluctance if it's a customer machine, or if
>you have to do this for 20 machines in a short period.
>
>There's a ton of things to do, I really don't want to ramble on,
>it's not an area I have great deal of knowledge or expertise on.
>I've read about this before, but never had to deal with it
>personally much.  You really should find a forensics expertise list.
>
>I've really just listed the things I've done to track down what
>happened to a machine when I can't get in touch with the admin.
>Looking in /var/log/messages, and examining .bash_history of various
>users is an easy way of seeing what someone has been doing on your
>machine.  However, a compentent malicious user won't leave those
>behind.
>
>	Thanks,
>		Kirby
>
>_______________________________________________
>Whitebox-users mailing list
>Whitebox-users@beau.org
>http://beau.org/mailman/listinfo/whitebox-users