Threat Level: green Handler on Duty: Jim Clausing

SANS ISC: SSHD rootkit in the wild - Internet Security | DShield SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
SSHD rootkit in the wild

There are a lot of discussions at the moment about a SSHD rootkit hitting mainly RPM based Linux distributions.
Thanks to our reader unSpawn, we received a bunch of samples of the rootkit. The rootkit is actually a trojanized library that links with SSHD and does *a lot* of nasty things to the system.

At this point in time we still do not know what the initial attack vector is – it is unknown how the attackers get root access on the compromised servers that is needed to change the legitimate libkeyutils library with a trojanized one. We are, of course, keeping an eye on the development and will post a new diary or update this one if we receive more information about the attack vectors.

The trojanized library is very, very nasty. Upon execution it performs a number of actions, as described below.

The code first deobfuscates the text strings needed for it to work. The original text is only XORed so this is very easy to retrieve and the deobfuscated strings have already been posted on a lot of sites.

Once that has been done, the library sets up everything needed for it to work. It resolves symbols for the following functions which are used later: PEM_write_RSAPrivateKey, PEM_write_DSAPrivateKey, MD5_Init, MD5_Update, and MD5_Final. As you can already see, it is definitely messing up with the authentication mechanism.

Besides resolving the symbols, the library also hooks the following functions: pam_authenticate, pam_start and crypt as well as audit_log_user_message and audit_log_acct_message. By hooking these functions, the rootkit can modify the flow of the SSHD – as you can see, this is a user-mode rootkit, as it does not affect the kernel.

The main activity of the rootkit consists in collection of credentials of authenticated users. Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information. The hooking of audit_log* functions was done to allow the attacker to stay as low profile as possible – if the attacker uses the hardcoded backdoor password to issue any commands to the rootkit, no logs will be created.

The current version of the rootkit supports three commands: Xver, Xcat and Xbnd. The first command just prints the rootkit’s version; the Xcat commands print the collected information back in the session for the attacker while the Xbnd command allows the attacker to setup a listener.

Besides this, the rootkit can automatically send collected credentials to the attacker. In order to do this the rootkit has a DGA (Domain Generation Algorithm) implemented that will create random looking domain names in the .biz, .info and .net domains (in that order). It will then send a DNS packet containing collected credentials to the target IP address, if it was able to resolve it (meaning the attacker has registered that day’s domain). If no domains have been resolved, the DNS packet is sent to the hard-coded IP address, which in all samples we received was 78.47.139.110.

The rootkit itself looks very similar to the Ebury trojan which was detected back in 2011. In fact, I’m pretty sure that a lot of the code has been directly copied, however, the Ebury trojan patched the whole SSHD and required the attacker to change it.

This was easier to detect and prone to being overwritten with patching. The libkeyutils library, which comes as part of the keyutils-libs package is not changed that often so the chance of it being overwritten automatically is much lower.

If you run a RPM based system you can check the integrity of the file with the rpm command:

# rpm -Vv keyutils-libs-1.2-1.el5
........    /lib/libkeyutils-1.2.so
S.5.....    /lib/libkeyutils.so.1
........    /usr/share/doc/keyutils-libs-1.2
........  d /usr/share/doc/keyutils-libs-1.2/LICENCE.LGPL

This will check a lot of things, the most important being the MD5 checksum so if you see the output as one above you have a trojanized library. Proper output should have all (and only) dots. Keep in mind that the RPM’s verification, of course, depends on the integrity of its database and the kernel itself.

We will keep an eye on the development and will update the diary accordingly – if you have samples or more information, especially on what the initial attack vector is please let us know.

I’d like to thanks again to unSpawn for supporting the SANS ISC.


--
Bojan (@bojanz)
INFIGO IS

I will be teaching next: Web App Penetration Testing and Ethical Hacking - SANS Munich July 2019

Bojan

376 Posts
ISC Handler
lots of speculation about the attack vector - local workstation.
This just in from cpanel: Salutations,

You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.

As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.
Anonymous
I wonder if a bit more clarification is in order.

I don't believe there is anything that SSHD can do, hacked or not, to get an inbound SSH user's private key - the private key never leaves the originating system (instead, SSHD asks the incoming SSH a question that only the holder of the private key could answer). So if you're coming into a hacked server from some other place (say, via the PuTTY or SecureCRT client), your private keys are not at risk.

The only private keys that SSHD has anything to do with, as far as I know, are *host* keys of the server it's on, but SSHD has access to those anyway.

However, if the badware is hacking libkeyutils, then that is *also* used by the ssh command-line client, so if you are on a hacked server and initiate an outbound SSH session, then it appears that it can grab your private keys there (and is bad news). But this doesn't appear to involve SSHD unless libkeyutils is somehow sending ssh-based intercepted keys to sshd for phoning home.

None of this applies to *passwords*, which SSHD of course has full access to.

Disclaimer: I don't know any more about this incident than I have read here.
Stephen

1 Posts
Quote: "Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information."

and

Quote: "...the attackers will get your passwords/keys *anyway*."

They cannot steal what is not there. So if you create a key pair off server (using PuttyGen for instance) and upload the public key only (because all you need on the server is the public key) and authorise it, they have no private key to steal so cannot access using keys.
Stephen
2 Posts
@SJFriedl You are right, I should have better checked what I wrote :)

They still hook the RSA/DSA key write functions so I am not sure why they do it. I see that ssh-keygen is also linked to libkeyutils so it might be that they try to steal private keys during the creation process.

Thanks for pointing to this, I'll update the diary shortly.
Bojan

376 Posts
ISC Handler
"Now you can verify if SSHD owns any of the shared memory segments"

SSHD runs as root on RHEL at least, so I'm not sure what
ipcs -mp
is supposed to show.

thanks
Bojan
28 Posts
I mentioned yesterday on twitter, but I don't know if this is completely valid, hopefully you can help clarify this for me. You mentioned earlier that it hooks into MD5_Init, MD5_Update, and MD5_Final. So how can we trust the MD5 sum calculated by rpm -vV, as the checksum could potentially be altered by the MD5_ functions it hooks into?
Bojan
1 Posts
Is it limited to hosts with the RPM-specific local escaliation hole https://isc.sans.edu/diary/Fedora+RedHat+Vulnerabilty+Released/15172 ?
Anonymous
seems to be related to this:
https://bbs.archlinux.org/viewtopic.php?pid=1234615
Quote: [some] got fresh systems installed just to find out that the file was there few minutes later. This might suggest attack from the administrators machine.
They claim: It was confirmed that problems exist in distros based on RHEL and with the cPanel, DirectAdmin and Plesk. One of the vulnerabilities used by the backdoor is CVE-2012-56-71, remote code execution in Exim.
and maybe this is also interesting (including some logs):
https://bbs.archlinux.org/viewtopic.php?pid=1234615

MiKa
Anonymous
sorry, second link should be (cpanel discussion):
https://forums.cpanel.net/f185/sshd-rootkit-323962.html#post1324702
Anonymous
Do you concur that, if the key pair is generated on a local system using something like PuttyGen and only the public key uploaded then the exploit cannot steal the private key.
Anonymous
@RPM based - I'm pretty sure that all Linux kernels are vulnerable. It requires a race to be won so might not be that simple to exploit.

@BJ - when you run ipcs -mp you will see the PID of the process owning the shared memory segment, so check with the ps command if it is SSHD or not. Normally SSHD should use no shared memory segments.

@Fidget - yep, I concur.
Bojan

376 Posts
ISC Handler
After sharing server credentials to cPanel support, two of the servers got compromised, cPanel had no answer.
Bojan
1 Posts
Is it possible that rpm -Vv keyutils-libs gives false positive and isn't enough by itself to confirm trojan presence?
For one of our systems, which is CentOS 5 box, I get the following result:
S.5..... /lib64/libkeyutils-1.2.so
........ /lib64/libkeyutils.so.1
........ /usr/share/doc/keyutils-libs-1.2
........ d /usr/share/doc/keyutils-libs-1.2/LICENCE.LGPL
Some time ago prelink was installed and running by default. It messed up AIDE (Advanced Intrusion Detection Environment) monitoring by changing lots of files. It was turned off and uninstalled because of this. However, I'm not sure what are all the changes that prelink does and if it could change libkeyutils-1.2.so. Any filesystem change made by trojan would be picked up by AIDE. It is possible, but not very likely that this would slip unnoticed in AIDE report which is generated and checked daily.
We don't use cPanel.
Bojan
1 Posts
To filter out possible prelink factor on the keyutils and keyutils-libs RPMs, do the following three steps:

#1, check whether the binaries and libraries are changed.

[root@testbox]# rpm -Vf `rpm -ql keyutils keyutils-libs | grep -P '\.so|bin' `
S.5..... /bin/keyctl
S.5..... /sbin/request-key
S.5..... /bin/keyctl
S.5..... /sbin/request-key
S.5..... /lib64/libkeyutils-1.2.so
S.5..... /lib64/libkeyutils-1.2.so
S.5..... /lib/libkeyutils-1.2.so
S.5..... /lib/libkeyutils-1.2.so

#2, filter the prelink factor by un-prelinking the binaries and libraries (symlinks can not be prelinked or un-prelinked so you see the output):

[root@testbox]# prelink -u `rpm -ql keyutils keyutils-libs | grep -P '\.so|bin' `
prelink: /lib64/libkeyutils.so.1 does not have .gnu.prelink_undo section
prelink: /lib/libkeyutils.so.1 does not have .gnu.prelink_undo section

#3, double check the results again with rpm. Here no output means not hacked -- good news! otherwise it will output some output like 'S.%..... blahblah' lines.

[root@testbox]# rpm -Vf `rpm -ql keyutils keyutils-libs | grep -P '\.so|bin' `
[root@testbox]#

Bojan
2 Posts
It may be better to turn off prelink at the beginning of Centos/Redhat installation.

The prelink may provides some performance boost according to some sources, but it brings more security issues like this case where it confuses System/Security administrators when faced with possible real serious security issues. Although we can un-prelink the affected files ad-hoc, it posts challenges to do it in cron or other automation methods.

Probably it it not a bad idea to sacrifice some performance for security, as nowadays the security becomes more and more secrious topic.

Any thoughts?
Bojan
2 Posts
I read here (http://www.webhostingtalk.com/showpost.php?p=8567829&postcount=978) that a local vulnerabilty is not off the table. In case of a compromised admin PC it could still be possible to even steal ssh-Login-Keys
Bojan
1 Posts
Nessus have released plugin 64913 to detect this trojaned software.

http://www.tenable.com/plugins/index.php?view=single&id=64913
Bojan
5 Posts
Thanks. Also firewall sshd to only accept your IP
Bojan
1 Posts
A point that we seem to be collectively missing here is that it's possible there was not just one sole entry point/root exploit to these systems. The group behind this is (obviously) quite sophisticated, so they could be keylogging, using some cPanel passwords, and using some local exploits as well.

This would be entirely consistent with what we're seeing and could be why it's hard to isolate a single entry point or initial root exploit.

Just a thought ...
Bojan
1 Posts

Sign Up for Free or Log In to start participating in the conversation!