ssh brute force attacks

[Ed Heron]

What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.

Most attackers (several dozen per day) try once every 15 to 20
minutes. I assume to avoid automatically being banned. Some were
throwing thousands of attempts at me from a single IP address. It might
have been happening on my old servers, but I'm afraid to look.

I didn't have any automatic banning software installed before, but I
do now.

However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).

We recently started using OpenVPN with certificates. I'm working on
adding a password so it takes both a certificate and a password to
connect a remote machine to the private network.

I'm thinking that OpenVPN makes external SSH obsolete. I could turn
it off forcing me to start a whole VPN in order to get access to the
internal interface.

Has everybody else already gotten to this conclusion or are there lots
of people still allowing remote SSH access?

I'm also thinking of setting up a honey pot on my external SSH port.
Are there any pre-configured honeypot distributions? Would a honeypot
that never lets anybody in because it doesn't have any valid login
combinations be good or should it let them in and let them waste their
time installing root kits then when they logout reset the machine.

Any thoughts?

[Geoff Chesshire]

Hi Ed,

Post by Ed Heron
We recently started using OpenVPN with certificates. I'm working on
adding a password so it takes both a certificate and a password to
connect a remote machine to the private network.

I've been using OpenVPN for years, and have found it extremely reliable.
In fact, not only do I use it for external access, but I use it even
within the LAN, so that all services are accessible only through OpenVPN
(even ssh, dns, nfs, samba, email, apache, cups). Laptops keep their same
VPN IP addresses wherever they connect from, making DNS administration
easy. I make WAN access to OpenVPN come in using UDP through a secure
email port such as 143 or 993, as these few are generally not blocked by
public networks e.g. at airports. I forward this port to the usual
OpenVNP port (1194) on the VPN server on the LAN. Laptops try to connect
to the server first over the LAN at port 1194, and if that fails, fall
back to the WAN address+port. I use client/server PKI certificates, with
a pre-shared key during the handshake. Passwords on the client
certificates are possible, but I haven't used them. For laptops, I think
an encrypted root partition with a boot-up passphrase may be the way to go.

Post by Ed Heron
I'm thinking that OpenVPN makes external SSH obsolete. I could turn it
off forcing me to start a whole VPN in order to get access to the
internal interface. Has everybody else already gotten to this
conclusion or are there lots of people still allowing remote SSH access?

The only time I ever enable external SSH access is temporarily to allow
guest access from an external machine that is not (yet) included in the
VPN, and even then I use an obscure port and forwarded that to port 22 on
the target machine. The only port normally left open on the WAN is the
one for the VPN connection. Heck, I can even reliably restart OpenVPN
remotely via ssh over OpenVPN, with no open sockets lost and the
connection barely misses a heartbeat.

Thanks,
Geoff

[Chris McMahon]

OK, I can't believe I'm posting this on Friday night, but do you have
performance issues with OpenVPN? Every VPN I've ever used has been a
serious drag to getting real work done.

-C

Post by Geoff Chesshire
Hi Ed,

Post by Ed Heron
We recently started using OpenVPN with certificates. I'm working on
adding a password so it takes both a certificate and a password to
connect a remote machine to the private network.

I've been using OpenVPN for years, and have found it extremely reliable.
In fact, not only do I use it for external access, but I use it even
within the LAN, so that all services are accessible only through OpenVPN
(even ssh, dns, nfs, samba, email, apache, cups). Laptops keep their same
VPN IP addresses wherever they connect from, making DNS administration
easy. I make WAN access to OpenVPN come in using UDP through a secure
email port such as 143 or 993, as these few are generally not blocked by
public networks e.g. at airports. I forward this port to the usual
OpenVNP port (1194) on the VPN server on the LAN. Laptops try to connect
to the server first over the LAN at port 1194, and if that fails, fall
back to the WAN address+port. I use client/server PKI certificates, with
a pre-shared key during the handshake. Passwords on the client
certificates are possible, but I haven't used them. For laptops, I think
an encrypted root partition with a boot-up passphrase may be the way to go.

Post by Ed Heron
I'm thinking that OpenVPN makes external SSH obsolete. I could turn it
off forcing me to start a whole VPN in order to get access to the
internal interface. Has everybody else already gotten to this
conclusion or are there lots of people still allowing remote SSH access?

The only time I ever enable external SSH access is temporarily to allow
guest access from an external machine that is not (yet) included in the
VPN, and even then I use an obscure port and forwarded that to port 22 on
the target machine. The only port normally left open on the WAN is the
one for the VPN connection. Heck, I can even reliably restart OpenVPN
remotely via ssh over OpenVPN, with no open sockets lost and the
connection barely misses a heartbeat.
Thanks,
Geoff
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[J. Marsden DeLapp]

Post by Ed Heron
What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.
Most attackers (several dozen per day) try once every 15 to 20
minutes. I assume to avoid automatically being banned. Some were
throwing thousands of attempts at me from a single IP address. It might
have been happening on my old servers, but I'm afraid to look.
I didn't have any automatic banning software installed before, but I
do now.
However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).

Here are a few things I do to increase SSH security.

Create a sshuser group and require anyone who needs ssh access to be in that group.

vim /etc/ssh/sshd_config
#change to not allow root login
PermitRootLogin no
#added sshuser group and
#Added to restrict ssh login to people in the sshuser group
AllowGroups sshuser

Anyone who is a ssh user should have a unique userID. Don't use common names
like bob, chuck, fred, admin, staff, etc. Things like bobjones, chucksmith are better choices.

Another cute trick to slow them down is to rate limit connections.

#These two commands will rate limit connection attempts on ssh
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

But that does not work against the distributed attacks coming from a slew of different
ip addresses. And it sets you up for a potential denial of service attack. If you do the
rate limiting thing, make sure you tell your users if they have three failed login attempts,
they need to wait for at least 60 seconds before trying again.

Mars
--
=============================================================
J. Marsden DeLapp, PE
President
DeLapp & Associates, Inc. dba DeLapp Engineering.
Providing lighting and power planning, design and analysis services
for commercial, industrial and large residential facilities.
1190 Harrison Road Ste 3a
Santa Fe NM 87507
[phone]
http://DeLapp.com
=============================================================

[Andres Paglayan]

Post by J. Marsden DeLapp

Post by Ed Heron
What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.
Most attackers (several dozen per day) try once every 15 to 20
minutes. I assume to avoid automatically being banned. Some were
throwing thousands of attempts at me from a single IP address. It might
have been happening on my old servers, but I'm afraid to look.
I didn't have any automatic banning software installed before, but I
do now.
However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).

Here are a few things I do to increase SSH security.
Create a sshuser group and require anyone who needs ssh access to be in that group.
vim /etc/ssh/sshd_config
#change to not allow root login
PermitRootLogin no
#added sshuser group and
#Added to restrict ssh login to people in the sshuser group
AllowGroups sshuser
Anyone who is a ssh user should have a unique userID. Don't use common names
like bob, chuck, fred, admin, staff, etc. Things like bobjones, chucksmith are better choices.
Another cute trick to slow them down is to rate limit connections.
#These two commands will rate limit connection attempts on ssh
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

Mode parent up! ;-)

I use those rules from Mars in every server since I got them as response
few years ago.
even with distributed attacks they are very good,

In the more sensitive servers I only allow ssh from two ips,

1./ my house dsl ip,
2./ and the ip of another collocated server of mine
so worst case, if I am not at home, I can ssh into machine two, then ssh
into one and change the rules.

there's one comment thought,
it counts for every ssh connection, not just failed,
so if you open 4 consoles and ssh in in less than a minute from them
all, you lock yourself out,
it's good to have a rule that always allows ssh from your ip

make sure to restore rules on startup,

Post by J. Marsden DeLapp
But that does not work against the distributed attacks coming from a slew of different
ip addresses. And it sets you up for a potential denial of service attack. If you do the
rate limiting thing, make sure you tell your users if they have three failed login attempts,
they need to wait for at least 60 seconds before trying again.
Mars

[Peter Espen]

In addition, I wrote a script that's in crontab and it examines auth.log and automatically blocks the offending originating IP address via iptables.

Peter
--

Post by J. Marsden DeLapp

Post by Ed Heron
What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.
Most attackers (several dozen per day) try once every 15 to 20
minutes. I assume to avoid automatically being banned. Some were
throwing thousands of attempts at me from a single IP address. It might
have been happening on my old servers, but I'm afraid to look.
I didn't have any automatic banning software installed before, but I
do now.
However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).

Here are a few things I do to increase SSH security.
Create a sshuser group and require anyone who needs ssh access to be in that group.
vim /etc/ssh/sshd_config
#change to not allow root login
PermitRootLogin no
#added sshuser group and
#Added to restrict ssh login to people in the sshuser group
AllowGroups sshuser
Anyone who is a ssh user should have a unique userID. Don't use common names
like bob, chuck, fred, admin, staff, etc. Things like bobjones, chucksmith are better choices.
Another cute trick to slow them down is to rate limit connections.
#These two commands will rate limit connection attempts on ssh
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
But that does not work against the distributed attacks coming from a slew of different
ip addresses. And it sets you up for a potential denial of service attack. If you do the
rate limiting thing, make sure you tell your users if they have three failed login attempts,
they need to wait for at least 60 seconds before trying again.
Mars
--
=============================================================
J. Marsden DeLapp, PE
President
DeLapp & Associates, Inc. dba DeLapp Engineering.
Providing lighting and power planning, design and analysis services
for commercial, industrial and large residential facilities.
1190 Harrison Road Ste 3a
Santa Fe NM 87507
[phone]
http://DeLapp.com
=============================================================
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Wesley J. Landaker]

Post by Peter Espen
In addition, I wrote a script that's in crontab and it examines auth.log
and automatically blocks the offending originating IP address via
iptables.

You may want to consider using fail2ban, which is a maintained open-source
project that does the same sort of thing:

http://www.fail2ban.org/

[Peter Espen]

Post by Wesley J. Landaker

Post by Peter Espen
In addition, I wrote a script that's in crontab and it examines auth.log
and automatically blocks the offending originating IP address via
iptables.

You may want to consider using fail2ban, which is a maintained open-source
http://www.fail2ban.org/

Thanks. However, sshd 22 is the only port I leave open on my personal linux box and something like fail2ban is overkill for me. Plus, my script does exactly what I want and it was fun to write.

[allan]

Here is a suggestion...
Close port 22 on the firewall, set up a forwarding rule for a public facing "unused" port that maps to your server ip
and port 22.
Accessing your public ip with "unused" port above will forward request to your server port 22. You can leave 22
untouched on the inside machine and let the firewall trap external requests for port 22. You can access yor machine from
the outside via the "unused" port.
Peace,
Allan

Post by Peter Espen

Post by Wesley J. Landaker

Post by Peter Espen
In addition, I wrote a script that's in crontab and it examines auth.log
and automatically blocks the offending originating IP address via
iptables.

You may want to consider using fail2ban, which is a maintained open-source
http://www.fail2ban.org/

Thanks. However, sshd 22 is the only port I leave open on my personal linux box and something like fail2ban is overkill for me. Plus, my script does exactly what I want and it was fun to write.
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[allan]

Years ago I started filtering ssh in my firewall for this same reason. I do not think it is a "new thing." I was
receiving ssh failed logins from groups of ip addrs (too large to filter?) multiple times. Since I did not require
external access I was able to shut it down at the firewall.
Another trick is to change the ssh port from the standard to something that would be hard to guess.

Peace,
Allan

Post by Ed Heron
What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.
Most attackers (several dozen per day) try once every 15 to 20
minutes. I assume to avoid automatically being banned. Some were
throwing thousands of attempts at me from a single IP address. It might
have been happening on my old servers, but I'm afraid to look.
I didn't have any automatic banning software installed before, but I
do now.
However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).
We recently started using OpenVPN with certificates. I'm working on
adding a password so it takes both a certificate and a password to
connect a remote machine to the private network.
I'm thinking that OpenVPN makes external SSH obsolete. I could turn
it off forcing me to start a whole VPN in order to get access to the
internal interface.
Has everybody else already gotten to this conclusion or are there lots
of people still allowing remote SSH access?
I'm also thinking of setting up a honey pot on my external SSH port.
Are there any pre-configured honeypot distributions? Would a honeypot
that never lets anybody in because it doesn't have any valid login
combinations be good or should it let them in and let them waste their
time installing root kits then when they logout reset the machine.
Any thoughts?
_______________________________________________
NMLUG mailing list
http://lists.b9.com/cgi-bin/mailman/listinfo/nmlug

[Matt Rechkemmer]

Post by Ed Heron
However, it makes me think about SSH. It is a secure protocol but a
bad password could open my system up to exploitation. This isn't a SSH
fault but a lack of confidence in my users. And we don't even use SSH
from outside the private network that often (just me for maintenance).
We recently started using OpenVPN with certificates. I'm working on
adding a password so it takes both a certificate and a password to
connect a remote machine to the private network.
I'm thinking that OpenVPN makes external SSH obsolete. I could turn
it off forcing me to start a whole VPN in order to get access to the
internal interface.
Has everybody else already gotten to this conclusion or are there lots
of people still allowing remote SSH access?
I'm also thinking of setting up a honey pot on my external SSH port.
Are there any pre-configured honeypot distributions? Would a honeypot
that never lets anybody in because it doesn't have any valid login
combinations be good or should it let them in and let them waste their
time installing root kits then when they logout reset the machine.

If you don't use SSH from outside the network, don't allow access from the
outside. Many of my boxes need remote login. Consequently, they're often
being probed by the many botnets out there. At one point, it got so bad
that sshd was being overwhelmed and wouldn't accept further connections.

I've been using Fail2ban (http://bit.ly/bFSCku) for sometime now. This
stops most attacks, but when the storm gets heavy, even Fail2ban can falter.
Not too long ago, I enhanced sshd's security further with knockd
(http://bit.ly/Z7wWM). The amount of attacks I see on a daily basis is now
near zero. If you use a fairly complicated knock sequence, I bet you'll see
straight zeroes. (I use only two ports that are ascendingly apart from each
other for simplicity. This makes it easy, but it also opens things to the
occasional rapid port scanning bot(s) - that's where Fail2ban comes in).

I've also seen a lot of success from running sshd on an alternate port
(e.g., 443). You could probably do this if you still wanted to allow for
"maintenance" access. Almost all the SSH probing bots I've seen only look
on port 22 for the daemon.

If you want to setup a honey pot for SSH, look at Kojoney:
http://kojoney.sourceforge.net/

OpenVPN is great. Just make sure you maintain your CRL to account for
stolen, "lost," and other misplaced equipment that was assigned a cert.

Hope this helps,

--
Matt Rechkemmer
nmlug-list-9d2Zkzbd0mmzjR9+/***@public.gmane.org

"All that is necessary for evil to succeed is for good men to do nothing."
- Edmund Burke

[Aaron Birenboim]

Post by Ed Heron
What is the deal with the SSH brute force attacks? I wasn't paying
attention until recently, but some of my new CentOS machines are giving
me reports of all the failed login attempts.

I get hammered all the time at work.
See if denyhosts will work for you.

Post by Ed Heron
Has everybody else already gotten to this conclusion or are there lots
of people still allowing remote SSH access?

key access only, with denyhosts is quite safe IMOHO, and
denyhosts will cut down the hacker SSH traffic fast.
It starts putting hacker/zombie IPs on a blacklist.

Warning: you will get some false positives,
when users try many times in a row to login, and fail.

aaron