I found this very nice tool called DenyHosts.

DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

If you’ve ever looked at your ssh log (/var/log/secure on Redhat, /var/log/auth.log on Mandrake, etc…) you may be alarmed to see how many hackers attempted to gain access to your server. Hopefully, none of them were successful (but then again, how would you know?). Wouldn’t it be better to automatically prevent that attacker from continuing to gain entry into your system?

DenyHosts attempts to address the above… and more.