ajaypal.com

I have been using FLOSS (GNU/ Linux) and related software on our servers for around 5 years now. Most of the servers face Internet.

As is the case with any evolving software even in GNU/ Linux(s)
vulnerabilities are discovered and ofcourse patched form time to time.

But alas we ppl who manage servers are sometimes lazy, or forget and do not patch servers in time. My excuse we are only human. And in general act as open invitations/ sitting ducks to crackers and
malicious ppl around the word.

I have seen this happen to a lot of ppl around, same has happened to me as well, sometimes we are made to realize, by the crackers, that we havn't been vigil in our duties. And whatever experience I have in managing server i still cant deny this fact that this can happen to me, that will hurt my ego.

But hey we learn from mistakes and usually grow up. But what about
newbies, ppl whom we GNU/ Linux fans motivate, we ask them to switch over to a better system from _you_know_what_$$_crap_they_are_using_.

I still remember the old days when the number of script kiddies was much less, atleast in India. Now ever Ram and Sham has Internet access and is eager to lay its hands on some script, trying to be the super kiddie.

And the poor newbie sysadmin we motivated to switch over to something better, is an easy target. They becoem the victims, kiddies exploit their servers. The machines are listed in XBL, RBL and in general cause disservice to their users and others also. And the newbie sysadmin also gets the impression that GNU/ Linux is difficult/ insecure/ whatever (which is not true). Usually we blame the newbie sysadmin that he havnt been too vigil, lax and did do the usual RTFM.

Most of such ppl remain clue less on what happened and why their systems have misbehaved, what is happed to their servers. Some of such guys also consider moving back to _you_know_what_$$_crap_they_were_using_. And ppl like us who motivated them are back to zero, all out time spent motivating
them goes down the drain.

Some of the glaring mistakes newbies make are Installing whatever version of older unpacthed versions of GNU/ Linux they can lay their hands on. And then not configuring firewalls, not closing unnecessary services and what not. All of this has been documented we all know that, but still happens.

And all of the above is true for experienced sysadmins also. I have seen so many websites being defaced/ mail servers being used for spam, IRC chat clients installed etc etc. Again my excuse after all we are only human.

Cant we as a community of good ppl/sysadmins do something about it. Is it possible to build a community where we can watch over each others back, and report any problems in time to vulnerable system/ or systemes that are already down the drain. And from community I do not mean another mailing list or user group. Is it possible to do something automated, to keep watch over servers, a distributed system. Where ppl who have subscribed to the system would have their system checked/ scanned periodically by other systems, and sysadmin can be forwarned of existing/ new problems. Something like an XBL, RBL but without the black list thing, and with a warning and maybe possible solutions to the problem for the sysadmins.

Similar services are offered by some commercial vendors, but i believe a community effort would be a better option, due to its very distributed nature and scale. (More technicalities can be discussed later)

I am trying to forge such an alliance with two other sysadmins i know, and hope something will come out of it. And we plan to make newbies, around our area, part of it, and maybe help them with their newly setup servers, so that they dont go back to _you_know_what_$$_crap_they_were_using_. Most of it would be initially manual, except periodic port scans to locate vulnerabilities, but later on more things can be automated.

If there is anything similar in place, or any advice or comments, please.

Please dont tell me that:

• a good sysadmin dont need such a crap.
• real sysadmins secure their systems like forts
• real sysadmins dont make mistakes
• pull out your network wire to secure your servers
• RTFM
• go away, you cant run a GNU/ Linux workstation, dont even think of servers
• blah blah

I think i have some experience, but still sometimes i need help and
confirmation that my servers are ok, what is wrong in third party
confirmations, if it is only a remove vulnerability scan. And why not i can do that same for others and others can do it for me. I have been doing this for 2-3 ppl already. And why cant we automate this process and in a distributed manner.

What is my motivation for writing all this
Recently i had some discussion with someone who is an advocate of FLOSS and a dedicated GNU/ Linux user pasting it here without his permission (this is part of an email discussion) some parts edited/ changed

What ever you observe, that was correct. However, I am only user of
******* services. I forwarded your Email to concerned man on Friday

.. and he told me today (Monday), that system was hanged on Saturday
on rebooting, it fail to boot. In nutshell, there is *real problem* with
server. He is trying to fix it.

another part of email, some part edited

Before we start discussing, I would like to know frank opinion, about
FLOSS, is it going to help us.

Another discussion with someone else
parts of a telephonic conversation (whatever i can remember)

Him: The nameserver lookup is not working
Me: Have you checked the logs
Him: I cant, somehow the logs aint showing anything at all
Me: (Puzzled) that should mean, maybe server has been compromised
Me: (after a port scan of his machine, next day) there is sshd server
running on port 1422, you server is definitly compromised. Time to
reinstall.

And on Internet a simple portscan can find a lot of machines which are either compromised or ready to be compromised and we do nothing about them, ofcourse unless the machines are honeypots and have been left like that intentionally.

Can we help each other!!