modrewrite to combat spam attacks
This is a simple ModRewrite i use to combat the file upload, then expand attack that is fairly common with poorly created upload scripts
Basically this reads, if the request or the refferer contains any of the bad words, redirect it to a script called 'nospam.php' (in the root of your server) and give a 404 error, so the page will be removed from search engines.
RewriteCond %{THE_REQUEST} tnb|torrent|bad-credit|4u|adipex|advicer|baccarrat|blackjack|bllogspot|booker|byob|car-rental-e-site|car-rentals-e-site|carisoprodol|casino|casinos|chatroom|cialis|coolcoolhu|coolhu|credit-card-debt|credit-report-4u|cwas|cyclen|cyclobenzaprine|dating-e-site|day-trading|debt-consolidation|debt-consolidation-consultant|discreetordering|duty-free|dutyfree|equityloans|fioricet|flowers-leading-site|freenet-shopping|freenet|gambling-|hair-loss|health-insurancedeals-4u|homeequityloans|homefinance|holdem|holdempoker|holdemsoftware|holdemtexasturbowilson|hotel-dealse-site|hotele-site|hotelse-site|incest|insurance-quotesdeals-4u|insurancedeals-4u|jrcreations|levitra|macinstruct|mortgage-4-u|mortgagequotes|online-gambling|onlinegambling-4u|ottawavalleyag|ownsthis|palm-texas-holdem-game|paxil|penis|pharmacy|phentermine|poker-chip|poze|pussy|rental-car-e-site|ringtones|roulette|shemale|shoes|slot-machine|texas-holdem|thorcarlson|top-site|top-e-site|tramadol|trim-spa|ultram|valeofglamorganconservatives|viagra|vioxx|xanax|zolus
RewriteRule (.*) nospam.php [R=404,L]
RewriteCond %{HTTP_REFERER} tnb|torrent|bad-credit|4u|adipex|advicer|baccarrat|blackjack|bllogspot|booker|byob|car-rental-e-site|car-rentals-e-site|carisoprodol|casino|casinos|chatroom|cialis|coolcoolhu|coolhu|credit-card-debt|credit-report-4u|cwas|cyclen|cyclobenzaprine|dating-e-site|day-trading|debt-consolidation|debt-consolidation-consultant|discreetordering|duty-free|dutyfree|equityloans|fioricet|flowers-leading-site|freenet-shopping|freenet|gambling-|hair-loss|health-insurancedeals-4u|homeequityloans|homefinance|holdem|holdempoker|holdemsoftware|holdemtexasturbowilson|hotel-dealse-site|hotele-site|hotelse-site|incest|insurance-quotesdeals-4u|insurancedeals-4u|jrcreations|levitra|macinstruct|mortgage-4-u|mortgagequotes|online-gambling|onlinegambling-4u|ottawavalleyag|ownsthis|palm-texas-holdem-game|paxil|penis|pharmacy|phentermine|poker-chip|poze|pussy|rental-car-e-site|ringtones|roulette|shemale|shoes|slot-machine|texas-holdem|thorcarlson|top-site|top-e-site|tramadol|trim-spa|ultram|valeofglamorganconservatives|viagra|vioxx|xanax|zolus
RewriteRule (.*) nospam.php [R=404,L]
This simple addition is a nice little tool in your web server admin toolbelt.
Vmware vSphere – wrong amount of free space reported.
Earlier today we began to upgrade our ESX 3.5 machines to VMware vSphere (esx 4.0) We normally have 3 ESX hosts, one has been offline and retasked to testing our new SAN hardware. We have enough capacity on our esx2 machine to house our entire collection of VM's under nominal load. So, I sent esx1 into maintenance mode, and upgraded it to vSphere. The install went perfectly, and i got the networking all set up and joined it to our cluster.
Everything seemed good as i started to migrate machines over to esx1, so i could do the same upgrade on esx2... then everything went horribly wrong.
our VMFS is a 4TB lun, in ESX 3.5 land the command vdf reports 1.5TB available. in vSphere land it reports 8GB!!!!!
When we moved machines over to run on vSphere their swapfiles were causing problems and vSphere thinks it doesn't have enough space to run VM's. Luckly before everything went down i was able to get esx2 out of maintenance mode, and migrate all the VM's back to esx2.
Right now our vSphere esx1 is dead in the water until i can figure out why it thinks 1.5TB = 8GB
Unless i can come up with the answer tonight i believe a call to VMware tech support will be in order for the morning.
UPDATE: We upgraded another esx node to vSphere and it sees the old VMFS disk the same way. I think its a LVM problem, either with our RAID hardware or something corrupt on the old VMFS volume. We've decided to create a new VMFS volume on our EMC AX4 and call it a day. I really HATE sidestepping problems like this. I want to know WHY its broken like this. Grrr.
SPAM SPAM GO AWAY!
It seems the new trend for successfully exploited weak web applications is that they will only show their seedy content to google or other search engines.
They will only be triggered to show their content when they are accessed by GoogleBot from a Google IP (yeah they are getting that specific).
When you click and view a page that google say's is full of Viagra spam, you won't see anything, its tricky and VERY frustrating and hard to troubleshoot.
So far, the common sign i've seen of successful exploits have been:
1. .bak files (installed as wordpress plugins, you have to scour your 'active_plugins' field in the database
2. .pngg .giff .jpgg and .old files, trying to upload malicious PHP and get around unsecure uploaders
3. the use of the base64_decode PHP function, while there are legit uses for this function, it can be a sign of a baddie
4. Use of the 'eval' function in PHP. Also, legit uses are out there, but i've seen it used for the dark side of the force.
5. a 'WordPress' user in your Wordpress user table.
If you want to scan a *nix system for the file names i've found to be 'bad' use the following commands.
find -name *_old.php*
find -name *.php.jpgg
find -name *.php.giff
find -name *.php.pngg
To look for those functions i talked about your can use your friend 'grep'
grep -inrH "eval(base64_decode(" <your dir here>
grep -inrH "gzinflate(base64_decode(" <your dir here>
For anyone interested I've recently installed mod_security with their core rule sets on our Apache webserver and after tweaking the config files and creating some white-lists I have be able to ward off a number of baddies and exploit attempts.
http://www.modsecurity.org/
Its worth the hassle of setting it up. It also has a 'detection only' mode which does a great job letting you know what you have running and tweak the rules before it starts to block requests.
The trials and tribulations of Wordpress SPAM
In the past month i've seen a huge increase hacked and spammed Wordpress installs.
At CIAS we have 1000+ users accounts with 203 individual installs managed by the students, so i usually try to allow the students the benifit of the doubt that they will update their own software... year... that hasn't been working out so well.
Over the years i've had to update a few accounts which have been hacked or otherwise compromised... but lately, the issue has been getting worse and worse. Last night i did a mass upgrade of 203 installs to wordpress 2.7..
I've created a script that watches the version of every install on the server. Once a new version of wordpress becomes available i will increment the version on the server and hopefully everyone's wordpress will get upgraded just like that. Hopefully i'll be able to keep us from getting hammered with spam.
Another issue we have on the server is poorly written upload scripts which are allowing .php files to be uploaded, and then execuded by the webserver, which then creates a nice little backdoor for hackers to then pepper the server with bad files and crap. Its annoying! LEAVE ME ALONE 
Well… that was an interesting night!
Tonight was supposed to be a very simple hard drive copy to a server, virtualize a physical server, get a beer and go home... I figured, i'd be at home by 6 or 7pm... well... Technology has a way of really screwing with you. So here's a little story about my stupid night.
So 5:00pm goes around and i turn off the old desktop class machine that has been used as a server for the better part of 5 years. The plan is to take the hard drive will all the unamed organizations data off of it, put it on to our super fast secure awesome server, and then call it a night, if I had time i was going to get rid of the old physical hardware once and for all and put the server on our virtual server cluster, no brainer, i've done this kind of thing a million times before.
Here is where the first snag came, i got the hard drive, put it in our hard drive to firewire adaptor, and i couldn't get it to mount. Hell, i couldn't get it to spin... finally after some fiddling and adding the external power plug for the adaptor i got it going, i started the file copy, i thought i was done. I promptly went to play COD4 and blow some 12 year noobs up. I turned around to check on the status of the file copy, and i noticed the drive wasn't spinning. "Strange" I think to myself. I try remounting the drive over and over again, and it just won't go, this is already pissing me off. I put the drive on my Windows XP machine and it mounts after a long time, and i start the file copy again, and the drive stops spinning AGAIN. So by now i'm pissed. Its fairly obvious the drive isn't working properly, and this explains the slow performance the organization had been seeing on their old file server. So what do I do? Only what any sane IT professional would do with a semi dead hard drive, i put it in the freezer. Yes you read that right, i put it in the freezer. Its a religous thing, the hard drive in freezer trick. Some people say its hog-wash, other swear by it. In our office, its saved our butts a few time, so "I want to believe"
At this point Jay and I decide that while the hard drive is on ice, we will take a walk down to the parking lot and pick up our vehicles from the far corner of the lot and bring them closer, if anything to get outside and get fresh air and not be around stupid dead hard drives... and then we saw something that was just... strange... I think the picture will speak for its self. A box truck, inbetween the two levels of parking.
So after wondering WTF was going on the with the box truck we headed back in to check on our frozen hard drive. We hooked it up, and BAM it started working right off the bat, real fast and everything... and then about 5 minutes into the copy it failed again. At this point i was starting to loose faith that we would get this going. So, we froze it again. Tried it again, and it would go for a while, then fail. This is the point where desperation is starting to set in. I really want to get this server finished by start of business Tuesday. So i thought to myself, "What would happen if i ran the hard drive from INSIDE the freezer"
Yes, thats a power cable, and firewire cable going into the office fridgem which we've moved next to my desk. The sick part? IT WORKS. The hard drive is running, and actually copying data. Its still a bit slow, but the hard drive is still working. I just don't understand, but whatever. ITS WORKING!!
Yup, so that was my night, a hard drive, in a static bag, in the freezer.
So, have you ever had any luck with the hard drive in a freezer? Have you ever gone to this lengths? That was one heck of a night at the office.
Now, time for a blast from the past, an mixer board and outboard rack:
Managing multiple Debian linux servers
At CIAS here we manage around 20 linux machines, of which only a handful of them are in a cluster.
This means that conventional tools such as puppet and other cluster management software suits don't fit well into what we want to do.
So what am i to do? I have 20 machines to manage, secure, audit, monitor, update and any number of other tasks. Any good system administrator will have his hands knee deep in BASH and perl and <insert favorite scripting language here>. I personally happen to be a web guy. I live and breath in PHP and MySQL. My specialty over the years has been to create nice little one off web applications that parse data, manage it, and present in a useful manner to myself and my co workers. Lately i've start to go a step beyond that and create web services which my machines begin to interact with. I've created clever little command line apps that do specific tasks, and are usually generic enough that they work on all our machines. I've even created a deployment method for these apps! So now i update a single repository of our scripts and auto-magicly our servers have the latest scripts.
This system has really started to work well, and its been growing day by day.
So far i've created systems for the following solutions:
- A global iptables blacklist - add an ip to the list and all our machines block that IP
- A command to block an IP from any of our hosts, which then is put in the global list
- A script that audits SSH attacks and blocks those ips
- A web interface for all those blocked IP's (Add, Edit and Delete from the list)
- A web interface to show all available APT updates on a host, and the ability to approved updates and have them install automatically at a certain time.
- A web reporting tool that monitors all our servers disk usage and sends warning on full or near full disks
- A interface to a long term archive solution we are custom building
- A script and web interface which aggregates all of our logwatch reports, and then converts them to RSS
As more and more problems need solution i keep creating and distributing these systems. To what end?
In the future i would love to create a portal based system where I log in and i can manage and interact with all of my servers from one place. Much like an Altiris Notification server for my linux machines.
So here's my question to the great internet... Does anyone else do anything kinda like this? Is this a set of solution that I should think about packaging up and creating an open source project for? Does anyone care? Does anyone else have a better solution than the hacks i've been working on? I want to hear your feed back!!!
And now for something completely random:
