Bradley Coudriet trying to make the world better, one line of code at a time.

13Nov/082

Managing multiple Debian linux servers

At CIAS here we manage around 20 linux machines, of which only a handful of them are in a cluster.

This means that conventional tools such as puppet and other cluster management software suits don't fit well into what we want to do.

So what am i to do? I have 20 machines to manage, secure, audit, monitor, update and any number of other tasks. Any good system administrator will have his hands knee deep in BASH and perl and <insert favorite scripting language here>. I personally happen to be a web guy. I live and breath in PHP and MySQL. My specialty over the years has been to create nice little one off web applications that parse data, manage it, and present in a useful manner to myself and my co workers. Lately i've start to go a step beyond that and create web services which my machines begin to interact with. I've created clever little command line apps that do specific tasks, and are usually generic enough that they work on all our machines. I've even created a deployment method for these apps! So now i update a single repository of our scripts and auto-magicly our servers have the latest scripts.

This system has really started to work well, and its been growing day by day.

So far i've created systems for the following solutions:

  • A global iptables blacklist - add an ip to the list and all our machines block that IP
  • A command to block an IP from any of our hosts, which then is put in the global list
  • A script that audits SSH attacks and blocks those ips
  • A web interface for all those blocked IP's (Add, Edit and Delete from the list)
  • A web interface to show all available APT updates on a host, and the ability to approved updates and have them install automatically at a certain time.
  • A web reporting tool that monitors all our servers disk usage and sends warning on full or near full disks
  • A interface to a long term archive solution we are custom building
  • A script and web interface which aggregates all of our logwatch reports, and then converts them to RSS

As more and more problems need solution i keep creating and distributing these systems. To what end?

In the future i would love to create a portal based system where I log in and i can manage and interact with all of my servers from one place. Much like an Altiris Notification server for my linux machines.

So here's my question to the great internet... Does anyone else do anything kinda like this? Is this a set of solution that I should think about packaging up and creating an open source project for? Does anyone care? Does anyone else have a better solution than the hacks i've been working on? I want to hear your feed back!!!

 

And now for something completely random:

DCA Finals

20Mar/080

Logwatch & RSS – A Perfect Union

Here at CIAS we have a bunch of linux servers, and we run logwatch on them to give us a daily look at our servers.

There are few options with logwatch and how you want to get the information, you can have it email you, which for one or two servers that is pretty standard, but we are managing 10+ servers, and i don't want to clog up my email and i don't want to use my inbox as a way to archive all of our logwatch reports.

So what did I come up with?

On The Server

Everything that happens here is using a simple upload script i created to facilitate the aggregation of the scripts. Machine's logwatch runs as a cron scripts and outputs its report to a directory on the machine. Right after that script runs another scripts run and copies the logwatch report to a central web server.

On the Web Server

The 'logwatch' app is just a nice simple front end to a bunch of directories from this structure

archive
    server1
        12-3-07
            logwatch.txt
        12-2-07
            logwatch.txt
     server2
            12-3-07
                logwatch.txt

I created a nifty little PHP class that takes care of this structure, and allows easy access to the files through code.

RSS and <insert favorite news reader here>

Cool, so now i have a nifty little web app that organizes our logwatch reports... so whats the next step?

I already run NetNewsWire on my Mac and RSS would be a perfect solution for seeing updated logwatch reports as they come in from the servers. So create the RSS is was pretty easy...

1. use the PHP class
2. output in RSS format the last 5 logwatch reports for the specified server
3. RSS MONEY!!

I'm pretty happy with this elegant solution. I always have a hard time remembering to 'go' to a web app... I always keep NetNewsWire open. So this solution makes me a better System Administrator

Tagged as: , , , No Comments
28Feb/080

Wildcard Certificate on cias.rit.edu

Earlier today we installed a wildcard certificate on cias.rit.edu.

What does this mean?

A: Before today, if you went to https://bjcpgd.cias.rit.edu (over SSL) you would be using the certifcate for cias.rit.edu, and in the eyes of your browser bjcpgd.cias.rit.edu and cias.rit.edu are two totally seperate domains. The installation of this wildcard certifcate now means that the certificate will cover and be valid for all sub-domains of cias.rit.edu. This means you can now easily use SSL to protect your web applications on cias.rit.edu.

Do I have to change anything to use this certificate?

A: Not at all! The certificate is active and valid for all accounts under cias.rit.edu. You may need to tweak your application to make sure its point to https:// in the areas that you want protected, but that is a very application specific issue.

What else has the cias tech team been doing to help secure our applications and data on cias.rit.edu?

A: In the past we have allowed all users to be able to browse other users directories. We've been slightly uncomfortable with this scenario for a while. Yesterday we made a change to everyone's home directory that will keep prying eyes out, yet still allow your web applications to 'roam free'. We specificly did the follow on each users home directory:

    chgrp www-data /home/<username>    chmod 750 /home/<username>

What exactly did that just do? First off we made the group owner for you home directory the web server's user. So you are the owner of your directory, and www-data is the group. The chmod of 750 means, The owner can read,write and execute. The group can read and write, everyone else has NO ACCESS.

 

28Feb/080

Apache Errors

Tonight i encountered a new error in Apache i've never seen before:

[Thu Feb 28 17:54:32 2008] [notice] child pid 2602 exit signal File size limit exceeded (25)

I saw MANY of these errors, and Apache wouldn't return any pages to the browser.

Here to find out, in 32bit Linux, there is a 2GB file size limit.. when this file size limit is reached on an Apache Log file..... these errors start to crazy.. and breaks Apache.

Lesson learned

So... with that said, once we get all 64bit ESX Nodes, i will start to slowly upgrade my web nodes up to 64bit.

Update:

I have since read on the 'Tubes' that this issue has been resolved in Apache 2.2.x... the server that was having this problem was Apache 2.0.x

9Feb/080

Developers BEWARE

Ok, this is a really funny thing that happened to me a few days ago while working with an outside vendor.

We were talking about an unnamed product by an unnamed vendor.

Tech: We connect using RSS to remote control the machine

Me: (With a dumb founded look) you use... RSS to connect?

Tech: Yeah, Remote System Support

Me: OOOHH That makes sense, So this webserver connects over port 80 and port 443, SSL right?

Tech: Oh No, 80 for HTTP and 443 is actually used for a java applet... not SSL...

Me: again dumb founded..... excuse me?

Ok, so what is the moral of this story? Developers, don't reuse already established acronyms, and when you use a standard port.. use it for what its intended, PLEASE?!

31Jan/081

I admire this!

This student's goal is to get his CIAS blog to the top of google

http://sjb0940.cias.rit.edu/wordpress/?p=103

Filed under: web development 1 Comment
21Nov/070

Leopard, ImageMagick gem

I was searching the internet looking for a way to install the ImageMagick gem on Leopard, and lone behold i found it!

YAY

http://nullstyle.com/2007/10/27/how-to-build-imagemagick-and-install-rmagick-with-macports-on-mac-os-x-leopard/

23Mar/070

BUG: Found and solution

There seems to be a bug in the latest versions of Ruby on Rails that is installed on the cias web server
If you are seeing in your ruby logs:

ActionController::RoutingError (no route found to match

and from the web you are seeing:
no route found to match "/blah" with {:method=>:get}

Then you need to do the following

edit your config/boot.rb line that reads
root_path = Pathname.new(root_path).cleanpath(true).to_s
to
root_path = Pathname.new(root_path).cleanpath(true).realpath().to_s

Run: sudo cias_killruby
This script is a new feature i've made that will kill your ruby processes on the server

The actual bug report is at: http://dev.rubyonrails.org/ticket/6755

I hope this helps everyone

18Feb/070

Creating a Rails Application on cias.rit.edu

**disclaimer**
I would like to thank Adam Miller very much for working with me to create an environment that can run ruby on rails, and for providing this great how-to for creating rails application on cias.rit.edu I personally cannot take any responsibility for the great work he's done documenting these steps. Bravo Adam.

Prerequisite:

You should be comfortable typing in commands to a command line interface. You should also be sure to have rails, mysql, and a server running on your local machine for development. Your rails app will run in production mode on the CIAS server, so you will not be able to access many of the debugging tools that development mode offers you.

That sound good? Alright, here we go!

Step 1: Creating the Apps Directory

For reasons of security and flexibillity, I'm going to recommend creating a directory for your applications to live in. We don't want to have your apps live in a public facing directory so lets create it as low as we can go (your home folder).

[code]
cd ~
mkdir apps
cd apps
[/code]

Step 2: Setting up your app

We're already to go now just call the rails command to build your new application:

[code]rails your-apps-dir[/code]

This will create a brand new rails app for you. If you already have been developing your application locally, you can also deploy your application here. Hopefully your using subversion to manage your files and you can just checkout a version of your app:

[code]svn co path-to-repository app-name[/code]

If not you could use ftp to get your local files up on the server, although I would not recommend this method at all.

Step 3: Set permissions

By default, the permissions will be set incorrectly, and we can't have that.

[code]cd ~/apps/your-apps-dir/public
chmod 755 *[/code]

I've also found that if your generating a new rails app, your tmp directory and its contents will have incorrect permissions as well.

[code]chmod 777 tmp/* tmp[/code]

Step 4: Create the symbolic link

Create a symbolic link to your apps public directory. For this tutorial, I'm assuming you will want to run your application from the apps directory we set up in step 1.

[code]ln -s ~/apps/your-apps-dir/public/ ~/public_html/app-shortcut-name[/code]

If you want your application to run at your root level, then replace your public_html folder with your symbolic link.

[code]mv ~/public_html ~/public_html_backup
ln -s ~/apps/your-apps-dir/public/ ~/public_html/
[/code]

Step 5: Edit your .htaccess file.

In your apps public directory, edit the .htaccess file using your favorite editor. Because your app is not going to be running at the root level (if it is, you may be able to just skip this step), you'll need to tell the server how it should map your urls. Above the very first RewriteRule add this line.

[code]
RewriteBase /~your-account/symbolic-link-name/
[/code]

Again, if your running your app in your public root, just leave off the symbolic-link-name.

We also need to make sure apache knows we want to use fastcgi to run our application. Add this line to the top of your .htaccess file:

[code]
AddHandler fcgid-script .fcgi
[/code]

Then we also need to rewrite this line:

[code]
RewriteRule ^(.*)$ dispatch.cgi [QSA,L]
[/code]

to be:

[code]RewriteRule ^(.*)$ dispatch.fcgi [QSA,L][/code]

So that we make sure we're using fastcgi insead of just cgi. Your app will be painfully slow (and consume a lot of resources) if your not running with fastcgi.

Now you should be able to visit
http://cias.rit.edu/~your-account/symbolic-link-name/
And you should see the rails welcome message. If your deploying an application that you have already started, you may see an error instead of the rails welcome message. Don't worry about this as we have not finished setting up the database yet. Onward ho!

Step 6: Database hookups

You'll need to edit the production database values in database.yml. Each cias account (as far as I know) comes with a mysql database in the name of your account. So we can edit it to look something like this:

[code]
production:
adapter: mysql
database: your-account-name
username: your-account-name
password: your-accounts-password
host: cias.rit.edu
[/code]

Step 7: Production mode

As of this writing, cias.rit.edu is not set up to force your app into production mode by default. It's important to run production mode because it is more secure and consumes less resources. In the app of your rails directory, open up config/enviornment.rb and add this line somewhere towards the top:

[code]ENV['RAILS_ENV'] ||= 'production'[/code]

Step 8: Enjoy!

And that should do it! Your app should be up and running. If you have problems feel free to shoot me an email: acm6603[at]gmail.com. Google is also very helpful even though it won't be specific to this server.

19Jan/072

Sent to all webserver users

The following message was sent to all CIAS Web server users tonight:


Dear web server users,

An exciting new feature was released tonight (1/18/2007)!

I would like to make you aware of a very nice new feature that has justbeen activated on the cias.rit.edu web server.
From now on you can either address your web space via the standard:
http://cias.rit.edu/~bjcpgd/
or the new:
http://bjcpgd.cias.rit.edu/

This new address gives your web space a bit more personality anduniqueness.

Enjoy your new personalized sub domain and if you have any suggestionsor new features you would like to see, or just general feedback pleaseemail me at:
bjcpgd@rit.edu
or comment on my blog at
http://bjcpgd.cias.rit.edu

Thank you for your time

~Brad

Archives

Recent Comments

Tags

Blogroll

My Social Sites