Are you ready for Amazon S3...Maybe?

This is sort of a continuation of my previous post about Amazon S3.

While was looking for a way to use Amazon S3 without doing any coding, I ran into a couple of beta programs. Some of which just didn't seem to work right. I was looking for a program that would copy the files to S3 daily and wouldn't change the name of the files. JungleDisk was one of the first I looked at and it would constantly be doing a get and put while uploading the data along with changing the file name (if you used only JungleDisk you won't realize this). As mentioned in my previous post, S3 backup and S3interface.com were good candidates but they didn't do automated backups. S3interface limits the file size to 10MB and you can only upload from within the web GUI. S3 backup on the other hand with the release of S3 backup beta 12 the automated feature now works and I have been using it successfully for 2 months.

I have S3 backup installed on my "monitoring" server (which monitors all of my web/sql servers) where the external HD is directly connected. So after my backups run I scheduled an upload within S3 backup to upload the changed files. I also installed this program on my desktop. Since I can connect to the same bucket I can see if, what time and which files where uploaded. It works great!

The only problem I had with this program was when I created two jobs. I was trying to use multiple buckets to store the files for the different servers that I backup. When I would run the second job it would put all of the files in the first bucket and overwrite the files from the previous job. I worked around this by creating a single job using one bucket with multiple folders. As a consequence, this is much easier to view all of the files at once. So when reviewing my backup files I can quickly look at all of the files quickly.

Since this has been working so well I have started to think about cancelling my other online off-site storage provider. With Amazon S3, their costs are so cheap that I am paying about $60-70 per month for about 30GB. This includes the daily uploads of the differential files (between 200MB and 3GB) and full backups of about 26GB. My current provider gives me 500GB for $600 per month. Although this is more space the cost savings is nearly 10 fold since with Amazon I only pay for what I use and I don't use all 500GB.

In summary, I have found that S3 backup is the best program (so far) out there to upload files to S3 without having to create your own. Also a great way to take advantage of Amazon S3's cost savings and data center reliability.

Are You Ready for Amazon S3?...Probably Not

Having only discovered Amazon S3 a couple of months ago I was totally blown away by the idea. For those of you who don't know what this it, it is storage space in Amazon's data center that is rented out on a monthly fee (and transfer fee). As soon as I read about this I started thinking about the current online/off site storage vendors that I was using. I couldn't believe that Amazon would offer this "service" at such a cheap price.

I started researching everything about it and how I could start using it. The catch is that you must use it by utilizing their APIs. Meaning that you just don't sign up for the service and download their program, you need to write you own program or use someone else's open-source program which is what I did. Since there are so many open-source type programs out there you can just pick the one that you think is the best for your needs and use it. (On a side note, there are also some "pay-for" programs and some online storage companies have moved their infrastructure over to Amazon and are already offering this as a "storage backed by Amazon" type of service.)

The problem I see is the maturity of these open source programs. If I use one now for the next year and the person (or company) stops development of it, then what? A program like JungleDisk changes the file names for some reason when the file is uploaded. If development stops on this program you may need another program to interact with Amazon S3, renaming all of your files could be an arduous task. The other programs that I have used S3backup and www.S3Interface.com both keep the file name unchanged. But I want to be able to automate the file transfer and with these programs this task is more of a manual one.

On another note, I believe that S3 will revolutionize the way we think about storage all together. It will turn storage into a commodity and something where a physical location is not important. Backing up data in one location and restoring it at another may not be a problem in the not too distant future.

Here is the link to Amazon's site if you want to read more about it. www.amazon.com/s3

Spiceworks - Network Management System

I found spiceworks while reading one of my RSS feeds about 2 months ago. Spiceworks is a network management program that is completely free and just about as robust as much larger and much much more expensive ones.

After reading about the program in the article and checking it out on their website (link here) I decided to download it and give it a try. The interface is web based. After some simple configuration to get it to work on your network it finds all of the devices. It will categorize and label everything. The program will tell you about the hardware and all of the specs that you need for your hardware inventory. It finds all of the software installed and can provide you with reports about who has what installed and whether or not you have enough licenses.

Spiceworks also includes a helpdesk function. I just set this up and I was worried that I would need to create individual users manually (it doesn't have Active Directory integration). But once the user goes to the helpdesk site and enters in the ticket spiceworks will associate the hardware device where the the user filled in the ticket. The username, computer name, IP address, and everything else that spiceworks already has in its database about the hardware device will show up. So far it is working great. I only wish that I could get the money in the paycheck for what I saved in a Network Management System.

HP Integrated Lights-Out 2 (iLO) - Management Network

My past experience was unfortunately with Dell servers. I could never convince my old employer that HP servers were better. I guess they didn't care or didn't mind that I needed to drive to the data center to reboot a frozen server in the middle of the night. At my new job with uptime being so important the obvious choice for servers was HP. Compared to Dell...well there is no comparison. Dell is always a generation behind on their management programs. I had limited experience with HP servers at some of my old clients and I could see that this is the way to go.

There are several versions of the iLO software and every HP Proliant DL server comes with the basic version. The basic version gives you virtual power switch control. You have a couple of options whether you want to just press the power button or press and hold for several seconds. The basic version is what I am using for now. My main concern is rebooting a frozen server at 3:00am without having to drive to the data center ;). The advanced version gives you full KVM access even to the boot screen. I haven't recommended to management the advanced version yet, but I can see that having it would be very useful.

The way I set up my server environment was with a separate private "iLO network". The network is totally closed with no access to the Internet. I have a separate server which I use for monitoring my production servers. (See my blog "My Latest Project" for a description of the hardware setup.) First I setup all of my production servers with an iLO "private" 192.168.0.0/24 network address. You do this when the server boots, press F8 when you see the iLO prompt. There are several options in this menu. You need to turn off "get a DHCP address" and manually input your private IP address. I chose a simple sequential numbering scheme starting with 192.168.0.3 for my servers (.1 and .2 are for my two firewalls). 192.168.0.9 is my last IP address which is for the monitoring server. I actually probably can never access this server when it crashes with iLO since my only access is from this same server.

Here is my monitoring server NIC setup.

NIC 1 - front-end IP address
NIC 2 - back-end 172.16.0.0/24 address
NIC 3 - iLO network 192.168.0.9
NIC 4 - iLO management 192.168.0.8

From this server I can see all of my other servers with the iLO web interface. As long as this server is up and running I will be able to reboot and control my production servers.

The NICs of my other production web/sql servers are setup in a similar way except they only have the front-end, back-end and iLO management NICs. These servers can't actually communicate over the iLO network they only listen for commands from the monitoring server. Within my production setup (photo of my firewall/switch setup here) I don't include my iLO switch. Since the production network can't have any downtime I bought a pair of high-end switches. But the uptime of the iLO network is not as critical so I just used a cheap workgroup switch. If it fails then no big deal. I will just go buy another one and pop it in during business hours. I think the risk of this switch failing and one of the servers failing at the same time is very low.

That's it...the iLO network is setup. I put some favorites in my web browser so I could quickly access each server without having to remember the IP address. Also for an cool effect use different color Ethernet cables for each network. I personally try to use a different color for each one; one for the front-end, one for the back-end and a different one for the iLO network. Makes it easier when it is time to move on and the new sys admin is trying to figure out how the system works...but we don't care about him/her now do we???? :)