Posted 7 months ago
I have been itching to dive into high availability clusters for quite some time now. As a network administrator uptime is a crucial part of my field. My goal in this series of blogs is to share my experience with setting up a high availability cluster using Linux systems.
Debriefing:
I will set up two virtual machines using VirtualBox. The two machines will both run Debian 5.0.2 i386 and will be on their own private network 192.168.2.0/24. The two nodes will together provide a high availability web and database service via Apache2 and MySQL using HA. With clustering an important factor is keeping data synchronized, that will be handled by DRDB.
Getting Up To Speed:
At this point I have setup my two virtual machines, Bravo and Echo, yes very original. I am using iptables on the host machine to masquerade the packets that need to go out to the Internet for the virtual machines. I have installed HA and drbd8 via the Debian packaging system. I configured HA using the Installing Heartbeat (Newbie) screencast tutorial. Similarly I configured a very simple setup of DRBD following the documentation. Now then, lets get into the details of my progress so far.
Networking Woes:
Ok, so the networking part of this mission was a slight barrier. I recall in the past I struggled to get multiple virtual machines to communicate with each other with VirtualBox. This time I seem to have planted that issue right on its ass, and I'm going to explain just how I did it.
In order to get virtual machines to network we need to change the guest operating systems from "NAT" to "Host Interface" in the "Network" settings. If only it were this simple! Well we actually need a "TAP" interface for the guest operating system to use. Lets hop into the trenches shall we?
First we need to let the kernel know wish to act like a router, forwarding packets:
Note this can also be set in /etc/sysctl.conf then it will not have to be set at every boot.
echo 1 > /proc/sys/net/ipv4/ip_forward
Next we need to create our TAP interfaces:
The interface names are arbitrary but it is important that the same name is used when assigning them to each virtual machine.
tunctl -t tap0 -u username
tunctl -t tap1 -u username
ifconfig tap0 up
ifconfig tap1 up
Now lets create a bridge interface and add our TAP interfaces:
The IP assigned is arbitrary but do make note of it
brctl addbr br0
brctl addif br0 tap0
brctl addif br0 tap1
ifconfig br0 up
ifconfig br0 192.168.2.1
Now assign each one of the TAP interfaces to each virtual machine and start them up. Assign an IP address to each virtual machine that is in the same network as the bridge interface on the host system. The virtual machines should be able to ping each other as well as the host system. Notice that they can not get out to the Internet though. For this we will need some help from iptables:
Beware that using MASQUERADE target is not technically proper with static IP addresses, however its the only way I can get NAT to work properly in this situation. The rule below should not be considered a permanent solution, I don't plan on leaving it that way at least.
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -j MASQUERADE
Now the virtual machines should be able to ping each other, and also make it out to the Internet, assuming that other barriers do not exist like firewalls etc.
I will end part 1 at this point. I will catch up on configuring HA and DRBD in part 2 and hopefully at that point also have some more information to share, for now its off to bed and preparing for Monday.
So long
Posted in Network Admin, Linux, Tech | no comments
Posted 10 months ago
I finally configured amavisd-new with my Postfix configuration. It is a simple setup using only ClamAV and Spamassassin. However I do want to implement DKIM through amavisd-new of course that requires a newer version of amavisd-new than is in the packaging system of the release of Ubuntu I am running.
Further I would like to switch my Linode to Debian over Ubuntu.
It is safe to expect some downtime soon likely tonight as a result of upgrading through two releases.
I will post a part 2 once the upgrade is complete and I implement DKIM with the details of what I did.
Posted in Website, Network Admin, Linux, Tech | Tags amavisd, clamav, debian, dkim, domain keys, new, postfix, spamassassin, ubuntu | no comments
Posted about 1 year ago
My war with Postfix wages on. However I have resolved an issue that has been haunting my email server for quite some time now!
I constantly received a mail loop error when postmaster notifications were being sent to mail.skiblerspot.net. I knew it was something simple but never spent the time to track down what was missing. Perhaps I made the assumption that if Postfix knows to accept mail for skiblerspot.net it will automatically accept mail for mail.skiblerspot.net. This is an incorrect assumption and rightly so as this would be a terrible feature.
My problem was solved by simply changing:
mydestination = localhost, localhost.localdomain
to
mydestination = localhost, localhost.localdomain skiblerspot mail.skiblerspot.net
Adding skiblerspot was unnecessary to fix my immediate issue but I figured it doesn't hurt. Notice that I could have also added mail.skiblerspot.net to my virtual mail boxes database. I think in my case adding the hostname to mydestination is a better solution. Though if I were expecting mail for other users and wanted to create aliases for this domain I would then add it to virtual mail box database instead. Note that a domain should NEVER be in both places (mydestination and virtual mail box database).
This page notes on postfix configuration helped me realize my mistake.
Sean
Posted in Network Admin, Linux, Tech | no comments
Posted about 1 year ago
I have managed to get Gentoo running on my Acer Aspire 5610 laptop. Here is an overview of the specs:
- CPU: Intel Core Duo 1.86Ghz
- RAM: 2GB PC25300
- HD: 160GB SATA 3G
- Network: Intel Pro/Wireless 3945ABG, Broadcom 44xx 10/100 Ethernet
- Removable Storage:Unknown 4-in-1 media card reader
I installed using the Gentoo 2008.0 i686 Live CD. I am wishing I would have taken the time to compile my own custom kernel but that can always be done later on. Anyway here are some tips to help anyone with the Acer 5610 series hoping to get Gentoo installed. I only ran into a couple of easy-to-fix problems after the install completed and I was booted into the system.
Getting Networking...Networked...
I had to manually insert the b44 module using
sudo modprobe b44
to get my wired network card to work properly. Before doing this eth0 did not exist and caused a few errors to spring up during boot up. This module can be loaded automatically by adding b44 into /etc/modules.autoload.d/kernel-2.6 as documented here.
Wireless networking has not been tested yet but I believe the 3945ABG is well supported among distributions. It seems to be recognized and I know there are modules for it. If you are having trouble try the following commands.
lsmod | grep 3945
If you don't see anything promising from that then try to find the appropriate module.
modprobe -l | grep 3945
Try inserting the modules that the previous command spat out, you only need to type in the module name not the extension nor the full path, look at this example.
modprobe iwl3945
Graphics Adapter Support?
I was very surprised with this one. Oddly letting Xorg -configure generate an xorg.conf clearly found an Intel 950 GMA integrated graphics controller. The xorg.conf looked pretty good, at least for now. However I got errors when trying to start X (errors will be posted later). I actually needed to also load a module for this as well, I forget the name of the module you can probably find it yourself though.
modprobe -l | grep intel
After loading that module my X server started up fine.
My laptop is not here, which is why some of the above information is not complete. The system was still updating before I left work and I wasn't about to pull the plug to the Inter Web. I will come back to this and update the information once I have it back. I need to see how the multimedia card reader works out so I will update with that info.
It might not be long before I have to take the Ubuntu stickers off the laptop and replace them with Gentoo counterparts.
Sean
Posted in Network Admin, Linux, Tech | no comments
Posted about 1 year ago
I have been wrestling with Virtual Box for the past couple of days trying to get a small network of virtual machines communicating using the host machine (Kubuntu 8.04) as a router. Have been able to get the guest machines to communicate to the host machine and even to my server then on to the Interweb! Routing through Linux is so much fun and not hard to configure.
I often get stuck on a very simple and humbling problem. For some reason I always forget that the machine(s) being routed to will need to know how to get back. It happens to me a lot. I have been banging my head on the desk wondering why I can't get to 10.23.23.1 but I can get to 10.23.23.2, because I CAN get to 10.23.23.1, but 10.23.32.1 just does not know which route to take to get back to me.
Anyway, I have the kernel set to forward packets with:
echo '1' > /proc/sys/net/ipv4/ip_forward
The host machine knows how to get to both guest machines, and they know how to get to the host machine. They are both on the same network so a route that says use 192.168.0.2 to get to 192.168.0.0/24 should certainly be enough.
If I figure out what the problem(what I am overlooking) is I will update this blog.
Up too late again...but what is the point of living life if you don't live it!
Sean
Posted in Network Admin, Linux, Tech | no comments
Posted about 1 year ago
Sorry for all the 503 errors lately. It seems to be a bug in Haproxy 1.3.12 which is the latest package available to the Ubuntu distribution running on my VPS (running this site). I don’t feel like compiling the latest version which sounds to have a fix. Instead I will be dropping Apache2 → Haproxy entirely, in favor of Nginx. This suits my VPS much better, the current setup is total overkill anyway.
I’m really excited about this, and it should greatly increase the response time of the site. I hope to have this going before the weekend so I can spend some time adding features and fixing code for the site.
Posted in Website, Network Admin, Linux, Tech | no comments
Posted about 1 year ago
I spent some time today and setup email on the new server. Much credit given to this how-to. It is now that I’m starting to fully enjoy my Linode. I basically have Postfix as the MTA and Dovecot performing IMAP/IMAPS and SASL. I am hitting the ceiling by now with memory usage, but I may be looking at taking Apache out of the picture in favor of something a bit smaller. Either Lighttpd or Nginx, leaning towards Nginx, if for no other reason than to just try something new.
Before all that I want to polish out my email setup by adding virus scanning solution which I haven’t decided on yet. I also will probably add postgrey but I don’t know anything about its resource usage yet, or even how it operates so that is another topic to research.
I also need to reconfigure the firewall so I can accept pings while resisting DoS attacks and the like. I actually took the time for once to write a SysV startup script for iptables. If I get some time to revise it and clean it up perhaps I’ll make it publicly available in case anyone wants to look at it.
Well I have plenty to do and as usual will pretend there is a chance of me going to bed at a good time tonight!
Sean
Posted in Network Admin, Linux, Tech | no comments
Posted about 1 year ago
I recently killed Mandriva Spring 2008 on my home server in favor of Debian based Ubuntu Server 8.04 64bit. Now of course I have an automated backup system, which is not great but it certainly gets the job done. My system was and to this moment is using the dump tool, in a very archaic manner with cron jobs. Its very simple but not all too efficient, I do a monthly level 0 dump, weekly level 4 dump, and a daily level 8 dump.
Here is the actual script, it just gets run from 3 different cron jobs:
#-----------------#
# Variables #
#-----------------#
Date=`date +%F`
MonthlyDir="/artemis/backup/linux/server/monthly"
WeeklyDir="/artemis/backup/linux/server/weekly"
DailyDir="/artemis/backup/linux/server/daily"
MonthlyLog="/artemis/backup/linux/server/monthly/${Date}.log"
WeeklyLog="/artemis/backup/linux/server/weekly/${Date}.log"
DailyLog="/artemis/backup/linux/server/daily/${Date}.log"
#---------------#
# Checks #
#---------------#
if [ "$UID" != '0' ]
then
printf "`date +'%F %H:%m:%S'`\nServer backup failed with:\tUser $USER is not root!" \
| mail -s "Backup failure!" sean@localhost
exit 1
fi
if [ ! -d /artemis/backup/linux/server ]
then
printf "Server backup failed with:\nDestination drive, artemis not mounted or path invalid!" \
| mail -s "Backup failure!" sean@localhost
exit 2
fi
for e in ${MonthlyDir} ${WeeklyDir} ${DailyDir}
do
if [ ! -d ${e} ]
then
printf "Server backup failed with:\nDirectory (${e}) does not exist or is not mounted!\n" \
| mail -s "Backup failure!" sean@localhost
exit 2
fi
done
case $1 in
#-----------------------------#
# Monthly Backup #
#-----------------------------#
--monthly|--full)
# Do dump on /home
/sbin/dump 0 -u -A ${MonthlyDir}/home_${Date}.archive \
-f ${MonthlyDir}/home_${Date}.dump -j9 /home 2>&1 >> $MonthlyLog
# Do a dump on /
/sbin/dump 0 -u -A ${MonthlyDir}/root_${Date}.archive \
-f ${MonthlyDir}/root_${Date}.dump -j9 / 2>&1 >> $MonthlyLog
;;
#-----------------------------#
# Weekly Backup #
#-----------------------------#
--weekly)
# Do dump on /home
/sbin/dump 4 -u -A ${WeeklyDir}/home_${Date}.archive \
-f ${WeeklyDir}/home_${Date}.dump -j9 /home 2>&1 >> $WeeklyLog
# Do a dump on /
/sbin/dump 4 -u -A ${WeeklyDir}/root_${Date}.archive \
-f ${WeeklyDir}/root_${Date}.dump -j9 / 2>&1 >> $WeeklyLog
;;
#-----------------------------#
# Daily Backup #
#-----------------------------#
--daily)
# Do dump on /home
/sbin/dump 8 -u -A ${DailyDir}/home_${Date}.archive \
-f ${DailyDir}/home_${Date}.dump -j9 /home 2>&1 >> $DailyLog
# Do a dump on /
/sbin/dump 8 -u -A ${DailyDir}/root_${Date}.archive \
-f ${DailyDir}/root_${Date}.dump -j9 / 2>&1 >> $DailyLog
;;
#-----------------------------#
# Help Menu #
#-----------------------------#
--help)
echo -e "\nThis script is used to backup Brutus using the dump command.\n"
echo -e "It is truly intended to be used in conjunction with Cron.\n"
echo -e "Valid options are as follows:\n
Option: Meaning:
------------------------|-------------------------------------------
--help | Show this menu.
--monthly|--full | Make a full backup.
--weekly | Make an incremental weekly backup.
--daily | Make an incremental daily backup."
;;
#-----------------------------#
# Anything Else #
#-----------------------------#
*)
printf "Server backup failed with:\tInvalid or missing option!" | mail -s "Backup failure!" sean@localhost
echo "server_backup: Invalid or missing option: please see --help"
;;
esac
I have read a little about The Tower of Hanoi but never thoroughly enough to completely understand it, let alone use it. This also seems to be geared more toward admins who are using tape to backup their systems where I simply use a hard drive.
Anyway, since I installed a completely new distribution I used nothing but a few server configs from my backups. So I didn’t really have to do any overly complex restoration. Matter of fact I pretty much just dumped the most recent monthly backup and pulled what I needed out and that sufficed fairly well. Until I came to subversion! Duh, duh, duh!!! Obviously that did not work very well and I have been holding off on figuring it out, and because my svn repo was very bare, I nearly just started a new one instead. However, I am always up for a challenge and could not help myself with this one.
It was actually really easy, but restoring from 20 different backups really wasn’t much fun! Anyway I started with my oldest monthly backup, and went to the last. Then began with the weekly backup dated just after the last monthly, and again went through to the newest. Then finally restored the remaining daily backups in the same manner.
My restore command looked like this:
sudo restore -x /svn -f /artemis/backup/linux/server/daily/root_2008-11-13.dump -T /
Yeah, I was restoring di rectly to the svn directory in root rather than /tmp, so what!? I probably could have taken extra time and scripted this, or perhaps just have a better backup rotation. However I think after actually having to use my backups, I will be changing more than just that. I think I am going to go to rsnapshot instead of dump. It just seems much cleaner and simpler to use. I will blog about it once I have it configured.
Well I’m off to bed at 3:00AM once again! I need to get a good workout in tomorrow so I’m going to get some rest, and hammer out chins and pull-ups tomorrow until I drop.
Good night,
Sean
Posted in Network Admin, Linux, Tech | no comments
