Linux server maintenance


For a server that was put in place mid-2007 and remained in service with only three or four reboots until being taken offline near the end of 2010, it has performed quite well. I was asked to perform some maintenance tasks on it, with the expectation that it will be reactivated and placed back online within the next month or so. Here are a few retrospective thoughts that arose during the routine:

The hardware was a solid choice. I had already used the chosen motherboard and identical RAM sticks in my own desktop machine and verified their usability and reliability. When I upgraded my desktop’s RAM to 4 GB, I simply bought new sticks instead of maxing out the available slots, since by that point, the original sticks were no longer sold. This left me with a spare set which I installed in the server as a last step. The hard drives were also a good decision, even if they do tend to run hotter than my own comfort level (hence the heat-sinks and fans).

On to software. Arch Linux is still my OS of choice for servers, mainly because I find it quick and easy to deploy. Plus, if I use the same OS everywhere, I can share configuration files and even binaries more readily, as well as reduce my own administrative learning curve. However, there were a few things that I did differently with this system. I decided upon a custom kernel, which turned out to be a headache when it comes to upgrades. Arch Linux likes to be up-to-date before installing any new software, since it is a rolling distribution and two months down the line dependencies may differ. Since it was a system intended to have guaranteed access at all hours, and wanting to reduce maintenance costs, I had decided not to upgrade the software on the operating system. In practice, this was a good thing, but when it came time to do maintenance more recently, it was a bit of a headache.

Security is another area in which this server performed mightily. Although it may have not been intentional, any and all passwords that I had set on my account (and on the root account) had long since been forgotten. Only the .htpasswd and userland passwords were ones that anybody in the company could recall. This required me to reset the passwords. In the process, I learned that it’s important to reassemble the array before changing passwords; otherwise the /etc/shadow file gets jumbled. Changing the data on each drive individually is not a valid solution! After rebooting several times and performing more than one contiguity check using fsck, I finally managed to set the passwords to more familiar strings. Having a separate partition with a copy of the root filesystem’s /etc folder was also an important feature that was designed-in. It made configuration using the install CD (using mdassemble) much quicker.

The kernel remaining at version 2.6.23 became problematic after running a full system upgrade. Since I had GRUB configured to only boot my custom kernel, and since my custom kernel would not be recompiled by the automatic system update using pacman, I was left with no choice but to burn an installer CD to get the system back up and running. The process taught me how to reinstall all packages in the system (just in case glibc or some other major component got zapped), and revealed to me some of the nuances that took place over the years of Arch Linux’s evolution, such as the way RAID is handled at boot-time by an initrd. I also learned that the operating system itself can stripe swap data across several physical disks, reducing the need for RAID configuration on the swap partitions. This of course means that instead of one mirrored 0.5 GB partition, the OS can now stripe over 1.5 GB on three separate disks. In principle, it should be way more efficient.

All said and done, the server is ready to be redeployed; software has been fully upgraded, and it even has more memory than before.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: