Tuesday, July 31, 2007

OUCHIES! I broke my big toe this morning!

I broke my big toe. I went to the hospital and had them X-RAY it. It's broke!

I was carrying my son (15 months old) down the steps, and I slipped. My only though was "Don't let Jason get hurt."

So I grabbed him and wrapped my arms around him, as my left foot missed a step, and my right foot slipped off the step it was on, hitting the step below that toe first, which toes consequently folded underneath that foot at the same time as they became the primary weight bearers for all 250 lbs of me.

Jason was not hurt. I think he was scared that daddy was screaching like his 19-month old cousin Jade when they're fighting over a toy (actually he's the screacher, not her), but he was fine.

Here's a pic of the X-RAY:

After I hurt myself, I took about 5 minutes to gather my wits, then I took Jason to daycare, and drove myself to the hospital, which is quite pleasant at 8:45 AM.

A few X-RAYS, and a silly post-op shoe later, and here I am on a diet of Advil and ice-packs. Hopefully the bones won't need to be pinned in place, but I won't find out till the end of the week, when I have a follow-up with the orthopedic specialist.

Wednesday, June 27, 2007

Using Solaris 10 Update 3, Sun Cluster 3.2, Zones, & ZFS in a Multi-Node Cluster of Sun Fire T-2000s

It all started with a conference call with one of our customers. We wanted a way to set up some highly available systems that could be used for various beta or QA purposes, or production services, or anywhere in between as needed. We also wanted a way to maximize the resources. We had 4 servers available to us, all Sun Fire T-2000s. If we used them as straight servers, they'd be great at anything they do, right? 8 cores, 4 threads per core, 32GB of RAM. Nice. Capable of running dozens of zones without skipping a beat. Perhaps even hundreds of zones.

Zones make perfect development boxes, right? You can blow them away and re-install them in a matter of minutes, or even seconds on ZFS. Zones also do pretty good as production environments as well. We're currently using a large number of zones in production, to supply a variety of services.

Zones on ZFS make particularly good dev boxes because you can take frequent snapshots and roll back as desired.

** Zones with their zoneroot on ZFS do encounter bug #6356600, which relates to how the live upgrade scratch zone used for installing packages into local zones can't access ZFS filesystems to upgrade zones with a ZFS zoneroot.

Sun Cluster 3.2 introduced support for ZFS as a failover filesystem, and for failover zones as well. We decided to make use of both of these features.

We built a 4-node cluster out of the 4 T-2000s, and began exporting individual disks from our SAN. Put 3 disks in a ZFS pool as a raidz filesystem, and installed the zone root at a ratio of 1 zone per zpool. (We're still doing some testing with our SAN and comparing performance of ZFS on individual disks, or ZFS on a RAID5 LUN exported by the SAN, but so far the way we're doing it is working nicely.)

So we built the cluster, and got it all configured and running. We then installed the first zone onto the ZFS pool. Then I copied the relevant portions of the zones configuration (in /etc/zones/*) to the other nodes in the cluster.

We then created a resource group in Sun Cluster 3.2, and added the zone into the resource group. We also added the ZFS pool into the resource group as an HA-Storage resource, and created a quick set of control scripts to start and stop the zone. The zone itself takes care of bringing up it's ip addresses, and starting the various applications installed within.

End result: Highly available servers, on a failover basis, that take less than 30 seconds to fail over from one host to another.

So far it's working really well. We're already getting more requests to build more of these multi-noded clusters, with zone/zpool combo's as the resource group. It's been a great solution for us.