"Linux Gazette...making Linux just a little more fun!"

Editor's note: In the original article on Adler's website, many of the inline images display a larger copy of themselves when clicked on. These larger images are not included in the Linux Gazette version, to keep the total size of the Gazette small.

A rainy day in New York City, and I'm hustling around looking for a parking lot.

Gucci bags and Rolex watches for sale abound. Where are the damn umbrella sellers!

It's 6:20pm and I'm heading up 3rd Ave in search of a parking lot. The meeting starts at 6:30pm. Damn, I'm always running just on time. With little effort, I was able to find a very reasonable parking lot which charged $10 'till closing. It's usually about $25 for a mid town parking lot. I dropped the car off, and dash out in search of the IBM building where the NYLUG was meeting. Rain is coming down, I'm getting soaked, and I'm looking all over the place for those street vendors who always have what you don't need at the time. Fake Rolex watches were up for sale, as were Gucci bags, but no umbrellas. I could feel the rain starting to seep onto my scalp as I ran across Madison, heading north towards 57 St.

IBM, a while back, started to get a clue about the benefits of Open Source/Free software and has now donated one of their meeting rooms for the NYLUG, who meet about once a month. (Rasterman is talking at the next one.) The IBM building stands very tall on the corner of 57th and Madison. It boasts some modern work of some sort at its entrance. One needs to sign in, in order to be let into the building. The meeting was being held on the 9th floor.

I arrive to the meeting room where NYLUG is gathered. A projector is setup with Donald's laptop plugged into it. There are about 30 or 40 people present. Jim Gleason, the organizer of the meeting, who works for VA Research, is there talking with Donald, looking rather busy. He sees me and introduces me to Donald. I have just drove in through about 2.5 hours of LIE traffic, dashed across several streets and avenues under the rain, and my bladder had been screaming at me since exit 40 on the LIE that it needed to be relieved. I couldn't concentrate much on what I was saying at the time. I shook hands with Donald, and muttered something like, "We use lots of computers at BNL". I don't remember how he responded, I think he didn't say anything. I then managed to get myself away, find a seat, store my laptop and look for a good place to take a photo of the room.

A shot of the NYLUG meeting room, courtesy of IBM. By the time Donald's talk was well underway, there was basically standing room only.

Jim Gleason took the mike, and called on people to sit down. He wanted to get the meeting going on time, (it was getting close to 7pm by now). I settled down into my seat, booted my laptop, and proceeded to ignore my aching bladder. I had more important business to take care at the time.

A solemn moment for Jim Gleason, the VA Research guy who is one of the NYLUG contacts and organizers. Actually, the shot was taken as he happened to look down at his notes. The only time he did so during his introduction. Murphy's law is at work here. Jim is a very energetic guy who is excited about his work.

His introductory slide showed two things. His affiliation with the NASA Goddard Space Center and a company called Scyld Computing Corporation. My guess is that he has been given the liberty at NASA to work with this Scyld startup to help bring the Beowulf into the private sector. Good for him. At this point, something rather annoying started to happen. The projector which was hooked up to Donald's laptop, started to lose sync with it. Donald, who has a bit of shyness to himself, was having a hard time giving his talk while at the same time, pressing various auto sync buttons on the projector to try and get his transparencies back up on the screen. This went on through his talk. It really didn't matter since he didn't bother to walk through his slides, rather he just talked from the top of his head about what he as been doing for that past 7 years.

Donald's talk went on until 8:20pm. During that time I got the following out of his talk.

A bad picture of Donald at the beginning of his talk. It looks like my camera is getting out of sync with the fabric of space-time. (One of these day's I'm going to upgrade my analog camera to a digital one. But with the increase in property taxes out on Long Island and the small salary a "junior" scientist makes at BNL, it will be some time be for I do so.)

Donald talked a bit about the open source nature of the project. In short, being an open source project was crucial in making it as reliable as it is. This also holds for the Linux kernel itself. While working on building Beowulf clusters, Donald ran across some problems with the Linux kernel which he had to fix. Things like only being able to mount 64 file systems got in his way. Having hundreds of PC's talking to each other on the network stressed the networking data structures in the Kernel which he also had to deal with. Being that he had the source code to the kernel he was able to make the Beowulf project work. He also took in contributions from outsiders. If the contributed software was of relevance and worked well, he would include it.

The side of the IBM building, as I face Madison Ave. Thank you IBM for letting the NYLUG use your meeting rooms so that we can hear Donald speak. Although it would be nice if you guys got a Linux friendly projector. Its OK if the projector is not Y2K certified. We'll take it any way.

Donald spent some time showing pictures of current Beowulf clusters in operation. Some were rack mounted systems, some were bunches of PC's on a shelfs. The PC's on shelfs Beowulf system is called LOBOS which stands for Lots of Boxes on Shelves. One of the systems built in the 19 inch racks was called the hive due to the noise the large cabinet fans made.

The art work standing at the entrance to the IBM building. Unfortunately, I can't tell the difference between this and a bunch of steel beams welded together.

Some of the specifics about Beowulf clusters he mentioned were the following. Usually one of the nodes is set aside and dedicated to managing the rest of the nodes in the cluster. It's the job distributor. Some very simple techniques are used to keep track of which systems have not crashed. A multicast coming from each machine is received by the controlling node or it pings the rest of the nodes in the cluster. If one of the nodes goes down, the controller quits submitting jobs to it. There are some mechanisms within the Beowulf software for process migration from one node to another. He also talked about how he uses RPM extensively to maintain the software on the nodes. He referred to RPM as a "key technology" enabling the easy maintainability, software wise, of large clusters of PC's. A question came up asking about how he maintains his Beowulf code. He didn't answer the question very well. He didn't really want to answer the question since he did not want to endorse any kind of source code management software like rcs or cvs. But he did stress that RPM was key in order to be able to distribute software to may PC's.

Who's that imposter! (I've gotta' upgrade that damn camera...)

Donald also spent some time talking about how one runs a Beowulf cluster and keeps it reliable. This is done by monitoring the hardware for possible future failures. The most common one is due to failing fans. There seems to be a host of applications which monitor system performance, from the temperature of the boxes, to network packet error checking. Keeping an eye on these problems helps keep a Beowulf cluster healthy.

Donald answering questions after his talk. I nice shot of his left back side.

One person in the audience asked why Donald used Linux as the kernel for building up his Beowulf project instead of one of the BSD kernels. Donald had an interesting answer to that question. First off, the BSD kernels were not as stable as Linux, back when he started working on his project. He then proceeded to complain that working with the BSD developers was very difficult. They tend to hide the development process thus making it harder to contribute the needed upgrades. (Remember that Donald had to work with the internal data structures of the kernel in order to make his project scale.) He then said that these BSD developers had very large egos. "Their ego's would fill this room" he said, thus implying the difficulty of working with them. He then went on to say that he was quite able to work with Linus. Linus was a laid back guy.

Another shot of Donald left backside. Although I'm starting to work around towards his front. If I'm lucky I may get him looking into the camera.

There were many other interesting questions which were discussed during Donald's talk. You can read my jumbled notes if you care to try and decipher them for more information.

Well, that's as far forward as I could get. Although I did get a nice shot of him and his book which I proudly display at the top of this write up.

People got up and started to leave the room after the talk was over. I stuck around to take some pictures of Donald as he talked to some NYLUGers. I eventually was able to get a chance to re-introduce myself to him. I have him my card and invited him out to BNL if he were ever in the area again. (I'm really bad at this sort of thing.) I then asked him if he had trouble getting funding for his first Beowulf system. He told me that he got the Beowulf idea back when he was working for the NSA. He presented the idea to his superiors. He needed $50K to put a cluster of nodes to together. For the NSA, $50K just too little to bother with and his request was declined. So he took his idea over to NASA. NASA thought it was worth funding, so he got a job there specifically to work on his Beowulf idea. The rest, as they say, is history.

My last shot of Donald as we start receiving our dinner orders. I was fortuitous enough to take this shot just as the waiter held Donald's plate right over his mouth. It is truly amazing how often Murphy's law kicks in. The guy looking right into the camera is named Judd. He works for Netscape and announced at the NYLUG meeting an install fest he was organizing.

I left the room and spent some time with Jim Gleason in the hallway just outside. VA Research is about to deliver a 36 node system to BNL this coming week and we talked about that a bit. Suddenly, my bladder screamed at me and I broke off in mid sentence, "Where's the men's room!". To my fortune, it was about 10 feet behind me. I don't know how I was able to ignore my bodily functions from exit 40 of the LIE until now...

A picture of the other table where the 2nd half of the NYLUGers hung out while waiting for their food to show up.

I sat across one guy who seemed to do business with wall street. One tidbit which I found rather interesting was that he had this friend who put systems together for wall street trading firms. One would assume that these systems are bullet proof; 100% reliable. It turns out that they crash all the time. There is enough redundancy in these systems so that these crashes can be afforded. After hearing Donald talk about large numbers of systems being up for 100's of days at a time, and then hearing that wall street trading systems crash continuously was a real shock. Maybe wall street will begin to understand the power of Open Source. Until then, my retirement fund will not be as safe as it could be.

Another shot of Jim Gleason along with Matthew Hunt and Ari. Ari is the guy in the back who also works for VA Research. He's coming out to BNL to setup the 36 node machine I'm aching to submit my jobs to. The guy in the middle is Matthew Hunt, President of the Linux Users of NY group (LUNY).

Driving along on the LIE always leads my mind to wandering in thought. This time, my mind wandered around open source land. I still cannot get a grip on the power of the Internet. What really made Donald's project possible was the fact that he had access to Linux. You could never build a Beowulf cluster out of windows 3.1 machines. Think about it, this is what was running on those powerful 100MHz DX486 machines back when he started this project. I can imagine going to one of the NSA administrators and trying to convince him that you could take all those PC's the secretaries were using to write up memos using MS Word, gang them together and turn them into a super computer. And do so for only $50K. Back in 1992, that was a radical idea! And look at what we have now, super computers popping up and the beginning of a new industry. Also, has anyone ever heard of an NT Beowulf cluster? I'm sure Micro Soft would boast of one if there was one. (And take credit for the idea as well.) That would be a good way to test the stability of NT. Run 100 NT machines in a cluster and see how long you would keep them all up and running. It would be nice to see Mindcraft perform such a feat. Having 100 Linux machines running for over a hundred days translates to 10,000 cpu days of continuous running. Benchmark that Mindcraft...

Exit number 67, exit number 68. Ahhh, exit 68, home at last.

Click here if you want to read other articles I've published on the Internet, or click here to view my home page.