Somehow I managed to squeeze myself and Free Geek Vancouver into today’s edition of Vancouver 24 Hours. Hoorah!
Somehow I managed to squeeze myself and Free Geek Vancouver into today’s edition of Vancouver 24 Hours. Hoorah!
Free Geek Server Room Build Part 4 AKA How I learned to love LTSP, Migrate OpenLDAP and get bind running all in one day!!Posted: May 8th, 2010 | Author: jordan | Filed under: DNS, Free Geek, Insight, LDAP, Linux, Migrate, SSH, Vancouver | 2 Comments » Today was awesome! We got so much done!!! And it all went without a problem… oh except for when we tried to create an LACP bond on our NFS server and crashed the whole network… yeah… Quick story on that. we have 10 VLANs all trunking between our switches and our router. The NFS server is on VLAN 5 untagged on port 17 on the switch, we the added port 18 and created a bond on the switch. We then created a bond0 interface on the NFS server and used ifenslave to assign the eth devices to the bond device. Then….
BAM! WHOLE NETWORK GOES DOWN. Not just vlan5, no no, the whole god damn network. No Internet access nothing, not even from the router, the router can’t ping a thing on the whole network.
Why God? Why?
Then the Network God ARP said, “Jordan did you check those top kwality DLink switches?” So off I went to check the switch I just modified. For some unknown reason the DLINK decided to plunk VLAN 6 tagged onto port 16 for no reason what-so-ever. So I fixed that, but no, nothing worked still. So Tyler says, just unplugg the ethernet cables to the NFS server. Voila! Problem solved. Basically we think the NFS was just spewing out crap across the network and making all the servers in VLAN 5 bail, including the router. We were getting some pretty crazy ARP poisoning happening the router. Now, on to the good stuff.
First the LTSP client boots looking for a PXE server, DHCP is handed out and tells the client to grab a boot image from yew. The LTSP client then boots into Ubuntu 10.04 (bleeding edge baby) where the login screen is presented. The user logs in and authenticates against our new fancy LDAP server on teak. When the client logs in their home directory is handed out via LDAP as /net/home/<$user> This directory is handled by AutoFS and mounts the NFS home from maple. The user now has full desktop experience with all their account info and home directory handled by 3rd parties.
Our LTSP server is now a 2 x Dual Core Xeon 3.20GHz with 4GB of RAM. A HUGE upgrade from what we running before. In addition to all this insanity Vicki was able to migrate our ticketing system for us as well as update all the LDAP records for home directories, install autofs on all servers, install the mount maps, comment out all the irrelevant fstab crap AND switch over all our servers to the LDAP server. Pow vicki, pow!
The backup system is pretty sweet as well. In our NFS server that holds the home directories is two RAID 5’s, a RAID 1 and some spares. One RAID 5 has a slice out of it that is for home directories. The other is 1TB for nothing but backups. What Tyler did was write a script that uses rsync to create incremental backups all done thru hard links. It’s pretty rad.
Now that I come to think of it, I didn’t really do much except play with the dogs!!
Yes, yes, I know. Two months have gone by since the last entry about Free Geek. Well finally I had some time to make it down there and to my enjoyment though Tyler from Free Geek had been busy at work. He managed to do quite a bit of work while I was away. Here’s a pic and some highlights!
Now first I must mention something that happened which was spectacular. I showed up to Free Geek with tools in hand ready to kick ass and chew bubblegum. I said ‘Hi’ to the gang and then got right to work going over what’s been done already and what we should do for the day. Then I heard a small voice coming from behind me. It all started with a simple ‘Hello.’ Tyler and I turned around and here stood this lady, she said that she had heard we were doing updates to the network and wondered if she could help. I have something to confess here, I judged at first sight. So my initial response was ….. uhhhhhh….. and in my head I was thinking “oh god I have SO much to do today, I can’t possibly teach and babysit someone else.” However we said ok you can help
“what’s your name?” I asked
“Vicki” she replied.
I said “OK, Vicki, I’m going to outline on this whiteboard what we hope to accomplish today.”I then began drawing out network topologies and what VLAN’s we were going to roll out that day. Tyler pulls up a network diagram I had done up briefly a few weeks ago to talk about subnet allocation and service assignments. All the while Vicki was quietly watching and listening. We then went about which of our new servers would be responsible of what task, such as “teak” was going to be our new LDAP and DNS server, maple the new NFS server, how authentication was going to happen for autoFS mounts and so on. Granted if you’ve been in this industry for a while this isn’t super complex stuff, LDAP migration, network topology planning, thinking ahead for future departments, etc etc. However, this isn’t childs play either, let’s be honest there are a lot of ‘sys admins’ out there and not all of them could roll out a network of this size.
We turned to Vicki and started going thru the tasks on the board, expecting (I was anyway) to see a lot of confusion. BUT NO! OMG! She knew just as much, if not MORE about this stuff than we did. In fact, over lunch we got into a discussion about proper use of VLAN’s and subnet routing between them. This woman was (is) AMAZING! It was like the network God looked down from heaven and with his noodley appendage, blessed our tech mecca for that day by sending us a worker! A worker that new how to install services, write config files, test connectivity and map VLAN’s!!! Quite literally she cut our work time by 40% if not more. If anyone needs a good sysadmin, or network engineer who knows their way around a linux terminal and learns by being shown ONCE! Contact me, I’ll send her details on to you.
Anywho, Tyler and I laid out the VLAN’s and what they would be responsible for. We had configured three switches thus far to trunk all the VID’s but when we got to the fourth and final switch, we had no admin credentials for it. (My fault!) Our plan at that point was to wait until the end of the day, reset the switch, recover the password and then move our core router to the rack. In the meantime I checked up on Vicki and she had gotten all of our services, OpenLDAP, bind, Zenoss, apt-cache, TFTP server, and some other stuff up and running and was ready for configuration. I migrated the database from an older version of OpenLDAP with a slapd.conf file to the new version with the slapd.d directory.Once the Free Geek came to an end Tyler and I moved the router from the bathroom server room to the upstairs rack, pushed the ADSL modem thru and VLAN, and then made an LACP trunk to our OpenBSD router. Put the VLAN interfaces in place and POW. Network configured. (For the most part) The final stage is migrating the servers to the proper VLAN’s and updating their services configurations.
The next and final post will be mostly diagram based. Stay Tuned! HOPEFULLY the next post will be really insightful IF I can get Luke and Kamil from Zymeworks to donate some time into rebuilding our Asterisk server and implementing a KDC
This isn’t really a whole post, more like a failure report.
I had such high hopes when I got to Free Geek about getting the rack powered up and connected to our network. No we didn’t get the 30A circuits wired in yet but I do have one 15A that will be able to power the 15A PDU I have installed. I got that all up and running, and quite proud that one of our two PDU’s has remote power management, however when it came to configuring the network it was a different story.
Enter the crusty 3COM switch. We have this old, and I mean old, 3COM switch that I swear to God moans as it passes packets. Our 3COM has one already configured LACP trunk heading the ‘server room’ downstairs in the bathroom, what I wanted to do was setup another trunk to go to the new server rack. Nope, not happening. I fought and fought and fought with that 3COM on web interface, console interface, and smoke/blanket interface but to avail. It just would not allow more than one LACP trunk to be configured. Now I’m back to looking for a switch to replace it. There is a start-up in town that said they would graciously donate their old gigabit switch hardware, I hope they pull through on that. (If you guys do, I’ll give you a good plug here.)
While I’m waiting for the switches to magic themselves over to me I’m going to start with OS install and configuration but that will have to wait for part 3.
So over the last couple days we’ve gotten in a decent 48 port switch built by extreme networks as well as two APC SmartUPS 3000’s. After we put these in I got to work building our new trunk back to an existing switch. As you can see from the pic to the side getting this cable from point A to B is going to be a little challenging. Not only does it have to get in behind that huge shelving unit, but that huge shelving unit is covered by more shelving 6ft deep.
This was the result! My whole body pinned between a crap load of iMacs and the upper shelf, it hurt like hell but at least I got an ab workout! 😉 Eventually I got the cables fished thru, crimped and nicely zap strapped in. At the end of the day Joshua and I were able to install:
Next up will be network configuration!
The Free Geek server room is not in the best of shape. It basically consists of about 9 desktop towers of varying speeds stacked in groups of 3 with plywood in between them. Oh! and it’s all housed inside of a bathroom. 😛 The power is being supplied by APC UPS’s with missing front bezels and malfunctioning controls…. they scream randomly. There’s no cooling, or even airflow and when the toilet’s water evaporates it begins to smell like dead fish. The gateway was just replaced with a dual core Xeon (thank you Joshua,) but the old one which ran like a trooper for 3 years was a Pentium II 366. A lot of the servers are beginning to show their age, failing hard drives, dead fans, strange smells. It’s time for a general overhaul.The first thing to do is to plan power and figure out a location for the new server rack. We’ll be feeding in 2 20A circuits and 2 15A as well. We decided on placing the rack beside the lunch room because it was accessible enough to work on and also allowed to kinda show off to all the volunteers. We then got this rack graciously donated to us to house all the new server that will be going in. It’s a 42U HP rack enclosure with removable doors and panels. Very nice rack inside, I would say even better than the standard APC Netshelter because the doors come off a lot easier and it’s nice and roomy to work inside of. However the only thing I did not like about this rack was the feet. To level it off I had to get an 11/16th wrench and from the bottom lower the feet where as on the APC Netshelter this is done inside the rack with a handy little Phillips screwdriver.
Now with the location for the rack picked out and rack itself chosen we can now do my favourite part. Cleaning. The first pic is of the dirt swept up in about 15sq ft around and under the rack location. After this is done the HP rack was put in place and then levelled out.
Since this rack was donated to Free Geek it had a lot of stuff in it already, therefore the stuff needed to be removed. These pictures were taken when most of the stuff had already been removed.
Now with the rack in place and cleaned up comes the fun part! Finding stuff to put in it!!! Free Geek has a lot of stuff to choose from.Normally the first thing I would install and get working is the power and UPS, however we are currently waiting for a UPS to come in through donation and the circuits have not been laid in yet either. Instead the first thing I’m going to do is find a nice switch to install. First on the block was a Cisco 2948G. At first glance it seemed like a great choice. Lots of ports, Gigabit ethernet, however after the password recovery and seeing that the IOS version was 6.1 and there was no way of accessing the flash memory I quickly gave up. I then tried a 10/100 Linksys switch but it had a dead console port so there was no way to configure it. Then fate it seems smiled on me. I found a super high end, low profile, sleek / sexy HP switch pictured on the right! (j/k) No don’t worry we didn’t use this blade style switch. It looks as though we’re going to have to wait for the just the right switch to come rolling in the door. That’s the beautiful thing about Free Geek, at some point what you want will just magically appear! 😀 With the search for a switch being a bust I then went on a hunt for servers. Luckily we’ve been cacheing them over the last few months, we now have many many HP Proliant DL360 G3’s and G4’s that we’re going to put into production. They’re usually dual core Xeon 2.8GHz and 3.2GHz. Also with all the extra carcasses we have plenty of spare parts such as power supplies and hard drives sleds. Yes, they are SCSI and thanks to Jeff up in build we also have a LOT of 18.2 SCSI hard drives. Thanks to all within the Free Geek crew especially the now estranged Ifny who will be missed! Coming up next will be (hopefully) switch installation and trunking to the main network as well as installing OSes and configuring services. Stay tuned!!!
As I touched on very briefly in one of my previous posts, I volunteer. I volunteer for a non-profit organization by the name of Free Geek. It’s the local chapter here in Vancouver and they have a very simple purpose. Their website states: Free Geek is a nonprofit community organisation that reduces the environmental impact of waste electronics by reusing and recycling donated technology. Through community engagement we provide education, job skills training, Internet access and free or low cost computers to the public.
When I first arrived at Free Geek I was amazed at how much computer hardware there actually was, but even MORE amazed the second time I went to see that it all had changed. I mean, all of it. The amount of hardware that that flows through their doors is mind boggling. Immediately I knew this was something I wanted to be a part of. I met one of the founders and explained my talents and I what I could do for them, Ifny LaChance, she hugged me! She explained that Free Geek was desperately in need of a good Systems Administrator as their current person could not adequately handle the failing systems. The Free Geek infrastructure suffered from network outages, power outages, overheating, and poor performance, and they were all considered part and parcel of the environment. Most of the users had learned to live with it, or work around the issues.
They use a system called LTSP for all of their public and private workstations. These workstations were at a crawl. The first thing I took care of was all the physical wiring between computers, switches, and servers. While pulling cable out I would find the oddest things, like a six foot ethernet cable running from a switch to hub, then from that hub another cable extending about four feet to a switch. This switch was not only then connected to three different computers but also plugged back into the same switch that this connection originated from! After the wiring had been taken care of, an 802.1d trunk was introduced between two main switches and then VLAN’s were placed on the network to segregate subnets up a little better and to give some added security.
Since then I’ve gone on to introduce an OpenLDAP directory system, integrate user authentication for all workstations and switches. Created an OpenVPN install and added services such as SNMP monitoring by way of Zenoss. I am on a never-ending mission to improve the stability and the scalability of their infrastructure while only being able to work with recycled hardware and open-sourced software. It has been a great challenge, but of all the years I have worked with technology I must say that nothing compares with the satisfaction I feel from knowing my hours of labour are enabling an amazing organization to help keep this planet green and give back to the community.
Please visit their website to see how you can help. freegeekvancouver.org