Posted: May 8th, 2010 | Author: jordan | Filed under: DNS, Free Geek, Insight, LDAP, Linux, Migrate, SSH, Vancouver | 2 Comments »
Free Geek Mascot #1
Today was awesome! We got so much done!!! And it all went without a problem… oh except for when we tried to create an LACP bond on our NFS server and crashed the whole network… yeah… Quick story on that. we have 10 VLANs all trunking between our switches and our router. The NFS server is on VLAN 5 untagged on port 17 on the switch, we the added port 18 and created a bond on the switch. We then created a bond0 interface on the NFS server and used ifenslave to assign the eth devices to the bond device. Then….
BAM! WHOLE NETWORK GOES DOWN. Not just vlan5, no no, the whole god damn network. No Internet access nothing, not even from the router, the router can’t ping a thing on the whole network.
Why God? Why?
Then the Network God ARP said, “Jordan did you check those top kwality DLink switches?” So off I went to check the switch I just modified. For some unknown reason the DLINK decided to plunk VLAN 6 tagged onto port 16 for no reason what-so-ever. So I fixed that, but no, nothing worked still. So Tyler says, just unplugg the ethernet cables to the NFS server. Voila! Problem solved. Basically we think the NFS was just spewing out crap across the network and making all the servers in VLAN 5 bail, including the router. We were getting some pretty crazy ARP poisoning happening the router. Now, on to the good stuff.
This is a basic (and poorly made) diagram of how the Free Geek Vancouver network looks. We’ll take a look at how an LTSP client boots and logs in.
First the LTSP client boots looking for a PXE server, DHCP is handed out and tells the client to grab a boot image from yew. The LTSP client then boots into Ubuntu 10.04 (bleeding edge baby) where the login screen is presented. The user logs in and authenticates against our new fancy LDAP server on teak. When the client logs in their home directory is handed out via LDAP as /net/home/<$user> This directory is handled by AutoFS and mounts the NFS home from maple. The user now has full desktop experience with all their account info and home directory handled by 3rd parties.
What? Centralized authentication AND home directories?! REALLY?
Our LTSP server is now a 2 x Dual Core Xeon 3.20GHz with 4GB of RAM. A HUGE upgrade from what we running before. In addition to all this insanity Vicki was able to migrate our ticketing system for us as well as update all the LDAP records for home directories, install autofs on all servers, install the mount maps, comment out all the irrelevant fstab crap AND switch over all our servers to the LDAP server. Pow vicki, pow!
The backup system is pretty sweet as well. In our NFS server that holds the home directories is two RAID 5’s, a RAID 1 and some spares. One RAID 5 has a slice out of it that is for home directories. The other is 1TB for nothing but backups. What Tyler did was write a script that uses rsync to create incremental backups all done thru hard links. It’s pretty rad.
Now that I come to think of it, I didn’t really do much except play with the dogs!!
She helped in her own way
What's going on?
Posted: May 4th, 2010 | Author: jordan | Filed under: Free Geek, LDAP, Linux, SSH | 2 Comments »
Yes, yes, I know. Two months have gone by since the last entry about Free Geek. Well finally I had some time to make it down there and to my enjoyment though Tyler from Free Geek had been busy at work. He managed to do quite a bit of work while I was away. Here’s a pic and some highlights!
Front of Rack
Now first I must mention something that happened which was spectacular. I showed up to Free Geek with tools in hand ready to kick ass and chew bubblegum. I said ‘Hi’ to the gang and then got right to work going over what’s been done already and what we should do for the day. Then I heard a small voice coming from behind me. It all started with a simple ‘Hello.’ Tyler and I turned around and here stood this lady, she said that she had heard we were doing updates to the network and wondered if she could help. I have something to confess here, I judged at first sight. So my initial response was ….. uhhhhhh….. and in my head I was thinking “oh god I have SO much to do today, I can’t possibly teach and babysit someone else.” However we said ok you can help
“what’s your name?” I asked
“Vicki” she replied.
I said “OK, Vicki, I’m going to outline on this whiteboard what we hope to accomplish today.”
Damn! That's nice wiring!
I then began drawing out network topologies and what VLAN’s we were going to roll out that day. Tyler pulls up a network diagram I had done up briefly a few weeks ago to talk about subnet allocation and service assignments. All the while Vicki was quietly watching and listening. We then went about which of our new servers would be responsible of what task, such as “teak” was going to be our new LDAP and DNS server, maple the new NFS server, how authentication was going to happen for autoFS mounts and so on. Granted if you’ve been in this industry for a while this isn’t super complex stuff, LDAP migration, network topology planning, thinking ahead for future departments, etc etc. However, this isn’t childs play either, let’s be honest there are a lot of ’sys admins’ out there and not all of them could roll out a network of this size.
We turned to Vicki and started going thru the tasks on the board, expecting (I was anyway) to see a lot of confusion. BUT NO! OMG! She knew just as much, if not MORE about this stuff than we did. In fact, over lunch we got into a discussion about proper use of VLAN’s and subnet routing between them. This woman was (is) AMAZING! It was like the network God looked down from heaven and with his noodley appendage, blessed our tech mecca for that day by sending us a worker! A worker that new how to install services, write config files, test connectivity and map VLAN’s!!! Quite literally she cut our work time by 40% if not more. If anyone needs a good sysadmin, or network engineer who knows their way around a linux terminal and learns by being shown ONCE! Contact me, I’ll send her details on to you.
Anywho, Tyler and I laid out the VLAN’s and what they would be responsible for. We had configured three switches thus far to trunk all the VID’s but when we got to the fourth and final switch, we had no admin credentials for it. (My fault!) Our plan at that point was to wait until the end of the day, reset the switch, recover the password and then move our core router to the rack. In the meantime I checked up on Vicki and she had gotten all of our services, OpenLDAP, bind, Zenoss, apt-cache, TFTP server, and some other stuff up and running and was ready for configuration. I migrated the database from an older version of OpenLDAP with a slapd.conf file to the new version with the slapd.d directory.
Tyler and Vicki (Respectively)
Once the Free Geek came to an end Tyler and I moved the router from the bathroom server room to the upstairs rack, pushed the ADSL modem thru and VLAN, and then made an LACP trunk to our OpenBSD router. Put the VLAN interfaces in place and POW. Network configured. (For the most part) The final stage is migrating the servers to the proper VLAN’s and updating their services configurations.
The next and final post will be mostly diagram based. Stay Tuned! HOPEFULLY the next post will be really insightful IF I can get Luke and Kamil from Zymeworks to donate some time into rebuilding our Asterisk server and implementing a KDC
Posted: April 14th, 2010 | Author: jordan | Filed under: Uncategorized | No Comments »
Cellular Internet connectivity is becoming more and more popular. So much so that a lot telcos are reporting problems keeping up with the demand. But to make matters worse the telcos, at least in North America, are gouging there customer base with ridiculous fees. Before we get into what the solution to this problem is, let’s just take a quick look into the telco pricing model. Thanks to an associate of mine who is the head IT administrator for an extremely large newspaper publishing house, I now know that Telus’ wholesale cost on talk time is 0.0018 cents per minute. That’s right, it’s not a typo, 0.0018 cents a minute. My associate, because his company does over 1.4 million in business per year with Telus was able to negotiate a flat rate of .002 cents a minute for the entire organization, and just to be clear… yes this is cellular talk time we’re talking about here.
The problem is that the telco industry is an old boys club that’s very very hard to penetrate. Not only from a consumer perspective but from an industry perspective as well. As a small business owner trying to work in the cellular space I can attest this statement. Implementing change or getting an answer to fairly basic question can be a monumental task when dealing with these companies. Which brings me to the problem within the telco pricing model. The cost of data allocation.
Take for example a family plan. There are multiple handsets that pull from a collective pool of minutes. When the pool is expunged the minutes are charged per, nothing like this exists for data. If you own an iPhone with a data plan and a rocket stick with a data plan they’re two separate allocated pools of data. What if you could get a cell phone plan and attach as many devices you like to it. An iPhone, a rocket stick and cell modem in your home. You could pay for one plan and use the allocated minutes and data for not only your cell phone but also your home Internet connection. Wouldn’t that be grand?
Posted: April 11th, 2010 | Author: jordan | Filed under: Uncategorized | No Comments »
Yes yes, I realize I haven’t posted in almost two months. I feel awful about it. I’m not gonna sit here and be like most bloggers “updates are coming yada yada” cause I know if I say that it’ll be another two months before I make a post. The good news is that Luke over at Zymeworks is ready to donate the switches to Free Geek so I’ll be able to finish that series up soon. I’m hoping to get over there on Tuesday to do the next steps. But for now, on to more important things….
LIKE THIS
This is the bacon cheeseburger from Five Guys Burger and Fries. There are many different styles of hamburger out there. There’s the restaurant style where you feel like you should eat it with a knife and fork. The backyard bbq style where everything is fresh and yummy, and the fast food style where everything is fake and preprocessed. But what if you made a fast food style burger but only used fresh ingredients, never frozen beef, fresh bread. What if you made a fast food burger that had mayonnaise chosen from a blind taste test from 15 other mayonnaises? What if you had pickles that were far superior than any other pickle before? What if you could chose your toppings from a selection that puts Harvey’s to shame? Well you would have the five guys burger.
This family owned and operated franchise chain is taking North America by storm with even the president himself taking a bite. I decided to try it last night with my always willing roommate and let me tell you. It did not disappoint. If you need that fatty fix and want something fast foodish, but with good quality ingredients. Here it is. The restaurant is West Vancouver in Park Royal. Check it out.
Posted: February 26th, 2010 | Author: jordan | Filed under: Mac OS X Server | 3 Comments »
I just battled this for the last three days and finally have a solution. If you’re like me, you don’t have 16 different Mac OS X server boxes in your infrastructure. You have one. I was using Lithium as a monitoring tool and recently have decided into install Daylite server so I can keep track of clients, sales leads, and what not.
After installing Daylite Server, the app would not run. Daylite kept exiting saying that it could not initialize the data storage environment and quoting an “error 300.” What I figured out thanks to James over at Lithium and Kamil from http://kamilkisiel.blogspot.com/ is Lithium has a post-flight script that jams a bunch of sysctl options into /etc/sysctl.conf. I erased those options, restarted my Mac OS X server and Daylite began to run afterwards.
I do not yet know the impact this has on Lithium, perhaps I could get James over at Lithium to comment on this.
Yay.
UPDATE: Ok just so you know, once the daylite database was created and setup I placed the sysctl options back in place and restarted my server, both lithium and daylite are A-OK.
Here’s the options to place back in, just copy and paste to a command line.
echo "" >> /etc/sysctl.conf echo "#Lithium 5.0.0 sysctl.conf additions" >> /etc/sysctl.conf echo "kern.sysv.shmmax=524288000" >> /etc/sysctl.conf echo "kern.sysv.shmmin=1" >> /etc/sysctl.conf echo "kern.sysv.shmmni=64" >> /etc/sysctl.conf echo "kern.sysv.shmseg=16" >> /etc/sysctl.conf echo "kern.sysv.semmns=130" >> /etc/sysctl.conf echo "kern.sysv.shmall=131072000" >> /etc/sysctl.conf echo "kern.sysv.maxproc=4096" >> /etc/sysctl.conf echo "kern.maxprocperuid=1024" >> /etc/sysctl.conf
Hey MarketCircle! It would be really great to get someone on your inside to follow up with me and resolve my unhappiness with your product support. I’d really appreciate a discount on the software because despite this hiccup I must say Daylite is amazing and I would love to deploy it in my infrastructure.
Posted: February 18th, 2010 | Author: jordan | Filed under: Free Geek | No Comments »
This isn’t really a whole post, more like a failure report.
I had such high hopes when I got to Free Geek about getting the rack powered up and connected to our network. No we didn’t get the 30A circuits wired in yet but I do have one 15A that will be able to power the 15A PDU I have installed. I got that all up and running, and quite proud that one of our two PDU’s has remote power management, however when it came to configuring the network it was a different story.
Enter the crusty 3COM switch. We have this old, and I mean old, 3COM switch that I swear to God moans as it passes packets. Our 3COM has one already configured LACP trunk heading the ’server room’ downstairs in the bathroom, what I wanted to do was setup another trunk to go to the new server rack. Nope, not happening. I fought and fought and fought with that 3COM on web interface, console interface, and smoke/blanket interface but to avail. It just would not allow more than one LACP trunk to be configured. Now I’m back to looking for a switch to replace it. There is a start-up in town that said they would graciously donate their old gigabit switch hardware, I hope they pull through on that. (If you guys do, I’ll give you a good plug here.)
While I’m waiting for the switches to magic themselves over to me I’m going to start with OS install and configuration but that will have to wait for part 3.
Posted: February 11th, 2010 | Author: jordan | Filed under: Free Geek | 2 Comments »
Old Server Room
Welp, so many people emailed me about my last post that I had to include this. Better late than ever, Ladies and Gentlemen! May I introduce you to the Free Geek server room. Now granted, I understand that this photo doesn’t look all too bad, but let me assure you this is its ‘good side.’ If we removed the angry Joshua, as well as a door we would find many horrible scene. Such as two APC UPS 1400, both with missing front bezels, only the PCB and connecting ribbon remains. Be careful, if you at them the wrong way, they turn off! Above them is the sink for the bathroom that has been turned into a shelf to hold up a 5 bay 160GB SCSI RAID box that is our ONLY backup.
So over the last couple days we’ve gotten in a decent 48 port switch built by extreme networks as well as two APC SmartUPS 3000’s. After we put these in I got to work building our new trunk back to an existing switch. As you can see from the pic to the side getting this cable from point A to B is going to be a little challenging. Not only does it have to get in behind that huge shelving unit, but that huge shelving unit is covered by more shelving 6ft deep.
This was the result! My whole body pinned between a crap load of iMacs and the upper shelf, it hurt like hell but at least I got an ab workout! 
Eventually I got the cables fished thru, crimped and nicely zap strapped in. At the end of the day Joshua and I were able to install:
2 x APC SmartUPS 3000 XL
2 x HP ProLiant DL360 G4’s
48 port Extreme Networks switch
10 port 1U 15A PDU
LACP trunk
Next up will be network configuration!
Posted: February 7th, 2010 | Author: jordan | Filed under: Free Geek | 2 Comments »
The Free Geek server room is not in the best of shape. It basically consists of about 9 desktop towers of varying speeds stacked in groups of 3 with plywood in between them. Oh! and it’s all housed inside of a bathroom. 
The power is being supplied by APC UPS’s with missing front bezels and malfunctioning controls…. they scream randomly. There’s no cooling, or even airflow and when the toilet’s water evaporates it begins to smell like dead fish. The gateway was just replaced with a dual core Xeon (thank you Joshua,) but the old one which ran like a trooper for 3 years was a Pentium II 366. A lot of the servers are beginning to show their age, failing hard drives, dead fans, strange smells. It’s time for a general overhaul.
Our New Rack Enclosure
The first thing to do is to plan power and figure out a location for the new server rack. We’ll be feeding in 2 20A circuits and 2 15A as well. We decided on placing the rack beside the lunch room because it was accessible enough to work on and also allowed to kinda show off to all the volunteers.
We then got this rack graciously donated to us to house all the new server that will be going in. It’s a 42U HP rack enclosure with removable doors and panels. Very nice rack inside, I would say even better than the standard APC Netshelter because the doors come off a lot easier and it’s nice and roomy to work inside of. However the only thing I did not like about this rack was the feet. To level it off I had to get an 11/16th wrench and from the bottom lower the feet where as on the APC Netshelter this is done inside the rack with a handy little Phillips screwdriver.
Now with the location for the rack picked out and rack itself chosen we can now do my favourite part. Cleaning. The first pic is of the dirt swept up in about 15sq ft around and under the rack location. After this is done the HP rack was put in place and then levelled out.
Since this rack was donated to Free Geek it had a lot of stuff in it already, therefore the stuff needed to be removed. These pictures were taken when most of the stuff had already been removed.
Now with the rack in place and cleaned up comes the fun part! Finding stuff to put in it!!! Free Geek has a lot of stuff to choose from. 
Mike and the big ass switch
Normally the first thing I would install and get working is the power and UPS, however we are currently waiting for a UPS to come in through donation and the circuits have not been laid in yet either. Instead the first thing I’m going to do is find a nice switch to install. First on the block was a Cisco 2948G. At first glance it seemed like a great choice. Lots of ports, Gigabit ethernet, however after the password recovery and seeing that the IOS version was 6.1 and there was no way of accessing the flash memory I quickly gave up. I then tried a 10/100 Linksys switch but it had a dead console port so there was no way to configure it. Then fate it seems smiled on me. I found a super high end, low profile, sleek / sexy HP switch pictured on the right! (j/k) No don’t worry we didn’t use this blade style switch. It looks as though we’re going to have to wait for the just the right switch to come rolling in the door. That’s the beautiful thing about Free Geek, at some point what you want will just magically appear!
Tedrek!!!
With the search for a switch being a bust I then went on a hunt for servers. Luckily we’ve been cacheing them over the last few months, we now have many many HP Proliant DL360 G3’s and G4’s that we’re going to put into production. They’re usually dual core Xeon 2.8GHz and 3.2GHz. Also with all the extra carcasses we have plenty of spare parts such as power supplies and hard drives sleds. Yes, they are SCSI and thanks to Jeff up in build we also have a LOT of 18.2 SCSI hard drives. Thanks to all within the Free Geek crew especially the now estranged Ifny who will be missed! Coming up next will be (hopefully) switch installation and trunking to the main network as well as installing OSes and configuring services. Stay tuned!!!
Posted: January 13th, 2010 | Author: jordan | Filed under: Insight, Work | No Comments »
So if you’ve been reading my blog you’ll know that a couple months ago I quit my 9-5 job. Since then to be honest, I haven’t done much in the way of systems administration. I’ve had a couple contracts here and there but nothing really big. I tell you what, do the skills dull quickly! I sat down the other day to help a friend with a pretty simple problem. He had 10.5 OSX Server and wanted to extend his LDAP schema…. I couldn’t remember how to do it! Or just little things, mostly in bash, like how to tell processes to stfu. Don’t get me wrong I know how to do it, its just not coming to me as quickly as I would like.
To any sysadmins out there thinking about quitting, make a list of things you think are cool and don’t want to forget. Cause soon after your notice, there won’t be anything upstairs!
Posted: December 15th, 2009 | Author: jordan | Filed under: AppleTV, Mac OS X Server, Media | No Comments »
First I’m going to take an overview of XBMC, the application that we’ll use to replace front row or AppleTV.
XBMC is great application. It has many features that put it above and beyond front row. First XBMC is a media manager and digital jukebox. It does not do any file system organization of files so it does require you to be a clean with your media. It is capable of playing most types of audio such as ogg and mp3 as well as able to play any video that you have a codec installed on your computer. It’s fully interoperable with the standard Apple remote and has a different yet still intuitive interface, (despite what Luke tells you.)
I’m going to show you how to customize XBMC to pull movies in from differing sources. First navigate into Movies and erase the sources that you won’t be using. Then click add new source and type in the location. Also you can specify an SMB server for XBMC to connect to, we’ll cover more on this later. My movies are located on my server and thus I will specify the SMB address along with a username and password. Then when I select on the source that I’ve just entered XBMC will make an SMB connection to my server and list all the movies I have. Now I can navigate to my Movies directory and select any movie I want.
Another great thing about XBMC is the control interface. Simply take you apple remote and press the play button, and on screen control appears. You can then navigate the controls and select whatever action you want with the play button. Or press menu to escape. Also while the movie is playing you can press the menu button and minimize the film so that you can browse for another, or if you want to return to the minimize film just keep pressing the menu button.
Now this can all be applied to the music and pictures features as well. Just specify the location of your music or pictures and XBMC will list them for you. Plus, thanks to a nifty iTunes plugin you can now load your iTunes playlists inside of XBMC so there’s no need to resort all your music!
But now for the piece de resistance. To replace front row altogether you want to enable the Apple remote to work with XBMC. To do so, in Settings, hit the Apple Remote section. If you’ve got a standard-issue Apple remote, set Mode to Standard. To start up XBMC instead of Front Row, check off “Always Running” in the Apple Remote section of XBMC’s Settings area.
You can download and install XBMC for your Mac from here or if you’re using an AppleTV please read this.
