I should have known the trip would have problems when right from the start a foggy day delayed our flight, but when travel is booked and boxes are shipped, what can you do but press forward and try to make the best out of a rough situation.
It was a Tuesday morning and I found myself sitting in the lounge at Victoria’s airport with two of my good friends tracking what should have been our plane as it did loops over Hope, BC. It was a foggy morning in Victoria and our plane wasn’t able to land until the fog had lifted.
This was to be my second working holiday in Coronation; the site of a property I had purchased the previous year. My first trip didn’t go exactly as planned, but I had higher hopes for this year since I now had grid-tied electricity and a better understanding of the challenges I would face. Nevertheless, despite all my budgeting and pre-planning the weather was making sure this trip would get off to a rough start.
As the fog lifted, and our plane began to make a beeline for YYJ, I remarked to my friends that at least we were getting the hiccups out of the way early on this trip. Oh how wrong I would turn out to be!
Our flight was uneventful, and we only arrived in Calgary about an hour later then scheduled and before long I spied our box of goodies being unloaded onto the oversize baggage carousal.
As an aside, traveling with a box, rather then a suitcase, turned out to be an ingenious life hack to get around airline weight restrictions on baggage. When checking in at the automated kiosk, I received the option to print the baggage tags, after which it was straight to the oversize baggage line, bypassing the airline’s baggage drop process. We did receive some funny looks and a lot of questions about the contents of the box – but it was worth it to avoid trying to cram everything into a couple of suitcases!
Anyway, with our box in hand we made our way to the exit to pick up our vehicle for the week. On the last trip to Coronation I used a traditional rental car agency, however, prices had since become so inflated that I ended up using Turo this time around. Turo is to car rental what AirBNB is to hotels – you rent from a private person, rather then an agency. All communication with the renter is via the app which eliminates most of the overhead for traditional car rentals. I’ll confess I was a bit nervous about using Turo, but my fears were quickly alleviated when our host pulled up to the departure gates and handed me the keys. The whole thing was a lot more convenient then a traditional car rental, and a hell of a lot less expensive!
I’ll admit it felt a little odd to be driving away without any pomp and circumstance, but our host seemed happy and there were no sirens or men with sunglasses running towards us so I figured all must be well. I proceeded to pull our 2012 Doge Caravan out of the parking spot and we were on our way!
Of course, if this was a sensible trip the next step would be to head out on the highway to Coronation, but this was not a sensible trip and so the next step was to drive to a local datacenter and begin doing some work for my hosting company.
As I have alluded to in the past, I am a co-owner of Free Range Cloud; a hosting company I’ve been running since 2018. In the weeks prior to this trip we received some distressing news – that the handshake deal I had arranged for our hosting in Winnipeg (a trade of IPs for rack space) was coming to an end after the small ISP I had made the deal with had been acquired. We needed to have our gear removed by the end of November, and since rack space in Calgary was cheaper and connectivity better, we decided it was a good time to decommission the legacy Winnipeg location and lift and shift our customers to Calgary.
Of course, rather then booking a special trip to setup the rack, I made the questionable decision to bolt this detour onto my existing Coronation trip – after all I was already flying through Calgary – but it did mean that our timeline had tightened up significantly, particularly since we now needed to be at the datacenter prior to the end of business in order to setup our access cards and get access to our rack. We then also needed to install all of our gear, which I had shipped to a friend local to Calgary.
Turns out the datcenter was really close to the airport, and after just 15 minutes I found myself standing in front of a locked door alongside Blair and Kyle, wondering where our account manager was. Turns out he had departed for lunch, so I left Blair behind while Kyle and I took the van to my friend’s place in order to pick up the server gear.
As it happens, my friend is also close to the datacenter, and after about 12 minutes, a few pleasantries and a couple of round trips to the car we were all set to return to the datacenter and see what fresh hell was in store for us.
Blair buzzed us in, and I dragged our boxes through the front door and into the data hall where our space had been assigned. Blair set to work mounting the equipment while I took a walk around to find our account manager in order to get my access card setup.
I’ve been in a lot of different datacenter’s and each one seems to have it’s own unique quirks. This one is physically located in an office park and aside from the cooling on the roof there isn’t much to indicate it’s actually a datacenter. The data hall itself was pretty standard, albeit smaller then most I have visited; a couple of rows of equipment racks, several UPS units, white raised flooring and dry, climate-controlled air. Some of the racks were in an interesting condition, and I found it quite amusing that I was able to reach around locked cabinet doors into customer equipment, but overall it was far from the worst facility I had been in. With my access card in hand and my cursory inspection complete, I turned back to see what was going on with our space – the news was not good.
Blair hadn’t made a lot of progress, namely owing to the fact that none of our equipment would physically fit on the posts in the rack allocated to us. To explain the situation, let me first describe how equipment is typically installed in server cabinets. Generally, racks have 4 posts – 2 at the front, 2 at the back – and equipment is typically installed on rails on the right and left sides which go between the front and back posts.
There are two styles of equipment rails – square hole and round hole. Round hole equipment is installed by inserting a screw through the front of the rail and into a cage nut which attaches to the rack. Square hole equipment attaches directly to the square holes on the rack posts and are typically more convenient to install. Our equipment was square holed, and try as we might we could not get it to latch in place.
I tried, Blair tried, even Kyle tried, but eventually we determined that the holes in these posts were not of a standard size and there was no way we were getting our equipment mounted to them. Off to our account manager to see what could be done!
As it turns out, we had been sold a brand new product – 1/5 of a rack colocation and were actually the first customers to try moving in. No one had actually tested to see if standard square-holed equipment would fit before we came along. Doh!
Fortunately, Blair managed to talk our way into getting extra space at the same rate. We were now able to move into a ¼ cabinet, with the understanding that we would only use the same amount of space promised originally. This was actually better, since it meant we could grow in the future without needing to move racks. Unfortunately, it also meant we would need to come back after the new space was ready – in a couple of hours’ time.
While this pushed our timeline back a bit, it did mean we had enough time to address another problem that I had discovered – when we packed our box we neglected to include power or network cables, so it was off to a local electronics store to pay expensive retail prices for both!
At the same time it gave me an opportunity to visit a hardware store and pick up some additional supplies I would need in Coronation, and it was in the checkout line of one of these hardware stores that I had an epiphany of sorts – I had forgotten the goddamn key to my shed in Coronation!!!!!
While waiting in line I thought to myself “I should have brought my points card,” when then caused me to realize my points card was on a keychain which was attached to my shed keys for Coronation!!!! I immediately called my Calgary friend who had a spare set for emergencies and thank goodness he was able to bail me out. A quick drive across town and I was back in business for my work later in the week.
By this time, dusk had arrived and we all headed back to the datacenter to move into our new rackspace, which fortunately was uneventful.
By the time our move-in was complete it was dark, and after way too much time fidgeting with the car, trying to make the headlights work we were on the road to Stettler, which would be our home for the next week.
By this time, you would hope that I would have begun to recognize a pattern, but in my infinite stupidity I delightfully exclaimed that I was happy to be getting all the bumps out of the way first, and the rest of the trip had to be smooth sailing, right? Right!?
Our drive into Stettler was uneventful, save for one incident where we pulled off the highway to pee and were immediately intimidated by barking dogs. An incident that while hundreds of kilometers from Coronation was eerily similar to an event that happened last time we needed to pee on the side of the road. In the end we made good time, and managed to check into the Airbnb before 11PM to get some much needed sleep.
We slept in a bit, but by 9 we were on the road to Coronation and 45 minutes later I felt a little twinge of joy as we rolled up to my property, and the all too familiar shed and dishes greeted me as I pulled along side the driveway.
Going into this trip, I had a several of what I hoped were very attainable goals:
- Get all three dishes put into service, receiving at least something
- Get my satellite tuner server operational
- Install a new router to replace the hastily acquired Ubiquity router from last year
- Run DC power cable to various junction boxes around the shed and tie it back to a network relay (this would give me the ability to remotely power things on and off)
- Kill all the weeds and grass anywhere near the property line – I had received a fine from the city for weed control, after Canada Thistle (some of the most infamous and hard to eliminate weeds) had crept onto my property from the neighboring one.
- Run some conduit between each of my satellite dishes in order to protect the coax cable from the elements and make adding new runs easier.
With these goals in mind, Blair and Kyle began digging some trenches between my dishes, while I busied myself with the network router, relay box and satellite splitters.
A couple of hours into my work I realized yet another oversight – while I had plenty of coax cable around the property, I had forgotten to pack connectors and the tools to install the connectors; meaning the dozen or so jumper cables I needed to make couldn’t be completed until after a visit to the nearest Home Depot – a 2.5 hour drive away! How I managed to forget so many crucial things on this trip is beyond me, but next time I travel anywhere I am going to make an actual list so I can cross things off as I pack them!
Blair, Kyle and I bit the bullet and decided to get Home Depot out of the way early, and we shut down our work party around 4 to begin our trek to Red Deer which was home to the nearest Home Depot. It was, of course, dark by the time we made it to the store, but they fortunately had plenty of stock of the connectors and tools I needed, along with a ladder that happened to be on clearance.
We also passed by a Value Village where I was able to pick up a keyboard (something else I had forgotten to bring), as I figured might need it to setup the satellite server – after all I wasn’t sure it even had an operating system, much less one I knew the credentials for. Boston Pizza in Red Deer was our dinner, being that it was the only thing open after 9PM, and after some food we began our 90 minute drive back to our Airbnb in Stettler. This drive was uneventful, save for the fact that about 30 minutes in we saw a strange green glow off in the distance – the Northern Lights! My first time seeing them in person!
We arrived back in Stettler around the same time as the day before – another late night and straight to bed.
The next few days are honestly a bit of a blur. We got up, went to Coronation, got shit done, ate lunch, got more shit done and came back late in the evening. We discovered that the only places open after 9 in Stettler were Boston Pizza and a Chinese buffet that currently only offered table service. We ate at Boston Pizza a lot.
We also revisited all of the restaurants in Coronation – not difficult to do since there are just three, plus a pizza place. One of the restaurants was under new ownership – Me-Maw’s kitchen was now the Big Mama’s Smokehouse featuring meat dishes in all forms. One thing I thought was pretty neat was that they did away with full menus, instead offering the choice between just two items that were on special each day. Overall, all the restaurants were good, and I would say that my personal ratings from last year were unchanged.
We also made use of the pool at the Canalta – one of three traditional hotels in Stettler. During last year’s trip Blair and Kyle made a point of visiting the hotel’s pool on several occasions, something which the front desk staff accommodated for a nominal fee. On day 2 after having a morning swim, Blair and Kyle came back complaining of higher prices for day use rates of the pool, but by the time I began joining them for some evening swims the price had dropped to just $5 – cash only. I’m pretty sure the nighttime front desk agent was happily pocketing the cash, but I was more then happy to use the pool for $5 so I’m not complaining! Blair insists that if he is to come back, we need to stay at a hotel with a pool. We’ll see.
We reached what I would consider peak hilarity when we borrowed the TV from the AirBNB to setup the satellite server. I wanted to have a monitor on hand just on case something went horribly wrong with the networking, and the TV was the only display readily available. I am glad no one spotted us leaving with the TV that day, because I’m sure they would have assumed we were stealing it.
Most of the work in Coronation went reasonably well. The trenches were dug in record time and the conduits placed and buried. We got the coax cables terminated and run, the satellite tuner server mounted to the wall and configured, the DC power cables routed to all corners of the shed. Kyle even made excellent progress on killing the weeds, grass and anything else living in my property’s dirt – by spraying cleaning vinegar all over the place!
While many things went well, the thing that didn’t go well was the bloody dishes! I expected aiming them to take time, but I was not ready for the sheer amount of time it would consume, and the sheer amount of dicking around to try and make them operational.
Going into this project, I had planned on replacing the LNBFs (the antenna which sits at the front of the dish that the cable connects to) with modern LNBFs I knew would work with my satellite receivers. What I didn’t anticipate was how difficult it was going to be to physically attach a modern LNB to to the dishes that I had. Being commercial-grade dishes from the 80’s the mounting mechanism they employed was completely different then what I was used to with consumer dishes, and I was never able to get it perfectly aligned so that enough signal would come in to drive the tuners.
Blair and I screwed with both of them for hours, spending I don’t even want to think about how much money on nuts, bolts and washers at the local Home Hardware. In the end, two of the three dishes I just left with the old LNBs, and the third one I affixed with zap straps (I am low key excited to find out if they survive the winter). While I did manage to make the original LNBFs work, they won’t work to receive all of the feeds I was hoping for, so more redneck engineering is in my future.
While messing around with the LNBFs was a massive pain, the other massive pain was actually aiming the dishes themselves. It had easily been 20-30 years since anyone had touched them, and 20-30 years of rust is… a lot of rust. We tried penetrating oil and power tools, but at the end of the day some really big wrenches and a whole lot of elbow grease is what eventually broke the final bolts free, allowing the dishes to free spin.
My original plan was to point one dish at 58W, over the Atlantic, and another one as 180E, over the Pacific, but given that both satellites were low to the horizon (making them more difficult to aim) and given the fact that I managed to damage my satellite finder I decided to aim both for more easy targets over North America. At least all three dishes are now producing, though not quite the the signals I had hoped. The bolts have now been broken free so next time I can concentrate on aiming, rather then brute force.
We worked right up until Monday (our return flights were on Tuesday) but fortunately only had a couple of hours of work on our last day leaving us with a bit of time for recreation. Unfortunately, just about everything we wanted to do was closed on Mondays so we had to entertain ourselves by visiting points of interest such as the Castor grain elevator historic site, and the Underwood flying machine. It was all still interesting stuff.
Tuesday arrived and we checked out of the Airbnb early to give ourselves plenty of time to make it back to Calgary. I had arranged a lunch with a couple of co-workers, which was a great way to spend part of the afternoon. Blair, Kyle and I then finished off the day by visiting the Devonian gardens (an indoor garden in the top of a shopping mall) and exploring the network of skywalks throughout downtown Calgary.
The skywalks are actually really cool – because Calgary gets so cold in the Winter, there exists a network of walkways that connect most of the downtown high rises above the roads. This makes it possible to travel from building to building to building without ever needing to go outside. It’s now an official item on my bucket list to go back and explore the entire network from end to end.
Our flight was scheduled for 18:30, and by 16:30 we were back at the airport, dropping our Turo rental off at the departures terminal without so much as a hitch. I have to say I am extremely satisfied with the whole experience and will likely be using it again.
Upon arriving back home, I spent a couple of days decompressing, before turning to the work of beginning to actually use the gear in Coronation and Calgary and it was during this process that I would make my final crucial mistakes.
Let’s start with the Free Range Cloud gear. While in Calgary, I had verified that everything had power, and that I had remote access to the routers and out of band management. What I didn’t validate was that all the servers completed their boot successfully, after all I figured that since I had out of band management, I could troubleshoot as needed. What this didn’t account for was hardware problems; which, as it so happened, is exactly what occurred. The RAID card, which connects the hard disks to the rest of the system was no longer detected, meaning the system couldn’t boot properly and even if it had, there were no drives to store any data on. I managed to find a replacement server locally, and sweet talked my friend into picking it up, harvesting the RAID card and installing it, but I seriously should have checked it before calling it good.
The next fatal mistakes were related to Coronation. I had brought with me two power supplies – one 24V boost converter, and one 48V boost converter. While the 48V one was to provide power to my security cameras, I couldn’t remember what the 24V one was for so I had left it on the shelf. After all everything seemed to be working fine, so it must not be needed, right?
Wrong. As it turns out the WISP Internet antenna was designed to run on 24V, and while it did run on the 12V coming off the batteries, every time the charger cycled on and off the voltage dropped low enough for the antenna to reboot. This meant that one of my two internet connections randomly reset itself roughly every 3 hours. Not great, but this is why I had two! I configured everything to use the DSL as primary, since it was now more reliable.
My final faux pas came in the form of my satellite tuner server. I had configured and tested remote access to it before leaving Coronation, and validated I could receive some satellite feeds, however, I failed to burn-in test the server, and it quickly became apparent that not all was functioning well. While the feeds worked, they also dropped out constantly; not something related to signal strength since I had plenty of that. I concluded that the server was running an extremely old OS and drivers, so I decided the natural thing to do would be to upgrade it, but any OS upgrade is a risky move and doing so on a server with no out of band management, where the nearest remote hands is 3 hours away was probably not the best idea. And as would happen, bit me in the ass. The server came back, but SSH access was broken, and the tuner cards were no longer recognized by the software. Without SSH I had no way to fix it, so my tuner server had become essentially useless.
I had fun on this trip but also had a few take aways. I vowed to always plan things out ahead of time, proceed with caution when performing major changes, and to verify, check and double check before leaving. All of my problems (forgotten items, untested hardware, broken systems) could have been prevented had I slowed down and really took the time to think through what I was doing. In the end, I had an interesting and fun trip, but not one that actually accomplished all of my goals. And I had a broken Internet connection and borked server to remind me of that fact each and every day.