It was a normal Thursday afternoon in the Free Range Cloud NOC. The time was around 3PM and I was looking forward to packing up and heading home for the day when all of a sudden things began to go south. The TV, tuned to an IP feed of CNN, locked up, Slack began throwing DNS errors, and alarms began to trickle in. Then my phone began to ring…
We had experienced site outages before, but this time it was different – the problems weren’t isolated to one site, and our office (which had redundant VPN tunnels to different cities) was also completely offline. An outage affecting over 10 POPs in isolated cities, none of which had a common colocation provider and few common Internet circuits. Either the entire Internet was exploding before my very eyes or something was, as they say, very rotten in the state of Denmark.
In my crippled capacity to work (all of our whitelisted management addresses were on the downed VPN) it would have taken me a while to figure out what exactly was happening, if it hadn’t been for sinister looking ticket that had glanced at some 30 minutes before – “DDoSed by BlackMatter.” The gist of the ticket was that some cyber-extortionist group had targeted us and was demanding payment or else they would launch a DDoS attack against our infrastructure. I had dismissed the message as a bluff, receiving cyber extortion emails is a weekly occurrence; if I took them all seriously I’d have filed for bankruptcy years ago, but this one was seemingly the real thing – my inbox full of alarms a testament to the effectiveness of the attack. I needed to do something, and fast, since my customers and suppliers were already on the phone biting my head off.
As luck would have it, I was already on a call with my business partner, which at least gave me someone to bounce ideas off of; a workflow that I find helpful in stressful situations.
Since the company’s inception we had largely been running without any form of DDoS protection. With attacks growing in magnitude and frequency, adding DDoS protection was in the back of my mind, but it wasn’t something I was ready to pull the trigger on just yet since solutions could be expensive. I had always assumed that one of our customers would be the target of such an attack, and I’d be able to leverage techniques like blackhole routing to mitigate it; never before had I considered we would be the target, but here we were stuck in a pretty awful situation.
My business partner and I decided to engage with the attacker. Signing up for any mitigation service was going to take days, if not weeks, and it was time we simply didn’t have. We sent a message to them, asking for them to temporarily suspend the attack so we could get into our systems and assess. The response was almost immediate – “Attack will stop in the next 200 seconds.” Very specific.
True enough, about 5 minutes later the sea of red that was my alarm dashboard began to clear, and our office regained full Internet connectivity. I had a moment to breath.
I once again assessed our situation. We had no sites with DDoS protection, and while we could potentially order a dedicated server somewhere that had it and begin routing our traffic though them it would probably be at least 24 hours before something like that could be set up – an eternity in the hosting industry.
I hate to say it, but we reluctantly decided to pay the attacker. They were asking for 2.5 Monero, which equated to about $1500 Canadian dollars. It was a painful blow to our tiny company, but it was the only quick fix I could think of.
The next problem was actually getting Monero. While it’s easy to accept cryptocurrencies as payment, exchanging it for fiat is more troublesome, and exchanging fiat for crypto even more difficult. We decided to engage with the attacker again – offer them a “downpayment” of about $400 USD (which was the amount we happened to have in our crypto wallet) in exchange for a 24 hour reprieve. The reply came almost immediately; “we have a deal.”
I found the least sketchy looking online conversion service (Monero doesn’t seem to be supported by a lot of the mainstream crypto gateways) and sent away our account balance of Ethereum. This was actually our first time withdrawing money from this new crypto processor, and I was impressed by how fast and easy it was. At least we had an excuse to test withdrawals.
The conversion happened quickly, and our attacker confirmed that he had received the money and would leave us alone for at least the next 24 hours. I wasn’t sure if it was the truth, but at least everything was online for the end of the day – I could have some dinner and think about next steps.
The problem with any extortion attempt, is that you have to trust the extortionist. If they’re holding incriminating evidence, you need to hope they never made copies, and in the case of a DDoS attack, you have to hope that they’re true to their word – because restarting the attack would be trivially easy. I figured since we had told them we’d be paying more money the next day, it made sense for them to hold off; but afterwards all bets were off.
After some dinner and some venting, we had to figure out how to buy crypto from our corporate account – something that proved to be challenging.
Up until now, we had only ever received crypto from customers, and then converted it to fiat using a service called Canadian Bitcoins. I assumed actually buying crytpo would be easier, boy was I wrong! While there exist a multitude of crypto exchanges for individuals, none of them seem to support corporations. I couldn’t even send money from the corporate bank-account into my personal crypto account, because the name on the bank account didn’t match! Frustrating beyond belief. In the end, I needed to send cash from our corporate account, to my personal bank account, then from my personal back account into my personal crypto exchange. This is going to be an accounting nightmare, but that’s Future Chris’ problem!
The next day, I reached out to our extortionist again, to confirm details, and generally see if they were even still around. They were. And so, I sent the next chunk of crypto off to a conversion service and hoped that it would arrive.
I assumed this would be the end of it. Perhaps this attacker would re-emerge in a few weeks time to try and extort us again, but by that time we would have DDoS protection in place and could hopefully withstand the attack. What happened next came as a total surprise.
15 or 30 minutes after the crytpo transaction was confirmed, I received a final message from the extortionist – they confirmed they had received the funds, and assured me they would no longer harass our network. They even apologized for causing harm to the business. Then they went on to give some totally unsolicited advice on how to mitigate such attacks in the future. I was blown away!
The advice was solid, and the message well written, even professional. I had spent a good portion of the day researching DDoS mitigation options, and had begun to formulate my own plan for building out mitigation infrastructure, and the advice I had just been given aligned fairly closely with my plan. Our attacker even went as far as recommending some suppliers – a few of which I had already added to my short list. The response was so well composed that I became significantly less mad and more just confused. I also had a little more confidence that maybe this would be the last time we heard from them. After all, if they intended to launch a fresh attack, why tell us exactly how to stop them.
I relayed my whole story to a buddy who works in cybersecurity, and confirmed that my plan (and the advice I had received) was solid. We figured this was a case of one of the “good guys” in the cyber criminal circles – someone overseas trying to feed their family who would stick to their word. I guess, in that respect, we lucked out.
While, I would have rather avoided this whole experience I have to say it was at least the best outcome of a bad situation. While we had been extorted for over $1500, I had a reasonable amount of confidence this would be the end of it. And out of the bad comes a better product – Free Range Cloud would now be deploying a battle-tested DDoS mitigation solution, which give us new sales opportunities and would better protect our customers. Plus, I can write this expense off as “DDoS consulting” since that’s essentially what we received.