On Tuesday, June 8th, 2021 at 09:47 UTC, Fastly's Content Delivery Network (CDN) went down.
Sites affected included Reddit, eBay, Twitch, GOV.UK, Pinterest, and Vimeo. Some were affected in smaller ways - Twitter emojis were no longer visible, for example.
Down Detector shortly after Fastly’s outage.
How Do CDNs Work?
Typically a CDN sits between your web application and users, providing content caching as close to users as possible.
Other functions may include security and content optimization, but caching is a CDN’s “core” purpose.
Making use of a CDN is generally a no-brainer: your web application’s assets require less computing power, your users get content faster and with fewer complications, and global networks afford great stability - often with flexible and reasonable pricing.
“Too big to fall” is a myth: while CDN outages may ordinarily be limited in scope, broader outages do happen, as we’ve seen. From Fastly’s own report, the outage was caused by “a bug that could be triggered by a specific customer configuration under specific circumstances”, affecting 85% of Fastly’s network.
It’s well worth noting that this was a very specific set of circumstances and that the software deployment in question was started on May 12th, 2021 - it existed for close to a month before being triggered.
While this scale of the outage isn’t regular, it’s not unheard of; some notable examples include:
- 50% of Cloudflare traffic drops, July 2020
- Google Cloud major outage/disruption, March 2020
- Amazon S3 outage, February 2017
The effects this type of outage has on user traffic (and therefore business) can be catastrophic. Diagrams below illustrate such an outage:
User access through CDN pre-outage
User access through CDN mid-outage
What Do I Do Now?
Continue to use CDNs! It pays to have as much content as possible cached and served by a CDN - see rationale above. Conversely, the more reliance you place on a single provider, the less room you have for recovery in good time if anything goes wrong. You can, however, avoid keeping all your proverbial “eggs in one basket” and stay agile enough to adjust on the fly, even in an automated manner.
We recommend that you consider using an Application Delivery Controller (ADC) alongside your CDN. ADCs operate close to your backend infrastructure and are therefore able to make decisions about routing and traffic management based on “internal” information, such as Service Discovery. CDNs and ADCs work hand-in-hand, and it’s not uncommon to see ADCs deployed behind CDNs. The result is globally-served content that’s resilient against outages while remaining agile and configurable.
Snapt Nova is a next-generation cloud-native ADC, comprising load balancing, web acceleration, and security. Nova nodes are scalable, ephemeral, and easily replaceable; regardless of your deployment preference. Your Nova node infrastructure can be orchestrated by the Nova Cloud Controller, scaled up and down according to traffic and compute resource load. A good example of how Nova would fit below a CDN follows:
Example of user access through a CDN and Nova pre-outage
Example of user access through a CDN and Nova mid-outage
Recovering from a broad outage is now as simple as a DNS record change. You could, for example, configure monitoring to automatically flip DNS records when downtime occurs, and even scale up your number of nodes in anticipation using Nova’s simple REST-based API. If containers are your poison, the Nova AutoJoin container allows for simple adoption of configuration templates - truly effortless scaling with your container orchestration platform of choice. This along with hourly usage-based pricing makes for an impressive combination.
Your Infrastructure, Your Control
Nova Cloud Controller will orchestrate your node deployments, but the compute resources involved (virtual or physical) is up to you - mix, match and optimize according to your own environment. We don’t anticipate any issues, but your Nova nodes will continue to operate as configured even if disaster strikes.
Another convenient side-effect is that data transferred is between you and your Nova nodes, ensuring privacy and compliance with data storage laws. In addition, should you wish to control every part of the equation, we’ll white-label and deploy a full Nova Cloud Controller within your own environment.
Security, Both Simple and Robust
Nova’s state-of-the-art Web Application Firewall (WAF) is included in every plan (including our perpetually-free Community edition!) and protects against the OWASP Top Ten Web Application Security Risks. Other security features include:
- Geofencing: not expecting traffic from a given country? Simple management of allow- and deny-lists.
- Active Threat AI: an intelligent engine that evaluates potential threats in real-time and acts accordingly.
- Denial of Service (DoS) mitigation: prevent malicious actors from consuming precious resources.
- Bot blocking: leverage our years of experience to keep ahead of the curve and prevent automated site scraping, vulnerability probes, and other unwanted activity.
- WAF ‘learning’ mode: receive reports on what would have been blocked, without disrupting existing workflows. Use this information to fine-tune your WAF configuration in order to match your use case.
- NovaSense: a separate but fully integrated threat intelligence service, to anticipate abuse and malicious activity long before it reaches your web applications.
We expect CDNs to succeed, be reliable, and improve greatly. We’re excited to see this happening and we celebrate the increased stability and affordability that comes with this progression.
We also believe in good planning and anticipation of the odd hiccup regardless. We’ve built Nova ADC to be complementary to whichever environment you find yourself in, including behind a global CDN or major cloud service.
To try the latest in ADC technology, request a free trial of Snapt Nova.