Case study · GoDaddy
DPS: the hosting platform behind GoDaddy's free trials and freemium
A Node.js, Redis and Cassandra platform that served every published Website Builder site at about 11–12 ms, across four datacenters on three continents, for at least eight years. It grew to tens of millions of websites and made GoDaddy's first free trial possible.
- 6,500requests per second, per server
- 12 msaverage response time
- 4datacenters on three continents
- 8+years in production
Problem
In 2013, publishing a Website Builder site meant provisioning hosting for it. Every published site was set up on GoDaddy's legacy shared hosting, and that step stood between a customer and a live website.
The legacy platform's response time, about 80 ms, was our target to beat. That platform handled traffic spikes by brute force. For the 2014 Super Bowl, GoDaddy ran an ad that sent viewers to one small-business customer's website. We estimated a peak of 10,000 requests per second to that single site.
12servers set aside for one customer's site during the 2014 Super Bowl
The safe answer was to move the site by hand onto its own cluster of 12 servers. That worked once, for one site. We couldn't do it for every site.
The business also wanted cheaper ways in. A free trial, and later a free tier, only work if a site can go live with no paid hosting behind it and each additional free site costs almost nothing to serve. Per-site provisioning could not get there.
Constraints
A stack none of us knew
We were a small team of .NET developers. I had never worked outside the Microsoft stack, and the first half of 2014 went to learning Linux, Node.js, Redis and Cassandra while we built on them. The steepest parts for me were Cassandra schema design and tuning the Linux network stack.
A hard date
The Super Bowl was in early February 2014, a few months after we started building in late 2013.
A team that got smaller
Four of us started DPS. By late 2014 I was the only developer assigned to it, and it had to fit around my other projects.
Approach & architecture
The idea was to turn publishing from a provisioning job into a write. A builder sends a site's content to a publishing API, and the API stores it in Cassandra, keyed by host name and page. Nothing is provisioned per site.
Serving is the other half, and it is the half that had to be fast. A visitor is routed to the nearest datacenter and through a load balancer to a pool of identical Node.js web servers. A server answers from its own memory if it can, from Redis if it cannot, and reads Cassandra only on a miss. More traffic means more servers, and nothing else changes.
The request path
Diagram of two paths. Serve path: a visitor is routed to the nearest of four datacenters, through a load balancer, to horizontally scaled Node.js web servers, which answer from an in-memory cache, then from Redis, and read Cassandra only on a miss. Publish path, kept separate: the site builders send content to a Node.js publishing API, which writes to the same Cassandra cluster.
The two paths meet at Cassandra. Simplified: how visitors were routed between datacenters is left out.
The schema that taught me the most
Our first Cassandra schema keyed each page by application, host name and route, with the application alone as the partition key. That put every site into a single physical row. Nothing looked wrong until we tried to seed a million test sites, and the seeding never finished.
30 minto seed 1 million sites, once the partition key was fixed
A composite partition key and a rewritten data layer fixed it. The same million sites loaded in 30 minutes.
Caching, then more caching
After that, most of the speed came from not reading the database at all. One caching change took the hit rate from 78% to 93%.
A gradual rollout
We moved customers gradually, behind an A/B switch in the publishing flow. Internal accounts went to DPS in the first quarter of 2014. Customers followed in the second quarter, starting at 5% and rising to 100%.
Closer to the visitor
DPS first served from two datacenters in the United States, Phoenix and Ashburn. Singapore started taking traffic around March 2016 and Amsterdam around June 2016. Each visitor was routed to the nearest of the four.
Four datacenters, three continents
- At launchPhoenix · AshburnTwo datacenters in the United States.
- About March 2016SingaporeA third datacenter, in Asia.
- About June 2016AmsterdamThe fourth, in Europe.
Stylised; not to scale. The dates are approximate.
A tenth of the hardware
By early 2015 DPS could be measured against the legacy hosting it was replacing, and the result fit on one slide. The slide in my Node Summit talk that February read: "We can handle the same load with only 10% of the hardware!"
Legacy hosting and DPS, as presented at Node Summit 2015
Comparison from Node Summit 2015 slides. Legacy shared hosting: 100% of the hardware, hundreds of servers with tens of thousands of sites on each, about 13,000 requests per second on average, about 87 milliseconds to first byte. DPS: the same load on 10% of the hardware, 10 milliseconds to first byte, and on track to serve 1.7 billion requests per month.
Figures from my Node Summit 2015 slides. Each square is 1% of the legacy hardware, not a server. The bars are to scale.
Results
6,500
requests per second, per server, at peak
12 ms
average response time
11 ms
in load tests, against an 80 ms legacy target
4
DDoS attacks absorbed with zero customer impact
93%
cache hit rate, up from 78%
30 min
to seed 1 million sites
99%
unit-test coverage
8+
years in production
At its peak a single server handled 6,500 requests per second with a 12 ms average response. The launch baseline, from our 2014 load tests, was about 3,000 requests per second on each of 13 servers, averaging about 11 ms against the 80 ms target.
The platform absorbed at least four DDoS attacks with no customer impact. The team received GoDaddy's 2014 Innovation Award for DPS.
Websites served
- 2015Hundreds of thousands
- About 2016More than a million
- 2021Tens of millions
Circle sizes are illustrative, not to scale.
Why it mattered
Because publishing became a write into one shared platform, with no hosting to provision per site, DPS made GoDaddy's first free trial possible, and later its first freemium product.
A free site is only affordable if it costs almost nothing to serve, and a serving layer that answers in milliseconds, mostly from cache, is what makes it cheap. I don't have a cost-per-site figure to publish, so this rests on the architecture rather than a number.
That freemium product was Websites + Marketing, part of GoDaddy's COVID-19 response to help people start small businesses. I also designed and built APIs that still power it.
DPS reached the Node Summit main stage in February 2015. I gave two talks on back-to-back days at Node Summit in San Francisco, one of them as a short-notice stand-in for a teammate who fell ill. At the same summit GoDaddy's VP of Engineering, Antonio Silveira, presented DPS on the main stage. His session was titled "The Node Dream, How Moving to Node.js Changed Our Team and Got Us 10x Better Scalability". Its abstract said the platform hosted "hundreds of thousands of websites" and could "easily scale to tens of millions".
It did: more than a million sites by about 2016, and tens of millions by the time I left GoDaddy in September 2021.
Stephen is one of the best software engineers I have ever worked with.
Lessons
Get the partition key right before real data arrives
One wrong partition key put every site in a single row, and it only showed up at a million sites. Load-test at the real shape of the data, early.
Missing the Super Bowl date didn't sink DPS
DPS was not ready to host the live Super Bowl site in 2014. By game day our load tests showed it could have carried the peak. Internal accounts were on it that quarter and customers the next.
Roll customers over gradually
Internal accounts, then 5% of customers, then everyone, behind an A/B switch.
Under attack, a hot cache is a defense
The DDoS traffic hit cached content, so average latency improved while the attacks were happening.
Fix the bus factor on purpose
For years I was the only person doing active development on DPS. In 2018 I made knowledge transfer a formal goal: another developer, and someone from another team, each had to ship a DPS change.
