Overview
The Codingscape team, through their previous company Wedgies.com, powered real-time surveys on whitehouse.gov for The Obama White House during the State of the Union address.
The live event was supported by our team through hosting polls, surveys, data collection on the whitehouse.gov webpage where the speech was live-streamed. This meant we'd have to build a system that could handle hundreds of thousands of concurrent users and 10s of millions of pageviews in a 2-hour window.
Codingscape’s team of senior-developers relied on their experiences building large, scalable systems at Amazon, Overstock, and other tech firms to build out a robust technical solution for this event.
Background
We want to run polls for a big event toward the end of January. Is it possible to remove your branding?”
Our startup is named Wedgies. We get that question a lot.
What made this conversation different was that the White House was asking. And the event was kind of a big deal.
“We will be hosting the livestream of the State of the Union on a special page we are developing on Whitehouse.gov. We want to let viewers vote on questions in real time during the speech using your platform.”
Redcated ;)
Obama Whitehouse Digital Team
Execution
If you don’t remember what you learned in High School Civics, the US Constitution requires the President to address Congress periodically to update them on the “state of the union”. 1965 was the first time the State of the Union address was televised publicly. These days, SOTU is streamed live on Youtube from Whitehouse.gov and they wanted online polls to go with it.
We said yes.
We've helped everyone from publishers like the Wall Street Journal collect poll data from their readers to regular people deciding where to go for lunch as a team. If it can be voted on, Wedgies can power it.
“We will be hosting the livestream of the State of the Union on a special page we are developing on Whitehouse.gov. We want to let viewers vote on questions in real time during the speech using your platform.”
“How many visitors are you expecting to that page?” we asked.
They told us the number of visitors they were preparing for. “Will you be able to support that?”
We said we’d be ready.
Now we had an agreement with the White House to power polling on Whitehouse.gov where our polls would be viewed by millions of Americans over a two hour period.
Wedgies needed to scale up to handle that many voters, and scale quickly. We only had 30 days. There was a lot of work to be done.
PREPARING OUR PLAYBOOK
No team goes into a game without a playbook and we needed to develop ours.
We had a big goal set by our customer, and hundreds of thousands of concurrent users isn’t a number you can really test against while load testing and scaling.
We started by breaking that number down into other metrics we could measure and test against.
We created plays, or a series of checklists, that we knew we would need to do before or during the SOTU. Once we had them set, we ran our playbook past every expert and service provider who would talk to us.
We asked them to challenge our assumptions and some of them were extremely helpful. A lot of them thought we were crazy. While we load tested we found bottlenecks that needed to be fixed. And, we found failure scenarios that were added to our playbook. e.g. What do we do if we lose a node during the event?
Over those 30 days we adopted #NoSleepTillSOTU as our company hashtag. Somewhere between the early morning load tests and late night emails I started to have the same nightmares I had in college. You know, the ones where you are taking a final in a class that you somehow went all semester without attending?
That’s the one.
GO FOR WEDGIES
Finally, it was January 20th, 2015. The day of the State of the Union. We had our playbook filled with checklists and started running through them early in the morning. We ran some last minute tests, continued to monitor everything, and set up our situation room.
Everything looked solid.
An hour before the event, we dialed into a phone number provided by our contacts with the administration and I found myself on a bridge conference call with the White House tech team. I pictured everyone else on the line seated around a table in a dark room like CTU from the TV Show 24.
“Jimmy from Wedgies here.”
“Welcome, Jimmy from Wedgies.”
I double-checked our checklists. There was one empty box. I had called an audible and decided not to run a small load test against production.
Earlier in the day, President Obama’s tech team started pushing our polls to the landing page as a test. We were already getting significant traffic to the polls, but nowhere near what we were expecting that evening.
I decided not to run the load test so we wouldn’t cause any potential slowness during their daytime testing efforts that could result in Wedgies not being used that night.
That turned out to be a near critical mistake.
When the first poll powered by Wedgies hit the Whitehouse.gov page, there was an empty box.
Something was not responding! My heart dropped the way only a developer’s can when something has broken production. With the added weight of a startup co-founder and CTO watching their product show up blank on the SOTU livestream.
After a quick diagnosis the team concluded there was one very likely reason. The Load Balancers, the front line of our system, weren’t quite ready for the load being thrown at them. And if that was the case, the only solution was waiting a short amount of time for more load balancers to spin up to handle the load.
“Is Wedgies good?”, the voice on the other end of the line asked.
I swallowed hard and crossed my fingers, “Wedgies is good. Go for Wedgies.”
The next wave of traffic hit us and the polls popped into existence on the White House’s website. For the next two hours we continued to monitor our servers. Everything was running smoothly now, like we thought it should from our testing. At the end of the night we had served over 5 million polls to the State of the Union livestream viewers!
We celebrated with a big group hug, turned Heroku down, and went to Arby’s for dinner.
The next President of the United States is going to look at the current Administration’s records and wonder why they paid for ‘Wedgies’ in January of 2015. We’re looking forward to that call!
We started Wedgies because we believe voting on the Internet can be better than it is. It was an honor and a challenge to power voting via polls during such an important event for a democratic country.
Of note, Wedgies’s partners are Codingscape’s partners, and much of the Codingscape team remains the same!
Conclusion
Real-time surveys on whitehouse.gov for 10s of millions of pageviews
Our senior software engineers built a system that could handle hundreds of thousands of concurrent users and 10s of millions of pageviews in a 2-hour window.
Obama's State of The Union was a success and so were the live polls on whitehouse.gov.