For more than five hours Monday, while Facebook and Instagram were dark, David Herrmann fretted about ads.
Herrmann, a freelance media buyer, said that everyone he worked with relied heavily on the platforms, which soak up the bulk of the $80 million to $100 million in ad spending he manages each year.
One company that advertises exclusively on Facebook watched its revenue plunge 70% during the outage from the same period a week earlier, Herrmann said. Sales slipped 30% at another company, which spends $40,000 a day on ads.
“I was more or less checking Facebook consistently throughout the day, hoping it would come back,” he said. “But without clear direction from Facebook, we just had to wait.”
Ads fuel Facebook, which rakes in more than 98% of its revenue from more than 10 million active advertisers. In the three months ending June 30, it pulled in an average of $78 million in ad sales every six hours.
Graham Mudd, Facebook’s vice president of ads and business product marketing, wrote on Twitter on Monday that the outage affected Facebook’s ad platform and apologized “for the disruption this creates for our customers.”
A cascade of mistakes made during maintenance on Facebook’s network caused the outage that took its services offline, the company said in a blog post published Tuesday.
Facebook’s family of apps, which includes Instagram, WhatsApp and Messenger, were offline for more than five hours as employees scrambled to repair the damage. More than 3.5 billion people around the world use Facebook’s services.
The initial problem occurred in a network Facebook calls its “backbone,” which connects its data centers around the world, Santosh Janardhan, a vice president of infrastructure at Facebook, wrote in the blog post.
During maintenance of the network, a command was issued to assess how much capacity was available. But the command backfired, disconnecting the network and blocking Facebook’s data centers from communicating, Janardhan said. An audit tool designed to catch mistaken commands failed to detect the error, he added.
But it was just the beginning of the problems. “This change caused a complete disconnection of our server connections between our data centers and the internet,” Janardhan wrote. “And that total loss of connection caused a second issue that made things worse.”
With Facebook’s data centers offline, the company’s servers that manage its internet addresses were also unavailable. “This made it impossible for the rest of the internet to find our servers,” Janardhan said.
Media buyers noted that Facebook went dark at the beginning of the most important period for many advertisers, as they kick off holiday campaigns during a season that is expected to be complicated this year by supply chain struggles and pandemic restrictions.
“There may be heads on pikes by the end of this,” Cory Dobbin, the founder of the Aaron Advertising digital agency, wrote on Twitter.
Many businesses rely exclusively on Facebook to reach customers, Dobbin, who manages roughly $50,000 a day in advertising spending, said in an interview. The majority of his clients’ spending goes to Facebook, with the rest to Google, Snap and other platforms.
“The name of the game for many advertisers, if it wasn’t already, is diversification,” he said. “This is a perfect example of why you can’t rely on a single channel to bring in all of your revenue.”
He continued: “It’s just far too risky to rely on Facebook to be there for your business long term.”
Dobbin said he would be surprised if Facebook refunded advertisers.
“This is how Facebook works,” he said. “Always has been, likely always will be.”
Many companies used the Facebook outage to evaluate how their ads on competing platforms were performing. As Facebook users flocked to alternate services, Twitter posted on its own platform “hello literally everyone,” garnering more than 3 million likes and a disoriented face emoji from Instagram’s account. Netflix posted a meme featuring its popular “Squid Game” show, portraying Twitter as a rescuer.
But Herrmann, the media buyer, said advertisers would continue to be shackled to Facebook because of its enormous size and reach.
“It can and does still have massive implications across the media buying space, so it’s not going anywhere,” he said. “TikTok is coming up quickly, but nobody at scale does it as well as Facebook.”