Airlines depend on huge, overlapping and complex IT systems to do just about everything, from operating flights to handling ticketing, boarding, websites and mobile-phone apps. Years of airline consolidation have left a hodgepodge of hard-pressed computer systems.
DALLAS — Twice in less than a month, a major airline was paralyzed by a computer outage that prevented passengers from checking in and flights from taking off.
Last month, it took Southwest days to recover from a breakdown it blamed on a faulty router. On Monday, it was Delta’s turn, as a power outage crippled the airline’s information-technology systems and forced it to cancel or delay hundreds of flights. Delta employees had to write out boarding passes by hand, and at one airport they resurrected a dot-matrix printer from the graveyard of 1980s technology.
Why do these kinds of meltdowns keep happening?
Checking on your flight
If you can’t get through to your airline on the web or by phone, your best strategy is to check for delays or cancellations at this page on the Port of Seattle website: http://www.portseattle.org/Sea-Tac/Flights-Airlines/Pages/default.aspx
The website incorporates data from the airline-tracking website FlightView as well as information provided by airlines to the Port of Seattle.
Source: Seattle Times staff
The answer is that airlines depend on huge, overlapping and complex IT systems to do just about everything, from operating flights to handling ticketing, boarding, websites and mobile-phone apps. And after years of rapid consolidation in the airline business, these computer systems may be a hodgepodge of parts of varying ages and from different merger partners.
These systems are also being worked harder, with new fees and options for passengers, and more transactions — Delta’s traffic has nearly doubled in the past decade.
“These old legacy systems are operating much larger airlines that are being accessed in many, many more ways,” said Daniel Baker, CEO of tracking service FlightAware.com. “It has really been taxing.”
The result: IT failures that can inconvenience tens of thousands of passengers and create long-lasting ill will.
The problem started with a 2:30 a.m. power outage that wreaked havoc on computer systems at Delta’s Atlanta hub.
Most Read Stories
- Aerospace firm Electroimpact agrees to pay $485K after AG finds ‘shocking’ discrimination against Muslims
- Huskies get commitment from Coeur d'Alene 4-star QB Colson Yankoff
- Price tag zooms up for light rail across I-90 bridge: $225 million more needed
- Poutine is the new nachos: where to find the best versions in the Seattle area
- 'I'm amazed tourists ever come back': Your comments on Seattle's poor tourism survey
Georgia Power said the outage was caused by the failure of a piece of Delta equipment called switch gear, which switches power flows within a system. That affected flight-planning and customer-tracking systems, and at 5 a.m. the airline declared a “ground stop,” halting takeoffs around the world.
Delta said that after the power outage “some critical systems and network equipment didn’t switch over to Delta’s backup systems,” adding that “investigation into the causes is ongoing.”
Outside experts discarded the possibility that computer hackers might have caused the outage.
Georgia Power spokesman John Kraft said no other businesses or residences were affected. He said Georgia Power crews determined that Georgia Power’s system “seems to be OK, and determined that it apparently was an issue on (Delta) equipment.”
Delta said it had canceled more than 740 flights, although its computer systems were fully functioning again.
The airline said it had operated about 2,340 of its almost 6,000 scheduled flights as of 3:40 p.m. It carried about 500,000 passengers daily during July, its busiest month, Delta said on its website.
The airline said it will offer $200 vouchers for future travel to customers whose flights were canceled or delayed more than three hours as a result of the episode.
Delta is waiving the change fees for any Monday flight that is rebooked through Friday. However, customers would have to pay the difference between the original and new tickets. And if they didn’t rebook their flights by Friday, change fees of as much as $500 would apply.
Airlines made more than $2.8 billion in change fees in 2013, according to the Bureau of Transportation Statistics.
The shutdown came just as travelers started flowing into Hartsfield-Jackson and other airports. Monday morning is usually busy as business travelers gear up and vacationers return home. Tens of thousands of affected customers struggled to figure out the status of their flights.
Delta had made major investments in technology upgrades in the late 1990s, but that was followed by hard times, a bankruptcy reorganization and a merger. In recent years, however, the company has enjoyed record profits.
As a side effect of the outage, many affected travelers said they did not get messages or updates from Delta on their flight status.
On Monday in Richmond, Va., Delta gate agents were writing out boarding passes by hand. In Tokyo, a dot-matrix printer was resurrected to keep track of passengers on a flight to Shanghai.
IT experts questioned whether Delta’s network was adequately prepared for the inevitable breakdown.
“One piece of equipment going out shouldn’t cause this,” said Bill Curtis, chief scientist at software-analysis firm Cast. “It’s a bit shocking.”
Curtis said IT systems should be designed so that when a part fails, its functions automatically switch over to a backup, preferably in a different location. “And if I had a multibillion-dollar business running on this, I would certainly want to have some kind of backup power,” he added.
IT problems are not unique to airlines. There have been high-profile breaches and breakdowns at banks and retailers, among others. Airlines have particular challenges because their systems are constantly undergoing changes and additions, including automation to handle the large volume of transactions with customers.
That degree of automation hindered Delta’s ability to inform passengers, many of whom didn’t know about the outage until they got to the airport. In the first several hours after the outage, when planes were grounded, Delta’s website and other systems showed flights as being on time.
Computer-network outages have affected nearly all the major carriers in recent years. After it combined IT systems with merger partner Continental, United suffered shutdowns on several days, most recently in 2015.
American Airlines also experienced breakdowns in 2015, including technology problems that briefly stopped flights at its big hub airports in Dallas, Chicago and Miami. American experienced delays after a bug in its iPad software meant that pilots did not have accurate airport maps.
Recovering from an outage can take several days, as Southwest proved last month. Southwest said it canceled 2,300 flights between July 20 and 24, about 12 percent of its schedule, and FlightStats said more than 8,000 flights were delayed.
The Southwest outage lasted about 12 hours, and the company blamed faulty routers for the disruption.
But the pilots and mechanics unions said Southwest’s chief executive had put too much focus on containing costs rather than upgrading outdated computer equipment. They demanded the ouster of Chief Executive Gary Kelly and Chief Operating Officer Mike Van de Ven.
That didn’t happen.
Until Monday, Delta had been considered among the leaders in operations and was thought to be immune to big IT problems.
Mergers create many chances for things to go wrong, as airlines that may have incompatible software combine their systems. From an IT standpoint, the United-Continental merger was seen as particularly awful, while Delta’s 2008 acquisition of Northwest was seen as so smooth that American copied it when it combined with US Airways in 2013.
Major airlines primarily use third-party processors like Sabre, Amadeus and Travelport to distribute their real-time flight data to travel booking sites like Expedia and Travelocity. They also contract with these services to run their own internal reservation systems, as well as their departure control systems to process boarding, last-minute bookings and seat assignments.
Delta uses an in-house system to process passenger services and flight operations, but the system infrastructure is run by Travelport in its Atlanta data center. Southwest uses Sabre for its domestic reservations and Amadeus for its international bookings, though it is migrating everything to the Amadeus system.
Each passenger on each plane represents multiple transactions: Each seat assignment, meal preference, child requirement and frequent-flier number is a separate log. Andrea Huguely, a spokeswoman for Sabre, said that each minute its system processed 164,000 requests and approximately $250,000 worth of travel spending.
Some travelers took Monday’s delays in stride.
“It sucks, but what are you gonna do?” said Lori Leszczynski, whose flight home to Milwaukee was delayed. “This is Atlanta, it’s a major hub, I’m sure they have top people working on it.”
Delta deployed carts to offer refreshments and beverages to customers at gate areas.
Lee McMillan, of Dothan, Ala., was headed to Seattle on business. His 10 a.m. departure was delayed until about 4 p.m.
“If it really was a power outage, you would think they have backup systems,” McMillan said. “You would just think they have redundancy.”
Sen. Richard Blumenthal, a Connecticut Democrat and member of the Senate Committee on Commerce, Science and Transportation, said Delta should do more than just waive the change fees for affected customers, who should be able to rebook on their own schedule.
The U.S. Department of Transportation lacks the authority to regulate airline customer service beyond a narrow set of issues, according to passenger advocates.
While the department can fine airlines tens of millions of dollars for safety violations, its penalties for non-safety-related issues are much smaller, typically in the tens of thousands of dollars.
Charlie Leocha, the chairman and co-founder of Travelers United, an airline consumer-advocacy group, said that airlines have fought new rules designed to protect passengers’ rights, but once they’re in place, they can be effective.
For example, he noted that there were more than 600 major tarmac delays a year before a 2010 rule prohibiting airlines from keeping passengers in a plane on a tarmac for more than three hours. That rule gave the department the authority to fine airlines as much as $27,500 per passenger per flight.
After the rule took effect, the number of such delays plummeted to fewer than 10 a year.
“It’s so dramatically better,” Leocha said.