A new way for grading food safety at King County restaurants was designed to address major challenges faced by other food-inspection systems.
PUBLIC Health Seattle-King County has unveiled a new grading system for food safety of restaurants.
While widely praised, some skepticism has emerged because a limited curve based on the last four inspections applies to grades within ZIP codes.
In this first year, roughly 50 percent in an area are rated as “excellent”; the next 40 percent as “good”; and the bottom 10 percent as “OK.” But grading on such a curve doesn’t determine everything.
The percentages reflect annual countywide performance based on food-safety regulation. For instance, the “excellent” category corresponds to 50 percent of restaurants exhibiting perfect compliance with critical code items in the prior year. If perfect compliance grew annually countywide, the top category would expand; conversely, if compliance dropped, the top category would shrink.
A restaurant with no critical violations in the past four inspections will always receive the top grade. And no curve applies for the worst performers — those subject to multiple returns by inspectors, or closures. Those restaurants will be rated as “need[ing] improvement” regardless of ZIP code.
The Seattle Times has reported some confusion and skepticism about this new approach. The co-owner of Dong Thap Noodles said, “I believe it should be rated by individual restaurant, rather than the rating being on a curve by the neighborhood.” More bluntly, the owner of Dot’s Butcher & Deli opined, “That’s the … dumbest thing I’ve ever heard.”
While the desire for simplicity is understandable, King County’s grading system was designed to address major challenges faced by other grading systems. I should know. I’ve studied and documented flaws of restaurant grading using data from some dozen jurisdictions. And for three years, without compensation from the county, my research team at Stanford University has studied King County’s inspection data, developed a randomized controlled trial to improve the quality of inspections, and helped design the methodology behind the county’s grading system.
The reality is that unadjusted grading — converting an inspection score from a single visit into a grade with uniform cutoffs — reflects chance and inspector differences more than food safety. A single inspection provides only a “snapshot in time,” which is why it is important to look at longer trends across multiple inspections. And even when inspectors visit at the same time, they vary dramatically in their propensity to cite violations. Our study documented that even when inspectors observe identical conditions in the same visit, they disagree on citing violations 60 percent of the time. Restaurateurs struggle with not knowing which inspector’s food code to implement in the kitchen. The owner of Sushi Yasuda in New York “received three different answers from three different inspectors about rules overseeing sushi rice.”
Such inconsistency does not mean that inspectors lack training or fail to observe risk. The FDA model Food Code spans nearly 800 pages of complex provisions and inherently entails discretion to apply to diverse establishments, from the mom-and-pop ice cream shop to the corporate burger franchise. “Going by the book” can be infeasible and often detrimental. As Peter Schuck wrote of meat inspections, “if all … regulations were enforced to the letter, no meat processor in America would be open for business.”
The result of such inspector differences is that when grading was adopted in New York, inspection scores had virtually no predictive power of how the same establishment would fare down the road. Good grades simply relied on winning the inspector lottery. In King County, where inspectors are assigned principally by ZIP code, unadjusted grading would create a similar lottery, where the winners are establishments in particular ZIP codes. Based on statistical simulations, moving from a lenient to a tough inspector (from the 10th to 90th percentile) would reverse grades for more than 50 percent of establishments.
Grading on a curve allows consumers to make a more informed choice about food safety by mitigating the inspector lottery. First, it is easier for consumers to recognize an area than to know the leniency of the assigned inspector. Second, distinctions within an area correspond more closely to food choices, which are overwhelmingly local. If every Bellevue restaurant received the top grade (which is actually the case in places like San Diego, where some 99.9 percent of restaurants earn As), that disclosure would offer little help to Bellevue diners hoping to distinguish restaurants on food safety.
If grading is to be done, the county should be applauded for recognizing these challenges and for transparently addressing them based on rigorous empirical evidence. The health department deployed a tested peer-review system to improve the accuracy and consistency of inspections. And it has implemented what educational institutions commonly do when differences in instructor grading standards swamp differences in student performance. By being the first to address, rather than hide, the challenges of grading, the county exemplifies governance based on evidence and facts.
Given discretion of the grader and the arbitrariness of which inspector or instructor may be assigned, the right choice is to grade on a curve.