Anthropic leak reveals Claude Code tracking user frustration and raises new questions about AI privacy
WTF, Anthropic’s Claude Code keeps track of every time you swear
Code that reads your frustration is the least interesting part of the story of this accidental leak from Anthropic. The leak reveals how AI tools are also concealing their own role in the work they help produce
On March 31 artificial intelligence company Anthropic accidentally leaked roughly 512,000 lines of code, and within hours, developers were poring over it. Among the surprises was code inside Claude Code, Anthropic’s AI coding assistant, that appears to scan user prompts for signs of frustration. It flags profanity, insults and phrases such as “so frustrating” and “this sucks,” and it appears to log that the user expressed negativity.
Developers also discovered code designed to scrub references to Anthropic-specific names—even the phrase “Claude Code”—when the tool is used to create code in public software repositories, making the latter code appear as though it was entirely written by a human. Alex Kim, an independent developer, posted a technical analysis of the leaked code in which he called it “a one-way door”—a feature that can be forced on but not off. “Hiding internal codenames is reasonable,” he wrote. “Having the AI actively pretend to be human is a different thing.” Anthropic did not respond to a request for comment from Scientific American.
The findings expose a problem emerging across the AI industry: tools that are designed to be useful and intimate are also quietly measuring the people who use them—and obscuring their own hand in the work they help produce. Anthropic, which has staked its reputation on AI safety, offers an early case study in how behavioral data collection can outpace governance.
On supporting science journalism
If you’re enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Technically, the frustration detector is simple. It uses regex, a decades-old pattern-matching technique—not artificial intelligence. “An LLM company using regexes for sentiment analysis is peak irony,” Kim wrote. But the choice, he notes in an interview with Scientific American, was pragmatic: “Regex is computationally free, while using an LLM to detect this would be costly at the scale of Claude Code’s global usage.” The signal, he adds, “doesn’t change the model’s behavior or responses. It’s just a product health metric: Are users getting frustrated, and is the rate going up or down across releases?”
Miranda Bogen, director of the AI Governance Lab at the Center for Democracy & Technology, says the more pressing issue is what happens to such information once a company has it. “Even if it’s a very legible and very simple prediction pattern, how you use that information is a separate governance question,” she says. A signal collected for one purpose can migrate into other parts of a product in ways users neither expect nor consent to.
Bogen says the pattern is familiar from older Internet platforms, where small behavioral cues became signals that shaped what users saw and how they were categorized. AI companies are reprising a similar privacy problem: users hand these systems enormous amounts of information precisely because the tools are designed to know them well enough to be useful. “Who is keeping track of things about users?” Bogen asks. “And how is that information being used to make determinations about them?” What the Anthropic leak made plain is that, at least at one company, such accounting is already written into the code.
It’s Time to Stand Up for Science
If you enjoyed this article, I’d like to ask for your support. Scientific American has served as an advocate for science and industry for 180 years, and right now may be the most critical moment in that two-century history.
I’ve been a Scientific American subscriber since I was 12 years old, and it helped shape the way I look at the world. SciAm always educates and delights me, and inspires a sense of awe for our vast, beautiful universe. I hope it does that for you, too.
If you subscribe to Scientific American, you help ensure that our coverage is centered on meaningful research and discovery; that we have the resources to report on the decisions that threaten labs across the U.S.; and that we support both budding and working scientists at a time when the value of science itself too often goes unrecognized.
In return, you get essential news, captivating podcasts, brilliant infographics, can’t-miss newsletters, must-watch videos, challenging games, and the science world’s best writing and reporting. You can even gift someone a subscription.
There has never been a more important time for us to stand up and show why science matters. I hope you’ll support us in that mission.
