Klarna's CEO toured the conference circuit as proof AI was about to gut customer service headcount. Fifteen months later, he was hiring humans again. The Klarna reversal is a case study in uniform AI confidence on tasks of very different difficulty.
In February 2024, Swedish buy-now-pay-later firm Klarna ran one of the more aggressive PR campaigns of the post-ChatGPT era. The company announced its AI assistant was doing the work of 700 full-time customer service agents, had handled 2.3 million conversations in its first month, and was directly responsible for an expected $40 million profit improvement in 2024. CEO Sebastian Siemiatkowski toured podcasts and conference stages as living proof that AI was about to gut customer-service headcount across the industry. Fifteen months later, in May 2025, Klarna began quietly hiring human customer service agents again. Siemiatkowski admitted, on the record, that the cost-cutting had gone too far and quality had suffered.
The reversal came in stages. Klarna's AI assistant was technically successful at the easy parts of customer service: order status lookups, basic refund flows, FAQ-style responses. What it couldn't do was tell the difference between a routine inquiry and a high-stakes situation. A customer in genuine distress about a payment dispute was treated to the same friendly, AI-generated responses as someone asking about delivery dates. Escalations got lost. Repeat customers churned. Trustpilot scores slipped.
By late 2024, internal data showed the AI was hitting roughly 80% accuracy, about the same as the human agents it replaced, but with a critical difference. The 20% it got wrong was the same 20% across all customers, every time. The errors weren't random. They clustered around exactly the cases where a human agent's judgment had been most valuable. Siemiatkowski's public reflection in 2025 was crisp: "We focused too much on efficiency and cost," and Klarna began rolling out a hybrid model where AI handled routine work and humans handled anything requiring discretion or empathy.
The broader takeaway, increasingly cited in AI-strategy circles, is that AI systems that don't know when they're wrong are dangerous in proportion to how often they get put in front of customers. The Klarna AI was confident on the easy questions and confident on the hard ones, and the hard ones were where the damage compounded.
A vanilla language-model deployment has uniform confidence: every response comes out with the same tone, the same fluency, the same apparent assurance. This is a feature of the architecture, not a bug: the model is trained to produce coherent text, and visibly uncertain text is, by definition, less coherent. So the model is implicitly punished during training for sounding uncertain, and rewarded for sounding fluent.
The downstream effect is that the user has no signal about which outputs are robust and which are guesses. A Klarna customer asking about a delivery date and a Klarna customer asking about a fraud charge got responses that felt equally authoritative. In reality, the first was a well-trodden query path with high-confidence training data; the second was a high-stakes edge case the model had limited exposure to. The model couldn't tell the user which was which.
The same uniform-confidence problem haunts AI analytics. An AI insight that says "Your conversion rate is 3.2% this month" (a routine lookup) and an AI insight that says "Your conversion drop was driven by a third-party tag breaking on mobile Safari" (a multi-step causal inference) come out with the same authoritative tone. They are vastly different in how trustworthy they are. Without a confidence layer that distinguishes them, decisions get made on both with the same weight.
Askalytics gives every insight a 0 to 100 confidence score, capped by code checks of the evidence and reviewed by a second AI.
The architectural fix has two parts: produce honest confidence scores based on the strength of the underlying evidence, and verify the model's interpretation against a second, structurally different model.
Askalytics' AI Insights surface every insight with a 0 to 100 confidence score that starts as the model's own rating and is then held to the evidence by code checks: every figure must appear in the data it cites, or the score is capped. A second AI on a different provider then reviews the same evidence, and the weaker verdict is the one you see. Every number is tied to the query that produced it so you can check the evidence yourself.
On top of that, the Confidence Check runs a different AI model (OpenAI's ChatGPT) over the same data and answer the narrator (Anthropic's Claude) produced. If the second model agrees with the interpretation, the score reflects consensus. If it disagrees, the conflict is surfaced. This is structurally different from asking the same model "are you sure?", which is notoriously weak, because a model tends to have the same blind spots on both passes.
Klarna's problem, translated to analytics, would have looked like: a routine "what was last week's revenue?" answer and a complex "what caused last quarter's drop in conversions?" answer coming out with the same confidence and the same tone. Cross-provider verification splits them. The simple one gets confirmed, ships at high confidence. The complex one might surface a disagreement between providers, ship at lower confidence, and explicitly say so, letting the user decide how much weight to put on it. Honest uncertainty, communicated cleanly, beats false confidence dressed up as fluency.
The Klarna reversal isn't a story about AI being bad. It's a story about AI being uniformly confident across tasks of very different difficulty. The fix is not to make the AI better, because frontier models will keep getting better and the problem will persist. The fix is to make the system around the AI honest about what it knows and what it doesn't.
For more on how Askalytics handles confidence, read how AI Insights are scored and our approach to cross-provider verification.