When AI Makes a Business Worse

Technology does not have to fail technically to make a business worse.

An AI system can operate exactly as configured and still increase cost, frustrate customers, weaken employee judgment, create legal exposure, damage a brand, or insert errors into a previously stable process.

That possibility deserves more attention because most business cases are presented asymmetrically. Teams describe what happens when the technology works. Executives also need to know what happens when it is wrong.

Accuracy is not a technical detail

Generative models produce probabilistic outputs. That characteristic is useful because the systems can work flexibly with language and information. It also means an output that sounds authoritative can be incorrect.

The business significance of an error depends on context. If AI creates a weak first draft of an internal presentation, an employee can correct it. If AI invents a customer refund policy, legal interpretation, safety procedure, or financial instruction and the recipient acts on it, the consequences may be much larger.

A widely discussed example came from Canada. In Moffatt v. Air Canada, a customer relied on incorrect information supplied by Air Canada's chatbot about bereavement fares. Air Canada argued, among other things, that it should not be responsible for information provided by the chatbot. The British Columbia Civil Resolution Tribunal rejected that argument and treated the chatbot as part of Air Canada's website.

The larger business lesson is simple. Delegating an interaction to software does not necessarily delegate accountability for the interaction.

Customer-service automation can optimize the wrong thing

A support leader can reduce average handling time and still worsen customer service. A chatbot can reduce labor cost and simultaneously increase frustration. A system can produce excellent containment numbers because customers give up. The metric selected determines what management sees.

Klarna offers an instructive example of how corporate AI strategy can evolve. The company became one of the most visible advocates of AI customer service and previously said its AI assistant was doing work equivalent to hundreds of human agents. It later moderated that approach. Reuters reported in September 2025 that CEO Sebastian Siemiatkowski acknowledged the company had overemphasized cost cutting and that service quality had suffered in parts of the business. The company remained committed to AI, but increased emphasis on human service and growth.

The point is not that AI customer service does not work. It often can. The point is that optimizing for cost per interaction without measuring the value of the customer relationship can produce the wrong answer.

Automation removes friction. Sometimes friction is useful.

Companies often describe manual review as inefficient. Sometimes it is. Sometimes the person reviewing the transaction is the control.

Consider a purchasing workflow. An employee notices that a supplier's banking information changed immediately before a large payment. An AI workflow designed to maximize processing speed may treat the transaction as another item to process. A human familiar with the supplier may find the change unusual.

This does not mean every payment requires manual handling. It means process design must distinguish between unnecessary friction and protective friction.

The same applies to fraud review, hiring, quality assurance, safety, legal analysis, financial approvals, and high-value customer escalation. Efficiency is not always the only objective.

The false confidence problem

AI can also change human behavior. When a system produces polished work quickly, employees may review it less carefully than they would review an obviously incomplete draft.

That creates an important distinction between human in the loop and meaningful human oversight. A person clicking "approve" on hundreds of AI-generated decisions is technically involved. That does not mean the person is evaluating each decision effectively.

NIST's AI Risk Management Framework emphasizes defining human oversight, measuring performance and safety, documenting limitations, and determining whether a system actually achieves its intended purpose before proceeding. Human review must therefore be designed according to the risk of the decision.

AI can make experts slower

The assumption that AI necessarily improves productivity can also fail.

METR conducted a randomized controlled trial involving experienced open-source developers working on repositories they knew well. With early-2025 AI tools, the developers took 19% longer to complete tasks despite believing AI would make them faster.

That result should not be generalized to current coding tools. METR later reported indications that newer tools were providing productivity benefits, but selection effects made its later estimates unreliable.

The important business insight is different. Perceived productivity and measured productivity can diverge. Employees may feel faster because part of the work feels easier. Management still needs objective measurement.

The integration layer creates its own failure modes

A standalone chatbot is relatively contained. An AI agent connected to email, customer records, financial systems, cloud storage, code repositories, purchasing systems, or external websites can do more useful work. It can also do more damage.

OWASP identifies prompt injection and sensitive-information disclosure among significant security risks for large-language-model applications. An attacker may attempt to manipulate model behavior through instructions embedded in content the AI processes. Poorly designed systems can also expose confidential data.

As companies move from "AI answers questions" toward "AI takes actions," access controls, permissions, logging, approval thresholds, and fail-safe mechanisms become more important. Capability and risk increase together.

Bad processes remain bad processes

A company may automate an unnecessary approval, duplicate data entry, poor customer segmentation, an obsolete report, or a badly designed escalation process. The technology then performs an unnecessary activity more efficiently.

That is not transformation. The process should be redesigned before it is automated.

For every proposed automation, executives should ask: If we were designing this business today, would this process exist? If no, automating it may preserve operational debt rather than remove it.

Institutional knowledge can disappear quietly

There is another risk that appears only after automation succeeds. People stop doing the work. Initially that seems desirable. Years later, the company encounters an unusual situation and discovers that nobody understands the process deeply enough to diagnose what went wrong.

This does not justify preserving manual work merely for tradition. It does justify deciding deliberately which expertise must remain inside the organization.

Critical systems need documentation, trained owners, escalation paths, testing, and recovery procedures. Automation without retained understanding creates dependency.

Vendor dependency deserves financial treatment

AI systems often rely on third-party models, APIs, platforms, or infrastructure. A vendor can change pricing, rate limits, model behavior, features, data policies, service terms, or product direction.

Management should therefore ask before implementation: How difficult would replacement be? Can the company export its information? How much proprietary workflow logic is tied to this platform? Would switching vendors require redesigning the application? Could the business operate temporarily without the system?

Vendor dependency is not necessarily bad. Every company relies on vendors. The risk needs to be understood and priced.

The right question is not whether AI can fail

Every business system can fail. Humans make mistakes too. The relevant comparison is between the proposed AI-enabled process and the current alternative.

Suppose humans make an error 4% of the time and AI makes an error 2%. That may be a great improvement. But perhaps AI errors are more correlated. Instead of occasional independent mistakes, one model defect causes the same error across 50,000 transactions. That risk profile differs.

Good evaluation therefore measures frequency, severity, detectability, reversibility, and concentration of errors. An error that can be corrected instantly is different from one that becomes irreversible before detection.

A pre-mortem for AI

Before implementing a significant system, executives should conduct a simple exercise. Assume the implementation failed badly 12 months from now. What happened?

Customers rejected it? Employees bypassed it? Costs exceeded projections? Was the data unreliable? The model made high-impact mistakes? The integration failed? The vendor changed pricing? Regulators objected? The system created a security incident? Could nobody demonstrate ROI?

Now design the pilot to test those failure modes. That is far more useful than treating risk as a compliance checklist after the technology has already been selected.

AI can make businesses significantly better. It can also perform the wrong task faster, distribute mistakes at scale, eliminate valuable human interaction, and create complexity that did not previously exist. The dividing line is not optimism versus pessimism. It is design.

Comdatus Consulting perspective: AI risk assessment should examine what happens when the system is wrong, unavailable, compromised, misunderstood, or rejected — not merely what happens when the demonstration works.

Primary sources and research

• Moffatt v. Air Canada, 2024 BCCRT 149
• Reuters reporting on Klarna's evolving AI customer-service strategy
• METR, Early-2025 AI Experienced Open-Source Developer Study
• METR, 2026 Productivity Uplift Update
• NIST AI Risk Management Framework
• OWASP GenAI Security Project — Sensitive Information Disclosure