Gartner: Bad AI governance could be deadly

Gartner expects that by 2029, legal claims over deaths caused by poorly deployed automation solutions will dramatically increase.

In a presentation as part of the 2025 Gartner Data & Analytics Summit in Sydney, a Gartner vice president analyst, Carlie Idoine, presented on the company’s top data and analytics predictions this year.

Iodine’s predictions all centred around AI; she said technology professionals working on AI deployment in enterprises needed to be fully cognisant of the risks.

“What we're doing could have very existential life and death consequences," she told the audience.

"We have to expand our legal and risk management by working … really tightly with those legal and risk management partners. We have to have more fit-for-purpose data quality. There’s lots of implications here.”

The risk of human death and the legal exposure that could result from it was the most dire prediction in the presentation, but others pointed to a drastically changed IT industry.

Software as a service (SaaS) may be headed toward displacement as, Idoine said, AI agents will replace 30% of SaaS application user interfaces (UIs), “relegating the SaaS application to be a semantically enriched domain data source”.

“We hear of early adopters replacing the UI of applications like Workday and Salesforce with AI agents, leveraging the application as a domain database - we don't need to have this UI piece of it anymore," she said.

She added this would likely eventuate in organisations questioning whether they need to pay for the applications at all and could instead build their own AI layer to deliver the same results.

“There's going to be a breakeven point ... you're going to need to continue to assess for that build versus buy discussion," she said.

"That would also mean that your existing data management platform and practice may be the key to lowering the cost and implementing this type of transition, so it’s a different way to think about the data management applications that you have within your organisation.”

Other predictions included that 50% of business decisions will be augmented or automated by AI agents for decision intelligence by 2027.

A major risk here is that without proper metadata labelling, these decisions could be based on false assumptions due to hallucinations.

She added that the worse the labelling, the more expensive a large language model is to use in the long run.

“Poor semantics equals more hallucinations," she stated.

"This is an established fact we know to be true and there's a direct correlation, therefore, to poor rounding or prompting of the LLM, and hence more tokens being used, and hence more cost.”

Idoine said Garnter’s prediction is that organisations that prioritise semantics in AI-ready data will increase their generative AI model accuracy by up to 80% and reduce costs by 60%.

She emphasised that as AI tools become more prevalent and ingrained in systems, good data governance will become of paramount importance.

“AI agents for decision intelligence aren't a cure all, nor are they infallible," she said.

"We have to be careful about that. They must go hand in hand with effective governance and risk management.”

She asked the audience: “Does it scare anybody, the power of AI agents? It should."