Incorporating artificial intelligence into an industrial process adds not just new risks to assess, but risks that are constantly evolving as the technology advances. However, research from assurance firm DNV concludes that existing risk management principles can still be applied to ensure safety in this new era of AI-enabled systems, albeit with adaptations.
In its new position paper, Assurance of AI-Enabled Systems, DNV says that, although AI-enabled systems add complexity and uncertainty, proven assurance methods developed in safety critical industries, such as energy, maritime, nuclear, and aerospace, already provide a robust starting point for addressing the challenges.
The authors say that AI reshapes risk because it does not operate as a fixed, predictable component, making traditional one-time assurance insufficient and necessitating continuous and adaptive assurance throughout the lifecycle.
“When AI fails, it is rarely just a ‘glitch’. It’s often the result of hidden risks and biases, unclear responsibilities, or unexpected interactions between AI, people, and the
environment where it operates. In addition, these systems evolve through learning, updates, and changing contexts, making traditional assurance methods insufficient,” according to the report.
The authors illustrate the pitfalls if continuous and holistic risk monitoring does not take place by looking at a number of variables that could compromise AI systems used in driverless robotaxis. For example, an update that improves pedestrian detection may also hamper the vehicle’s performance in fog. Alternatively, a car that has never encountered a snowdrift in training might make a poor driving decision if it finds one blocking its way out on the road and need re-training.
However, Christian Agrell, Programme Director for AI Assurance at DNV, says the more complex risk assessment needed does not require a completely new approach, but can build on existing building blocks.
“Creating trustworthy artificial intelligence does not require us to start from zero. We already have strong foundations in modern assurance and risk science and our long experience managing digital technologies in high-risk environments,”
He says that by applying those principles thoughtfully, systems could be built that remain safe and reliable, even as they evolve.
Guiding principles
In the report, DNV draws up a series of foundational principles that it says can help create trustworthy AI.
A system model that captures the entire AI-enabled system needs to be developed, reflecting how AI interacts with humans, digital and physical components, and its operational environment. This should provide an understanding of new types of AI behaviour, unintended interactions and context-specific risks that would not be detected by examining the AI component alone.
A modular approach needs to be used in devising a risk model, so that it breaks down complex systems and the emergent risks into manageable parts across system levels. Meanwhile, claims that a system is safe need to be tied to verifiable evidence, assumptions and rationale.
Assurance that adapts as AI evolves is crucial. To remain trustworthy, assurance needs to include real-time monitoring, regular updates to evidence, and re-evaluation of risks and requirements so that confidence in the system remains valid throughout its lifecycle, DNV says.
The report concludes with calls for organisations to recognize AI risks as systemic, not isolated, to acknowledge the new challenges through a shift in risk and assurance thinking, to build on appropriate methods ready for AI, and to ensure they comply with evolving regulatory requirements.
The report also highlights the dangers of failing to address ethical and societal concerns.“All risks with AI are not yet uncovered, understood, and captured by regulation. For responsible use of AI, it is important to take ethical and societal concerns seriously. Failing to do so may also cause significant reputational risk,” the authors say.
(Photo: Shutterstock)
