Breaking open the Agentic AI “Black Box”

Now, new “Agentic AI”, or "AI agents", systems designed for goal-directed autonomy, promise to tackle complex tasks with minimal human supervision and transform how software and automation are conceived. [1],[2],[3],[4],[5]

While the hype around fully autonomous AI is strong, the reality is tempered by technical limitations, data quality issues, and compliance needs. The core tension lies in balancing the value of autonomy by offloading complex tasks and achieving results at scale with the inherent risks of “black box” decision-making. The probabilistic nature of LLMs means outputs can be unpredictable or erroneous, demanding robust guardrails. [6]

Security, privacy, and regulatory frameworks like revDSG, GDPR and the EU AI Act further necessitate human oversight and traceability, especially in critical sectors. [7],[8],[9] True value from Agentic AI, therefore, emerges when it is anchored in controlled workflows and human oversight. We explored the promises and challenges of Agentic AI in a Proof of Concept (PoC) involving a joint team of Swisscom's Data C AI Consulting and Noumena Digital engineers(opens in new tab).

Agentic AI systems represent a shift from reactive AI models to proactive systems capable of multi-step workflows and decision-making to achieve goals. These LLM-powered autonomous programs perceive their environment, plan, invoke tools e.g. through APIs and Model Context Protocol (MCP) servers, and interact with digital surroundings and other agents, using e.g.Agent-to-Agent (A2A) implementations. [10]

Agentic AI operates on a continuous iterative cycle, described as a Perception-Reasoning- Action-Feedback loop or, similarly, a Thought-Action-Observation cycle: [1],[10],[11],[12],[26]

1. Perception (Observation): The system ingests and interprets diverse data, structured (databases, APIs) and unstructured (images, emails).

2. Reasoning (Thought): LLMs or multimodal models determine the next best action based on the interpreted data and the overall goal, selecting from available tools.

3. Action: The agent executes the chosen action, such as calling an external API or an MCPserver, under predefined guardrails.

4. Feedback (Observation): The action's outcome is returned to the system, informing the next reasoning step or plan adjustment.

Humans can be integrated at any step for guidance or approval ("human-in-the-loop"). Agents are envisioned to exist on a spectrum of autonomy, from single LLM calls within human-coded workflows to highly autonomous systems that dynamically choose their actions. [4]

Agentic AI offers various benefits but also introduces significant risks that businesses mustprevent. In the following, find a list of successful use cases:

• Insurance - Faster claims processing: Agentic AI streamlines the claims lifecycle, from initial claim submission to settlement, using NLP and image recognition to autonomously assess claims, detect fraud, and automate communication, leading to faster, (potentially) more accurate resolutions. [13],[14],[15], [16]
• Manufacturing - Predictive maintenance G digital twins: AI agents analyze machinery data (e.g., from IoT sensors) to predict failures and trigger proactive maintenance. [17],[18] A collaboration by KION Group, Accenture, and NVIDIA uses digital twins to simulate and optimize warehouse operations, where virtual autonomous agents manage goods and routes. [19],[20] Swiss startups like www.ethon.ai or www.cerrion.com use AI agents to deliver process insights to operators and/or trigger further workflows upon problem detections.
• Public sector - Workflow automation: The UK government's “Humphrey” AI suite automates administrative tasks like transcribing meetings and analyzing public consultations, aiming to free up civil servants for strategic work. [21],[22]

However, AI also has its prominent failure cases. In a highly publicized case in 2024, an Air Canada chatbot provided incorrect information about bereavement fare policies. The airline was held liable for it, showing that businesses can be responsible for errors made by their AI systems. [23],[24]

For the PoC, we utilized Noumena’s technology stack, which emphasizes enterprise-grade security, fine-grained access control, audit trails, and permissioned orchestration. To put this to the test, Swisscom and Noumena partnered in a proof-of-concept (PoC) project to explore secure Agentic AI in insurance claims processing.

Traditionally, claims are classified, reviewed, and adjusted. Classification directs the claim to the correct department. The review assesses the completeness, trustworthiness, and validity of the information, determining eligibility and compensation. Denied claims require clear explanations. A department-specific adjustor cross-checks reviews for quality and fairness.

AI can automate classification and review. In this PoC, an LLM classified claims and provided access to department-specific LLM reviewers. To ensure human oversight, the adjustor typically remains human. However, to illustrate the flexible implementation of policies, the PoC included a rule allowing reviewer recommendations for small amounts to bypass human adjustment.The PoC focused on integrating services seamlessly, orchestrating AI and human interactions along traceable workflows, restricting agent (whether AI or human) access to data on a need-to-know basis, and implementing transparent AI delegation policies.

Processing ffow for insurance claims as implemented in the PoC. Claims processing (health, car, household) used Al for classification and department-specific review, leveraging Claude 3.5 Sonnet via AWS Bedrock. Human adjustors handled final decisions.

This PoC demonstrated Agentic AI's practical application in automating complex tasks,specifically in:

• Reasoning: LLM services successfully replaced human classification and review,providing accurate answers using claim and policy data.
• Process automation: AI services enabled full automation, with human intervention only for explicitly required pathways.

The PoC also validated the effectiveness of Noumena’s technological approach in addressingsome key challenges of Agentic AI in the enterprise context:

• Access control: AI agents require context-aware permissions. The Noumena Access model provided a framework for orchestrating AI agents with security guarantees, combining a declarative language for policies with runtime attributes to ensure data privacy and timely, need-to-know access through fine-grained control.
  
• Auditability: Enterprises must enforce data minimization, ensuring agents access only the information required for specific tasks. GDPR compliance necessitates strict controls over data deletion and localization. Traceability is required for every action taken by an AI agent. The solution ensured complete traceability by logging all AI agent actions and decisions, crucial for compliance and accountability.
  
• Accountability G flexible delegation policy development: AI agents need temporary, process-linked access with non-repudiation (logged, traceable actions). Organizations require clear audit trails of agent decisions. Noumena Access provides dynamic access and ensures non-repudiation through an audit trail that links actions to authorized users. The PoC enabled flexible workflow-as-code for AI service delegation and human oversight (e.g., claim amount thresholds).
  

Overall, the PoC successfully showcased Agentic AI's potential to reshape business processes through efficiency gains. It also showed Noumena’s technology's ability to enable secure, auditable Agentic AI systems by providing execution guardrails.

Beyond technology choices, Agentic AI projects need to follow a well-thought-out strategy to ensure success. A phased, strategic approach that embeds risk mitigation is crucial for the deployment of Agentic AI. Follow some guiding principles when implementing agentic AI PoCs and deployments:

• Right use case: Start with essential, low-to-moderate risk internal tasks.
• Clear boundaries: Strictly limit agent capabilities and action space initially. Make use of the human-in-the-loop strategy.
  
• Measure G log: Define success metrics and record all agent activities for audit and compliance purposes. [6]
  

Moreover, make sure to follow a phased rollout approach:

1. Controlled pilot and beta deployment: Limited scope, heavy oversight, with goal to validate core functionality. With a limited production and restricted user base, test of scalability and real-world performance.

2. Broad adoption: Wider rollout, incremental expansion of responsibilities, formal integration into business processes with governance.

3. Ongoing improvement G maintenance: Continuous model updates, performance reviews, and adaptation to new rules or data.

Transparency with stakeholders and fostering cultural readiness are crucial throughout. [25]

Agentic AI marks a new frontier in automation and intelligence. To realize its promise, companies must break open the black box. This makes systems auditable, understandable, and controllable through oversight and constraints as explored in the PoC explored above.

Pragmatism is paramount. Organizations should experiment boldly yet govern responsibly. They should invest in human expertise and transparency tools. AI should be seen as a powerful assistant. However, its outputs should be questioned and verified, especially in high-risk scenarios. Policymakers and industry groups must develop standards and audit frameworks. This could be similar to those in finance or safety engineering, to ensure responsible innovation. [7]

Auditability and control will enable, rather than stifle, innovation. Trustworthy, transparent, and controllable agents will be deployed more widely and boldly. "Pragmatic agency" means building agents that are as autonomous as possible within well-understood guardrails, continually illuminating their inner workings, and maintaining a human-in-command approach. This journey demands diligence. But by transforming the AI "black box" into a "glass box," we can harness its power for a future of effective human-AI collaboration.

Authors:

• Pierre Lugan, Head of Product (Noumena Digital AG)
  LinkedIn(opens in new tab)
• Tim Giger, Principal Consultant, CTO Data & AI Consulting (Swisscom)
  LinkedIn(opens in new tab)
• Severin Husmann, Senior Consultant Data & AI (Swisscom)
  LinkedIn(opens in new tab)