SGTech Community Sharing · AI Security

Trusted Architecture for the Future

Contributed by Genie Sugene Gan (Dr.). Written from the leadership seat: what changes for a security function once the systems it protects stop executing instructions and start making decisions.

Control points 4.4-01 Human-in-the-loop · 4.3-11 Continuous monitoring and logging · 2.1-01 Supply chain securityAlso touches 2.4-01, 1.1-01, 4.4-06.

Cybersecurity's next job isn't guarding access. It's governing decisions.

For almost thirty years, cybersecurity has rested on one assumption: computers do exactly what we tell them. If something went wrong, someone had broken in.

That assumption shaped everything we built — how we designed networks, managed identity, and responded to incidents. We built systems we could understand, and therefore systems we could secure.

That assumption no longer holds. Today's systems don't just execute instructions. They learn. They reason. Increasingly, they decide. And once machines start deciding, the question that matters most changes.

It's no longer who is allowed to access this system. It's what is this system allowed to decide.

Those sound similar. They aren't. One is about access. The other is about judgement — and judgement is far harder to secure, because a system can make the wrong call without ever being compromised.

Fig 1 Thirty years of security practice assumes the left-hand picture: to get a bad outcome, someone has to get in. On the right, every control held and the outcome is still wrong. Nothing in an access-shaped defence detects that.

01When nothing is broken, but everything is wrong

Consider Australia's Robodebt programme. An automated system generated wrongful debt notices for hundreds of thousands of citizens. There was no cyberattack, no stolen credentials, no breach of any kind. The system worked exactly as designed. Its conclusions were simply wrong, at scale.

The cost wasn't just financial. It was public trust — and trust is far harder to rebuild than it is to lose.

Every automated decision should be able to pass three tests. Can we explain it? Can we challenge it? Can we audit it? If the answer to any one of those is no, we haven't removed uncertainty. We've automated it.

Fig 2 The three tests are a gate, not a scorecard: they are joined by AND. Robodebt failed all three while every access control in the system was working perfectly.

02The decision supply chain

For years, the software industry has invested in securing its supply chain: where code came from, who wrote it, whether it can be verified. AI needs the same discipline. Except its output isn't software. It's a decision.

Every recommendation, prediction, and autonomous action is the product of a chain — and if any link in that chain loses integrity, so does the decision. I think about that chain in four layers.

Fig 3 Four links, four different failures, none of which requires an intruder. The bar is the reason the order matters: the same error caught at the data layer is a correction, and caught at the action layer is an incident.

Data. Every decision begins here. The real question isn't how intelligent the model is, it's what reality the model has actually been taught. Bad data is a distorted map: the traveller can navigate it perfectly and still end up in the wrong place.

Models. Accuracy and latency matter, but they're no longer sufficient alone. A highly accurate model can still be steered off course by prompt injection or adversarial inputs — techniques that don't stop it from working, just persuade it to work differently.

Agents. This is the biggest shift in AI. Models generate answers; agents make choices — which tools to use, which actions to take. Once execution enters the picture, architecture becomes everything. An autonomous agent shouldn't be trusted because it's intelligent. It should be trusted because its architecture constrains what it can do.

Action. This is where a decision becomes reality — money moves, access is granted, treatment begins. By this point, the decision has left the digital world, and its consequences are far harder to reverse. Every critical action deserves one last question: should this actually happen?

03Three principles that don't expire

Technology will keep changing. These three won't.

The three that outlive the tooling
Principle	What it means	The question it answers
Provenance	Every important decision should be traceable within minutes: what data informed it, which model generated it, which agent executed it.	If we can't reconstruct a decision, on what basis are we trusting it?
Guardrails	Intelligence without boundaries isn't autonomy, it's unpredictability. Every autonomous system should know not just what it can do, but what it must never do.	What is this system architecturally unable to do, regardless of what it concludes?
Continuous verification	AI keeps learning after deployment, so correctness can't be assumed once and forgotten. It has to be demonstrated, continually.	When was this last shown to be right — not when was it last approved?

Trust isn't certified. It's earned, every day.

04Trust is not a feature

If there's one idea worth taking away, it's this: trust isn't a product, a checklist, or a line in a compliance report. It's the outcome of architecture.

Every time we build an AI system, we're deciding what evidence matters, what's permitted, and what oversight remains. We're deciding the conditions under which people are willing to trust the technology we build. That's not only a technical decision. It's a leadership one.

Fig 4 The two lines have different mechanics. Capability rises as a by-product of everything else the industry is doing. Trust has no such slope of its own — it only moves when someone deliberately builds a step, and the dashed line is what happens when no one does.

Power scales naturally. Capability scales naturally. Autonomy scales naturally. Trust does not. It has to be designed, deliberately, and earned continuously.

The most important infrastructure we build over the next decade won't be digital. It will be trust itself.

Edited by Ding Sun, AI Workgroup, SGTech CSC Chapter.
SGTech Community Sharing · AI Security | From the leadership seat