C.S.O.
Crickets asked about risks by senior security officials at NCSC meeting
cyberark Peter Garraghan, CEO of Mindgard and Professor of Distributed Systems at Lancaster University, asked the CYBERUK audience for a show of hands. How many of you have banned generative AI in your organization? Three hands went up.
“So how many of us actually, deep down, raise our hands and really understand the security risks that come with controlling AI systems?”
Not a single person in the crowd of 200 security-savvy people raised their hands.
“So everyone is using generative AI, but no one understands how secure it is in the system,” Garrahan responded. “The cat came out of the bag.”
Excerpts from a session at last week’s annual conference of the UK’s National Cyber Security Center (NCSC) vividly illustrate how some organizations are deploying AI haphazardly, without much consideration for the broader implications.
This is also something that government agencies are actively trying to dissuade companies and government departments from doing, as these dangerous introductions increase the attack surface, especially for companies with critical supply chain roles.
The NCSC published a report on this issue on the first day of CYBERUK 2025. The report not only noted that there is a “realistic chance” that critical systems will be vulnerable to sophisticated attackers by 2027, but also that all organizations that have failed to integrate AI into their cyber defenses by then will be significantly more vulnerable to new types of cybercriminals.
The report, released by Senior Minister Pat McFadden, claimed that by 2027, AI-powered attackers will further reduce the time it takes to exploit vulnerabilities. In recent years, this period has been reduced to a few days, and officials believe it will continue to be reduced as AI-powered vulnerability research becomes more widespread.
An NCSC spokesperson said: register: “Organizations and systems that fail to respond to AI-powered threats will be more exposed to vulnerabilities and subsequent exploitation, risking further vulnerabilities within supply chains. This will strengthen the overall threat to the UK’s digital infrastructure and supply chains across the economy.”
“NCSC’s Supply Chain Guidance is designed to help organizations effectively manage and monitor their supply chains. We encourage organizations to use this resource to better understand and manage their risks.
“This is also why market incentives need to exist to build resilience at scale and at an accelerated rate.”
Establishment of AI…before safety measures are taken
Ensuring that cybersecurity basics are applied across the board when deploying AI systems is critical to mitigating the threat AI poses to businesses, which experts expect will be developed faster than safety in the rush to gain market share.
AI models are rapidly becoming deeply ingrained in organizations’ systems, data, and operational technology, making common AI-related attacks dangerous to those business assets, the report notes.
Consider direct and indirect prompt injection, as well as software vulnerabilities and supply chain attacks. With AI-connected systems, all of these attacks have the potential to facilitate widespread access to the enterprise environment, and the necessary controls must be put in place to mitigate these risks.
Garrahan talked about a recent penetration test his company conducted on Candleshop’s AI chatbot. This type of AI technology is currently being rapidly implemented by most companies to keep up with the corporate Joneses.
The chatbot used large-scale language models (LLM) to help the company sell candles. Garraghan said the security was in place and his company was able to breach it, posing security, safety and business risks.
The security risk in this case is that a quick engineer could run a reverse shell on the application, allowing an attacker to extract system data. Safety risks could include designing a chatbot to provide instructions on how many candles it takes to burn down a house, and business risks could arise if a chatbot was designed to divulge information about how a company makes its candles.
Although these specific outcomes did not occur in the same company, in Gallagan’s view they serve as realistic consequences of deploying AI tools without proper governance in place.
The NCSC also warned of potential risks, saying that insecure data processing processes or configurations could allow transmitted data to be intercepted, credentials stolen, or user data misused in targeted attacks.
When asked how it intends to support UK organizations to meet the demand for cyber resilience against AI-enabled cyber-attacks, the NCSC said it would keep a close eye on guidance and advice published throughout the year.
a spokesperson said The Leg: “Cyber threat actors are almost certainly already using AI to enhance their existing tactics, techniques, and procedures, so it’s important that organizations of all sizes have a strong cybersecurity baseline to protect themselves.
“The NCSC, in partnership with the Government, continues to focus on improving digital resilience across the UK. This includes publishing a range of advice and guidance to help organizations take action and become more resilient to cyber threats.”
“For those most in need, we expect the largest technology companies, often their suppliers, to adapt to future threats and fulfill their corporate social responsibilities.” ®
