AI has huge potential to transform procurement. Generative AI can automate routine processes, analyse large volumes of information and support decision-making, while agentic AI will take on tasks and interact with other business systems.
But as procurement teams adopt AI, there is a capability issue that has to be addressed. As well as ensuring employees have the skills to use it, organisations need procurement professionals with the security awareness and judgement to recognise potential risks, challenge unsupported claims and know when specialist cybersecurity expertise is required.
In addition, while procurement teams need to be able to use AI securely in their own work, organisations need specialist expertise to assess, procure and secure the AI being introduced into other parts of the business. These are complementary capabilities but they are not the same.
Using AI securely in procurement
AI is becoming part of everyday procurement activity. It is used to analyse information, automate processes and support decisions, which means teams need enough security literacy to understand that AI can be exploited, recognise its limitations and know when its outputs need to be challenged or independently verified.
Elements of procurement roles will also shift from doing tasks to directing and supervising the AI systems that do them. Because of this, AI-augmented roles are emerging, where the value is not only in having the right AI tools but in knowing how to use them safely and effectively.
In practical terms, this means procurement teams need to develop the judgement to understand what an AI system is doing and when human intervention or further expert scrutiny is required. A system producing a supplier summary or analysing information presents one set of considerations. An agent capable of accessing data, making decisions or taking actions across multiple systems presents another. Essentially, the greater the autonomy, the more important it is that people understand the system’s capabilities, limitations and potential failures.
This does not mean procurement professionals need to become cybersecurity specialists. They do, however, need to know what questions to ask, recognise when an answer or claim needs to be queried and know when to bring in the appropriate security expertise.
Securing AI in procurement
When AI is embedded in procurement processes, securing the technology itself is clearly essential. It is vital to secure AI systems, defend against AI-enabled attacks and put appropriate governance around autonomous technologies. However, organisations also need the cybersecurity capability to put these measures into practice.
Security professionals need to understand new threats including prompt injection, data poisoning, model exploitation and attacks against AI agents. They also need to understand how AI changes established security assumptions, including data flows, permissions and trust boundaries.
This is also important when procurement teams are evaluating AI-enabled technology for other areas of the business. Vendor assessments have to go beyond conventional security questionnaires and certifications. It is essential to understand what an AI system can access, what permissions it has, where its data comes from, how its outputs are validated and what happens when the system is manipulated or behaves unexpectedly.
Procurement can’t be used as a substitute for the security team’s involvement. Its role is to help identify the right questions, challenge unsupported vendor claims and ensure that the relevant issues are properly considered. But the deeper technical assessment, testing and security validation has to sit with cybersecurity specialists.
The challenge for organisations is connecting these two capabilities effectively.
Cyber skills have to evolve alongside AI
AI security can’t be seen as a niche specialism that is understood only by a small group of experts. With AI being embedded across business applications and workflows, the security capabilities to assess and secure it also needs to develop. The latest Hack The Box Workforce Intelligence Report shows AI security is already a prominent area of professional development, with AI penetration testing, prompt injection, model exploitation and agentic AI security among the areas practitioners are focusing on.
When an organisation is either evaluating or deploying AI, it has to have access to people who understand these threats well enough to ask the right questions, assess the answers and test the technology where appropriate. In other words, an organisation’s cyber capability has to develop alongside its AI adoption.
The development also needs to be practical. Security professionals should have the opportunity to work with AI systems in realistic environments, test how they behave under pressure and understand how attackers might exploit them. Capability needs to be developed, practised and demonstrated. It can’t just be assumed.
Knowing when not to trust AI
AI can produce outputs that appear convincing but are inaccurate, incomplete or inappropriate and there is a danger people will accept something that seems plausible without questioning it. This automation bias is a risk as human judgement can weaken when people get used to accepting AI-generated outputs without checking them.
Procurement professionals using AI need to know when an output should be challenged or independently verified and when an issue requires specialist security input. Cybersecurity professionals, meanwhile, need to be able to interrogate AI and understand what information it has considered, what assumptions it has made and what it may have missed.
These capabilities can’t be developed by teaching people how to write better prompts. They have to have an understanding of how AI systems behave, where their limitations lie and for security professionals, how an attacker might exploit them.
Don’t automate away cyber expertise
There is also a longer-term workforce challenge. AI is effective at automating routine and medium-complexity tasks but those tasks have traditionally helped junior professionals develop their expertise. If AI takes over too much of this work without a development strategy in place, organisations could weaken their future talent pipeline, creating a capability gap over time.
When it comes to cybersecurity, AI adoption and cyber workforce development must happen together. Security professionals should be able to use AI to accelerate their work while continuing to develop the underlying knowledge and judgement to investigate independently, recognise failures and intervene when necessary.
For procurement teams, the challenge is ensuring that increasing reliance on AI does not remove the human judgement needed to question outputs and recognise when an issue needs to be escalated.
Building an AI-ready procurement function
Procurement teams need enough security knowledge to understand how AI can be exploited and when its outputs cannot be trusted. They also need access to cybersecurity expertise that can challenge vendor claims, understand the potential attack surface and determine what security testing is appropriate.
Procurement professionals should not be expected to carry out these technical assessments themselves. But they do need to know when they are necessary, ask the right questions and ensure the relevant specialists are brought into the process.

AI cybersecurity capability also has to be part of ongoing workforce development. A penetration tester needs to understand how to assess AI-enabled applications and agents. A SOC analyst needs to know when AI-generated analysis requires independent validation. Security architects need to understand how AI changes data flows, permissions and trust boundaries.
New AI models will emerge, with agents becoming more autonomous and further blurring the boundary between human and machine activity. For procurement, AI and security literacy and informed judgement will be essential. For cybersecurity teams, they need to develop and maintain the specialist technical capability needed to assess, test and secure those technologies.
The biggest barrier to successful and safe AI in procurement is having the right combination of procurement judgement and specialist cybersecurity expertise and connecting those capabilities effectively.
By Nikos Fountas, Chief Operations Officer, Hack The Box.























