OpenAI has announced its decision to acquire the AI security platform, Promptfoo, to embed security testing into its AI agents.
This integration will expand its recently released enterprise platform, Frontier, to strengthen how systems are tested, evaluated, and secured before deployment.
The acquisition aims to address the risks of AI agents when they access internal data, call APIs, and workflows, so organizations can deploy agents with more reliability, governance, and compliance controls.
Srinivas Narayanan, CTO of B2B Applications at OpenAI, highlights how this acquisition will allow customers to build and deploy AI systems with built-in testing and security.
“Promptfoo brings deep engineering expertise in evaluating, securing, and testing AI systems at enterprise scale,” he explained.
“Their work helps businesses deploy secure and reliable AI applications, and we’re excited to bring these capabilities directly into Frontier.”
Understanding Risks as AI Agents Interact with Real Business Data
This acquisition highlights a growing problem as more AI agents are being deployed and start to interact with real systems inside companies, reading internal documents, query databases, call APIs, and may be required to trigger actions such as ticket approval, sending emails, or executing workflows.
This shifts an AI agent from being a text generator to an automated employee, introducing security risks that traditional software security tools are not designed to handle.
Problems can begin to occur when a language model misinterprets an instruction, taking in both the prompt and external content that may include malicious instructions, causing the model to treat it as legitimate guidance.
In the case of prompt injection attacks, a model cannot reliably distinguish between developer-written and data embedded instructions, leading to data leakage or unauthorized actions.
Other risks can evolve from agent access to internal data stores, such as customer databases, financial reports, and company knowledge bases, which are often connected, meaning a model can reveal sensitive information if the wrong question is asked.
These failures can occur quietly and at scale, meaning an agent could perform unintended actions without developers detecting, leading to significant data breaches, compliance violations, or operational disruption.
How Promptfoo Enhances Testing, Reliability, and Deployment of AI Agents
By solving the issue of how to safely build and operate AI agents inside real organizations, this acquisition can allow enterprise customers to improve their security, testing, and governance for AI systems that can take actions.
This ensures that AI systems are safer and easier to deploy at scale as more companies move from simple chatbots to more capable AI agents that influence real business operations, requiring them to test AI behavior before it’s released into production.
Instead of writing traditional unit tests, customer developers can define scenarios and expected behavior with the AI in large batches to judge where the model might fail in a system.
In return, Promptfoo can simulate prompt-injection attacks and attempt to extract confidential information or edge cases that might cause the model to misuse a connected tool.
Having recorded the responses, the system then evaluates the model against the defined criteria, producing a report that shows where the model fails or behaves unpredictably.
