The Emerging AI Security Opportunity

The AI revolution is here. What does that mean for security teams?

Generative AI products are being adopted by individuals and enterprises at a rate we’ve never seen before: ChatGPT is the fastest product to reach 100M users. AutoGPT is the fastest open source project to reach 100K GitHub Stars. Much like the cloud, SaaS, mobile, BYOD, or hybrid work, this change in how we work creates new cybersecurity threats. Unlike those previous waves, enterprises don’t have years to migrate and ensure security, leading companies like Apple and Samsung to outright ban tools that could make their employees dramatically more productive.

I’ve spent time over the last few weeks and months with founders and CISOs to better understand the new security threats that generative AI creates and to learn more about the emerging approaches for countering those threats. The space is nascent—it’s hard to predict what next week will bring in AI innovation, let alone set security strategy for the next year—but I’m increasingly excited about the opportunity for new startups to enable secure AI.

At a high level, we can break the problem down into four categories:

1. Using AI securely

2. Protecting against AI-enabled attacks

3. Building AI securely

4. Protecting models in production

These categories are in descending order of predominance. Only a fraction of companies need to deeply invest in protecting models in production right now, but every CISO needs to think about how their employees are using AI on a day-to-day basis.

Using AI Securely

Rapid, bottom-up adoption of AI tools that make it easier to do everything from writing to coding to creating slideshows forces the question, “What data am I sharing with models and the companies behind them?”

At the moment, it’s difficult for security teams to know. This can be particularly concerning because models are constantly improving based on the data shared with them, amplifying the risk of sensitive data getting out into the wild.

Of course, AI-enabled companies can and do take measures to ensure the security and privacy of customer data. But as we’ve seen in traditional software, assurances are not enough; CISOs look for third-party validation through compliance certifications like SOC 2 or ISO 27001. We can expect that similar standards will emerge in AI, and as a result, companies will emerge to help AI companies meet those standards, creating opportunity for a “Drata* for AI.”

To continue the parallel to traditional SaaS, ensuring SOC 2 for all vendors is a necessary but not sufficient requirement for CISOs. Internal teams still need to take their own measures. There are two clear areas where security measures can be inserted:

• Prompt layer: Security teams can monitor (or in more drastic cases limit) what is shared via prompts. The integration point here is not yet clear, but a few companies have pitched a GPT “firewall.”

• Model/Agent access control: AI-enabled agents (e.g. AutoGPT, BabyAGI) are increasingly being used to solve complex tasks and will soon begin to automate routine work. To do so, they’ll need read and write access to other software systems within an enterprise, which will need to be monitored and limited where possible.

Protecting Against AI-Enabled Attacks

The same AI tools improving productivity for employees are improving productivity for attackers. ChatGPT can very easily generate phishing emails and has already led to a massive rise in phishing attacks. AI agents can at a minimum increase the scale and efficiency of existing cyberattacks, and potentially help generate entirely new attacks.

Part of the answer here for CISOs will be to double down on their existing security approaches. Leveraging detection and prioritization tools to filter noise and automated remediation tools for greater scale are two ways to match the increase in attacks.

Phishing is likely the first attack vector to be completely reimagined by AI. GPT-3 (and GPT-4) can quickly generate more personalized, more convincing phishing emails. We’re already seeing companies emerge to better train employees to detect AI-generated phishing attacks and expect to see additional solutions crop up that detect and filter these attacks before they hit employees’ inboxes.

Building AI Securely

The vast majority of companies building AI in any capacity are doing so by fine-tuning an existing proprietary or open-source model. Both of these approaches require some degree of trust in the underlying foundation model. As a recent OpenAI breach has shown, model providers have a ways to go in proving they are secure.

If we’re to believe the recent leaked Google memo, AI will soon be dominated by smaller open-source models built for narrower tasks. This will create a challenge similar to the one we’ve seen in traditional open source software—open source code is exposed to vulnerabilities, forcing security teams to keep track of the libraries in their codebase and patch vulnerabilities where they exist.

It may only be a matter of time until we reach a “Log4J moment” for open source AI, in which thousands of companies realize they built on top of a model that is not secure. Attackers can insert backdoor triggers or poison training data while staying invisible to the data scientist or ML engineer downloading the model from HuggingFace. Enterprises will need developer-friendly tools that validate and monitor the security of open-source models, similar to Snyk in traditional software.

Regardless of whether you build on a proprietary or open-source model, fine-tuning for company-specific use requires you to use your own data, which introduces another set of risks. It’s difficult to precisely control to what degree a model may expose training data, making it very risky to fine-tune a model on PII or other sensitive data. We’re seeing developers turn to synthetic data tools like Tonic* to build powerful AI without exposing sensitive data. 

Protecting Models in Production

Finally, enterprises run the risk of attackers exploiting a model that they’ve put into production. ChatGPT itself offers a few examples of attack types:

• “Adversarial attacks: Adversarial attacks are a type of attack where an attacker deliberately manipulates the input data to an AI model, causing it to produce incorrect results. For example, an attacker may add small perturbations to an image that are imperceptible to the human eye but can cause the AI model to misclassify the image.”

• “Model stealing: Model stealing is a type of attack where an attacker tries to obtain a copy of an AI model by querying it with carefully crafted inputs. By analyzing the model's responses to these inputs, the attacker can reverse-engineer the model and create a copy of it.”

• “Evasion attacks: Evasion attacks are a type of attack where an attacker tries to evade detection by an AI model. By carefully crafting input data, an attacker can cause the model to misclassify or ignore certain inputs, allowing malicious inputs to slip through undetected.”

Ensuring security best practices throughout the development of the model is helpful in preventing these attacks or making them difficult for hackers to execute. But like in traditional security, we expect that a layer of runtime security will be necessary. HiddenLayer, which recently won the RSA Innovation Sandbox, has built an MLDR (ML Detection and Response) product that monitors model inputs and outputs and flags potential attacks. We’re not yet sure if this will prove to be the optimal runtime security approach, but enterprises will likely look for options with some urgency.

Startup Velocity

We’re still climbing the hype curve of generative AI, meaning that companies are far more focused on how they will use and build AI to power their businesses than the security implications of their decisions. However, a few prominent AI hacks can quickly turn boardroom conversations from “What is our AI strategy?” to “What is our AI security strategy?” In a world of tightening security budgets, this could create a rare and massive opportunity for founders.

Part of what’s been so inspiring about the generative AI boom is the unexpected sources of rapid innovation. On a weekly basis, individual developers are matching or exceeding the accomplishments of massive companies. Such a fast-paced environment gives lean teams who can move quickly to secure the use and development of AI a rare head start vs. incumbents. If you’re building in this space, I’d love to learn more. 

* Indicates a GGV portfolio company