In most conversations with fellow investors about AI safety, I encounter one of two reactions. The first is dismissal: AI safety is an interesting academic research agenda, but it is not a commercial category with near-term investment returns. The second is confusion about what "AI safety" actually means in a commercial context, often conflating long-term existential risk research with the practical, near-term safety and reliability challenges facing enterprises deploying AI today.
Both of these reactions stem from a misunderstanding of what AI safety actually means as an investment thesis, and both are costing investors significant returns. I want to make the case here that AI safety -- properly understood -- is the most compelling category-creating investment opportunity in enterprise technology today, with a total addressable market we estimate at $50B or more by the early 2030s, and where the category leaders are being established right now.
This piece reflects my perspective specifically as an investor, not as an AI researcher. I am not making claims about the long-term trajectory of AI capabilities or the probability of various tail risk scenarios. What I am arguing is that there is a large, real, growing market for commercial AI safety products today, that this market is severely underfunded relative to its size and importance, and that the companies building in this category will be extremely valuable over the next decade.
Reframing AI Safety: What the Commercial Market Actually Needs
The AI safety community, broadly defined, encompasses everything from theoretical alignment research on hypothetical superintelligent systems to practical work on reducing hallucinations in deployed language models. For commercial investment purposes, we focus on the practical end of this spectrum -- the near and medium-term safety challenges that are blocking or slowing enterprise AI deployment right now.
The practical AI safety challenges that enterprises face today fall into four broad categories. The first is output reliability: ensuring that AI systems produce accurate, honest, and appropriate outputs without hallucinating, generating harmful content, or producing outputs that violate regulatory requirements. The second is operational security: protecting AI systems from adversarial attacks, prompt injection, data poisoning, model theft, and other security threats that are specific to AI systems. The third is governance and compliance: maintaining the audit trails, explainability documentation, bias monitoring, and regulatory reporting required for AI deployment in regulated industries. The fourth is alignment with organizational values and policies: ensuring that AI systems behave consistently with organizational policies, ethical guidelines, and brand standards across all deployment contexts.
None of these challenges are theoretical. Every enterprise organization deploying AI at meaningful scale is dealing with all four of them right now, and the available solutions are inadequate. This inadequacy is the market opportunity for AI safety companies.
The Scale of the Problem: Why This Market Is Large
To understand why we estimate the AI safety market at $50B+, it helps to understand how large enterprises are currently failing to address these challenges and what the consequences of those failures cost them. The numbers are striking.
AI hallucination costs in enterprise deployments are substantial. A 2025 survey by a major consulting firm found that enterprises deploying AI in customer-facing applications are experiencing an average of 3-5 significant AI errors per week per deployed application. Each of these errors requires human remediation at an average cost of $500-$2,000 per incident, plus potential customer relationship costs. Across a typical enterprise with dozens of AI deployments, annual hallucination remediation costs are running $1-5M per year -- before accounting for the reputational costs of AI errors that reach customers without remediation.
AI security incidents are rising rapidly. Enterprise AI systems are now facing systematic prompt injection attacks, adversarial input attacks designed to cause misbehavior, and data exfiltration attacks that exploit AI system behaviors to extract proprietary data. These attacks are not hypothetical -- our portfolio company Guardrail Systems documented a 340% increase in prompt injection attempts against its customers' AI systems between Q1 and Q4 2025. The financial services industry, which is both a major AI deployer and a major target for adversarial attacks, is reporting AI security incidents at a rate that is drawing serious regulatory attention.
Regulatory compliance costs for AI are growing rapidly and will accelerate dramatically as AI regulations take effect. The EU AI Act, which came into full force in 2025, requires enterprises deploying AI in high-risk categories to conduct formal conformity assessments, maintain technical documentation, and implement human oversight mechanisms. Financial regulators in the US and Europe have issued binding guidance on AI governance for credit decisions and customer communications. Healthcare regulators require clinical validation studies and ongoing performance monitoring for AI-assisted diagnosis. The cost of building compliance programs for AI from scratch, without purpose-built tools, is running $2-5M per major AI deployment at large enterprises.
Taken together, these costs create a clear willingness-to-pay basis for AI safety tools. An enterprise spending $5M per year on AI hallucination remediation, security incident response, and compliance program management will pay $500K-$1M per year for AI safety tools that cut those costs significantly. This is the commercial foundation of the AI safety market -- it is not philanthropy, it is pragmatic cost reduction and risk management.
The Investment Landscape: Why This Category Is Underfunded
Despite the clear commercial opportunity, AI safety companies are dramatically underfunded relative to their market potential. We estimate that AI safety startups received approximately $1.2B in venture funding in 2024 -- less than 2% of total AI startup funding. This is striking given that the market opportunity we are describing is larger than many AI categories that are receiving ten times more funding.
Several factors explain this underfunding. The term "AI safety" carries academic connotations that make commercial investors less comfortable than terms like "AI productivity" or "AI automation." The enterprise sales cycles for security and compliance products are long, which means AI safety companies have less impressive ARR growth curves at seed stage than consumer-facing AI applications. And there is a genuine lack of investors with both deep AI research backgrounds and commercial investment experience -- the combination required to evaluate AI safety startups accurately -- which has meant the category has been evaluated primarily by generalist investors who underestimate both the market size and the technical differentiation of the leading companies.
At Milestone AI Ventures, our team's technical background positions us well to evaluate AI safety companies. Before founding Milestone, I led the Natural Language Understanding team at Google DeepMind, where I worked on core research problems in language model safety and robustness. This background gives me the technical credibility to evaluate AI safety startups' core technology claims with appropriate rigor, and to distinguish genuine technical innovation from marketing positioning. It is one of the key reasons we believe we have an advantage in this category.
The Companies We Are Backing
Guardrail Systems, our most mature AI safety investment, addresses the enterprise governance and output safety layer. Their platform provides real-time monitoring of production LLM applications for hallucination detection, bias monitoring, prompt injection detection, and policy compliance checking. They have found particularly strong product-market fit in financial services and healthcare -- the two industries where AI errors carry the highest regulatory and reputational consequences -- and are growing rapidly as AI deployment in these industries accelerates.
What makes Guardrail Systems defensible is not just the quality of their detection algorithms, though those are strong. It is the proprietary dataset of production AI incidents that their platform has accumulated across all customer deployments. Every hallucination, every prompt injection attempt, every policy violation that their system detects becomes a training signal that improves their detection models. This creates a classic data flywheel: more customers generate more training data, which improves detection accuracy, which attracts more customers. Competitors launching today cannot replicate this dataset without years of production deployment at scale.
We are also actively evaluating several earlier-stage companies in adjacent parts of the AI safety landscape. AI red-teaming platforms -- tools that systematically probe AI systems for vulnerabilities and failure modes before deployment -- represent a particularly compelling opportunity as enterprises mature their AI security programs. AI alignment and policy management platforms -- tools that help organizations define, implement, and monitor organizational AI policies across all deployed systems -- are another category we are watching closely as the regulatory environment matures.
The Next Three Years: Catalysts for Market Growth
Several specific catalysts will drive AI safety market growth over the next three years, and understanding these catalysts is important for timing investments in the category correctly.
The EU AI Act enforcement timeline is the most significant near-term catalyst. Enterprises deploying AI in high-risk categories -- which includes most financial services, healthcare, and HR applications -- must demonstrate compliance with EU AI Act requirements beginning in 2025, with full enforcement by 2026. Meeting these requirements without purpose-built AI governance tools is impractical at scale. The EU AI Act is essentially a mandate for enterprises to invest in AI safety infrastructure, and we are already seeing compliance-driven purchasing accelerate in our portfolio company's pipeline.
Major AI security incidents at large enterprises will drive a second wave of investment. The cybersecurity industry knows this pattern well: major security incidents at high-profile organizations create industry-wide urgency that accelerates security product adoption by years. The first major AI security incident at a systemically important financial institution, or a high-profile AI failure that causes material patient harm in a hospital network, will do for AI safety what the major cybersecurity breaches of the 2010s did for endpoint security and cloud security posture management. We do not hope for such incidents, but we think it is important to be honest that they will likely occur and that they will dramatically accelerate market development when they do.
The expansion of AI deployment into higher-stakes autonomous decision-making is the third catalyst. As AI systems move from generating text for human review to autonomously executing actions -- placing trades, approving loans, modifying system configurations, sending customer communications -- the consequences of AI errors increase dramatically. The need for real-time safety monitoring and intervention capabilities for autonomous AI systems will create a wave of demand for AI safety infrastructure that does not yet exist in mature commercial form.
The Investment Thesis in Summary
Our AI safety investment thesis can be stated simply: AI safety tools will be as mandatory for enterprise AI deployments as security tools are for enterprise cloud deployments. Just as no CFO would approve deploying an enterprise application in the cloud without security controls, observability, and access management in place, no enterprise will be able to justify deploying AI in customer-facing or high-stakes applications without equivalent AI safety infrastructure within three to five years.
The companies that establish category leadership in AI safety today will benefit from the same compounding advantages that drove the cybersecurity category's spectacular growth over the past twenty years: growing regulatory requirements that create demand, security incidents that drive urgency, and technical complexity that creates high barriers to entry and switching costs once established. The market will be large, the margins will be high, and the winners will be extremely durable businesses.
We are investing in this category with urgency because we believe the category leadership positions are being established now, in 2025, while the market is still early. The founders building AI safety companies today who establish deep enterprise customer relationships and accumulate proprietary incident datasets will be extraordinarily difficult to displace once the market reaches scale. This is the moment to invest in AI safety -- not because it is the fashionable thing to do, but because the underlying commercial opportunity is large, the technical defensibility of leading companies is high, and the competitive window is limited.
Dr. Sarah Chen is the Managing Partner and Co-Founder of Milestone AI Ventures. She previously served as Research Director at Google DeepMind, where she led work on language model safety. The views expressed here are her own and do not constitute investment advice.