A few weeks ago, we published our analysis, “Claude Mythos: Breakthrough or Brilliant Marketing?”, where we examined the growing discussion around Anthropic’s Claude Mythos and claims about its potential for vulnerability discovery and offensive security assistance.
While much of the narrative around these systems remains speculative, early real-world testing is starting to provide more grounded signals about how Mythos-style models perform in practical security environments.
Public evaluations, including work shared by Daniel Stenberg from the curl project, are beginning to shed light on where Mythos AI-assisted vulnerability discovery performs effectively and where it still faces clear limitations.
At the same time, it is worth noting that Mythos has not been broadly released for independent evaluation, so its full capability set is still not publicly verifiable. What is already clear, however, is that the AI security space continues to be shaped heavily by early claims and expectations that often outpace available evidence.
1. AI Security Models Are Extremely Good at Pattern Recognition
Early Mythos-style testing suggests that modern AI security models are extremely effective at recognizing common and well-documented vulnerability patterns. These include issues such as SQL injection, cross-site scripting (XSS), buffer overflows, insecure API usage, unsafe deserialization, taint flow problems, and a wide range of recurring parsing and validation mistakes.
This capability is largely driven by the scale of training data. These models are trained on vast corpora of code repositories, bug reports, and security discussions, allowing them to quickly identify patterns such as:
- previously observed vulnerability structures
- common insecure coding patterns
- recurring misconfigurations and logic mistakes
However, pattern recognition should not be confused with deep security understanding. The ability to flag vulnerable-looking code is only one component of vulnerability discovery, and does not necessarily imply reliable reasoning about exploitability or full system behavior.
External evaluations, including work referenced by the AI Security Institute UK, suggest that such systems can detect exploitable weaknesses in poorly secured environments. “Mythos Preview can exploit systems with weak security posture, and it is likely that more models with these capabilities will be developed.”
Importantly, these findings come from constrained testing environments and do not demonstrate autonomous capability to escape sandboxing or operate beyond the boundaries of the evaluation setup.
Some early evaluations also suggest that such systems may exhibit limited goal-directed behavior in constrained settings, where models can:
- pursue objectives across multiple steps
- adapt strategies based on intermediate feedback
- use tools or external signals to refine outputs
However, these behaviors should be understood as emergent properties within controlled experiments, not evidence of unrestricted offensive autonomy in real-world environments.
2. Mythos Is Not the Same Thing as Penetration Testing
Mythos-like systems appear to function primarily as code-centric security reasoning tools rather than full penetration testing systems that independently simulate real attackers in live environments. In practice, they often resemble highly advanced static application security testing (SAST) or security analysis assistants.
Static analysis focuses on what might be wrong based on code structure, logic, and known vulnerability patterns. Penetration testing, by contrast, evaluates what an attacker can actually achieve against a running system, including authentication flows, runtime behavior, network conditions, misconfigurations, privilege escalation paths, and chained weaknesses.
This distinction matters because many high-impact vulnerabilities emerge only during execution or from interactions between systems, workflows, permissions, and trust boundaries.
As a result, AI security systems may be highly effective at surfacing potential weaknesses while still facing limitations in reasoning about real-world exploitability and attack impact in dynamic environments.
3. The curl Case Study, A Reality Check for AI Vulnerability Discovery
One of the most closely watched public evaluations of Mythos-style vulnerability discovery involved the curl codebase, a mature open-source project maintained by Daniel Stenberg.
Curl is an unusually difficult target because it is already one of the most heavily audited, fuzzed, and security-reviewed open-source projects on the internet. That makes it a useful real-world test of whether AI can uncover issues that years of human review may have missed.
The assessment produced several findings, but the results were more modest than many expected:
- Multiple issues were reported for review.
- Most findings were ultimately classified as false positives or non-security bugs.
- Only one finding resulted in a confirmed security vulnerability.
- No major new class of vulnerability was discovered.
The confirmed issue demonstrates that AI can contribute to vulnerability discovery, even in mature software. However, the results also highlight a key limitation. Generating findings is not the same as finding exploitable vulnerabilities.
Most importantly, every reported issue still required human validation. Security engineers had to determine whether a finding represented a genuine vulnerability, a harmless bug, or simply an incorrect assumption about how the software behaved.
The curl case study suggests that AI is becoming a valuable force multiplier for security research, but it remains an assistant rather than a replacement for expert judgment.
Reference: Mythos finds a curl vulnerability.
4. Benchmark Performance Does Not Equal Real-World Vulnerability Discovery
Benchmark performance and real-world vulnerability discovery are not the same thing.
Many of the headline claims surrounding Mythos and similar AI systems come from controlled evaluations where models are tested against datasets containing known vulnerabilities. These benchmarks are valuable for measuring progress, but they do not necessarily reflect how a model will perform against production software that has been maintained, tested, and secured for years.
The difference becomes clear when examining mature projects such as curl.
- Benchmark environments contain vulnerabilities that are intentionally present and measurable.
- Real-world software may contain very few remaining flaws.
- Security controls, fuzzing, audits, and years of review can significantly reduce the number of discoverable issues.
- Vulnerabilities that do exist are often buried behind complex code paths and system interactions.
This is why strong benchmark scores should be interpreted carefully. A model may perform exceptionally well in controlled testing while producing far fewer meaningful findings when applied to highly scrutinized production code.
The curl evaluation illustrates this challenge. Despite impressive benchmark results and growing expectations around AI-assisted security research, the practical findings were limited and required substantial human validation.
Ultimately, the most important metric is not how many issues a model reports, but how many survive expert validation and lead to meaningful security improvements.
5. Why False Positives Become an Operational Problem
A recurring observation from early testing is that AI-generated vulnerability reports often appear highly detailed and technically convincing, even when the reported issue is not exploitable in practice.
These outputs may include:
- Structured exploit chains describing attack paths
- Step-by-step attack narratives that appear realistic
- Confident technical reasoning with plausible explanations
- Severity framing that makes findings seem immediately actionable
However, many findings fail validation due to factors such as:
- Incorrect assumptions about system behavior
- Missing runtime conditions required for exploitation
- Overlooked compensating controls elsewhere in the environment
- Incomplete architectural or business context
Importantly, this is not always simple hallucination. In many cases, the model is performing reasonable analysis on incomplete or partially inaccurate system representations. The problem is that technically plausible reasoning can still produce conclusions that do not hold up under real-world validation.
The result is a significant triage burden, where engineers must investigate large volumes of findings that initially appear critical but ultimately provide limited actionable value. This challenge is not entirely new, as traditional SAST tools have long struggled with false positives that reduce trust and slow remediation workflows.
6. Duplicate Findings Can Inflate Vulnerability Counts
Human researchers generally identify a single root cause and trace its impact through one exploit chain or a limited set of related scenarios. AI systems often behave differently.
A single implementation flaw may generate multiple findings because the model explores several theoretical attack paths, alternative privilege escalation routes, or slightly different protocol interpretations.
For example, one authentication flaw might produce separate findings for session bypass, token manipulation, privilege escalation, and API abuse, even though all stem from the same underlying weakness.
The result is inflated vulnerability counts.
This creates operational friction across triage workflows, remediation planning, severity scoring, and engineering prioritization. Security teams may initially believe they are dealing with dozens of independent issues when the reality is a much smaller number of root causes.
Again, the challenge becomes signal-to-noise ratio rather than raw detection capability. Finding more issues only matters if teams can realistically prioritize and fix them.
7. Business Logic Vulnerabilities Remain Extremely Difficult
Business logic vulnerabilities remain one of the most challenging classes of issues for AI systems to detect reliably.
Unlike technical vulnerabilities that follow recognizable patterns, business logic flaws are highly context-dependent. They emerge from how systems are intended to operate within specific workflows, organizational rules, and human decision-making processes.
These issues can include fraud via process manipulation, abuse of refund flows, exploitation of approval chains, or misuse of legitimate features in unintended ways. Because there is no consistent signature or universal pattern, automated systems struggle to identify them reliably.
Detecting such issues typically requires adversarial thinking, domain knowledge, and an understanding of business intent that goes beyond code-level reasoning.
8. Technically Exploitable Does Not Always Mean Meaningful
Another recurring issue is that AI-generated findings can assume unrealistic exploit conditions. Some reports describe scenarios that require prior compromise, such as existing root access, authentication bypass, or control of infrastructure layers. While technically valid in isolation, these conditions may already represent a “post-breach” state, limiting their practical relevance.
For example, a statement like “an attacker with root access can disable logging” is correct, but does not meaningfully describe a vulnerability in most threat models.
Security impact depends on attacker position, trust boundaries, system architecture, and realistic exploitation paths. Without that context, technically correct findings can still be misleading in severity assessment.
Final Thoughts
Early Mythos-style evaluations show a consistent pattern: strong performance in pattern recognition and scalability, but limitations in contextual reasoning, validation, and business logic understanding.
Cybersecurity remains fundamentally context-driven. It is not only about identifying suspicious code, but about understanding real-world behavior, attacker intent, and meaningful risk within a system’s operational environment.
AI can significantly accelerate vulnerability discovery and improve coverage, but it does not remove the need for human judgment. The most realistic outcome is augmentation, where AI improves the productivity of security researchers rather than replacing them.
