ZeroDay.Tools - Gen AI Hardening & Attack Suite
Note: For per-integration logging & monitoring, see LatentSpace.Tools.
This repository provides an up-to-date AI/ML Hardening Framework and a Multimodal Attack Suite for Generative AI. It is built around the security notions of a Kill Chain x Defense Plan, primarily focusing on Gen AI, with illustrative examples from Discriminative ML and Deep Reinforcement Learning. This work is predicated on:
- The universal and transferable nature of attacks against Auto-Regressive models.
- The conserved efficiency of text-based attack modalities (see: Figure 3) even for multimodal models.
- The non-trivial nature of hardening GenAI systems.
Our approach to AI security is systematically structured around understanding, identifying, and mitigating threats across a defined AI Kill Chain. This framework enables a robust defense plan for Generative AI, Discriminative ML, and Deep Reinforcement Learning systems.
This GIF demonstrates an attack utilizing per-model templates to generate adversarial strings. It employs Greedy Coordinate Gradient optimization of target input/outputs, achieving results in minutes on consumer hardware when starting from a template.
The following checklist summarizes key exposures and core dependencies for each step in the AI kill chain. For detailed takeaways, mitigation strategies, and in-line citations, please refer to the links provided, which lead to the "Detailed Vulnerability Remediation & Mitigation Strategies" section.
Download the Observability Powerpoint for additional context on monitoring and defense.
🚨 Gen AI Vulnerabilities x Exposures (Click to Expand)
Key Exposure: Brand Reputation Damage & Performance Degradation Dependency: Requires specific API fields; no pre-processing
Key Exposure: Documentation & Distribution of System Vulnerabilities; Non-Compliance with AI Governance Standards Dependency: Requires API Access over time; ‘time-based blind SQL injection’ for Multimodal Models
Key Exposure: Documentation & Distribution of Model-Specific Vulnerabilities Dependency: API Access for context window retrieval; VectorDB Access for decoding embeddings
Key Exposure: Data Loss via Exploitation of Distributed Systems Dependency: Whitebox Attacks require a localized target of either Language Models or Mutlimodal Models; multiple frameworks (e.g. SGA, VLAttack, etc) also designed to enable Transferable Multimodal Blackbox Attacks and evade 'Guard Models'
Key Exposure: Legal Liability from Data Licensure Breaches; Non-Compliance with AI Governance Standards Dependency: Requires API Access over time; ‘rules’ defeated via prior system and model context extraction paired with optimized attacks
Key Exposure: IP Loss, Brand Reputational Damage & Performance Degradation; Non-Compliance with AI Governance Standards, especially for “high-risk systems” Dependency: System Access to GPU; net-new threat vector with myriad vulnerable platforms
Key Exposure: Brand Reputation Damage & Performance Degradation; Non-Compliance with AI Governance Standards, especially for “high-risk systems” Dependency: Target use of compromised data & models; integration of those vulnerabilities with CI/CD systems
Key Exposure: Documentation & Distribution of System Vulnerabilities; Brand Reputation Damage & Performance Degradation Dependency: Lack of Active Assessment of Sensitive or External Systems
This section provides in-depth information on the dependencies, key exposures, and mitigation takeaways for each vulnerability outlined in the checklist.
- Dependency: Requires specific API fields; no pre-processing.
- Key Exposure: Brand Reputation Damage & Performance Degradation.
- Takeaway: Mitigate low-complexity priming attacks via evaluation of input/output embeddings against moving windows of time, as well as limits on what data is available via API (e.g., Next-Token Probabilities aka Logits). This also mitigates DDoS attacks and indicates instances of poor generalization.
- Dependency: Requires API Access over time; ‘time-based blind SQL injection’ for Multimodal Models.
- Key Exposure: Documentation & Distribution of System Vulnerabilities; Non-Compliance with AI Governance Standards.
- Takeaway: Mitigate retrieval of information about the system and application controls from Time-Based Blind Injection Attacks via Application-Specific Firewalls and Error Handling Best-Practices. Augment detection for sensitive systems by evaluating conformity of inputs/outputs against pre-embedded attack strings, and flagging long-running sessions for review.
- Dependency: API Access for context window; Access to Embeddings for Decoding (e.g., VectorDB).
- Key Exposure: Documentation & Distribution of Model Vulnerabilities & Data Access.
- Takeaway: Reduce the risk from discoverable rules, extractable context (e.g., persistent attached document-based systems context), etc., via pre-defined rules. Prevent decodable embeddings (e.g., additional underlying data via VectorDB & Backups) by adding appropriate levels of noise or using customized embedding models for sensitive data.
- Dependency: Whitebox Attacks require a localized target; multiple frameworks (e.g., SGA, VLAttack, etc.) support Transferable Multimodal Blackbox Attacks and evade 'Guard Models'.
- Key Exposure: Data Loss via Exploitation of Distributed Systems.
- Takeaway: Defeat pre-processed optimization attacks by pre-defining embeddings for 'good' and 'bad' examples, logging, clustering, and flagging of non-conforming entries pre-output generation, as well as utilizing windowed evaluation of input/output embeddings against application-specific baselines.
- Dependency: Requires API Access over time; ‘rules’ defeated via prior system and model context extraction paired with optimized attacks.
- Key Exposure: Legal Liability from Data Licensure Breaches; Non-Compliance with AI Governance Standards.
- Takeaway: Prevent disclosure of underlying data while mitigating membership or attribute inference attacks with pre-defined context rules (e.g., “no repetition”), whitelisting & monitoring of allowed topics, as well as DLP paired with active statistical monitoring via pre/post-processing of inputs/outputs.
- Dependency: System Access to GPU; net-new threat vector with myriad vulnerable platforms.
- Key Exposure: IP Loss, Brand Reputational Damage & Performance Degradation; Non-Compliance with AI Governance Standards, especially for “high-risk systems”.
- Takeaway: Multiple Open-Source Attack frameworks are exploiting a previously underutilized data exfiltration vector in the form of GPU VRAM, which has traditionally been a shared resource without active monitoring. Secure virtualization and segmentation tooling exists for GPUs, but mitigating this vulnerability is an active area of research.
- Dependency: Target use of compromised data & models; integration of those vulnerabilities with CI/CD systems.
- Key Exposure: Brand Reputation Damage & Performance Degradation; Non-Compliance with AI Governance Standards, especially for “high-risk systems”.
- Takeaway: Mitigate Supply Chain & Data Poisoning attacks via use of Open-Source Foundation Models and Open-Source Data wherein Data Provenance/Lineage can be established, versions can be hashed, etc. Thereafter, affect access and version control of fine-tuning data, contextual data (i.e., augmented generation), etc.
- Dependency: Lack of Active Assessment of Sensitive or External Systems.
- Key Exposure: Documentation & Distribution of System Vulnerabilities; Brand Reputation Damage & Performance Degradation.
- Takeaway: Utilize a Defense in Depth approach (e.g., Purple Teaming), especially for Auto Regressive Models, while staying up to date on the latest attack & defense paradigms. Utilize open-source code-generation and vulnerability assessment frameworks, contribute to the community, etc.
This framework and the accompanying attack suite can be utilized for:
-
Manipulation of AI Systems: Targeting Self-Supervised Systems, AI Assistants, Agentic Frameworks, and connected tools/plugins. This is achieved via direct or indirect injection of adversarial strings optimized to make Models designed to call external functions or access tooling frameworks return specific arguments.
- Example Impact: Unauthorized IAM Actions, Internal Database Access, aiding in privilege escalation.
-
Inference Attack Definition: Defining Membership & Attribute Inference Attacks for open-source, semi-closed, and closed-source models. This involves targeting behavior that elicits high-precision recall of underlying training data.
- Example Application: Validation of GDPR-compliant data deletion (alongside layer validation), Red/Blue Teaming of LLM Architectures & Monitoring.
While the primary focus is Generative AI, these security principles and vulnerabilities also extend to other AI paradigms.
🔍 Examples of Traditional ML and Deep/Reinforcement Learning Vulnerabilities (Click to Expand)
- Key Exposure: System-Specific Vulnerability & Performance Degradation.
- Dependency: Lack of Actively Monitored & Versioned RL Policies.
- Takeaway: Mitigate the compounding nature of poorly aligned & incentivized reward functions and resultant RL policies by actively logging, monitoring & alerting such that divergent policies are identified. While adversarial training increases robustness, these systems remain susceptible to attack.
- Dependency: Requires Out-Of-Date Vulnerability Definitions and/or lack of image scanning when deploying previous builds.
- Key Exposure: Brand Reputation Damage & Performance Degradation.
- Takeaway: Mitigate commonly exploited repos and analytics packages by establishing best-practices with respect to vulnerability management, repackaging, and image scanning.
- Explore the Attack Suite: Launch the Colab Notebook to see various attacks in action.
- Review the Hardening Checklist: Familiarize yourself with the Gen AI Vulnerabilities x Exposures to understand potential risks.
- Dive Deeper into Remediation: Use the Detailed Vulnerability Remediation & Mitigation Strategies section for specific guidance.
- Understand Observability: Download the Observability Powerpoint for broader context on AI system monitoring.
- Integrate with Monitoring Tools: For advanced per-integration logging & monitoring solutions, refer to LatentSpace.Tools.
Understanding and addressing the vulnerabilities outlined in this repository provides significant advantages:
- 🛡️ Enhanced Security Posture: Proactively identify and mitigate a wide range of AI-specific threats.
- 📉 Reduced Risk Exposure: Minimize potential brand reputation damage, data loss, intellectual property theft, and performance degradation.
- ⚖️ Improved Compliance & Governance: Better align with AI governance standards and legal requirements (e.g., GDPR, regulations for “high-risk AI systems”).
- 💡 Informed Defense Strategies: Develop more robust and effective defense mechanisms based on a clear understanding of evolving attack vectors.
- 🤝 Community Engagement & Knowledge: Stay updated with the latest attack and defense paradigms and contribute to a safer AI ecosystem.