Attackers can manipulate RAG-based AI systems with ConfusedPilot attack

Attackers can introduce a malicious document in systems like Microsoft 365 Copilot to manipulate AI responses, potentially causing misinformation and compromising decision-making. The ConfusedPilot attack affects retrieval augmented generation (RAG)-based AI systems, including Microsoft 365 Copilot, Llama, Vicuna, and OpenAI. By adding malicious content to documents referenced by AI systems, attackers can manipulate responses, bypass current AI security measures, and impact organizations, including Fortune 500 companies implementing RAG-based AI systems

This attack, discovered by the Spark Research Lab at the University of Texas, circumvents security measures, persists even after content removal, and can lead to content suppression, misinformation generation, and false attribution. Mitigations include data access controls, integrity audits, and data segmentation to prevent such attacks in enterprise systems relying on RAG-based AI. ```
https://www.darkreading.com/cyberattacks-data-breaches/confusedpilot-attack-manipulate-rag-based-ai-systems