Artificial intelligence as a threat and a tool

2) AI as a threat

2.1 Deepfakes & social engineering

Synthetic media lowers the cost of convincing fraud. In Feb 2024, a finance worker in Hong Kong was duped during a video call where every participant (including a fake CFO) was an AI‑generated deepfake—HK$200 million (~US$25.6 million) was wired before the ruse was discovered. National authorities and independent testers warn that deepfakes, voice cloning and generative text increase the believability and volume of phishing and business‑email compromise (BEC), challenging traditional awareness training and filters.

2.2 Augmented intrusion campaigns

Multiple governments and vendors have reported early but real use of generative AI by state‑aligned actors for reconnaissance, phishing content, and research support—not yet “new classes” of attacks, but observable scaling and localization of existing ones. Expect AI to steadily raise baseline capability for criminal groups as tooling matures.

2.3 Risks in LLM‑powered applications (OWASP)

The OWASP GenAI project maintains a living Top 10 for LLM applications. Key risks include: prompt injection and data exfiltration; insecure output handling (e.g., blindly executing generated code); poisoning of training/fine‑tuning/embedding data; model DoS and excessive agency in agentic systems; insecure plugin/tool design; over‑reliance and model theft.

2.4 Adversarial machine learning (attacks on models)

Beyond application‑level issues, ML models are vulnerable to evasion (adversarial examples at inference), poisoning (corrupting training or embedding data), privacy attacks (inversion, membership inference), and extraction (stealing parameters/behavior). NIST’s taxonomy and MITRE ATLAS catalog these tactics and mitigations; ENISA synthesizes European perspectives.

2.5 Model & data supply chain

Model hubs and pipelines introduce new supply‑chain risk. Researchers have identified malicious models that executed code via unsafe serialization or embedded payloads; platform maintainers have since expanded scanning and safer formats, but enterprises must “trust but verify” by scanning, signing, and isolating models in runtime containers.

2.6 RAG and agentic patterns

Retrieval‑augmented generation (RAG) improves factuality but increases the attack surface via indirect prompt injection (malicious instructions in retrieved content), index poisoning, and leakage of sensitive data through vector stores. Recent demonstrations show email/Gmail summarizers and enterprise assistants can be manipulated via HTML/CSS‑hidden prompts unless robust filters and isolation are used. OWASP now documents vector/embedding weaknesses explicitly.