Hallucination

LLMs generate confident-sounding text that may be factually wrong. In security contexts this is dangerous. A model might:

  • Fabricate a CVE number that doesn’t exist
  • Invent tool flags or command syntax
  • Cite statistics with no real source
  • Describe a vulnerability incorrectly

Mitigation: Always verify AI-generated technical claims against primary sources — NVD, vendor advisories, official documentation.

Prompt Injection

Prompt injection is an attack where malicious content in data you feed to an LLM hijacks its behavior. For example, if you paste a suspicious email into an AI tool for analysis, that email could contain hidden instructions telling the model to ignore your request and do something else.

Mitigation: Be cautious feeding untrusted data directly into AI tools. Sanitize inputs where possible and be skeptical of unexpected model behavior.

Data Privacy

Anything you send to a cloud-based LLM may be logged, used for training, or exposed in a breach. Never send:

  • Personally identifiable information (PII)
  • Credentials or API keys
  • Proprietary client data
  • Internal network details or architecture diagrams

Mitigation: Use local models (Ollama, LM Studio) for sensitive data. Review the privacy policy of any AI service before use in a professional context.

Over-Reliance

AI tools can create false confidence. A model that produces a polished, well-structured incident report doesn’t mean the analysis is correct. Over-reliance risks include:

  • Missing attacker TTPs the model didn’t flag
  • Accepting incorrect remediation advice
  • Reducing critical thinking in analysts over time

Mitigation: Treat AI output as a first draft or a second opinion — not a final answer. Maintain and develop your own analytical skills independently.

Model Knowledge Cutoffs

LLMs have a training cutoff date. They won’t know about:

  • CVEs disclosed after their cutoff
  • New threat actor TTPs
  • Recent tool releases or updates

Mitigation: Use AI with web search capabilities for current threat intelligence, or supplement with real-time feeds.

Bias and Blind Spots

Models are trained on internet text which contains biases, outdated practices, and gaps. Security advice from an LLM may reflect older frameworks or miss emerging attack patterns.

Mitigation: Cross-reference AI recommendations with current industry sources — MITRE ATT&CK, CISA advisories, vendor threat reports.