Security has used machine learning for a long time. Spam filters, malware classifiers, and network anomaly detection have all been built on it for well over a decade. What changed recently is generative AI, which produces language, code, images, and audio on demand.
That change is real, and it is also heavily oversold. This page tries to separate the two.
What Actually Changed for Attackers
The honest answer is speed and scale, not new capabilities. Almost nothing an attacker does with AI was impossible before. It was just slower, more expensive, or required a skill the attacker did not have.
Phishing is the clearest example. Bad grammar and awkward phrasing used to be a reliable warning sign, partly because many operators were writing in a second or third language. That signal is gone. A model will write clean, native sounding email in any language, in any tone, at any volume.
Voice and video cloning moved from research to commodity in about two years. That has already produced real financial fraud, including a widely reported case in Hong Kong in 2024 where an employee at the engineering firm Arup transferred a large sum after a video call with what appeared to be colleagues and was not. More on this in AI powered phishing and deepfakes.
Code assistance helps attackers the same way it helps everyone else. It lowers the effort of writing a script, adapting a public proof of concept, or understanding an unfamiliar codebase. The major model providers publish threat reports describing exactly this kind of misuse and the accounts they have banned for it.
What Did Not Change for Attackers
The initial access methods did not change. Stolen credentials, unpatched internet facing systems, and people clicking things remain how most intrusions start, exactly as they did before.
There is also no credible public evidence of AI inventing genuinely novel attack classes on its own. What the research shows so far is assistance and acceleration in tasks that already existed.
That matters for how you spend money. If your defense against phishing was hoping people notice typos, you have a problem now. If your defense was phishing resistant multi factor authentication, that still works, because the technical control does not care how well the email was written.
What Changed for Defenders
Defenders got something attackers did not: a tool that is genuinely good at reading enormous volumes of text quickly. Security is a text problem more than people admit. Logs, alerts, tickets, configuration files, vendor documentation, and policies are all text.
The concrete wins so far are alert enrichment and summarization, explaining unfamiliar commands or code to junior analysts, translating between query languages, drafting reports from notes, and first pass code review. See AI in the SOC for the detail.
There is a structural advantage here too. Defenders can run these tools against their own data, inside their own environment, with their own context. An attacker guessing at your environment from outside has much less to work with.
The New Attack Surface
The bigger change may be that organizations are now deploying AI systems themselves, which creates something new to attack. Prompt injection, where instructions hidden in content the model reads redirect its behavior, has no clean fix and gets worse as models are given permission to take actions rather than just answer.
The Open Worldwide Application Security Project maintains a Top 10 for Large Language Model Applications and MITRE maintains ATLAS, which is an ATT&CK style knowledge base for attacks against AI systems. Both are free and worth reading.
Then there is the governance problem: employees using AI tools nobody approved, with company data. That is covered in shadow AI.
How to Read Vendor AI Claims
Almost every security product now advertises AI. Some of it is genuine, some is a rebrand of statistical detection that shipped in 2015, and some is a text box bolted onto a dashboard.
Four questions cut through most of it. What specific task does it do, and would a human doing that task be useful? What data does it need, and where does that data go? How do you check whether its output was right? And what happens when it is wrong, meaning does it fail loudly or quietly?
Quiet failure is the one to watch. A tool that closes alerts incorrectly produces no error message and no complaint. It just produces a clean queue, which looks like success.
Who Benefits More
The current balance favors defenders slightly, for a boring reason. Defenders have the data. These tools are most valuable when applied to large amounts of context you already own, and defenders own their logs, their code, their configuration, and their history.
What AI does clearly do is compress the low end of attacker skill. Things that used to require some ability now require less. That raises the floor on attack volume and quality, which mostly hurts organizations that were relying on attackers being sloppy.
The practical response has not changed much: know what you own, patch what matters, use phishing resistant authentication, log the right things, and practice responding. Those were the right answers before and they are the right answers now. See good AI versus bad AI for the defender playbook.
Learn This at HackRange
Week 11 of the HackRange Mentorship is a full week on generative AI in security operations, and Week 7 covers governing AI use inside a company. Both are hands on.