Generative AI has landed in the middle of cybersecurity, and it is working for both sides.
If the term is new to you, generative AI is the kind of artificial intelligence that creates new content: text, code, images and even audio. Chatbots such as ChatGPT are the famous examples. Security work is mostly reading, writing and sorting huge piles of information, so it is no surprise that these tools fit the job so well.
The problem is that they also fit the attacker’s job. And when your own team uses them carelessly, they open a third set of risks inside your own company.
This guide walks through all three sides in plain English: how defenders use generative AI, how criminals use the same tools, and what can go wrong when you use the tools yourself.
A One-Minute Definition
When security people say generative AI, they usually mean a large language model. That is a system trained on enormous amounts of text that can answer questions, summarize long documents, translate, and write or explain code.
Older security software mostly followed fixed rules written by humans: if this exact pattern appears, raise an alarm. A generative model is different. It works more like a very fast junior analyst: tireless, widely read, quick to produce a draft, and sometimes wrong in ways that sound completely convincing.
It also helps to know what generative AI is not. It does not think, and it does not store facts the way a database does. It predicts words that sound right, based on patterns it learned during training. Security teams have used other kinds of AI for years, mainly machine learning systems that score events as normal or suspicious. What is new here is the conversation: you can ask questions in plain language and get summaries, drafts and explanations back. That is why the technology spread through security teams so quickly.
Keep that junior analyst picture in mind. Almost every good use, and every serious risk, in this article comes from it.
How Defenders Use Generative AI
Catching phishing and strange behavior
The oldest job in security is spotting the bad message in a sea of normal ones. Generative AI helps because it reads language the way a person does, not just a list of banned words. It can judge whether an email sounds like your finance director, whether a request feels out of character, and whether the story in the message matches how your company normally works.
That matters because phishing, the fake messages that trick people into clicking or paying, is still the main door attackers walk through. We cover one recent wave in our report on Gmail users warned about sophisticated AI driven phishing attacks.
Noticing behavior that does not fit
Not every threat arrives as a message. Defenders also watch behavior for anomalies, meaning events that do not fit the normal pattern: a sign in from a country the employee never visits, an account suddenly reading thousands of files at three in the morning, a password reset followed minutes later by new mailbox rules.
Generative AI helps analysts investigate these alerts because it can gather the surrounding facts and explain, in plain words, why a pattern looks wrong and what to check next. The analyst still makes the final call, but the background homework that used to take an hour now arrives in minutes.
Triaging the alert pile
A security operations center, the team that watches a company’s systems around the clock, can receive thousands of alerts a day. Most are harmless. A few are the start of a real attack, and finding them fast is the whole game.
Analysts call the sorting step triage, a word borrowed from hospital emergency rooms. Generative AI can group related alerts into one incident, pull in the background an analyst would normally hunt for by hand, and rank the pile so the most dangerous items get human eyes first.
Summarizing logs and writing reports
Security systems keep logs, which are long records of everything that happened: who signed in, which files were opened, which connections were made. After an incident, someone has to read those logs and write a clear report for managers, insurers or regulators.
This is slow, dull, essential work, and it is where language models shine. An analyst can ask for a timeline of an incident in plain words, then check and correct the draft instead of writing it from a blank page.
Helping analysts write rules and review code
Defenders also use these tools as a coding partner. The AI can draft a detection rule, the pattern that tells a monitoring system what to watch for, and a human expert reviews it before it goes live. It can explain an unfamiliar script line by line, which helps when an analyst is staring at code an attacker left behind. And it can review your own company’s code for common security mistakes before the software ships.
In every case the pattern is the same: the AI drafts and sorts, the human decides.
Training people with practice attacks
Finally, defenders use generative AI to train their own staff. The tool can write realistic practice phishing emails, tailored to your industry, so employees learn to spot the real thing in a safe setting. Good training matters more than ever, because the real attacks have become much harder to spot, as the next section explains.
How Attackers Use the Same Tools
Phishing that reads perfectly
The UK National Cyber Security Centre, the NCSC, assessed in 2024 that AI will almost certainly increase the volume and impact of cyber attacks, and that social engineering is where attackers gain the most. Social engineering means tricking people rather than breaking machines. As the NCSC put it, generative AI can already produce convincing messages and lure documents without the translation, spelling and grammar mistakes that once gave phishing away.
In plain terms: the old safety advice, look for bad spelling, is dying. A criminal who speaks no English can now write a flawless message in your language, in your boss’s style, in seconds.
Cloned voices and fake documents
The FBI warned in December 2024 that criminals exploit generative AI to commit fraud on a larger scale, because the technology reduces the time and effort needed to deceive people. Its alert described criminals generating short audio clips containing a loved one’s voice, used to impersonate a close relative in a fake crisis and pressure the family into sending money quickly. The same alert covered fake profiles, fake identification documents and polished written scams.
Notice what changed. The scam itself is decades old. The voice on the phone used to require a talented impersonator. Now it requires a short recording and a cheap tool.
A faster helper, not a new mastermind
Attackers also use chatbots the way defenders do: to research targets, translate, fix errors in their code and work faster. Google’s Threat Intelligence Group reported in January 2025 that more than 57 government backed hacking groups, linked to China, Iran, North Korea and Russia, had experimented with its Gemini chatbot. The important detail is the balance: Google found these groups gained productivity, not brand new capabilities. The NCSC reached a similar view, judging that AI help with malware and exploit development would mostly benefit attackers who are already skilled.
So the honest picture is not robot hackers taking over. It is ordinary criminals and spy agencies doing familiar work faster, cheaper and at a much larger scale.
Scale is the quiet change. The NCSC assessment warned that AI services reduce the barriers to entry, increasing the number of cyber criminals and the scale, speed and effectiveness of existing attack methods. One person can now research a target, write the lure in flawless local language, translate the replies and keep dozens of conversations going at once. None of those steps is new. Doing all of them, all day, at almost no cost, is new.
The Third Problem: Securing Your Own Use of AI
Here is the part many teams miss. The moment your staff start using generative AI, the tool itself becomes a security question. Official guidance from the NCSC, CISA and the NSA returns to three concerns again and again.
1. Your own data can leak
Anything an employee pastes into a public chatbot leaves your company’s control. That can include passwords, customer records, contracts, or private source code pasted in to ask for a quick fix. The joint guidelines for secure AI system development, published by the NCSC with CISA and partners, are built around the idea that AI systems should function as intended without revealing sensitive data to unauthorised parties. That duty starts with simple rules for staff, and we go deeper in our guide on how to prevent an LLM from accessing confidential data in development.
2. Prompt injection
Prompt injection means hidden instructions are planted inside content the AI reads: an email, a web page, a document. When the AI processes that content, it may follow the hidden instruction instead of yours, for example by forwarding data or approving an action it should not.
The NCSC has warned that this may never be fully fixed, because a language model does not truly separate data from instructions: to the model, it is all just text. The practical answer is to design around it. Limit what the AI can access, require human approval for serious actions, and watch what the system does. The risk grows sharply when AI agents can browse, read email and act on their own, which we examine in our article on browser agent security risk.
3. Confident answers that are wrong
A language model would rather sound sure than admit doubt. In security work that is dangerous: a wrong detection rule leaves a hole, a wrong summary hides the one log line that mattered, and a wrong fix can break a working system. This is why every official guide treats AI output as a draft for review, never as a finished answer.
There is one more concern behind all three. A deployed AI system is itself a valuable target. The NSA’s AI Security Center, together with CISA, the FBI and allied agencies, published Deploying AI Systems Securely in April 2024, advising organizations to protect the model, its data and its logs with the same care as any other critical system: strong access controls, validation before use, and constant monitoring.
Practical Rules for Using It Safely
You do not need to ban these tools. You need rules around them. The guidance from CISA, which keeps an overview of its work on its artificial intelligence page, and from the NCSC and NSA, points to the same practical set. None of it requires special software. It is mostly management: decide which tools are allowed, decide what data may never be pasted into them, and make sure a named human stays responsible for whatever the AI produces.
- Keep secrets out. No passwords, private keys, customer data or unreleased code in public AI tools. Ever.
- Use approved tools. Company accounts with logging and access controls beat personal accounts nobody can see.
- Limit the blast radius. Give the AI the least access it needs, and require a human to approve payments, deletions and other serious actions.
- Check the work. Rules, code and summaries from an AI are drafts. A person verifies them before they go live.
- Train for the new scams. Staff should expect flawless phishing and cloned voices. Simple habits help: verify unusual requests through a second channel, and agree a family safe word for emergency calls.
The Bottom Line
So, how can generative AI be used in cybersecurity? As a tireless assistant that reads, sorts, summarizes and drafts, on both sides of the fight. Defenders use it to catch phishing, triage alerts and train their people. Attackers use it to write better lures, clone voices and work faster. And every organization that adopts it takes on a new duty: guarding the data, the instructions and the output of the tool itself.
The teams that benefit most will not be the ones with the fanciest model. They will be the ones with clear rules, careful humans, and a healthy habit of checking the machine’s work.
For more guides like this one, browse the cybersecurity and networking category.
