A payroll employee signs in from an unfamiliar device. Minutes later, a new mailbox forwarding rule appears. A cloud account then downloads documents it rarely touches. Each event could have an innocent explanation. Together, they may describe an account takeover. The hard part is finding the connection early enough to act, without treating every employee who travels or changes laptops as an attacker.
That is the useful starting point for AI cybersecurity: not a machine that somehow knows a network is safe, but software that helps people identify, connect and evaluate signals. Security teams already use rules, signatures, statistical analysis and machine learning. Generative AI adds another layer, allowing an analyst to ask questions, draft queries and turn scattered records into a readable investigation.
The distinction matters to anyone buying a security product. A polished chatbot demonstration can establish that an interface is convenient. It cannot establish that the product catches a stolen session token, misses fewer attacks or prevents a breach. Those are separate claims requiring separate evidence. This article is an explanation and evaluation guide, not an original product benchmark or a report of an incident we observed.
Two meanings of AI security that should not be confused
The first is AI for security: applying models to email inspection, endpoint telemetry, account behavior, vulnerability prioritization or incident investigation. The second is security for AI: protecting a model, its training data, its retrieval sources and its connected tools from manipulation or unauthorized access. An organization deploying an AI security assistant needs both.
There is a third distinction inside the first category. A predictive model scores or classifies an event. A generative model produces an explanation, query or proposed action. They may appear in the same product, but their failure modes differ. A classifier can mislabel legitimate behavior. A language model can invent a causal connection or a command that looks plausible but is wrong.
NIST's March 2025 adversarial machine-learning taxonomy distinguishes predictive and generative systems and organizes attacks by their objectives, capabilities and place in the AI lifecycle. The practical lesson is to evaluate a particular use case rather than approve an entire product because it carries an AI label. A phishing classifier and an assistant allowed to disable accounts do not present the same risk.
How threat detection works, from raw events to an investigation
The pipeline begins with telemetry: identity-provider sign-ins, endpoint process activity, cloud audit records, network connections, email events and application access. A security information and event management platform, or SIEM, can collect and search those records. Endpoint detection and response, or EDR, provides visibility into device activity. Extended detection and response, or XDR, attempts to connect signals across multiple security domains. None of these labels guarantees complete coverage.
Records need usable timestamps, consistent account and device identifiers, and enough detail to reconstruct what happened. Collection delays, missing endpoints and changes in log format can turn a good detection method into a blind spot. A model cannot reliably establish that a user never accessed a document if the application did not record document access in the first place.
Different methods then do different jobs. A signature can identify a known malicious artifact. A rule can flag a risky combination of events. A supervised model learns distinctions from labeled examples. An anomaly detector identifies departures from a baseline. A correlation system groups related activity into an incident. A generative assistant may summarize that incident and help an analyst retrieve supporting records.
CISA and its international partners' August 2024 event-logging guidance emphasizes the visibility needed to detect malicious activity, including attackers using legitimate system tools. It is not an AI endorsement. It explains why the underlying logging program remains necessary regardless of which analytics layer sits above it.
Where AI can earn its place
Email is one practical use case. A model can combine language, sender history, links and attachment characteristics to prioritize suspicious messages. That is more informative than treating spelling errors as a reliable phishing signal. It is still imperfect: a legitimate vendor may send an unusual invoice, and a compromised vendor account may send a convincing malicious one.
Identity analysis is another. An account's usual devices, applications and access patterns can provide context around a suspicious sign-in. But a location anomaly by itself is weak evidence: travel, corporate gateways and virtual private networks complicate the picture. The stronger question is whether several independent signals support unauthorized access.
On endpoints, behavior analysis can help identify suspicious sequences even when a file hash is unfamiliar. In cloud environments, it can prioritize unusual permission changes, new credentials or access to sensitive resources. These capabilities remain dependent on the events actually collected and the baseline used to judge them.
Generative assistance is especially useful as a starting point for bounded work: translating an analyst's question into a search, explaining a script, drafting an incident timeline or preparing a handoff. Every factual item in the output should point back to an event, document or validated tool result. Saving reading time is valuable; replacing the evidence with a fluent narrative is not.
A worked example: investigating a suspicious mailbox rule
Consider a hypothetical US professional-services firm. Its monitoring system flags a new rule forwarding a finance employee's email to an external address. The employee also has a recent sign-in from an unmanaged device. This is an illustrative scenario, not a reported breach or evidence that any particular product detected one.
An assistant could assemble the sign-in history, the rule-creation event, the destination address and nearby account changes. The analyst should check the original records, confirm the account and time window, and contact the employee through a trusted channel. The investigation should also look for legitimate explanations, such as an approved workflow, instead of only searching for evidence of compromise.
If the account takeover is confirmed, a documented response may involve removing the rule, revoking sessions, securing credentials and determining which messages or files were accessed. Resetting a password alone may not address every active session or delegated permission. The appropriate sequence depends on the identity provider and incident scope.
The model's contribution is organizing evidence and reducing repetitive work. The authorization to act comes from the organization's response procedure, not from how confident the model sounds. A decision affecting a production account should have an identifiable approver and an audit trail.
Why high accuracy can still produce an unusable alert queue
Attack activity is often rare relative to ordinary activity. That makes headline accuracy a poor buying criterion. A model that labels almost everything benign can look accurate on a mostly benign dataset while missing the cases that matter.
Here is deliberately simplified arithmetic, not a measured benchmark. Suppose a test contains 10,000 events: 100 malicious and 9,900 benign. A detector catches 90 malicious events and wrongly flags 1 percent of benign events. It produces 90 true positives and 99 false positives. Precision is 90 divided by 189, or about 47.6 percent; recall is 90 divided by 100, or 90 percent. Its overall accuracy is 9,891 divided by 10,000, or 98.91 percent, yet more than half its alerts are false alarms.
The calculation is illustrative because real systems often group many events into one incident. Changing the grouping changes the denominator. Ask whether a vendor is reporting results per event, alert, account or incident, and require the number of true positives, false positives, true negatives and false negatives behind its percentages.
A lower alert count is not automatically an improvement either. It might mean better deduplication, or it might mean relevant alerts were suppressed. Review both the analyst workload and the attacks the system failed to surface.
What the published evidence does—and does not—show
Microsoft has published controlled studies of Security Copilot assistance and an analysis of operational productivity. These are useful examples of evaluating assistance on defined tasks rather than relying only on customer anecdotes. They are also vendor-authored evidence about a particular product and workflow, not proof that AI security systems as a category reduce breaches.
Controlled tasks can show improvements in task completion time or answer quality under the study's conditions. They cannot reproduce every dependency, interruption, missing log or disputed incident label in a real security operations center. Observational operational data adds context, but adoption can correlate with better staffing, training or other changes; it does not by itself isolate the model's causal effect.
Likewise, NIST's AI Risk Management Framework is a voluntary risk-management framework, not a product certification. MITRE ATT&CK organizes observed adversary behaviors; MITRE ATLAS focuses on threats to AI-enabled systems. Mapping a product to these resources helps explain its coverage, but a long list of mapped techniques is not a measured detection rate.
The defensible conclusion is narrower than the sales pitch: AI assistance can be evaluated and may improve specific workflows, while local detection effectiveness still needs testing against the organization's own threats, telemetry and response capacity.
The model is also part of the attack surface
An attacker can influence some of the material a security assistant reads: a phishing email, a ticket, a file name, a web page or an incident description. An instruction embedded in that material may try to redirect the assistant, conceal evidence or induce an unauthorized action. This is indirect prompt injection. Retrieved text is evidence to inspect, not an authority allowed to rewrite the assistant's instructions.
Other risks occur at different stages. Poisoning can contaminate training data or other data sources; evasion aims to make malicious input pass as benign; privacy attacks seek to expose protected information. Not every model is vulnerable in the same way, and keeping training protected does not automatically protect a tool-using assistant at inference time.
A sound design restricts tool access outside the model, checks action parameters against policy, and logs what was retrieved and executed. The assistant should not be able to grant itself a broader permission because an email told it to. Keep credentials and privileged execution separate from untrusted content, and test malicious instructions in the same kinds of records the product will actually encounter.
NIST's adversarial machine-learning report and MITRE ATLAS provide a vocabulary for these threats. They do not offer a universal fix. The safeguards need to match the assistant's data access, tools and possible consequences.
Hallucinations, missing context and changing baselines
A language model may invent a command's meaning, attribute activity to the wrong account or collapse several unrelated events into a persuasive timeline. Requiring citations helps only when the cited record actually supports the claim. Analysts must be able to open it and inspect the relevant field, timestamp and scope.
A behavioral model has a different problem: normal changes. A merger, a new cloud application or a shift in working hours can alter legitimate behavior. A quiet baseline can also be misleading if an attacker was already present during the period used to establish normality. Track changes in alert quality and data coverage rather than assuming a model stays calibrated indefinitely.
Feedback can create another blind spot. If dismissed alerts become training labels automatically, a tired analyst's mistake may teach the system to ignore the same pattern later. Preserve uncertainty, audit a sample of dismissals and separate confirmed benign outcomes from cases nobody had time to investigate.
The assistant should distinguish what the records show, what it infers and what remains unknown. Missing evidence is a reason to narrow a conclusion, not a blank space for the model to fill.
Privacy and governance for US organizations
Security logs can contain employee names, email subjects, device identifiers, IP addresses and details of customer or patient activity. Connecting them to a model changes who can process that information and potentially where it is stored. Before a pilot, establish which data leaves the organization, retention periods, subprocessors, access controls and whether prompts or outputs may be used for model training.
For healthcare, financial services and other regulated settings, involve the relevant privacy, legal and compliance staff. Whether a particular obligation applies depends on the data, the organization and the processing arrangement. An AI feature does not automatically establish HIPAA compliance, satisfy a financial regulator or remove state privacy obligations.
Minimize data where it does not undermine the investigation, restrict analysts to authorized cases, and avoid pasting production logs into consumer chat services without approval. Redaction itself needs checking: a useful identifier may be needed to join events, while a full document body may not be needed at all.
CISA's Zero Trust Maturity Model describes identity, devices, networks, applications and data as distinct pillars, supported by visibility, automation and governance. Although designed for federal agencies, it offers a useful structural reminder: analytics does not replace access control. A security assistant should receive least-privilege access just like another application.
How to run a pilot that can fail honestly
Start with one question. Can the assistant shorten verified investigation of suspicious mailbox rules? Can a model improve detection of a defined cloud-permission abuse pattern? Do not combine summarization, detection, response and prevention into a single success claim.
Establish the current baseline, then evaluate on held-out historical cases and authorized exercises with known outcomes. Include ordinary work that resembles an attack: approved bulk exports, travel, administrator maintenance and new software rollouts. Test missing logs and delayed ingestion as well as clean data. Keep evaluation cases separate from the examples used to tune the system.
For detection, measure precision and recall at a stated threshold, false positives per relevant unit, and time from observable activity to alert. For assistance, measure time to a verified disposition, incorrect factual claims and the amount of correction required. For response, track failed actions, unintended effects and rollback. Report both typical and slow cases; a median alone can hide a serious tail.
Run initially in observation mode, without allowing the assistant to change production systems. Compare against the existing process on comparable cases, with reviewers who can inspect the evidence. Record the model version, configuration, data sources and evaluation period so a later update can be retested.
Define failure criteria before seeing the results. If the product exposes sensitive data, repeatedly invents evidence, misses priority scenarios or saves less time than verification consumes, the pilot should not pass. A result that says 'not ready' is more useful than a successful demonstration with no measurable objective.
Automate preparation before high-impact response
There is a sensible progression from retrieving logs, to drafting a case, to recommending an action, to executing a tightly scoped action. Each step adds consequences. Summarizing an incident incorrectly is bad; disabling the wrong identity or isolating a critical server can interrupt the business immediately.
Require approval for consequential changes until the specific workflow has demonstrated reliable behavior. Bound actions by account, asset, duration and permission. Preserve the reason, supporting records, tool inputs, result and approver. Provide a working rollback path and a way to stop the automation when data quality or alert volume changes unexpectedly.
In operational technology, healthcare delivery and other availability-sensitive environments, isolation can have safety implications. The security team needs the system owner's response procedure, not a generic instruction to quarantine anything suspicious. Speed is valuable only when the action is correct for the system being protected.
What smaller businesses should prioritize
A smaller US company may gain more from securing its identity platform, enabling useful audit logs and ensuring someone reviews alerts than from purchasing a separate AI layer. Phishing-resistant multifactor authentication where supported, timely patches, tested backups and limited administrative access remain foundational. Detection identifies trouble; these controls can prevent it or limit the damage.
If there is no internal security team, ask a managed provider who investigates alerts, how quickly it can respond, which services it actually monitors and what happens outside business hours. Confirm that important cloud logs are included in the service and license tier. 'AI-powered' says nothing about whether a person will handle a confirmed account takeover at night.
The buying question is practical: does this service improve the ability to recognize and contain the threats facing this business? An assistant that produces more readable reports may be useful, but that should not be mistaken for broader monitoring coverage.
The bottom line: better evidence, not just faster explanations
AI can help security teams work through more material and connect signals they might otherwise inspect separately. It can also add misleading certainty, new data exposure and an overly powerful route from an attacker's text to a production action. Both possibilities belong in the same evaluation.
The strongest deployment is not the one that advertises the most autonomous agent. It is the one that detects a defined threat with measurable coverage, shows its evidence, stays within its permissions and makes the response team more effective without hiding uncertainty. Watch verified outcomes, missed cases and unintended actions—not the fluency of the incident summary.
Sources and guidance reviewed for this article on October 13, 2026. Standards explain mechanisms and risks; vendor research is identified as vendor evidence; the mailbox scenario and alert arithmetic are explicitly illustrative. No proprietary tool was independently benchmarked for this guide.