For most of the last decade, the 'smart' in a smart assistant lived somewhere else. You spoke to your phone, the audio went to a data center, and an answer came back. That arrangement is changing. The newest phones and laptops sold in the US carry chips designed specifically to run AI models, and the big platform companies now ship language models that live on the device itself.
At the same time, the industry's ambition has shifted from assistants that answer to assistants that act — software that reads what is on your screen, understands what you are trying to do, and takes the next step for you. Put those two trends together and you get the idea behind 'on-device AI agents.' This explainer separates what is actually running locally today from what is marketing, and what the change means for privacy, reliability and everyday use. It relies on the companies' own technical documentation and on government and security-industry guidance; it is not a hands-on product review.
What 'on-device' actually means
A language model is a large file of numbers (its 'weights') plus software that uses them to predict text. Running it 'on-device' means those weights are stored on your phone or laptop and the computation happens on its own processor. Nothing needs to be sent to a server for that request to be answered.
The catch is size. The models behind the most capable cloud chatbots are far too large for a phone. On-device models are deliberately small. Apple's 2025 technical report describes its on-device foundation model as roughly 3 billion parameters, compressed to fit in a phone's memory. Google's Gemini Nano, which runs on supported Android phones through a system service called AICore, is in the same small-model category. Those sizes are enough for summarizing, rewriting, classifying and short replies; they are not enough for long research questions or complex reasoning.
That is why the honest description of most 2026 products is hybrid: a local model handles what it can, and the system decides — sometimes visibly, sometimes not — when to send a request to a larger model in the cloud.
Apple Machine Learning Research — Apple foundation models 2025 updates
The hardware that made it possible
The enabling change is the neural processing unit, or NPU — a part of the chip built to do the repetitive math of AI models efficiently, without draining the battery the way the main processor or graphics chip would. Phones have carried such units for years, but they were mostly used for photography and face unlock.
Microsoft made the NPU a buying criterion for Windows laptops in May 2024, when it defined 'Copilot+ PCs' as machines with an NPU capable of more than 40 trillion operations per second, along with minimum memory and storage. That threshold is a product-eligibility line Microsoft chose, not a universal measure of AI quality, but it signaled to buyers that local AI was becoming a standard feature rather than a novelty.
Memory matters as much as processing speed. A model has to fit in RAM alongside everything else the device is doing, which is one reason newer AI-capable phones and laptops ship with more memory — and one reason the current memory shortage, covered in our PC-prices explainer, reaches consumers.
From assistant to agent: what changes
An assistant responds. An agent pursues a goal: it breaks a request into steps, uses tools such as apps, files or a browser, and checks whether each step worked. 'Find the restaurant my sister texted me about and add a reservation reminder for Friday' is an agent-style task, because it requires reading a message, identifying the restaurant, opening a calendar and creating an entry.
On a device, the agent's advantage is context. Your phone already knows your messages, calendar, photos, location history and which apps you use. A local model can draw on that information without uploading it, which is both the main selling point and the main reason to think carefully about what you allow it to do.
In practice, the most common 'agent' features today are narrower than the demos: notification and email summaries, suggested replies, searching photos by description, transcribing and summarizing calls or recordings, cleaning up writing, and handing a request to a specific app. Fully independent multi-step action across third-party apps exists in limited, often opt-in forms and remains the least mature part of the picture.
Why local processing helps privacy — and where it stops
The privacy argument is simple: data that never leaves the device cannot be intercepted in transit, stored on a company server, or reviewed by a contractor. For tasks such as summarizing your own messages, that is a real improvement over sending everything to the cloud.
The limits are equally real. First, hybrid systems send some requests out. Apple's answer is Private Cloud Compute, which it says runs larger models on its own servers without retaining user data and with server software published for outside researchers to inspect. That is a stronger design than a typical cloud service, but it is still a cloud service, and users depend on Apple's implementation matching its description. Other platforms make different promises, and third-party apps can send data wherever their own privacy policies allow.
Second, local does not mean invisible. A feature that indexes everything you see on screen creates a valuable target on the device itself. Microsoft's Recall feature was delayed in 2024 after security researchers criticized how its snapshots were stored; Microsoft's revised design made it opt-in, encrypted the data and tied access to Windows Hello authentication. The episode is a useful reminder that a local database of your activity is only as safe as the protections around it.
The new risk: an assistant that can be talked into things
Once an assistant can read content and take actions, it can be manipulated through the content it reads. Security researchers call this prompt injection: hidden or disguised instructions in an email, web page or document that the model treats as a command — 'forward the last three messages to this address,' for example. OWASP lists prompt injection as the top risk for applications built on large language models, and NIST's generative-AI risk profile treats it as a distinct security concern.
Running the model locally does not fix this. The weakness is in how the model interprets text, not where it runs. The practical defenses are on the product side — asking for confirmation before sensitive actions, limiting which apps an agent can control, and keeping a visible record of what it did — and on the user side: be slow to grant an assistant permission to send messages, make payments or change account settings on its own.
Speed, offline use and battery
Local models remove the round trip to a server, so short tasks such as rewriting a sentence or summarizing a notification can feel instant and work without a signal — on a plane, in a rural dead zone, or in a building with poor coverage. Google's documentation for Gemini Nano highlights offline availability and keeping data on the device as core reasons developers would choose it.
The cost is energy and heat. Even with an efficient NPU, sustained AI work uses battery, and devices manage that by limiting how long or how often heavy tasks run. Users are unlikely to notice for occasional summaries; they may notice with continuous transcription or long sessions.
What the small models still get wrong
Small models make the same kinds of mistakes as large ones, and often more of them: confidently wrong facts, misread tone, summaries that drop the one detail that mattered. Notification summaries have produced well-publicized misfires, including Apple pausing AI summaries for news apps in early 2025 after they misstated headlines. A summary of your own messages that gets a time or a name wrong is a smaller problem than a garbled headline, but it is the same failure.
Small models also know less. They are trained to be good at language tasks, not to store a large amount of world knowledge, so questions about current events or specialized topics are usually routed to the cloud or answered poorly. And because features differ by device, chip and region, two people with 'the same' assistant may get very different capabilities.
Who can actually use these features
On-device AI is uneven by design. The features generally require recent, higher-end hardware: newer iPhones and Macs for Apple Intelligence, specific Android models for Gemini Nano, and Copilot+ laptops for Windows' local features. Older or budget devices may get the cloud version, a reduced version or nothing. Before buying a device for its AI features, check the manufacturer's current support list rather than the advertising, and consider whether the features you would actually use need local processing at all.
Practical settings for everyday users
Check what is on. Assistant and AI-feature settings on iPhone, Android and Windows show which features are enabled and, increasingly, whether a given request is handled locally or in the cloud. Turn off what you do not use.
Grant actions slowly. Reading and summarizing are low risk; sending, buying, deleting and changing settings are not. Keep confirmation prompts on for those, and review which apps an assistant can control.
Protect the device itself. An assistant with deep personal context raises the value of your unlocked phone. A strong passcode, current updates and theft protections matter more, not less — our smartphone security guide covers the settings in detail.
Verify what matters. Treat summaries as a pointer to the original message, not a replacement, especially for dates, amounts and anything from a bank, doctor or employer.
The strongest arguments on each side
The optimistic case: moving routine AI work onto the device is the most credible privacy improvement in consumer AI so far, it makes useful features work offline and quickly, and agents that handle small chores could save real time for people who find phones and paperwork tedious.
The skeptical case: 'on-device' is often a partial claim in a hybrid system, the agent features most heavily promoted are the least reliable, and giving software permission to act on your accounts creates a new attack surface that local processing does nothing to close. Both readings fit the evidence; the difference is mostly in how much autonomy a user chooses to grant.
What to watch next
Three questions will decide how independent these devices really become. Whether platforms show users clearly, per request, what ran locally and what went to the cloud. Whether agent permissions become as granular and visible as camera and location permissions are today. And whether independent researchers can verify privacy claims, as Apple has invited for Private Cloud Compute, rather than relying on company statements.
DailyTech — Open AI text generators and open-weight models explained
What this explainer can and cannot establish
This article describes how on-device AI works and what the platform companies say their systems do, based on their published technical and privacy documentation. We did not independently test every device, measure accuracy rates, or audit any company's privacy claims. Model sizes and features change with software updates, and availability varies by device and country, so check current documentation before relying on a specific capability.