You don’t need an assistant that knows everything.
You need an assistant that can prove what it can access, when, and why—and that can’t act beyond your boundaries.
This is what AI assistant privacy looks like in practice: not a vague promise, but a set of controls you can verify.
Key Takeaway: The safest assistants behave like delegated staff, not a cross-app superuser: least privilege, time-bounded access, auditable actions, and approval gating for anything sensitive.
Why AI assistants end up accessing “too much”
Most privacy failures don’t start with malicious intent. They start with convenience.
An assistant is asked to “help with email,” “manage my calendar,” “summarize my files,” and “send a few messages.” That sounds reasonable—until you realize you’ve created a system that can:
read across inboxes, documents, chats, and contacts
correlate context across apps (what you meant to keep separate)
keep persistent access tokens even after the task ends
The risk isn’t only what the assistant sees today. It’s what it can continue to see tomorrow, and what that access enables if something goes wrong.
Security teams have a name for the anti-pattern: provisioning broad, standing access “because it’s easier.” Oso describes teams that “naively provision an independent service account with universal access,” warning that it violates least privilege and expands the blast radius when anything is misconfigured or abused, in its guide to delegated access and just-in-time credentials (2025): Setting Permissions for AI Agents.
Permission boundaries: how to manage AI assistant permissions without killing usefulness
If you remember one rule, make it this:
Permissions define maximum reach. Everything else is secondary.
VERTU’s own mobile security guidance puts it plainly: “App permissions are your first line of defense because they define the assistant’s maximum reach,” recommending “least privilege with escalation,” in its guide to mobile AI security and protecting data on AI-enabled phones.
1) Prefer delegated access over “service account” access
A privacy-respecting assistant should work on your behalf, under your identity, inside your permissions.
That means:
The assistant inherits your access rules.
If you lose access to a folder or record, the assistant loses it too.
The assistant can’t quietly become more privileged than you are.
Oso calls this “delegated access,” noting that permissions remain real-time: “if the user loses access to a record or a file, so does the agent.”
2) Default to read-only—and make “write” a step-up
For sensitive work, read-only is the safest default.
Grant write capabilities only when you can bound the outcomes:
sending messages
sharing files
changing settings
moving money
WorkOS recommends treating AI agents with the same rigor as users—authentication, scoped permissions, and auditability—using least privilege and explicit scopes in AI agent access control (2025).
Pro Tip: When you see permission choices like “Allow once” or “While using,” prefer them over “Always.” You want access to be situational, not permanent.
3) Separate roles (and don’t let one assistant do everything)
In high-trust environments, the temptation is to build a single assistant that can do it all.
That’s exactly how privilege creep happens.
A better model:
one assistant role can read documents but cannot send external email
another can draft messages but cannot access financial systems
a third can schedule travel but cannot open deal-room files
This is the logic behind role-based access control and granular permissions WorkOS describes (e.g., permissions like tickets:read and tickets:comment).
Controlled data access: the difference between “can access” and “is allowed to access now”
Permissions are the outer boundary. Controlled access is how you avoid over-sharing inside the boundary.
Think in four variables:
Purpose — what specific task is being executed?
Scope — which exact objects can be touched (this thread, this folder, this project)?
Time — how long does access last (minutes, not months)?
Auditability — can you reconstruct what happened?
The Cloud Security Alliance emphasizes that “data collection must follow data minimization and purpose limitation principles,” and highlights least privilege, JIT access, and audit logs in Data Security within AI Environments (2025).
A workable model: just-in-time access + auditable actions
Just-in-time access means the assistant gets credentials only when invoked, then those credentials expire.
Oso summarizes it directly: “An agent requests a credential each time it’s invoked; after the task, the system discards the credential.”
The goal is to avoid a dangerous default: an assistant that stays “logged in” forever.
AI assistant memory privacy explained (without hand-waving)
Memory is where many assistants quietly cross the line.
There are three different things people call “memory,” and you should treat them differently:
- Chat historypast conversations (helpful, but risky)
- Saved memory/personalizationstable preferences you approve (“use my assistant’s name,” “my home airport”)
- Training / improvementwhether your content is used beyond serving you
For sensitive work, the safest position is simple: memory should be minimal and intentional.
What an assistant should be allowed to remember
your approved preferences (tone, formatting, travel preferences)
your chosen workflows (“always confirm before sending external mail”)
project labels that don’t expose the content itself
What an assistant should not remember by default
raw documents, contracts, deal terms
credentials, recovery codes, authentication secrets
location history, itinerary details, personal identifiers
anything you wouldn’t want repeated verbatim later
The Cloud Security Alliance explicitly calls out retention and deletion controls (“Ensure retention and deletion practices align with business needs and legal requirements”) and the importance of monitoring.
The privacy test: can you review, clear, and revoke memory?
If you can’t:
see what is stored
delete individual items
wipe all memory
revoke connected integrations
…then “memory” is no longer a convenience feature. It’s a risk you can’t bound.
Hermes Agent is explicit about this control posture: “Permissions, integrations, sessions and memories can be reviewed, cleared or withdrawn when needed,” on the Hermes Agent page.
User confirmation: where “AI with Control” becomes real
An assistant that can act is categorically different from an assistant that can suggest.
Confirmation should not be a polite checkbox. It should be a security boundary.
At minimum, require explicit approval for:
sending external messages (email, WhatsApp/WeChat, Slack/Teams)
sharing or exporting files
changing account security settings
approving purchases, payments, or wire-related steps
deleting or moving records
Oso describes human-in-the-loop as an authorization strategy where a human grants access before a sensitive action proceeds.
VERTU’s Hermes framing matches the same idea: “Significant actions require your confirmation before they proceed.”
⚠️ Warning: A weak confirmation prompt becomes a social-engineering channel. The preview must be clear, the approval must be bound to the exact action, and the system should fail closed when it can’t classify risk confidently.
A 10-minute decision checklist for private AI assistants
Use this to evaluate any assistant you’re considering for sensitive work.
Permissions & identity
Can it operate via delegated access (your permissions), not a universal service account?
Can it be read-only by default?
Are permissions granular (folder/project/app), not “all or nothing”?
Controlled data access
Can you scope access to this thread / this folder / this project?
Is access time-bounded (just-in-time), or persistent?
Can you see an audit trail of what it accessed and what it did?
Memory
Can you view, edit, and delete what it remembers?
Can you disable memory entirely for sensitive workflows?
Is there a clear separation between “memory” and “training/improvement”?
Confirmation & revocation
Are high-impact actions approval-gated with a meaningful preview?
Can you revoke integrations immediately (and does revocation actually stop access)?
If any answer is “I’m not sure,” treat that as a “no.” The assistant is not ready for sensitive work.
Where Hermes Agent fits: user authorisation, controlled data access, AI with Control
If you’re looking for an assistant model aligned with this control framework, Hermes Agent is positioned around three ideas you can verify:
- User authorisation“Significant actions require your confirmation before they proceed.”
- Controlled data access“App access, private spaces, system zones and enterprise permissions can define what Hermes can see and what it cannot.”
- AI with Controlworkflows can be prepared, but action is gated by your approval.
That posture is consistent with VERTU’s broader security framing: isolate, then grant narrowly, as described in VERTU’s mobile AI security guide and in its broader control approach to autonomous agents in AI agent security risks and how to keep autonomous AI under control.
FAQ
Can AI assistants read my emails and files?
They can—if you grant them that permission through app scopes, connectors, or account-level access. The real question is whether you can limit access to the minimum needed (least privilege), scope it to a task, and revoke it.
What permissions are most dangerous to grant?
Anything that creates broad, silent visibility or action: full mailbox access, full drive access, system-wide accessibility/overlay permissions, and any “send/share/delete” capability without approval gating.
Is on-device AI always private?
On-device is often a stronger default because it reduces transmission. But privacy still depends on storage (retention), backups, connected integrations, and whether the assistant can export data elsewhere.
What does “AI assistant memory” actually mean?
It can mean chat history, saved preferences, or training use. For sensitive work, require a system where memory is minimal, reviewable, and deletable—and where training use is clearly separated.
Next steps
If you use an assistant for sensitive work, treat it like you would treat a new member of staff: define a role, set boundaries, and require approvals.
If you want a private, approval-gated assistant model designed around those boundaries, explore Hermes Agent—including its “you approve,” “you define boundaries,” and “you can revoke” control posture.
Disclosure: This article references VERTU pages. Editorial judgment remains the priority.
