Administration guide

Roles

Two roles. A member holds what the letters on each server give (see Groups): r-- sees, rw- works, rwx lets Heinzel change alone; beside that a member reads the knowledge base and the audit log. An administrator holds everything, including users, settings and every server, whatever the letters say; a server's ceiling still caps what Heinzel does there. The second approval of destructive commands is a named right, given to a member on the rights page. A member takes a seat.

The roles of earlier releases (viewer, operator, approver, nothing by itself) still work as they did: an operator is a member, an approver a member with the second approval, a viewer a member whose working rights are taken away. They are not offered any more; saving a role replaces them.

Invite users under Users. They receive a login link and set a password or second factor afterwards. Disabling a user ends their sessions.

Rights of one person

Under Users, "Rights" lists every right as a sentence with three answers: as it stands, yes, no. "As it stands" is what the role and the letters give, and the line says which. "No" wins over "yes". The ten rights that follow from the letters can be taken away here, so that somebody who works on servers by their letters still only looks; the named rights can be given.

The page says the outcome in one sentence before you save, and whether it takes a seat: rights to see are free, any right to act takes one. Nobody changes their own rights, gives more than they hold, or changes somebody who holds more. An administrator of the installation holds every right in every tenant; that cannot be narrowed. A page that is closed says which right is missing, and everybody finds their own rights on their account page.

Groups, and who may do what on which server

Rights are written as a file's mode is, in three letters that build on one another:

One rule says who may do what: what a person may do on a server is the highest access of their teams to the groups the server is in, and never above a ceiling. Four words carry it.

• A team is a set of people (Groups, Teams). Its managers put people in and take them out, name other managers, rename or remove it; whoever manages the users does that on every team, and whoever makes a team manages it. A member leaves a team on their own and never joins one.
• A group of servers is a set of servers (Groups, Groups of servers). Its managers put servers in and take them out, set its ceiling and maintenance window, and say which teams reach it. A server's form offers the groups its editor manages. There is no membership by tag: a server is in a group because somebody put it there.
• An access is one line: a team, a group of servers, letters. It is given by whoever manages that group of servers (on the group's page or on the team's); an access to a single server on the server's page, and to every server, the ground rule, on the page "Who may do what", by whoever manages the users. Everybody and every server are groups too, so "everybody works on every server" is one line, the ground rule a fresh tenant starts with (rw-), and one person on one server is one line as well. Every person and every server is a group of its own, made by Heinzel, so a single one is named the same way in a share, a model grant, as a schedule's owner or target.
• A ceiling is the most anybody may do, administrators included: on a server, on a group of servers, on a login, and the installation's for the tenant. The lowest holds. --- on a server blocks it for everybody, r-- makes it read-only; whoever saw it keeps seeing it, as ls lists a file nobody may open. Only an administrator raises a ceiling to rwx; whoever looks after the server or manages the group lowers it.

The page "Who may do what" (Groups) explains it for one person, server by server: the letters they hold, through which team to which group, and which ceiling held them down; whoever manages the users picks anybody, everybody else reads their own. Below that stands every access there is, each with the page it is changed on. A team's page shows what it reaches and who manages it, a group's page who reaches it; a server's page the ceiling, the groups it is in and the accesses that reach it; the rights page of a person what is theirs alone, beside what comes through their teams. The servers list marks a server with a + where an access names it or a group it is in.

Whoever adds a server works on it, through the group of their own, where they would not reach it otherwise. Somebody who does not see every server loses the rights that reach over the whole tenant (users, the audit log, the knowledge base, rollouts, everything under "Administer"), because those cannot be cut down to a few servers; servers and schedules can, so those rights stay and hold on the servers reached. The right "act across groups", held by administrators, lifts every such limit, but no ceiling.

A group of servers can carry a maintenance window: weekdays, from and to, in the tenant's time zone, over midnight if need be. Outside the window Heinzel only reads alone on the group's servers: every command that changes something waits for a person (two where the matrix asks for two at rw-), and a scheduled run skips what it would have changed and lists it under "needs an operator". A server in two groups is held to the window of each. The policy check on the commands page says when a window holds a command back, and a person who approves outside the window overrides it, which the audit log shows. A group without days has no window.

How much Heinzel does on its own

The x says it, and a session chooses below it:

The lowest of several counts, at every command:

• The tenant's ceiling (Installation, Tenants, by an administrator of the installation): rw- or rwx. Nobody in the tenant goes above it.
• The person's letters on the server, which the ceilings of the server, its groups and its login cap. Somebody in several teams gets the highest.
• The session: chosen when it starts, up to what the person may use. "Every command waits" is a session's choice only; nobody is held to it from outside, and a server that asked for everything before the upgrade is rw- now: reading runs alone, as it always did in unattended runs.

When somebody writes into a colleague's session, that turn is held to both of them. Read-only and blocked servers and the block list are not touched by any choice. What runs alone is not approved by anybody: give the x to people and servers where that is meant.

The matrix: what a class does at a level

What rw- and rwx mean exactly is a table under Settings, Policy: one row per risk class (write, package, service, firewall, identity, reboot, destructive), one column per level, and in each cell "runs at once", "waits for one person", "waits for two people" or "never runs". By default everything but reading waits for one person at rw-, destructive commands and identities for two; at rwx everything runs at once and identities wait for two. A company where Heinzel never reboots and two people see every firewall change sets that here. Three cells nobody moves: reading runs, a blocked command never runs, an identity (an account, a key, a sudo line) waits for a person at least. A server's form holds a cell stricter for that server alone, never milder. A scheduled run skips what waits; the second of two people is an approver or administrator who signed in within the last five minutes.

A change that could lock Heinzel out

Before Heinzel changes an address, a route or the firewall, it arms a timer on the server that restores the old state after two minutes, in the same command as the change. Stopping the timer is a service command: at rwx Heinzel decides it alone, below it somebody approves it in time. Either way it goes out over a new login, never the open connection, which outlives a firewall that already refuses new ones. If that login fails, the timer runs and the server undoes the change by itself. After an address change Heinzel logs in at the new address first and, when the host there shows the pinned host key, enters it for the server itself; the audit log says "Followed to its new address". A server known by name follows its DNS instead: until DNS names the new address, the change is rolled back. The session says so on the command that arms the timer, and again on a stop that failed.

Schedules

A schedule runs with nobody there, so it cannot ask. It names one of two levels. rw- reads, whatever anybody set. rwx goes as far as a session of its owner could go alone on that server: the lowest of the tenant's ceiling, the owner's level and the server's limit counts, at every command. Only somebody who may let a session change things alone can choose it, and whoever saves a change to the level, the task or the model becomes the owner. What a run may not do alone is skipped, named in the report and kept on the approvals page for a week, where somebody takes it up in a session of their own. Read-only and blocked servers and the block list hold for schedules too.

Limits a tenant draws itself

Beside the level, a tenant has four tools. None of them lifts the block list or opens a read-only server.

Emergency stop (Settings, Policy, and a banner on every page): no session starts, no command runs, no schedule fires, and what is running is cut off. It needs a reason. An administrator lifts it.

Block list (Settings, Commands): commands that never run here, on every server or on some, in words or as a regular expression. Instead of blocking a command, a rule can hold it to a higher class ("counts as"): rm under /srv/db as destructive, a restart of the database as a reboot. The matrix then decides as it does for that class. No rule lowers a class; the standing approval with its exact line is for that.

Standing approvals (Settings, Commands): commands that are always fine. An entry is one exact line; a proposal that equals it runs without a person and is logged as such. Destructive commands are never covered.

"Try a command against the policy" in the settings shows what all of this decides for a command on a server, without running it.

Control after the fact

Countersign (Sessions): a finished session in which a changing command ran without an approval waits for somebody who had no part in it to countersign it. It is off in a new tenant; switch it on under Settings, Policy, where an audit asks for four eyes on every change.

Access review (Users): who both asks and approves, who has full autonomy, who has not signed in for ninety days, who has no second factor. An administrator attests it; the page says when that is a quarter old.

Change record (session page): a Markdown file with who asked, who decided, how each command ended, and where its entries stand in the audit chain.

Incident draft (session page): the same facts as a timeline, with the sections a person has to write (impact, root cause, what to change) marked TODO. No model writes it.

Roll back (session page): every changing command is proposed with a note on how to take it back, shown on its card. Roll back puts those notes, newest first, into the task of a new session for a person to read and send. Nobody checked the notes, so nothing is undone by the button itself: each step is proposed and approved like any other command.

Summary (top of the session page): what Heinzel says the session came to, as changes, verification, open points and how to undo. It is the model's own account, not checked against what ran; the change record is the record. Counters at the top say how much stands in each part, and a part with nothing in it takes no room below them. A way back is spoken of only where something was changed. Open points are added to the server's to-do.

The second approval of a destructive command needs a sign-in no older than five minutes.

Tenants

A tenant is one customer, department or environment. It has its own servers, SSH keys, sessions, schedules, audit log, model provider, policy and alerts. Nothing of it is visible from another tenant, to people or to the agent. An installation starts with one tenant and needs no more; with one tenant nothing in the interface mentions tenants except the Tenants page.

• Accounts belong to the installation, roles to a tenant. One person has one account (one password, one passkey, one second factor) and a role in each tenant they are a member of. Someone with several tenants lands in the one last worked in and changes with Switch at the top of the navigation.
• One tenant per tab. A tab works in the tenant its address names (/acme/servers), so several tabs work in several tenants side by side. A tab that is loaded anew, or comes back after the connection was lost, comes back in its tenant, whatever the other tabs did: a half-filled form is never carried over into another one. With several tenants the tab's title names the tenant. The first request of a sign-in into another tenant is written to that tenant's log. If the tenant of an address is closed to somebody, they are sent to choose again; nothing they were about to do is done elsewhere. A change of role in one tenant leaves the pages of another alone.
• One account per browser. The tabs of a browser share the sign-in. Signing in with another account while one is signed in is refused with "Log out first". For two accounts at once use a second browser profile or a private window.
• Administrators of the installation create, rename, close and delete tenants, keep the tariffs, and alone change what spans tenants: sign-in rules, single sign-on, the license, the rule set, mail delivery and another person's account (lock, second factor, passkeys). All of it is under Installation (see below), outside every tenant. They see each tenant from outside (its tariff, limits, use and administrators) and enter only the tenants they are a member of, with the role they hold there, narrowed like anybody's. The first account of an installation is one; they appoint further ones under Installation, Accounts. Entering a tenant is written to that tenant's audit log.
• A new tenant names the address of its first administrator. That address is invited by mail, as any invitation, and holds every right in the tenant once it says yes; it then adds its own people, servers and model provider. Whoever created the tenant stays outside. A further administrator can be invited the same way under Installation, Tenants, Edit.
• Tariffs (Installation, Tariffs) are presets of a tenant's limits: seats, servers and how far Heinzel may go on its own. Choosing one for a tenant fills in its limits, which can then be changed for that tenant alone; the list marks a tenant whose limits differ from its tariff as "adjusted". Changing or removing a tariff leaves its tenants' limits as they are. Payment is not part of it.
• A tenant's administrator manages its members and their roles, its servers, keys and settings, except the addresses the server calls (model base URL, webhook), which an administrator of the installation sets. Remove takes a member out of the tenant and leaves the account alone.
• Invitations need a yes. Inviting an address mails the person: a login link for a new account, a note for somebody who already has one. After signing in they accept or decline, on the page they land on or under Switch tenant. Until then the member list shows the address as invited and nothing of the account.
• Closing a tenant keeps its data, lets nobody in and rests its schedules, and can be undone. Deleting is for a closed tenant only and asks for its address to be typed: everything in it goes, its audit log included, while the accounts and other tenants stay. Each tenant's audit log is a chain of its own, so the others still verify; the deletion itself is recorded in the installation's log. The work runs in the background and picks up again by itself after a restart.
• Seats are counted per account for the whole installation: someone who works in three tenants takes one seat. An invitation takes its seat when it is accepted; if none is free then, the person is told to ask an administrator. All tenants draw on the one pool of the license.
• Seats per tenant: an administrator of the installation can give a tenant a number of seats it may use (Tenants, Edit). The tenant's own administrators see the use on the Users page and are refused, with the limit named, when a role, a right to act or an accepted invitation would exceed it. This limit counts who acts in that tenant, so a person who acts in two tenants takes one seat of each, and one of the license. Without a number a tenant is held to the license's total alone, which stands above all tenants either way. A limit set below what is in use takes nothing from anybody: the Tenants page shows the row in red, and no further person gets in until it fits again.
• Servers per tenant: next to it stands the number of servers a tenant may have. The server form refuses, with the limit named, when it is full; a removed server leaves its place to the next one. Without a number only the license's total counts. A limit set below what is in place removes no server. Below the form the page says what the license allows the whole installation in seats and servers.
• Single sign-on: with one tenant a new account joins it with the default role. With several it joins none and waits for an administrator.

Locked out

When nobody can receive mail, an operator with shell access on the host prints a login link for any account:

docker compose exec app bin/login-link admin@example.com

The link is single use and valid for 15 minutes. Open it, then set a password or second factor under Account.

Passkeys are added by each user under Account. When a device with a passkey is lost, "Remove passkeys" on the Users page clears that user's passkeys; the user signs in with password or login link and adds a new one.

Installation

The installation's own pages stand outside every tenant, at /installation, behind the entry Installation at the bottom of the navigation. Only administrators of the installation see the entry and open the pages, with or without a tenant of their own; a way back into the tenant they came from stands at the top. The pages: Tenants (create, edit, close, delete, with the installation's emergency stop), Tariffs, Accounts (everybody who can sign in, with their tenants; lock an account, reset its second factor or passkeys, send a login link, make it an administrator of the installation, and put a person into a tenant or take them out of one when the tenant's own administrators cannot, say because its only one is locked out), Mail delivery, Settings (sign-in, license, rule set) and Log (the installation's own audit chain: logins, accounts, tenants; and whether the installation runs: version, license, mail, migrations, background jobs). A single-tenant installation has the same pages and needs them rarely.

Settings

The settings are pages with a navigation beside them, seven for the tenant. Sign-in, license and the rule set are the installation's and are under Installation, Settings.

General: organization name, language for system mails, time zone (the clock schedules run by and times are shown on), retention.

Policy: approval mode, the matrix (what a class does at a level), after how many minutes a waiting approval is escalated, what new sessions are for everybody, links for people without an account, output scrubbing, command timeout and the longest timeout Heinzel may ask for with a single command (a package upgrade). The emergency stop is on this page, and so are the capabilities below.

Capabilities

The open Heinzel rule set (version 3.0 and later) reads partition tables, boot loaders, accounts and secrets and changes none of them: that is Heinzel Pro's work, and it is switched on per tenant on the policy page.

• Identity: creating and changing accounts, SSH keys and sudo rights. On from the start. Such a command is of the class "identity": it waits for a person at every level, for two where destructive commands do, and never runs unattended. Deleting an account or changing a password stays destructive.
• Secrets and certificates: rotating a secret with every consumer mapped first, watching certificates across the fleet and renewing them. On from the start; off for a tenant that leaves secrets to a vault.
• Partitioning: writing partition tables. Off from the start.
• OS replacement: making filesystems, writing raw disks, changing boot loaders and EFI entries. Off from the start.

Rollouts, schedules, the fleet audit, OS upgrades in waves, the version matrix and maintenance windows need no switch: they are the runtime's own, and their rules are always in the agent's list.

A switch does three things at once: the rule files for the work (priv/rules_pro, shown on the rules page with their capability) go into the agent's instructions, the classifier lets the commands through as destructive, and the policy asks for the approvals. With the capability off the same commands are refused with a reason that names the switch, and the agent is told to read and report instead. A server's form switches a capability off again for that server alone, so a database host is never repartitioned however the tenant is set. Rollouts, schedules and the fleet audit need no switch. Every change is in the audit log and in the policy file.

The copy of the open rule set an image carries is one tagged release (priv/rules/VERSION and SOURCE say which); a newer tag of the same major version is taken every hour on its own, a new major version from Installation, Settings, Rule set, both without a new image (upgrade.md, "Rule set").

Your own rules

Under Rules, "Your own rules" holds what the organization adds to the rule set, in the shape Heinzel keeps it in a checkout: a text per rule, or all for every rule, for every server of the tenant or for one server. Headings say what a section does: ## Add: adds, ## Replace: replaces the section of the rule it names, ## Remove: drops it; a heading without a prefix adds. Later wins, as in Heinzel: the rule set, then the text for every server, then the server's own. The all texts go into every session's instructions below the rules and above the knowledge notes; a rule's own sections follow the rule when the agent reads it.

A few rules take additions only: the connection pipeline, anomaly detection, secrets hygiene, and every rule of Heinzel Pro's own that the manifest does not mark as overridable. The ethics rule takes nothing. A refused section stays on the form with its reason and is written to the audit log; nothing is dropped in silence. Writing the texts takes the right to shape the policy; reading them is open to every member.

A team that moves from Heinzel imports its checkout: the files of memory/custom-rules (all.md, one per rule) and a server's rules.md, memory.md and todo.md (its memory and to-do list go into the server), uploaded as they are, with a preview of what is taken and what is skipped (a file named after no rule of the set in use, or one with a refused section). HEINZEL_RULES_DIR stays the installation's level: a mounted directory replaces the public rule set for every tenant, and the texts here lie over it as they lie over a release.

Commands: the block list, the standing approvals, and a form that says what the policy would do with a command.

Sign-in (Installation, Settings): password login, mandatory 2FA, sign-out after idle time, single sign-on.

Model providers: a provider is one account at a vendor, or one server in your own network: a name, Anthropic or OpenAI-compatible, the key and, for an administrator of the installation, the base URL. It offers as many of its models as you tick. A typical setup is one Anthropic account with three models: a balanced one for everybody, the strongest for the senior team, the quickest for small jobs. Add the provider once, tick the models Heinzel Pro knows or type other ids, and test. The key is entered in one place and changing it reaches every model of the provider; an empty key field keeps the stored key. Advanced options cover Anthropic thinking and effort, OpenAI reasoning effort, and compatibility switches for Azure and older servers.

Each model then has a page of its own for what people see and may do: its name, the efforts it offers, its prices, and who may use it. Whoever starts a session chooses among the models open to them. A provider keeps its kind; a second vendor, or the same vendor's EU endpoint, is a second provider.

• Who may use it. A model is open to every member of the tenant, or restricted to the groups and people you tick. A small team leaves everything open and never sees this. Administrators are asked like everybody else.
• A stronger model (on the model's form): start sessions on an inexpensive model and name the one a session can be handed to. The agent hands over when the task turns out hard, and so can whoever may write in the session. It goes one way, the audit log says who asked and why, and the session page shows how many tokens each model used.
• A fallback (on the model's form): the model that answers a step when this one's provider is overloaded or does not answer, after the usual retries. It is used only for people who may use it too, the session and the audit log say so, and the next step asks the first model again.
• Groups (Administration, Groups) are named sets of members, such as a team. Grant by group and a newcomer gets the right models by joining it; a person ticked by name is granted through the group of their own.
• Effort. Tick the efforts the model takes; tick none for a server that rejects the setting. Then say who may go how far: every member up to one level, and groups or people further. Whoever starts a session picks among what they may use, or leaves it at Standard (as you set the provider up under Advanced options). It is passed on as effort to Anthropic and as reasoning_effort to OpenAI-compatible servers. Schedules run on Standard.
• Model catalog. The model field offers the models Heinzel Pro knows; picking one fills in its name and efforts. Any other id can be typed. A newer catalog, signed by the vendor, is pasted at the foot of Settings, Model providers by an administrator of the installation; a catalog also comes with every release. Heinzel Pro never fetches it.
• A model the vendor retires. Vendors retire models with weeks of notice, by mail and on their web pages, never through the API; requests to a retired model fail. The catalog carries these dates. A model on its way out is marked under Model providers and on the dashboard, with the day and the successor, and one click switches it: the id, the efforts and the catalog's name change, who may use it, where it is prescribed, its fallback and its place as the default stay. Its prices are cleared, since the successor costs something else. The alert "A model is being retired by its vendor" goes out when the retirement is announced, two weeks before the day, and when the day has come. A session that hits a retired model fails and says so; it is continued in a new session once the model is switched. Nothing switches on its own: a model prescribed for a server is a promise about where its output goes.
• Default. One provider is the default. Whoever may use several picks one when starting a session; with only one, no choice is shown. A schedule names a provider or takes the default at the time of the run.
• A session keeps its provider. Access is checked with every message and before every model call. A provider that is switched off or removed, or access that was withdrawn, ends the session there; no other model takes over silently.
• A schedule runs in the name of the person who last changed its task, its model or its level, on a provider that person may use. Which group may change it is a different matter, see Schedules.
• Removing a member takes them out of every group and grant of the tenant, their own group included. An administrator of the installation who is no member of the tenant cannot be granted a restricted provider; add them as a member first.
• Prices (optional). Enter what you pay per million tokens: input, output, cached input read, cache write, and the currency. Every model call is then priced when it is made, at the prices of that moment, and sessions show their model and what they cost. Providers return token counts, not prices, so without your prices none is shown and nothing is estimated. A session shows a cost only if every one of its calls could be priced; calls made before 2.1 cannot, because cached and fresh input were not told apart then. Taxes, discounts and the vendor's invoice rounding are not in the figure.
• Prompt caching is on by default (Settings, Sessions). The provider then keeps the instructions and the course of a session in its cache between steps, and a step pays for what is new at the full price and for the rest at the cache price. Anthropic does this; other providers cache as they see fit. What changes from step to step, such as a host's memory, is sent apart from what does not, so the cached part survives it.

Internal model in your own network: choose OpenAI-compatible, set the base URL of your server (Ollama http://ollama.intranet:11434/v1, vLLM or LM Studio http://host:8000/v1, LocalAI) and the model name it serves, and leave the key empty if the server has no authentication. Nothing leaves the network then; the container only needs to reach that host. Models with tool calling work best (for Ollama, a model whose card lists "tools").

When the container cannot reach that host directly, pick under "Reached through" one of your VPN connections or servers. The address is then dialled inside your network: through a VPN connection, a name in your tailnet or LAN (LM Studio on a laptop, http://macbook:1234/v1); through a server, an address as that server sees it, so http://localhost:11434/v1 is an Ollama that listens only on the server itself. The server forwards the port over SSH as its first login without sudo; sshd must allow it (AllowTcpForwarding, on by default). On a hosted installation a tenant's administrator may set the address only this way.

The policy as a file (Settings, Policy file): how much Heinzel does on its own, the four-eyes rule, sharing, timeouts, the block list and the standing approvals as one JSON file with a SHA-256 over its content. Paste a file to compare it with what is set; applying it makes the tenant's policy equal to the file and is recorded with the hash. Rules for one server name it by host name. Per-server settings, addresses and secrets are not in the file.

Alerts: a list of email addresses and an optional webhook (Slack, Teams, Mattermost or generic JSON). Pick which events alert: approval needed, command failed or denied, host key changed, scheduled run finished, repeated failed logins, user created, an approval that waited too long, and a weekly mail of what was left undone. Events added by an upgrade are off until you switch them on. For Teams, make the webhook with Workflows ("Post to a channel when a webhook request is received"); the older Office 365 connectors are retired and do not take these cards.

Alerts from the monitoring (Settings, Integrations): make a token and have the monitoring post to /api/alerts/<tenant> with it as a bearer token. Alertmanager's webhook is taken as it is; any other system sends JSON with host, title and text. An alert whose host is one of the tenant's servers starts a session there that reads only and reports cause and proposed fix; the answer to the post carries the session's address. The same alert within an hour finds its session, and at most twenty sessions start in an hour. The sessions run in the name of whoever made the token.

The same page marks changes in the monitoring: with an endpoint that takes annotations as Grafana's API does (https://…/api/annotations, a service account token), every command above reading that ran appears in the graphs with host, command, who approved it and how it ended.

The same page ships the audit log: an administrator of the installation gives a tenant an HTTPS endpoint and an optional bearer token, and every audit event of that tenant is posted there within a minute as one line of JSON with its position and hash in the chain. What the endpoint does not take stays owed. An event carries what the audit log shows, the text of commands included.

Single sign-on (Installation, Settings, Sign-in): any OpenID Connect issuer. Redirect URI is <base url>/auth/oidc/callback. New SSO users get the default role; an allowed-domain list restricts who may log in.

License (Installation, Settings): paste the key. Trial, grace period and seat use are shown.

Servers

Add a server with host name, port and first login (SSH user), and set its ceiling, the most anybody may do on it, in the letters of a file's mode (see Groups): --- blocks it, r-- makes it read-only, rw- lets people work with Heinzel asking before every change, rwx lets Heinzel change alone, which only an administrator switches on. Who reaches the server below the ceiling is said by the accesses of teams to the groups it is in; whoever adds a server works on it.

A server also carries tags, words or key:value such as env:prod role:db. The servers list filters by them, and several tags narrow to the hosts that carry all. Block rules and standing approvals hold on every server or on one, not by tags.

Findings (navigation): what a scheduled run found wrong, one entry per server and problem. Seen again the next night it stays one entry with a count; no longer seen by a later run of the same schedule it closes by itself. Operators take a finding, close it by hand, or start a session from it after reading the task the check proposes.

On many servers (New session, "The same on all, in parallel"): tick the servers, by hand or by tag, and write the task once. The list of sessions shows the rollout as one row that leads to its page. It runs on the first server as an ordinary session with every approval it needs. When that session is through, somebody who may approve commands opens the rollout, looks at the first server's session and says go; the task then runs on the others, four at a time. The rollout's page lists, before the go, the commands it covers: those a person approved on the first server and that succeeded there, none of them destructive. On each other server these run without asking, in that order, each only as the same text on that session's own server and only after the one before it succeeded there, and only in the session's first turn. The person who said go is recorded as who approved, and has to be able to read the first server's session. Anything else asks as always, and so does a command for two people, one that names a secret and a file edit. The rollout's page shows a row a server with its state and its commands; each server keeps its own session, record and stop button.

Managed files (navigation, administrators): name the files that Ansible, Terraform or another tool writes, as a path or a pattern (/etc/nginx/**), with where a change belongs instead. The list holds on every server. Heinzel is told the list for the session's server, a command that would change such a file is refused with that pointer, and so is a file edit. A file whose first ten lines say it is managed (Ansible managed, This file is managed by …) is refused without an entry: before the approval where the file may be read then, and in any case by the write itself, which looks at the file's first lines on the host. A pattern that ends in /** covers the directory too. A command is refused when a managed file is what it writes to: a copy from one, an archive of one or a test of one is left alone. The list is kept by hand; no repository is read. The check on commands finds a path that stands in the command, not one built from variables or reached through a symlink.

An entry takes a mode, the way Heinzel's memory/protected-paths.md has them. Managed by a tool is the above. Read-only refuses every change, with the reason you give. Hidden refuses reading the content as well, not even a line count: a secrets file, a directory with other people's data. Changes wait for a person has every change there read by a person before it runs, whatever the session may otherwise do alone. The agent is told the list with each mode, and that a glob is a hint, not a sandbox: a recursive copy of a parent touches the path too.

Attachments (session page): the message box takes files by the paper-clip button, by dropping them on it and by pasting from the clipboard: text files up to 300 kB, and for a session whose model is an Anthropic one also images (PNG, JPEG, GIF, WebP) and PDFs up to 3.5 MB, five a message and 12 MB a session, since everything attached goes to the model with every step. What a file is, is decided by how its bytes begin and end. Files are kept in the database with the session and deleted with it; whoever may write in the session can take one out again, and a file taken for a message that is never sent is deleted a day later. A text is scrubbed of the kept secrets and like a command's output, and one that reads like instructions to Heinzel is kept from it until somebody who may write in the session lets it through. An image cannot be checked that way.

File edits (command card): when Heinzel changes a configuration file it names the file and its whole new content, and the card shows the difference as a diff. The file as it is gets read for that only where reading runs without approval; elsewhere the card shows the new content, and the diff comes back from the host once the command ran. Where a diff was shown, the write first checks that the file is still what the diff was shown against, and refuses otherwise. A file edit always asks on its own; no plan and no standing approval stands in for it. A copy of the old file is kept beside it as <file>.heinzel-<time>.bak; these copies are not cleaned up. The write goes through a symlink to its target and is not atomic.

Plans (session page): for a task of three or more changing commands Heinzel first proposes a plan: the goal, each step with its exact command, what could go wrong and how to undo it. Whoever may approve commands approves or denies it as a whole. An approved step lets exactly that command run once on the server named; a command that differs by a character asks on its own, and so does one that needs two people or names a secret. The steps run in their order, the next only after the one before it succeeded, and the plan holds for its turn: a stop or the next message ends it. A waiting plan shows on the Approvals page and in its count; no mail or push is sent for it yet. Unattended runs have no plans.

sudo has to work without a password for the user Heinzel logs in as, or Heinzel logs in as root. Heinzel keeps no password for a host. Where sudo asks for one, a command that uses it fails at once, and the agent works without privileges and says so.

Logins (the server's page, "Logins and key"): a login is a user Heinzel signs in as. A server has one or more, such as root and stefan. The server's key opens each; add it to the authorized_keys of every user. Whoever opens a session picks the login where there is more than one. A session that goes on from another, rolls it back or takes up one of its commands signs in as the same user. A schedule and a rollout name a user that holds on each of their servers (without one, the first login their owner may use); a server without that login is left out, not worked on as somebody else.

A login has a ceiling of its own, which only lowers what the server allows, as a file narrows its folder: root@web1 with r-- lets everybody only read as root while they work as stefan. A login may also name who uses it: nobody ticked, everybody who reaches the server; a team ticked, that team alone, a root login for the administrators. Only somebody who may work as a command's login approves it.

Without sudo: for a user other than root the login's form asks whether it may use sudo. With the box off, Heinzel works with that user's own rights and refuses every command that names sudo, sudoedit, doas, su, pkexec or run0, even as an argument. That is a net, not a wall: what the user may really do is set in the host's sudoers.

A login nobody needs any more is retired, not deleted: it is offered to nobody, a session that works as it runs nothing more, and the old sessions still show who they worked as. A server's last login stays.

Secrets (navigation, administrators): keep a password or a token under a name, and Heinzel uses it as $HEINZEL_SECRET_NAME in a command. The value is never sent to the model; it reaches the command on its standard input, not on the command line, and is blacked out of the output before that is stored. Use random values of twelve characters or more: a value that can be guessed can be found out by a command that prints guesses and sees which one was blacked out. A command that names a secret waits for a person at every level of autonomy and does not run unattended, because a command can pass a value on in another shape (encoded, written to a file). Read such a command before you approve it.

A schedule can name tags instead of one server. It then runs on every server that carries all of them, four at a time, skips blocked servers, and mails one report: the hosts that need somebody first, then a line for each that is fine. A server tagged later joins the next run.

Copy the public key into authorized_keys of every login, then Test connection. The host key is pinned on first contact. If a host was reinstalled, use "Trust new host key". "Rotate key" issues a new pair.

A host that takes SSH only from a jump host names that server under "Reached through (bastion)". Heinzel connects to the bastion with the bastion's key and pinned host key, tunnels from there and logs in to the host with the host's own key. The bastion must allow TCP forwarding (AllowTcpForwarding), and one hop is supported.

Handover (server page) is a Markdown file with what is stored about the server: system, access, the agent's notes, open to-dos and findings, the journal, its sessions and its schedules. It names only the sessions its reader may read, and no model is asked to write it.

Memory is what Heinzel remembers about a host: OS, services, paths, quirks. The agent updates it after work; you can edit it. The journal lists changes.

A host that others administer too

Two tenants can add the same host, each with its own row, key pair, host-key pin, memory, journal and to-dos; neither sees that the other has it, and the comment of the public key in authorized_keys is the same for all (heinzel-pro). What Heinzel Pro knows about a host stays in the tenant.

The host itself is common ground: whoever has a shell on it sees its files, packages, logs and shell history, with or without Heinzel. One thing adds to that: Heinzel's handover notes in the system journal. A new server leaves and reads them. A tenant that shares a host with strangers who must not read along switches both off in the server form; the next section says what the notes are and what can be set.

Handover notes in the system journal

A handover note is one line in the host's system journal under the tag heinzel. Whoever runs journalctl -t heinzel there sees what Heinzel changed: colleagues with a shell, and a second Heinzel instance that looks after the same host.

The server form has two boxes for it. "Heinzel leaves handover notes" gives something away, "Heinzel reads the notes it finds there" brings a stranger's text to the model. On a new server both are on. Whoever looks after servers may switch either off; only an administrator switches it on again. Notes that are read count as facts about the host, never as an instruction.

What a note says is set by the tenant under Settings, Policy, and a server may differ. Both pages show the same controls, and above them the line the current choices make:

The server form asks one question about it: "Like every server" or "Different for this server". Only the second answer shows the five settings, and a dot marks what every server has. Whoever picks that value goes on following the settings. The form does not speak of a tenant: who looks after servers knows the settings, not the tenant behind them.

The form shows the line the current choices make, such as [Ada Lovelace/root] Updated nginx to 1.28.0, because the old version had a security hole. After the slash stands the account Heinzel works with on the host. A scheduled run names no person, since nobody was there; it does name the contact address.

The model gives only the sentence. Heinzel Pro puts sender, account and link around it and writes the line itself; a logger -t heinzel of the model's own is refused, and so is every read where the server reads no notes, at every autonomy level and also when the command was approved before the box was unticked. The note runs as a command of the session and is treated like a command that reads: it waits for an approval where reading waits. Language and wording remain the model's; they are asked for in the prompt and cannot be checked in code. The gate knows the ways the rules spell such a command and the obvious others; it is a filter, not a proof.

A journal is often forwarded to a central log server. The shared contact address keeps a single person's address out of it.

SSH keys

Heinzel Pro owns every private key it uses. Keys are generated inside the application, encrypted at rest and never exported; there is no way to upload a private key, and that is deliberate: a key that also lives on a laptop cannot be revoked by Heinzel Pro alone, and its use could not be told apart from a person's.

Two shapes:

• Own key per server (default). Least privilege: one compromised host reveals nothing about the others. Rotate under the server.
• Shared key (Administration, SSH keys). One key pair for a group of hosts, so configuration management installs one public line everywhere. Operators pick it in the server form. Rotation keeps the previous key as "retired": hosts that still carry only the old line keep working and are marked, and the administrator drops the retired key once every host has the new line.

Access for people is not a question of keys. Everyone with the Operator role works through the same keys; the audit log records who asked for what. Give each host a dedicated heinzel user with the sudo rights it needs rather than personal accounts, so host logs and the audit log tell the same story.

VPN connections

For servers Heinzel cannot reach from the internet: behind a home router, in an office network, without a public address. A VPN connection is an encrypted tunnel (WireGuard) between Heinzel and your network. Heinzel logs in through it with SSH, with the same key and the same pinned host key as everywhere else. Administrators set connections up under "VPN".

There are three ways. If you are unsure and do not use Tailscale, take the first; if you already use Tailscale or Headscale, take the third.

Way 1: your server connects to Heinzel

1. "VPN", "New VPN connection", enter a name, save.
2. Install WireGuard on the server: sudo apt install wireguard (Debian, Ubuntu) or sudo dnf install wireguard-tools (Fedora, RHEL, AlmaLinux, Rocky Linux).
3. Copy the block from the connection's page and paste it into the server's terminal. It saves /etc/wireguard/heinzel.conf and starts the tunnel, now and after every restart.
4. Wait for the status to show "connected", usually within ten seconds. The page updates by itself.
5. "Add this machine as a server". Address and connection are filled in. From there on as with any server: install the public SSH key, test the connection.

Nothing else about the server's network changes: only what is meant for Heinzel's tunnel address goes through the tunnel. The block contains the key of the connection, so treat it like a password.

You can switch the tunnel off on your side at any time: sudo systemctl disable --now wg-quick@heinzel

A whole network through one machine. Devices that cannot run WireGuard (a NAS, a switch) are reached through a Linux machine in the same network that is always on. When creating the connection, enter the network under "Reach a whole network through one machine", for example 192.168.1.0/24. The block then sets up forwarding as well (install iptables too) and lets Heinzel into the named networks only. Add the other devices with the address they have in your network and select the VPN connection.

Way 2: Heinzel connects to your router

Add Heinzel to the router as you would add a laptop: as a new WireGuard device. The router then offers a configuration file. Open it with a text editor and paste all of it under "New VPN connection", "Heinzel connects to my VPN router". Heinzel stores the key in it encrypted and never shows it again.

FRITZ!Box: "Internet", "Permit Access", tab "VPN (WireGuard)", add a connection, connect a single device, name it, confirm on the box, download the settings. The labels differ a little between FRITZ!OS versions. The box needs a MyFRITZ! address and a line with a public address; on a DS-Lite line take way 1.

Other routers: create a WireGuard peer (client) for Heinzel and export its configuration. Allow it the managed servers and their SSH port only.

Add servers with the address they have in your network, for example 192.168.178.20, and select the VPN connection.

Way 3: Heinzel as a device in your Tailscale network

For those who already use Tailscale or Headscale. Heinzel joins your network as one more device, named heinzel. Nothing is installed on the servers and nothing is changed on a router.

1. In the Tailscale admin console: "Settings", "Keys", "Generate auth key". Leave "Ephemeral" off. If you use tags, give the key one, for example tag:heinzel: a tagged device does not expire. Without a tag, switch off key expiry for the device heinzel on the "Machines" page later, or it has to join anew after some months.
2. In Heinzel: "VPN", "New VPN connection", "My servers are in a Tailscale network", enter a name and the key, save. For Headscale, also enter the address of your control server.
3. Wait for the status to show "connected". With device approval switched on, approve the device first; the page says when it waits for that.
4. In your network's access rules, allow the device the managed servers on their SSH port, and nothing else.
5. Add servers with their Tailscale address (it starts with 100.) or their name in the Tailscale network, and select the VPN connection.

Heinzel uses the key once and forgets it after joining. Tailscale SSH is not used: Heinzel logs in to the ordinary SSH service, with its own key and the pinned host key. Add a server behind a subnet router with its address in that network.

To switch it off on your side, remove the device heinzel under "Machines" or take its access rule away.

If the page says Heinzel is not signed in, the key was refused, has expired, or the device was removed: make a new key and paste it under "Change". If a server is not reached, check the access rules first.

If it stays on "waiting for the other side"

• Way 1: sudo wg show heinzel on the machine shows "latest handshake" once the two sides have spoken. The machine must be allowed to send UDP to the port named on the connection's page; a company firewall needs a rule.
• Way 1, connected, but the server does not answer: a firewall on the server must allow SSH from Heinzel's tunnel address, with ufw sudo ufw allow in on heinzel to any port 22 proto tcp
• Way 2: the router must be reachable from the internet under its name. If the connection was deleted or made anew on the router, paste the new file under "Change".
• If no connection of way 1 has ever come up, the installation's UDP ports are probably closed, see installation.md.

Limits

• A tenant cannot have two servers with the same address and port, not even in two different networks. Way 1 gives every connection addresses of its own and does not meet the problem.
• Behind a bastion, the VPN connection belongs on the bastion.
• After a restart, Heinzel Pro calls the other side where it was last heard from. If its address has changed meanwhile, the tunnel is back only once the other side calls; that can take some minutes.
• A connection of way 3 is a device in exactly one Tailscale network. For a second network, make a second connection.

Sessions

Pick a server, describe the task. Each proposed command appears as a card with its risk class and rationale. Approve or deny; a denial can carry a reason the model reads. Output is shown unredacted to you. The Approvals page lists pending commands across all sessions.

Several at once

Sessions run side by side: start one, start the next, and go where one needs you. The list and the rail beside an open session show what each is doing:

What needs you is on top. For your own sessions (started by you, or working on your message) the navigation shows a count, the browser tab carries it in its title, and with "Notify me in this browser" the browser shows a notification while you are in another tab or window. The tab of a session speaks for that session alone: its title begins with "● Approval needed", "● Failed", "● Stopped" or "● Answered" when it needs you, and with "…" while Heinzel works, so among several session tabs you see which one to go to. Two tabs of one session show the same. An answer counts as read once its page is in front of you, not while the tab sits in the background. Sessions shared with you are listed with their state and do not ping you; their answers count as unread only after you opened them once.

Sharing

A session belongs to whoever started it. Sharing is set for each session, in the session itself: People shows who is in, and there the owner brings a colleague or a group in, changes what they may do, or shuts them out again, one at a time and at once. Somebody shut out who has the session open loses it at that moment. The owner opens it:

• to everybody in the tenant, for reading or for writing,
• to groups and people, for reading or for writing,
• to people without an account, for reading, by a link that ends after 1, 7 or 30 days or when it is taken back. An administrator allows links under Settings, Policy; they are off at first. The page behind a link leaves out names and blacks out secrets in command output; host names and commands are shown as they are.

Writing means sending messages, so a colleague can go on where the owner stopped. Every message shows who sent it, and the audit log records each message, each change of sharing and each link with its actor. A turn somebody else asked for runs with the lower of both people's rights. With Take over, an administrator, or somebody the session is shared with for writing by name or through a group, becomes the owner; the previous owner keeps writing, and a link ends. Being one of "everybody" is not enough to take a session over.

Defaults: Settings, Policy says what new sessions are for everybody in the tenant (private, readable, writable). Each member can set their own under Sessions, Sharing, for example "private, and the group On-call may write".

Administrators of the tenant can open every session. The Approvals page shows every waiting command to those who decide on it, with a link to the session only where it is shared with them. The audit log leaves out what happened in sessions the reader may not open. Scheduled runs are readable for the whole tenant. Two things stay visible beyond a private session: the server's journal lists every change made on it, with the command, and notifications about a waiting command go to whoever the tenant set up to receive them. Messages written before 2.4 carry no sender.

Schedules

A schedule runs on a group of servers, a single server being its own group: one unattended session per server, a few at a time, and one report at the end. It belongs to a group of people, to one person, or to everybody; the people of that group change, remove and start it, and everybody's schedule is changed by whoever manages schedules. Whoever sets where it runs must work on those servers; whoever does not see every server sees the schedules that run on a server they see.

Cron expressions in UTC. The task runs unattended: read-only commands execute, and changes too where the schedule is set to "Changes alone" (see Schedules above). Anything else is skipped and listed under "needs an operator" in the report, which goes to the schedule's destinations: mail addresses, and the team's channel where the tenant has a webhook. A mail carries the report as text and as HTML.

Reports and messages

A session sends its report to the destinations set on its page (More, Report and messages). It goes by itself when Heinzel has finished a task and written the summary, and when the session fails; a turn that ends with a question sends nothing. "Send the report now" sends it at any time. The same panel lists what was sent from the session and whether it left.

Under Account, "Reports of my sessions" sets the destinations every session a person starts takes along.

A person can also ask in a message: "and then send me a mail to ops@example.com". Heinzel writes that message with the tool send_message. It reaches only a destination a person named: an address written in a message of the session, the session's report destinations, the people of the session, or the team's channel. An address Heinzel read on a host or in a file is refused, as is every address in a session an alert started. A session sends ten such messages an hour at most, each is scrubbed of what looks like a secret, and each is in the audit log.

A destination is a mail address or a channel. Two channels exist: mail, and the tenant's webhook (Settings, Notifications), which is one place for the whole team. A delivery that fails is tried five times.

Audit log

Filter by action, actor or text; export CSV; verify the chain. Target links jump to the session or server.