Running a hub

What it is

A hub is the server your machines log in to, wdx-hub, which keeps accounts: it knows your nodes and which are online, keeps the docs your projects push, and passes the sessions between your nodes, end to end encrypted (Hub). One person runs it, for themselves or a team, on a server every node can reach. wdx works the same without one.

Operating a hub makes you responsible for its deployment, updates, account access, backups, retained data, and incident response. Decide what documents may be uploaded and tell your users how access, retention, and deletion are handled. See Privacy and data and Security.

How the hub and its nodes work together, and what it keeps and never sees: The hub, in Architecture.

The hub is a program of its own, with its own home, database and settings, released beside wdx with the same version. Its home, WDX_HUB_HOME, holds the admin socket, whose path must fit the system’s limit, 107 bytes on Linux and 103 on macOS: wdx-hub serve in a deeper home fails, naming the path and its length. It need not run the version its nodes run: they agree on the API. A node newer than the hub is welcomed, so nodes may update before the hub; one whose wdx has a feature the hub is too old for refuses it with hub_version_unsupported, asking for the hub to be updated. A node older than the hub serves is refused with node_version_unsupported, and wdx hub status says which side to update.

Install it

On Linux with systemd, amd64 or arm64, as root:

$ curl -fsSL https://raw.githubusercontent.com/srjn45/wdx/main/scripts/install-hub.sh | sudo sh
==> downloading wdx-hub 0.5.0 (linux/amd64)
public url, where browsers and nodes reach this hub (such as https://hub.example.com): https://hub.acme.dev
how does the web endpoint get its TLS?
  1) a proxy in front of it terminates TLS (Caddy, nginx, a load balancer)
  2) certificate files on this machine
choice: 1
==> made the master key
==> the settings in /var/lib/wdx-hub/.wdx-hub/config.yaml list every setting of this version
==> installed the bash completion /usr/share/bash-completion/completions/wdx-hub
==> installed wdx-hub 0.5.0 -> /usr/local/bin/wdx-hub, the service wdx-hub.service
==> wdx-hub answers, version 0.5.0
==> point the proxy at 127.0.0.1:8080, the web endpoint
==> the node endpoint listens on :8443 and must be reachable from the nodes, untouched by the proxy
==> invite the first user with this link, which works once, until 2026-11-10T15:42:07Z:
    https://hub.acme.dev/signup?invite=…
==> back up /var/lib/wdx-hub/.wdx-hub/config.yaml: it holds the master key, without which the hub's secrets are lost
==> run admin commands with sudo: sudo wdx-hub user list
==> Tab completes wdx-hub's commands and flags in a new shell

• What it makes: the system user wdx-hub, with no login shell, its home in /var/lib/wdx-hub that only it can read, and the hub’s own home in it, /var/lib/wdx-hub/.wdx-hub: the config file, the database hub.db, the certificate authority, the admin socket, the logs and the backups. The binary goes to /usr/local/bin/wdx-hub and the service to /etc/systemd/system/wdx-hub.service, started at once and at every boot. For each of bash, zsh and fish the machine has, the completion goes where that shell finds it for every user (below).
• It asks two things, the only settings without a default: the public url, where browsers and nodes reach the hub (http:// only for localhost and 127.0.0.1, which skips the TLS question), and how the web endpoint gets its TLS (below). A value the config file has already is not asked, so the script also installs over a hub set up by hand.
• Unattended, with no terminal, it takes them from the environment and fails naming the one missing: WDX_HUB_PUBLIC_URL, WDX_HUB_WEB_TLS (off for a proxy, or files), and with files WDX_HUB_WEB_TLS_CERT and WDX_HUB_WEB_TLS_KEY. These reach only the script, never the running hub. Pass them through sudo: curl … | sudo WDX_HUB_PUBLIC_URL=https://hub.acme.dev WDX_HUB_WEB_TLS=off sh.
• Which release: the newest stable one; WDX_CHANNEL=dev the newest of either kind, and WDX_VERSION=0.5.0 that one. The channel is kept as the hub’s update.channel, which wdx-hub update follows. Until a stable release has the hub, the stable channel says to use WDX_CHANNEL=dev. WDX_DOWNLOAD_BASE takes the releases from elsewhere, laid out as GitHub’s, and is kept as update.base_url, which wdx-hub update looks at; --apply takes it only from root’s environment (below), as the script run again gives it.
• Run again on a machine with the hub installed, the script does not install over it: it updates it, through the installed wdx-hub update --apply (below).
• Every other setting has a default; sudo wdx-hub help settings lists them, and sudo wdx-hub setting set changes one, having the running hub read it again or saying it takes effect at the next start (sudo systemctl restart wdx-hub).

Not on macOS, and not as a user service: to run it without a server of your own, see On Fly.io; to try a hub on a laptop, run wdx-hub serve in a terminal (deploy/hub says how, and how to run it in a container).

TLS

The hub has two endpoints: the web endpoint, for browsers and the nodes’ logins, and the node endpoint, :8443, where the nodes keep their connection. The node endpoint always serves TLS itself, with a certificate from the hub’s own authority, which the nodes check: a proxy must never terminate it. The web endpoint gets its TLS one of two ways.

A proxy in front of it (choice 1, web.tls off): the hub listens on 127.0.0.1:8080 in plain HTTP, which only the proxy on the same machine reaches, and the proxy serves https://. Caddy gets and renews its certificate from Let’s Encrypt on its own; its whole /etc/caddy/Caddyfile:

hub.acme.dev {
  reverse_proxy 127.0.0.1:8080
}

Then tell the hub to believe the address the proxy forwards, which its rate limits and audit chain use:

$ sudo wdx-hub setting set --name trusted_proxies --value 127.0.0.1

Until it is set, every client counts as the proxy, so one client’s wrong tries make everyone wait. since 0.5.0 With web.tls off and trusted_proxies empty, the hub warns in its log, no proxy trusted at its start and request from a peer not on this machine at the first request from another machine, with that peer’s address.

Certificate files on this machine (choice 2, web.tls files): the hub serves TLS itself on :443, with a certificate (PEM, with its chain) and a key at the absolute paths you give, which the wdx-hub user must be able to read: sudo chgrp wdx-hub key.pem && sudo chmod g+r key.pem. The service then gets the one capability it needs to listen on a port below 1024. Renewing the files is up to you; the hub reads them at its start.

There is no third way: the hub does not get certificates itself. A public hub uses a proxy that does.

Ports to open

Port	What	From
443	the web endpoint: the proxy’s, or the hub’s with certificate files	browsers and nodes
8443	the node endpoint, TLS of the hub’s own, never through the proxy	nodes

With a proxy, 127.0.0.1:8080 stays closed to everyone else. The node endpoint’s port is listen.nodes; when the nodes reach it at another address than the public url’s host with that port, set nodes_url.

Moving the hub to another name

The nodes keep the node endpoint’s address from their login, and the hub tells it them again each time they connect: when it changed, they keep the new one, and stay logged in. To move the hub to another name:

1. Point the new name at the hub, and set nodes_url to it.
2. Set nodes_url_previous to the old address, such as https://old.example.com:8443, comma-separated for several. The endpoint’s certificate then names the old name too, so nodes still dialling it connect and are told the new one.
3. Restart the hub: sudo systemctl restart wdx-hub.
4. Once every node has connected since, as the last seen of each user’s nodes in wdx-hub user get shows, empty nodes_url_previous and restart it again.

A node offline the whole time still has the old address: while the old name points at the hub and is in nodes_url_previous, it follows when it next connects.

Users

The hub’s registration is closed: a person joins with an invite, a link that works once, for 24 hours. The install prints the first; make others with:

$ sudo wdx-hub user invite
invited: the link works once, until 2026-11-10 16:05
https://hub.acme.dev/signup?invite=…

They sign up on that page, set up their second factor, and from then on log their machines in with wdx hub login --url https://hub.acme.dev: every login names its hub with --url, which is required. --expires 72h gives an invite longer; the setting registration, open, lets anyone sign up.

Rate limits

The hub’s pages limit the tries, counted in memory, so a restart forgets them. A refused try is answered 429 with how long to wait. An IPv6 client is counted by its /64, the prefix one machine is given whole, so that a new address of it for every try counts as the same client.

• Per address, at most 20 passwords, codes or tokens a minute, wrong invite and reset links counted, and links to approve a node login that is unknown, and 10 sign-ups. Each try is counted as it is let through, so a burst cannot pass before any is counted.
• Per username and address, after 5 wrong passwords or codes in a row the next try from that address waits 1 second, doubling up to 15 minutes, until one passes. Wrong passwords from one address never lock the user out from another: the second factor, needed after the password, stops whoever guesses from many. The first refusal of a user’s lockout is in the audit chain, as user.locked_out.
• Per username, from any address, 50 wrong tries in a day write user.password_attacked to the audit chain, once; they refuse nothing. Neither row is written for a username no user has.
• Passwords hashed at once: two, each with 19 MiB, and 16 more may wait for them. A try beyond those is refused at once: 429, “too many tries at once, try again in a moment”.
• A username no user can have, one that is not 1 to 39 letters, digits and -, is answered as a wrong password at once, with no hash and no count of its own. The counts are kept for at most 10,000 usernames; when that many are counted, the address limits still hold.

Admin commands

The hub’s commands talk to the running hub over its admin socket, which only root and the wdx-hub user reach. Run them with sudo: as root on a machine with the hub installed, wdx-hub becomes the wdx-hub user before anything else, so that it reads the hub’s home and every file it writes there stays the hub’s.

$ sudo wdx-hub version
$ sudo wdx-hub user list
$ sudo wdx-hub user disable --name bob
$ sudo wdx-hub user reset --name bob --second-factor
$ sudo wdx-hub setting set --name log.level --value debug
$ sudo wdx-hub audit verify
$ journalctl -u wdx-hub

A user is named by --name, their username whatever its case, or by --id, the usr- id that wdx-hub user list --json and user get show: exactly one of them, as with wdx’s commands. --username, the flag’s name in the dev releases before, still works.

Disabling or removing a user, as removing a node, cuts it off at once: the hub closes its control connections and every stream open between its nodes, and the nodes close their sessions.

since 0.5.0 wdx-hub setting set --value - reads the value from stdin, one line, so that a secret is in no command line, which ps shows to every user:

$ head -c 32 /dev/urandom | base64 | sudo wdx-hub setting set --name master_key --value -

Only replace a master key that has sealed nothing yet: the secrets sealed under the old one are lost with it.

sudo -u wdx-hub -H wdx-hub … works too. Each has --help, and --json as wdx’s commands do.

Tab completion

since 0.5.0 In a shell started after the install, Tab completes wdx-hub’s commands and flags, after wdx-hub or sudo wdx-hub, as it does wdx’s (Shell completion). The install wrote the script of each shell it found, for every user: /usr/share/bash-completion/completions/wdx-hub (bash, with the bash-completion package), /usr/local/share/zsh/site-functions/_wdx-hub (zsh) and /usr/share/fish/vendor_completions.d/wdx-hub.fish (fish); wdx-hub update --apply writes them again. wdx-hub completion <shell> prints one, for setting it up by hand.

Help and completion work even where the hub’s settings do not load, as for a user other than the hub’s. Usernames after --name, and ids after --id, come from the running hub, over its admin socket, so they are offered only to a user who reaches it; to others Tab offers nothing there.

Updating

wdx-hub update works as wdx update does. Without a flag it only looks:

$ sudo wdx-hub update
update available: wdx-hub 0.5.0 -> 0.5.1 (stable channel)
  run sudo wdx-hub update --apply

$ sudo wdx-hub update --apply
==> downloading wdx-hub 0.5.1
==> checking that it runs on the settings
==> backing up the database to /var/lib/wdx-hub/.wdx-hub/backups/hub-0.5.0.db
==> stopping the daemon
==> backing up wdx-hub 0.5.0, the database and the settings
==> installing wdx-hub 0.5.1
==> starting the daemon
updated wdx-hub 0.5.0 -> 0.5.1
  schema:   7 -> 8
  rollback: sudo wdx-hub update --rollback goes back to 0.5.0

• --apply needs root, as it replaces the binary and the service: root_required otherwise. It updates only a hub the script installed; one in a container or copied by hand is not_installed, and is updated the way it was installed.
• As root it trusts only what root chose. The config file is the hub’s user’s to write, so --apply refuses an update.base_url set there to anything but GitHub’s: update_source_untrusted, nothing changed. To update from elsewhere, give the address in root’s environment for that run: sudo WDX_HUB_UPDATE_BASE_URL=<url> wdx-hub update --apply. Before root writes anything in the hub’s home, the home, logs/ and backups/ must be directories of the hub’s user’s, not symlinks: home_unsafe otherwise, naming the path. What root writes there, its log included, is left the hub’s user’s. A release whose unit does not start /usr/local/bin/wdx-hub as the user wdx-hub is not started, and the old one is put back (update_failed).
• Before anything changes, the new release runs on the settings and writes the config file anew in a staging home: one that refuses them is settings_invalid, and nothing changed.
• It keeps the old binary, a copy of the database and the config file in /usr/local/lib/wdx-hub/previous/, then stops the service, installs the new binary and the unit it prints, and starts it.
• If the new release does not answer within 30 seconds, everything is put back and the old one started: update_failed, naming the step, and journalctl -u wdx-hub says why.
• Restarting closes every node’s connection and every session’s stream through the hub. The nodes connect again on their own, waiting longer each time up to 1 minute; an open attach or remote TIDE sees the connection lost, as for a network drop.
• --channel dev or --version 0.5.1 choose another release this once.
• The shells’ completions are written again from the new binary, even when it is up to date; one it cannot write is a warning, and the update stands.

Changed your mind:

$ sudo wdx-hub update --rollback

It puts the kept version back, with the config file as it was. The database stays when that version can use it. When a migration since the update changed it in a way the older release cannot read, the copy taken at the update must be restored, losing every signup, login and doc pushed since: it asks first on a terminal, and from a script --discard-changes agrees and --cancel refuses (restore_required). Only one previous version is kept.

Backups and the master key

• The master key, in the config file, seals the secrets the hub keeps: its users’ authenticator secrets and the keys of their docs. The install made it and never printed it. Without it they are lost for good, with any backup of the database. Keep a copy of /var/lib/wdx-hub/.wdx-hub/config.yaml somewhere safe, apart from the database’s backups.
• A hub installed by a release before 0.5.0 has its master key in the log of the day it was installed, /var/lib/wdx-hub/.wdx-hub/logs/wdx-hub-<date>.log, as has any day a key was set with wdx-hub setting set. Delete those files, sudo rm /var/lib/wdx-hub/.wdx-hub/logs/wdx-hub-<date>.log, since the key cannot be changed; the hub deletes them anyway once they are older than log.retention_days, 7 days by default. Since 0.5.0, setting set logs only the length of the value it is given, and the install hands the key over on stdin, so that it is in no command line either.
• Each update backs the database up to /var/lib/wdx-hub/.wdx-hub/backups/hub-<version>.db, and keeps the last 3.
• Any time, back it up while the hub runs, to a path the wdx-hub user can write:

$ sudo wdx-hub backup --out /var/lib/wdx-hub/.wdx-hub/backups/nightly.db

It writes the database’s copy to that path and the certificate authority the nodes trust, <home>/ca/, to the directory beside it with .ca added, here backups/nightly.db.ca/; neither may be there already, and a backup that fails leaves neither. That directory holds the certificate authority’s private key: keep the backup as safe as the master key, which it still does not hold. To restore it, with the hub stopped, put the copy at <home>/hub.db and its .ca directory at <home>/ca/, both owned by wdx-hub, and the master key in the config file as before. A hub whose database has nodes refuses to start without its certificate authority (ca_missing), rather than make a new one that would log every node out. On Fly.io the volume’s snapshots hold ca/ with the database already.

To move the hub to another machine, install it there, stop both (sudo systemctl stop wdx-hub), copy the old home, /var/lib/wdx-hub/.wdx-hub, over the new one, owned by wdx-hub, and start it: the database, the certificate authority the nodes trust and the master key go together.

Uninstalling

$ curl -fsSL https://raw.githubusercontent.com/srjn45/wdx/main/scripts/install-hub.sh | sudo sh -s -- --uninstall

It stops and removes the service, the binary, the shells’ completions and /usr/local/lib/wdx-hub/. The wdx-hub user and its home stay, with the database, the certificate authority, the master key and the backups, so that installing again goes on where it was. --purge removes them too, after asking on the terminal, since every account and doc of the hub goes with them; --purge --yes does it without asking.

On Fly.io

since 0.5.0 Fly.io runs the hub’s container for you, from the repository’s deploy/hub/fly.toml and Dockerfile: no server to keep, no proxy to set up, TLS for https://<name>.fly.dev by Fly. Choose it when the hub is for yourself or a few people and you would rather not look after a machine; choose a server of your own, with the install script above, when you want the hub on hardware or a network you control, a domain of your own without Fly in front, or wdx-hub update and its rollback.

What it runs: one machine, shared-cpu-1x with 256 MB, always on, since the nodes keep a connection open that a stopped machine would drop; its web endpoint behind Fly’s proxy, which terminates TLS on 443, and its node endpoint on 8443 as raw TCP, untouched, since the hub terminates the nodes’ TLS itself; the hub’s home on a 1 GB volume at /data, which Fly snapshots daily.

The first deploy

You need flyctl, a Fly account and a checkout of the repository. Choose the app’s name and region once: the name makes the hub’s address, https://<name>.fly.dev, and changing it later is a new hub to every passkey and every node’s login; the region is the city closest to your machines, since every key typed into an attached terminal goes there and back.

1. Log in and create the app:

   $ fly auth login
   $ fly apps create <name>

2. Give the app its addresses. The node endpoint is raw TCP, which only a dedicated IPv4 carries; IPv6 alone is free, but many offices, cafés and hotels have none. The IPv4 costs about $2 a month, and flyctl asks before allocating it:

   $ fly ips allocate-v4 --app <name>
   $ fly ips allocate-v6 --app <name>

3. Make the master key, save it in your password manager first, then set it with the public url as the app’s secrets:

   $ head -c 32 /dev/urandom | base64
   $ fly secrets set --app <name> --stage WDX_HUB_MASTER_KEY=<the key> WDX_HUB_PUBLIC_URL=https://<name>.fly.dev

   A Fly secret cannot be read back. Without the master key, a restored volume’s docs and authenticator secrets cannot be read.

4. From a clean checkout of a release tag, deploy to your region (fly platform regions lists them):

   $ git checkout v0.5.0
   $ make hub-fly-deploy APP=<name> REGION=<region>

   The first deploy creates the hub’s machine there, and its volume beside it, the one fly.toml mounts. Later deploys need no REGION: the machine stays where it is.

5. Invite yourself, as the image’s user, nonroot, which reaches the hub’s admin socket:

   $ fly ssh console --app <name> --user nonroot -C "wdx-hub user invite"

   Open the link it prints, sign up and set up your authenticator.

6. On each machine, log in:

   $ wdx hub login --url https://<name>.fly.dev

Then tell the hub to believe Fly’s proxy. Every request comes to it from the proxy, so until then the page that approves a login shows the proxy’s address under From, not yours, the rate limits count every client as one, and the audit chain sees the same; the hub’s log warns of it. Two addresses are needed: the range Fly’s proxy connects from, 172.16.0.0/12, and the app’s dedicated IPv4, which fly ips list --app <name> shows, since the X-Forwarded-For Fly sends ends with the address the client reached. With the range alone, every client is that IPv4. Set both, once:

$ fly ips list --app <name>
$ fly secrets set --app <name> WDX_HUB_TRUSTED_PROXIES=172.16.0.0/12,<the app's dedicated IPv4>

Setting a secret restarts the machine; the next approval page shows your own address. The hub’s log says, for each request on the web endpoint, the client’s address, as the hub believes it, and the connection’s peer when it differs: the address should be yours, not the IPv4. Registration stays closed, as on any hub: others join by the invite you make for them the same way, and every other admin command runs through fly ssh console --app <name> --user nonroot -C "wdx-hub …". The hub logs to Fly: fly logs --app <name>. Nothing completes on Tab there: the image has no shell, so each command runs on its own.

What make hub-fly-deploy does

make hub-fly-deploy APP=<name>, scripts/hub-fly-deploy.sh, refuses before anything reaches Fly unless flyctl is on the PATH, APP is set, the tree is clean and HEAD is exactly a release tag: a hub always runs a release’s version. Then:

• When the app has no machine yet, it refuses unless REGION is set, before anything changes on Fly, and deploys the machine and its volume there.
• When the app has a machine, it snapshots the volume and prints the snapshot’s id: note it, since it is what a rollback across a migration needs. A REGION given then is ignored, and it says so: the machine stays where it is.
• It deploys the tag with fly deploy, the image built by Fly’s builder from deploy/hub/Dockerfile with the tag’s version and commit, on one machine.
• It waits until https://<name>.fly.dev/v1/version answers the tag’s version, at most 5 minutes, and fails with its last answer otherwise.

Updating

wdx-hub update does not apply on Fly: there is no installed binary for it to replace, so it answers not_installed. A hub on Fly is updated by deploying the new tag, by hand:

$ git checkout v0.5.1
$ make hub-fly-deploy APP=<name>

The restart drops every open session, attach and remote TIDE through the hub, as a network drop would; terminals keep running on their hosts, and the nodes connect again on their own.

Rolling back

Each deploy is a release of the app with its image. List them, and deploy the previous one again:

$ fly releases --app <name> --image
$ fly deploy --app <name> --config deploy/hub/fly.toml --image <the previous image>

That works while the older release can use the database. When the update ran a migration the older release cannot read, it refuses to start, and fly logs --app <name> says database schema version … needs a binary that knows schema version …. Then go back to the snapshot the deploy took: create a volume from it, make it the one the machine runs on, and deploy the older image on it. Everything since the deploy is lost: signups, logins and the docs pushed, which the nodes push again.

$ fly volumes create wdx_hub_data --app <name> --region <region> --size 1 --snapshot-id <the snapshot's id>

So before each update, the two things to note are the image fly releases --image lists for the running release, and the snapshot’s id the deploy prints.

Cost

About $4.50 a month: the shared-cpu-1x machine with 256 MB, always on, about $2; the dedicated IPv4, $2; the 1 GB volume and its snapshots, a few cents; and a little outbound data, the sessions between your nodes.