SETUP_NOTES
===========
Plan 9 cpu/auth/fs server setup notes.
This file documents the actual procedure used to bring up cube as
the cpu/auth/fs server for the baddcafe authentication domain.
It is written in the spirit of the 9grid howto cited below, with
the deviations called out at each step.
Names used in this file:
cpu/auth/fs server: cube
terminal: gnot
Plan 9 authdom: baddcafe
public DNS name: baddcafe.com
bootstrap user: glenda
host owner: bootes
operator user: mospak (day-to-day account; bootes is the
system identity used by auth services)
kernel: i386
install disk: sdE0
Sections:
SETUP STEPS 1-17 install + auth foundation (required)
SETUP STEPS 18-22 web - plain HTTP (optional)
SETUP STEPS 23 cube as authoritative DNS (optional)
SETUP STEPS 24-30 TLS via Let's Encrypt (optional)
FINAL CHECKLIST per-section verification matrix
ROUTER CONFIG MikroTik gateway example
TESTED HARDWARE one known-working configuration
SOURCE NOTES
------------
This file follows the procedure of:
http://mirror.9grid.fr/mirror.9grid.fr/plan9-cpu-auth-server-howto.html
In-tree references used in this setup:
/sys/man 8/auth, 8/changeuser, 8/secstore, 6/ndb,
6/users, 4/fossilcons
/sys/doc Bell Labs papers
/sys/src live source for kernel rebuild and rc scripts
BELL LABS CONVENTION
--------------------
Per-host changes go in /cfg/<sysname>/, not in /rc/bin/.
/rc/bin/cpurc shipped, generic, do NOT edit
/rc/bin/termrc shipped, generic, do NOT edit
/cfg/cube/cpurc cube's cpu-mode startup overrides
/cfg/cube/termrc cube's terminal-mode overrides
/cfg/gnot/termrc gnot's terminal startup overrides
The shipped /rc/bin/cpurc sources /cfg/$sysname/cpurc when it
exists. Anything turned on at boot lives in /cfg/cube/cpurc.
In particular: the keyfs, cron, and secstored lines that the
9grid howto Step 6 puts in /rc/bin/cpurc go in /cfg/cube/cpurc
in this setup. The shipped tree stays pristine.
If asked "do I edit /rc/bin/X or /cfg/cube/X?", the answer is
always /cfg/cube/X.
MENTAL MODEL
------------
Three identity systems, three databases, three "who am I"s:
file-server users /adm/users (managed via fossil console)
Controls file ownership and groups.
Does not create any login password.
con -l /srv/fscons
uname bootes bootes
uname sys +bootes
uname adm +bootes
auth users /adm/keys (managed via auth/changeuser)
Controls login for cpu, drawterm, and
the auth protocol on tcp567.
auth/changeuser -p bootes
NVRAM identity /dev/sdE0/nvram (set via auth/wrkey
or the kernel boot-time prompt)
Tells the kernel who the host owner
is when cube boots in cpu mode.
auth/wrkey
A given identity (e.g. bootes) usually exists in all three.
Three secret stores, three lifetimes:
factotum in-memory authentication agent. Mounted at
/mnt/factotum. Loaded keys are lost at reboot
or factotum restart. Control file is
/mnt/factotum/ctl.
Load a key for the current session:
echo 'key proto=p9sk1 dom=baddcafe user=mospak' \
>/tmp/key
echo ' !password=PASSWORD' >>/tmp/key
cat /tmp/key >/mnt/factotum/ctl
rm /tmp/key
secstore persistent encrypted store served by
auth/secstored. Survives reboot. Per-user.
bootes secstore is not mospak secstore. The
conventional file inside secstore is named
exactly "factotum".
Store a local file:
auth/secstore -p factotum
Fetch and load into live factotum:
auth/secstore -G factotum | read -m \
>/mnt/factotum/ctl
keyfs host-side decryption of /adm/keys. Mounted at
/mnt/keys. Started at boot from cpurc.
Captures the NVRAM machine key once at startup;
reboot required to pick up a new machine key.
Three passwords per host owner. This setup uses three distinct
passwords for defense in depth. The KISS option of one shared
password works equally well functionally.
1. login password set via auth/changeuser; protects
bootes identity at the auth protocol
2. machine key set via auth/convkeys (encrypts
/adm/keys) and auth/wrkey (writes
the NVRAM copy); the two must match
byte-for-byte
3. secstore password set via auth/wrkey; identifies bootes
to auth/secstored
Two NDB files, two purposes:
/lib/ndb/local network names, addresses, and service
attributes (authdom, auth, cpu, fs,
dns, ipgw — typically inheriting from
an ipnet stanza)
/lib/ndb/auth host-owner rule:
hostid=bootes
uid=!sys uid=!adm uid=*
Means: bootes is the host owner, may
authenticate any user except sys and
adm.
Two listener layouts, both canon:
aux/listen runs services as none. Used for
non-auth services.
aux/listen -t \ searches a service.auth/ directory
/rc/bin/ and runs the auth services as the
service.auth host owner. This is what serves
tcp567 (the Plan 9 auth protocol).
This setup uses the standard /rc/bin/service.auth/tcp567 layout.
Two reboot paths, two semantics:
fshalt -r warm reboot via #c/reboot. Bypasses
9load and plan9.ini. Same kernel
continues running. OK for same-arch
reboots.
cold reboot power cycle. Reads plan9.ini, follows
the menu. Required after plan9.ini
edits and after switching kernel
architecture.
SAFETY RULES
------------
Always fshalt before power-off. Never hard-reset a running
Plan 9 system. fossil and venti both prefer a clean shutdown.
Change one layer at a time:
install
network
auth service
bootes login
secstore
Do not start the same service twice. Common mistakes are two
aux/listen commands, two auth/keyfs commands, or two httpd
commands.
SETUP STEPS
-----------
1. Install Plan 9.
Boot the 9legacy install ISO. Use a single PLAN9 fdisk
partition with sub-partitions for 9fat, nvram, swap, fossil,
isect, arenas, bloom. Choose fossil+venti at the
configfs prompt (not fossil-only — venti is needed for
snapshot/backup later).
bootsetup installs the terminal kernel (9pcf) and writes a
plan9.ini that boots terminal mode by default. Override
this at the Enable boot method: prompt to add a [cpu] menu
entry.
2. First boot in terminal mode as glenda.
At the boot prompt:
root is from (tcp, local)[local!#/sdE0/fossil]: <ENTER>
user[none]: glenda
3. Add glenda to adm temporarily and fix /tmp permissions.
The first command lets glenda write /adm/timezone/local in
step 4. The second fixes /tmp from shipped 0755 (which blocks
most users from writing) to the canonical 0777 - needed by
ape/patch and other POSIX-style ports.
con -l /srv/fscons
prompt: uname adm +glenda
prompt: fsys main
main: wstat /active/tmp - - - 777 -
Ctrl-\
>>> q
Note: glenda's adm membership is TEMPORARY for bootstrap.
After the host owner (bootes) is established and verified
(around steps 14-16), demote glenda from privileged groups —
see step 41.
4. Set the timezone.
lc /adm/timezone
cp /adm/timezone/<your-zone> /adm/timezone/local
5. Prepare the per-host config directory.
mv /cfg/example /cfg/cube
6. Edit /cfg/cube/cpurc.
Per-host cpu-mode startup script. The shipped /rc/bin/cpurc
sources this file and stays unchanged.
#!/bin/rc
dmaon
ip/ipconfig
ndb/dns -r
auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
auth/cron >>/sys/log/cron >[2=1] &
auth/secstored
aux/listen -q -t /rc/bin/service.auth \
-d /rc/bin/service tcp
sleep 3
The keyfs, cron, and secstored lines correspond to the lines
the 9grid howto Step 6 says to uncomment in /rc/bin/cpurc.
Putting them in /cfg/cube/cpurc instead leaves the shipped
tree untouched.
7. Enable the auth service stub.
mv /rc/bin/service.auth/authsrv.tcp567 \
/rc/bin/service.auth/tcp567
One-shot rename, not part of cpurc.
8. Edit /lib/ndb/local.
Read cube's MAC address:
cat /net/ether0/addr; echo
Append an ipnet stanza so attributes inherit to all hosts on
the LAN, plus per-host tuples:
ipnet=baddcafe ip=192.168.88.0 ipmask=255.255.255.0
authdom=baddcafe
auth=cube
cpu=cube
fs=cube
dns=192.168.88.1
dnsdomain=baddcafe.com
ipgw=192.168.88.1
ip=192.168.88.11 sys=cube ether=<cube-mac> \
dom=cube.baddcafe.com
ip=192.168.88.12 sys=gnot ether=<gnot-mac> \
dom=gnot.baddcafe.com
ip=192.168.88.11 dom=baddcafe.com
ip=192.168.88.11 dom=www.baddcafe.com
FQDN form (`baddcafe.com`, `cube.baddcafe.com`) is canonical
per `man/6/ndb` definition of `dom=` as Internet FQDN.
`authdom=baddcafe` (no .com) is separate — that's the Plan 9
authentication domain identifier, not a DNS name.
Verify:
ndb/query sys cube
ndb/ipquery sys cube authdom auth cpu fs ipgw dns
ndb/query returns the matching tuple's attributes.
ndb/ipquery follows ipnet inheritance — what ndb/cs actually
returns to clients.
9. Build and install the cpu kernel.
cd /sys/src/9/pc
mk 'CONF=pccpuf'
9fat:
cp 9pccpuf /n/9fat/
bootsetup never installs a cpu kernel. This step is
load-bearing.
10. Edit /n/9fat/plan9.ini.
Replace the flat bootfile= line with a menu:
[menu]
menuitem=cpu, Plan 9 CPU/Auth Server
menuitem=terminal, Plan 9 Terminal
menudefault=cpu, 10
[cpu]
bootfile=sdE0!9fat!9pccpuf
[terminal]
bootfile=sdE0!9fat!9pcf
[common]
nobootprompt=local!#S/sdE0/fossil
bootargs=local!#S/sdE0/fossil
bootdisk=local!#S/sdE0/fossil
venti=#S/sdE0/arenas
console=0
dmamode=on
mouseport=ps2
monitor=vesa
vgasize=1024x768x16
user=glenda
`user=glenda` auto-logs into terminal mode without a
username prompt. Comment it out (`# user=glenda`) once
bootes is fully working if you prefer the username prompt
at boot.
11. Create bootes as a file-server user.
On cube, still as glenda:
con -l /srv/fscons
prompt: uname bootes bootes
prompt: uname sys +bootes
prompt: uname adm +bootes
prompt: uname upas +bootes
prompt: fsys main
main: create /active/cron/bootes bootes bootes d775
main: create /active/sys/log/cron sys sys a666
Ctrl-\
>>> q
fsys main MUST come before any create.
If /active/sys/log/cron already exists:
wstat /active/sys/log/cron - sys sys a666 -
12. Add the host-owner rule to /lib/ndb/auth.
Append:
hostid=bootes
uid=!sys uid=!adm uid=*
Verify (this file is not in cs's default DB; query it
explicitly):
ndb/query -f /lib/ndb/auth hostid bootes
13. Reboot into cpu mode for the first time.
fshalt
# cold reboot (power-cycle), then select the cpu menu
# entry at the boot menu
NVRAM is empty on this first cpu boot. The kernel prompts:
authid: bootes
authdom: baddcafe
secstore key: SECSTORE_KEY
password: MACHINE_KEY
The password becomes both the NVRAM machine key and the
/adm/keys encryption key for cpurc's auth/keyfs. Pick any
non-empty value; it is rotated to a permanent value in step
15.
After the kernel prompt completes, cube comes up in cpu
mode running auth services as bootes via the NVRAM
identity. cpurc has started auth/keyfs, auth/cron,
auth/secstored, and aux/listen.
14. Create bootes as an auth user.
Steps 14–16 run at cube's local console (rc shell on cube's
attached monitor + keyboard, running as bootes after step
13).
auth/changeuser -p bootes
Set the bootes login password. This is the password
drawterm and cpu(1) will prompt for; it is independent of
the NVRAM machine key set in step 13.
15. Re-key /adm/keys and write NVRAM (terminal-mode boot).
/adm/keys was created in step 14 with the temporary machine
key entered at the boot prompt in step 13. Rotate it to a
permanent password.
The safe form runs convkeys + wrkey in terminal mode where
no keyfs is alive. Doing this in cpu mode with keyfs
running risks a race documented in auth(8) BUGS: any
keyfs-write between convkeys and reboot re-encrypts
/adm/keys with the OLD machkey from keyfs's stale memory
snapshot, leaving NVRAM and disk out of sync at next boot.
1. Clean shutdown:
fshalt
2. Cold reboot (power-cycle), select terminal at the boot
menu. Terminal mode does not start auth services; no
keyfs to race.
3. At the terminal-mode rc prompt:
auth/convkeys /adm/keys
# New password (×2): MACHINE_KEY
auth/wrkey
# authid: bootes
# authdom: baddcafe
# auth password: MACHINE_KEY (must equal convkeys's)
# secstore password: SECSTORE_KEY (independent;
# distinct value for defense in depth)
4. Reboot back to cpu mode:
fshalt -r
Warm reboot is fine after this — no kernel arch change,
NVRAM holds the new key, /adm/keys is freshly re-encrypted
with the matching key. cpurc's auth/keyfs reads the new
NVRAM machkey at startup and decrypts /adm/keys cleanly.
16. Verify cube is running as bootes.
cat /dev/user # expect: bootes
cat /env/service # expect: cpu
ps | grep keyfs # bootes ... keyfs
ps | grep secstored # bootes ... secstored
ps | grep cron # bootes ... cron
From a remote host:
nc -z -v 192.168.88.11 567 # auth listener up
17. Provision bootes secstore.
The boot-time TLS key load (added later, when TLS is set up)
reads bootes's secstore via `auth/secstore -n -G factotum`,
which needs both:
- NVRAM secstore key (set in step 13's kernel prompt)
- on-disk verifier at /adm/secstore/who/bootes
Step 13 set the NVRAM half. This step writes the on-disk
verifier with the same password.
auth/secuser bootes
# bootes password: SECSTORE_KEY (same as step 13's
# secstore key)
# retype password: SECSTORE_KEY
# expires [...]: <DDMMYYYY> or <Enter> for +365 days
# Enabled or Disabled: <Enter> (Enabled)
# require STA?: <Enter> (no)
# comments: <Enter> (or any note)
auth/secuser auto-creates /adm/secstore (0755), /who (0755),
/store (0700). Tighten /adm/secstore to canonical 0770 per
man/8/secstore via fossil console:
con -l /srv/fscons
prompt: fsys main
main: wstat /active/adm/secstore - - - 770 -
Ctrl-\
>>> q
Verify:
ls -ld /adm/secstore # d-rwxrwx---
cat /adm/secstore/who/bootes # exp + PAK-Hi lines
18. (Optional) Create web user/group + /usr/web.
Plan 9 ships ip/httpd/httpd as the canonical web server.
It serves /usr/web by default. This step creates a `web`
group for content-admin collaboration and the /usr/web
directory; httpd's runtime identity is independent (see
step 22).
Fossil console:
con -l /srv/fscons
prompt: uname web web
# if not at main: prompt: fsys main
main: create /active/usr/web bootes web d775
Ctrl-\
>>> q
Result: drwxrwxr-x bootes web /usr/web. bootes writes via
the owner bit; users added to group `web` (e.g. mospak via
`uname web +mospak`) write via the group bit.
If /usr/web already exists from an earlier pass, the create
will return `file already exists`. Use wstat to set just
the attributes you need to change — fields after the path
are `name uid gid mode mtime`; `-` means "don't change".
Example for fixing only the owner (when /usr/web exists
with a uid other than bootes):
main: wstat /active/usr/web - bootes - - -
19. Minimize /lib/namespace.httpd.
The shipped /lib/namespace.httpd is full of mounts specific
to the original Bell Labs site (alice, netlib, etc.) that
will fail or do nothing on a fresh install. httpd auto-
binds /usr/web onto / regardless of namespace.httpd
contents, so empty is the minimum that works.
cp /lib/namespace.httpd /lib/namespace.httpd.orig
> /lib/namespace.httpd
20. Place initial content.
echo '<html><body><h1>your-site</h1></body></html>' \
> /usr/web/index.html
Default mode 0664 (rw-rw-r--) is world-readable, which
httpd-as-none requires (see step 22).
21. Start httpd by hand and test.
httpd self-backgrounds (rfork + parent exits) so the prompt
returns immediately:
ip/httpd/httpd
ps | grep httpd # none ... httpd
From another LAN host:
curl http://<cube-ip>/
# expect: HTTP/1.1 200 OK with your index content
22. Persist httpd for boot.
/rc/bin/cpurc sources /cfg/<sysname>/cpustart via `.` (no
exec bit needed):
echo 'ip/httpd/httpd' > /cfg/cube/cpustart
Note on httpd's runtime identity: regardless of who starts
httpd, it switches to user `none` immediately (httpd.c
becomenone() writes "none" to #c/user, rebuilds the
namespace, overlays /lib/namespace.httpd). Files in
/usr/web therefore need world-readable mode bits for httpd
to serve them. File ownership matters only for who EDITS
content, not for httpd's serving role.
23. (Optional) Run cube as authoritative DNS for baddcafe.com.
By default the canonical Plan 9 setup uses the registrar's
nameservers and only resolves DNS locally. If you want
cube to be the authoritative nameserver for your public
domain (so DNS records live declaratively on cube alongside
the rest of your config), follow the steps below.
a. Edit /lib/ndb/local — add three blocks:
Root-server hints (for cube's recursive resolver; verify
current IPs from IANA before pasting):
dom=
ns=A.ROOT-SERVERS.NET
... (M.ROOT-SERVERS.NET)
dom=A.ROOT-SERVERS.NET ip=198.41.0.4
... (12 more)
SOA for baddcafe.com:
dom=baddcafe.com soa=
refresh=3600 ttl=3600
ns=ns1.baddcafe.com
ns=ns2.baddcafe.com
mb=postmaster@baddcafe.com
Public-facing A records (use your public IP, not LAN IP):
ip=<public-ip> dom=baddcafe.com
ip=<public-ip> dom=www.baddcafe.com
ip=<public-ip> dom=ns1.baddcafe.com
ip=<public-ip> dom=ns2.baddcafe.com
Update the ipnet stanza so LAN clients use cube as DNS:
dns=<cube-lan-ip>
b. Edit /cfg/cube/cpurc — change `ndb/dns -r` to
`ndb/dns -s`. This enables the UDP/53 listener; cube
walks the root hints itself for non-authoritative
queries. TCP/53 is served by the shipped
/rc/bin/service/tcp53 stub via the existing aux/listen.
c. fshalt -r and verify after reboot:
dig @<cube-lan-ip> baddcafe.com SOA +short
dig @<cube-lan-ip> google.com A +short # tests recursion
d. Add NAT rules at your gateway router for WAN UDP/53 +
TCP/53 → cube (see Router config appendix).
e. At your registrar, register glue records for ns1 and ns2
(subdomain nameserver hostnames + IP), then change the
NS records to point to those nameservers.
f. Wait for propagation (1-60 minutes typical, up to 48h
worst case for resolvers stuck on old NS). Verify
externally with a public probe (check-host.net or
similar) since macOS-from-inside-the-LAN dig may be
intercepted by your ISP's transparent DNS proxy.
Caveat: under this setup, cube becomes a public-facing
nameserver. By default ndb/dns also recurses for incoming
queries (open recursive resolver - DDoS amplification
hazard, modest for a small zone). Hardening options
discussed in man/8/ndb (-R flag) and in your gateway
firewall (drop WAN-side UDP/53 with RD bit set).
Caveat 2: vanilla Plan 9 ndb/dns returns NOTIMP for
unsupported record types (CAA, TLSA, SVCB, etc.) - non-RFC
behavior per RFC 8659 sec 6.2. Modern public CAs require CAA
queries to return NOERROR and refuse to issue certificates if
they get NOTIMP back. Practical effect: with vanilla ndb/dns
serving your zone authoritatively, you cannot obtain a cert
from any public CA.
Resolved locally: small patch to `dnserver.c` returning
NOERROR + SOA for in-area queries on unsupported RR types
(RFC 1035 + RFC 3597 + RFC 2308 compliant). Roughly 20-line
diff against `/sys/src/cmd/ndb/dnserver.c` at the
`!rrsupported(...)` arm; expected to land in 9legacy
stable. Apply this patch (build + `mk install` in
`/sys/src/cmd/ndb`, then `fshalt -r` to restart ndb/dns
cleanly) before attempting step 24 (TLS). Verify with
`dig CAA baddcafe.com @<cube-lan-ip>` returning NOERROR +
SOA in the authority section.
24. (Optional) TLS for baddcafe.com via Let's Encrypt.
Plan 9 4E ships libsec with TLS 1.2 RSA-CBC. This step
obtains a publicly-trusted cert from Let's Encrypt and
deploys it so cube serves HTTPS on port 443.
Prerequisites:
- Step 23 done (cube is authoritative DNS).
- dnserver.c CAA patch applied (caveat 2 in step 23).
- bootes secstore provisioned (step 17).
- Gateway router forwards WAN tcp/80 + tcp/443 to cube.
- Domain registrar has glue records for ns1/ns2 of
baddcafe.com pointing at your public IP.
Hybrid challenge: HTTP-01 for the apex (one TXT-at-a-time
sidesteps a multi-TXT response bug in `dn.c` rrattach1
that deduplicates two TXT records with the same name);
DNS-01 for the wildcard SAN (mandatory per ACME spec for
`*.baddcafe.com`).
On macOS, dry-run via Let's Encrypt staging first (prod
has hard rate limits):
sudo certbot certonly \
--staging \
--manual \
--preferred-challenges http,dns \
--cert-name baddcafe.com-staging \
--domains 'baddcafe.com,*.baddcafe.com' \
--key-type rsa --rsa-key-size 2048 \
-m <email> --agree-tos
Certbot pauses for each challenge. For DNS-01: append a
`dom=_acme-challenge.baddcafe.com / txtrr=<value>` stanza
to cube's `/lib/ndb/local`, then `echo refresh
>/net/dns` to make ndb/dns pick up the change without a
process restart. For HTTP-01: write the token at
`/usr/web/.well-known/acme-challenge/<token>` on cube
(httpd serves dot-prefix paths fine). Verify propagation
externally before pressing Enter — macOS dig/curl from
inside the LAN may miss hairpin-NAT issues; use a public
probe (check-host.net or similar) for ground truth.
After staging succeeds end-to-end, run the same command
minus `--staging` and with `--cert-name baddcafe.com`
(distinct cert-name avoids clobbering the staging tree;
prod and staging share the same archive layout). Then
`sudo certbot delete --cert-name baddcafe.com-staging` to
drop the staging tree.
25. Stage cert files for transfer to cube.
LE provides leaf (cert.pem) and intermediate (chain.pem)
as separate files in `/etc/letsencrypt/live/baddcafe.com/`
— Plan 9 wants them separate (no concatenation needed).
Convert the private key from PKCS#8 (OpenSSL 3.x default)
to PKCS#1 RSAPrivateKey (auth/asn12rsa input format):
mkdir -p ~/cert
sudo cp /etc/letsencrypt/live/baddcafe.com/cert.pem \
~/cert/cert.pem
sudo cp /etc/letsencrypt/live/baddcafe.com/chain.pem \
~/cert/chain.pem
sudo openssl rsa \
-in /etc/letsencrypt/live/baddcafe.com/privkey.pem \
-outform DER -traditional \
-out ~/cert/key.der
sudo chown $USER:staff ~/cert/{cert,chain}.pem \
~/cert/key.der
chmod 600 ~/cert/key.der
`-traditional` is load-bearing on OpenSSL 3.x — the
default encoding is PKCS#8 which `auth/asn12rsa` won't
parse.
Transfer ~/cert/{cert.pem, chain.pem, key.der} to cube by
whatever path you have available — USB stick, drawterm
paste, 9p import (`9p write` from plan9port), or a u9fs
bridge. The cube-side procedure below assumes the three
files land at /tmp/<name> on cube; adjust paths if you
stage them elsewhere.
26. Convert key + load secstore (cube, as bootes).
Stage the asn12rsa output in `ramfs -p`, NOT plain /tmp.
Plan 9's /tmp is fossil-backed; plaintext private keys
written there survive reboots and may be captured by
fossil snaps. ramfs(4)'s `-p` flag makes the memory
private (noswap + /proc-private) so the key never touches
disk and dies with the process.
mkdir -p /tmp/secret
ramfs -p -m /tmp/secret
cd /tmp/secret
auth/asn12rsa -t 'service=tls role=client owner=*' \
/tmp/key.der > key
auth/secstore -g factotum
cat key >> factotum
auth/secstore -p factotum
cd /
unmount /tmp/secret
rm /tmp/secret # empty mountpoint on fossil
rm /tmp/key.der # transferred-in key
`role=client` is load-bearing per the server-side lookup
in libsec (`tlshand.c` looks up the key by literal
`proto=rsa service=tls role=client`). Do NOT add
`proto=rsa` via `-t` — asn12rsa emits it itself.
`unmount` frees the only plaintext copy of the key in
memory; `rm /tmp/key.der` deletes the transferred-in DER
file. After this, the key exists only inside bootes's
secstore vault (encrypted at rest).
27. Place cert files at /sys/lib/ssl/.
mkdir -p /sys/lib/ssl
cp /tmp/cert.pem /sys/lib/ssl/cert.pem
cp /tmp/chain.pem /sys/lib/ssl/chain.pem
chmod 600 /sys/lib/ssl/cert.pem /sys/lib/ssl/chain.pem
rm /tmp/cert.pem /tmp/chain.pem
Mode 0600 bootes:sys is correct: httpd reads cert files
in main() before becomenone() drops privs, so post-drop
`none` doesn't need read access. /sys/lib/ssl/ is the
canonical location per `man/2/pushtls` FILES.
28. Boot-time factotum load — edit /cfg/cube/cpurc.
Find the `auth/secstored` line and add the following
immediately after it:
auth/secstore -n -G factotum | read -m \
>>/mnt/factotum/ctl
Note `>>` (append) — not `>` (truncate, breaks factotum
ctl semantics). `-n` = no prompt (uses NVRAM password
from step 13). `-G` = get-and-print to stdout. `read -m`
reads to EOF and emits one write per line — required, as
factotum rejects a ctl write of more than one line.
Result at next boot: secstored starts, this line fetches
the encrypted factotum blob from the vault, decrypts via
NVRAM key, pipes the result to factotum's ctl device.
factotum now holds the TLS key with no operator
interaction.
29. httpd TLS flags — edit /cfg/cube/cpustart.
Replace the single line from step 22:
ip/httpd/httpd -c /sys/lib/ssl/cert.pem \
-C /sys/lib/ssl/chain.pem
`-c` flips httpd from HTTP to HTTPS (the listener binds
443 instead of 80 when a cert is present); also supplies
the leaf cert. `-C` supplies the intermediate chain so
clients can build a trust path to the root CA. Plan 9
splits leaf and chain where nginx/apache concatenate.
30. Reboot and verify.
fshalt -r
Same-arch warm reboot. Cube returns in 30-60 s. After
boot:
cat /mnt/factotum/ctl | grep 'service=tls role=client'
# expect: key proto=rsa service=tls role=client ...
ps | grep httpd # expect: none ... httpd
From a compatible-client host (macOS curl with LibreSSL/
SecureTransport, browsers using platform TLS, Plan 9
native libsec, older OpenSSL <3.0):
curl -v --resolve baddcafe.com:443:<cube-lan-ip> \
https://baddcafe.com/
Expect: TLSv1.2 / cipher AES256-SHA, issuer Let's
Encrypt, verify ok, HTTP 200.
External port-443 reachability (genuine external probe):
curl -sS \
'https://check-host.net/check-tcp?host=baddcafe.com:443'
Caveat: modern OpenSSL 3.x strict mode (e.g. brew openssl,
check-host.net's HTTPS probe) rejects with "unsafe legacy
renegotiation disabled" because Plan 9 4E libsec predates
RFC 5746 (2010) and doesn't send the `renegotiation_info`
extension on the initial handshake. Compatible clients
handshake fine; this is an accepted limitation of the
canonical Plan 9 TLS scope.
Cleanup:
- Wipe macOS staging files (key.der is the prod private
key in plaintext): `shred -u ~/cert/key.der || rm -P
~/cert/key.der; rm ~/cert/{cert,chain}.pem`.
- Strip the `_acme-challenge` stanza from cube's
/lib/ndb/local and `echo refresh >/net/dns`.
- `rm -r /usr/web/.well-known` on cube.
- Renewal cadence: LE certs expire after 90 days.
Track the expiry date and repeat steps 24-30 before
then. LE sends warning emails 20/10/1 days before
expiry to the address registered with certbot.
31. Mail TLS prep — relax /sys/lib/ssl mode + derive fullchain.
Listener-spawned TLS services (smtpd STARTTLS, pop3 STLS,
tlssrv-wrapped imap4d/pop3) run as user `none` per
/rc/bin/service/ convention. They need world-readable cert
files. Also: smtpd and pop3 with the upas-certchain patch
(in 9legacy stable, present on the 9legacy CD) send the
certificate chain via STARTTLS when the -c flag points at
a multi-PEM file (leaf + intermediate concatenated).
On cube as bootes:
chmod 0644 /sys/lib/ssl/cert.pem /sys/lib/ssl/chain.pem
cat /sys/lib/ssl/cert.pem /sys/lib/ssl/chain.pem \
> /sys/lib/ssl/fullchain.pem
chmod 0644 /sys/lib/ssl/fullchain.pem
Both cert.pem and chain.pem are leaf-only / chain-only PEMs
(no `RSA PRIVATE KEY` block); the private key remains
factotum-only. 0644 leaks no secrets — leaf cert is public
information by definition.
Renewal cadence (step 30): after each cert refresh, also
re-run the cat above to refresh fullchain.pem.
32. Customize /rc/bin/cpurc.local with site env vars + reboot.
The shipped /rc/bin/cpurc.local has placeholder values
(site=DOM.EXAMPLE.COM, fileserver=kfs, facedom=FACEDOM)
which are read at boot and made live in the env. $site is
consumed by upas/smtp (HELO + From: completion); $fileserver
by some legacy scripts; $facedom by faces(1).
Edit /rc/bin/cpurc.local — comment the placeholder lines
for documentation, add real values below:
# site=DOM.EXAMPLE.COM
# fileserver=kfs
# facedom=FACEDOM
site=baddcafe.com
fileserver=cube
facedom=baddcafe.com
Apex form everywhere (the bare domain, not a host-prefix
like mail.baddcafe.com) — matches MX target and PTR; smtpd
-h conflates HELO + default-domain in a single flag, so
apex avoids per-user upasname overrides.
Reboot for env to propagate to running services:
fshalt -r
After boot, verify via bridge or local console:
echo $site $fileserver $facedom
# expect: baddcafe.com cube baddcafe.com
33. Add MX record to /lib/ndb/local + refresh DNS.
For your domain to advertise as a mail handler, add an mx=
attribute to the SOA stanza in /lib/ndb/local. Apex form
(MX target == apex domain) is RFC 2181 §10.3 compliant —
the apex must be a direct A record (not a CNAME), which is
naturally the case for a host serving its own domain.
Edit /lib/ndb/local SOA stanza:
dom=baddcafe.com soa=
refresh=3600 ttl=3600
ns=ns1.baddcafe.com
ns=ns2.baddcafe.com
mx=baddcafe.com pref=10
mb=postmaster@baddcafe.com
Refresh ndb/dns + ndb/cs (canonical after any /lib/ndb/
edit; ndb/dns auto-detects mtime change but explicit
refresh is safer):
echo refresh >/net/dns
echo refresh >/net/cs
Verify locally + externally:
dig @192.168.88.11 baddcafe.com MX +short # cube authoritative
# expect: 10 baddcafe.com.
dig @1.1.1.1 baddcafe.com MX +short # Cloudflare
dig @8.8.8.8 baddcafe.com MX +short # Google
# both should return: 10 baddcafe.com.
34. Edit /mail/lib/rewrite for own-MX local delivery.
The shipped /mail/lib/rewrite is empty + comments — the
operator MUST select either rewrite.direct (for own-MX
hosts) or rewrite.gateway (for smarthost-relay hosts).
Own-MX is the canonical choice for a host serving its
own apex domain (cube).
Three edits to rewrite.direct:
- drop stale "replace YOURDOMAIN.DOM" comment
- substitute YOURDOMAIN.DOM with your apex domain
(one rule line, the cube-FQDN strip pattern)
- add an apex-strip rule (without it, inbound mail
addressed to user@<your-apex> would fall through to
the queue rule and loop back via SMTP)
Final "your local names" section:
# your local names
\l!(.*) alias \1
\l\.baddcafe\.com!(.*) alias \1
baddcafe\.com!(.*) alias \1
First rule strips the bare hostname (\l = local sysname).
Second strips the host-FQDN form. Third strips the apex-
domain form (the new rule). All three feed into the
aliasmail / local!user / mailbox dispatch chain.
upas/send re-reads /mail/lib/rewrite per-message; no
service restart needed. Verify the rewrite chain
canonical-correct:
echo | upas/send -x bootes
# expect: unknown user: local!bootes
# ('unknown user' just means /mail/box/bootes/ doesn't
# exist yet — see bootes mailbox creation step)
echo | upas/send -x bootes@baddcafe.com
# expect: unknown user: local!bootes
# (apex-strip rule fired; without it would queue for
# SMTP self-loopback)
echo | upas/send -x glenda@gmail.com
# expect: gmail.com!glenda
# (queue rule fires; hand to upas/qmail for SMTP)
35. Create the host owner's home + mailbox.
For local mail to deliver to a user (host owner first; any
additional operator users later via the same procedure),
the user needs a mailbox at /mail/box/$user/mbox. The
canonical creator is /sys/lib/newuser, which scaffolds
the home directory ($home/{bin,lib,tmp}) AND creates the
mailbox via mail -c, the cron user dir via auth/cron -c,
and writes a default $home/lib/profile from a heredoc
template.
Run AS the user (cat /dev/user must return the target —
via local console after switching, or drawterm cpu -h
cube as the user):
/sys/lib/newuser
Expected output:
creating new mbox: /mail/box/<user>/mbox
Mailbox mode is `alrw--w--w-` per libsys.c:960
creatembox(): `0622 | DMAPPEND | DMEXCL` — append-only,
exclusive lock, world-write so smtpd-as-`none` can append
on inbound delivery.
Possible benign messages:
- `auth/cron: couldn't create /cron/<user>: ... is a
directory` — the cron dir already exists from earlier
setup (e.g., T3-era cpurc); harmless, mailbox creation
already succeeded.
- On a headless cpu server, /sys/lib/newuser ends with
`. lib/profile` which sources the just-created profile.
The 4E canonical profile's `case cpu` branch ends with
`if (! test -e /mnt/term/mnt/wsys) { ... exec rio }`
intended for drawterm-without-rio sessions; on a local
headless console where /mnt/term doesn't exist at all,
rio fails with `no frame buffer`. Non-fatal: the
mailbox was created BEFORE the rio attempt; parent
shell is intact. Future fresh interactive logins
re-trigger; canonical fix is to edit
$home/lib/profile and add `test -d /mnt/term &&` to
the inner condition so local console skips silently.
Verify:
ls -lq /mail/box/<user>/mbox
# expect: alrw--w--w- M ... <user> <user> 0 ...
echo | upas/send -x <user>
# expect: local!<user> (no "unknown user" prefix)
Repeat for each additional local user once they're added
via auth/changeuser + uname. newuser is idempotent — it
self-aborts if $home/lib/profile already exists ("user
directories already made"), so safe to invoke again.
36. Customize /mail/lib mail-policy files (smtpd.conf, blocked, names.local).
Three files configure smtpd's policy and alias behavior. The
canonical edits for an own-MX site:
/mail/lib/smtpd.conf — replace placeholder values, set policy
per man/6/smtpd:120-205. Canonical for own-MX inside a private
LAN with apex form throughout (matches MX, PTR, $site):
defaultdomain <your-apex-domain>
norelay on # restrict relay to ournets/ourdomains
verifysenderdom on # cheap DNS junk filter
saveblockedmsg off # discard rather than archive
ournets <your-LAN-CIDR>
ourdomains <your-apex>, *.<your-apex>
`norelay on` requires explicit `ournets` (trusted source nets
allowed to relay) + `ourdomains` (destination domains accepted
for relay) per man/6/smtpd:136-150.
/mail/lib/blocked — ship file has 2003-era cruft (AOL/UUNet
modem CIDRs, defunct domains like cyberpromo.com, AOL-account
chain-letter senders). Two paths:
(a) clear to header-only and populate reactively as actual
abuse arrives — cleanest baseline
(b) keep as-is — most rules don't match anything anymore
(defunct CIDRs/domains); ship state is harmless cosmetic
noise. One rule (`dial 63.0.0.0/11` — 32M IPs, was
UUNet dial-up) has minor false-positive risk for
legitimate hosts in re-allocated ranges; accepted as
low-volume risk for personal install
/mail/lib/names.local — system aliases per RFC 5321 §4.5.1
(postmaster mandatory) + RFC 2142 (recommended). Minimum
required edit: change `postmaster glenda` to
`postmaster <host-owner>` so postmaster mail accepts and
bounces-from-postmaster are deliverable. Optional canonical
expansion (RFC 2142 — review reactively if peer MTAs ever
flag missing aliases):
postmaster <host-owner>
abuse <host-owner>
hostmaster <host-owner>
webmaster <host-owner>
root <host-owner>
All five route to the host-owner's mailbox via aliasmail's
default-to-local recursion (per man/8/aliasmail:32-33).
Note: real users (e.g. the host owner, operator users) do
NOT need entries in names.local — aliasmail returns
`local!<name>` by default for unknown names, which dispatches
to /mail/box/<name>/mbox via the rewrite chain. names.local
is for VIRTUAL aliases that don't map to real users.
Effect immediately after deploy — smtpd reads conf in
getconf() per connection; rewrite + aliasmail re-read
per-message. No service restart needed.
Verify:
cat /mail/lib/smtpd.conf | head -10 # syntax visible
echo | upas/send -x postmaster@<your-apex>
# expect: local!<host-owner> (alias resolution working)
37. Enable smtpd listener — edit /rc/bin/service/tcp25.
/rc/bin/service/tcp25 is the canonical stub aux/listen forks
when an incoming SMTP connection arrives. The ship version
runs upas/smtpd with `-n $3` only — no STARTTLS, no explicit
receiving domain. Two flags need adding for an own-MX site:
-c /sys/lib/ssl/fullchain.pem # gates STARTTLS advertisement
# (per smtpd.c:448-449)
-h <your-apex-domain> # receiving domain + EHLO
# server-name (per smtpd.c:198-201,
# smtpd.c:445)
Final stub (one-line change):
#!/bin/rc
#smtp serv net incalldir user
user=`{cat /dev/user}
exec upas/smtpd -n $3 -c /sys/lib/ssl/fullchain.pem -h <your-apex-domain>
Mode preserved at 0775 sys:sys (matches ship). No aux/listen
restart needed — the listener forks rc to exec the stub on
each call; new stub picked up on next inbound connection.
LAN smoke test (from any LAN host while WAN NAT not yet
configured):
{ sleep 4; echo "EHLO mac.lan"; sleep 1; echo "QUIT"; } \
| nc -w 8 <cube-LAN-ip> 25
Expect:
220-<your-apex-domain> ESMTP
220
250-<your-apex-domain> you are mac.lan
250-ENHANCEDSTATUSCODES
250-STARTTLS
250 AUTH CRAM-MD5
221 2.0.0 Successful termination
Common test gotchas (canonical smtpd anti-spam in action):
- "554 5.7.0 Spammer!" — peer sent input during the 3-second
post-banner pause (smtpd.c:289-305 sayhi()); use sleep 4
before EHLO
- "554 5.7.0 Liar!" — HELO name uses RFC 2606 reserved
domain (example.com / example.net / example.org / .test /
.invalid / localhost / localdomain). Use any other
plausible name (e.g. mac.lan, your.actual.hostname)
Other smtpd flags considered but typically NOT added to tcp25:
`-a` (mandatory AUTH; for tcp587 Submission only),
`-f` `-r` `-s` `-g` (redundant — smtpd.conf already configures
norelay / verifysenderdom / saveblockedmsg).
38. Open WAN dst-NAT for tcp25 on the router.
With smtpd's tcp25 listener configured (step 37), external
MTAs need a routing path through the gateway to reach it.
Add a dst-NAT rule on the router translating WAN-side
tcp/25 traffic to cube's LAN address.
On RouterOS (MikroTik), CLI form:
/ip firewall nat add chain=dstnat in-interface-list=WAN \
protocol=tcp dst-port=25 \
action=dst-nat to-addresses=<cube-LAN-ip> to-ports=25 \
comment="SMTP -> cube"
Or web UI: IP / Firewall / NAT / +
Chain: dstnat
In. Interface List: WAN
Protocol: 6 (tcp)
Dst. Port: 25
Action tab → Action: dst-nat,
To Addresses: <cube-LAN-ip>, To Ports: 25
Comment: SMTP -> cube
Mirrors the existing tcp80/443/53 dst-NAT rules in your
ROUTER CONFIG appendix.
Optional LAN-side hairpin — only if LAN clients should be
able to submit mail via SMTP to the public hostname (mostly
relevant if you later enable client-side Submission/SMTPS
on tcp587/465). Skip for first-pass own-MX scope:
/ip firewall nat add chain=dstnat \
src-address=<LAN-CIDR> dst-address=<your-public-IP> \
protocol=tcp dst-port=25 \
action=dst-nat to-addresses=<cube-LAN-ip> to-ports=25 \
comment="hairpin LAN->cube:25"
/ip firewall nat add chain=srcnat \
src-address=<LAN-CIDR> dst-address=<cube-LAN-ip> \
protocol=tcp dst-port=25 \
action=masquerade \
comment="hairpin masquerade for LAN replies"
Verify external reachability (run from any host outside
your LAN, or use a third-party probe service):
# check-host.net TCP-port probe (free, no auth):
curl -s -A "Mozilla/5.0" \
"https://check-host.net/check-tcp?host=<your-apex>:25&max_nodes=3" \
-H "Accept: application/json"
# returns request_id; wait ~10s, then:
curl -s -A "Mozilla/5.0" \
"https://check-host.net/check-result/<request_id>" \
-H "Accept: application/json"
# expect: each external node reports successful TCP
# connection to <your-public-IP>
# Or browser: https://check-host.net/check-tcp?host=<apex>:25
# Or hotspot-from-phone trick: switch your test host to
# cellular hotspot (forces WAN routing), then nc to apex:25
# — should return canonical 220-<apex> ESMTP banner.
39. Edit /mail/lib/remotemail HELO fallback.
/mail/lib/remotemail is the rc script called by upas/runq
for each queued outbound message. It derives the HELO
name announced to peer MTAs from the sender's domain via
aliasmail -f. When that fails (sender unqualified, or
/mail/lib/fromfiles missing — both common on a fresh
install), it falls back to a placeholder string at line
12. Ship default is `yourdomain.dom` — invalid (no .dom
TLD); receiving MTAs reject HELO with this name.
Edit /mail/lib/remotemail line 12:
case *
fd=baddcafe.com # was: fd=yourdomain.dom
Apex form again (matches HELO with MX + PTR for clean
FCrDNS validation by peer MTAs). Mode 0775 upas:upas
preserved. runq invokes remotemail fresh per pass; effect
immediate on next queued message.
40. Request rDNS PTR record from your ISP.
Mail providers (Gmail, Microsoft, etc.) require Forward-
Confirmed reverse DNS (FCrDNS) on the connecting IP. The
PTR record is in the in-addr.arpa zone delegated to your
ISP — you cannot set it from cube.
Find your ISP's NOC contact via `whois <your-public-ip>`
(look at admin-c / tech-c / abuse-mailbox; or `dig <isp>
SOA +short` for the noc-style address). Email a polite
request:
Please set PTR <your-public-ip> → <your-apex-domain>
Apex form — matches your HELO + MX for clean FCrDNS.
Many ISPs ship a default PTR pointing to a generic hostname
that has no matching forward A record; this fails FCrDNS
even worse than no PTR at all (mention it in the request
if so).
Verify after the ISP applies the change:
dig -x <your-public-ip> +short # expect: <your-apex-domain>.
dig <your-apex-domain> +short # expect: <your-public-ip>
Both should round-trip cleanly. Until this lands, outbound
mail to strict providers may be rejected or filtered to spam.
41. Demote glenda from privileged groups (post-T4 hardening).
Step 3 added glenda to the adm group temporarily for install
bootstrapping (so glenda could write /adm/timezone/local etc.
before bootes existed). After the host owner (bootes) is
established and verified (steps 14-16) and provisioned with
secstore (step 17), glenda's adm membership is no longer
needed. In some 4E install paths glenda is also a member
of sys (granting write access to system code, libraries,
binaries). Both are bootstrap leftovers; should be removed
for least-privilege canonical state.
Run any time after bootes-as-host-owner is verified. Via
fossil console as bootes:
con -l /srv/fscons
prompt: uname adm -glenda
prompt: uname sys -glenda
Ctrl-\
>>> q
Verify:
cat /adm/users
# expect: adm:adm:adm:sys,bootes (no glenda)
# sys:sys::bootes (no glenda)
What this changes for glenda:
- cannot modify /adm/keys (auth user database)
- cannot modify /adm/users (group memberships)
- cannot modify files in /sys/* (binaries, scripts, source)
- still listed as a user in /adm/users for file-ownership
attribution; nothing else affected
Why both groups (not just adm):
- adm controls auth-subsystem write access (most sensitive)
- sys controls system-code write access (less direct, but
an attacker with sys can edit rc scripts that run as
bootes — escalation path); removing both closes the
full surface
Glenda's auth state should also be verified:
ls /mnt/keys
# expect: only /mnt/keys/bootes (and any operator users
# added later via auth/changeuser). glenda absent means
# she has no Plan 9 auth credential — cannot drawterm in
# at all. This is the canonical safe state for an
# inactive bootstrap user.
If glenda IS in /mnt/keys (has an /adm/keys entry), set a
strong password to prevent drawterm-as-glenda from being
a privilege-escalation entry point:
auth/changeuser glenda
# interactively, set a strong password (or paste one
# from secstore-generated random)
Or remove glenda from /adm/keys entirely if she's truly
inactive.
Cosmetic optional: `rm -r /usr/glenda` if her home directory
is empty and you want a clean filesystem. Files owned by
glenda elsewhere (rare; mostly a few install artifacts) can
stay — nobody can become glenda to modify them.
42. Create a regular user (three-step canonical procedure).
Plan 9 user creation spans three subsystems: fossil
(file-ownership identity), keyfs (login password for
p9sk1 cpu/drawterm auth), and secstore (encrypted on-disk
credential archive auto-loaded into factotum at login).
Two distinct passwords — login and secstore — never reuse.
Example below uses `mospak`. Run on the auth/fs server
as the host owner:
# 1. Fossil user
con -l /srv/fscons
prompt: uname mospak mospak
Ctrl-\
>>> q
# 2. Login password only
auth/changeuser mospak
# password: <mospak login password>
# confirm
# 3. Secstore enrollment
auth/secuser mospak
# password: <mospak secstore password — distinct from login>
# expires / Enabled / require STA / comments — defaults
Step 1: `uname <user> <leader>`; self-leadered means the
user is their own group leader (no separate group).
Step 2: sets the keyfs password used for p9sk1 cpu /
drawterm auth. Step 3: enrolls the user in secstore so
factotum can auto-load their stored keys at login.
After all three steps, the new user logs in (drawterm or
cpu) and runs `/sys/lib/newuser` to scaffold home directory,
mailbox, and per-user cron file. See man/8/newuser.
NOTES
-----
Shipped log cruft.
/sys/log/mail (~914 B) and /sys/log/runq (~182 B) ship with
2003-2007 entries from Bell Labs dev hosts (olive, achille).
These are append-mode logs — upas binaries don't care about
pre-existing content, so the cruft is harmless. Truncate
via `> /sys/log/mail` etc. for a fresh baseline if you want
clean logs from day 1; or leave as-is (purely cosmetic; no
functional impact). Every fresh 4E install has these —
they're a community-shared shipping artifact (Bell Labs
packaged the install ISO from a working dev host without
truncating logs first).
mailnews — future setup.
/cron/upas/cron's mailnews entry fires every 15 min; built
to broadcast /lib/news/* content to /sys/lib/subscribers
via mail. On a fresh install both /lib/news/ and
/sys/lib/subscribers are empty, so the cron line is a
no-op (writes /mail/lib/newstime mtime each run; nothing
delivered). To use later for cube-internal announcements:
- populate /lib/news/ with one announcement file per
topic (any plain-text content)
- populate /sys/lib/subscribers with one username per
line
- ensure each subscriber has a mailbox (`mail -c` as
the user, or run /sys/lib/newuser which calls mail -c
as part of home scaffolding)
- rewrite chain (step 34) is the prerequisite for the
outgoing `mail` invocation in mailnews to actually
dispatch
Until populated, the cron line firing 4×/hr is harmless
overhead. Comment it out in /cron/upas/cron if you want
to silence it; restore when ready.
Add a key to the host owner's secstore.
Operational procedure for adding or updating a key in the
host owner's secstore archive. The next boot's secstore-G
in cpurc loads the new key into factotum. Without this
push, manually-loaded keys (echo >> /mnt/factotum/ctl)
disappear on reboot.
Procedure (run on the auth server, as the host owner):
ramfs -p # -p sets noswap + private;
# keeps the key out of swap
cd /tmp
auth/secstore -g factotum # fetch existing
wc -l factotum # SANITY: expect >0 lines
echo 'key proto=...' >> factotum
auth/secstore -p factotum # push back
unmount /tmp
Gotcha 1: use `ramfs -p`, never plain `ramfs`. `-p` sets
`noswap` + `private` (ramfs.c:786-798); without `noswap` the
plaintext key can be paged out to swap. Verify the mount
with a marker: after `ramfs -p`,
`echo m > /tmp/m && ls /tmp/m && unmount /tmp && ls /tmp/m`
— present after mount, absent after unmount.
Gotcha 2: a failed `-g` destroys the vault. If `-g` fails
it creates no file. The following `>>` then makes a new
one-line file, and `-p` replaces the whole vault entry with
it. Prior keys are gone, and secstore keeps no backups.
`wc -l` does not catch this. A failed decrypt leaves a
full-length file of random bytes with a plausible line
count. Check the exit status of `-g` instead.
`auth/secstore -G .` lists the vault — names, sizes, dates
— and writes nothing. Run it before and after, and compare
the size of `factotum`.
Gotcha 3: the secstore password is prompted twice (once
by `-g`, once by `-p`). The inline `!password=...` slot
inside the appended key line is a different password
(typically the host-owner login password set during T4).
FINAL CHECKLIST
---------------
File-server users:
bootes exists in /adm/users.
bootes is in sys, adm, upas.
Auth users:
bootes exists in /adm/keys.
NVRAM:
authid=bootes, authdom=baddcafe.
Machine key matches /adm/keys's encryption key.
Secstore key set.
Startup files:
/cfg/cube/cpurc starts ip/ipconfig, ndb/dns -r, auth/keyfs,
auth/cron, auth/secstored, and aux/listen -t.
/rc/bin/cpurc is unchanged from the shipped 9legacy tree.
Listener:
/rc/bin/service.auth/tcp567 exists (renamed from
authsrv.tcp567).
nc to cube:567 from outside succeeds.
NDB:
ndb/query sys cube returns the expected tuple.
ndb/ipquery sys cube ... shows inherited ipnet attributes.
ndb/query -f /lib/ndb/auth hostid bootes returns the rule.
Secstore:
/adm/secstore exists, mode 0770, owner bootes.
/adm/secstore/who/bootes exists with `exp` and `PAK-Hi` lines.
Verifier password matches the NVRAM secstore key from
step 13.
Web (plain HTTP, optional):
/usr/web exists, owner bootes, group web, mode 0775.
/usr/web/index.html exists, world-readable.
/lib/namespace.httpd exists (can be empty).
/cfg/cube/cpustart contains `ip/httpd/httpd`.
Remote `curl http://<cube-ip>/` returns the test page.
DNS authoritative (optional, when running step 23):
/lib/ndb/local has SOA for baddcafe.com, root-server
hints, and public A records at your WAN IP.
/cfg/cube/cpurc starts `ndb/dns -s`.
Gateway router forwards WAN UDP/53 + TCP/53 to cube.
Registrar has glue records for ns1/ns2 and NS records
pointing to them.
External `dig @<wan-ip> baddcafe.com SOA` returns your SOA.
External resolver `dig @1.1.1.1 baddcafe.com A` returns
your public IP.
dnserver.c CAA patch applied (CAA queries return NOERROR
+ SOA in authority section, not NOTIMP).
TLS (optional, when running steps 24-30):
/etc/letsencrypt/live/baddcafe.com/{cert,chain,fullchain,
privkey}.pem exist on macOS (root-owned).
Cube /sys/lib/ssl/{cert,chain}.pem exist, mode 0600
bootes:sys.
Cube secstore vault has `factotum` entry:
ls -l /adm/secstore/store/bootes/factotum
/cfg/cube/cpurc has the boot-time secstore->factotum line
after `auth/secstored`.
/cfg/cube/cpustart invokes httpd with `-c` and `-C` flags.
cat /mnt/factotum/ctl shows the TLS key:
key proto=rsa service=tls role=client owner=* size=
48
From a compatible-client host:
curl --resolve baddcafe.com:443:<cube-lan-ip> \
https://baddcafe.com/
# expect: 200 OK with cipher AES256-SHA, verify ok
External port 443 reachable (check-host.net or similar).
macOS staging files wiped post-deploy (~/cert/key.der in
particular — it's the prod private key in plaintext).
Cube /usr/web/.well-known/ removed and _acme-challenge
stanza stripped from /lib/ndb/local.
Cert expiry date noted for renewal (LE 90-day cycle).
ROUTER CONFIG (MikroTik example)
--------------------------------
This is the router-side configuration the per-host setup above
assumes is in place. Generic to RouterOS-based MikroTiks; adapt
field names for other router platforms.
Find your public IP:
/ip/dhcp-client/ print # look at WAN client
# or via web UI: IP / DHCP Client / IP Address
DHCP leases (IP / DHCP Server / Leases) — pin LAN hosts to
fixed IPs by MAC:
192.168.88.11 <CUBE_MAC> # cube
192.168.88.12 <GNOT_MAC> # gnot terminal
192.168.88.13 <BR1_MAC> # wireless bridge near cube
192.168.88.14 <BR2_MAC> # wireless bridge near gnot
NAT rules (IP / Firewall / NAT, chain=dstnat, in-interface=<WAN>):
Chain dstnat, proto tcp, dst port 25, → 192.168.88.11:25
comment="SMTP -> cube"
Chain dstnat, proto tcp, dst port 80, → 192.168.88.11:80
comment="HTTP -> cube"
Chain dstnat, proto tcp, dst port 443, → 192.168.88.11:443
comment="HTTPS -> cube"
Chain dstnat, proto udp, dst port 53, → 192.168.88.11:53
comment="DNS UDP -> cube"
Chain dstnat, proto tcp, dst port 53, → 192.168.88.11:53
comment="DNS TCP -> cube"
CLI form (RouterOS terminal):
/ip firewall nat add chain=dstnat in-interface=<WAN> \
protocol=udp dst-port=53 \
action=dst-nat to-addresses=192.168.88.11 to-ports=53 \
comment="DNS UDP -> cube"
(analogous for tcp/53, tcp/80, tcp/443)
NAT loopback (hairpin) for LAN reach to public hostname:
Without these rules, LAN clients cannot reach the site by
its public hostname. DNS resolves to the WAN IP but the
dst-NAT rules above match `in-interface=<WAN>` only —
LAN-origin traffic arrives on the gateway's LAN side and
fails the rule. Add two extra rules per port: a dst-NAT
for LAN-origin traffic, plus a srcnat masquerade so cube's
reply rides back through the gateway.
For tcp/443 (required for LAN reach):
/ip firewall nat add chain=dstnat \
src-address=192.168.88.0/24 \
dst-address=<PUBLIC_IP> \
protocol=tcp dst-port=443 \
action=dst-nat to-addresses=192.168.88.11 to-ports=443 \
comment="hairpin LAN->cube:443"
/ip firewall nat add chain=srcnat \
src-address=192.168.88.0/24 \
dst-address=192.168.88.11 \
protocol=tcp dst-port=443 \
action=masquerade \
comment="hairpin masquerade for LAN replies"
For tcp/80 (optional): same shape with `dst-port=80` in both
rules. Useful only if LAN clients should follow the HTTP→
HTTPS redirect from `http://baddcafe.com`; most LAN workflows
go straight to HTTPS, so skip unless needed.
Test from a LAN client:
curl -v https://baddcafe.com/
# expect: 200 OK, TLS cert validates CN=baddcafe.com
Move router admin off port 80 (avoid colliding with HTTP NAT):
IP / Services: www, port 8080
Router admin panel now at http://192.168.88.1:8080
When step 23 (cube as authoritative DNS) is in use, also
consider hardening the WAN-facing DNS exposure — drop UDP/53
packets where the DNS header's RD (Recursion Desired) bit is
set, so cube only answers authoritative queries to the world
while still serving recursive lookups to the LAN. Specific
RouterOS syntax for this varies by firmware version; consult
MikroTik documentation for "content matcher" and "RAW filter"
features.
TESTED HARDWARE
---------------
Reference for one known-working configuration; the procedure
applies to most Plan-9-compatible x86 hardware. "Compatible"
in practice means: a kernel-supported NIC (most Intel/Realtek
gigabit chipsets), a SATA controller in AHCI mode (or older
IDE/ATAPI), and either VGA or VESA-compatible video. See
/sys/src/9/pc/etherif.h and the various sd*.c drivers for
the exact list.
Server: Supermicro X7SPA-H-D525 (Atom D525, 4 GB RAM, i386)
Terminal: ThinkPad X61 (in docking station) or ThinkPad T61
Router: MikroTik mAP 2nD (RBmAP2nD) — LAN gateway
Bridges: MikroTik mAP lite (RBmAPL-2nD) x 2 — ethernet-to-
wireless bridges for server and terminal
Bridge mode caveat — the wireless bridges must be configured as
`wlan1.mode=station-bridge`, NOT `station-pseudobridge`.
Pseudobridge does MAC NAT and only carries one downstream device
transparently; station-bridge is L2-transparent for arbitrarily
many. Symptom of getting this wrong: DHCP fails on the wired
side of the bridge.
Server kernel: cube boots `9pccpuf` (i386, self-built — see
step 9), not the amd64 `9k10cpuf`, because the amd64 9k kernel
is multiboot-only and 9load isn't a multiboot loader. The
hardware is amd64-capable; the choice is software-driven.
END
|
