SETUP_NOTES
===========
Plan 9 cpu/auth/fs server setup notes.
This file documents the actual procedure used to bring up cube as
the cpu/auth/fs server for the baddcafe authentication domain.
It is written in the spirit of the 9grid howto cited below, with
the deviations called out at each step.
Names used in this file:
cpu/auth/fs server: cube
terminal: gnot
Plan 9 authdom: baddcafe
public DNS name: baddcafe.com
bootstrap user: glenda
host owner: bootes
operator user: mospak (day-to-day account; bootes is the
system identity used by auth services)
kernel: i386
install disk: sdE0
Sections:
SETUP STEPS 1-17 install + auth foundation (required)
SETUP STEPS 18-22 web - plain HTTP (optional)
SETUP STEPS 23 cube as authoritative DNS (optional)
SETUP STEPS 24-30 TLS via Let's Encrypt (optional)
FINAL CHECKLIST per-section verification matrix
ROUTER CONFIG MikroTik gateway example
TESTED HARDWARE one known-working configuration
SOURCE NOTES
------------
This file follows the procedure of:
http://mirror.9grid.fr/mirror.9grid.fr/plan9-cpu-auth-server-howto.html
In-tree references used in this setup:
/sys/man 8/auth, 8/changeuser, 8/secstore, 6/ndb,
6/users, 4/fossilcons
/sys/doc Bell Labs papers
/sys/src live source for kernel rebuild and rc scripts
BELL LABS CONVENTION
--------------------
Per-host changes go in /cfg/<sysname>/, not in /rc/bin/.
/rc/bin/cpurc shipped, generic, do NOT edit
/rc/bin/termrc shipped, generic, do NOT edit
/cfg/cube/cpurc cube's cpu-mode startup overrides
/cfg/cube/termrc cube's terminal-mode overrides
/cfg/gnot/termrc gnot's terminal startup overrides
The shipped /rc/bin/cpurc sources /cfg/$sysname/cpurc when it
exists. Anything turned on at boot lives in /cfg/cube/cpurc.
In particular: the keyfs, cron, and secstored lines that the
9grid howto Step 6 puts in /rc/bin/cpurc go in /cfg/cube/cpurc
in this setup. The shipped tree stays pristine.
If asked "do I edit /rc/bin/X or /cfg/cube/X?", the answer is
always /cfg/cube/X.
MENTAL MODEL
------------
Three identity systems, three databases, three "who am I"s:
file-server users /adm/users (managed via fossil console)
Controls file ownership and groups.
Does not create any login password.
con -l /srv/fscons
uname bootes bootes
uname sys +bootes
uname adm +bootes
auth users /adm/keys (managed via auth/changeuser)
Controls login for cpu, drawterm, and
the auth protocol on tcp567.
auth/changeuser -p bootes
NVRAM identity /dev/sdE0/nvram (set via auth/wrkey
or the kernel boot-time prompt)
Tells the kernel who the host owner
is when cube boots in cpu mode.
auth/wrkey
A given identity (e.g. bootes) usually exists in all three.
Three secret stores, three lifetimes:
factotum in-memory authentication agent. Mounted at
/mnt/factotum. Loaded keys are lost at reboot
or factotum restart. Control file is
/mnt/factotum/ctl.
Load a key for the current session:
echo 'key proto=p9sk1 dom=baddcafe user=mospak' \
>/tmp/key
echo ' !password=PASSWORD' >>/tmp/key
cat /tmp/key >/mnt/factotum/ctl
rm /tmp/key
secstore persistent encrypted store served by
auth/secstored. Survives reboot. Per-user.
bootes secstore is not mospak secstore. The
conventional file inside secstore is named
exactly "factotum".
Store a local file:
auth/secstore -p factotum
Fetch and load into live factotum:
auth/secstore -G factotum | read -m \
>/mnt/factotum/ctl
keyfs host-side decryption of /adm/keys. Mounted at
/mnt/keys. Started at boot from cpurc.
Captures the NVRAM machine key once at startup;
reboot required to pick up a new machine key.
Three passwords per host owner. This setup uses three distinct
passwords for defense in depth. The KISS option of one shared
password works equally well functionally.
1. login password set via auth/changeuser; protects
bootes identity at the auth protocol
2. machine key set via auth/convkeys (encrypts
/adm/keys) and auth/wrkey (writes
the NVRAM copy); the two must match
byte-for-byte
3. secstore password set via auth/wrkey; identifies bootes
to auth/secstored
Two NDB files, two purposes:
/lib/ndb/local network names, addresses, and service
attributes (authdom, auth, cpu, fs,
dns, ipgw — typically inheriting from
an ipnet stanza)
/lib/ndb/auth host-owner rule:
hostid=bootes
uid=!sys uid=!adm uid=*
Means: bootes is the host owner, may
authenticate any user except sys and
adm.
Two listener layouts, both canon:
aux/listen runs services as none. Used for
non-auth services.
aux/listen -t \ searches a service.auth/ directory
/rc/bin/ and runs the auth services as the
service.auth host owner. This is what serves
tcp567 (the Plan 9 auth protocol).
This setup uses the standard /rc/bin/service.auth/tcp567 layout.
Two reboot paths, two semantics:
fshalt -r warm reboot via #c/reboot. Bypasses
9load and plan9.ini. Same kernel
continues running. OK for same-arch
reboots.
cold reboot power cycle. Reads plan9.ini, follows
the menu. Required after plan9.ini
edits and after switching kernel
architecture.
SAFETY RULES
------------
Always fshalt before power-off. Never hard-reset a running
Plan 9 system. fossil and venti both prefer a clean shutdown.
Change one layer at a time:
install
network
auth service
bootes login
secstore
Do not start the same service twice. Common mistakes are two
aux/listen commands, two auth/keyfs commands, or two httpd
commands.
SETUP STEPS
-----------
1. Install Plan 9.
Boot the 9legacy install ISO. Use a single PLAN9 fdisk
partition with sub-partitions for 9fat, nvram, swap, fossil,
isect, arenas, bloom. Choose fossil+venti at the
configfs prompt (not fossil-only — venti is needed for
snapshot/backup later).
bootsetup installs the terminal kernel (9pcf) and writes a
plan9.ini that boots terminal mode by default. Override
this at the Enable boot method: prompt to add a [cpu] menu
entry.
2. First boot in terminal mode as glenda.
At the boot prompt:
root is from (tcp, local)[local!#/sdE0/fossil]: <ENTER>
user[none]: glenda
3. Add glenda to adm temporarily and fix /tmp permissions.
The first command lets glenda write /adm/timezone/local in
step 4. The second fixes /tmp from shipped 0755 (which blocks
most users from writing) to the canonical 0777 - needed by
ape/patch and other POSIX-style ports.
con -l /srv/fscons
prompt: uname adm +glenda
prompt: fsys main
main: wstat /active/tmp - - - 777 -
Ctrl-\
>>> q
4. Set the timezone.
lc /adm/timezone
cp /adm/timezone/<your-zone> /adm/timezone/local
5. Prepare the per-host config directory.
mv /cfg/example /cfg/cube
6. Edit /cfg/cube/cpurc.
Per-host cpu-mode startup script. The shipped /rc/bin/cpurc
sources this file and stays unchanged.
#!/bin/rc
dmaon
ip/ipconfig
ndb/dns -r
auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
auth/cron >>/sys/log/cron >[2=1] &
auth/secstored
aux/listen -q -t /rc/bin/service.auth \
-d /rc/bin/service tcp
sleep 3
The keyfs, cron, and secstored lines correspond to the lines
the 9grid howto Step 6 says to uncomment in /rc/bin/cpurc.
Putting them in /cfg/cube/cpurc instead leaves the shipped
tree untouched.
7. Enable the auth service stub.
mv /rc/bin/service.auth/authsrv.tcp567 \
/rc/bin/service.auth/tcp567
One-shot rename, not part of cpurc.
8. Edit /lib/ndb/local.
Read cube's MAC address:
cat /net/ether0/addr; echo
Append an ipnet stanza so attributes inherit to all hosts on
the LAN, plus per-host tuples:
ipnet=baddcafe ip=192.168.88.0 ipmask=255.255.255.0
authdom=baddcafe
auth=cube
cpu=cube
fs=cube
dns=192.168.88.1
dnsdomain=baddcafe.com
ipgw=192.168.88.1
ip=192.168.88.11 sys=cube ether=<cube-mac> \
dom=cube.baddcafe.com
ip=192.168.88.12 sys=gnot ether=<gnot-mac> \
dom=gnot.baddcafe.com
ip=192.168.88.11 dom=baddcafe.com
ip=192.168.88.11 dom=www.baddcafe.com
FQDN form (`baddcafe.com`, `cube.baddcafe.com`) is canonical
per `man/6/ndb` definition of `dom=` as Internet FQDN.
`authdom=baddcafe` (no .com) is separate — that's the Plan 9
authentication domain identifier, not a DNS name.
Verify:
ndb/query sys cube
ndb/ipquery sys cube authdom auth cpu fs ipgw dns
ndb/query returns the matching tuple's attributes.
ndb/ipquery follows ipnet inheritance — what ndb/cs actually
returns to clients.
9. Build and install the cpu kernel.
cd /sys/src/9/pc
mk 'CONF=pccpuf'
9fat:
cp 9pccpuf /n/9fat/
bootsetup never installs a cpu kernel. This step is
load-bearing.
10. Edit /n/9fat/plan9.ini.
Replace the flat bootfile= line with a menu:
[menu]
menuitem=cpu, Plan 9 CPU/Auth Server
menuitem=terminal, Plan 9 Terminal
menudefault=cpu, 10
[cpu]
bootfile=sdE0!9fat!9pccpuf
[terminal]
bootfile=sdE0!9fat!9pcf
[common]
nobootprompt=local!#S/sdE0/fossil
bootargs=local!#S/sdE0/fossil
bootdisk=local!#S/sdE0/fossil
venti=#S/sdE0/arenas
console=0
dmamode=on
mouseport=ps2
monitor=vesa
vgasize=1024x768x16
user=glenda
`user=glenda` auto-logs into terminal mode without a
username prompt. Comment it out (`# user=glenda`) once
bootes is fully working if you prefer the username prompt
at boot.
11. Create bootes as a file-server user.
On cube, still as glenda:
con -l /srv/fscons
prompt: uname bootes bootes
prompt: uname sys +bootes
prompt: uname adm +bootes
prompt: uname upas +bootes
prompt: fsys main
main: create /active/cron/bootes bootes bootes d775
main: create /active/sys/log/cron bootes adm a664
Ctrl-\
>>> q
fsys main MUST come before any create.
If /active/sys/log/cron already exists from copydist:
wstat /active/sys/log/cron - bootes adm a664 -
12. Add the host-owner rule to /lib/ndb/auth.
Append:
hostid=bootes
uid=!sys uid=!adm uid=*
Verify (this file is not in cs's default DB; query it
explicitly):
ndb/query -f /lib/ndb/auth hostid bootes
13. Reboot into cpu mode for the first time.
fshalt
# cold reboot (power-cycle), then select the cpu menu
# entry at the boot menu
NVRAM is empty on this first cpu boot. The kernel prompts:
authid: bootes
authdom: baddcafe
secstore key: SECSTORE_KEY
password: MACHINE_KEY
The password becomes both the NVRAM machine key and the
/adm/keys encryption key for cpurc's auth/keyfs. Pick any
non-empty value; it is rotated to a permanent value in step
15.
After the kernel prompt completes, cube comes up in cpu
mode running auth services as bootes via the NVRAM
identity. cpurc has started auth/keyfs, auth/cron,
auth/secstored, and aux/listen.
14. Create bootes as an auth user.
Steps 14–16 run at cube's local console (rc shell on cube's
attached monitor + keyboard, running as bootes after step
13).
auth/changeuser -p bootes
Set the bootes login password. This is the password
drawterm and cpu(1) will prompt for; it is independent of
the NVRAM machine key set in step 13.
15. Re-key /adm/keys and write NVRAM (terminal-mode boot).
/adm/keys was created in step 14 with the temporary machine
key entered at the boot prompt in step 13. Rotate it to a
permanent password.
The safe form runs convkeys + wrkey in terminal mode where
no keyfs is alive. Doing this in cpu mode with keyfs
running risks a race documented in auth(8) BUGS: any
keyfs-write between convkeys and reboot re-encrypts
/adm/keys with the OLD machkey from keyfs's stale memory
snapshot, leaving NVRAM and disk out of sync at next boot.
1. Clean shutdown:
fshalt
2. Cold reboot (power-cycle), select terminal at the boot
menu. Terminal mode does not start auth services; no
keyfs to race.
3. At the terminal-mode rc prompt:
auth/convkeys /adm/keys
# New password (×2): MACHINE_KEY
auth/wrkey
# authid: bootes
# authdom: baddcafe
# auth password: MACHINE_KEY (must equal convkeys's)
# secstore password: SECSTORE_KEY (independent;
# distinct value for defense in depth)
4. Reboot back to cpu mode:
fshalt -r
Warm reboot is fine after this — no kernel arch change,
NVRAM holds the new key, /adm/keys is freshly re-encrypted
with the matching key. cpurc's auth/keyfs reads the new
NVRAM machkey at startup and decrypts /adm/keys cleanly.
16. Verify cube is running as bootes.
cat /dev/user # expect: bootes
cat /env/service # expect: cpu
ps | grep keyfs # bootes ... keyfs
ps | grep secstored # bootes ... secstored
ps | grep cron # bootes ... cron
From a remote host:
nc -z -v 192.168.88.11 567 # auth listener up
17. Provision bootes secstore.
The boot-time TLS key load (added later, when TLS is set up)
reads bootes's secstore via `auth/secstore -n -G factotum`,
which needs both:
- NVRAM secstore key (set in step 13's kernel prompt)
- on-disk verifier at /adm/secstore/who/bootes
Step 13 set the NVRAM half. This step writes the on-disk
verifier with the same password.
auth/secuser bootes
# bootes password: SECSTORE_KEY (same as step 13's
# secstore key)
# retype password: SECSTORE_KEY
# expires [...]: <DDMMYYYY> or <Enter> for +365 days
# Enabled or Disabled: <Enter> (Enabled)
# require STA?: <Enter> (no)
# comments: <Enter> (or any note)
auth/secuser auto-creates /adm/secstore (0755), /who (0755),
/store (0700). Tighten /adm/secstore to canonical 0770 per
man/8/secstore via fossil console:
con -l /srv/fscons
prompt: fsys main
main: wstat /active/adm/secstore - - - 770 -
Ctrl-\
>>> q
Verify:
ls -ld /adm/secstore # d-rwxrwx---
cat /adm/secstore/who/bootes # exp + PAK-Hi lines
18. (Optional) Create web user/group + /usr/web.
Plan 9 ships ip/httpd/httpd as the canonical web server.
It serves /usr/web by default. This step creates a `web`
group for content-admin collaboration and the /usr/web
directory; httpd's runtime identity is independent (see
step 22).
Fossil console:
con -l /srv/fscons
prompt: uname web web
# if not at main: prompt: fsys main
main: create /active/usr/web bootes web d775
Ctrl-\
>>> q
Result: drwxrwxr-x bootes web /usr/web. bootes writes via
the owner bit; users added to group `web` (e.g. mospak via
`uname web +mospak`) write via the group bit.
If /usr/web already exists from an earlier pass, the create
will return `file already exists`. Use wstat to set just
the attributes you need to change — fields after the path
are `name uid gid mode mtime`; `-` means "don't change".
Example for fixing only the owner (when /usr/web exists
with a uid other than bootes):
main: wstat /active/usr/web - bootes - - -
19. Minimize /lib/namespace.httpd.
The shipped /lib/namespace.httpd is full of mounts specific
to the original Bell Labs site (alice, netlib, etc.) that
will fail or do nothing on a fresh install. httpd auto-
binds /usr/web onto / regardless of namespace.httpd
contents, so empty is the minimum that works.
cp /lib/namespace.httpd /lib/namespace.httpd.orig
> /lib/namespace.httpd
20. Place initial content.
echo '<html><body><h1>your-site</h1></body></html>' \
> /usr/web/index.html
Default mode 0664 (rw-rw-r--) is world-readable, which
httpd-as-none requires (see step 22).
21. Start httpd by hand and test.
httpd self-backgrounds (rfork + parent exits) so the prompt
returns immediately:
ip/httpd/httpd
ps | grep httpd # none ... httpd
From another LAN host:
curl http://<cube-ip>/
# expect: HTTP/1.1 200 OK with your index content
22. Persist httpd for boot.
/rc/bin/cpurc sources /cfg/<sysname>/cpustart via `.` (no
exec bit needed):
echo 'ip/httpd/httpd' > /cfg/cube/cpustart
Note on httpd's runtime identity: regardless of who starts
httpd, it switches to user `none` immediately (httpd.c
becomenone() writes "none" to #c/user, rebuilds the
namespace, overlays /lib/namespace.httpd). Files in
/usr/web therefore need world-readable mode bits for httpd
to serve them. File ownership matters only for who EDITS
content, not for httpd's serving role.
23. (Optional) Run cube as authoritative DNS for baddcafe.com.
By default the canonical Plan 9 setup uses the registrar's
nameservers and only resolves DNS locally. If you want
cube to be the authoritative nameserver for your public
domain (so DNS records live declaratively on cube alongside
the rest of your config), follow the steps below.
a. Edit /lib/ndb/local — add three blocks:
Root-server hints (for cube's recursive resolver; verify
current IPs from IANA before pasting):
dom=
ns=A.ROOT-SERVERS.NET
... (M.ROOT-SERVERS.NET)
dom=A.ROOT-SERVERS.NET ip=198.41.0.4
... (12 more)
SOA for baddcafe.com:
dom=baddcafe.com soa=
refresh=3600 ttl=3600
ns=ns1.baddcafe.com
ns=ns2.baddcafe.com
mb=postmaster@baddcafe.com
Public-facing A records (use your public IP, not LAN IP):
ip=<public-ip> dom=baddcafe.com
ip=<public-ip> dom=www.baddcafe.com
ip=<public-ip> dom=ns1.baddcafe.com
ip=<public-ip> dom=ns2.baddcafe.com
Update the ipnet stanza so LAN clients use cube as DNS:
dns=<cube-lan-ip>
b. Edit /cfg/cube/cpurc — change `ndb/dns -r` to
`ndb/dns -s`. This enables the UDP/53 listener; cube
walks the root hints itself for non-authoritative
queries. TCP/53 is served by the shipped
/rc/bin/service/tcp53 stub via the existing aux/listen.
c. fshalt -r and verify after reboot:
dig @<cube-lan-ip> baddcafe.com SOA +short
dig @<cube-lan-ip> google.com A +short # tests recursion
d. Add NAT rules at your gateway router for WAN UDP/53 +
TCP/53 → cube (see Router config appendix).
e. At your registrar, register glue records for ns1 and ns2
(subdomain nameserver hostnames + IP), then change the
NS records to point to those nameservers.
f. Wait for propagation (1-60 minutes typical, up to 48h
worst case for resolvers stuck on old NS). Verify
externally with a public probe (check-host.net or
similar) since macOS-from-inside-the-LAN dig may be
intercepted by your ISP's transparent DNS proxy.
Caveat: under this setup, cube becomes a public-facing
nameserver. By default ndb/dns also recurses for incoming
queries (open recursive resolver - DDoS amplification
hazard, modest for a small zone). Hardening options
discussed in man/8/ndb (-R flag) and in your gateway
firewall (drop WAN-side UDP/53 with RD bit set).
Caveat 2: vanilla Plan 9 ndb/dns returns NOTIMP for
unsupported record types (CAA, TLSA, SVCB, etc.) - non-RFC
behavior per RFC 8659 Section 6.2. Modern public CAs require CAA
queries to return NOERROR and refuse to issue certificates if
they get NOTIMP back. Practical effect: with vanilla ndb/dns
serving your zone authoritatively, you cannot obtain a cert
from any public CA.
Resolved locally: small patch to `dnserver.c` returning
NOERROR + SOA for in-area queries on unsupported RR types
(RFC 1035 + RFC 3597 + RFC 2308 compliant). Roughly 20-line
diff against `/sys/src/cmd/ndb/dnserver.c` at the
`!rrsupported(...)` arm; expected to land in 9legacy
stable. Apply this patch (build + `mk install` in
`/sys/src/cmd/ndb`, then `fshalt -r` to restart ndb/dns
cleanly) before attempting step 24 (TLS). Verify with
`dig CAA baddcafe.com @<cube-lan-ip>` returning NOERROR +
SOA in the authority section.
24. (Optional) TLS for baddcafe.com via Let's Encrypt.
Plan 9 4E ships libsec with TLS 1.2 RSA-CBC. This step
obtains a publicly-trusted cert from Let's Encrypt and
deploys it so cube serves HTTPS on port 443.
Prerequisites:
- Step 23 done (cube is authoritative DNS).
- dnserver.c CAA patch applied (caveat 2 in step 23).
- bootes secstore provisioned (step 17).
- Gateway router forwards WAN tcp/80 + tcp/443 to cube.
- Domain registrar has glue records for ns1/ns2 of
baddcafe.com pointing at your public IP.
Hybrid challenge: HTTP-01 for the apex (one TXT-at-a-time
sidesteps a multi-TXT response bug in `dn.c` rrattach1
that deduplicates two TXT records with the same name);
DNS-01 for the wildcard SAN (mandatory per ACME spec for
`*.baddcafe.com`).
On macOS, dry-run via Let's Encrypt staging first (prod
has hard rate limits):
sudo certbot certonly \
--staging \
--manual \
--preferred-challenges http,dns \
--cert-name baddcafe.com-staging \
--domains 'baddcafe.com,*.baddcafe.com' \
--key-type rsa --rsa-key-size 2048 \
-m <email> --agree-tos
Certbot pauses for each challenge. For DNS-01: append a
`dom=_acme-challenge.baddcafe.com / txtrr=<value>` stanza
to cube's `/lib/ndb/local`, then `echo refresh
>/net/dns` to make ndb/dns pick up the change without a
process restart. For HTTP-01: write the token at
`/usr/web/.well-known/acme-challenge/<token>` on cube
(httpd serves dot-prefix paths fine). Verify propagation
externally before pressing Enter — macOS dig/curl from
inside the LAN may miss hairpin-NAT issues; use a public
probe (check-host.net or similar) for ground truth.
After staging succeeds end-to-end, run the same command
minus `--staging` and with `--cert-name baddcafe.com`
(distinct cert-name avoids clobbering the staging tree;
prod and staging share the same archive layout). Then
`sudo certbot delete --cert-name baddcafe.com-staging` to
drop the staging tree.
25. Stage cert files for transfer to cube.
LE provides leaf (cert.pem) and intermediate (chain.pem)
as separate files in `/etc/letsencrypt/live/baddcafe.com/`
— Plan 9 wants them separate (no concatenation needed).
Convert the private key from PKCS#8 (OpenSSL 3.x default)
to PKCS#1 RSAPrivateKey (auth/asn12rsa input format):
mkdir -p ~/cert
sudo cp /etc/letsencrypt/live/baddcafe.com/cert.pem \
~/cert/cert.pem
sudo cp /etc/letsencrypt/live/baddcafe.com/chain.pem \
~/cert/chain.pem
sudo openssl rsa \
-in /etc/letsencrypt/live/baddcafe.com/privkey.pem \
-outform DER -traditional \
-out ~/cert/key.der
sudo chown $USER:staff ~/cert/{cert,chain}.pem \
~/cert/key.der
chmod 600 ~/cert/key.der
`-traditional` is load-bearing on OpenSSL 3.x — the
default encoding is PKCS#8 which `auth/asn12rsa` won't
parse.
Transfer ~/cert/{cert.pem, chain.pem, key.der} to cube by
whatever path you have available — USB stick, drawterm
paste, 9p import (`9p write` from plan9port), or a u9fs
bridge. The cube-side procedure below assumes the three
files land at /tmp/<name> on cube; adjust paths if you
stage them elsewhere.
26. Convert key + load secstore (cube, as bootes).
Stage the asn12rsa output in `ramfs -p`, NOT plain /tmp.
Plan 9's /tmp is fossil-backed; plaintext private keys
written there survive reboots and may be captured by
fossil snaps. ramfs(4)'s `-p` flag makes the memory
private (noswap + /proc-private) so the key never touches
disk and dies with the process.
ramfs -p /tmp/secret
cd /tmp/secret
auth/asn12rsa -t 'service=tls role=client owner=*' \
/tmp/key.der > key
auth/secstore -g factotum
cat key >> factotum
auth/secstore -p factotum
cd /
unmount /tmp/secret
rm /tmp/key.der # transferred-in key
`role=client` is load-bearing per the server-side lookup
in libsec (`tlshand.c` looks up the key by literal
`proto=rsa service=tls role=client`). Do NOT add
`proto=rsa` via `-t` — asn12rsa emits it itself.
`unmount` frees the only plaintext copy of the key in
memory; `rm /tmp/key.der` deletes the transferred-in DER
file. After this, the key exists only inside bootes's
secstore vault (encrypted at rest).
27. Place cert files at /sys/lib/ssl/.
mkdir -p /sys/lib/ssl
cp /tmp/cert.pem /sys/lib/ssl/cert.pem
cp /tmp/chain.pem /sys/lib/ssl/chain.pem
chmod 600 /sys/lib/ssl/cert.pem /sys/lib/ssl/chain.pem
rm /tmp/cert.pem /tmp/chain.pem
Mode 0600 bootes:sys is correct: httpd reads cert files
in main() before becomenone() drops privs, so post-drop
`none` doesn't need read access. /sys/lib/ssl/ is the
canonical location per `man/2/pushtls` FILES.
28. Boot-time factotum load — edit /cfg/cube/cpurc.
Find the `auth/secstored` line and add the following
immediately after it:
auth/secstore -n -G factotum | read -m \
>>/mnt/factotum/ctl
Note `>>` (append, single ctl write) — not `>`
(truncate, breaks factotum ctl semantics). `-n` = no
prompt (uses NVRAM password from step 13). `-G` =
get-and-print to stdout. `read -m` joins lines into a
single message that factotum's ctl interprets as one
`key proto=rsa ...` command.
Result at next boot: secstored starts, this line fetches
the encrypted factotum blob from the vault, decrypts via
NVRAM key, pipes the result to factotum's ctl device.
factotum now holds the TLS key with no operator
interaction.
29. httpd TLS flags — edit /cfg/cube/cpustart.
Replace the single line from step 22:
ip/httpd/httpd -c /sys/lib/ssl/cert.pem \
-C /sys/lib/ssl/chain.pem
`-c` flips httpd from HTTP to HTTPS (the listener binds
443 instead of 80 when a cert is present); also supplies
the leaf cert. `-C` supplies the intermediate chain so
clients can build a trust path to the root CA. Plan 9
splits leaf and chain where nginx/apache concatenate.
30. Reboot and verify.
fshalt -r
Same-arch warm reboot. Cube returns in 30-60 s. After
boot:
cat /mnt/factotum/ctl | grep 'service=tls role=client'
# expect: key proto=rsa service=tls role=client ...
ps | grep httpd # expect: none ... httpd
From a compatible-client host (macOS curl with LibreSSL/
SecureTransport, browsers using platform TLS, Plan 9
native libsec, older OpenSSL <3.0):
curl -v --resolve baddcafe.com:443:<cube-lan-ip> \
https://baddcafe.com/
Expect: TLSv1.2 / cipher AES256-SHA, issuer Let's
Encrypt, verify ok, HTTP 200.
External port-443 reachability (genuine external probe):
curl -sS \
'https://check-host.net/check-tcp?host=baddcafe.com:443'
Caveat: modern OpenSSL 3.x strict mode (e.g. brew openssl,
check-host.net's HTTPS probe) rejects with "unsafe legacy
renegotiation disabled" because Plan 9 4E libsec predates
RFC 5746 (2010) and doesn't send the `renegotiation_info`
extension on the initial handshake. Compatible clients
handshake fine; this is an accepted limitation of the
canonical Plan 9 TLS scope.
Cleanup:
- Wipe macOS staging files (key.der is the prod private
key in plaintext): `shred -u ~/cert/key.der || rm -P
~/cert/key.der; rm ~/cert/{cert,chain}.pem`.
- Strip the `_acme-challenge` stanza from cube's
/lib/ndb/local and `echo refresh >/net/dns`.
- `rm -r /usr/web/.well-known` on cube.
- Renewal cadence: LE certs expire after 90 days.
Track the expiry date and repeat steps 24-30 before
then. LE sends warning emails 20/10/1 days before
expiry to the address registered with certbot.
FINAL CHECKLIST
---------------
File-server users:
bootes exists in /adm/users.
bootes is in sys, adm, upas.
Auth users:
bootes exists in /adm/keys.
NVRAM:
authid=bootes, authdom=baddcafe.
Machine key matches /adm/keys's encryption key.
Secstore key set.
Startup files:
/cfg/cube/cpurc starts ip/ipconfig, ndb/dns -r, auth/keyfs,
auth/cron, auth/secstored, and aux/listen -t.
/rc/bin/cpurc is unchanged from the shipped 9legacy tree.
Listener:
/rc/bin/service.auth/tcp567 exists (renamed from
authsrv.tcp567).
nc to cube:567 from outside succeeds.
NDB:
ndb/query sys cube returns the expected tuple.
ndb/ipquery sys cube ... shows inherited ipnet attributes.
ndb/query -f /lib/ndb/auth hostid bootes returns the rule.
Secstore:
/adm/secstore exists, mode 0770, owner bootes.
/adm/secstore/who/bootes exists with `exp` and `PAK-Hi` lines.
Verifier password matches the NVRAM secstore key from
step 13.
Web (plain HTTP, optional):
/usr/web exists, owner bootes, group web, mode 0775.
/usr/web/index.html exists, world-readable.
/lib/namespace.httpd exists (can be empty).
/cfg/cube/cpustart contains `ip/httpd/httpd`.
Remote `curl http://<cube-ip>/` returns the test page.
DNS authoritative (optional, when running step 23):
/lib/ndb/local has SOA for baddcafe.com, root-server
hints, and public A records at your WAN IP.
/cfg/cube/cpurc starts `ndb/dns -s`.
Gateway router forwards WAN UDP/53 + TCP/53 to cube.
Registrar has glue records for ns1/ns2 and NS records
pointing to them.
External `dig @<wan-ip> baddcafe.com SOA` returns your SOA.
External resolver `dig @1.1.1.1 baddcafe.com A` returns
your public IP.
dnserver.c CAA patch applied (CAA queries return NOERROR
+ SOA in authority section, not NOTIMP).
TLS (optional, when running steps 24-30):
/etc/letsencrypt/live/baddcafe.com/{cert,chain,fullchain,
privkey}.pem exist on macOS (root-owned).
Cube /sys/lib/ssl/{cert,chain}.pem exist, mode 0600
bootes:sys.
Cube secstore vault has `factotum` entry:
ls -l /adm/secstore/store/bootes/factotum
/cfg/cube/cpurc has the boot-time secstore->factotum line
after `auth/secstored`.
/cfg/cube/cpustart invokes httpd with `-c` and `-C` flags.
cat /mnt/factotum/ctl shows the TLS key:
key proto=rsa service=tls role=client owner=* size=2048
From a compatible-client host:
curl --resolve baddcafe.com:443:<cube-lan-ip> \
https://baddcafe.com/
# expect: 200 OK with cipher AES256-SHA, verify ok
External port 443 reachable (check-host.net or similar).
macOS staging files wiped post-deploy (~/cert/key.der in
particular — it's the prod private key in plaintext).
Cube /usr/web/.well-known/ removed and _acme-challenge
stanza stripped from /lib/ndb/local.
Cert expiry date noted for renewal (LE 90-day cycle).
ROUTER CONFIG (MikroTik example)
--------------------------------
This is the router-side configuration the per-host setup above
assumes is in place. Generic to RouterOS-based MikroTiks; adapt
field names for other router platforms.
Find your public IP:
/ip/dhcp-client/ print # look at WAN client
# or via web UI: IP / DHCP Client / IP Address
DHCP leases (IP / DHCP Server / Leases) — pin LAN hosts to
fixed IPs by MAC:
192.168.88.11 <CUBE_MAC> # cube
192.168.88.12 <GNOT_MAC> # gnot terminal
192.168.88.13 <BR1_MAC> # wireless bridge near cube
192.168.88.14 <BR2_MAC> # wireless bridge near gnot
NAT rules (IP / Firewall / NAT, chain=dstnat, in-interface=<WAN>):
Chain dstnat, proto tcp, dst port 80, → 192.168.88.11:80
comment="HTTP -> cube"
Chain dstnat, proto tcp, dst port 443, → 192.168.88.11:443
comment="HTTPS -> cube"
Chain dstnat, proto udp, dst port 53, → 192.168.88.11:53
comment="DNS UDP -> cube"
Chain dstnat, proto tcp, dst port 53, → 192.168.88.11:53
comment="DNS TCP -> cube"
CLI form (RouterOS terminal):
/ip firewall nat add chain=dstnat in-interface=<WAN> \
protocol=udp dst-port=53 \
action=dst-nat to-addresses=192.168.88.11 to-ports=53 \
comment="DNS UDP -> cube"
(analogous for tcp/53, tcp/80, tcp/443)
Move router admin off port 80 (avoid colliding with HTTP NAT):
IP / Services: www, port 8080
Router admin panel now at http://192.168.88.1:8080
When step 23 (cube as authoritative DNS) is in use, also
consider hardening the WAN-facing DNS exposure — drop UDP/53
packets where the DNS header's RD (Recursion Desired) bit is
set, so cube only answers authoritative queries to the world
while still serving recursive lookups to the LAN. Specific
RouterOS syntax for this varies by firmware version; consult
MikroTik documentation for "content matcher" and "RAW filter"
features.
TESTED HARDWARE
---------------
Reference for one known-working configuration; the procedure
applies to most Plan-9-compatible x86 hardware. "Compatible"
in practice means: a kernel-supported NIC (most Intel/Realtek
gigabit chipsets), a SATA controller in AHCI mode (or older
IDE/ATAPI), and either VGA or VESA-compatible video. See
/sys/src/9/pc/etherif.h and the various sd*.c drivers for
the exact list.
Server: Supermicro X7SPA-H-D525 (Atom D525, 4 GB RAM, i386)
Terminal: ThinkPad X61 (in docking station) or ThinkPad T61
Router: MikroTik mAP 2nD (RBmAP2nD) — LAN gateway
Bridges: MikroTik mAP lite (RBmAPL-2nD) x 2 — ethernet-to-
wireless bridges for server and terminal
Bridge mode caveat — the wireless bridges must be configured as
`wlan1.mode=station-bridge`, NOT `station-pseudobridge`.
Pseudobridge does MAC NAT and only carries one downstream device
transparently; station-bridge is L2-transparent for arbitrarily
many. Symptom of getting this wrong: DHCP fails on the wired
side of the bridge.
Server kernel: cube boots `9pccpuf` (i386, self-built — see
step 9), not the amd64 `9k10cpuf`, because the amd64 9k kernel
is multiboot-only and 9load isn't a multiboot loader. The
hardware is amd64-capable; the choice is software-driven.
END
|
