Incident notes and pitfalls from a one-person homelab — a security engineer's actual infrastructure, written up as it breaks and gets fixed. No vendor pitch, no tutorial fluff.
Run by one operator, no teamStack Debian · systemd · Suricata · FastAPIAds / trackingnone
iOScampus, 12,000 people
Windowsregional office
NOXone desk, one lamp
Method
The map you remember is not the network you have
I thought I knew my own lab. Asked to write its layout down, I did it from memory and a hand drawing — and got several things wrong. Not the big shape, but the parts that decide how safe it actually is. Only measurement caught them, and it caught them one correction at a time.
A wall I thought was there was not. I had a device marked in my head as translating addresses — a boundary, its own little world behind it. It was doing no such thing. It was bridging: the machines wired to it sat on the same network as everything upstream, not behind another wall. I had inferred the boundary from how traffic behaved; the device's own mode setting said the opposite. Everything I concluded downstream of that assumption was built on a wall that was not there.
"Unreachable" meant "I did not open the other door." I declared a whole segment unreachable and moved on — the tidy, defeated conclusion. It was not isolated. I simply had not brought up the interface that reaches it. A second radio, on a profile marked so it can never become the default route (no risk of quietly sending the internet the wrong way), and the segment was just there. The machines I had written off as gone answered on the first probe.
A diagram is a claim. A measurement is a fact. The distance between the two is exactly where the surprises live — and on your own network, those are the surprises you can least afford to wave away.
The one machine I could not reach was the point. There is a host built to be a target, deliberately buried behind another layer with no wireless of its own. From every remote path I had, it stayed silent. That is not a failed measurement — it is the result. For that machine, being reachable would have been the defect. Verifying isolation and failing to connect are the same event seen from two sides.
The thread holding all three together is how you name a machine. Not by its address — addresses drift with every lease, and when two segments renumber themselves overnight, yesterday's list is fiction. You name it by its cryptographic host key, which does not move. When the numbers had all changed, the key was the only thing still telling me which machine I was actually talking to — and, in two cases, quietly correcting which one I thought it was.
None of this was a network problem. It was a knowledge problem: the confident internal map, drawn once and trusted since. The fix is not cleverer tooling. It is refusing to let the drawing be the evidence — and going to look.
Pitfall
The page was perfect. It just never started.
A small interactive page went live: a terminal you can type into, with a second panel below that explains what each command would mean to a defender. Locally it worked. Published, it rendered correctly and did absolutely nothing — no cursor, no output, no reaction to any key. The markup was there, the styles were there, the script was there in the served source. Nothing ran.
The instinct is to suspect the code. It was easier than that, and worse: the code was fine and it was never asked to run at the moment it expected.
The page initialised on DOMContentLoaded, which is the conventional and usually correct choice. But the site sits behind a CDN with an optimisation that defers and re-schedules inline scripts to speed up first paint. By the time the deferred script was activated, DOMContentLoaded had already fired — and an event that has already fired never fires again for a listener added afterwards. The listener sat there, correct and permanently unreached.
# fragile: assumes this code runs before the document finishes loading
window.addEventListener('DOMContentLoaded', init);
# robust: run now if the document is already past that point
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', init);
} else {
init();
}
There is no error when this happens. No exception, no warning, no failed request — the console is clean, because nothing went wrong. A function simply waited for a moment that had already passed. Silent non-execution is harder to notice than a crash, and it looks identical to "the feature was never written".
The optimisation can also be turned off per script, which is worth doing for anything interactive rather than presentational — most CDNs honour an attribute on the tag for exactly this. Belt and braces: opt the script out and write the readiness check, because the opt-out lives in someone else's configuration and can change without you.
The wider lesson is about where a bug can live. This one was not in the application, not in the server, and not in the browser: it was in a transformation applied between them, by a layer added for an unrelated reason. When something works locally and not in production, the difference is rarely the code — it is everything that sits in the path that the local version does not have.
Incident
There was a third one
A few days ago I wrote up a confusion between two services that shared a name: one defined at the system level, one in the user session, with only the second one actually serving. The conclusion was tidy, the fix was applied, and the lesson — ask the port who owns it — seemed learned. It was not tidy. It was one layer short.
While disabling the redundant unit, a routine check printed the control group of whatever process was holding the port. It named a service that had not appeared anywhere in the previous investigation: a third unit, under a different name entirely, running since the previous afternoon. Neither of the two services I had spent an evening comparing was serving anything. Both were failing to bind, because the port had been taken by something I never thought to look for — I had been comparing two candidates and never asked whether the answer was outside the list.
# the line that named a service nobody was looking for
ps -o cgroup= -p <pid holding the port>
0::/user.slice/.../app.slice/a-completely-different.service
The previous write-up was right about the mechanism and wrong about the participants. Two services with one name is a real failure mode and the reasoning held. It simply described the wrong two. A correct explanation of the wrong facts is the most durable kind of error, because everything about it feels finished.
The consequence was not cosmetic. Earlier that evening I had edited the configuration and restarted "the service" to apply it — and then confirmed the change by watching a number on the page move in the right direction. The process actually serving that page had been started the previous afternoon and had never read the new configuration. The number moved for its own reasons. I had accepted a coincidence as verification of a change that had not taken effect.
What makes this worth publishing is how ordinary the mistake is. The check I ran was the right one; I just ran it on a name instead of on a port, and then confirmed the outcome by looking at an indicator that had other reasons to move. Two small shortcuts, each defensible on its own, and together they produced an hour of confident wrongness.
The habits that came out of it are unglamorous and worth the trouble:
Start from the port, not the name. Whatever is listening is the ground truth; everything else is a claim about it. A service manager reporting a unit as healthy tells you about that unit, not about the service being provided.
Verify a configuration change by something the change itself controls. If an edit renames a label, look for the new label — not for a count that might drift on its own. A metric with independent reasons to move cannot confirm anything.
Treat the tidy conclusion as the suspect one. The two-service explanation was satisfying, which is exactly why it stopped the search. Nothing in it was false; it was just complete too early.
Three units now exist where one is needed. The redundant pair are stopped, disabled, and the system-level one is masked — so the next person to reach for the obvious command gets an error instead of silence. That error message is the real fix. Everything else was cleanup.
Incident
Every restart worked. None of them restarted anything.
The dashboard showed fewer machines online than it should. The obvious suspect was a recent network change, so the first move was to compare what the dashboard was told to poll against what actually answers — and two entries pointed at addresses that had stopped being reachable weeks earlier. Fixing those took a minute. What took the hour was everything that happened next.
After editing the configuration, the service was restarted the usual way. It reported success. The page kept serving the old data. Restart again, success again, still stale. The temptation at that point is to suspect caching, or the config parser, or the browser — three plausible theories, all wrong.
# the line that ended the guessing
ERROR: [Errno 98] error while attempting to bind on address
('0.0.0.0', 8888): address already in use
There were two services with the same name. One defined at the system level, one in the user session. The user-level one was running and holding the port. The system-level one had been failing to start ever since — restarting cleanly, failing to bind, backing off, trying again — and every "restart" command issued with elevated privileges was addressing the copy that was not serving anything.
Both halves of the confusion were true at once. The restart genuinely succeeded: the unit started, tried to bind, and exited. The page genuinely kept working: a different process under a different manager was serving it. Neither observation was wrong, and together they pointed at a conclusion that did not exist.
This is worth stating plainly because the same shape recurs anywhere two supervisors can own the same thing: a system unit and a user unit, a container and a host service, a process manager and an init script left behind from an earlier setup. The name collides, the port does not, and the tooling reports success on both sides of the misunderstanding.
The check that settles it takes one command — ask the port who owns it, then ask the owner which manager it belongs to:
ss -tlnp | grep :<port>
cat /proc/<pid>/cgroup | tail -1
# .../[email protected]/... means a user unit, not the system one
The cgroup line is the part that ends the argument. A process listed under a user slice was not started by the system manager, no matter how confidently that manager reports its own unit as healthy.
One more finding worth keeping: after the correct service finally restarted, the board showed zero machines online for about a minute before filling in. That is not a fault — the status is gathered by a background poller, and a freshly started process has not polled yet. It is, however, exactly the kind of empty screen that invites another round of unnecessary debugging. A dashboard that cannot yet distinguish "nothing is up" from "I have not looked yet" will eventually cost someone an evening.
Incident
Four alerts, four different stories, one music player
The monitor put four lines on the board within six minutes: CPU load 100%, 92%, 100%, and — from the disk daemon — 16 currently unreadable (pending) sectors. Read together they suggest an obvious conclusion: a machine grinding against a failing drive. Every layer of that conclusion turned out to be wrong, and the way each one failed is worth writing down.
The load average was real; the CPU pressure was not. Load sat above the core count while the processor was 76% idle, I/O wait was flat zero, and not a single process was in uninterruptible sleep. Load counts runnable and blocked tasks over a window; a burst of short-lived processes lifts it long after the burst is over. The number was true and the story it implied was false.
# the reading that ended the disk theory
us=16 sy=8 id=76 wa=0# nothing waiting on storage, nothing in D state
The biggest consumer was not the workload. Cumulative CPU time, not instantaneous percentage, is the honest measure of who has been busy — and by that measure the top process was a music player with an hour and forty minutes to its name, with the desktop shell second. The security tooling everyone assumes is heavy did not appear near the top. A machine can be loaded by what someone left open, not by what it is for.
Instantaneous CPU% answers "who is busy now"; accumulated CPU time answers "who has been burning the machine all day". The second question is almost always the one being asked, and the default view of most tools answers the first.
And the disk that raised the alarm is not the disk doing the work. The drive reporting unreadable sectors has 28,000 hours on it — over three years of continuous spinning — but it is a secondary data disk, near-empty, with a few thousand reads to its name. The system itself runs from a solid-state drive with an order of magnitude more traffic and a clean report. The warning was accurate. Its importance was not what it looked like.
The SMART numbers are worth separating, because two of them are routinely confused. Pending sectors are reads the drive could not complete and has not yet given up on: a successful re-read or an overwrite makes them disappear. Reallocated sectors are the ones it has permanently retired from its spare pool. On this drive the pending count is non-zero and the reallocated count is still zero — meaning it is struggling but has not yet lost anything. The number to watch is the second one. If it starts climbing, the drive is spending its reserve.
What makes this an incident worth recording is not the fix — there wasn't much of one — but the shape of the error. Four alerts arrived close together and appeared to corroborate each other. Correlation in a monitoring feed is not evidence; it is often just two unrelated things happening while you happen to be watching.
Incident
A counter that wraps faster than you sample
We wanted a rough idea of what six always-on machines cost to run. No meter, no budget — just the CPU energy counters every Intel chip has exposed since Sandy Bridge, under /sys/class/powercap/. The plan was a cron job reading the counter once a day and subtracting yesterday's value. That plan is wrong, and the way it is wrong produces plausible numbers rather than obvious errors.
The register is 32-bit and it wraps. Each domain publishes its own ceiling in max_energy_range_uj. On the oldest desktop here that ceiling is about 65,500 joules — which at an idle draw of roughly 16 W is reached in seventy minutes, and under load in under twenty. A daily read doesn't measure a day. It measures the remainder after an unknown number of wraps.
# the ceiling is per-machine — read it, don't assume it
cat /sys/class/powercap/intel-rapl:0/max_energy_range_uj
# sample at =< 1/4 of the wrap period; accumulate, never subtract once a day
So the sampler runs every 60 seconds and accumulates joules, with a modular delta: if the new reading is lower than the previous one, add the ceiling before subtracting. That covers the wrap. It does not cover the second trap.
Detect reboots from the kernel, not from the clock. The counter resets to zero at boot while your state file does not — so the next delta looks exactly like a wrap and silently inflates. Our first version derived a boot identifier from wall-clock time minus uptime. An NTP correction shifts that by a second or two, every run then looks like a fresh boot, the state file is discarded each time, and every delta becomes zero. The logs fill with well-formed rows of nothing. Use /proc/sys/kernel/random/boot_id: the kernel generates it once per boot and no clock adjustment can move it.
One more rounding error of judgement, found only by comparing against a direct measurement: the state-initialising sample writes a legitimate-looking 60-second interval carrying zero joules. Counted naively, it drags the day's average down. On one machine it made the difference between 15.3 W and 9.6 W — a 40% error that looked entirely reasonable on screen.
What the counters cover is CPU package and, on some generations, DRAM. Everything else — board, disks, fans, PSU losses — is outside. We store raw joules and apply a per-machine platform baseline at read time, so the estimate improves retroactively when a baseline is measured rather than guessed. On the two laptops that baseline came free: unplug AC, read power_now from the battery, and you have the true whole-machine draw. One of our guesses was off by a factor of two.
Field report
Three wrong conclusions about one Wi-Fi network
A few machines in the outbuilding were drifting between two SSIDs — sometimes on the network they belong to, sometimes on the house one. The cause was mundane: every saved profile had autoconnect enabled at priority zero, so NetworkManager picked by signal strength rather than by intent. One decoy network we keep around had been left at priority 10, which meant it outranked the network the machines were supposed to prefer.
# intent, expressed as a number
nmcli con mod "<preferred>" connection.autoconnect-priority 100
nmcli con mod "<other>" connection.autoconnect no
That part took ten minutes. The next hour was spent being wrong three times, and the three mistakes share one shape.
Wrong once: a stale address in our own notes. Testing whether a machine could reach the server, we pinged the address recorded in the documentation. Silence — which reads exactly like isolation. The address had changed months earlier. With the correct one, everything answered immediately, and the "isolated" host turned out to have an open path to a management interface.
Wrong twice: a scan that returned its own cache. Asked to list visible access points, the tool reported only two networks, not including the one we were looking for — with --rescan yes on the command line. The conclusion drawn was that the machine physically cannot see that AP. An explicit rescan followed by a few seconds of patience returned the same AP at strong signal. The flag requests a rescan; it does not wait for one.
A missing answer and a missing thing look identical. Every one of these failures produced an empty result that was read as evidence of absence. The cheap defence is a second vantage point: another machine had been associated with that "invisible" AP the whole time, which should have ended the theory before it started.
Wrong three times — and this one was self-inflicted. Applying the fix in bulk disabled autoconnect for the house network on a machine we believed had no alternative uplink. It would have stayed reachable until its next reboot, then gone silent in a building nobody visits daily. It did have a wired uplink; the interface simply hadn't appeared in the listing we checked. Both halves of that belief were wrong, and only one of them was harmless.
Moving a host between networks then surfaced a real defect. Our overlay VPN pins each peer to a list of underlay addresses. The server was listed only under the address it holds on the house network — unreachable from the other subnet. The service reported itself as active, the interface existed, the address was assigned, and the handshake timed out every thirty seconds against a peer it could no longer route to.
Adding the address the peer actually holds on the new subnet fixed it in one restart. The general form is worth keeping: when a host changes networks, the peer list of every overlay it participates in becomes a set of claims about the old topology. Nothing errors. The tunnel simply never comes up, while every status command reports success.
Field report
The dashboard that never needed a browser
The obvious way to watch a small fleet is a web dashboard: a service, a port, a frontend, a websocket, an auth layer in front of it because now it is reachable. We built one of those. The thing actually used every day is a terminal program, and the reasons turned out to be less about taste than about cost.
It has no attack surface, because it is not a service. A web dashboard is a listening port that must be authenticated, kept patched, and firewalled — for a tool whose only user is the person already holding an SSH key. The terminal version inherits the security of the channel it runs over. There is nothing new to defend.
It costs nothing to leave running. No browser tab, no node process, no persistent websocket per viewer. It draws a screen a couple of times a second from data it already has to collect anyway, and it does that over a link slow enough that a web UI would be unusable.
The part that earns its place is the event log, not the gauges. A dashboard you have to watch is a toy; the bars are pleasant and tell you nothing you would not have found later. The single most useful pane is the one that stays empty until something happens, then says which machine and what. Everything above it is context for that line.
Some design constraints the terminal imposes turn out to be improvements. Fixed-width layout forces you to decide what actually matters, because there is no room to add one more panel. Colour has to carry meaning rather than decoration — green, amber and red end up as the only three states worth having, and anything that is merely interesting stays grey. Sparklines drawn from block glyphs are coarse enough that you read them as trend rather than value, which is all you want at a glance.
Refresh cadence is the one number worth tuning rather than defaulting. Polling a fleet is not free: too fast and the collector becomes the load you are watching for; too slow and the log stops being useful for diagnosis. The panes here update on different clocks — a few seconds for process and memory, minutes for anything that has to reach across a network or read a disk.
The feature to be careful with is the one that acts. Ours can stop or restart a selected process, which is genuinely useful at 3 a.m. and genuinely dangerous with a single mistyped key. It is restricted to the local machine, the destructive variant needs a different key from the ordinary one, and the selection is always visible before it can be used. A read-only dashboard has no bad day; a dashboard with a kill command has one waiting.
One detail turned out to matter more than the interface itself: how you start it. The monitor is one program (nox-tui), but almost nobody runs it directly. A two-line launcher (nox-watch) checks which host it is on: if it is the machine that does the collecting, it execs the monitor locally; if not, it execs an SSH session to that machine and runs it there. The result is that the same command produces the same screen from any machine in the estate, and you never have to remember where you are sitting. It cost almost nothing to write and it is the reason the tool gets used daily rather than occasionally.
A live demo of this view — with invented hosts and generated numbers — runs at /monitor/: you can trigger a load spike, a disk error, an intrusion attempt or a dropped machine, and watch how each one reads on the board.
None of this argues against web interfaces — they win the moment a second person needs the view, or when the display must survive the operator's laptop being closed. It argues for asking which of those you actually have. For one operator and a handful of machines, the terminal you are already in is not a downgrade.
Field Report
Your AI coding agent trusts your vendor's website more than it should
Not one of our own incidents this time — third-party research, worth flagging because it touches the exact toolchain this journal is written with.
Google recommends that companies publish an llms.txt file at their domain root — a plain-text, machine-readable summary meant to help AI agents understand a product faster than crawling the whole site. Security researcher Alon Hertz and his team scanned 15,000 major companies, pulled 8,565 of these files across 6,214 live domains, and found 237 install instructions pointing at packages that don't exist — names referenced as setup steps, in ecosystems from PyPI to npm to crates.io, that nobody had ever published.
So they published them. Minimal packages, registered under the exact names cited in real companies' own documentation, doing nothing but phoning home when installed. A machine inside a company Hertz describes as worth "multi-hundred-billion-dollars" ran their code in under four minutes — no phishing, no intrusion, just an AI agent following instructions its own vendor had published.
15,000companies scanned
237unclaimed install targets found
<4 minto first callback
The coding agents that followed these instructions in testing included Claude, OpenAI's Codex, and Nous Research's Hermes — this is a toolchain many of us actually run, not a hypothetical. A live campaign was also found in the wild: attackers impersonated the auth vendor Clerk by registering a package name its own llms.txt referenced.
Hertz's framing is the part worth sitting with: "The trust model is broken. Agents treat vendor docs as ground truth and don't question them — and neither do the humans supervising them." A coding agent has no reason to distrust the company's own website. Neither, usually, does the developer who approved the install.
No fix is offered here beyond the obvious one: an agent proposing to install a package is still proposing to run arbitrary code on your machine, regardless of where it read the instruction. Reviewing that step doesn't stop mattering just because the source looked official.
Two unrelated-looking symptoms, one night, one root cause: a multi-homed server whose own security tooling didn't know which of its own IP addresses belonged to it.
Symptom one. The SSH log showed a successful public-key login every 30–40 seconds, forever. Not a brute-force attempt — a clean, successful login, immediately followed by the session closing. Nothing in the log looked wrong on its own: Accepted publickey, session opened, session closed. A login that succeeds isn't usually where you look for a bug.
20:51:56 sshd-session: Accepted publickey for wizardg from 192.168.88.17
20:51:56 sshd-session: session opened for user wizardg
20:51:57 sshd-session: session closed for user wizardg
20:52:34 sshd-session: Accepted publickey for wizardg from 192.168.88.17
...repeats every ~35s, indefinitely
The source, 192.168.88.17, turned out to be this machine's own primary NIC. The homelab dashboard polls a list of machines over SSH to report their status — and this box was in that list twice: once correctly marked local: true (queried in-process, no network involved), and once under its LAN IP with no such flag. It was dialing itself.
The fix was one field. Adding "local": true to the second config entry stopped the SSH polling entirely — verified: 3 logins in the 2 minutes before the fix, 0 in the 90 seconds after.
Symptom two, same night. Suricata — the IDS watching this box's traffic — was logging roughly 9,000 alerts a day, almost all "Nmap SYN scan detected," "Port scan detected," and "ICMP flood," all sourced from — again — 192.168.88.17. The dashboard's own routine network sweep (an nmap -sn ping scan across the LAN, run periodically to refresh the topology map) was tripping the IDS's own scan-detection rules.
The cause: HOME_NET in the Suricata config only listed 192.168.1.0/24 — a secondary, less-used subnet. This machine's actual primary LAN (192.168.88.0/24) was never declared "home," so by definition it fell into EXTERNAL_NET. A box scanning its own network looked, to its own IDS, exactly like an outside attacker.
# before
HOME_NET: "[192.168.1.0/24]"# after
HOME_NET: "[192.168.1.0/24,192.168.88.0/24]"
~9,000false alerts / day, before
0in a clean 2-minute window, after
1YAML line changed
The part worth admitting: my first attempt to verify the fix was wrong, too. I compared alert timestamps using UTC while Suricata's eve.json logs local time with a +02:00 offset — a naive string comparison across mismatched formats quietly let hundreds of stale, pre-fix events count as "new." The fix was real; my first proof of it wasn't. Re-running the check with matching timestamp formats gave the honest answer: zero.
Neither of these was a security incident in the usual sense — nothing got in. But both cost real time to diagnose precisely because the logs looked correct: a successful login, a well-formed alert. The lesson generalizes past this one box: on any host with more than one network interface, traffic from its own other subnet can look external to tools that only know about one of its addresses.
suricatasshmulti-homedidsfalse-positive
Pitfall
A venv doesn't know the OS upgraded Python under it
A dashboard service had been down. Not crashed — just quietly absent, for an unknown number of days, because nothing was watching it. When restarted by hand, it failed immediately:
ModuleNotFoundError: No module named 'uvicorn'
The module was, in fact, installed. uvicorn sat exactly where pip had put it — inside the virtualenv's lib/python3.13/site-packages/. The problem: the system's Python had since been upgraded to 3.14, and the venv's own interpreter symlink now pointed at 3.14 too. A 3.14 interpreter has no reason to look inside a directory named python3.13. The packages weren't missing — they were orphaned, sealed inside a version-numbered path nothing was reading anymore.
A venv's site-packages path is version-pinned at creation, not re-linked on interpreter upgrade. An in-place Python version bump silently strands every existing virtualenv, and the failure mode — "module not found" for a module that's plainly installed — doesn't hint at the real cause.
Two compounding failures made this take days to notice, not minutes:
1. No systemd unit existed for the service — it had only ever been
started by hand, so a crash was permanent until someone noticed.
2. The launch script used --reload, which spawns two processes
(a watcher + a worker) sharing one port. A prior run left a stale
process holding it, and the next start failed with:
[Errno 98] address already in use
The fix was three parts, in order: rebuild the venv against the current interpreter and regenerate a real requirements.txt (the old one didn't exist at all — freezing the working 3.13 environment first meant nothing was lost); write a systemd user unit with Restart=always and no --reload; enable it so it survives a reboot, not just a manual start.
[Service]
WorkingDirectory=%h/lab-dashboard
# --reload deliberately omitted: it forks two processes and was the
# original cause of the port collision that killed this service
ExecStart=%h/lab-dashboard/.venv/bin/python -m uvicorn main:app --host 0.0.0.0 --port 8888
Restart=always
RestartSec=10
Nothing here is exotic. A Python point-release upgrade, a venv that predates it, a service with no supervisor. Individually each is a one-line fix. Together, on a homelab where nobody's paged when something goes quiet, they add up to a service that's been dark for days before anyone happens to click the link.
