---
name: octolux-app-debug
description: >
  Debug the Octolux Node.js app server (the thing on port 8830 that serves
  the Dashboard, Projects, Deploy, VNC, Screencast, Remote sections). Use
  for both startup failures (dashboard tab won't load, `octolux health` is
  failing, port 8830 is unbound, server crashes silently) and runtime
  feature bugs (section buttons on the dashboard not opening, click does
  nothing on first click, every click silently fails, tab created but
  invisible, Hydrogen workspace mutations returning 409, server claiming
  success but no tab actually appearing). Covers a generic diagnostic flow
  (proxy `ECONNREFUSED` is ALWAYS upstream — read the server log, not the
  proxy), six documented symptoms with root causes and fixes (including
  the `cli()` wrapper bug that silently dropped Hydrogen 409s and the Adom
  HTTPS proxy 5xx-body-stripping gotcha that requires HTTP 200 with
  `{ok:false}` instead of 5xx), the "do not do these" rules (e.g. never
  `pkill node` — VS Code runs as Node and you will kill the container
  session), and prevention guidance for cutting recurrence on fresh
  containers.
  Scope: the Octolux app server only. NOT device-side issues (Weston,
  Chromium kiosk, HMI won't boot) — those belong in `octolux-platform`.
  NOT push-to-device flow — that's `octolux-deploy`.
  Trigger words: octolux server not running, octolux dashboard error,
  octolux dashboard blank, octolux dashboard won't load, octolux health
  failing, octolux app server, octolux server.js crash, octolux 8830,
  ECONNREFUSED 0.0.0.0:8830, octolux econnrefused, octolux ws module not
  found, ERR_MODULE_NOT_FOUND octolux, octolux first run, octolux fresh
  container, octolux app crashes on startup, octolux server log, where do
  octolux logs go, can't reach octolux dashboard, octolux EADDRINUSE,
  octolux port already in use, octolux silent crash, octolux vnc button
  doesn't work, octolux dashboard button does nothing, clicking VNC opens
  nothing, octolux deploy button broken, octolux section button first
  click, dashboard tab created but invisible, hydrogen open-or-refresh
  active-tab, hydrogen browser session disconnected, hydrogen 409, no
  browser session is connected, hydrogen workspace mutations failing,
  refresh hydrogen editor tab, octolux dashboard every click does nothing,
  octolux server returns ok true but nothing happens, adom-cli silent
  failure, execFile error swallowed, dashboard 502 stripped body, adom
  proxy body stripping, octolux toast error.
---

# octolux-app-debug — debug the Octolux app server

> **Scope.** This skill covers the **Octolux Node.js app server** in `~/ntx-embedded/apps/octolux/` — the thing that serves the Dashboard, Projects, Deploy, VNC, Screencast, and Remote sections on port 8830. Device-side issues (HMI won't boot, Chromium kiosk crashes, Weston compositor, audio, GPIO) belong in the `octolux-platform` skill. Deploy / push-to-device flow belongs in `octolux-deploy`.

## When to use this skill

- User reports the Octolux Dashboard webview is blank, shows a proxy error, or won't load.
- `octolux health` returns anything other than `OK: Octolux server is running`.
- Nothing is bound to port 8830 (`ss -ltn | awk '$4 ~ /:8830/'` is empty).
- User says "the dashboard isn't loading," "the octolux server keeps dying," "port 8830 isn't listening," "ECONNREFUSED on the dashboard."

## Diagnostic flow — start here, every time

**The Hydrogen dashboard tab's `ECONNREFUSED 0.0.0.0:8830` is *always* upstream of the proxy.** It never means the proxy itself is broken; it means the server on the other side of the proxy is not listening. Diagnose upstream:

```bash
# 1. Is the server process alive?
octolux health

# 2. If not, did it crash on startup? Read the log.
cat /tmp/octolux-server.log    # or wherever the server's stdout was redirected

# 3. Is anything on the port?
ss -ltn | awk '$4 ~ /:8830/'

# 4. Did npm install actually run?
ls ~/ntx-embedded/apps/octolux/node_modules/ws 2>/dev/null \
  && echo "ws present" || echo "ws MISSING — run npm install"
```

The log is the source of truth. `octolux health` only tells you *that* the server is dead, not *why*. Skipping straight to "restart it" without reading the log is the #1 mistake — you'll restart it into the same crash and be back where you started.

## Symptoms

Each symptom below has a recognition pattern, the actual root cause, and the fix.

### Symptom 1 — `ECONNREFUSED 0.0.0.0:8830` on the dashboard (and `node_modules/` is missing)

The fresh-container default. Documented in detail because most first-time users hit it.

**Recognition**

| Layer | What you see |
|---|---|
| Webview | Dashboard tab shows only `connect ECONNREFUSED 0.0.0.0:8830` |
| CLI | `octolux health` → `ERROR: Octolux server is not running` |
| Process | Nothing on port 8830 |
| Server log | `Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'ws' imported from .../server.js` |

**Root cause.** `~/ntx-embedded/apps/octolux/node_modules/` was never populated — `ws` and other runtime deps are missing because the container doesn't pre-run `npm install` inside `apps/octolux/`, and there is no top-level `package.json` at `~/ntx-embedded/` that would cover it.

**Fix.**

```bash
cd ~/ntx-embedded/apps/octolux
npm install --no-audit --no-fund

nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &

octolux health
#   → OK: Octolux server is running (octolux v1.0.0)
```

Then refresh the Dashboard tab:

```bash
adom-cli hydrogen webview open-or-refresh --name "Octolux" \
  --url "$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')dashboard/" \
  --panel-id "$(adom-cli hydrogen workspace tabs 2>/dev/null \
      | python3 -c 'import sys,json; print(json.load(sys.stdin)["tabs"][0]["panelId"])' 2>/dev/null)"
```

### Symptom 2 — `EADDRINUSE 0.0.0.0:8830` (or server log says the port is taken)

**Recognition.** `octolux health` says the server isn't running, but `ss -ltn` shows something IS on port 8830. Starting a new server.js gets `Error: listen EADDRINUSE: address already in use 0.0.0.0:8830`.

**Root cause.** A stale `node server.js` from a previous shell, or a different process that grabbed 8830.

**Fix.** Find the specific PID and kill only that one — **never** `pkill node` (see *Do not do these*):

```bash
pgrep -af 'node.*octolux/server.js'
# → 12345 node /home/adom/ntx-embedded/apps/octolux/server.js
kill 12345
# verify the port is free, then start fresh:
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
```

If `ss -ltn` shows 8830 held by a non-`node` process, that's a different problem — figure out what it is before killing it.

### Symptom 3 — `octolux health` is OK but the dashboard is still blank

**Recognition.** Server says it's running. `ss` confirms it's listening on 8830. But the webview tab is still empty or stuck.

**Root cause.** Almost never the server. Two likely culprits:
- The Hydrogen webview is caching the old `ECONNREFUSED` page and hasn't reloaded.
- The browser-side JS in the dashboard page is failing (a CSP / cross-origin / proxy-rewrite issue).

**Fix.** Refresh the tab via `adom-cli hydrogen webview open-or-refresh` (it does a hard reload). If still blank after that, capture the panel for inspection:

```bash
adom-cli hydrogen screenshot panel --name "Octolux" --reason "dashboard blank despite healthy server"
```

Read the screenshot for browser-side errors (often visible as a small text overlay in the corner) before assuming the server is bad.

### Symptom 4 — server starts, then crashes a few seconds later

**Recognition.** `octolux health` is briefly OK, then immediately starts returning errors again. Or `nohup` shows the process exited shortly after launch.

**Root cause.** The server got past dependency loading and crashed inside its own init code — typically a missing env var, a malformed config file, or an uncaught exception inside a route handler that fires on startup.

**Fix.** Tail the log live to catch the actual error:

```bash
tail -f /tmp/octolux-server.log
# in another shell:
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
```

Then fix whatever the log surfaces — there is no canonical fix here, it depends on the specific exception.

### Symptom 5 — section buttons (VNC / Deploy / Stream / Camera / Projects) do nothing on first click

A runtime feature bug, not a startup failure. The server is healthy but the dashboard's per-device toolbar buttons appear dead on first interaction. Documented 2026-05-29.

**Recognition.** `octolux health` is OK; the dashboard webview loads and renders the device cards. But clicking a section button on a device card — VNC, Deploy, Stream, Camera — or the top-bar Projects button does nothing the first time. The button looks like it depresses, then nothing visible happens. Curling the underlying endpoint directly returns success:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
# → {"ok":true,"action":"created","tab":"Octolux VNC"}
```

The server *does* create the tab. The user just can't see it.

**Root cause.** In `~/ntx-embedded/apps/octolux/server.js`, the `/dashboard/api/open-app` handler creates the section tab via `adom-cli hydrogen webview open-or-refresh` but only foregrounds it (via `active-tab`) when the tab already existed. On a *first* click, the new tab is created in the same Hydrogen pane as the dashboard (the server correctly resolves the dashboard's pane via the `panelState.url` heuristic) but is placed *behind* the active dashboard tab. The dashboard stays foreground, so the user concludes the click did nothing.

The original code (look around line 723 of `server.js`):

```javascript
} else {
  const args = ['hydrogen', 'webview', 'open-or-refresh', '--name', targetName, '--url', appUrl];
  if (panelId) args.push('--panel-id', panelId);
  await cli(...args);
  if (tabExists) await cli('hydrogen', 'workspace', 'active-tab', '--name', targetName);  // ← bug
  action = tabExists ? 'navigated' : 'created';
}
```

**Fix.** Always call `active-tab` after `open-or-refresh`, regardless of `tabExists`:

```javascript
} else {
  const args = ['hydrogen', 'webview', 'open-or-refresh', '--name', targetName, '--url', appUrl];
  if (panelId) args.push('--panel-id', panelId);
  await cli(...args);
  // Always foreground the tab — a newly created tab is placed behind whichever tab
  // is already active in the same pane (i.e. the Dashboard), so the user sees no
  // change without this explicit activation.
  await cli('hydrogen', 'workspace', 'active-tab', '--name', targetName);
  action = tabExists ? 'navigated' : 'created';
}
```

Then restart the server with the canonical "kill specific PID, never `pkill`" pattern:

```bash
OCTOLUX_PID=$(pgrep -af 'node.*octolux/server\.js' | awk 'NR==1{print $1}')
[ -n "$OCTOLUX_PID" ] && kill $OCTOLUX_PID
sleep 1
nohup node ~/ntx-embedded/apps/octolux/server.js > /tmp/octolux-server.log 2>&1 &
octolux health
```

**Verification.** After the restart, click any section button and confirm `isActive`:

```bash
adom-cli hydrogen workspace find-tab "Octolux VNC"
# → "isActive": true
```

**Why this was easy to miss.** Once you've clicked a section button once in a session, the tab persists. Future clicks hit the `tabExists` branch and *do* call `active-tab`, so the button "works" — only the very first click is dead. If your dev workflow always opens a section before testing, you never see the bug. This is the kind of regression that wants a smoke test that runs against a *fresh* container state where no section tabs exist yet.

**Scope of the bug.** Affects **all six** sections that route through `/dashboard/api/open-app`: VNC, Deploy, Stream (Screencast), Camera, plus the top-bar Projects button. It was reported as a VNC issue, but the fix applies to the whole class.

### Symptom 6 — section buttons do nothing on *every* click because the Hydrogen browser session is disconnected

Same surface symptom as Symptom 5, completely different root cause. If Symptom 5's fix is in place and clicking a section button *still* does nothing — including subsequent clicks, not just the first one — this is the case. Documented 2026-05-29.

**Recognition.** All section buttons silently fail every time, not just the first click. `octolux health` is OK. The server logs `[open-app] failed: …` lines and the response body — when read against `localhost:8830` directly — contains:

```text
adom-cli hydrogen webview open-or-refresh ... failed:
  {"error": "add-tab returned 409: ..."}
  {"error": "No browser session is connected.
            The editor tab may need to be refreshed.",
   "hint":  "Workspace mutations require a live browser session
             to take effect. Try refreshing the Hydrogen editor tab."}
```

Diagnostic command:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
# → {"ok":false, "error":"...409 No browser session...", "hint":"Refresh..."}
```

**Root cause (the environmental one).** Hydrogen workspace mutations (`webview open-or-refresh`, `workspace add-tab`, `workspace active-tab`) require a **live browser-side connection** to Hydrogen — i.e., your editor browser tab open and connected to its workspace WebSocket. When that browser session drops (tab closed, network blip, sleep / wake), every workspace mutation from anywhere in the container fails with HTTP 409 from `hydrogen.adom.inc`.

**Root cause (the server bug that hid it).** The original `cli()` wrapper in `~/ntx-embedded/apps/octolux/server.js` `/dashboard/api/open-app` handler resolved with `{err, stdout}` but callers only destructured `stdout` — so subprocess failures were silently dropped and the handler returned `{"ok":true,"action":"created"}` even when nothing had actually been created. The 409 from Hydrogen was completely invisible to the dashboard until you patched the wrapper.

**Fix part 1 — make `cli()` actually surface failures** (the server change):

```javascript
// Before — silently drops errors:
const cli = (...args) => new Promise(r => {
  execFile('adom-cli', args, { timeout: 8000 }, (err, stdout) => r({ err, stdout }));
});

// After — fails loudly when the CLI exited non-zero OR when stdout JSON
// contains an `error` field (Hydrogen 409 surfaces this way: exit-0,
// JSON body {"error": "..."}). Either case rejects so the handler can
// return a proper error to the dashboard.
const cli = (...args) => new Promise((resolve, reject) => {
  execFile('adom-cli', args, { timeout: 8000 }, (err, stdout, stderr) => {
    let parsed = null;
    try { parsed = JSON.parse(stdout || ''); } catch { /* not JSON */ }
    if (err) {
      const msg = (parsed && parsed.error) || (stderr && String(stderr).trim()) || err.message;
      return reject(new Error(`adom-cli ${args.join(' ')} failed: ${msg}`));
    }
    if (parsed && parsed.error) {
      return reject(new Error(`adom-cli ${args.join(' ')} failed: ${parsed.error}`));
    }
    resolve({ stdout, parsed });
  });
});
```

Wrap the body of the handler in `try { ... } catch (e) { return json(res, 200, { ok: false, error: e.message, hint: ... }); }`. Detect the "No browser session is connected" substring and surface a user-facing hint pointing at the editor-tab refresh.

**Fix part 2 — return 200 with `ok:false`, NOT 5xx** (the proxy gotcha):

The Adom container's HTTPS proxy at `*.adom.cloud` strips response bodies on 5xx (Cloudflare-style stock error page). If the server returns `HTTP 502 { ... }` the dashboard only sees `error code: 502` and the hint never reaches the user. Always return HTTP 200 with `{ ok: false, error: ..., hint: ... }` for application-level failures. The dashboard JS must key off `body.ok`, not `r.ok`:

```javascript
const r = await fetch(base + '/api/open-app', { ... });
const body = await r.json().catch(() => ({}));
if (!r.ok || body.ok === false) {
  showToast(`Couldn't open ${action.toUpperCase()}`, body.error, body.hint);
}
```

**Fix part 3 — show the toast on the dashboard** (the visible part):

Add a small toast renderer to `ui/dashboard/index.html` so the user sees the title + error + hint instead of a button that looks dead. The hint is the key payload — it tells them to refresh the editor tab.

**The user-side recovery action.** Once these patches are in place, the user-side fix is always the same: refresh the Hydrogen editor browser tab (right-click → reload, or `Cmd/Ctrl-R`). That re-establishes the workspace session, and the next click works.

**Why this kept reproducing in our session.** Symptom 5's fix to always-call-active-tab was correct, but Hydrogen had silently disconnected (e.g., the editor tab paused in the background or lost its WebSocket). With the old `cli()` wrapper, every `add-tab` / `open-or-refresh` after that point looked like it succeeded — including on the very first click and every subsequent click — so the surface symptom "button looks dead" persisted even though the *original* root cause was fixed. The lesson: **don't ignore subprocess errors**, and **don't rely on 5xx semantics through a body-stripping proxy**.

**Verification.** After applying parts 1+2+3:

```bash
PROXY=$(echo $VSCODE_PROXY_URI | sed 's/{{port}}/8830/')
curl -s -X POST "${PROXY}dashboard/api/open-app" \
  -H "Content-Type: application/json" -d '{"app":"vnc"}'
```

When Hydrogen is disconnected, the response body must contain `"ok":false`, a real `error` string, and `"hint":"Refresh the Hydrogen editor browser tab ..."`. Status code stays 200 so the proxy passes the body through.

## Other ERR_MODULE_NOT_FOUND variants

`ERR_MODULE_NOT_FOUND` for any package name, not just `ws`, has the same root cause as Symptom 1 (missing `node_modules/`). The fix is identical: `npm install` in `~/ntx-embedded/apps/octolux/`. If npm install completes successfully but a *different* module is still missing on the next start, the app's `package.json` is incomplete or out of sync with what `server.js` actually imports — file it as a bug against the app rather than working around it locally.

## Do NOT do these

- **Do not `pkill node`** — VS Code Server runs as Node in this container; killing all Node processes kills the editor and effectively ends the container session (only an Adom admin can restart it). Always find the specific Octolux server PID with `pgrep -af 'node.*octolux/server.js'` and kill only that PID.
- **Do not bounce the container's HTTPS proxy** in response to `ECONNREFUSED 0.0.0.0:8830` — the proxy is working; it's accurately reporting an upstream failure. The fix is on the Octolux server side.
- **Do not assume Octolux deploy or device-side issues share root causes** with app-server issues — they have separate skills (`octolux-deploy`, `octolux-platform`).
- **Do not "fix" the dashboard by restarting the Hydrogen tab repeatedly** — if `ECONNREFUSED` is reproducing, the tab is doing its job. Diagnose the server first.

## Preventing recurrence

Every new container hits Symptom 1 on first dashboard open. Options for cutting that recurrence (ordered cheapest → most invasive):

1. **Document it in `~/.claude/CLAUDE.md` for this project.** Add a line under "Quick Start" or a new "First-run setup" section saying explicitly: *"Run `cd ~/ntx-embedded/apps/octolux && npm install` before the first `octolux health` on a new container."* Cheap, no infrastructure change, surfaces to every Claude thread that loads the project context.
2. **Auto-install on `octolux health` if `node_modules/` is missing.** Modify the `octolux` CLI's `health` command so that, before reporting "server not running," it checks for `node_modules/ws` and if absent prints `HINT: deps are not installed. Run \`cd ~/ntx-embedded/apps/octolux && npm install\` first.` — converts the current bare error into a self-diagnosing one. The CLI already prints a `Hint:` line for the start command; this is the same pattern.
3. **Add the npm install to the container bootstrap.** The gallia installer (`~/gallia/install.mjs`) runs on every fresh container — adding a step that `npm install`s any `~/ntx-embedded/apps/*` that has a `package.json` would eliminate the symptom entirely. More invasive (changes shared infrastructure), but the right long-term fix.
4. **Have `server.js` print a useful error before crashing.** Wrap the top-level imports in a try/catch that detects `ERR_MODULE_NOT_FOUND` and prints `MISSING DEPS — run npm install in this directory` before re-throwing. Less helpful than option 2 because users still have to read the log.

Option 1 is what we did today (via this skill itself, which now ships in the wiki). Option 2 is the right next step if the issue keeps biting people.

## Why this happens (architecture)

The Adom Docker container ships the `~/ntx-embedded/` repo cloned in place but does **not** pre-run `npm install` inside `apps/octolux/`. There is no top-level `package.json` at `~/ntx-embedded/` that could cover it — the Octolux app's `package.json` is local to its own directory.

When the user opens the Octolux Dashboard for the first time via `adom-cli hydrogen webview …`, Hydrogen opens a tab against the in-container HTTPS proxy at `…/proxy/8830/dashboard/`. The proxy hits `0.0.0.0:8830`, finds nothing listening, and renders the error page with body `connect ECONNREFUSED 0.0.0.0:8830`.

If the user follows the `octolux health` hint and runs `node server.js`, Node's ESM resolver fails on the first `import 'ws'` and exits before any port is bound. A `nohup`'d background start swallows the crash output — the user sees the same dashboard error and concludes "the start command didn't work," when in fact the server briefly ran and died.

The fix is therefore always **(a) read the server log to confirm what actually crashed it, then (b) install the missing piece, then (c) start the server again** — never just "try the start command harder."

## File map

| Path | What |
|---|---|
| `~/ntx-embedded/apps/octolux/server.js` | The Node.js app server entry point |
| `~/ntx-embedded/apps/octolux/package.json` | The app's Node dep manifest |
| `~/ntx-embedded/apps/octolux/node_modules/` | Created by `npm install` — must exist for server to start |
| `/tmp/octolux-server.log` | Recommended stdout/stderr redirect for the detached server |
| `/usr/local/bin/octolux` | CLI for `octolux health`, `octolux projects list`, `octolux deploy …`, etc. |
| `/usr/local/bin/adom-cli` | Hydrogen webview / screenshot driver used to open the Dashboard tab |

## Related skills

- `octolux-platform` — device-side issues (Weston, Chromium kiosk, boot sequence, audio, GPIO). Use when the HMI device itself misbehaves, not when the app server misbehaves.
- `octolux-deploy` — push HTML to a device, manage orientation, the deploy API. Use after the server is up and you're trying to ship an interface.
- `octolux-hmi-design` — design the HTML/CSS for an HMI surface.
- `adom-workspace-control` — open / refresh the Octolux Dashboard panel from any agent.