Editing: Web Serial HMI + USB CDC Firmware (Octolux ↔ RP-series)

Field: Description

---
name: web-serial-hmi
description: >
  Build a paired NTX HMI (Chromium kiosk HTML on an Octolux device) +
  firmware (USB CDC on an RP2040 / RP2350) that auto-connects over USB
  with no permission picker. Covers the whole loop: project layout under
  `~/ntx-embedded/customers/<id>/`, Pico SDK USB CDC firmware template
  with line-based ASCII protocol, HMI-side Web Serial using `getPorts()`
  + VID filter + auto-connect polling, the Chromium enterprise policy
  `SerialAllowAllPortsForUrls` (the key enabler — without it
  `requestPort()` always shows a picker), the udev rule that makes
  `/dev/ttyACM*` openable by the kiosk's Chromium user, the SCP +
  chromium-restart + screencast-bounce deploy sequence, a visible
  page-side debug strip so you can diagnose what Chromium UI dialogs
  (which don't render in VNC) are doing, and a CDP-via-SSH-tunnel
  pattern for live page inspection. Includes a working firmware +
  HMI template (~80 lines of C, ~80 lines of JS) plus a failure-mode
  reference table. Scope: NTX Octolux devices + RP-series targets.
  Trigger words: build hmi firmware pair, hmi over usb, web serial
  hmi, web serial kiosk, octolux web serial, chromium serial port,
  Web Serial no devices, navigator.serial.getPorts, requestPort vs
  getPorts, SerialAllowAllPortsForUrls, DefaultSerialGuardSetting,
  no port selected by user, no compatible devices found, web serial
  picker empty, auto-connect serial port, rp2040 usb cdc, rp2350 usb
  cdc, pico_enable_stdio_usb, kiosk udev ttyACM, /dev/ttyACM
  permissions, chromium policy serial, octolux device deploy reset
  vnc, visible debug strip, CDP page reload screencast stale, port
  open failed serial, ntx hmi firmware template, new customer hmi
  project, ports[0] open fails.
---

# web-serial-hmi — NTX HMI ↔ USB CDC firmware pattern

> **Scope.** Building a Chromium-kiosk HMI on an **NTX Octolux device** that auto-connects to a **USB-attached RP2040 or RP2350** target over CDC-ACM, with **no Web Serial picker** ever showing. End-to-end working pattern, not a per-customer recipe — instantiate it for any new project under `~/ntx-embedded/customers/<id>/`.
>
> Out of scope: Android USB Host (different stack); Web Bluetooth / WebHID; non-RP-series USB devices (mostly the same pattern, just substitute the VID); CAN-FD bridges; the SWD/flash side of the firmware lifecycle — that's covered by `rp2350-flash` and the `target/rp2040.cfg` openocd patterns and the local `firmware/` build is just plain `cmake + make`.

## When to use this skill

- Starting a new customer HMI project that needs to control hardware over a USB-attached MCU.
- Porting an Android-tablet HMI app to an Octolux Linux/Chromium device and you want the same "just plug it in" feel.
- The Web Serial picker is showing — *"No compatible devices found"*, *"No port selected by user"*, or just appearing at all — and the project shouldn't have a picker.
- `requestPort()` works once but the operator has to tap-and-pick every session and that's wrong for a kiosk.
- The HMI page can't see what's happening because Chromium permission dialogs don't render in VNC.

## Architecture

```
┌─────────────────────────────────┐      USB-C       ┌──────────────────────┐
│ NTX Octolux device (10.0.10.x)  │ ◄──────────────► │ RP2040 / RP2350 board│
│ ─────────────────────────────── │ CDC-ACM, 115200  │ ───────────────────  │
│ Chromium 109 kiosk @ localhost  │   8N1 line ASCII │ Pico SDK USB CDC     │
│  ├─ Web Serial API              │                  │  ├─ getchar polling  │
│  ├─ SerialAllowAllPortsForUrls  │                  │  ├─ ASCII line parser│
│  └─ getPorts() filtered by VID  │                  │  └─ GPIO drive       │
│ Apache at /                     │                  │                      │
│ /dev/ttyACM0 (root:root, 0666)  │                  │                      │
└─────────────────────────────────┘                  └──────────────────────┘
        ▲                                                       ▲
        │  SSH from container (deploy + udev + policy)          │  SWD via PiProbe (flash)
        │                                                       │
   Adom Docker container                                  see rp2350-flash
   ~/ntx-embedded/customers/<id>/{hmi,firmware}
```

Two physical links from the workcell to the target board:

- **USB-C** between Octolux and target — runtime command path (this skill).
- **SWD** between the workcell's PiProbe and the target — flash path (`rp2350-flash`).

The container is the build host for both: HMI HTML edits go SCP-ed to Octolux; firmware ELF is built locally and flashed via the PiProbe.

## Required components

A working NTX HMI+firmware pair has **five** moving parts. Skip any one and connect won't happen. Pieces 3 and 4 are one-time setup *per Octolux device*; pieces 1, 2, 5 are per-project.

| # | Layer | Where it lives | Per-… |
|---|---|---|---|
| 1 | **Firmware** — USB CDC stdio, command parser, GPIO drive | `customers/<id>/firmware/src/main.c` (in container) | project |
| 2 | **HMI HTML** — connect logic + UI | `customers/<id>/hmi/index.html` (in container, deployed to device) | project |
| 3 | **Chromium policy** — `SerialAllowAllPortsForUrls` | `/etc/chromium/policies/managed/octolux-serial.json` (on device) | device |
| 4 | **udev rule** — `MODE="0666"` for VID 0x2e8a | `/etc/udev/rules.d/99-rp-cdc-acm.rules` (on device) | device |
| 5 | **Deploy** — SCP HMI + chromium restart + screencast bounce | shell pattern run from container | project |

## Quick start — new project, end to end

Run from the Adom container. Substitute `<id>` (e.g. `acme-control`), `<device-ip>` (e.g. `10.0.10.98`), and your domain commands.

### 1. Scaffold

```bash
ID=acme-control
DEVICE_IP=10.0.10.98
DEVICE_MODEL='Octolux 15.6"'
VIEW_W=1920 VIEW_H=1080

mkdir -p ~/ntx-embedded/customers/$ID/{hmi,firmware/src}

cat > ~/ntx-embedded/customers/$ID/project.json <<EOF
{
  "id": "$ID",
  "name": "Acme Control",
  "customer": "Acme",
  "summary": "One-line description of what this HMI does.",
  "accent": "#e87a1e",
  "accentDeep": "#a64f10",
  "deviceModel": "$DEVICE_MODEL",
  "viewport": { "w": $VIEW_W, "h": $VIEW_H },
  "orientation": "landscape",
  "defaultIP": "$DEVICE_IP",
  "tagline": "one-line operator-facing tagline"
}
EOF
```

### 2. Drop in the firmware + HMI templates

Use the templates further down this doc verbatim, customizing only:

- **Firmware:** `FW_VERSION_STRING` (the `<NAME> FW x.y.z` boot banner), the `dispatch()` command list, and your specific GPIO pin numbers / domain logic.
- **HMI:** the visible controls (buttons, sliders, etc.), the brand colors from your `project.json`, the response-line handlers in `handleResponse()`.

Keep the Web Serial connect logic and the visible debug strip **unchanged** — that's the point of this skill.

### 3. Build + flash the firmware (first time)

```bash
export PICO_SDK_PATH=$HOME/pico-sdk
cd ~/ntx-embedded/customers/$ID/firmware
mkdir -p build && cd build && cmake .. && make -j$(nproc)

# Flash via PiProbe over SWD — pick the cfg for your chip:
openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg \
  -c "adapter speed 5000; program ${ID//-/_}.elf verify reset exit"
# (or target/rp2350.cfg for an RP2350 — see rp2350-flash)
```

### 4. One-time device setup (Chromium policy + udev rule)

Skip if this Octolux device has been set up for Web Serial before. Verify by SSH'ing in: if `/etc/chromium/policies/managed/octolux-serial.json` and `/etc/udev/rules.d/99-rp-cdc-acm.rules` both exist, you're done.

```bash
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
set -e
# Chromium policy: SerialAllowAllPortsForUrls turns getPorts() into a
# blanket enumeration for http://localhost; DefaultSerialGuardSetting=2
# also disables requestPort() pickers as a safety net.
mkdir -p /etc/chromium/policies/managed
cat > /etc/chromium/policies/managed/octolux-serial.json <<'POL'
{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
POL

# udev: /dev/ttyACM* is root-only by default on the Yocto distro, and
# the kiosk Chromium can't open it. MODE=0666 fixes that for any
# Raspberry Pi (RP2040/RP2350) USB CDC device.
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'UDEV'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
UDEV
udevadm control --reload-rules

# Re-process any ALREADY-plugged-in device so the new rule applies now.
# (Without this, the perms only apply to *future* plug events.)
BUSPORT=$(lsusb -t 2>/dev/null | awk '/Pico|2e8a/ {sub(/^[ \t]+/, ""); print}' | head -1)
echo "USB device tree row: ${BUSPORT:-(none seen)}"
# Cycle the parent USB device (substitute bus-port from `lsusb -t` if your
# topology differs; 4-1 is the common Octolux internal hub port).
echo 4-1 > /sys/bus/usb/drivers/usb/unbind 2>/dev/null || true
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind 2>/dev/null || true

systemctl restart octolux-chromium
REMOTE

# Per the [[hmi-deploy-reset-vnc]] rule: chromium restart leaves the
# screencast bound to a dead CDP target. Bounce the screencast on the
# device — see the Deploy sequence below for the exact commands.
```

### 5. Deploy the HMI to the device

See the "Standard deploy sequence" section below — it's the same shell pattern you'll run on every HMI edit.

### 6. Verify it auto-connected (CDP eval from container)

```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

python3 - <<'PY'
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def evx(expr, await_=False):
    ws.send(json.dumps({"id": 1, "method": "Runtime.evaluate",
        "params": {"expression": expr, "returnByValue": True,
                   "awaitPromise": await_}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == 1: return r["result"]["result"]["value"]
print("connected:", evx("state.connected"))
print("fwVersion:", evx("state.fwVersion"))
print("ports    :", evx("navigator.serial.getPorts().then(p => p.length)", await_=True))
PY
```

Expected once the firmware is running and the cable is plugged in:

```
connected: True
fwVersion: 1.0.0
ports    : 9          # 8 platform UARTs + your 1 RP-series CDC
```

If `connected: True` and `fwVersion` is populated → you're done; the HMI is talking to the MCU. If not, jump to the failure table.

## Firmware template (RP2040 / RP2350 USB CDC)

`firmware/CMakeLists.txt`:

```cmake
cmake_minimum_required(VERSION 3.13)
set(PICO_BOARD pico CACHE STRING "")            # or "pico2" for RP2350
set(PICO_PLATFORM rp2040 CACHE STRING "")       # or "rp2350"
include($ENV{PICO_SDK_PATH}/external/pico_sdk_import.cmake)
project(<id_underscored> C CXX ASM)
pico_sdk_init()
add_executable(<id_underscored> src/main.c)
target_link_libraries(<id_underscored> pico_stdlib hardware_gpio)
pico_enable_stdio_usb(<id_underscored>  1)      # CDC-ACM on USB-C
pico_enable_stdio_uart(<id_underscored> 0)      # no UART — USB only
pico_add_extra_outputs(<id_underscored>)
```

`firmware/src/main.c`:

```c
#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdarg.h>
#include "pico/stdlib.h"
#include "pico/stdio_usb.h"      // stdio_usb_connected()
#include "hardware/gpio.h"

#define FW_VERSION_STRING "<NAME> FW 1.0.0"
#define CMD_BUF_SIZE 96

// Status LEDs (wire your board's status LEDs to these or change as needed).
// LED_USB: solid ON while USB CDC is connected to a host.
// LED_ACT: 50 ms blink on every valid command.
// LED_ERR: solid ON when the last response was ERR; clears on next OK.
// LED_IDLE: solid ON when system is in its safe / inactive state.
#define LED_USB   9
#define LED_ACT  10
#define LED_ERR  16
#define LED_IDLE 18

static char cmd_buf[CMD_BUF_SIZE];
static int  cmd_len = 0;
static bool act_on = false;
static absolute_time_t act_off_at;

static void blink_activity(void) {
    gpio_put(LED_ACT, 1);
    act_on = true;
    act_off_at = make_timeout_time_ms(50);
}
static void say_ok(const char *fmt, ...) {
    va_list a; va_start(a, fmt);
    fputs("OK ", stdout); vprintf(fmt, a); fputs("\r\n", stdout);
    va_end(a);
    gpio_put(LED_ERR, 0);
    blink_activity();
}
static void say_err(const char *m) {
    printf("ERR %s\r\n", m);
    gpio_put(LED_ERR, 1);
    blink_activity();
}
static void to_upper(char *s) { for (; *s; s++) *s = (char)toupper((unsigned char)*s); }

// === Domain commands ====================================================
//
// Replace these with the commands your project actually needs. Keep
// the canonical built-ins (PING, VERSION, STATUS, HELP) — the HMI
// template's handshake assumes they exist.
//
static void dispatch(char *line) {
    char *tok = strtok(line, " \t");
    if (!tok) return;

if (!strcmp(tok, "PING"))    { say_ok("PONG"); return; }
    if (!strcmp(tok, "VERSION")) { printf("%s\r\n", FW_VERSION_STRING); blink_activity(); return; }
    if (!strcmp(tok, "STATUS"))  { /* print one-line state snapshot */ blink_activity(); return; }
    if (!strcmp(tok, "HELP"))    { /* print "OK …" lines */ blink_activity(); return; }

// your-domain command parsing here, e.g.:
    //   if (!strcmp(tok, "DO")) { … say_ok("DO %d", arg); return; }

say_err("unknown command");
}

// === Boilerplate main =================================================

int main(void) {
    stdio_init_all();

// Init your GPIOs here (LEDs + outputs). Drive outputs LOW first
    // so the system boots in a safe state regardless of host behavior.
    const uint leds[] = { LED_USB, LED_ACT, LED_ERR, LED_IDLE };
    for (size_t i = 0; i < sizeof(leds)/sizeof(leds[0]); i++) {
        gpio_init(leds[i]); gpio_set_dir(leds[i], GPIO_OUT); gpio_put(leds[i], 0);
    }
    gpio_put(LED_IDLE, 1);

bool was_conn = false, banner_done = false;

while (true) {
        bool conn = stdio_usb_connected();
        gpio_put(LED_USB, conn ? 1 : 0);
        if (conn && !was_conn) banner_done = false;
        was_conn = conn;

// Boot banner on every CDC connect — the HMI parses this to
        // populate the FW version pill.
        if (conn && !banner_done) {
            sleep_ms(50);                     // give the host's reader a beat
            printf("%s\r\n", FW_VERSION_STRING);
            puts("TYPE 'HELP' FOR COMMANDS");
            banner_done = true;
        }

// Drain available input into cmd_buf; dispatch on EOL.
        for (int budget = 0; budget < 32; budget++) {
            int c = getchar_timeout_us(0);
            if (c == PICO_ERROR_TIMEOUT) break;
            if (c == '\n' || c == '\r') {
                if (cmd_len > 0) {
                    cmd_buf[cmd_len] = '\0';
                    to_upper(cmd_buf);
                    dispatch(cmd_buf);
                    cmd_len = 0;
                }
            } else if (cmd_len < CMD_BUF_SIZE - 1) {
                cmd_buf[cmd_len++] = (char)c;
            } else { cmd_len = 0; say_err("command too long"); }
        }

// Activity LED timeout.
        if (act_on && absolute_time_diff_us(get_absolute_time(), act_off_at) <= 0) {
            gpio_put(LED_ACT, 0); act_on = false;
        }
    }
}
```

### Protocol conventions that work well

- **Newline-terminated ASCII** — accept `\n` and `\r\n`. Case-insensitive (uppercase before dispatch).
- **One line in, exactly one line out** — `OK …`, `ERR …`, or a single-token status line. One-to-one keeps the HMI's send/receive trivial.
- **Standard built-ins always present:**
  - `PING → PONG` (liveness)
  - `VERSION → <NAME> FW x.y.z` (capability detection + version pill)
  - `STATUS → STATE …` (one-line state snapshot)
  - `HELP → multi-line OK …` (operator self-discovery)
- **Boot banner on every CDC connect.** Print `<NAME> FW x.y.z\r\n` first thing after `stdio_usb_connected()` flips true. The HMI matches `/^([\w-]+) FW (\S+)/i` to populate the version pill.
- **LED1 (USB) mirrors `stdio_usb_connected()`** — instant visual confirmation that the host opened the port.
- **LED2 (activity) blinks 50 ms on every valid command** — independent visual signal that the firmware *parsed* a command.
- **Default-safe on boot** — all outputs LOW, critical actuators OFF, idle LED ON. Don't rely on the host to send `ALL OFF` first.

## HMI template

```html
<script>
"use strict";

const state = {
  port: null, writer: null, reader: null,
  connected: false, fwVersion: null,
  autoConnectTimer: null,
};

// ─── Visible debug strip ──────────────────────────────────────────────
// Mirror every UI event + Web Serial step into the page DOM, because
// Chromium permission dialogs (the Web Serial picker, geo prompts,
// etc.) render above the page surface and DON'T appear in the
// screencast / VNC feed. Without this strip, you'd be blind to what's
// happening on the kiosk from your dashboard.
function dbg(line) {
  const strip = document.getElementById('debugStrip');
  if (!strip) return console.log(line);
  const d = document.createElement('div');
  d.textContent = new Date().toISOString().slice(11,23) + '  ' + line;
  strip.appendChild(d);
  while (strip.children.length > 5) strip.removeChild(strip.firstChild);
}

// ─── Line-splitter transformer for the reader stream ──────────────────
function lineSplitter() {
  let buf = '';
  return {
    transform(chunk, ctrl) {
      buf += chunk;
      let i;
      while ((i = buf.search(/\r?\n/)) >= 0) {
        ctrl.enqueue(buf.slice(0, i));
        buf = buf.slice(i + (buf[i] === '\r' ? 2 : 1));
      }
    },
  };
}

async function openExisting(port) {
  await port.open({
    baudRate: 115200, dataBits: 8, stopBits: 1, parity: 'none',
    bufferSize: 256, flowControl: 'none',
  });
  state.port = port;

const enc = new TextEncoderStream();
  enc.readable.pipeTo(port.writable);
  state.writer = enc.writable.getWriter();

const dec = new TextDecoderStream();
  port.readable.pipeTo(dec.writable);
  state.reader = dec.readable
    .pipeThrough(new TransformStream(lineSplitter()))
    .getReader();

state.connected = true;
  readLoop();
  handshake();
}

async function tryAutoConnect() {
  if (state.connected || !('serial' in navigator)) return state.connected;
  const ports = await navigator.serial.getPorts();
  // The kiosk policy returns the FULL set of /dev/tty* devices the
  // kernel sees — typically ~8 platform UARTs (empty getInfo()) plus
  // your real USB CDC. Always filter; never open ports[0] blindly.
  // 0x2e8a is the Raspberry Pi VID (RP2040 + RP2350). Substitute your
  // chip's VID if you use a non-RP target.
  const rp     = ports.filter(p => (p.getInfo() || {}).usbVendorId === 0x2e8a);
  const anyUsb = ports.filter(p => typeof (p.getInfo() || {}).usbVendorId === 'number');
  const candidates = rp.length ? rp : anyUsb;
  dbg(`getPorts() → ${ports.length} total, ${rp.length} RP-series`);
  for (const port of candidates) {
    try {
      await openExisting(port);
      const info = port.getInfo();
      dbg(`auto-connected to VID:${(info.usbVendorId||0).toString(16)} PID:${(info.usbProductId||0).toString(16)}`);
      return true;
    } catch (e) {
      dbg('  open failed on one candidate: ' + e.message);
    }
  }
  return false;
}

if (!(await tryAutoConnect())) {
    // Keep polling — if the cable is plugged in *after* the page loads,
    // the next tick picks it up. Cheap (one promise every 2 s).
    state.autoConnectTimer = setInterval(async () => {
      if (await tryAutoConnect() && state.autoConnectTimer) {
        clearInterval(state.autoConnectTimer);
        state.autoConnectTimer = null;
      }
    }, 2000);
  }
});

async function readLoop() {
  while (state.reader) {
    try {
      const { value, done } = await state.reader.read();
      if (done) { state.connected = false; break; }
      if (value) handleResponse(value.trim());
    } catch { state.connected = false; break; }
  }
}

async function handshake() {
  await send('PING');
  await send('VERSION');
  await send('STATUS');
}

async function send(cmd) {
  if (!state.writer) return;
  dbg('→ ' + cmd);
  await state.writer.write(cmd + '\n');
}

function handleResponse(line) {
  dbg('← ' + line);
  const m = line.match(/^([\w-]+) FW (\S+)/i);
  if (m) state.fwVersion = m[2];
  // Your domain handlers here (parse STATUS reply, react to OK/ERR, etc.)
}
</script>
```

The HMI's HTML body and CSS are entirely up to your project — see `octolux-hmi-design` for the brand-language guide (dark mode, accent colors, touch-target sizes, Inter font). The only DOM element this template requires is `<div id="debugStrip"></div>` somewhere visible during development. You can hide it (or remove the global tap listeners) after the project ships.

## One-time Chromium policy — the single biggest enabler

> Without `SerialAllowAllPortsForUrls`, `requestPort()` shows a picker, the picker isn't visible through VNC, and the kiosk is unreliable. With it, `getPorts()` returns every kernel serial device with no user gesture.

```json
{
  "SerialAllowAllPortsForUrls": ["http://localhost"],
  "DefaultSerialGuardSetting": 2
}
```

- **`SerialAllowAllPortsForUrls: ["http://localhost"]`** — pages served from `http://localhost` (which is where the Octolux's Apache serves the HMI) get blanket permission to enumerate and open serial ports. `getPorts()` now returns the full list.
- **`DefaultSerialGuardSetting: 2`** — `2` means *"Do not allow any site to request access"*. Belt-and-suspenders: if your code accidentally calls `requestPort()`, the picker is a no-op instead of presenting UI nobody can see.

Path: `/etc/chromium/policies/managed/octolux-serial.json`. Restart Chromium after install (`systemctl restart octolux-chromium`). After restart, you **must** also bounce the screencast (see `hmi-deploy-reset-vnc`).

Verify the policy is live via CDP eval:

```javascript
navigator.serial.getPorts().then(p => p.length)
// Expected: a positive integer (typically 9 on Octolux 15.6") when an RP
// device is plugged in. Zero means the policy didn't load — re-check
// the file path and the chromium restart.
```

## One-time udev rule

The Octolux Yocto distro ships `/dev/ttyACM0` as `-rw-r--r-- root:root`. The kiosk Chromium can't open it. Fix with a rule that targets the RP-series VID:

```bash
cat > /etc/udev/rules.d/99-rp-cdc-acm.rules <<'EOF'
SUBSYSTEM=="tty", ATTRS{idVendor}=="2e8a", MODE="0666"
EOF
udevadm control --reload-rules
```

For an **already-plugged** device, the rule doesn't retroactively re-permission. Either physically unplug+replug, or cycle the USB driver from a shell:

```bash
echo 4-1 > /sys/bus/usb/drivers/usb/unbind
sleep 1
echo 4-1 > /sys/bus/usb/drivers/usb/bind
```

(`4-1` is the typical Octolux internal hub port; substitute from `lsusb -t` if your topology differs.)

Note: the Yocto distro lacks the `usb_id` udev helper, so `udevadm info /dev/ttyACM0` returns *"Unknown device: Inappropriate ioctl"*. **That's fine.** Chromium reads `getInfo().usbVendorId` from the underlying USB descriptor, not from udev's metadata database. You only need the permission rule, not metadata population.

## Standard deploy sequence

Every HMI edit goes through this. Don't skip the screencast bounce — see `hmi-deploy-reset-vnc`.

```bash
ID=acme-control
DEVICE_IP=10.0.10.98

# 1. Push the HTML
scp ~/ntx-embedded/customers/$ID/hmi/index.html \
    root@$DEVICE_IP:/usr/share/apache2/default-site/htdocs/index.html

# 2. Reload the kiosk — restart chromium pulls the new HTML AND
#    reloads any new chromium policy / udev rule deployed alongside.
ssh root@$DEVICE_IP 'systemctl restart octolux-chromium'

# 3. Wait for the kiosk to come back
sleep 10

# 4. Bounce the screencast — chromium restart killed its CDP target
#    binding and the screencast won't auto-reattach. Without this,
#    the dashboard VNC tile freezes on a pre-deploy frame.
#    NOTE: busybox `ps aux` truncates cmdlines so `ps aux | grep
#    screencast` returns NOTHING even when it's running. Walk /proc
#    instead. And NEVER walk /proc/PID/net/tcp to find port owners —
#    those views are namespace-wide and you will mass-kill systemd.
ssh root@$DEVICE_IP 'bash -s' <<'REMOTE'
for d in /proc/[0-9]*; do
  pid=${d##*/}
  grep -q 'screencast' "$d/cmdline" 2>/dev/null && kill -9 "$pid" 2>/dev/null
done
sleep 2
cd /tmp && nohup node screencast.js --quality 80 --pipeline 4 > /tmp/screencast.log 2>&1 &
REMOTE

# 5. (optional) confirm the dashboard VNC is alive — successive snaps
#    should differ unless the page is genuinely static.
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/a.jpg -w "%{size_download}\n"
sleep 2
curl -s http://$DEVICE_IP:8085/snapshot -o /tmp/b.jpg -w "%{size_download}\n"
```

If `/tmp/screencast.js` or `/tmp/node_modules/ws/` ever go missing on the device (they live in tmpfs and can be wiped by a bad kill spree), SCP them back from the container:

```bash
scp ~/ntx-embedded/apps/octolux/ui/screencast/screencast.js \
    root@$DEVICE_IP:/tmp/screencast.js
cd ~/ntx-embedded/apps/octolux/node_modules && tar cz ws | \
  ssh root@$DEVICE_IP 'mkdir -p /tmp/node_modules && cd /tmp/node_modules && tar xz'
```

## Live diagnostics — CDP from the container

The Octolux kiosk exposes Chrome DevTools at `localhost:9222`. Tunnel it back:

```bash
ssh -fN -L 9222:localhost:9222 -o ExitOnForwardFailure=yes root@$DEVICE_IP

# install python websocket client once
pip install --user --break-system-packages websocket-client
```

Then any time you want to inspect the live page state:

```python
import json, urllib.request, websocket
t = json.loads(urllib.request.urlopen("http://localhost:9222/json/list").read())
page = next(x for x in t if x.get("type") == "page")
ws = websocket.create_connection(page["webSocketDebuggerUrl"])
def cdp(method, params=None, _id=[0]):
    _id[0] += 1
    ws.send(json.dumps({"id": _id[0], "method": method, "params": params or {}}))
    while True:
        r = json.loads(ws.recv())
        if r.get("id") == _id[0]: return r
def evx(expr, await_=False):
    return cdp("Runtime.evaluate", {"expression": expr, "returnByValue": True, "awaitPromise": await_})["result"]["result"]["value"]

# Useful queries:
evx("state.connected")
evx("state.fwVersion")
evx("navigator.serial.getPorts().then(p => p.length)", await_=True)
evx("Array.from(document.querySelectorAll('#debugStrip > div')).map(d=>d.textContent).join('\\n')")

# Force a reload (and remember: bounce screencast afterwards!):
cdp("Page.reload", {"ignoreCache": True})
```

### Hit-test if a tap doesn't fire the button

```javascript
const b = document.getElementById('btnConnect');
const r = b.getBoundingClientRect();
const hit = document.elementFromPoint(r.left + r.width/2, r.top + r.height/2);
JSON.stringify({ same: hit === b, hit_id: hit?.id, hit_tag: hit?.tagName });
```

If `same` is false, something is covering the button.

## Failure-mode reference

Every row in this table corresponds to a real failure observed during this pattern's development. Use it as the diagnostic flow.

| Symptom | Root cause | Fix |
|---|---|---|
| Picker shows *"No compatible devices found"* | Chromium policy not in place AND/OR `/dev/ttyACM*` not world-readable | Install both the policy and the udev rule; restart chromium; bounce screencast |
| Picker shows the device but every connect ends in *NotFoundError: No port selected by user* | User dismissed the picker, or `requestPort()` was called from a non-user-gesture context (programmatic CDP `.click()` triggers this) | Don't use `requestPort()` at all — use the `getPorts()` auto-connect pattern in the HMI template |
| `getPorts()` returns ports but `port.open()` throws *Failed to execute 'open' on 'SerialPort': Failed to open serial port* | You opened `ports[0]`; it's a platform UART without a backing device | Filter by `getInfo().usbVendorId === 0x2e8a` before opening |
| `getPorts().length === 0` even though `lsusb` shows the device on the kernel side | Policy file missing, malformed, or wrong path; OR Chromium wasn't restarted after the policy install; OR `/dev/ttyACM0` perms are still root-only | Re-check `/etc/chromium/policies/managed/octolux-serial.json` is valid JSON; restart chromium; `ls -la /dev/ttyACM0` should be `0666` |
| Tap on a button does nothing — no visible page reaction | Touch input might not be reaching the page; check via the debug strip (it should log `pointerdown` / `click`). If those don't appear → input layer (Weston/touchscreen) issue. If they do appear but `connect() called` doesn't → the click hit something covering the button — use `elementFromPoint` | Diagnose via debug strip + hit-test |
| The debug strip shows `connect() called → requestPort()...` and hangs — then eventually fails *"No port selected by user"* | The HMI still has `requestPort()` somewhere | Replace with `getPorts()` + VID filter (HMI template) |
| Page reloaded via CDP `Page.reload` — VNC dashboard tile froze on the pre-reload frame | The screencast bound to the old CDP target ID; reload creates a new one; the binding is dead | Bounce screencast (Standard deploy sequence step 4). Applies to ANY Chromium target change — restart, reload, navigate |
| `octolux deploy push` times out | Octolux server's deploy API is flaky on this build | Use the SCP + restart bash sequence directly (it's what the API does internally anyway) |
| `ps aux \| grep screencast` returns nothing but screencast IS running and bound to port 8085 | Busybox `ps aux` truncates cmdlines | Walk `/proc/[0-9]*/cmdline` to find PIDs by argv match |
| Killed a system PID and lost SSH / had to recover the device | You walked `/proc/[0-9]*/net/tcp` to find owners of port 8085 — those views are namespace-wide and match every PID | **NEVER** identify port owners via `/proc/PID/net/tcp` on this distro. Use `/proc/PID/cmdline` matching instead |
| `/tmp/screencast.js` or `/tmp/node_modules/ws/` missing after a bad kill / reboot | They live in tmpfs and the original placement path doesn't auto-restore | SCP from `~/ntx-embedded/apps/octolux/ui/screencast/screencast.js` and tar+SCP `~/ntx-embedded/apps/octolux/node_modules/ws/` |
| Adom HTTPS proxy returns `error code: 502` with no body when the HMI's own dashboard endpoint fails | The Adom proxy strips bodies on 5xx (Cloudflare-style stock error page) | Have the HMI server return HTTP 200 with `{ok:false, error, hint}` instead of 5xx — see `octolux-app-debug` Symptom 6 |
| Dashboard section button (VNC / Deploy / Stream) does nothing | Different issue — Hydrogen browser session disconnected or server.js handler — see `octolux-app-debug` Symptoms 5 and 6 | Not in this skill |

## Things NOT to do

- **Don't blindly `open(ports[0])`** — always filter by VID. The kiosk policy returns the full set of `/dev/tty*` and most of them are platform UARTs that won't open.
- **Don't rely on `requestPort()`** — it's user-gesture-gated and shows a picker, and the picker is invisible to VNC, and the operator shouldn't have to tap anyway.
- **Don't walk `/proc/[0-9]*/net/tcp`** to find what's bound to a port. Namespace-wide view → `xargs kill` hits systemd PIDs.
- **Don't `pkill node`** on the kiosk. node-red, the screencast, and other helpers all run as `node`. Same rule that applies on the Adom container.
- **Don't restart Chromium without resetting screencast** — see `hmi-deploy-reset-vnc`. Same applies to CDP `Page.reload` and `Page.navigate`.
- **Don't return 5xx** for app errors that need an explanatory body — the Adom HTTPS proxy strips bodies on 5xx. Return HTTP 200 with `{ok: false, error, hint}`.
- **Don't assume udev metadata is populated** on the Yocto distro — `udevadm info /dev/ttyACM0` returns *"Inappropriate ioctl"*. Doesn't matter; Chromium reads VID/PID from the USB descriptor via `getInfo()`.

## Related skills

- `hmi-deploy-reset-vnc` (auto-memory) — the screencast-bounce rule. Triggered by ANY Chromium target change.
- `octolux-hmi-design` — visual design language (dark mode, accent colors, touch targets, Inter font) for the HTML body and CSS that wraps this skill's connect logic.
- `octolux-platform` — Octolux device-side reference (Weston, Chromium kiosk launcher, Apache config, paths).
- `octolux-deploy` — the wider HMI-deploy story; this skill's deploy sequence is a more reliable shell-equivalent of `octolux deploy push`.
- `octolux-app-debug` — when the issue is the Octolux Node server / dashboard, not the HMI/firmware pair.
- `rp2350-flash` — the SWD/flash side of the firmware lifecycle. This skill is the USB-runtime side; that one is the flash side.

Cancel

You must be logged into Adom to save. Edits are applied immediately but subject to review.