fix(hero_db): silence BrokenPipe ERROR spam on client disconnect #32

Merged

mahmoud merged 2 commits from fix/silence-broken-pipe-error into development

2026-05-19 12:31:55 +00:00

mahmoud commented

2026-05-19 12:07:15 +00:00

Owner

Summary

Two related ERROR-level log-spam bugs in hero_db, both triggered by normal client disconnects mid-response. The servers function correctly; the bugs are invisible from the UI and visible only as noise in hero_db_server / hero_db_admin logs. Matches the issue Kristof flagged in chat ("its 10 min to fix hero_db"). Omar correctly reported the admin UI works.

Bug 1 — `hero_db_server`: BrokenPipe spam (every ~5s)

crates/hero_db/src/server.rs:371 and :412 filter out UnexpectedEof and ConnectionReset but not BrokenPipe. On mahmoud-ashraf-devbox this produced:

1779189623  [ERROR] Client error: Broken pipe (os error 32)
1779189618  [ERROR] Client error: Broken pipe (os error 32)
1779189613  [ERROR] Client error: Broken pipe (os error 32)

Every 5s for 25+ hours (the admin stats poller closing mid-response). Fix: add io::ErrorKind::BrokenPipe to the filters on both the TCP and Unix code paths.

Bug 2 — `hero_db_admin`: hyper connection-error spam

crates/hero_db_admin/src/main.rs:237-241 filters out "shutting down", "connection closed", and "Connection reset" via substring match but missed "Broken pipe" and "error writing a body". Observed:

1779185647  [ERROR] Connection error: error writing a body to connection
1779189460  [ERROR] Connection error: error writing a body to connection

Fix: lowercase the comparison (so Broken pipe / broken pipe both match) and add broken pipe + error writing a body to the benign-disconnect list. Demote message wording from "during shutdown" (no longer accurate) to "Client connection closed".

Test plan

cargo check -p hero_db passes
cargo check -p hero_db_admin passes
Built release binaries on the devbox, dropped patched hero_db_server into /home/common/hero/bin/, restarted via hero_proc service.start → Broken pipe ERROR events in the last 90s: 0 (was every 5s). Server-side filter alone is sufficient; spam stays gone even with the older admin running.
Patched hero_db_admin not deployed on devbox yet — a fresh build from development HEAD throws Template error: Template 'index.html' not found (HTTP 500). Unrelated to this PR; looks like fallout from 5d9aaa5 chore: migrate to hero_admin_lib shared assets. Rolled back to the previous admin binary; admin UI is HTTP 200 again. The Bug 2 fix in this PR is still correct and merge-worthy; deploy on devbox is blocked by that separate issue.
Reviewer to confirm no regression in normal request handling (existing benign-disconnect filters still match; new ones are additive).

Follow-up

File a separate issue / PR for the hero_db_admin template-not-found regression on development so this PR can land without waiting on it.

## Summary Two related ERROR-level log-spam bugs in hero_db, both triggered by normal client disconnects mid-response. The servers function correctly; the bugs are invisible from the UI and visible only as noise in `hero_db_server` / `hero_db_admin` logs. Matches the issue Kristof flagged in chat ("its 10 min to fix hero_db"). Omar correctly reported the admin UI works. ## Bug 1 — `hero_db_server`: BrokenPipe spam (every ~5s) `crates/hero_db/src/server.rs:371` and `:412` filter out `UnexpectedEof` and `ConnectionReset` but **not** `BrokenPipe`. On `mahmoud-ashraf-devbox` this produced: ``` 1779189623 [ERROR] Client error: Broken pipe (os error 32) 1779189618 [ERROR] Client error: Broken pipe (os error 32) 1779189613 [ERROR] Client error: Broken pipe (os error 32) ``` Every 5s for 25+ hours (the admin stats poller closing mid-response). Fix: add `io::ErrorKind::BrokenPipe` to the filters on both the TCP and Unix code paths. ## Bug 2 — `hero_db_admin`: hyper connection-error spam `crates/hero_db_admin/src/main.rs:237-241` filters out `"shutting down"`, `"connection closed"`, and `"Connection reset"` via substring match but missed `"Broken pipe"` and `"error writing a body"`. Observed: ``` 1779185647 [ERROR] Connection error: error writing a body to connection 1779189460 [ERROR] Connection error: error writing a body to connection ``` Fix: lowercase the comparison (so `Broken pipe` / `broken pipe` both match) and add `broken pipe` + `error writing a body` to the benign-disconnect list. Demote message wording from "during shutdown" (no longer accurate) to "Client connection closed". ## Test plan - [x] `cargo check -p hero_db` passes - [x] `cargo check -p hero_db_admin` passes - [x] Built release binaries on the devbox, dropped patched `hero_db_server` into `/home/common/hero/bin/`, restarted via hero_proc `service.start` → **`Broken pipe` ERROR events in the last 90s: 0** (was every 5s). Server-side filter alone is sufficient; spam stays gone even with the older admin running. - [ ] **Patched `hero_db_admin` not deployed on devbox yet** — a fresh build from `development` HEAD throws `Template error: Template 'index.html' not found` (HTTP 500). Unrelated to this PR; looks like fallout from `5d9aaa5 chore: migrate to hero_admin_lib shared assets`. Rolled back to the previous admin binary; admin UI is HTTP 200 again. The Bug 2 fix in this PR is still correct and merge-worthy; deploy on devbox is blocked by that separate issue. - [ ] Reviewer to confirm no regression in normal request handling (existing benign-disconnect filters still match; new ones are additive). ## Follow-up File a separate issue / PR for the `hero_db_admin` template-not-found regression on `development` so this PR can land without waiting on it.

mahmoud added 1 commit

2026-05-19 12:07:15 +00:00

fix(hero_db): silence BrokenPipe ERROR spam on client disconnect

Build and Test / build (pull_request) Successful in 5m4s

Details

89f8601d74

The TCP and Unix client-error filters already swallow UnexpectedEof and
ConnectionReset (normal client-side disconnects) but not BrokenPipe.
A periodic admin/stats poller that closes mid-response triggers
"Client error: Broken pipe (os error 32)" every 5s in production logs.

Add io::ErrorKind::BrokenPipe to both filters.

mahmoud added 1 commit

2026-05-19 12:23:37 +00:00

fix(hero_db_admin): silence benign client-disconnect ERROR spam

Build and Test / build (pull_request) Successful in 4m7s

Details

859bc04898

The hyper serve_connection error filter caught "shutting down" /
"connection closed" / "Connection reset" but missed "Broken pipe" and
"error writing a body to connection" — both observed in production logs
as ERROR-level noise when a client disconnects mid-response.

Lowercase the comparison and add the two missing substrings so all
benign client-disconnect cases log at debug instead of error.

mahmoud merged commit 5fbbb2f638 into development

2026-05-19 12:31:55 +00:00

mahmoud referenced this pull request from a commit

2026-05-19 12:31:56 +00:00

Merge pull request 'fix(hero_db): silence BrokenPipe ERROR spam on client disconnect' (#32) from fix/silence-broken-pipe-error into development

mahmoud deleted branch fix/silence-broken-pipe-error

2026-05-19 12:31:59 +00:00

mahmoud referenced this pull request

2026-05-19 13:10:58 +00:00

hero_db_admin: HTTP 500 "Template 'index.html' not found" — templates loaded from env!("CARGO_MANIFEST_DIR") at runtime #33

mahmoud commented

2026-05-19 13:11:39 +00:00

Author

Owner

Filed the template-not-found regression that's blocking devbox deploy of the hero_db_admin half of this PR: #33. The hero_db_server fix in this PR (Bug 1) is unaffected and is fully deployed and verified on mahmoud-ashraf-devbox.

Filed the template-not-found regression that's blocking devbox deploy of the `hero_db_admin` half of this PR: #33. The `hero_db_server` fix in this PR (Bug 1) is unaffected and is fully deployed and verified on `mahmoud-ashraf-devbox`.