slash/slash-dispatch-plugin
3
0
Fork 0
Code Issues 1 Pull requests Projects Releases 3 Packages Wiki Activity Actions

fix: stop the 5am notification storm - toggles off by default, plus change detection #8

Merged
mohlec merged 5 commits from fix/notification-storm into main 2026-08-06 20:33:32 +00:00
Conversation 0 Commits 5 Files changed 5 +84 -3
mohlec commented 2026-08-06 20:07:48 +00:00
Owner
Copy link

Stops the 5am message storm, and fixes the reason it happened.

What happened

The billing sync rewrites every appointment in the active window on a schedule, and handleWebhook called notifyAppointment on every row it received - whether or not anything had actually changed. So a re-sync DM'd each tech their entire upcoming week, at 05:00 local.

Confirmed from the KV store. Counting appointment records by the hour they were last written:

2026-08-06T09    26     <- this morning's storm (09:00 UTC = 05:00 local)
2026-08-02T09    24
2026-07-26T09    29
2026-07-19T09    28
2026-07-12T09    29
2026-07-05T09    18
2026-06-28T09    18
2026-06-21T09    21
2026-06-14T09    19

Bulk writes at 09:00 UTC, every Sunday, plus today. This has been happening all along - it only became audible when the TechMap was populated and the DMs finally had somewhere to go.

Immediate mitigation already applied

dispatch-bot is disabled in production, which stops every DM path without touching the TechMap or the board. The own-column-first ordering people like is unaffected.

Note this is a stopgap, not a fix: OnActivate calls EnsureBotUser, so a plugin restart or redeploy may well re-enable the bot. This PR needs to land before the next deploy, or the storm comes back on its own.

Changes

Three toggles, all defaulting OFF (plugin.json, server/plugin.go, server/notify.go):

Setting Controls
NotifyNewAppointment DM on a newly assigned appointment
NotifyChanges DM on update / cancellation
NotifyReminders the 30-minute-before reminder

Defaulting off means the deploy itself is the fix - nothing to remember to switch off afterwards.

Change detection (server/store.go, server/api.go) - the part that makes these safe to ever turn back on:

apptFingerprint() hashes the fields a human would notice, deliberately ignoring UpdatedAt (rewritten on every save) and Action. handleWebhook now reads the stored record before writing and skips the DM when the fingerprint is unchanged. Cancellation only notifies if it was not already cancelled.

The board still updates on every webhook exactly as before - only the DM is gated.

Deliberately not done

No quiet hours. With reminders off by default it is not urgent, and I did not want to guess your working hours. Worth adding before NotifyReminders is ever switched on: a 06:00 appointment still means a 05:30 message, and that is a smaller version of this same problem.

Verification

go build   exit 0
plugin.json parses; version 0.2.1
settings: WebhookSecret, TechsChannelId, TechMap,
          NotifyNewAppointment(false), NotifyChanges(false), NotifyReminders(false)

Version bumped to 0.2.1 so this build is distinguishable from what is running - the exact ambiguity that hid Phase 6 for a month.

Stops the 5am message storm, and fixes the reason it happened. ## What happened The billing sync rewrites **every appointment in the active window** on a schedule, and `handleWebhook` called `notifyAppointment` on every row it received - whether or not anything had actually changed. So a re-sync DM'd each tech their entire upcoming week, at 05:00 local. Confirmed from the KV store. Counting appointment records by the hour they were last written: ``` 2026-08-06T09 26 <- this morning's storm (09:00 UTC = 05:00 local) 2026-08-02T09 24 2026-07-26T09 29 2026-07-19T09 28 2026-07-12T09 29 2026-07-05T09 18 2026-06-28T09 18 2026-06-21T09 21 2026-06-14T09 19 ``` Bulk writes at 09:00 UTC, every Sunday, plus today. This has been happening all along - it only became audible when the TechMap was populated and the DMs finally had somewhere to go. ## Immediate mitigation already applied `dispatch-bot` is **disabled** in production, which stops every DM path without touching the TechMap or the board. The own-column-first ordering people like is unaffected. Note this is a stopgap, not a fix: `OnActivate` calls `EnsureBotUser`, so a plugin restart or redeploy may well re-enable the bot. **This PR needs to land before the next deploy**, or the storm comes back on its own. ## Changes **Three toggles, all defaulting OFF** (`plugin.json`, `server/plugin.go`, `server/notify.go`): | Setting | Controls | |---|---| | `NotifyNewAppointment` | DM on a newly assigned appointment | | `NotifyChanges` | DM on update / cancellation | | `NotifyReminders` | the 30-minute-before reminder | Defaulting off means the deploy itself is the fix - nothing to remember to switch off afterwards. **Change detection** (`server/store.go`, `server/api.go`) - the part that makes these safe to ever turn back on: `apptFingerprint()` hashes the fields a human would notice, deliberately ignoring `UpdatedAt` (rewritten on every save) and `Action`. `handleWebhook` now reads the stored record **before** writing and skips the DM when the fingerprint is unchanged. Cancellation only notifies if it was not already cancelled. The board still updates on every webhook exactly as before - only the DM is gated. ## Deliberately not done **No quiet hours.** With reminders off by default it is not urgent, and I did not want to guess your working hours. Worth adding before `NotifyReminders` is ever switched on: a 06:00 appointment still means a 05:30 message, and that is a smaller version of this same problem. ## Verification ``` go build exit 0 plugin.json parses; version 0.2.1 settings: WebhookSecret, TechsChannelId, TechMap, NotifyNewAppointment(false), NotifyChanges(false), NotifyReminders(false) ``` Version bumped to **0.2.1** so this build is distinguishable from what is running - the exact ambiguity that hid Phase 6 for a month.
The billing system re-sends every appointment in the active window on a
schedule (observed: bulk writes at 09:00 UTC, weekly). Every one of those
rows called notifyAppointment, so a re-sync DM'd each tech their entire
week at 5am. Compare a fingerprint of the stored record against the
incoming one and skip the DM when nothing a human would notice changed.
Cancellation only notifies if it was not already cancelled.
mohlec merged commit 93d33f6f7a into main 2026-08-06 20:33:32 +00:00
mohlec deleted branch fix/notification-storm 2026-08-06 20:33:32 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
slash/slash-dispatch-plugin!8
No description provided.