slash/slash-mobile
slash/slash-mobile
3
0
Fork 0
Code Issues 6 Pull requests Projects Releases 3 Packages Wiki Activity Actions

ci: serialise Android builds and cap Gradle/npm to fit the shared host #25

Merged
mohlec merged 1 commit from ci/cap-android-build-resources into ncw/main 2026-09-01 20:06:34 +00:00
Conversation 1 Commits 1 Files changed 1 +27 -4
mohlec commented 2026-09-01 19:46:02 +00:00
Owner
Copy link

Stops a routine Android build from taking the git server down, and serialises builds so they
supersede instead of stacking.

What happened

The Android build runs on the same 16 GB host as Forgejo, Postgres and tailscaled, configured for
full runner capacity (1b8896a4e). One build already runs it hot. On 2026-09-01 a
workflow_dispatch on a branch and a push to ncw/main built the same commit concurrently:
15-min load average 228
HTTPS + SSH both stopped answering
Both builds died without reporting a result
Internal-ref audit queued behind starved 64 minutes
timeout-minutes: 60 did not reap either job - still listed running an hour past deadline, no build processes alive, 14.5 of 16 GB free

I caused the second concurrent build by dispatching one while a push build was queued. But one
build at full capacity on that host is the underlying condition; the collision only made it
visible.

Changes

  1. concurrency group, constant rather than keyed on github.ref. Per-ref grouping would
    not have prevented this - the two builds were on different refs. There is one runner, so only
    one of these builds may hold it. Tag builds are exempt from cancellation because they publish
    versioned releases and must not be superseded by a later branch push.
  2. Gradle: workers 8 -> 4, heap 6g -> 4g, --no-parallel so the RN module graph does not fan
    out. 4g is deliberately not lowered further - RN Android builds OOM below roughly that.
  3. npm: --maxsockets 4, --no-audit --no-fund, install heap 4096m -> 3072m.

Verification

Dispatched against this branch head, which sits on top of ncw/main and therefore also carries
the NtfyPushService specialUse change from #18 - so this run doubles as the compile check that
never completed for that fix.

Worth reviewing specifically: cancel-in-progress uses an expression
(${{ !startsWith(github.ref, 'refs/tags/') }}). If this Forgejo version does not evaluate
expressions there it could silently resolve falsy and never cancel, which would look identical to
working. The dispatch run is the test - if the workflow fails to parse or never supersedes, this
should become a plain true and accept that tag builds can be cancelled.

Not addressed here

  • timeout-minutes did not reap the orphaned jobs. A wedged build can hold the queue
    indefinitely. Needs its own look.
  • Moving Actions to a runner off the Forgejo host would remove the shared-fate problem
    entirely rather than rationing around it. That is the durable fix; this PR is the containment.
Stops a routine Android build from taking the git server down, and serialises builds so they supersede instead of stacking. ### What happened The Android build runs on the same 16 GB host as Forgejo, Postgres and tailscaled, configured for full runner capacity (`1b8896a4e`). One build already runs it hot. On 2026-09-01 a `workflow_dispatch` on a branch and a push to `ncw/main` built **the same commit** concurrently: | | | |---|---| | 15-min load average | **228** | | HTTPS + SSH | both stopped answering | | Both builds | died without reporting a result | | Internal-ref audit queued behind | starved **64 minutes** | | `timeout-minutes: 60` | **did not reap either job** - still listed running an hour past deadline, no build processes alive, 14.5 of 16 GB free | I caused the second concurrent build by dispatching one while a push build was queued. But one build at full capacity on that host is the underlying condition; the collision only made it visible. ### Changes 1. **`concurrency` group, constant rather than keyed on `github.ref`.** Per-ref grouping would *not* have prevented this - the two builds were on different refs. There is one runner, so only one of these builds may hold it. Tag builds are exempt from cancellation because they publish versioned releases and must not be superseded by a later branch push. 2. **Gradle**: workers 8 -> 4, heap 6g -> 4g, `--no-parallel` so the RN module graph does not fan out. 4g is deliberately *not* lowered further - RN Android builds OOM below roughly that. 3. **npm**: `--maxsockets 4`, `--no-audit --no-fund`, install heap 4096m -> 3072m. ### Verification Dispatched against this branch head, which sits on top of `ncw/main` and therefore also carries the `NtfyPushService` `specialUse` change from #18 - so this run doubles as the compile check that never completed for that fix. Worth reviewing specifically: `cancel-in-progress` uses an expression (`${{ !startsWith(github.ref, 'refs/tags/') }}`). If this Forgejo version does not evaluate expressions there it could silently resolve falsy and never cancel, which would look identical to working. The dispatch run is the test - if the workflow fails to parse or never supersedes, this should become a plain `true` and accept that tag builds can be cancelled. ### Not addressed here - **`timeout-minutes` did not reap the orphaned jobs.** A wedged build can hold the queue indefinitely. Needs its own look. - **Moving Actions to a runner off the Forgejo host** would remove the shared-fate problem entirely rather than rationing around it. That is the durable fix; this PR is the containment.
ci: serialise Android builds and cap Gradle/npm to fit the shared host
All checks were successful
Internal-reference audit / internal-refs (pull_request) Successful in 17s
Details
cf6f726795 
The Android build runs on the same 16 GB host that serves Forgejo, Postgres
and tailscaled, and it was configured for full runner capacity (1b8896a4e).
One build already runs that host hot; two do not fit at all.

On 2026-09-01 a workflow_dispatch on a branch and a push to ncw/main built
the same commit concurrently. The 15-minute load average reached 228, HTTPS
and SSH both stopped answering, both builds died without ever reporting a
result, and the internal-ref audit queued behind them starved for 64
minutes. Neither run was reaped by timeout-minutes: 60 - Forgejo still
listed both as running an hour past their deadline, with no build processes
alive and 14.5 GB of the 16 GB free.

Three changes:

  - concurrency group, constant rather than keyed on github.ref. Per-ref
    grouping would not have prevented this incident, because the two builds
    were on different refs. There is one runner; only one of these builds
    may hold it. Tag builds are exempt from cancellation since they publish
    versioned releases.
  - Gradle: workers 8 -> 4, heap 6g -> 4g, and --no-parallel so React
    Native's module graph does not fan out. 4g is deliberately not lowered
    further; RN Android builds OOM below roughly that.
  - npm: --maxsockets 4 to cap concurrent fetches, --no-audit --no-fund to
    drop work CI has no use for, and install heap 4096m -> 3072m.

Not addressed here, and worth its own decision: timeout-minutes did not
reap the orphaned jobs, so a wedged build can hold the queue indefinitely.
Moving Actions onto a runner separate from the Forgejo host would remove
the shared-fate problem entirely rather than rationing around it.
mohlec merged commit 28db7c6d15 into ncw/main 2026-09-01 20:06:34 +00:00
mohlec deleted branch ci/cap-android-build-resources 2026-09-01 20:06:34 +00:00
mohlec commented 2026-09-01 20:07:48 +00:00
Author
Owner
Copy link

Correction to this PR description: the dominant cause is storage latency, not memory or CPU

I framed this PR as "two builds do not fit in 16 GB" and sized the caps accordingly. Having
actually measured the host during a build, that attribution was wrong, and I would rather correct
it here than let it become the received explanation.

What the host is: an LXC container, 24 cores, 16 GB RAM with no swap, and the root
filesystem on /dev/rbd2 - a Ceph RBD network block device, currently 80% full. Because the
job runs runs-on: native, the build shares that device with Forgejo, Postgres and the git repos
themselves.

What it looks like mid-build:

procs      -----io----   -system--  ------cpu-----
 r  b        bi    bo    in     cs   us sy id wa
 6  6      4986 35386 31289  45280   8  1 77 13
 9  3      5344 11712 23746  47783   3  1 84 12
  • CPU is 77-84% idle during an active build. It is not CPU-bound.
  • 3-6 processes permanently blocked in uninterruptible I/O.
  • 45,000+ context switches per second.
  • Pressure stall info: io some avg10=12.46 against cpu some avg10=5.51 - I/O stalls are
    roughly 2.3x CPU stalls.
  • Block-device counters work out to roughly 67 ms of accumulated wait per write versus ~9 ms
    per read. (/sys/block write_ms includes queueing, so treat it as an upper bound rather than
    pure per-op latency.) Local NVMe would be ~0.1 ms.

Why this profile is pathological here. npm install for React Native unpacks tens of
thousands of tiny files, each one a metadata operation crossing the network to the Ceph cluster;
Gradle, Kotlin and the NDK then layer large caches on top. Every write costs milliseconds instead
of microseconds, so workers sit in D state.

And load average counts blocked-on-I/O processes, not just runnable ones. That is the entire
explanation for a 15-minute load average of 228 on a box whose CPUs were mostly idle: a queue
waiting on storage, not a CPU stampede. It is also why the git service itself stopped answering -
same block device.

Aggravating, but secondary: 24 cores invites tools to scale parallelism to core count while 16 GB
cannot feed it, and with no swap a memory spike is fatal rather than slow.

What this PR does and does not fix

The changes here are still worth having, and they are cheap - the verification build ran in
8 min 18 s with workers at 4 and a 4 GB heap, on a host that stayed in single-digit load
throughout. Serialising builds halves device contention, and --maxsockets 4 plus fewer Gradle
workers genuinely reduce concurrent I/O.

But none of it reduces the latency per write. This is containment, not a fix.

What would actually fix it, in order of leverage

  1. Move the runner workspace and GRADLE_USER_HOME onto local SSD/NVMe scratch, off Ceph.
    This converts the bulk of build I/O from network round-trips to local writes. Far and away the
    biggest win.
  2. Stop rewriting node_modules every build. --prefer-offline avoids the network fetch but
    still writes every file; a cache keyed on package-lock.json removes most of the small-file
    storm.
  3. Move Actions onto a runner separate from the Forgejo host. Better justified now than when I
    first raised it: the contention is not just CPU and RAM, it is the same block device as the git
    repositories and the database.
  4. Add swap, and reclaim disk headroom (80% full worsens thin-provisioned write latency).
    Mitigations, not fixes.

The in-file comments added by this PR still describe the sizing in terms of fitting alongside
Forgejo and Postgres on a 16 GB host. That is true but incomplete; happy to follow up with a small
commit rewording them to point at storage latency instead.

### Correction to this PR description: the dominant cause is storage latency, not memory or CPU I framed this PR as "two builds do not fit in 16 GB" and sized the caps accordingly. Having actually measured the host during a build, that attribution was wrong, and I would rather correct it here than let it become the received explanation. **What the host is:** an LXC container, **24 cores**, **16 GB RAM with no swap**, and the root filesystem on **`/dev/rbd2` - a Ceph RBD network block device**, currently 80% full. Because the job runs `runs-on: native`, the build shares that device with Forgejo, Postgres and the git repos themselves. **What it looks like mid-build:** ``` procs -----io---- -system-- ------cpu----- r b bi bo in cs us sy id wa 6 6 4986 35386 31289 45280 8 1 77 13 9 3 5344 11712 23746 47783 3 1 84 12 ``` - **CPU is 77-84% idle during an active build.** It is not CPU-bound. - 3-6 processes permanently **blocked** in uninterruptible I/O. - 45,000+ context switches per second. - Pressure stall info: `io some avg10=12.46` against `cpu some avg10=5.51` - I/O stalls are roughly **2.3x** CPU stalls. - Block-device counters work out to roughly **67 ms of accumulated wait per write** versus ~9 ms per read. (`/sys/block` write_ms includes queueing, so treat it as an upper bound rather than pure per-op latency.) Local NVMe would be ~0.1 ms. **Why this profile is pathological here.** `npm install` for React Native unpacks tens of thousands of tiny files, each one a metadata operation crossing the network to the Ceph cluster; Gradle, Kotlin and the NDK then layer large caches on top. Every write costs milliseconds instead of microseconds, so workers sit in D state. **And load average counts blocked-on-I/O processes, not just runnable ones.** That is the entire explanation for a 15-minute load average of 228 on a box whose CPUs were mostly idle: a queue waiting on storage, not a CPU stampede. It is also why the git service itself stopped answering - same block device. Aggravating, but secondary: 24 cores invites tools to scale parallelism to core count while 16 GB cannot feed it, and with no swap a memory spike is fatal rather than slow. ### What this PR does and does not fix The changes here are still worth having, and they are cheap - the verification build ran in **8 min 18 s** with workers at 4 and a 4 GB heap, on a host that stayed in single-digit load throughout. Serialising builds halves device contention, and `--maxsockets 4` plus fewer Gradle workers genuinely reduce *concurrent* I/O. But none of it reduces the *latency per write*. This is containment, not a fix. ### What would actually fix it, in order of leverage 1. **Move the runner workspace and `GRADLE_USER_HOME` onto local SSD/NVMe scratch**, off Ceph. This converts the bulk of build I/O from network round-trips to local writes. Far and away the biggest win. 2. **Stop rewriting `node_modules` every build.** `--prefer-offline` avoids the network fetch but still writes every file; a cache keyed on `package-lock.json` removes most of the small-file storm. 3. **Move Actions onto a runner separate from the Forgejo host.** Better justified now than when I first raised it: the contention is not just CPU and RAM, it is the same block device as the git repositories and the database. 4. **Add swap, and reclaim disk headroom** (80% full worsens thin-provisioned write latency). Mitigations, not fixes. The in-file comments added by this PR still describe the sizing in terms of fitting alongside Forgejo and Postgres on a 16 GB host. That is true but incomplete; happy to follow up with a small commit rewording them to point at storage latency instead.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
slash/slash-mobile!25
No description provided.