CANCELLED — stream closed unexpectedly
`Status { code: Cancelled }` on an active stream almost always means the TCP connection was closed by a middlebox (Kubernetes pod restart, load balancer, NAT timeout). It's not an error in the programmer sense — it's a normal event to handle with reconnect logic.
Root causes
Ranked by frequency. First cause is the one to check first.
- 01Kubernetes rolling update of the client pod — SIGTERM gracefully closed the stream.
- 02AWS NLB or GCP L4 LB idle timeout (typically 350s) with no keepalive configured.
- 03Docker bridge network dropping idle connections after 5 minutes.
- 04Provider rolling out a new server version — brief disconnect while connections drain.
- 05Your client-side code threw inside the `for await` loop and awaited the exception without cancelling the iterator.
Fix steps
- 1
Always wrap subscribe in a reconnect loop
CANCELLED is normal on long-running streams. Treat any stream termination (CANCELLED, UNAVAILABLE, or silent EOF) as a signal to reconnect with backoff.
- 2
Enable keepalive
Configure 30s keepalive with 10s timeout. This forces the client to send PING frames that refresh NAT state and LB session tables. Without this, an idle stream dies ~5 min in on most cloud networks.
- 3
Implement a watermark for resume
Track the last processed slot. After reconnect, discard updates below that slot (or above+skip) to avoid double-processing. For bonding-curve sniping, missing 2 slots is acceptable — for arb, maintain a dedupe cache keyed on txSig.
Related errors
- UNAVAILABLE: connection refusedYour gRPC client got `Status { code: Unavailable }` with `connection refused` (or `transport is closing`). The TCP handshake never completed — either you're hitting the wrong port, TLS is misconfigured, or the endpoint is genuinely down.
- DEADLINE_EXCEEDED on Yellowstone subscribe`Status { code: DeadlineExceeded }` means the RPC didn't produce a response within the deadline you (or the default) set. On a streaming subscribe this usually means the initial handshake took too long, or the client's deadline was accidentally set on the whole stream instead of the initial call.
- RESOURCE_EXHAUSTED — rate limit on gRPC`Status { code: ResourceExhausted }` means the server refused to serve your call because you hit a limit — concurrent streams, messages-per-second, or filter complexity. Unlike HTTP 429, this is a hard rejection: the stream is closed and must be reconnected, not retried on the same channel.
Want an endpoint that just works?
Subglow is flat-priced Solana gRPC + JSON-RPC on a single API key. Pre-parsed JSON, dedicated sendTransaction bucket, 99.9% latency SLA on Dedicated. No credit juggling, no surprise bills.