OpenTelemetry

The zero-cache service embeds the JavaScript OTLP Exporter and can send logs, traces, and metrics to any standard otel collector.

To enable otel, set the following environment variables then run zero-cache as normal:

OTEL_EXPORTER_OTLP_ENDPOINT="<your otel endpoint>"
OTEL_EXPORTER_OTLP_HEADERS="<auth headers from your otel collector>"
OTEL_RESOURCE_ATTRIBUTES="<resource attributes from your otel collector>"
OTEL_NODE_RESOURCE_DETECTORS="env,host,os"

Grafana Cloud Walkthrough

Here are instructions to setup Grafana Cloud, but the setup for other otel collectors should be similar.

  1. Sign up for Grafana Cloud (Free Tier)
  2. Click Connections > Add Connection in the left sidebar add-connection
  3. Search for "OpenTelemetry" and select it
  4. Click "Quickstart" quickstart
  5. Select "JavaScript" javascript
  6. Create a new token
  7. Copy the environment variables into your .env file or similar copy-env
  8. Start zero-cache
  9. Look for logs under "Drilldown" > "Logs" in left sidebar

Distributed Tracing

You can enable end-to-end trace correlation from your frontend through zero-cache to your API server. This allows you to see the full request flow in your tracing UI.

To enable this, provide a getTraceparent callback when creating your Zero client:

import {ZeroProvider} from '@rocicorp/zero/react'
import {propagation, context} from '@opentelemetry/api'
 
function getTraceparent() {
  const carrier: Record<string, string> = {}
  propagation.inject(context.active(), carrier)
  return carrier.traceparent
}
 
return (
  <ZeroProvider
    /* ... other options ... */
    getTraceparent={getTraceparent}
  >
    <App />
  </ZeroProvider>
)

This callback is called before sending WebSocket messages that trigger API server calls (push, changeDesiredQueries, initConnection). The returned W3C traceparent header is forwarded through zero-cache to your API server, where it can be used to continue the trace.

Metrics Reference

zero.server

MetricTypeUnitDescription
uptimeGaugesCumulative uptime, starting from when requests are served
api.requestsCounterAPI requests to user mutate, query, cleanup, and auth endpoints
api.request_durationHistogramsEnd-to-end API request duration, including retries
api.attemptsCounterHTTP fetch attempts made while calling user API endpoints
api.attempt_durationHistogramsAPI HTTP fetch attempt duration, excluding retry sleep
api.in_flightUpDownCounterAPI requests currently in flight
startup_durationHistogramsDuration from starting zero-cache to its ready signal
worker_startup_durationHistogramsDuration from starting a worker to its ready signal

zero.replica

MetricTypeUnitDescription
db_sizeGaugebytesSize of the replica's main db file (excludes WAL)
wal_sizeGaugebytesSize of the replica's WAL file
wal2_sizeGaugebytesSize of the replica's WAL2 file (only if using wal2 mode)
backup_lagGaugemsTime since last litestream backup. Expected to sawtooth from 0 to ZERO_LITESTREAM_INCREMENTAL_BACKUP_INTERVAL_MINUTES
purge_blockedCounterNumber of change-log purges blocked because the actual backup state could not be verified or is stale
litestream.restore.runsCounterLitestream restore runs
litestream.restore.attemptsCounterLitestream restore subprocess attempts
litestream.restore.db_bytesCounterbytesSQLite database bytes restored by successful Litestream restores
litestream.restore.durationHistogramsWall-clock duration of Litestream restore runs
litestream.restore.wait_durationHistogramsTime spent waiting for replication-manager snapshot status before restoring
litestream.restore.process_durationHistogramsWall-clock duration of Litestream restore subprocesses
litestream.restore.validation_durationHistogramsTime spent validating restored replica databases
litestream.backup.process_runsCounterLitestream backup process exits
litestream.backup.process_durationHistogramsRuntime duration of Litestream backup subprocesses before they exit
litestream.backup.list_durationHistogramsDuration of Litestream backup destination listing commands
litestream.backup.verification_durationHistogramsDuration of verifying actual backup state in the backup destination
litestream.snapshot.reservation_durationHistogramsDuration of snapshot reservations while view-syncers restore and subscribe

zero.replication

MetricTypeUnitDescription
upstream_lagGaugemsLatency from sending a replication report to receiving it in the stream
replica_lagGaugemsLatency from receiving a replication report to it reaching the replica
total_lagGaugemsEnd-to-end replication latency. Grows as an estimate if the next report hasn't arrived
last_total_lagGaugemsEnd-to-end latency of the most recently received report. Unlike total_lag, does not grow if reports stop arriving
eventsCounterNumber of replication events processed
transactionsCounterCount of replicated transactions
changesCounterCount of replicated changes, including DML and DDL statements
slot_healthGauge1Health of the active logical replication slot. Uses slot and status attributes
slot_retained_wal_bytesGaugebytesWAL bytes retained by the active logical replication slot
slot_safe_wal_bytesGaugebytesWAL bytes the active logical replication slot can retain before loss
initial_sync_runsCounterNumber of initial-sync runs
initial_sync_durationHistogramsWall-clock duration of an initial-sync run
initial_sync_copy_durationHistogramsWall-clock duration of the COPY phase for a successful initial-sync run
initial_sync_copy_other_durationHistogramsInitial-sync duration excluding SQLite flush and index time for a successful run
initial_sync_flush_durationHistogramsTotal SQLite flush time for a successful initial-sync run
initial_sync_index_durationHistogramsSQLite index creation time for a successful initial-sync run
initial_sync_rowsCounterRows copied during successful initial-sync runs
initial_sync_copy_streamCounterbytesPostgreSQL COPY stream bytes processed during initial sync, including in-progress and failed runs
initial_sync_completed_copy_streamCounterbytesPostgreSQL COPY stream bytes processed during successful initial-sync runs
initial_sync_copy_chunksCounterPostgreSQL COPY stream chunks processed during initial sync
shadow-sync-runsCounterNumber of shadow initial-sync runs
shadow-sync-durationHistogramsWall-clock duration of a shadow initial-sync run
flow_control.active_subscribersGaugeActive change-stream subscribers receiving live changes
flow_control.queued_subscribersGaugeChange-stream subscribers waiting for the current transaction to finish before activation
flow_control.pending_messagesGaugeDownstream change-stream messages not yet acknowledged by subscribers
flow_control.backlog_messagesGaugeLive change-stream messages buffered while subscribers catch up
flow_control.backlog_bytesGaugebytesLive change-stream bytes buffered while subscribers catch up
flow_control.max_backlog_bytesGaugebytesMaximum live change-stream bytes buffered by a single subscriber
flow_control.waitsCounterCompleted flow-control checkpoints
flow_control.wait_durationHistogramsTime replication waits at flow-control checkpoints

zero.sync

MetricTypeUnitDescription
max-protocol-versionGaugeHighest sync protocol version seen from connecting clients
active-clientsUpDownCounterNumber of currently connected sync clients
active-client-groupsGaugeNumber of active ViewSyncerService instances in a syncer worker
queriesGaugeActive IVM pipelines across all client groups in a syncer worker
rowsGaugeCVR-tracked rows across all client groups in a syncer worker
serving_lagGaugemsMaximum time active ViewSyncer client groups have had replica changes ready locally but not yet served to clients
serving_lag_statsGaugemsDistribution of serving lag across active ViewSyncer client groups
serving_lagging_client_groupsGaugeNumber of active ViewSyncer client groups with replica changes ready locally but not yet served to clients
view_syncer_lagHistogramsLag from replica-ready changes to ViewSyncer output for active client groups
view_syncer_hydrationHistogramsTime from ViewSyncer query sync requiring hydration to output for a client group
lock-wait-timeHistogramsTime spent waiting to acquire the ViewSyncerService lock per operation
pipeline-resetsCounterCount of pipeline resets
hydrationCounterNumber of query hydrations
hydration-timeHistogramsTime to hydrate a query
advance-timeHistogramsTime to advance all queries for a client group after applying a transaction
poke.timeHistogramsTime per poke transaction (excludes canceled/noop pokes)
poke.transactionsCounterCount of poke transactions
poke.rowsCounterCount of poked rows
cvr.load_attemptsCounterCVR load attempts
cvr.load_durationHistogramsTime to load a CVR
cvr.flush_attemptsCounterCVR flush attempts
cvr.flush-timeHistogramsTime to flush a CVR transaction
cvr.rows-flushedCounterNumber of changed rows flushed to a CVR
websocket.open_connectionsUpDownCounterOpen client WebSocket connections
websocket.connection_attemptsCounterClient WebSocket connection attempts
websocket.connection_successesCounterClient WebSocket connections successfully initialized
websocket.connection_failuresCounterClient WebSocket connection attempts that failed before initialization
websocket.errorsCounterClient WebSocket error events
ivm.advance-timeHistogramsTime to advance IVM queries in response to a single change
ivm.conflict-rows-deletedCounterRows deleted because they conflicted with an added row
query.transformationsCounterNumber of query transformations performed
query.transformation-timeHistogramsTime to transform custom queries via API server
query.transformation-hash-changesCounterTimes a query transformation hash changed
query.transformation-no-opsCounterTimes a query transformation was a no-op
query.row-set-signature-driftsCounterTimes unchanged query rehydration produced a different row-set signature than the CVR, forcing a config-version bump

zero.mutation

MetricTypeUnitDescription
crudCounterNumber of CRUD mutations processed
customCounterNumber of custom mutations processed
pushesCounterNumber of pushes processed