Skip to main content

​
Service Health Endpoints

Each service exposes a /health endpoint for liveness checks and uptime monitoring.

​
Health Check Reference

GET http://localhost:3000/health

Response:
{
  "status": "ok",
  "service": "api-gateway"
}

​
Automated Health Checks

Use the CLI doctor command to check all services at once: 
npm run cli -- doctor
This command:
  • Checks gateway reachability
  • Validates all service health endpoints
  • Reports connectivity issues
  • Returns non-zero exit code on failure (suitable for CI/CD)

​
Protocol Adapter Health

Protocol adapters expose health check endpoints to verify configuration and on-chain program deployment.

​
Check All Protocols

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/protocols/health
Response: 
{
  "protocols": [
    {
      "protocol": "escrow",
      "ok": true,
      "configured": true,
      "deployed": true
    },
    {
      "protocol": "jupiter",
      "ok": true
    }
  ]
}

​
Check Specific Protocol

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/protocols/escrow/health
Response: 
{
  "ok": true,
  "configured": true,
  "deployed": true,
  "programId": "EscrowProgramIdXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
The escrow adapter health check verifies:
  • ESCROW_PROGRAM_ID is set in environment
  • Program account exists on-chain
  • Program is executable

​
Audit Events

The audit-observability service collects and stores structured audit events for all critical operations.

​
Event Types

  • transaction.created
  • transaction.simulated
  • transaction.policy_evaluated
  • transaction.signed
  • transaction.submitted
  • transaction.confirmed
  • transaction.failed
  • wallet.created
  • agent.created
  • agent.started
  • agent.stopped
  • agent.paused
  • policy.evaluated

​
Query Audit Events

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/audit/events

​
Create Audit Event

Post custom audit events: 
curl -X POST -H 'x-api-key: dev-api-key' \
  -H 'Content-Type: application/json' \
  http://localhost:3000/api/v1/audit/events \
  -d '{
    "eventType": "custom.event",
    "message": "Custom audit event",
    "metadata": {
      "customField": "value"
    }
  }'

​
Transaction Proofs

Every transaction generates a cryptographic proof artifact for auditability and replay.

​
Proof Structure

interface ExecutionProof {
  txId: string;
  walletId: string;
  agentId?: string;
  intentHash: string;        // SHA-256 of intent payload
  policyHash: string;        // SHA-256 of policy decision
  simulationHash: string;    // SHA-256 of simulation result
  signature?: string;        // On-chain transaction signature
  proofHash: string;         // SHA-256 of combined proof
  createdAt: string;
}

​
Retrieve Transaction Proof

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/proof
Response: 
{
  "txId": "tx_abc123",
  "walletId": "wallet_xyz789",
  "intentHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "policyHash": "cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce",
  "simulationHash": "2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae",
  "signature": "5vJ8Q...",
  "proofHash": "fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9",
  "createdAt": "2026-03-08T10:30:00.000Z"
}

​
Replay Transaction

Retrieve the complete execution context for replay: 
curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/transactions/<txId>/replay
Response: 
{
  "txId": "tx_abc123",
  "request": { /* original intent */ },
  "policyDecision": { /* policy evaluation result */ },
  "simulation": { /* simulation output */ },
  "proof": { /* execution proof */ },
  "history": [ /* stage transitions */ ]
}
Use the replay endpoint to debug failures, verify policy decisions, and reconstruct transaction execution flow.

​
Metrics

The metrics store tracks counters for system events and operations.

​
Increment Metric

curl -X POST -H 'x-api-key: dev-api-key' \
  -H 'Content-Type: application/json' \
  http://localhost:3000/api/v1/metrics/inc \
  -d '{
    "name": "custom.metric",
    "value": 1
  }'

​
Get All Metrics

curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/metrics
Response: 
{
  "data": {
    "audit_event.transaction.created": 142,
    "audit_event.transaction.confirmed": 138,
    "audit_event.wallet.created": 12,
    "custom.metric": 5
  }
}

​
Built-in Metrics

The system automatically tracks:
  • audit_event.<eventType> - Count of each audit event type
  • Transaction stage transitions
  • Policy evaluation results
  • RPC pool failover events
  • Outbox retry attempts

​
RPC Pool Status

Monitor RPC endpoint health and failover behavior: 
curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/rpc/pool/status
Response: 
{
  "endpoints": [
    {
      "url": "https://api.devnet.solana.com",
      "score": 0.95,
      "successes": 142,
      "failures": 3,
      "avgLatencyMs": 230,
      "failStreak": 0,
      "lastCheckedAt": "2026-03-08T10:30:00.000Z"
    }
  ]
}

​
Health Score Calculation

  • Initial score: 1.0
  • Success: score += 0.06 - latencyPenalty
  • Failure: score *= 0.7
  • Minimum score: 0.05
Latency Penalties:
  • ≤ 200ms: no penalty
  • 201-500ms: -0.04
  • 501-1000ms: -0.08
  • 1000ms: -0.12
Endpoints are automatically sorted by score (then by latency) for failover selection.

​
Outbox Queue Status

Monitor the durable outbox queue for pending and failed transactions: 
curl -H 'x-api-key: dev-api-key' \
  http://localhost:3000/api/v1/outbox/stats
Response: 
{
  "pending": 3,
  "processing": 2,
  "failed": 0,
  "done": 145
}

​
Outbox Job Lifecycle

  1. pending - Job queued, waiting for worker
  2. processing - Worker has claimed lease
  3. done - Successfully completed
  4. failed - Exceeded max retry attempts
Jobs in processing state have an active lease. If a worker crashes, the lease expires and the job returns to pending.Configuration:
  • TX_OUTBOX_LEASE_MS - Lease duration (default: 30000ms)
  • TX_OUTBOX_MAX_ATTEMPTS - Max retries (default: 6)
  • TX_OUTBOX_POLL_MS - Worker poll interval (default: 2000ms)