Layers 1-3: Candle Cache System

The candle cache is a 3-layer system that eliminates REST API rate limits by subscribing to Bitget WebSocket once and sharing the data across all scanners.

---

Layer 1: Candle Cache Service

File: /app/trading_engine/candle_cache_service.mjs PM2: candle-cache

What It Does

1. Fetches top 200 symbols by 24h volume (≥$1M floor)
2. Opens 4 WebSocket connections (shards)
3. Subscribes each shard to 50 symbols × 6 granularities
4. Maintains rolling candle history in memory
5. Writes consolidated cache to disk every 10 seconds

Why 4 Shards?

Bitget limits subscriptions per WebSocket connection. 4 shards × 50 symbols = 200 symbols tracked reliably.

Shard Architecture

graph LR
    subgraph Shard0["Shard 0"]
        WS0["WS Connection 0"]
        S0["50 symbols"]
    end
    subgraph Shard1["Shard 1"]
        WS1["WS Connection 1"]
        S1["50 symbols"]
    end
    subgraph Shard2["Shard 2"]
        WS2["WS Connection 2"]
        S2["50 symbols"]
    end
    subgraph Shard3["Shard 3"]
        WS3["WS Connection 3"]
        S3["50 symbols"]
    end

    Bitget --> WS0
    Bitget --> WS1
    Bitget --> WS2
    Bitget --> WS3

Auto-Reconnect Logic

// On disconnect: exponential backoff 2s → 4s → 8s → max 30s
// On connect: resubscribe all symbols
// On error: log, schedule reconnect, don't crash

Cache Format (Consolidated)

{
  "updatedAt": 1714224000000,
  "candles": {
    "BTCUSDT": {
      "5m": [[ts, open, high, low, close, vol, qVol], ...],
      "1H": [[ts, open, high, low, close, vol, qVol], ...],
      "4H": [[ts, open, high, low, close, vol, qVol], ...]
    },
    "ETHUSDT": {
      "5m": [[ts, open, high, low, close, vol, qVol], ...]
    }
  }
}

History Lengths

| Granularity | Candles | Time Coverage | |————-|———|—————| | 5m | 200 | ~16 hours | | 15m | 200 | ~50 hours | | 30m | 200 | ~100 hours | | 1H | 48 | ~48 hours | | 4H | 200 | ~33 days | | 1Dutc | 30 | ~30 days |

---

Layer 2: Candle Cache Bridge

File: /app/trading_engine/scripts/candle_cache_bridge.mjs PM2: candle-bridge

Why It Exists

The consolidated cache is ~8MB. Reading the entire file for just 5m candles is wasteful. The bridge splits it into per-granularity files.

What It Does

1. Reads /app/trading_engine/data/candle_cache.json
2. Extracts each granularity into separate files in /tmp/
3. Runs every 30 seconds

Output Files

/tmp/candle_cache_5m.json     # Only 5m candles for all symbols
/tmp/candle_cache_15m.json    # Only 15m candles
/tmp/candle_cache_30m.json    # Only 30m candles
/tmp/candle_cache_1H.json     # Only 1H candles
/tmp/candle_cache_4H.json     # Only 4H candles
/tmp/candle_cache_1Dutc.json  # Only 1D candles

Format (Per-Granularity)

{
  "updatedAt": 1714224000000,
  "candles": {
    "BTCUSDT": [[ts, open, high, low, close, vol, qVol], ...],
    "ETHUSDT": [[ts, open, high, low, close, vol, qVol], ...]
  }
}

Why /tmp/?

• Faster writes (tmpfs on most systems)
• Automatic cleanup on reboot
• Scanners don’t block on large I/O

---

Layer 3: Core Reader Module

File: /app/trading_engine/core/candle_cache.mjs Usage: import { readCandleCache } from '../core/candle_cache.mjs'

API

readCandleCache(symbol, granularity)
// Returns: [[ts, o, h, l, c, vol, qVol], ...] or null

Supported Granularities

• 5m, 15m, 30m, 1H, 4H, 1Dutc

Stale-But-Valid Philosophy

// Returns stale data with warning instead of null
// Why? A 15-min old 5m candle is still useful
// A null triggers a REST API call which might 429

if (cacheAge > maxAge) {
  console.warn('[CACHE STALE] serving old data');
  return candles; // Still return them!
}

Staleness Thresholds

| Granularity | Max Age | Why | |————-|———|—–| | 5m | 10 min | 2× candle period | | 15m | 30 min | 2× candle period | | 30m | 1 hour | 2× candle period | | 1H | 2 hours | 2× candle period | | 4H | 8 hours | 2× candle period | | 1Dutc | 2 hours | Daily refresh is slow |

In-Memory Cache

// Avoids disk reads within a 5-second window
// Multiple scanners calling readCandleCache() in same cycle share memory

Error Handling

• File missing → returns null → scanner falls back to REST
• File corrupt → returns null
• Wrong granularity → logs warning, returns null

---

Deployment Checklist

# 1. Install dependency
npm install ws

# 2. Start cache service
pm2 start /app/trading_engine/candle_cache_service.mjs --name candle-cache

# 3. Start bridge
pm2 start /app/trading_engine/scripts/candle_cache_bridge.mjs --name candle-bridge

# 4. Verify cache exists
ls -la /app/trading_engine/data/candle_cache.json
ls /tmp/candle_cache_*.json

# 5. Save PM2 config
pm2 save