Monitoring

Guide for monitoring GLM Proxy health and performance.

Related:

• Dashboard Guide - Visual dashboard tour and features
• Configuration Guide - Environment variables and settings
• Z.ai Documentation - Understanding quotas, rate limits, and tier capabilities
• Metrics Reference - Prometheus metrics definitions
• Events SSE Contract - Real-time event streaming

Dashboard Monitoring

The dashboard provides real-time visual monitoring of all proxy metrics. Open it at http://127.0.0.1:18765/dashboard.

Key Monitoring Sections

Connection Status - See if the proxy is connected to the upstream API:

Keys Heatmap - Monitor the health of all your API keys at a glance:

• Green cells = healthy keys
• Yellow/Red cells = degraded or failing keys
• Pulsing cells = active requests

Request Charts - Track request rate and latency over time:

Circuit Breaker Status - Monitor circuit breaker states for all keys:

Health Ribbon - Quick overview of system health at the top of the dashboard:

Cost Panel - Track token usage and costs in real-time:

API Endpoints

Health Check

Check proxy health and key status:

curl http://127.0.0.1:18765/health

Response:

{
  "status": "OK",
  "healthyKeys": 5,
  "totalKeys": 5,
  "uptime": 3600,
  "backpressure": {
    "current": 2,
    "max": 200,
    "available": 198,
    "percentUsed": 1
  }
}

Fields

Real-time Statistics

View detailed real-time statistics:

curl http://127.0.0.1:18765/stats

Response:

{
  "uptime": 3600,
  "uptimeFormatted": "1h 0m 0s",
  "totalRequests": 1500,
  "requestsPerMinute": 25.0,
  "successRate": 99.3,
  "activeConnections": 3,
  "latency": {
    "avg": 1245,
    "min": 296,
    "max": 5420,
    "spread": 18.3,
    "samples": 1492
  },
  "keys": [
    {
      "index": 0,
      "state": "CLOSED",
      "inFlight": 1,
      "total": 300,
      "successes": 298,
      "successRate": 99.3,
      "failures": 2,
      "latency": {
        "avg": 1200,
        "min": 312,
        "max": 4521,
        "p50": 1050,
        "p95": 2800,
        "p99": 4100,
        "samples": 100
      }
    }
  ],
  "errors": {
    "timeouts": 1,
    "socketHangups": 0,
    "dnsErrors": 0,
    "tlsErrors": 0,
    "clientDisconnects": 0,
    "rateLimited": 0,
    "totalRetries": 5,
    "retriesSucceeded": 3,
    "retrySuccessRate": 60.0
  },
  "queue": {
    "current": 0,
    "max": 100,
    "available": 100,
    "percentUsed": 0,
    "metrics": {
      "totalEnqueued": 10,
      "totalDequeued": 10,
      "totalTimedOut": 0,
      "peakSize": 3
    }
  }
}

Key Metrics

Latency Metrics

Error Metrics

Queue Metrics

Backpressure

View current load and backpressure status:

curl http://127.0.0.1:18765/backpressure

Response:

{
  "current": 2,
  "max": 200,
  "available": 198,
  "percentUsed": 1
}

Interpreting Backpressure

Adaptive Concurrency (AIMD)

The proxy uses TCP-inspired AIMD (Additive Increase / Multiplicative Decrease) to dynamically adjust per-model concurrency limits based on upstream 429 feedback.

Modes:

GET /adaptive-concurrency

Returns current AIMD state including per-model windows.

curl -H "x-admin-token: TOKEN" http://127.0.0.1:18765/adaptive-concurrency

Response includes: mode, per-model effectiveMax, staticMax, consecutiveCleanTicks, and global window state.

PUT /adaptive-concurrency

Toggle between observe_only and enforce mode at runtime.

curl -X PUT \
  -H "x-admin-token: TOKEN" \
  -H "content-type: application/json" \
  -d '{"mode": "enforce"}' \
  http://127.0.0.1:18765/adaptive-concurrency

Response: { "success": true, "previousMode": "observe_only", "currentMode": "enforce", "snapshot": {...} }

Notes:

• Requires admin authentication when enabled
• Switching to enforce immediately pushes computed windows to the key manager
• Switching to observe_only restores static concurrency limits
• Can also be set via GLM_ADAPTIVE_CONCURRENCY_MODE environment variable
• The dashboard AIMD card includes a toggle button for this endpoint

Persistent Statistics

View historical usage statistics:

curl http://127.0.0.1:18765/persistent-stats

Circuit Breaker Monitoring

Per-key circuit breaker state is included in /stats:

System Page provides detailed circuit breaker analytics:

Alerts and Monitoring

Security: See Security Configuration for admin authentication and endpoint protection. Load Testing: See Load Testing Guide for performance baseline validation.

Recommended Alerts

Monitoring Integration

Integrate with monitoring tools:

# Prometheus exporter example
while true; do
  curl -s http://127.0.0.1:18765/stats | jq -r '
    # Convert to Prometheus metrics format
    # ...
  '
  sleep 15
done