--- title: Monitoring and Observability description: Monitor Alita Robot health, metrics, and errors. --- # Monitoring and Observability Alita Robot provides comprehensive monitoring capabilities including health checks, Prometheus metrics, and resource monitoring. ## Health Endpoint The health endpoint provides real-time status of the bot and its dependencies. ### Endpoint ``` GET /health ``` ### Response Format ```json { "status": "healthy", "checks": { "database": true, "redis": true }, "version": "2.21.0", "uptime": "24h30m15s" } ``` ### Response Fields | Field | Type | Description | |-------|------|-------------| | `status` | string | Overall health: `healthy` or `unhealthy` | | `checks.database` | boolean | PostgreSQL connection status | | `checks.redis` | boolean | Redis connection status | | `version` | string | Bot version | | `uptime` | string | Time since bot started | ### HTTP Status Codes :::tip Use the HTTP status code (not the response body) in your load balancer or uptime monitoring tool. A 503 response means at least one dependency is unhealthy. ::: | Code | Meaning | |------|---------| | 200 | All systems healthy | | 503 | One or more checks failed | ### Health Check Logic The health check performs: 1. **Database check**: Pings PostgreSQL with a 2-second timeout 2. **Redis check**: Sets and gets a test key with a 2-second timeout Both checks must pass for `healthy` status. ### Usage Examples ```bash # Simple health check curl http://localhost:8080/health # Docker health check (built-in) /app/alita_robot --health # Kubernetes liveness probe livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 30 periodSeconds: 10 ``` ## Prometheus Metrics The metrics endpoint exposes Prometheus-compatible metrics for monitoring. ### Endpoint ``` GET /metrics ``` ### Bot Metrics Besides the Go runtime metrics, the bot exports: | Metric | Type | Labels | Meaning | |--------|------|--------|---------| | `alita_updates_processed_total` | counter | `update_type`, `outcome`, `command` | Telegram updates processed | | `alita_handler_errors_total` | counter | `module`, `expected` | Errors handlers returned; `expected` is `true` for expected Telegram API errors | | `alita_redis_commands_total` | counter | `command` | Redis commands sent (`get`, `set`, `del`, ...) | | `alita_db_queries_total` | counter | `operation` | Database statements (`query`, `create`, `update`, `delete`, `row`, `raw`) | | `alita_db_query_duration_seconds` | histogram | `operation` | Database statement latency | | `alita_db_query_errors_total` | counter | `operation` | Failed database statements (a lookup that finds no row is not a failure) | | `go_sql_*` | gauge/counter | `db_name` | Connection pool stats: open, in-use and idle connections, waits | | `alita_telegram_api_request_duration_seconds` | histogram | `method` | Bot API request latency (`sendMessage`, `getChatMember`, ...); `getUpdates` is left out, as it long-polls | | `alita_telegram_api_errors_total` | counter | `method`, `code` | Failed Bot API requests: HTTP status, or `transport` when no response came back | | `alita_webhook_queue_depth` | gauge | none | Webhook updates accepted and waiting for a dispatcher worker | | `alita_webhook_rejected_total` | counter | `reason` | Webhook updates answered with 503: `queue_full` or `stopped` | | `alita_job_runs_total` | counter | `job` | Background job ticks run (`digestSchedulerTick`, `captchaCleanup`, ...) | | `alita_job_failures_total` | counter | `job` | Job ticks that returned an error or panicked | | `alita_cache_local_hits_total` | counter | none | Read-through cache lookups served by the in-process layer (`CACHE_LOCAL_TTL`) | | `alita_cache_local_misses_total` | counter | none | Read-through cache lookups that missed the in-process layer and fell through to Redis | `outcome` is the worst of `ok`, `expected_error`, `error` and `panic` any handler gave the update. `panic` counts panics recovered by command handlers or the dispatcher; a module's own recover hides its panic. `command` is the command a new message starts with, without `/`, when it is a help-menu command, `other` for any other command, and empty otherwise. `module` is the help module of that command, or `other`, even when a different handler on the message failed. Error rate: ```promql sum(rate(alita_updates_processed_total{outcome=~"error|panic"}[5m])) / sum(rate(alita_updates_processed_total[5m])) ``` Per-update cost: ```promql sum(rate(alita_redis_commands_total[5m])) / sum(rate(alita_updates_processed_total[5m])) sum(rate(alita_db_queries_total[5m])) / sum(rate(alita_updates_processed_total[5m])) ``` Bot API rate limiting: ```promql sum by (method) (rate(alita_telegram_api_errors_total{code="429"}[5m])) ``` In-process cache hit ratio: ```promql rate(alita_cache_local_hits_total[5m]) / (rate(alita_cache_local_hits_total[5m]) + rate(alita_cache_local_misses_total[5m])) ``` ### Prometheus Scrape Configuration Add to your `prometheus.yml`: ```yaml scrape_configs: - job_name: 'alita-robot' static_configs: - targets: ['alita:8080'] scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics # Required when METRICS_AUTH_TOKEN is set authorization: type: Bearer credentials: # or credentials_file: /etc/prometheus/alita_token ``` ### Authentication When `METRICS_AUTH_TOKEN` is set, `/metrics` and `/db_metrics` return `401 Unauthorized` unless the request sends `Authorization: Bearer `. When it is empty, both endpoints are open and the bot logs a warning at startup. See [METRICS_AUTH_TOKEN](/api-reference/environment/#metrics_auth_token). ### Docker Compose with Prometheus ```yaml services: prometheus: image: prom/prometheus:latest volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - prometheus_data:/prometheus ports: - "9090:9090" depends_on: - alita volumes: prometheus_data: ``` ### Grafana Dashboard For visualization, add Grafana: ```yaml services: grafana: image: grafana/grafana:latest volumes: - grafana_data:/var/lib/grafana ports: - "3000:3000" environment: - GF_SECURITY_ADMIN_PASSWORD=admin depends_on: - prometheus volumes: grafana_data: ``` ## Resource Monitoring Alita Robot includes automatic resource monitoring to prevent resource exhaustion. ### Configuration ```bash # Maximum goroutines before triggering cleanup # Default: 1000, Recommended: 1000-5000 RESOURCE_MAX_GOROUTINES=1000 # Maximum memory usage in MB before triggering cleanup # Default: 500, Recommended: 500-2000 RESOURCE_MAX_MEMORY_MB=500 # Memory threshold for triggering garbage collection # Default: 400, Recommended: 80% of RESOURCE_MAX_MEMORY_MB RESOURCE_GC_THRESHOLD_MB=400 ``` ### Auto-Remediation :::note Auto-remediation runs automatically in the background. No manual intervention is required unless Tier 3 is triggered, at which point you should investigate and restart the bot. ::: The system uses a 4-tier auto-remediation approach when resource thresholds are exceeded: 1. **Tier 0 — Warning**: Logs warnings at 80% goroutine threshold or 50% memory usage 2. **Tier 1 — GC Trigger**: Triggers garbage collection at 60% memory usage or 50ms GC pause times 3. **Tier 2 — Aggressive Cleanup**: Runs multiple GC cycles when memory exceeds the GC threshold 4. **Tier 3 — Restart Recommendation**: Logs a restart recommendation at 150%+ goroutines or 160%+ memory usage ## Activity Monitoring Track group activity automatically: ```bash # Days of inactivity before marking chat as inactive # Default: 30, recommended 1-365 (not enforced) INACTIVITY_THRESHOLD_DAYS=30 # Hours between automatic activity checks # Default: 1, recommended 1-24 (not enforced) ACTIVITY_CHECK_INTERVAL=1 # Enable automatic cleanup of inactive chats # Default: true ENABLE_AUTO_CLEANUP=true ``` ### Activity Metrics The system tracks group activity: - **DAG**: Daily Active Groups - **WAG**: Weekly Active Groups - **MAG**: Monthly Active Groups The activity monitor also tracks individual user activity, calculating Daily Active Users (DAU), Weekly Active Users (WAU), and Monthly Active Users (MAU). Groups are automatically: - Marked inactive after the threshold period - Reactivated when they become active again ## Performance Monitoring ### Enable Performance Tracking ```bash # Enable automatic performance monitoring ENABLE_PERFORMANCE_MONITORING=true # Enable background statistics collection ENABLE_BACKGROUND_STATS=true ``` ### Collected Metrics - Message processing time - Database query duration - Cache hit/miss rates - API response times - Goroutine count - Memory usage ## Database Metrics The `/db_metrics` endpoint exposes real-time database connection pool statistics. ### Endpoint ``` GET /db_metrics ``` When `METRICS_AUTH_TOKEN` is set, send the same bearer header: `curl -H "Authorization: Bearer $METRICS_AUTH_TOKEN" http://localhost:8080/db_metrics`. ### Response Format ```json { "open_connections": 45, "idle": 38, "in_use": 7, "wait_count": 0, "wait_duration": 0, "max_idle_closed": 0, "max_lifetime_closed": 0 } ``` `wait_duration` is the total time spent waiting for a connection, in nanoseconds. The endpoint is always registered. It queries the underlying `*sql.DB` connection pool stats via `db.Stats()`. ## Background Database Monitoring In addition to the `/db_metrics` endpoint, a background monitoring goroutine periodically logs pool health metrics. ### Configuration ```bash # Enable background database monitoring (logs pool stats every minute at debug level; needs DEBUG=true to show) # Controls the background goroutine, NOT the /db_metrics endpoint registration ENABLE_DB_MONITORING=true ``` When enabled, the monitoring goroutine logs pool stats every minute at **debug** level, so the line appears only when `DEBUG=true`. With `DEBUG=false` the goroutine still runs, but only its start and stop messages are logged; use `/db_metrics` to read the stats in production. ``` [DBMonitoring] Connections: 12 open, 7 in-use, 5 idle | Wait: 0 calls, 0s total | Closed: 0 (max idle), 0 (max lifetime) ``` ## Debug Mode For detailed logging during development: ```bash DEBUG=true ``` :::caution Debug mode disables background performance monitoring and significantly increases log volume. Never enable in production as it degrades overall performance. ::: Debug mode: - Increases log verbosity - Disables background monitoring - Shows detailed error stack traces ## Log Analysis ### Log Fields Structured log entries include: | Field | Description | |-------|-------------| | `update_id` | Telegram update ID | | `error_type` | Error type (e.g., `*TelegramError`) | | `file` | Source file where error occurred | | `line` | Line number | | `function` | Function name | ### Example Log Entry ```json { "level": "error", "msg": "Handler error occurred: user blocked bot", "update_id": 123456789, "error_type": "*gotgbot.TelegramError", "file": "alita/modules/admin.go", "line": 45, "function": "handleAdminCommand", "time": "2024-03-15T10:30:00Z" } ``` ### Log Levels | Level | Description | |-------|-------------| | `DEBUG` | Verbose debugging (DEBUG=true only) | | `INFO` | Normal operation events | | `WARN` | Expected issues (e.g., user blocked bot) | | `ERROR` | Unexpected errors | | `FATAL` | Critical errors that stop the bot | ## Alerting :::tip At minimum, set up alerts for `AlitaUnhealthy` and monitor the `/health` endpoint with an external HTTP check (e.g., blackbox exporter) to catch critical outages early. ::: ### Prometheus Alerting Rules Create `alerts.yml`: ```yaml groups: - name: alita-alerts rules: - alert: AlitaUnhealthy expr: up{job="alita-robot"} == 0 for: 1m labels: severity: critical annotations: summary: "Alita Robot is down" - alert: HighMemoryUsage expr: process_resident_memory_bytes > 500000000 for: 5m labels: severity: warning annotations: summary: "High memory usage detected" - alert: HighGoroutineCount expr: go_goroutines > 1000 for: 5m labels: severity: warning annotations: summary: "High goroutine count detected" ``` ### Database Health The `/health` endpoint returns a 503 status code when the database or Redis connection is unhealthy. Because the health check is a JSON API (not a Prometheus metric), monitor it with an external HTTP health check such as the Prometheus Blackbox Exporter or your uptime monitoring tool.