Monitoring and Observability

Monitor Alita Robot health, metrics, and errors.

Monitoring and Observability

Alita Robot provides comprehensive monitoring capabilities including health checks, Prometheus metrics, and resource monitoring.

Health Endpoint

The health endpoint provides real-time status of the bot and its dependencies.

Endpoint

GET /health

Response Format

{
  "status": "healthy",
  "checks": {
    "database": true,
    "redis": true
  },
  "version": "2.21.0",
  "uptime": "24h30m15s"
}

Response Fields

HTTP Status Codes

:::tip Use the HTTP status code (not the response body) in your load balancer or uptime monitoring tool. A 503 response means at least one dependency is unhealthy. :::

Health Check Logic

The health check performs:

1. Database check: Pings PostgreSQL with a 2-second timeout
2. Redis check: Sets and gets a test key with a 2-second timeout

Both checks must pass for healthy status.

Usage Examples

# Simple health check
curl http://localhost:8080/health

# Docker health check (built-in)
/app/alita_robot --health

# Kubernetes liveness probe
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

Prometheus Metrics

The metrics endpoint exposes Prometheus-compatible metrics for monitoring.

Endpoint

GET /metrics

Bot Metrics

Besides the Go runtime metrics, the bot exports:

outcome is the worst of ok, expected_error, error and panic any handler gave the update. panic counts panics recovered by command handlers or the dispatcher; a module’s own recover hides its panic. command is the command a new message starts with, without /, when it is a help-menu command, other for any other command, and empty otherwise. module is the help module of that command, or other, even when a different handler on the message failed.

Error rate:

sum(rate(alita_updates_processed_total{outcome=~"error|panic"}[5m])) / sum(rate(alita_updates_processed_total[5m]))

Per-update cost:

sum(rate(alita_redis_commands_total[5m])) / sum(rate(alita_updates_processed_total[5m]))
sum(rate(alita_db_queries_total[5m])) / sum(rate(alita_updates_processed_total[5m]))

Bot API rate limiting:

sum by (method) (rate(alita_telegram_api_errors_total{code="429"}[5m]))

In-process cache hit ratio:

rate(alita_cache_local_hits_total[5m]) / (rate(alita_cache_local_hits_total[5m]) + rate(alita_cache_local_misses_total[5m]))

Prometheus Scrape Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'alita-robot'
    static_configs:
      - targets: ['alita:8080']
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    # Required when METRICS_AUTH_TOKEN is set
    authorization:
      type: Bearer
      credentials: <METRICS_AUTH_TOKEN value>   # or credentials_file: /etc/prometheus/alita_token

Authentication

When METRICS_AUTH_TOKEN is set, /metrics and /db_metrics return 401 Unauthorized unless the request sends Authorization: Bearer <token>. When it is empty, both endpoints are open and the bot logs a warning at startup. See METRICS_AUTH_TOKEN.

Docker Compose with Prometheus

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    depends_on:
      - alita

volumes:
  prometheus_data:

Grafana Dashboard

For visualization, add Grafana:

services:
  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    depends_on:
      - prometheus

volumes:
  grafana_data:

Resource Monitoring

Alita Robot includes automatic resource monitoring to prevent resource exhaustion.

Configuration

# Maximum goroutines before triggering cleanup
# Default: 1000, Recommended: 1000-5000
RESOURCE_MAX_GOROUTINES=1000

# Maximum memory usage in MB before triggering cleanup
# Default: 500, Recommended: 500-2000
RESOURCE_MAX_MEMORY_MB=500

# Memory threshold for triggering garbage collection
# Default: 400, Recommended: 80% of RESOURCE_MAX_MEMORY_MB
RESOURCE_GC_THRESHOLD_MB=400

Auto-Remediation

:::note Auto-remediation runs automatically in the background. No manual intervention is required unless Tier 3 is triggered, at which point you should investigate and restart the bot. :::

The system uses a 4-tier auto-remediation approach when resource thresholds are exceeded:

1. Tier 0 — Warning: Logs warnings at 80% goroutine threshold or 50% memory usage
2. Tier 1 — GC Trigger: Triggers garbage collection at 60% memory usage or 50ms GC pause times
3. Tier 2 — Aggressive Cleanup: Runs multiple GC cycles when memory exceeds the GC threshold
4. Tier 3 — Restart Recommendation: Logs a restart recommendation at 150%+ goroutines or 160%+ memory usage

Activity Monitoring

Track group activity automatically:

# Days of inactivity before marking chat as inactive
# Default: 30, recommended 1-365 (not enforced)
INACTIVITY_THRESHOLD_DAYS=30

# Hours between automatic activity checks
# Default: 1, recommended 1-24 (not enforced)
ACTIVITY_CHECK_INTERVAL=1

# Enable automatic cleanup of inactive chats
# Default: true
ENABLE_AUTO_CLEANUP=true

Activity Metrics

The system tracks group activity:

• DAG: Daily Active Groups
• WAG: Weekly Active Groups
• MAG: Monthly Active Groups

The activity monitor also tracks individual user activity, calculating Daily Active Users (DAU), Weekly Active Users (WAU), and Monthly Active Users (MAU).

Groups are automatically:

• Marked inactive after the threshold period
• Reactivated when they become active again

Performance Monitoring

Enable Performance Tracking

# Enable automatic performance monitoring
ENABLE_PERFORMANCE_MONITORING=true

# Enable background statistics collection
ENABLE_BACKGROUND_STATS=true

Collected Metrics

• Message processing time
• Database query duration
• Cache hit/miss rates
• API response times
• Goroutine count
• Memory usage

Database Metrics

The /db_metrics endpoint exposes real-time database connection pool statistics.

Endpoint

GET /db_metrics

When METRICS_AUTH_TOKEN is set, send the same bearer header: curl -H "Authorization: Bearer $METRICS_AUTH_TOKEN" http://localhost:8080/db_metrics.

Response Format

{
  "open_connections": 45,
  "idle": 38,
  "in_use": 7,
  "wait_count": 0,
  "wait_duration": 0,
  "max_idle_closed": 0,
  "max_lifetime_closed": 0
}

wait_duration is the total time spent waiting for a connection, in nanoseconds.

The endpoint is always registered. It queries the underlying *sql.DB connection pool stats via db.Stats().

Background Database Monitoring

In addition to the /db_metrics endpoint, a background monitoring goroutine periodically logs pool health metrics.

Configuration

# Enable background database monitoring (logs pool stats every minute at debug level; needs DEBUG=true to show)
# Controls the background goroutine, NOT the /db_metrics endpoint registration
ENABLE_DB_MONITORING=true

When enabled, the monitoring goroutine logs pool stats every minute at debug level, so the line appears only when DEBUG=true. With DEBUG=false the goroutine still runs, but only its start and stop messages are logged; use /db_metrics to read the stats in production.

[DBMonitoring] Connections: 12 open, 7 in-use, 5 idle | Wait: 0 calls, 0s total | Closed: 0 (max idle), 0 (max lifetime)

Debug Mode

For detailed logging during development:

DEBUG=true

:::caution Debug mode disables background performance monitoring and significantly increases log volume. Never enable in production as it degrades overall performance. :::

Debug mode:

• Increases log verbosity
• Disables background monitoring
• Shows detailed error stack traces

Log Analysis

Log Fields

Structured log entries include:

Example Log Entry

{
  "level": "error",
  "msg": "Handler error occurred: user blocked bot",
  "update_id": 123456789,
  "error_type": "*gotgbot.TelegramError",
  "file": "alita/modules/admin.go",
  "line": 45,
  "function": "handleAdminCommand",
  "time": "2024-03-15T10:30:00Z"
}

Log Levels

Alerting

:::tip At minimum, set up alerts for AlitaUnhealthy and monitor the /health endpoint with an external HTTP check (e.g., blackbox exporter) to catch critical outages early. :::

Prometheus Alerting Rules

Create alerts.yml:

groups:
  - name: alita-alerts
    rules:
      - alert: AlitaUnhealthy
        expr: up{job="alita-robot"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Alita Robot is down"

      - alert: HighMemoryUsage
        expr: process_resident_memory_bytes > 500000000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage detected"

      - alert: HighGoroutineCount
        expr: go_goroutines > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High goroutine count detected"

Database Health

The /health endpoint returns a 503 status code when the database or Redis connection is unhealthy. Because the health check is a JSON API (not a Prometheus metric), monitor it with an external HTTP health check such as the Prometheus Blackbox Exporter or your uptime monitoring tool.