Health Probes

Every Revali server answers two probe endpoints by default. Point your orchestrator's probes at them. When you need readiness to depend on something, such as a database, add readiness checks.

• Probes are served outside prefix. They answer at /healthz, not /api/healthz.
• Liveness never runs checks. If a database outage failed liveness, the orchestrator would restart every instance at once.
• During shutdown, liveness keeps returning 200, while readiness returns 503 {"status":"draining"} without running any checks. See Graceful Shutdown.

Responses#

{ "status": "ok" }

With checks registered, readiness names each one, and any failure makes the response 503:

{
  "status": "unhealthy",
  "checks": {
    "database": { "status": "ok" },
    "queue": { "status": "unhealthy", "detail": "broker down" }
  }
}

Readiness Checks#

import 'package:revali_router/revali_router.dart';

class DatabaseIsReachable implements HealthCheck {
  const DatabaseIsReachable(this.db);

  final Database db;

  @override
  String get name => 'database'; // the key in the response body

  @override
  Future<HealthCheckResult> check() async {
    try {
      await db.ping();
      return const HealthCheckResult.healthy();
    } catch (e) {
      return HealthCheckResult.unhealthy('$e');
    }
  }
}

import 'package:my_app/health/database_is_reachable.dart';
import 'package:revali_router/revali_router.dart';

@App()
final class MainApp extends AppConfig {
  MainApp() : super.fromEnv();

  final db = Database.connect(Env.current.require('DATABASE_URL'));

  @override
  Future<void> configureDependencies(DI di) async {
    di.registerSingleton<Database>(db);
  }

  @override
  HealthSettings get health => HealthSettings(
        checks: [DatabaseIsReachable(db)],
        checkTimeout: const Duration(seconds: 2),
      );
}

How checks behave:

• Checks run concurrently. A probe takes as long as its slowest check.
• A check that throws or times out counts as unhealthy, with the error as its detail. It never causes a 500.
• detail is visible to anyone who can reach the pod, so never put credentials in it.
• Only add checks for dependencies the service can't work without. If a cache the service can do without becomes a check, one slow cache node takes every instance out of rotation.

HealthSettings#

const HealthSettings.disabled() turns off both probes. A path that's turned off returns 404.

Kubernetes#

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
    timeoutSeconds: 10 # above checkTimeout