Skip to main content

Cluster Management

Scheduler0 runs as a Raft cluster. These endpoints let operators inspect and manage cluster membership, leadership, and internal scheduler state. For database backup/restore, see Backup and Restore.

Self-hosting only

All cluster endpoints are available only when self-hosting Scheduler0 and use Basic Authentication with the X-Peer header (peer/basic auth). They bypass the API-key scope check entirely. They are not available to managed-service users.

warning

Several of these operations are destructive or disruptive (reset-raft deletes local state and exits the process; force-rebuild rebuilds membership). Use them only when you understand the cluster impact.

Membership​

List Nodes​

Returns all nodes currently in the Raft cluster, including their IDs, addresses, and voter/non-voter status.

Endpoint: GET /api/v1/cluster/list-nodes

curl -X GET "https://your-host/api/v1/cluster/list-nodes" \
  -u username:password

Response:

{
  "success": true,
  "data": [
    { "id": 1, "address": "10.0.0.1:7070", "isLeader": true, "suffrage": "Voter" }
  ]
}

Add Node​

Adds a node to the Raft cluster. Leader-only — returns 403 if the receiving node is not the leader.

Endpoint: POST /api/v1/cluster/add-node

Query Parameters:

  • nodeId (required): ID of the node to add
  • nodeAddress (required): Raft (peer) address of the node, e.g. 10.0.0.2:7070
  • clientAddress (required): Client/HTTP address of the node
curl -X POST "https://your-host/api/v1/cluster/add-node?nodeId=2&nodeAddress=10.0.0.2:7070&clientAddress=10.0.0.2:9090" \
  -u username:password

Response: { "success": true, "data": { "status": "node added" } }

Remove Node​

Removes a node from the Raft cluster. Leader-only — returns 403 if not the leader.

Endpoint: POST /api/v1/cluster/remove-node

Query Parameters:

  • nodeId (required): ID of the node to remove
curl -X POST "https://your-host/api/v1/cluster/remove-node?nodeId=2" \
  -u username:password

Response: { "success": true, "data": { "status": "node removed" } }

Add Self / Remove Self​

Convenience endpoints a node calls on itself to (de)register from etcd and join/leave Raft membership.

Endpoints:

  • POST /api/v1/cluster/add-self → { "status": "added" }
  • POST /api/v1/cluster/remove-self → { "status": "removed" }
curl -X POST "https://your-host/api/v1/cluster/add-self" -u username:password
curl -X POST "https://your-host/api/v1/cluster/remove-self" -u username:password

Leadership​

Promote Node​

Promotes a non-voter node to a voter. Leader-only — returns 403 if not the leader.

Endpoint: POST /api/v1/cluster/promote-node

Query Parameters:

  • nodeId (required): ID of the node to promote
curl -X POST "https://your-host/api/v1/cluster/promote-node?nodeId=2" \
  -u username:password

Response: { "success": true, "data": { "status": "node promoted to voter" } }

Demote Node​

Demotes a voter node to a non-voter. Leader-only — returns 403 if not the leader.

Endpoint: POST /api/v1/cluster/demote-node

Query Parameters:

  • nodeId (required): ID of the node to demote
curl -X POST "https://your-host/api/v1/cluster/demote-node?nodeId=2" \
  -u username:password

Response: { "success": true, "data": { "status": "node demoted to non-voter" } }

Transfer Leadership​

Transfers Raft leadership away from the current leader to another eligible node. Leader-only — returns 403 if the receiving node is not the leader.

Endpoint: POST /api/v1/cluster/transfer-leadership

curl -X POST "https://your-host/api/v1/cluster/transfer-leadership" \
  -u username:password

Response: { "success": true, "data": { "status": "leadership transferred" } }

Recovery​

Force Rebuild​

Forces a rebuild of the Raft cluster around a designated seed node. Should be run only on the seed node during recovery from a loss of quorum.

Endpoint: POST /api/v1/cluster/force-rebuild

Query Parameters:

  • seedNodeId (required): ID of the node to seed the rebuilt cluster from
curl -X POST "https://your-host/api/v1/cluster/force-rebuild?seedNodeId=1" \
  -u username:password

Response: { "success": true, "data": { "status": "rebuild initiated" } }

Reset Raft​

Clears this node's local Raft state on disk and exits the process. The HTTP response is flushed before the process terminates, so the call returns { "status": "raft reset" } and then the node shuts down (your process supervisor is expected to restart it, after which it rejoins clean).

Endpoint: POST /api/v1/cluster/reset-raft

curl -X POST "https://your-host/api/v1/cluster/reset-raft" \
  -u username:password
danger

This deletes local Raft data and stops the node. Only use it on a node you intend to re-bootstrap.

Debug Dumps​

These read-only endpoints dump in-memory scheduler state from the node that serves the request, for debugging. They are not paginated.

EndpointReturns
GET /api/v1/cluster/dump/schedule-queueAll items currently in the in-memory schedule (priority) queue
GET /api/v1/cluster/dump/job-executions-cacheThe job-executions cache, keyed by job ID
GET /api/v1/cluster/dump/job-queuesAll job queues from the repository
GET /api/v1/cluster/dump/job-queue-versionsJob queue version records
curl -X GET "https://your-host/api/v1/cluster/dump/schedule-queue" \
  -u username:password

Health Check​

Returns basic liveness information plus Raft statistics for the node. This endpoint is always allowed regardless of credential scope.

Endpoint: GET /api/v1/healthcheck

curl -X GET "https://your-host/api/v1/healthcheck"

Response:

{
  "success": true,
  "data": {
    "leaderAddress": "10.0.0.1:7070",
    "leaderId": "1",
    "raftStats": {
      "state": "Leader",
      "term": "4",
      "last_log_index": "1024",
      "num_peers": "2"
    }
  }
}

Response Fields:

  • leaderAddress: Raft address of the current cluster leader
  • leaderId: Node ID of the current cluster leader
  • raftStats: Raw Raft statistics for the node (keys and values are strings) — useful for confirming which node is the leader and how far each follower has replicated