Skip to main content

Cluster Management

Scheduler0 runs as a Raft cluster with an embedded SQLite database. These endpoints let self-hosting operators inspect and change cluster membership and leadership, recover from quorum loss, and dump in-memory scheduler state. Database backup and restore are on a separate page.

All cluster endpoints require either a credential with the admin scope or HTTP Basic Auth with the server's operator username/password plus X-Peer: cmd. The examples use Basic Auth against a node's client address.

warning

reset-raft deletes local Raft state and exits the process; force-rebuild rewrites membership. Use them only during recovery.

Endpoints​

MethodPathParametersResponse data
GET/cluster/list-nodes—[{nodeId, nodeAddress, clientAddress}]
POST/cluster/add-nodenodeId, nodeAddress, clientAddress (query){"status": "node added"}
POST/cluster/remove-nodenodeId (query){"status": "node removed"}
POST/cluster/promote-nodenodeId (query){"status": "node promoted to voter"}
POST/cluster/demote-nodenodeId (query){"status": "node demoted to non-voter"}
POST/cluster/transfer-leadership—{"status": "leadership transferred"}
POST/cluster/add-self—{"status": "added"}
POST/cluster/remove-self—{"status": "removed"}
POST/cluster/force-rebuildseedNodeId (query){"status": "rebuild initiated"}
POST/cluster/reset-raft—{"status": "raft reset"}
GET/cluster/dump/schedule-queue—In-memory schedule queue
GET/cluster/dump/job-executions-cache—Job-executions cache keyed by job id
GET/cluster/dump/job-queues—Job queue records
GET/cluster/dump/job-queue-versions—Job queue version records

add-node, remove-node, promote-node, demote-node and transfer-leadership must be sent to the current leader; any other node returns 403. Find the leader with GET /healthcheck.

Membership​

# List nodes
curl "https://node1.example.com/api/v1/cluster/list-nodes" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"
# → { "success": true, "data": [ { "nodeId": 1, "nodeAddress": "10.0.0.1:7071", "clientAddress": "10.0.0.1:9091" } ] }

# Add a node (send to the leader). nodeAddress is the Raft port, clientAddress the HTTP API port.
curl -X POST "https://node1.example.com/api/v1/cluster/add-node?nodeId=2&nodeAddress=10.0.0.2:7071&clientAddress=10.0.0.2:9091" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

# Remove a node (send to the leader)
curl -X POST "https://node1.example.com/api/v1/cluster/remove-node?nodeId=2" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

add-self and remove-self are sent to the node itself and make it register with / deregister from service discovery and join or leave the Raft membership.

Voters and leadership​

New nodes join as non-voters. Promote a node once it has caught up; demote before removing a voter to keep quorum arithmetic simple.

curl -X POST "https://node1.example.com/api/v1/cluster/promote-node?nodeId=2" -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"
curl -X POST "https://node1.example.com/api/v1/cluster/demote-node?nodeId=2"  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"
curl -X POST "https://node1.example.com/api/v1/cluster/transfer-leadership"   -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

transfer-leadership hands leadership to another eligible voter; use it before taking the leader down for maintenance.

Recovery​

Force rebuild rebuilds the cluster around a seed node after quorum is lost. Run it on the seed node:

curl -X POST "https://node1.example.com/api/v1/cluster/force-rebuild?seedNodeId=1" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

Reset Raft clears the receiving node's Raft state on disk and exits the process after the response is written. Your process supervisor is expected to restart it, after which it rejoins cleanly.

curl -X POST "https://node2.example.com/api/v1/cluster/reset-raft" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

Debug dumps​

The four GET /cluster/dump/* endpoints return the in-memory state of the node that serves the request. They are unpaginated and intended for debugging.

curl "https://node1.example.com/api/v1/cluster/dump/schedule-queue" \
  -u "$SCHEDULER0_USER:$SCHEDULER0_PASSWORD" -H "X-Peer: cmd"

Health check​

GET /healthcheck needs no authentication and reports the current leader and the node's Raft statistics.

curl "https://node1.example.com/api/v1/healthcheck"
{
  "success": true,
  "data": {
    "leaderAddress": "10.0.0.1:7071",
    "leaderId": "1",
    "raftStats": { "state": "Leader", "term": "4", "last_log_index": "1024", "num_peers": "2" }
  }
}

raftStats is the raw hashicorp/raft stats map (string keys and values).

CLI equivalents​

scheduler0 cluster list-nodes | add-node | remove-node | promote-node | demote-node | transfer-leadership | add-self | remove-self | force-rebuild | reset-raft | dump ... wrap these endpoints; see CLI.