Fuzzball 4.4 ties node eviction to Prometheus alerts

Image
Fuzzball 4.4 ties node eviction to Prometheus alerts

GLOBAL – November 12, 2026 -- Fuzzball 4.4 lets the alert rules in a customer's Prometheus and Alertmanager stack decide when a failing compute node is taken out of service. It also adds persistent endpoints that keep a model's URL fixed when the model behind it is replaced. In Fuzzball 4.3, an endpoint kept its URL only for the life of the workflow that declared it, so every resubmitted workflow produced a new address and every connected agent needed reconfiguring.

Alert rules decide when Fuzzball takes a failing node out of service

Each compute node runs the standard Prometheus Node Exporter, and a custom Fuzzball exporter adds metrics Node Exporter cannot see. Operators set thresholds and wait windows in their alert rules, and Fuzzball acts on the alerts Alertmanager sends it.

A firing alert can place one of two holds on a node. A cordon hold stops new work from being placed there. An evict hold also takes running work off. A resolved alert releases the hold unless the rule requires an administrator to do so. Every hold records who took the node out of service and why.

By default, monitoring can hold at most 20 percent of a pool's nodes at once, or one node in five. In a 10-node pool, that caps automated holds at two nodes, by FairsOnline's calculation.

Workflow authors control what happens to work on an evicted node. A step marked retry.on: node restarts from the beginning on a healthy node. Every other step fails, with the reason recorded in the workflow events. A long job without that marking therefore fails when its node is evicted.

Deployments through the Fuzzball Kubernetes operator install Prometheus, Alertmanager and Grafana by default, preconfigured with alert rules, routes and dashboards. Operators with an existing Prometheus instance can switch the bundled stack off and load the same rules, routes and dashboards into their own.

Persistent endpoints keep an agent's model URL when the workflow changes

Operators create a persistent endpoint by name, and a service attaches to it. To replace a model, they submit a new workflow against the same endpoint. By default, the running workflow keeps serving until the new service is ready, and then Fuzzball cancels the previ

Published by:
fairsonline_team
Industries