Background jobs & hosted services

Genie runs background work through two complementary systems:

1. Hosted services — long-running in-process background workers (notification delivery, search sync, SLA/monitor loops) that run inside the engine and are observable + controllable at runtime.
2. Hangfire jobs — durable, storage-backed fire-and-forget / scheduled / recurring jobs, managed through a dedicated admin surface.

Hosted services

Engine workers derive from MonitoredBackgroundService (in the Services/HostedServices area). Each runs a processing loop and reports live metrics to an IHostedServiceMonitor, so operators can watch throughput and pause/resume a worker without restarting the app. Examples include the notification workers and the Meilisearch sync worker.

API — /api/v1/genie/hosted-services

Method	Route	Purpose
GET	/	live metrics for every registered service
GET	/{name}	metrics for one service
POST	/{name}/pause	pause a service’s processing loop (it idles until resumed)
POST	/{name}/resume	resume a paused service
GET	/{name}/history	last hour of cycle-summary history (from Redis)

Pausing stops the inner loop while leaving the background task alive — the worker polls until resumed. Cycle history is best-effort and returns an empty list when Redis is unavailable. The UI exposes all of this through the components/hosted-services admin dashboard.

Hangfire jobs

For durable background processing Genie integrates Hangfire (wired via the app builder’s ConfigureHangfire). Its admin API is restricted to the System role and short-circuits with 503 when Hangfire storage is not configured on the server.

API — /api/v1/hangfire

Method	Route	Purpose
GET	/stats	storage statistics (counts per state, servers, recurring jobs)
GET	/jobs	paginated, filterable job list (state, queue, method, date, id)
GET	/jobs/{id}	full job detail (parameters, state history, exception)
POST	/jobs/{id}/requeue	requeue a failed/deleted job
DELETE	/jobs/{id}	delete a job
GET	/queues	queues with current counts
GET	/servers	active Hangfire server instances
GET	/recurring	recurring job definitions
POST	/recurring/{id}/trigger	run a recurring job now
POST	/recurring/bulk-trigger · /jobs/bulk-requeue · /jobs/bulk-delete	batch operations

Which to use

Need	Use
Continuous in-process loop, live pause/resume, engine-internal	Hosted service (MonitoredBackgroundService)
Durable, retryable, scheduled/recurring, survives restarts	Hangfire job