Background jobs & hosted services

Genie runs background work through two complementary systems:

1. Hosted services — long-running in-process background workers (notification delivery, search sync, SLA/monitor loops) that run inside the engine and are observable + controllable at runtime.
2. Hangfire jobs — durable, storage-backed fire-and-forget / scheduled / recurring jobs, managed through a dedicated admin surface.

Hosted services

Engine workers derive from MonitoredBackgroundService (in the Services/HostedServices area). Each runs a processing loop and reports live metrics to an IHostedServiceMonitor, so operators can watch throughput and pause/resume a worker without restarting the app. The engine’s workers:

Dashboard name	Class	Cycle
App Notification Worker	AppNotificationWorkerService	notification delivery, 1–5 s
Email / SMS Notification Worker	EmailNotificationWorkerService, SmsNotificationWorkerService	queue drains
Warehousing Sync Worker	WarehousingPollingWorker	warehousing poll
Workflow SLA Monitor	WorkflowSlaMonitorHostedService	calls IWorkflowSlaMonitor once a minute; a failed sweep also waits a minute
Wizard Storage Worker	WizardStorageWorker	first cycle is the startup migrate + sweep; then drains designer saves, idling in 30 s cycles

Every worker waits for model migration to finish (or be disabled) before its first cycle. The SLA monitor and wizard storage worker are registered by AddGenieWorkflow() / AddGenieWizard() and are not governed by AddWorkers(...).

Three startup hosted services are deliberately not monitored workers, because they run once rather than in a loop: EngineSqlBootstrapHostedService (a plain IHostedService whose StartAsync must finish before model migration starts), ModelMigrationService (the producer of the migration-ready signal every worker waits on; a failure must stop the host, not retry) and GenieWarmupHostedService (one-shot, best-effort). A worker that runs on a long fixed interval should override ErrorDelay with that interval, or a persistent fault retries every two seconds.

API — /api/v1/genie/hosted-services

Method	Route	Purpose
GET	/	live metrics for every registered service
GET	/{name}	metrics for one service
POST	/{name}/pause	pause a service’s processing loop (it idles until resumed)
POST	/{name}/resume	resume a paused service
GET	/{name}/history	last hour of cycle-summary history (from Redis)

Pausing stops the inner loop while leaving the background task alive — the worker polls until resumed. Cycle history is best-effort and returns an empty list when Redis is unavailable. The UI exposes all of this through the components/hosted-services admin dashboard.

Hangfire jobs

For durable background processing Genie integrates Hangfire (wired via the app builder’s ConfigureHangfire). Its admin API is restricted to the System role and short-circuits with 503 when Hangfire storage is not configured on the server.

API — /api/v1/hangfire

Method	Route	Purpose
GET	/stats	storage statistics (counts per state, servers, recurring jobs)
GET	/jobs	paginated, filterable job list (state, queue, method, date, id)
GET	/jobs/{id}	full job detail (parameters, state history, exception)
POST	/jobs/{id}/requeue	requeue a failed/deleted job
DELETE	/jobs/{id}	delete a job
GET	/queues	queues with current counts
GET	/servers	active Hangfire server instances
GET	/recurring	recurring job definitions
POST	/recurring/{id}/trigger	run a recurring job now
POST	/recurring/bulk-trigger · /jobs/bulk-requeue · /jobs/bulk-delete	batch operations

Which to use

Need	Use
Continuous in-process loop, live pause/resume, engine-internal	Hosted service (MonitoredBackgroundService)
Durable, retryable, scheduled/recurring, survives restarts	Hangfire job