Skip to content

Job Automator Project Template

You can find the official source for the template here:
GitHub – JobAutomator Template

What this template provides

  1. Background service (Windows Service host by default) that runs every ExecutionIntervalSeconds
  2. Trigger execution engine (AutomationExecutor) that evaluates conditions and returns dynamic task parameters
  3. Host-group–aware job creation using a routing map (e.g., Minion, Titan, …)
  4. Scalar writes for:
    • …/<HostGroup>/Last Job Id
    • …/<HostGroup>/<TriggerId>/Is Met (or …/<HostGroup>/Is Met for “all-AND” mode)
  5. Cached automations with freshness check via the Automations API /api/automations/version
  6. Token management (username/password -> access + refresh token) for calling external Domain Services APIs
  7. Serilog logging (console + rolling file)

Prerequisites

Configuration

All settings live in appsettings.json (environment overrides supported).

Example

{
  "Authentication": {
    "Url": "http://DS-DEV1:81",
    "UserName": "demo_admin_to_be_deleted",
    "Password": "webapi"
  },
  "ConnectionStrings": {
    "Scalars": "http://DS-DEV1:82",
    "Automations": "http://DS-DEV1:82",
    "Jobs": "http://DS-DEV1:82"
  },
  "JobAutomator": {
    "ExecutionIntervalSeconds": 10.0,
    "LogTriggerStatus": false
  },
  "JobHostGroupRouting": {
    "DefaultKey": "wf-jobs-Minion",
    "ContainsMap": {
      "minion": "wf-jobs-Minion",
      "titan":  "wf-jobs-Titan"
    }
  },
  "Cache": {
    "LocalVersionFilePath": "C:\\Services\\JobAutomator\\Automations\\Version"
  },
  "Serilog": {
    "Using": ["Serilog.Sinks.Console","Serilog.Sinks.File"],
    "MinimumLevel": "Debug",
    "WriteTo": [
      { "Name": "Console" },
      {
        "Name": "File",
        "Args": {
          "path": "C:\\Services\\JobAutomator\\Log\\JobAutomation-.log",
          "rollingInterval": "Day",
          "retainedFileCountLimit": 30
        }
      }
    ]
  }
}

Key settings

  1. Authentication Base URL of Auth Server and credentials used by the automator to obtain/refresh JWTs.
  2. ConnectionStrings These are base URLs for the Domain Services web APIs (the template uses DS providers that call HTTP endpoints, not direct DB connections):
    • Automations: e.g., http://.../api/automations
    • Scalars: e.g., http://.../api/scalars
    • Jobs: e.g., http://.../api/jobs and …/api/tasks
  3. JobAutomator
    • ExecutionIntervalSeconds: poll interval
    • LogTriggerStatus: log a line for each automation with the trigger result
  4. JobHostGroupRouting
    • DefaultKey: fallback service key (must match a Jobs Web API service key)
    • ContainsMap: maps substrings in HostGroup (from the automation JSON) to service keys used by Jobs Web API, e.g. "minion" -> "wf-jobs-Minion"
  5. Cache
  6. LocalVersionFilePath: path to a local version.txt that mirrors the server-side /api/automations/version timestamp (used by the caching repo to invalidate)
  7. Serilog
  8. Console + rolling file out-of-the-box

Tip: Store secrets via environment variables or user-secrets. Credentials here are for development only.

How it works

Job Automator – runtime flow

Where things live (at a glance)

  1. Service host / lifecycle
    • AutomatorBackgroundService.StartAsync() -> starts timers and token refresh
    • AutomatorBackgroundService.StopAsync() -> stops timers
  2. Main loop & decisions
    • JobAutomator.Start() / Stop() / ExecutionTimerElapsed(...)
    • AutomationExecutor.Execute(...) (trigger evaluation & “Is Met” scalars)
  3. Job routing & submission
    • JobServiceFactory.GetJobService(hostGroup) (maps host group -> jobs service key)
    • JobAutomator.CreateNewJob(...) (constructs & enqueues the job)
  4. Last job state (duplicate-prevent)
    • JobAutomator.ReadLastJobIdScalar(...)
    • JobAutomator.WriteLastJobIdScalar(...)
  5. HTTP providers, token, and caching
    • DependencyFactory (builds DS HTTP providers)
    • AccessTokenProvider (auth + refresh)
    • CachingAutomationRepository (server-driven cache invalidation)

1) Dependency wiring (HTTP-based DS providers)

DependencyFactory builds the HTTP-based providers with circuit breakers/retries:

  • Automations: DHI.Services.Provider.DS.AutomationRepository ({Automations}/api/automations)
  • Scalars: DHI.Services.Provider.DS.ScalarRepository ({Scalars}/api/scalars/wf-scalars)
  • Jobs: DHI.Services.Provider.DS.JobRepository ({Jobs}/api/jobs/{serviceKey})
  • Tasks: DHI.Services.Provider.DS.CodeWorkflowRepository ({Jobs}/api/tasks/wf-tasks)

It also constructs:

  • AccessTokenProvider: handles login + refresh token rotation against Auth Server.
  • CachingAutomationRepository: wraps the HTTP repo with in-memory cache and server-driven invalidation via /api/automations/version.

2) Caching & freshness

CachingAutomationRepository caches GetAll, Get(id), GetByGroup(...), and ContainsGroup(...). On every read it compares:

  • Server timestamp = GET {Automations}/api/automations/version (requires Bearer token)
  • Local timestamp = Cache.LocalVersionFilePath\version.txt

If server > local, it clears cache and rewrites local timestamp.

Note: In this template, only LocalVersionFilePath is used from Cache; TTLs are fixed internally.

3) Execution loop

JobAutomator runs every ExecutionIntervalSeconds:

  1. Pull all automations (AutomationService.GetAll()).
  2. Skip if IsEnabled = false.
  3. Evaluate with AutomationExecutor:
    • Two modes
      • All-AND (no Conditional string): every enabled trigger must pass.
      • Expression mode (Conditional: e.g., A && (B || C)), where A/B/C are trigger IDs.
    • Each trigger returns AutomationResult with optional dynamic task parameters.
    • Writes trigger “Is Met” scalars for observability.
  4. If met:
    • Read …/<HostGroup>/Last Job Id scalar (if any).
    • If last job is Pending/Starting/InProgress, do not enqueue again.
    • Else create job:
      • TaskId from the automation
      • Parameters = automation’s TaskParameters trimmed + dynamic params from triggers
      • HostGroup, Priority, Tag copied across
    • Write …/<HostGroup>/Last Job Id scalar to the new job id.

Scalar paths written by the automator

  • Trigger result: Job Automator/<MachineName>/<AutomationId>/<HostGroup>[/<TriggerId>]/Is Met -> bool
  • Last job id: Job Automator/<MachineName>/<AutomationId>/<HostGroup>/Last Job Id -> Guid

4) Host-group routing

JobServiceFactory resolves which Jobs Web API service key to use from an automation’s HostGroup:

  • Check ContainsMap (substring match on hostGroup) -> map to a service key, e.g. "minion" -> "wf-jobs-Minion".
  • If unmatched, fall back to DefaultKey.
  • Registers one JobService per service key, backed by {Jobs}/api/jobs/{serviceKey}.

This mirrors how the Automations Web API reads from multiple host job databases (it registers all and composes for read). Here, the automator writes to the single host selected per automation.

Triggers & dynamic parameters

Automations include a TriggerCondition with a list of $typed triggers (e.g., SqlTrigger). At runtime, the automator deserializes trigger instances (you must reference their assemblies) and calls Execute():

  • Execute() returns AutomationResult.Met(Dictionary<string,string> parameters) or NotMet().
  • Returned parameters are merged (and trimmed to the workflow’s expected keys) before job creation.

Example: SqlTrigger (built-in) supports Postgres, SQL Server/Azure SQL, SQLite, MySQL. For bespoke logic, add your own trigger class to a referenced assembly and point $type at it in your automation JSON.

Running the Job Automator

Windows Service (default)

The template configures a Windows Service host:

applicationBuilder.Services.AddWindowsService(options => {
  options.ServiceName = AutomatorBackgroundService.ServiceName; // "DHI Job Automator"
});
  • Build & install your service (e.g., with sc.exe or your deploy scripts).
  • Logs go to console (when run interactively) and to the Serilog file sink configured in appsettings.json.

Console / container

If you prefer a console app or container:

  • Remove/guard the AddWindowsService(...) call with environment conditions.
  • Run the host normally; health is in logs.

Deploy & Install (Windows Service or Container)

The template runs best as a Windows Service (already wired via AddWindowsService). You can also run it as a console or in a container.

0) Prereqs

  1. .NET 8 Runtime (for framework-dependent publish) or publish self-contained.
  2. A service account with:
    • Read access to the app folder and appsettings*.json
    • Write access to:
      • Cache.LocalVersionFilePath directory (e.g., C:\Services\JobAutomator\Automations\Version)
      • Serilog log folder (e.g., C:\Services\JobAutomator\Log\)
    • Outbound HTTPS/HTTP to your DS endpoints (Authentication.Url, ConnectionStrings:Automations/Scalars/Jobs)
  3. Auth Server + DS Web APIs reachable and configured.

1) Publish the app

From the solution root:

# Framework-dependent (requires .NET 8 runtime on the server)
dotnet publish JobAutomator.csproj -c Release -o .\publish\win-x64

# OR self-contained (no runtime needed on the server)
dotnet publish JobAutomator.csproj -c Release -r win-x64 --self-contained true -p:PublishSingleFile=true -o .\publish\win-x64-sc

Copy the output to the server, e.g. C:\Services\JobAutomator\app\.

Put your environment config next to the exe:

  • appsettings.json (shared)
  • appsettings.Prod.json (overrides)
  • Logs: C:\Services\JobAutomator\Log\
  • Cache version file directory: C:\Services\JobAutomator\Automations\Version\

2) Configure settings

Edit appsettings.json (or appsettings.Prod.json) on the server:

  • Authentication:Url/UserName/Password
  • ConnectionStrings:Automations/Scalars/Jobs -> base URLs of your DS Web API
  • JobAutomator:ExecutionIntervalSeconds
  • JobHostGroupRouting -> ensure keys (e.g., wf-jobs-Minion) match your Jobs Web API service keys
  • Cache:LocalVersionFilePath -> a writable folder path (the template writes version.txt there)

3) Install as a Windows Service

The binary path must include the full path to the exe. If you use appsettings.Prod.json, pass --Environment Prod or set an env var.

$svcName = "DHI Job Automator"
$exePath = "C:\Services\JobAutomator\app\JobAutomator.exe"
$bin    = "`"$exePath`" --Environment Prod"

New-Service -Name $svcName `
  -BinaryPathName $bin `
  -DisplayName $svcName `
  -StartupType Automatic `
  -Description "Polls Automations, evaluates triggers, creates jobs, writes scalars."

# Set recovery (restart on failure)
sc.exe failure  "DHI Job Automator" reset= 86400 actions= restart/5000/restart/5000/restart/5000 | Out-Null
sc.exe failureflag "DHI Job Automator" 1 | Out-Null

# Start it
Start-Service "DHI Job Automator"

Using sc.exe directly

sc create "DHI Job Automator" binPath= "\"C:\Services\JobAutomator\app\JobAutomator.exe\" --Environment Prod" start= auto
sc description "DHI Job Automator" "Polls Automations, evaluates triggers, creates jobs, writes scalars."
sc start "DHI Job Automator"

Tip: If you run under a custom account, grant “Log on as a service” and NTFS Modify on Log\ and the Cache.LocalVersionFilePath directory.

4) Verify

  • Get-Service "DHI Job Automator" -> Running
  • Check logs at C:\Services\JobAutomator\Log\JobAutomation-*.log
    • You should see:
      • “Authenticating… / Token refreshed”
      • “Execution timer started.”
      • Lines about automations count, trigger status (enable JobAutomator:LogTriggerStatus=true for detail)

5) Update (in-place)

Stop-Service "DHI Job Automator"
# Replace files in C:\Services\JobAutomator\app\ with new publish output
Start-Service "DHI Job Automator"

If you change the service name or move paths, uninstall/reinstall (below).

6) Uninstall

Stop-Service "DHI Job Automator"
sc.exe delete "DHI Job Automator"

Running as a Console (no service)

For quick testing:

cd C:\Services\JobAutomator\app
.\JobAutomator.exe --Environment Prod

Press Ctrl+C to stop.

Container deployment (optional)

If you prefer to run the automator in a container:

  1. Guard the Windows service registration (optional): if you keep AddWindowsService, run on a Windows container; for Linux containers, run as a plain console host (you can conditionally add it only on Windows).

  2. Dockerfile (Linux)

FROM mcr.microsoft.com/dotnet/runtime:8.0
WORKDIR /app
COPY publish/ ./
# Environment overrides (examples):
# ENV ASPNETCORE_ENVIRONMENT=Prod \
#     Authentication__Url=https://auth.example.com \
#     ConnectionStrings__Automations=https://ds.example.com \
#     ConnectionStrings__Scalars=https://ds.example.com \
#     ConnectionStrings__Jobs=https://ds.example.com
ENTRYPOINT ["./JobAutomator"]
  1. Run
docker run -d --name job-automator \
  -e ASPNETCORE_ENVIRONMENT=Prod \
  -e Authentication__Url=https://auth.example.com \
  -e Authentication__UserName=automator \
  -e Authentication__Password=*** \
  -e ConnectionStrings__Automations=https://ds.example.com \
  -e ConnectionStrings__Scalars=https://ds.example.com \
  -e ConnectionStrings__Jobs=https://ds.example.com \
  -v C:/Services/JobAutomator/Log:/app/Log \
  -v C:/Services/JobAutomator/Automations/Version:/app/Version \
  yourrepo/job-automator:latest

Mount volumes for logs and for Cache.LocalVersionFilePath (or override the path to a container-local folder).

Common Pitfalls

  • 401/403 to DS APIs -> wrong Authentication config, user lacks roles, or clock skew; verify logs show “Token refreshed…”.
  • No jobs created -> trigger not met; enable JobAutomator.LogTriggerStatus=true and check “Is Met” scalars; also ensure HostGroup maps to a registered service key.
  • Cache never refreshes -> Cache.LocalVersionFilePath not writable or Automations API /api/automations/version unreachable.
  • TLS issues -> use trusted certs on DS endpoints; install root CA on the server if using internal PKI.

Minimal wiring checklist

  1. Auth Server reachable at Authentication.Url with valid credentials.
  2. Jobs/Scalars/Automations base URLs set under ConnectionStrings.
  3. Job service keys exist in your Jobs Web API (e.g., wf-jobs-Minion, wf-jobs-Titan) and your routing maps to them.
  4. Automations contain HostGroup values that match your routing.
  5. Automations Web API exposes /api/automations/version (used by cache freshness).

Troubleshooting

  • 401/403: Auth URL or credentials wrong; token not attached; clock skew; or missing roles/policies on the target APIs.
  • Jobs never enqueue
    • Trigger not met (enable JobAutomator.LogTriggerStatus = true).
    • HostGroup not matching any routing rule -> falls back to default (check logs).
  • Cache never refreshes
    • Verify Automations API GET /api/automations/version returns an ISO-8601 timestamp.
    • Check Cache.LocalVersionFilePath is writable.
  • Custom trigger not executing
    • Ensure the assembly containing the trigger type is referenced and $type in JSON is fully qualified.