Jobs Automations Web API Project Template

This template primarily uses:

• DHI.Services.Jobs.WebApi -> automation + jobs endpoints
• DHI.Services.Scalars.WebApi (when included) -> scalars endpoints
• DHI.Services.PostgreSQL -> Jobs / Scalars / Filters / Notifications stores
• File-based DirectoryAutomationRepository for storing automations

You can find the official source for the template here: GitHub – Jobs Automations Web API Template

---

Overview

This template provides:

1. JWT auth with “AdministratorsOnly/EditorsOnly” policies (configurable bypass via AllowAnonymousAccess).
2. CORS, API versioning, HSTS, response compression, exception handling.
3. Swagger/OpenAPI (XML comments ready) + Serilog (console, file, optional Seq).
4. Automations Repository: file-backed via DirectoryAutomationRepository (one JSON per automation).
5. Triggers Repository via MEF: discover trigger types + their parameters at runtime.
6. Read-only composite Job repository: read jobs across multiple Postgres DBs (to enrich automations with last job info).
7. SignalR hub (/notificationhub) enabled for live updates.

Typical flow:

1. DS Web Ops calls this API to CRUD Automations.
2. Job Automator polls this API, evaluates triggers, and creates jobs in the correct HostGroup DB.
3. Scalars are created/updated by Job Automator and surfaced here for UI enrichment (e.g., IsMet, LastJob).

---

Installing Packages

Add the base packages (adjust to your needs):

dotnet add package DHI.Services.Jobs.WebApi
dotnet add package DHI.Services.Scalars.WebApi
dotnet add package DHI.Services.PostgreSQL
dotnet add package Serilog.AspNetCore

If you plan to export logs to Seq:

dotnet add package Serilog.Sinks.Seq

---

Configuration

All configuration is in appsettings.json.

Required keys (sample)

{
  "AppConfiguration": { 
    "HstsMaxAgeInDays": 1
  },
  "JobAutomations": "C:\\Services\\JobAutomator\\Automations",
  "Tokens": {
    "Issuer": "dhigroup.com",
    "Audience": "dhigroup.com",
    "PublicRSAKey": "[env:PublicRSAKey]"
  },
  "Postgres-webAPI": {
    "ConnectionString": "Server=localhost;Port=5432;Database=JobWorkflow;User Id=Solutions;Password=P@ssword!"
  },
  "Postgres-filter": {
    "ConnectionString": "Server=localhost;Port=5432;Database=JobWorkflow;User Id=Solutions;Password=P@ssword!"
  },
  "Postgres-log": {
    "ConnectionString": "Server=localhost;Port=5432;Database=JobWorkflow;User Id=Solutions;Password=P@ssword!;Table=public.logging;"
  },
  "Postgres-scalars": {
    "ConnectionString": "Server=localhost;Port=5432;Database=JobWorkflow;User Id=Solutions;Password=P@ssword!;"
  },
  "Postgres-jobs": [
    { 
      "JobRepositoryConnectionString": "Server=localhost;Port=5432;Database=JobMinion;User Id=Solutions;Password=P@ssword!", 
      "Id": "wf-jobs-Minion"
    },
    {
      "JobRepositoryConnectionString": "Server=localhost;Port=5432;Database=JobTitan;  User Id=Solutions;Password=P@ssword!",
      "Id": "wf-jobs-Titan" 
    }
  ],
  "TaskRepositoryConnectionString": "[AppData]Workflows.json",
  "Swagger": {
    "SpecificationName": "MyApplicationOpenAPISpecification",
    "DocumentName": "MyApplicationWebAPI",
    "DocumentTitle": "My Application Web API",
    "DocumentDescription": "[AppData]SwaggerInfo.md"
  },
  "JobLogPath": "C:\\Services\\WorkflowLogService",
  "AllowAnonymousAccess": "True",
  "PermittedOrigins": [ "http://localhost:82", "http://localhost:83" ],

  "Serilog": {
    "Using": ["Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.Seq"],
    "MinimumLevel": "Debug",
    "WriteTo": {
      "0": { "Name": "Console" },
      "1": { "Name": "File", "Args": { "path": "Log/webApi.log" } },
      "2": { "Name": "Seq",  "Args": { "serverUrl": "http://localhost:5341" } }
    }
  },

  "ConnectionStrings": {
    "TriggerCatalogDirectory": "C:\\Services\\JobAutomator\\TriggerPlugins"
  }
}

What each setting does

1. JobAutomations – root folder where automation JSON files live (one file per automation).
2. Postgres-jobs[] – list of host groups (by Id) with dedicated Job repositories.
   • These are registered as JobService instances under their Id.
   • Automations use HostGroup to select the target job DB when executed.
3. Postgres-scalars – Postgres DB for scalars.
4. TaskRepositoryConnectionString – points to your workflow/task repository (e.g., [AppData]Workflows.json) so the API can validate referenced tasks.
5. AllowAnonymousAccess – when true, the API bypasses authentication.
6. PermittedOrigins – CORS allowed origins for DS Web Ops.
7. ConnectionStrings:TriggerCatalogDirectory – optional folder containing MEF trigger parameter assemblies.

Tip: use [env:...] placeholders for secrets (e.g., RSA keys, DB credentials).

---

Repository & Services (what’s wired up)

• Automations -> DirectoryAutomationRepository (file-based); versioning via a version.txt in the same folder.
• Jobs (read) -> ReadOnlyCompositeJobRepository combining all Postgres-jobs DBs to find the last job per automation across hosts.
• Jobs (write) -> not performed by this Web API. Jobs are created by Job Automator, targeting the HostGroup’s job repository.
• Scalars -> Postgres ScalarRepository; API/Service uses them to enrich automations (IsMet, LastJob).
• Filters / Notifications / Logging -> Postgres providers + Serilog (console, file, optional Seq).
• Triggers -> MEF catalog combining built-in triggers + optional plugin directory; exposed via Triggers endpoints.

---

Endpoints

Automations

Base route: GET/POST/PUT/DELETE /api/automations (v1; secured)

• GET /api/automations List all automations (enriched with IsMet, LastJob when available).
• GET /api/automations?group=BlueCast List automations within a group.
• GET /api/automations/{id} Get a single automation by FullName (URL-encoded).
• GET /api/automations/count Count of automations.
• GET /api/automations/fullnames (or ?group=...) FullName strings for all or a group.
• GET /api/automations/ids Short IDs for all automations.
• POST /api/automations (body = Automation JSON) Create an automation. Requires $type for concrete automation & triggers.
• PUT /api/automations (body = Automation JSON) Update an automation by FullName in body.
• DELETE /api/automations/{id} Delete by FullName.
• PUT /api/automations/{id}/enable (body: { "flag": true|false }) Enable/disable automation (supports snooze patterns downstream via Job Automator logic).
• GET /api/automations/version Returns repository change timestamp (O format) if supported by repo.

Trigger Discovery (MEF)

Base route: GET /api/automations/triggers (v1; secured)

• GET /api/automations/triggers Returns a list of TriggerParameters describing available trigger types and their properties/requirements (from MEF).
• GET /api/automations/triggers/{id} Returns the parameter schema for one trigger type (by Id).

Use these Trigger endpoints in DS Web Ops to build dynamic forms for automation creation without hardcoding trigger fields.

---

Automation JSON (example)

Key idea: different triggers have different fields. We model that with $type on the trigger and extra properties captured by JsonExtensionData.

{
  "$type": "DHI.Services.Jobs.Automations.Automation, DHI.Services.Jobs.Automations",
  "Id": "BlueCast/vdjtesttable2024b",
  "Name": "vdjtesttable2024b",
  "FullName": "BlueCast/vdjtesttable2024b",
  "Group": "BlueCast",
  "IsEnabled": true,
  "Priority": 1,
  "HostGroup": "Minion",
  "Tag": "vdjtesttable2024b",
  "TaskId": "Workflows.UpdateSysProgressTableTestWF",
  "TaskParameters": {
    "TerminationGracePeriod": "0.00:00:10",
    "WorkflowTimeout": "0.00:05:00",
    "BaseTime10": "",
    "BlockID": "",
    "MySqlTable": "vdjtesttable2024b",
    "MySqlCredentials": "C:/Services/JobExecutor/systemprogress_wfdev_rw.json"
  },
  "TriggerCondition": {
    "Conditional": "bluecastTriggerCon",
    "Triggers": [
      {
        "$type": "DHI.Services.Jobs.Automations.Triggers.BluecastTrigger, DHI.Services.Jobs.Automations",
        "ConnectionString": "Server=wf-depl1;User ID=web;Password=webaccess;Database=systemprogress",
        "DbmsType": "MySQL",
        "Description": "bluecast Trigger",
        "Id": "bluecastTriggerCon",
        "IsEnabled": true,

        // trigger-specific fields below (captured via JsonExtensionData)
        "UseBulkCaching": "true",
        "JobTable": "vdjtesttable2024b",
        "JobType": "vdjtesttable2024b",
        "BasetimeIntervalHours": "6",
        "BlockOrder": "H06;B1;B2;B3;B4;B5",
        "InitiateHours": "-3.25",
        "ExpiryHours": "H06=>-120.25;B1=>-48;-18",
        "MaxRunCount": "H06=>10;3",
        "RestartDelayMinutes": "H06=>15;6",
        "HotWaitBlocks": "H06",
        "ChainWaitBlocks": "B*",
        "PrerequisiteWaitJobs": ""
      }
    ]
  }
}

Important fields:

• HostGroup – maps to a job repository (Postgres-jobs[].Id) where new jobs will be created by the Job Automator.
• TaskId / TaskParameters – the workflow to run (validated against the wf-tasks service).
• TriggerCondition – either expressed via Conditional (logical expression of trigger ids) or AND-combined when omitted.
• IsEnabled – quick switch to pause/resume the automation.

---

Triggers & MEF

• Triggers implement ITrigger (or derive from BaseTrigger).
• Parameter discovery is provided by small classes exported via MEF (ITriggerParameters) using [TriggerParameter] attributes.
• Drop your parameter exporters into the folder pointed to by ConnectionStrings:TriggerCatalogDirectory to make them show up in /api/automations/triggers.

Example (SQL Trigger parameter export):

[Export(typeof(ITriggerParameters))]
[ExportMetadata("Id", nameof(SqlTrigger))]
public class SqlTriggerParameters : ITriggerParameters, ISqlTriggerParameters
{
    [TriggerParameter(true, title: "The queries to run", description: "SQLs executed in sequence")]
    public string[] Queries { get; set; }

    [TriggerParameter(true, title: "Database Connection String")]
    public string ConnectionString { get; set; }

    [TriggerParameter(true, title: "Database Type")]
    public DbmsType DbmsType { get; set; }

    [TriggerParameter(true, title: "Description")]
    public string Description { get; }
}

---

Running the Automations Web API

Prereqs

• .NET SDK (check the template on GitHub for the current target framework)
• PostgreSQL instances for Jobs/Scalars/Filters/Log (per connection strings)
• A directory for JobAutomations (the repo will create it if missing)

Steps

1. Configure appsettings.json (tokens, Postgres, repo paths, CORS, Serilog).
2. Ensure the JobAutomations folder exists and is writable.
3. (Optional) Provide a TriggerCatalogDirectory with MEF trigger parameter assemblies.
4. Run:

dotnet run

5. Open Swagger at:

http://localhost:5000/swagger

---

Security Notes

• Auth: JWT via Auth Server; keep ClockSkew = 0 for strict validation.
• Policies: By default the template allows any authenticated user (RequireAuthenticatedUser) for Admin/Editor policies. For production, revert to claims (uncomment the GroupSid policies in code).
• CORS: set PermittedOrigins for DS Web Ops.
• Secrets: Use [env:...] for keys/connection strings.

For Auth Server details, see the Authorization Server Project Template.

---

How it fits with Job Automator

• This Web API stores and serves Automations (file-based repo), exposes Triggers, and reads Jobs/Scalars to enrich UI.
• Job Automator (separate worker) polls this API, evaluates triggers, creates Jobs in the correct HostGroup database, and updates Scalars.

See the high-level system diagram: Architecture Overview

---

Quick API Examples

List triggers (for dynamic forms):

curl -H "Authorization: Bearer <token>" http://localhost:5000/api/automations/triggers

Create an automation:

curl -X POST -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d @automation.json \
     http://localhost:5000/api/automations

Enable/disable:

curl -X PUT -H "Authorization: Bearer <token>" \
     -H "Content-Type: application/json" \
     -d '{ "flag": true }' \
     "http://localhost:5000/api/automations/BlueCast%2Fvdjtesttable2024b/enable"

---

Related Documentation

• Architecture Overview
• Authorization Server Project Template

---