Skip to content

DHI.Services.Jobs.WebApi — internal developer guide

DHI.Services.Jobs.WebApi is an ASP.NET Core package that exposes REST endpoints for:

  • Jobs (create, update status/heartbeat, cancel, query, health checks)
  • Tasks (discoverable executable units)
  • Hosts (machines/agents that run jobs)
  • Automations & Triggers (schedule/condition-based job creation)
  • Scenarios (domain object with optional linkage to jobs)
  • SignalR notifications for job events
  • Filesystem automation versioning (optional watcher)

It ships with:

  • opinionated JSON serialization (SerializerOptionsDefault)
  • file-backed JSON repositories that read/write under App_Data
  • reflection-based connection drivers (ServiceConnection types) that can instantiate your own repository implementations by type name + connection string

You can host the API in minutes using either ServiceLocator (programmatic) or the Connections module (JSON config). Everything below is ready to copy-paste.

1) Install

dotnet add package DHI.Services.Jobs.WebApi
# Typical companions:
dotnet add package DHI.Services.Scalars
dotnet add package DHI.Services.WebApiCore

2) Host application (Program + Startup)

Create a minimal ASP.NET Core project and use the following boilerplate.

Program.cs

using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;

namespace YourCompany.JobsApi
{
    public class Program
    {
        public static void Main(string[] args) =>
            CreateHostBuilder(args).Build().Run();

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(wb => wb.UseStartup<Startup>());
    }
}

appsettings.json (starter)

{
  "Tokens": {
    "Issuer": "https://your-issuer",
    "Audience": "dhi-jobs-api",
    "PublicRSAKey": "[AppData]rsa-public.pem"
  },
  "Swagger": {
    "SpecificationName": "v1",
    "DocumentName": "DHI Jobs Web API",
    "DocumentTitle": "DHI Jobs Web API",
    "DocumentDescription": "[AppData]swagger-description.md"
  },
  "AppConfiguration": {
    "HstsMaxAgeInDays": 30,
    "ContentRootPath": "",        // leave empty to use env.ContentRootPath
    "LazyCreation": true
  },
  "AutomationFolderWatcher": {
    "Enabled": true,
    "DebounceMilliseconds": 2000
  },
  "ConnectionStrings": {
    "TriggerCatalogDirectory": "" // optional MEF plugin dir for triggers
  }
}

Startup.cs (authoritative template)

Replace file names/paths as needed. Everything compiles if the referenced packages are present.

using System;
using System.Collections.Generic;
using System.IO;
using System.Security.Claims;
using DHI.Services.Jobs;
using DHI.Services.Jobs.Automations;
using DHI.Services.Jobs.Scenarios;
using DHI.Services.Jobs.WebApi;
using DHI.Services.Scalars;
using DHI.Services.WebApiCore;
using DHI.Services.WebApiCore.Notifications;
using DHI.Services.WebApiCore.Filters;
using DHI.Services.Jobs.Workflows;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.Versioning;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.IdentityModel.Tokens;
using Microsoft.OpenApi.Models;
using Swashbuckle.AspNetCore.SwaggerUI;

public class Startup
{
    public Startup(IConfiguration configuration, IWebHostEnvironment env)
    {
        Configuration = configuration;
        Environment = env;
    }

    public IConfiguration Configuration { get; }
    public IWebHostEnvironment Environment { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        // --- Authentication (JWT) ---
        services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
            .AddJwtBearer(options =>
            {
                options.TokenValidationParameters = new TokenValidationParameters
                {
                    ValidateIssuer = true,
                    ValidateAudience = true,
                    ValidateLifetime = true,
                    ValidateIssuerSigningKey = true,
                    ValidIssuer = Configuration["Tokens:Issuer"],
                    ValidAudience = Configuration["Tokens:Audience"],
                    IssuerSigningKey = RSA.BuildSigningKey(Configuration["Tokens:PublicRSAKey"].Resolve())
                };
            });

        // --- Authorization policies ---
        services.AddAuthorization(options =>
        {
            options.AddPolicy("AdministratorsOnly",
                policy => policy.RequireClaim(ClaimTypes.GroupSid, "Administrators"));
            options.AddPolicy("EditorsOnly",
                policy => policy.RequireClaim(ClaimTypes.GroupSid, "Editors"));
        });

        // --- API Versioning ---
        services.AddApiVersioning(o =>
        {
            o.ReportApiVersions = true;
            o.AssumeDefaultVersionWhenUnspecified = true;
            o.DefaultApiVersion = new ApiVersion(1, 0);
            o.ApiVersionReader = ApiVersionReader.Combine(
                new QueryStringApiVersionReader("api-version", "version", "ver"),
                new HeaderApiVersionReader("api-version"));
        });

        // --- MVC / JSON options aligned with our serializers ---
        services
            .AddCors()
            .AddResponseCompression()
            .AddControllers()
            .AddJsonOptions(o =>
            {
                o.JsonSerializerOptions.DefaultIgnoreCondition = SerializerOptionsDefault.Options.DefaultIgnoreCondition;
                o.JsonSerializerOptions.PropertyNamingPolicy   = SerializerOptionsDefault.Options.PropertyNamingPolicy;
                o.JsonSerializerOptions.AddConverters(SerializerOptionsDefault.Options.Converters);
            });

        // --- HSTS ---
        services.AddHsts(o =>
        {
            o.Preload = true;
            o.MaxAge = TimeSpan.FromDays(Configuration.GetValue<double>("AppConfiguration:HstsMaxAgeInDays"));
        });

        // --- Swagger ---
        services.AddSwaggerGen(setup =>
        {
            setup.SwaggerDoc(Configuration["Swagger:SpecificationName"], new OpenApiInfo
            {
                Title = Configuration["Swagger:DocumentTitle"],
                Version = "1",
                Description = File.Exists(Configuration["Swagger:DocumentDescription"].Resolve())
                    ? File.ReadAllText(Configuration["Swagger:DocumentDescription"].Resolve())
                    : "DHI Jobs Web API"
            });
            setup.EnableAnnotations();
            setup.IncludeXmlComments(Path.Combine(AppContext.BaseDirectory, "DHI.Services.Jobs.WebApi.xml"), includeControllerXmlComments: true);
            setup.AddSecurityDefinition("Bearer", new OpenApiSecurityScheme
            {
                Description = "Enter the word 'Bearer' followed by a space and your JWT.",
                Name = "Authorization",
                In = ParameterLocation.Header,
                Type = SecuritySchemeType.ApiKey,
                Scheme = "Bearer"
            });
            setup.AddSecurityRequirement(new OpenApiSecurityRequirement
            {
                { new OpenApiSecurityScheme{ Reference = new OpenApiReference{ Type = ReferenceType.SecurityScheme, Id = "Bearer"}}, new List<string>() }
            });
        });

        // --- SignalR (notifications hub) ---
        services.AddSignalR(hubOptions => hubOptions.EnableDetailedErrors = true);

        // --- Domain repositories (JSON under App_Data) ---
        services.AddScoped<IHostRepository>(_ => new GroupedHostRepository("grouped_hosts.json"));
        services.AddScoped<IAutomationRepository>(_ => new AutomationRepository("automations.json"));
        services.AddScoped<IScalarRepository<string,int>>(_ => new ScalarRepository("[AppData]scalars.json".Resolve()));
        services.AddSingleton<IFilterRepository>(_ => new FilterRepository("[AppData]signalr-filters.json".Resolve()));

        // Job repositories example: composite read-only
        services.AddSingleton<IJobRepository<Guid,string>>(_ =>
        {
            var repo1 = new JobRepository("[AppData]jobs.json".Resolve());
            var repo2 = new JobRepository("[AppData]jobs2.json".Resolve());
            return new ReadOnlyCompositeJobRepository(new[] { repo1, repo2 });
        });

        // Directory-backed automations (optional + temp)
        string tempAutomationsPath = "";
        services.AddScoped<DirectoryAutomationRepository>(sp =>
        {
            var env = sp.GetRequiredService<IWebHostEnvironment>();
            var path = Path.Combine(env.ContentRootPath, "App_Data", "automations_dir");
            tempAutomationsPath = path + "_temp";
            Directory.CreateDirectory(path);
            return new DirectoryAutomationRepository(path);
        });
        services.AddScoped<TempDirectoryAutomationRepository>(_ => new TempDirectoryAutomationRepository(tempAutomationsPath));

        // Automation folder watcher (optional)
        var autosRoot = Path.Combine(Environment.ContentRootPath, "App_Data", "automations_dir");
        services.AddAutomationFolderWatcher(Configuration, autosRoot);

        // If you want a very simple file logger:
        services.AddScoped<Microsoft.Extensions.Logging.ILogger>(_ => new DHI.Services.Notifications.SimpleLogger("[AppData]log.log".Resolve()));
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment()) app.UseDeveloperExceptionPage(); else app.UseHsts();

        app.UseAuthentication();
        app.UseHttpsRedirection();
        app.UseSwagger();
        app.UseSwaggerUI(ui =>
        {
            var spec = Configuration["Swagger:SpecificationName"];
            ui.SwaggerEndpoint($"../swagger/{spec}/swagger.json", Configuration["Swagger:DocumentName"] ?? "Jobs API");
            ui.DocExpansion(DocExpansion.None);
        });

        app.UseExceptionHandling();        // from WebApiCore
        app.UseResponseCompression();
        app.UseRouting();
        app.UseAuthorization();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
            endpoints.MapHub<NotificationHub>("/notificationhub");
        });

        // Pin App_Data:
        var contentRoot = Configuration.GetValue("AppConfiguration:ContentRootPath", env.ContentRootPath);
        AppDomain.CurrentDomain.SetData("DataDirectory", Path.Combine(contentRoot, "App_Data"));

        // ------------------------------------------------------------------
        // Choose ONE: (A) programmatic service registration  OR  (B) Connections
        // ------------------------------------------------------------------

        // (A) Programmatic registration via ServiceLocator (uncomment to use)
        /*
        ServiceLocator.Register(
            new JobService<Workflow, string>(
                new JobRepository("jobs.json"),
                new TaskService<Workflow, string>(new WorkflowRepository("[AppData]workflows.json", SerializerOptionsDefault.Options.Converters)),
                null),
            "wf-jobs"
        );

        ServiceLocator.Register(
            new TaskService<Workflow, string>(new WorkflowRepository("[AppData]workflows.json")),
            "wf-tasks"
        );

        ServiceLocator.Register(
            new ScenarioService(
                new ScenarioRepository("[AppData]scenarios.json".Resolve(), SerializerOptionsDefault.Options.Converters),
                new JobRepository("jobs.json")),
            "json-scenarios"
        );

        // Code workflow example:
        var cwRepo = new CodeWorkflowRepository("[AppData]workflows2.json".Resolve());
        var cwSvc  = new CodeWorkflowService(cwRepo);
        ServiceLocator.Register(cwSvc, "wf-tasks2");
        ServiceLocator.Register(
            new JobService<CodeWorkflow,string>(new JobRepository("jobs2.json"), cwSvc),
            "wf-jobs2");
        */

        // (B) Connections module (JSON-driven) — recommended for environments
        // Provide "connections.json" and enable the repository loader:
        /*
        var lazyCreation = Configuration.GetValue("AppConfiguration:LazyCreation", true);
        Services.Configure(new ConnectionRepository("connections.json"), lazyCreation);
        */
    }
}

3) Using the Connections module (JSON-driven services)

Enable (see Startup comment) and create connections.json at your content root or under App_Data. Example:

{
  "$type": "System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[DHI.Services.IConnection, DHI.Services]], mscorlib",
  "wf-tasks": {
    "$type": "DHI.Services.Jobs.WebApi.TaskServiceConnection, DHI.Services.Jobs.WebApi",
    "ConnectionString": "[AppData]workflows.json",
    "RepositoryType": "DHI.Services.Jobs.Workflows.WorkflowRepository, DHI.Services.Jobs",
    "Name": "Task service connection for workflows",
    "Id": "wf-tasks"
  },
  "wf-jobs": {
    "$type": "DHI.Services.Jobs.WebApi.JobServiceConnection, DHI.Services.Jobs.WebApi",
    "JobRepositoryConnectionString": "jobs.json",
    "JobRepositoryType": "DHI.Services.Jobs.WebApi.JobRepository, DHI.Services.Jobs.WebApi",
    "TaskRepositoryConnectionString": "[AppData]workflows.json",
    "TaskRepositoryType": "DHI.Services.Jobs.Workflows.WorkflowRepository, DHI.Services.Jobs",
    "Name": "Job service connection for workflows",
    "Id": "wf-jobs"
  },
  "json-scenarios": {
    "$type": "DHI.Services.Jobs.WebApi.ScenarioServiceConnection, DHI.Services.Jobs.WebApi",
    "ConnectionString": "[AppData]Scenarios.json",
    "RepositoryType": "DHI.Services.Jobs.Scenarios.ScenarioRepository, DHI.Services.Jobs",
    "JobRepositoryConnectionString": "jobs.json",
    "JobRepositoryType": "DHI.Services.Jobs.WebApi.JobRepository, DHI.Services.Jobs.WebApi",
    "Name": "Scenario service connection",
    "Id": "json-scenarios"
  }
}

You may replace repository types with your own providers; [AppData] resolves automatically.

4) Automation folder watcher (optional)

Registers a background watcher that writes version.txt under your automations directory whenever any *.json changes (debounced).

  • Configure in appsettings.json (AutomationFolderWatcher.Enabled=true)
  • Register via services.AddAutomationFolderWatcher(Configuration, <rootPath>) (we pass App_Data/automations_dir in Startup)

Use GET /api/automations/version (or /api/temp-automations/version) for a serialized timestamp clients can poll.

5) Routes you get (summary)

Notes

FullNameString: if an ID contains /, use | in URLs. The server maps Group|Sub|Name ↔ Group/Sub/Name.

Jobs — /api/jobs/{connectionId} (Authorize)

  • POST / (EditorsOnly) → create job (JobDTO)
  • GET /{id} → get job
  • GET /last?account&status&task&tag
  • GET / → query by querystring (account, since, status, task, tag)
  • POST /query → query by body (QueryDTO)
  • GET /count

System/ops:

  • PUT /status/{id} (EditorsOnly) → update status/progress (JobStatusUpdateDTO); terminal states (Completed/Error/Cancelled/TimedOut) are enforced
  • PUT /heartbeat/{id} (EditorsOnly)
  • PUT /cancel (EditorsOnly) array of GUIDs
  • PUT /{id}/cancel (EditorsOnly)
  • DELETE /{id} (AdministratorsOnly) optional status guard
  • DELETE / (AdministratorsOnly) account, before

SignalR notifications: job add/update/delete broadcasts to /notificationhub groups computed from filters in IFilterRepository.

Job health — /api/jobs/{connectionId}/health (Authorize)

  • GET /gap/{field}/{pastHours}/{taskId?}
  • GET /incomplete/{pastHours}/{taskId?}
  • GET /errorratio/{pastHours}/{maxErrorRatio}
  • GET /delay/{pastHours}/{maxDelayMinutes}
  • GET /minimumstarted/{pastHours}/{expectedNumberOfJobs}

Job hosts — /api/jobhosts (AdministratorsOnly)

  • POST / (HostDTO), PUT /, GET /{id}, GET /, GET /count, GET /fullnames, GET /ids, DELETE /{id}

Tasks — /api/tasks/{connectionId} (Authorize)

  • GET /{id} (supports Workflow and CodeWorkflow)
  • GET /, GET /ids, GET /count

Automations — /api/automations (AdministratorsOnly)

  • GET /{fullName}, GET /, GET /count, GET /fullnames, GET /ids
  • POST / (body must include $type for automation + predicates)
  • PUT / (validates id)
  • DELETE /{fullName}
  • PUT /{fullName}/enable with { "flag": true|false }
  • GET /version, POST /version (touch)

Temp automations (Trigger-Now queue) — /api/temp-automations (AdministratorsOnly)

  • Same as Automations, plus DELETE / to clear all

Triggers — /api/automations/triggers (AdministratorsOnly)

  • GET /{id}, GET /, GET /count, GET /ids

Scenarios — /api/scenarios/{connectionId} (Authorize; EditorsOnly to mutate)

  • GET / by time (from, to, optional dataSelectors=[a,b])
  • POST /query (QueryDTO<Scenario>, optional dataSelectors)
  • GET /{id} (optional dataSelectors)
  • POST / (EditorsOnly), PUT / (EditorsOnly), DELETE /{id}?softDelete=true|false (EditorsOnly)

6) Directory-backed automations (optional controller)

If you want each automation as a separate file under a directory (instead of a single JSON), expose the DirectoryAutomationRepository via your own controller. It plays nicely with the folder watcher.

7) Security model (must-have)

  • Auth: JWT Bearer.
  • Policies:
    • AdministratorsOnly → claim GroupSid=Administrators
    • EditorsOnly → claim GroupSid=Editors
  • Typical: Editors execute/cancel/update; Admins manage hosts/automations and system deletes.

8) JSON & polymorphism

We ship SerializerOptionsDefault:

  • enums serialized as strings
  • case-insensitive properties
  • number reading from strings
  • converters for polymorphic Automations/Triggers, connection types, dictionaries, etc. Use it in MVC (already done in Startup) and any custom code that serializes our domain types.

9) Request examples

Execute a job

curl -X POST "https://your.api/api/jobs/wf-jobs" \
  -H "Authorization: Bearer <JWT>" -H "Content-Type: application/json" \
  -d '{
        "taskId": "Ops|Nightly|Ingest",
        "parameters": { "InputPath": "s3://bucket/prefix", "Max": 10 },
        "tag": "nightly",
        "hostGroup": "prod",
        "priority": 1
      }'

Update status (stateless)

curl -X PUT "https://your.api/api/jobs/wf-jobs/status/7299875c-ee1f-4f20-bb30-066cc267f1bd" \
  -H "Authorization: Bearer <JWT>" -H "Content-Type: application/json" \
  -d '{ "jobStatus": "Completed", "statusMessage": "OK", "progress": 100 }'

Enable an automation

curl -X PUT "https://your.api/api/automations/Ops|Nightly|Ingest/enable" \
  -H "Authorization: Bearer <JWT>" -H "Content-Type: application/json" \
  -d '{ "flag": true }'

Automation version (for UI cache busting)

curl "https://your.api/api/automations/version"

10) Operational notes & pitfalls

  • Terminal states: once Completed, Error, Cancelled, or TimedOut, further status changes are rejected (409) unless idempotent (same status).
  • Health endpoints: the {field} path segment must be a DateTime/DateTime? property of Job (case-insensitive; we normalize the first letter).
  • FullNameString: use | in URLs where your logical key contains /. Internally we map it back.
  • SignalR groups: groups are driven by filters from IFilterRepository. Every job event is matched against filters; matching filter IDs are used as SignalR groups.
  • App_Data: we set AppDomain.CurrentDomain.SetData("DataDirectory", <contentRoot>/App_Data) at startup; our WebApi repos resolve paths relative to this.
  • Trigger catalog (optional): supply a directory path in ConnectionStrings:TriggerCatalogDirectory to MEF-load additional trigger types at runtime.

11) Extensibility surface

  • Swap any repository with your own by setting RepositoryType to a fully-qualified type name and providing the ConnectionString (either via ServiceLocator or Connections).
  • Add your own controllers alongside ours; our SerializerOptionsDefault ensures consistent payloads.
  • Plug your own logger; we use ILogger everywhere. A simple file logger (SimpleLogger) exists for quick starts.

That’s the whole contract. Drop this package into a new ASP.NET Core project, paste the Startup above, decide between ServiceLocator or Connections, and you have a fully working Jobs API with hosts, tasks, automations, scenarios, health checks, and SignalR notifications.