DHI.Services.Jobs.Automations — Internal Developer Guide

This is a concise guide to help DHI developers use, extend, and integrate DHI.Services.Jobs.Automations. The module ships on NuGet and is safe to consume without access to the private source.

---

What the module does

An Automation represents “when X happens, run task Y with parameters Z”.

• Automation (Automation<TTaskId>): named, grouped unit that targets a task id and carries task parameters.
• Triggers (ITrigger): conditions that decide if an automation is met (ready to run).
• TriggerCondition: either “implicit AND” (all enabled triggers must pass) or a boolean expression over trigger ids, e.g. A AND (B OR C).
• Executor (AutomationExecutor): evaluates triggers, merges/filters parameters, logs, and returns an AutomationResult.
• Repositories/Services: JSON-backed stores for automations + a MEF-based TriggerRepository that exposes trigger parameter metadata for Web/API UIs.

---

The important types

• Automation<TTaskId> & Automation
  • TaskId, TaskParameters, HostGroup, Priority, Tag, IsEnabled, TriggerCondition, LastJob, IsMet.
• TriggerCondition
  • Triggers : List<ITrigger>, Conditional : string? (boolean expression using trigger Ids).
• ITrigger / BaseTrigger
  • Id, Description, IsEnabled, Type, Extra (JSON extension data), Execute(ILogger, IDictionary<string,string>?).
• Built-in triggers (namespace ...Automations.Triggers):
  • ScheduledTrigger (time-based),
  • JobCompletedTrigger (checks last job status),
  • SqlTrigger (runs 1..N SQL statements sequentially, passing row params).
• AutomationExecutor
  • Evaluates triggers, writes Scalars: per-trigger Is Met, plus Last Log and Last Met Log.
• Repos/Services
  • Automation: AutomationRepository (single JSON file), DirectoryAutomationRepository (one file per automation), TempDirectoryAutomationRepository (scratch space; doesn’t touch version).
  • Trigger metadata (for Web/API UI generation): TriggerRepository (MEF composition) and TriggerService.

---

Parameter flow (how values are built)

When the executor runs a trigger it builds execution parameters as:

1. Automation.TaskParameters

   → 2) Public readable properties on the trigger instance (converted to invariant strings)

   → 3) BaseTrigger.Extra (JSON-injected per-trigger params)

   → 4) Scheduled-only overlay for keys utcNow, utcPrev, toleranceSeconds coming from runParams (if provided)

Special keys recognized anywhere in the bag:

• TriggerNow (bool-ish): if true, bypasses the real condition and returns Met() for all built-in triggers. Also causes executor to ignore IsEnabled flags.
• Scheduled overlays:
  • utcNow, utcPrev: UTC timestamps to evaluate boundaries deterministically (useful for workers)
  • toleranceSeconds: acceptable drift window (default 30s)

Trimming: the executor returns only the subset of parameters whose keys exist in Automation.TaskParameters (to avoid accidental leaks).

---

Built-in triggers (how to use)

1) ScheduledTrigger

• Ctor: (id, description, startTimeUtc, interval) (interval ≥ 1 minute)
• Execute logic:
  • If TriggerNow true → met.
  • Otherwise: met when now crosses a boundary of start + k*interval.
  • First-ever evaluation (no utcPrev) uses a tolerance window (default 30s, override with toleranceSeconds).
• Optional run params: utcNow, utcPrev, toleranceSeconds (seconds, non-negative).

2) JobCompletedTrigger

• Generic form checks last job for (TJobId,TTaskId) via an IJobRepository.
• Concrete provided: JobCompletedTrigger : JobCompletedTrigger<Guid,string>
• Ctor: (id, taskId, repositoryType, connectionString)
  • repositoryType must implement IJobRepository<Guid,string> and have a ctor (string connectionString).
  • connectionString supports:
    • [env:VARIABLE] → replaced by environment variable.
    • [AppData] → replaced with AppDomain.CurrentDomain.DataDirectory (must be set at app startup).

3) SqlTrigger

• Ctor: (connectionString, dbmsType, queries[], id, description) (or internal ctor with an existing IDbConnection).
• Supported DBMS: Postgres, SqlServer/AzureSql, SqLite, MySQL. (Undefined throws.)
• Behavior:
  • Runs Queries[0] with AutomationParameters → must return ≥1 row.
  • For each row, runs Queries[1] with merged params {row + automationParams}, collects rows, and so on.
  • If last query returns rows, returns Met(finalRowAsDictionary); otherwise NotMet.

---

Trigger metadata → Web/API UI

To make triggers discoverable in Web/API and generate forms, implement parameter classes exported via MEF:

• Implement marker ITriggerParameters (must have a Description prop).
• Decorate with:
  • [Export(typeof(ITriggerParameters))]
  • [ExportMetadata("Id", nameof(YourTriggerClass))]
• Use [TriggerParameter(required: bool, title: "...", description: "...", format: "...")] on public settable properties you want exposed.
• The TriggerRepository reflects these classes and builds a JSON-schema-like TriggerParameters object (with types, enums, arrays, nested objects, required list, etc.).

---

Repositories you’ll actually use

Pick one persistence style:

• Single file JSON: var repo = new AutomationRepository(pathToJsonFile);
• Directory (1 file per automation): var repo = new DirectoryAutomationRepository(rootDir);
• Temp directory (ephemeral, pre-sets TriggerNow=true and __UpdateToScalars=true if missing): var repo = new TempDirectoryAutomationRepository(rootDir);

And then the service:

var automationSvc = new AutomationService(
    repository: repo,
    scalarRepository: scalarRepo,      // IScalarRepository<string,int>
    jobRepository: jobRepo             // IJobRepository<Guid,string>
);

Repos update a version.txt (except temp) so a separate host can detect changes.

---

Quickstart: define, store, evaluate

using DHI.Services.Jobs.Automations;
using DHI.Services.Jobs.Automations.Triggers;

// 1) Define your automation
var auto = new Automation(
    name: "Nightly Analytics",
    group: "analytics/batch",
    taskId: "Analytics.RunNightly")
{
    HostGroup = "prod",
    Priority = 5,
    Tag = "nightly",
    TaskParameters = new Dictionary<string, string>
    {
        // only keys you want surfaced to the task
        ["Region"] = "EU",
        ["OutputBucket"] = "analytics-output"
        // "TriggerNow" intentionally omitted for scheduled runs
    },
    TriggerCondition = new TriggerCondition(new List<ITrigger>
    {
        new ScheduledTrigger(
            id: "EveryMidnightUTC",
            description: "Fire daily at midnight UTC",
            startTimeUtc: DateTime.Parse("2025-01-01T00:00:00Z"),
            interval: TimeSpan.FromDays(1))
    })
};

// 2) Persist it
var repo = new DirectoryAutomationRepository("/var/app/automations");
var svc  = new AutomationService(repo, scalarRepo, jobRepo);
svc.Add(auto);

// 3) Evaluate (e.g., from a worker)
var executor = new AutomationExecutor(logger, scalarService, rootGroup: "Job Automator Log");
var result   = executor.Execute(auto /*, runParams: optional overlays */);

if (result.IsMet)
{
    // Merge result.TaskParameters into your job request and enqueue
}

Boolean expressions: give triggers stable ids and use them in TriggerCondition.Conditional:

var cond = new TriggerCondition(
    new List<ITrigger> {
        new ScheduledTrigger("T1", "hourly", DateTime.UtcNow, TimeSpan.FromHours(1)),
        new JobCompletedTrigger("T2", "ETL-Done", typeof(MyJobRepo), "[env:CONN]"),
        new SqlTrigger("[env:SQL]", DbmsType.SqlServer,
                       new[] { "select count(*) C from dbo.Ready where Flag=1" },
                       "T3", "SQL gate")
    },
    conditional: "T1 AND (T2 OR T3)"
);
auto.TriggerCondition = cond;

One-off manual run: set TriggerNow=true either in auto.TaskParameters or when executing:

var result = executor.Execute(auto, runParams: new Dictionary<string,string> {
    ["TriggerNow"] = "true"
});

---

Example: build your own trigger

You generally need two things:

1. The runtime trigger (derive from BaseTrigger)
2. The parameters export (MEF) so UI/Web knows how to build a form

1) Runtime trigger

using DHI.Services.Jobs.Automations;
using Microsoft.Extensions.Logging;
using System.Collections.Generic;
using System.IO;

namespace DHI.Services.Jobs.Automations.Triggers
{
    [Serializable]
    public class FileExistsTrigger : BaseTrigger
    {
        public string Path { get; }
        public bool CaseSensitive { get; }

        public FileExistsTrigger(string id, string description, string path, bool caseSensitive = false)
            : base(id, description)
        {
            Path = path ?? throw new ArgumentNullException(nameof(path));
            CaseSensitive = caseSensitive;
        }

        public override AutomationResult Execute(ILogger logger, IDictionary<string,string> parameters = null)
        {
            // Honor the global bypass
            if (parameters.IsTriggerNow())
            {
                logger?.LogInformation("FileExistsTrigger {Id}: TriggerNow=true → met", Id);
                return AutomationResult.Met();
            }

            var exists = File.Exists(Path);
            if (!exists && !CaseSensitive)
            {
                var dir = Path.GetDirectoryName(Path) ?? ".";
                var name = System.IO.Path.GetFileName(Path);
                if (Directory.Exists(dir))
                {
                    foreach (var f in Directory.GetFiles(dir))
                    {
                        if (string.Equals(System.IO.Path.GetFileName(f), name, StringComparison.OrdinalIgnoreCase))
                        { exists = true; break; }
                    }
                }
            }

            return exists ? AutomationResult.Met(new Dictionary<string,string>
            {
                ["TriggeredBy"] = "FileExists",
                ["FilePath"] = Path
            })
            : AutomationResult.NotMet();
        }
    }
}

2) Parameters export (for Web/API form generation)

using System.ComponentModel.Composition;
using DHI.Services.Jobs.Automations.TriggerParametersExport;

namespace DHI.Services.Jobs.Automations.TriggerParametersExport
{
    [Export(typeof(ITriggerParameters))]
    [ExportMetadata("Id", nameof(DHI.Services.Jobs.Automations.Triggers.FileExistsTrigger))]
    public sealed class FileExistsTriggerParameters : ITriggerParameters
    {
        [TriggerParameter(true,  title: "Path", description: "Absolute or relative file path")]
        public string Path { get; set; }

        [TriggerParameter(false, title: "Case Sensitive", description: "Match filename case")]
        public bool CaseSensitive { get; set; }

        [TriggerParameter(true,  title: "Description")]
        public string Description { get; set; }

        // Required by ITriggerParameters
        public string Description { get; set; }
    }
}

Drop these classes in an assembly that your TriggerRepository scans (MEF). The Id metadata must equal nameof(YourTriggerClass) so UI can connect the schema to the runtime type.

---

Using the trigger metadata

To list all available trigger types and build UIs:

var triggerRepo = new TriggerRepository(searchDirectory: AppContext.BaseDirectory);
foreach (var tp in triggerRepo.GetAll())
{
    // tp.Id, tp.Properties, tp.Required, tp.AssemblyQualifiedTypeName, etc.
}

---

Connection strings & environment

• "[env:XYZ]" is replaced with Environment.GetEnvironmentVariable("XYZ").
• "[AppData]" requires AppDomain.CurrentDomain.SetData("DataDirectory", path).
• Bad/missing env vars throw clear ArgumentExceptions.

---

Operational tips

• Intervals: keep ScheduledTrigger.Interval >= 1 minute. Use toleranceSeconds (e.g., 10–60s) to accommodate clock skew / queueing.
• Ids: choose simple alpha/num Ids—used in expressions and scalar keys.
• Logging/Scalars: the executor captures all logs into Last Log and, on success, Last Met Log. Per-trigger Is Met scalars are written under rootGroup/automationId/hostGroup/(triggerId)/Is Met.
• Security: avoid embedding secrets directly—prefer [env:SECRET_NAME].
• Temp repo: great for ad-hoc runs and tests; it defaults TriggerNow=true unless you override.

---

Minimal end-to-end sample (multiple triggers + expression)

var a = new Automation("Weekly Report", "reports", "Reports.Generate")
{
    HostGroup = "ops",
    TaskParameters = new()
    {
        ["Recipient"] = "team@dhi.example",
        ["Format"] = "pdf"
    },
    TriggerCondition = new TriggerCondition(
        new List<ITrigger>
        {
            new ScheduledTrigger("SCHED", "Monday 07:00 UTC", new DateTime(2025,1,6,7,0,0, DateTimeKind.Utc), TimeSpan.FromDays(7)),
            new JobCompletedTrigger("ETL_DONE", "ETL.Daily", typeof(MyJobRepo), "[env:ETL_DB]"),
        },
        conditional: "SCHED AND ETL_DONE"
    )
};

svc.AddOrUpdate(a);

// Worker evaluation:
var res = executor.Execute(a);
if (res.IsMet)
{
    // enqueue job with res.TaskParameters
}

---

Troubleshooting

• My schedule never fires Check that StartTimeUtc ≤ current UTC, Interval >= 00:01:00, and your worker passes consistent utcNow/utcPrev across polls.
• JobCompletedTrigger throws Ensure the RepositoryType implements IJobRepository<Guid,string> and has (string connectionString) ctor. Verify env/DataDirectory substitutions.
• SqlTrigger returns NotMet Confirm the first query returns rows; inspect debug logs for serialized parameters; remember the last query must return at least one row for Met.