Pipeline Specs

Reproducible pipelines. A pipeline spec is a small JSON file that declares a pipeline's inputs and its steps. Save any finished run as a spec, re-run it on new files, keep it in your own repo, and trace every result back to the exact tools and inputs that produced it. Requires the tools scope.

What a spec is

The predefined pipelines are specs too — a spec is simply a pipeline you own. It has two parts:

• inputs — the values that change between runs (the files to process, a sample name, a reference).
• steps — a graph of tool invocations. Each step names a registered tool, its parameters, and the steps it waits for.

Saved specs live in your workspace at pipelines/specs/<name>.pipeline.json. They are ordinary files: open them in the viewer, download them, edit them by hand, or commit them to your own repository and run them from there.

Worked example

A two-step pipeline: sort a BAM with samtools sort, then index the sorted file with samtools index. The first step reads the inputBam input; the second reads the first step's output.

{
  "specVersion": "1.0",
  "id": "sort-and-index",
  "name": "sort-and-index",
  "description": "Coordinate-sort a BAM file and build its index",
  "category": "custom",
  "executionMode": "process-manager",
  "inputs": [
    { "name": "inputBam", "type": "file", "description": "Unsorted BAM file" },
    { "name": "sampleName", "type": "string", "description": "Sample identifier used to name output files", "default": "sample" }
  ],
  "steps": [
    {
      "stepId": "sort",
      "toolName": "samtools",
      "description": "Sort BAM by coordinate",
      "parameters": {
        "samtoolsCommand": "sort",
        "inputFile": "${inputBam}",
        "outputFile": "${sampleName}_sorted.bam",
        "outputFormat": "bam",
        "threads": 4
      },
      "dependsOn": []
    },
    {
      "stepId": "index",
      "toolName": "samtools",
      "description": "Index the sorted BAM",
      "parameters": {
        "samtoolsCommand": "index",
        "inputFile": "${sort.outputFile}",
        "outputFile": "${sampleName}_sorted.bam.bai",
        "outputFormat": "bai"
      },
      "dependsOn": ["sort"]
    }
  ]
}

Save it as pipelines/specs/sort-and-index.pipeline.json, check it, and run it:

# 1. Check the spec — same checks a run applies
curl -X POST https://api.smarts.bio/v1/pipelines/validate \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "workspace_id": "ws_abc123", "spec_file": "pipelines/specs/sort-and-index.pipeline.json" }'
# → { "valid": true, "errors": [], "warnings": [] }

# 2. Run it on a sample
curl -X POST https://api.smarts.bio/v1/pipelines \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "spec_file": "pipelines/specs/sort-and-index.pipeline.json",
    "workspace_id": "ws_abc123",
    "input": { "inputBam": "input/sample1.bam", "sampleName": "sample1" }
  }'
# → { "data": { "id": "pipe_abc123", "status": "queued", ... } }

# 3. Poll until it finishes
curl "https://api.smarts.bio/v1/pipelines/pipe_abc123?workspace_id=ws_abc123" \
  -H "Authorization: Bearer sk_live_..."

Spec format

Top-level fields

inputs[]

file inputs take workspace-relative paths (input/sample1.bam); the platform expands them to the full storage key at run time.

steps[]

Interpolation

Any string parameter can reference values with ${…}:

Every reference must resolve — a typo such as ${inputBamm} is a validation error, not a silently missing argument. If a step reads ${producer.outputFile} but the producer declares no outputFile, you get a warning: the file is then matched by position, which can pick the wrong one when a tool writes several outputs.

Save a run and re-run it on different files

You rarely need to write a spec by hand. Run a pipeline once — a predefined one, or one the agent designed for you in chat — then save the run. Saving turns it into a reusable spec:

• Workspace files the run read become declared file inputs; the values it used are kept in provenance.previousValues for reference.
• Files produced by an earlier step are wired as ${stepId.outputFile} — never turned into inputs.
• A shared output-filename prefix (e.g. S1_sorted.bam, S1_sorted.bam.bai) becomes a sampleName input.
• File inputs never keep a default, so a re-run can't silently reuse the previous run's data.
• Anything that needs a human look — a list of files left as literals, a run that ended as failed — comes back as a warning.

A name that is already taken is refused (409) unless you ask to overwrite. From any surface:

Web app

In the Processes panel, open a finished (completed or failed) pipeline run and click Save as pipeline. The spec is saved to pipelines/specs/; open it to see its inputs and steps, fill in new inputs, and click Run pipeline.

VS Code & JupyterLab

Open any *.pipeline.json from the smarts.bio file explorer. The pipeline viewer shows the inputs (with the values the last run used), the step graph, and a run form — fill in the required inputs and click Run pipeline.

API & CLI

# 1. Save a finished run
curl -X POST https://api.smarts.bio/v1/pipelines/pipe_abc123/save-spec \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "workspace_id": "ws_abc123", "name": "sort-and-index" }'
# → 201 { "path": "pipelines/specs/sort-and-index.pipeline.json", "spec": { ... }, "warnings": [] }

# 2. Re-run it on a different sample
curl -X POST https://api.smarts.bio/v1/pipelines \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "spec_file": "pipelines/specs/sort-and-index.pipeline.json",
    "workspace_id": "ws_abc123",
    "input": { "inputBam": "input/sample2.bam", "sampleName": "sample2" }
  }'

The SDKs don't wrap spec_file, spec, save-spec or validate yet — call the HTTP endpoints directly or use the CLI. Full endpoint details are in the Pipelines reference.

AI assistants (MCP)

The hosted and local MCP servers expose the same flow, so you can simply ask your assistant to "save that run as a pipeline and run it on sample2":

• smarts_list_pipelines — built-in pipelines, plus the specs saved in a workspace when workspace_id is given.
• smarts_get_pipeline — a pipeline's inputs (required or not, defaults) and steps, by pipeline_id or spec_file.
• smarts_run_pipeline — run by exactly one of pipeline_id or spec_file.
• smarts_save_pipeline — save a finished run (run_id, optional name and overwrite).

Run a spec from your own repo

A spec doesn't have to live in the workspace. Send it inline as spec — handy when specs are versioned alongside your analysis code. The CLI does this automatically when --spec points at a local file (prefix a path with @ to force the workspace copy instead).

curl -X POST https://api.smarts.bio/v1/pipelines \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d "$(jq -n --slurpfile spec pipelines/sort-and-index.pipeline.json '{
    spec: $spec[0],
    workspace_id: "ws_abc123",
    input: { inputBam: "input/sample3.bam", sampleName: "sample3" }
  }')"

Provenance: run.json

Every pipeline run writes a run.json manifest into its output folder, pipelines/<run folder>/run.json (the folder is a timestamp such as 2026-09-19_14-03-22). It is written when the run starts and updated when it finishes, so a set of results always records what produced it. The manifest embeds the spec exactly as executed — editing a saved spec later never rewrites the history of past runs.

{
  "schemaVersion": 1,
  "executionId": "…",
  "pipelineId": "sort-and-index",
  "pipelineName": "sort-and-index",
  "status": "completed",
  "workspaceId": "ws_abc123",
  "organizationId": "…",
  "outputFolder": "2026-09-19_14-03-22",
  "startedAt": "2026-09-19T14:03:22.000Z",
  "completedAt": "2026-09-19T14:09:51.000Z",
  "durationMs": 389000,
  "stepsTotal": 2,
  "stepsCompleted": 2,
  "stepsFailed": 0,
  "parameters": { "inputBam": "…/input/sample2.bam", "sampleName": "sample2" },
  "inputChecksums": { "inputBam": { "s3Key": "…/input/sample2.bam", "checksum": "…", "size": 1843200 } },
  "pipelineDefinition": { "…": "the spec exactly as executed" },
  "steps": [
    {
      "stepId": "sort",
      "stepNumber": 1,
      "toolName": "samtools",
      "toolVersion": "…",
      "imageDigest": "sha256:…",
      "taskDefinitionArn": "…:12",
      "status": "completed",
      "dependsOn": [],
      "startedAt": "…",
      "completedAt": "…",
      "durationMs": 241000,
      "outputFiles": [{ "filename": "sample2_sorted.bam", "s3Key": "…", "size": 1790000, "checksum": "…" }]
    }
  ]
}

Limits