Skip to main content

CI green logs, row absent

CI passed on workflow logs; database side effect never showed up

CI can pass on log replay or synthetic success while the integration database never received the side effect you assumed. Pipeline fixtures and mocked stores hide that until production. Read-only verification closes the loop on the real database you point at during the job.

Emit structured tool observations from the workflow under test, keep a registry aligned with migration truth, and fail the job when AgentSkeptic returns inconsistent with ROW_ABSENT. That is a different signal from exit code zero on curl mocks: it is state alignment, not proof that a particular HTTP request executed.

Pair this with assurance manifests for multi-scenario sweeps when you need time-bounded freshness on saved reports. The commercial npm path adds enforce and CI lock fixtures; OSS verify stays local without a license server for snapshot checks.

Treat verification artifacts as part of the audit trail: attach JSON workflow results and human truth reports to the change record so reviewers see expected versus observed, not only green pipeline tiles.

What to do next

Example: activity that looked successful in logs or traces, missing row (ROW_ABSENT)

The block below uses the bundled `wf_missing` demo so this page stays aligned with the engine.

Verification report

Private verification snapshot for sharing in tickets or Slack. This URL is not indexed for search; see Security & Trust for how the site handles data.

Kind: workflow

Structured decision summary is available for Outcome Certificate v3 shares only.

Canonical human report

workflow_id: wf_missing
workflow_status: inconsistent
trust: NOT TRUSTED: At least one step failed verification against expected downstream state (determinate failure).
execution_path: Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.
diagnosis:
  summary: Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
  primary_origin: downstream_system_state
  confidence: medium
  actionable_failure: category=ambiguous severity=high recommended_action=manual_review automation_safe=false
  - evidence: scope=step codes=ROW_ABSENT seq=0 tool=crm.upsert_contact
  alternative_origin: downstream_system_state
    rationale: The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
  alternative_origin: tool_use
    rationale: The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
failure_explanation:
expected: Verification expected downstream state to satisfy verify_target "table=contacts identity=[id=missing_id] required_fields=[name, status]" and intended_effect "Upsert contact "missing_id" with fields {"name":"X","status":"Y"}" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].
observed: Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0
divergence: Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.
known_facts:
  - id=trust_summary value=NOT TRUSTED: At least one step failed verification against expected downstream state (determinate failure).
  - id=workflow_status value=inconsistent
  - id=verification_policy value=consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0
  - id=primary_origin value=downstream_system_state
  - id=classification_confidence value=medium
  - id=failure_analysis_summary value=Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
  - id=primary_scope value=step
  - id=primary_codes value=ROW_ABSENT
  - id=primary_tool_id value=crm.upsert_contact
  - id=primary_seq value=0
  - id=verify_target value=table=contacts identity=[id=missing_id] required_fields=[name, status]
  - id=intended_effect_narrative value=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}
  - id=evidence_summary_row_count value=0
unknowns:
  - id=classification_confidence_band value=medium
  - id=competing_hypothesis value=origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
  - id=competing_hypothesis value=origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
correctness_definition:
  enforcement_kind: step_registry_expectation
  must_always_hold: Must: after tool_observed seq=0 toolId=crm.upsert_contact, authoritative downstream state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.
  enforce_as:
    - Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.
    - Read-only verification reads at verify time (SQL, HTTP witnesses, object metadata, vector fetches, Mongo findOne, etc.) SHALL match verificationRequest.
  enforceable_projection: {"projectionKind":"step_registry_expectation","workflowId":"wf_missing","verificationPolicyFragment":"consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0","seq":0,"toolId":"crm.upsert_contact","verificationRequest":{"kind":"sql_row","table":"contacts","identityEq":[{"column":"id","value":"missing_id"}],"requiredFields":{"name":"X","status":"Y"}}}
  remediation_alignment: recommended_action=manual_review automation_safe=false
run_level: (none)
event_sequence: normal
steps:
  - seq=0 tool=crm.upsert_contact
    declared: tool=crm.upsert_contact; intent=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}; parameters_digest={"fields":{"name":"X","status":"Y"},"recordId":"missing_id"}
    expected: table=contacts identity=[id=missing_id] required_fields=[name, status]
    observed_database: rowCount=0
    verification_verdict: outcome=FAILED_ROW_MISSING; Expected state at the verification target was not found (for example a missing row or absent witness result).; failure_category=workflow_execution
    observations: evaluated=1 of 1 in_capture_order
    detail: No row matched key (table=contacts id=missing_id)
    reference_code: ROW_ABSENT
    user_meaning: Success was implied, but no matching row was found in the database.

Machine JSON (integration SSOT)

{
  "schemaVersion": 15,
  "workflowId": "wf_missing",
  "status": "inconsistent",
  "runLevelReasons": [],
  "verificationPolicy": {
    "consistencyMode": "strong",
    "verificationWindowMs": 0,
    "pollIntervalMs": 0
  },
  "eventSequenceIntegrity": {
    "kind": "normal"
  },
  "steps": [
    {
      "seq": 0,
      "toolId": "crm.upsert_contact",
      "releaseCritical": false,
      "intendedEffect": {
        "narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
      },
      "observedExecution": {
        "paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
      },
      "verificationRequest": {
        "kind": "sql_row",
        "table": "contacts",
        "identityEq": [
          {
            "column": "id",
            "value": "missing_id"
          }
        ],
        "requiredFields": {
          "name": "X",
          "status": "Y"
        }
      },
      "status": "missing",
      "reasons": [
        {
          "code": "ROW_ABSENT",
          "message": "No row matched key (table=contacts id=missing_id)"
        }
      ],
      "evidenceSummary": {
        "rowCount": 0
      },
      "repeatObservationCount": 1,
      "evaluatedObservationOrdinal": 1,
      "failureDiagnostic": "workflow_execution"
    }
  ],
  "verificationRunContext": {
    "maxWireSchemaVersion": 1,
    "retrievalEvents": [],
    "controlEvents": [],
    "modelTurnEvents": [],
    "toolSkippedEvents": [],
    "toolObservedIngestIndexBySeq": {
      "0": 0
    },
    "firstToolObservedIngestIndex": 0,
    "hasRunCompletedControl": false,
    "lastRunEvent": {
      "ingestIndex": 0,
      "type": "tool_observed"
    }
  },
  "workflowTruthReport": {
    "schemaVersion": 9,
    "workflowId": "wf_missing",
    "workflowStatus": "inconsistent",
    "trustSummary": "NOT TRUSTED: At least one step failed verification against expected downstream state (determinate failure).",
    "runLevelIssues": [],
    "eventSequence": {
      "kind": "normal"
    },
    "steps": [
      {
        "seq": 0,
        "toolId": "crm.upsert_contact",
        "releaseCritical": false,
        "outcomeLabel": "FAILED_ROW_MISSING",
        "observations": {
          "evaluatedOrdinal": 1,
          "repeatCount": 1
        },
        "reasons": [
          {
            "code": "ROW_ABSENT",
            "message": "No row matched key (table=contacts id=missing_id)"
          }
        ],
        "intendedEffect": {
          "narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
        },
        "observedExecution": {
          "paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
        },
        "verifyTarget": "table=contacts identity=[id=missing_id] required_fields=[name, status]",
        "observedStateSummary": "rowCount=0",
        "failureCategory": "workflow_execution"
      }
    ],
    "failureAnalysis": {
      "summary": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.",
      "primaryOrigin": "downstream_system_state",
      "confidence": "medium",
      "unknownReasonCodes": [],
      "evidence": [
        {
          "scope": "step",
          "seq": 0,
          "toolId": "crm.upsert_contact",
          "codes": [
            "ROW_ABSENT"
          ]
        }
      ],
      "alternativeHypotheses": [
        {
          "primaryOrigin": "downstream_system_state",
          "rationale": "The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
        },
        {
          "primaryOrigin": "tool_use",
          "rationale": "The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
        }
      ],
      "actionableFailure": {
        "category": "ambiguous",
        "severity": "high",
        "recommendedAction": "manual_review",
        "automationSafe": false
      }
    },
    "executionPathFindings": [],
    "executionPathSummary": "Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.",
    "failureExplanation": {
      "schemaVersion": 1,
      "expected": "Verification expected downstream state to satisfy verify_target \"table=contacts identity=[id=missing_id] required_fields=[name, status]\" and intended_effect \"Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}\" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].",
      "observed": "Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0",
      "divergence": "Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.",
      "knownFacts": [
        {
          "id": "trust_summary",
          "value": "NOT TRUSTED: At least one step failed verification against expected downstream state (determinate failure)."
        },
        {
          "id": "workflow_status",
          "value": "inconsistent"
        },
        {
          "id": "verification_policy",
          "value": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0"
        },
        {
          "id": "primary_origin",
          "value": "downstream_system_state"
        },
        {
          "id": "classification_confidence",
          "value": "medium"
        },
        {
          "id": "failure_analysis_summary",
          "value": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state."
        },
        {
          "id": "primary_scope",
          "value": "step"
        },
        {
          "id": "primary_codes",
          "value": "ROW_ABSENT"
        },
        {
          "id": "primary_tool_id",
          "value": "crm.upsert_contact"
        },
        {
          "id": "primary_seq",
          "value": "0"
        },
        {
          "id": "verify_target",
          "value": "table=contacts identity=[id=missing_id] required_fields=[name, status]"
        },
        {
          "id": "intended_effect_narrative",
          "value": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
        },
        {
          "id": "evidence_summary_row_count",
          "value": "0"
        }
      ],
      "unknowns": [
        {
          "id": "classification_confidence_band",
          "value": "medium"
        },
        {
          "id": "competing_hypothesis",
          "value": "origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
        },
        {
          "id": "competing_hypothesis",
          "value": "origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
        }
      ]
    },
    "correctnessDefinition": {
      "schemaVersion": 1,
      "enforcementKind": "step_registry_expectation",
      "mustAlwaysHold": "Must: after tool_observed seq=0 toolId=crm.upsert_contact, authoritative downstream state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.",
      "enforceAs": [
        "Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.",
        "Read-only verification reads at verify time (SQL, HTTP witnesses, object metadata, vector fetches, Mongo findOne, etc.) SHALL match verificationRequest."
      ],
      "enforceableProjection": {
        "projectionKind": "step_registry_expectation",
        "workflowId": "wf_missing",
        "verificationPolicyFragment": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0",
        "seq": 0,
        "toolId": "crm.upsert_contact",
        "verificationRequest": {
          "kind": "sql_row",
          "table": "contacts",
          "identityEq": [
            {
              "column": "id",
              "value": "missing_id"
            }
          ],
          "requiredFields": {
            "name": "X",
            "status": "Y"
          }
        }
      },
      "remediationAlignment": {
        "recommendedAction": "manual_review",
        "automationSafe": false
      }
    }
  }
}