CI green but missing database side effect
CI can stay green on workflow logs while the database side effect is missing until a read-only gate surfaces ROW_ABSENT before production
CI can stay green when logs never hit the database row your workflow promised; add read-only SQL verification that reads structured tool activity the same way AgentSkeptic does in the bundled demos.
Copy the /integrate first-run flow into your pipeline so every PR runs read-only checks against a pinned database fixture or ephemeral Postgres before merge.
- Replay workflow logs into NDJSON observations for the side-effect tools you care about.
- Fail the job when read-only SQL returns
ROW_ABSENTor field mismatches—even if the trace step logged success.
Example: activity that looked successful in logs or traces, missing row (ROW_ABSENT)
The block below uses the bundled `wf_missing` demo so this page stays aligned with the engine.
Verification report
Private verification snapshot for sharing in tickets or Slack. This URL is not indexed for search; see Security & Trust for how the site handles data.
Kind: workflow
Human report
workflow_id: wf_missing
workflow_status: inconsistent
trust: NOT TRUSTED: At least one step failed verification against the database (determinate failure).
execution_path: Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.
diagnosis:
summary: Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
primary_origin: downstream_system_state
confidence: medium
actionable_failure: category=ambiguous severity=high recommended_action=manual_review automation_safe=false
- evidence: scope=step codes=ROW_ABSENT seq=0 tool=crm.upsert_contact
alternative_origin: downstream_system_state
rationale: The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
alternative_origin: tool_use
rationale: The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
failure_explanation:
expected: Verification expected post-execution database state to satisfy verify_target "table=contacts identity=[id=missing_id] required_fields=[name, status]" and intended_effect "Upsert contact "missing_id" with fields {"name":"X","status":"Y"}" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].
observed: Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0
divergence: Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.
known_facts:
- id=trust_summary value=NOT TRUSTED: At least one step failed verification against the database (determinate failure).
- id=workflow_status value=inconsistent
- id=verification_policy value=consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0
- id=primary_origin value=downstream_system_state
- id=classification_confidence value=medium
- id=failure_analysis_summary value=Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
- id=primary_scope value=step
- id=primary_codes value=ROW_ABSENT
- id=primary_tool_id value=crm.upsert_contact
- id=primary_seq value=0
- id=verify_target value=table=contacts identity=[id=missing_id] required_fields=[name, status]
- id=intended_effect_narrative value=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}
- id=evidence_summary_row_count value=0
unknowns:
- id=classification_confidence_band value=medium
- id=competing_hypothesis value=origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
- id=competing_hypothesis value=origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
correctness_definition:
enforcement_kind: step_sql_expectation
must_always_hold: Must: after tool_observed seq=0 toolId=crm.upsert_contact, database state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.
enforce_as:
- Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.
- Authoritative SQL state SHALL match identity, required fields, and relational checks encoded in verificationRequest.
enforceable_projection: {"projectionKind":"step_sql_expectation","workflowId":"wf_missing","verificationPolicyFragment":"consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0","seq":0,"toolId":"crm.upsert_contact","verificationRequest":{"kind":"sql_row","table":"contacts","identityEq":[{"column":"id","value":"missing_id"}],"requiredFields":{"name":"X","status":"Y"}}}
remediation_alignment: recommended_action=manual_review automation_safe=false
run_level: (none)
event_sequence: normal
steps:
- seq=0 tool=crm.upsert_contact
declared: tool=crm.upsert_contact; intent=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}; parameters_digest={"fields":{"name":"X","status":"Y"},"recordId":"missing_id"}
expected: table=contacts identity=[id=missing_id] required_fields=[name, status]
observed_database: rowCount=0
verification_verdict: outcome=FAILED_ROW_MISSING; Expected row is missing from the database (the log implies a write that is not present).; failure_category=workflow_execution
observations: evaluated=1 of 1 in_capture_order
detail: No row matched key (table=contacts id=missing_id)
reference_code: ROW_ABSENT
user_meaning: Success was implied, but no matching row was found in the database.Machine JSON
{
"schemaVersion": 15,
"workflowId": "wf_missing",
"status": "inconsistent",
"runLevelReasons": [],
"verificationPolicy": {
"consistencyMode": "strong",
"verificationWindowMs": 0,
"pollIntervalMs": 0
},
"eventSequenceIntegrity": {
"kind": "normal"
},
"steps": [
{
"seq": 0,
"toolId": "crm.upsert_contact",
"intendedEffect": {
"narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
},
"observedExecution": {
"paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
},
"verificationRequest": {
"kind": "sql_row",
"table": "contacts",
"identityEq": [
{
"column": "id",
"value": "missing_id"
}
],
"requiredFields": {
"name": "X",
"status": "Y"
}
},
"status": "missing",
"reasons": [
{
"code": "ROW_ABSENT",
"message": "No row matched key (table=contacts id=missing_id)"
}
],
"evidenceSummary": {
"rowCount": 0
},
"repeatObservationCount": 1,
"evaluatedObservationOrdinal": 1,
"failureDiagnostic": "workflow_execution"
}
],
"verificationRunContext": {
"maxWireSchemaVersion": 1,
"retrievalEvents": [],
"controlEvents": [],
"modelTurnEvents": [],
"toolSkippedEvents": [],
"toolObservedIngestIndexBySeq": {
"0": 0
},
"firstToolObservedIngestIndex": 0,
"hasRunCompletedControl": false,
"lastRunEvent": {
"ingestIndex": 0,
"type": "tool_observed"
}
},
"workflowTruthReport": {
"schemaVersion": 9,
"workflowId": "wf_missing",
"workflowStatus": "inconsistent",
"trustSummary": "NOT TRUSTED: At least one step failed verification against the database (determinate failure).",
"runLevelIssues": [],
"eventSequence": {
"kind": "normal"
},
"steps": [
{
"seq": 0,
"toolId": "crm.upsert_contact",
"outcomeLabel": "FAILED_ROW_MISSING",
"observations": {
"evaluatedOrdinal": 1,
"repeatCount": 1
},
"reasons": [
{
"code": "ROW_ABSENT",
"message": "No row matched key (table=contacts id=missing_id)"
}
],
"intendedEffect": {
"narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
},
"observedExecution": {
"paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
},
"verifyTarget": "table=contacts identity=[id=missing_id] required_fields=[name, status]",
"observedStateSummary": "rowCount=0",
"failureCategory": "workflow_execution"
}
],
"failureAnalysis": {
"summary": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.",
"primaryOrigin": "downstream_system_state",
"confidence": "medium",
"unknownReasonCodes": [],
"evidence": [
{
"scope": "step",
"seq": 0,
"toolId": "crm.upsert_contact",
"codes": [
"ROW_ABSENT"
]
}
],
"alternativeHypotheses": [
{
"primaryOrigin": "downstream_system_state",
"rationale": "The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
},
{
"primaryOrigin": "tool_use",
"rationale": "The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
}
],
"actionableFailure": {
"category": "ambiguous",
"severity": "high",
"recommendedAction": "manual_review",
"automationSafe": false
}
},
"executionPathFindings": [],
"executionPathSummary": "Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.",
"failureExplanation": {
"schemaVersion": 1,
"expected": "Verification expected post-execution database state to satisfy verify_target \"table=contacts identity=[id=missing_id] required_fields=[name, status]\" and intended_effect \"Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}\" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].",
"observed": "Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0",
"divergence": "Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.",
"knownFacts": [
{
"id": "trust_summary",
"value": "NOT TRUSTED: At least one step failed verification against the database (determinate failure)."
},
{
"id": "workflow_status",
"value": "inconsistent"
},
{
"id": "verification_policy",
"value": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0"
},
{
"id": "primary_origin",
"value": "downstream_system_state"
},
{
"id": "classification_confidence",
"value": "medium"
},
{
"id": "failure_analysis_summary",
"value": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state."
},
{
"id": "primary_scope",
"value": "step"
},
{
"id": "primary_codes",
"value": "ROW_ABSENT"
},
{
"id": "primary_tool_id",
"value": "crm.upsert_contact"
},
{
"id": "primary_seq",
"value": "0"
},
{
"id": "verify_target",
"value": "table=contacts identity=[id=missing_id] required_fields=[name, status]"
},
{
"id": "intended_effect_narrative",
"value": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
},
{
"id": "evidence_summary_row_count",
"value": "0"
}
],
"unknowns": [
{
"id": "classification_confidence_band",
"value": "medium"
},
{
"id": "competing_hypothesis",
"value": "origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
},
{
"id": "competing_hypothesis",
"value": "origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
}
]
},
"correctnessDefinition": {
"schemaVersion": 1,
"enforcementKind": "step_sql_expectation",
"mustAlwaysHold": "Must: after tool_observed seq=0 toolId=crm.upsert_contact, database state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.",
"enforceAs": [
"Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.",
"Authoritative SQL state SHALL match identity, required fields, and relational checks encoded in verificationRequest."
],
"enforceableProjection": {
"projectionKind": "step_sql_expectation",
"workflowId": "wf_missing",
"verificationPolicyFragment": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0",
"seq": 0,
"toolId": "crm.upsert_contact",
"verificationRequest": {
"kind": "sql_row",
"table": "contacts",
"identityEq": [
{
"column": "id",
"value": "missing_id"
}
],
"requiredFields": {
"name": "X",
"status": "Y"
}
}
},
"remediationAlignment": {
"recommendedAction": "manual_review",
"automationSafe": false
}
}
}
}