Green trace, missing Postgres row

Green LangGraph or agent trace but wrong or missing Postgres row

When your graph finishes with a green step list, you still need a read-only check that the row your business logic cares about is present in Postgres with the expected columns. Traces record what the runtime believed happened; they do not substitute for SELECT results at verification time.

AgentSkeptic ingests structured tool activity as NDJSON or JSON, derives expected row identity from your registry, and runs read-only SQL. A missing row surfaces as inconsistent with reason ROW_ABSENT even when the trace narrative looked successful. Use this pattern after LangGraph runs, before you treat outcomes as safe for customers or downstream automation.

Start from the bundled demo contrast (wf_complete vs wf_missing), then wire your own tool IDs and tables. First-run on your database is documented at /integrate; contract mode with a registry is the audit-grade path when you need explicit per-tool expectations rather than inferred checks from Quick Verify.

Operational teams often discover this gap only after a silent failure: the ticket never moved, the contact never landed, or analytics disagrees with CRM. Running verification as a gate catches that class before release trains or compliance sign-off, without pretending the trace proves causality.

workflow_id: wf_missing
workflow_status: inconsistent
trust: NOT TRUSTED: At least one step failed verification against the database (determinate failure).
execution_path: Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.
diagnosis:
  summary: Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
  primary_origin: downstream_system_state
  confidence: medium
  actionable_failure: category=ambiguous severity=high recommended_action=manual_review automation_safe=false
  - evidence: scope=step codes=ROW_ABSENT seq=0 tool=crm.upsert_contact
  alternative_origin: downstream_system_state
    rationale: The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
  alternative_origin: tool_use
    rationale: The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
failure_explanation:
expected: Verification expected post-execution database state to satisfy verify_target "table=contacts identity=[id=missing_id] required_fields=[name, status]" and intended_effect "Upsert contact "missing_id" with fields {"name":"X","status":"Y"}" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].
observed: Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0
divergence: Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.
known_facts:
  - id=trust_summary value=NOT TRUSTED: At least one step failed verification against the database (determinate failure).
  - id=workflow_status value=inconsistent
  - id=verification_policy value=consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0
  - id=primary_origin value=downstream_system_state
  - id=classification_confidence value=medium
  - id=failure_analysis_summary value=Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.
  - id=primary_scope value=step
  - id=primary_codes value=ROW_ABSENT
  - id=primary_tool_id value=crm.upsert_contact
  - id=primary_seq value=0
  - id=verify_target value=table=contacts identity=[id=missing_id] required_fields=[name, status]
  - id=intended_effect_narrative value=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}
  - id=evidence_summary_row_count value=0
unknowns:
  - id=classification_confidence_band value=medium
  - id=competing_hypothesis value=origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation.
  - id=competing_hypothesis value=origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written.
correctness_definition:
  enforcement_kind: step_sql_expectation
  must_always_hold: Must: after tool_observed seq=0 toolId=crm.upsert_contact, database state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.
  enforce_as:
    - Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.
    - Authoritative SQL state SHALL match identity, required fields, and relational checks encoded in verificationRequest.
  enforceable_projection: {"projectionKind":"step_sql_expectation","workflowId":"wf_missing","verificationPolicyFragment":"consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0","seq":0,"toolId":"crm.upsert_contact","verificationRequest":{"kind":"sql_row","table":"contacts","identityEq":[{"column":"id","value":"missing_id"}],"requiredFields":{"name":"X","status":"Y"}}}
  remediation_alignment: recommended_action=manual_review automation_safe=false
run_level: (none)
event_sequence: normal
steps:
  - seq=0 tool=crm.upsert_contact
    declared: tool=crm.upsert_contact; intent=Upsert contact "missing_id" with fields {"name":"X","status":"Y"}; parameters_digest={"fields":{"name":"X","status":"Y"},"recordId":"missing_id"}
    expected: table=contacts identity=[id=missing_id] required_fields=[name, status]
    observed_database: rowCount=0
    verification_verdict: outcome=FAILED_ROW_MISSING; Expected row is missing from the database (the log implies a write that is not present).; failure_category=workflow_execution
    observations: evaluated=1 of 1 in_capture_order
    detail: No row matched key (table=contacts id=missing_id)
    reference_code: ROW_ABSENT
    user_meaning: Success was implied, but no matching row was found in the database.

{
  "schemaVersion": 15,
  "workflowId": "wf_missing",
  "status": "inconsistent",
  "runLevelReasons": [],
  "verificationPolicy": {
    "consistencyMode": "strong",
    "verificationWindowMs": 0,
    "pollIntervalMs": 0
  },
  "eventSequenceIntegrity": {
    "kind": "normal"
  },
  "steps": [
    {
      "seq": 0,
      "toolId": "crm.upsert_contact",
      "intendedEffect": {
        "narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
      },
      "observedExecution": {
        "paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
      },
      "verificationRequest": {
        "kind": "sql_row",
        "table": "contacts",
        "identityEq": [
          {
            "column": "id",
            "value": "missing_id"
          }
        ],
        "requiredFields": {
          "name": "X",
          "status": "Y"
        }
      },
      "status": "missing",
      "reasons": [
        {
          "code": "ROW_ABSENT",
          "message": "No row matched key (table=contacts id=missing_id)"
        }
      ],
      "evidenceSummary": {
        "rowCount": 0
      },
      "repeatObservationCount": 1,
      "evaluatedObservationOrdinal": 1,
      "failureDiagnostic": "workflow_execution"
    }
  ],
  "verificationRunContext": {
    "maxWireSchemaVersion": 1,
    "retrievalEvents": [],
    "controlEvents": [],
    "modelTurnEvents": [],
    "toolSkippedEvents": [],
    "toolObservedIngestIndexBySeq": {
      "0": 0
    },
    "firstToolObservedIngestIndex": 0,
    "hasRunCompletedControl": false,
    "lastRunEvent": {
      "ingestIndex": 0,
      "type": "tool_observed"
    }
  },
  "workflowTruthReport": {
    "schemaVersion": 9,
    "workflowId": "wf_missing",
    "workflowStatus": "inconsistent",
    "trustSummary": "NOT TRUSTED: At least one step failed verification against the database (determinate failure).",
    "runLevelIssues": [],
    "eventSequence": {
      "kind": "normal"
    },
    "steps": [
      {
        "seq": 0,
        "toolId": "crm.upsert_contact",
        "outcomeLabel": "FAILED_ROW_MISSING",
        "observations": {
          "evaluatedOrdinal": 1,
          "repeatCount": 1
        },
        "reasons": [
          {
            "code": "ROW_ABSENT",
            "message": "No row matched key (table=contacts id=missing_id)"
          }
        ],
        "intendedEffect": {
          "narrative": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
        },
        "observedExecution": {
          "paramsCanonical": "{\"fields\":{\"name\":\"X\",\"status\":\"Y\"},\"recordId\":\"missing_id\"}"
        },
        "verifyTarget": "table=contacts identity=[id=missing_id] required_fields=[name, status]",
        "observedStateSummary": "rowCount=0",
        "failureCategory": "workflow_execution"
      }
    ],
    "failureAnalysis": {
      "summary": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state.",
      "primaryOrigin": "downstream_system_state",
      "confidence": "medium",
      "unknownReasonCodes": [],
      "evidence": [
        {
          "scope": "step",
          "seq": 0,
          "toolId": "crm.upsert_contact",
          "codes": [
            "ROW_ABSENT"
          ]
        }
      ],
      "alternativeHypotheses": [
        {
          "primaryOrigin": "downstream_system_state",
          "rationale": "The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
        },
        {
          "primaryOrigin": "tool_use",
          "rationale": "The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
        }
      ],
      "actionableFailure": {
        "category": "ambiguous",
        "severity": "high",
        "recommendedAction": "manual_review",
        "automationSafe": false
      }
    },
    "executionPathFindings": [],
    "executionPathSummary": "Full upstream execution-path visibility requires schemaVersion 2 run events (retrieval, model_turn, control, tool_skipped) with run graph fields.",
    "failureExplanation": {
      "schemaVersion": 1,
      "expected": "Verification expected post-execution database state to satisfy verify_target \"table=contacts identity=[id=missing_id] required_fields=[name, status]\" and intended_effect \"Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}\" for seq=0 toolId=crm.upsert_contact under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0].",
      "observed": "Step verification outcome: code=ROW_ABSENT detail=No row matched key (table=contacts id=missing_id); rowCount=0",
      "divergence": "Divergence at step seq=0 toolId=crm.upsert_contact: primary_code=ROW_ABSENT meaning=Success was implied, but no matching row was found in the database.",
      "knownFacts": [
        {
          "id": "trust_summary",
          "value": "NOT TRUSTED: At least one step failed verification against the database (determinate failure)."
        },
        {
          "id": "workflow_status",
          "value": "inconsistent"
        },
        {
          "id": "verification_policy",
          "value": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0"
        },
        {
          "id": "primary_origin",
          "value": "downstream_system_state"
        },
        {
          "id": "classification_confidence",
          "value": "medium"
        },
        {
          "id": "failure_analysis_summary",
          "value": "Primary failure at seq 0 tool crm.upsert_contact (code ROW_ABSENT); origin: downstream_system_state."
        },
        {
          "id": "primary_scope",
          "value": "step"
        },
        {
          "id": "primary_codes",
          "value": "ROW_ABSENT"
        },
        {
          "id": "primary_tool_id",
          "value": "crm.upsert_contact"
        },
        {
          "id": "primary_seq",
          "value": "0"
        },
        {
          "id": "verify_target",
          "value": "table=contacts identity=[id=missing_id] required_fields=[name, status]"
        },
        {
          "id": "intended_effect_narrative",
          "value": "Upsert contact \"missing_id\" with fields {\"name\":\"X\",\"status\":\"Y\"}"
        },
        {
          "id": "evidence_summary_row_count",
          "value": "0"
        }
      ],
      "unknowns": [
        {
          "id": "classification_confidence_band",
          "value": "medium"
        },
        {
          "id": "competing_hypothesis",
          "value": "origin=downstream_system_state|rationale=The database has no row at the verified key; the tool log may not have committed a write, or replication lag prevented observation."
        },
        {
          "id": "competing_hypothesis",
          "value": "origin=tool_use|rationale=The database has no row at the verified key; the registry key/value or pointer resolution from tool params may not match the row that was written."
        }
      ]
    },
    "correctnessDefinition": {
      "schemaVersion": 1,
      "enforcementKind": "step_sql_expectation",
      "mustAlwaysHold": "Must: after tool_observed seq=0 toolId=crm.upsert_contact, database state SHALL satisfy the verification contract in verificationRequest under policy [consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0] for workflowId=wf_missing.",
      "enforceAs": [
        "Registry (or synthetic events plus registry) SHALL keep verificationRequest aligned with declared tool parameters for seq=0.",
        "Authoritative SQL state SHALL match identity, required fields, and relational checks encoded in verificationRequest."
      ],
      "enforceableProjection": {
        "projectionKind": "step_sql_expectation",
        "workflowId": "wf_missing",
        "verificationPolicyFragment": "consistencyMode=strong; verificationWindowMs=0; pollIntervalMs=0",
        "seq": 0,
        "toolId": "crm.upsert_contact",
        "verificationRequest": {
          "kind": "sql_row",
          "table": "contacts",
          "identityEq": [
            {
              "column": "id",
              "value": "missing_id"
            }
          ],
          "requiredFields": {
            "name": "X",
            "status": "Y"
          }
        }
      },
      "remediationAlignment": {
        "recommendedAction": "manual_review",
        "automationSafe": false
      }
    }
  }
}

Run first-run integration