Almost every software company we look at in diligence now has an AI agent somewhere in its product or engineering workflow. Some of those agents only draft text. More of them take actions: they issue refunds, size loan offers, suspend accounts and label dishes as nut-free. Once software is making calls like these, a buyer needs to know how it makes them, what limits it, and what happens when it gets one wrong.
That is the job of the Agentic AI Workflow & Guardrail Audit (AWGA), part of the Quandary Peak Light House framework. We work with the target’s developers to understand what the system was built to do, and with its production team to see what it has actually been doing. This post explains how the audit runs and walks through three engagements where it found problems serious enough to change the deal.
Why Buyers Are Asking About This Now
Market.us puts agentic AI workflow audit services at USD 0.6 billion in 2025 and projects USD 22.4 billion by 2035, a compound annual growth rate of 42.9%. Financial services firms are the largest buyers at 27.9% of the market, and multi-agent workflows make up 54.7% of what gets audited. Pre-deployment audits are the biggest service line today, and runtime monitoring is growing fastest [1].
Regulation explains part of the demand, though the picture is messy. US banking regulators replaced their model risk guidance in April 2026, and the new version (SR 26-2) says plainly that generative and agentic AI models are outside its scope [2]. The CFPB withdrew its circulars on AI and adverse action notices in 2025 [3], but Regulation B still requires a lender to give the specific principal reasons for a decline, including to small-business applicants with revenue of USD 1 million or less [4]. Europe pushed the deadline for high-risk AI obligations such as credit scoring to December 2027 without changing the obligations [5]. The Digital Services Act already requires platforms to disclose automated moderation decisions [6]. And in Moffatt v. Air Canada, a tribunal held the airline responsible for what its chatbot told a customer [7].
So there is less official guidance on how to govern an agent and no less liability when one misbehaves. In a transaction that gap usually lands on the buyer, and diligence is the cheapest place to measure it.
How the Audit is Put Together
Light House is Quandary Peak’s AI-assisted platform for technical due diligence. Its Scalability Assessment Kit and Tool Maturity Impact Analysis score architecture, code and engineering practice on a five-level maturity scale [8]. AWGA reuses the same evidence handling and sandboxed scanning and adds tooling built for agents.
We follow three rules. We never run adversarial tests against production. We don’t report a finding without hashed evidence behind it, such as a file at a pinned commit, a trace ID or a recorded test run. And we score controls by how they perform when we measure them, not by whether they appear in a policy document. A guardrail that catches 99% of unsafe content when traffic is quiet and 40% at peak gets credit for 40%.
Before testing, we map the target onto a reference model with four trust zones and ten control points (C1 to C10), covering guardrails, budgets, checkpoints, memory isolation, model pinning, the tool gateway, the policy engine, human review, decision records and kill switches. Each maps to the OWASP Top 10 for Agentic Applications [9].
Every finding gets an Agentic Exposure Score (AES) from 0 to 100. It multiplies impact (financial, data, regulatory or safety) by likelihood (seen in production, reachable from untrusted input, easy to trigger), then discounts for compensating controls based on how they performed in testing. We treat 60 and above as critical and 40 to 60 as high.
Working With the Development Team
With the people who built the system, we want to answer one question: as built, what can this thing do, and what stops it from doing the wrong thing?
- D1, inventory: We parse the code for every agent, tool, model binding, prompt and MCP server. The list is almost always longer than the one we were given.
- D2, capability graph: We label each tool with what it can touch (untrusted input, private data, state changes, outbound messages, money) and check every agent session against Meta’s Agents Rule of Two: no session should combine untrusted input, sensitive data and consequential actions without a human approving [10]. It builds on Simon Willison’s “lethal trifecta” [11].
- D3, policy conformance: We look for business rules that exist only in a prompt. “Never refund more than $40” in a system prompt is a suggestion, and nothing enforces it.
- D4, supply chain: We flag model aliases a provider can repoint without notice, and fingerprint MCP tool descriptions to catch changes after approval.
- D5, reliability lab: We run each task many times and measure pass^k, the chance of getting it right on all k tries. Consistency drops fast as k grows, even for strong models [12].
- D6, adversarial harness: We plant payloads with unique canary strings in every untrusted channel and watch whether any reach a mocked tool.
- D7, handover: Every check ships as a CI job the buyer’s team can keep running after close.
Here is D2’s output on a real support-agent codebase. Text a customer types into a ticket can reach a Stripe refund through two agents, with nothing deterministic in between:
PATH get_ticket_messages -> SupportTriage -> RefundSpecialist -> issue_refund
['irreversible', 'money_movement', 'state_change'] UNGATED
PATH get_ticket_messages -> SupportTriage -> message_courier
['external_comm'] UNGATED
violates_rule_of_two: true
ungated_consequential_tools: ['issue_refund', 'message_courier']pass^k is why “our agent is 87% accurate” doesn’t tell you whether it can run on its own. An agent that solves a task in 6 of 8 tries has a pass^1 of 0.75, a pass^4 of 0.21 and a pass^8 of zero.
# file: lighthouse_awga/reliability.py
def pass_hat_k(n: int, c: int, k: int) -> float:
if k > n:
raise ValueError("k must be <= n")
return math.comb(c, k) / math.comb(n, k)Working With the Production Team
With the production team, we want to know what the agents have actually done and whether anyone would notice if they went wrong. We work from exported OpenTelemetry GenAI spans, incident tickets, review-queue logs and billing data.
- P1, telemetry sufficiency: Can we rebuild one decision end to end, including inputs, model version, tool arguments, policy result and approver?
- P2, trace invariants: We check the rules the target says always hold, such as “every side effect follows a policy allow” and “no order is refunded twice,” against the full export.
- P3, replay and counterfactuals: We replay frozen inputs against the pinned model to test stability, then change one field at a time to test sensitivity.
- P4, guardrail chaos: We inject latency and errors into guardrail services at the target’s real peak concurrency.
- P5, oversight: We measure reviewer agreement, time per item and whether evidence gets opened. One reviewer we looked at approved 300 of 300 recommendations at a median of 3.4 seconds each.
- P6, kill switch and cost drills: We time how long it takes to switch off one agent, one tool or all autonomy, and find which conversations drive the LLM bill.
Most production findings come from invariant queries like this one, which finds side effects that happened more than once:
-- file: lighthouse_awga/trace_invariants.sql
-- Q3. Duplicate side effects: same business key, more than one successful refund.
SELECT
attrs->>'$."app.order_id"' AS order_id,
count(*) AS successful_refunds,
count(DISTINCT trace_id) AS traces,
sum((attrs->>'$."app.amount_cents"')::BIGINT) AS total_cents,
list(attrs->>'$."gen_ai.tool.call.id"') AS call_ids
FROM spans
WHERE attrs->>'$."gen_ai.tool.name"' = 'issue_refund' AND status_code = 'OK'
GROUP BY 1
HAVING count(*) > 1
ORDER BY total_cents DESC;Three Engagements
These engagements are anonymized and the figures rounded. The code is simplified from each target’s code, with the defect left in.
Case A: A social network with 10K+ concurrent users
A consumer social app with about 10,000 concurrent users at its evening peak and a large EU audience. It ran a safety classifier on the posting path, a three-agent moderation pipeline (triage, policy, enforcement) that could ban accounts, and a “Creator Copilot” with long-term memory. The buyer was a strategic media acquirer.
The guardrail failed open under load (A-01, AES 50.1). At peak, the classifier’s p99 latency hit 2.3 seconds against an 800 ms timeout, and every timeout counted as “safe,” so about 380,000 posts went live unchecked over 60 days. The copilot could recall other users’ memories (A-02, AES 80.0). Every probe from a new account surfaced a canary we had planted in someone else’s memory, and production logs showed 2,870 cross-user recalls in 30 days, some drawn from DMs. And hidden text in a reported post could turn the enforcement agent against the reporter (A-03, AES 60.8). We found 37 production bans where the banned user had reported the same case.
How Light House identified and located the issues
- Static analysis flagged a broad except Exception around the guardrail call that returned “safe.”
- A Locust load test at 10,000 concurrent users with 2% seeded harmful posts measured a 6.1% bypass rate. The same query over 60 days of production spans found 14 high-bypass peak windows.
- The memory module used one shared vector namespace, filtered by owner only after top-k retrieval, and fell back to anyone’s top hits when a user had none.
- Canary probes measured the leak, and joining retrieval spans to memory ownership gave counsel the list of affected users.
- The capability graph showed an ungated path from fetch_post through all three moderation agents to ban_user. Our injection harness flipped the policy verdict in 58% of trials.
The fail-open handler as we found it, then the fix that moves tenancy into the database:
# file: cases/case_a_social.py
GUARD_TIMEOUT_S = 0.8
async def moderate_and_publish_vulnerable(post: Post, guard_client, publish, quarantine):
try:
verdict = await asyncio.wait_for(guard_client.classify(post.text), GUARD_TIMEOUT_S)
except Exception as exc: # TimeoutError, HTTP 5xx, connection reset ...
log.warning("guardrail unavailable, defaulting to allow: %s", exc)
verdict = Verdict(label="safe", score=0.0, source="default") # <-- fail-open
if verdict.label == "unsafe":
return await quarantine(post, verdict)
return await publish(post)
# file: cases/case_a_social.py
RLS_DDL = """
ALTER TABLE memories ENABLE ROW LEVEL SECURITY;
ALTER TABLE memories FORCE ROW LEVEL SECURITY;
CREATE POLICY memories_owner_only ON memories
USING (owner_id = current_setting('app.user_id', true)::uuid)
WITH CHECK (owner_id = current_setting('app.user_id', true)::uuid);
REVOKE ALL ON memories FROM PUBLIC;
GRANT SELECT, INSERT ON memories TO copilot_app; -- copilot_app does not own the table
"""Clarifying questions Light House asked
- What happens on the posting path when the classifier times out? Is it the same on teen and DM surfaces?
- What are the classifier’s p99 latencies at peak, and how much capacity sits behind it?
- Which agents see raw user text, and which handoffs pass the whole transcript?
- Can the enforcement agent ban someone without a human, and what non-LLM signals does it need?
- Is the copilot’s owner filter enforced in the vector store, the database or application code?
What it meant for the deal: The purchase agreement added a specific indemnity for pre-closing privacy claims, and row-level security and fail-closed guardrails became closing conditions.
Case B: An online marketplace with 1 million+ subscribers
A marketplace with about one million paying subscribers and a fast-growing seller-financing line: working-capital loans made through a bank partner, around USD 210 million a year. Four agents read uploaded bank-statement PDFs, made underwriting calls on top of an XGBoost score, sized and sent offers, and wrote decline notices. The buyer was a fintech private equity fund.
The cap limiting each loan to 15% of the seller’s trailing GMV existed only in a prompt (B-01, AES 61.3). Over 90 days, 2.3% of offers broke it, by USD 6.8 million in total. Applicants could hide white-on-white “underwriter memos” in their PDFs. In our harness those memos raised the offer 44% of the time, and 11 production documents contained the same kind of text (B-02, AES 52.9). The underwriting agent used a floating model alias. When the provider changed the model behind it, borderline approval rates moved 4.1 points in a week with no deploy, and replaying the same cases flipped 7.9% of decisions (B-03, AES 69.5). And 41% of sampled decline notices didn’t give the reasons the model actually used, while 17% said “did not meet our internal criteria,” which Regulation B says is not enough (B-04, AES 72.0).
How Light House identified and located the issues
- The policy checker pulled the cap sentence from the offer agent’s instructions and found no check in create_offer and no policy bundle in the repository.
- Joining offers with daily seller snapshots listed every over-cap offer and the amount over.
- The capability graph marked PDF parsing as untrusted input reaching create_offer, and a scan of 61,000 stored extractions found the hidden instructions.
- Grouping chat spans by gen_ai.response.model showed the served model changing the same week approval rates moved.
- A faithfulness check compared each notice’s reasons with the top negative feature contributions behind the score.
Our fix moves the cap into policy-as-code that reads authoritative seller data:
# file: cases/policies/capital_offer.rego
package marketplace.capital.offer
import rego.v1
default allow := false
cap_pct := 15
max_amount := ((input.seller.gmv_ttm_usd * cap_pct) / 100) - input.seller.existing_advance_usd
deny contains msg if {
input.offer.amount_usd > max_amount
msg := sprintf("amount %v exceeds 15%% of TTM GMV net of existing advances (%v)", [input.offer.amount_usd, max_amount])
}
deny contains "factor rate below approved band" if input.offer.factor_rate < 1.06
deny contains "factor rate above approved band" if input.offer.factor_rate > 1.18
deny contains "seller KYB not verified" if input.seller.kyb_status != "verified"
deny contains "score below program floor" if input.score < 0.42
allow if count(deny) == 0Clarifying questions Light House asked
- Where is the loan cap enforced, and which system is the source of truth for GMV?
- Does each agent call a dated model snapshot or an alias? Who finds out when the provider changes it?
- Does intake read the PDF text layer or OCR of the rendered page? Is invisible text stripped?
- What does the notice writer get as input? Does it ever see reason codes?
- For any decision in your retention window, can you produce the model version, inputs, score contributions and policy result?
What it meant for the deal: An escrow was sized to cover look-back remediation and the LLM override was switched off before closing. Pinned models, the policy engine and complete decision records became closing conditions.
Case C: A food delivery platform
A regional delivery platform whose agents handled customer support (triage plus a refund specialist), turned restaurant PDFs into allergen-tagged menus, and messaged couriers. The buyer was a competitor counting on savings from automated support.
Refunds were being paid twice during retry storms (C-01, AES 41.7). A retry decorator wrapped the whole agent step, so when the follow-up model call hit a rate limit, the step ran again, refund and all. That affected 1.9% of agent refunds, USD 412,000 over 90 days. Courier delivery notes were editable free text that the agents read, and instructions planted there produced unauthorized refunds in 38% of our trials (C-02, AES 47.1). The menu agent tagged dishes nut-free whenever a menu didn’t mention nuts. Of those tags, 2.7% had nothing behind them, and 41 were on dishes whose own descriptions listed peanuts, pesto or similar (C-03, AES 60.0, safety-critical). And 0.4% of conversations used 31% of the LLM budget because agents could pass them back and forth without limit (C-04).
How Light House identified and located the issues
- Static analysis flagged issue_refund as money-moving with no idempotency key, and the orchestrator showed the retry wrapping tool execution.
- Grouping successful refund spans by order ID found the duplicates, clustered in three model-provider incident windows.
- The capability graph traced courier notes to issue_refund through a handoff, and a sweep of 2.4 million notes found 1,140 with text aimed at the agent.
- A grounding check compared each allergen tag with the dish’s own text on 3,000 items labelled by a food-safety reviewer.
- Ranking conversations by token share led us to the loop between the triage and refund agents.
The vulnerable step, then the idempotency key we recommended. The same key goes to the payment provider’s idempotency_key, so a retry can’t pay out twice.
# file: cases/case_c_food_delivery.py
@retry(stop=stop_after_attempt(4), wait=wait_exponential(multiplier=0.5, max=8),
retry=retry_if_exception_type((RateLimitError, APITimeoutError)))
async def run_step_vulnerable(llm, tools, state):
response = await llm.chat(messages=state.messages, tools=tools.schemas())
for call in response.tool_calls:
result = await tools.call(call.name, **call.arguments) # side effect
state.messages.append({"role": "tool", "tool_call_id": call.id, "content": str(result)})
followup = await llm.chat(messages=state.messages) # 429 here => whole step retried
state.messages.append(followup.message)
return state
# file: cases/case_c_food_delivery.py
def refund_key(order_id: str, ticket_id: str, reason_class: str) -> str:
return hashlib.sha256(f"refund:v1:{order_id}:{ticket_id}:{reason_class}".encode()).hexdigest()Clarifying questions Light House asked
- What exactly does the retry decorator wrap? Could a retry rerun a tool that already succeeded?
- Is the refund idempotency key built from the business event or the model’s tool-call ID?
- Which ticket fields can couriers, restaurants or customers edit, and which do the agents read?
- What does menu ingestion do when a menu says nothing about an allergen?
- Is there a per-conversation cap on tokens, model calls and time?
What it meant for the deal: Fixing allergen tags on every live menu became a closing condition, backed by a specific indemnity. Duplicate refunds and refund abuse went into a working-capital adjustment, and the budget circuit breaker shipped two weeks after close.
What the Buyer Walks Away With
Each engagement produces a red-flag memo, purchase agreement language (indemnities, escrows, closing conditions) tied to hashed evidence, and a 100-day remediation plan with CI checks that show whether each fix is holding. The Red Flag pass can run before LOI with only repository access, the data room and a sample of production telemetry.
If the company you’re evaluating relies on agents, we’d be glad to talk through what a Light House AWGA review would cover.
