Agent Eval Set

33 work-order scenarios · regenerate when scenarios.mjs changes

Every scenario the work-order agent is tested against, one per section: the situation as a text thread, the setup (pre-loaded state), and the expected behavior the harness asserts. Source of truth is agent/evals/scenarios.mjs; run with node agent/evals/run.mjs --filter <name>. The demo (sales) surface is tested separately and not covered here.

Each scenario is one orchestrator turn: the harness pre-loads setup (sent-log = open candidates, supabase = mocked issue rows incl. status), fires one event (new_issue or incoming_user_message), then asserts the tool calls / drafts / outbox. Mockup phrasing is representative; the exact assertions live in the named scenario.

Conversational presence: the agent always replies to the PM with a brief, weight-matched message (an ack, an answer, or "not sure what that's about") even when it takes no work-order action. What changes per case is the tool calls (dispatch / close / learn / lookup), not whether it speaks. Every PM-facing message is a draft today (never auto-sent); under full autonomy the same replies send directly. The scenario keys below are unchanged code identifiers — some still read "no_match" though the agent now replies.

Sections
  1. New work orders — 5 · process_wo
  2. PM replies — dispatch — 7
  3. PM replies — closing the loop (status) — 4
  4. PM replies — clarify & no-match — 6
  5. PM replies — learning & memory lookup — 6
  6. PM replies — follow-up answer — 2
  7. Voice — vendor short names & issue casing — 3 · deterministic

Legend: dispatch close status clarify no-match learn memory lookup reply

1 · New work orders — event new_issue

A work order lands already enriched. The agent reads memory, picks a vendor if it has a candidate, and stages ONE summary bubble to the PM. No greetings, no urgency prefix, line 2 is one natural sentence.

Standard issue with a vendor candidate

process_wo: standard issue with vendor candidate → one multi-line draft
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Setup issue: faucet @ Unit 701, vendor candidate Mario
Expect summary 1 draft
Tools read_memoryupdate_issuesend_text
draft includes "Unit 701", "Mario", "Should I send"

No confident vendor → no vendor question

process_wo: no candidate vendor → read_memory + 1 draft, no vendor question
Unit 701 Hub Champaign Has no wifi.
Setup issue: wifi down, vendor = null
Expect summary 1 draft, 2 lines
Tools read_memorysend_text update_issue
no "Should I send", natural grammar on line 2

Missing unit → location is the property

process_wo: missing unit → location line is just the property
Hub Champaign There's a roof leak in the lobby. Should I send Mario?
Setup issue: roof leak, unit = null
Expect summary line 1 = property, no "Unit "
Tools read_memoryupdate_issuesend_text

Awkward "has X" title → natural grammar

process_wo: awkward "has X" title → natural grammar on line 2
Unit 701 Hub Champaign The dryer isn't working. Should I send Mario?
Setup title "dryer not working"
Expect summary natural sentence
never "has dryer not working"; no colon/em-dash

Long description → compact line 2

process_wo: long description gets compact line 2
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Setup multi-paragraph description (towels, cabinet, days)
Expect summary ONE short sentence
excludes "towels", "cabinet", "several days"

2 · PM replies — dispatch — event incoming_user_message

PM approves a listed work order. Agent stages an ack, drafts the tenant + vendor messages, advances the WO to dispatched, and logs the decision.

Clear "yes", one open candidate

chat: clear "yes" with one candidate → ack + tenant + vendor drafts + observation
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
yes
Setup 1 candidate: faucet (Mario)
Expect dispatch 3 drafts
Tools send_textdraft_tenantdraft_vendorupdate_issuewrite_memory

Affirmative naming the same vendor

chat: affirmative naming same vendor ("yep send Mario") → drafts
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
yep send Mario
Setup 1 candidate: faucet (Mario)
Expect dispatch 3 drafts
Tools send_textdraft_tenantdraft_vendorupdate_issuewrite_memory

Vendor swap ("no send Luigi instead")

chat: vendor swap ("no send Luigi instead") → drafts with Luigi + write_memory
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
no send Luigi instead
Setup 1 candidate: faucet (we suggested Mario)
Expect dispatch drafts name Luigi; ack asks why
Tools send_textdraft_tenantdraft_vendorupdate_issuewrite_memory

Bare swap with no "instead" cue

chat: bare vendor swap ("send Yonic") → drafts with Yonic, not Mario
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
send Yonic
Setup 1 candidate: faucet (we suggested Mario)
Expect dispatch drafts name Yonic, not Mario
regression for 2026-05-24 "Send yonic" miss

Burst settles to one turn (T1)

chat: settled burst ("yes" / "wait" / "send Luigi not Mario") → one turn, dispatch Luigi
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
yes go ahead
J
wait
J
actually send Luigi instead of Mario
Setup 1 candidate: faucet (suggested Mario). 3 rapid bubbles the poller settles into one turn (newline-joined)
Expect dispatch reads the whole burst — drafts name Luigi, never Mario off the first "yes"
Tools send_textdraft_tenantdraft_vendorwrite_memory

Override ack is action-neutral (no "sending" verb)

chat: vendor override ack is action-neutral (no "sending"/"dispatching" verb)
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
send Luigi instead
on it. any reason you prefer Luigi over Yonic?
Setup 1 candidate: faucet (suggested Mario). PM overrides to Luigi
Expect dispatch + ask why, but the ack must NOT contain "sending"/"dispatching"/"going with" — it narrates an in-progress send that's still only a staged draft
Tools send_textdraft_tenantdraft_vendorwrite_memory

Dispatch + side-channel preference in one message

chat: dispatch + side-channel preference in one message → both fire
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
yes go ahead. and just so you know we always use Yonic for plumbing here.
Setup 1 candidate: faucet
Expect dispatchlearn both fire
Tools send_textdraft_tenantdraft_vendorwrite_memory

3 · PM replies — closing the loop — the 2026-05-30 fix

Every reply that resolves a WO advances its status so it leaves the open-candidate pool. This is what stops answered orders resurfacing as phantom candidates.

PM self-handled → pm_handling

chat wo_status: "I already took care of those" → update_issue pm_handling, no dispatch
Unit 1 1447 Harvard St The dryer smells like it's burning. Should I send Cross Appliance?
Unit 2 1829 11th St The bathroom light fixture needs replacement. Should I send Abraham?
Jose
J
I already took care of those work orders
great, thanks for handling those
Setup 2 candidates: dryer, light
Expect close ×2reply status pm_handling
Tools update_issuesend_text draft_tenantdraft_vendor
brief ack of their update; closes both, no dispatch, no auto-send

PM redirects to tenant triage → triaging

chat wo_status: "have the tenant send a photo first" → update_issue triaging
Unit 3 205 Horizon Ave The refrigerator drawer is broken. Should I send Cross Appliance Inc?
Jose
J
Have the resident send a photo of the drawer and the model number of the fridge first
got it
Setup 1 candidate: fridge
Expect triagereply status triaging
Tools update_issuesend_text
brief ack (a bare "got it" passes; restating "photo + model number" is fine but not required); must not claim dispatch

Lone "Yes, please" → dispatch + dispatched

chat wo_status: lone "Yes, please" → dispatch + update_issue dispatched
Unit 4 824 11th St The laundry machine is not draining. Should I send Cross Appliance Inc?
Jose
J
Yes, please
Setup 1 candidate: laundry
Expect dispatch 3 drafts, status dispatched
Tools send_textdraft_tenantdraft_vendorupdate_issue

Incident replay — resolved WOs excluded

chat wo_status: resolved WOs excluded → lone "Yes, please" dispatches, not clarifies
Unit 4 824 11th St The laundry machine is not draining. Should I send Cross Appliance Inc?
Jose
J
Yes, please
Liked "Yes, please"
Setup 4 in sent-log; dryer/light = pm_handling, fridge = triaging, laundry = awaiting_pm
Expect dispatch NOT clarify
candidate filter leaves only laundry → 3 drafts; judge: ack, not "which one?"

4 · PM replies — clarify & no-match

Clarify ONLY when 2+ genuinely-open candidates and the reply doesn't say which. Everything else here takes no work-order action — but the agent still replies: it answers the question, acks the update, or surfaces that nothing's pending. "No-match" now means no dispatch/close, not silence.

"yes" with two open candidates → clarify

chat: "yes" with two open candidates → drafts a clarifying question
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Unit 701 Hub Champaign The AC isn't blowing cold. Should I send California Heat?
Jose
J
yes
Setup 2 candidates: faucet, AC
Expect clarify 1 draft
Tools send_text (draft:true) draft_tenantdraft_vendor
judge: asks faucet vs AC, picks neither

PM self-handles outside the loop → ack

chat: PM self-handles issue ("assigned to guox, will talk tomorrow") → silent no_match
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
Assigned plumbing to guox and I'll talk to them about the filter tomorrow
got it, thanks for the heads-up
Setup 1 candidate: faucet
Expect replyclose brief ack; may close faucet → pm_handling
Tools send_text draft_tenantdraft_vendor
regression: outbox MUST stay empty (draft only, never the robotic "No action taken.")

Question back to the agent → answers honestly

chat: info question ("what is the lockbox code?") → no_match
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
what is the lockbox code for hub champaign?
i don't have a lockbox code for hub champaign on file. do you have it?
Setup 1 candidate: faucet
Expect replylookup answers it lacks the code
Tools read_memorysend_text
no invented code; no dispatch; no false "let me check"

Bare "ok" with no recent sends

chat: ambiguous "ok" with no recent sends → no_match
Jose
J
ok
all good?
Setup 0 candidates
Expect reply brief reply or none — the ambiguous edge
Tools send_text draft_tenantdraft_vendor
never a "which work order?" question (none open)

"yes go ahead" with no sent-log

chat: reply with no sent-log history → no_match
Jose
J
yes go ahead
not sure what you're referring to, what's up?
Setup 0 candidates
Expect reply approves nothing unlisted; surfaces the gap
Tools send_text draft_tenantdraft_vendor
the case that prompted this work: don't ghost an unmappable "yes"

Prod-chat sends don't leak into test-chat

chat: prod-chat sends do not leak into test-chat candidates
Jose (test chat)
J
yes
not sure what that's about, anything open on your end?
Setup a send exists, but in a DIFFERENT chat_guid
Expect reply candidates scoped per chat; never touches the prod WO
Tools send_text draft_tenantdraft_vendorupdate_issue

5 · PM replies — learning & memory lookup

Side-channel preferences and corrections get recorded (write_memory) plus a brief ack, no dispatch. Questions that hinge on prior context trigger a read_memory lookup, then the agent answers with what it found.

Stated preference → learn

chat: stated preference ("always use Yonic for plumbing") → write_memory, no dispatch
Jose
J
btw always use Yonic for plumbing at the Hub Champaign building
noted, Yonic for plumbing at Hub Champaign going forward
Setup 0 candidates
Expect learnreply no dispatch
Tools write_memorysend_text

Per-property quirk → learn

chat: per-property quirk (elevator vendor) → write_memory, no dispatch
Jose
J
fyi the elevator vendor for 1234 Main St is Acme Elevators, they have a contract
good to know, noted
Setup 0 candidates
Expect learnreply no dispatch
Tools write_memorysend_text

Stated preference recorded directly

chat: stated preference recorded as memory write (no read_memory needed)
Jose
J
always use Yonic for plumbing at Hub Champaign
got it, noted
Setup 0 candidates
Expect learnreply write, don't read first
Tools write_memorysend_text

Vendor-pick question with property hint → lookup

chat: read_memory fires for vendor-pick question with property hint
Jose
J
who do we use for plumbing at 17 Ozone Ave?
looks like we've used Yonic for plumbing there
Setup 0 candidates
Expect lookupreply reads with property hint, then answers
Tools read_memorysend_text

Vendor history question → lookup

chat: read_memory fires for vendor history question
Jose
J
have we used Yonic before?
yep, Yonic's done a few plumbing jobs for us
Setup 0 candidates
Expect lookupreply reads, then answers
Tools read_memorysend_text

Open-ended "yes, whoever's best" → lookup before dispatch

chat: read_memory used to verify before dispatch on ambiguous "yes"
Unit 701 Hub Champaign The kitchen faucet is leaking. Should I send Mario?
Jose
J
yes go with whoever you think is best for plumbing here
Setup 1 candidate: faucet
Expect lookup open-ended → read first
Tools read_memory

6 · PM replies — follow-up answer

When history shows we asked WHY a vendor was swapped and the PM answers with a reason, record it and give a brief ack of the reason — no dispatch (that already happened on the override turn).

Answering "any reason you prefer him?"

chat: follow-up answer ("he's cheaper") → write_memory only, no dispatch
Jose
J
no send Luigi instead
on it. any reason you prefer Luigi over Yonic?
Jose
J
he's cheaper for drain jobs
makes sense, thanks
Setup 0 live candidates; history = the swap + our "why?"
Expect learnreply records reason + brief ack
Tools write_memorysend_text draft_tenantdraft_vendor

Terse answer, unrelated WOs still open

chat: terse follow-up answer ("local plumber") with only unrelated WOs open → write_memory, no clarify
Jose
J
send it to the drain specialist
on it. any reason you prefer the drain specialist over Yonic?
Jose
J
local plumber
noted
Setup drain WO already dispatched (off the list); only an AC + a light WO open (both unrelated); history = the swap + our "why?"
Expect learnreply records reason + brief ack — recency beats the open-WO list, so the terse reason is NOT bounced with "which one? point me to the work order"
Tools write_memorysend_text draft_tenantdraft_vendor
Regression 2026-06-03 prod — the reason evaporated into a clarify draft and was never recorded

7 · Voice — vendor short names & issue casing — deterministic draft render

Drafts speak the way the PM does. The renderers shorten a vendor's legal name (drop Inc/LLC/INC.; first name for individuals, unless that first name collides with another vendor in the workspace) and lowercase a Title-cased issue phrase that lands mid-sentence. Pure string transforms — asserted on exact template substrings, so the trailing "." after the vendor proves the shortening regardless of the model's ack.

Legal name + Title-cased issue

chat voice: legal name + Title-cased issue → suffix stripped + issue lowercased
Jose
J
yes
[tenant] I sent this to Osalpa Electric. Please expect a call from them. They will schedule with you.
[vendor] Please schedule with the tenant for dead outlets after spark.
Setup vendor "Osalpa Electric, INC."; issue "Dead outlets after spark"
Expect dispatch tenant + vendor drafts
Tools send_textdraft_tenantdraft_vendor
draft includes "to Osalpa Electric." (no "INC."), "for dead outlets after spark" (lowercased)

Individual vendor → first name

chat voice: individual vendor ("Yonic Herrera") → first name in tenant draft
Jose
J
yes
[tenant] I sent this to Yonic. Please expect a call from them. They will schedule with you.
Setup vendor "Yonic Herrera"; roster has no other "Yonic"
Expect dispatch tenant + vendor drafts
Tools send_textdraft_tenantdraft_vendor
draft includes "to Yonic." — first name, not "Yonic Herrera"

First-name collision → keep full name

chat voice: two "Luis" in workspace → collision keeps full name in draft
Jose
J
yes
[tenant] I sent this to Luis Herrera. Please expect a call from them. They will schedule with you.
Setup vendor "Luis Herrera"; workspace also has "Luis Ramos"
Expect dispatch tenant + vendor drafts
Tools send_textdraft_tenantdraft_vendor
draft includes "to Luis Herrera." — first name would be ambiguous, so full name kept