agent-eval-lab · report

agent-eval-lab — v1-hybrid vs v0-keyword, 24 cases

idf-weighted document ranking with acronym expansion and a title prior, extracts the best-supported sentence from the winning document, and abstains below a threshold calibrated on the dev split.

83.3%
pass rate · 192 executions
66.7%–95.8%
95% cluster bootstrap CI
0.833
pass@3 (unbiased)
0.833
pass^3 (all 3 pass)
24
cases · 0.8333333333333334 fully reliable

Failures by grader

graderfailed executionsshare of failures
facts2433.3%
citations2433.3%
refusal2433.3%

Cases · 4 with at least one failure

casequestionpass / samplesevidence
team-price
pricing
How much does Team cost per seat each month?
8/8
all samples passed
starter-price
pricing
What does Starter cost per seat per month?
8/8
all samples passed
business-price
pricing
What is the Business plan price per seat?
8/8
all samples passed
annual-discount
pricing
How much does annual billing save on the list price?
8/8
all samples passed
monthly-refund-window
billing
How long do I have to get a full refund on a monthly charge?
8/8
all samples passed
annual-refund-method
billing
How are annual plans refunded if we cancel?
8/8
all samples passed
refund-cutoff
billing
When do refunds stop being issued?
8/8
all samples passed
purge-grace
retention
How long is the grace period after deleting a workspace?
8/8
all samples passed
backup-retention
retention
How long are backups kept after deletion?
8/8
all samples passed
enterprise-retention
retention
Can Enterprise keep data longer than the default retention window?
8/8
all samples passed
export-latency
retention
How fast can I get a full export of my data?
8/8
all samples passed
sso-protocols
identity
Which SSO protocols are supported?
8/8
all samples passed
scim-plans
identity
Which plans include SCIM provisioning?
8/8
all samples passed
session-timeout-range
identity
What is the configurable range for session timeout?
0/8
first failure · sample 0

I don't know based on the provided documents.

citations: none

  • facts — missing: 15 minutes, 12 hours
  • citations — answerable case answered without a citation
  • refusal — expected an answer, got a refusal
team-rate-limit
api
How many API requests per minute can a Team workspace send?
8/8
all samples passed
business-rate-limit
api
What is the API request limit for Business?
8/8
all samples passed
burst-bucket
api
How large can a burst be before throttling starts?
0/8
first failure · sample 0

I don't know based on the provided documents.

citations: none

  • facts — missing: twice, 10 seconds
  • citations — answerable case answered without a citation
  • refusal — expected an answer, got a refusal
uptime-sla
reliability
What uptime SLA applies to paid plans?
8/8
all samples passed
credit-tier-99
reliability
What credit do I get if a month lands below 99.0%?
8/8
all samples passed
credit-claim-window
reliability
How long do I have to claim a service credit?
0/8
first failure · sample 0

A month below 99.9% earns a 10% service credit.

citations: KB-006

  • facts — missing: 30 days
self-hosted
unanswerable
Do you offer an on-premises or self-hosted deployment?
8/8
all samples passed
dedicated-slack
unanswerable
Which plan includes a dedicated Slack channel with the support team?
0/8
first failure · sample 0

The Slack integration posts to a channel of your choice.

citations: KB-009

  • citations — refusal cited KB-009
  • refusal — expected refusal, got an answer
enterprise-seats
unanswerable
How many seats are included in the Enterprise plan?
8/8
all samples passed
addon-seat-refund
unanswerable
Are add-on seats refundable?
8/8
all samples passed

Provenance

candidate
v1-hybrid
lab version
1.0.0
corpus
Kestrel (8b2de3583a6a)
dataset hash
c9f0a16c1ee5
split
all
samples per case
8
seed
42
started
2026-11-16T10:47:14.376Z
duration
61 ms
runtime
v22.23.4
generated
2026-11-16T10:47:14.446Z