agent-eval-lab · report
idf-weighted document ranking with acronym expansion and a title prior, extracts the best-supported sentence from the winning document, and abstains below a threshold calibrated on the dev split.
| grader | failed executions | share of failures |
|---|---|---|
facts | 24 | 33.3% |
citations | 24 | 33.3% |
refusal | 24 | 33.3% |
| case | question | pass / samples | evidence |
|---|---|---|---|
team-price |
How much does Team cost per seat each month? | 8/8 | all samples passed |
starter-price |
What does Starter cost per seat per month? | 8/8 | all samples passed |
business-price |
What is the Business plan price per seat? | 8/8 | all samples passed |
annual-discount |
How much does annual billing save on the list price? | 8/8 | all samples passed |
monthly-refund-window |
How long do I have to get a full refund on a monthly charge? | 8/8 | all samples passed |
annual-refund-method |
How are annual plans refunded if we cancel? | 8/8 | all samples passed |
refund-cutoff |
When do refunds stop being issued? | 8/8 | all samples passed |
purge-grace |
How long is the grace period after deleting a workspace? | 8/8 | all samples passed |
backup-retention |
How long are backups kept after deletion? | 8/8 | all samples passed |
enterprise-retention |
Can Enterprise keep data longer than the default retention window? | 8/8 | all samples passed |
export-latency |
How fast can I get a full export of my data? | 8/8 | all samples passed |
sso-protocols |
Which SSO protocols are supported? | 8/8 | all samples passed |
scim-plans |
Which plans include SCIM provisioning? | 8/8 | all samples passed |
session-timeout-range |
What is the configurable range for session timeout? | 0/8 | first failure · sample 0I don't know based on the provided documents. citations: none
|
team-rate-limit |
How many API requests per minute can a Team workspace send? | 8/8 | all samples passed |
business-rate-limit |
What is the API request limit for Business? | 8/8 | all samples passed |
burst-bucket |
How large can a burst be before throttling starts? | 0/8 | first failure · sample 0I don't know based on the provided documents. citations: none
|
uptime-sla |
What uptime SLA applies to paid plans? | 8/8 | all samples passed |
credit-tier-99 |
What credit do I get if a month lands below 99.0%? | 8/8 | all samples passed |
credit-claim-window |
How long do I have to claim a service credit? | 0/8 | first failure · sample 0A month below 99.9% earns a 10% service credit. citations: KB-006
|
self-hosted |
Do you offer an on-premises or self-hosted deployment? | 8/8 | all samples passed |
dedicated-slack |
Which plan includes a dedicated Slack channel with the support team? | 0/8 | first failure · sample 0The Slack integration posts to a channel of your choice. citations: KB-009
|
enterprise-seats |
How many seats are included in the Enterprise plan? | 8/8 | all samples passed |
addon-seat-refund |
Are add-on seats refundable? | 8/8 | all samples passed |