WINDWAY DATA

03 / evidence over impressions

Private Evaluation

Would the result survive another run?

An answer-only benchmark can grade the response while missing what the agent did. Environment-grounded evaluation records actions and checks their consequences against explicit rules and task predicates.

Private Evaluation starts with your agent interface, authorized environment and evaluation question. We agree held-out tasks, allowed tools, run conditions, failure scenarios and reporting boundaries before execution.

Current GameOps evidence demonstrates dataset verification, not measured performance of a customer agent. Commercial measurement capabilities are configured and validated against the agreed environment before a score is reported.

Public chatbot evaluations

No real chatbot evaluations have been approved for publication yet.

Published runs
0
Published turns
0

Read the evaluation summary

Try Explorer ↗

Inspect the three offers ↗ · WFRB v0 methodology & results ↗ · Explore the design partner program ↗

Answers versus consequences.

ANSWER-ONLY BOUNDARY

Grade the response.

  1. Returned answer
  2. Reference comparison
  3. Answer grade

ENVIRONMENT-GROUNDED BOUNDARY

Inspect the run.

  1. Action + observed state
  2. Constraint + outcome checks
  3. Verifier evidence

A comparison of instrument boundaries, not an executed customer benchmark.

Two evaluation boundaries
QuestionAnswer-only benchmarkEnvironment-grounded evaluation
What is checked?The returned answer against a reference or rubric.Actions, transitions and final state against environment rules.
What may be missed?Invalid actions or side effects behind a convincing response.Behavior outside the instrumented environment and defined checks.
What is retained?An answer and its grading evidence.A replayable trajectory, verifier outcome and scoped provenance.

Define the measurement contract.

These dimensions are not a claim that a customer agent has already been scored.

Existing foundations and configurable commercial measures
DimensionCurrent evidence / proposed measurementBoundary
Task successCanonical episode final-state predicates.Dataset verification today; agent success rate requires actual runs.
Action validityInvalid moves and argument constraints in controlled fixtures.Validated only for the implemented action schema.
Trajectory efficiencyAgreed step, time or cost budgets.Configurable; no existing efficiency score claimed.
Failure handlingRecorded failure types and failure states.Live tool failures require environment-specific instrumentation.
RecoveryReplay checked recovery trajectories.Agent recovery measurements require controlled failure trials.
Constraint complianceExplicit environment rule checks.Client constraints need implementation and negative tests.
State integrityInitial/final states and retained failure state.External side effects require an agreed observation boundary.
ReproducibilityReset, replay and provenance gates.External services may remain nondeterministic.

A private evaluation package.

Your inputs
Agent interface, authorized tasks, constraints, failure scenarios and success predicates.
Agreed protocol
Held-out task split, run conditions, retry policy and which checks are implemented.
Delivery
Per-task outcomes, trajectories, verifier evidence, aggregate measures actually executed and explicit limitations.
Scope a private evaluation ↗