06 / thinking in public
Field notes
Engineering perspectives on data and behavior. These notes explain our approach; they are not executed benchmark results or completed research.
An example of success says little about what happens when a tool times out or a state assumption becomes invalid. Failure data is useful when it identifies the state at the point of interruption and records the actions that follow.
A recovery trajectory needs a checkable outcome. The agent’s explanation is evidence of what it said; the environment and verifier establish what it changed. Keep those two kinds of evidence separate.
Replay turns a recorded trajectory into a testable sequence. The initial state, tool interfaces and dependencies must be sufficient to execute the same task again. A reset operation makes that starting point explicit.
Reproducibility is scoped. A deterministic fixture is useful for checking logic, but does not prove that a live external service will behave identically. Record what is controlled, what is simulated and what remains outside the test boundary.
A synthetic world can expose the effects of actions without relying on an explanation in natural language. Design a task family, define the permitted transitions and choose seeds that give identifiable initial states.
Accept valid outcomes and reject plausible incorrect outcomes. Negative tests help reveal a verifier that passes too easily. A visual render illustrates the world; it does not independently certify an episode.
Schema validity is one gate, not the whole result. Reset, replay, final-state verification and provenance each answer a different question about a record. A rejected record should retain the reason it failed.
Independent reruns strengthen an evidence trail when they actually execute the interfaces. State the scope of the audit, keep its artifacts and avoid presenting dataset QA as an agent performance score.