A frontier AI lab needed to measure how its agents plan, reason, and adapt across multi-step tool use. Labelbox built the simulated environments and expert-graded benchmark that made agentic performance measurable.

The challenge

Truly autonomous agents are built and tested in environments that mirror real-world complexity. A leading AI lab at the frontier of agentic development needed to measure how its models plan, reason, and adapt across intricate, multi-step workflows — agents interacting with tools and databases, adapting to dynamic feedback. Evaluating that demands more than instruction-following: models had to interpret ambiguous natural language, chain correct API calls based on evolving responses, and reach a programmatically verifiable end state. Agentic systems run on rich structured feedback, verifiable outcomes, and task-specific evaluation. The lab needed all three at scale.

The approach

Labelbox built the evaluation environment. It constructed multiple distinct domain environments — each a self-contained ecosystem with its own database and API interfaces covering actions like querying availability, inserting reservations, updating user data, and modifying transaction details. For every interface, Labelbox produced a high volume of multi-step tasks crafted to test whether an agent could:

• Interpret natural language instructions with precision: parse complex, often ambiguous human requests into actionable steps.

• Plan a series of tool calls effectively: understand dependencies between API calls and orchestrate them in the correct logical sequence.

• Adapt to tool responses dynamically: handle success, failure, and unexpected data, adjusting subsequent actions.

• Reach a programmatically verifiable end state: achieve an outcome the system could objectively confirm.

Example tasks show the difficulty:

• "Help me book a hotel for three people arriving on different dates, then cancel one night if the price exceeds $300." Sequential planning, conditional logic, and modifying prior decisions based on new information.

• "Return an item purchased online, update the refund method to store credit, and send a confirmation email." Multiple system updates and external communication.

• "Schedule a car rental, then adjust pickup time based on flight delay info." Real-time adaptation and integration of external, time-sensitive data.

Labelbox's platform captured two tiers of domain expertise. One defined the architectural foundation: building the interfaces, defining tool schemas, and ensuring the technical accuracy of each API function in languages like Python. The other invented realistic task prompts and mapped the ground-truth action plans, validating the resulting database state. API sequences had to reflect conditional logic, so agents adapted with branching logic based on tool outputs. Every task was a unique planning challenge, and Labelbox graded its difficulty by whether it stumped the lab's base model versus other leading models.

The outcome

Labelbox delivered a benchmark of 25 distinct API interfaces, hundreds of defined API functions, and over 1,000 complex tool-use tasks spanning multiple domains. Each task was scored for difficulty and verified through structured outputs. The lab gained an objective mechanism to evaluate how well its agentic models reason through multi-step goals, adapt under uncertainty, and achieve structured outcomes through tool interfaces — surfacing gaps in reasoning and execution, and accelerating product development and model iteration with high-quality frontier data and evaluation frameworks.

Where this goes

This is RL infrastructure for the frontier: simulated environments, programmatically verifiable rewards, and expert-graded signal that make agentic capability measurable — and trainable.