Opus 5.5: measure the cost of a completed job

Consider a small product team handing an agent a bug report: reproduce a broken settings form, make the smallest fix and return a patch with a regression test. The potential saving is developer time. If the patch needs repeated correction or changes unrelated behavior, the team still pays for that time.

That is a useful prospective evaluation for Kerber AI and other teams building agent workflows. It is a hypothetical task, not a report from a Kerber AI deployment or a hands-on test of Opus 5.5.

Before adding a model, check whether ordinary code already covers the job. A validator or a deterministic migration should stay ordinary code. Where diagnosis across a repository does need a model, a proven agent setup is the baseline. Opus 5.5, released on September 22, is a candidate to test against it.

Price the failed attempts too

Anthropic lists claude-opus-5-5 at $4 per million input tokens and $20 per million output tokens, with cache reads at $0.20 per million tokens. These are the standard API rates, not the separate fast-mode prices. In its announcement, Anthropic says typical workloads at default settings cost 40% less than Opus 5 in its tests. That is a provider result, not a forecast for your application.

For the settings-form task, define acceptance before running either model: the reported bug is reproduced, the regression test passes after the fix and existing behavior remains intact. The patch must stay within the requested scope. It must not deploy or merge anything.

Divide the cost of all attempts, including failures and retries, by the number of accepted jobs. Include tool and execution costs alongside API spend. Record human correction and review time separately, or price it using an explicit hourly assumption. If no jobs pass, report that result rather than a cost per success.

A cheap failed run cannot disappear from the comparison. Nor should a patch count as an unassisted success after a developer has rewritten it. Keep first-pass acceptance separate from acceptance after correction, and record elapsed time until the reviewer accepts the result.

Check the integration before comparing the models

The migration guide covers Messages API integrations. Anthropic says Claude Managed Agents requires only the model-name change; custom agent loops have more to check. A failed request caused by an obsolete parameter is a migration failure, not evidence about the model's ability to fix the form.

• Thinking is always on. Requests that disable thinking or set a manual thinking budget return a 400 error. Remove those settings and choose effort explicitly. The default is now medium, rather than Opus 5's high. Read content blocks by type and return thinking blocks unchanged in the tool loop.
• Forced tool use is rejected. tool_choice types any and tool return errors. The guide recommends auto with strict tool use, or structured outputs for schema-valid JSON. Strict arguments do not guarantee that a tool gets called. The application still needs to verify the required action happened.
• Thinking blocks are bound to the model and conversation. Keep prior messages, tools and the system prompt unchanged, appending new context instead. For accounts created on or after August 31, 2026 at 00:00 UTC, replaying a block after changing its preceding context returns an error by default. The guide documents opt-in controls for older accounts and what happens when a router switches models; do not assume reasoning transfers with the transcript.
• On the Claude API and Google Cloud, computer_20251124 is rejected. Move to computer_toolset_20260801 and update the loop for member tool calls, multiple actions per turn and toolset_name on results. The older tool still works on Amazon Bedrock. Check your platform rather than applying the change indiscriminately.

Progress reporting can also break without an error. Text between tool calls now arrives in thinking blocks. With the default thinking.display of omitted, that text is empty. The guide specifies updates with the thinking-display-updates-2026-08-18 beta header to receive progress updates while keeping reasoning hidden. Render non-empty thinking blocks before the tool calls they precede and pass the blocks back unchanged. The alternative summarized setting mixes progress updates with reasoning summaries.

Also check completion status rather than HTTP success alone. The behavior notes describe refusals returned as HTTP 200 with stop_reason: "refusal". Log the category and end the attempt clearly. Any retry or fallback belongs in the evaluation record and must stay within the same permissions and budget.

Run a bounded comparison, keep the working default

Use a fixed set of representative bug reports, fresh copies of the same repository state and the same acceptance checks. Give both setups the same tools and permissions. Set spend, runtime and retry limits before starting. Include an incomplete report that should produce a clarification request, so confident guessing cannot score as useful autonomy.

Record the baseline's current configuration and set 5.5's effort explicitly. Anthropic recommends a fresh effort sweep: its documentation says it tends to think more per turn at the same effort level than Opus 5. Matching the effort label alone does not establish an equal compute budget.

Inspect the resulting diffs as well as the test output. A settings-form fix that rewrites the surrounding application has failed the scope check even if the tests pass. An agent that reports a blocked test accurately gives the reviewer a different result from one that claims success without running it. Preserve that distinction in the scorecard.

Keep the proven model in production while this runs. Adopt 5.5 for the tested workload only if the accepted results justify the combined spend and review burden, with no unacceptable permission or scope failures. If the evidence is mixed, keep the current default.