# ruby_llm-contract

> Handle LLM output variance for ruby_llm — retry on malformed JSON or rule violations, escalate to a smarter model, measure variance on datasets, gate CI on regressions.

- **URL**: https://www.freshcrate.ai/projects/ruby_llm-contract
- **Author**: justi
- **Category**: Testing
- **Latest version**: `v0.10.1` (2026-06-01)
- **License**: MIT
- **Source**: https://github.com/justi/ruby_llm-contract
- **Language**: Ruby
- **GitHub**: 25 stars
- **Registry**: github (`justi/ruby_llm-contract`)
- **Tags**: `ai`, `anthropic`, `cost-tracking`, `eval`, `json-schema`, `llm`, `model-comparison`, `openai`, `prompt-engineering`, `ruby`

## Description

Handle LLM output variance for ruby_llm — retry on malformed JSON or rule violations, escalate to a smarter model, measure variance on datasets, gate CI on regressions.

## Recent releases

| Version | Date | Urgency | Changes |
| --- | --- | --- | --- |
| `v0.10.1` | 2026-06-01 | High | Patch release fixing gem packaging. 0.10.0 was yanked from rubygems.org; **0.10.1 is the recommended upgrade target**. No code behavior change vs 0.10.0.  ## Fixed  - **Gem no longer ships internal tracker / dev configs.** Excluded from `spec.files`: `TODO.md`, `.rspec`, `.rubycritic.yml`, `.simplecov`, and the `.revive/` directory. Pre-0.10.1 the published gem contained these files; adopters who already extracted 0.10.0 can safely delete them.  ## Also in this release (from 0.10.0)  This is the |
| `v0.9.0` | 2026-05-31 | High | **Multimodal input — route PDFs/images/audio through your contract.**  First adopter-driven feature after the 0.8 narrative repositioning. Attachments now travel via `Step.run(input, context: { attachment: ... })`; `max_cost`, `validate`, `retry_policy escalate(...)`, and trace observability still apply.  ### Added  - **`Step.run(input, context: { attachment: file_or_io_or_url })`** — adapter forwards `chat.ask(content, with: attachment)`. RubyLLM ≥ 1.15 normalises wire format per provider (Anth |
| `v0.8.0` | 2026-04-26 | High | Narrative repositioning + small API additions. Internal architecture unchanged: no `Step::Base` refactor, no breaking changes to existing DSL.  ### Added  - **`thinking(effort:, budget:)` class macro on `Step::Base`** — mirrors `RubyLLM::Agent.thinking` signature exactly. Stored as `{ effort:, budget: }` hash; reader returns the hash; supports `:default` reset semantics; superclass inheritance like `model`/`temperature`. The convenience alias `reasoning_effort(:low)` is implemented as `thinking |
| `v0.7.1` | 2026-04-22 | High | ## Behavioral change (follow-up to v0.7.0)  Closes the known limitation called out in the v0.7.0 CHANGELOG.  **Before:** `Step::Base#run_once` wrapped the entire Runner chain in `rescue ArgumentError` to convert DSL misconfiguration (e.g. `prompt has not been set`) into `:input_error`. Side effect: any `ArgumentError` raised from adapter code during `Runner#call` — wrong arity, bad config arg, any programmer bug — was silently coerced into `:input_error` and re-tried as if the user had supplied |
| `v0.7.0` | 2026-04-21 | High | ## Breaking changes  Both changes target redundancy between `ruby_llm-contract` and upstream `ruby_llm` 1.14.x.  ### 1. `:adapter_error` removed from `DEFAULT_RETRY_ON`  New default: `[:validation_failed, :parse_error]`.  `ruby_llm`'s Faraday middleware already retries transport errors (`RateLimitError`, `ServerError`, `ServiceUnavailableError`, `OverloadedError`, timeouts) with backoff. Retrying on `:adapter_error` against the same model re-ran what transport had already retried — retry × retry |
| `v0.6.4` | 2026-04-19 | High | ## Highlights  - **`production_mode: { fallback: "..." }`** on `compare_models` / `optimize_retry_policy` — measures retry-aware, end-to-end cost per successful output. Each candidate runs with a runtime-injected `[candidate, fallback]` retry chain. - **New metrics**: `escalation_rate`, `single_shot_cost`, `effective_cost`, `single_shot_latency_ms`, `effective_latency_ms`, `latency_percentiles` — on both `Report` and `AggregatedReport` (averaged across `runs:`). - **Extended `ModelComparison#tab |

## Citation

- HTML: https://www.freshcrate.ai/projects/ruby_llm-contract
- Markdown: https://www.freshcrate.ai/projects/ruby_llm-contract.md
- Dependencies JSON: https://www.freshcrate.ai/api/projects/ruby_llm-contract/deps

_Generated by freshcrate.ai. Indexes github releases for AI-agent ecosystem packages._