Deep dive · advanced topics · ~30 minInternal talk · July 2026
WIP — draft

Measuring
What Works.

Evals and traces turn vibes into numbers — then use them to route across models and harnesses, and the daily habits that compound.

David Budáč
measure → route → habits
2
Agenda

Three parts, ~30 minutes

PartTopicNote
01Measuring what works: evals & tracesmost important for every engineer · live demo
02Alternative harnesses, models & APIslive demo
03Day-to-day tips & closelive demo

All pricing and product claims are date-stamped July 2026 and sourced — see the closing Sources slide.

01

Measuring what works: evals & traces

The most important part for every engineer in this room — evals are the core skill, not a nice-to-have.

3
Evals · why vibes don't scale

"Feels better" doesn't scale

  • One person's intuition about a prompt or CLAUDE.md change doesn't survive a team of ten making changes independently.
  • An eval turns a feeling into a number two people can agree on without a meeting.
  • This is the core AI skill for an engineer in 2026 — on par with writing tests.
4
Evals · eval types

Task-completion suites

  • A fixed set of real tasks, each with a pass/fail or scored check.
  • Build it from actual past work — five to ten representative tasks beat fifty synthetic ones.
5
Evals · eval types

LLM-as-judge

  • For tasks with no clean pass/fail — writing quality, code style, clarity — score output against a written rubric using a second model.
  • Known failure mode: a judge tends to prefer its own family's outputs. Use a different model as judge than the one being evaluated.
6
Evals · eval types

Pass@k on a fixed repo task set

  • Run the same task k times; check whether at least one attempt succeeds.
  • Accounts for run-to-run variance in agentic work — a single run can mislead either direction.
  • The mirror metric: pass^k — all k attempts succeed. Use it when you need reliability, not just best-case ability.
the same task, run 5 times ✗ ✗ ✓ ✗ ✗ pass@5 ✓ passes — one success is enough answers: can it do the task at all? pass^5 ✗ fails — every run must succeed answers: can you rely on it unattended? same five runs — two different questions
7
Evals · traces

Traces: capture the full transcript

  • Every tool call, every turn, every reasoning block — the raw material for analysis.
  • Also what turns a failed run from "it didn't work" into something debuggable.
8
Evals · traces

Reading traces for wasted turns

  • Look for: a file read twice, a plan restated repeatedly, a failed tool call retried blind.
  • This is where deck 05, Cost & Context's cost anatomy and context hygiene become actionable on a real session.
t1 read auth.ts t2 write the plan t3 read auth.ts — again ⚑ re-read t4 restate the plan ⚑ repeat t5 retry failed call, same args ⚑ blind retry t6 edit + tests pass ✓ every ⚑ is tokens paid twice — half this trace is waste
9
Evals · tooling

Tools: OTel & provider dashboards

  • OTel-based observability: capture agent traces the same way you'd capture microservice traces.
  • Claude Code ships OTel support out of the box — token, cost, and tool-usage metrics via standard exporters. Zero new vendors to start.
  • Provider dashboards: most model providers now ship usage/trace views out of the box — check what your org already has before adding a new tool.
10
Evals · the minimal loop

A minimal eval loop, this week

fixed prompts
→
written rubric
→
score in a spreadsheet
→
compare variants

Deliberately unglamorous — the bar to start is nearly zero, no new tooling required.

11
Evals · live demo
Live

Run a small eval, live

  • A small fixed task set (3–5 tasks), two variants — e.g. the two CLAUDE.md variants from deck 05, or two models on the identical prompt.
  • Score each run against a written rubric, tally in a spreadsheet on screen.

Live Same method as the minimal loop — no special tooling, just discipline.

12
Evals · fallback

Fallback: recorded run

▢
Placeholder — screenshot of the scored spreadsheet comparing two variants goes here.

Use only if a live run stalls or scoring runs long. Narrate the rubric scores and the winning variant from rehearsal.

02

Alternative harnesses, models & APIs

Why not monoculture: route by cost, intelligence, and taste — the same discipline at every layer.

13
Alt. harnesses · why not monoculture

Route by cost, intelligence, taste

  • No single model or harness wins on all three axes at once.
  • Cheap and fast for bulk, mechanical work; smarter models for hard unsupervised problems; highest-taste models only where UI/copy quality is the actual bottleneck.
A cheap & fast B workhorse C frontier cheaper → C B A smarter → A B C finer taste → A B C

No column sweeps all three rows — route per task. This is the same discipline at every layer: model choice, harness choice, even open-vs-hosted choice.

14
Alt. harnesses · Cursor

Cursor as a harness

  • IDE-native, not terminal-native — strongest for engineers who live in an editor rather than a terminal.
  • Auto mode and Cursor Router (deck 05, Cost & Context) route per-request across models automatically.
  • Beats a terminal agent when routing quality and IDE integration matter more than terminal-native workflows.
15
Alt. harnesses · OpenRouter

OpenRouter: one API, many models

  • 315+ models from Anthropic, OpenAI, Google, DeepSeek, Meta, Mistral, xAI, and others — one key, one bill.
  • Pay-per-token; roughly a 5.5% fee on credit purchases rather than a heavy per-token markup on most models.
  • Caveat: failover is a feature, but it means a request can silently route to a different provider than expected.

Some trackers cite a "30-60% markup in some cases" — that figure is inconsistent across sources; the 5.5% credit-fee framing is the more defensible one.

16
Alt. harnesses · Pi

Pi: a minimal harness

  • Terminal coding-agent harness by Mario Zechner (now Earendil Inc) — system prompt under ~1,000 tokens.
  • Supports 15+ providers/hundreds of models, plus local models via Ollama for zero-cost inference.
  • Tradeoff: using Claude models through Pi bills at raw API rates — a Claude Max subscriber effectively pays twice.
17
Alt. harnesses · open models, July 2026

The frontier gap is closing — for coding

  • Several open-weight releases in 2026 close in on frontier coding-benchmark scores at a fraction of the parameter count or cost.
  • Closing, not closed — the next slides give specific, verified examples rather than the trend in the abstract.
18
Alt. harnesses · Gemma 4

Gemma 4: laptop-viable

  • Google, arrived ~March 2026. Sizes: E2B, E4B, a 12B unified multimodal (June 2026), a 26B MoE (3.8B active), a 31B dense.
  • The 12B model runs on a 16GB laptop (e.g. 16GB MacBook Air) at Q4_0 quantization — nearly matching the 26B model at under half the memory.
19
Alt. harnesses · Qwen 3.6

Qwen 3.6: dense and MoE flagships

  • Alibaba, Apache 2.0. Two April 2026 flagships: 27B dense and 35B-A3B MoE (only 3B active params/token).
  • Native 256K context, extensible to ~1M via YaRN; reportedly outperforms a 397B MoE model on agentic coding benchmarks.
  • The 35B-A3B MoE is the realistic local-laptop pick given its low active-parameter count. [exact laptop RAM requirements not independently verified]
20
Alt. harnesses · live demo
Live

Local models, small hardware

  • Gemma 4 (12B) or Qwen 3.6 (35B-A3B) running locally on the presenter's own laptop via Ollama.
  • A real coding task from this team's repo, run to completion on screen.

Live Message: absolutely viable even on modest hardware — not "as good as the frontier."

21
Alt. harnesses · fallback

Fallback: recorded run

▢
Placeholder — screenshot or recording of a local Gemma 4 / Qwen 3.6 run completing a real coding task goes here.

Use only if the live run is too slow to finish on stage. Keep the honest framing: slower than hosted, genuinely usable.

22
Alt. harnesses · Colibri + GLM-5.2

Colibri: GLM-5.2 without a GPU

  • GLM-5.2 (Z.ai): open-weight MoE, ~753B total params (~40B active), 1M context. Reportedly matches/beats GPT-5.5 on several long-horizon coding tasks. [total param count varies 744B–753B across sources]
  • Colibri: a ~2,400-line pure-C inference engine — keeps ~17B dense params in RAM (int4), streams routed experts from disk via an LRU cache.
  • Runs on ordinary hardware, no GPU, about 25GB RAM. Speed: ~0.05–1 tokens/sec — value is running it at all, not fast.
SSD all routed experts the bulk of ~753B params never all in memory per token: only the routed few RAM — ~25GB total dense core ~17B · int4 · always resident hot experts — LRU cache no GPU anywhere · ~0.05–1 tok/s — the value is that it runs at all
23
Alt. harnesses · when local makes sense

When local makes sense — and when it doesn't

Makes sense

Privacy-sensitive or air-gapped work, driving marginal cost toward zero, latency requirements loose enough to tolerate it.

Doesn't

Need frontier quality or fast turnaround; hardware/ops overhead outweighs the savings.

03

Day-to-day tips & closing

The habits that compound daily — then the recap, then the sources.

24
Day-to-day · voice mode
Live

Voice mode: dictate, don't type

  • Claude Code shipped native /voice dictation ~March 2026 — Hold (default) and Tap modes, tuned for coding vocabulary (regex, OAuth, JSON, localhost).
  • Hands-free review: dictate a prompt, keep typing while speaking, review output without touching the keyboard.

Live "Hlas" (Czech voice tooling per earlier notes) [TODO: verify tool name — could not confirm as a real product; may be a generic reference]. Demo Claude Code's own /voice command instead, which is solidly verified.

25
Day-to-day · fallback

Fallback: recorded run

▢
Placeholder — screenshot or recording of Claude Code's /voice command dictating and executing a real prompt goes here.

Use only if the venue's microphone makes live dictation unreliable.

26
Day-to-day · habits

Habits that compound daily

  • Keyboard-first — fewer context switches to the mouse.
  • Parallel sessions on unrelated tasks instead of waiting serially.
  • Background/async tasks — kick off, check back later.
  • Checkpoint with git — small commits so an agent's mid-task state is never the only copy.
27
Day-to-day · recap

Measure → route → habits

01 measure
→
02 route
→
03 habits
  • Measure first. Five real tasks and a written rubric beat any amount of prompt debate — start this week, in a spreadsheet.
  • Route by the numbers. No model or harness wins on cost, intelligence, and taste at once — pick per task, not by loyalty.
  • Habits compound. Voice, parallel sessions, background tasks, git checkpoints — small daily wins that add up.

Act on measurement (part 01) first — without evals, the routing choices in part 02 are just a more expensive set of vibes.

28
Close · sources

Sources — all claims dated July 2026

Harnesses & routing

Cursor pricing guide 2026nxcode.io OpenRouteropenrouter.ai Ollama — local inferenceollama.com

Models

Qwen 3.6github.com/QwenLM/Qwen3.6 NIST CAISI assessment of GLM-5.2nist.gov

Full source list with per-claim citations: deck05-facts.md (internal research doc). One item flagged unverified: "Hlas" as a named product.