LOCAL LLMs · REAL APPLICATIONS · MEASURED EVIDENCE
LLM Lab measures quality, performance, security and operability of local LLMs in real-world applications, Finboard first and Media Search Agent next, across models, runtimes and hardware.
Reference run · Finboard / Qwen 2.5 14B / Metal run evidence
run evidence
Each dimension starts from application-level outcomes and extends to the mechanisms, implementation and optimisation beneath them. Each has its own page.
Does the model perform the application's task reliably, and how is a change in that reliability detected?
Quality and the Eval Ledger → planned absent — a lab Performance capture carries no Quality payload
Where latency originates, how the model executes, and what the runtime and the GPU actually did.
Performance lenses → live
Whether data, runtimes, tools and model actions remain within explicit, testable boundaries.
Security boundaries → planned absent — a lab Performance capture runs no confinement probe
Whether the local LLM stack can be packaged, provisioned, observed, upgraded and recovered as a product dependency.
Operability lifecycle → planned absent — no Operability payload schema yet
The same questions are asked of real-world applications with different failure modes and of a controlled reference app. Each app's status and scenarios come from its catalogue; the live app comes first.
real-world application live
Personal Finance App · Local assistant
Finboard is a local-first portfolio look-through desktop app with a local LLM assistant over typed, read-only capabilities. It exercises tool selection, multi-turn context, narration, privacy and confinement, the runtime's lifecycle and inference performance. Finboard keeps its own eval harness and ledger; the Lab consumes what Finboard exports and links the rest.
runtime-direct recordedassistant-turn plannednarration plannedmultiturn plannedreal-world application planned
Media search app · AI search
Media Search Agent would bring a second domain — retrieval, ranking, multimodal context and orchestration over local data — once its generative integration exists. Its scenarios are defined with its first bundle; until then nothing is measured.
reference app planned
Command-line test app · Controlled test runs
LLM Probe will be a small, deterministic command-line app for exploring the model and runtime stack without a domain: change one variable, observe what happens. It will be the control, not a substitute for a real-world app, and every byte of its prompts will be authored for the Lab.
completion plannedcontext-scaling plannedtool-call plannedlong-generation plannedrun evidence
Application, model, runtime, backend, hardware, context and quantization are recorded as explicit variables on every run, each with the provenance its bundle records.
One streamed request sent by the collector directly to the runtime Finboard ships, on loopback, with a synthetic prompt and the runtime's default system prompt. It exercises the runtime the app ships; it does not exercise Finboard's prompt assembly, capabilities or narration.
Capture mode · text retained
run page · Performance view · source experiments/public/finboard/20261022T064227Z-finboard-runtime-direct/bundle.json
run evidence
Each dimension extends from application symptoms to framework internals and hardware behaviour as the investigation requires. Performance already goes the whole way down.
From user-perceived latency to the operator, runtime, backend and GPU behaviour behind it. Seven lenses over the same recorded run; the label on each row is the provenance of its evidence in the bundle.
The same scenario can be compared across models, runtimes, backends and hardware without changing the application. These are the values recorded in the current committed runs; comparisons arrive with a second run.