Build a local AI feature
Test the local AI feature
Test deterministic behavior directly and exercise the real model with a small evaluation instead of making every test depend on local inference.
Split verification into two layers. Mixing them creates flaky CI and false confidence.
Unit tests should cover the deterministic parts:
- prompt construction
- schema and output validation
- timeout and error mapping
- fallback sentence
- provider adapter contract
These tests run without Ollama. They should be fast and repeatable. A pull request should not require a GPU on the runner.
Integration checks exercise the installed model. Keep a small set of representative activity records and run them against the exact model tag used by the application.
Do not assert one exact generated sentence. Check the contract: valid shape, supported facts, maximum length, acceptable latency, and no invented activity. Nondeterministic wording is fine when the facts stay grounded.
Record failures beside the input and model version. A later model upgrade must pass the same cases before becoming the default.
The model is nondeterministic. Your acceptance boundary should not be.
I run integration checks manually or on a scheduled job on a known machine, not on every push. The unit layer catches regressions in code you control. The integration layer catches regressions in model behavior you only partially control.
When an integration case fails, save the raw JSON response. Model issues often look like application bugs until you read the actual output.
Name the layers in your test folder so newcomers do not mix them:
tests/unit/summarizer.test.js
scripts/eval/run-ollama-eval.mjsCI runs the first. Your laptop runs the second when Ollama is up.
Try this on your own project: add one unit test for fallbackSummary and one script that loops five evaluation inputs through Ollama when it is available. Document which command is safe in CI versus local-only.
Lesson completed