Skip to content
Civware

Know it works before launch,and keep knowing after

Evaluation means testing an AI system against your own examples before launch, with each result marked Passed or Needs review. Monitoring means checking the same things once people use it, and telling an owner when something changes.

Group
Run
First step
An evaluation set for one system
You decide
What is good enough to launch
Works with
Any AI system, ours or not

When it fits

Useful for any AI system, including ones you've built yourself or bought.

  1. 01

    You have an AI system and no clear way to say whether it is good enough to launch.

  2. 02

    A model or prompt change is planned, and you need to know what it will affect.

  3. 03

    An AI system is live, and nobody checks how it is doing until something goes wrong.

What we build

  1. 01An evaluation set: real examples from your work, with the answer you'd accept.
  2. 02A launch check that marks each example Passed or Needs review.
  3. 03Monitoring of quality, cost and usage once the system is live.
  4. 04Alerts to a named owner when results change.
  5. 05A rerun of the evaluation set before every model or prompt change.

How it works

The same examples are used before launch and after, so any change shows up as a clear difference.

  1. 01Your examples are marked Passed or Needs review before launch.
  2. 02Weekly checks repeat a sample once people use it.
  3. 03When a check needs review, its owner hears about it, with examples.
Tested before launch, watched afterFive of your own examples are tested before launch: four passed and one, an angry complaint, needs review. After the launch decision, weekly checks run. Weeks 1 to 3 passed and week 4 needs review, so an alert goes to the owner with the examples attached.FIG. 01TESTED BEFORE, WATCHED AFTERYOUR EXAMPLERESULTRefund over limitPassedAddress changePassedAngry complaintNeeds reviewMissing order numberPassedTwo questions in onePassedLAUNCH DECISIONAFTER LAUNCH: WEEKLY CHECKSWEEK 1PassedWEEK 2PassedWEEK 3PassedWEEK 4Needs reviewALERT TO THE OWNERweek 4 needs review, examples attached
Fig. 01 Tested before launch, watched after. A check that needs review goes to its owner with the examples attached.

Safeguards

An evaluation is only useful if it reflects real work and someone acts on it.

Examples that don't reflect real work
Examples come from your own work, including the awkward and unusual cases.
A pass mark that hides real problems
Results are shown case by case, in words, so a single serious failure isn't averaged away.
Monitoring that nobody reads
Each check has a named owner, and only changes that matter trigger an alert.
Changes that slip through
The evaluation set reruns before every model or prompt change, and the results are compared.

How to start

First engagement · Evaluation set

A launch check for one AI system

It delivers:

  • An evaluation set built from your own examples
  • A launch report, case by case, marked Passed or Needs review
  • Monitoring of quality, cost and usage, with alerts
  • A rerun process for future changes
Price
Fixed, agreed before we start
Length
Usually 2 to 4 weeks
Decision pointYou decide what counts as good enough to launch, case by case.

Questions

Can you evaluate a system we didn't build?

Yes. We need access to test it, and examples of the work it is meant to do.

Why words instead of a score?

A single score can hide one serious failure among many passes. Case-by-case results make it clear what needs attention.

How many examples do we need?

Enough to cover the common cases and the awkward ones. We'll help you choose them in the first week.

What does monitoring cost to run?

It depends on how much the system is used. We'll estimate it before you decide.

Tell us one job you'd like AI to do

We'll tell you honestly whether AI is the right fit, and what a sensible first step looks like.

One job is enough to start.

We use your details to reply to you, as described in our privacy policy.