aillm-evaluationfine-tuninglorafilemakerlocal-llmmodel-testingai-governance

How Do You Know Whether Fine-Tuning an AI Model Actually Improved It?

Shubham·

Training an LLM is not proof of better answers. Learn how to compare a base model and a fine-tuned model on unseen cases, measure business impact, and avoid common FileMaker data traps.

A model finishes fine-tuning. The training dashboard looks good. It now uses your company's terminology and responds with impressive confidence. So is it actually better than the model you started with?

You cannot answer that from the training run alone. You need to test the original model and the tuned version on the same unseen business cases, measure the errors that matter, and check that the improvement survives outside your training examples.

In a FileMaker or ERP workflow, an impressive-sounding model that misroutes an urgent invoice dispute is not an improvement.

What should fine-tuning improve in the first place?

Fine-tuning adapts a pretrained model using examples of the behavior you want. With methods such as LoRA (Low-Rank Adaptation), only a relatively small set of additional parameters needs training, reducing resource requirements compared with changing all the base-model weights.

Useful targets include consistent ticket classification, specialized terminology, extraction into a defined structure, or a particular writing style. But fine-tuning is generally the wrong repair for missing current facts. If an assistant does not know today's stock level, connect it to the live system. If it cannot find a policy clause, examine document retrieval first.

Before training anything, write down one sentence: "The tuned model will be better at [specific task], measured by [specific test]." If you cannot finish that sentence, you do not yet have a training objective.

How do you compare a base model with a tuned model fairly?

1. Keep a real baseline

Start with the original model, a clear prompt, and a few good examples. Save its answers to a fixed evaluation set. Record the model version, prompt version, input format, sampling settings, response time, and cost. Then run the fine-tuned candidate on the same questions and grading rules.

Give both candidates comparable prompts; otherwise the result does not isolate training's value.

2. Separate training, validation, and test examples

  • Training set: examples the model learns from.
  • Validation set: examples used while choosing training settings or checkpoints.
  • Test set: a separate, held-out collection used for the final decision.

The test set must not be repeatedly used to redesign the model; otherwise it gradually becomes another development set. Remove duplicates and near-duplicates. In business data, also consider splitting by customer, conversation thread, document, or time period. Ten emails from the same conversation should not be scattered across training and testing and then counted as ten independent successes.

Google's guidance warns that test examples must be representative and distinct from training. Familiar records can produce misleadingly good scores.

3. Measure the failure that costs the business money

A single 'accuracy' number rarely tells the whole story.

Task Useful measure Important hidden failure
Classifying tickets Precision and recall for each category Urgent complaints mislabeled as routine
Extracting order details Field-level correctness and valid JSON rate Wrong order number or delivery date
Drafting replies Human review score and factual grounding Fluent but unsupported promises
Answering with company documents Correct evidence and appropriate refusal Invented policy or stale source
Local inference Response time, memory use, and cost per request Improvement too slow for normal work

Define a release threshold before testing. Minor internal categorization errors differ from mistakes involving credit limits, legal terms, or payments, which may demand human review.

What would this look like in FileMaker?

Imagine an existing FileMaker support system. Incoming emails need to be categorized as Billing, Delivery, Product, Technical, or Other. Staff have already corrected hundreds of category assignments, which makes these records potentially useful training examples.

A sensible experiment would be:

  1. Export approved, anonymized ticket text and the reviewed correct category.
  2. Reserve a realistic held-out test set, including difficult and uncommon categories.
  3. Run the original model with a strong classification prompt and store its proposed category.
  4. Fine-tune a candidate on the remaining reviewed examples and rerun the held-out cases.
  5. Compare overall accuracy and each category's errors, review time, latency, and cost.
  6. Keep the candidate out of production until a reviewer approves the actual improvement.

Suppose, purely as an illustration, the original model correctly classifies 78 of 100 held-out tickets and the tuned model gets 88. That looks promising. But if the tuned version now misroutes four urgent billing disputes that the original handled correctly, it may still be the wrong model to deploy. Inspect the error pattern, not only the headline percentage.

There is one FileMaker-specific trap worth noticing. Claris documents that its Fine-Tune Model script step can take training data from a table or a JSONL file. In the table mode, the completion field is the desired answer and all other fields in the selected table occurrence become part of the user prompt. If those records include names, phone numbers, pricing, or confidential notes, you could send much more information than intended. The documentation recommends generating JSONL when you need control over exactly which fields are included.

Claris also documents supported providers for that step as OpenAI or Claris AI Model Server on supported hardware. A custom OpenAI-compatible inference endpoint does not mean that every local model is supported by this FileMaker training step. You can still experiment with open-weight models and LoRA using a separate training stack; just keep training compatibility distinct from inference compatibility.

When is the 'better' model actually worse?

Watch for four failure patterns:

Memorization: The model performs beautifully on training examples but fails when the wording, customer, or context changes. Usually the test data were too similar to training, or the model overfit.

Regression: One category improves while a different category deteriorates. Report per-category results and retain a general-purpose test set, not only examples of the specialty you trained.

False confidence: The tuned model gives a polished answer when it should say it lacks enough evidence. Track appropriate refusals, unsupported facts, and uncertainty rather than rewarding assertiveness.

Operational cost: A modest quality gain may not justify additional model memory, inference latency, retraining work, security review, or support effort. The right comparison is business value after operating cost, not just a benchmark score.

A practical release checklist

  • We defined the task and success criteria before training.
  • The base model with a strong prompt is a recorded baseline.
  • Training, validation, and final test cases are separated without duplicates or conversation leakage.
  • The final test set includes realistic rare and high-risk cases.
  • We reviewed errors by category, not just overall accuracy.
  • Sensitive fields were removed or explicitly approved before export.
  • Performance, cost, permissions, and refusal behavior were tested.
  • A rollback path exists, and a human approves production promotion.

Frequently asked questions

Does lower training loss mean the model is better?

No. Lower loss indicates better fit to the training objective. It does not demonstrate better results on new customer requests. That requires separate evaluation.

Is LoRA the same thing as a better model?

No. LoRA is a more parameter-efficient method of adaptation. It makes some training scenarios less resource-intensive, but quality depends on the base model, examples, configuration, and evaluation.

How many test examples do we need?

There is no universal number. You need enough independent, representative examples to detect the errors relevant to your business. Small pilots are useful for finding obvious problems, but a handful of successes cannot establish production reliability.

Can an LLM grade its own output?

An LLM can help flag structure or compare candidate answers, but model grading should be calibrated against reviewed human examples. For exact fields such as IDs, amounts, category labels, and JSON structure, deterministic checks are often more reliable.

The actual answer

A fine-tuned model is better only when it produces fewer meaningful errors on new, representative business cases without creating unacceptable regressions, security problems, or operational costs. A training job is an experiment, not a promotion decision.

For FileMaker developers, this is familiar territory. We already test scripts against expected records and protect production workflows. A trained model deserves the same discipline: versioned inputs, repeatable checks, clear ownership, and a safe way back. Start small, measure the failures that matter, and only then let the new model handle real work.

Related Loggix reading

Sources