What is regression testing?

You just fixed a bug in your order module. Two days later, a customer calls: invoicing is broken. Nobody touched the invoicing code — or so everyone assumed. This is the single most common failure mode in custom business software, and it has a name: regression.

What is regression testing, in plain terms?

Regression testing is the practice of re-checking that things which used to work still work, after you change something else in the system.

The word "regression" means a step backward — a feature that used to function correctly has quietly stopped functioning correctly. Regression testing is not about finding new bugs in new features. It's about catching the side effects your change had on parts of the system you didn't mean to touch.

A concrete example: a developer adjusts a calculation script in a FileMaker order layout so it correctly handles a new VAT rate. That script happens to be triggered from three other places too — a quote conversion, a credit note, and a monthly recap report. The VAT fix is correct. But the credit note now silently subtracts VAT twice, because it relied on the old behavior of that same script. Nobody asked "what else calls this script?" That's a regression, and regression testing is the discipline built specifically to catch it before your customer does.

Why does custom software break in places nobody touched?

Because custom business systems are webs of dependencies, not isolated screens. In a typical FileMaker or ERP solution:

• One script is called from a button, a server schedule, and an API endpoint.
• One table relationship feeds a dashboard, an export, and a printed document.
• One value list is shared between a sales form and a support ticket form.

Change the underlying logic once, and every dependent place inherits that change — for better or worse. The more the system has grown over the years (more modules, more integrations, more custom scripts bolted on by different developers), the larger and less visible this dependency web becomes. That's exactly why regression risk grows with the age and success of a system, not with its size at launch.

When should you run regression tests?

Regression testing isn't a one-time event — it belongs at several points in the lifecycle of a change:

1. Before every release, however small. A one-line fix deserves the same discipline as a big feature, because small fixes cause a disproportionate share of regressions — they're made quickly, under less scrutiny.
2. After a platform or version upgrade — for example, moving to a new FileMaker version, upgrading a plug-in, or updating an API connector. Platform vendors change default behaviors between versions more often than people expect.
3. After refactoring, even when the visible behavior is supposed to stay identical. Refactoring is precisely the kind of change that regression testing is designed to validate — "nothing looks different, so nothing should have broken."
4. On a schedule, for critical business flows (invoicing, stock updates, payroll exports) — run automated checks nightly or weekly regardless of whether anyone remembers to trigger them manually.

How do you actually do regression testing in a FileMaker or custom ERP environment?

There are three realistic approaches, and most mature systems end up combining them.

1. Manual regression checklists

A written list of core business flows — "create an order," "apply a discount," "generate an invoice," "export to accounting" — that someone walks through by hand before every release. This is the minimum viable version of regression testing. It's cheap to start but expensive to maintain as the checklist grows, and it depends entirely on human discipline (which is precisely where it tends to fail under deadline pressure).

2. Scripted test scenarios inside the platform

In FileMaker, this often means building test scripts that simulate a user's actions — creating a test record, running a calculation, checking the result against an expected value, then cleaning up. Tools like FmBetterForms are useful here not just for building richer, more consistent UI, but because a standardized form layer makes scripted testing more predictable: you're testing against a known, versioned form structure instead of a layout that quietly drifts over time as different developers tweak it.

3. AI-assisted test generation and monitoring

This is where the practice is evolving fastest. Tools such as Klai — AI integrated directly into a FileMaker workflow — can help generate test cases from existing scripts, flag scripts with unusually high fan-in (many callers, meaning many places a regression could surface), and summarize what a code change actually touches before it ships. This doesn't replace human judgment, but it closes a real gap: most regressions happen because nobody had a complete map of what depended on what. AI-assisted analysis makes that map visible in minutes instead of requiring a developer to trace it by hand.

What should a regression test actually check?

A good regression test isn't "does the app open." It checks specific, previously-correct outcomes against specific inputs. For each critical flow, define:

• The input: e.g., an order with 3 line items, one discounted, shipped to a EU country outside the home country.
• The expected output: e.g., total VAT calculated at the destination country's rate, invoice number sequential, stock quantity decremented by exactly 3.
• The trigger for re-checking: which changes should force this test to be re-run (script edits, layout changes, plug-in updates, server upgrades).

That third column is the one most teams skip — and it's the one that turns a checklist into an actual regression system, because it tells you which tests matter for which change, instead of re-running everything every time (which is slow enough that people eventually stop doing it).

How is regression testing different from other kinds of testing?

• Unit testing checks one small piece of logic in isolation — does this one calculation return the right number for these inputs.
• Integration testing checks that two connected systems (say, FileMaker and an accounting API) exchange data correctly.
• Regression testing checks that previously verified behavior — anywhere in the system, at any level — hasn't broken because of an unrelated change.

You need all three. But regression testing is the one that protects your existing customers' daily reality, which is why it deserves its own discipline rather than being treated as a subset of "testing in general."

A short checklist before your next release

• Do you have a written list of the 10–15 business flows that must never break (order entry, invoicing, stock, payroll, key exports)?
• Is there a record of which scripts, layouts, or tables each change actually touches?
• Has someone (or something automated) re-run the checklist against this specific change, not just "tested it seemed fine"?
• Do you know which changes historically caused regressions in this system, so you can weight your testing effort accordingly?
• Is there a rollback plan if a regression reaches production anyway?

FAQ

Does regression testing require expensive tools? No. A disciplined manual checklist, consistently followed, catches the majority of regressions in small-to-mid-sized systems. Automation and AI-assisted analysis become worth the investment once the system, the team, or the release frequency grows past what a human can reliably re-check by hand every time.

Can regression testing be fully automated in FileMaker? Large parts of it can — scripted test scenarios, scheduled server-side checks, and AI-assisted dependency mapping all reduce manual effort significantly. Full automation of every business rule is rarely practical or worth it; the goal is covering the flows where a silent failure would actually hurt the business.

Who should be responsible for regression testing — the developer or a separate tester? Ideally not the same person who just made the change, even in a small team. A second set of eyes (or an automated check) catches the assumptions the original developer didn't think to question, precisely because they were the one who made them.

Is regression testing only relevant for large systems? No — it's relevant the moment a system has more than one dependent flow, which is almost immediately. It becomes more critical as the system grows, but skipping it early is exactly how small systems accumulate the kind of hidden dependency web that makes later changes risky.

Regression testing is one concrete piece of a much bigger question: how do you keep custom business software reliable and easy to hand over as your team, your tools, and your business processes change? That question is explored in more depth in our guide on how to make custom business software reliable and transferable.

If your FileMaker system, ERP, or API integrations have grown complex enough that every release feels like a gamble, that's usually a sign the underlying structure — not just the testing habits — needs a second look. Loggix helps teams map those dependencies, build proper test coverage into FileMaker solutions, add AI-assisted tools like Klai to spot risk before it ships, and — where it makes sense — restructure the system so future changes are safer to make in the first place.