How to prepare an incident response procedure

Your API integration stops responding at 7:45 on a Tuesday morning. Orders from your webshop are no longer reaching your ERP. Your FileMaker system — the one your warehouse, sales team, and finance department all depend on — is throwing errors nobody has seen before. Within twenty minutes, five people are sending panicked messages, three are trying to fix it simultaneously, and nobody is sure who is actually in charge.

This article gives you a practical, step-by-step framework to build an incident response procedure from scratch — one that defines exactly who does what, in what order, the moment something breaks.

What exactly is an incident response procedure — and why isn't a backup enough?

An incident response procedure (IRP) is a documented, pre-agreed set of actions your organization takes when a critical system fails. It is not the same as a disaster recovery plan (which focuses on restoring data), a business continuity plan (which keeps operations alive during disruption), or an SLA with a vendor (which sets response time expectations). An IRP sits one layer above all of those: it is the human choreography that activates everything else.

Having a backup is necessary but not sufficient. If nobody knows who is authorized to trigger the restore, which backup is current, or how long the restore will take — a backup is just a file sitting on a server. The procedure is what turns that file into a recovered system.

What counts as an incident worth a formal response?

Not every bug or slowdown warrants a full incident response. Before writing your procedure, agree on a classification system. A practical starting point:

• P1 — Critical: Business operations are fully stopped. Example: the API connector between your FileMaker system and your logistics partner goes silent; no shipments can be confirmed. Revenue impact begins immediately.
• P2 — High: A major function is degraded but a workaround exists. Example: the automated invoice generation in FileMaker fails, but staff can create invoices manually — slowly.
• P3 — Medium: A non-critical workflow is broken. Example: a reporting dashboard is not refreshing, but underlying data is intact.
• P4 — Low: A minor inconvenience with no immediate business impact.

Your incident response procedure only activates automatically for P1 and P2. P3 and P4 go through normal support channels.

Who needs to be in your incident response team?

The single biggest failure mode in incident response is unclear ownership. When everyone feels responsible, no one acts. Your IRP must name specific people — not job titles, but actual names — for each role:

Incident Commander (IC): The one person with authority to make decisions and coordinate the response. This is not necessarily the most technical person in the room. It is the person who can keep the response organized under pressure, communicate with leadership, and prevent five people from all trying to fix the same thing at once.

Technical Lead: The person who diagnoses and implements the fix. For a FileMaker failure, this is typically your in-house developer or your external FileMaker partner. For an API failure, this may be your integration specialist or the vendor's support team.

Business Owner: The department head or manager most affected by the outage. Their job is to assess business impact in real time, authorize workarounds, and communicate with customers or partners if needed.

Communications Lead: Someone responsible for internal and external messaging. During a P1 incident, silence is worse than an imperfect update. Customers, suppliers, and colleagues all need a single source of truth.

Scribe: Someone who logs everything — timestamps, actions taken, decisions made, who was contacted. This log is critical for your post-incident review.

For smaller organizations, one person may wear two hats. That is fine. What is not fine is leaving any of these roles unfilled.

How do you structure the response itself — step by step?

A well-designed incident response follows five phases. Each phase has a clear exit condition — a moment when you move to the next phase, rather than cycling back endlessly.

Phase 1: Detection and triage (target: under 15 minutes)

1. The incident is reported — by a monitoring alert, a staff member, or a customer complaint.
2. The first responder confirms it is a real failure, not a local issue (one user's browser, one machine, one network connection).
3. The first responder classifies the severity (P1–P4) using the agreed criteria.
4. If P1 or P2: the Incident Commander is notified immediately. The IRP activates.

Exit condition: IC acknowledges and assumes command.

Phase 2: Containment (target: under 30 minutes)

Containment means stopping the damage from spreading before you start fixing anything. For a FileMaker system failure, this might mean:

• Taking the system offline so no further corrupt data is written.
• Switching the API connector to a maintenance mode so external partners receive a clear error rather than silent failures.
• Notifying affected departments so they stop trying to use the broken system and switch to a manual fallback.

Do not skip this phase. Attempting to fix a live, broken system while users are still hammering it is one of the most common ways a two-hour incident becomes a two-day one.

Exit condition: The failure is isolated. No new damage is occurring.

Phase 3: Diagnosis (target: time-boxed, 30–60 minutes)

The Technical Lead investigates root cause. Useful starting points for a FileMaker or API failure:

• Check FileMaker Server logs (Event.log, Access.log) for the first error timestamp.
• Check the API connector's transaction log — did the last successful call happen before or after a deployment, a credential rotation, or a server restart?
• Check infrastructure: did a server certificate expire? Did a firewall rule change? Did a scheduled task fail silently?
• Isolate whether the failure is on your side, the vendor's side, or the integration layer between them.

Time-box this phase. If the Technical Lead has not identified the root cause within 60 minutes, escalate — either to a senior specialist or to your external partner. Diagnosis paralysis is real, and it costs hours.

Exit condition: Root cause is identified, or the decision to escalate is made.

Phase 4: Recovery

Recovery actions depend entirely on the root cause, but your IRP should pre-document the most likely scenarios:

• FileMaker database corruption: restore from last verified backup, replay transaction log if available, verify data integrity before bringing users back.
• API authentication failure (expired token, rotated key): reissue credentials, update connector configuration, run a test call, confirm successful data flow before reopening to production traffic.
• Server or hosting failure: fail over to secondary environment if available, or restore to a clean host from your backup, then re-test all integrations before reopening.
• Third-party API outage (the failure is on the vendor's side, not yours): activate your manual workaround, communicate clearly to affected teams, monitor the vendor's status page, and set a review interval (every 30 minutes, for example).

For each scenario, your IRP should include: who performs the action, what the action is step by step, and how to verify that recovery is successful before declaring the incident resolved.

Exit condition: The system is restored, tested, and verified. Users are notified they can resume work.

Phase 5: Post-incident review (within 48 hours)

This is the phase most organizations skip — and the reason the same incident happens again six months later.

Within 48 hours of resolution, hold a blameless post-mortem. The Scribe's log is your input. Work through:

• What exactly failed, and why?
• What in the detection or response could have been faster?
• Were roles clear? Did anyone not know what to do?
• What monitoring or alert would have caught this earlier?
• What changes to the system, the procedure, or the training are needed?

Document the answers and assign owners with deadlines. A post-incident review with no action items is a wasted meeting.

What workarounds should you pre-define?

The question "what do we do manually if the system is down?" should never be answered for the first time during an incident. For every critical workflow your system handles, define a fallback before anything breaks:

• If the FileMaker-to-ERP API connector fails: orders are entered into a shared spreadsheet by sales staff, imported into the ERP manually by finance at end of day.
• If the FileMaker scheduling module goes down: the operations team falls back to the printed weekly schedule pinned in the warehouse.
• If the customer portal loses its backend connection: the support team takes orders by phone and logs them in a temporary form.

Workarounds do not need to be elegant. They need to exist, be written down, and be known to the people who will use them.

What should your incident response procedure document actually contain?

Here is a practical checklist for what to include:

• Incident severity classification table (P1–P4 with examples from your own systems)
• Named roles: Incident Commander, Technical Lead, Business Owner, Communications Lead, Scribe — with backup names for each
• Contact list: phone numbers, not just email — for internal team, external partners, and critical vendors
• Activation trigger: exactly what event or report triggers the IRP
• Phase-by-phase response steps with exit conditions
• Pre-documented recovery playbooks for your top 3–5 failure scenarios
• Pre-defined manual workarounds for each critical workflow
• Communication templates: what to send to staff, customers, and partners during a P1 — written in advance, not improvised
• Post-incident review template
• Review schedule: the IRP itself should be reviewed and tested at least once per year, or after any significant change to your systems

How do you test a procedure before you actually need it?

An untested procedure is a guess dressed up as a plan. Test yours in one of three ways:

1. Tabletop exercise: Gather your incident response team, present a hypothetical scenario ("Your FileMaker Server is unreachable at 8 AM on a Monday — go"), and walk through the procedure verbally. No systems are touched. The goal is to find gaps in roles and decision-making.
2. Partial simulation: Deliberately trigger a non-critical failure in a test or staging environment and run the response procedure for real — diagnosis, recovery, communication, and all.
3. Red team drill: Ask someone outside the core team (an internal IT colleague or an external partner) to create a surprise failure scenario during a low-risk window, and measure how long it takes your team to detect, classify, and begin response.

Even a 90-minute tabletop exercise once a year will surface problems that no amount of documentation review will catch.

Frequently asked questions

How long should our incident response procedure document be? Long enough to be complete, short enough to be used under pressure. A practical IRP for a mid-sized business is typically 4–8 pages plus appendices (contact lists, playbooks, templates). If it runs to 30 pages, nobody will read it during a live incident.

Do we need a separate procedure for every system? Not necessarily. A single IRP framework can cover multiple systems, with system-specific playbooks as appendices. The roles, phases, and communication approach stay the same; the technical recovery steps differ per system.

What if our critical system is maintained by an external vendor or partner? Your IRP must still exist internally. You need to know: who calls the vendor, what number, what SLA applies, what you do while waiting, and at what point you escalate beyond the first support tier. Outsourcing maintenance does not outsource incident ownership.

How often should we update the procedure? At minimum: after every P1 or P2 incident, after any major change to your systems or integrations, and on a scheduled annual review. Treat it like a living document, not a one-time project.

What is the single most important thing to get right? Clear, named ownership of the Incident Commander role. Everything else in incident response depends on one person having unambiguous authority to coordinate the response. Without that, even a perfect written procedure collapses under the pressure of a real incident.

---

Building and maintaining a solid incident response procedure is ultimately a governance question as much as a technical one: who is responsible, what are they responsible for, and how does the organization prove it can recover when something goes wrong. If your business runs on a custom FileMaker environment, a network of API integrations, or a combination of both, the failure scenarios are specific enough that generic templates rarely cover them fully. Loggix works with organizations to map those specific failure points, design recovery playbooks that match the actual architecture, and where needed, build in monitoring, redundancy, and AI-assisted alerting that shortens the gap between something breaking and someone knowing about it — before a customer does.