A Production Playbook for Healthcare API Integrations

A healthcare integration is not finished when the first payload reaches the destination.

It is finished when both teams can operate it: detect failures, explain decisions, recover safely, and change the contract without surprising each other.

I have led support engineering and integration programs where that distinction determined whether a partner launch became routine or permanently fragile.

This is the production playbook I use now.

TL;DR

• Give each integration an owner, a named outcome, and a lifecycle.
• Define the data, authorization, delivery, and failure contracts before onboarding.
• Make replay and idempotency part of the design, not incident-time patches.
• Observe integration decisions without copying protected health information into logs and metric labels.
• Build one runnable onboarding path and test it continuously.
• Turn repeated support work into validation, diagnostics, documentation, or product behavior.

1. Name the product and its owner

An integration may be implemented as endpoints, jobs, queues, and files. Operationally, it is one product with external users.

Before implementation, I write down:

• the business or clinical workflow it supports;
• the accountable product and engineering owners;
• the source and destination owners;
• the supported environments;
• the launch, maintenance, and deprecation states; and
• the escalation path when the workflow is unsafe or unavailable.

Ownership should follow the workflow, not the repository boundary.

I also separate service objectives from contractual promises. A service-level objective (SLO) guides engineering. A service-level agreement (SLA) is an external commitment with business consequences. They should not become accidental synonyms.

2. Define four contracts

Most integration escalations are contract defects disguised as payload defects.

Data contract

Specify:

• schema or FHIR profile versions;
• required, optional, and conditionally required fields;
• code systems and terminology versions;
• identifier rules;
• time zone, precision, and null behavior; and
• examples for edge cases, not only the happy path.

“FHIR R4” is not enough. Name the profiles and supported interactions.

Authorization contract

Specify:

• the supported user and machine flows;
• registration steps per environment;
• scopes and audience rules;
• token and key rotation behavior;
• revocation and offboarding; and
• the expected response when access is denied.

For SMART applications, use the server's SMART configuration discovery. Do not copy authorization endpoints from a sandbox into production configuration.

Delivery contract

Specify:

• synchronous, asynchronous, batch, or subscription behavior;
• ordering and duplicate-delivery expectations;
• idempotency-key scope and lifetime;
• timeout ownership;
• retryable versus terminal failures; and
• retention, quarantine, replay, and reconciliation rules.

“At least once” without an idempotency contract means “duplicates are someone else's incident.”

Failure contract

Specify stable error codes, safe messages, and operator action.

For FHIR APIs, use OperationOutcome where the standard calls for it. For custom APIs, return an equivalent structured envelope. A human-readable message should not be the only machine contract.

Parameter interactions belong here too. If a mode changes the meaning of a filter, publish a truth table or reject the combination.

3. Build one executable onboarding path

The best quickstart proves the smallest useful workflow against a supported environment.

It should include:

1. how to obtain credentials without placing secrets in source control;
2. a synthetic request or fixture;
3. the expected success response;
4. one expected authorization failure;
5. one validation failure; and
6. a cleanup or rollback step.

I track time to first verified transaction, not just time to first HTTP 200. Verification means the destination received the intended result and both sides can trace it.

The same example should run in continuous integration against a conformance environment. Otherwise, the quickstart is only prose with shell formatting.

4. Make delivery recoverable

Retries are safe only when the system knows whether an operation already happened.

My default delivery state machine is:

accepted -> durable -> attempted -> delivered
                         |             |
                         v             v
                      retrying      reconciled
                         |
                         v
                    quarantined -> replayed

The important properties are:

• acceptance becomes durable before asynchronous work begins;
• one logical request keeps one stable idempotency key;
• retry policy depends on the failure class;
• backoff includes jitter and a maximum attempt count;
• terminal failures enter a bounded quarantine or dead-letter queue; and
• replay is authorized, audited, and uses the original delivery identity.

Do not retry every 4xx. Do not give every 5xx infinite attempts. Do not replay a message by editing database rows during an incident.

A replay tool should support preview, bounded selection, operator identity, and a durable audit record.

5. Observe the workflow, not only the infrastructure

CPU and pod health can look perfect while every partner request fails authorization.

I want the system to answer:

• Which integration is affected?
• Which contract version processed the message?
• Where did the transaction stop?
• Was the failure transient, terminal, or policy-driven?
• Can the operator recover it without duplicating work?

Logs

Use structured events with correlation, receipt, integration, and attempt identifiers.

Keep names, birth dates, member identifiers, clinical text, and raw payloads out of routine logs. Restricted audit data needs separate access, encryption, retention, and deletion controls.

Metrics

Useful service-level indicators include:

• accepted, delivered, and terminal-failure counts;
• end-to-end delivery latency;
• authorization and validation failure rates;
• retry and quarantine volume;
• oldest pending message age; and
• replay success rate.

Use bounded integration identifiers as labels. Raw patient, request, or unconstrained client identifiers create both privacy and cardinality problems.

Traces

Distributed traces help when a transaction crosses services, but only if trace context survives queue and batch boundaries.

Instrumentation claims should match the implementation. As of this August 2026 refresh, fi-fhir exposes Prometheus metrics and correlation fields. Its serve path does not yet wire the configured OpenTelemetry exporter. I call that a known gap, not “full tracing.”

6. Treat security as an operating constraint

Security guidance needs to be implementable.

My baseline includes:

• least-privilege scopes tied to a named workflow;
• separate credentials per environment and integration;
• short exposure windows and documented rotation;
• asymmetric client authentication where the applicable profile supports it;
• transport encryption and explicit destination verification;
• no production secrets in examples; and
• an auditable break-glass and replay process.

SMART App Launch 2.2 prefers asymmetric private-key JWT authentication for clients that can authenticate. It also supports public-client cases, so “every client has a secret” is not a safe assumption.

Authorization telemetry should explain the failed policy check without exposing tokens or sensitive payloads.

7. Run support like an incident system

Good support does not depend on remembering which engineer solved the issue last time.

Each integration needs:

• severity definitions based on workflow impact;
• a first-response checklist;
• dependency and partner escalation paths;
• status-update expectations;
• a recovery procedure; and
• a post-incident owner.

The post-incident question is not “Who made the mistake?” It is “Which system change prevents the same investigation?”

That change might be:

• a validator;
• a clearer error code;
• a synthetic monitor;
• a contract test;
• a dashboard panel;
• an onboarding check; or
• a documentation correction.

This is how support work compounds instead of repeating.

8. Measure outcomes with definitions

Metrics are useful only when a team can reproduce them.

Measure	Start	Stop	Important exclusions
Time to first verified transaction	Credentials and environment are ready	Destination confirms expected result	Waiting on unsigned agreements
Delivery success rate	Durable accepted messages	Reconciled deliveries	Explicitly cancelled messages
End-to-end latency	Durable acceptance	Destination acknowledgement	Scheduled batch hold time, if reported separately
Recovery time	Impact begins	Backlog is safe and processing normally	Unreconciled duplicates

I avoid a single “integration health score” unless its formula and weighting are visible. Composite scores can hide a severe failure behind several healthy inputs.

Ticket deflection is also a weak success metric by itself. Fewer tickets can mean better docs, or it can mean partners stopped reporting problems. Pair it with verified transactions, error rates, and partner feedback.

9. Use a readiness gate

I do not call an integration production-ready until the team can show evidence for each row:

Area	Evidence
Ownership	Named product, engineering, source, and destination owners
Contract	Versioned schemas, examples, errors, and compatibility policy
Security	Approved flow, scoped credentials, rotation, and audit path
Delivery	Idempotency, retry matrix, quarantine, and tested replay
Observability	Correlated logs, bounded metrics, alerts, and runbook links
Onboarding	Continuously tested synthetic quickstart
Operations	SLOs, severity model, escalation path, and recovery exercise

This gate is intentionally boring. Boring launches are good launches.

How fi-fhir informs this playbook

My fi-fhir project is where I test these ideas in public.

The current implementation includes:

• Source Profiles for feed-specific parsing and normalization;
• validated integration contracts with correlation and idempotency fields;
• dry-run and replay workflows;
• durable delivery, bounded retry, and dead-letter handling;
• Prometheus metrics; and
• docs and browser tools for profiles, pipelines, and mappings.

It also documents unfinished work, including the tracing gap above. That is part of operational maturity: distinguish shipped behavior from planned behavior.

You can inspect the fi-fhir operations guide, Source Profile guide, and interactive tools.

Takeaways

Reliable integration programs make uncertainty explicit.

They define the contract, preserve delivery identity, expose decisions safely, and make recovery routine. They also convert partner pain into a smaller future support surface.

The endpoint is only one component. The product is the operating system around it.

Further reading

• SMART App Launch 2.2
• FHIR R4 HTTP and error handling
• Case Study: The Patient-Matching Parameter Trap
• Healthcare Interoperability: From SMART Apps to TEFCA-Scale Exchange
• fi-fhir docs hub