Ugo ChukwuAI & Financial Systems
← Work

Revenue Recovery / Payment Operations

Failed Payment Recovery Engine

An event-driven system that treats a failed recurring payment as a classified event with its own recovery path — rather than putting every decline through the same retry schedule and hoping.

StatusValidated prototype
DomainSubscriptions & billing
RoleArchitecture & build
Core stackPython · webhooks · Postgres

01 — The problem

A declined renewal is not one problem. It is six, and they need six answers.

Insufficient funds, an expired card, a soft issuer decline, a hard decline, a fraud block and a processor outage all arrive as the same word: failed. Most billing stacks answer all six with one retry schedule.

The result is involuntary churn — subscribers who intended to keep paying and left because the system gave up, or annoyed them, at the wrong moment. It rarely shows up as a line item, which is precisely why it persists.

Cost of the gap

Recurring revenue lost to customers who never chose to leave.

Risk

Blind retries that irritate issuers and customers alike.

Blocker

No visibility into which failures are recoverable and which are not.

02 — The system

Classify first. Then choose a recovery path per class.

Every failure webhook is classified from the decline code and context, then routed to a strategy: retry on a schedule tuned to that failure type, prompt the customer to update a payment method, hold and wait for the processor, or escalate to a human before the account is touched.

Webhook ingestion

Signed, verified, idempotent; the raw event is stored before anything acts on it.

Failure classification

Decline codes mapped to a stable internal taxonomy that survives processor changes.

Configurable strategies

Retry counts, spacing and stop conditions per class — changed in config, not in code.

Customer notifications

Dunning sequences that stop the moment payment succeeds, not one email later.

Escalation rules

High-value accounts and edge cases reach a person before any suspension.

Recovery analytics

Recovery rate by failure class, attempt number and elapsed time — the feedback loop.

03 — Architecture

How a failure moves

payment_failedsubscription eventscustomer record
Event ingestionsignature check · idempotency key · raw store
Failure classifiersoft · hard · funds · expiry · fraud · outage
Strategy selectorretry · notify · hold · escalate
Scheduler & outcome ledgerstate machine · attempt history · hard stop
Dunning sequencesEscalation queueRecovery analytics
Recovery timelineattempt 3 of 4

04 — Engineering decisions

Where the judgment actually lives.

A stable internal taxonomy

Processor decline codes are mapped into our own failure classes. When a processor changes its codes — and they do — the mapping changes; the strategies and the analytics history do not.

Scheduling as a state machine

Not a cron job scanning for stale invoices. Each recovery case has an explicit state, a next action and a next action time — which makes behaviour inspectable and replayable.

Retry timing tuned to cause

Insufficient funds wants to land after payday. An expired card will never succeed on retry and needs the customer. A processor outage wants a short hold. One schedule cannot express all three.

Idempotent retry execution

Every attempt carries a key derived from invoice and attempt number. A worker that crashes mid-charge cannot produce a double charge on restart.

Hard stops and caps

Maximum attempts, maximum elapsed days and an absolute stop on fraud-flagged declines. A recovery system that will not stop is a liability.

Comms decoupled from charging

Notification failures must never block a retry, and a successful payment cancels queued messages immediately. Nothing erodes trust faster than a dunning email after the money cleared.

05 — Reliability & controls

What the system does when things go wrong.

Failure modeBehaviour
Duplicate webhookRejected on the event key; the recovery case is not advanced twice.
Customer pays mid-sequenceSuccess event closes the case, cancels scheduled retries and pulls queued notifications.
Processor outageCircuit breaker pauses attempts globally; cases are held, not consumed, and resume on recovery.
Worker dies mid-attemptAttempt is idempotent; the job is re-leased and the outcome reconciled against the processor.
Notification provider downQueued and retried independently; charging continues unaffected.
Unknown decline codeRouted to a conservative default strategy and raised for taxonomy review — never guessed aggressively.

06 — Evidence

Project statusValidated prototype

Verified results

None published. No client production recovery figures are claimed on this page.

Benchmarked results

  • Recovery rate by failure classmeasurement pending
  • Attempts required per recovered invoicemeasurement pending
  • Double-charge incidents under fault injectionmeasurement pending

These metrics have not been measured yet. Nothing is claimed for them until a reproducible run exists.

Simulated results

Generated failure mixes across decline classes, with injected duplicates, outages and mid-sequence customer payments.

Projected business value

Modelled from your renewal volume, current involuntary churn rate and average subscription value — calculated in the diagnostic, never presented as achieved.

Verified, benchmarked, simulated and projected figures are kept separate on purpose. A benchmarked number is never presented as a verified client result.

07 — Technology

PythonFastAPIPostgreSQLQueues & workersPayment APIsDocker

Technology supports the story. The architecture and the controls are the story.

Losing renewals you never meant to lose?

We can size the involuntary churn first, then decide whether a recovery system pays for itself.

Book a Discovery Call