teardown
Why we stopped trusting third-party sandbox environments
A short write-up of a failure mode we have hit more than once, and the testing approach we adopted because of it.
The engagement
The situation
Sandbox environments offered by third-party APIs routinely differ from production in ways their documentation does not mention — rate limits, pagination behaviour, field nullability and webhook ordering are the usual suspects. Code that passes every sandbox test can still fail on its first real request.
What we did
We now treat the sandbox as a contract check rather than a confidence check, and schedule a reversible low-value transaction against production before any integration goes live. Where the provider supports it, we also replay captured production payloads against our own parser in CI.
The outcome
This is a practice rather than a project. It has caught several differences before launch that the sandbox did not surface, most commonly around pagination and null fields.
Detail
The failure mode
A sandbox is a simulation, and simulations are maintained with less care than the thing they simulate. The differences that bite are rarely dramatic:
- Pagination. The sandbox has forty records, so nobody discovers the cursor is not stable under concurrent writes.
- Nullability. Fields that are always populated in test data are optional in production, and your parser has never seen a null.
- Rate limits. Often absent or far more generous in the sandbox, so the backoff path has never executed.
- Webhook ordering. Delivered neatly in sequence in test; out of order and occasionally twice in production.
What we do instead
Sandbox tests verify the shape of the contract. Before launch we run one real, reversible, low-value transaction against production and inspect the actual payload against our parser. Where providers allow payload capture, we keep those recordings and replay them in CI, so production reality becomes a regression test rather than a one-off check.
Related
Start a project like this.
If something here resembles your situation, describe yours. We will tell you how close it actually is.
- 01
Send a short brief
Five minutes. What you are building, or what is broken.
- 02
We reply within one business day
From an engineer, with a real question or observation.
- 03
A 30-minute technical call
Free, no deck, no obligation.