r/webdev • u/Choice_Row_2025 • 2d ago
Discussion How do you handle local dev against third-party APIs you don't control?
Been going back and forth on this for a while and I don't think I've landed anywhere good.
The situation is the usual one. App talks to a handful of external APIs. Some have sandbox environments, some don't. The ones that do have sandboxes that don't quite match prod, and the gap is never documented anywhere, you just find it eventually.
Options as far as I can tell:
Hit the real sandbox in tests. Slow, rate limited, and you can't run it in CI on every PR without burning through quota. The sandbox data is also usually three fake records, so pagination bugs and anything volume related never show up until a real customer hits them.
Record and replay, VCR style cassettes. Works great the day you record them. Six months later nobody remembers how to re-record, a chunk of them are for endpoints that have since changed, and the suite is green the entire time.
Hand-written mocks. This is what most places I've worked have done and I've slowly come around to thinking it's the worst of the three. You write the mock from your reading of the docs, so the mock encodes your misunderstanding of the API, and then it passes forever. The test isn't checking that your code works against the API, it's checking that your code works against your idea of the API, and those two drift apart silently.
Contract testing, pact and friends. Makes sense when both ends are yours. When the other end is Slack, nobody is publishing a contract for you.
So what do teams actually do in practice? Specifically curious about:
- whether anyone has anything that detects when the real API has drifted away from whatever you're testing against, or if you just find out from a bug report
- whether you bother making mock data realistic in volume, or accept that pagination and perf issues are things you'll only ever see in prod
- vendors that genuinely do sandboxes well, if any exist
Not looking for "just use msw" or a list of tools, I know they're out there. I'm asking what your team actually settled on and whether you'd do it that way again.
7
u/tswaters 2d ago
Depends on what the integration is and what it needs to accomplish. Are we firing one-off analytical events to them (placed order, signed up, etc.) or is it a deeper connection that requires an actual dataset in the remote system?
I've seen cases where a bit of integration code is untested because it's incredibly niche, relatively simple and is easy enough to verify manually, because you talk to the engineers on the other side, and they either see or don't see the firehose of events. If it dies, it dies in prod and you know immediately because it'll log a failure, and you are looking for those failures. If the schema or api changes, it will have been announced, versioned & there really is no value in keeping a green unit test suite to describe something that never changes. Maybe if there's a team to support it, documentation & tests help to describe that thing but if it's 50 lines of app code and 500 lines of SQL, the app is dumb, who wants to keep it's mocks up to date, gross.
There are other cases where you do realistically need to hit a third party's sandbox during any development, manual testing & end to end tests. I'm thinking payment providers and their iframes. If you want to make a functional ecommerce site with a stubbed payment form, it's a little scary having free orders kept at bay by an if statement that we flag into when testing 🙃 like, that is precarious.
If there's no sandbox, I've had success in asking the vendor really nicely for a separate account completely just so we can dev against it without affecting prod. Sometimes there's a way to disable billing during development, then afterwards you need to pay $$ to spin up sandbox, have it populated with data from a subset of prod & do development & run tests against. It really depends on the vendor, someone like meta will laugh in your faces, if they respond at all.. other places it's part of the SLA.
The fact of the matter is, software quality is expensive and most places I've been the focus has been on building product and fixing prod. It's hard to get integration environments right. It's a FTE just to keep all the test datasets, services & sandboxes up to date and in a working state. Without a dedicated resource and any non-trivial release cadence, it's not hard to spend more than half your time futsing with test environments... and you'll still miss bugs!
0
u/Choice_Row_2025 2d ago
the vendor account thing is a good call and i've never actually tried just asking. how do you usually frame it, is it a support ticket or do you need to already have an account manager for anyone to care? meta laughing at you tracks, but i'd assume most mid-size vendors would rather give you a sandbox than have you testing against prod.
and the last paragraph is the real answer to my question, i think. it's a full time job to keep test environments honest, nobody staffs it, so everyone runs on stale fixtures and finds out in prod. that's not a tooling problem and no amount of pact is going to fix it. i was sort of hoping someone would tell me otherwise but four hours of this thread says no
2
u/tswaters 1d ago
Meta is insane, and shouldn't be used as a bar for anything. To be fair, they do have test endpoints you can dev against but everything you do is with your own Facebook account. They have no-duplicate rules and don't allow service accounts or anything like that. Anything you do on their UI needs to be logged in, can't use
@work.comemails, infuriating.But yea, hopefully I would've been invited to whatever kickoff meeting there was and I'd be asking about sandbox and how test environments work. It's literally different at every place. Some of them are wacky to work with. Sometimes it's a slack channel where you ask and they explain. That one I was thinking of was "aventri" in 2018-2022 around there? Events software where we synced contacts via api, they would just create an account for devs and you'd set up your own API keys then flip to prod when the time comes. Pretty difficult to test, but it never changed so 🤷
5
u/PrimaryFamous6139 full-stack 2d ago
Teams usually hear about drift from bug tickets. There is not a neat fix for drift detection. What helps in practice is simple. Use hand-made mocks so tests run fast. Also set up a small sandbox run that happens once a night. That way you see failures before users see them. Stripe is the only provider I trust for a sandbox like this. Most other options end up feeling uncertain. In the end, other teams accept that the edges of an integration are where problems show up. Local development will not catch all of it.
3
u/Significant_Pick8297 2d ago
A good middle ground is a nightly canary against the real API, validating responses against JSON Schema/OpenAPI and flagging only breaking changes like removed fields, type changes or new required fields. Keep fixtures for CI, but let the canary tell you when those fixtures stopped representing reality.
-6
u/Choice_Row_2025 2d ago
yeah, this is the shape of it. filtering to breaking changes only is the part i hadn't thought through, since a raw diff would fire on every new optional field and get muted within a month.
have you actually run one? curious how noisy it was in practice, and who ends up owning it. my worry is that a canary that fires on a monday gets acked, fires again tuesday, and by week three it's a slack channel nobody's in
1
u/ujay-007 front-end 1d ago
Why would it be noisy if it’s properly setup? It’s not like there are breaking API changes several times a week. Just have it send an alert if something actually changes which shouldn’t be that often
4
u/TiddoLangerak 1d ago
Another common approach is to use some decoupling mechanism like ports-and-adapters. Your core domain is then entirely unaware of the specifics of the integration, and you can write the vast majority of your tests without considering the integration at all. Of course, you still want some tests to test the integration itself (e.g. the adapter in ports-and-adapters), and as many stated here, just dumping a few payloads (or slightly adopting them) works fine. But these, I wouldn't bother making them realistic in volume at all, there's no way that you can get something remotely similar to real production without hitting real production. For prod, just make sure you have good instrumentation.
1
u/Maxion 1d ago
This is the way to do it, not just for situations when you don't have enough API keys to test anywhere but staging, but also for when the API/Vendor is changed.
If you're on a larger team, this also abstracts away the vendor/API specifics away from the app which makes the whole structure of the app better/easier.
ports-and-adaptersalso make it way easier to have better logging around the API.
3
u/thekwoka 1d ago
Unfortunately, you just gotta suck it up.
Either mock them, just use good type checking to try to at least be stable, or all the above.
But yeah, it gets obnoxious when apis have undocumented behaviors and incorrect types.
It's honestly shocking how many quick well used APIs are so shit, like returning numbers or strings arbitrarily for id type...
3
u/Khavel_dev 1d ago
Record-replay is the only approach that survived long term for us. Capture real responses from the API once, store them as fixtures, replay during dev and tests. The manual mocks always drift because nobody updates them when the upstream API changes, and the sandbox accounts either cost money or have rate limits that slow everyone down.
What made it stick was a thin local proxy that checks if a fixture exists for the request and falls back to the real API if not. New endpoints get captured automatically the first time someone hits them, so the fixture set grows without anyone thinking about it. Not glamorous, but it's the setup that doesn't rot after two months.
2
u/stack_craft 1d ago
Very true, mocks test your assumption of the API, not the API itself.
For vendor APIs without contract testing or decent sandboxes, the most pragmatic setup usually looks like:
- Hand-written Mocks for CI, Nightly Egress Smoke Tests for Reality: Use fast internal mocks/fixtures for everyday unit tests in CI so PRs stay fast. But pair that with a nightly scheduled smoke job that hits the actual third-party read-only endpoints to compare live response schemas against your recorded mock fixtures.
- Contract Adapters: Wrap the third-party client inside an Anti-Corruption Layer (ACL). If an external API silently changes pagination or response types, the failure is caught and handled at the border adapter layer rather than breaking core application logic.
- Accepting Volume Limits: For volume and pagination issues, sandbox environments are almost universally useless. The only real defence is robust defensive logging, explicit payload validation at the edge, and retry/circuit-breaker wrappers around vendor HTTP calls.
1
u/brass_crafted_soul 1d ago
hand-written mocks encode your assumptions and drift silently. I stopped trusting them entirely and only use recorded traffic validated against a schema.
i run a nightly job that hits the real sandbox and compares responses to my stored fixtures using JSON Schema validation. If the vendor changes a field type or removes a key, the build fails before customers notice. This catches drift without burning CI quota on every PR.
for volume issues I generate synthetic data matching the recorded schema structure. Pagination bugs surface in tests because the shape is guaranteed valid even when the fixture set is small
1
u/BlueScreenJunky php/laravel 1d ago
I think to test your code you need to have hard coded mocks (ideally copied from the production rather than guessed from the docs), because your CI tests are supposed to test that you have no regression in the code you're pushing. If the online API drifted it sucks but it shouldn't make your test suite fail because whoever is trying to merge their code has not broken anything, and your production is already broken anyway.
Now ideally you may want to have a separate test (not even part of you CI, maybe it can run daily ?) to check that the API you're hitting has not drifted ?
What I usually do is have alert logs whenever an API call fails, and hopefully have good enough log monitoring to catch the issue before all our clients start calling support.
-1
u/Choice_Row_2025 1d ago
"whoever is merging hasn't broken anything, and your production is already broken anyway" is a better way of putting it than anything in my post. a CI failure there is telling the wrong person about the wrong thing.
where i'd push is the error logs, because they only catch the loud half. a call that fails is the easy case. it fails, it logs, somebody goes and looks.
the one that actually hurts returns 200. a field goes nullable and you start writing nulls into your db. a new enum value falls through to your default branch. they change pagination and you process page one forever, and it looks fine, because you did get a page. nothing throws. nothing logs. monitoring is green the whole time and the data underneath is quietly wrong.
and the real cost lands when you finally notice, because you have no idea when it started. an outage hands you a timestamp, you replay from there, done. drift hands you three months of records and no way to tell which ones are bad.
that's what the daily job actually buys you, i think. not detection so much as a date
1
u/Veraxo1 10h ago
Can't understand sh1t m8 but I think you are paranoid. Just implement it against real endpoint as it is now and have good enough logging and alert system to react fast when it breaks. Stop trying to prevent something inevitable especially if you don't manage the external endpoint at all.
Also mature systems (and those written with any standards) tend to have API versioning, so once a version is published it doesnt just change over night. You should be notified before hand and usually v1 still works for some time when v2 is live just for you to be able to migrate. If it's not the case for you, then I have no idea who are you integrating with, but it is clearly not something slack-like.
1
u/kemalios 1d ago
A pragmatic middle ground: record real responses as fixtures, but keep a small script that runs on a schedule, hits the actual API or sandbox, and diffs the response shape against those fixtures. It doesn't run on every PR so quota is fine, and it catches drift before customers do. Hand-written mocks from docs are the worst of both worlds because they encode your misunderstanding and pass forever. For APIs with no sandbox, ask the vendor. A support ticket explaining you need a test account often gets you one, especially on a paid plan. The suite stays fast, and you get an early warning instead of a green test that means nothing.
1
1d ago
[removed] — view removed comment
1
u/webdev-ModTeam 1d ago
Your post/comment has been determined to be a low-effort post or comment. This includes title-only posts, easily searchable questions, vague/open-ended discussion prompts, LLM generated posts or comments, and posts/comments that do not provide enough context for meaningful replies or discussion.
1
u/LividRequirement4364 1d ago
We ended up doing a bit of both since neither option really worked for us by itself. We use hand-written mocks for local dev and the usual CI stuff because they are fast and we do not have to deal with rate limits.
The annoying part is that mocks can start lying to you once the real API changes.
So we have a really small test suite that makes safe read-only calls to the actual APIs. It only runs once a night, not on every PR. If it breaks, we know something probably changed in the API and then we update the mocks. Local testing stays fast but we still get some warning when a third party changes something.
For pagination we just throw huge fake arrays at the mocks and make sure our code does not fall over. Weird data still somehow finds a way into prod though.
And yeah, out of the ones we use, Stripe is probably the closest to having a sandbox that actually behaves like the real API.
1
u/freb97 1d ago
You can let something automatically check the API by calling it and generating typescript types from the responses for example, or zod schemas. Then just run tsc to see if everything still fits. I built something for this a while ago, but it’s probably better to use openAPI schemas directly if the API provides any. If not you can try it out, not my proudest work though: https://github.com/bussmann-io/autodisco
1
u/luodaint 1d ago
Hand-written mocks that return happy JSON are how you ship pagination bugs. What worked is a fake server for the behaviors you actually care about: 429s, partial pages, stale fields, empty lists. Keep it behind the same adapter the real client uses so prod and fake share the request shape. Record/replay one golden path per vendor as a canary, not the whole suite, and re-record when their changelog moves. Sandbox is for the occasional manual check, never CI on every PR.
1
u/muharremyurtsever 1d ago
The cassette problem gets a lot better if a cron reruns them against the real sandbox instead of waiting for someone to remember. Then drift shows up as one broken build the week the API changes, not six months of silent green.
1
u/hennell 1d ago
I've usually done hand writen mocks - send a test request, get the response and then build a mock / response generator from those responses.
If the API has a lot of frequent changes or isn't documented well I'll add tests that hit the API directly that only run manually. If there's a production issue you can run the API tests to check if anything has changed from expectations.
For critical APIs you want some contract or strict validation on the production calls though. API data is input, validate it like input. If you're coding around an api that returns an entity value as "open" or "closed" add a path and test for it if returns anything else. User gets a graceful error, you get an API log with relevant info and /or automatic ticket.
Also you should have an API docs page/folder with internal notes about that API.
How you get access tokens, how you run any live tests (or re-record replay tests!) or view error logs on your or their systems etc. And a line for any errors or weird discoveries like - "Sandbox returns custref in all caps, prod can be any case", or "added pending to open/closed status without warning". Really helps when things go wrong or you're developing a new feature to see what issues have happened in the past.
1
u/julesbuildstuff 1d ago
we landed on hand-written mocks plus a nightly contract check, and honestly the nightly check is the only reason the mocks are trustworthy. once a night a job hits the real sandbox with a handful of representative requests and diffs the response shape against what our fixtures claim. it asserts nothing about business logic, just "did a field change type, vanish, or start coming back null". that catches the silent drift you're describing without burning quota on every PR.
the thing that helped more than i expected: keep fixtures ugly. real payloads have nulls in weird places, empty arrays, three pages of results, unicode in names. when someone hand-writes a mock they write the happy shape they imagined, so pagination and volume bugs get found by a customer. we seeded ours from actual recorded sandbox responses once, then let them rot on purpose and let the diff job tell us when the rot matters.
on vendors, sandboxes are wildly uneven. the big payment/comms players are usually fine, most tiers below that are theatre. and +1 to asking for a second account. framing it as "we want to test our integration before we send you volume in prod" lands with an account manager way more often than you'd think.
would i do it this way again? mostly. the part i'd change is writing the diff job on day one instead of after the second surprise.
1
u/jaimittal91 1d ago
the nightly canary is the right shape but there's a cheaper version that catches drift same day instead of overnight, validate the response shape right where you parse it in prod, not just in a scheduled job somewhere else. if a field's type changes or a new enum value shows up, log it and fall back to a safe default instead of writing whatever came back. you're not blocking the request so nothing degrades for the user, but the "three months of bad records and no idea when it started" problem in your post becomes an alert with a timestamp on day one. costs almost nothing since you're already parsing that response anyway.
1
u/PLBjt 1d ago
What I usually land on is a thin adapter behind a local fake for day-to-day work, plus a small set of recorded fixtures for the weird responses. Sandbox creds are fine when the vendor's sandbox actually behaves like prod; a lot of them don't, so I still keep one or two recorded error payloads (rate limit, partial success, expired token) and replay those in tests. The check I run before trusting the mock is a periodic contract hit against the real sandbox in CI — if their schema drifts, that fails first and I update the fixtures. Hitting live from every laptop gets messy fast with shared test accounts and rate limits.
1
u/FilmWeasle 1d ago
There's no guarantee that a third-party APIs will integrate well with testing. In fact, it probably won't. The API should really be versioned. I've been simulating APIs with what is virtually real-world data, and then I just running Django's test suite as usual. I used to have Django's test suite launch a separate REST server which would then simulate a sequence of exchanges between two servers.
1
u/Bubbly_Orange_3502 1d ago
Put an expiry on the cassettes. A fixture older than your re-record window fails the suite instead of passing green, which turns drift into a build error rather than something you find in a bug ticket.
1
u/farzad_meow 22h ago
the approach that worked for my team was to encapsulate external api services in their own independent service. we would not talk to external service directly ever.
for our e2e test we could easily mock what we wanted.
con,,: you had to maintain the damn service and had proper monitoring to catch problems on their end.
it is not a one size fit all approach but helped dealing with annoying company that had unreliable apis
-1
48
u/[deleted] 2d ago
[deleted]