Instagram Comment Gate

I stopped paying for the SaaS and ran it myself.

When a post does well, hundreds of people comment a keyword and every one of them expects a link in their DMs a second later. The tool that does this bills per person contacted, so it gets more expensive exactly when it works. I took an open-source clone of that one feature, deployed the whole thing myself, and ran it live against real traffic for about five dollars a month with no cap.

incident — 31 Jul

  symptom   duplicate public replies + duplicate DMs
            every send logging FAILED, on a post at 57k views

  the tell  the duplicate's button was missing an emoji
            my template always sends that emoji. so it was not mine.

  cause     a second tool still connected to the same account
            IG allows one private reply per comment. theirs landed first.
            mine came back Error 1 and logged FAILED.

  fix       disconnect it. everything went green.
  found     on the way: a retry storm and a webhook/poll race

The deployment is two services on purpose. The web app runs on Vercel and handles the dashboard, the OAuth callback and the incoming webhooks. The queue worker runs on Railway alongside Postgres and Redis, because a queue consumer has to stay awake and a serverless function sleeps between requests. Four deployment failures were worth keeping: Prisma and tsx had to move into runtime dependencies because the build was pruning dev deps, a pinned build and start command was needed to stop the platform launching the website instead of the worker, the encryption key has to be byte-identical across both services or every send fails to decrypt, and the two datastores each expose an internal and a public URL that must be matched to the right consumer or the app hangs forever.

Then it went live and broke. The interesting part is that the cause was not in the code at all, and the evidence that proved it was a missing emoji on a button. Once I knew a second tool was double-firing, disconnecting it fixed the symptom. But the investigation, run against the live database with a temporary read-only script, surfaced two genuine bugs underneath: a retry storm, where failed comment jobs retried three times and each retry sent again, and a race where the webhook and the five-minute reconciliation poll could both claim the same comment at worker concurrency five. The insert into the send log is now the atomic claim, with a lease on pending rows. Both went back to my fork as fixes.

The lesson generalises past this stack: when an automation double-fires, check that nothing else is connected to the same account doing the same job before you touch a line of code. A second tool leaves no trace inside the first one's logs.

production incident root-cause
queue workers
Postgres + Redis
concurrency and idempotency
OAuth and webhooks
multi-service deployment

two-day build, live since

What it actually took. I spent the first stretch of the incident reading my own code, because that is where you look when your thing misbehaves. It was the wrong place, and the thing that turned it around was noticing a detail that could not have come from my template. Reading the evidence for what it was, rather than for what I expected it to say, was the whole job.

The application is diwenne/openreply, not my build. Mine is the deployment and the fixes on my fork.