Social Automation Rescue

A client ran an automated content pipeline that generated and published daily social posts for three separate brands, using AI-generated content, on self-hosted virtualized infrastructure. On paper it worked. Every dashboard was green. Every service reported itself healthy.

In practice it had stopped doing the one thing it existed to do. Content was being generated correctly and nothing was reaching the platform. Because nothing reported an error, nobody found out for days at a time. One earlier outage ran for over two weeks before anyone noticed the brands had gone quiet. That is the situation I was brought in to end.

What the system was

Four moving parts, all self-hosted: an AI content generation service, a scheduling platform that held and released the posts, a workflow engine tying the steps together, and a message queue passing work between them. Each had its own health check and its own idea of whether it was fine. None of those signals were connected to whether a post had actually been published.

Green did not mean working

This is the whole lesson of the engagement, and it transfers to almost any production system with more than two components in it. Conventional health monitoring here was not merely unhelpful. It was actively misleading. It reported healthy while the system did no useful work, and it did so confidently enough that people stopped looking.

Two examples make the point. The message queue accepted connections and reported itself serving. It had no workers pulling from it, so messages arrived and sat there forever. Nothing about that state looks like failure from the outside. Separately, the process manager reported a backend process as online. That process had loaded and was internally wedged, answering nothing. The manager checked that the process existed. It never checked that the process worked.

A health check tells you a component has an opinion about itself. It does not tell you the business outcome happened. Once you accept that, the diagnostic approach changes completely, and so does what you build to watch the system afterward.

How I worked it

1. Monitor outcomes, not health

I stopped trusting internal health signals entirely and verified the only thing that mattered: did a real post reach the platform today, with a live public URL that I could open. That became the single source of truth. Every diagnostic decision after that flowed backward from the end result rather than from any intermediate status a component reported about itself.

2. Systematic root-cause isolation

Each incident was worked by halving the problem space with one diagnostic question, then following the evidence rather than a theory. Several plausible root causes were ruled out by checking a single real number, a process’s memory footprint, a timestamp, a connection count, instead of trusting a dashboard that had already proven unreliable. The diagnostic tooling produced false positives of its own, and those were discarded rather than acted on. Chasing a false signal during a live outage costs hours you do not have.

3. Fix the cause, not the symptom

Where a failure recurred, the work targeted the condition underneath it. Automatic credit top-ups were enabled so a billing failure could not halt generation again. An under-provisioned container was right-sized so memory pressure could not cascade. A fragile data-parsing step was hardened so malformed AI output could not crash a run. Restarting the service would have cleared each symptom for a day. None would have stayed cleared.

Six failures wearing the same mask

Six distinct failure modes were found and resolved. Every one produced the identical visible symptom, which was no symptom at all: content generated, nothing published, everything green. They were structurally unrelated problems presenting as the same silence, which is exactly why outages ran for days.

• A message-queue worker with no active pollers. It came back after a host reboot serving and healthy, and processing nothing.
• A billing failure in the AI content service. Generation silently halted. Closed permanently with automatic recharge.
• A fragile response-parsing step. Malformed AI output crashed one brand’s run outright.
• A scheduling platform outage. The backend failed to serve behind its gateway.
• A backend process that never bound its port. It loaded fully, wedged during startup, and the process manager reported it online throughout.
• Container memory exhaustion. An under-provisioned service met restart attempts that stacked duplicate processes and drove the host into thrash.

Where it ended up

All three brands were restored to reliable daily publishing, verified end to end against live post URLs rather than internal status. The recurring failure modes were eliminated at the root instead of patched over.

I then built a publish-verification monitor designed around the correct signal. It confirms an actual post was published, asserts every post type rather than sampling one, and alerts once per incident instead of flooding the inbox on every check, because an alert channel people mute is worse than none.

The strategy is layered on purpose. Prevent the known root causes, and guarantee that any future unforeseen failure surfaces as an immediate alert rather than a multi-day silence. The operational principle I handed over was this: a system that can fail in a new way every week cannot be fully prevented from breaking, but it can be made to reliably announce when it does.

Measure	Result
Brands automated	3
Distinct failure modes resolved	6
Days of silent outage eliminated	16, reduced to 0

What this work involved

Linux systems administration across containerized and virtualized infrastructure. Production incident response, with root-cause analysis spanning several interacting services rather than sitting inside any one of them. Message-queue and database diagnostics, monitoring and alerting design, and reliability engineering aimed at the failure surface rather than the last thing that broke. Much of it was pragmatic prioritization under live-outage pressure: deciding which of six broken things to chase first while the brands were still dark.

For other IT providers

I take this kind of work on a subcontract basis, white-label, under your name. I do not approach your client and do not need to be visible to them at any point. If you have an account with something failing intermittently that nobody has been able to pin down, that is the specific problem I am useful for. You can get in touch here.