Keep one speaker sharp, blur the crowd behind him: masks drift the moment the camera moves. CPU-only VPS. What is the right pipeline?

[u/Exotic_Accountant565]

Source: a ~29 minute C-SPAN rally recording. One speaker at a podium in the foreground, a dense crowd of roughly 40 people behind him. The camera is not locked, it drifts, zooms slightly and reframes. I want everyone except the speaker blurred (removal would be even better), and I want it scriptable from a terminal so I can batch it, not hand-rotoscoped in a GUI.

What I have tried and how it fails. ffmpeg boxblur over a static or keyframed crop region is fine on a still frame, but the moment the camera moves the region is in the wrong place: background faces pop out, and the speaker's own face gets caught in the blur. Generic per-frame face-blur tools (deface-style detect-then-blur) flicker frame to frame with no identity persisting, so the "keep this one sharp" exception does not survive a pan. Remotion overlays hit the same wall one layer up: I still need per-frame mask geometry and I do not have it.

What I think I need but have not wired up: a real video object segmentation pass that propagates masks temporally (SAM 2 with a click on the speaker to build a keep-mask? a person detector plus ByteTrack / BoT-SORT to hold track IDs across the pan?), exported as a per-frame alpha sequence that I composite in ffmpeg with alphamerge against a blurred copy, or feed into ProPainter for actual removal.

Hardware, which I suspect is the real constraint here: a Hetzner VPS with 32 GB RAM, an i5 CPU, and no GPU. Everything has to run on CPU. That is what makes me unsure whether SAM 2 over 29 minutes is remotely practical or whether I should be reaching for something much lighter.

Questions. Is SAM 2 video propagation the right tool, and is CPU-only inference on a clip this long realistic (hours? days?) or a non-starter, meaning I should rent a GPU box for the mask pass and do the composite on the VPS? Would a person detector plus ByteTrack with an inverted mask (blur everything except track ID N) stay stable over 29 minutes on CPU, or does re-ID break too often after occlusions? Any practical guidance on mask dilation and temporal smoothing amounts that stop the edges crawling, and on the ffmpeg side of compositing a PNG alpha sequence back over the source? And is there something better in 2026 that I am missing for "keep one subject sharp, anonymize everyone else" as a CLI tool?

Local and open source strongly preferred. Happy to write the glue code.