Status: in build
Active R&D, code on the bench, nothing shipped. What follows is the working design and the constraints that shaped it — published because the constraints are more instructive than the product.
The premise
I want an agent that watches my working day — what's on screen, what I'm actually doing versus what I planned — and reports to me over Telegram. Useful idea, terrible privacy proposition if any frame of my screen leaves the machine. So the defining requirement is brutal and simple: everything runs locally, on hardware I already own.
That hardware is a laptop with a Ryzen 7 4800H, 16 GB of RAM, and a GTX 16-series GPU with 4 GB of VRAM. No tensor cores. This is not the machine anyone writes AI tutorials for, which is exactly why the exercise is worth doing. Most people's machines look like this one.
What 4 GB actually buys you
The headline lesson from the research phase: nearly every "run a VLM locally" guide assumes 8 GB and up. At 4 GB the 7B and 11B vision models simply do not fit, whatever the blog posts imply. What fits is the 2B class — Qwen3-VL-2B or MiniCPM-V — quantized to Q4, with roughly 30 to 45 tokens per second of generation and a few seconds of image-encode latency per screenshot. Since screenshots arrive every 30 to 60 seconds rather than in real time, that latency is acceptable.
Two hardware traps cost me real time. The GTX 16-series reports itself to CUDA as compute capability 7.5, the same as cards with tensor cores, so inference engines happily assume fast FP16 — which this silicon does not have. The fix is forcing integer-quantized matmul kernels at build time and refusing FP16 weights entirely. And the vision projector, a separate ~0.9 GB component that must stay at F16, gets evicted to system RAM so the weights and KV cache keep the VRAM. The VLM owns the GPU; OCR, entity recognition, and embeddings all run on the CPU's 8 cores instead.
Privacy as a pipeline, not a promise
The part of the design I care most about is what happens before any screenshot reaches the model:
- —A blocklist skips capture entirely for password managers and banking apps. Those windows are never captured, not captured-then-deleted.
- —OCR runs over every captured frame, and the extracted text goes through two detectors in series: regex for the structured stuff (emails, card numbers, phone numbers) and a small local NER model for names and addresses.
- —Anything flagged is blacked out at the pixel level before the frame moves on. The VLM sees a redacted image or nothing.
Honesty requires saying this cannot guarantee zero leakage — OCR misses things, and NER has false negatives. Layered redaction reduces risk; the blocklist eliminates it for the highest-stakes apps. That's the truthful shape of every privacy system I know of.
The agent proposes, the human disposes
The agent improves its own configuration over time — new blocklist entries, changed nudge thresholds — but only through a proposal loop. It sends a card to my Telegram with approve, reject, and modify buttons, and only an explicit approval mutates the live config. Every proposal and decision lands in an audit table.
This mirrors the human-in-the-loop gate in the QG-Synthesis Engine, scaled down to a personal tool: the agent can want things, but it can't have them without a signature.
Build order
Five phases, each independently useful: a dumb capture-to-Telegram loop first (value on day one), then event-driven capture off the Windows foreground hook, then the redaction pipeline, then structured memory in SQLite with vector recall over the past month, and finally the proposal loop. The system earns its complexity one phase at a time — the same rule that governs everything I ship.