Building long-running local AI agents

A conversation is not a job database

A video-rendering task has several lifetimes: the conversation that planned it, the driver that submitted it, the worker generating frames, and the final assembly. Treating those as one process makes recovery fragile.

In my H3 workflows, a submission driver can finish or fail after jobs are queued. The queue and output artifacts can still exist, letting assembly or download resume separately. That separation is useful, but not magical: I have also had orphaned jobs stuck in a loading state after a worker disappeared. Recovery needed reconciliation, not another optimistic status message.

The practical contract is a stable task identifier, the inputs, the latest observed state, an output location, and a record of what has already happened. A process restart is not proof that the intended work has resumed correctly.

Different agents need different boundaries

I am tailoring agents to video rendering, personal assistance, media management, and coding. I do not want the same permissions or success criteria for all four. The design boundary I use is:

• Video rendering: prepare and submit a scoped job, inspect the output, and retry within an agreed budget.
• Personal assistance: distinguish preparing a recommendation from sending a message or making a commitment.
• Media management: separate discovery and organization from irreversible deletion or overwriting a master.
• Coding: verify the produced artifact with tests, and distinguish an implementation failure from a broken harness.

A retry is a hypothesis, not a habit

The H3 operating notes use a small fresh-seed retry budget—typically two or three candidates. Repeating the same failure is a reason to inspect the prompt or recipe, not to keep spending GPU time indefinitely.

This changes how I think about agent progress. “Still running” is not necessarily useful progress. The relevant question is whether the next action tests a new explanation for the failure. If it does not, the agent should preserve its evidence and ask for a decision.

Local hardware makes scheduling part of the product

Two large GPUs give me room to experiment, but a rendering job and an inference server can still compete for the same memory. Some render workflows park a local language-model engine. An agent that needs that engine to supervise the render can lose its own ability to act.

That is why I treat ownership and recovery paths as first-class concerns: know what holds the GPU, avoid interrupting unrelated work, and retain a way to inspect the queue when the preferred model is unavailable. Raw tokens per second tell me very little about whether the whole workflow will finish.

The wiki is the memory that earns its place

My wikis separate executable recipes, experimental findings, and failure modes. A useful entry records what worked, when it was tested, what failed, and how to recognize the same failure again. Planned work is labeled as planned; an old working command is not automatically a current one.

The loop is artifact → evaluation → human inspection → a small, traceable rule → the next preflight. I have learned to update the records after a repair, too: a fixed service with a stale “broken” note creates a phantom blocker for the next agent.

I use AI to help turn observations into these records, but promoting a lesson into a durable recipe still needs judgment. The wiki is not an autonomous self-improvement claim. It is a way to stop paying for the same mistake twice.

What is working, and what is still uneven

H3 has queued jobs, output artifacts, and a post-render evaluation sidecar. Targeted reroll orchestration exists, but it is not uniformly integrated into every driver. In the LTX story wrapper, script validation blocks errors before rendering while visual postflight is advisory and can be skipped if its model service is unavailable.

That distinction matters more than calling the system “autonomous.” My goal is reliable, inspectable work with explicit human intervention—not a promise that every task runs unattended.