A private AI assistant, tested end to end
Confidential Research is a desktop app that runs an open model on your own computer, encrypts the chats you save and, when you allow it, searches the web through Tor or your own VPN. Before calling it ready I tested it the way a new user meets it: a clean profile, every screen driven automatically, and the app’s network connections captured at the system level while it searched.
- Stack
- Rust + Tauri 2 · Svelte 5 · llama.cpp in-process
- Private search
- Built-in Tor or in-app WireGuard, no admin rights
- Status
- Beta on Linux · open source (AGPL-3.0)
What I found
The earlier build passed hundreds of unit tests while its Tor mode fetched pages from the real IP address.
After the fixes, every remote endpoint the app opened during a search was a Tor relay, and none was a website.
After an encrypted session, a search of the whole profile directory for the chat’s name, question and answer found nothing.
On a 32-case grounded-answer benchmark run twice per model, Gemma 4 E4B scored 98.4% and Qwen3.8 27B 96.9%, at 223 and 56 tokens/s.
The central idea
A privacy feature is a claim about network traffic, so the test has to capture the traffic. The earlier build passed hundreds of unit tests while its Tor mode fetched pages from the real IP address.
- 01Clean profile
- 02Drive every screen
- 03Capture connections
- 04Fix
- 05Re-run and re-capture
Contents
The test: a new user and a packet-level witness
The harness drives the real desktop app through WebDriver: it installs, chats, turns on private search, creates an encrypted chat and restarts, taking a screenshot at each step. While the app searched, a root-level socket capture listed every remote endpoint it had open, and each address was looked up in the Tor Project’s relay directory.
The capture matters because it does not trust the app. A badge that says Tor, a log line that says the tunnel is up and a passing unit test are all things the code says about itself. The socket table is what the operating system says.
What the first run found
The earlier build did not work as a product, and its privacy features protected no one. The Tor leak came down to one line: the HTTP client was given the Tor proxy and then told to ignore proxies, which in the current version of the library clears the proxy it had just been given. Pages went out directly while Tor sat idle.
| Area | Result | What happened |
|---|---|---|
| First-run install | Broken | The installer stepped through its screens without downloading anything |
| Chat | Broken | Raw reasoning shown, answers cut off at 512 tokens, everyday questions refused |
| Web search over Tor | Leaked the real IP | Page fetches bypassed Tor; the capture showed direct connections to the sites |
| VPN | Never routed | Needed root to create a network device and sent no traffic through it |
| Encryption | Could not be enabled | Setting a passphrase always failed; saved chats were plain text on disk |
After the fixes: evidence instead of assertions
Tor is now built in, and searches go to DuckDuckGo’s onion service, so the query never reaches an exit relay. In the re-run, every remote endpoint the app opened during a search was a Tor relay and none was a website. The VPN is WireGuard running inside the app, with no admin rights and no system changes. Its only non-loopback socket is the WireGuard connection, so if the tunnel drops, requests fail instead of leaking.
Saved chats are encrypted with the user’s passphrase. After a session, a search of the whole profile directory for the chat’s name, question and answer found nothing. Documents attached to an encrypted chat go into an encrypted index: questions about a PDF lease were answered exactly, citing the page, with none of the lease text readable on disk.
Choosing the model with the app’s own benchmark
The benchmark runs 32 cases through the app’s real prompts, each model’s own chat template and the same inference loop: answering from given sources with correct citations, saying so when the sources do not contain the answer, conflicting sources, a prompt-injection trap inside a source, a Spanish source, formatting and date arithmetic. Every model ran twice with fixed seeds, and every failure was read by hand. Several were the scorer’s fault rather than the model’s, and the scorer was fixed before any ranking was trusted.
The table is not the whole decision. Small models do well on grounded answering, which is what this benchmark measures, while larger models are far ahead on general knowledge and reasoning. So setup recommends Qwen3.8 27B wherever it fits, a 2-bit version of it on 12 to 16 GB graphics cards, and the small Gemma 4 E4B only on 8 GB machines.
| Model | Score | Tokens/s |
|---|---|---|
| Gemma 4 E4B | 98.4% | 223 |
| Qwen3.8 27B | 96.9% | 56 |
| Qwen3.6 35B-A3B | 96.9% | 182 |
| Qwen3.8 27B, 2-bit | 93.8% | 75 |
| Qwen3.5 9B | 93.8% | 163 |
| Ornith 1.5 35B-A3B | 92.2% | 184 |
| Nemotron 3.5 Lightning 30B-A3B | 84.4% | 248 |
| MiniCPM5 2B | 75.0% | 308 |
Which search engines answer over Tor
Search over Tor is its own problem, because most engines treat Tor exits as bots. Probed through a stock Tor client over three rounds of fresh circuits, DuckDuckGo’s onion service and the Wikipedia API answered every time. Bing returned HTTP 200 with results unrelated to the query, which is worse than an error. Brave, Startpage, Qwant, Mojeek, MetaGer and Yahoo returned CAPTCHAs, rate limits or errors. The app uses the onion service plus Wikipedia on Tor, and DuckDuckGo plus Wikipedia over a VPN.
What this does not claim
The boundaries, stated the way I would want to read them on anyone else’s privacy tool:
- Linux is the tested platform. Mac and Windows builds are not produced yet.
- There has been no external security audit.
- Private search hides your address from websites. It does not hide from your internet provider that you use Tor, and it cannot protect identifying details you type into a question.
- Nothing here protects against malware already on the computer.
- The benchmark measures grounded answering, citation and abstention on 32 cases, not general intelligence, and every speed comes from one workstation.
Questions about this note? Get in touch.
sipratt@gmail.com