AI agents at work without surrendering your data: a practical setup

The clock ticks past 4 PM. Your screen glows with a half-finished report, a dozen browser tabs open, and an inbox holding 38 unread threads. You feel the drag, the mental resistance to another cycle of research, synthesis, and drafting, knowing a cloud-based AI could accelerate it all, but at the cost of your data, funnelling your proprietary insights into the algorithms of a third party. The machine is designed for extraction, not your sovereignty.

The Default Trap: Efficiency at a Cost

Most current AI agent setups offer a seductive bargain: immense productivity in exchange for your data. You upload documents, connect to cloud services, and allow the agent to process, summarise, and generate. This convenience is built on an architecture where your information, often sensitive or proprietary, leaves your controlled environment. It is ingested by remote servers, processed by models you cannot inspect, and potentially retained for periods you cannot verify, regardless of what a vendor’s privacy policy claims. The default is efficiency coupled with an unseen cost: the erosion of your digital sovereignty.

This isn’t just about direct data leakage. When your work product, your internal memos, your client strategies, or your code snippets are processed by a third-party LLM, they become part of a vast, opaque training corpus, or at the very least, they reside on someone else’s infrastructure. This creates a dependency, a vendor lock-in that makes it difficult to switch providers without leaving a digital trail behind. The real reason you hesitate isn’t a lack of trust in AI’s capabilities; it’s a lack of control over your information once it enters another entity’s system. You’re not choosing between AI and privacy; you’re choosing whose infrastructure you trust, and the default option isn’t built to serve your interests first.

Defining Your Private AI Workstation

A private AI workstation reclaims this control. It is an environment where the entire AI agent workflow—from data ingestion to model inference and output generation—occurs exclusively under your direct supervision, on hardware you own or control, within an isolated ecosystem. “Private” here means your data never traverses an untrusted network to an external LLM provider. It means the model runs locally, processing your information without sending it to a third party.

The goals are clear:
1. Data Never Leaves Your Control: All sensitive inputs and outputs remain on your local machine or private network.
2. Model Transparency (Relative): You use open-source models, allowing for community inspection and understanding of their architecture, even if you don’t inspect every training weight yourself.
3. Isolated Environment: The agent operates within a sandboxed context, limiting its access to only what is strictly necessary.
4. No Vendor Lock-in for Core Processing: You are free to swap out models, frameworks, or hardware without losing control of your data or being beholden to a single provider’s terms.

A common failure mode is believing that “anonymised” or “pseudonymised” data sent to a cloud provider is truly private. Re-identification techniques have advanced significantly, and the context of your work data often provides enough unique identifiers to link it back to you or your organisation. For true sovereignty, the data must remain under your physical and logical control.

Component 1: The Local Processing Unit

The foundation of your private AI setup is robust local hardware. Running large language models and agentic frameworks demands significant computational resources.

Hardware Requirements

* Processor (CPU): While the GPU handles the bulk of LLM inference, a modern multi-core CPU (e.g., Intel i7/i9, AMD Ryzen 7/9 from the last 2-3 generations) is essential for overall system responsiveness, running the operating system, and managing the agent’s logic. Aim for at least 8 physical cores.
* System Memory (RAM): This is critical. Even if your GPU has ample VRAM, the operating system, the LLM runner, and the agent framework itself consume RAM. For smaller models (7B-13B parameters) and light agentic tasks, 32 GB of RAM is a practical minimum. For larger models (30B+ parameters) or complex multi-agent setups, 64 GB or even 128 GB is advisable. Running a 30B parameter model, even quantised, might consume 20-25 GB of system RAM if offloaded partly to CPU.
* Graphics Card (GPU) with VRAM: This is often the bottleneck. LLMs are memory-intensive. The more VRAM your GPU has, the larger and more capable models you can run entirely on the GPU, which drastically increases inference speed.
* Minimum: 12 GB VRAM (e.g., NVIDIA RTX 3060/4060 Ti with 16GB, or older RTX 2080 Ti) for running smaller 7B-13B parameter models quantized to 4-bit.
* Recommended: 24 GB VRAM (e.g., NVIDIA RTX 3090, 4090, or professional cards like an A6000) allows for 30B-40B parameter models at 4-bit or even 7B models at higher precision (8-bit), significantly enhancing performance and model quality.
* AMD GPUs are gaining support (e.g., ROCm for Linux), but NVIDIA’s CUDA ecosystem remains the most mature for local LLM inference.
* Storage (SSD): A fast NVMe SSD is crucial for quick loading of LLM weights and for handling the agent’s temporary files. A 1 TB drive is a good starting point, but consider 2 TB if you plan to experiment with multiple models, as some can be 30-60 GB each.

The tradeoff is clear: higher cost for better performance and the ability to run larger, more capable models. Power consumption will also increase with more powerful GPUs. For example, an NVIDIA RTX 4090 can draw over 400 watts under load.

Operating System

For maximum control and access to open-source tooling, a Linux distribution is highly recommended.
* Ubuntu LTS (Long Term Support) or Debian Stable: These offer a balance of up-to-date packages, extensive community support, and stability. They provide a solid base for installing GPU drivers (especially NVIDIA’s proprietary drivers), Docker, and Python environments.
* Containerization (Docker/Podman): Regardless of your chosen OS, containerisation is a powerful tool for isolating your AI agent environment. Running the LLM server and the agent framework within separate containers ensures dependencies don’t conflict and provides a clear boundary for resource access. This also simplifies deployment and teardown.

A common failure mode here is using a general-purpose OS (like Windows or macOS) without understanding its background processes, telemetry, and default network configurations. While tools exist for these platforms, Linux offers a more transparent and auditable base for a sovereignty-focused setup.

Component 2: The Local Large Language Model (LLM)

The core of your private AI agent is a large language model that runs entirely on your local hardware.

Choosing an Open-Source Model

Open-source LLMs evolve quickly. Key considerations include:
* Model Family: Popular families include Llama (Meta), Mistral (Mistral AI), Gemma (Google), and various fine-tuned derivatives. These models are generally released with permissive licenses for research and commercial use, though always check the specific license of any model you download.
* Parameter Count: This indicates the model’s size and, generally, its capability. Common sizes are 7B (7 billion parameters), 13B, 34B, 70B. Larger models are more capable but require more VRAM and compute.
* Quantization: This is crucial for running models on consumer hardware. Quantization reduces the precision of the model’s weights (e.g., from 16-bit floating point to 4-bit integer), significantly reducing VRAM requirements and disk size, often with a tolerable impact on performance. Common quantization levels are Q4 (4-bit), Q5, Q8. A 7B parameter model quantized to Q4 requires approximately 5 GB VRAM, while a 70B model at Q4 might need 40 GB. You will typically download “GGUF” or “safetensors” files for local inference.

Failure mode: Downloading models from unverified sources. Always prefer models from reputable open-source communities (e.g., Hugging Face with many downloads and active discussions) and verify checksums if provided.

Running the Model Locally

Several tools simplify running open-source LLMs on your machine:
* Ollama: This is a user-friendly tool that allows you to download, run, and manage various open-source LLMs with a simple command-line interface. It provides a local API endpoint (typically `http://localhost:11434/api/generate`) that your agent framework can easily connect to. Ollama handles the complexities of GPU acceleration and model loading.
Installation:* `curl -fsSL https://ollama.com/install.sh | sh` (for Linux).
Downloading a model:* `ollama run mistral` (downloads Mistral 7B and starts an interactive chat).
* LM Studio: A GUI-based application (Windows, macOS, Linux) that offers a similar experience to Ollama, allowing you to search, download, and run models. It also exposes a compatible OpenAI-like API endpoint.
* Direct Python Libraries (llama.cpp, transformers): For more advanced users who want granular control, libraries like `llama.cpp` (for GGUF models) or Hugging Face’s `transformers` library allow you to load and run models directly within your Python scripts. This offers maximum flexibility but requires more setup.

Once installed, your chosen runner will expose an API endpoint. This is the critical piece that allows your agent to “talk” to the LLM without sending data outside your machine.

Component 3: The Agentic Framework

An agentic framework provides the orchestration layer for your AI. It enables the LLM to perform more complex tasks by giving it “tools” (functions it can call), memory (to retain context across interactions), and a planning capability (to break down complex goals into sub-tasks).

What is an Agentic Framework?

Think of it as the operating system for your LLM. Instead of just asking a single question and getting a single answer, an agent can:
* Plan: Deconstruct a high-level goal into a sequence of actionable steps.
* Tool Use: Interact with external systems or local scripts (e.g., execute a local Python script, search a local document database, run a `grep` command).
* Memory: Maintain a conversational history or a scratchpad of working notes.
* Reflection: Evaluate its own progress and self-correct.

Open-Source Agentic Frameworks

Several robust open-source frameworks are suitable for local deployment:
* AutoGen (Microsoft Research): Designed for multi-agent conversations, where different agents (e.g., a “planner” agent, a “coder” agent, a “reviewer” agent) collaborate to achieve a goal. AutoGen is designed to be model-agnostic and can easily connect to local LLM endpoints.
* CrewAI: Focuses on role-playing and collaborative AI agents, allowing you to define distinct roles, tasks, and a shared goal for a “crew” of agents. It’s built on LangChain.
* LangChain: A more foundational framework, providing abstractions for LLMs, prompt management, chains (sequences of LLM calls), agents, and tools. While it can be complex, it offers immense flexibility for building custom agentic workflows. It supports local LLM integration via various “loaders” and “wrappers.”

Connecting to Your Local LLM

The process generally involves configuring the framework to point to your local LLM’s API endpoint.
* Ollama Integration: Most frameworks can be configured to use an OpenAI-compatible API. Ollama provides this. You would typically set an environment variable like `OPENAI_API_BASE=http://localhost:11434/v1` and `OPENAI_API_KEY=ollama` (the key is often a placeholder when using local instances).
* LM Studio Integration: Similar to Ollama, LM Studio also provides an OpenAI-compatible endpoint.
* Direct Integration: For frameworks like LangChain, you might use specific `Ollama` or `LlamaCpp` classes to directly interface with your local model runner.

Failure mode: Accidentally configuring the agent to call external APIs (e.g., OpenAI’s default endpoint) instead of your local one. Always double-check your API base URLs and environment variables. A `curl` command to your local endpoint (e.g., `curl http://localhost:11434/api/tags`) should confirm it’s running.

Component 4: Isolated Data Environment

The agent is only as secure as the data it processes. An isolated data environment ensures that your sensitive information remains confined and protected.

Data Storage

* Encrypted Local Drives: This is non-negotiable. On Linux, use LUKS (Linux Unified Key Setup) to encrypt your entire drive or dedicated partitions. This ensures that even if your machine is physically compromised, the data remains unreadable without the passphrase.
* Dedicated Storage: Consider a separate physical drive or a dedicated encrypted partition specifically for your AI agent’s input, output, and working files. This compartmentalises your data. Mount a dedicated, encrypted 500 GB drive for agent work.

Data Ingestion

* Manual File Upload/Controlled Directory Access: The agent should only access data you explicitly provide. This means placing relevant files in a specific, sandboxed directory that the agent has permission to read. Avoid giving the agent broad file system access.
* No Automatic Cloud Sync: Ensure the directories used by the agent are not automatically synchronised with cloud storage services (e.g., Dropbox, Google Drive, OneDrive) unless those services are themselves end-to-end encrypted and under your control.

Sandboxing

* Containerisation (Revisited): Running your agentic framework and its tools within Docker or Podman containers provides a strong sandbox. You can precisely control which directories the container can access (using bind mounts) and whether it has network access.
Example:* `docker run -v /path/to/your/data:/app/data –network none my_agent_image` This command mounts your local data directory into the container and explicitly denies network access, ensuring the agent cannot exfiltrate data.
* Virtual Machines (VMs): For extreme isolation, especially for highly sensitive tasks, consider running your entire AI workstation (OS, LLM, agent) within a virtual machine (e.g., using QEMU/KVM on Linux). This creates a completely separate execution environment.

Failure mode: Granting the agent broad network access (“just in case it needs to look something up”) or wide-ranging file system permissions. Each permission granted is a potential vector for data leakage. Default to the principle of least privilege.

Practical Workflow: Building Your First Private Agent

Let’s outline a practical scenario: summarising a collection of proprietary research papers and drafting an internal executive summary, all without sending a single byte to an external service.

1. Prepare Your Hardware and OS:
* Install Ubuntu LTS on your machine.
* Install NVIDIA drivers (if applicable) and CUDA toolkit.
* Set up LUKS encryption for your primary drive and a dedicated data partition.
* Install Docker or Podman.
2. Install Ollama and Download an LLM:
* Install Ollama: `curl -fsSL https://ollama.com/install.sh | sh`
* Download a suitable model, e.g., Mistral 7B: `ollama pull mistral`
* Verify it’s running: `ollama run mistral` (then Ctrl+D to exit).
3. Set Up Your Agentic Framework (e.g., AutoGen):
* Create a dedicated Python virtual environment: `python3 -m venv ai_agent_env && source ai_agent_env/bin/activate`
* Install AutoGen: `pip install pyautogen`
* Create a configuration file (e.g., `OAI_CONFIG_LIST`) that points to your local Ollama instance:
“`json
[
{
“model”: “mistral”,
“api_key”: “ollama”,
“base_url”: “http://localhost:11434/v1”
}
]
“`
4. Configure the Agent to Use the Local LLM Endpoint:
* In your Python script, load this configuration. AutoGen will automatically pick up the `base_url` and `api_key` for your local Ollama endpoint when you define your LLM configuration.
5. Define Agent Capabilities (Tools):
* Create a directory for your research papers, e.g., `/home/user/private_research/`. Ensure it’s mounted into your agent’s container if using Docker.
* Define tools for your agent. For example, a Python function that uses `pypdf` to extract text from a PDF, or a tool that runs `grep` on a directory of text files.
* Example tool (Python script called by agent):
“`python
# read_pdf.py
import pypdf
def extract_text(pdf_path):
reader = pypdf.PdfReader(pdf_path)
text = “”
for page in reader.pages:
text += page.extract_text()
return text
“`
6. Feed It Local, Isolated Data:
* Place your research papers in `/home/user/private_research/`.
* Instruct your agent to summarise these specific files using the `read_pdf.py` tool.
* Example AutoGen script snippet:
“`python
import autogen
# … setup config_list …

llm_config = {“config_list”: config_list, “cache_seed”: 42}

# Create a user proxy agent to ask questions
user_proxy = autogen.UserProxyAgent(
name=”user_proxy”,
human_input_mode=”NEVER”, # Or “ALWAYS” for interactive control
max_is_auto_reply=10,
is_termination_msg=lambda x: x.get(“content”, “”).rstrip().endswith(“TERMINATE”),
code_execution_config={“work_dir”: “coding”, “use_docker”: False}, # Set to True for Docker
)

# Create an assistant agent
assistant = autogen.AssistantAgent(
name=”assistant”,
llm_config=llm_config,
system_message=”You are a helpful assistant. You can read PDFs and summarise them. Use the ‘read_pdf.py’ tool.”,
)

# Register the tool
user_proxy.register_function(
function_map={
“read_pdf”: read_pdf.extract_text # assuming read_pdf.py is accessible
}
)

# Start the conversation
user_proxy.initiate_chat(
assistant,
message=”Summarise the key findings from all PDF documents in the ‘/home/user/private_research/’ directory into a concise executive summary. Highlight any conflicting data points.”,
)
“`
7. Monitor Its Output:
* The agent’s output will be displayed in your terminal or written to a local file you specify. Review this output critically.

Failure mode: Over-relying on the agent without validation. The agent is a tool. Its outputs, especially summaries or generated text, must be reviewed for hallucinated facts, logical inconsistencies, or misinterpretations of your data.

Managing Risks and Maintaining Sovereignty

Building a private AI workstation is an ongoing commitment to digital sovereignty. It requires active management.

* Updates: Regularly update your operating system, GPU drivers, Ollama/LM Studio, and agentic frameworks. Security vulnerabilities are discovered and patched frequently. Neglecting updates is a common vector for compromise.
* Network Isolation: For tasks involving highly sensitive data, consider running your AI agent on a machine with limited or no internet access. A physical air gap is the ultimate security measure against network exfiltration. If network access is required for model downloads, perform them on a separate, less sensitive machine or temporarily enable network access, download, and then disable it.
* Data Lifecycle: Understand how your LLM runner and agent framework handle temporary files and context. Ensure that sensitive data passed into the model’s context window is securely purged from memory and disk after the task is complete. Implement secure deletion practices (e.g., `shred` command on Linux) for any temporary files.
* Human-in-the-Loop: Never fully automate critical decision-making or content generation. The AI agent is a powerful assistant, not an autonomous decision-maker. Always review its outputs, verify facts, and apply your own judgment and expertise. This is a critical safeguard against both AI errors and potential data handling issues.
* Model Audit: While inspecting every weight of a large LLM is impractical, stay informed about the open-source community’s findings regarding model biases, safety issues, and data leakage risks associated with specific models.
* Resource Monitoring: Keep an eye on your system’s resource usage (CPU, RAM, VRAM). Unexpected spikes could indicate an issue or a runaway process.

Frequently asked questions

Can I use a cloud LLM if I encrypt my data first?

Encrypting data before sending it to a cloud LLM provides a layer of protection against passive eavesdropping during transit. However, the data must be decrypted by the cloud provider’s servers for the LLM to process it. At that point, it is in plain text on their infrastructure, subject to their security practices, retention policies, and potential internal access. True data sovereignty means the decryption and processing happen only on your controlled hardware.

Is this setup suitable for real-time collaboration?

This local-first setup prioritises privacy and control over real-time, multi-user collaboration in the traditional sense. While you could share agent outputs manually, it’s not designed for synchronous, shared document editing or concurrent agent interactions across multiple users. For collaborative work, you would need to integrate a secure, private collaboration layer on top of your local AI, ensuring any shared data remains within a trusted, encrypted environment.

What about open-source models trained on public data? Don’t they “know” my data already?

Open-source models are trained on vast datasets, often scraped from the internet. While a model might have encountered similar public information, it does not “know” your specific, private data unless you explicitly feed it in. The risk with cloud LLMs is that your private data becomes part of their ongoing operational data, which could implicitly or explicitly influence future model behaviour or be retained. With a local model, its knowledge base is fixed at the time of download, and your private inputs are processed in isolation.

How much does this setup cost?

The primary cost is hardware. A capable system with a good GPU (e.g., RTX 4090) could range from £2,000 to £4,000 or more, depending on other components. Lower-end setups (e.g., RTX 3060 12GB) could start around £1,000-£1,500. Beyond hardware, the software is largely open-source and free. There is an investment of your time in setup, configuration, and ongoing maintenance.

Takeaway

You are no longer a passive user of someone else’s AI. You are the architect of your own intelligent tools, building on your own rails. This is the first step in reclaiming agency in an increasingly automated world.

Recommended: if you want a vetted option here, we use pCloud (encrypted cloud storage). Affiliate link — we may earn a commission; our verdict is not for sale.

Related reading: explore more in our Work pillar.