Sponsored by
How to run your own AI model locally with AnythingLLM: Step-by-step guide
You probably use AI tools for a ton of everyday tasks, such as researching topics, summarising articles, and creating images. During this process, you share a lot of information, including sensitive documents and personal information, with these chatbots.
However, most of the AI tools you interact with are cloud-based, which means that the information you input leaves your computer, travels through the internet to the provider’s server, and then returns with the answer.
This increases the risk of data interception and data breaches, both in transit and at rest. You're essentially trusting a third-party AI provider with your personal information.
Enter local AI. Unlike cloud AI, local AI runs entirely on your computer, and your data never leaves your system. Several local AI tools are available that can help you achieve results similar to those of cloud AI without compromising your privacy.
One such tool is AnythingLLM. Let’s understand what AnythingLLM is, along with a step-by-step guide on how to install and use it.
What is AnythingLLM?
AnythingLLM is a free, open-source, all-in-one AI application that lets you chat with an AI model, interact with your own documents, and run AI agents, all from a single interface without the need for any coding.
AnythingLLM sits on top of AI models and is not an AI model in itself. You can choose which model powers it, such as Llama through its built-in provider or Ollama. You can also connect it to a cloud AI model such as GPT or Claude using their respective APIs.
AnythingLLM adopts a local-first approach. By default, everything runs on your own computer, and cloud services are an optional add-on rather than a mandatory requirement. Your tasks first run locally on your device, and if you need more power, you can choose to connect a cloud model to a specific workspace while keeping everything else local.
AnythingLLM allows you to create dedicated workspaces for specific tasks and chat with your documents. This comes in handy when you need to scour complicated documents for information. It also comes with its own AI agents, which can help you scrape websites for more information. You can also schedule jobs and create agent workflows.
How to set up AnythingLLM
AnythingLLM is available as a desktop app on macOS, Windows, and Linux. You can also run it on any machine or server that supports Docker, including your own computer or a cloud server on AWS, Google Cloud, or Azure. You get several one-click templates for Railway, Render, DigitalOcean, and others.
If you don’t want to run it yourself, you can also use its cloud version, hosted by Mintplex Labs. Downloading AnythingLLM to your system is pretty simple.
Step 1: Visit the official AnythingLLM website, and you will see a Download Free button on the home page.
Step 2: Click on it and then choose the system you are working on: macOS, Windows, or Linux. We downloaded AnythingLLM for Windows x64.
Step 3: Once your download is complete, you will see a setup page requiring you to confirm the destination folder.
Step 4: After doing so, click on Install and let the installation run to completion. It might take five to ten minutes, depending on your internet speed and processing capabilities. Once everything is set up, you’ll see a Welcome screen. Click on Get Started.
Step 5: After this, you can choose the recommended model you want to connect AnythingLLM to. The platform automatically picks the best model based on your hardware. However, you can also choose a different model according to your needs.
Step 6: Once done, you will be taken through various optional questions. You’ll see the option to enter details such as your email address. You can skip this part if you’re not comfortable sharing any data.
After this, you will see the main AnythingLLM interface, with workspaces on the left panel and a central pane for you to work in.
What can you do with AnythingLLM?
I tested AnythingLLM to see how it holds up on a real task. The test task is exploring a job switch, which involves a lot of personal paperwork, such as my CV, current employment contract, offer letters, and salary details.
These are clearly the kinds of information and documents I wouldn’t want to upload to a cloud AI, which is why I picked AnythingLLM for this.
Let’s understand each feature AnythingLLM offers through this example.
Workspaces and threads
Everything in AnythingLLM is organised into workspaces, which are like project folders with their own documents, settings, chat modes, and even AI models. You can create threads inside each workspace.
These are separate conversations that share the workspace documents but keep their own chat histories. Nothing carries over between workspaces, so documents in one workspace cannot be seen from another.
I created a workspace called “Job Switch” and set up a thread for each company I was considering: Brightline Systems and Halcyon Cloud.
When I asked about Company A’s offer, the discussion didn’t get mixed up with what I had been asking about Company B.
Chatting with your documents
This is one of AnythingLLM’s core features. There are two ways you can upload documents to AnythingLLM: embedding and attaching.
- Embedding: The document is split into small chunks and stored in the local vector database. When you ask a question, only the most relevant chunks are pulled in. An embedded document is available in every thread within your workspace.
- Attaching: You can drop the document directly into a chat by clicking the plus icon. AnythingLLM reads the full text, but the document stays limited to that one thread.
I embedded my CV and current employment contract into the workspace because I would need them in every conversation. The offer letters, on the other hand, were different from company to company, so I attached each one in the relevant company thread.
Attaching documents is pretty easy: simply drag and drop your documents into the chat window or click on the + icon in the prompt input. Then I started asking questions, such as, “What is the notice period in my current contract?” The AI thought for 49.3 seconds and pointed to the correct clause in the employment contract I had uploaded.
Sure, you can argue that a local AI agent takes longer than a cloud AI agent. However, in this case, you do not have to upload your sensitive employment contract to the cloud.
Chat mode: query vs chat
AnythingLLM gives each workspace a chat mode, which controls where the answers come from.
- Query mode answers only from your documents and tells you when it can’t find the answer instead of guessing blindly.
- Chat mode blends your documents with the model’s general knowledge, which makes it more conversational.
- Agent mode, the default for new workspaces, can also use tools on its own.
In my testing, I used Query mode for factual questions such as notice periods, joining dates, and salary compensation because I did not want the model filling gaps with made-up details. For broader questions, such as how I should negotiate a higher salary or joining bonus, I switched to Chat mode. It drew on my offer details and gave me general advice.
AI agents and web scraping
Besides helping you work locally, AnythingLLM can also scrape web pages and pull information when needed. In older workspaces, you can start an AI agent by typing @agent before your prompt. In newer workspaces, where Agent mode is on by default, the model decides on its own when a tool is needed.
In my example, I needed a quick brief on the company before I sat for an interview. So, I typed, “@agent, scrape the company’s website and summarise what the company does, its main product, and anything about its work culture.”
Agent flows
AnythingLLM also allows you to create custom agent flows without writing a single line of code. You can build them in a simple visual editor by stacking steps.
For example, you can visit a website, summarise what you find, and then save the results to a file. Once saved, the flow works like any other agent skill. You can ask AnythingLLM in plain language to run the whole sequence for you.
Flows are most useful for repetitive tasks that involve small changes. For instance, during a job search, you could build one flow that checks a company’s career page and lists only roles that match your profile. Other common uses include daily news roundups, pulling data from websites into a report, or summarising new posts from a blog.
You only need to set up the instructions once, and from then on, AnythingLLM does the work for you with a single request.
Scheduled jobs
AnythingLLM also offers a scheduled jobs feature, which allows the agent to run a task automatically on a recurring schedule. You only need to give the agent a prompt, a schedule, and the tools it’s allowed to use, and AnythingLLM runs it in the background.
Every run is saved, so you can review the settings later or continue the conversation in a regular thread.
Memories
AnythingLLM can also remember useful facts about you using its memory feature, such as your role, preferences, and ongoing projects, and then use them in future conversations. These come in two kinds: workspace memories apply only to specific workspaces, whereas global memories apply everywhere.
The feature is turned off by default. You can turn it on from the chat settings menu under Memories, and AnythingLLM can then build memories automatically from your past conversations.
AnythingLLM desktop features
Besides the chat and workspace features, AnythingLLM also offers several desktop features that work outside the main chat window across your whole computer.
Meeting Assistant
The Meeting Assistant records, transcribes, and summarises your meetings, just as a paid tool would, but it runs entirely on your device. It listens locally instead of having a bot join the meeting and identifies who said what. It can write summaries using the model you choose and suggest follow-up actions with just one click.
You can also upload recordings you already have, including podcasts and YouTube videos. Every meeting is indexed, allowing you to search by topic, speaker, or date and chat with any transcript. The best part is that the recordings never leave your device. Even if you use a cloud model for summaries, only the transcript text is sent, not the actual voice recording.
Desktop Assistant
This is a pop-up overlay you can open from anywhere with CMD + / on Mac or Ctrl + / on Windows and Linux. This allows you to chat with AnythingLLM without switching to the app.
The Desktop Assistant is screen-aware, which means it can see whichever app you’re working in, such as Chrome or Slack. So, you can ask questions about what’s on the screen. It can also use all your agent skills and MCP tools.
Magic features
Magic features come with a free daily allowance and are unlimited on the Pro plan. There are three main tools under this category, available on Mac and Windows only, not Linux.
- Magic Echo: You can press Option + Z or Alt + Z and speak. Your words are transcribed, cleaned up with punctuation and grammar corrections, and stripped of filler words. The text is then automatically typed wherever your cursor is placed. It can use what’s on your screen for context, and you can even set voice shortcuts, such as saying “sign off” to paste your email signature.
- Magic Beacon: This feature lets you highlight text in any app, after which a small dot appears. You can click it to summarise, translate, explain, or rewrite the text without leaving the app.
- Magic Tap: This is AnythingLLM’s smart autocomplete feature, which suggests how you can finish your sentence using faded ghost text, similar to what you see in apps like WhatsApp or Instagram. You can press Tab to accept the suggestion or choose to ignore it.
Final thoughts
AnythingLLM is an open-source, local-first AI application that runs locally on your system, helping keep your data private and secure. Unlike cloud AI, it does not send your inputs to the servers of third-party AI providers when you use a local model. You can achieve this privacy and security without necessarily compromising on the quality of your AI-related tasks.
AnythingLLM allows you to chat with your documents, embed them in workspaces, and extract information locally. There’s also an option to connect the model to the internet and scour web pages using AI agents. Besides this, you can schedule jobs to run at specific dates and times every day and even create agent flows to carry out repetitive tasks without the need for coding.
There are also several free desktop features, such as Meeting Assistant, which lets you record and transcribe your meetings without having to upload your recordings to the cloud, and Desktop Assistant, a screen-aware tool that lets you ask questions about what’s on your screen.
Sure, at times, you might find AnythingLLM to be a tad slow, just like any other local AI tool. It might take a few extra seconds to get the results you want. However, this is a trade-off you might have to accept if you want the security and privacy benefits of a local AI tool like AnythingLLM.