Private AI for small business

When running AI on your own hardware beats sending everything to the cloud, and when it doesn't.

Most businesses meet AI through ChatGPT, Claude or Gemini. They're cloud services: you type something in, it goes to the provider's servers, and an answer comes back. For a lot of everyday work that's exactly right. But I keep hearing the same worry from business owners: "I'd love to use this on our client files, but I'm not comfortable sending them off to someone else's computer."

That's where private AI comes in. Instead of renting a model in the cloud, you run an AI model on hardware you own, in your office, your workshop or a small box on a shelf. Your data never leaves the building. This guide explains when that's worth doing, what it involves, and when you're better off sticking with the cloud.

What "local" or "private" AI actually means

Over the last few years, a lot of capable AI models have been released as open-weight models. That means anyone can download them and run them on their own computer. Families like Llama, Qwen, Mistral and Gemma come in a range of sizes, from small models that run on a compact device to large ones that need a serious workstation.

A private AI system usually has three parts:

• The hardware: anything from a small NVIDIA Jetson board to a PC with a good graphics card, or a dedicated AI workstation
• The model: an open-weight model chosen and set up for your job
• The tooling around it: how it reads your documents, remembers context, connects to your systems, and how your staff actually use it (a chat window, a form, a voice note, or an automatic process running in the background)

The third part is where most of the value is. A raw model is like a smart new employee with no training. The setup around it is what makes it useful for your business.

When local models beat cloud AI

1. Your data is sensitive or confidential

Client records, legal documents, medical notes, financials, unreleased product designs, tender pricing. If a breach or leak would cost you a client or land you in trouble, keeping the AI on-site removes a whole category of risk. There's no third-party server, no question about where the data is stored, and no worrying about what a provider's terms might allow. For Australian businesses dealing with personal information, keeping it on-site also makes your privacy obligations simpler to explain to clients.

2. The job is narrow and repetitive

Local models are at their best on focused tasks done hundreds of times: turning site notes into a daily report, sorting incoming enquiries, pulling key details out of invoices or job sheets, summarising long documents, or answering staff questions from your own manuals and procedures. You don't need the world's biggest model to do one job well. A well-set-up smaller model running on modest hardware often does it reliably.

3. You use it a lot

Cloud AI is usually charged per user or per amount of text processed. That's cheap when you use it now and then, but it adds up when a process runs all day. A local system is mostly an upfront cost, plus power and maintenance. Once it's running, the hundredth report costs the same as the first.

4. The internet is unreliable, or there isn't any

Rural properties, job sites, sheds, vehicles and events often have patchy connections. A local model keeps working with no internet at all. That's also why it suits devices: an AI built into a product can't stop working just because the Wi-Fi dropped out.

5. You want to own it

Cloud providers change prices, retire models and update their terms. With a local system, the model you set up today behaves the same way next year unless you decide to change it. For a core business process, that kind of stability matters.

When cloud AI is the better choice

Private AI isn't always the answer, and I'd rather tell you that up front.

• You need the most capable model available. For difficult reasoning, complex coding or polished long-form writing, the largest cloud models are still ahead of what most small businesses can run at home.
• You only use it occasionally. If AI is something you use a few times a week, a cloud subscription is cheaper and simpler.
• Nobody can look after it. A local system needs updates and backups like any other computer. If that isn't covered, a managed cloud service may suit you better.
• The data isn't sensitive. Drafting social media posts or brainstorming product names doesn't need to stay in the building.

Many businesses end up with a hybrid: cloud AI for general work, and a private system for the sensitive or high-volume jobs.

A quick comparison

What it looks like in practice

My own system, Polly, is a local AI with persistent memory running on an NVIDIA Jetson board. It remembers context across conversations, works without the internet, and keeps everything on hardware I control. It's my research assistant, and it's a working example of the kind of system I build for clients.

Here's an example of what that could mean for a business: a builder records a voice note at the end of each day on site. A small private AI box in the office turns it into a tidy daily report with the weather, trades on site, delays and next steps, then files it against the right job. No client details go to the cloud, it works even when the site has poor reception and the note syncs later, and the cost doesn't go up the more reports are written.

What does the hardware cost?

It depends on the size of the model and how many people use it at once. In rough terms:

• Compact edge devices (such as the NVIDIA Jetson Orin range) cost a few hundred dollars and suit one focused job, a single user, or AI built into a product
• A desktop PC with a 16GB-or-larger graphics card runs mid-sized models well for a small team
• Dedicated AI workstations cost a few thousand dollars and run large models for heavier use or several staff

The right choice comes from the job, not the other way round. I'd always start by proving the task on modest hardware before anyone buys a bigger box.

Security still matters

Keeping AI local removes the third-party risk, but it doesn't make the system secure by itself. A good private AI setup still needs proper user logins, a sensible network setup (it shouldn't be reachable from the open internet unless that's designed in), encrypted backups, and clear rules about who can see what. Treat it like any other system that holds your business data.

← More guides