Local AI Is a Scam? My Honest Take, and What I Actually Use It For
According to my comment section, local models are overrated, a huge scam and a dumb, expensive hobby. Because of that hobby, AI is causing RAM shortages, hardware shortages, gamers are suffering, and so on. To be fair, there's some truth to all of that. Local AI is not for everyone and it does not replace a frontier lab subscription. So this one is a rant: the good, the bad, my honest take, and what I actually use my Strix Halo box for.
It's not the rig of my dreams. It's not a bunch of RTX 6000s and a stack of DGX Sparks. It's a small box, and I do a lot with it.

The good
Compared to two years ago, local AI is in a really good place. The models are shockingly good now. They're a lot smaller than they used to be and much smarter for their size. Open licenses mean people take them, tweak them and make them better.
The harnesses around the models got better too: Hermes Agent (I can never figure out how to pronounce it), Pi, OpenCode, Claude Code and OpenAI's Codex. Take a decently smart model on reasonable hardware, add a really good harness, MCPs and tools, and you can get a lot done with something some people would call not enough.
Hardware caught up as well. Apple runs models very well, there's Strix Halo, the new Gorgon Halo chips, and Nvidia's GB10 boxes, which are sold out and stupid expensive. Hardware is no longer the bottleneck. It's a price problem now. And the tools are easier than ever: read a few guides, watch a few videos, and in five or ten minutes you have a local model running, even on a phone.

The bad
There are a lot fewer guardrails. You don't get the cushions of a ChatGPT or Claude subscription, where they really baby you. Dive into gray territory like medical questions or security work and a cloud model shuts you off quickly. Local models mostly don't, and that's probably intentional, because nobody wants to neuter something that's meant to be free and open.

There's still a ton of tinkering: the harness, the model, your hardware, maybe running without a display. You end up learning what MTP and n-grams are, terms you may not even want to know. I like the back end stuff. Most people won't.
Uncensored models have even fewer guardrails. Sometimes they're dumber than the originals, but they won't tell you no, and that puts a bad taste in regulators' mouths. And now there are poisoned models: give one a code phrase and it can start sending your environment variables to a server. I made a whole video about that.
My honest take
For most people, cloud frontier models are the easiest and biggest return on investment. I'm never going to set someone who has never touched AI up with a local model. You sign up for a $20 plan and you're chatting with a bot within minutes. The latest models get much better results with less pain and friction. Local models are always behind, and to get comparable results you have to babysit them, use a good harness, tinker and add guardrails.
If you buy hardware expecting to get your money back, you most likely won't. My guess is two to five years to replace a subscription, and that assumes the box runs almost nonstop, before electricity, maintenance and your own time. The exception is if you sell it as a service to people who care about security and compliance more than the latest model.

Local AI is also more than a web UI that copies ChatGPT. There's a whole world of specialized models to connect together, and it changes daily. I took a couple of weeks off and came back to three new model releases.
What I actually use it for
Mostly privacy, max privacy. The box sits at home on my network, and I reach it over Tailscale.
A declutter bot. I forward Hermes pictures, ideas, screenshots and plans. It runs on my local box and files them into Obsidian: a place to visit or eat, a quote, a date night idea. No more digging through screenshots from months ago.

My own document library. PDFs, docs, Excel sheets, Markdown and HTML files, school records, medical files, bills, contracts. Local AI extracts the contents, writes the scripts, does the transcriptions and checks the result with vision. I was basically doing retrieval-augmented generation (RAG) before I knew the name. A good Qwen model, a good harness and context about me, on a modest box, can do a lot, and none of it sits in someone's cloud drive.

A life retro. I've journaled for years, on paper and digitally, and I record voice memos. Small transcription models turn those into text, and then I ask things I'd never send to Anthropic or OpenAI: what did I do on this date five years in a row, what was I thinking about this five years ago, what were my goals this year. It's a little cheesy, but I use it for gratitude too.

Finances. CSV files and bank statements get organized and categorized into charts and a debt payoff plan. I started from a YNAB-style budget structure and had AI adapt it to our finances.
Health. Apple Health exports, a smart scale, blood work, doctor's notes and supplements go into one folder for a small model that specializes in medicine. It's not my doctor. It explains trends, and I take the questions to my doctor, who usually agrees.

None of this is groundbreaking. It just makes my life easier in a more private way.
Where I think it's going
Who knows what happens when the AI market pops, subscription prices go through the roof and tokens get metered harder than ever. Local AI might become the only option, or a luxury. My bet is that local AI is the future: smaller models tailored to specific tasks on modest hardware. Massive data centers and subscriptions running at a loss don't look sustainable to me.
So if it's privacy focused, go local. For anything else, look at the cheap subscriptions. I have both: the local box, a Claude subscription and a $20 OpenAI plan. Behind the scenes, Claude Code designs my benchmarks at max settings, local Qwen 3.8 Flash-Next runs them for days for free, and GPT-6 Astra checks Qwen's work, Claude's work and my assumptions.

What are your top three local AI use cases? Tell me what I'm missing.
Sources
- Upgrading my GMKtec EVO-X2 with a second SSD
- Can a local AI model phone home?
- Qwen 3.8 Flash-Next benchmarks
- GPT-6 Astra first impression
- Hermes Agent on GitHub
- Obsidian
- Tailscale
🎥 Gear links
- Sony a6700 with 16-50mm kit lens
- Shure SM7B dynamic microphone
- Shure MVX2U Gen 2 audio interface
- Audio-Technica ATH-M50x headphones
- ADAM Audio D3V speakers
- Logitech G PRO X Superlight 2 mouse
- NEEWER NL288 LED video light
Affiliate links may earn me a commission at no extra cost to you. As an Amazon Associate I earn from qualifying purchases.
