Every few days some dude posts this exact “thinkpost” to LinkedIn. It is always giving the same energy: A dude who discovered last Tuesday that VRAM exists, and would now like to explain it to you. Slowly, and in the most remedial way possible. It reminds me of the way I try to coax my hound dog Dory through a revolving door. If you’ve ever tried to get a dog to understand a revolving door, you feel my pain.
Basically, this dude has no idea what he’s doing, and it’s awkward as hell watching him try to work through something he doesn’t understand. He throws the numbers out like they mean something to the average person. 400 billion parameters. 1.5TB for full precision. 25 billion active. Context window. He references a gaming RTX 5090 for some reason, as if a gaming GPU is an economical way to do this. (It’s not. Gaming GPU’s are meant for gaming. This is why they are built and marketing for gaming.) Then he crashes the plane: “You cannot run frontier models on your own hardware, and anyone telling you otherwise is either lying or too dumb to know they’re lying.”
He’s about half right. Which, on the internet, means people will think he’s right. Being right on the internet, among people who don’t know a damn thing, is the single most dangerous amount of right a person can be.
The end result of virtually every version of this thinkpost is the same thesis: A dude proves that you can’t run a giant open-weight model on a gaming GPU. That part is true. No fucking shit, dude. A Porsche 911 can’t tow 10,000 pound boat. I have real questions about anyone using a Porsche 911 for towing… period.
A 5090 has 32GB of VRAM. A 5080 has 16GB. (Usually.) Neither one is going to run a frontier model alone. Most consumer motherboards only have 4 x16 PCI-E slots, which means best case, you’re getting 128GB of total VRAM, and spending around $16,000 to get just the 5090’s. (I know MSRP was supposed to be $2,000, but that shit ain’t happening.) But then our thinkpost writer took that true thing and swapped it, mid-thought, for a completely different thing: That you can’t run frontier models on domestic hardware at all. Those are not the same sentence.
Once again, I know a Porsche 911 wasn’t designed to tow a boat. But from that conclusion, I can’t just assume that diesel trucks don’t exist, and a Porsche 911 is the only means of getting a boat from the tarmac to a boat ramp. The space between a gaming GPU and a $500,000 8-flamethrower-deep-B200 server is not empty. It is, in fact, full of hardware that was specifically designed for the exact purpose of serving a mix of model sizes, for different tasks.
Then there are two other ideas this dude never says, because they’re complicated. Quantization and mixture-of-experts (MoE).
Quantization is the boring one, so let’s speed through it. Nobody serving a model at home runs it at full precision, because it’s kinda fucking dumb. You run it at 4-bit, which cuts the memory footprint by roughly a factor of four at basically no quality cost for inference. His “400GB minimum, 1.5TB full precision” numbers are the sticker price for a Corvette ZR-1. Yes, you can technically go out and buy a 5.5 liter, flat plane V8, being force-fed by turbochargers for like $200,000 — but the reason Chevrolet makes the ZR-1 is because they know you aren’t buying it. They make it so you’ll buy the $65,000 2LT package, volume seller. To be clear, if you get into a 2LT package Corvette, and mash the right pedal, you’ll think you’re in a fighter plane getting steam-launched off the deck of an aircraft carrier. I have yet to meet a person think the C8 Corvette isn’t all the sports car that anyone needs.
So yes, you can technically get the full-fat model with 1.5TB of VRAM… but absent the fuck-you money of some VC firm lighting cash on fire to fuel your company, you don’t need it. It’s a vanity item, not a requirement.
The second term is mixture-of-experts, and this is the one that really matters. It’s the reason these thinkposts are dumb, and they seem to feed from circa-2023 Reddit posts when hobbyists were just trying to figure it out. Modern open-weight models aren’t one giant brain where every neuron fires on every word you type. Take Qwen3-235B-A22B. That’s 235 billion parameters total, but only about 22 billion are active on any given token. The model is enormous, sure. But the thing it’s doing at any instant is not. I have an amazing number of food options in my fridge, pantry, and freezer at any given moment, but I’m only going to cook a meal from a small number of them at any time. So the question was never, “Can your hardware hold 235 billion parameters at once?” It’s “Can your hardware stream 22 billion at a time?” And that, it turns out, is a solved problem that costs about the same as a lightly used Mazda Miata NC chassis. (Also, holy shit you can get a nice NC Miata for like $10,000 now? Damn.)
Which brings us to the thing the size of a shoebox that AMD and Nvidia and a few other hardware companies are beginning to talk about, but not much, because it’s still a niche product for super-nerds.
The most affordable one is a GMKtec EVO-X2. It has a Ryzen AI Max+ 395, 128GB of unified memory and it’s about… $3,500 for the 128GB configuation. It’s about the size of the Pelican case I put my DJI Osmo Pocket 3 into. On its own, one of these loads Qwen3-235B-A22B and generates around 12 tokens per second, which is comfortable reading speed, and it runs 30B-class MoE models at 60 to 100 tokens per second, which is faster than you can follow. The 5090 our LinkedIn dude waived at you, the $2,000 card… that cannot load that 235B model at all. Not slowly. Not badly. It just won’t fit. The $3,400 shoebox “AI-PC” does it while consuming one quarter of the wattage.
Want to go bigger? Buy two. Two of them is 256GB of combined memory for about $6,800, and you cluster them over Ethernet using llama.cpp’s RPC engine. This is not a hack I found on a forum at 2am. AMD publishes the official walkthrough. Their own playbook clusters two of these units to run GLM 4.7, a 358-billion-parameter model, across both machines. Their developer team wired up four of them and ran Kimi K2.5, which is a trillion-parameter model, on 512GB of what is essentially four Jordan 6 boxes filled with magic GPU’s.
The only thing AMD is hiding here is the network connection limitation, because it’s not exactly clean. Pairing two of these AMD boxes is going to create a bottleneck, and that bottleneck is… ethernet cables.
They cluster over a 10-gigabit Ethernet link, and that link is slow enough that the network, not the chips, becomes your bottleneck. So if a model fits inside a single 128GB box, you run it in a single box and you don’t split it. You only pair them when the model is genuinely too fat for one. If you want the cleaner version of this, NVIDIA’s DGX Spark pairs two units over a purpose-built 200-gigabit connection, roughly twenty times the bandwidth, and handles models up to 405 billion parameters. It costs more. I think the cable alone is like $200? It’s an expensive cable.
That’s the trade. Adults make trades. If you want to remove the bottleneck, you pony up for the Nvidia hardware.
At some point in the thinkpost you read, the dude will bring up the RTX 6000 Pro, the $13,000 super-GPU.
Here’s the thing about the $13,000 RTX 6000 Pro. It’s fucking awesome at what it does. But you need to understand why it’s expensive. It’s not expensive because it has a lot of DDR7 RAM. It’s the speed of the inference it offers. That’s 96GB at 1,792 GB/s, and one RTX 6000 Pro pushes around 8,400 tokens per second on a 30B model. It can batch out thousands of requests at once. An RTX is meant to serve hundreds of people at once. It’s what you buy when you have a 500 person company, roughly speaking. You don’t need bandwidth meant to serve a high school gymnasium full of people.
You, at home, serving exactly one human being, run a batch size of one. You do not need to even consider buying an RTX 6000 Pro. Buying an RTX 6000 Pro to run your agents, even if you have a dozen… is stupid. If you don’t own a boat, or a farm, or a horse trailer, it does not make sense to buy a dually F-350 with a diesel, because the fact that you can throw 25,000 pounds on a goose-neck is undermined by the fact that the heaviest thing you’ll tow is your WIFE. (BOOM! ROASTED!)
One other small note while I’m here: Anthropic and OpenAI are not racking RTX 6000 Pros. That’s a workstation card for small to medium sized enterprises. The companies serving hundreds of thousands of people at once run datacenter Blackwell, the B200s and GB200s, and they buy them in container ship volumes. That is the entire point, I’m trying to make here, I guess. They buy oceans (and ocean-going-vessels) of high-bandwidth compute because they are serving millions of people simultaneously. Concurrency is their problem. It was never your problem. You are not a hyperscaler. You are a person with a mini PC who wants to own his own compute, and for that, the math has never been more in your favor — provided you don’t mind spending “lightly used Mazda Miata money” on something that will mean you need to tinker with it. (And for some people, like me, who are the type to buy project cars, and don’t mind getting our hands dirty, this is a fun hobby.)
I’ll even give our thinkposter the two things he got right, because unlike his post, I can hold more than one idea at a time under this luxurious mullet. Memory bandwidth genuinely is the king of per-token speed, and a discrete GPU will smoke these unified-memory boxes on any small model that fits in its VRAM. It’s true. The single best thing about open weights probably isn’t running them on your desk at all; It’s the pricing competition they create in cloud hosting, where a dozen providers undercut each other on the same model. Competition means pricing is going to be a race to the bottom, and that will only help consumers. Both of these ideas can be true at the same time.
So here’s how it actually plays out: You’re going to spend money on hardware today that will lose value, because that’s the early adopter tax. If you don’t mind being an early adopter, and you understand fabs are going to be able to ramp up, and 2-3 years from now, these same desktop AI-PC’s are probably going to cost… $1,000 or so? (I don’t know, but the RAM shortage can’t last forever.) Whatever. You know what you’re getting into.
Meaningful Addendum: Who This Is Actually For?
Everything I just said is true. It is also a hobby. I think it’s dumb as fuck that so much VC cash is being pushed into all these unproven agentic systems, harnesses, apps, etc.
All of it assumes you are comfortable in front of a command line. It assumes you can run containers without googling what a container is. It assumes a baseline tolerance for open source applications that behave, let’s say, less than predictably. Software that ships with a README written by… probably another agent. Software where the fix for your problem is a… figuring it out yourself, motherfucker. That’s the tax. Nobody advertises these systems being a total pain in the dick.
A lot of people are doing this on Apple hardware, and honestly, good for them. I have seen various guides using Mac Minis, and also the higher-end Mac Studios with a truly obscene amount of unified memory. Apple has been putting the CPU, GPU, and memory in one pool for years now, which means the memory accessible to the graphics side isn’t some sad little walled-off 16GB like in Windows and Linux systems. Their unified Apple Silicon means they share the whole pool. Which was, for a long time, a cool architectural flex that was great for people like me who are content creators, because it made video editing dumbass fast. Then large language models showed up and turned Apple Silicon with unified memory into the single most important design decision anybody made this decade, entirely by accident.
Which positions Apple absurdly well, and I don’t think they’ve fully shown their hand yet — mostly because for hobbyists, Apple kind of… sucks. Most of us are running some derivative of Ubuntu as far as I can tell. Apple is famously not for tinkering. It’s stable. It’s reliable. But they don’t want you messing with their hardware.
I can see the world clearly from here: Apple gets into the model business the exact way Apple gets into every business. Not first. They’re never first. They’ll wait for everyone else to figure out the market, and then introduce a polished product. If you want to access this procuct, you subscribe. The model runs locally on your hardware.
I can see it now. They’re going to ditch Siri, because it never really met the AI-moment. (I hope they call it Ronnie, because it’s the most trailer park name possible.) They’ll say the word “privacy” roughly eleventy-thousand times in the keynote in front of the big black projector screen. Apple pushes the updates. It’s iCloud, but the thing being synced is a brain. They already sell you the memory. They already sell you the silicon. The only missing piece is the subscription, and Apple has never once in its corporate life failed to find the subscription model.
I’m so confident this is the long game that I’m certain a few friends of mine at Apple are reading this, silently nodding, and very deliberately not liking this post. Because a like is a signal, and a signal is a paper trail, and a paper trail is an NDA-shaped exit wound for the HR meeting. It’s okay y’all. It’s fine. I know. Blink twice next time you see me out at the bar.
Anyway, I digress…
Where we are today, with local models, is exactly where building a gaming PC was circa 2012.
You could do it. But it meant you had to know things. You installed drivers you didn’t know existed. You found weird plugins. You learned the difference between UEFI and BIOS, mostly at 1am, mostly against your will, after you couldn’t figure out why the CPU fans weren’t spinning. You learned what thermal throttling meant because your machine got progressively dumber the longer you used it. You flirted with overclocking.
You thought about water cooling. And then you did the thing every single one of us has done at least once, which is decide that a normal loop wasn’t enough and you needed a hardline loop, with the bent acrylic tubing and the colored coolant and the neon, and then somewhere in there you bought an at-home acid-etching kit so you could put a custom design on the glass side panel.
And when it was finally done, you stood back and understood, with total clarity, that you had built the coolest object in the history of objects.
And then your wife walked into the office, looked at it for about a second and a half, and said, “Oh. I guess I like the lights?”
That’s where we are. That’s this whole scene, right now. “Own your model” is going to be a weird hobbyist phase for a while, until the people out on the bleeding edge sand down all the rough spots and hand it to everyone else in a box that just works.
I’m not speaking from some lofty position here. As I write this, my Jetson AGX Orin is being deeply stupid. The models are loading and producing exactly zero output. Why? Because I was messing with the GGUF files. Was I supposed to be messing with the GGUF files? No. Did I do it anyway? Duh. I broke a working system because I was playing with it, which is why you can’t (and shouldn’t) do this unless you’re committed to learning a lot while you fail. (This isa lso a fairly complete summary of my entire engineering journey.)
So if that sounds fun to you: Cool. Genuinely. Have at it. Tinker. Break it. Fix it at 2am. If this gives you the same joy and frustration it gives me, dope. Go do it. 1 in 1,000 people will like this process.
But let’s be realistic about the other 999 of you reading this. You are going to keep opening Claude or ChatGPT, getting the answer, and going about your day, and you will be correct to do so. Engineers are going to keep being engineers, which mostly means voluntarily creating problems that we then get to solve.
If you’re reading this and you followed most of it, you’re probably one in a thousand.
Congratulations. You’re about to break something today. I hope you don’t suck at fixing it.