AI that runs in your building: open weights, real hardware, honest limits

Three honest reasons, and one bad one

Running models on your own hardware costs more engineering effort than calling an API. There are three reasons that justify it and one that does not.

1. The data genuinely cannot leave. Contract terms, client confidentiality, a group policy, or a PDPA cross-border transfer you would rather not have to justify — and since the approved-country list was abolished in 2025, justifying one means writing your own transfer assessment and standing behind it for three years. This is the strongest reason and it is usually the real one. Note that it is a policy question, not a technical one — decide it with the people who own the policy.

Running the model in your building removes the transfer. It does not remove the rest. Where the model is deciding something about a person — scoring, screening, ranking, matching — Malaysia's April 2026 guideline on automated decision-making applies at any scale: say so in the privacy notice, keep a trained human in the loop rather than letting the model be the sole factor, and carry out an impact assessment before you switch it on. On-premise makes that easier to evidence, not unnecessary.

2. The cost per question has to be zero. Metered inference makes people ration themselves, and an assistant nobody uses casually returns nothing. Once the hardware exists, the marginal cost of the four-hundredth question is electricity. Marginal, not total — the machine and the person who keeps it running are paid for whether anyone asks anything or not, and at genuinely low volume a metered API is the cheaper invoice. The argument is about behaviour rather than arithmetic: what changes is that nobody hesitates before asking, and that is worth more than the sums suggest.

3. It must still work in three years. Hosted models are deprecated, repriced and re-tuned on someone else's schedule. A weights file on your own disk does exactly what it did the day you tested it. For anything embedded in a regulated or contractual process, that reproducibility is worth real money.

And the bad reason: because it will be better. It will not be, on the work that is actually hard. The gap has narrowed a great deal and keeps narrowing, but as of August 2026 it has narrowed unevenly. On everyday coding it has effectively closed, and on agentic command-line work the best open models sit within a point of the best hosted ones. On repository-scale coding, where one change has to be right across a dozen files at once, the hosted models still lead by a wide margin — and on reasoning-heavy work generally they still lead. That is the honest split, and a project sold on closing what is left of that gap is a project sold on the wrong thing. Run local because of where the data is and what the question costs, not because you expect to win a benchmark.

What the hardware actually costs

The number that surprises people is how small a useful first deployment is.

• Pilot, small team — the fast route. One workstation-class GPU with 24 GB of memory runs a capable quantised model at conversational speed for a handful of concurrent users. This is a machine under a desk, not a rack.
• Pilot, small team — the roomy route. A single box with 128 GB of memory shared between the CPU and an integrated GPU. Several brands now build these around the same idea, and they hold a model four or five times larger than the 24 GB card can — at a fraction of the speed.
• Department. One or two data-centre GPUs, batching properly through a serving stack, handles a department's daily use.
• Company-wide, or heavy document processing. Now it is a real server with real cooling and a real conversation about where it lives.

Those two pilot machines cost about the same and are not the same purchase, because capacity decides what you can run and bandwidth decides how fast it answers. A 24 GB card reads its memory at roughly 1,000 GB/s. The 128 GB boxes are specified at 250 to 275 GB/s and measure nearer 215 in real use — about a fifth of the speed, four to five times the room. So the card gives you a mid-sized model at reading pace, and the box gives you a model with three figures of billions of parameters at a pace closer to waiting for a colleague to reply. Neither is the better machine in the abstract. The question is whether your use case is one somebody sits and watches.

Mostly it is not. Overnight extraction across a folder of scanned delivery orders, a weekly pass over incident reports, a retrieval index rebuilt at 2 am — none of these care about tokens per second, and all of them get better answers from the larger model. Interactive question-and-answer at a desk is the case that does care, and it is the case the 24 GB card wins.

Since the heading promises a cost: as of August 2026 both pilot routes land between roughly RM 15,000 and RM 25,000 for a complete machine — the card as part of a workstation build, the box as a finished unit priced largely by brand. Department scale is several times that, once you count data-centre cards and the server around them. Treat all of those as this month's figures rather than a price list. The 2026 memory shortage has pushed consumer memory and GPU street prices far above list, with no expected relief before late 2027, and the 128 GB boxes are the most exposed to it because memory is most of what you are buying.

One thing the coverage rarely states plainly: the open models that closed the gap are not the models either pilot machine runs. The current leaders are trillion-parameter mixture-of-experts files — a model split into many sub-networks, with only a small fraction of it active on any one answer — and the weights alone run to hundreds of gigabytes. Both pilot machines sit two tiers below that. What they do run is a mid-sized model that is very good at extraction, retrieval, classification and drafting, which is exactly the work described below. It is not the model in the headline, and a proposal that quotes a leaderboard score beside a machine under a desk is quoting a number that machine cannot produce.

The other driver is concurrency, not model size. Forty people asking questions between nine and ten in the morning is a fundamentally different machine from one person asking all day. Size the pilot small on purpose, measure actual concurrent use for a month, then buy for the measurement rather than for the fantasy.

The serving layer is not exotic. vLLM where throughput matters, Ollama or llama.cpp where simplicity matters, an OpenAI-compatible endpoint in front of it either way so that nothing in your applications is coupled to the choice. Swapping the model behind that endpoint should be a configuration change, and if it is not, the integration is wrong.

Start with retrieval, not with chat

The highest-return first deployment is almost never a chatbot. It is search over your own documents.

Every established company sits on a decade of quotations, contracts, specifications, method statements, incident reports and correspondence. All of it is technically searchable and practically not, because filename search requires you to already know the answer.

Retrieval-augmented generation over that corpus answers "what did we quote Company X for a project like this in 2023" in seconds, with the source document attached. Three properties make it the right first project:

• It is verifiable. Every answer cites the document it came from, so a wrong answer is caught by the person reading it. That is what makes it safe to deploy to non-technical staff early.
• The value is immediate and obvious. Nobody needs convincing that finding a document in five seconds beats twenty minutes.
• It exercises the whole stack honestly. Ingestion, chunking, embeddings, retrieval quality, permissions. If retrieval works, everything downstream is easier; if it does not, you have learned that cheaply.

The unglamorous truth is that most of the work is in ingestion and permissions, not in the model. Scanned PDFs need OCR. Two hundred versions of the same specification need deduplicating. And a retrieval system that ignores your access rules will happily quote the payroll file to the wrong person — permissions belong in the retrieval layer, not in the prompt.

Agents, and the GUI agent question

An agent is a model given tools and a goal instead of a single question. The distinction that matters in practice is what it can touch.

Agents with proper interfaces. Where an API, a database or a protocol exists, the agent uses it. Structured, fast, testable, auditable. This is where nearly all real value sits, and it is the design we use inside Odoo — named intents with no destructive tool in the surface.

GUI agents. An agent that reads a screen and drives a keyboard and mouse, operating software that has no other interface. This is real, it works, and it is the right answer only when there is genuinely nothing better — a vendor system with no API, a government portal, a legacy application nobody can modify.

Three rules whenever a GUI agent is deployed, and they are not negotiable:

1. Its own account, with least privilege. Never a person's credentials. Everything it does must be attributable to it.
2. A hard stop before anything irreversible. Same principle as the ERP assistant: prepare, then hand to a human for the commit. Payments, submissions and confirmations are human actions.
3. A complete action log. If you cannot reconstruct what it did on a given afternoon, you cannot use it in any process that gets audited.

Treated as an unsupervised junior with a restricted account, a GUI agent is useful. Treated as automation, it eventually does something confidently wrong at scale.

What you give up by running it yourself

Said plainly, because a solutions page that lists only strengths is a brochure:

• Hard, long-horizon reasoning. Multi-step analysis where a single mistake invalidates the chain. This is where the remaining gap lives — narrower every quarter, and still real. Everyday coding has left this list. Repository-scale coding has not.
• Very long context. Improving quickly, still a gap, and the gap costs memory — which costs hardware.
• No easy reach for the web. A hosted model can simply be handed a search tool. On-premise, that tool means the query leaves the building even when the documents do not — so outside your own corpus the model answers from memory, and it does so confidently.
• Keeping up. Hosted models improve without you. A local deployment improves when someone plans a model refresh, tests it against your evaluation set, and rolls it out. Budget that quarterly, or the deployment silently ages.

The pragmatic pattern for most companies is a split: local for anything touching confidential data and for the high-volume routine work; hosted for the occasional hard question that contains nothing sensitive. Deciding which is which is a policy conversation, and it should be written down before either is switched on.

The engagement shape

1. Use-case selection. What would actually change if this worked. Most candidate ideas die here, correctly and cheaply.
2. Pilot. Modest hardware, one use case, an evaluation set of real examples with known answers, and a measurement — not a demonstration.
3. Production. Sized to measured concurrency, wired into the systems people already use, with permissions and logging.
4. Operate. Model refreshes, evaluation runs against the same set, and capacity as use grows.

If the pilot fails its evaluation, we say so and stop. A pilot that cannot fail was a purchase decision wearing a lab coat.

Questions