On-premise AI inference is on the rise. Can Australian enterprises make it work?
Unpacking the latest Uptime Institute insights and mapping real-world solutions for local conditions.
Developing large language models (LLMs) from scratch is a monumental undertaking. It demands specialised infrastructure, deep technical talent, and a clear path to differentiation. As Uptime Institute highlights in their recent blog, “Enterprises will deploy inference in-house - if they can” a decisive trend is emerging: very few enterprises will actually train their own frontier models. Instead, most will lean on commercial models and open-weight alternatives.
The real fork in the road comes after the model is chosen: where do you actually run the inference workload?
Uptime Intelligence’s latest report analyses the economics of AI inference across different data centre venues and lands on three critical observations. For Australian IT leaders, these observations carry extra weight - given our unique energy costs, climate challenges, and the rapid growth of local colocation hubs. Let’s break them down.
Observation 1: On-prem can be the lowest-cost option - but few can pull it off
Uptime points out that sustained, high hardware utilisation is the secret sauce for low-cost AI compute. Hyperscalers win on price largely through economies of scale and virtualisation that keep their IT infrastructure humming at maximum capacity.
To compete on cost, an enterprise must leverage its existing infrastructure footprint and internal technical capabilities while maintaining high GPU utilisation. Yet the reality is sobering: according to Uptime’s 2024 IT & Power Efficiency survey, 53% of respondents have no utilisation targets for their overall server fleets. Among those who do, only 29% report average utilisation above 65% - and 65% is the critical threshold where on-prem AI infrastructure becomes cost-effective.
For Australian businesses, this is particularly tricky. LLM inference workloads are notoriously unpredictable; they fluctuate based on hidden system prompts, the complexity of user inputs, and the number of tokens generated. This unpredictability often makes the public cloud more attractive, since you only pay for what you actually consume.
Observation 2: Model choice sometimes dictates infrastructure and vice versa
While latency, data sovereignty, governance, and operational control are all crucial, enterprises also face a hard reality: limited model portability.
If you plan to use flagship models from OpenAI or Anthropic, you can only consume them as a service or via cloud platforms. You simply cannot deploy them on-prem or in a local colocation facility. Furthermore, opting for custom inferencing hardware developed by cloud vendors locks you into that vendor’s cloud ecosystem.
The report stresses that decisions about hardware and data centre venues must not be made in isolation from LLM selection. Conversely, freely distributed open-weight models, paired with GPU servers from vendors like Dell, HPE, Lenovo, and Supermicro, offer the greatest deployment flexibility. For Australian organisations in regulated sectors (finance, government, healthcare), this flexibility is non-negotiable.
Observation 3: Smaller models are less capable, but drastically cheaper to run
The average rack power for enterprise IT in 2025 hovers around 7kW. But frontier AI clusters demand much more. If an AI cluster can’t fit into your existing data hall and requires a brand-new wing, any cost advantage over the public cloud evaporates instantly.
Here’s the silver lining: not every inference job needs a massive, liquid-cooled behemoth. Smaller, simpler LLMs—perfectly adequate for transcription, translation, summarisation, and basic Q&A - can be deployed within existing facilities with only minor modifications to cooling and power distribution. For enterprises starting small, utilising existing internal space or current colocation cabinets is far more attractive; even with moderate utilisation, the cost per token remains low.
This is likely why, despite the commercial pull of the cloud, Uptime Intelligence’s surveys still show on-prem data centres as the most popular destination for AI workloads.
“On-prem data centres as the most popular destination for AI workloads.”
Making it happen: Infrastructure solutions for every stage of your AI journey
Rather than letting infrastructure become the bottleneck, Australian enterprises can choose from a simplified, three-tiered approach that matches their AI ambition with the right physical foundation.
1. Inferencing and edge AI (testing, pilots, edge deployments)
For organisations starting small or rolling out AI at the edge, space and simplicity are paramount.
Vertiv™ SmartCabinet™ and Vertiv™ SmartRow™ provide fully enclosed, self-contained micro data centres that combine power, cooling, UPS, and monitoring in a compact footprint - ideal for office environments, remote sites, or proof-of-concept clusters.
The Vertiv™ 360AI high-density rack solution fits seamlessly into existing data centre halls, supporting incremental AI deployments without major facility overhauls.
2. Enterprise and data centre AI (AI labs, inference, training within white space)
Once AI moves into production within your main data centre, retrofitting existing space becomes the priority.
Vertiv’s pre-engineered retrofit designs scale from 70kW to 400kW per solution, with hybrid cooling options (air, liquid, or both) that adapt to your facility’s existing chilled water or air-cooled systems.
The Vertiv™ PowerNexus and Liebert® UPS family (including the EXM2, rated for 50°C ambient temperatures) allow you to scale power capacity without consuming additional floorspace or disrupting live workloads.
For thermal management, Liebert® DCD rear-door heat exchangers (up to 50kW per rack) and Direct-to-Chip CDUs handle dense GPU loads while preserving valuable white space.
3. Prefabricated modular AI data centres (large-scale training, full AI facilities)
When AI scales to model training or enterprise-wide production, prefabricated modular solutions offer the fastest, most predictable path to deployment.
The Vertiv™ SmartMod™ HDX delivers a fully equipped modular data centre - complete with power module, IT hall, and chiller skid—that can be deployed without disturbing existing IT workloads. Factory integration cuts deployment time by up to 50% and saves up to 25% in floorspace compared to traditional builds.
For even greater scale, Vertiv™ MegaMod HDX supports rack densities above 100kW and power capacities up to 10MW, combining direct-to-chip liquid cooling with air-cooled systems.
A locally manufactured alternative
For Australian enterprises prioritising local supply chains, ECANET’s bespoke modular data centres offer an Australian-built, ISO-certified alternative. Each modular unit - or series of interlinked modules - can be configured to integrate Vertiv components (or equipment of the customer’s choice), with a power train from 0.5MW to 2MW and cooling plant of the same capacity, supporting 6 to 100 racks in a scalable configuration. Crucially, ECANET’s solution is backed by a comprehensive service regime - from assessment and design through deployment, commissioning, and ongoing maintenance - to keep your AI infrastructure functioning at peak performance.
The bottom line
Uptime Institute’s insights make one thing crystal clear: Australian enterprises want to run inference on-premise, provided the infrastructure doesn’t become a bottleneck.
The “if” comes down to whether the existing facility can deliver the necessary power density, thermal efficiency, and operational simplicity - without breaking the bank. Whether you’re starting with a SmartCabinet for a proof-of-concept, retrofitting your data centre with Vertiv’s pre-engineered designs, or deploying a full-scale modular data centre from Vertiv or ECANET, the infrastructure is ready to make on-premise AI inference a reality.
With the right infrastructure partner, on-premise AI inference isn't just a technical possibility; it’s a strategic advantage. And in the race to adopt AI, Australian enterprises can’t afford to be left waiting for a cloud response when they could be inferring at the edge - right here, right now.
Ready to explore how Vertiv or ECANET can support your AI journey Down Under? Reach out for a site assessment and infrastructure roadmap tailored to your unique power, cooling, and deployment constraints.