releases.shpreview
PulumiPulumi/Pulumi Blog

Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi

June 30, 2026Pulumi BlogView original ↗

Putting Ollama on a cloud GPU is something I keep coming back to. A while ago I wrote up running open-source LLMs on an AWS EC2 box with Ollama and Pulumi, and the shape never really changes: a GPU instance, a model server, and a firewall rule in front. Infrastructure as code earned its place by making that kind of setup predictable and repeatable, and AI infrastructure is no exception. A GPU box serving a model is still a VM, a disk, and a firewall rule, and it should be declared like one.

Thorsten Hans made exactly that case in his Akamai post, Fully Automated AI Infrastructures with Terraform and Akamai Cloud, which stands up a single GPU instance on Linode, installs the drivers, runs Ollama, and pulls a model, with no manual steps after terraform apply.

I liked the shape of it, so this post ports the same idea to Pulumi and runs it across AWS, Azure, and Google Cloud instead of one. The result is one program shape per cloud: a single pulumi up brings up a GPU box that installs its own driver, runs Ollama, and pulls a model with no manual steps, and a single pulumi destroy takes it back down. Along the way it drops the two imperative bits the Terraform version leans on: a static access token sitting in an environment variable, and a null_resource running a shell loop to wait for the model. The first becomes an OIDC login from a Pulumi ESC environment, so no long-lived key lives anywhere. The second turns out not to be a resource at all.

What you are building

Strip away the per-cloud naming and every version of this is the same three things: a GPU virtual machine, a firewall in front of it, and a cloud-init script that turns a bare Ubuntu box into a running inference server. The model serving runs on Ollama, which exposes an HTTP API on port 11434 and keeps the model resident in GPU memory between requests.

flowchart LR
Dev([Your machine / curl]) -->|"HTTP :11434"| FW["Firewall / security group<br/>(allow 11434, optional 22)"]
FW --> VM["GPU VM (Ubuntu 24.04)<br/>NVIDIA driver + Ollama"]
VM -->|"resident in GPU memory"| Model["qwen2.5:14b"]
ESC["Pulumi ESC<br/>pulumi-idp/auth"] -.->|"OIDC login (short-lived creds)"| Pulumi["pulumi up"]
Pulumi -.->|"AWS / Azure / GCP"| VM

Component

Port

Description

GPU VM

-

Ubuntu 24.04 with an NVIDIA T4-class GPU. Installs the driver and Ollama from cloud-init, then pulls the model.

Ollama

11434

Serves the model over an HTTP API and keeps it in GPU memory between calls.

Firewall / security group

-

Allows inbound 11434 (and 22 when you ask for it). Open to the world for a demo; lock the CIDR down for anything real.

pulumi-idp/auth (ESC)

-

One environment that brokers an OIDC login into AWS, Azure, and GCP. No static keys in code or CI.

One detail here is worth pausing on, and it explains why the Pulumi version comes out shorter than the Terraform one. The Akamai project ends with a null_resource that runs a curl loop in local-exec to block terraform apply until the model finishes downloading. That is not infrastructure but a runtime check wearing a resource costume. Pulumi has a Command provider that would let you reproduce it line for line, and this post deliberately does not. The program provisions the box, the box pulls the model on its own, and the program prints the endpoint. Whether the model has finished downloading yet is a question you answer with a curl, not a question your IaC tool should be holding a deployment open to ask.

Prerequisites

Before getting started, ensure you have:

  • Pulumi CLI installed and configured
  • A Pulumi Cloud account (ESC and OIDC live here)
  • An account on at least one of AWS, Azure, or Google Cloud, with an OIDC trust set up for Pulumi (covered below)
  • Enough GPU quota in your target region for one T4-class instance; a fresh account often starts at zero, which is the most common reason a first deploy fails, so check it before you run pulumi up
  • Node.js 18+ for the TypeScript program
  • curl to talk to the inference endpoint once it comes up

This post serves the qwen2.5:14b model, the same one the Akamai article uses, because the 4-bit quant fits comfortably on a single 16 GB T4. The model is one config value (model), so swap in anything Ollama supports; match the GPU to the model’s memory footprint, because a model that overflows VRAM still runs but spills onto the CPU and crawls.

Credentials without the copy-paste

The Terraform version authenticates the only way a single-cloud demo can: you mint a personal access token, export it as LINODE_TOKEN, and the provider reads it from the environment. It works, but that token is long-lived, it sits in your shell history and your CI secrets, and you mint one per cloud. Across three clouds that is three static keys to rotate and worry about.

Pulumi ESC (Environments, Secrets, and Configuration) replaces it all with an OIDC login. The idea: instead of storing a cloud key, you store a trust relationship. At deploy time, ESC presents a short-lived OIDC token to AWS, Azure, or Google Cloud, and each one hands back temporary credentials scoped to a role you control. Nothing long-lived is ever written down.

I keep that wiring in one environment, pulumi-idp/auth, and every stack imports it. Here is the whole thing:

<span class="line"><span class="cl"><span class="c"># pulumi-idp/auth: one ESC environment that brokers an OIDC login into all three</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># clouds. Every stack imports it. No static cloud key lives here or anywhere else.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">values</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">aws</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn::open::aws-login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">oidc</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">roleArn</span><span class="p">:</span><span class="w"> </span><span class="l">arn:aws:iam::123456789012:role/pulumi-esc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">sessionName</span><span class="p">:</span><span class="w"> </span><span class="l">pulumi-esc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">azure</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn::open::azure-login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">clientId</span><span class="p">:</span><span class="w"> </span><span class="l">aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">tenantId</span><span class="p">:</span><span class="w"> </span><span class="l">aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">subscriptionId</span><span class="p">:</span><span class="w"> </span><span class="l">/subscriptions/00000000-0000-0000-0000-000000000000</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">oidc</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">gcp</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">fn::open::gcp-login</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">project</span><span class="p">:</span><span class="w"> </span><span class="m">123456789012</span><span class="w"> </span><span class="c"># numeric project number, not the project ID</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">oidc</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">workloadPoolId</span><span class="p">:</span><span class="w"> </span><span class="l">pulumi-esc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">providerId</span><span class="p">:</span><span class="w"> </span><span class="l">pulumi-esc</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">serviceAccount</span><span class="p">:</span><span class="w"> </span><span class="l">pulumi-esc@my-project.iam.gserviceaccount.com</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">environmentVariables</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># AWS: read by the Pulumi AWS provider, the AWS SDKs, and the aws CLI</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">AWS_ACCESS_KEY_ID</span><span class="p">:</span><span class="w"> </span><span class="l">${aws.login.accessKeyId}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">AWS_SECRET_ACCESS_KEY</span><span class="p">:</span><span class="w"> </span><span class="l">${aws.login.secretAccessKey}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">AWS_SESSION_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${aws.login.sessionToken}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Azure: read by the azure-native provider</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ARM_USE_OIDC</span><span class="p">:</span><span class="w"> </span><span class="s2">"true"</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ARM_CLIENT_ID</span><span class="p">:</span><span class="w"> </span><span class="l">${azure.login.clientId}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ARM_TENANT_ID</span><span class="p">:</span><span class="w"> </span><span class="l">${azure.login.tenantId}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ARM_SUBSCRIPTION_ID</span><span class="p">:</span><span class="w"> </span><span class="l">${azure.login.subscriptionId}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">ARM_OIDC_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${azure.login.oidc.token}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Google Cloud: read by the Pulumi Google Cloud provider</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">GOOGLE_CLOUD_PROJECT</span><span class="p">:</span><span class="w"> </span><span class="l">${gcp.login.project}</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">GOOGLE_OAUTH_ACCESS_TOKEN</span><span class="p">:</span><span class="w"> </span><span class="l">${gcp.login.accessToken}</span><span class="w">
</span></span></span>

You only need the blocks for the clouds you actually deploy to. Opening this environment makes ESC perform a live OIDC login for each cloud listed, so trim it to the one you use, or keep all three if, like me, you bounce between them.

Each cloud needs a one-time trust setup so this login is allowed at all: an IAM OIDC identity provider and role on AWS, a federated credential on an Azure app registration, and a Workload Identity Pool on Google Cloud. You do that once; after that, pulumi-idp/auth is the only thing any stack references for credentials.

A stack opts into it with one block in its stack config. The AWS stack’s Pulumi.aws.yaml:

<span class="line"><span class="cl"><span class="c"># Pulumi.aws.yaml</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">environment</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">pulumi-idp/auth</span><span class="w">
</span></span></span>

That is the entire credential story. No keys in the program, no keys in CI, nothing to rotate. The program below never mentions a secret; it only creates resources, and the ambient credentials from pulumi-idp/auth carry the request.

One cloud-init, three clouds

The part that turns a bare Ubuntu box into an inference server is identical on every cloud, so it lives in one file, cloud-init.yaml, that all three programs read. It follows the Akamai cloud-config closely, with a couple of robustness tweaks for a multi-cloud run, and it runs in a deliberate order because the GPU driver needs a reboot before Ollama can see the card:

  1. Update packages and install the kernel headers and ubuntu-drivers.
  2. Install the NVIDIA driver, then reboot so the kernel loads it.
  3. On the next boot, a one-shot systemd service installs Ollama, binds it to 0.0.0.0:11434, and pulls the model.
  4. The service disables itself, so it never runs again.
<span class="line"><span class="cl"><span class="c">#cloud-config</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># Zero-touch Ollama GPU box. Cloud-agnostic: no provider metadata, no static creds.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># The Pulumi program substitutes the model name on the `ollama pull` line below</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c"># before this is passed as user-data (AWS/GCP) or custom-data (Azure).</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">write_files</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Make Ollama listen on every interface and never unload the model from VRAM.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/systemd/system/ollama.service.d/override.conf</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">content</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> [Service]
</span></span></span><span class="line"><span class="cl"><span class="sd"> Environment="OLLAMA_HOST=0.0.0.0:11434"
</span></span></span><span class="line"><span class="cl"><span class="sd"> Environment="OLLAMA_KEEP_ALIVE=-1"</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># Runs once on the post-reboot boot, after the GPU driver is loaded.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/local/bin/ollama-setup.sh</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">permissions</span><span class="p">:</span><span class="w"> </span><span class="s1">'0755'</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">content</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> #!/usr/bin/env bash
</span></span></span><span class="line"><span class="cl"><span class="sd"> set -euxo pipefail
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> # Install Ollama; the installer creates and starts the ollama systemd unit.
</span></span></span><span class="line"><span class="cl"><span class="sd"> curl -fsSL https://ollama.com/install.sh | sh
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> # Pick up the OLLAMA_HOST / OLLAMA_KEEP_ALIVE override written above.
</span></span></span><span class="line"><span class="cl"><span class="sd"> systemctl daemon-reload
</span></span></span><span class="line"><span class="cl"><span class="sd"> systemctl enable ollama.service
</span></span></span><span class="line"><span class="cl"><span class="sd"> systemctl restart ollama.service
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> # Wait for the daemon to bind its socket before pulling. This is the box
</span></span></span><span class="line"><span class="cl"><span class="sd"> # waiting on its OWN local daemon, not an external readiness gate.
</span></span></span><span class="line"><span class="cl"><span class="sd"> until curl -fsS http://127.0.0.1:11434/api/tags >/dev/null 2>&1; do
</span></span></span><span class="line"><span class="cl"><span class="sd"> sleep 2
</span></span></span><span class="line"><span class="cl"><span class="sd"> done
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> # Pre-pull the model so the endpoint answers on the very first request.
</span></span></span><span class="line"><span class="cl"><span class="sd"> ollama pull __MODEL__
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> # One-shot: never run again on future boots.
</span></span></span><span class="line"><span class="cl"><span class="sd"> systemctl disable ollama-setup.service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/systemd/system/ollama-setup.service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">content</span><span class="p">:</span><span class="w"> </span><span class="p">|</span><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> [Unit]
</span></span></span><span class="line"><span class="cl"><span class="sd"> Description=One-time Ollama install and model pull
</span></span></span><span class="line"><span class="cl"><span class="sd"> After=network-online.target
</span></span></span><span class="line"><span class="cl"><span class="sd"> Wants=network-online.target
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> [Service]
</span></span></span><span class="line"><span class="cl"><span class="sd"> Type=oneshot
</span></span></span><span class="line"><span class="cl"><span class="sd"> ExecStart=/usr/local/bin/ollama-setup.sh
</span></span></span><span class="line"><span class="cl"><span class="sd"> RemainAfterExit=true
</span></span></span><span class="line"><span class="cl"><span class="sd"> # The model pull runs for several minutes; without this, systemd's default
</span></span></span><span class="line"><span class="cl"><span class="sd"> # 90s start timeout kills the service mid-download and the model never lands.
</span></span></span><span class="line"><span class="cl"><span class="sd"> TimeoutStartSec=0
</span></span></span><span class="line"><span class="cl"><span class="sd">
</span></span></span><span class="line"><span class="cl"><span class="sd"> [Install]
</span></span></span><span class="line"><span class="cl"><span class="sd"> WantedBy=multi-user.target</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">runcmd</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">apt-get update</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">DEBIAN_FRONTEND=noninteractive apt-get -y upgrade</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># linux-headers-generic also covers the kernel the upgrade above may have pulled</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># in, so DKMS builds the NVIDIA module for the kernel that boots next.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">DEBIAN_FRONTEND=noninteractive apt-get install -y "linux-headers-$(uname -r)" linux-headers-generic ubuntu-drivers-common</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># --gpgpu selects the headless server driver branch, the right one for a compute</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="c"># GPU like the T4; bare `ubuntu-drivers install` would pull the desktop stack.</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">ubuntu-drivers install --gpgpu</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span>- <span class="l">systemctl enable ollama-setup.service</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="nt">power_state</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="l">reboot</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">message</span><span class="p">:</span><span class="w"> </span><span class="l">Rebooting to load the NVIDIA driver before Ollama setup</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"> </span><span class="nt">condition</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span>

The model name is the literal token __MODEL__. The Pulumi program reads this file at deploy time and substitutes your model config value before handing it to the instance. Two environment settings are doing quiet but important work: OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on every interface instead of localhost alone, and OLLAMA_KEEP_ALIVE=-1 keeps the model pinned in GPU memory so only the first request pays the load cost.

The programs

Every program has the same five beats: read the config, template the cloud-init.yaml, open the inbound ports, launch the GPU instance with that cloud-init as its user data, and export the endpoint. What differs is only the dialect each cloud speaks for “GPU instance” and “firewall rule.”

The shared contract keeps the three listings legible side by side: the same model and allowSsh config keys, the same cloud-init.yaml, and the same four exports (publicIp, ollamaEndpoint, generateEndpoint, tagsEndpoint). Pick your cloud:

<span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">pulumi</span> <span class="kr">from</span> <span class="s2">"@pulumi/pulumi"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">aws</span> <span class="kr">from</span> <span class="s2">"@pulumi/aws"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">fs</span> <span class="kr">from</span> <span class="s2">"fs"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">cfg</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">Config</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">model</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"model"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"qwen2.5:14b"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">allowSsh</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="nx">getBoolean</span><span class="p">(</span><span class="s2">"allowSsh"</span><span class="p">)</span> <span class="o">??</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">region</span> <span class="o">=</span> <span class="p">(</span><span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"region"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"us-east-1"</span><span class="p">)</span> <span class="kr">as</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">Region</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">userData</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s2">"cloud-init.yaml"</span><span class="p">,</span> <span class="s2">"utf8"</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/__MODEL__/g</span><span class="p">,</span> <span class="nx">model</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Region comes from config; credentials arrive ambiently from pulumi-idp/auth.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">provider</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">Provider</span><span class="p">(</span><span class="s2">"aws"</span><span class="p">,</span> <span class="p">{</span> <span class="nx">region</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Most recent Ubuntu 24.04 LTS (amd64, hvm, gp3) published by Canonical.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ubuntu</span> <span class="o">=</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">ec2</span><span class="p">.</span><span class="nx">getAmi</span><span class="p">({</span>
</span></span><span class="line"><span class="cl"> <span class="nx">mostRecent</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">owners</span><span class="o">:</span> <span class="p">[</span><span class="s2">"099720109477"</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">filters</span><span class="o">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">"name"</span><span class="p">,</span> <span class="nx">values</span><span class="o">:</span> <span class="p">[</span><span class="s2">"ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"</span><span class="p">]</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">{</span> <span class="nx">name</span><span class="o">:</span> <span class="s2">"virtualization-type"</span><span class="p">,</span> <span class="nx">values</span><span class="o">:</span> <span class="p">[</span><span class="s2">"hvm"</span><span class="p">]</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">{</span> <span class="nx">provider</span> <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ingress</span>: <span class="kt">aws.types.input.ec2.SecurityGroupIngress</span><span class="p">[]</span> <span class="o">=</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">description</span><span class="o">:</span> <span class="s2">"Ollama HTTP API. NOTE: restrict cidrBlocks to your own range in production."</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fromPort</span>: <span class="kt">11434</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">toPort</span>: <span class="kt">11434</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"tcp"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">cidrBlocks</span><span class="o">:</span> <span class="p">[</span><span class="s2">"0.0.0.0/0"</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">}];</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="p">(</span><span class="nx">allowSsh</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">ingress</span><span class="p">.</span><span class="nx">push</span><span class="p">({</span>
</span></span><span class="line"><span class="cl"> <span class="nx">description</span><span class="o">:</span> <span class="s2">"SSH"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fromPort</span>: <span class="kt">22</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">toPort</span>: <span class="kt">22</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"tcp"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">cidrBlocks</span><span class="o">:</span> <span class="p">[</span><span class="s2">"0.0.0.0/0"</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="p">});</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">sg</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">ec2</span><span class="p">.</span><span class="nx">SecurityGroup</span><span class="p">(</span><span class="s2">"ollama"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">description</span><span class="o">:</span> <span class="s2">"Ollama inferencing access"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">ingress</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">egress</span><span class="o">:</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">description</span><span class="o">:</span> <span class="s2">"Allow all outbound"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">fromPort</span>: <span class="kt">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">toPort</span>: <span class="kt">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"-1"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">cidrBlocks</span><span class="o">:</span> <span class="p">[</span><span class="s2">"0.0.0.0/0"</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">provider</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">aws</span><span class="p">.</span><span class="nx">ec2</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="s2">"ollama"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">ami</span>: <span class="kt">ubuntu.then</span><span class="p">(</span><span class="nx">a</span> <span class="o">=></span> <span class="nx">a</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="nx">instanceType</span><span class="o">:</span> <span class="s2">"g4dn.xlarge"</span><span class="p">,</span> <span class="c1">// 1x NVIDIA T4
</span></span></span><span class="line"><span class="cl"> <span class="nx">vpcSecurityGroupIds</span><span class="o">:</span> <span class="p">[</span><span class="nx">sg</span><span class="p">.</span><span class="nx">id</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">associatePublicIpAddress</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">userData</span><span class="p">,</span> <span class="c1">// plain text; the AWS provider base64-encodes it for you
</span></span></span><span class="line"><span class="cl"> <span class="nx">rootBlockDevice</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">volumeSize</span>: <span class="kt">40</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">volumeType</span><span class="o">:</span> <span class="s2">"gp3"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">tags</span><span class="o">:</span> <span class="p">{</span> <span class="nx">Name</span><span class="o">:</span> <span class="s2">"ollama"</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">provider</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">publicIp</span> <span class="o">=</span> <span class="nx">server</span><span class="p">.</span><span class="nx">publicIp</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">ollamaEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">generateEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/generate`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">tagsEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/tags`</span><span class="p">;</span>
</span></span>
<span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">pulumi</span> <span class="kr">from</span> <span class="s2">"@pulumi/pulumi"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">gcp</span> <span class="kr">from</span> <span class="s2">"@pulumi/gcp"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">fs</span> <span class="kr">from</span> <span class="s2">"fs"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">cfg</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">Config</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">model</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"model"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"qwen2.5:14b"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">allowSsh</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="nx">getBoolean</span><span class="p">(</span><span class="s2">"allowSsh"</span><span class="p">)</span> <span class="o">??</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">project</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"project"</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">zone</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"zone"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"us-central1-a"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">userData</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s2">"cloud-init.yaml"</span><span class="p">,</span> <span class="s2">"utf8"</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/__MODEL__/g</span><span class="p">,</span> <span class="nx">model</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Latest Ubuntu 24.04 LTS amd64 image.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ubuntu</span> <span class="o">=</span> <span class="nx">gcp</span><span class="p">.</span><span class="nx">compute</span><span class="p">.</span><span class="nx">getImage</span><span class="p">({</span>
</span></span><span class="line"><span class="cl"> <span class="nx">family</span><span class="o">:</span> <span class="s2">"ubuntu-2404-lts-amd64"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">project</span><span class="o">:</span> <span class="s2">"ubuntu-os-cloud"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Network tag binds the firewall rule to this instance.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">networkTag</span> <span class="o">=</span> <span class="s2">"ollama"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">firewall</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">gcp</span><span class="p">.</span><span class="nx">compute</span><span class="p">.</span><span class="nx">Firewall</span><span class="p">(</span><span class="s2">"ollama-fw"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">network</span><span class="o">:</span> <span class="s2">"default"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">project</span>: <span class="kt">project</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="c1">// Inbound to Ollama from anywhere; restrict this CIDR for production.
</span></span></span><span class="line"><span class="cl"> <span class="nx">allows</span>: <span class="kt">allowSsh</span>
</span></span><span class="line"><span class="cl"> <span class="o">?</span> <span class="p">[{</span> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"tcp"</span><span class="p">,</span> <span class="nx">ports</span><span class="o">:</span> <span class="p">[</span><span class="s2">"11434"</span><span class="p">]</span> <span class="p">},</span> <span class="p">{</span> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"tcp"</span><span class="p">,</span> <span class="nx">ports</span><span class="o">:</span> <span class="p">[</span><span class="s2">"22"</span><span class="p">]</span> <span class="p">}]</span>
</span></span><span class="line"><span class="cl"> <span class="o">:</span> <span class="p">[{</span> <span class="nx">protocol</span><span class="o">:</span> <span class="s2">"tcp"</span><span class="p">,</span> <span class="nx">ports</span><span class="o">:</span> <span class="p">[</span><span class="s2">"11434"</span><span class="p">]</span> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sourceRanges</span><span class="o">:</span> <span class="p">[</span><span class="s2">"0.0.0.0/0"</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">targetTags</span><span class="o">:</span> <span class="p">[</span><span class="nx">networkTag</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">instance</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">gcp</span><span class="p">.</span><span class="nx">compute</span><span class="p">.</span><span class="nx">Instance</span><span class="p">(</span><span class="s2">"ollama"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">machineType</span><span class="o">:</span> <span class="s2">"n1-standard-4"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">zone</span>: <span class="kt">zone</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">project</span>: <span class="kt">project</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">tags</span><span class="o">:</span> <span class="p">[</span><span class="nx">networkTag</span><span class="p">],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">bootDisk</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">initializeParams</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">image</span>: <span class="kt">ubuntu.then</span><span class="p">(</span><span class="nx">i</span> <span class="o">=></span> <span class="nx">i</span><span class="p">.</span><span class="nx">selfLink</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"> <span class="nx">size</span>: <span class="kt">40</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">guestAccelerators</span><span class="o">:</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="kr">type</span><span class="o">:</span> <span class="s2">"nvidia-tesla-t4"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">count</span>: <span class="kt">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"> <span class="c1">// GPUs cannot live-migrate, so host maintenance must terminate (and restart) the VM.
</span></span></span><span class="line"><span class="cl"> <span class="nx">scheduling</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">onHostMaintenance</span><span class="o">:</span> <span class="s2">"TERMINATE"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">automaticRestart</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">networkInterfaces</span><span class="o">:</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">network</span><span class="o">:</span> <span class="s2">"default"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">accessConfigs</span><span class="o">:</span> <span class="p">[{}],</span> <span class="c1">// an empty config requests an ephemeral public IP
</span></span></span><span class="line"><span class="cl"> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"> <span class="nx">metadata</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="s2">"user-data"</span><span class="o">:</span> <span class="nx">userData</span><span class="p">,</span> <span class="c1">// Ubuntu cloud-init reads this key, not startup-script
</span></span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">dependsOn</span>: <span class="kt">firewall</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">publicIp</span> <span class="o">=</span> <span class="nx">instance</span><span class="p">.</span><span class="nx">networkInterfaces</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">nics</span> <span class="o">=></span> <span class="nx">nics</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">accessConfigs</span><span class="o">!</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="nx">natIp</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">ollamaEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">generateEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/generate`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">tagsEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/tags`</span><span class="p">;</span>
</span></span>
<span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">pulumi</span> <span class="kr">from</span> <span class="s2">"@pulumi/pulumi"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">resources</span> <span class="kr">from</span> <span class="s2">"@pulumi/azure-native/resources"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">network</span> <span class="kr">from</span> <span class="s2">"@pulumi/azure-native/network"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">compute</span> <span class="kr">from</span> <span class="s2">"@pulumi/azure-native/compute"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">random</span> <span class="kr">from</span> <span class="s2">"@pulumi/random"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="o">*</span> <span class="kr">as</span> <span class="nx">fs</span> <span class="kr">from</span> <span class="s2">"fs"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">cfg</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">Config</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">model</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"model"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"qwen2.5:14b"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">allowSsh</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="nx">getBoolean</span><span class="p">(</span><span class="s2">"allowSsh"</span><span class="p">)</span> <span class="o">??</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">location</span> <span class="o">=</span> <span class="nx">cfg</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">"location"</span><span class="p">)</span> <span class="o">??</span> <span class="s2">"eastus"</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">userData</span> <span class="o">=</span> <span class="nx">fs</span><span class="p">.</span><span class="nx">readFileSync</span><span class="p">(</span><span class="s2">"cloud-init.yaml"</span><span class="p">,</span> <span class="s2">"utf8"</span><span class="p">).</span><span class="nx">replace</span><span class="p">(</span><span class="sr">/__MODEL__/g</span><span class="p">,</span> <span class="nx">model</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">resourceGroup</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">resources</span><span class="p">.</span><span class="nx">ResourceGroup</span><span class="p">(</span><span class="s2">"ollama-rg"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">location</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">vnet</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">network</span><span class="p">.</span><span class="nx">VirtualNetwork</span><span class="p">(</span><span class="s2">"ollama-vnet"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">addressSpace</span><span class="o">:</span> <span class="p">{</span> <span class="nx">addressPrefixes</span><span class="o">:</span> <span class="p">[</span><span class="s2">"10.0.0.0/16"</span><span class="p">]</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">subnet</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">network</span><span class="p">.</span><span class="nx">Subnet</span><span class="p">(</span><span class="s2">"ollama-subnet"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">virtualNetworkName</span>: <span class="kt">vnet.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">addressPrefix</span><span class="o">:</span> <span class="s2">"10.0.1.0/24"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Standard SKU public IPs use static allocation.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">publicIpAddress</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">network</span><span class="p">.</span><span class="nx">PublicIPAddress</span><span class="p">(</span><span class="s2">"ollama-pip"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sku</span><span class="o">:</span> <span class="p">{</span> <span class="nx">name</span>: <span class="kt">network.PublicIPAddressSkuName.Standard</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">publicIPAllocationMethod</span>: <span class="kt">network.IPAllocationMethod.Static</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Azure permits all outbound by default, so only inbound rules are needed.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">nsg</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">network</span><span class="p">.</span><span class="nx">NetworkSecurityGroup</span><span class="p">(</span><span class="s2">"ollama-nsg"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">securityRules</span><span class="o">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl"> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">"allow-ollama"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">priority</span>: <span class="kt">1000</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">direction</span>: <span class="kt">network.SecurityRuleDirection.Inbound</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">access</span>: <span class="kt">network.SecurityRuleAccess.Allow</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">protocol</span>: <span class="kt">network.SecurityRuleProtocol.Tcp</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sourcePortRange</span><span class="o">:</span> <span class="s2">"*"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">destinationPortRange</span><span class="o">:</span> <span class="s2">"11434"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sourceAddressPrefix</span><span class="o">:</span> <span class="s2">"0.0.0.0/0"</span><span class="p">,</span> <span class="c1">// prod: restrict to your client CIDR
</span></span></span><span class="line"><span class="cl"> <span class="nx">destinationAddressPrefix</span><span class="o">:</span> <span class="s2">"*"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">...(</span><span class="nx">allowSsh</span> <span class="o">?</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">"allow-ssh"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">priority</span>: <span class="kt">1001</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">direction</span>: <span class="kt">network.SecurityRuleDirection.Inbound</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">access</span>: <span class="kt">network.SecurityRuleAccess.Allow</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">protocol</span>: <span class="kt">network.SecurityRuleProtocol.Tcp</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sourcePortRange</span><span class="o">:</span> <span class="s2">"*"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">destinationPortRange</span><span class="o">:</span> <span class="s2">"22"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sourceAddressPrefix</span><span class="o">:</span> <span class="s2">"0.0.0.0/0"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">destinationAddressPrefix</span><span class="o">:</span> <span class="s2">"*"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">}]</span> <span class="o">:</span> <span class="p">[]),</span>
</span></span><span class="line"><span class="cl"> <span class="p">],</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">nic</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">network</span><span class="p">.</span><span class="nx">NetworkInterface</span><span class="p">(</span><span class="s2">"ollama-nic"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">networkSecurityGroup</span><span class="o">:</span> <span class="p">{</span> <span class="nx">id</span>: <span class="kt">nsg.id</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">ipConfigurations</span><span class="o">:</span> <span class="p">[{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">"ipconfig1"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">subnet</span><span class="o">:</span> <span class="p">{</span> <span class="nx">id</span>: <span class="kt">subnet.id</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">publicIPAddress</span><span class="o">:</span> <span class="p">{</span> <span class="nx">id</span>: <span class="kt">publicIpAddress.id</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">privateIPAllocationMethod</span>: <span class="kt">network.IPAllocationMethod.Dynamic</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">primary</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Azure requires an admin credential even though we never log in. Generate one
</span></span></span><span class="line"><span class="cl"><span class="c1">// instead of hard-coding it; it stays a Pulumi secret and is never exported.
</span></span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">adminPassword</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">random</span><span class="p">.</span><span class="nx">RandomPassword</span><span class="p">(</span><span class="s2">"ollama-admin-password"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">length</span>: <span class="kt">24</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">special</span>: <span class="kt">true</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">overrideSpecial</span><span class="o">:</span> <span class="s2">"!#$%*"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">minLower</span>: <span class="kt">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">minUpper</span>: <span class="kt">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">minNumeric</span>: <span class="kt">1</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">minSpecial</span>: <span class="kt">1</span><span class="p">,</span> <span class="c1">// Azure requires 3 of 4 character classes
</span></span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">vm</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">compute</span><span class="p">.</span><span class="nx">VirtualMachine</span><span class="p">(</span><span class="s2">"ollama-vm"</span><span class="p">,</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">resourceGroupName</span>: <span class="kt">resourceGroup.name</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">hardwareProfile</span><span class="o">:</span> <span class="p">{</span> <span class="nx">vmSize</span><span class="o">:</span> <span class="s2">"Standard_NC4as_T4_v3"</span> <span class="p">},</span> <span class="c1">// 1x NVIDIA T4
</span></span></span><span class="line"><span class="cl"> <span class="nx">networkProfile</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">networkInterfaces</span><span class="o">:</span> <span class="p">[{</span> <span class="nx">id</span>: <span class="kt">nic.id</span><span class="p">,</span> <span class="nx">primary</span>: <span class="kt">true</span> <span class="p">}],</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">osProfile</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">computerName</span><span class="o">:</span> <span class="s2">"ollama"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">adminUsername</span><span class="o">:</span> <span class="s2">"azureuser"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">adminPassword</span>: <span class="kt">adminPassword.result</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">customData</span>: <span class="kt">Buffer.from</span><span class="p">(</span><span class="nx">userData</span><span class="p">).</span><span class="nx">toString</span><span class="p">(</span><span class="s2">"base64"</span><span class="p">),</span> <span class="c1">// Azure wants base64
</span></span></span><span class="line"><span class="cl"> <span class="nx">linuxConfiguration</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">disablePasswordAuthentication</span>: <span class="kt">false</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">storageProfile</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">imageReference</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">publisher</span><span class="o">:</span> <span class="s2">"Canonical"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">offer</span><span class="o">:</span> <span class="s2">"ubuntu-24_04-lts"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">sku</span><span class="o">:</span> <span class="s2">"server"</span><span class="p">,</span> <span class="c1">// Gen2; NC4as_T4_v3 is a Gen2 size
</span></span></span><span class="line"><span class="cl"> <span class="nx">version</span><span class="o">:</span> <span class="s2">"latest"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="nx">osDisk</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nx">name</span><span class="o">:</span> <span class="s2">"ollama-osdisk"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">createOption</span><span class="o">:</span> <span class="s2">"FromImage"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">diskSizeGB</span>: <span class="kt">40</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nx">managedDisk</span><span class="o">:</span> <span class="p">{</span> <span class="nx">storageAccountType</span><span class="o">:</span> <span class="s2">"StandardSSD_LRS"</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"> <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// A Standard/Static public IP is reserved at creation, so its address is known
</span></span></span><span class="line"><span class="cl"><span class="c1">// once the resource exists; no separate lookup needed.
</span></span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">publicIp</span> <span class="o">=</span> <span class="nx">publicIpAddress</span><span class="p">.</span><span class="nx">ipAddress</span><span class="p">.</span><span class="nx">apply</span><span class="p">(</span><span class="nx">ip</span> <span class="o">=></span> <span class="nx">ip</span><span class="o">!</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">ollamaEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">generateEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/generate`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kr">const</span> <span class="nx">tagsEndpoint</span> <span class="o">=</span> <span class="nx">pulumi</span><span class="p">.</span><span class="nx">interpolate</span><span class="sb">`http://</span><span class="si">${</span><span class="nx">publicIp</span><span class="si">}</span><span class="sb">:11434/api/tags`</span><span class="p">;</span>
</span></span>

A few per-cloud details are worth calling out, since they are the places the “same program” abstraction leaks:

  • AWS is the shortest program, because the GPU comes with the instance shape: a g4dn.xlarge is a T4 box, so there is no separate accelerator to attach. The one thing to do before you deploy is raise the Running On-Demand G and VT instances vCPU quota in your region; a fresh account starts at zero, and pulumi up fails with VcpuLimitExceeded until you do.
  • Google Cloud attaches the GPU explicitly with guestAccelerators, and that brings the rule that trips people up: a GPU instance cannot live-migrate, so scheduling.onHostMaintenance must be "TERMINATE" or the apply is rejected. The cloud-init also has to ride on the user-data metadata key, not startup-script, and the empty accessConfigs: [{}] is what hands the box a public IP. T4 quota is also zero on a new project.
  • Azure is the longest listing, because the network is à la carte: the resource group, virtual network, subnet, public IP, security group, and NIC are each their own resource before you reach the VM. One detail surprises people: a Linux VM requires an admin credential even when you never log in, so the program generates a throwaway password with random.RandomPassword rather than committing one. The NC4as_T4_v3 is a compute GPU, so the standard server (CUDA) driver from the cloud-init is the right choice here; the GRID driver is only needed for GPU-accelerated visualization workloads.

The full programs, all three Pulumi.<cloud>.yaml files, and the shared cloud-init.yaml are in the companion repo:

[GitHub repository: dirien/fully-automated-ai-inference-pulumi

github.com/dirien/fully-automated-ai-inference-pulumi

](https://github.com/dirien/fully-automated-ai-inference-pulumi)

Deploying

Create a project, install the provider for your cloud, point the stack at pulumi-idp/auth, and deploy. For AWS:

<span class="line"><span class="cl">mkdir ai-inference <span class="o">&&</span> <span class="nb">cd</span> ai-inference
</span></span><span class="line"><span class="cl">pulumi new typescript
</span></span><span class="line"><span class="cl">npm install @pulumi/aws
</span></span>

Drop the AWS listing into index.ts, copy the cloud-init.yaml shown earlier into the same directory (or grab it from the companion repo), point the stack at the auth environment, and set your region:

<span class="line"><span class="cl">pulumi stack init aws
</span></span><span class="line"><span class="cl"><span class="c1"># add `environment: [pulumi-idp/auth]` to Pulumi.aws.yaml (shown above)</span>
</span></span><span class="line"><span class="cl">pulumi config <span class="nb">set</span> region us-east-1
</span></span><span class="line"><span class="cl">pulumi config <span class="nb">set</span> model qwen2.5:14b
</span></span><span class="line"><span class="cl">pulumi up
</span></span>

The other two clouds are the same flow with a different provider package and a couple of config keys: npm install @pulumi/gcp with pulumi config set project <your-project-id> and pulumi config set zone us-central1-a, or npm install @pulumi/azure-native @pulumi/random with pulumi config set location eastus.

The slow part is the GPU box: it boots, installs the driver, reboots, installs Ollama, and pulls the model, all without you. Depending on the instance, the region, and the model size, plan on roughly 10 to 15 minutes before the endpoint answers. When pulumi up finishes you get the endpoints straight back as stack outputs:

<span class="line"><span class="cl">Outputs:
</span></span><span class="line"><span class="cl"> generateEndpoint: <span class="s2">"http://<public-ip>:11434/api/generate"</span>
</span></span><span class="line"><span class="cl"> ollamaEndpoint : <span class="s2">"http://<public-ip>:11434"</span>
</span></span><span class="line"><span class="cl"> publicIp : <span class="s2">"<public-ip>"</span>
</span></span><span class="line"><span class="cl"> tagsEndpoint : <span class="s2">"http://<public-ip>:11434/api/tags"</span>
</span></span>

Testing the inference server

pulumi up returns as soon as the infrastructure exists, which is before the model has finished downloading. This is exactly where Terraform reached for that null_resource loop, and where Pulumi hands you a URL instead. To check whether the model is ready, ask Ollama what it has loaded:

<span class="line"><span class="cl">curl -s <span class="k">$(</span>pulumi stack output tagsEndpoint<span class="k">)</span> <span class="p">|</span> grep -q <span class="s2">"qwen2.5:14b"</span> <span class="o">&&</span> <span class="nb">echo</span> <span class="s2">"ready"</span> <span class="o">||</span> <span class="nb">echo</span> <span class="s2">"still pulling"</span>
</span></span>

Once it reports ready, send it a prompt. This is the same request the Akamai post makes, pointed at your generateEndpoint output:

<span class="line"><span class="cl">curl -s <span class="k">$(</span>pulumi stack output generateEndpoint<span class="k">)</span> -d <span class="s1">'{
</span></span></span><span class="line"><span class="cl"><span class="s1"> "model": "qwen2.5:14b",
</span></span></span><span class="line"><span class="cl"><span class="s1"> "system": "Answer every question with a three-line poem.",
</span></span></span><span class="line"><span class="cl"><span class="s1"> "prompt": "Why is the sky blue?",
</span></span></span><span class="line"><span class="cl"><span class="s1"> "stream": false
</span></span></span><span class="line"><span class="cl"><span class="s1">}'</span>
</span></span>

The first call is slower, because Ollama loads the model into GPU memory before it answers. Every call after that is fast, since OLLAMA_KEEP_ALIVE=-1 keeps it resident:

<span class="line"><span class="cl"><span class="p">{</span>
</span></span><span class="line"><span class="cl"> <span class="nt">"model"</span><span class="p">:</span> <span class="s2">"qwen2.5:14b"</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nt">"response"</span><span class="p">:</span> <span class="s2">"Sunlight scatters, short waves fly,\nBlue light paints the open sky,\nViolet fades as day drifts by."</span><span class="p">,</span>
</span></span><span class="line"><span class="cl"> <span class="nt">"done"</span><span class="p">:</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span>

When you are done, take it all down with one command, so the GPU stops billing:

<span class="line"><span class="cl">pulumi destroy
</span></span>

Cost

A GPU instance is the whole bill, and you pay by the hour whether the model is busy or idle. The numbers below are rough on-demand rates for a single T4-class box left running 24/7; in practice you spin it up, use it, and pulumi destroy it, so what you actually pay tracks the hours it stays up.

Cloud

Instance

GPU

~Hourly

~Monthly (24/7)

AWS

g4dn.xlarge

1× T4 (16 GB)

~$0.53

~$384

Azure

Standard_NC4as_T4_v3

1× T4 (16 GB)

~$0.53

~$384

Google Cloud

n1-standard-4 + 1× T4

1× T4 (16 GB)

~$0.54

~$394

These are on-demand Linux rates in a cheap US region (us-east-1, eastus, us-central1) as of mid-2026, and the GPU dominates the bill. Spot or preemptible capacity cuts it sharply if your workload tolerates interruption (often 60 to 70 percent off), Google Cloud’s sustained-use discount trims an always-on instance on its own, and the 40 GB disk adds only a few dollars a month.

A GPU box left running overnight is the expensive mistake here. Because the whole stack is declarative, the safe habit is cheap: pulumi destroy when you stop using it, pulumi up when you need it back. The state and config are version-controlled, so standing it up again is one command, not a rebuild.

Security considerations

The architecture in this post matches the Akamai original on purpose, which means it carries the same caveat: Ollama is exposed over plain HTTP on a port open to the entire internet. That is fine for a demo on a box you tear down the same day. It is not fine for anything that outlives the afternoon, and self-hosted AI endpoints get found fast: scanners like Shodan enumerate a freshly exposed one within hours, and an open Ollama instance is free compute for whoever finds it first.

The credential side, though, is genuinely better than a static-token setup, and that is the part worth keeping. There is no long-lived cloud key in the program, in your shell, or in CI; every deploy gets short-lived credentials minted through pulumi-idp/auth and thrown away when it finishes.

Concern

Akamai/Terraform demo

This deployment

Cloud credentials

Static PAT in an env var

OIDC login, short-lived, via pulumi-idp/auth

Inbound exposure

Port 11434 open to 0.0.0.0/0

Same by default; one config flag scopes the CIDR

Transport

Plain HTTP

Plain HTTP (front with TLS for real use)

Readiness wait

null_resource + local-exec shell loop

A runtime curl, no resource

My recommendations for taking this past a demo:

  • Scope the inbound rule to your own CIDR instead of 0.0.0.0/0; the allowSsh flag in the program is the pattern to copy for the Ollama port
  • Put a reverse proxy with TLS in front of Ollama, or keep the box off the public internet entirely and reach it over a private network or a Tailscale tailnet, the way I did for the Hermes agent
  • Keep credentials on OIDC through ESC; never fall back to a static key to “just get it working”
  • Treat the box as disposable, and pulumi destroy it when you are not using it, which shrinks both the bill and the attack window

What’s next?

The program is a foundation, and the interesting work is what you layer on once a model is a pulumi up away:

  • Lock it down. Scope the firewall, add TLS, or move the box behind a private network so the endpoint is not on the public internet at all.
  • Scale the model to the GPU. qwen2.5:14b on a T4 is the entry point. Bigger models want more memory, which means an A10G, an L4, or an A100, and the only change in the program is the instance type and the model value.
  • Swap the serving engine. Ollama is the simplest thing that works. For higher throughput, the same shape holds with vLLM in place of Ollama in the cloud-init.
  • Generate the next one. Pulumi Neo can take a target like “a GPU inference box on Azure behind a private endpoint” and produce a first draft of the program, which you then review in a PR like any other change.

Conclusion

Declaring an AI inference server as code buys you the same thing it buys for any other infrastructure: you can reproduce it on any of the three clouds, version-control it, and tear it down with a single pulumi destroy. Porting the Akamai project made the contrast sharp. The credentials moved from a static token to an OIDC login that leaves nothing behind, and the readiness wait disappeared entirely once you stop treating a runtime check as a resource.

That is the line worth carrying to the next thing you deploy. Not every step in a runbook is a resource. A GPU box and a firewall rule are; waiting for a download to finish is not. Draw that line first, and the program gets shorter on its own.

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions. New to Pulumi? Get started here.

Fetched June 30, 2026