The Mini Rack Saga - Part 4: The Self-Hosted AIML Stack

This is the one we’ve been building toward. In Parts 1 through 3 we built a physical rack, set up two Raspberry Pi 5s, and stood up a highly available RKE2 cluster on three Intel Mac minis with Tailscale handling all the connectivity. Now we’re putting that hardware to work.

In Part 4 we’re deploying a fully self-hosted AI stack on an M4 Pro Mac mini — Ollama running Gemma 4 natively, LiteLLM as the model proxy with per-user usage tracking, and Open WebUI as the frontend with Keycloak SSO. No cloud, no subscriptions, no data leaving your environment.

Let’s get into it.

Architecture Overview Link to heading

Before we dive in, here’s how everything fits together:

The flow looks like this:

User → Tailscale → Open WebUI → LiteLLM → Ollama (native)
                       ↓              ↓
                   Keycloak       Postgres (storage_pi)

Ollama runs natively on macOS, taking full advantage of the M4 Pro’s unified memory for fast local inference. LiteLLM and Open WebUI run as Docker containers on the same machine, talking to Ollama over localhost. Postgres lives on the storage_pi from Part 2 — we’re putting that dedicated database node to good use.

Both LiteLLM and Open WebUI authenticate via Keycloak SSO — no shared passwords, no local accounts, just clean identity management across the whole stack.

Step 1: Install Ollama and Pull Gemma 4 Link to heading

The M4 Pro Mac mini is an excellent inference machine. Apple Silicon’s unified memory architecture means the GPU and CPU share the same memory pool, so large models fit comfortably and run fast. Gemma 4 runs beautifully on it.

Install Ollama Link to heading

Head to ollama.com and download the Mac app, or install via the terminal:

curl -fsSL https://ollama.com/install.sh | sh

Pull Gemma 4 Link to heading

ollama pull gemma4:27b

Give it a few minutes to download. Once it’s done, do a quick sanity check:

ollama run gemma4:27b "Hello, are you there?"

You should get a response back. Ollama is up and running.

Verify the API Link to heading

Ollama exposes a local API on port 11434. By default it only listens on localhost, which is exactly what we want since LiteLLM will be talking to it from the same machine:

curl http://localhost:11434/api/tags

You should see Gemma 4 listed in the response.

Step 2: Set Up Tailscale on the Mac Mini Link to heading

Tailscale is what makes this whole setup accessible from anywhere on your tailnet without exposing anything to the public internet.

Install Tailscale Link to heading

Download the .pkg installer from pkgs.tailscale.com — this is the recommended path as it installs the macOS System Extension automatically. Run the installer, then launch Tailscale from your Applications folder and sign in.

Once connected, note down the Mac mini’s Tailscale IP — you’ll need it when configuring LiteLLM’s SSO redirect URI.

Step 3: Configure Keycloak Link to heading

This guide assumes Keycloak is already running. If you need to set it up, that’s a topic for a dedicated article.

Both LiteLLM and Open WebUI need their own Keycloak clients. Head to your Keycloak admin console and create two clients in your realm.

Client 1: LiteLLM Link to heading

1. Go to Clients → Create client
2. Set Client ID to litellm
3. Set Client authentication to On (confidential)
4. Under Valid redirect URIs add:

   http://<your-mac-mini-tailscale-ip>:4000/sso/callback
   

5. Save, go to the Credentials tab and copy the Client Secret

Client 2: Open WebUI Link to heading

1. Go to Clients → Create client
2. Set Client ID to open-webui
3. Set Client authentication to On (confidential)
4. Under Valid redirect URIs add:

   https://<your-mac-mini-tailscale-hostname>/oauth/oidc/callback
   

5. Save, go to the Credentials tab and copy the Client Secret

Note the client secrets for both — you’ll need them in the .env file next.

Step 4: Environment File Link to heading

Sensitive values live in a .env file alongside your docker-compose.yml. Never commit this to git.

Postgres prerequisite: LiteLLM and Open WebUI each need their own database on the storage Pi. Before continuing, make sure litellm and openwebui databases exist on your Postgres instance. If you set up the storage Pi in Part 2, you already have Postgres running — just create the two databases.

Create .env in your project directory:

# Postgres
POSTGRES_PASSWORD=<your-postgres-password>

# LiteLLM
LITELLM_MASTER_KEY=<your-litellm-master-key>
LITELLM_SALT_KEY=<your-litellm-salt-key>
LITELLM_KEYCLOAK_CLIENT_SECRET=<your-litellm-keycloak-client-secret>

# Open WebUI
WEBUI_SECRET_KEY=<your-webui-secret-key>
WEBUI_URL=https://<your-mac-mini-tailscale-hostname>
OAUTH_CLIENT_ID=open-webui
OAUTH_CLIENT_SECRET=<your-openwebui-keycloak-client-secret>
OPENID_PROVIDER_URL=https://<your-keycloak-tailscale-hostname>/realms/<your-realm>/.well-known/openid-configuration

Step 5: Docker Compose Link to heading

With the .env file in place, create docker-compose.yml in the same directory:

networks:
  openwebui:
    driver: bridge

volumes:
  openwebui_data:

services:
  litellm:
    image: litellm/litellm-non_root:main-latest
    container_name: litellm
    restart: unless-stopped
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"
    environment:
      - DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@<storage-pi-ip>:5433/litellm
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
      - LITELLM_SALT_KEY=${LITELLM_SALT_KEY}
      - PROXY_ADMIN_ID=<your-keycloak-username>
      - GENERIC_DEFAULT_USER_ROLE=internal_user
      - GENERIC_CLIENT_ID=litellm
      - GENERIC_CLIENT_SECRET=${LITELLM_KEYCLOAK_CLIENT_SECRET}
      - GENERIC_AUTHORIZATION_ENDPOINT=https://<your-keycloak-tailscale-hostname>/realms/<your-realm>/protocol/openid-connect/auth
      - GENERIC_TOKEN_ENDPOINT=https://<your-keycloak-tailscale-hostname>/realms/<your-realm>/protocol/openid-connect/token
      - GENERIC_USERINFO_ENDPOINT=https://<your-keycloak-tailscale-hostname>/realms/<your-realm>/protocol/openid-connect/userinfo
      - GENERIC_REDIRECT_URI=http://<your-mac-mini-tailscale-ip>:4000/sso/callback
    volumes:
      - ./litellm-config.yaml:/app/config.yaml:ro
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"]
      interval: 10s
      timeout: 5s
      retries: 10
      start_period: 15s
    networks:
      - openwebui

  openwebui:
    image: openwebui/open-webui:0.9
    container_name: openwebui
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      - DATABASE_URL=postgresql://postgres:${POSTGRES_PASSWORD}@<storage-pi-ip>:5433/openwebui
      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
      - WEBUI_URL=${WEBUI_URL}
      - ENABLE_OLLAMA_API=false
      - OPENAI_API_BASE_URLS=http://litellm:4000/v1
      - OPENAI_API_KEYS=${LITELLM_MASTER_KEY}
      - ENABLE_LOGIN_FORM=false
      - ENABLE_OAUTH_SIGNUP=true
      - ENABLE_OAUTH_ROLE_MANAGEMENT=true
      - OAUTH_ADMIN_ROLES=admin
      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
      - OAUTH_PROVIDER_NAME=Keycloak
      - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
      - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
      - OPENID_PROVIDER_URL=${OPENID_PROVIDER_URL}
      - OAUTH_SCOPES=openid email profile
    volumes:
      - openwebui_data:/app/backend/data
    depends_on:
      litellm:
        condition: service_healthy
    networks:
      - openwebui

A few things worth calling out:

• litellm-non_root — we’re using the rootless LiteLLM image, which is the more secure option
• ENABLE_LOGIN_FORM=false — Open WebUI has no local login form; Keycloak is the only way in
• ENABLE_OAUTH_ROLE_MANAGEMENT=true — Keycloak roles map directly to Open WebUI roles; users with the admin role in Keycloak get admin access in Open WebUI automatically
• depends_on with health check — Open WebUI won’t start until LiteLLM is fully healthy, preventing startup race conditions
• PROXY_ADMIN_ID — your Keycloak username, grants you admin access to the LiteLLM UI via SSO

LiteLLM Config File Link to heading

Create litellm-config.yaml in the same directory:

model_list:
  - model_name: gemma4
    litellm_params:
      model: ollama/gemma4:27b
      api_base: http://host.docker.internal:11434

general_settings:
  store_model_in_db: true

The host.docker.internal hostname lets the LiteLLM container reach Ollama running natively on the Mac mini without any extra networking configuration.

Bring It Up Link to heading

docker compose up -d

Check everything is running:

docker compose ps

Both containers should show as running. LiteLLM will take a moment to pass its health check before Open WebUI starts.

Expose Services via Tailscale Serve Link to heading

With both containers up, use tailscale serve to put them on the tailnet with HTTPS and automatic TLS:

# Expose Open WebUI
tailscale serve --https=443 http://localhost:8080

# Expose LiteLLM
tailscale serve --https=4000 http://localhost:4000

Both services will now be accessible via your Mac mini’s MagicDNS hostname on your tailnet.

Step 6: Create LiteLLM API Keys Per User Link to heading

Once LiteLLM is up, head to http://<your-mac-mini-tailscale-ip>:4000 and sign in with Keycloak. Navigate to Virtual Keys and create a key per user. Each key:

• Routes to the same Ollama backend
• Tracks token usage independently
• Can have spend limits and rate limits applied

Open WebUI uses the master key to talk to LiteLLM on behalf of all users — the per-user keys are for direct API access or granular usage reporting.

Step 7: Verify the Stack Link to heading

Test Ollama directly Link to heading

curl http://localhost:11434/api/tags

Test LiteLLM Link to heading

curl http://localhost:4000/health

Test the full flow Link to heading

1. Open Open WebUI at https://<your-mac-mini-tailscale-hostname>
2. Click Sign in with Keycloak
3. Log in and get redirected back to Open WebUI
4. Start a conversation with Gemma 4
5. Head to the LiteLLM UI and verify your request shows up in the usage dashboard

If all of that works — you’re done.

Wrapping Up Link to heading

Let’s take a moment to appreciate what we’ve built across this series:

• ✅ Part 1 — Physical rack build
• ✅ Part 2 — Raspberry Pi software setup, NVMe boot, Docker stacks
• ✅ Part 3 — HA RKE2 cluster on Mac minis, Tailscale Operator
• ✅ Part 4 — Self-hosted AIML stack: Ollama, LiteLLM, Open WebUI, Keycloak SSO

A full self-hosted AI stack, running on your own hardware, with no data leaving your environment. Per-user usage tracking. SSO via Keycloak across both LiteLLM and Open WebUI. Accessible from anywhere on your tailnet. That’s not a toy — that’s a production-grade homelab AI setup.

In Part 5 we put this stack to work as a coding agent — installing opencode-offline on a developer laptop and pointing it at LiteLLM over Tailscale for a fully offline AI coding experience. See you there.