Documentation

osmoss.ai Docs

Everything you need to install, configure, and integrate osmoss.ai.

Quick Start

Get osmoss.ai running locally in under five minutes. The install script detects your OS and hardware, downloads the runtime, and starts the API server automatically.

1. Install

Run the one-line install script. On macOS you can also use Homebrew.

⌁ bash

# Linux & macOS
curl -fsSL https://get.osmoss.ai | sh

# macOS (Homebrew)
brew install osmoss/tap/osmoss

# Windows (PowerShell, run as Administrator)
irm https://get.osmoss.ai/win | iex

2. Pull a model

Download a quantised model from the registry. Mistral 7B Q4 is a good starting point — fast on most hardware.

⌁ bash

osmoss pull mistral-7b-q4

3. Start the server

Launch the runtime. The OpenAI-compatible API starts on port 11434.

⌁ bash

osmoss serve

4. Make your first request

Send a chat completion request with cURL to verify everything is working.

⌁ bash

curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistral-7b-q4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

API Reference

osmoss.ai exposes an OpenAI-compatible REST API on port 11434. Any client that works with the OpenAI API works here without modification — just change the base URL.

POST /v1/chat/completions

Generate a chat completion. Supports streaming via Server-Sent Events when stream is true.

⌁ json

# Request
curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistral-7b-q4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain quantum entanglement briefly."}
    ],
    "temperature": 0.7,
    "max_tokens": 512,
    "stream": false
  }'

# Response
{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "mistral-7b-q4",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "Quantum entanglement is..."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 28, "completion_tokens": 94, "total_tokens": 122}
}

POST /v1/completions

Raw text completion. Useful for non-chat models or prompt-completion workflows.

⌁ bash

curl http://localhost:11434/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistral-7b-q4",
    "prompt": "The capital of France is",
    "max_tokens": 16,
    "temperature": 0
  }'

POST /v1/embeddings

Generate vector embeddings for a string or array of strings. Returns a 1536-dimension float array per input.

⌁ json

curl http://localhost:11434/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistral-7b-q4",
    "input": "The quick brown fox"
  }'

# Response
{
  "object": "list",
  "data": [{"object": "embedding", "index": 0, "embedding": [0.0023, -0.0091, ...]}],
  "model": "mistral-7b-q4",
  "usage": {"prompt_tokens": 5, "total_tokens": 5}
}

GET /v1/models

List all models currently available on the runtime.

⌁ json

curl http://localhost:11434/v1/models

# Response
{
  "object": "list",
  "data": [
    {"id": "mistral-7b-q4", "object": "model", "created": 1700000000, "owned_by": "osmoss"},
    {"id": "llama3-8b-q5", "object": "model", "created": 1700000001, "owned_by": "osmoss"}
  ]
}

GET /health

Health check endpoint. Returns 200 OK when the runtime is ready to serve requests. Use this for load balancer probes.

⌁ bash

curl http://localhost:11434/health
# {"status": "ok", "model": "mistral-7b-q4", "uptime": 3821}

CLI Reference

The osmoss CLI manages models, controls the runtime, and exposes diagnostics. All commands accept --help for full option listings.

osmoss pull

Download a model from the registry. Quantisation variant is specified with a suffix (q4, q5, q8). Omitting the suffix pulls the recommended default.

⌁ bash

osmoss pull mistral-7b-q4
osmoss pull llama3-8b         # pulls recommended variant
osmoss pull phi3-mini-q8
osmoss pull gemma-7b-q4

osmoss serve

Start the API server. Flags override config.toml values.

⌁ bash

osmoss serve
osmoss serve --host 0.0.0.0 --port 11434
osmoss serve --model llama3-8b-q5 --gpu-layers 35

osmoss list

List all locally available models with their size, quantisation, and hash.

⌁ bash

osmoss list

# NAME              SIZE    QUANT  HASH
# mistral-7b-q4     4.1 GB  Q4_K_M 3f2a...
# llama3-8b-q5      5.7 GB  Q5_K_M 9c1b...

osmoss set-default

Set the default model served when no model is specified in the request. Hot-swaps at runtime with zero downtime.

⌁ bash

osmoss set-default llama3-8b-q5

osmoss remove

Delete a model from local storage to free disk space.

⌁ bash

osmoss remove mistral-7b-q4

osmoss info & doctor

Display hardware detection results and run a full diagnostic report.

⌁ bash

osmoss info      # GPU, CPU, memory, active backend
osmoss doctor    # full diagnostic with pass/fail checks

osmoss service

Install and manage the background service (systemd on Linux, launchd on macOS, Windows Service on Windows).

⌁ bash

osmoss service install
osmoss service start
osmoss service stop
osmoss service status
osmoss service uninstall

Configuration

osmoss.ai is configured via config.toml. Every value can be overridden with an environment variable prefixed OSMOSS_. CLI flags take the highest precedence.

config.toml — full reference

The config file lives at ~/.osmoss/config.toml on Linux/macOS and %APPDATA%\osmoss\config.toml on Windows.

⌁ toml

[runtime]
host        = "127.0.0.1"    # bind address
port        = 11434          # API port
ctx_size    = 4096           # context window (tokens)
gpu_layers  = -1             # layers to offload; -1 = all
threads     = 8              # CPU threads (ignored when GPU is active)
batch_size  = 512            # prompt eval batch size

[model]
default     = "mistral-7b-q4"
dir         = "~/.osmoss/models"

[logging]
level       = "info"         # debug | info | warn | error
format      = "text"         # text | json
file        = ""             # path to log file; empty = stdout

[dashboard]
enabled     = true
port        = 11435

Environment variables

Every config key maps to an OSMOSS_ environment variable. Underscores replace dots; keys are uppercased.

⌁ bash

OSMOSS_HOST=0.0.0.0
OSMOSS_PORT=11434
OSMOSS_CTX_SIZE=8192
OSMOSS_GPU_LAYERS=35
OSMOSS_THREADS=16
OSMOSS_MODEL_DEFAULT=llama3-8b-q5
OSMOSS_LOGGING_LEVEL=debug
OSMOSS_LOGGING_FORMAT=json

Model Catalogue

osmoss.ai ships curated, hardware-optimised weights for every major open-weight model family. Quantised variants trade a small accuracy loss for dramatically lower memory requirements.

Supported models

Pull any model by its registry name. The Recommended column shows the best variant for most hardware.

⌁ text

# MODEL              FAMILY    PARAMS  RECOMMENDED
# mistral-7b-q4      Mistral   7B      mistral-7b-q4
# mistral-8x7b-q4    Mistral   56B     mistral-8x7b-q4
# llama3-8b-q5       LLaMA 3   8B      llama3-8b-q5
# llama3-70b-q4      LLaMA 3   70B     llama3-70b-q4
# phi3-mini-q8       Phi-3     3.8B    phi3-mini-q8
# phi3-medium-q4     Phi-3     14B     phi3-medium-q4
# gemma-2b-q8        Gemma     2B      gemma-2b-q8
# gemma-7b-q4        Gemma     7B      gemma-7b-q4
# qwen2-7b-q4        Qwen 2    7B      qwen2-7b-q4
# deepseek-r1-7b-q4  DeepSeek  7B      deepseek-r1-7b-q4
# falcon-7b-q4       Falcon    7B      falcon-7b-q4

Quantisation formats

osmoss.ai supports GGUF and safetensors. GGUF quantised variants are recommended for most deployments.

⌁ text

# VARIANT  BITS   VRAM (7B)  QUALITY
# q4       4-bit  ~4 GB      good — recommended default
# q5       5-bit  ~5 GB      better accuracy, slightly more VRAM
# q8       8-bit  ~8 GB      near-lossless, best for production
# f16      16-bit ~14 GB     full precision, maximum quality

Auth & Security

By default osmoss.ai binds to 127.0.0.1 and requires no authentication — it is only reachable from the local machine. When exposing the API on a network, enable the API key middleware.

Enabling API key auth

Set one or more API keys in config.toml. The server then requires an Authorization: Bearer <key> header on every request.

⌁ toml

[auth]
enabled  = true
keys     = ["sk-local-abc123", "sk-local-def456"]

Using a key in requests

Pass the key as a Bearer token. This is identical to how you would authenticate with the OpenAI API.

⌁ bash

curl http://your-server:11434/v1/chat/completions \
  -H 'Authorization: Bearer sk-local-abc123' \
  -H 'Content-Type: application/json' \
  -d '{"model": "mistral-7b-q4", "messages": [{"role": "user", "content": "Hi"}]}'

TLS / HTTPS

osmoss.ai does not terminate TLS itself. Place it behind a reverse proxy such as nginx or Caddy for HTTPS.

⌁ text

# Caddy example — automatic HTTPS with Let's Encrypt
ai.example.com {
  reverse_proxy localhost:11434
}

Deployment

osmoss.ai can be deployed as a systemd service, a Docker container, or a Kubernetes workload. All three approaches are production-ready.

systemd (Linux)

The CLI generates and installs a systemd unit file automatically.

⌁ bash

osmoss service install
systemctl enable --now osmoss
journalctl -u osmoss -f    # follow logs

Docker

The official image is available on Docker Hub. Mount a volume for model persistence.

⌁ bash

docker run -d \
  --name osmoss \
  --gpus all \
  -p 11434:11434 \
  -v ~/.osmoss/models:/root/.osmoss/models \
  osmoss/osmoss:latest serve --host 0.0.0.0

Kubernetes (Helm)

Install the Helm chart for a production-grade Kubernetes deployment with GPU node affinity and a PersistentVolumeClaim for models.

⌁ bash

helm repo add osmoss https://charts.osmoss.ai
helm repo update

helm install osmoss osmoss/osmoss \
  --set runtime.gpuLayers=-1 \
  --set model.default=mistral-7b-q4 \
  --set persistence.size=50Gi

SDK Examples

osmoss.ai is a drop-in replacement for the OpenAI API. Set the base URL to http://localhost:11434/v1 and use any api_key string — it is not validated locally.

Python

Use the official openai Python package. Works with both sync and async clients.

⌁ python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="local",
)

# Streaming chat completion
with client.chat.completions.stream(
    model="mistral-7b-q4",
    messages=[{"role": "user", "content": "Write a haiku about on-device AI."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Node.js / TypeScript

Use the official openai npm package.

⌁ typescript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'local',
});

const stream = await client.chat.completions.stream({
  model: 'mistral-7b-q4',
  messages: [{ role: 'user', content: 'Explain GGUF quantisation.' }],
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}

cURL

Raw HTTP — useful for testing, shell scripts, and CI pipelines.

⌁ bash

# Non-streaming
curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mistral-7b-q4","messages":[{"role":"user","content":"Hello"}]}'

# Streaming (SSE)
curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  --no-buffer \
  -d '{"model":"mistral-7b-q4","stream":true,"messages":[{"role":"user","content":"Count to 5"}]}'

Something missing or unclear?Get Early Access