The engine
for local AI

The engine

Between the model
and the metal

From the moment a model file arrives to the last token of a response, eight systems do the work. Scroll to step through them, drag to turn the engine, or pick a layer.

01 · Model loadingProbe

Reads the model, not a rulebook

A model arrives as a single GGUF file. Loculus Core reads its architecture, tokenizer and chat template straight from it, so a new model from a supported family runs without new code.

• Zero-copy GGUF loading
• Architecture auto-detection
• Tokenizer auto-config
• Native chat templates
• Tool-call format detection

17 architectures · 20 quantization formats

02 · Auto-tuningAttune

Measured on your machine

First start fingerprints your GPU, CPU, memory and SSD, then benchmarks the fastest kernel settings for this exact hardware, in seconds and once. Every start after that is instant.

• Hardware fingerprinting
• Per-shape kernel autotuning
• GPU cache & SSD calibration
• Runs with or without matrix cores
• Persisted tuning profiles

Zero configuration

03 · Memory & KV cacheTide

Every spare byte. Never one too many.

No fixed split between engine and system: memory grows into what is free while the engine works and is handed back the instant anything else needs it. The KV cache grows the same way, only as the context fills.

• Elastic memory allocation
• Pressure-aware reclamation
• On-demand KV cache growth
• Adaptive KV-cache compression
• Up to 1M-token context

KV memory scales with use, not with max context

04 · MoE expert offloadingStrata

Bigger than your memory

Mixture-of-experts models larger than RAM keep their experts on the SSD. The experts each layer needs are predicted ahead and streamed in before they are asked for; recent ones stay in spare memory.

• SSD → RAM → GPU expert tiers
• Predictive expert prefetch
• High-throughput SSD streaming
• Elastic expert cache
• Models larger than RAM

748B parameters on a 93 GB machine

05 · Prefill & prefix cachingRecall

Only new tokens cost time

Conversations are checkpointed as they grow and long prompts are cached on disk, so an agent’s next turn prefills only what is new, even after a restart.

• Prefix caching
• State checkpointing
• Persistent prompt cache
• Matrix-core prefill
• Chunked long-context prefill

8k-token prompts resume in under a second

06 · DecodingForesight

Faster, with exactly the same answer

Draft tokens race ahead and the model keeps only those it would have chosen itself. The output is identical; code edits generate up to 2.3× faster.

• Lossless speculative decoding
• No separate draft model
• Batched verification
• Long-context decode attention
• Cache-aware kernels

Same tokens, fewer steps

07 · API serverConduit

Speaks the APIs you already use

One engine behind the Anthropic Messages, OpenAI Responses and Chat Completions APIs. Point existing agents and apps at it and go.

• OpenAI & Anthropic APIs
• Structured outputs
• Native tool calling
• Reasoning streams
• Streaming (SSE)

Drop-in for existing clients

08 · Runtime & kernelsNucleus

Built from first principles

No borrowed runtime. Every CPU and GPU kernel is written from scratch for this engine, and every model is validated against an exact reference before it ships.

• Graph-free runtime
• Custom Vulkan & CPU kernels
• Low-overhead GPU dispatch
• Fused kernels
• 20 quantization formats

Validated against an exact reference