Skip to content

Agent skill tunes local LLM serving speed by measured runs

The MIT-licensed repository packages a measurement-first loop for speeding up self-hosted model servers, plus the scripts to run it.

Draftship Editorial4 min read

On 2026-11-17 the repository sybil-solutions/inference-optimisation-skill was trending on GitHub with 394 stars and 44 forks, one day after it was created on 2026-11-16. It is an MIT-licensed Python project that turns local LLM inference optimization into an agent skill: a written procedure an AI coding agent loads and follows to make a self-hosted model server faster. According to the project's README, it covers single-user and concurrent decode, speculative decoding, prefill, KV cache size, startup time and time to first token, and is meant to work on any hardware and any quantization, under a memory cap.

The framing is the interesting part. Most tuning advice is a list of flags. This is a loop with rules about how to measure, when to stop and what counts as a win.

What is in the repository

Three parts. SKILL.md holds the procedure: inventory, ceilings, ground rules, the loop and lever order, plus how to report. A references/ folder is loaded only when needed and covers ceiling maths with a worked example, measurement traps, levers tagged general or stack-specific, known traps in vLLM, SGLang, llama.cpp and ExLlama across Intel, NVIDIA, Apple and AMD hardware, and publishing rules. The scripts/ folder is standard-library Python and bash: bench.py benchmarks any OpenAI-compatible server, ceiling.py computes theoretical maxima from measured bandwidth, FLOPs and component sizes, queue.sh runs a serialized queue of interleaved control and candidate runs, and mem.sh reports a container's real footprint including driver residency on unified memory.

bench.py measures startup and short-prompt time to first token, single-user decode as a median of several runs, aggregate decode at concurrency, cold prefill at several lengths, long-context needle retrieval and peak memory with a cap watchdog. It flags looping output, failed requests and runs on battery. Token counts come from the server's usage stats rather than streamed chunks, and each result lands in a ledger file.

The rules the agent has to follow

Six steps: compute the theoretical ceiling first, change one thing at a time, measure against interleaved controls, log every result including failures, gate on long-context retrieval, and qualify a clean pinned build before publishing. A candidate is accepted only if it beats both neighbouring controls and regresses nothing by more than 3%. The project puts noise at plus or minus 5 to 10%, and treats most single-run wins as drift.

The ceiling step is the unusual one. Without a known maximum, the README argues, you cannot tell whether 53 tokens per second is good or whether 100 is possible at all.

The numbers it ships with

The README lists before-and-after results measured on named hardware. On an Intel Arc B390 laptop iGPU running a 35B model at 4 bits, decode for one user went from 43.6 to 53 tokens per second, four-user decode from 84.7 to 103, prefill at 256K context from 297 to 628 tokens per second, startup from 113 to 43 seconds, and KV cache from 264K tokens to four 256K slots. On four H200 NVL cards, a setup that crashed under load reached 229.1 tokens per second decode and 17,546 tokens per second prefill, with all six gates passing.

The B390 write-up is candid about its own ceiling: the maths put a practical target near 88 to 97 tokens per second for one user, and the run reached roughly 55% of that, with the remainder blamed on kernel efficiency.

Limits and open questions

Every number is self-reported, with methods and raw data linked to the authors' own repositories, and the project is one day old. The README does not say how well the levers transfer to hardware, engines or models outside those listed, nor how much time the loop saves against manual tuning. Compatibility is described as Claude Code plus any agent that loads SKILL.md-style skills, with PROMPT.md as a fallback for other models; how that works in practice is not spelled out.

What this means for people building apps

If you call a hosted model API, none of this applies directly. The habits do: fix a ceiling before optimising, interleave controls, use medians, count real tokens, and keep a record of what failed. A starting point the source supports is running ceiling.py on the bundled example config, benchmarking a baseline with bench.py, then queueing base, candidate and base again with queue.sh before trusting a change.

Draftship is a different route to a working app: you describe it in plain language and Draftship writes, builds and hosts it, with a built-in database and sign-in, and model choice is a setting on paid plans rather than a server you tune. The inference skill and Draftship are unrelated projects.

Primary sources

This article was drafted with AI assistance from the primary sources listed below and checked automatically against them. An editor hasn't reviewed it yet.