--- url: "https://github.com/noonghunna/club-3090" title: club-3090 source_kind: github author: noonghunna captured: "2026-08-06T16:50:57+00:00" comment_tree: false topics: [llm-frontend] summary: A GitHub repo with recipes and configs for running LLMs locally on RTX 3090 GPUs using vLLM, llama.cpp, and ik_llama, including setup scripts and benchmarks. status: ok --- # noonghunna/club-3090 - stars: 1896 - language: Python - topics: β€” - license: Apache-2.0 - created: 2026-04-28T10:08:59Z - archived: False --- # club-3090 **Recipes for serving LLMs locally on RTX 3090s.** Multi-engine (vLLM, llama.cpp, ik_llama), multi-model, model-agnostic by design. If you have one or two RTX 3090s and want to run modern LLMs at home, in a homelab, or as a dev backend β€” this repo collects the working configs, patches, and benchmarks. > 🎯 **4090 or 5090 owner?** The composes run cross-rig β€” contributors have benched both with measured numbers: **[Can I use a 4090? β†’](docs/FAQ.md#can-i-use-a-4090-instead-of-a-3090)** Β· **[Can I use a 5090? β†’](docs/FAQ.md#can-i-use-a-5090)**. The tooling is calibrated for 3090s but the configs are class-aware; per-class gotchas (4090's tighter idle VRAM, 5090's 32 GB envelope) + cross-rig benchmark rows live in the FAQ. > 🎨 **Want image generation too?** The **[Image Studio bundle](docs/ai-studio/image.md)** runs Ideogram-4 image gen + a chat model + Open WebUI together on two GPUs β€” one command: `bash scripts/setup-image-studio.sh`. --- ## Quick start > πŸͺŸ **On Windows?** These steps assume Linux/macOS. Set up **WSL2** first β†’ **[docs/WSL_SETUP.md](docs/WSL_SETUP.md)** (start-to-finish). Native Windows runs only the *upstream* llama.cpp binary β€” none of this repo's tooling. ```bash # 1. Clone the repo git clone https://github.com/noonghunna/club-3090.git cd club-3090 # Profile compatibility tooling requires PyYAML. Ubuntu LTS usually has it via # python3-yaml; otherwise run: python3 -m pip install pyyaml # 2. Pick/download + SHA-verify the model (interactive hardware-aware picker) # (asks you which model, then where to put model weights β€” pick in-repo # default, ~/models, or a custom path on a different drive. To skip prompts: # `export MODEL_DIR=/path/to/models` and pass the model name. See FAQ.) bash scripts/setup.sh # Or scripted: # bash scripts/setup.sh qwen3.6-27b # 3. Pick a config + boot it (interactive wizard: asks model β†’ GPUs β†’ projects VRAM budget) bash scripts/launch.sh # Or let the resolver pick for your model + hardware (.env pin β€– curated default): # bash scripts/launch.sh --variant qwen3.6-27b/default # YOUR default for this model # Or skip the wizard with an explicit config: # bash scripts/launch.sh --variant beellama/dflash # single-card BLESSED default β€” code-fast (~100 code / 50 narr TPS), DFlash spec-dec (⚠️ unofficial multi-arch image; sm_89/120 unvalidated β€” see docs/INFERENCE_ENGINES.md) # bash scripts/launch.sh --variant ik-llama/iq4ks-mtp # single-card BALANCED alt β€” ~63/69 TPS, 200K ctx + vision, leanest VRAM (ik_llama IQK quant) # bash scripts/launch.sh --variant llamacpp/default # single-card cliff-immune ALT β€” 200K @ -ub 512, ~51/60 TPS # bash scripts/launch.sh --variant llamacpp/mtp-vision # single-card 49K + MTP + vision # bash scripts/launch.sh --variant vllm/dual # dual-card 262K + vision (vLLM single-card paths blocked on #167) # Or partial flags (wizard fills the rest): # bash scripts/launch.sh --model qwen3.6-27b --gpus 0,1 # bash scripts/launch.sh --tp 2 --pp 1 # override vLLM parallelism # See the variants this machine can run + the per-model defaults view # (hardware-filtered by GPU count; add --all to see every variant): # bash scripts/switch.sh --list # runnable here # bash scripts/switch.sh --list --all # everything # Pin your own default so bare `launch.sh` goes straight there: # bash scripts/switch.sh --set-default ik-llama/iq4ks-mtp # e.g. prefer the balanced ik-llama path over the beellama default; clear: --clear-default qwen3.6-27b # 4. Sanity test (launcher already printed this curl) curl -sf http://localhost:8020/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Capital of France?"}],"max_tokens":200}' # 5. Run the canonical benchmark bash scripts/bench.sh # 6. Switch later without re-clicking through the wizard: bash scripts/switch.sh vllm/long-vision # for example # 7. Keep your install up-to-date as the stack moves (Genesis pin bumps, # new compose variants, vendored patch updates): bash scripts/update.sh ``` `launch.sh` calls `switch.sh` (down old, up new) and then `verify-full.sh` so you know it's serving cleanly before you point a client at it. See [`scripts/`](scripts/) for all helpers. ### Prefer a screen to the CLI? β€” the serve cockpit (`c3`) **`c3`** is a lazydocker-style terminal UI that wraps the same flow β€” **discover β†’ serve β†’ operate β†’ validate** β€” in one keyboard-driven cockpit: browse the catalog and serve a variant with `⏎`, watch live GPU / scenes / containers, run Doctor health checks, and (producer lane) walk the add-a-model pipeline. ```bash # Install from the checkout (the cockpit + its in-repo core package): uv pip install -e tools/serve-cockpit # with uv (recommended β€” one command) # …or plain pip: pip install -e tools/tui-core && pip install -e tools/serve-cockpit c3 # launch (also: python -m club3090_cockpit) ``` **First run:** press **`S`** β†’ set your **Model Dir** (where weights download) + **HuggingFace token** β†’ **`Ctrl+S`** to save; then **`r`** to browse the catalog and serve one. **`c3 --lean`** (or **`[C]`** in-app) hides the producer lane for a consumer-only view. After a `git pull`, re-run the install to pick up new deps + UI changes. Full keybindings + details β†’ [`tools/serve-cockpit/`](tools/serve-cockpit/). > ⚠️ **Single-card long-context note:** Cliff 2 (GDN prefill OOM at >~50K single-prompt) is **open** on 24 GB single-card vLLM. Genesis v7.72.2 PN59 was intended as the fix but doesn't engage on chunked-prefill. **Workarounds:** [`vllm/dual`](docs/DUAL_CARD.md) (TP=2 escapes it) or [`llamacpp/default`](docs/SINGLE_CARD.md#bulletproof-no-cliffs) (different engine, no cliff). Full diagnosis at [`docs/CLIFFS.md`](docs/CLIFFS.md). --- ## TL;DR β€” what this is - **Two complementary routes** β€” pick by what your workload breaks on: - 🏎 **vLLM dual** = max throughput. Up to **127 TPS code** (DFlash) or **4 concurrent streams @ 262K** (turbo). Full feature stack (vision Β· tools Β· MTP Β· streaming). - πŸ›‘ **llama.cpp single** = max robustness. Full **200K context** on one 3090 (max-safe β€” fills cleanly with margin; see [CLIFFS](docs/CLIFFS.md)). Stress-tested clean: no prefill cliffs, 25K-token tool returns work, 91K needle ladder passes. **~51 / 60 TPS** (Q4_K_M + MTP) β€” slower than vLLM dual but doesn't crash on real-world tool-using agents. - **Validated docker compose configs** for both routes β€” drop-in OpenAI-compatible API on `localhost:8020` - **Multi-engine**: vLLM (full features), llama.cpp (max ctx + robustness), ik_llama (best GGUF quants). _(SGLang was evaluated β€” currently blocked on Ampere; see [`docs/engines/SGLANG.md`](docs/engines/SGLANG.md).)_ - **Model-agnostic**: today ships curated configs for Qwen3.6-27B and friends; structure scales as we add models - **Universal `pull`** (v0.8.0; extended in v0.8.2) β€” evaluate any safetensors HF repo, get an honest one-line fit verdict (`--recommend`), and when a pull hard-blocks, send the redacted diagnostic back in one consented step (`--submit-last`). Broader arch coverage each release. See [`docs/PULL.md`](docs/PULL.md) **New to local AI itself?** β†’ [`docs/LOCAL_AI_PRIMER.md`](docs/LOCAL_AI_PRIMER.md) β€” plain-English: how hardware / engines / model sizes / quants fit together. **New here?** β†’ [`docs/GETTING_STARTED.md`](docs/GETTING_STARTED.md) β€” 5-minute clone-to-curl path. **Already running, want to compare engines?** β†’ [docs/engines/](docs/engines/) **Picking an engine** (vLLM / llama.cpp / ik_llama)? β†’ [docs/INFERENCE_ENGINES.md](docs/INFERENCE_ENGINES.md) **Confused by quant names** (Q4_K_M vs IQ4_KS vs AWQ)? β†’ [docs/QUANTIZATION.md](docs/QUANTIZATION.md) **Hardware questions** (4090, NVLink, power caps)? β†’ [docs/HARDWARE.md](docs/HARDWARE.md) **Don't know what TPS / KV / MTP mean?** β†’ [docs/GLOSSARY.md](docs/GLOSSARY.md) --- ## Pick your path | You have | Start here | |---|---| | **1Γ— RTX 3090** | [`docs/SINGLE_CARD.md`](docs/SINGLE_CARD.md) β€” workload β†’ config β†’ quick start | | **2Γ— RTX 3090** (PCIe / NVLink auto-detected) | [`docs/DUAL_CARD.md`](docs/DUAL_CARD.md) β€” workload β†’ config β†’ quick start | | **3+ GPUs** (any class β€” 4Γ— 3090, 8Γ— A6000, mixed) | [`docs/MULTI_CARD.md`](docs/MULTI_CARD.md) β€” TP scaling math, derivation from `dual.yml`, valid TP values | | **A model not in the supported list** / any HF safetensors repo | [`docs/PULL.md`](docs/PULL.md) β€” universal `pull` flow: evaluate against the KV math, honest about confidence | | Considering self-host vs cloud APIs | [`docs/COMPARISONS.md`](docs/COMPARISONS.md) β€” cost crossover + when each wins | Each hardware page lists every supported model with the working composes for that card count, plus measured TPS and per-workload pitfalls. Model-specific deep dives (quants, Genesis patches, engine internals) live under [`models//`](models/). --- ## Supported models | Model | Status | Card counts | Engines | Highlights | |---|---|---|---|---| | **[Qwen3.6-27B](models/qwen3.6-27b/)** | Production-ready ⭐ | 1Γ— / 2Γ— 3090 | vLLM βœ… Β· llama.cpp βœ… Β· ik_llama βœ… | Vision Β· tools Β· MTP n=3 Β· up to 262K ctx Β· vLLM dual = 89/127 TPS Β· llama.cpp single = 200K max-safe, no prefill cliffs Β· ik_llama IQ4_KS = ~60/69 TPS (fastest single-card) | | **[Gemma 4 31B](models/gemma-4-31b/)** | Production-ready | 1Γ— ΒΉ / 2Γ— 3090 | vLLM βœ… (dual) Β· llama.cpp ⚠️ (community fork; mainline blocked on FA hdim=512) | Vision Β· tools Β· MTP n=3 (Google official drafter) **OR** DFlash n=7 (z-lab drafter) Β· up to 262K ctx via INT8 PTH KV (PR [#40391](https://github.com/vllm-project/vllm/pull/40391) vendored) Β· MTP dual = 106/141 TPS at 32K, 95/126 at 262K Β· DFlash dual = 105/177 TPS at 32K (code-optimal) Β· single-card: no functional config since the beellama retirement (2026-07-27 β€” engine deprecated; historical: 47/88 TPS via beellama DFlash, [discussion #239](https://github.com/noonghunna/club-3090/discussions/239)) | | **[Qwen3.6 35B-A3B](models/qwen3.6-35b-a3b/)** | Production-ready (ik-llama single-card Β· vLLM dual) | 1Γ— / 2Γ— 3090 | vLLM βœ… Β· ik_llama βœ… Β· llama.cpp βœ… (mainline runs it β€” see `docs/HARDWARE.md`; ik_llama is the shipped single-card path) | **MoE (256 experts Γ— 8 active, ~3 B active params)** Β· vision Β· tools Β· **ik_llama `fit-mtp.yml` single-card (Mudler APEX I-Compact)** = 103/149 TPS at 196K, hermes 11/20 + aider 12/30 + cli 12/40 ([PR #243](https://github.com/noonghunna/club-3090/pull/243)) Β· ik_llama `byteshape-iq4xs` single-card = 113/129 TPS at full 262K, 110/150 8-pack ([PR #293](https://github.com/noonghunna/club-3090/pull/293)) Β· vLLM dual = 178/174 TPS at 262K + vision (v0.22.0 stable; MTP net-negative on this MoE at TP=2) | | **[Gemma 4 26B-A4B](models/gemma-4-26b-a4b/)** | Production via AWQ (Intel AutoRound INT4 blocked on Ampere) | 2Γ— 3090 Β² | vLLM βœ… (AWQ overlay) Β· llama.cpp ❌ | **MoE (128 experts Γ— 8 active, ~4 B active params)** Β· vision Β· tools Β· AWQ dual = **139/139 TPS at 32K**, CV 0.2% / 0.0% | ΒΉ Single-card Gemma 4 on Ampere 24 GB: vLLM + mainline llama.cpp are blocked (head_dim=512 FA wall, no Ampere FA kernel yet). The community **beellama.cpp** fork builds with `FA_ALL_QUANTS=ON` and runs cleanly on a single 3090 β€” 47/88 TPS, 100–150K ctx, 109/114 8-pack (validated 2026-05-27). 32 GB+ GPUs run the standard vLLM path (validated on RTX 5090 32 GB by [@apnar](https://github.com/noonghunna/club-3090/discussions/67#discussioncomment-16832042)). Β² Gemma 4 26B-A4B single-card not yet tested on Ampere; should fit on a 24 GB 3090 at modest context but the configs are dual-card-only today. More models coming β€” they go under `models//` with the same internal pattern. --- ## Measured TPS at a glance ![Qwen3.6-27B TPS by config](https://raw.githubusercontent.com/noonghunna/club-3090/HEAD/docs/img/performance.png) Bench protocol: 3 warm + 5 measured runs. See [`scripts/bench.sh`](scripts/bench.sh) for methodology. Per-config details + run-by-run numbers + VRAM + AL/accept rates: [models/qwen3.6-27b/CHANGELOG.md](models/qwen3.6-27b/CHANGELOG.md). --- ## Benchmarks Reproduce the numbers above on your own rig. All benchmarks run against the **currently-running** compose (boot one first via `launch.sh`). **Throughput (TPS)** β€” the canonical narrative + code bench (3 warmup + 5 measured per prompt): ```bash bash scripts/bench.sh ``` **Behavioral quality** β€” tool-call correctness, instruction-following, structured output, etc. via `benchlocal-cli`: ```bash bash scripts/quality-test.sh # --medium: 5 packs (default, ~15-25 min, no Docker) bash scripts/quality-test.sh --quick # 2 packs (~5-10 min, no Docker) bash scripts/quality-test.sh --full # 8 packs / 150 scenarios (~25-40 min, needs Docker) bash scripts/quality-test.sh --pack aider-polyglot-30 # a single named pack bash scripts/quality-test.sh --reasoning # HE+/LCB/GPQA(gated)/GSM reasoning suite β€” separate from --full; code packs need Docker ``` **Full rebench (one model, everything)** β€” the canonical 5-step pipeline (`bench` β†’ `verify-stress` β†’ `quality-test --full` β†’ `soak` β†’ `aider-polyglot-30`), ~1.75-2 hr per leg. All artifacts land under `results/rebench//`: ```bash bash scripts/rebench-full.sh # auto-tag from MODEL bash scripts/rebench-full.sh --tag qwen-int8 # explicit tag bash scripts/rebench-full.sh --skip soak,aider # skip phases (CSV) bash scripts/rebench-full.sh --resume # resume an interrupted run (skip completed steps) # Endpoint-first mode (non-Docker engines: llama-swap, ramalama, raw llama-server, …): bash scripts/rebench-full.sh \ --url http://HOST:PORT --model 'MODEL-NAME' --engine llama-cpp # vllm|llama-cpp|sglang|other ``` Run `rebench-full.sh` twice on different models to assemble a matched-config head-to-head. Full test-pipeline reference: [`docs/QUALITY_TEST.md`](docs/QUALITY_TEST.md). --- ## Diagnostics When filing a bug, sharing cross-rig data, or replying to a triage thread, generate a paste-ready triage report β€” it captures hardware, OS, GPU, container runtime, stack version, and active container state as markdown. **Home paths, hostnames, usernames, and HF tokens are redacted by default**, so it's safe to paste into a public issue or discussion. ```bash # Quick report (~2 sec) β€” hardware + stack + boot-log highlights bash scripts/report.sh # Capture the full cross-rig pass to a file, ready to paste into an issue/discussion # (~35 min β€” drop --full for a ~2 sec hardware-only capture) bash scripts/report.sh --full > my-rig.md # Add live test output (pick what the thread needs): bash scripts/report.sh --verify # + verify-full.sh (~1-2 min) bash scripts/report.sh --stress # + verify-stress.sh 7/7 (~5-10 min) bash scripts/report.sh --soak # + continuous soak (~25 min) β€” catches Cliff 2b bash scripts/report.sh --bench # + bench.sh TPS (~3 min) bash scripts/report.sh --full # ALL four β€” the canonical "everything" cross-rig pass (~35 min) # Internal sharing only (disable redaction): bash scripts/report.sh --no-redact ``` `--soak` is its own flag because a config can pass verify + stress + bench and still fail the multi-turn continuous soak (Cliff 2b at ~25K accumulated tokens) β€” soak is currently the only test that catches that agentic-workload failure mode. See [`docs/CLIFFS.md`](docs/CLIFFS.md). ### If `launch.sh` / `switch.sh` won't boot β€” load a compose directly The launcher scripts wrap the boot in a preflight (hardware / free-VRAM checks), `.env` parsing, and a Python-driven variantβ†’compose registry. If any of those misfire β€” a false preflight failure, a CRLF/`.env` quirk on Windows, or missing PyYAML β€” bypass them and bring the compose up with plain Docker. Two escalations (assumes you've already downloaded the weights β€” Quick start step 2): ```bash # 1. Skip ONLY the hardware / free-VRAM preflight (keeps .env + registry): bash scripts/switch.sh --force llamacpp/default # 2. Bypass the scripts entirely β€” boot the compose file with Docker directly. # Set MODEL_DIR to wherever your weights live; -f points at the compose. # Layout: models///compose///.yml # single-card llama.cpp (cliff-immune fallback; no nightly/Genesis dependency) β€” serves on :8020 MODEL_DIR=/path/to/models docker compose \ -f models/qwen3.6-27b/llama-cpp/compose/single/unsloth-q4km/mtp.yml up -d # single-card ik_llama (fastest single-card path) β€” :8020 MODEL_DIR=/path/to/models docker compose \ -f models/qwen3.6-27b/ik-llama/compose/single/ubergarm-iq4ks/mtp.yml up -d # dual-card vLLM β€” :8010 MODEL_DIR=/path/to/models docker compose \ -f models/qwen3.6-27b/vllm/compose/dual/autoround-int4/fp8-mtp.yml up -d # verify it's serving (use the port from the comment above), then stop it the same way: curl -s http://localhost:8020/v1/models | jq . docker compose -f down ``` `MODEL_DIR` is the only env var you must set β€” it's mounted as `/models`, and defaults to the in-repo `models-cache/` if your weights live there. Everything else has a sane default baked in; each compose **header** documents its own overrides (`GGUF_FILE`, `CTX_SIZE`, `UBATCH_SIZE`, …) and the exact `docker compose` line. `bash scripts/switch.sh --list` lists the variants runnable on this machine (hardware-filtered by GPU count; `--all` shows every variant), and a successful `switch.sh` run prints the compose path it used β€” so you can always recover the `-f` target. > ⚠ Launching directly skips the preflight that catches under-VRAM / wrong-GPU-count mistakes. If the container exits, check `docker logs 2>&1 | tail -50`. --- ## Repo layout ``` club-3090/ β”œβ”€β”€ README.md this file β€” start here β”œβ”€β”€ CHANGELOG.md cross-cutting changes (engine pin bumps, script updates) β”œβ”€β”€ LICENSE Apache-2.0 β”œβ”€β”€ docs/ β”‚ β”œβ”€β”€ LOCAL_AI_PRIMER.md plain-English on-ramp: hardware / engines / sizes / quants β”‚ β”œβ”€β”€ ARCHITECTURE.md how this stack thinks about LLM serving on 24 GB β”‚ β”œβ”€β”€ HARDWARE.md Ampere SM 8.6+, NVLink note, 24 GB ceilings β”‚ β”œβ”€β”€ WSL_SETUP.md Windows (WSL2) from-scratch setup walkthrough β”‚ β”œβ”€β”€ GLOSSARY.md plain-language definitions (TPS / KV / MTP / TP / etc.) β”‚ β”œβ”€β”€ UPSTREAM.md every upstream issue / PR we depend on or have filed β”‚ β”œβ”€β”€ CLIFFS.md full synopsis of the prefill cliffs (root causes + fix landscape) β”‚ β”œβ”€β”€ img/ chart sources (performance.svg, vram-budget-{single,dual,combined}.svg) + PNG exports β”‚ └── engines/ cross-model engine comparison + per-engine deep dives β”‚ β”œβ”€β”€ README.md decision tree, pros/cons matrix β”‚ β”œβ”€β”€ VLLM.md vLLM general docs + tuning β”‚ β”œβ”€β”€ LLAMA_CPP.md llama.cpp general docs + 262K recipe β”‚ β”œβ”€β”€ IK_LLAMA.md advanced-quant engine (IQK quants, two-stage spec-dec) β”‚ └── SGLANG.md blocked status + watch list β”œβ”€β”€ models/ β”‚ └── qwen3.6-27b/ all Qwen3.6-27B-specific stuff β”‚ β”œβ”€β”€ README.md model overview + variants + recommendations β”‚ β”œβ”€β”€ INTERNALS.md engineering rationale (Genesis, Marlin pad, DFlash, upstream tracker) β”‚ β”œβ”€β”€ CHANGELOG.md model-specific dated history β”‚ β”œβ”€β”€ vllm/ β”‚ β”‚ β”œβ”€β”€ README.md "vLLM recipes for Qwen3.6-27B" β”‚ β”‚ β”œβ”€β”€ compose/// compose files (e.g. dual/autoround-int4/fp8-mtp.yml) β”‚ β”‚ └── patches/ tolist_cudagraph + Marlin pad README + Genesis pointer β”‚ β”œβ”€β”€ llama-cpp/ β”‚ β”‚ β”œβ”€β”€ README.md "llama.cpp composes for Qwen3.6-27B" β”‚ β”‚ └── compose/single/unsloth-q4km/ mtp.yml + mtp-vision.yml + bounded-thinking.yml β”‚ β”œβ”€β”€ ik-llama/ β”‚ β”‚ └── compose/single/ubergarm-iq4ks/ mtp.yml + mtp-vision.yml + two-stage.yml (IQK quant) β”‚ └── sglang/ β”‚ └── README.md blocked status β€” what would unblock it on this model β”œβ”€β”€ scripts/ shared, model-aware β”‚ β”œβ”€β”€ setup.sh bash setup.sh β†’ preflight + downloads + verifies + Genesis β”‚ β”œβ”€β”€ launch.sh interactive wizard: model β†’ GPUs β†’ KV projection β†’ boots compose + verifies β”‚ β”œβ”€β”€ switch.sh stateless variant switcher (bring down old, up new) β”‚ β”œβ”€β”€ update.sh one-shot upgrade: git pull + re-pin Genesis + re-vendor patches β”‚ β”œβ”€β”€ health.sh runtime health probe (KV %, MTP AL, recent TPS, errors) β”‚ β”œβ”€β”€ preflight.sh sourceable lib: docker / GPU / disk / repo-drift / Genesis-pin checks β”‚ β”œβ”€β”€ verify.sh quick smoke test (engine-aware via env) β”‚ β”œβ”€β”€ verify-full.sh fast functional test (8 checks, ~1-2 min) β”‚ β”œβ”€β”€ verify-stress.sh boundary-case stress test (longctx ladder + tool prefill OOM, ~5-10 min) β”‚ β”œβ”€β”€ soak-test.sh runtime VRAM accretion / multi-turn agent traffic (~10-30 min, opt-in) β”‚ β”œβ”€β”€ bench.sh canonical TPS bench β”‚ β”œβ”€β”€ offload-matrix.sh CPU-offloaded MoE config sweep (llama.cpp forks) β”‚ └── report.sh paste-ready triage report (run before filing a bug or sharing bench numbers) └── tools/ └── charts/ re-generate docs/img/* SVGs and PNG exports (matplotlib) β”œβ”€β”€ gen-perf.py perf bar charts (combined + single + dual) └── gen-vram.py VRAM stacked bars (combined + single + dual) ``` --- ## What you'll need | For any model on this stack | Notes | |---|---| | 1Γ— or 2Γ— NVIDIA RTX 3090 (24 GB each) | Larger Ampere/Ada cards (4090, A6000) work; smaller cards (12 GB) don't fit 27B-class models. | | Linux (Ubuntu 22.04+ tested) | macOS/Windows: vLLM is Linux + CUDA only. Llama.cpp works on macOS/Windows but recipes assume Linux paths. **On Windows? See [docs/WSL_SETUP.md](docs/WSL_SETUP.md)** for the from-scratch WSL2 walkthrough. | | Docker + NVIDIA Container Toolkit | For vLLM. llama.cpp works without Docker. | | NVIDIA driver 580.x+ | For CUDA 13 runtime in vLLM nightly. | | ~30 GB free disk | Per model. More for multiple models. | vLLM image pins live in `scripts/lib/profiles/engines/*.yml` and are exported by `scripts/launch.sh` / `scripts/switch.sh` as `VLLM_NIGHTLY_SHA`. Set `VLLM_IMAGE` to override the full image ref β€” e.g. to pin a specific upstream nightly, or to run a current image when a pinned nightly has been purged: ```bash VLLM_IMAGE=vllm/vllm-openai:latest bash scripts/launch.sh --variant vllm/dual ``` See [docs/HARDWARE.md](docs/HARDWARE.md) for hardware-specific notes (PCIe vs NVLink, power draw, etc.). --- ## How this is structured **Engines and hardware are general** β€” the docs in `docs/` apply across models. vLLM works the same way regardless of whether you're serving Qwen, GLM, or Llama; the engine docs cover that once. **Models are specific** β€” under `models//`, you find that model's quants, quirks, recommended configs, and engine-specific recipes. Adding a new model means adding a new subdir with the same internal pattern. **Scripts are shared but model-aware** β€” `bash scripts/setup.sh qwen3.6-27b` downloads the right model + clones the right patches. When we add another model, you'd run `bash scripts/setup.sh glm-4.6` and the same script handles it. This separation keeps the stack maintainable as it grows. We don't want a model-specific README at the top; we want the top to be "stack docs" and the model details under their dedicated subdirs. --- ## Community - πŸ’¬ **[Discord](https://discord.gg/gzdfjhj5yN)** β€” casual chat, hardware questions, share what you're running. Use for synchronous Q&A. - πŸ“‹ **[GitHub Discussions](https://github.com/noonghunna/club-3090/discussions)** β€” async, searchable. Best for cross-rig benchmark drops, "should I tune X" type threads, and anything you want others to find via search. - πŸ› **[GitHub Issues](https://github.com/noonghunna/club-3090/issues)** β€” bug reports, regression repros, concrete asks. Triage ladder in [FAQ](docs/FAQ.md#before-symptom-matching--boot-the-simplest-stack-first) before filing. ## Community projects Projects in the club-3090 ecosystem maintained outside this repo: - **[VykosX/club-3090-server](https://github.com/VykosX/club-3090-server)** β€” single-file installer adding a server-management layer on top of club-3090: browser admin panel on `:8008/admin`, OpenAI-compatible reverse proxy on `:8009` with multi-backend routing, GPU-aware multi-instance orchestration, fan/power controls, audit logs, and per-user API auth/quota. Headless Arch + Debian/Ubuntu friendly. Started 2026-05-05, AGPL-3.0; see [discussion #108](https://github.com/noonghunna/club-3090/discussions/108) for the announcement and current WIP status. **Not yet officially adopted** β€” listed here as a community pointer until it converges on a stable surface area. If you've built something that integrates with club-3090 and you'd like a pointer added here, open a discussion. --- ## Migration history - **2026-04-28** β€” Repo created. Consolidates and supersedes: - [`noonghunna/qwen36-27b-single-3090`](https://github.com/noonghunna/qwen36-27b-single-3090) (single-card recipe; archived for issue history) - [`noonghunna/qwen36-dual-3090`](https://github.com/noonghunna/qwen36-dual-3090) (dual-card recipe; archived for issue history) Old repos remain readable for existing issue threads, external links (Medium articles, Reddit posts), and historical context. New issues should be filed here. See [CHANGELOG.md](CHANGELOG.md) for the merged dated history. --- ## Credits The stack stands on a lot of shoulders: - **Qwen team** ([@Alibaba_Qwen](https://huggingface.co/Qwen)) β€” for the base models and the MTP head architecture - **[Lorbus](https://huggingface.co/Lorbus/Qwen3.6-27B-int4-AutoRound)** β€” for the AutoRound INT4 quant with preserved BF16 `mtp.fc` (the model this whole stack runs on) - **[Sandermage](https://github.com/Sandermage/genesis-vllm-patches)** β€” Genesis patch tree for TurboQuant + hybrid models on consumer Ampere; root-causing #40880 and shipping the v7.14 fix - **[vibhavagarwal5](https://github.com/vllm-project/vllm/pull/38479)** β€” TurboQuant landing PR + tracking issue #40069 - **[vLLM project](https://github.com/vllm-project/vllm)** β€” the engine + active maintenance - **[llama.cpp](https://github.com/ggerganov/llama.cpp)** β€” the alternative engine path - **[Luce z-lab](https://github.com/luce-spec)** β€” DFlash N=5 draft model for Qwen3.6-27B - **Intel AutoRound** β€” quantization framework - **All cross-rig contributors** β€” [@ampersandru](https://github.com/ampersandru), [@walmis](https://github.com/walmis), [@3dluvr](https://github.com/3dluvr), and the Reddit / X local-LLM community for benchmark data and bug reports. --- ## License Apache 2.0. Do what you want with it. If you get better numbers on your rig β€” open an issue. If you add a new model with working configs β€” open a PR.