Muse Spark Open ↗Nebius Token Factory ↗

Did Muse catch the bug that actually shipped?

About Muse Spark Open · Try it on Token Factory

Read the reviews Code and raw data on GitHub

Shipped bugs caught

CaughtPartialMissed

Results

What the reviews found

Proof

Catches confirmed by running code

By ecosystem

Nine open-source projects, Django to Prettier

Same prompt and tools for every PR. Most Django bugs are backend logic with a clear failing input; the JavaScript set has more rendering and event-handling bugs, which are harder to spot in a diff.

In Muse's words

Three reviews worth reading

Every case

Every pull request, graded against the real fix

Open a card to read Muse's full review. The finding that matched the real defect is marked.

Usage

What each review cost

Method

How the benchmark works

Questions you might ask

How the results were checked

Raw data

Every record behind this page

The exact files this page is built from.

Every review (JSONL) Every model request and its tokens Grades and rationales Grading rubric Memory probe answers Summary Case list

Reproduce

Run it on your own key

The cases, grades, reproduction scripts, every review and the per-request token ledger are in the repository. Clone it, add a Token Factory key, and rerun everything.

Studio1-OSS/muse-spark-open-bench

git clone https://github.com/Studio1-OSS/muse-spark-open-bench
npm ci && cp .env.example .env   # add NEBIUS_API_KEY
npm run cases -- repos   # build the case folders
npm run bench -- run     # Muse reviews every PR
npm run repro -- repos   # before / merged / fix runs
npm run site             # rebuild this page