Jose Romero

← All benchmarks← Back to the post

Benchmark report · November 3 to 4, 2026

Is the base M6 Mac mini actually good for local AI?

I gave the cheapest Mac mini (M6, 16 GB) real work with five small local models, then ran the same models on my M5 Max Mac Studio held to the same 11.4 GB model budget. 1,214 core requests, graded answers, and every failure kept.

One overnight run on my two Macs. An AI agent built the harness and ran the queue, and a separate AI judge graded the answers blind. This is a hobby test, not a lab. Details in Methods and limits.

What I found

5 / 5Qwen 3.5 4B fixed the same small bug every time on the Mini, in about 69 seconds on average. One bug, five attempts.
0 / 10usable answers from LFM 2.5, the fastest model on the Mini at 121.2 tokens per second. Most of those were empty answers, not wrong ones. What happened.
2.2x to 3.3xhow much faster the Studio generated text on the same models with the same model budget.
2 / 100longer workflow tasks fully passed on the Mini (9 of 100 on the Studio). Unattended work was unreliable.
1.05 sto transcribe a minute of audio with Whisper small on the Mini (1.60 s with large v3 turbo). Short clips only.
7 / 30generated images judged usable. All 6 of 6 product photos passed; almost nothing else did.
0 / 15image edits judged usable with plain image to image.
0memory guard stops in the whole run. The failures were not the Mini running out of room.

Short version. The base Mini is useful for a few jobs you can check: drafts with Gemma 4 12B, a small code change with tests on Qwen 3.5 4B, a first transcript with Whisper, quick product-style images. It was not dependable for longer jobs left to run on their own. The Studio runs the same models about three times faster; what its 64 GB can do with bigger models is a separate test I have not run. I returned the Mini. That is my call for my use, not a verdict on the machine.

Read this first

  • Small samples. Most cells are five attempts. One overnight run, one unit of each Mac.
  • An AI graded the answers. One judge model, one pass, blind to model and machine. Not a human panel.
  • Fixed settings, no tuning, no retries. Every model got the same output caps and the same agent. Some zeros below are a model and this harness not fitting together, not proof the model cannot do the task.
  • Coverage is not a pass rate. 179 of 200 workflow tasks got a score; 11 of 200 fully passed (Mini 2 of 100, Studio 9 of 100).
  • The code fix is one bug, five times. Repeatable on that bug. Not a coding benchmark.
  • Same model budget, not a 16 GB simulation. The Studio's limit was watched, not enforced, and its full 64 GB was not tested.
  • No cloud model and no cost math. Nothing here compares against a subscription.
  • Short audio clips. The longest is 93 seconds. Do not scale the speed to an hour-long meeting.

The full list is in Methods and limits. Four things I said wrong on camera are in Corrections.

Speed: same models, two chips

Generation speed is new output tokens per second on matched first turns (the same request on both machines, warm-up excluded). A token is a piece of a word, so this is how fast a model types, not how good the answer is. For LFM 2.5 most of those tokens were hidden reasoning that never reached the answer.

Mac mini (M6, 16 GB) Mac Studio (M5 Max, same model budget) · output tokens per second (higher is faster)

Qwen 3.5 4Bn = 5 matched pairs · 2.93x49.5145.2Qwen 3.5 9Bn = 5 matched pairs · 3.28x29.195.4Gemma 4 12Bn = 4 matched pairs · 3.15x19.361.0LFM 2.5 8Bn = 5 matched pairs · 2.17x121.2263.2Phi-4 minin = 5 matched pairs · 2.90x58.7170.5
ModelMac mini tok/sMac Studio tok/sStudio vs MiniMatched pairs
Qwen 3.5 4B49.5145.22.93x5
Qwen 3.5 9B29.195.43.28x5
Gemma 4 12B19.361.03.15x4
LFM 2.5 8B121.2263.22.17x5
Phi-4 mini58.7170.52.90x5

Reading a long prompt (prefill)

Before a model writes anything it has to read the prompt. This is prompt tokens per second on a matched document of 480 records. All five models read 3.09 to 3.45 times faster on the Studio, so long documents are where the gap is felt most.

Mac mini (M6, 16 GB) Mac Studio (M5 Max, same model budget) · prompt tokens per second on a 480-record document

Qwen 3.5 4Bn = 5 · 3.13x1,6905,286Qwen 3.5 9Bn = 4 · 3.32x999.13,318Gemma 4 12Bn = 3 · 3.45x551.91,903LFM 2.5 8Bn = 5 · 3.09x4,51913,984Phi-4 minin = 5 · 3.11x1,7055,299
ModelMac mini prompt tok/sMac Studio prompt tok/sStudio vs MiniMatched pairs
Qwen 3.5 4B1,6905,2863.13x5
Qwen 3.5 9B9993,3183.32x4
Gemma 4 12B5521,9033.45x3
LFM 2.5 8B4,51913,9843.09x5
Phi-4 mini1,7055,2993.11x5

The wait before the first word

What that reading speed means in seconds: the time from sending the 480-record document (about 19,000 prompt tokens) until the first visible text. LFM 2.5 returned no visible text on this task.

ModelPrompt tokensMac mini, secondsMac Studio, secondsRequests (Mini / Studio)
Qwen 3.5 4B19,30411.53.75 / 5
Qwen 3.5 9B19,30419.45.94 / 5
Gemma 4 12B19,31535.110.13 / 5
LFM 2.5 8B17,976no visible answerno visible answer5 / 5
Phi-4 mini16,5139.02.95 / 5

Loading a model

The first tiny request right after each model's server started, which includes loading the model into memory. One measurement per model and machine, files already on the internal SSD, disk cache not controlled, so read these as rough: loading took seconds, not minutes.

ModelModel fileMac mini, secondsMac Studio, seconds
Qwen 3.5 4B3.0 GB1.82.2
Qwen 3.5 9B6.0 GB2.00.5
Gemma 4 12B6.7 GB3.71.9
LFM 2.5 8B4.8 GB2.10.4
Phi-4 mini2.2 GB1.00.2
Every core task family, both machines (68 rows)

Means over the listed attempts. "prefix reused" means the runtime had the start of the prompt cached from the previous turn. Do not average unrelated tasks into one chip speed.

ModelMachineTaskCacheAttemptsWall sPrompt tok/sOutput tok/sHit length capEmpty answers
Qwen 3.5 4BMac minichatuncached1004.061,82948.900
Qwen 3.5 4BMac Studiochatuncached1001.375,460142.200
Qwen 3.5 4BMac minidocuments-240uncached57.581,86744.400
Qwen 3.5 4BMac Studiodocuments-240uncached52.485,835131.800
Qwen 3.5 4BMac minidocuments-480uncached514.841,69037.900
Qwen 3.5 4BMac Studiodocuments-480uncached54.725,286125.200
Qwen 3.5 4BMac minidocuments-60uncached53.411,93948.000
Qwen 3.5 4BMac Studiodocuments-60uncached51.145,976141.000
Qwen 3.5 4BMac minitoolsprefix reused51.1456749.200
Qwen 3.5 4BMac Studiotoolsprefix reused50.391,482145.900
Qwen 3.5 4BMac minitoolsuncached51.521,51149.905
Qwen 3.5 4BMac Studiotoolsuncached50.703,920145.605
Qwen 3.5 4BMac miniwritinguncached54.611,12449.600
Qwen 3.5 4BMac Studiowritinguncached51.582,738145.200
Qwen 3.5 9BMac minichatuncached838.241,06128.800
Qwen 3.5 9BMac Studiochatuncached1002.603,27793.200
Qwen 3.5 9BMac minidocuments-240uncached412.951,09027.600
Qwen 3.5 9BMac Studiodocuments-240uncached54.043,49189.600
Qwen 3.5 9BMac minidocuments-480uncached423.7299925.800
Qwen 3.5 9BMac Studiodocuments-480uncached57.183,31985.700
Qwen 3.5 9BMac minidocuments-60uncached46.181,12628.700
Qwen 3.5 9BMac Studiodocuments-60uncached51.963,49391.500
Qwen 3.5 9BMac minitoolsprefix reused41.7741829.700
Qwen 3.5 9BMac Studiotoolsprefix reused50.571,11894.900
Qwen 3.5 9BMac minitoolsuncached42.4587629.704
Qwen 3.5 9BMac Studiotoolsuncached50.962,46897.005
Qwen 3.5 9BMac miniwritinguncached48.4465829.300
Qwen 3.5 9BMac Studiowritinguncached52.541,77195.700
Gemma 4 12BMac minichatprefix reused267.3240319.000
Gemma 4 12BMac Studiochatprefix reused332.361,10759.300
Gemma 4 12BMac minichatuncached439.4665118.700
Gemma 4 12BMac Studiochatuncached673.022,05658.300
Gemma 4 12BMac minidocuments-240uncached324.0560118.300
Gemma 4 12BMac Studiodocuments-240uncached57.392,02356.300
Gemma 4 12BMac minidocuments-480uncached343.0655218.000
Gemma 4 12BMac Studiodocuments-480uncached512.741,90956.900
Gemma 4 12BMac minidocuments-60prefix reused313.3963418.600
Gemma 4 12BMac Studiodocuments-60prefix reused54.332,00257.400
Gemma 4 12BMac minitoolsprefix reused33.4034619.500
Gemma 4 12BMac Studiotoolsprefix reused51.1196260.600
Gemma 4 12BMac minitoolsuncached32.1438619.903
Gemma 4 12BMac Studiotoolsuncached50.9494664.805
Gemma 4 12BMac miniwritinguncached315.6236219.300
Gemma 4 12BMac Studiowritinguncached55.0893160.100
LFM 2.5 8BMac minichatuncached1004.183,047120.78883
LFM 2.5 8BMac Studiochatuncached1001.927,971261.88883
LFM 2.5 8BMac minidocuments-240uncached56.594,685110.555
LFM 2.5 8BMac Studiodocuments-240uncached52.7314,554245.555
LFM 2.5 8BMac minidocuments-480uncached59.054,519101.855
LFM 2.5 8BMac Studiodocuments-480uncached53.5813,984229.055
LFM 2.5 8BMac minidocuments-60uncached54.934,462117.155
LFM 2.5 8BMac Studiodocuments-60uncached52.1912,869257.455
LFM 2.5 8BMac minitoolsuncached54.461,881121.155
LFM 2.5 8BMac Studiotoolsuncached52.223,664263.255
LFM 2.5 8BMac miniwritinguncached54.301,685121.355
LFM 2.5 8BMac Studiowritinguncached51.983,954263.255
Phi-4 miniMac minichatprefix reused1006.7118650.5460
Phi-4 miniMac Studiochatprefix reused1001.87425151.6320
Phi-4 miniMac minidocuments-240prefix reused56.242,12040.700
Phi-4 miniMac Studiodocuments-240prefix reused52.066,440124.300
Phi-4 miniMac minidocuments-480prefix reused512.211,70530.200
Phi-4 miniMac Studiodocuments-480prefix reused53.925,29999.100
Phi-4 miniMac minidocuments-60prefix reused53.152,29253.400
Phi-4 miniMac Studiodocuments-60prefix reused51.116,613153.600
Phi-4 miniMac minitoolsprefix reused50.9845860.600
Phi-4 miniMac Studiotoolsprefix reused50.501,119174.000
Phi-4 miniMac miniwritingprefix reused57.1779358.610
Phi-4 miniMac Studiowritingprefix reused52.521,978168.910
Core requests that never ran (12 cells, all on the Mini)

Each model had a fixed time window. On the Mini the two slowest models ran out of it before finishing the plan.

ModelMachineTaskRanPlanned
Qwen 3.5 9BMac minichat83100
Qwen 3.5 9BMac minidocuments-24045
Qwen 3.5 9BMac minidocuments-48045
Qwen 3.5 9BMac minidocuments-6045
Qwen 3.5 9BMac minitools45
Qwen 3.5 9BMac miniwriting45
Gemma 4 12BMac minichat69100
Gemma 4 12BMac minidocuments-24035
Gemma 4 12BMac minidocuments-48035
Gemma 4 12BMac minidocuments-6035
Gemma 4 12BMac minitools35
Gemma 4 12BMac miniwriting35
Memory actually used per model

Peak sampled process footprint during the core queue, one sample per second (short peaks can be missed). Budget: 11.4 GB on both machines. The highest Mini peak was 10.0 GB.

ModelMachinePeak footprintPeak MLX allocationSwap growthGuard stop
Qwen 3.5 4BMac mini5.4 GB4.8 GB0 MBno
Qwen 3.5 4BMac Studio5.7 GB5.4 GB0 MBno
Qwen 3.5 9BMac mini8.2 GB7.5 GB0 MBno
Qwen 3.5 9BMac Studio8.1 GB7.6 GB0 MBno
Gemma 4 12BMac mini10.0 GB9.6 GB0 MBno
Gemma 4 12BMac Studio10.3 GB9.8 GB0 MBno
LFM 2.5 8BMac mini6.2 GB5.9 GB0 MBno
LFM 2.5 8BMac Studio6.5 GB6.0 GB0 MBno
Phi-4 miniMac mini7.4 GB7.1 GB0 MBno
Phi-4 miniMac Studio7.6 GB7.1 GB0 MBno

Graded answers: conversation and writing

Each model got five multi-turn conversations (planning a small workshop from fixed numbers, 20 turns) and five writing tasks (a short video introduction with three titles from supplied notes). Both prompts are fictional. A separate AI judge graded each saved answer without knowing the model, the machine or the speed. Usable means worth working with, not ready to publish. Results that never arrived stay in the denominator.

judged usable planned answers (10 per model)

Qwen 3.5 4B3 / 10Qwen 3.5 9B0 / 10 · 1 never ranGemma 4 12B7 / 10 · 3 never ranLFM 2.5 8B0 / 10Phi-4 mini0 / 10
ModelMini · conversationMini · writingStudio · conversationStudio · writing
Qwen 3.5 4B0 / 53 / 50 / 53 / 5
Qwen 3.5 9B0 / 50 / 5 1 never ran0 / 50 / 5
Gemma 4 12B4 / 5 1 never ran3 / 5 2 never ran5 / 55 / 5
LFM 2.5 8B0 / 50 / 50 / 50 / 5
Phi-4 mini0 / 50 / 50 / 50 / 5

On the Mini, Gemma 4 12B was the pick of this group: 4 of 5 conversations and 3 of 5 writing tasks usable, at about 19 tokens per second. Every Gemma answer that existed was judged usable; the misses are the 3 that never ran inside the Mini's time window. 96 of 100 planned answers existed to be graded.

Read all 96 saved answers with the judge's notes

What the zeros actually were

A zero can mean a wrong answer or no answer. They are not the same thing, so here is the split for every graded item.

ModelMachineGradedUsableNo visible answer at allMostly empty turnsVisible answer, judged not usableEmpty turnsTurns stopped at the output cap
Qwen 3.5 4BMac mini1030070 of 1050 of 105
Qwen 3.5 4BMac Studio1030070 of 1050 of 105
Qwen 3.5 9BMac mini900090 of 870 of 87
Qwen 3.5 9BMac Studio10000100 of 1050 of 105
Gemma 4 12BMac mini770000 of 720 of 72
Gemma 4 12BMac Studio10100000 of 1050 of 105
LFM 2.5 8BMac mini10055088 of 10593 of 105
LFM 2.5 8BMac Studio10055088 of 10593 of 105
Phi-4 miniMac mini10000100 of 10547 of 105
Phi-4 miniMac Studio10000100 of 10533 of 105

LFM 2.5: fast, and mostly empty rather than wrong. In the video I call it "super fast and super wrong". The saved data is more specific. On the Mini, 5 of its 10 graded items had no visible answer at all and 5 were conversations where most turns came back empty. Across its Mini core requests, 108 of 125 returned nothing visible and stopped at the 512 token output cap, and all 125 carried hidden reasoning text: the model spent its output budget thinking and ran out before it wrote the answer.

The judge's own words on a writing task: "The sole assistant output is empty, so no introduction, titles or useful factual response can be assessed." And on a conversation: "All 20 attempted turns were reviewed. Seventeen are empty and only three invitations are visible. These do not substitute for the absent plan, calculations and risk answers; there is insufficient substantive dialogue to establish financial consistency."

So this is likely an interaction between this model's reasoning habit, its chat template and my 512 token cap, not proof that LFM 2.5 cannot do these tasks. I did not raise the cap, turn reasoning off for it, or retry. With the settings I ran, it was not usable; with other settings it may be.

Phi-4 mini is the opposite case. It answered every turn, and the judge rejected the answers on content: 10 of 10 graded Mini items had visible answers judged not usable. For example: "The original full-capacity profit is correct, but repeated break-even and volunteer calculations are materially wrong. Invitations invent a location, discounts and early closing, while every risk response exceeds the length limit. These defects recur across all four cycles."

The code fix

Each model got the same small bug five times through the Pi coding agent: a failing test suite in a small Python inventory project, with the instruction to find the root causes and fix them without touching the tests. Output was capped at 2,048 tokens per model turn, with five minutes per attempt. A pass means the failing tests, the full visible suite and hidden edge-case checks all passed and no stray files were left behind.

passed every check attempts (5 per model)

Qwen 3.5 4B5 / 5Qwen 3.5 9B2 / 5Gemma 4 12B0 / 5LFM 2.5 8B0 / 5Phi-4 mini0 / 5
ModelMini · bug fixMini · log analyzerStudio · bug fixStudio · log analyzer
Qwen 3.5 4B5 / 5 69 s avg0 / 55 / 5 22 s avg0 / 5
Qwen 3.5 9B2 / 5 92 s avg0 / 52 / 5 38 s avg0 / 5
Gemma 4 12B0 / 50 / 50 / 50 / 5
LFM 2.5 8B0 / 50 / 50 / 50 / 5
Phi-4 mini0 / 50 / 50 / 50 / 5

On the Mini, Qwen 3.5 4B passed 5 of 5 (individual runs: 88, 92, 43, 72, 49 seconds). The larger Qwen 3.5 9B passed 2 of 5. The other three models passed none. This is one bug, five times: it shows repeatability on that bug, not general coding ability.

What the failed attempts actually did

None of the failures was a memory stop or a five-minute timeout. They fall into three groups, read from the saved agent sessions (the raw session files are not published).

ModelMachineBug fixLog analyzer
Qwen 3.5 4BMac mini5 of 5: passed5 of 5: explored with tools, then hit the output cap before changing a file
Qwen 3.5 4BMac Studio5 of 5: passed5 of 5: explored with tools, then hit the output cap before changing a file
Qwen 3.5 9BMac mini3 of 5: explored with tools, then hit the output cap before changing a file; 2 of 5: passed5 of 5: explored with tools, then hit the output cap before changing a file
Qwen 3.5 9BMac Studio2 of 5: passed; 2 of 5: explored with tools, then hit the output cap before changing a file; 1 of 5: changed files; checks still failed5 of 5: explored with tools, then hit the output cap before changing a file
Gemma 4 12BMac mini5 of 5: explored with tools, then hit the output cap before changing a file5 of 5: no tool call; hit the output cap while writing text
Gemma 4 12BMac Studio5 of 5: explored with tools, then hit the output cap before changing a file5 of 5: no tool call; hit the output cap while writing text
LFM 2.5 8BMac mini5 of 5: no tool call; answered in text and stopped5 of 5: no tool call; answered in text and stopped
LFM 2.5 8BMac Studio5 of 5: no tool call; answered in text and stopped5 of 5: no tool call; answered in text and stopped
Phi-4 miniMac mini4 of 5: no tool call; answered in text and stopped; 1 of 5: no tool call; hit the output cap while writing text3 of 5: no tool call; hit the output cap while writing text; 2 of 5: no tool call; answered in text and stopped
Phi-4 miniMac Studio5 of 5: no tool call; answered in text and stopped4 of 5: no tool call; answered in text and stopped; 1 of 5: no tool call; hit the output cap while writing text
  • LFM 2.5 and Phi-4 mini never called a tool. LFM 2.5 wrote its tool request as plain text instead of in the format the agent reads, so the agent saw a normal reply and stopped after a second or two. Phi-4 mini described a plan in prose and stopped. Neither produced a wrong patch; they produced no patch. That is a tool-format mismatch with this agent as much as a coding result.
  • Gemma 4 12B started correctly (listed the files, ran the tests) and then used the whole 2,048 token turn on reasoning before making an edit.
  • Qwen 3.5 9B explored for many turns and in its failed runs hit the same cap before a working edit.
  • The log analyzer task went 0 for 50 at this cap (0 passes). It asks for a complete command-line program in one file, and 34 of 50 attempts ended at the 2,048 token cap, mostly while writing that file. Read it as "does not fit in this output budget", not "cannot write the program". The same task with an 8,192 token cap is in the next section.
The actual fix from attempt 1 (Qwen 3.5 4B, Mini)
 b/stockroom/loader.py
index d6144e0..fc560b7 100755
--- a/stockroom/loader.py
+++ b/stockroom/loader.py
@@ -38,7 +38,7 @@ def parse_row(row: dict) -> Movement:
             raise ValueError("receive rows need unit_cents")
         return Movement(normalize_sku(row["sku"]), qty, cost, note=note)
     if kind == "ship":
-        return Movement(row["sku"], -qty, _cents(row.get("unit_cents")), note=note)
+        return Movement(normalize_sku(row["sku"]), -qty, _cents(row.get("unit_cents")), note=note)
     raise ValueError(f"unknown movement kind {kind!r}")
 
 

Checks passed: failing tests pass, full visible suite, hidden ledger edges, hidden loader edges, footprint clean.

Structured-answer checks in the core suite (0 of 188 passed)

Short single-shot tasks with an exact expected structure (document extraction at three sizes and a tool-call task), 512 token output cap, strict automatic scoring: one wrong value fails the whole answer. No model passed any of them. In 47 of the 141 document answers every required field was present but at least one value was wrong; the rest were missing fields or were not valid structured output. LFM 2.5's zeros here are the same empty answers described above.

ModelMachinedocuments-240documents-480documents-60tools
Qwen 3.5 4BMac mini0 / 50 / 50 / 50 / 5
Qwen 3.5 4BMac Studio0 / 50 / 50 / 50 / 5
Qwen 3.5 9BMac mini0 / 40 / 40 / 40 / 4
Qwen 3.5 9BMac Studio0 / 50 / 50 / 50 / 5
Gemma 4 12BMac mini0 / 30 / 30 / 30 / 3
Gemma 4 12BMac Studio0 / 50 / 50 / 50 / 5
LFM 2.5 8BMac mini0 / 50 / 50 / 50 / 5
LFM 2.5 8BMac Studio0 / 50 / 50 / 50 / 5
Phi-4 miniMac mini0 / 50 / 50 / 50 / 5
Phi-4 miniMac Studio0 / 50 / 50 / 50 / 5

Longer jobs with moving parts

Closer to everyday automation: reconcile three synthetic documents, build a small working billing website, organize a folder of synthetic files, plus the log analyzer again with a larger output budget. Five attempts per cell, five minutes each, 8,192 output tokens, scored by automatic checks. All names, invoices and files are made up for the test.

179 of 200 scheduled tasks finished far enough to get a score. That is coverage, not a pass rate: across both machines and all five models, 11 of 200 fully passed. On the Mini, 2 of 100. On the Studio, 9 of 100. The full passes: Qwen 3.5 4B 7 (Mac mini documents 1, Mac mini website 1, Mac Studio documents 4, Mac Studio website 1); Qwen 3.5 9B 4 (Mac Studio website 3, Mac Studio folders 1). Gemma 4 12B, LFM 2.5 and Phi-4 mini had none.

fully passed scheduled tasks (20 per model per machine)

Qwen 3.5 4B · Mac mini2 / 20 · 15 scoredQwen 3.5 4B · Mac Studio5 / 20 · 20 scoredQwen 3.5 9B · Mac mini0 / 20 · 14 scoredQwen 3.5 9B · Mac Studio4 / 20 · 20 scoredGemma 4 12B · Mac mini0 / 20 · 10 scoredGemma 4 12B · Mac Studio0 / 20 · 20 scoredLFM 2.5 8B · Mac mini0 / 20 · 20 scoredLFM 2.5 8B · Mac Studio0 / 20 · 20 scoredPhi-4 mini · Mac mini0 / 20 · 20 scoredPhi-4 mini · Mac Studio0 / 20 · 20 scored
ModelMachineDocumentsWebsiteFoldersLog analyzer (8,192 tokens)
Qwen 3.5 4BMac mini1 / 5 4 scored; mean score 68; 1 never started1 / 5 4 scored; mean score 69; 1 never started0 / 5 4 scored; mean score 35; 1 never started0 / 5 3 scored; mean score 10; 1 never started; 1 no score
Qwen 3.5 4BMac Studio4 / 5 5 scored; mean score 971 / 5 5 scored; mean score 300 / 5 5 scored; mean score 320 / 5 5 scored; mean score 18
Qwen 3.5 9BMac mini0 / 5 4 scored; mean score 10; 1 never started0 / 5 3 scored; mean score 33; 1 never started; 1 no score0 / 5 4 scored; mean score 20; 1 never started0 / 5 3 scored; mean score 27; 2 never started
Qwen 3.5 9BMac Studio0 / 5 5 scored; mean score 103 / 5 5 scored; mean score 661 / 5 5 scored; mean score 360 / 5 5 scored; mean score 34
Gemma 4 12BMac mini0 / 5 3 scored; mean score 10; 2 never started0 / 5 2 scored; mean score 2; 2 never started; 1 no score0 / 5 3 scored; mean score 10; 2 never started0 / 5 2 scored; mean score 0; 3 never started
Gemma 4 12BMac Studio0 / 5 5 scored; mean score 100 / 5 5 scored; mean score 10 / 5 5 scored; mean score 100 / 5 5 scored; mean score 0
LFM 2.5 8BMac mini0 / 5 5 scored; mean score 100 / 5 5 scored; mean score 50 / 5 5 scored; mean score 100 / 5 5 scored; mean score 0
LFM 2.5 8BMac Studio0 / 5 5 scored; mean score 100 / 5 5 scored; mean score 50 / 5 5 scored; mean score 100 / 5 5 scored; mean score 0
Phi-4 miniMac mini0 / 5 5 scored; mean score 100 / 5 5 scored; mean score 50 / 5 5 scored; mean score 100 / 5 5 scored; mean score 0
Phi-4 miniMac Studio0 / 5 5 scored; mean score 100 / 5 5 scored; mean score 50 / 5 5 scored; mean score 100 / 5 5 scored; mean score 0

With Qwen 3.5 4B on the Mini: documents 1 of 5, website 1 of 5, folders 0 of 5. A partial score means some checks passed; for example, leaving the source files untouched earns points even when nothing useful was produced.

The instruction: documents
Reconcile the three source files in this folder (invoice.pdf, amendment.docx and activity.html). Create reconciled.json with exactly these keys: invoice_id, currency, customer, subtotal_cents, shipping_cents, tax_cents, approved_credit_cents, deposit_paid_cents, balance_due_cents, current_due_date, supplier_contact_email, source_files. Use integer cents, null for unknown contact email, and the list of the three filenames for source_files. Do not double count the deposit or include void/example records. Also create reconciliation.csv with one header row and one record using all scalar fields (omit source_files). Create summary.txt with a concise explanation of the changes and uncertainty. Use actual source contents; do not alter them. Python 3 and pypdf are available. You may inspect DOCX with Python zipfile/XML. No network access is needed.

Attempt 1 of 5. The other four use the same instruction with different synthetic files.

The instruction: website
Build an offline billing dashboard for the supplied invoices.json. Create index.html, styles.css and app.mjs, using plain HTML/CSS/JavaScript with no dependencies or external requests. It will be served by python3 -m http.server. Load invoices.json and show a readable table of invoice, customer, due date, status and formatted USD balance. Provide labelled status, minimum-balance (dollars) and customer-search controls; filters combine. Include a visible empty-results state, a clear loading/error state, and a button that downloads CSV for the filtered rows. Do not insert customer text as raw HTML. Make the layout usable on desktop and mobile and keyboard accessible.

Implement and export these pure functions from app.mjs so the logic can be tested by Node as well as used in the browser. Guard browser-only code so importing app.mjs in Node works.
selectInvoices(records, status, minBalanceCents, search): status is "all", "open", "paid" or "void"; filter by exact status unless all, balance >= minBalanceCents, and case-insensitive customer substring search. Return a new array sorted by due_date then id; do not mutate the input.
summarize(records, asOf): return exactly {outstandingCents, overdueCents, unpaidCount}. Only status=open records with positive balances count. Overdue means due_date strictly earlier than asOf. The dashboard summary uses all source records, independent of filters, and asOf=2026-11-01.
toCSV(records): return CSV text with header id,customer,due_date,status,balance_cents and one record per supplied row in order. Quote fields correctly when they contain comma, quote or newline; double embedded quotes. No currency conversion in CSV.

Keep invoices.json unchanged. Also write README.md with the launch command and a short verification checklist. Build the actual files, not just an explanation.

Attempt 1 of 5. The other four use the same instruction with different synthetic files.

The instruction: folders
Organize only the files in inbox. Infer category and date from file contents, not filenames or modification dates. Copy each unique, confidently classified file to organized/<category>/<YYYY-MM>/<original-filename>, where categories are invoices, receipts and meetings. If byte-identical duplicates exist, choose the lexicographically first filename as canonical and do not copy the duplicates. Leave uncertain files uncopied. Preserve every inbox file unchanged. Write manifest.json as an array sorted by source, each entry with exactly source (inbox/name), destination (relative path or null), sha256, status (copied, duplicate, or needs_review), and duplicate_of (canonical inbox path for duplicates, otherwise null). Write a short review.txt listing anything needing human attention. No deletion or network access.

Attempt 1 of 5. The other four use the same instruction with different synthetic files.

All 200 attempts, one by one

Score is 0 to 100 from the automatic checks. "Checks failed" lists the first few failed checks by name.

ModelMachineTaskAttemptStatusScoreFull passSecondsChecks failed
Qwen 3.5 4BMac miniDocuments1scored100pass48
Qwen 3.5 4BMac miniDocuments2scored10no46field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 4BMac miniDocuments3scored75no300csv values, summary present
Qwen 3.5 4BMac miniDocuments4scored85no294csv values
Qwen 3.5 4BMac miniDocuments5never started (time window ended)n/anon/a
Qwen 3.5 4BMac miniWebsite1scored15no297sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 4BMac miniWebsite2scored80no182summary
Qwen 3.5 4BMac miniWebsite3scored100pass221
Qwen 3.5 4BMac miniWebsite4scored80no300summary
Qwen 3.5 4BMac miniWebsite5never started (time window ended)n/anon/a
Qwen 3.5 4BMac miniFolders1scored40no39manifest exact
Qwen 3.5 4BMac miniFolders2scored40no38manifest exact
Qwen 3.5 4BMac miniFolders3scored40no216manifest exact
Qwen 3.5 4BMac miniFolders4scored20no140manifest exact, copies and hashes
Qwen 3.5 4BMac miniFolders5never started (time window ended)n/anon/a
Qwen 3.5 4BMac miniLog analyzer (8,192 tokens)1scored10no330c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac miniLog analyzer (8,192 tokens)2scored10no330c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac miniLog analyzer (8,192 tokens)3scored10no330c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac miniLog analyzer (8,192 tokens)4started, no scorecardn/anon/a
Qwen 3.5 4BMac miniLog analyzer (8,192 tokens)5never started (time window ended)n/anon/a
Qwen 3.5 4BMac StudioDocuments1scored85no14csv values
Qwen 3.5 4BMac StudioDocuments2scored100pass36
Qwen 3.5 4BMac StudioDocuments3scored100pass26
Qwen 3.5 4BMac StudioDocuments4scored100pass32
Qwen 3.5 4BMac StudioDocuments5scored100pass26
Qwen 3.5 4BMac StudioWebsite1scored15no146sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 4BMac StudioWebsite2scored15no170sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 4BMac StudioWebsite3scored100pass52
Qwen 3.5 4BMac StudioWebsite4scored15no100sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 4BMac StudioWebsite5scored5no87sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Qwen 3.5 4BMac StudioFolders1scored20no13manifest exact, copies and hashes
Qwen 3.5 4BMac StudioFolders2scored40no14manifest exact
Qwen 3.5 4BMac StudioFolders3scored40no11manifest exact
Qwen 3.5 4BMac StudioFolders4scored40no12manifest exact
Qwen 3.5 4BMac StudioFolders5scored20no13manifest exact, copies and hashes
Qwen 3.5 4BMac StudioLog analyzer (8,192 tokens)1scored10no115c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac StudioLog analyzer (8,192 tokens)2scored10no141c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac StudioLog analyzer (8,192 tokens)3scored10no125c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 4BMac StudioLog analyzer (8,192 tokens)4scored60no112c4 malformed and validation, c5 multi file and rankings
Qwen 3.5 4BMac StudioLog analyzer (8,192 tokens)5scored0no60c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac miniDocuments1scored10no5field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac miniDocuments2scored10no24field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac miniDocuments3scored10no191field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac miniDocuments4scored10no40field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac miniDocuments5never started (time window ended)n/anon/a
Qwen 3.5 9BMac miniWebsite1scored5no300sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Qwen 3.5 9BMac miniWebsite2scored5no300sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Qwen 3.5 9BMac miniWebsite3scored90no300deliverables present
Qwen 3.5 9BMac miniWebsite4started, no scorecardn/anon/a
Qwen 3.5 9BMac miniWebsite5never started (time window ended)n/anon/a
Qwen 3.5 9BMac miniFolders1scored20no69manifest exact, copies and hashes
Qwen 3.5 9BMac miniFolders2scored20no300manifest exact, copies and hashes
Qwen 3.5 9BMac miniFolders3scored10no300manifest exact, copies and hashes, review present
Qwen 3.5 9BMac miniFolders4scored30no300manifest exact, review present
Qwen 3.5 9BMac miniFolders5never started (time window ended)n/anon/a
Qwen 3.5 9BMac miniLog analyzer (8,192 tokens)1scored30no330c2 filters correct, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac miniLog analyzer (8,192 tokens)2scored0no330c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac miniLog analyzer (8,192 tokens)3scored50no330c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac miniLog analyzer (8,192 tokens)4never started (time window ended)n/anon/a
Qwen 3.5 9BMac miniLog analyzer (8,192 tokens)5never started (time window ended)n/anon/a
Qwen 3.5 9BMac StudioDocuments1scored10no2field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac StudioDocuments2scored10no2field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac StudioDocuments3scored10no2field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac StudioDocuments4scored10no2field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac StudioDocuments5scored10no2field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Qwen 3.5 9BMac StudioWebsite1scored100pass156
Qwen 3.5 9BMac StudioWebsite2scored15no300sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 9BMac StudioWebsite3scored15no262sort, combined filter, customer search, empty results, summary, input immutable and 1 more
Qwen 3.5 9BMac StudioWebsite4scored100pass262
Qwen 3.5 9BMac StudioWebsite5scored100pass220
Qwen 3.5 9BMac StudioFolders1scored100pass32
Qwen 3.5 9BMac StudioFolders2scored20no20manifest exact, copies and hashes
Qwen 3.5 9BMac StudioFolders3scored20no21manifest exact, copies and hashes
Qwen 3.5 9BMac StudioFolders4scored20no21manifest exact, copies and hashes
Qwen 3.5 9BMac StudioFolders5scored20no21manifest exact, copies and hashes
Qwen 3.5 9BMac StudioLog analyzer (8,192 tokens)1scored70no283c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac StudioLog analyzer (8,192 tokens)2scored10no190c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac StudioLog analyzer (8,192 tokens)3scored10no162c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac StudioLog analyzer (8,192 tokens)4scored10no225c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Qwen 3.5 9BMac StudioLog analyzer (8,192 tokens)5scored70no182c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac miniDocuments1scored10no300field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac miniDocuments2scored10no300field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac miniDocuments3scored10no300field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac miniDocuments4never started (time window ended)n/anon/a
Gemma 4 12BMac miniDocuments5never started (time window ended)n/anon/a
Gemma 4 12BMac miniWebsite1scored0no300sort, combined filter, customer search, empty results, summary, input immutable and 3 more
Gemma 4 12BMac miniWebsite2scored5no300sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Gemma 4 12BMac miniWebsite3started, no scorecardn/anon/a
Gemma 4 12BMac miniWebsite4never started (time window ended)n/anon/a
Gemma 4 12BMac miniWebsite5never started (time window ended)n/anon/a
Gemma 4 12BMac miniFolders1scored10no300manifest exact, copies and hashes, review present
Gemma 4 12BMac miniFolders2scored10no300manifest exact, copies and hashes, review present
Gemma 4 12BMac miniFolders3scored10no300manifest exact, copies and hashes, review present
Gemma 4 12BMac miniFolders4never started (time window ended)n/anon/a
Gemma 4 12BMac miniFolders5never started (time window ended)n/anon/a
Gemma 4 12BMac miniLog analyzer (8,192 tokens)1scored0no330c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac miniLog analyzer (8,192 tokens)2scored0no330c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac miniLog analyzer (8,192 tokens)3never started (time window ended)n/anon/a
Gemma 4 12BMac miniLog analyzer (8,192 tokens)4never started (time window ended)n/anon/a
Gemma 4 12BMac miniLog analyzer (8,192 tokens)5never started (time window ended)n/anon/a
Gemma 4 12BMac StudioDocuments1scored10no170field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac StudioDocuments2scored10no178field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac StudioDocuments3scored10no173field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac StudioDocuments4scored10no172field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac StudioDocuments5scored10no145field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Gemma 4 12BMac StudioWebsite1scored5no300sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Gemma 4 12BMac StudioWebsite2scored0no151sort, combined filter, customer search, empty results, summary, input immutable and 3 more
Gemma 4 12BMac StudioWebsite3scored0no152sort, combined filter, customer search, empty results, summary, input immutable and 3 more
Gemma 4 12BMac StudioWebsite4scored0no300sort, combined filter, customer search, empty results, summary, input immutable and 3 more
Gemma 4 12BMac StudioWebsite5scored0no153sort, combined filter, customer search, empty results, summary, input immutable and 3 more
Gemma 4 12BMac StudioFolders1scored10no12manifest exact, copies and hashes, review present
Gemma 4 12BMac StudioFolders2scored10no163manifest exact, copies and hashes, review present
Gemma 4 12BMac StudioFolders3scored10no205manifest exact, copies and hashes, review present
Gemma 4 12BMac StudioFolders4scored10no51manifest exact, copies and hashes, review present
Gemma 4 12BMac StudioFolders5scored10no12manifest exact, copies and hashes, review present
Gemma 4 12BMac StudioLog analyzer (8,192 tokens)1scored0no144c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac StudioLog analyzer (8,192 tokens)2scored0no147c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac StudioLog analyzer (8,192 tokens)3scored0no144c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac StudioLog analyzer (8,192 tokens)4scored0no144c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Gemma 4 12BMac StudioLog analyzer (8,192 tokens)5scored0no144c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac miniDocuments1scored10no12field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac miniDocuments2scored10no9field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac miniDocuments3scored10no72field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac miniDocuments4scored10no72field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac miniDocuments5scored10no5field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac miniWebsite1scored5no20sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac miniWebsite2scored5no72sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac miniWebsite3scored5no72sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac miniWebsite4scored5no34sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac miniWebsite5scored5no31sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac miniFolders1scored10no38manifest exact, copies and hashes, review present
LFM 2.5 8BMac miniFolders2scored10no74manifest exact, copies and hashes, review present
LFM 2.5 8BMac miniFolders3scored10no35manifest exact, copies and hashes, review present
LFM 2.5 8BMac miniFolders4scored10no8manifest exact, copies and hashes, review present
LFM 2.5 8BMac miniFolders5scored10no45manifest exact, copies and hashes, review present
LFM 2.5 8BMac miniLog analyzer (8,192 tokens)1scored0no6c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac miniLog analyzer (8,192 tokens)2scored0no7c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac miniLog analyzer (8,192 tokens)3scored0no3c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac miniLog analyzer (8,192 tokens)4scored0no2c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac miniLog analyzer (8,192 tokens)5scored0no4c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac StudioDocuments1scored10no3field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac StudioDocuments2scored10no6field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac StudioDocuments3scored10no4field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac StudioDocuments4scored10no4field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac StudioDocuments5scored10no10field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
LFM 2.5 8BMac StudioWebsite1scored5no28sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac StudioWebsite2scored5no16sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac StudioWebsite3scored5no1sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac StudioWebsite4scored5no3sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac StudioWebsite5scored5no1sort, combined filter, customer search, empty results, summary, input immutable and 2 more
LFM 2.5 8BMac StudioFolders1scored10no8manifest exact, copies and hashes, review present
LFM 2.5 8BMac StudioFolders2scored10no4manifest exact, copies and hashes, review present
LFM 2.5 8BMac StudioFolders3scored10no4manifest exact, copies and hashes, review present
LFM 2.5 8BMac StudioFolders4scored10no16manifest exact, copies and hashes, review present
LFM 2.5 8BMac StudioFolders5scored10no3manifest exact, copies and hashes, review present
LFM 2.5 8BMac StudioLog analyzer (8,192 tokens)1scored0no1c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac StudioLog analyzer (8,192 tokens)2scored0no2c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac StudioLog analyzer (8,192 tokens)3scored0no1c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac StudioLog analyzer (8,192 tokens)4scored0no22c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
LFM 2.5 8BMac StudioLog analyzer (8,192 tokens)5scored0no1c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac miniDocuments1scored10no9field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac miniDocuments2scored10no176field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac miniDocuments3scored10no13field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac miniDocuments4scored10no175field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac miniDocuments5scored10no175field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac miniWebsite1scored5no21sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac miniWebsite2scored5no25sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac miniWebsite3scored5no28sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac miniWebsite4scored5no29sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac miniWebsite5scored5no27sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac miniFolders1scored10no176manifest exact, copies and hashes, review present
Phi-4 miniMac miniFolders2scored10no175manifest exact, copies and hashes, review present
Phi-4 miniMac miniFolders3scored10no24manifest exact, copies and hashes, review present
Phi-4 miniMac miniFolders4scored10no9manifest exact, copies and hashes, review present
Phi-4 miniMac miniFolders5scored10no175manifest exact, copies and hashes, review present
Phi-4 miniMac miniLog analyzer (8,192 tokens)1scored0no186c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac miniLog analyzer (8,192 tokens)2scored0no185c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac miniLog analyzer (8,192 tokens)3scored0no21c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac miniLog analyzer (8,192 tokens)4scored0no185c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac miniLog analyzer (8,192 tokens)5scored0no185c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac StudioDocuments1scored10no60field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac StudioDocuments2scored10no6field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac StudioDocuments3scored10no60field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac StudioDocuments4scored10no60field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac StudioDocuments5scored10no60field invoice id, field currency, field customer, field subtotal cents, field shipping cents, field tax cents and 9 more
Phi-4 miniMac StudioWebsite1scored5no7sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac StudioWebsite2scored5no60sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac StudioWebsite3scored5no60sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac StudioWebsite4scored5no7sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac StudioWebsite5scored5no9sort, combined filter, customer search, empty results, summary, input immutable and 2 more
Phi-4 miniMac StudioFolders1scored10no7manifest exact, copies and hashes, review present
Phi-4 miniMac StudioFolders2scored10no2manifest exact, copies and hashes, review present
Phi-4 miniMac StudioFolders3scored10no60manifest exact, copies and hashes, review present
Phi-4 miniMac StudioFolders4scored10no6manifest exact, copies and hashes, review present
Phi-4 miniMac StudioFolders5scored10no60manifest exact, copies and hashes, review present
Phi-4 miniMac StudioLog analyzer (8,192 tokens)1scored0no62c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac StudioLog analyzer (8,192 tokens)2scored0no6c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac StudioLog analyzer (8,192 tokens)3scored0no5c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac StudioLog analyzer (8,192 tokens)4scored0no62c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes
Phi-4 miniMac StudioLog analyzer (8,192 tokens)5scored0no4c1 runs and summary format, c2 filters correct, c3 json core, c4 malformed and validation, c5 multi file and rankings, c6 exit codes

What the results looked like

The first scheduled attempt of each kind was also reviewed by the AI judge from screenshots and saved files: 11 sets existed, 19 were missing, 4 were judged usable (0 of 5 websites). This is a second, stricter standard than the automatic checks, and it covers only first attempts. A page can look fine and still miss the requirements.

Gemma 4 12B · Mac mini · website not usable

Gemma 4 12B website result, desktop view

AI judge: A rudimentary dashboard shell exists, but the supplied missing-script finding and empty rendered state leave the billing workflow unusable. Visually, the tightly packed default-styled controls and table headings lack useful separation, and stray text makes the page look unfinished.

Specific problems
  • The packet reports index.html references app.mjs but that file is absent.
  • Summary values and invoice rows are missing.
  • The screenshot includes stray text autoestima>.

Qwen 3.5 9B · Mac Studio · website not usable

Qwen 3.5 9B website result, desktop view

AI judge: The filters are readable and the failure is visible, but initialization stops before a usable invoice table. Simultaneous loading and error messages give an ambiguous state; the available interface cannot serve the requested billing workflow.

Specific problems
  • Initialization reports Cannot set properties of null (setting innerHTML).
  • Loading remains visible alongside the error, with no invoice table or usable billing results.

Qwen 3.5 9B · Mac mini · website not usable

Qwen 3.5 9B website result, desktop view

AI judge: The visible artifact offers only a loading message, with no practical dashboard content or interactions. That message is readable but does not communicate the recorded script failure or offer recovery.

Specific problems
  • The supplied browser observation reports SyntaxError: Unexpected token export in app.mjs.
  • The page remains at Loading invoices after network idle without the requested dashboard controls or results.

Qwen 3.5 4B · Mac mini · website not usable

Qwen 3.5 4B website result, desktop view

AI judge: The desktop shell has a clear heading, labelled stacked filters, a prominent CSV action and separated table columns. Those visual strengths do not establish functionality: a duplicate-export error prevents populated results, and typing in search leaves an unexplained blank table.

Specific problems
  • The supplied browser observation reports SyntaxError: Duplicate export of formatBalanceCents.
  • The visible summary stays at $0.00 outstanding, $0.00 overdue and 0 unpaid with no invoice rows.
  • The recorded no-such-customer search produces neither results nor an empty-result explanation.

Qwen 3.5 4B · Mac Studio · website not usable

Qwen 3.5 4B website result, desktop viewQwen 3.5 4B website result, after-filter viewQwen 3.5 4B website result, filtered viewQwen 3.5 4B website result, mobile view

AI judge: This is a partially working dashboard with legible populated rows, correctly displayed sample summary arithmetic after interaction, working recorded search recovery, mobile containment and a keyboard-triggered download. It still fails the initial-load workflow and the labelled dollar minimum: a $50 minimum includes $25 and $5 invoices. Those material correctness defects require fixes before use.

Specific problems
  • Initial loading does not finish visually after network idle; summaries remain zero and the invoice table is empty until an input event.
  • The packet attributes the initial failure to loadInvoices not calling applyFilters after loading JSON.
  • With Open status and Minimum Balance ($) set to 50, INV-105 ($25.00) and INV-106 ($5.00) remain visible, violating the dollar threshold.

Qwen 3.5 4B · Mac mini · documents usable

AI judge: All three requested outputs are supplied. The JSON exactly matches the reference schema and values; the CSV carries the scalar fields consistently. The explanation clearly accounts for the credit, single deposit, changed due date, excluded void payment and unknown contact.

Qwen 3.5 4B · Mac Studio · documents usable

AI judge: The reconciliation values, JSON and explanation are strong and consistent with the reference. The CSV is malformed and violates its requested schema, but correction is localized to regenerating that one record/header from the already correct JSON; the underlying reconciliation remains useful.

Specific problems
  • reconciliation.csv omits the required supplier_contact_email column and includes the prohibited source_files field.
  • The CSV header has 11 fields while its record has 13: invoice.pdf, amendment.docx and activity.html are unquoted separate cells after the due date, so the row does not match its header.

Qwen 3.5 4B · Mac Studio · folders not usable

AI judge: The submission includes a five-entry sorted manifest, a clear uncertainty note, one correct receipt copy and the requested duplicate/uncertain-file omissions. However, the two other copies exchange the original filenames associated with their bytes, and every recorded manifest hash is wrong. This makes the provenance unreliable and requires substantial repair even though the copied invoice and meeting contents land in their respective category/month directories.

Specific problems
  • The actual invoice bytes expected for inbox/notes-final.txt are copied under organized/invoices/2026-10/old-invoice.txt; the actual meeting bytes expected for inbox/old-invoice.txt are copied under organized/meetings/2026-09/notes-final.txt. The original-filename requirement is violated.
  • manifest.json assigns inbox/notes-final.txt to organized/meetings/2026-09/notes-final.txt instead of organized/invoices/2026-10/notes-final.txt, and inbox/old-invoice.txt to organized/invoices/2026-10/old-invoice.txt instead of organized/meetings/2026-09/old-invoice.txt.
  • All five manifest sha256 values differ from the reference and are not full 64-character SHA-256 hex digests.

Qwen 3.5 9B · Mac mini · folders not usable

AI judge: All source hashes and the copied file bytes are retained correctly in the supplied inventory, and the duplicate and uncertain file are omitted appropriately. But two of the three unique classified files are routed to incorrect categories, one also to the wrong month, and four manifest entries omit a required key. The primary organization task needs substantial correction.

Specific problems
  • notes-final.txt is copied to organized/meetings/2026-09/notes-final.txt; the reference requires organized/invoices/2026-10/notes-final.txt.
  • old-invoice.txt is copied to organized/invoices/2026-09/old-invoice.txt; the reference requires organized/meetings/2026-09/old-invoice.txt.
  • manifest.json omits duplicate_of from the mystery.txt, notes-final.txt, old-invoice.txt and photo.txt entries; the exact schema requires null for those four entries.

Qwen 3.5 4B · Mac mini · folders usable

AI judge: The actual copied paths and hashes fully match the reference, including deduplication and leaving the uncertain file uncopied. The manifest has the correct five-key entries and statuses but needs two hash replacements and sorting by source. Those localized metadata corrections leave the core organization work useful.

Specific problems
  • manifest.json gives scan-copy.txt the invalid/incomplete sha256 value 'a0d63f019652b7f33450fb78edadbc7b' and mystery.txt '37ec065c7a140f8358694df753941f66', both different from the full reference hashes.
  • The manifest order is notes-final.txt, old-invoice.txt, photo.txt, scan-copy.txt, mystery.txt. It is not sorted by source; inbox/mystery.txt should be first.

Qwen 3.5 9B · Mac Studio · folders usable

AI judge: The manifest matches the expected reference in schema, values and source order. All three actual copies match the required destinations and hashes, the lexicographically first duplicate is retained, and the uncertain file is left uncopied with a clear human-review note. No defect is visible in the supplied artifact evidence.

Transcription

Two Whisper models through MLX, 264 transcriptions in total (22 clips, three repeats, two models, two machines). Speed is seconds of work per minute of audio across all clips, model load excluded.

ModelMini s per audio minuteStudio s per audio minuteStudio vs Mini
Whisper small1.050.561.89x
Whisper large v3 turbo1.600.592.72x

On five natural clips from one speaker, Whisper small got 9.3% of words wrong and large v3 turbo 6.0% (lower is better). The transcripts were identical on both machines.

ModelClip group (Mini)AttemptsAverage clipWord error rateInvented speechs per audio minute
Whisper smallNatural read speech (1 speaker)1512 s9.3% 42 of 453 wordsn/a1.16
Whisper smallNatural speech + added noise1512 s12.6% 57 of 453 wordsn/a1.14
Whisper smallSynthetic practical speech1518 s8.5% 63 of 744 wordsn/a1.01
Whisper smallSynthetic speech + added noise1518 s10.5% 78 of 744 wordsn/a1.05
Whisper smallLonger synthetic session393 s8.1% 60 of 744 wordsn/a1.03
Whisper smallSilence (control)310 sn/a3 of 30.43
Whisper large v3 turboNatural read speech (1 speaker)1512 s6.0% 27 of 453 wordsn/a1.77
Whisper large v3 turboNatural speech + added noise1512 s7.3% 33 of 453 wordsn/a2.05
Whisper large v3 turboSynthetic practical speech1518 s8.1% 60 of 744 wordsn/a1.51
Whisper large v3 turboSynthetic speech + added noise1518 s9.3% 69 of 744 wordsn/a1.51
Whisper large v3 turboLonger synthetic session393 s8.1% 60 of 744 wordsn/a1.33
Whisper large v3 turboSilence (control)310 sn/a3 of 31.68

Word error rate is lexical: case and punctuation are normalized, but "5" and "five" count as different, which inflates the error on the synthetic clips with many numbers. The natural clips are public-domain audiobook readings.

It heard words in silence

Given a clip of pure silence, both models wrote words anyway: 3 of 3 times for small and 3 of 3 for large v3 turbo on the Mini, and the same on the Studio. This is a known habit of this kind of model, and a reason to skim a transcript before trusting it.

ModelMachineAttemptSilence lengthWhat it wrote
Whisper large v3 turboMac mini110 sThank you.
Whisper large v3 turboMac mini210 sThank you.
Whisper large v3 turboMac mini310 sThank you.
Whisper large v3 turboMac Studio110 sThank you.
Whisper large v3 turboMac Studio210 sThank you.
Whisper large v3 turboMac Studio310 sThank you.
Whisper smallMac mini110 syou
Whisper smallMac mini210 syou
Whisper smallMac mini310 syou
Whisper smallMac Studio110 syou
Whisper smallMac Studio210 syou
Whisper smallMac Studio310 syou
One clip, reference and both transcripts

Reference (synthetic speech)

This is a billing note for Cedar Desk Workshop. The invoice total is two hundred twenty nine dollars and forty eight cents. Apply one six dollar credit and the twenty dollar deposit already received. Do not count the deposit twice. The revised due date is November eleventh. The supplier contact email is still unknown.

Whisper small · 15 word errors of 54

This is a billing note for Cedar Desk Workshop. The invoice total is $229.48. Apply $1.6 credit and the $20 deposit already received. Do not count the deposit twice. The revised due date is November 11. The supplier contact email is still unknown.

Whisper large v3 turbo · 14 word errors of 54

This is a billing note for Cedar Desk Workshop. The invoice total is $229.48. Apply one $6 credit and the $20 deposit already received. Do not count the deposit twice. The revised due date is November 11th. The supplier contact email is still unknown.

Image generation

SDXL Turbo through MLX at 512 by 512, five briefs, three seeds, at one and four steps. Both machines produced byte-identical images, so the 30 distinct outputs were judged once. Timing excludes loading the model. Every image below is AI-generated; none shows a real place, product or person.

SettingMini seconds per imageStudio seconds per imageStudio vs Mini
1 step0.880.541.62x
4 steps1.490.742.01x
BriefUsable at 1 stepUsable at 4 steps
Product photo3 / 33 / 3
Cafe hero image0 / 31 / 3
Video thumbnail0 / 30 / 3
Newsletter illustration0 / 30 / 3
Sign with text0 / 30 / 3

All 30 images, with the judgment for each

Product photo

Brief: Generic product image for a small online shop
Prompt: Studio product photograph of one plain white ceramic coffee mug on a pale beige tabletop, soft window light from the left, subtle shadow, uncluttered background, no letters, no logo.

AI-generated image, Product photo, 1 step, seed 42
1 step · seed 42 usable

The plain white mug is well framed on a beige tabletop with light from the left and a rightward shadow. A small unexplained projection near the right rim is a localized defect that could be retouched; otherwise the product image is useful.

Specific problems
  • A small triangular projection protrudes from the right side near the rim, opposite the main handle, making the mug silhouette slightly implausible.
AI-generated image, Product photo, 1 step, seed 314
1 step · seed 314 usable

A clean, believable single-mug product image with the requested neutral surface, left-side light and restrained rightward shadow. The entire mug and handle are visible with useful margin and no lettering or logo.

AI-generated image, Product photo, 1 step, seed 2026
1 step · seed 2026 usable

The single plain white mug is cleanly framed on a neutral beige surface with no text or logo. Its geometry and finish are strong; the dominant illumination and shadow direction differ from the specified left light.

Specific problems
  • The cast shadow runs left and the strongest bright rim/handle areas are on the right, indicating dominant light from the right rather than left.
AI-generated image, Product photo, 4 step, seed 42
4 steps · seed 42 usable

The image provides a well-framed single white mug on a neutral beige surface with left-side illumination and no branding. Minor softness and irregularities around the rim and lower handle join reduce finish slightly.

Specific problems
  • The rim contour is a little uneven and the lower handle connection appears pinched/angular.
AI-generated image, Product photo, 4 step, seed 314
4 steps · seed 314 usable

The composition closely matches the product brief: one plain mug, beige background and left-side illumination. Slightly soft, uneven rim and handle transitions prevent a fully polished product finish but are minor at this use scale.

Specific problems
  • The rim edge and handle-to-body junctions look slightly soft and irregular, especially the lower angled join.
AI-generated image, Product photo, 4 step, seed 2026
4 steps · seed 2026 usable

The single white mug, neutral tabletop and clean product framing are strong fits. The visible lighting direction is opposite the request, but a minor presentation adjustment could make this useful for the generic shop image.

Specific problems
  • The dominant cast shadow extends left, and the brightest rim/handle highlights are on the right, rather than indicating light from the left.

Cafe hero image

Brief: Local cafe website hero artwork
Prompt: Wide editorial photograph of a cozy neighborhood cafe interior with wooden tables, green plants and warm morning sunlight, the left side mostly an empty cream wall for a headline overlay, no people, no text.

AI-generated image, Cafe hero image, 1 step, seed 42
1 step · seed 42 not usable

The image is a polished, warm cafe photograph, but its square framing and busy left foreground do not meet the intended wide hero and empty-left-wall composition.

Specific problems
  • The image is square rather than wide.
  • The left side is filled with foliage, a hanging light, a window and a foreground table/chair.
  • Strong furniture silhouettes and window highlights compete with a headline in the requested position.
AI-generated image, Cafe hero image, 1 step, seed 314
1 step · seed 314 not usable

The warm, plant-filled cafe is convincing as a photograph, but its square framing and window-filled left side do not provide the specified wide hero with an empty cream headline wall.

Specific problems
  • The image is square rather than wide.
  • The left side contains windows, hanging plants, a potted plant and furniture instead of an empty cream wall.
  • Dense foreground and background shapes leave no clean left headline field.
AI-generated image, Cafe hero image, 1 step, seed 2026
1 step · seed 2026 not usable

The cafe is warm, inviting and photographically finished, with wood and abundant greenery. It lacks both the wide format and a mostly empty cream wall on the left, so it is poorly suited to the specified headline-overlay hero without substantial recomposition.

Specific problems
  • The image is square rather than wide.
  • The left half is occupied by plants, hanging fixtures, furniture and bright windows rather than blank cream wall.
  • Foreground tables/chairs and strong light-dark edges make a large headline difficult to place in the requested location.
AI-generated image, Cafe hero image, 4 step, seed 42
4 steps · seed 42 not usable

The cafe atmosphere, wood, greenery and warm sunlight are convincing, but the layout does not provide the specified wide hero composition or empty cream wall on the left. It would require substantial recomposition for the requested headline treatment.

Specific problems
  • The image is square rather than wide.
  • The left side contains large plants, a hanging bulb, a wall planter and a framed object; the wall is green, not an empty cream field.
  • Dense foreground furniture and foliage leave little clean headline space.
AI-generated image, Cafe hero image, 4 step, seed 314
4 steps · seed 314 usable

This cafe image offers a substantial uncluttered cream wall in the upper middle, alongside wood, plants and warm light. It is not already wide and the clearest overlay area is more central than left, but a modest crop and headline-placement adjustment could make it a useful hero.

Specific problems
  • The source is square, not wide.
  • The far-left side is a window and plant; the clean cream headline area is upper-center rather than mostly left.
AI-generated image, Cafe hero image, 4 step, seed 2026
4 steps · seed 2026 not usable

The warm sunlight, wooden furniture and abundant greenery fit a neighborhood cafe, but the square and crowded composition lacks the requested mostly empty cream wall on the left.

Specific problems
  • The image is square rather than a wide hero.
  • Large hanging foliage, windows and furniture occupy the left side.
  • The dense chair/table arrangement and bright window flare leave limited clean headline space.

Video thumbnail

Brief: Original background for a technology review thumbnail
Prompt: A compact silver desktop computer on a dark wooden desk, dramatic blue lighting on the right and orange lighting on the left, large empty dark space above the computer for title overlay, realistic photograph, no text, no logo.

AI-generated image, Video thumbnail, 1 step, seed 42
1 step · seed 42 not usable

The desk scene offers some dark space overhead, but the main subject is a black monitor rather than a compact silver computer. A bright half-white screen competes with title text, and the orange lighting is on the wrong side.

Specific problems
  • No compact silver desktop unit is visible; the scene centers on a large black monitor and peripherals.
  • The screen has a stark black/white vertical split, drawing attention away from a potential title.
  • Orange light appears at right, not left; blue dominates the backdrop.
  • The overhead area is only moderately large and visibly textured/light-banded.
AI-generated image, Video thumbnail, 1 step, seed 314
1 step · seed 314 not usable

This is an attractive, coherent desktop-monitor setup, but it misses the requested compact silver computer and the large empty dark area above it. The blue/orange placement also differs from the brief.

Specific problems
  • The dominant object is a large black monitor with keyboard and mouse; no compact silver computer unit is visible.
  • The monitor rises almost to the top, leaving only a narrow band above it rather than large title-overlay space.
  • The clearest orange light is at the right edge, while blue light dominates behind/left of the monitor, reversing the requested directional emphasis.
  • A small light rectangular mark is visible on the lower monitor bezel, so the completely unmarked/no-logo appearance is not assured.
AI-generated image, Video thumbnail, 1 step, seed 2026
1 step · seed 2026 not usable

A coherent, polished computer scene is present, but the dark tower plus large monitor and extra desk items miss the compact silver-computer focus. Blue-left/orange-right lighting is opposite the brief, and only a shallow overhead strip is available for a title.

Specific problems
  • The computer tower is black, and the composition includes a prominent monitor, keyboard and other items rather than one compact silver unit.
  • The strongest blue illumination is on the left screen and orange on the right wall/tower.
  • The space above the monitor is relatively shallow and textured, not a large empty dark title field.
AI-generated image, Video thumbnail, 4 step, seed 42
4 steps · seed 42 not usable

The scene includes a desktop tower and wooden desk with some overhead wall space, but the devices are dark and bulky rather than compact silver. The split-bright screen and reversed color emphasis also weaken the requested thumbnail composition.

Specific problems
  • A large monitor and tall black tower replace the compact silver-computer focus.
  • The screen is divided into black and bright white areas, creating a strong competing focal point.
  • Orange appears as a glow on the right tower; the left side has no comparable orange illumination.
  • Keyboard details and the closely adjoining monitor/tower edges are somewhat soft and visually ambiguous.
AI-generated image, Video thumbnail, 4 step, seed 314
4 steps · seed 314 not usable

The dark desktop scene is recognizable, but a large black deep-sided monitor/computer housing replaces the compact silver unit. It leaves little space above for title text and lacks the requested orange-left/blue-right lighting arrangement.

Specific problems
  • The dominant device is large and black with a deep side housing, not a compact silver desktop unit.
  • The device extends nearly to the top edge, leaving little clear overhead title area.
  • Blue light dominates the backdrop; the only obvious orange accent is at the far right, not the left.
  • A tiny label-like mark appears on the lower bezel despite the no-text/no-logo request.
AI-generated image, Video thumbnail, 4 step, seed 2026
4 steps · seed 2026 not usable

This polished technology setup contains a tower and monitor, but not the requested compact silver focus or lighting orientation. The screen, tower and textured wall consume much of the title area, so the exact thumbnail brief needs substantial recomposition.

Specific problems
  • The main tower is predominantly black with silver trim, and the large monitor/keyboard broaden the scene beyond the requested compact silver unit.
  • Blue illumination is concentrated at left and orange at right, opposite the prompt.
  • Only a modest upper-left dark area is empty; the upper-right is textured stone and orange light, limiting a large title above the computer.

Newsletter illustration

Brief: Illustration for a practical home gardening newsletter
Prompt: Clean watercolor illustration of three small herb plants in terracotta pots on a white kitchen windowsill, a small green watering can beside them, gentle natural colors, white background, no text.

AI-generated image, Newsletter illustration, 1 step, seed 42
1 step · seed 42 not usable

The soft watercolor and white windowsill are attractive, but four planted containers are shown, several green, with no separate green watering can. The specified three terracotta pots plus can arrangement would require substantial correction.

Specific problems
  • Four planted containers are visible rather than three.
  • Several pots are green rather than terracotta, and the center terracotta vessel has a mug-like handle.
  • No separate small green watering can is visible.
  • The rightmost plant/pot is cropped by the frame.
AI-generated image, Newsletter illustration, 1 step, seed 314
1 step · seed 314 not usable

The muted watercolor treatment and white window setting are attractive, but the right herb grows from the watering-can-like vessel and only two conventional pots are present. This is not the requested three terracotta pots plus a separate watering can.

Specific problems
  • There are only two conventional plant pots; the third plant is in a tall green can-like container.
  • The center pot is predominantly green, so three terracotta pots are not depicted.
  • The green watering-can-like object is planted rather than sitting separately beside the herbs.
AI-generated image, Newsletter illustration, 1 step, seed 2026
1 step · seed 2026 not usable

The gentle watercolor is appealing, but the arrangement adds an extra planted vessel and merges large loop handles with the center pot instead of showing a separate small green watering can. The exact gardening brief needs substantial object correction.

Specific problems
  • In addition to three foreground pots, a gray planted vessel is visible behind the left pot, adding a fourth plant/container.
  • Two large loops rise behind the center pot, but no separate recognizable green watering can is present.
  • The right plant/pot is partly cropped.
AI-generated image, Newsletter illustration, 4 step, seed 42
4 steps · seed 42 not usable

The watercolor style and bright windowsill suit a gardening theme, but the container arrangement does not match three herbs in terracotta pots beside a separate green watering can. The crowded crop would need substantial content correction for this exact illustration brief.

Specific problems
  • At least four planted containers are visible, plus a rear terracotta pot, rather than three small terracotta-potted herbs.
  • Two prominent planted containers are green and handled; no separate unplanted, clearly functional watering can is shown.
  • Large leaves and the rightmost pot are cut by the frame, making the arrangement crowded.
AI-generated image, Newsletter illustration, 4 step, seed 314
4 steps · seed 314 not usable

The clean watercolor and white windowsill are suitable stylistically, but only one herb is in a terracotta pot. The other herbs occupy green can/pitcher-like vessels, and a separate small watering can is absent.

Specific problems
  • Only the left vessel is terracotta; the center and right containers are green.
  • Plants grow inside the can-like containers instead of a separate watering can standing beside three pots.
  • The center spout/handle region overlaps the neighboring vessel, making their functions less clear.
AI-generated image, Newsletter illustration, 4 step, seed 2026
4 steps · seed 2026 not usable

The illustration has a clean watercolor finish, but it shows an extra plant in a blue vessel and a watering-can spout attached to the right planted container. It does not deliver three terracotta herbs beside a separate green can.

Specific problems
  • Four plant groupings/containers are visible, including the large blue vessel at back-left.
  • A pale diagonal spout appears joined to the right planted pot, merging the watering-can and planter forms.
  • No separate small green watering can is visible.
  • The large plants and right pot are cut by the frame.

Sign with text

Brief: Deliberately demanding exact-text sign for a shop
Prompt: A simple front-facing rectangular cream shop sign with exactly the words OPEN TODAY in large dark green capital letters, centered, crisp readable typography, thin green border, no other words or decorations.

AI-generated image, Sign with text, 1 step, seed 42
1 step · seed 42 not usable

The sign has multiple malformed text bands and never visibly spells OPEN TODAY. The angled, heavily framed storefront treatment also misses the requested simple front-facing thin-border design.

Lettering as the judge read it: Top: O[extra narrow vertical stroke]PEEN. Middle: an illegible line of tiny pseudo-text. Bottom: O[overprinted/boxed glyph]ON.

Specific problems
  • The large word appears as O, an extra narrow vertical stroke, then PEEN; it is not OPEN TODAY.
  • A tiny illegible pseudo-text line and a second malformed lower word add unrequested text.
  • The lettering includes an overprinted/boxed lower glyph, and the surrounding heavy frame and facade are extraneous.
AI-generated image, Sign with text, 1 step, seed 314
1 step · seed 314 not usable

The image is a cleanly rendered mounted sign, but its large readable word is OPEEN and TODAY is absent. The exact-text requirement therefore fails despite the otherwise crisp presentation.

Lettering as the judge read it: OPEEN

Specific problems
  • Visible text is OPEEN, not OPEN TODAY.
  • The sign has a thick layered green outer frame and surrounding masonry rather than only a thin green border and undecorated face.
AI-generated image, Sign with text, 1 step, seed 2026
1 step · seed 2026 not usable

The mounted sign has a clear large OPEEN word but no readable TODAY, and additional illegible lettering appears above and below. The photograph-like facade does not rescue the essential text failure.

Lettering as the judge read it: Main: OPEEN. Above: small illegible glyphs. Below: two lines of illegible pseudo-text.

Specific problems
  • Large text reads OPEEN instead of OPEN TODAY.
  • Small distorted text appears in multiple extra lines.
  • A thick layered frame and masonry surround replace the requested simple thin border.
AI-generated image, Sign with text, 4 step, seed 42
4 steps · seed 42 not usable

The sign contains malformed main lettering and an extra pseudo-text line, with no readable OPEN TODAY phrase. The thick facade framing and ornamental surroundings also conflict with the simple exact-text design.

Lettering as the judge read it: Top: OPENN (the final N-shaped strokes are crowded/fused). Lower line: OPEING A!Y, followed by a small unclear glyph; there is also a small stray mark before the line.

Specific problems
  • Main text visually reads OPENN, with crowded/fused N-shaped final strokes; it does not read OPEN TODAY.
  • A second line of malformed lettering and small stray glyphs is present, contrary to the no-other-words/decorations instruction.
  • The heavy green facade/frame, ornate top trim and worn wall add substantial unrequested elements.
AI-generated image, Sign with text, 4 step, seed 314
4 steps · seed 314 not usable

The sign is crisp, front-facing and visually clean, but the only large word is OPEEN and TODAY is missing. Exact wording is essential, so it cannot serve the specified sign without replacing the text.

Lettering as the judge read it: OPEEN

Specific problems
  • Visible word is OPEEN, not OPEN TODAY.
  • The green outer frame is substantially thicker than the requested thin border, although thin inset lines are present.
AI-generated image, Sign with text, 4 step, seed 2026
4 steps · seed 2026 not usable

The hanging sign is recognizable, but its main word is OPEEN and the lower line is malformed rather than TODAY. Extra small text and heavy surrounding framing also violate the simple exact-text brief.

Lettering as the judge read it: Main: OPEEN. Lower line: appears to read HOCH DATY (irregular letterforms). Upper small line: illegible pseudo-text.

Specific problems
  • Main word has an extra E and TODAY is absent.
  • There are two additional text bands, including unreadable small characters.
  • The thick frame, hanging chains and brick/wall context depart from the requested undecorated thin-border sign.

Image edits

The same model in plain image-to-image mode (strength 0.5, 2 actual denoising steps) was asked to change five of its own images. About 1.62 seconds per edit on the Mini and 0.97 on the Studio. 0 of 15 were judged usable: the edits mostly kept the source and did not make the requested change. This is a test of one simple technique, not of purpose-built image editors.

Product photo

Edit prompt: Studio product photograph of one plain cobalt blue ceramic coffee mug on a pale beige tabletop, soft window light from the left, subtle shadow, uncluttered background, no letters, no logo.
Original prompt: Studio product photograph of one plain white ceramic coffee mug on a pale beige tabletop, soft window light from the left, subtle shadow, uncluttered background, no letters, no logo.

AI-generated source image, Product photoAI-edited result, Product photo, seed 42
Seed 42 source left, edit right not usable

A coherent plain mug photograph preserves the source well, but no cobalt-blue recoloring is visible.

Specific problems
  • The requested cobalt blue color is absent.
AI-generated source image, Product photoAI-edited result, Product photo, seed 314
Seed 314 source left, edit right not usable

A coherent mug image preserves the source setting well, but the requested recoloring is absent; visual polish does not make it an effective revision.

Specific problems
  • The required cobalt blue color is missing; the mug remains off-white.
AI-generated source image, Product photoAI-edited result, Product photo, seed 2026
Seed 2026 source left, edit right not usable

The scene remains a plausible neutral mug product photograph, but its central color revision is completely missing.

Specific problems
  • The mug remains cream rather than cobalt blue.
  • Handle material appears unusually metallic/reflective relative to the ceramic body.

Cafe hero image

Edit prompt: Photograph of the same cozy neighborhood cafe interior with wooden tables and green plants, now warmly lit in the evening by amber lamps, preserve the large blank cream wall on the left for a headline overlay, no people, no text.
Original prompt: Wide editorial photograph of a cozy neighborhood cafe interior with wooden tables, green plants and warm morning sunlight, the left side mostly an empty cream wall for a headline overlay, no people, no text.

AI-generated source image, Cafe hero imageAI-edited result, Cafe hero image, seed 42
Seed 42 source left, edit right not usable

The edit adds amber fixtures while preserving the broad cafe arrangement, but strong daylight and the occupied green/plant-covered wall leave both key revision goals unmet.

Specific problems
  • The scene still reads as daytime despite the lit lamps.
  • The required blank cream headline wall is not present.
AI-generated source image, Cafe hero imageAI-edited result, Cafe hero image, seed 314
Seed 314 source left, edit right not usable

The cafe remains recognizable and lamps are emphasized, but it still reads as a daylight interior and does not provide the requested blank cream headline wall.

Specific problems
  • The requested evening atmosphere is not convincingly achieved.
  • No large blank cream wall on the left is provided; plants and wall objects occupy that area.
AI-generated source image, Cafe hero imageAI-edited result, Cafe hero image, seed 2026
Seed 2026 source left, edit right not usable

Warm lamps and a cozy cafe setting are visible, but the scene remains daylight-oriented and the headline area is absent. Source layout is recognizable with noticeable furnishing changes.

Specific problems
  • Evening atmosphere is not convincingly established.
  • No large blank cream wall for the headline overlay.
  • Additional left-side furnishings and decorations further occupy the requested overlay area.

Video thumbnail

Edit prompt: A compact silver desktop computer on the same dark wooden desk, dramatic purple lighting on the right and green lighting on the left, preserve large empty dark space above for a title overlay, realistic photograph, no text, no logo.
Original prompt: A compact silver desktop computer on a dark wooden desk, dramatic blue lighting on the right and orange lighting on the left, large empty dark space above the computer for title overlay, realistic photograph, no text, no logo.

AI-generated source image, Video thumbnailAI-edited result, Video thumbnail, seed 42
Seed 42 source left, edit right not usable

The original workstation and desk are recognizably retained, but the requested compact silver computer and two-color lighting are not achieved.

Specific problems
  • No compact silver desktop computer.
  • Required purple lighting on the right and green lighting on the left are absent.
  • A small white marking remains on the monitor bezel; its text is unreadable, so the no-logo/no-text requirement is not confidently clean.
AI-generated source image, Video thumbnailAI-edited result, Video thumbnail, seed 314
Seed 314 source left, edit right not usable

The source workstation and desk are preserved, but the computer and lighting remain substantially outside the requested revision.

Specific problems
  • No compact silver desktop computer.
  • Required purple-right/green-left illumination absent.
  • Small white/bright bezel markings remain, preventing confident confirmation of a completely unbranded/text-free image.
AI-generated source image, Video thumbnailAI-edited result, Video thumbnail, seed 2026
Seed 2026 source left, edit right not usable

The workstation is recognizable but the requested computer replacement and split lighting do not occur.

Specific problems
  • Compact silver computer absent.
  • Requested directional purple and green lighting absent; a yellow-green tower light is not the requested scene lighting.
  • Small bright bezel markings remain, although unreadable.

Newsletter illustration

Edit prompt: Clean black ink line drawing of three small herb plants in terracotta pots on a white kitchen windowsill, a small watering can beside them, monochrome white background, preserve the layout, no text.
Original prompt: Clean watercolor illustration of three small herb plants in terracotta pots on a white kitchen windowsill, a small green watering can beside them, gentle natural colors, white background, no text.

AI-generated source image, Newsletter illustrationAI-edited result, Newsletter illustration, seed 42
Seed 42 source left, edit right not usable

Stronger outlines partly approach an ink style, but the image remains a colored illustration with the wrong container/count details and no recognizable watering can.

Specific problems
  • Not monochrome black ink on white; extensive color fills remain.
  • Not three herbs in terracotta pots; handled mug-like planters and additional containers remain.
  • Requested small watering can absent.
AI-generated source image, Newsletter illustrationAI-edited result, Newsletter illustration, seed 314
Seed 314 source left, edit right not usable

The scene becomes a cleaner colored illustration, but it is not the requested monochrome ink drawing and the required three terracotta pots plus separate watering can are not established.

Specific problems
  • Extensive green/orange fills and shading remain instead of monochrome black line work on white.
  • More than three containers are visible and not all are terracotta.
  • No clearly identifiable separate small watering can beside the plants; the brown partial spout-like form is ambiguous.
AI-generated source image, Newsletter illustrationAI-edited result, Newsletter illustration, seed 2026
Seed 2026 source left, edit right not usable

More defined outlines preserve the plant arrangement, but the colored fills, excess containers and absent watering can prevent fulfillment of the requested revision.

Specific problems
  • Not a monochrome black-ink line drawing on white.
  • More than three containers remain, including two mug-like handled planters rather than three terracotta pots.
  • Small watering can absent.

Sign with text

Edit prompt: The same front-facing rectangular cream shop sign, now with exactly the words CLOSED TODAY in large dark green capital letters, centered, crisp readable typography, preserve the thin green border, no other words or decorations.
Original prompt: A simple front-facing rectangular cream shop sign with exactly the words OPEN TODAY in large dark green capital letters, centered, crisp readable typography, thin green border, no other words or decorations.

AI-generated source image, Sign with textAI-edited result, Sign with text, seed 42
Seed 42 source left, edit right not usable

The sign retains its broad layout while replacing some gibberish with different gibberish. The required closure wording is absent.

Specific problems
  • CLOSED TODAY is not present.
  • Extra lower-line text remains and is malformed.
  • The broad green surround does not meet the requested thin-border, text-only result.
AI-generated source image, Sign with textAI-edited result, Sign with text, seed 314
Seed 314 source left, edit right not usable

The sign structure remains recognizable, but the requested closure message is absent and unrelated malformed lettering remains. It cannot serve as a CLOSED TODAY notice.

Specific problems
  • Neither CLOSED nor TODAY appears.
  • Extra lower-line words and decorative strokes violate the exact-text/no-decoration requirement.
  • The border remains a broad green panel rather than a thin green border.
AI-generated source image, Sign with textAI-edited result, Sign with text, seed 2026
Seed 2026 source left, edit right not usable

The sign's general structure survives but the lettering becomes different gibberish. It provides neither the exact words nor a usable closure notice.

Specific problems
  • Requested CLOSED TODAY wording missing.
  • Unrelated, malformed text remains in two lines.
  • Broad green surround and extra text fail the thin-border/no-other-words brief.

Video edits with FFmpeg

Each model, through the Pi agent, was asked to cut and assemble short synthetic clips with FFmpeg: five tasks per model per machine, 50 attempts. This is an AI running an editing tool, not generating video.

8 / 50attempts produced a video that plays (35 produced no file, 7 would not decode).
2judged usable from six sampled frames each.
1passed every automatic check: Qwen 3.5 9B on the Mac Studio, 33.8 seconds.
ModelMachinePassed all checksScoredVideo filesTimeouts
Qwen 3.5 4BMac mini0 / 5500
Qwen 3.5 4BMac Studio0 / 5520
Qwen 3.5 9BMac mini0 / 5532
Qwen 3.5 9BMac Studio1 / 5520
Gemma 4 12BMac mini0 / 5553
Gemma 4 12BMac Studio0 / 5531
LFM 2.5 8BMac mini0 / 5500
LFM 2.5 8BMac Studio0 / 5500
Phi-4 miniMac mini0 / 5500
Phi-4 miniMac Studio0 / 5500
The instruction for video task 1
Create a short local workshop preview from the supplied synthetic media. Use ffmpeg (available on PATH) or Python; do the actual conversion, not just instructions. No network is needed.
Deliver output.mp4: first setup.mp4 from 2 to 4 seconds, then demo.mp4 from 3 to 5 seconds, then closing.png held for2 seconds. Total6 seconds,640x360,24fps,H.264, yuv420p, no audio track. Preserve source files unchanged. Do not add transitions or extra frames. Also write edit.json as exactly this JSON array describing the requested edit (numeric types may be integer or float):
[{"source": "setup.mp4", "start_seconds": 2, "duration_seconds": 2}, {"source": "demo.mp4", "start_seconds": 3, "duration_seconds": 2}, {"source": "closing.png", "start_seconds": 0, "duration_seconds": 2}]
You may inspect inputs and verify your output. This is an ordinary deterministic video editing workflow, not a request to understand video content. ffmpeg -version confirms the available offline tool. Finish with a brief statement of what you created.

The 8 videos that play: expected frames on the left, submitted on the right

Expected and submitted frames, Qwen 3.5 9B on the Mac Studio
Qwen 3.5 9B · Mac Studio · video-3 usable

All six sampled output frames match their expected source content and order, and the reported format, six-second duration and exact manifest satisfy the brief within the stated review scope.

Expected and submitted frames, Gemma 4 12B on the Mac mini
Gemma 4 12B · Mac mini · video-4 not usable

The output is decodable and readable, but the sampled sequence starts from setup second 0 and stays on setup instead of reaching the requested demo and closing card. Its ten-second duration contradicts both the brief and its otherwise correct manifest.

Specific problems
  • Incorrect setup trim: actual begins at setup second 0 rather than the requested second 2 at the first sample.
  • Demo and closing content are absent at their required sampled positions; setup persists instead.
  • Reported duration is 10.00 seconds, four seconds longer than requested.
Expected and submitted frames, Qwen 3.5 9B on the Mac mini
Qwen 3.5 9B · Mac mini · video-4 not usable

The duration and format are correct, and the manifest is correct, but the sampled visual content is an untrimmed setup sequence rather than the requested setup/demo/closing edit. Readability cannot compensate for the missing sequence.

Specific problems
  • Wrong setup start at the initial samples.
  • All six samples show setup footage; required demo and closing frames are absent at their expected times.
  • A correct manifest and total duration conceal a substantially incorrect visual sequence.
Expected and submitted frames, Qwen 3.5 9B on the Mac mini
Qwen 3.5 9B · Mac mini · video-5 not usable

This is a partially decodable file but not a usable workshop preview. The only decoded sample displays dense technical text instead of source artwork; the file ends far before the required sequence and the edit manifest is missing.

Specific problems
  • Only 1.13 seconds delivered instead of 6 seconds.
  • The one decoded sample contains unrelated dense technical text rather than readable source artwork.
  • Five of six required sample times have no decoded output frame.
  • Required edit manifest missing.
Expected and submitted frames, Gemma 4 12B on the Mac mini
Gemma 4 12B · Mac mini · video-3 not usable

The file is readable and has the requested encoding, but the samples show setup seconds 0-5 rather than the requested trimmed sequence, and the output is too long. The correct edit manifest is not implemented by the sampled visuals.

Specific problems
  • Wrong setup trim at initial samples.
  • Setup footage occupies the required demo and closing sample positions.
  • Ten-second duration exceeds the requested six seconds.
Expected and submitted frames, Qwen 3.5 4B on the Mac Studio
Qwen 3.5 4B · Mac Studio · video-2 not usable

The sampled decoded stream contains readable setup footage with the wrong trim and no demo/closing at the expected times. Six reported video streams and no manifest further prevent this from being a clearly delivered sequential preview.

Specific problems
  • Wrong setup trim and incorrect content at all sampled edit positions.
  • Required demo and closing content not present at their expected times in the sampled stream.
  • Multiple video streams with two defaults do not provide clear evidence of a single intended concatenated preview.
  • Required edit manifest missing.
Expected and submitted frames, Qwen 3.5 9B on the Mac mini
Qwen 3.5 9B · Mac mini · video-2 usable

All sampled frames and the manifest match the requested edit, with clear preservation of source content. The small reported duration and frame-rate deviations need correction for exact delivery specifications but do not undermine the sampled preview's usefulness.

Specific problems
  • Reported duration is 6.02 seconds, not exactly 6.00.
  • Reported frame rate is 23.92 fps, not the requested 24 fps; exact cadence compliance is not established.
Expected and submitted frames, Qwen 3.5 9B on the Mac Studio
Qwen 3.5 9B · Mac Studio · video-4 not usable

The video is decodable and some source cards are preserved, but the sampled sequence begins with black frames, presents closing content in the demo slot and setup content in the closing slot. Correct-looking metadata and manifest do not fix the wrong visual edit.

Specific problems
  • First two sampled frames are black instead of setup footage.
  • Closing card appears in the demo sample positions; setup appears in the closing positions.
  • Required demo content is absent at the sampled times.
  • Reported duration is 6.08 seconds rather than exactly 6.00.

Model by model

Everything above, regrouped per model. Open one.

Qwen 3.5 4B 49.5 tok/s on the Mini · 3 of 10 usable answers · bug fix 5 of 5
  • Files: mlx-community/Qwen3.5-4B-MLX-4bit at revision 32f3e8ecf6, 3.0 GB of weights.
  • Speed: 49.5 output tokens per second on the Mini, 145.2 on the Studio (2.93x, 5 matched pairs). Reading the 480-record document: 1,690 and 5,286 prompt tokens per second (3.13x, n = 5).
  • Memory: peak sampled footprint 5.4 GB on the Mini and 5.7 GB on the Studio, against the 11.4 GB budget. Guard stops: none.
  • Graded answers: Mac mini 0 of 5 conversations and 3 of 5 writing tasks usable; Mac Studio 0 of 5 conversations and 3 of 5 writing tasks usable. Read the saved answers.
  • Of the graded items that were not usable (Mini): 0 had no visible answer at all, 0 were mostly empty turns, 7 had visible answers the judge rejected. Across the Mini core requests, 0 of 130 returned nothing visible and stopped at the output cap; 0 of 130 carried hidden reasoning text.
  • Bug fix (2,048 token cap): Mini 5 of 5 (5 of 5: passed). Studio 5 of 5 (5 of 5: passed).
  • Workflow suite, full passes: Mac mini documents 1 of 5, website 1 of 5, folders 0 of 5, log analyzer 0 of 5; Mac Studio documents 4 of 5, website 1 of 5, folders 0 of 5, log analyzer 0 of 5.
  • Video edits: Mini 0 of 5 passed, 0 video files; Studio 0 of 5 passed, 2 video files.
Qwen 3.5 9B 29.1 tok/s on the Mini · 0 of 10 usable answers · bug fix 2 of 5
  • Files: mlx-community/Qwen3.5-9B-MLX-4bit at revision 938d891994, 6.0 GB of weights.
  • Speed: 29.1 output tokens per second on the Mini, 95.4 on the Studio (3.28x, 5 matched pairs). Reading the 480-record document: 999 and 3,318 prompt tokens per second (3.32x, n = 4).
  • Memory: peak sampled footprint 8.2 GB on the Mini and 8.1 GB on the Studio, against the 11.4 GB budget. Guard stops: none.
  • Graded answers: Mac mini 0 of 5 conversations and 0 of 5 writing tasks usable (1 never ran); Mac Studio 0 of 5 conversations and 0 of 5 writing tasks usable. Read the saved answers.
  • Of the graded items that were not usable (Mini): 0 had no visible answer at all, 0 were mostly empty turns, 9 had visible answers the judge rejected. Across the Mini core requests, 0 of 107 returned nothing visible and stopped at the output cap; 0 of 107 carried hidden reasoning text.
  • Bug fix (2,048 token cap): Mini 2 of 5 (3 of 5: explored with tools, then hit the output cap before changing a file; 2 of 5: passed). Studio 2 of 5 (2 of 5: passed; 2 of 5: explored with tools, then hit the output cap before changing a file; 1 of 5: changed files; checks still failed).
  • Workflow suite, full passes: Mac mini documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5; Mac Studio documents 0 of 5, website 3 of 5, folders 1 of 5, log analyzer 0 of 5.
  • Video edits: Mini 0 of 5 passed, 3 video files; Studio 1 of 5 passed, 2 video files.
Gemma 4 12B 19.3 tok/s on the Mini · 7 of 10 usable answers · bug fix 0 of 5
  • Files: mlx-community/gemma-4-12B-it-4bit at revision 73bcf09092, 6.7 GB of weights.
  • Speed: 19.3 output tokens per second on the Mini, 61.0 on the Studio (3.15x, 4 matched pairs). Reading the 480-record document: 552 and 1,903 prompt tokens per second (3.45x, n = 3).
  • Memory: peak sampled footprint 10.0 GB on the Mini and 10.3 GB on the Studio, against the 11.4 GB budget. Guard stops: none.
  • Graded answers: Mac mini 4 of 5 conversations and 3 of 5 writing tasks usable (3 never ran); Mac Studio 5 of 5 conversations and 5 of 5 writing tasks usable. Read the saved answers.
  • Of the graded items that were not usable (Mini): 0 had no visible answer at all, 0 were mostly empty turns, 0 had visible answers the judge rejected. Across the Mini core requests, 0 of 87 returned nothing visible and stopped at the output cap; 3 of 87 carried hidden reasoning text.
  • Bug fix (2,048 token cap): Mini 0 of 5 (5 of 5: explored with tools, then hit the output cap before changing a file). Studio 0 of 5 (5 of 5: explored with tools, then hit the output cap before changing a file).
  • Workflow suite, full passes: Mac mini documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5; Mac Studio documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5.
  • Video edits: Mini 0 of 5 passed, 5 video files; Studio 0 of 5 passed, 3 video files.
LFM 2.5 8B 121.2 tok/s on the Mini · 0 of 10 usable answers · bug fix 0 of 5
  • Files: mlx-community/LFM2.5-8B-A1B-MLX-4bit at revision 146590a491, 4.8 GB of weights.
  • Speed: 121.2 output tokens per second on the Mini, 263.2 on the Studio (2.17x, 5 matched pairs). Reading the 480-record document: 4,519 and 13,984 prompt tokens per second (3.09x, n = 5).
  • Memory: peak sampled footprint 6.2 GB on the Mini and 6.5 GB on the Studio, against the 11.4 GB budget. Guard stops: none.
  • Graded answers: Mac mini 0 of 5 conversations and 0 of 5 writing tasks usable; Mac Studio 0 of 5 conversations and 0 of 5 writing tasks usable. Read the saved answers.
  • Of the graded items that were not usable (Mini): 5 had no visible answer at all, 5 were mostly empty turns, 0 had visible answers the judge rejected. Across the Mini core requests, 108 of 125 returned nothing visible and stopped at the output cap; 125 of 125 carried hidden reasoning text.
  • Bug fix (2,048 token cap): Mini 0 of 5 (5 of 5: no tool call; answered in text and stopped). Studio 0 of 5 (5 of 5: no tool call; answered in text and stopped).
  • Workflow suite, full passes: Mac mini documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5; Mac Studio documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5.
  • Video edits: Mini 0 of 5 passed, 0 video files; Studio 0 of 5 passed, 0 video files.
Phi-4 mini 58.7 tok/s on the Mini · 0 of 10 usable answers · bug fix 0 of 5
  • Files: mlx-community/Phi-4-mini-instruct-mlx-4Bit at revision d848c30f6d, 2.2 GB of weights.
  • Speed: 58.7 output tokens per second on the Mini, 170.5 on the Studio (2.90x, 5 matched pairs). Reading the 480-record document: 1,705 and 5,299 prompt tokens per second (3.11x, n = 5).
  • Memory: peak sampled footprint 7.4 GB on the Mini and 7.6 GB on the Studio, against the 11.4 GB budget. Guard stops: none.
  • Graded answers: Mac mini 0 of 5 conversations and 0 of 5 writing tasks usable; Mac Studio 0 of 5 conversations and 0 of 5 writing tasks usable. Read the saved answers.
  • Of the graded items that were not usable (Mini): 0 had no visible answer at all, 0 were mostly empty turns, 10 had visible answers the judge rejected. Across the Mini core requests, 0 of 125 returned nothing visible and stopped at the output cap; 0 of 125 carried hidden reasoning text.
  • Bug fix (2,048 token cap): Mini 0 of 5 (4 of 5: no tool call; answered in text and stopped; 1 of 5: no tool call; hit the output cap while writing text). Studio 0 of 5 (5 of 5: no tool call; answered in text and stopped).
  • Workflow suite, full passes: Mac mini documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5; Mac Studio documents 0 of 5, website 0 of 5, folders 0 of 5, log analyzer 0 of 5.
  • Video edits: Mini 0 of 5 passed, 0 video files; Studio 0 of 5 passed, 0 video files.

Methods and limits

Setup

ModelModel filesRevisionWeights on disk
Qwen 3.5 4Bmlx-community/Qwen3.5-4B-MLX-4bit32f3e8ecf63.0 GB
Qwen 3.5 9Bmlx-community/Qwen3.5-9B-MLX-4bit938d8919946.0 GB
Gemma 4 12Bmlx-community/gemma-4-12B-it-4bit73bcf090926.7 GB
LFM 2.5 8Bmlx-community/LFM2.5-8B-A1B-MLX-4bit146590a4914.8 GB
Phi-4 minimlx-community/Phi-4-mini-instruct-mlx-4Bitd848c30f6d2.2 GB

Who did what

I chose the question, the machines and the models, and I own both computers. An AI coding agent wrote the test harness, launched and supervised the queues overnight, and built the analysis. Answer quality was graded by a separate AI model (GPT-6 Astra, high reasoning effort) working from blinded packets: it was not shown the model, the machine or the speed. Automatic checks (tests, validators, word error rate) are separate from those judgments and are labelled as such above. No person re-graded the answers.

Product links on my site are affiliate links. I returned the Mini after this test, which is my decision for my use and not a claim about anyone else's.

Limits

Corrections to the video

Four things I said wrong on camera. The numbers on this page are the correct ones.

WhatSaid in the videoCorrect
Image edits"zero out of five"0 of 15. Five briefs, three seeds each, so 15 edits, none judged usable.
GMKtec EVO-X2 memory"120 gigabytes"128 GB of unified memory.
Code fixes worth keeping"small unsupervised code fixes"Supervised. Small change, the model runs the tests, a person reviews the diff.
Long-prompt reading speed, Studio vs Mini"3.1 and 3.5 times faster"3.09x to 3.45x across the five models.

Data

The data behind every table on this page. Raw agent session logs are not published; the saved answers and judgments are.