| Model | Parser | Pass | Fail | Skip | Total | Status |
|---|---|---|---|---|---|---|
| mlx-community/gemma-4-e4b-it-4bit | auto | 103 | 25 | 0 | ❌ [details] |
| # | Test | Group | Coverage | Status | Duration | Details |
|---|---|---|---|---|---|---|
| 1 | swift test execution | UnitTest | unit smoke standard full | FAIL | 79.7s | Expected: test output\nActual: FAIL: no parseable output |
| 2 | Binary exists at .build/release/afm | Preflight | unit smoke standard full | PASS | 0.0s | file exists |
| 3 | Server reachable at http://127.0.0.1:9998 | Preflight | unit smoke standard full | PASS | 0.0s | 200 OK |
| 4 | /v1/models contains model ID | Lifecycle | unit smoke standard full | PASS | 0.1s | model in response |
| 5 | Basic completion returns content | Lifecycle | unit smoke standard full | PASS | 0.3s | non-empty content or reasoning |
| 6 | Stop string '5' absent from output | Stop | unit smoke standard full | PASS | 0.2s | no '5' in content |
| 7 | finish_reason is 'stop' with stop sequence | Stop | unit smoke standard full | PASS | 0.0s | stop |
| 8 | Multi-word stop 'and' truncates correctly | Stop | unit smoke standard full | PASS | 2.6s | no 'and' in output |
| 9 | Stop on newline produces single line | Stop | unit smoke standard full | PASS | 0.1s | single line |
| 10 | Multiple stop sequences [7, 12] | Stop | unit smoke standard full | PASS | 0.4s | neither found |
| 11 | Empty stop array is no-op | Stop | unit smoke standard full | PASS | 0.2s | valid output |
| 12 | Streaming: stop string '5' absent | Stop | unit smoke standard full | FAIL | 0.3s | Expected: no '5' in stream\nActual: FAIL: found '5' |
| 13 | Stop sequence '3.' truncates list | Stop | standard full | PASS | 0.3s | no '3.' in output |
| 14 | Stop 'stopped' doesn't fire on 'stopping' | Stop | standard full | PASS | 1.8s | output produced |
| 15 | Stop 'llo' fires mid-word in 'hello' | Stop | standard full | PASS | 0.1s | no 'llo' |
| 16 | ChoiceLogprobs JSON schema valid | Logprobs | smoke standard full | PASS | 0.2s | valid schema |
| 17 | top_logprobs count <= requested (5) | Logprobs | smoke standard full | PASS | 0.2s | count valid |
| 18 | logprobs=false returns null | Logprobs | smoke standard full | PASS | 0.2s | null logprobs |
| 19 | top_logprobs=99 returns 400 | Logprobs | smoke standard full | PASS | 0.0s | 400 |
| 20 | Streaming logprobs present and valid | Logprobs | standard full | PASS | 0.2s | valid |
| 21 | top_logprobs=0 returns empty arrays | Logprobs | standard full | PASS | 0.2s | empty top_logprobs |
| 22 | reasoning_content present in response | Think | smoke standard full | PASS | 0.0s | present |
| 23 | No <think> tags in content field | Think | smoke standard full | PASS | 0.0s | clean content |
| 24 | Streaming: reasoning_content in deltas | Think | smoke standard full | PASS | 1.7s | present |
| 25 | Stop sequence doesn't break think extraction | Think | smoke standard full | PASS | 3.5s | response ok |
| 26 | reasoning_content has meaningful length (>5 chars) | Think | smoke standard full | PASS | 0.0s | >5 chars |
| 27 | Basic tool call: finish_reason=tool_calls, valid args | Tools | smoke standard full | PASS | 3.5s | valid |
| 28 | tool_choice=none suppresses tool calls | Tools | smoke standard full | PASS | 1.0s | no tool calls |
| 29 | Tool arguments are valid JSON dict | Tools | standard full | PASS | 4.2s | valid JSON |
| 30 | Streaming: tool calls with finish_reason | Tools | standard full | PASS | 4.1s | valid |
| 31 | Multi-tool: at least 1 tool call with 2 tools | Tools | standard full | PASS | 5.2s | >=1 calls |
| 32 | Array param: todos is JSON array (not string) | Tools | standard full | FAIL | 1.2s | Expected: array\nActual: FAIL: todos is string (not array) |
| 33 | Nullable param: anyOf [string, null] does not crash | Tools | standard full | FAIL | 0.0s | Expected: no crash\nActual: FAIL: 'str' object has no attribute 'get' |
| 34 | Multi-turn: tool result produces valid follow-up | Tools | standard full | PASS | 6.2s | text response |
| 35 | Multi-turn nullable: Jinja crash on anyOf [type, null] | Tools | standard full | FAIL | 0.1s | Expected: no crash\nActual: FAIL: ERROR:'str' object has no attribute 'get' |
| 36 | No tools: normal text response | Tools | standard full | PASS | 0.4s | text response |
| 37 | First unique request: uncached suffix remains | Cache | standard full | PASS | 0.4s | cached_tokens < prompt_tokens |
| 38 | Shared-prefix request: cached_tokens>0 | Cache | standard full | FAIL | 0.2s | Expected: >0\nActual: FAIL: cached_tokens=0 |
| 39 | Exact replay: cached response matches cold response | Cache | standard full | PASS | 0.4s | same content |
| 40 | Different unique prompt: uncached suffix remains | Cache | standard full | PASS | 0.5s | cached_tokens < prompt_tokens |
| 41 | Streaming shared-prefix: cached_tokens>0 in usage chunk | Cache | standard full | FAIL | 0.8s | Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage |
| 42 | Streaming replay: content matches non-streaming warmup | Cache | standard full | PASS | 1.0s | same content |
| 43 | Concurrent x8 shared-prefix: uncached suffix remains on every branch | Cache | standard full | FAIL | 3.4s | Expected: 0 < cached_tokens < prompt_tokens for all 8\nActual: FAIL: 1:0/57, 2:0/57, 3:0/57, 4:0/57, 5:0/57, 6:0/57, 7:0/57, 8:0/57 |
| 44 | Concurrent x8 shared-prefix: divergent suffix responses stay isolated | Cache | standard full | FAIL | 3.4s | Expected: each of 8 responses keeps only its own marker\nActual: FAIL: 1:error=Expecting value: line 1 column 1 (char 0); 2:error=Expecting value: line 1 column 1 (char 0); 3:error=Expecting value: line 1 column 1 (char 0); 4:error=Expecting value: line 1 column 1 (char 0); 5:error=Expecting value: line 1 column 1 (char 0); 6:error=Expecting value: line 1 column 1 (char 0); 7:error=Expecting value: line 1 column 1 (char 0); 8:error=Expecting value: line 1 column 1 (char 0) |
| 45 | Two simultaneous requests: both 200 | Concurrent | standard full | PASS | 0.2s | 200+200 |
| 46 | Three simultaneous requests: all 200 | Concurrent | standard full | PASS | 0.2s | all 200 |
| 47 | Empty messages → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 48 | Malformed JSON → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 49 | Missing messages field → 400 | Error | smoke standard full | PASS | 0.0s | 400 |
| 50 | response_format json_object returns valid JSON | Error | smoke standard full | PASS | 0.3s | valid JSON |
| 51 | max_tokens=5 is respected | Error | smoke standard full | PASS | 0.1s | <=10 tokens |
| 52 | OPTIONS /v1/chat/completions → 200 (CORS) | Error | smoke standard full | PASS | 0.0s | 200 |
| 53 | developer role accepted (mapped to system) | Error | smoke standard full | PASS | 6.0s | valid response |
| 54 | enable_thinking=false disables thinking | Kwargs | standard full | PASS | 0.1s | no_think |
| 55 | Streaming: enable_thinking=false disables thinking | Kwargs | standard full | PASS | 0.1s | no_think |
| 56 | Default (no kwargs) retains thinking | Kwargs | standard full | PASS | 1.7s | thinking |
| 57 | enable_thinking=false (2K tokens) returns content | Kwargs | standard full | PASS | 0.1s | content present, no reasoning |
| 58 | enable_thinking=true explicitly keeps thinking | Kwargs | standard full | PASS | 1.5s | thinking |
| 59 | Prefix cache reuse across requests | Cache | standard full | PASS | 0.5s | valid response on cache hit |
| 60 | Issue #32: sequential guided-json → nullable tool (no crash) | Cache | standard full | FAIL | 3.6s | Expected: both succeed\nActual: FAIL: req1=PASS req2=FAIL: 'str' object has no attribute 'get' |
| 61 | Sequential: 3 nullable tool calls (cache reuse) | Cache | standard full | FAIL | 0.1s | Expected: 3/3 succeed\nActual: FAIL: 0/3 req3=FAIL:'str' object has no attribute 'get' |
| 62 | Multi-turn: 5-msg tool conversation with prefix cache | Cache | standard full | PASS | 5.2s | valid response |
| 63 | Long system prompt: 3 sequential requests (prefix reuse) | Cache | standard full | PASS | 7.6s | 3/3 succeed |
| 64 | Sequential stress: 5 mixed requests (text/tool/json/stream/nullable) | Cache | standard full | FAIL | 4.4s | Expected: 5/5 succeed\nActual: FAIL: 4/5 req5=nullable_fail |
| 65 | Multi-turn: 5-turn growing conversation (prefix reuse) | Cache | standard full | PASS | 8.0s | 5/5 turns |
| 66 | Streaming + non-streaming sequential (shared prefix) | Cache | standard full | PASS | 0.4s | 3/3 succeed |
| 67 | Issue #32 comprehensive: nullable tool + multi-turn + prefix | Cache | standard full | FAIL | 0.1s | Expected: no crash\nActual: FAIL: ERROR:'str' object has no attribute 'get' |
| 68 | json_schema produces valid schema-matching JSON | Structured | standard full | PASS | 0.5s | valid JSON |
| 69 | TTFT < 5s | Perf | full | PASS | 0.7s | <5000ms |
| 70 | tok/s > 1 (72.3 tok/s) | Perf | full | PASS | 0.7s | >1 |
| 71 | Long context (~2K tokens) no crash | Perf | full | PASS | 5.9s | valid response |
| 72 | Very long context (~4K tokens) no NaN/garbage | Perf | full | PASS | 7.6s | clean output |
| 73 | Function name correctly extracted | XMLTools | standard full | PASS | 5.0s | get_weather |
| 74 | Parameter values are correct string types | XMLTools | standard full | PASS | 5.4s | strings |
| 75 | Mixed-type params (string+bool+int) parse correctly | XMLTools | standard full | PASS | 2.4s | valid types |
| 76 | Nested object param survives XML parsing | XMLTools | standard full | FAIL | 1.7s | Expected: valid dict\nActual: FAIL: options is str, not dict |
| 77 | tool_choice=required forces tool call | XMLTools | standard full | PASS | 5.1s | tool_calls |
| 78 | tool_choice={function: get_time} calls correct function | XMLTools | standard full | PASS | 3.6s | get_time |
| 79 | Tool call IDs are unique | XMLTools | standard full | PASS | 4.8s | unique IDs |
| 80 | Streaming: XML tool call assembles valid JSON args | XMLTools | standard full | PASS | 2.9s | valid |
| 81 | Streaming: array param is JSON array (not string) | XMLTools | standard full | FAIL | 0.9s | Expected: array\nActual: FAIL: todos is string in stream (not array) |
| 82 | Tool call matches OpenAI schema (id, type, function.name, function.arguments) | XMLTools | standard full | PASS | 4.7s | valid schema |
| 83 | Normal XML tool call works with afm_adaptive_xml | AdaptiveXML | standard full | PASS | 4.1s | valid tool call |
| 84 | Streaming tool call emits valid deltas | AdaptiveXML | standard full | PASS | 5.1s | name + JSON args |
| 85 | Multi-turn: model responds after tool result | AdaptiveXML | standard full | PASS | 1.5s | non-empty content |
| 86 | tool_choice=none suppresses tool calls | AdaptiveXML | standard full | PASS | 3.3s | no tool_calls |
| 87 | Multiple tools: correct function selected | AdaptiveXML | standard full | PASS | 5.1s | get_time |
| 88 | Argument types coerced (string, bool, int) | AdaptiveXML | standard full | PASS | 2.2s | correct types |
| 89 | Tool call valid (with or without grammar constraints) | AdaptiveXML | standard full | PASS | 5.0s | valid |
| 90 | Array of objects coercion (question tool pattern) | AdaptiveXML | standard full | FAIL | 2.8s | Expected: array of dicts\nActual: FAIL: questions is string (not coerced to array) |
| 91 | Number (float) and boolean coercion | AdaptiveXML | standard full | PASS | 5.1s | correct types |
| 92 | Nested object with typed fields coercion | AdaptiveXML | standard full | FAIL | 3.5s | Expected: correct types\nActual: FAIL: ['settings=str (not coerced to object)'] |
| 93 | Streaming array coercion (incremental path) | AdaptiveXML | standard full | FAIL | 2.5s | Expected: array in stream\nActual: FAIL: questions is string (streaming incremental path did not coerce) |
| 94 | XML entity decoding in tool call values | AdaptiveXML | standard full | PASS | 6.8s | decoded |
| 95 | EBNF grammar enforces all required params present | AdaptiveXML | standard full | PASS | 1.5s | command + description |
| 96 | EBNF structured params: array gets json_array constraint | AdaptiveXML | standard full | FAIL | 1.6s | Expected: tags is array\nActual: FAIL: tags is string (not parsed as array — JSON grammar not enforced) |
| 97 | Calculator tool call (non-streaming) | Grammar | standard full | PASS | 4.1s | calculate(expression) |
| 98 | Calculator tool call (streaming) | Grammar | standard full | PASS | 1.3s | calculate(expression) via SSE |
| 99 | Two tools: grammar allows correct selection | Grammar | standard full | PASS | 3.3s | get_weather selected |
| 100 | Two tools: grammar selects calculate | Grammar | standard full | PASS | 3.9s | calculate selected |
| 101 | Grammar enforces 3 required params (send_email) | Grammar | standard full | PASS | 3.4s | to + subject + body |
| 102 | Grammar constrains array param at generation time | Grammar | standard full | FAIL | 4.6s | Expected: tags is array\nActual: FAIL: tags is string — grammar should have enforced json_array |
| 103 | Grammar array param via streaming | Grammar | standard full | FAIL | 5.0s | Expected: tags is array (SSE)\nActual: FAIL: tags is string in streaming path |
| 104 | Complex schema: string + int + array + object | Grammar | standard full | FAIL | 7.1s | Expected: correct types\nActual: FAIL: ['assignees=str (not array)'] |
| 105 | Header absent when grammar enabled (tool strict:true) | StrictWiring | smoke standard full | PASS | 2.1s | no header |
| 106 | Header absent when grammar enabled (schema strict:true) | StrictWiring | smoke standard full | PASS | 0.5s | no header |
| 107 | No header when strict absent | StrictWiring | smoke standard full | PASS | 3.2s | no header |
| 108 | Streaming json_schema strict:true returns valid JSON | StrictWiring | smoke standard full | PASS | 0.4s | valid JSON with name+age |
| 109 | Streaming tool strict:true returns valid tool call | StrictWiring | smoke standard full | PASS | 4.3s | get_weather(location) |
| 110 | strict:false does not error (best-effort) | StrictWiring | smoke standard full | FAIL | 2.7s | Expected: valid response\nActual: FAIL: no tool call and no content |
| 111 | POST /v1/files upload returns file ID | Batch | standard full | PASS | 0.0s | file-* |
| 112 | GET /v1/files/:id returns file metadata | Batch | standard full | PASS | 0.0s | file object |
| 113 | POST /v1/batches creates batch | Batch | standard full | PASS | 0.0s | in_progress/completed |
| 114 | GET /v1/batches/:id polls to completed | Batch | standard full | PASS | 2.1s | completed |
| 115 | Output JSONL contains both results (2/2) | Batch | standard full | PASS | 0.0s | assert-1 + assert-2 |
| 116 | GET /v1/batches lists completed batch | Batch | standard full | PASS | 0.0s | list with ≥1 batch |
| 117 | DELETE /v1/files/:id removes uploaded file | Batch | standard full | PASS | 0.0s | deleted=true |
| 118 | SSE multiplex: 2 non-streaming requests tagged with custom_id | Batch | standard full | PASS | 0.3s | sse-a + sse-b + [DONE] |
| 119 | SSE multiplex: streaming interleaved with finish_reason | Batch | standard full | PASS | 0.4s | str-x + str-y chunks + stop + [DONE] |
| 120 | SSE multiplex rejects duplicate custom_ids | Batch | standard full | PASS | 0.0s | 400 |
| 121 | SSE multiplex rejects empty requests | Batch | standard full | PASS | 0.0s | 400 |
| 122 | batch + top_k + streaming | PairwiseSmoke | standard full | PASS | 0.2s | content present |
| 123 | batch + presence_penalty + non-streaming | PairwiseSmoke | standard full | PASS | 0.3s | finish_reason present |
| 124 | batch + repetition_penalty + logprobs | PairwiseSmoke | standard full | PASS | 0.2s | response valid |
| 125 | batch + all sampling params combined | PairwiseSmoke | standard full | PASS | 0.2s | response valid |
| 126 | streaming parity (same seed → same output) | PairwiseSmoke | standard full | PASS | 0.3s | match |
| 127 | cache idempotency (same seed → same output) | PairwiseSmoke | standard full | FAIL | 0.3s | Expected: r1='' vs r2=''\nActual: FAIL: mismatch |