AFM Multi-Model Assertion Report

Generated: 2026-04-04 09:58:51 | Tier: standard | Models: 1 configurations
Binary: .build/release/afm | Cache: /Volumes/edata/models/vesta-test-cache
0%   120 / 0 passed across all models — 4 failures

Summary

ModelParserPassFailSkipTotalStatus
mlx-community/Qwen3.5-35B-A3B-4bit auto 120 4 0 ❌ [details]

Detailed Results

mlx-community/Qwen3.5-35B-A3B-4bit auto

120 passed / 4 failed / total
#TestGroupCoverageStatusDurationDetails
1 swift test execution UnitTest unit smoke standard full FAIL 75.8s
Expected: test output\nActual: FAIL: no parseable output
2 Binary exists at .build/release/afm Preflight unit smoke standard full PASS 0.0s
file exists
3 Server reachable at http://127.0.0.1:9998 Preflight unit smoke standard full PASS 0.0s
200 OK
4 /v1/models contains model ID Lifecycle unit smoke standard full PASS 0.0s
model in response
5 Basic completion returns content Lifecycle unit smoke standard full PASS 5.0s
non-empty content or reasoning
6 Stop string '5' absent from output Stop unit smoke standard full PASS 1.7s
no '5' in content
7 finish_reason is 'stop' with stop sequence Stop unit smoke standard full PASS 0.0s
stop
8 Multi-word stop 'and' truncates correctly Stop unit smoke standard full PASS 1.7s
no 'and' in output
9 Stop on newline produces single line Stop unit smoke standard full PASS 1.7s
single line
10 Multiple stop sequences [7, 12] Stop unit smoke standard full PASS 1.7s
neither found
11 Empty stop array is no-op Stop unit smoke standard full PASS 2.8s
valid output
12 Streaming: stop string '5' absent Stop unit smoke standard full PASS 1.8s
no '5' in stream
13 Stop sequence '3.' truncates list Stop standard full PASS 0.9s
no '3.' in output
14 Stop 'stopped' doesn't fire on 'stopping' Stop standard full PASS 1.9s
output produced
15 Stop 'llo' fires mid-word in 'hello' Stop standard full PASS 2.7s
no 'llo'
16 ChoiceLogprobs JSON schema valid Logprobs smoke standard full PASS 0.1s
valid schema
17 top_logprobs count <= requested (5) Logprobs smoke standard full PASS 0.1s
count valid
18 logprobs=false returns null Logprobs smoke standard full PASS 0.1s
null logprobs
19 top_logprobs=99 returns 400 Logprobs smoke standard full PASS 0.0s
400
20 Streaming logprobs present and valid Logprobs standard full PASS 0.1s
valid
21 top_logprobs=0 returns empty arrays Logprobs standard full PASS 0.1s
empty top_logprobs
22 reasoning_content present in response Think smoke standard full PASS 0.0s
present
23 No <think> tags in content field Think smoke standard full PASS 0.0s
clean content
24 Streaming: reasoning_content in deltas Think smoke standard full PASS 0.9s
present
25 Stop sequence doesn't break think extraction Think smoke standard full PASS 1.7s
response ok
26 reasoning_content has meaningful length (>5 chars) Think smoke standard full PASS 0.0s
>5 chars
27 Basic tool call: finish_reason=tool_calls, valid args Tools smoke standard full PASS 1.4s
valid
28 tool_choice=none suppresses tool calls Tools smoke standard full PASS 0.9s
no tool calls
29 Tool arguments are valid JSON dict Tools standard full PASS 0.7s
valid JSON
30 Streaming: tool calls with finish_reason Tools standard full PASS 1.0s
valid
31 Multi-tool: at least 1 tool call with 2 tools Tools standard full PASS 1.2s
>=1 calls
32 Array param: todos is JSON array (not string) Tools standard full PASS 1.0s
array
33 Nullable param: anyOf [string, null] does not crash Tools standard full PASS 0.9s
no crash
34 Multi-turn: tool result produces valid follow-up Tools standard full PASS 2.1s
text response
35 Multi-turn nullable: full round-trip (anyOf params) Tools standard full PASS 2.6s
valid follow-up
36 No tools: normal text response Tools standard full PASS 0.2s
text response
37 First unique request: uncached suffix remains Cache standard full PASS 0.3s
cached_tokens < prompt_tokens
38 Shared-prefix request: cached_tokens>0 Cache standard full PASS 0.2s
>0
39 Exact replay: cached response matches cold response Cache standard full PASS 0.3s
same content
40 Different unique prompt: uncached suffix remains Cache standard full PASS 0.3s
cached_tokens < prompt_tokens
41 Streaming shared-prefix: cached_tokens>0 in usage chunk Cache standard full PASS 0.7s
>0
42 Streaming replay: content matches non-streaming warmup Cache standard full PASS 0.8s
same content
43 Concurrent x8 shared-prefix: uncached suffix remains on every branch Cache standard full FAIL 2.5s
Expected: 0 < cached_tokens < prompt_tokens for all 8\nActual: FAIL: 2:0/59, 3:0/59, 4:0/59, 5:0/59, 6:0/59, 7:0/59, 8:0/59
44 Concurrent x8 shared-prefix: divergent suffix responses stay isolated Cache standard full FAIL 2.5s
Expected: each of 8 responses keeps only its own marker\nActual: FAIL: 1:error=Expecting value: line 1 column 1 (char 0); 2:error=Expecting value: line 1 column 1 (char 0); 3:error=Expecting value: line 1 column 1 (char 0); 4:error=Expecting value: line 1 column 1 (char 0); 5:error=Expecting value: line 1 column 1 (char 0); 6:error=Expecting value: line 1 column 1 (char 0); 7:error=Expecting value: line 1 column 1 (char 0); 8:error=Expecting value: line 1 column 1 (char 0)
45 Two simultaneous requests: both 200 Concurrent standard full PASS 0.2s
200+200
46 Three simultaneous requests: all 200 Concurrent standard full PASS 0.2s
all 200
47 Empty messages → 400 Error smoke standard full PASS 0.0s
400
48 Malformed JSON → 400 Error smoke standard full PASS 0.0s
400
49 Missing messages field → 400 Error smoke standard full PASS 0.0s
400
50 response_format json_object returns valid JSON Error smoke standard full PASS 0.2s
valid JSON
51 max_tokens=5 is respected Error smoke standard full PASS 0.1s
<=10 tokens
52 OPTIONS /v1/chat/completions → 200 (CORS) Error smoke standard full PASS 0.0s
200
53 developer role accepted (mapped to system) Error smoke standard full PASS 4.2s
valid response
54 enable_thinking=false disables thinking Kwargs standard full PASS 0.1s
no_think
55 Streaming: enable_thinking=false disables thinking Kwargs standard full PASS 0.1s
no_think
56 Default (no kwargs) retains thinking Kwargs standard full PASS 1.4s
thinking
57 enable_thinking=false (2K tokens) returns content Kwargs standard full PASS 0.1s
content present, no reasoning
58 enable_thinking=true explicitly keeps thinking Kwargs standard full PASS 1.4s
thinking
59 Prefix cache reuse across requests Cache standard full PASS 0.3s
valid response on cache hit
60 Issue #32: sequential guided-json → nullable tool (no crash) Cache standard full PASS 2.9s
both succeed
61 Sequential: 3 nullable tool calls (cache reuse) Cache standard full PASS 3.5s
3/3 succeed
62 Multi-turn: 5-msg tool conversation with prefix cache Cache standard full PASS 1.5s
valid response
63 Long system prompt: 3 sequential requests (prefix reuse) Cache standard full PASS 4.6s
3/3 succeed
64 Sequential stress: 5 mixed requests (text/tool/json/stream/nullable) Cache standard full PASS 3.3s
5/5 succeed
65 Multi-turn: 5-turn growing conversation (prefix reuse) Cache standard full PASS 3.9s
5/5 turns
66 Streaming + non-streaming sequential (shared prefix) Cache standard full PASS 0.9s
3/3 succeed
67 Issue #32 comprehensive: nullable tool + multi-turn + prefix Cache standard full PASS 2.0s
valid response
68 json_schema produces valid schema-matching JSON Structured standard full PASS 0.2s
valid JSON
69 Function name correctly extracted XMLTools standard full PASS 1.2s
get_weather
70 Parameter values are correct string types XMLTools standard full PASS 1.0s
strings
71 Mixed-type params (string+bool+int) parse correctly XMLTools standard full PASS 1.3s
valid types
72 Nested object param survives XML parsing XMLTools standard full PASS 1.4s
valid dict
73 tool_choice=required forces tool call XMLTools standard full PASS 0.9s
tool_calls
74 tool_choice={function: get_time} calls correct function XMLTools standard full PASS 0.9s
get_time
75 Tool call IDs are unique XMLTools standard full PASS 1.6s
unique IDs
76 Streaming: XML tool call assembles valid JSON args XMLTools standard full PASS 0.9s
valid
77 Streaming: array param is JSON array (not string) XMLTools standard full PASS 1.0s
array
78 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools standard full PASS 1.2s
valid schema
79 Normal XML tool call works with afm_adaptive_xml AdaptiveXML standard full PASS 1.1s
valid tool call
80 Streaming tool call emits valid deltas AdaptiveXML standard full PASS 0.9s
name + JSON args
81 Multi-turn: model responds after tool result AdaptiveXML standard full PASS 0.8s
non-empty content
82 tool_choice=none suppresses tool calls AdaptiveXML standard full PASS 1.7s
no tool_calls
83 Multiple tools: correct function selected AdaptiveXML standard full PASS 0.8s
get_time
84 Argument types coerced (string, bool, int) AdaptiveXML standard full PASS 1.2s
correct types
85 Tool call valid (with or without grammar constraints) AdaptiveXML standard full PASS 1.3s
valid
86 Array of objects coercion (question tool pattern) AdaptiveXML standard full PASS 2.3s
array of dicts
87 Number (float) and boolean coercion AdaptiveXML standard full PASS 1.1s
correct types
88 Nested object with typed fields coercion AdaptiveXML standard full PASS 1.2s
correct types
89 Streaming array coercion (incremental path) AdaptiveXML standard full PASS 1.3s
array in stream
90 XML entity decoding in tool call values AdaptiveXML standard full PASS 2.2s
decoded
91 EBNF grammar enforces all required params present AdaptiveXML standard full PASS 0.9s
command + description
92 EBNF structured params: array gets json_array constraint AdaptiveXML standard full PASS 1.5s
tags is array
93 Calculator tool call (non-streaming) Grammar standard full PASS 1.3s
calculate(expression)
94 Calculator tool call (streaming) Grammar standard full PASS 0.9s
calculate(expression) via SSE
95 Two tools: grammar allows correct selection Grammar standard full PASS 1.4s
get_weather selected
96 Two tools: grammar selects calculate Grammar standard full PASS 0.9s
calculate selected
97 Grammar enforces 3 required params (send_email) Grammar standard full PASS 1.7s
to + subject + body
98 Grammar constrains array param at generation time Grammar standard full PASS 1.7s
tags is array
99 Grammar array param via streaming Grammar standard full PASS 1.6s
tags is array (SSE)
100 Complex schema: string + int + array + object Grammar standard full PASS 2.8s
correct types
101 Header absent when grammar enabled (tool strict:true) StrictWiring smoke standard full PASS 0.9s
no header
102 Header absent when grammar enabled (schema strict:true) StrictWiring smoke standard full PASS 0.3s
no header
103 No header when strict absent StrictWiring smoke standard full PASS 0.6s
no header
104 Streaming json_schema strict:true returns valid JSON StrictWiring smoke standard full PASS 0.2s
valid JSON with name+age
105 Streaming tool strict:true returns valid tool call StrictWiring smoke standard full PASS 1.3s
get_weather(location)
106 strict:false does not error (best-effort) StrictWiring smoke standard full PASS 1.0s
valid response
107 POST /v1/files upload returns file ID Batch standard full PASS 0.0s
file-*
108 GET /v1/files/:id returns file metadata Batch standard full PASS 0.0s
file object
109 POST /v1/batches creates batch Batch standard full PASS 0.0s
in_progress/completed
110 GET /v1/batches/:id polls to completed Batch standard full PASS 2.1s
completed
111 Output JSONL contains both results (2/2) Batch standard full PASS 0.0s
assert-1 + assert-2
112 GET /v1/batches lists completed batch Batch standard full PASS 0.0s
list with ≥1 batch
113 DELETE /v1/files/:id removes uploaded file Batch standard full PASS 0.0s
deleted=true
114 SSE multiplex: 2 non-streaming requests tagged with custom_id Batch standard full PASS 0.2s
sse-a + sse-b + [DONE]
115 SSE multiplex: streaming interleaved with finish_reason Batch standard full PASS 0.3s
str-x + str-y chunks + stop + [DONE]
116 SSE multiplex rejects duplicate custom_ids Batch standard full PASS 0.0s
400
117 SSE multiplex rejects empty requests Batch standard full PASS 0.0s
400
118 batch + top_k + streaming PairwiseSmoke standard full PASS 0.1s
content present
119 batch + presence_penalty + non-streaming PairwiseSmoke standard full PASS 0.1s
finish_reason present
120 batch + repetition_penalty + logprobs PairwiseSmoke standard full PASS 0.1s
response valid
121 batch + all sampling params combined PairwiseSmoke standard full PASS 0.1s
response valid
122 streaming parity (same seed → same output) PairwiseSmoke standard full PASS 0.2s
match
123 cache idempotency (same seed → same output) PairwiseSmoke standard full PASS 0.2s
match