AFM Multi-Model Assertion Report

Generated: 2026-03-06 20:01:00 | Tier: standard | Models: 3 configurations
Binary: .build/release/afm | Cache: /Volumes/edata/models/vesta-test-cache
0%   318 / 0 passed across all models — 34 failures

Summary

ModelParserPassFailSkipTotalStatus
mlx-community/Qwen3-Coder-Next-4bit auto 46 6 0 ❌ [details]
mlx-community/Qwen3-Coder-Next-4bit qwen3_xml 46 6 0 ❌ [details]
mlx-community/Qwen3.5-9B-MLX-4bit auto 55 7 0 ❌ [details]
mlx-community/Qwen3.5-9B-MLX-4bit qwen3_xml 55 7 0 ❌ [details]
mlx-community/Qwen3.5-35B-A3B-4bit auto 58 4 0 ❌ [details]
mlx-community/Qwen3.5-35B-A3B-4bit qwen3_xml 58 4 0 ❌ [details]

Detailed Results

mlx-community/Qwen3-Coder-Next-4bit auto

46 passed / 6 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 1.3s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 0.2s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 0.3s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 0.3s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 0.6s
neither found
10 Empty stop array is no-op Stop PASS 0.2s
valid output
11 Streaming: stop string '5' absent Stop FAIL 0.2s
Expected: no '5' in stream\nActual: FAIL: found '5'
12 Stop sequence '3.' truncates list Stop PASS 0.3s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 0.1s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 0.1s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.2s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.2s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 Think extraction (model lacks <think> support) Think SKIP
skip
22 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 0.7s
valid
23 tool_choice=none suppresses tool calls Tools FAIL 0.7s
Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'function': {'arguments': '{"location":"Paris","unit":"celsius"}', 'name': 'get_weather'}, 'type': 'function', 'id': 'call_YYMQ3Ua38byiUXXsdPXByFy0'}]
24 Tool arguments are valid JSON dict Tools PASS 0.8s
valid JSON
25 Streaming: tool calls with finish_reason Tools PASS 0.7s
valid
26 Multi-tool: at least 1 tool call with 2 tools Tools PASS 0.9s
>=1 calls
27 Array param: todos is JSON array (not string) Tools PASS 0.7s
array
28 Nullable param: anyOf [string, null] does not crash Tools PASS 0.6s
no crash
29 No tools: normal text response Tools PASS 0.2s
text response
30 First request: cached_tokens=0 Cache PASS 0.4s
0
31 Second identical request: cached_tokens>0 Cache FAIL 0.4s
Expected: >0\nActual: FAIL: cached_tokens=0
32 Different prompt: cached_tokens=0 Cache PASS 0.4s
0
33 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.5s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
34 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
35 Three simultaneous requests: all 200 Concurrent PASS 0.2s
all 200
36 Empty messages → 400 Error PASS 0.0s
400
37 Malformed JSON → 400 Error PASS 0.0s
400
38 Missing messages field → 400 Error PASS 0.0s
400
39 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
40 max_tokens=5 is respected Error PASS 0.2s
<=10 tokens
41 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
42 developer role accepted (mapped to system) Error PASS 2.4s
valid response
43 chat_template_kwargs (model lacks thinking) Kwargs SKIP
skip
44 Function name correctly extracted XMLTools PASS 0.8s
get_weather
45 Parameter values are correct string types XMLTools PASS 0.8s
strings
46 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 0.9s
valid types
47 Nested object param survives XML parsing XMLTools PASS 0.9s
valid dict
48 tool_choice=required forces tool call XMLTools PASS 0.7s
tool_calls
49 tool_choice={function: get_time} calls correct function XMLTools PASS 0.6s
get_time
50 Tool call IDs are unique XMLTools PASS 0.9s
unique IDs
51 Streaming: XML tool call assembles valid JSON args XMLTools PASS 0.8s
valid
52 Streaming: array param is JSON array (not string) XMLTools PASS 0.8s
array
53 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 0.7s
valid schema

mlx-community/Qwen3-Coder-Next-4bit qwen3_xml

46 passed / 6 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 1.0s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 0.3s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 0.3s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 0.3s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 0.6s
neither found
10 Empty stop array is no-op Stop PASS 0.2s
valid output
11 Streaming: stop string '5' absent Stop FAIL 0.2s
Expected: no '5' in stream\nActual: FAIL: found '5'
12 Stop sequence '3.' truncates list Stop PASS 0.3s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 0.1s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 0.1s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.2s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.2s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 Think extraction (model lacks <think> support) Think SKIP
skip
22 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 0.7s
valid
23 tool_choice=none suppresses tool calls Tools FAIL 0.7s
Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'function': {'arguments': '{"location":"Paris","unit":"celsius"}', 'name': 'get_weather'}, 'id': 'call_UYr0J6kzHpbD0gODl8bUoGo0', 'type': 'function'}]
24 Tool arguments are valid JSON dict Tools PASS 0.8s
valid JSON
25 Streaming: tool calls with finish_reason Tools PASS 0.7s
valid
26 Multi-tool: at least 1 tool call with 2 tools Tools PASS 0.9s
>=1 calls
27 Array param: todos is JSON array (not string) Tools PASS 0.7s
array
28 Nullable param: anyOf [string, null] does not crash Tools PASS 0.6s
no crash
29 No tools: normal text response Tools PASS 0.2s
text response
30 First request: cached_tokens=0 Cache PASS 0.4s
0
31 Second identical request: cached_tokens>0 Cache FAIL 0.4s
Expected: >0\nActual: FAIL: cached_tokens=0
32 Different prompt: cached_tokens=0 Cache PASS 0.4s
0
33 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.5s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
34 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
35 Three simultaneous requests: all 200 Concurrent PASS 0.2s
all 200
36 Empty messages → 400 Error PASS 0.0s
400
37 Malformed JSON → 400 Error PASS 0.0s
400
38 Missing messages field → 400 Error PASS 0.0s
400
39 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
40 max_tokens=5 is respected Error PASS 0.2s
<=10 tokens
41 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
42 developer role accepted (mapped to system) Error PASS 2.3s
valid response
43 chat_template_kwargs (model lacks thinking) Kwargs SKIP
skip
44 Function name correctly extracted XMLTools PASS 0.8s
get_weather
45 Parameter values are correct string types XMLTools PASS 0.8s
strings
46 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 0.9s
valid types
47 Nested object param survives XML parsing XMLTools PASS 0.9s
valid dict
48 tool_choice=required forces tool call XMLTools PASS 0.7s
tool_calls
49 tool_choice={function: get_time} calls correct function XMLTools PASS 0.6s
get_time
50 Tool call IDs are unique XMLTools PASS 0.9s
unique IDs
51 Streaming: XML tool call assembles valid JSON args XMLTools PASS 0.7s
valid
52 Streaming: array param is JSON array (not string) XMLTools PASS 0.8s
array
53 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 0.7s
valid schema

mlx-community/Qwen3.5-9B-MLX-4bit auto

55 passed / 7 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 3.7s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 1.9s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 1.8s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 2.9s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 1.8s
neither found
10 Empty stop array is no-op Stop PASS 4.4s
valid output
11 Streaming: stop string '5' absent Stop PASS 1.8s
no '5' in stream
12 Stop sequence '3.' truncates list Stop PASS 1.0s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 2.2s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 1.5s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.1s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.1s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 reasoning_content present in response Think PASS 0.0s
present
22 No <think> tags in content field Think PASS 0.0s
clean content
23 Streaming: reasoning_content in deltas Think PASS 0.9s
present
24 Stop sequence doesn't break think extraction Think PASS 1.8s
response ok
25 reasoning_content has meaningful length (>5 chars) Think PASS 0.0s
>5 chars
26 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 1.1s
valid
27 tool_choice=none suppresses tool calls Tools FAIL 1.1s
Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'id': 'call_Z70ubDpU4wS6kVzQGE8yNl1s', 'type': 'function', 'function': {'name': 'get_weather', 'arguments': '{"location":"Paris"}'}}]
28 Tool arguments are valid JSON dict Tools PASS 1.3s
valid JSON
29 Streaming: tool calls with finish_reason Tools PASS 1.1s
valid
30 Multi-tool: at least 1 tool call with 2 tools Tools PASS 1.5s
>=1 calls
31 Array param: todos is JSON array (not string) Tools PASS 1.0s
array
32 Nullable param: anyOf [string, null] does not crash Tools PASS 1.2s
no crash
33 No tools: normal text response Tools PASS 0.2s
text response
34 First request: cached_tokens=0 Cache PASS 0.3s
0
35 Second identical request: cached_tokens>0 Cache FAIL 0.3s
Expected: >0\nActual: FAIL: cached_tokens=0
36 Different prompt: cached_tokens=0 Cache PASS 0.3s
0
37 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.3s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
38 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
39 Three simultaneous requests: all 200 Concurrent PASS 0.3s
all 200
40 Empty messages → 400 Error PASS 0.0s
400
41 Malformed JSON → 400 Error PASS 0.0s
400
42 Missing messages field → 400 Error PASS 0.0s
400
43 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
44 max_tokens=5 is respected Error PASS 0.1s
<=10 tokens
45 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
46 developer role accepted (mapped to system) Error PASS 4.4s
valid response
47 enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
48 Streaming: enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
49 Default (no kwargs) retains thinking Kwargs FAIL 1.8s
Expected: thinking\nActual: FAIL: state=empty
50 enable_thinking=false (2K tokens) returns content Kwargs PASS 0.1s
content present, no reasoning
51 enable_thinking=true explicitly keeps thinking Kwargs FAIL 1.8s
Expected: thinking\nActual: FAIL: state=empty
52 Function name correctly extracted XMLTools PASS 1.1s
get_weather
53 Parameter values are correct string types XMLTools PASS 1.0s
strings
54 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 1.5s
valid types
55 Nested object param survives XML parsing XMLTools PASS 1.5s
valid dict
56 tool_choice=required forces tool call XMLTools PASS 1.1s
tool_calls
57 tool_choice={function: get_time} calls correct function XMLTools PASS 1.2s
get_time
58 Tool call IDs are unique XMLTools PASS 1.7s
unique IDs
59 Streaming: XML tool call assembles valid JSON args XMLTools PASS 1.0s
valid
60 Streaming: array param is JSON array (not string) XMLTools PASS 1.4s
array
61 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 1.1s
valid schema

mlx-community/Qwen3.5-9B-MLX-4bit qwen3_xml

55 passed / 7 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 3.6s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 1.9s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 1.8s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 2.9s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 1.8s
neither found
10 Empty stop array is no-op Stop PASS 4.4s
valid output
11 Streaming: stop string '5' absent Stop PASS 1.8s
no '5' in stream
12 Stop sequence '3.' truncates list Stop PASS 1.0s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 2.2s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 1.5s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.1s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.1s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 reasoning_content present in response Think PASS 0.0s
present
22 No <think> tags in content field Think PASS 0.0s
clean content
23 Streaming: reasoning_content in deltas Think PASS 0.9s
present
24 Stop sequence doesn't break think extraction Think PASS 1.8s
response ok
25 reasoning_content has meaningful length (>5 chars) Think PASS 0.0s
>5 chars
26 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 1.1s
valid
27 tool_choice=none suppresses tool calls Tools FAIL 1.1s
Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'function': {'name': 'get_weather', 'arguments': '{"location":"Paris"}'}, 'type': 'function', 'id': 'call_woQH7pd6sAkUe0hXzGzgjz3a'}]
28 Tool arguments are valid JSON dict Tools PASS 1.0s
valid JSON
29 Streaming: tool calls with finish_reason Tools PASS 1.1s
valid
30 Multi-tool: at least 1 tool call with 2 tools Tools PASS 1.6s
>=1 calls
31 Array param: todos is JSON array (not string) Tools PASS 1.0s
array
32 Nullable param: anyOf [string, null] does not crash Tools PASS 1.3s
no crash
33 No tools: normal text response Tools PASS 0.2s
text response
34 First request: cached_tokens=0 Cache PASS 0.3s
0
35 Second identical request: cached_tokens>0 Cache FAIL 0.3s
Expected: >0\nActual: FAIL: cached_tokens=0
36 Different prompt: cached_tokens=0 Cache PASS 0.3s
0
37 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.3s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
38 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
39 Three simultaneous requests: all 200 Concurrent PASS 0.3s
all 200
40 Empty messages → 400 Error PASS 0.0s
400
41 Malformed JSON → 400 Error PASS 0.0s
400
42 Missing messages field → 400 Error PASS 0.0s
400
43 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
44 max_tokens=5 is respected Error PASS 0.1s
<=10 tokens
45 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
46 developer role accepted (mapped to system) Error PASS 4.5s
valid response
47 enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
48 Streaming: enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
49 Default (no kwargs) retains thinking Kwargs FAIL 1.8s
Expected: thinking\nActual: FAIL: state=empty
50 enable_thinking=false (2K tokens) returns content Kwargs PASS 0.1s
content present, no reasoning
51 enable_thinking=true explicitly keeps thinking Kwargs FAIL 1.8s
Expected: thinking\nActual: FAIL: state=empty
52 Function name correctly extracted XMLTools PASS 1.1s
get_weather
53 Parameter values are correct string types XMLTools PASS 1.4s
strings
54 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 1.4s
valid types
55 Nested object param survives XML parsing XMLTools PASS 1.6s
valid dict
56 tool_choice=required forces tool call XMLTools PASS 1.3s
tool_calls
57 tool_choice={function: get_time} calls correct function XMLTools PASS 1.1s
get_time
58 Tool call IDs are unique XMLTools PASS 1.7s
unique IDs
59 Streaming: XML tool call assembles valid JSON args XMLTools PASS 1.0s
valid
60 Streaming: array param is JSON array (not string) XMLTools PASS 1.3s
array
61 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 1.1s
valid schema

mlx-community/Qwen3.5-35B-A3B-4bit auto

58 passed / 4 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 4.8s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 1.8s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 1.8s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 1.5s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 1.8s
neither found
10 Empty stop array is no-op Stop PASS 2.2s
valid output
11 Streaming: stop string '5' absent Stop PASS 1.8s
no '5' in stream
12 Stop sequence '3.' truncates list Stop PASS 0.9s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 2.0s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 1.8s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.1s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.1s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 reasoning_content present in response Think PASS 0.0s
present
22 No <think> tags in content field Think PASS 0.0s
clean content
23 Streaming: reasoning_content in deltas Think PASS 0.9s
present
24 Stop sequence doesn't break think extraction Think PASS 1.8s
response ok
25 reasoning_content has meaningful length (>5 chars) Think PASS 0.0s
>5 chars
26 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 1.5s
valid
27 tool_choice=none suppresses tool calls Tools PASS 1.0s
no tool calls
28 Tool arguments are valid JSON dict Tools PASS 1.1s
valid JSON
29 Streaming: tool calls with finish_reason Tools PASS 1.0s
valid
30 Multi-tool: at least 1 tool call with 2 tools Tools PASS 1.6s
>=1 calls
31 Array param: todos is JSON array (not string) Tools PASS 0.9s
array
32 Nullable param: anyOf [string, null] does not crash Tools PASS 1.2s
no crash
33 No tools: normal text response Tools PASS 0.2s
text response
34 First request: cached_tokens=0 Cache PASS 0.2s
0
35 Second identical request: cached_tokens>0 Cache FAIL 0.3s
Expected: >0\nActual: FAIL: cached_tokens=0
36 Different prompt: cached_tokens=0 Cache PASS 0.2s
0
37 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.3s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
38 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
39 Three simultaneous requests: all 200 Concurrent PASS 0.3s
all 200
40 Empty messages → 400 Error PASS 0.0s
400
41 Malformed JSON → 400 Error PASS 0.0s
400
42 Missing messages field → 400 Error PASS 0.0s
400
43 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
44 max_tokens=5 is respected Error PASS 0.1s
<=10 tokens
45 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
46 developer role accepted (mapped to system) Error PASS 4.3s
valid response
47 enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
48 Streaming: enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
49 Default (no kwargs) retains thinking Kwargs PASS 1.5s
thinking
50 enable_thinking=false (2K tokens) returns content Kwargs PASS 0.1s
content present, no reasoning
51 enable_thinking=true explicitly keeps thinking Kwargs PASS 1.5s
thinking
52 Function name correctly extracted XMLTools PASS 1.1s
get_weather
53 Parameter values are correct string types XMLTools PASS 0.9s
strings
54 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 1.3s
valid types
55 Nested object param survives XML parsing XMLTools PASS 1.1s
valid dict
56 tool_choice=required forces tool call XMLTools PASS 1.4s
tool_calls
57 tool_choice={function: get_time} calls correct function XMLTools PASS 1.1s
get_time
58 Tool call IDs are unique XMLTools PASS 1.6s
unique IDs
59 Streaming: XML tool call assembles valid JSON args XMLTools PASS 1.0s
valid
60 Streaming: array param is JSON array (not string) XMLTools PASS 1.0s
array
61 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 1.1s
valid schema

mlx-community/Qwen3.5-35B-A3B-4bit qwen3_xml

58 passed / 4 failed / total
GroupTestExpectedResultTime
1 Binary exists at .build/release/afm Preflight PASS 0.0s
file exists
2 Server reachable at http://127.0.0.1:9998 Preflight PASS 0.0s
200 OK
3 /v1/models contains model ID Lifecycle PASS 0.0s
model in response
4 Basic completion returns content Lifecycle PASS 4.9s
non-empty content or reasoning
5 Stop string '5' absent from output Stop PASS 1.8s
no '5' in content
6 finish_reason is 'stop' with stop sequence Stop PASS 0.0s
stop
7 Multi-word stop 'and' truncates correctly Stop PASS 1.8s
no 'and' in output
8 Stop on newline produces single line Stop FAIL 1.5s
Expected: single line\nActual: FAIL: multi-line output
9 Multiple stop sequences [7, 12] Stop PASS 1.7s
neither found
10 Empty stop array is no-op Stop PASS 2.2s
valid output
11 Streaming: stop string '5' absent Stop PASS 1.8s
no '5' in stream
12 Stop sequence '3.' truncates list Stop PASS 0.9s
no '3.' in output
13 Stop 'stopped' doesn't fire on 'stopping' Stop PASS 2.0s
output produced
14 Stop 'llo' fires mid-word in 'hello' Stop PASS 1.8s
no 'llo'
15 ChoiceLogprobs JSON schema valid Logprobs PASS 0.1s
valid schema
16 top_logprobs count <= requested (5) Logprobs PASS 0.1s
count valid
17 logprobs=false returns null Logprobs PASS 0.1s
null logprobs
18 top_logprobs=99 returns 400 Logprobs PASS 0.0s
400
19 Streaming logprobs present and valid Logprobs PASS 0.1s
valid
20 top_logprobs=0 returns empty arrays Logprobs PASS 0.1s
empty top_logprobs
21 reasoning_content present in response Think PASS 0.0s
present
22 No <think> tags in content field Think PASS 0.0s
clean content
23 Streaming: reasoning_content in deltas Think PASS 0.9s
present
24 Stop sequence doesn't break think extraction Think PASS 1.8s
response ok
25 reasoning_content has meaningful length (>5 chars) Think PASS 0.0s
>5 chars
26 Basic tool call: finish_reason=tool_calls, valid args Tools PASS 1.4s
valid
27 tool_choice=none suppresses tool calls Tools PASS 1.0s
no tool calls
28 Tool arguments are valid JSON dict Tools PASS 0.9s
valid JSON
29 Streaming: tool calls with finish_reason Tools PASS 1.1s
valid
30 Multi-tool: at least 1 tool call with 2 tools Tools PASS 1.6s
>=1 calls
31 Array param: todos is JSON array (not string) Tools PASS 0.9s
array
32 Nullable param: anyOf [string, null] does not crash Tools PASS 0.9s
no crash
33 No tools: normal text response Tools PASS 0.2s
text response
34 First request: cached_tokens=0 Cache PASS 0.3s
0
35 Second identical request: cached_tokens>0 Cache FAIL 0.3s
Expected: >0\nActual: FAIL: cached_tokens=0
36 Different prompt: cached_tokens=0 Cache PASS 0.2s
0
37 Streaming: cached_tokens>0 in usage chunk Cache FAIL 0.3s
Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage
38 Two simultaneous requests: both 200 Concurrent PASS 0.2s
200+200
39 Three simultaneous requests: all 200 Concurrent PASS 0.2s
all 200
40 Empty messages → 400 Error PASS 0.0s
400
41 Malformed JSON → 400 Error PASS 0.0s
400
42 Missing messages field → 400 Error PASS 0.0s
400
43 response_format json_object returns valid JSON Error PASS 0.2s
valid JSON
44 max_tokens=5 is respected Error PASS 0.1s
<=10 tokens
45 OPTIONS /v1/chat/completions → 200 (CORS) Error PASS 0.0s
200
46 developer role accepted (mapped to system) Error PASS 4.3s
valid response
47 enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
48 Streaming: enable_thinking=false disables thinking Kwargs PASS 0.1s
no_think
49 Default (no kwargs) retains thinking Kwargs PASS 1.4s
thinking
50 enable_thinking=false (2K tokens) returns content Kwargs PASS 0.1s
content present, no reasoning
51 enable_thinking=true explicitly keeps thinking Kwargs PASS 1.5s
thinking
52 Function name correctly extracted XMLTools PASS 1.5s
get_weather
53 Parameter values are correct string types XMLTools PASS 0.9s
strings
54 Mixed-type params (string+bool+int) parse correctly XMLTools PASS 1.3s
valid types
55 Nested object param survives XML parsing XMLTools PASS 1.5s
valid dict
56 tool_choice=required forces tool call XMLTools PASS 0.9s
tool_calls
57 tool_choice={function: get_time} calls correct function XMLTools PASS 1.7s
get_time
58 Tool call IDs are unique XMLTools PASS 1.6s
unique IDs
59 Streaming: XML tool call assembles valid JSON args XMLTools PASS 1.0s
valid
60 Streaming: array param is JSON array (not string) XMLTools PASS 0.9s
array
61 Tool call matches OpenAI schema (id, type, function.name, function.arguments) XMLTools PASS 1.1s
valid schema