| # | Test | Group | Status | Duration | Details |
|---|---|---|---|---|---|
| 1 | Binary exists at .build/release/afm | Preflight | PASS | 0.0s | file exists |
| 2 | Server reachable at http://127.0.0.1:9998 | Preflight | PASS | 0.0s | 200 OK |
| 3 | /v1/models contains model ID | Lifecycle | PASS | 0.0s | model in response |
| 4 | Basic completion returns content | Lifecycle | PASS | 3.6s | non-empty content or reasoning |
| 5 | Stop string '5' absent from output | Stop | PASS | 1.9s | no '5' in content |
| 6 | finish_reason is 'stop' with stop sequence | Stop | PASS | 0.0s | stop |
| 7 | Multi-word stop 'and' truncates correctly | Stop | PASS | 1.8s | no 'and' in output |
| 8 | Stop on newline produces single line | Stop | FAIL | 2.9s | Expected: single line\nActual: FAIL: multi-line output |
| 9 | Multiple stop sequences [7, 12] | Stop | PASS | 1.8s | neither found |
| 10 | Empty stop array is no-op | Stop | PASS | 4.4s | valid output |
| 11 | Streaming: stop string '5' absent | Stop | PASS | 1.8s | no '5' in stream |
| 12 | Stop sequence '3.' truncates list | Stop | PASS | 1.0s | no '3.' in output |
| 13 | Stop 'stopped' doesn't fire on 'stopping' | Stop | PASS | 2.2s | output produced |
| 14 | Stop 'llo' fires mid-word in 'hello' | Stop | PASS | 1.5s | no 'llo' |
| 15 | ChoiceLogprobs JSON schema valid | Logprobs | PASS | 0.1s | valid schema |
| 16 | top_logprobs count <= requested (5) | Logprobs | PASS | 0.1s | count valid |
| 17 | logprobs=false returns null | Logprobs | PASS | 0.1s | null logprobs |
| 18 | top_logprobs=99 returns 400 | Logprobs | PASS | 0.0s | 400 |
| 19 | Streaming logprobs present and valid | Logprobs | PASS | 0.1s | valid |
| 20 | top_logprobs=0 returns empty arrays | Logprobs | PASS | 0.1s | empty top_logprobs |
| 21 | reasoning_content present in response | Think | PASS | 0.0s | present |
| 22 | No <think> tags in content field | Think | PASS | 0.0s | clean content |
| 23 | Streaming: reasoning_content in deltas | Think | PASS | 0.9s | present |
| 24 | Stop sequence doesn't break think extraction | Think | PASS | 1.8s | response ok |
| 25 | reasoning_content has meaningful length (>5 chars) | Think | PASS | 0.0s | >5 chars |
| 26 | Basic tool call: finish_reason=tool_calls, valid args | Tools | PASS | 1.1s | valid |
| 27 | tool_choice=none suppresses tool calls | Tools | FAIL | 1.1s | Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'function': {'name': 'get_weather', 'arguments': '{"location":"Paris"}'}, 'type': 'function', 'id': 'call_woQH7pd6sAkUe0hXzGzgjz3a'}] |
| 28 | Tool arguments are valid JSON dict | Tools | PASS | 1.0s | valid JSON |
| 29 | Streaming: tool calls with finish_reason | Tools | PASS | 1.1s | valid |
| 30 | Multi-tool: at least 1 tool call with 2 tools | Tools | PASS | 1.6s | >=1 calls |
| 31 | Array param: todos is JSON array (not string) | Tools | PASS | 1.0s | array |
| 32 | Nullable param: anyOf [string, null] does not crash | Tools | PASS | 1.3s | no crash |
| 33 | No tools: normal text response | Tools | PASS | 0.2s | text response |
| 34 | First request: cached_tokens=0 | Cache | PASS | 0.3s | 0 |
| 35 | Second identical request: cached_tokens>0 | Cache | FAIL | 0.3s | Expected: >0\nActual: FAIL: cached_tokens=0 |
| 36 | Different prompt: cached_tokens=0 | Cache | PASS | 0.3s | 0 |
| 37 | Streaming: cached_tokens>0 in usage chunk | Cache | FAIL | 0.3s | Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage |
| 38 | Two simultaneous requests: both 200 | Concurrent | PASS | 0.2s | 200+200 |
| 39 | Three simultaneous requests: all 200 | Concurrent | PASS | 0.3s | all 200 |
| 40 | Empty messages → 400 | Error | PASS | 0.0s | 400 |
| 41 | Malformed JSON → 400 | Error | PASS | 0.0s | 400 |
| 42 | Missing messages field → 400 | Error | PASS | 0.0s | 400 |
| 43 | response_format json_object returns valid JSON | Error | PASS | 0.2s | valid JSON |
| 44 | max_tokens=5 is respected | Error | PASS | 0.1s | <=10 tokens |
| 45 | OPTIONS /v1/chat/completions → 200 (CORS) | Error | PASS | 0.0s | 200 |
| 46 | developer role accepted (mapped to system) | Error | PASS | 4.5s | valid response |
| 47 | enable_thinking=false disables thinking | Kwargs | PASS | 0.1s | no_think |
| 48 | Streaming: enable_thinking=false disables thinking | Kwargs | PASS | 0.1s | no_think |
| 49 | Default (no kwargs) retains thinking | Kwargs | FAIL | 1.8s | Expected: thinking\nActual: FAIL: state=empty |
| 50 | enable_thinking=false (2K tokens) returns content | Kwargs | PASS | 0.1s | content present, no reasoning |
| 51 | enable_thinking=true explicitly keeps thinking | Kwargs | FAIL | 1.8s | Expected: thinking\nActual: FAIL: state=empty |
| 52 | Function name correctly extracted | XMLTools | PASS | 1.1s | get_weather |
| 53 | Parameter values are correct string types | XMLTools | PASS | 1.4s | strings |
| 54 | Mixed-type params (string+bool+int) parse correctly | XMLTools | PASS | 1.4s | valid types |
| 55 | Nested object param survives XML parsing | XMLTools | PASS | 1.6s | valid dict |
| 56 | tool_choice=required forces tool call | XMLTools | PASS | 1.3s | tool_calls |
| 57 | tool_choice={function: get_time} calls correct function | XMLTools | PASS | 1.1s | get_time |
| 58 | Tool call IDs are unique | XMLTools | PASS | 1.7s | unique IDs |
| 59 | Streaming: XML tool call assembles valid JSON args | XMLTools | PASS | 1.0s | valid |
| 60 | Streaming: array param is JSON array (not string) | XMLTools | PASS | 1.3s | array |
| 61 | Tool call matches OpenAI schema (id, type, function.name, function.arguments) | XMLTools | PASS | 1.1s | valid schema |