| # | Test | Group | Status | Duration | Details |
|---|---|---|---|---|---|
| 1 | Binary exists at .build/release/afm | Preflight | PASS | 0.0s | file exists |
| 2 | Server reachable at http://127.0.0.1:9998 | Preflight | PASS | 0.0s | 200 OK |
| 3 | /v1/models contains model ID | Lifecycle | PASS | 0.0s | model in response |
| 4 | Basic completion returns content | Lifecycle | PASS | 1.3s | non-empty content or reasoning |
| 5 | Stop string '5' absent from output | Stop | PASS | 0.2s | no '5' in content |
| 6 | finish_reason is 'stop' with stop sequence | Stop | PASS | 0.0s | stop |
| 7 | Multi-word stop 'and' truncates correctly | Stop | PASS | 0.3s | no 'and' in output |
| 8 | Stop on newline produces single line | Stop | FAIL | 0.3s | Expected: single line\nActual: FAIL: multi-line output |
| 9 | Multiple stop sequences [7, 12] | Stop | PASS | 0.6s | neither found |
| 10 | Empty stop array is no-op | Stop | PASS | 0.2s | valid output |
| 11 | Streaming: stop string '5' absent | Stop | FAIL | 0.2s | Expected: no '5' in stream\nActual: FAIL: found '5' |
| 12 | Stop sequence '3.' truncates list | Stop | PASS | 0.3s | no '3.' in output |
| 13 | Stop 'stopped' doesn't fire on 'stopping' | Stop | PASS | 0.1s | output produced |
| 14 | Stop 'llo' fires mid-word in 'hello' | Stop | PASS | 0.1s | no 'llo' |
| 15 | ChoiceLogprobs JSON schema valid | Logprobs | PASS | 0.2s | valid schema |
| 16 | top_logprobs count <= requested (5) | Logprobs | PASS | 0.2s | count valid |
| 17 | logprobs=false returns null | Logprobs | PASS | 0.1s | null logprobs |
| 18 | top_logprobs=99 returns 400 | Logprobs | PASS | 0.0s | 400 |
| 19 | Streaming logprobs present and valid | Logprobs | PASS | 0.1s | valid |
| 20 | top_logprobs=0 returns empty arrays | Logprobs | PASS | 0.1s | empty top_logprobs |
| 21 | Think extraction (model lacks <think> support) | Think | SKIP | skip |
|
| 22 | Basic tool call: finish_reason=tool_calls, valid args | Tools | PASS | 0.7s | valid |
| 23 | tool_choice=none suppresses tool calls | Tools | FAIL | 0.7s | Expected: no tool calls\nActual: FAIL: finish_reason=tool_calls, tool_calls=[{'function': {'arguments': '{"location":"Paris","unit":"celsius"}', 'name': 'get_weather'}, 'type': 'function', 'id': 'call_YYMQ3Ua38byiUXXsdPXByFy0'}] |
| 24 | Tool arguments are valid JSON dict | Tools | PASS | 0.8s | valid JSON |
| 25 | Streaming: tool calls with finish_reason | Tools | PASS | 0.7s | valid |
| 26 | Multi-tool: at least 1 tool call with 2 tools | Tools | PASS | 0.9s | >=1 calls |
| 27 | Array param: todos is JSON array (not string) | Tools | PASS | 0.7s | array |
| 28 | Nullable param: anyOf [string, null] does not crash | Tools | PASS | 0.6s | no crash |
| 29 | No tools: normal text response | Tools | PASS | 0.2s | text response |
| 30 | First request: cached_tokens=0 | Cache | PASS | 0.4s | 0 |
| 31 | Second identical request: cached_tokens>0 | Cache | FAIL | 0.4s | Expected: >0\nActual: FAIL: cached_tokens=0 |
| 32 | Different prompt: cached_tokens=0 | Cache | PASS | 0.4s | 0 |
| 33 | Streaming: cached_tokens>0 in usage chunk | Cache | FAIL | 0.5s | Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage |
| 34 | Two simultaneous requests: both 200 | Concurrent | PASS | 0.2s | 200+200 |
| 35 | Three simultaneous requests: all 200 | Concurrent | PASS | 0.2s | all 200 |
| 36 | Empty messages → 400 | Error | PASS | 0.0s | 400 |
| 37 | Malformed JSON → 400 | Error | PASS | 0.0s | 400 |
| 38 | Missing messages field → 400 | Error | PASS | 0.0s | 400 |
| 39 | response_format json_object returns valid JSON | Error | PASS | 0.2s | valid JSON |
| 40 | max_tokens=5 is respected | Error | PASS | 0.2s | <=10 tokens |
| 41 | OPTIONS /v1/chat/completions → 200 (CORS) | Error | PASS | 0.0s | 200 |
| 42 | developer role accepted (mapped to system) | Error | PASS | 2.4s | valid response |
| 43 | chat_template_kwargs (model lacks thinking) | Kwargs | SKIP | skip |
|
| 44 | Function name correctly extracted | XMLTools | PASS | 0.8s | get_weather |
| 45 | Parameter values are correct string types | XMLTools | PASS | 0.8s | strings |
| 46 | Mixed-type params (string+bool+int) parse correctly | XMLTools | PASS | 0.9s | valid types |
| 47 | Nested object param survives XML parsing | XMLTools | PASS | 0.9s | valid dict |
| 48 | tool_choice=required forces tool call | XMLTools | PASS | 0.7s | tool_calls |
| 49 | tool_choice={function: get_time} calls correct function | XMLTools | PASS | 0.6s | get_time |
| 50 | Tool call IDs are unique | XMLTools | PASS | 0.9s | unique IDs |
| 51 | Streaming: XML tool call assembles valid JSON args | XMLTools | PASS | 0.8s | valid |
| 52 | Streaming: array param is JSON array (not string) | XMLTools | PASS | 0.8s | array |
| 53 | Tool call matches OpenAI schema (id, type, function.name, function.arguments) | XMLTools | PASS | 0.7s | valid schema |