| # | Test | Group | Status | Duration | Details |
|---|---|---|---|---|---|
| 1 | Binary exists at .build/release/afm | Preflight | PASS | 0.0s | file exists |
| 2 | Server reachable at http://127.0.0.1:9998 | Preflight | PASS | 0.0s | 200 OK |
| 3 | /v1/models contains model ID | Lifecycle | PASS | 0.0s | model in response |
| 4 | Basic completion returns content | Lifecycle | FAIL | 4.7s | Expected: non-empty content\nActual: FAIL: got '' |
| 5 | Stop string '5' absent from output | Stop | PASS | 1.9s | no '5' in content |
| 6 | finish_reason is 'stop' with stop sequence | Stop | PASS | 0.0s | stop |
| 7 | Multi-word stop 'and' truncates correctly | Stop | PASS | 1.9s | no 'and' in output |
| 8 | Stop on newline produces single line | Stop | FAIL | 0.5s | Expected: single line\nActual: FAIL: multi-line output |
| 9 | Multiple stop sequences [7, 12] | Stop | PASS | 1.8s | neither found |
| 10 | Empty stop array is no-op | Stop | FAIL | 0.2s | Expected: valid output\nActual: FAIL |
| 11 | Streaming: stop string '5' absent | Stop | PASS | 1.9s | no '5' in stream |
| 12 | Stop sequence '3.' truncates list | Stop | PASS | 1.0s | no '3.' in output |
| 13 | Stop 'stopped' doesn't fire on 'stopping' | Stop | FAIL | 0.3s | Expected: output produced\nActual: FAIL |
| 14 | Stop 'llo' fires mid-word in 'hello' | Stop | PASS | 0.3s | no 'llo' |
| 15 | ChoiceLogprobs JSON schema valid | Logprobs | PASS | 0.3s | valid schema |
| 16 | top_logprobs count <= requested (5) | Logprobs | PASS | 0.1s | count valid |
| 17 | logprobs=false returns null | Logprobs | PASS | 0.1s | null logprobs |
| 18 | top_logprobs=99 returns 400 | Logprobs | FAIL | 0.0s | Expected: 400\nActual: FAIL: HTTP 400000 |
| 19 | Streaming logprobs present and valid | Logprobs | PASS | 0.1s | valid |
| 20 | top_logprobs=0 returns empty arrays | Logprobs | PASS | 0.1s | empty top_logprobs |
| 21 | reasoning_content present in response | Think | PASS | 0.0s | present |
| 22 | No <think> tags in content field | Think | PASS | 0.0s | clean content |
| 23 | Streaming: reasoning_content in deltas | Think | PASS | 0.9s | present |
| 24 | Stop sequence doesn't break think extraction | Think | PASS | 1.8s | response ok |
| 25 | reasoning_content has meaningful length (>5 chars) | Think | PASS | 0.0s | >5 chars |
| 26 | Basic tool call: finish_reason=tool_calls, valid args | Tools | FAIL | 1.6s | Expected: valid\nActual: FAIL: finish_reason=stop, tool_calls=0 |
| 27 | tool_choice=none suppresses tool calls | Tools | PASS | 1.1s | no tool calls |
| 28 | Tool arguments are valid JSON dict | Tools | FAIL | 1.1s | Expected: valid JSON\nActual: FAIL: no tool calls |
| 29 | Streaming: tool calls with finish_reason | Tools | FAIL | 1.2s | Expected: valid\nActual: FAIL: no tool_calls in stream |
| 30 | Multi-tool: at least 1 tool call with 2 tools | Tools | FAIL | 1.3s | Expected: >=1 calls\nActual: FAIL: 0 tool calls |
| 31 | No tools: normal text response | Tools | PASS | 0.2s | text response |
| 32 | First request: cached_tokens=0 | Cache | PASS | 0.3s | 0 |
| 33 | Second identical request: cached_tokens>0 | Cache | FAIL | 0.3s | Expected: >0\nActual: FAIL: cached_tokens=0 |
| 34 | Different prompt: cached_tokens=0 | Cache | PASS | 0.2s | 0 |
| 35 | Streaming: cached_tokens>0 in usage chunk | Cache | FAIL | 0.3s | Expected: >0\nActual: FAIL: no cached_tokens>0 in stream usage |
| 36 | Two simultaneous requests: both 200 | Concurrent | PASS | 0.2s | 200+200 |
| 37 | Three simultaneous requests: all 200 | Concurrent | PASS | 0.3s | all 200 |
| 38 | Empty messages → 400 | Error | FAIL | 0.0s | Expected: 400\nActual: FAIL: HTTP 400000 |
| 39 | Malformed JSON → 400 | Error | FAIL | 0.0s | Expected: 400\nActual: FAIL: HTTP 400000 |
| 40 | Missing messages field → 400 | Error | FAIL | 0.0s | Expected: 400\nActual: FAIL: HTTP 400000 |
| 41 | response_format json_object returns valid JSON | Error | FAIL | 0.5s | Expected: valid JSON\nActual: FAIL: not valid JSON: |
| 42 | max_tokens=5 is respected | Error | PASS | 0.1s | <=10 tokens |
| 43 | OPTIONS /v1/chat/completions → 200 (CORS) | Error | PASS | 0.0s | 200 |
| 44 | developer role accepted (mapped to system) | Error | FAIL | 0.3s | Expected: valid response\nActual: FAIL |
| 45 | enable_thinking=false disables thinking | Kwargs | PASS | 0.1s | no_think |
| 46 | Streaming: enable_thinking=false disables thinking | Kwargs | PASS | 0.1s | no_think |
| 47 | Default (no kwargs) retains thinking | Kwargs | PASS | 1.5s | thinking |
| 48 | enable_thinking=false (2K tokens) returns content | Kwargs | PASS | 0.1s | content present, no reasoning |
| 49 | enable_thinking=true explicitly keeps thinking | Kwargs | PASS | 1.6s | thinking |
| 50 | TTFT < 5s | Perf | FAIL | 0.1s | Expected: <5000ms\nActual: FAIL: ms |
| 51 | tok/s > 1 (99.4 tok/s) | Perf | PASS | 0.5s | >1 |
| 52 | Long context (~2K tokens) no crash | Perf | FAIL | 0.7s | Expected: valid response\nActual: FAIL: empty |
| 53 | Very long context (~4K tokens) no NaN/garbage | Perf | FAIL | 1.3s | Expected: clean output\nActual: FAIL: empty content |