Conversation
Source commit: trylitmus/litmus@25274156c014dca5bee2a4dd236a2b947dbe6fa2
|
| Filename | Overview |
|---|---|
| contract/openapi.yaml | Adds $sessionend event type and 8 new top-level Event fields (model, provider, token counts, latency, cost); the Python SDK currently sends model via metadata rather than as a top-level field, creating a contract-SDK drift. |
Sequence Diagram
sequenceDiagram
participant App
participant LitmusClient
participant Consumer
participant IngestAPI
App->>LitmusClient: generation(session_id, model_name)
LitmusClient->>LitmusClient: build msg, model stored in metadata
LitmusClient->>Consumer: enqueue $generation event
App->>LitmusClient: track(event_type, metadata with token counts)
Note over LitmusClient: New fields like input_tokens<br/>not yet first-class params
LitmusClient->>Consumer: enqueue event with token data in metadata
Consumer->>IngestAPI: POST /v1/events
Note over IngestAPI: Contract expects top-level<br/>model, input_tokens, cost etc.
IngestAPI-->>Consumer: 202 Accepted
Prompt To Fix All With AI
This is a comment left during a code review.
Path: contract/openapi.yaml
Line: 138-162
Comment:
**SDK sends `model` (and new fields) via `metadata`, not as top-level fields**
The contract now defines `model`, `provider`, `input_tokens`, `output_tokens`, `total_tokens`, `duration_ms`, `ttft_ms`, and `cost` as top-level `Event` properties, but the Python SDK never serializes them at the top level. `client.py`'s `track()` signature has no parameters for any of these fields, and `Feature.generation()` explicitly puts `model` into `metadata` — confirmed by the test at `tests/test_client.py:251` which asserts `gen_event["metadata"]["model"] == "gpt-4o"`. If the server-side is updated to read `model` (and the other new fields) from the event's top-level object, the SDK will silently fail to deliver this data. The SDK's `track()` method, `Generation.event()`, and the event-building block in the constructor all need to be updated to forward these fields at the wire level.
How can I resolve this? If you propose a fix, please make it concise.
---
This is a comment left during a code review.
Path: contract/openapi.yaml
Line: 144-161
Comment:
**Missing `minimum: 0` constraints on numeric fields**
`input_tokens`, `output_tokens`, `total_tokens`, `duration_ms`, `ttft_ms`, and `cost` are all non-negative by definition, but the schema allows any integer/number value. Adding `minimum: 0` closes the door on accidental negative submissions that would corrupt reporting.
```suggestion
input_tokens:
type: integer
minimum: 0
description: Number of input tokens consumed.
output_tokens:
type: integer
minimum: 0
description: Number of output tokens produced.
total_tokens:
type: integer
minimum: 0
description: Total tokens (input + output).
duration_ms:
type: integer
minimum: 0
description: End-to-end generation latency in milliseconds.
ttft_ms:
type: integer
minimum: 0
description: Time to first token in milliseconds.
cost:
type: number
format: double
minimum: 0
description: Generation cost in USD.
```
How can I resolve this? If you propose a fix, please make it concise.Reviews (1): Last reviewed commit: "chore: sync OpenAPI contract from monore..." | Re-trigger Greptile
| model: | ||
| type: string | ||
| description: Model identifier (e.g. "claude-sonnet-4-20250514", "gpt-4o"). Typically set on $generation events. | ||
| provider: | ||
| type: string | ||
| description: LLM provider (e.g. "anthropic", "openai", "openrouter"). | ||
| input_tokens: | ||
| type: integer | ||
| description: Number of input tokens consumed. | ||
| output_tokens: | ||
| type: integer | ||
| description: Number of output tokens produced. | ||
| total_tokens: | ||
| type: integer | ||
| description: Total tokens (input + output). | ||
| duration_ms: | ||
| type: integer | ||
| description: End-to-end generation latency in milliseconds. | ||
| ttft_ms: | ||
| type: integer | ||
| description: Time to first token in milliseconds. | ||
| cost: | ||
| type: number | ||
| format: double | ||
| description: Generation cost in USD. |
There was a problem hiding this comment.
SDK sends
model (and new fields) via metadata, not as top-level fields
The contract now defines model, provider, input_tokens, output_tokens, total_tokens, duration_ms, ttft_ms, and cost as top-level Event properties, but the Python SDK never serializes them at the top level. client.py's track() signature has no parameters for any of these fields, and Feature.generation() explicitly puts model into metadata — confirmed by the test at tests/test_client.py:251 which asserts gen_event["metadata"]["model"] == "gpt-4o". If the server-side is updated to read model (and the other new fields) from the event's top-level object, the SDK will silently fail to deliver this data. The SDK's track() method, Generation.event(), and the event-building block in the constructor all need to be updated to forward these fields at the wire level.
Prompt To Fix With AI
This is a comment left during a code review.
Path: contract/openapi.yaml
Line: 138-162
Comment:
**SDK sends `model` (and new fields) via `metadata`, not as top-level fields**
The contract now defines `model`, `provider`, `input_tokens`, `output_tokens`, `total_tokens`, `duration_ms`, `ttft_ms`, and `cost` as top-level `Event` properties, but the Python SDK never serializes them at the top level. `client.py`'s `track()` signature has no parameters for any of these fields, and `Feature.generation()` explicitly puts `model` into `metadata` — confirmed by the test at `tests/test_client.py:251` which asserts `gen_event["metadata"]["model"] == "gpt-4o"`. If the server-side is updated to read `model` (and the other new fields) from the event's top-level object, the SDK will silently fail to deliver this data. The SDK's `track()` method, `Generation.event()`, and the event-building block in the constructor all need to be updated to forward these fields at the wire level.
How can I resolve this? If you propose a fix, please make it concise.| input_tokens: | ||
| type: integer | ||
| description: Number of input tokens consumed. | ||
| output_tokens: | ||
| type: integer | ||
| description: Number of output tokens produced. | ||
| total_tokens: | ||
| type: integer | ||
| description: Total tokens (input + output). | ||
| duration_ms: | ||
| type: integer | ||
| description: End-to-end generation latency in milliseconds. | ||
| ttft_ms: | ||
| type: integer | ||
| description: Time to first token in milliseconds. | ||
| cost: | ||
| type: number | ||
| format: double |
There was a problem hiding this comment.
Missing
minimum: 0 constraints on numeric fields
input_tokens, output_tokens, total_tokens, duration_ms, ttft_ms, and cost are all non-negative by definition, but the schema allows any integer/number value. Adding minimum: 0 closes the door on accidental negative submissions that would corrupt reporting.
| input_tokens: | |
| type: integer | |
| description: Number of input tokens consumed. | |
| output_tokens: | |
| type: integer | |
| description: Number of output tokens produced. | |
| total_tokens: | |
| type: integer | |
| description: Total tokens (input + output). | |
| duration_ms: | |
| type: integer | |
| description: End-to-end generation latency in milliseconds. | |
| ttft_ms: | |
| type: integer | |
| description: Time to first token in milliseconds. | |
| cost: | |
| type: number | |
| format: double | |
| input_tokens: | |
| type: integer | |
| minimum: 0 | |
| description: Number of input tokens consumed. | |
| output_tokens: | |
| type: integer | |
| minimum: 0 | |
| description: Number of output tokens produced. | |
| total_tokens: | |
| type: integer | |
| minimum: 0 | |
| description: Total tokens (input + output). | |
| duration_ms: | |
| type: integer | |
| minimum: 0 | |
| description: End-to-end generation latency in milliseconds. | |
| ttft_ms: | |
| type: integer | |
| minimum: 0 | |
| description: Time to first token in milliseconds. | |
| cost: | |
| type: number | |
| format: double | |
| minimum: 0 | |
| description: Generation cost in USD. |
Prompt To Fix With AI
This is a comment left during a code review.
Path: contract/openapi.yaml
Line: 144-161
Comment:
**Missing `minimum: 0` constraints on numeric fields**
`input_tokens`, `output_tokens`, `total_tokens`, `duration_ms`, `ttft_ms`, and `cost` are all non-negative by definition, but the schema allows any integer/number value. Adding `minimum: 0` closes the door on accidental negative submissions that would corrupt reporting.
```suggestion
input_tokens:
type: integer
minimum: 0
description: Number of input tokens consumed.
output_tokens:
type: integer
minimum: 0
description: Number of output tokens produced.
total_tokens:
type: integer
minimum: 0
description: Total tokens (input + output).
duration_ms:
type: integer
minimum: 0
description: End-to-end generation latency in milliseconds.
ttft_ms:
type: integer
minimum: 0
description: Time to first token in milliseconds.
cost:
type: number
format: double
minimum: 0
description: Generation cost in USD.
```
How can I resolve this? If you propose a fix, please make it concise.
Automated sync of `contract/openapi.yaml` from the monorepo.
Source commit: trylitmus/litmus@25274156c014dca5bee2a4dd236a2b947dbe6fa2
Review the diff, then merge. If the SDK has codegen (e.g. `pnpm generate`),
run it after merging to update generated types.