Scores and feedback

Every response carries x-vatan-request-id, generated before the request is served so it exists even on a streamed call. Without an id to refer to, feedback can only ever be aggregated and never traced back to the answer that earned it.

curl https://api.vatan.one/v1/feedback \
  -H "Authorization: Bearer $VATAN_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "request_id": "the x-vatan-request-id from the response",
    "value": true,
    "comment": "exactly right"
  }'

Authenticated with your gateway key, because whatever is scoring is your application: a thumbs-up in your UI, or a grader running over yesterday's traffic. Neither has anybody logged into our dashboard.

A thumbs-up is stored as 1 and a thumbs-down as 0, so everything averages and "what fraction were useful" is the same query as "what did the grader say". Pass "source": "eval" for an automated score; one request can carry both, and them disagreeing is the interesting case.

Scoring the same name twice corrects rather than duplicates. People change their minds, and averaging a thumbs-down with the thumbs-up that replaced it is simply wrong.