Leaderboard method

Counted on your computer. Shared only if you say yes.

The Model Leaderboard is built from a small daily summary that some Cinderpaw installations send. Sharing is off unless you turn it on. It is a box on the last setup screen that starts unticked, a one-time question on the home screen if you had Cinderpaw before this existed, or a switch in Settings, Privacy. Until you say yes, nothing is sent.

What is counted

Once a day, for each model that did tasks on a day that has finished, Cinderpaw sends one set of counts. Days from before you said yes are never sent. The set holds:

• The model and its provider. For a big cloud service, that is the service’s name and the public model name. A model on your own computer or your home network is named only if it is a known model family. Anything else is sent as local-other. A model on a server you set up yourself is sent as custom, with its name only if it is a known model family, and never the server’s address. A file name or a server name can carry a person’s name. One exception: a model you trained yourself on a cloud service, such as Together or Hugging Face, is sent by the name that service gave it, and that name can include your account name. It shows on the leaderboard only if 20 installations use it.
• How many tasks the model did, and how many it only helped with. A task that used several models counts once, for the model that did the most of the work.
• How each task ended: finished, got stuck, ran out of time, hit a limit, was stopped, gave no answer, or ended with an error.
• What kind of error it was, when there was one: network, sign-in, billing, rate limit, context too long, provider, tool or other.
• How many 👍 and 👎 you gave.
• How many tokens went in and out, and how many were reused from a cache when the provider says so. Tokens are the small pieces of text a model reads and writes.
• The cost in dollars, with the reported part and the estimated part kept apart.
• How long tasks took, as counts in eight ranges.
• How many tool calls, fallbacks and retries there were.
• Whether memory recall and self-improvement were on.
• The day, the Cinderpaw version, your operating system (windows, macos, linux or other) and a random code. The code has its own section below.

What is never sent

Never your messages. Never file names. Never links. Never keys. Never session, task or message ids. Never exact times. Only counts leave your computer, and nothing in them says what a task was about.

The monthly code

Each summary carries a random code of 32 letters and numbers, made on your computer. It changes every calendar month. It lets me count how many installations used a model in a month, and it stops one day from being counted twice. It cannot follow you from one month to the next, and neither can I. The leaderboard starts over each month for the same reason. The code is saved in ~/.cinderpaw/usage-code.json. Turn sharing off and that file is deleted.

Why 20 installations and 100 tasks

A model is listed only when at least 20 different installations have used it, and it has done at least 100 tasks, in the same month. With fewer, one person’s habits would be the number, and that would say more about the person than about the model.

Every rate also shows how many tasks it rests on, and a ± that says how sure it is. 92% (±4) of 140 means 92% of 140 tasks, give or take 4 points. The ± is a 95% range. It gets smaller as the tasks add up.

What 👍 means and does not mean

The 👍 number is the share of rated tasks that got a 👍. Many tasks are never rated, so the “of” number beside it is how many were. It shows how happy people were. It is not proof that the task was done right: a person can like a wrong answer, and skip a good one.

Done is the same kind of number. It means Cinderpaw finished the task without getting stuck, running out of time or hitting an error. It does not check that the result was right.

Why times are ranges

Exact times are never sent. On your computer each task is put in one of eight ranges: under 5 s, 5-15 s, 15-30 s, 30-60 s, 1-2 min, 2-5 min, 5-15 min, over 15 min. Only how many tasks fell in each range is sent. The Usual time column shows the range that holds the middle task.

Estimated vs reported cost

Some providers, such as OpenRouter, tell Cinderpaw what each call cost. That is reported cost. Others do not, so Cinderpaw works it out from the tokens used and a rough price for the model. That is an estimate, and it is kept apart from the reported cost. In the table, a cost marked (est.) includes an estimate. Models on your own computer, your home network or a server you set up yourself count as $0, because there is no price to look up.

How to check your own data

• Every summary is written to ~/.cinderpaw/telemetry-sent.jsonl before it is sent. It is a plain text file with one line for each day. On Windows, look for the .cinderpaw folder inside your user folder.
• In Cinderpaw, open Settings, then Privacy, and press “See what was sent”.
• The switch is in the same place. Turn it off and sending stops right away. The code file is deleted.
• In Models, the Leaderboard tab shows your own numbers for this month. They stay on your computer, whether sharing is on or off.

This is promise 4 in my promises. The numbers themselves are on the leaderboard.