How we test

How a project earns its numbers

Every project goes through the same three stages, and each page tells you which stage it's in.

1. Draft

We've written the steps and the agent instructions, but haven't run them on our computers yet. You'll see a yellow note at the top and no speed figures.

2. Testing

We're running it on one or more of our machines and fixing whatever the instructions got wrong.

3. Published

Someone followed the written steps from scratch and it worked. The results file is saved in our public repository, and the page shows them along with the machine and the date.

What we record

• The exact model and version, and the software versions used
• Which of our computers ran it
• Speed (for chat, words per second; for transcription, minutes of audio per minute) and memory used
• A simple quality check written for that project

What we won't do

• Show estimated speeds from calculators as if they were measured
• Present a rented cloud computer's results as a home computer's
• Leave old numbers up without a note once the software has changed a lot