The 15% number is based on two runs from the ./experiment folder, on an instance from the SWE Bench Pro benchmark. I picked two trajectory logs, from the pro_pilot_openlibrary_haiku_v2 folder, that I consider to be representative of the token savings one can expect. There's huge variance just between individual runs of the same model on the same instance, though, so it's somewhat difficult (and expensive, at API rates) to cut through the noise. But I ran over 80 trials, and I believe that number is accurate.
And I'm publishing the "token science" research in the repo itself, so anyone who's curious about it can review the logs and inspect the python scripts that I used. And if someone out there has the time, inclination, and money, to run some of those experiments themselves, it'd be great to get consensus on some of those results.
And if you'd like to share feedback directly, rather than publicly, my email is on my profile page.
got an hn acct just to comment here -- this tool is amazing. the schema is brilliant. been using it since february and eager to try out the new version. based on my experience i know it's token-efficient, but curious how you determined that ~15% token reduction?
Here's the original blog post from last December: https://ebcode.com/blog/2025/12/17/sourceminder/
The 15% number is based on two runs from the ./experiment folder, on an instance from the SWE Bench Pro benchmark. I picked two trajectory logs, from the pro_pilot_openlibrary_haiku_v2 folder, that I consider to be representative of the token savings one can expect. There's huge variance just between individual runs of the same model on the same instance, though, so it's somewhat difficult (and expensive, at API rates) to cut through the noise. But I ran over 80 trials, and I believe that number is accurate.
And I'm publishing the "token science" research in the repo itself, so anyone who's curious about it can review the logs and inspect the python scripts that I used. And if someone out there has the time, inclination, and money, to run some of those experiments themselves, it'd be great to get consensus on some of those results.
And if you'd like to share feedback directly, rather than publicly, my email is on my profile page.
got an hn acct just to comment here -- this tool is amazing. the schema is brilliant. been using it since february and eager to try out the new version. based on my experience i know it's token-efficient, but curious how you determined that ~15% token reduction?