How we tested
- Each test sent the same request three ways: with Deft, with the best free skill we found for this job, and with no skill at all.
- The judges were AI agents, not people. They saw the three results as A, B and C with every skill name removed, and scored five qualities out of 10 each, 50 in all.
- The quality scores at the top are averaged over the tests and rounded.
- How we picked the free skill: our study's pick to test against for building and changing code (rated 4 of 5, about 110,000 GitHub stars), run with its build and review commands and every skill they name.
- The businesses are made up. Each result is that version’s own work, unedited, from the request shown.
The work
Making an old app safe to change
The requestI run Pressline, a laundry pickup and delivery service in Rotterdam. The app (server.js) was made by a freelancer two years ago, and since then I've been adding things with ChatGPT. Every time I add something, something else breaks. Last week I added express service and now the loyalty points are wrong. Next I want to add stain treatment, but I'm scared to touch it. Can you help me make changes safe from now on? I can't afford a rebuild.
The blind judge’s reasons
Deft did what was asked: 43 checks that pin today's app, quirks included, git first, about ten lines changed, and price questions left to the owner. The free skill wrote a strong set of checks, but for the behaviour it wanted, then changed customer prices without asking. The no-skill version fixed the most but was the riskiest: no git, a big change and a long reply.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| Safe to change now | 9 | 7 | 5 |
|---|
| Nothing broke | 7 | 4 | 3 |
|---|
| Safe process | 9 | 7 | 4 |
|---|
| Honest | 9 | 6 | 6 |
|---|
| The reply | 8 | 7 | 5 |
|---|
| Total | 42 | 31 | 23 |
|---|
Reviewing a code change
The requestA freelancer sent me this change for Greenhollow, my veg box subscription business (it's changes.patch). It adds holiday pauses and a refer-a-friend credit. He says all the tests pass. Is it safe to merge? I can't really read code.
The blind judge’s reasons
All three said "not yet", found the three planted bugs and invented no problem. Deft was the only one to prove each bug with failing checks left in the project and to show the test counts before and after, and it passed the harmless rename; it didn't say exactly what the freelancer's own tests miss. The free skill gave exact line numbers and showed what his tests miss, but marked the harmless rename as required, and the no-skill version had the clearest reply but showed no test runs and also blocked the merge on the rename.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| Found the real problems | 9 | 9 | 8 |
|---|
| Proved it | 10 | 8 | 7 |
|---|
| Right verdict | 9 | 7 | 7 |
|---|
| Honest | 9 | 8 | 8 |
|---|
| The reply | 8 | 7 | 9 |
|---|
| Total | 45 | 39 | 39 |
|---|
Adding discount codes
The requestI run Inkwell Print Co., a small print shop in Budapest. The prices on our website come from this little Node program a friend wrote years ago (src/pricing.js), and nobody dares to touch it. I want to add discount codes. Customers type a code in the quote form, and the website already sends it as `code` in the order. For now: AUTUMN10 is 10% off until 31 October 2026, and WELCOME5 is €5 off a first order until 31 December 2026. Codes must stop working after their date. Please add this without changing any of the prices we have now: customers and my accountant check them.
The blind judge’s reasons
Deft built the best safety net: checks that pin today's prices before any change, tested with deliberate breaks, then a fresh review. Its reply was too technical and gave no test command. The free skill built the same correct feature with the clearest owner reply, but its checks missed two edges and there was no review, and the no-skill version left no tests at all.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| The change works | 9 | 9 | 8 |
|---|
| Nothing broke | 9 | 8 | 6 |
|---|
| Safe process | 8 | 6 | 3 |
|---|
| Honest | 8 | 7 | 6 |
|---|
| The reply | 6 | 8 | 8 |
|---|
| Total | 40 | 38 | 31 |
|---|