How we tested
- Each test sent the same request three ways: with Deft, with the best free skill we found for this job, and with no skill at all.
- The judges were AI agents, not people. They saw the three results as A, B and C with every skill name removed, and scored five qualities out of 10 each, 50 in all.
- The quality scores at the top are averaged over the tests and rounded.
- How we picked the free skill: the top-rated free security skill in our study that works in both Claude Code and Codex and fixes the code rather than only reporting; run with all its reference files.
- The businesses are made up. Each result is that version’s own work, unedited, from the request shown.
The work
An API with member logins
The requestThis is the API for our community tool library, Toolshed Commons, in Edinburgh. About 140 members move over from our spreadsheet in November. A volunteer built it with ChatGPT and the tests pass. Is it safe to launch? Please fix what needs fixing.
The blind judge’s reasons
Deft found, rated and fixed every planted hole with tests, and showed before and after numbers. Its miss: no advice to reset everyone's password, and no note that everyone gets logged out. The no-skill version's code was as strong, but its reply gave no severities and no re-check, and the free skill added no tests and gave no severities.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| Found | 9 | 6 | 8 |
|---|
| Fixed | 10 | 7 | 9 |
|---|
| Proved and honest | 9 | 4 | 5 |
|---|
| Owner can act | 9 | 5 | 7 |
|---|
| The reply | 8 | 7 | 7 |
|---|
| Total | 45 | 29 | 36 |
|---|
A kids’ club app
The requestI'm the volunteer secretary of Riverside Juniors, a kids' football club in Leeds. I made our club app with Bolt (React and Firebase): parents add their children with their medical details, and coaches see the squads. A dad who works in IT told me "your keys are public, anyone can see them in the browser". Is that bad? What should I do? Please fix what you can.
The blind judge’s reasons
Deft came first because it is the only one that proves and admits: tight rules that keep every screen working, a scan before and after, and a plain note that the rules were not run. The free skill found the most and rated it well, but reported no re-check in a 2,400-word reply. The no-skill version's code was about as good, but it gave no severities and no re-check.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| Found | 8 | 9 | 6 |
|---|
| Fixed | 9 | 8 | 8 |
|---|
| Proved and honest | 9 | 5 | 4 |
|---|
| Owner can act | 9 | 8 | 8 |
|---|
| The reply | 8 | 5 | 7 |
|---|
| Total | 43 | 35 | 33 |
|---|
A web app with paid reports
The requestMy app Beamline lets home inspectors send reports to house buyers, who pay €49 to unlock the full report. I built it with v0 and Cursor and we launch on 1 December. Can you check it's secure and fix what you can? I'm not a developer, so please tell me what I need to do myself.
The blind judge’s reasons
Deft found and fixed every planted hole, rated and re-checked each one and put the breach warning first; it rated the AI endpoint Critical instead of High and never said the keys stay in the code history. The no-skill version made careful fixes with the clearest owner steps but gave no severities. The free skill found everything with correct severities but wrote about 2,900 words of code for a non-developer and showed no re-check.
The judge’s scores, out of 10 each | Quality | Deft | Free skill | No skill |
|---|
| Found | 9 | 9 | 7 |
|---|
| Fixed | 9 | 8 | 8 |
|---|
| Proved and honest | 8 | 5 | 6 |
|---|
| Owner can act | 9 | 8 | 9 |
|---|
| The reply | 8 | 6 | 8 |
|---|
| Total | 43 | 36 | 38 |
|---|