jessicayjung

ARC-style 3D puzzle (wires) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

Text-to-CAD agent training env (free 11-task sample of 90): model exact mechanical parts in CadQuery from a spec or a drawing, graded on geometry. ...

ARC-style 3D puzzle (pile) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (lifts) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

Finance-ops spreadsheet agent training env (free 19-task sample of 248): xlsx analyst tasks from AR aging to month-end and group close, graded cell...

ARC-style 3D puzzle (scales) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (gears) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (beam) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (sort) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

Real-time rhythm eval frontier agents fail (0/9 Opus, Sonnet, Haiku; best 4 of 86 notes): play a full song from pixels

ARC-style 3D puzzle (roll) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (garden) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

ARC-style 3D puzzle (domino) frontier agents rarely solve (Opus 4/30, Sonnet 2/30, Haiku 0/30 held-out): find the unstated rule from pixels

Computer-use eval frontier agents fail (0/3 each: Opus, Sonnet, Haiku): cross a 3D maze by reading real clues past decoys, no wrong doorway