0

Abacus.AI

AI platform startup offering enterprise LLM tooling; co-creator with Yann LeCun's NYU group of the LiveBench contamination-resistant LLM benchmark.

Visit
Type
startup
HQ
San Francisco, CA, USA
Founded
2019
Funding
Series C ~$100M (2021–2022)
Website
abacus.ai
GitHub
github.com/abacusai
Twitter
twitter.com/abacusai

Cite

Notes

Only stored in your browser.

Evals
7
Tools
0
Models
0
Papers
0
Boards
1
People
3

Leaderboards

1
LiveBenchContinuously refreshed, contamination-resistant benchmark covering math, reasoning, coding, language, data analysis, and instruction-following with automatic objective scoring.

Evals

7

LiveBench - Language

Language sub-leaderboard of LiveBench: typo correction, plot unscrambling, NYT Connections.

LiveBench - Coding

Coding sub-leaderboard of LiveBench: LiveCodeBench generation + code-completion tasks.

LiveBench - Instruction Following

Instruction-following sub-leaderboard of LiveBench: paraphrase + story-generation tasks.

LiveBench - Math

Math sub-leaderboard of LiveBench: AMPS-Hard, math competitions, integrals-with-game, and Olympiad tasks.

LiveBench - Data Analysis

Data-analysis sub-leaderboard of LiveBench: consecutive-events, table-join, and table-reformat tasks.

LiveBench - Reasoning

Reasoning sub-leaderboard of LiveBench: theory-of-mind, zebra puzzle, spatial, and logic-with-navigation tasks.

LiveBench: A Challenging, Contamination-Free LLM Benchmark

LiveBench is a benchmark designed with test set contamination and objective evaluation in mind by releasing new questions regularly, as well as having questions based on recently-released datasets. Each question has verifiable, objective ground-truth answers, allowing hard questi

ActiveKnowledge