Epoch AI Launches FrontierMath AI Benchmark to Take a look at Capabilities of AI Fashions

Epoch AI, a California-based analysis institute launched a brand new synthetic intelligence (AI) benchmark final week. Dubbed FrontierMath, the brand new AI benchmark exams giant language fashions (LLMs) on their functionality of reseasoning and mathematical problem-solving. The AI agency claims that current math benchmarks are usually not very helpful as a result of components like knowledge contamination and AI fashions scoring very excessive scores on them. Epoch AI claims that even the main LLMs have scored lower than two % on the brand new benchmark.

Epoch AI Launches FrontierMath Benchmark

In a publish on X (previously generally known as Twitter), the AI agency defined that it collaborated with greater than 60 mathematicians to create a whole lot of origins and unpublished math issues. Epoch AI claims that these questions would take even mathematicians hours to unravel. The rationale behind growing the brand new benchmark was cited as the constraints with current benchmarks akin to GSM8K and MATH, the place AI fashions typically rating a excessive level.

The corporate claimed that the excessive scores achieved by LLMs are largely as a result of knowledge contamination. This implies the questions someway have been already fed into the AI fashions, leading to them simply fixing the questions.

FrontierMath solves the issue by together with new issues which can be distinctive and haven’t been printed anyplace, mitigating the dangers related to knowledge contamination. Additional, the benchmark consists of a variety of questions together with computationally intensive issues in quantity concept, actual evaluation, and algebraic geometry, in addition to matters akin to Zermelo–Fraenkel set concept. The AI agency says all of the questions are “guess proof”, that means they can’t be solved by accident with out robust reasoning.

Epoch AI highlighted that to measure AI’s aptitude, benchmarks must be created on artistic problem-solving the place the AI has to take care of reasoning over a number of steps. Notably, many trade veterans imagine that the present benchmarks are usually not enough to appropriately measure how superior an AI mannequin is.

Responding to the brand new benchmark in a publish, Noam Brown, an OpenAI researcher who was behind the corporate’s o1 mannequin welcomed the brand new benchmark and stated, “I like seeing a brand new eval with such low cross charges for frontier fashions.”