AI Eval Costs Now a Compute Bottleneck

AI evaluations have reached a cost threshold that is impacting the ability of entities to perform them effectively. Notably, the Holistic Agent Leaderboard incurred costs of around $40,000 for specific evaluations.

The increasing costs of AI evaluations are challenging existing computational infrastructures and altering the dynamics of who can afford such evaluations.

Researchers and organizations involved in AI development may face significant challenges due to these rising evaluation costs, potentially limiting innovation and advancements.

There is uncertainty in how these costs will influence the future accessibility and equity in AI research and the development of new models.