The price is wrong: AI cost calculation has to consider task completion rates, not just token costs

REG AD AI and ML Cheap can be expensive When it comes to AI services, you don't necessarily get what you pay for.It turns out that AI models with expensive tokens may cost less than models with cheap tokens for particular tasks. And the tooling attached to those models can have a significant effect on cost and output quality.Databricks, which sells data analytics software and services, recently devised an internal coding benchmark to assess the tradeoff between price and performance using various AI models. REG AD Matei Zaharia, CTO of Databricks and associate professor of computer science at UC Berkeley, said the company undertook the evaluation because models are often tuned to existing benchmark tests like SWE-Bench – which is "broken," according to OpenAI. REG AD Databricks devised its