glass.bio blog

Evals update: Ox Alpha outclasses DeepSeek Flash

Model performance by cost

Overall, Ox Alpha slightly outperforms DeepSeek, scoring 6.46 compared with 6.01. Like the other models, Ox performs worse on blinded examples. Firecrawl has the biggest effect on these runs, raising the blinded score from 6.13 to 6.70. Giving Ox access to OpenAI web search does not improve performance, but the model rarely uses the tool in blinded runs. GPT 5.6 Sol still performs substantially better than Ox and DeepSeek. Ox Alpha was free during these experiments, so we plot it at $0.01 on the logarithmic axis. We will update the graph when pricing is available next week.

Average rubric score across 25 cases against cost per 10 questions for benchmark runs.5.05.56.06.57.07.58.08.59.09.5$0.005$0.01$0.02$0.05$0.10$0.20$0.50$1$2$5$10$20Cost per 10 questionsAverage rubric score across 25 casesDeepSeekDeepSeek +Firecrawl searchDeepSeek +web searchGPT 5.6 SolGPT 5.6 Sol +web searchGPT 5.6 Sol +Firecrawl searchOx AlphaOx Alpha +Firecrawl searchOx Alpha +web search
Non-blinded ×Blinded

Does Firecrawl help beyond paper recognition?

When neither run recognizes the paper, Firecrawl does not improve the score. When both runs recognize it, Firecrawl adds 0.57 points. This suggests that the search results can provide useful evidence beyond identifying the paper.

Firecrawl score gain for Ox Alpha by paper recognitionFirecrawl adds 0.44 points across all matched attempts. It adds no points when neither run recognizes the paper, 0.83 when only Firecrawl recognizes it, and 0.57 when both runs recognize it. Lines show case-bootstrap 95% intervals.-1.00-0.50+0.00+0.50+1.00+1.50All matched attemptsn=139+0.44Never recognizedn=30-0.03Recognized only with Firecrawln=52+0.83Recognized with and without Firecrawln=52+0.57Paired score gain with Firecrawl