AI Tools Surpass Lawyers in Legal Research Accuracy, Vals Report Finds

A new Vals AI report shows tools like Alexi, Counsel Stack, Midpage, and ChatGPT outperform lawyers in legal research accuracy and authoritativeness.

Key points:

  • AI tools scored higher than lawyers in a 200-question legal research evaluation.
  • Alexi, Counsel Stack, Midpage, and ChatGPT all surpassed the human lawyer baseline.
  • ChatGPT performed well despite not being purpose-built for legal work.
  • AI systems struggled with multi-jurisdictional and citation-specific questions.
  • Experts say human review remains essential for accuracy and interpretation.

The report from LLM evaluation startup Vals AI compared the performance of Alexi, Counsel Stack, Midpage, and OpenAI’s ChatGPT against human lawyers on 200 U.S. legal research questions. The questions were sourced from attorneys at firms including Reed Smith, Fisher Phillips, McDermott Will & Emery, Ogletree Deakins, Paul Hastings, and Paul, Weiss, Rifkind, Wharton & Garrison.

Each response—AI and human—was scored for accuracy, authoritativeness, and clarity. The lawyer baseline averaged 69%, while the AI tools outperformed: Counsel Stack led at 78%, followed by Alexi at 77%, Midpage at 76%, and ChatGPT at 74%.

Tara Waters, the project’s lead, said she expected ChatGPT to excel in citation quality but found the opposite. “ChatGPT doesn't seem to be, yet, well-engineered for the sourcing and citation,” she told Legaltech News. The generalist AI tended to rely on broad web-based materials rather than pinpointing authoritative statutes or cases, she said.

Still, the legal-focused tools showed weaknesses too. When prompted to survey all 50 states for a single statute, they underperformed ChatGPT. “That was surprising,” said Vals AI CEO Rayan Krishnan, who noted that the systems “should be able to check each one procedurally” without fatigue. He speculated that some tools may have jurisdictional coverage limits or outdated data.

Krishnan cautioned that despite their strong aggregate scores, AI outputs still leave critical gaps. “Even if these tools are getting 70% accuracy, that remaining 30% is really valuable to have human input for,” he said.

Waters added that Vals AI plans to make its evaluation process more repeatable and automated but emphasized the need for continued human involvement in review and scoring. “There won't ever be a pure automated answer for this,” she said, “but we’ll be able to do it more frequently and consistently.”

This study follows Vals AI’s February benchmark that evaluated legal AI platforms from Thomson Reuters, Harvey, vLex, LexisNexis, and Vecflow, assessing how accurately they handled case analysis and transactional work. The new results suggest that, while AI continues to narrow the performance gap with lawyers, the future of legal research may depend as much on oversight as on automation.

Customer Stories

See how leading enterprise in-house teams have scaled smarter with Legal.io's high-caliber flex talent.

More from Legal.io


Community Discussions: How do you advance the role of legal operations within the legal department and within your organization?
Oct 10, 2023
Read More
Optimizing Law Firm Operations: The Strategic Role of Resource Management in DEI and Talent Development
Optimizing Law Firm Operations: The Strategic Role of Resource Management in DEI and Talent Development

Law firms are increasingly adopting resource management tools to address declining client services and financial pressures, as highlighted in a BigHand survey. These tools not only streamline operations but also play a crucial role in supporting diversity, equity, inclusion (DEI), and career development initiatives, essential for talent retention and aligning with client expectations.

Nov 16, 2023
Read More
Google to Mandate Disclosures of AI Content in Political Ads
Google to Mandate Disclosures of AI Content in Political Ads

Ahead of the election season, the tech giant is taking steps to put proper disclaimers on its political content.

Sep 09, 2023
Read More
AI Hallucinations Land Butler Snow in Hot Water in Alabama Prison Case

A federal judge is weighing sanctions against Butler Snow after one of its lawyers used ChatGPT-generated fake case citations in a high-profile Alabama prison lawsuit.

May 24, 2025
Read More
Lawyers Leaving US Government Accelerate Workforce Shift

An exodus of federal government lawyers in 2025 is reshaping the U.S. legal labor market and boosting private sector hiring as agency ranks shrink and DOJ sees heavy losses.

Feb 03, 2026
Read More
Ready to hire?

Schedule a free consultation to discuss your hiring needs.

Free 15-min consultation
Legal.io Platform
5 star reviews
Hiring made smarter

Easy-to-use platform for hiring legal talent, managing spend, and optimizing your panel — plus an average savings of 50%.

Need Immediate Help?

Submit a hiring request and let our experts handle the entire process for you.