California is moving to set formal boundaries on how lawyers and arbitrators use artificial intelligence as courts grapple with errors tied to AI assisted legal work. The state Senate passed SB 574, which would require attorneys to independently verify the accuracy of any material produced using AI tools, including legal citations and factual claims, according to Reuters.

The bill now heads to the California State Assembly, positioning the state among the first to codify professional standards for AI use in legal practice. If enacted, lawyers would be required to take reasonable steps to confirm AI output, correct fabricated or inaccurate content, and remove biased material before relying on it in filings or legal work.

The proposal also extends to alternative dispute resolution. Arbitrators would be barred from delegating decision making to generative AI and from relying on AI generated information outside the record without notifying the parties. Lawmakers framed the provision as a safeguard for transparency and procedural fairness.

Judicial concern over AI hallucinations has intensified as courts across the country sanction lawyers for submitting filings containing nonexistent cases or quotations generated by chatbots. California lawmakers cited a state appeals court decision fining a lawyer $10,000 for fabricated citations as evidence that existing duties already require close human review.

SB 574 would further restrict lawyers from inputting confidential or nonpublic client information into publicly available AI tools and require that AI use not result in unlawful discrimination. Senator Tom Umberg, who introduced the bill, said the goal is to ensure accountability as AI becomes more common in legal work and to keep legal decision making in human hands.

The measure builds on existing California rules governing attorney conduct and mirrors guidance already in place for judges and court staff. It also aligns with the state’s broader effort to regulate AI risks, including recent laws requiring major technology companies to disclose how they plan to manage potential harms from advanced AI systems.

For law firms and corporate legal departments, the bill signals more explicit compliance expectations around AI use, even as adoption accelerates. With Assembly review still pending, California’s approach is likely to inform similar efforts in other jurisdictions.