Washington, DC — The Association for the Advancement of Artificial Intelligence (AAAI), a leading nonprofit dedicated to advancing scientific research and collaboration, today announced a pilot program that strategically incorporates Large Language Models (LLMs) to enhance the academic paper review process for the AAAI-26 conference. This initiative aims to improve efficiency while maintaining the highest standards of scientific rigor and human oversight.
Enhancing Scientific Review, Not Replacing Human Expertise
The pilot program will thoughtfully integrate LLM technology at two specific points in the established review process:
- Supplementary First-Stage Reviews: LLM-generated reviews will be included as one component of the
initial review stage, providing an additional perspective alongside traditional human expert evaluations. - Discussion Summary Assistance: LLMs will assist the Senior Program Committee (SPC) members by
summarizing reviewer discussions, helping to highlight key points of consensus and disagreement among
human reviewers.
“This pilot represents a careful, measured approach to incorporating new technology into the scientific review process,” said Stephen Smith, AAAI President. “We’re exploring how LLMs can complement—not replace —the irreplaceable expertise and judgment of our human reviewers.”
Preserving Human Decision-Making and Scientific Integrity
AAAI emphasizes that this pilot maintains the primacy of human expertise and judgment in several important ways:
- No Displacement of Human Reviewers: No human reviewers are being replaced at any stage of the
process. - No Automated Decision-Making: LLM-generated content will not be used for automated accept/reject
decisions. - No Numerical Ratings from LLMs: The technology will not provide numerical scores or ratings for papers.
- Human Oversight at All Stages: Human experts will review all LLM-generated content.
Cutting-Edge Methods with Rigorous Safeguards
The pilot employs state-of-the-art methodologies in the responsible deployment of LLM technology, including:
- Multi-step reasoning processes at inference time
- Web search capabilities as tools in the reasoning chain
- Rigorous checks for proper data source attribution
- Comprehensive monitoring and evaluation of LLM contributions
Stringent Privacy and Data Protection
AAAI has implemented strict privacy controls throughout the pilot:
- All paper data and reviewer information will remain confidential
- No data will be used for LLM training
- Information will not be repurposed for any use beyond generating the intended reviews and summaries
- Complete compliance with all data protection regulations
“Scientific integrity and data privacy are non-negotiable priorities in this pilot,” said Smith. “We’ve designed this program with multiple layers of protection to ensure these principles are upheld.”
Looking Ahead
The results of this pilot will be thoroughly evaluated to determine the value and appropriate role of LLM technology in future scientific review processes. AAAI plans to publish a comprehensive analysis of the pilot’s outcomes, contributing to the growing body of knowledge on responsible AI integration in academic contexts.
About the Association for the Advancement of Artificial Intelligence
Founded in 1979, the Association for the Advancement of Artificial Intelligence (AAAI) is a nonprofit scientific society devoted to advancing the scientific understanding of the mechanisms underlying thought and intelligent behavior and their embodiment in machines. AAAI aims to promote research in and responsible use of artificial intelligence and increase public understanding of the field. For more information, see www.aaai.org.
AAAI Media Contact
Meredith Ellison
AAAI
ellison@aaai.org
Download “AAAI Launches AI-Powered Peer Review Assessment System”