United States

AI evaluation is the process of assessing the performance, reliability, and safety of AI systems.&lt;br&gt;&lt;br&gt;
**Main Takeaways**&lt;br&gt;&lt;br&gt;
* AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods. 
* Current approaches to evaluation focus on benchmark-driven testing, e.g., of the quality of (generative) models, with insufficient attention paid to other critical factors such as usability, transparency, and adherence to ethical guidelines. 
* New insights and methods for evaluating AI systems are needed to provide the assurance for trustworthy, wide-scale deployments.&lt;br&gt;&lt;br&gt;

**Context &amp; History**&lt;br&gt;&lt;br&gt;
The recent advances in AI have spurred tremendous innovation in potential applications for the technology.  However, many organizations are hesitant to deploy AI systems due to risks that include reputational damage from generative AI hallucinations, leakage of proprietary data, and lack of assurance that legal and ethical guardrails will be enforced. &lt;br&gt;&lt;br&gt;
Empirical methods have long played a role in AI research (e.g., [1]). Indeed, the research community has developed a robust body of metrics and methods for evaluating individual AI algorithms that has enabled the field to quantify performance and track progress.  In contrast, less attention has been paid to evaluating AI systems and their deployment in real-world settings, including their usage by non-AI experts. &lt;br&gt;&lt;br&gt;
AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods.  The generality, complexity, and breadth of AI capabilities makes it impossible to test them exhaustively, requiring new thinking as to what constitutes sufficiency in testing. Run-time adaptivity and the evolution of learned models can change system behavior on the fly, introducing the need for continuous monitoring and validation. Many AI systems are designed to be used interactively, making collaborative usage and its impact on humans an important consideration. &lt;br&gt;&lt;br&gt;
**Current State &amp; Trends** &lt;br&gt;&lt;br&gt;
Current practice for evaluating generative AI systems focuses on model-level testing relative to  a growing body of benchmarks.  Some benchmarks seek to measure general capabilities (e.g., GLUE [2], ARC-AGI [3], MMLU [4]) while others address particular types of reasoning and knowledge (e.g., MATH for mathematics [5], GPQA for logic [6], HumanEval for coding [7]). Benchmark-driven testing provides valuable insights into capabilities and shortcomings, as well as a principled means to evaluate progress over time.  Benchmarks are used as proxies for AI capabilities but have an inherent contextualization that does not necessarily generalize well to new domains. Furthermore, benchmark-based testing is insufficient for ensuring successful deployment given the lack of attention to expected usage in real-world settings and aspects of human use. Benchmark-driven evaluation further raises issues related to overfitting and contamination of test data with training data. As embodied in Goodhart’s law, *“When a measure becomes a target, it ceases to be a good measure”*.  &lt;br&gt;&lt;br&gt;
Evaluating AI systems is inherently complex, especially if these systems are broadly applicable and capable of learning after deployment, requiring a balanced approach that is clear and transparent while avoiding overfitting to specific metrics at the expense of broader reliability, fairness, and real-world applicability. System-level evaluation, when done, explores representative use cases rather than seeking to be comprehensive.  Redteaming serves as a complementary method through the use of adversarial interactions to identify misalignment with desired behavior models.  &lt;br&gt;&lt;br&gt;
Moving forward, AI evaluation needs to consider multiple dimensions of a system’s performance. Most evaluation efforts focus on capability, i.e., producing correct answers or behaviors in response to queries or tasking, for the scope of problems over which the system is expected to operate.  Meeting capability requirements is essential for use; however, other aspects of performance must be considered. &lt;br&gt;&lt;br&gt;
*Usability* is another critical dimension for evaluation. A principal factor of usability is transparency, meaning that mechanisms are provided that enable users to understand the basis for system actions and responses.  Usability further requires directability, meaning that users can control and modify the behavior of the system to meet current and specialized needs (now often referred to as “alignment”).  For AI systems being deployed to aid humans, evaluation must necessarily consider whether the technology ultimately improves combined humansystem performance.  &lt;br&gt;&lt;br&gt;
*Adherence* to legal requirements and ethical guidelines constitutes another important dimension for evaluation.  Increasingly, geo-political entities are introducing legislation to restrict what and how AI systems will be allowed to operate within their jurisdictions, requiring validation that performance will stay within defined guardrails. Both government and commercial organizations have ethical and financial motivations to ensure that their use of AI is fair and unbiased. To support these goals, various trustworthy AI assessment frameworks have been developed to guide organizations in evaluating AI systems for fairness, transparency, robustness, and compliance with ethical standards. Notable frameworks include the EU Trustworthy AI Assessment Framework, the NIST AI Risk Management Framework, and the ISO/IEC 42001:2023 AI Management System Standard.&lt;br&gt;&lt;br&gt;
AI systems introduce multiple operational issues related to their deployment. Privacy is a main concern: protecting personal or corporate information within a model from being leaked.  AI systems have become attack surfaces themselves, with adversaries seeking to exfiltrate data or model weights, or to bias responses for purposes at odds with the model’s creators.  Resource consumption and the cost for both training and deployment are additional considerations in evaluating overall performance of an AI system. &lt;br&gt;&lt;br&gt;
These various factors must be weighed together, including fairness, robustness, interpretability, and compliance with evolving regulations. A comprehensive evaluation framework must balance these diverse considerations, ensuring AI systems are secure, efficient, and aligned with ethical and legal standards.&lt;br&gt;&lt;br&gt;
**Research Challenges**&lt;br&gt;&lt;br&gt;
There is a need for a science of evaluation for AI systems that will inject additional rigor into the evaluation process. This science will build on existing metrics and methodologies but incorporate  new approaches that will increase confidence in our ability to deploy AI systems in missioncritical settings (e.g., [8] for evaluating Retrieval-Augmented Generation systems). Frameworks for auditing and reproducibility will be important to ensure the reliability and robustness of results [9]; as well, more attention should be paid to education within the field on proper empirical methodology.  Below are key challenges for advancing our understanding of how to conduct effective evaluations for AI systems.
* Develop a better understanding how to monitor and assess AI systems that are deployed over extended periods of time, especially for those that evolve their behavior. 
* Develop frameworks for evaluating the safety of agentic AI systems that can take actions in the world. 
* Create methods to provide increased transparency into machine learning models. 
* Develop evaluation methodologies that directly address human engagement with AI capabilities (as was the case with the Turing test). 
* Understand the trade-offs between different dimensions of evaluation, such as whether increased transparency justifies higher costs, or adherence to guardrails outweighs potential impacts on privacy, or how other crossdimensional considerations might influence overall outcomes.
&lt;br&gt;&lt;br&gt;
**Community Opinion** &lt;br&gt;&lt;br&gt;
The responses to the community survey show that there is significant concern regarding the state of practice for evaluating AI systems.  More specifically, 75% of the respondents either agreed or strongly agreed with the statement “The lack of rigor in evaluating AI systems is  impeding AI research progress.” Only 8% of respondents disagreed or strongly disagreed, with 17% neither agreeing nor disagreeing. These results reinforce the need for the community to devote more attention to the question of evaluation, including creating new methods that align better with emerging AI approaches and capabilities. &lt;br&gt;&lt;br&gt;
Given the responses to the first question, it is interesting that only 58% of respondents agreed or strongly agreed with the statement *“Organizations will be reluctant to deploy AI systems without more compelling evaluation methods.”*  Approximately 17% disagreed or strongly disagreed with this statement while 25% neither agreed nor disagreed. If one assumes that the lack of rigor for AI research transfers to a lack of rigor for AI applications, then the responses to these two statements expose a concern that AI applications are being rushed into use without suitable assessments having been conducted to validate them. &lt;br&gt;&lt;br&gt;
For the question *“What percentage of time do you spend on evaluation compared to other aspects of your work on AI?”* the results show 90% of respondents spend more than 10% of their time on evaluation and 30% spend more than 30% of their time. This clearly indicates that respondents take evaluation seriously and devote significant effort towards it. While the prioritization of evaluation is commendable, the results would also seem to indicate that evaluation is a significant burden, raising the question of what measures could be taken to reduce the effort that it requires. Potential actions might include promoting an increased focus on establishing best practices and guidelines for evaluation practices, increased sharing of datasets, and furthering the current trend of community-developed benchmarks. &lt;br&gt;&lt;br&gt;
The most widely selected response to the question* “Which of the following presents the biggest challenge to evaluating AI systems?”* was a lack of suitable evaluation methodologies (40%), followed by the black-box nature of systems (26%), and the cost/ time required to conduct evaluations (18%). These results underscore the need for the community to evolve approaches to evaluation that align better with current techniques and broader deployment settings.&lt;br&gt;&lt;br&gt;
1. Cohen, P.R. (1995). Empirical Methods for Artificial Intelligence. MIT Press. &lt;br&gt;2. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. BlackboxNLP@ EMNLP. &lt;br&gt;3. Chollet, F. (2019). On the Measure of Intelligence. ArXiv, abs/1911.01547. &lt;br&gt;4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.X., and  Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. ArXiv, abs/2009.03300. &lt;br&gt;5. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D.X., and Steinhardt, J. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. ArXiv, abs/2103.03874. &lt;br&gt;6. Rein, D., Hou, B.L., Stickland, A.C., Petty, J., Pang, R., Dirani, J., Michael, J., and  Bowman, S.R. (2023). GPQA: A Graduate-Level Google-Proof Q&amp;A Benchmark. ArXiv, abs/2311.12022. &lt;br&gt;7. Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. ArXiv abs/2107.03374 &lt;br&gt;8. Shahul, E., James, J., Anke, L.E., and Schockaert, S. (2023). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Conference of the European Chapter of the Association for Computational Linguistics. &lt;br&gt;9. Gundersen, O.E., Helmert, M., and Hoos, H. (2024). Improving Reproducibility in AI Research: Four Mechanisms Adopted by JAIR. J. Artif. Intell. Res. 81, 1019-1041.


The Future of AI Research - Webinar Series

AI Evaluation

The AAAI 2025 presidential panel on the future of AI research aims to help all AI stakeholders navigate the recent significant transformations in AI capabilities, as well as AI research methodologies, environments, and communities. It includes 17 chapters, each covering one topic related to AI research, and sketching its history, current trends and open challenges. The study has been conducted by 25 AI researchers and supported by 15 additional contributors and 475 respondents to a community survey.

AI evaluation is the process of assessing the performance, reliability, and safety of AI systems. 
**Main Takeaways** 
* AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods. 
* Current approaches to evaluation focus on benchmark-driven testing, e.g., of the quality of (generative) models, with insufficient attention paid to other critical factors such as usability, transparency, and adherence to ethical guidelines. 
* New insights and methods for evaluating AI systems are needed to provide the assurance for trustworthy, wide-scale deployments. 

**Context & History** 
The recent advances in AI have spurred tremendous innovation in potential applications for the technology. However, many organizations are hesitant to deploy AI systems due to risks that include reputational damage from generative AI hallucinations, leakage of proprietary data, and lack of assurance that legal and ethical guardrails will be enforced. 
Empirical methods have long played a role in AI research (e.g., [1]). Indeed, the research community has developed a robust body of metrics and methods for evaluating individual AI algorithms that has enabled the field to quantify performance and track progress. In contrast, less attention has been paid to evaluating AI systems and their deployment in real-world settings, including their usage by non-AI experts. 
AI systems introduce unique evaluation challenges that extend far beyond the scope of standard software validation and verification methods. The generality, complexity, and breadth of AI capabilities makes it impossible to test them exhaustively, requiring new thinking as to what constitutes sufficiency in testing. Run-time adaptivity and the evolution of learned models can change system behavior on the fly, introducing the need for continuous monitoring and validation. Many AI systems are designed to be used interactively, making collaborative usage and its impact on humans an important consideration. 
**Current State & Trends** 
Current practice for evaluating generative AI systems focuses on model-level testing relative to a growing body of benchmarks. Some benchmarks seek to measure general capabilities (e.g., GLUE [2], ARC-AGI [3], MMLU [4]) while others address particular types of reasoning and knowledge (e.g., MATH for mathematics [5], GPQA for logic [6], HumanEval for coding [7]). Benchmark-driven testing provides valuable insights into capabilities and shortcomings, as well as a principled means to evaluate progress over time. Benchmarks are used as proxies for AI capabilities but have an inherent contextualization that does not necessarily generalize well to new domains. Furthermore, benchmark-based testing is insufficient for ensuring successful deployment given the lack of attention to expected usage in real-world settings and aspects of human use. Benchmark-driven evaluation further raises issues related to overfitting and contamination of test data with training data. As embodied in Goodhart’s law, *“When a measure becomes a target, it ceases to be a good measure”*. 
Evaluating AI systems is inherently complex, especially if these systems are broadly applicable and capable of learning after deployment, requiring a balanced approach that is clear and transparent while avoiding overfitting to specific metrics at the expense of broader reliability, fairness, and real-world applicability. System-level evaluation, when done, explores representative use cases rather than seeking to be comprehensive. Redteaming serves as a complementary method through the use of adversarial interactions to identify misalignment with desired behavior models. 
Moving forward, AI evaluation needs to consider multiple dimensions of a system’s performance. Most evaluation efforts focus on capability, i.e., producing correct answers or behaviors in response to queries or tasking, for the scope of problems over which the system is expected to operate. Meeting capability requirements is essential for use; however, other aspects of performance must be considered. 
*Usability* is another critical dimension for evaluation. A principal factor of usability is transparency, meaning that mechanisms are provided that enable users to understand the basis for system actions and responses. Usability further requires directability, meaning that users can control and modify the behavior of the system to meet current and specialized needs (now often referred to as “alignment”). For AI systems being deployed to aid humans, evaluation must necessarily consider whether the technology ultimately improves combined humansystem performance. 
*Adherence* to legal requirements and ethical guidelines constitutes another important dimension for evaluation. Increasingly, geo-political entities are introducing legislation to restrict what and how AI systems will be allowed to operate within their jurisdictions, requiring validation that performance will stay within defined guardrails. Both government and commercial organizations have ethical and financial motivations to ensure that their use of AI is fair and unbiased. To support these goals, various trustworthy AI assessment frameworks have been developed to guide organizations in evaluating AI systems for fairness, transparency, robustness, and compliance with ethical standards. Notable frameworks include the EU Trustworthy AI Assessment Framework, the NIST AI Risk Management Framework, and the ISO/IEC 42001:2023 AI Management System Standard. 
AI systems introduce multiple operational issues related to their deployment. Privacy is a main concern: protecting personal or corporate information within a model from being leaked. AI systems have become attack surfaces themselves, with adversaries seeking to exfiltrate data or model weights, or to bias responses for purposes at odds with the model’s creators. Resource consumption and the cost for both training and deployment are additional considerations in evaluating overall performance of an AI system. 
These various factors must be weighed together, including fairness, robustness, interpretability, and compliance with evolving regulations. A comprehensive evaluation framework must balance these diverse considerations, ensuring AI systems are secure, efficient, and aligned with ethical and legal standards. 
**Research Challenges** 
There is a need for a science of evaluation for AI systems that will inject additional rigor into the evaluation process. This science will build on existing metrics and methodologies but incorporate new approaches that will increase confidence in our ability to deploy AI systems in missioncritical settings (e.g., [8] for evaluating Retrieval-Augmented Generation systems). Frameworks for auditing and reproducibility will be important to ensure the reliability and robustness of results [9]; as well, more attention should be paid to education within the field on proper empirical methodology. Below are key challenges for advancing our understanding of how to conduct effective evaluations for AI systems.
* Develop a better understanding how to monitor and assess AI systems that are deployed over extended periods of time, especially for those that evolve their behavior. 
* Develop frameworks for evaluating the safety of agentic AI systems that can take actions in the world. 
* Create methods to provide increased transparency into machine learning models. 
* Develop evaluation methodologies that directly address human engagement with AI capabilities (as was the case with the Turing test). 
* Understand the trade-offs between different dimensions of evaluation, such as whether increased transparency justifies higher costs, or adherence to guardrails outweighs potential impacts on privacy, or how other crossdimensional considerations might influence overall outcomes.
 
**Community Opinion** 
The responses to the community survey show that there is significant concern regarding the state of practice for evaluating AI systems. More specifically, 75% of the respondents either agreed or strongly agreed with the statement “The lack of rigor in evaluating AI systems is impeding AI research progress.” Only 8% of respondents disagreed or strongly disagreed, with 17% neither agreeing nor disagreeing. These results reinforce the need for the community to devote more attention to the question of evaluation, including creating new methods that align better with emerging AI approaches and capabilities. 
Given the responses to the first question, it is interesting that only 58% of respondents agreed or strongly agreed with the statement *“Organizations will be reluctant to deploy AI systems without more compelling evaluation methods.”* Approximately 17% disagreed or strongly disagreed with this statement while 25% neither agreed nor disagreed. If one assumes that the lack of rigor for AI research transfers to a lack of rigor for AI applications, then the responses to these two statements expose a concern that AI applications are being rushed into use without suitable assessments having been conducted to validate them. 
For the question *“What percentage of time do you spend on evaluation compared to other aspects of your work on AI?”* the results show 90% of respondents spend more than 10% of their time on evaluation and 30% spend more than 30% of their time. This clearly indicates that respondents take evaluation seriously and devote significant effort towards it. While the prioritization of evaluation is commendable, the results would also seem to indicate that evaluation is a significant burden, raising the question of what measures could be taken to reduce the effort that it requires. Potential actions might include promoting an increased focus on establishing best practices and guidelines for evaluation practices, increased sharing of datasets, and furthering the current trend of community-developed benchmarks. 
The most widely selected response to the question* “Which of the following presents the biggest challenge to evaluating AI systems?”* was a lack of suitable evaluation methodologies (40%), followed by the black-box nature of systems (26%), and the cost/ time required to conduct evaluations (18%). These results underscore the need for the community to evolve approaches to evaluation that align better with current techniques and broader deployment settings. 
1. Cohen, P.R. (1995). Empirical Methods for Artificial Intelligence. MIT Press. 2. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. BlackboxNLP@ EMNLP. 3. Chollet, F. (2019). On the Measure of Intelligence. ArXiv, abs/1911.01547. 4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.X., and Steinhardt, J. (2020). Measuring Massive Multitask Language Understanding. ArXiv, abs/2009.03300. 5. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D.X., and Steinhardt, J. (2021). Measuring Mathematical Problem Solving With the MATH Dataset. ArXiv, abs/2103.03874. 6. Rein, D., Hou, B.L., Stickland, A.C., Petty, J., Pang, R., Dirani, J., Michael, J., and Bowman, S.R. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. ArXiv, abs/2311.12022. 7. Chen, M. et al. (2021). Evaluating Large Language Models Trained on Code. ArXiv abs/2107.03374 8. Shahul, E., James, J., Anke, L.E., and Schockaert, S. (2023). RAGAs: Automated Evaluation of Retrieval Augmented Generation. Conference of the European Chapter of the Association for Computational Linguistics. 9. Gundersen, O.E., Helmert, M., and Hoos, H. (2024). Improving Reproducibility in AI Research: Four Mechanisms Adopted by JAIR. J. Artif. Intell. Res. 81, 1019-1041.

webinar

Agents and multi-agent systems (MAS) have evolved from
autonomous problem-solving entities to integrating generative
AI and LLMs, ultimately leading to cooperative AI frameworks
that enhance adaptability, scalability, and collaboration. 
**Main Takeaways**
* Multi-agent systems have evolved from rule-based autonomy to cooperative AI,
emphasizing collaboration, negotiation, and ethical alignment.
* The rise of Agentic AI, driven by LLMs, introduces new opportunities for flexible
decision-making but raises challenges in efficiency and complexity.
* Integrating cooperative AI with generative models requires balancing
adaptability, transparency, and computational feasibility in multi-agent
environments 

**Context & History** 
The field of multi-agent systems emerged in the late 1980s/early 1990s,
with its main influences coming from two disparate areas [1,2]. One was the
field of AI robotics, which had begun to seriously address the issue of integrated agent architectures: how do we assemble several cognitive components of intelligence (planning, reasoning, learning, vision,…) into an integrated computational agent? The second was the nascent area of distributed AI, which studied how multiple AI systems could be made to solve problems cooperatively, by dynamically sharing information and tasks. By the mid1990s, these ideas had given rise to the new field, driven by the vision of having (semi)autonomous AI systems – agents – working on behalf of individual users in pursuit of their users’ goals, possibly interacting with other such agents in order to do so. A key insight
was that, since delegated goals might not necessarily be in harmony, it would be necessary to equip such agents with the ability to reason socially. Thus, while AI historically emphasised components of intelligence such as reasoning and problem-solving, the new field of multi-agent systems emphasised social skills such as cooperation, coordination, argumentation and negotiation. In order to underpin those skills, the development of models of Theory of Mind became central to the field. 
By the late 1990s, the field had its own conferences and journal, and was
firmly established as a key sub-field of AI. The area flourished from the late
1990s, with enormous energy devoted to (e.g.) communication languages
for autonomous agents, protocols for cooperation, coordination, and
negotiation, and the underpinning theory of these social skills. With
respect to the latter, while in the early years the practical reasoning paradigm of AI planning had been the dominant influence on the theory of multiagent systems, by the early part of this century, game theory had become the dominant theoretical foundation. Game theory, which emerged from the field of economics, is the theory of interaction between self-interested agents. Although originally devised as a tool for studying interactions between humans and human organisations, it nevertheless seemed a natural framework for studying interactions between artificial agents. A huge body of work emerged, studying (for example), how auctions might be used to allocate scarce resources, the theory
of negotiation between self-interested artificial agents, and how agents might optimally form teams to solve problems and share the associated benefits of cooperation. Interestingly, although learning in multi-agent systems was a key component of the field from the outset, it was not the centre of attention within the field in the first decade or so. 
This initial boom period for multiagent systems lasted roughly from
the mid 1990s to around 2010-15. By the end of that time, though,
some uncomfortable questions were beginning to be asked. While the field
had generated impressive quantities of scientific results, applications seemed to be thin on the ground. For sure there were some high-profile applications. The field of security games, which emerged from multi-agent systems, used ideas from game theory to allocate scarce security resources to defend high-profile targets such as airports. This work led to deployed
applications at US airports and ports. Automated high-frequence trading
systems, which plan and execute the bulk of trades on the world’s markets,
are multi-agent systems on a global scale. And agent-based modelling,
which models socio-technical systems at the level of individual decisionmakers, received a huge boost after the 2008 financial crisis, and again after the 2020 COVID-19 pandemic, where it was demonstrated to be an important tool for modelling the spread of contagion: financial in the first case; epidemiological and social policy in the second. But for all these successes, the core vision of multi-agent systems - where agents function in the context of other agents is an active area of research, exploring social concepts such as norms, organisations, practices and also values, is ongoing work, but for a large part outside the AAMAS community, but within social simulation research, and with results informing and
shaping policy-making in several areas from public health to transportation,
and urban transformation. 
While applications of multi-agent systems research (AI agents interacting
with other AI agents) has not, as yet, lived up to early expectations,
individual dialogue agents such as Alexa, Siri, Cortana are now an everyday
reality, and trace their historical roots both to work on intelligent agents
in the 1990s and work from the NLP community on dialog systems. Many
other applications of this thread of work have achieved success over the past three decades: automated call center assistants, customer service assistants, smartphone virtual assistants, smart speaker assistants, home robot assistants, that can converse with human users and accomplish tasks like ticket booking, restaurant reservations, online shopping, medical and health assistance, and sales assistance, empowered by AI modules like
speech recognition, natural language understanding, dialog management (i.e. state tracking and dialog policy), and natural language generation and speech synthesis. 
As ML surged in the early part of this century, activity in this area increased
within the multi-agent systems field. Multi-agent reinforcement
learning (MARL) grew to become the single biggest area within the field,
possibly driven in part by the fact that developing MARL experiments can be done relatively quickly and without recourse to expensive hardware. At the time of writing, while MARL represents a significant sub-field of ML as a whole, it seems to lack any clear unifying vision or direction – or application. 
**Current State & Trends** 
The emergence of LLMs from 2020 onwards has also led to increased
interest in agents [3]. LLMs can be used as part of a workflow to
automate routine tasks, and the general capabilities of such “agents” for
planning and problem solving is widely discussed. In this context, the concept of Agentic AI refers to the integration of generative AI and LLMs into autonomous agent frameworks aiming to leverage the generative capabilities of such models to enhance interaction, creativity, and real-time decision making in dynamic environments. As we write this (late 2024) there has been an explosion of startup companies hoping to commercialise such agents. Despite this renewed enthusiasm, the original aims of AAMAS from 30 years ago, such as building robust, autonomous multiagent systems capable of complex coordination and long-term reasoning, have not been fully realized. The extent to which this new wave of agent activity is informed by what went before is unclear. 
The challenge now is to understand what multi-agent systems mean in
the era of LLMs. The current direction of agentifying LLMs may lead to
overly complex and unnecessary architectures and heavy computational
costs, whereas adopting a multi-agent paradigm to the development and use of LLMs may offer a sustainable way to compose, diversify, and integrate approaches effectively. Even though distribution was one of the original drivers for the MAS field, this is still a largely unexplored direction under the current paradigm. Another trend nowadays is to recover ideas from classical cognitive architectures to add common sense skills to autonomous agents. 
An emerging trend is multi-agent architectures, which structure
AI components into modular systems that improve transparency,
adaptability, and ethical alignment. The focus on cooperative agents
highlights a shift toward AI that prioritizes collaboration, negotiation, and shared decision-making. By applying modularity, encapsulation,
and separation of concerns, these architectures enable scalable teamwork
between autonomous agents and humans, making them ideal for
hybrid AI applications requiring trust, explainability, and domain-specific
expertise. 
**Research Challenges** 
* Identify challenges and benefits of embedding GenAI-driven agents
into MAS, focusing on enhancing collaboration without disrupting
existing dynamics.
* Investigate how LLM-powered agents can improve negotiation and
decision-making in dynamic multiagent environments while ensuring
ethical alignment and safety.
* Develop architectures that integrate LLM-driven agents while maintaining
scalability, transparency, and computational efficiency in multiagent settings. 
**Community Opinion** 
The survey responses indicate a
majority of respondents finding this
theme relevant to their research, with
a growing interest in integrating Large
Language Models (LLMs) into multiagent systems. Many participants
already use AI agents, with LLMs being
the most common technique (29.34%),
highlighting their expanding role in AI driven applications. 
The potential of multi-agent systems
leveraging LLMs is seen in areas such
as collaborative problem-solving
(68.86%), distributed decision-making
(54.49%), and social simulations
(41.32%). However, challenges persist,
including misalignment between LLMs’
general knowledge and specific system
needs (59.88%), lack of interpretability
(59.28%), and security risks (50.90%).
These concerns suggest a need for
improved explainability, alignment
strategies, and robust security measures
to ensure effective deployment. 
There is also a debate on the necessity
of agentifying LLMs—while 51.5%
believe multi-agent LLM paradigms
are essential for sustainable AI,
42.33% disagree that they introduce
unnecessary complexity. The
computational cost-benefit balance
remains uncertain, with responses
divided on whether LLMs outweigh
their costs. 
The textual responses highlight a broad
spectrum of perspectives on integrating
Large Language Models (LLMs) into
multi-agent systems (MAS), with some
advocating for hybrid approaches
rather than relying solely on LLMs. Many
respondents stressed the need for
diverse AI architectures, emphasizing
modular, multi-technology systems
where LLMs play a role but do not
dominate. Governance, coordination,
and adaptability emerge as key
advantages of MAS, while concerns
include increased complexity, lack
of theoretical guarantees, and high
computational costs. Several responses
criticize the overemphasis on LLMs,
questioning whether they are truly
essential or merely a current trend.
Others highlight practical challenges
such as grounding, alignment, and
robust communication protocols,
pointing out the need for new
frameworks that integrate symbolic
reasoning, structured governance,
and scalable architectures. Overall,
the discussion reflects a critical but
open stance toward agentifying LLMs,
suggesting that context, application
domain, and technological diversity will
shape their effectiveness in multi-agent
environments. 
In summary, the survey reflects
optimism about LLM-driven multiagent systems, but also underscores
the need for addressing key challenges
before widespread adoption.
 
1. Yoav Shoham. Agent-oriented programming. Artificial Intelligence. Artificial Intelligence. 60 (1): 51–92.
2. Michael Wooldridge. An Introduction to Multi-agent Systems 2e. Wiley, 2009.
3. Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic. Agents. Google whitepaper. https://archive.org/details/google-ai-agents-whitepaper

AI agents

The ability to reason has been a salient characteristic of human intelligence, and there is a critical need for verifiable reasoning in AI systems. 
**Main Takeaways** 
* Reasoning has always been seen as a core characteristic of human intelligence.
Reasoning is used to derive new information from given base knowledge; this
new information is guaranteed correct when sound formal reasoning is used,
otherwise it is merely plausible.
* AI research has led to a range of automated reasoning techniques. These
reasoning techniques have given rise to AI algorithms and systems, including
SAT, SMT, and constraints solvers as well as probabilistic graphical models, all of
which play a key role in critical real-world applications.
* While large pre-trained systems (such as LLMs) have made impressive
advancements in their reasoning capabilities, more research is needed to
guarantee correctness and depth of the reasoning performed by them; such
guarantees are particularly important for autonomously operating AI agents 

**Context & History** 
Reasoning is a core component of
human intelligence. From the dawn
of humanity, abductive reasoning
has been used to predict danger and
inductive reasoning made it possible
to learn regularities governing the
world. Beginning in Ancient Greece,
deductive reasoning techniques were
developed to draw valid conclusions
that follow logically from premises
known to be true. The development
of reasoning methods with such a
priori guarantees was a key factor in
the advancement of modern science,
mathematics, and engineering; notably,
according to philosophers such as
Charles Sanders Peirce, the interplay
between abduction, deduction, and
induction forms the basis of the
scientific method and hence all modern
science. Attempts to mechanize
logical reasoning can be traced back to
13th-century philosopher Ramon Lull
and lie at the heart of the concept of
computation. Probabilistic reasoning
and inference have also profoundly
impacted reasoning, often relying on
the celebrated theorem by Thomas
Bayes on inverse probability that also
forms the basis for many machine
learning and statistics approaches.
Finally, the evaluation of correct (sound)
reasoning lies at the heart of most
quantitative assessments of human
cognition. 
Not surprisingly, reasoning has been
central to the AI enterprise. Indeed,
the earliest research in AI – from Logic
Theorist onwards [1] – had a strong
focus on reasoning [2]. Since the 1960s,
AI has also embraced probabilistic
reasoning and models, initially for
medical diagnosis [3]. Since then,
the reasoning tasks addressed in AI 
research have covered the gamut from
planning and temporal reasoning to
diagnosis and explanation. While early
AI has paid attention to both plausible
reasoning (case-based, analogical,
qualitative) and sound formal reasoning
with guarantees (logical, probabilistic,
constraint-based), over the years,
the focus has shifted more towards
reasoning with formal guarantees.
There are good reasons for this when
designing AI systems and techniques
that compensate for human limitations
and weaknesses since reasoning
with guarantees is challenging for
humans. This has led to practically
impactful applications of AI systems
such as SAT, SMT, and constraints
solvers, including the verification of
correctness properties of computer
hardware and software, the safety of
communications protocols, the design
of new proteins, and, more recently, the
robustness of neural networks against
adversarial attacks. It has also resulted
in probabilistic graphical models [4,
5], which are powerful modeling and
inference tools that have found their
way into numerous applications of
reasoning in medicine, robotics, and
beyond. 
**Current State & Trends** 
The emergence of the Internet and the
associated technology that made it
possible to capture the human digital
footprint at scale, as well as the leaps in
computing power, have made possible
novel approaches to learning bottom up from data. Of particular interest are
large pre-trained models, such as LLMs,
that have shown surprising abilities in
plausible reasoning. Unlike the earlier
research on reasoning in AI, LLMs
have focused on plausible reasoning patterns as they emerge automatically
after large-scale training on petabyte
corpora. While the results have been
quite remarkable so far, the reasoning in
this context has been of the “plausible”
variety with no guarantees. 
Meanwhile, sound formal reasoning
techniques remain key to important
and impactful applications of cuttingedge AI technology for the verification
of computer hardware and software,
as well as for real-world planning and
resource allocation problems. They
are also increasingly recognized as a
crucial basis for the formal verification
of machine learning techniques such
as neural networks, e.g., in the context
of local robustness against adversarial
attacks [6]. Significant research activity
takes place in these areas, focusing on
improving various types of reasoning
algorithms (notably with respect to their
computational complexity), leveraging
learning within sound formal reasoning,
and combining reasoning and learning
techniques [7, 8]. 
**Research Challenges** 
Bringing some of the rigorous a priori or
post hoc guarantees back into plausible
reasoning patterns turbocharged by
the pre-trained models has become an
active and promising area of research
– especially where AI systems need to
work autonomously in safety-critical
domains. Research on so-called “large
reasoning models” as well as on neurosymbolic approaches is addressing
these challenges. 
Furthermore, even though formal
reasoning with correctness guarantees
is currently considerably less in vogue
than the use of generative AI techniques
for plausible reasoning, formidable and essential challenges also remain in that
area. In this context, the combination
of machine learning techniques with
formal reasoning techniques holds
considerable promise for economically
and socially valuable breakthroughs,
notably in the area of AI safety and
transparency.
The questions and challenges we face
range from the philosophical:
* What exactly is “reasoning”?

to the practical:
* Can LLM ‘reasoning’ be trusted?

and include: 
What does the future hold for the
advancement and role of symbolic
reasoning?
* To what extent can LLMs or other
generative models reproduce or
replace symbolic reasoning?
* To what degree will symbolic
reasoning be necessary or sufficient
to overcome the current limitations
of LLMs?
* How well can AI reasoning,
especially LLM ‘reasoning,’ be
explained and understood?
* How can computers better
understand and simulate human
reasoning?
* What is the role of collaborative
reasoning between humans and
computers?
* How best can LLMs and symbolic
reasoning be integrated into “neurosymbolic reasoning”?
* Are further breakthroughs, beyond
both LLMs and traditional symbolic
reasoning, required to achieve AGIlevel reasoning?
* What forms of reasoning can best
support humans when dealing with
various challenges, e.g., in medical,
scientific, engineering, and legal
domains? 

**Community Opinion** 
The AAAI community appears to
strongly agree on the importance
of reasoning in AI systems. In our
community survey, slightly over 55%
of the respondents chose to answer
specific questions related to the topic
of reasoning. Of these, 79% indicated
that the topic of reasoning is relevant
to their research (with 44.7% marking
it as “very relevant”). Of the properties
required for referring to a process
as reasoning, 77.5% of the survey
participants marked “Knowledge can
be incorporated”, 72.5% “Explanations
can be provided,” and 56.9% “Involves
multiple steps to arrive at a conclusion”.
Interestingly, merely 37.4% indicated
“Guaranteed correctness of inference
results/outcomes”, and only 23.7% that
“A formal system and solver is used,”
which reflects the recent focus on
informal, plausible reasoning, likely in
the context of generative AI methods.
This suggests that an effort may be
warranted to better communicate the
importance and success of formal,
sound reasoning techniques. Finally,
44.7% of respondents agreed that
“Reasoning involves a search process.” 
There was broad agreement among
survey participants that focusing
reasoning research in AI on humanlevel reasoning is valuable (41.6%) or
even essential (47%); similarly, a focus
on domain-specific reasoning abilities
was seen by 49.6% of respondents as
valuable, and by 42.8% as essential.
This clearly reflects the importance
attributed to a research focus on
reasoning. 
The community also sees an exciting
potential of synergy offered by logical
and probabilistic models of reasoning
that were developed in AI prior to large
pre-trained models. This is clearly
reflected in the fact that 76.9% of survey
participants marked the integration
of learning and reasoning approaches
as very important (6 or 7 on a scale
of 7); interestingly, the percentage
of respondents that considered
Explainability and verifiability as very
important was similarly high (at 71.7%). 
Finally, 61.8% of survey participants
estimated the minimal percentage of
symbolic AI techniques required for
reaching human-level reasoning to be
at least 50% (with 24.8% estimating it
at 75% or more, compared to 38.2%
estimating it at 25% or below). What
remains unclear is the degree to
which AI researchers and practitioners
realize that decidedly superhuman
levels of reasoning are required for
and displayed in the prominent and
successful applications of formal AI
reasoning techniques for scientific
and mathematical discovery and
engineering applications, as well as in AI
safety. 
1. Newell, A. & Simon, H. (1956). The logic theory machine: A complex information processing system. IRE Transactions on Information Theory 2: 61-79. 
2. Brachman, R. and Levesque, H. (2004) Knowledge Representation and Reasoning (1st Ed). Morgan Kaufman. 
3. Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach (4th Ed). Pearson. 
4. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman. 
5. Koller, D. and Friedmann, N. (2009) Probabilistic Graphical Models. The MIT Press. 
6. König, M. et al. (2024) Critically Assessing the State of the Art in Neural Network Verification. Journal of Machine Learning Research 25(12): 1-53 
7. Guo, D. et. al. (2025) DeepSeek-R: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948 
8. Kambhampati, S. (2024). Can Large Language Models Reason and Plan? Annals of New York Academy of Sciences. March 2024.

AI Reasoning

Improving factually and trustworthiness of AI systems is the
single largest topic of AI research today, and while significant
progress has been made, most scientists are pessimistic that
the problems will be solved in the near future. 
**Main Takeaways** 
* An AI system is factual if it refrains from outputting false statements. Improving
factually of AI systems based on neural-network large language models is
arguably the biggest area of AI research today.
* Trustworthiness extends trustworthiness to include criteria such as human
understandability, robustness, and the incorporation of human values. Lack of
trustworthiness has always been an obstacle for deploying AI systems in critical
applications.
* Approaches to improving the factuality and trustworthiness of AI systems
include fine-tuning, retrieval-augmented generation, verification of machine
outputs, and replacing complex models with simple understandable models. 

**Context & History** 
A factual AI system does not output
erroneous information or hallucinate
answers. Before the era of generative
AI, problems with factuality arose when
systems were trained on bad data,
as captured by the slogan, “garbage
in, garbage out”. Work on methods
for improving data quality has a long
history in AI [1]. 
Generative AI, and in particular
large language models, employ
reconstructive memory - that is, they
rebuild memories as needed on the
basis of distributed bits of information
rather than retrieve memories from
a fixed store. The earliest generative
LLMs made an impact with their ability
to generate coherent but entirely
imaginary stories [2]. Factuality of LLMs
on a given domain was improved by
fine-tuning the model on domain data
[3]. 
Trustworthiness is a broader concept
than factuality because it includes
criteria such as understandability,
robustness, and respect of human
values. A traditional approach for
improving understandability of AI
systems is to replace complex blackbox models with simple humanunderstandable models - such as
naive Bayes [4] or generalized linear
regression [5]. Research on robustness
of machine learning studies how the
outputs of a model vary with small
changes in its training data. For
example, contrastive learning is a
method to train deep neural nets with
increased robustness [6]. Further
discussion of robustness in generative
AI appears in
this report’s section on Reasoning.
Discussion of respect of human values
by AI systems appears in many other
sections of this report and so will not be
discussed here. 

**Current State & Trends** 
As noted, fine-tuning remains the
main approach used by scientists
and engineers to improve factuality
of generative AI systems. In addition
to fine-tuning on domain-specific
documents, modern fine-tuning
includes reinforcement learning with
human feedback from thousands of
people. The cost of employing such
large numbers of human evaluators
is a major bottleneck for scaling AI
systems, and so there is much interest
in discovering methods to reduce the
amount of human feedback needed [7]. 
The second main technique for
improving factuality of generative AI
is retrieval-augmented generation
(RAG) [8]. In response to a question,
the system gathers a set of relevant
documents using traditional
information retrieval algorithms. The
AI system is then prompted to generate
an answer by combing through and
summarizing the retrieved documents.
While RAG can improve factuality, it
is dependent on the quality of data
retrieved. For example, if the target
document set is the entire web, it
can end up incorporating incorrect
information and even satirical stories in
its answer. 
Related to RAG is enabling the
generative AI system to use tools for
fact checking. Tools used by generative
AI systems include calculators, factual
databases such as citation indexes, and
formal planning and reasoning systems
[9]. A recent approach to improving
factuality is to provide the system with
a set of rules that state constraints on
space of answers. The output from the
model is verified against these rules and
inconsistent responses are culled [10].
Amazon Web Services already supports
this approach with “automated
reasoning checks” [11]. 
A third technique for improving
factuality of generative AI is chainof-thought (CoT), where a series of
prompts breaks down a question into
smaller units [12]. CoT often includes
steps where the model is asked to
reflect back on its tentative conclusions
and see if any are hallucinations. CoT
is discussed in more detail in the
Reasoning section of this report. 
The impact of data quality on factuality
was mentioned above. In addition
to fine-tuning on human curated
data, there is recent work on creating
synthetic data that is guaranteed to be
high quality for fine-tuning [13]. 
Trustworthiness, we noted, generalizes
factually and includes understandability
and robustness. One approach to
making neural network models more
understandable is to factor them into a
set of recognizers for high level features
and then combine the features using an
understandable model such as additive
regression [15]. Another approach is to
tease out how concepts and rules are
actually represented in a trained [16].
Understandability can also be
improved by employing CoT techniques
to ask a generative AI system to explain
the steps in its reasoning [17] or tell
the user when the system is uncertain
about a conclusion [18]. Finally, a
generative AI system can be asked not
to output a single answer, but instead
to distill a complex set of information into a simple human understandable
representation such as a decision tree
[19]. 
**Research Challenges** 
Factuality is far from solved. There
are a growing number of benchmark
dataset designed to test the factuality
of LLMs. One of the latest, SimpleQA
from Google, is a collection of simple,
unambiguous, timeless, and challenging
factual questions and answers
[14]. As of December 2024, the best
models from OpenAI and Anthropic
correctly answered less than half of the
questions. 
Robustness in generative AI can
be improved, as noted above, by
employing robust loss functions such
as contrastive learning. Adversarial
training, which applies perturbations in
the embedding space during training,
can improve both robustness and
generalization [20]. In addition, the
techniques for factuality generally
improve robustness as well. 
**Community Opinion** 
Over 75% of the AAAI community
strongly agreed that factuality and
trustworthiness were relevant or very
relevant to their own research. 
All six approaches mentioned in
the survey for improving factuality - external fact-checking tools,
reinforcement, improving data quality,
data curation, synthetic training, and
new neural net architectures - found
support. The greatest demand was
for more research on new neural net
architectures (73% marked important
or very important), followed closely by
external fact-checking tools (70%). 
For trustworthiness, new neural
net architectures were also viewed
as most important (77% important
or very important), followed by
enabling models to describe their
reasoning processes (70%) and use
of understandable models instead of
neural networks (61%). Notably, few
viewed research on giving AI systems
human-like personalities as important
to improving trustworthiness (24%).
Finally, the community mostly agreed
(59%) that trustworthiness as currently
formulated is ill-defined. Most also
disagreed (around 60%) that either
factuality or trustworthiness would
soon be solved. 
The AAAI community suggested
a number of additional aspects of
factuality and trustworthiness that were
not covered above. These included:
* The ability to understand and
present different sides of the same
issues, including pros and cons for
each one.
* Understanding that trustworthiness
depends upon the context of the
domain, organizational objectives,
and user objectives. It is wrong
to think of an AI system as simply
being trustworthy or untrustworthy
without regard to context.
* Transparency needs to go beyond
the models used to the actual
sources of training data. This should
include multi-source verification of
facts ingested by the models.
* The focus of work should be
on risk and mitigation rather
than on “solving” factuality and
trustworthiness.
* Attention needs to paid to giving
AI agents the ability to update
their knowledge while maintaining
reliability.
 

1. Budach, Lukas, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Hazar Harmouch and Felix Naumann (2022). The Effects of Data Quality on Machine
Learning Performance. https://arxiv.org/pdf/2207.14529
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. Retrieved from https://cdn.openai.com/betterlanguage-models/language_models_are_unsupervised_multitask_learners.pdf
3. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.48550/
arXiv.1810.04805
4. Pedro Domingos & Michael Pazzani (1997). On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Learning, 29(2-3), 103–130. https://doi.
org/10.1023/A:1007413511361
5. Caruana, Rich, Yin Lou, Johannes Gehrke, Paul Koch, M. Sturm and Noémie Elhadad (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day
Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015). https://people.dbmi.columbia.edu/noemie/
papers/15kdd.pdf
6. Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR 2006), 2, 1735–1742. https://doi.org/10.1109/CVPR.2006.100
7. Hu, C., Hu, Y., Cao, H., Xiao, T., & Zhu, J. (2024). Teaching language models to self-improve by learning from language feedback. Findings of the Association for Computational Linguistics
(ACL 2024). Retrieved from https://aclanthology.org/2024.findings-acl.364/
8. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for
knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33, 9459–9474. Retrieved from https://arxiv.org/abs/2005.11401
9. Guan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023). Leveraging pre-trained large language models to construct and utilize world models for model-based task
planning. Proceedings of the 33rd International Conference on Automated Planning and Scheduling (ICAPS 2023). Retrieved from https://arxiv.org/abs/2305.14909
10. Backes, J., Bolignano, P., Cook, B., Dodge, C., Gacek, A., Luckow, K., Rungta, N., Tkachuk, O., & Varming, C. (2018). Semantic-based automated reasoning for AWS access policies using
SMT. In 2018 Formal Methods in Computer-Aided Design (FMCAD) (pp. 1–9). IEEE. https://doi.org/10.23919/FMCAD.2018.8602994
11. Barth, Antje (2024). Prevent factual errors from LLM hallucinations with mathematically sound Automated Reasoning checks (preview). Posted 3 Dec 2024, retrieved 8 Feb 2025. AWS
News Blog, permalink https://aws.amazon.com/blogs/aws/prevent-factual-errors-from-llm-hallucinations-with-mathematically-sound-automated-reasoning-checks-preview
12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in
Neural Information Processing Systems (Vol. 35, pp. 24824–24837). https://proceedings.neurips.cc/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
13. Ding, Bosheng, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu and Shafiq R. Joty (2024). Data Augmentation using LLMs: Data
Perspectives, Learning Paradigms and Challenges. Annual Meeting of the Association for Computational Linguistics (2024). https://aclanthology.org/2024.findings-acl.97.pdf
14. Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus (2024). Measuring short-form factuality in large language
models. https://doi.org/10.48550/arXiv.2411.04368
15. Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., & Hinton, G. E. (2021). Neural additive models: Interpretable machine learning with neural nets. Advances in
Neural Information Processing Systems, 34, 2021. https://proceedings.neurips.cc/paper/2021/hash/251bd0442dfcc53b5a761e050f8022b8-Abstract.html?utm_source=chatgpt.com
16. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin,
A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., & Henighan, T. (2024). Scaling Monosemanticity: Extracting
Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
17. Yeo, W. J., Ng, X. X., Le, T. K. C., & Lu, X. (2024). How interpretable are reasoning explanations from prompting large language models? Findings of the Association for Computational
Linguistics: NAACL 2024. Retrieved from https://aclanthology.org/2024.findings-naacl.138
18. Lin, S., Hilton, J., & Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research. https://openreview.net/pdf?id=8s8K2UZGTZ
19. Chen, Y., Zhang, L., Wang, H., & Li, J. (2025). Zero-Shot Decision Tree Construction via Large Language Models. arXiv preprint arXiv:2501.16247. Retrieved from https://arxiv.org/
abs/2501.16247
20.Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020). Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994. Retrieved from https://
arxiv.org/abs/2004.08994

AI Factuality & Trustworthiness

Embodied AI creates intelligent agents that perceive,
understand, and interact with the physical world. 
**Main Takeaways**
* Intelligence emerges through the coupling of a physical body with a real environment.
* Embodied AI insists that coupling is essential to achieving real intelligence in situated agents.
* Robots are good scientific and engineering platforms for developing Embodied AI.
 

**Context & History** 
In a cartoon view of AI’s historical
development there were two distinct
paradigms. The first is based on explicit
representations of knowledge, either
built-in or learned. The second is built
around learning, from tabula rasa,
in artificial neural networks. Both
approaches are usually disembodied.
A third approach insists that
embodiment is essential to intelligence
for situated agents [2]. The hypothesis is
that intelligence emerges, in evolution
and individual development, through
ongoing interaction and coupling of a
physical body with a real environment.
We call this third paradigm Embodied
AI (EAI). 
Similar but distinct themes, based on
the centrality of embodiment, have
emerged in some of the other cognitive
sciences, including psychology [9],
neuroscience [4,7] and philosophy [3,5].
The embodiment movement is
characterized by the six ‘E’s. The focus
is on Embodied, Embedded, Enactive,
Extended, Emergent and Evolving
intelligence. An embodied agent has
a physical body. A situated agent is
embedded in a particular environment,
which may include other embodied
agents. Enactivism argues that cognition
arises through a dynamic interaction
between the agent and its environment.
Intelligence is not just in the controller
of the agent: it is extended into the
body and into its coupling with the
environment. Intelligence emerges
through the evolution of that coupling.
A robot is an artificial purposive
embodied agent. EAI emphasizes the
tight coupling of perception and action.
Indeed, often perception is action and
vice versa. It follows that robotics is the
ideal test domain for EAI. That was the
motivation behind the building of robot
soccer players as an Embodied
AI challenge [6]. The RoboCup challenge
has led to new experiments and
theories for embodied multiagent realtime learning, decision-making and action [8,10]. 
Embodiment can be seen as an
essential scientific requirement on the
path to intelligence. But it can also be
seen as an engineering requirement
in any application scenario that
requires real-world interaction, such
as a self-driving car or a factory robot.
The form of embodiment, such as, for
example, a humanoid robot versus
a non-humanoid, will offer differing
affordances to humans interacting with
the robot. 
**Current State & Trends** 
If an agent is passively observing the
world through, for example, text or
video, it cannot learn how to make
decisions and act for itself in the world.
Text sometimes contains explicit true
information about the world but it
does not contain the implicit mundane
knowledge that is assumed to be
shared common sense. An embodied
agent in the real world needs that
common sense [1] which can only come
from interaction. Similarly, passively
watching video does not allow the agent
to learn how it should act in the world.
In contrast to passive agents, which
typically learn correlational models,
embodied agents have the ability to
learn, test, and revise causal models of
the world. Embodiment is a sufficient
basis for achieving that ability but not
strictly necessary. 
Accordingly, currently there is a new
emphasis on robots learning with
reinforcement learning over very large
numbers of trials, in both simulated and
physical worlds. There is also good work
going on in adapting Large Language
Models (LLMs) to generate robot plans.
Another frontier involves inverting
forward probabilistic causal models
to infer causality for robots interacting
with a world, real or artificial. 
**Research Challenges** 
There are many other open research
questions and challenges. Can an
embodied agent be trained purely
end-to-end successfully using current
techniques? Do we require a new
synthesis of AI and control theory to
make progress? Can existing pretrained language and/or vision models
be leveraged to improve embodied
cognition? Can simulators and world
models be made that are realistic
enough to train entirely (or mostly) in
simulation? Or, are simulated agents
“doomed to succeed”? How can
formal methods be used to prove that
an embodied agent (almost always)
achieves its goals without violating
safety constraints? 
We are not yet able to build an intelligent situated agent with humanlevel performance across a broad range of tasks, but we may have some (or most?) of the building blocks required to develop one. The main challenge is coping with the realities of
the world. So far, there seem to be no intrinsic obstacles to building intelligent embodied agents capable of humanlevel performance or beyond. 
**Community Opinion** 
The Community Survey gives
perspectives on the reactions to the
Embodied AI (EAI) theme. First, the
results of the survey are summarized
here. 31% of the survey respondents
chose to answer the questions for this
theme. This is the summary breakdown
of the responses to each question: 
**1. How relevant is this Theme for your
own research?** 74% of respondents
said it was somewhat relevant (27%),
relevant (25%) or very relevant (22%) 
**2. Is embodiment important for
the future of AI research?** 75% of
respondents agreed (43%) or strongly
agreed (32%) 
**3. Does embodied AI research require
robotics or can it be done in simulated
worlds?** 72% said that robotics is useful
(52%) or robotics is essential (20%). 
**4. Is artificial evolution a promising
route to realizing embodied AI?** 35%
agreed (28%) or strongly agreed (7%)
with that statement. 
**5. Is it helpful to learn about
embodiment concepts in the
psychological, neuroscience or
philosophical literature to develop
embodied AI?** 80% agreed (50%)
or strongly agreed (30%) with that
statement. 
Since the respondents to this theme
are self-selected (about a third of all
respondents), that bias must be kept in
mind. Nevertheless, it is significant that
about three-quarters felt that EAI is
relevant to their research, and a similar
fraction agreed on its importance for
future research. Moreover, a similar
fraction view robotics (contrasted
with simulation) as useful or essential
for EAI. Only a third viewed artificial
evolution as a promising route to EAI.
However, there is a strong consensus
that the cognitive sciences related
to AI have important insights useful
for developing EAI. Overall, these
results give us a unique perspective
on the future of Embodied Artificial
Intelligence research. 
1. Brachman, R. J. and Levesque, H. J. [2022]. Machines like Us: Toward AI with Common Sense. MIT Press.
2. Brooks, R. A. [1991]. Intelligence without representation. Artificial Intelligence, 47:139–159.
3. Clark, Andy. [2010] Supersizing the mind: Embodiment, action, and cognitive extension. Oxford University Press.
4. Damasio, Antonio [2021]. Feeling & knowing: making minds conscious. New York: Pantheon Books.
5. Di Paolo, Ezequiel A., and Evan Thompson. “The Enactive Approach.” The Routledge Handbook of Embodied Cognition. Routledge, 2024. 85-97.
6. Mackworth, A. K. [1993]. On seeing robots. In Basu, A. and Li, X. (eds.), Computer Vision: Systems, Theory, and Applications, pp. 1–13. World Scientific Press.
7. Shanahan, Murray. [2010] Embodiment and the Inner Life - Cognition and Consciousness in the Space Of Possible Minds. Oxford University Press.
8. Stone, P. [2007]. Learning and multiagent reasoning for autonomous agents. In The 20th International Joint Conference on Artificial Intelligence (IJCAI- 07), pp. 13–30. http://www.
cs.utexas.edu/∼pstone/Papers/bib2html-links/IJCAI07-award.pdf.
9. Varela, Francisco J., Evan Thompson, and Eleanor Rosch. The embodied mind, revised edition: Cognitive science and human experience. MIT press, 2017.
10. Visser, U. and Burkhard, H.-D. [2007]. Robocup: 10 years of achievements and challenges. AI Magazine, 28(2):115–130.

Embodied AI

AI has much to learn from other areas in cognitive science, and
can in turn contribute much to them. 
**Main Takeaways**
* Cognitive Science is a multidisciplinary field that was inspired by AI’s
exploration of the hypothesis of computation as a scientific language for
understanding cognition.
* Some continued interactions between AI and other areas in cognitive science
have yielded valuable insights and systems, notably cognitive architecture.
* Expanding these interactions could yield important advances for both fields. 

**Context & History** 
AI was the first field founded on
the intellectual hypothesis that
computation could become a scientific
language for understanding the nature
of intelligence, no matter what the
substrate. Cognitive Science was the
second, a multidisciplinary gathering
of researchers in AI, psychology,
linguistics, neuroscience, anthropology,
and other disciplines. Computational
ideas from AI were highly influential
in early cognitive science. However,
over time, AI has drifted apart from the
rest of cognitive science, for a variety
of reasons [3]. We believe that there
are now important benefits to be
gained from rebuilding those bridges
and exploring how progress in AI can
help understand human cognition (and
animal cognition more broadly) and
how progress in other areas of cognitive
science can help us build better AI
systems. In some cases, this will be
learning how to achieve in software the
kinds of cognitive capabilities organisms
have, and in other cases, deliberately
choosing to be different in ways that
complement human cognition so that
human-AI teams are more productive. 
**Current State & Trends** 
Cognitive Science is broad, so we focus
on three areas where research is likely
to be synergistic with AI. 
**Human-like learning and reasoning.**
Many animals learn, but humans are
pre-eminent learners and reasoners
in many ways. A surprising amount
of human learning is, in machine
learning terms, incremental, continual,
and data-efficient, often producing
articulable models (e.g. Gentner
& Maravilla, 2018). While today’s industrial knowledge graphs reach into
the tens of billions of facts, they lack the
expressiveness of human conceptual
structure [4]. Today’s reasoning systems,
like SAT solvers and model checkers,
are often superhuman in the size
of the problems they address and
the complexity of the solutions they
generate [2]. But today’s AI reasoning
systems cannot reason robustly with
incomplete and partially incorrect
domain theories, nor can they reason
from large bodies of experience as
people do. 
**Cognitive architectures** are systems
that explore hypotheses about the fixed
structures that define the processes
and representations used for cognition
[7]. They are used to investigate how
to build AI systems that do real-time
integration of perception, cognition, and
motor control across many tasks, and to
better understand human intelligence.
For example, cognitive architectures
have been used to simulate findings
(and make predictions) from cognitive
psychology and cognitive neuroscience
(e.g. [1,8]). While every cognitive
architecture involves multiple
processes and representations, they
vary considerably in the subset of
human cognition they explore and the
granularity of assumptions made. 
**Social agents.** One of the signature
properties of humans is that we
construct and live in a world of
collaboration where we learn about
each other and culturally-specific social
norms through interaction with others.
Progress in understanding how to build
social agents is essential to building
AI systems that live in our world as
collaborators and partners [9]. Social
AI is often developed independently
of findings and theories from social
science and learns social behavior quite differently from how people acquire social skills. 
**Research Challenges** 
Progress on these challenges will
lead to more adaptable AI systems
and reduce the computational and
environmental burdens of our systems,
better understand human cognition,
including social cognition, and provide
better tools for thought. 
**Human-like Learning and Reasoning** 
1. How can we develop human-like
incremental, data-efficient learning
methods that can produce articulable
models?
2. Develop formal ontologies that
span the range of human conceptual
structures, both concerning abstract
concepts and sensory-motor grounded
concepts.
3. How can AI systems robustly reason
with incomplete and partially incorrect
domain theories, and use experience in
reasoning, with human-scale bodies of
knowledge? 
**Cognitive Architectures** 
1. Expanding the higher-level cognitive
capabilities of cognitive architectures
to include the dynamical integration
of the full range of human capabilities
in response to task demands: diverse
forms of reasoning, metacognition,
online, lifelong continual learning across
modalities and knowledge types, and
engaging in ongoing human interaction
(e.g.[6]). These capabilities will require
learning and reasoning over models of
the physical world, abstractions, and
other agents using symbolic relational
and modality-specific representations of the current, future, and past
situations. 
2. Exploring the integration of
foundation models within cognitive
architectures, including sources for
knowledge and to interpret/generate
natural modalities (e.g. [10]). Can
the incremental learning capabilities
exhibited by cognitive architectures
overcome the limitations of stale
information in foundation models? 
3. Developing a comprehensive
benchmark task suite to evaluate the
breadth and integration of human
cognitive capabilities in end-to-end
performance as described above.
The tasks should be diverse and
broad to ensure robust assessments.
Additionally, the tasks should
be diagnostic, isolating cognitive
capabilities and their interactions to
provide insights into specific strengths
and weaknesses. 
**Social Agents** 
1. Facilitate learning through interaction:
The current generation of AI systems
learn by passively observing social
behavior rather than participating
in social behavior (analogous to the
distinction between decision theory
and game theory). In contrast, people
continuously co-construct behavior by mutually adapting to each other. At best, AI systems simulate interaction
by training on frozen simulated users
(e.g., RLHF), but this fails to account
for mutual adaptation. Thus, we need
research into ways to support (or
simulate) interactive learning at scale. 
2. Facilitate Privacy-preserving methods
to acquire social data: Human social
cues (face, voice) typically reveal the
identity of the social actor. Interpreting
social cues requires acquiring intrusive
situational information (e.g., the
meaning of a smile depends on not
just the face of the target person but
who else is in the situation, what they
are doing, the nature of the physical
environment, etc.). The ability to collect
this information is wisely restricted
by law (e.g., the EU’s AI act). Yet this
dramatically restricts the ability to
acquire data and deploy applications.
How do we create algorithms that
identify socially meaningful information
while proving that no future algorithm
could recover privacy/anonymity violating information from what is
stored? Developing methods that can
collect yet provably de-identify social
data is crucial for the advancement of
social agents. 
3. Developing interactional
benchmarks: Given that AI aspires
to build systems with general social
capabilities, we need a reliable way
to measure and assess if new models
are improvements. This includes
characterizing potential for bias, issues
of value alignment, whether the model
is willing to engage in deception, etc.
People now propose ad hoc collections
of tasks, but research is needed to
develop a comprehensive taxonomy
of tasks and measures. In contrast, the
social sciences have developed theorybased ontologies for characterizing
social situations. Research is needed to
translate these findings into systematic
and comprehensive benchmarks
of human social and interactional
behavior. 
**Community Opinion** 
Engagement with other areas of cognitive science varies across
the survey respondents, with 30% responding to the questions in this
section. Among those who responded, in terms of influence on their research, 18% said always, 32% said usually, 32% said sometimes. Only 2.8% said never and 12% said rarely. Thus 82% are influenced to a reasonable degree by research in other areas of cognitive science. Which other areas provide the most influence? Our respondents report psychology (82%), Neuroscience (44%), Linguistics (40%), and Anthropology (22%), with 13% mentioning other fields, Philosophy
being the most common. In terms of what issues are most relevant, a
broad set of creative responses were produced, some quite creative, e.g.
studying how minds completely different to our own might function. 
1. Anderson, J. R. (2007). How can the human mind exist in the physical universe? New York, NY: Oxford University Press.
2. Biere, A., Heule, M., et al. (2021) Handbook of Satisfiability, 2nd Edition, IOS Press
3. Forbus, K. (2010). AI and Cognitive Science: The Past and Next 30 years. Topics in Cognitive Science, 2(3), p. 346-356, https://doi.org/10.1111/j.1756-8765.2010.01083.x
4. Forbus, K. (2021). Evaluating revolutions in artificial intelligence from a human perspective. In OECD, AI and the Future of Skills, Volume 1: Capabilities and Assessments, OECD
Publishing, Paris. DOI:https://doi.org/10.1787/004710fe-en
5. Gentner, D. & Maravilla, F. (2018). Analogical reasoning. L. J. Ball & V. A. Thompson (eds.) International Handbook of Thinking & Reasoning (pp. 186-203). NY, NY: Psychology Press.
6. Gluck, K. & Laird, J. (2019) Interactive Task Learning: Humans, Robots, and Agents Acquiring New Tasks through Natural Interactions. MIT Press.
7. Kotseruba, I., & Tsotsos, J. K. (2020). 40 years of cognitive architectures: Core cognitive abilities and practical applications. Artificial Intelligence Review, 53(1), 17–94. https://doi.
org/10.1007/s10462-018-9646-y
8. Laird, J. (2012) The Soar Cognitive Architecture, MIT Press.
9. Lugrin, Birgit; Pelachaud, Catherine; Traum, David (Ed.) (2021). The Handbook on Socially Interactive Agents, pp. 433–462, ACM, New York, NY, USA, 2021, ISBN: 978-1-4503-8720-0.
10. Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023). Cognitive architectures for language agents. Transactions on Machine Learning Research, arXiv preprint arXiv:2309.02427.

AI & Cognitive Science

AI for social good is a subdiscipline of AI research where
measurable societal impact, particularly for vulnerable and
under-resourced groups, is a primary objective, focusing on
areas that have historically lacked sufficient AI research and
development. 
**Main Takeaways**
* **Ethical and Impact-Driven AI Development** – AI4SG projects have grown significantly in the past decade spurred by advancements in AI and ML, and they have prioritized ethical considerations, fairness, and societal benefits, ensuring
solutions address real-world problems in a responsible manner.
* **Interdisciplinary Collaboration is Crucial** – Successful AI4SG initiatives require strong partnerships between AI researchers, domain experts, policymakers, and local communities to ensure relevance and long-term sustainability.
* **Scalability and Sustainability Challenges** – While AI4SG has demonstrated significant potential, maintaining and scaling solutions in resource-constrained environments remains a key challenge. 

**Context & History** 
 “AI for Social Impact” (AI4SI / AI4SG)
has emerged as a distinct sub-discipline
within AI, characterized by its focus on
measurable societal impact, particularly
for vulnerable and underserved
populations. Unlike traditional AI
research, which often prioritizes
methodological advancements, AI4SI
places direct social impact as a primary
objective. It addresses problems that
have historically lacked sufficient
attention in AI research, aiming to
bridge the gap between AI capabilities
and real-world societal challenges such
as poverty, agriculture, public health,
and environmental conservation. The
goal is to create impactful solutions
tailored to real-world problems, often
in resource-constrained environments. 
AI4SI research necessitates deep
engagement with domain experts
and community members to identify
relevant problems, design effective
interventions, and rigorously evaluate
their impact. This interdisciplinary
approach draws insights from fields like
human-computer interaction, public
health, and social work, emphasizing
the importance of understanding
and addressing the specific needs
of targeted communities. These
projects require a balance of technical
innovation, ethical considerations, and
practical feasibility. 
The “AI for Social Good” workshop
organized by the White House Office
of Science and Technology Policy in
2016 may be credited as the single
event that sparked a significant
interest in this topic [1]. It ended up
unifying diverse efforts focused on
social impact under one umbrella
emerging field. This surge in interest is
attributed to several key factors. First,
the remarkable advancements in AI
technologies, including deep learning,
natural language processing, and
reinforcement learning, have provided
powerful tools applicable to a wide
range of social issues. The availability of
increased computing power and large
datasets has further accelerated this
progress, enabling the development of
sophisticated AI models. 
Secondly, the establishment of
government and industry-supported
funding programs, dedicated
workshops, conferences, and special
tracks within major AI conferences has
increased awareness and attracted
researchers to AI4SG[2,3]. This
increasing academic focus has led to a
substantial rise in publications related
to AI4SI, demonstrating the field’s
growing maturity[4]. 
**Current State & Trends** 
Advancing interdisciplinary
collaboration is becoming the norm in
AI for Social Good work. This necessity
stems from the complex nature of
societal challenges, which often require
insights from diverse fields. Close
partnerships with domain experts,
local practitioners, and policymakers
ensure that AI solutions are not only
technically sound but also relevant
and effective in the real world. This
collaborative approach fosters a shared
understanding of the problem and
enables the development of solutions
that are tailored to the specific needs of
the communities they serve. 
Emphasizing ethical AI is also critical,
with fairness, transparency, and privacy
as paramount considerations. Major
concerns, such as bias in data collection
and the unintended consequences of
AI deployment, must be addressed
from the outset to ensure that AI
systems align with societal values and
avoid harm. AI4SG projects, by their
very nature, work with vulnerable
populations and sensitive data, making
ethical considerations even more
crucial. By proactively addressing
potential ethical issues, researchers
can build trust with communities and
ensure that AI is used for good, rather
than exacerbating existing inequalities. 
Several emerging opportunities are
shaping the future of AI4SG. Firstly,
leveraging cloud-based platforms for
scalable AI deployments, allows for
wider reach and impact. Cloud-based
solutions enable the deployment of AI
tools to remote or resource-constrained
areas, democratizing access to AI
technologies. Secondly, incorporating
explainability and transparency in AI
solutions is crucial for gaining trust
from practitioners and beneficiaries,
particularly in high-stakes domains
like disaster response and public
health. When AI systems can explain
their reasoning and decision-making
processes, they are more likely to be
accepted and used effectively. Thirdly,
emphasizing localized AI4SI solutions
that can be sustainably maintained by
end users fosters long-term impact
and community ownership. By
empowering local communities
to manage and maintain AI tools,
AI4SG projects can ensure that their
benefits continue long after the initial
development phase. Finally, exploiting
available foundation models presents
a significant opportunity to accelerate
the development and deployment of
AI4SG applications. These pre-trained
models can serve as a starting point
for developing AI solutions for specific
social problems, reducing the time and
resources required for development [5]. 
**Research Challenges** 
A significant challenge facing AI for
Social Good research revolves around
the design of AI systems that are
not only technically effective but
also deeply contextually relevant
within social impact settings. This
requires a nuanced understanding
of the specific needs, cultural
sensitivities, and practical constraints
of the communities being served.
Researchers must move beyond purely
algorithmic considerations and engage
in participatory design processes that
prioritize the voices and experiences of
end-users, ensuring that AI solutions
are truly aligned with their real-world
needs and challenges. 
Another critical hurdle lies in
overcoming the limitations of data. AI
for Social Good projects frequently
grapple with scarce, low-quality, or
biased data, which can significantly
impact the performance and fairness
of AI models. Developing robust
data collection strategies, employing
techniques for data augmentation
and bias mitigation, and exploring
alternative data sources are essential
for building reliable and equitable AI
systems. This also requires a careful
consideration of the cultural context in
which data is gathered, ensuring that AI
data collection methods are adapted
to local practices, rather than imposing
external standards. 
Ensuring the sustainability and
scalability of AI deployments in
resource-constrained environments
presents a further complex challenge.
Beyond the initial prototype phase,
the long-term viability of AI solutions
depends on their ability to be
maintained and scaled by organizations
with limited resources, such as NGOs
and government agencies. This
necessitates the development of
sustainable software architectures, the
creation of user-friendly interfaces,
and the provision of adequate training
and support for local stakeholders.
Moreover, funding these efforts, often
because of the lack of commercial
viability, is in itself a major concern. 
Additionally, robust evaluation
frameworks are crucial for assessing the
impact of AI solutions in field settings
and building stakeholder trust. These
frameworks must go beyond traditional
performance metrics and incorporate
measures of social impact, user
satisfaction, and ethical considerations.
The accessibility community’s mantra,
“Nothing about us without us,” [6]
serves as a powerful reminder of the
importance of involving stakeholders
in all stages of the evaluation process.
Furthermore, we must be vigilant
against the potential for corporate
ethics-washing or green-washing,
ensuring that AI initiatives are genuinely
driven by a commitment to social good,
rather than serving as mere public
relations exercises. 
Finally, there is currently also a gap
between traditional AI education
and the specific skills required for
impactful social work. Standard
AI curricula primarily focus on
algorithm design and analysis, often
emphasizing theoretical concepts and
performance on benchmark datasets.
This approach, while essential for
advancing core AI methodologies,
leaves students ill-equipped to address
the complex, real-world problems
that AI4SG tackles. Effective AI4SG
research demands a broader skillset,
extending beyond purely technical
expertise. It requires the ability to
collaborate effectively with domain
experts, such as public health officials,
environmental scientists, or social
workers, and to engage meaningfully
with community members whose lives
are directly affected by the technology.
Understanding the nuanced socioeconomic and cultural contexts of social challenges, and translating
technical advancements into practical,
user-centered interventions, are crucial
competencies. 
**Community Opinion** 
The AAAI Community Survey on
the Future of AI research, with 475
responses, explored various aspects
of AI, particularly its application to
social good. A significant portion
of respondents (119) engaged with
questions related to AI for social good. 
Regarding the relevance of AI for
Social Good, a majority of the 119
respondents who answered this
question found it relevant or very
relevant to their research. Specifically,
33.61% considered it very relevant,
26.89% relevant, and 21.01% somewhat
relevant. When asked about barriers
to integrating AI into social impact
projects, respondents cited several
challenges. The most significant
barrier, with 47.06% of respondents
selecting it, was “Testing the solution
in the field.” Other substantial barriers
included “Scaling up the solution”
(38.66%), “Connecting to non-profit or
government organizations” (32.77%),
“Problem definition” (36.13%), and
“Assessing AI readiness” (30.25%).
“Sustainability of the business model”
was also a concern for 42.86% of
respondents. 
The survey also explored crucial
resources for scaling AI-driven solutions
for social impact. “Money” and
“Data” were identified as extremely
important by a large majority of
respondents (the exact percentages
are cut off in the provided snippets).
Other important resources included
“Government-companies partnerships,”
“Government-university partnerships,”
and “Technical support.” Finally,
regarding measuring the success of
AI interventions in addressing social
challenges, “Improvement outcome”
was the most frequently cited metric,
with 47.90% of the 119 respondents
choosing it. “Sustainable results” were
also considered important by 37.82%,
and “Adoption rate” by 48.74%. 

1. United Nations. (2015). Transforming our world: the 2030 Agenda for Sustainable Development. https://sdgs.un.org/2030agenda
2. White House Office of Science and Technology Policy Workshop on AI for Social Good 2016. https://cra.org/ccc/events/ai-social-good/
3. AAAI Conference Call for the Special Track on AI for Social Impact, 2024 https://aaai.org/aaai-24-conference/call-for-the-special-track-on-ai-for-social-impact/
4. IJCAI conference Call For Papers And Projects: Multi-Year Track On AI And Social Good https://2025.ijcai.org/call-for-papers-and-projects-multi-year-track-on-ai-and-social-goodspecial-track/
5. Shi, Z., Wang, C., Fang, F. “AI for social good: A survey”, 2020. https://arxiv.org/abs/2001.01818
6. United Nations Division for Social Inclusive Development “AI for Good Impact Report” https://aiforgood.itu.int/newsroom/publications-and-reports/
7. Zhao, Y., Boehmer, N., Taneja, A., Tambe, M. Towards Foundation-model-based Multiagent System to Accelerate AI for Social Impact, AAMAS 2025
8. Charlton, J. I. Nothing About Us Without Us: Disability Oppression and Empowerment. University of California Press, 1998 http://www.jstor.org/stable/10.1525/j.ctt1pnqn9

AI for Social Good

AI & Sustainability

Artificial Intelligence (AI) is revolutionizing scientific discovery
by accelerating the entire research cycle from knowledge
extraction and hypothesis generation to automation of
experimentation and verification at an unprecedented speed. 
**Main Takeaways**
* Progress of AI for Scientific Discovery will accelerate the pace of scientific
discovery in unprecedented ways and transform the way we do science.
* Highly automated AI systems for scientific discovery are emerging: While their capabilities are limited, they can execute the entire scientific discovery cycle with minimal human intervention for defined tasks.
* AI’s role in scientific discovery raises new challenges in ethics, collaboration, and reliability, requiring interdisciplinary efforts to address them. 

**Context & History** 
Scientific discovery has historically
been driven by human ingenuity, but
Artificial Intelligence (AI) is emerging as
a pivotal tool in reshaping this process
and accelerating it. 
Early AI systems, such as DENDRAL,
designed in the 1960s, were among the
first to automate hypothesis generation
and problem-solving in organic
chemistry [1]. Similarly, systems like
EURISKO explored heuristic learning
and adaptation, demonstrating the
potential of AI in creative problemsolving [2]. 
These pioneering efforts laid the
groundwork for AI’s role in science
by showcasing its ability to process
datasets and formulate hypotheses. 
With the progress of technology for
comprehensive and high-precision
measurements, scientists are
overwhelmed by the massive data
generated and the complexity of
systems behind it. Technologies to
discover patterns from massive data
that can be linked to novel hypotheses
to be tested by experiments are
essential for scientific research. AI for
Scientific Discovery is a much-needed
research area and is expected to drive
practice of science in a data-driven
approach. 
There is a spectrum of approaches in
how AI transforms scientific discovery.
First, we envision increasingly powerful
AI tools that will assist human scientists
to make discoveries faster and enable
them to tackle even more challenging
problems. The alternative approach is
to develop integrated AI and robotics
systems aiming at performing the
entire cycle of scientific research
with minimum interventions from
human scientists. The middle ground
is to position AI as a collaborator
with scientists that can interactively
solve scientific problems. AI for
Scientific Discovery embraces a wider
spectrum of relationship between AI
and scientists, and rapid progress is
expected in all ranges of this spectrum
that will fundamentally transform the
way we do science. A series of papers,
reports and workshops concluded
that AI for Science is one of the most
important research areas in coming
years[3–6]. 
**Current State & Trends** 
**1. From Tools to AI Collaborators and
Autonomous AI Scientists** 
AI’s capability in supporting scientific discovery has expanded dramatically,
with systems like AlphaFold2 from DeepMind achieving groundbreaking
success in structural biology[7,8]. AlphaFold2 solved the decadesold problem of protein folding, revolutionizing molecular biology
and enabling applications in drug discovery and biomedicine. AlphaFold2
represents the most successful case of AI as a tool for scientific discovery
that transforms biomedicine and biochemistry research and resulted
in a 2024 Nobel Prize in Chemistry. An increasing number of AI/robotics
systems have been developed to be effective tools for scientists. Beyond
biology, numerous AI tools have been developed for chemistry[9–11], material science[12], mathematics[13–15] and many other scientific fields to
accelerate scientific discovery. For example, the Ramanujan Machine is
an early effort to generate conjectures on fundamental constants that can be a tool for mathematicians undertaking a specific task in mathematics[13]. AI and robotics integrated systems have been developed that perform a specific type of chemistry experiment automatically[16,17]. 
The other end of the spectrum is
to develop highly autonomous AI/
robotics systems to perform scientific
discovery without (or with minimum)
human interventions. The Adam and
Eve systems, developed by Ross King,
represent the early effort on the other
side of the spectrum where scientific
discovery occurs without human-inthe-loop[18,19]. These ‘robot scientists’
not only generate hypotheses but also
design and execute experiments to
test them. For example, Eve identified
potential drugs for malaria through
automated high-throughput screening,
showcasing AI’s ability to iterate
scientific cycles autonomously[20]. The
Nobel Turing Challenge represents the
extreme end of the challenge gearing
toward highly autonomous AI and
robotics systems with the capability for
high impact scientific discovery[21,22].
Proposed as a grand vision, it aims to
develop AI systems capable of scientific
discoveries on par with Nobel Prizewinning work. This challenge entails an
instance of the Feigenbaum test - Can
I replicate the best human expert in
defined domains? - that is a variation
of the Turing test[23]. These systems
would not merely assist researchers
but act as autonomous entities capable
of proposing, testing, and refining
theories. 
**2. Impact on Science** 
AI’s integration into scientific workflows
heralds a paradigm shift:
* **Accelerated Discovery:** By
automating an entire cycle of scientific discovery, AI can reduce
the time required to achieve
breakthroughs, enabling a rapid
expansion of knowledge.
* **Enhanced Collaboration:** AI systems
like AlphaFold demonstrate how
interdisciplinary approaches—
combining AI, biology, and
physics—can tackle long standing
challenges. Eventually, a network of
collaboration may be formed among
AI systems enabling extensive
exchange of ideas and data at
the scale not possible by human
scientists.3. Exploration
* **Beyond Human Intuition:** AI’s ability to exhaustively explore hypothesis spaces enables discoveries that might elude human researchers
constrained by cognitive and methodological biases.
* **Transformation in Data Handling:**
An AI-centric approach dramatically changes the way researchers handle
experimental data. In the AI-centric approach all data is important,
not just the subset that strongly supports the expected outcome, but
also data that are not consistent with the expectation because most, if not
all, data need to be provided to AI system training for better hypothesis
generation and prediction. 

**3. Social and Ethical Implications** 
AI-driven science will likely have profound societal impacts:
* **Healthcare Transformation:** With
AI accelerating drug discovery and
personalized medicine, millions of
lives could be saved or improved.
* **Environmental and Climate Change:**
Rapid progress of material science,
chemistry, and environmental
sciences may provide us with
discoveries that can be used to
mitigate climate change and improve
the state of the environment.
* **Ethical Considerations:** The
autonomy of AI systems raises
questions about accountability,
credit for discoveries, and the
potential displacement of human
researchers. There are concerns
that hazardous materials may be
designed and produced with highly
autonomous AI scientists. Ethical
and safety measures shall be taken
to prevent malevolent uses of
powerful synthesis abilities with
materials and biology[24]. Proper
measures must be taken to prevent
such uses. 

**Research Challenges** 
1. Communication issues: A major
challenge may be to be able to
understand and communicate with
human scientists because accumulation
of knowledge and communications with
peers mostly takes the form of natural
languages with ample ambiguities,
analogies, and often with cultural
context. Of particular importance:
* **Commonsense knowledge:**
Professional knowledge is grounded
in everyday knowledge. Construction
of a shared broad conceptual
knowledge base of the world, to
ground reasoning and models and
provide grist for analogies.
* **Collaboration:** Science is often
a team sport, being able to work
effectively with others (human or AI)
will be important.
* **Communication:** Scientists talk,
draw, and use multiple forms of
interactive media. AIs for science
will need to be able to read and
understand the scientific literature
and communicate well with human
partners.
* **Models of scientific reasoning and
approaches** to developing modeling
and inference machinery that can
augment human cognition for
scientific discovery[25]. 

2. Defining Hypothesis Space: Science
is an open-ended problem: Unlike
most games, such as chess or the game
of GO, the structure and scale of the
problem space is not obvious and likely
to be unbounded and could be with
very high dimensionality. Extracting
or acquiring knowledge and properly
placing it within the space of scientific
knowledge is a non-trivial problem.
Similarly, the process of hypothesis
generation faces the problem of
identifying the dimensionality and scale
of hypothesis space it should work on. 
3. Inaccuracies, noise, and
reproducibility of data: Data in science
can be very noisy, inaccurate and
not reproducible in some fields. In
biology, inaccuracy and noise in data
is considered to be inevitable due to
artifacts introduced in experiments and
with intrinsic variability of experimental
samples. Studies revealed that a
significant proportion of data reported
in publications cannot be reproduced
properly in biomedical research[26,
27]. This may pose problems for the
quality of hypotheses generated or their
verification process at the early stage of
the research. 
**Community Opinion** 
In the community opinion survey, 32%
of those responding consider this is
somewhat relevant. The area of study
that AI to be most useful is ranked as
(1) Biology (47%), (2) Physics (14%), (3)
Chemistry (12%), with 26% responding
for other domains. In the question of if
an AI system can ever make discovery
worthy of the Nobel Prize, only 13% of
those responding said never, with 25%
saying no idea. 11% thought it might
happen in the 2020s, and 45% thought
it might happen by the 2050s. 

1. Lindsay R, Buchanan B, Feigenbaum E, Lederberg J. DENDRAL: A Case Study of the First Expert System for Scientific Hypothesis Formation. Artif Intell. 1993;61: 209–261.
2. Lenat D, Brown J. Why AM and EURISKO appear to work. Artif Intell. 1984;23: 269–294.
3. Wang H, Fu T, Du Y, Gao W, Huang K, Liu Z, et al. Scientific discovery in the age of artificial intelligence. Nature. 2023;620: 47–60.
4. Science and Engineering Capacity Development Unit, Computer Science and Telecommunications Board, Policy and Global Affairs, Division on Engineering and Physical Sciences,
National Academies of Sciences, Engineering, and Medicine. AI for scientific discovery: Proceedings of a workshop. Pool R, editor. Washington, D.C.: National Academies Press; 2024.
doi:10.17226/27457
5. Gil Y, Greaves M, Hendler J, Hirsh H. Artificial Intelligence. Amplify scientific discovery with artificial intelligence. Science. 2014;346: 171–172.
6. President’s Council of Advisors on Science and Technology. PCAST: Report to the president on supercharging research: Harnessing artificial intelligence to meet global challenges.
Office of Scientific and Technical Information (OSTI); 2024 Apr. doi:10.2172/2481685
7. Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green T, et al. Improved protein structure prediction using potentials from deep learning. Nature. 2020;577: 706–710.
8. Jumper J, Evans R, Pritzel A, Green T, Figurnov M, Ronneberger O, et al. Highly accurate protein structure prediction with AlphaFold. Nature. 2021;596: 583–589.
9. Peplow M. ChatGPT for chemistry: AI and robots join forces to build new materials. Nature. 2023 [cited 30 Nov 2024]. doi:10.1038/d41586-023-03745-5
10. Dias AL, Rodrigues T. Large language models direct automated chemistry laboratory. Nature. 2023;624: 530–531.
11. Boiko DA, MacKnight R, Kline B, Gomes G. Autonomous chemical research with large language models. Nature. 2023;624: 570–578.
12. Manica M, Born J, Cadow J, Christofidellis D, Dave A, Clarke D, et al. Accelerating material design with the generative toolkit for scientific discovery. Npj Comput Mater. 2023;9: 1–6
13. Raayoni G, Gottlieb S, Manor Y, Pisha G, Harris Y, Mendlovic U, et al. Generating conjectures on fundamental constants with the Ramanujan Machine. Nature. 2021;590: 67–73.
14. Davies A, Veličković P, Buesing L, Blackwell S, Zheng D, Tomašev N, et al. Advancing mathematics by guiding human intuition with AI. Nature. 2021;600: 70–74.
15. He Y-H. AI-driven research in pure mathematics and theoretical physics. Nature Reviews Physics. 2024; 1–8.
16. Coley CW, ThomasIII DA, Lummiss JAM, Jaworski JN, Breen CP, Schultz V, et al. A robotic platform for flow synthesis of organic compounds informed by AI planning. Science. 2019;365.
doi:10.1126/science.aax1566
17. Burger B, Maffettone PM, Gusev VV, Aitchison CM, Bai Y, Wang X, et al. A mobile robotic chemist. Nature. 2020;583: 237–241.
18. King RD, Rowland J, Oliver SG, Young M, Aubrey W, Byrne E, et al. The Automation of Science. Science. 2009;324: 85–89.
19. King RD, Whelan KE, Jones FM, Reiser PG, Bryant CH, Muggleton SH, et al. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature. 2004;427:
247–252.
20.Williams K, Bilsland E, Sparkes A, Aubrey W, Young M, Soldatova LN, et al. Cheaper faster drug development validated by the repositioning of drugs against neglected tropical diseases.
J R Soc Interface. 2015;12: 20141289.
21. Kitano H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Systems Biology and Applications. 2021;7: 29.
22.Kitano - 2016 - Artificial Intelligence to Win the Nobel Prize and Beyond Creating the Engine for Scientific Discovery, AI Magazine, 37:1 2016.
23.Feigenbaum, Edward A. (2003). “Some challenges and grand challenges for computational intelligence”. Journal of the ACM. 50 (1): 32–40. doi:10.1145/602382.602400. S2CID 15379263
24.Wittmann BJ, Alexanian T, Bartling C, Beal J, Clore A, Diggans J, et al. Toward AI-resilient screening of nucleic acid synthesis orders: Process, results, and recommendations. bioRxiv.
2024. p. 2024.12.02.626439. doi:10.1101/2024.12.02.626439
25.Hope, T, Downey, D., Weld, D.S., Etzioni, O. and Horvitz, E. (2023). A Computational Inflection for Scientific Discovery. Communications of the ACM 66, 8 (August 2023), 62–73. https://
doi.org/10.1145/3576896
26.Prinz F, Schlange T, Asadullah K (August 2011). “Believe it or not: how much can we rely on published data on potential drug targets?”. Nature Reviews. Drug Discovery. 10 (9): 712.
doi:10.1038/nrd3439-c1. PMID 21892149.
27. Begley CG, Ellis LM (March 2012). “Drug development: Raise standards for preclinical cancer research”. Nature (Comment article). 483 (7391): 531–533. Bibcode:2012Natur.483..531B.
doi:10.1038/483531a

Next from The Future of AI Research - Webinar Series

AI agents

Similar lecture

Artificial General Intelligence (AGI)