New Zealand

André Filipe de Moraes Batista, Maria das Graças Bruno Marietto, Wagner Tanaka Botelho, Guiou Kobayashi, Brunno dos Passos Alves, Sidney de Castro and Terry Lima Ruas (April 1st 2011). Principles of Agent-Oriented Programming, Multi-Agent Systems - Modeling, Control, Programming, Simulations and Applications, Faisal Alkhateeb, Eslam Al Maghayreh and Iyad Abu Doush, IntechOpen, DOI: 10.5772/14248.

Reactive behaviour in BDI-based models and architectures adopted in Agent Programming is typically modelled in terms of reactive plans not bound to any specific goal. In this paper we present and discuss an extension of the plan model used in BDI in which goal-based plans encapsulate both proactive and reactive behaviour. This brings important benefits both in the practice of agent program- ming and in supporting agent reasoning at runtime. The paper first introduced the model formally and then its evaluation through two concrete implementations, based on two existing agent program- ming platforms i.e. Jason and ASTRA.

AAMAS 2020

Encapsulating Reactive Behaviour in Goal-based Plans in BDI Agents Programming

agentspeak(er)

astra

agentspeak(l)

agent programming languages

agent-oriented programming

jason

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

technical paper

We propose Stable Emergent Policy (STEP) approximation, a distributed policy iteration scheme to stably approximate decentralized policies for partially observable and cooperative multi-agent systems. STEP offers a novel training architecture, where function approximation is used to learn from action recommendations of a decentralized planning algorithm. Planning is enabled by exploiting a training simulator, which is assumed to be available during centralized learning, and further enhanced by reintegrating the learned policies.


A Distributed Policy Iteration Scheme for Cooperative Multi-Agent Policy Approximation

"Neural networks are effective function approximators, but hard to train in the reinforcement learning (RL) context mainly because samples are correlated. For years, scholars have got around this by employing experience replay or an asynchronous parallel-agent system. This paper proposes Discrete-to-Deep Supervised Policy Learning (D2D-SPL) for training neural networks in RL. D2D-SPL discretises the continuous state space into discrete states and uses actor-critic to learn a policy. It then selects from each discrete state an input value and the action with the highest numerical preference as an input/target pair. Finally it uses input/target pairs from all discrete states to train a classifier. D2D-SPL uses a single agent, needs no experience replay and learns much faster than state-of-the-art methods. We test our method with two RL environments, the Cartpole and an aircraft manoeuvring simulator."


Discrete-to-Deep Supervised Policy Learning

Predictive auxiliary tasks have been shown to improve performance in numerous reinforcement learning works, however, this effect is still not well understood. The primary purpose of the work presented here is to investigate the impact that an auxiliary task's prediction timescale has on the agent's policy performance. We consider auxiliary tasks which learn to make on-policy predictions using temporal difference learning. We test the impact of prediction timescale using a specific form of auxiliary task in which the input image is used as the prediction target, which we refer to as temporal difference autoencoders (TD-AE). We empirically evaluate the effect of TD-AE on the A2C algorithm in the VizDoom environment using different prediction timescales. While we do not observe a clear relationship between the prediction timescale on performance, we make the following observations: 1) using auxiliary tasks allows us to reduce the trajectory length of the A2C algorithm, 2) in some cases temporally extended TD-AE performs better than a straight autoencoder, 3) performance with auxiliary tasks is sensitive to the weight placed on the auxiliary loss, 4) despite this sensitivity, auxiliary tasks improved performance without extensive hyper-parameter tuning. Our overall conclusions are that TD-AE increases the robustness of the A2C algorithm to the trajectory length and while promising, further study is required to fully understand the relationship between auxiliary task prediction timescale and the agent's performance.


Work in Progress: Temporally Extended Auxiliary Tasks

Describing an accurate simulation model of the driving behavior of real-world vehicles is a laborious or even impossible task, because a driver reacts to a dynamically changing environment. As multiple external factors determine driving behavior, it is usually difficult to obtain an accurate model, owing to a lack of sensors or inability to collect data. In this paper, we propose a novel technique to combine driving behavior in vehicular Internet of Things (IoT) big data with a multi-agent system. This enables correct and scalable simulation without modeling the behavior of vehicular IoT devices or the environment. We develop an extensible simulation framework, called FlowSim, that demonstrates the application of our technique for a simulation of camera-image data collection from connected cars.


Simulating Vehicular IoT Applications by Combining a Multi-agent System and Big Data

"Patients are often required to follow a medical treatment after discharge, e.g., for a chronic condition, rehabilitation after surgery, or for cancer survivor therapies. The need to adapt to new lifestyles, medication, and treatment routines, can produce an individual burden to the patient, who is often at home without the full support of healthcare professionals. Although technological solutions -in the form of mobile apps and wearables- have been proposed to mitigate these issues, it is essential to consider individual characteristics, preferences, and the context of a patient in order to oer personalized and effective support. The specific events and circumstances linked to an individual prole can be abstracted as a patient trajectory, which can contribute to a better understanding of the patient, her needs, and the most appropriate personalized support. Although patient trajectories have been studied for different illnesses and conditions, it remains challenging to effectively use them as the basis for data analytics methodologies in decentralized eHealth systems. In this work, we present a novel approach based on the multi-agent paradigm, considering patient trajectories as the cornerstone of a methodology for modelling eHealth support systems. In this design, semantic representations of individual treatment pathways are used in order to exchange patient-relevant information, potentially fed to AI systems for prediction and classification tasks. This paper describes the major challenges in this scope, as well as the design principles of the proposed agent-based architecture, including an example of its
use through a case scenario for cancer survivors support."


Agent-based modeling for Ontology-driven analysis of patient trajectories

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible.
In this work, we focus on learning to coordinate.
Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. 
We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks with Bernoulli-distributed rewards. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in applications with sparse neighbourhood structures.


Thompson Sampling for Factored Multi-Agent Bandits

"Reinforcement learning (RL) is a powerful learning paradigm in which agents can learn to maximize sparse and delayed reward signals. Although RL has had many impressive successes in complex domains, learning can take hours, days, or even years of training data. A major challenge of contemporary RL research is to discover how to learn with less data. Previous work has shown that domain information can be successfully used to shape the reward; by adding additional reward information, the agent can learn with much less data. Furthermore, if the reward is constructed from a potential function, the optimal policy is guaranteed to be unaltered. While such potential-based reward shaping (PBRS) holds promise, it is limited by the need for a well-defined potential function. Ideally, we would like to be able to take arbitrary advice from a human or other agent and improve performance without affecting the optimal policy. The recently introduced dynamic potential based advice (DPBA) method tackles this challenge by admitting arbitrary advice from a human or other agent and improves performance without affecting the optimal policy. The main contribution of this paper is to expose, theoretically and empirically, a flaw in DPBA. 
Alternatively, to achieve the ideal goals, we present a simple method called policy invariant explicit shaping (PIES) and show theoretically and empirically that PIES succeeds where DPBA fails."


Useful Policy Invariant Shaping from Arbitrary Advice

Reinforcement Learning (RL) has been widely used to solve sequential decision-making problems. However, RL algorithms suffer from poor sample efficiency and require a long time to learn a suitable policy, especially when multiple agents are learning without prior knowledge. This problem can be alleviated through reusing knowledge from other agents during the learning process. One notable approach is advising actions based on a teacher-student relationship, where the decision of a student agent during learning is aided by an experienced teacher agent. A critical assumption in teacher-student paradigm is that the communication may be limited, so that a student may wait for a while and learn by itself before receiving the next advice. More importantly, in some noisy or stochastic environments, the student may not be able to mas- ter the advised actions when they are only performed once. We propose three methods for agents choosing between learning by exploration, asking for advice and reusing previous advice. The results show that our approaches significantly outperform existing advising methods without reusing advice.


Learning by Reusing Previous Advice in Teacher-Student Paradigm

"This paper evaluates the applicability of reinforcement learning (RL) to multi-product inventory management in supply chains. The novelty of this problem with respect to supply chain literature is (i) we consider concurrent inventory management of a large number (hundreds) of products under realistic constraints such as shared capacity, and (ii) the number of products (size of the problem) can change frequently, implying that the RL agent needs to work in this regime without retraining. We approach the problem as a special class of dynamical system control, and explain why the generic problem cannot be satisfactorily solved using classical optimisation techniques. Subsequently, we formulate the problem in a
reinforcement learning framework that can be used for parallelised decision-making, and use the advantage actor critic (A2C) and deep Q-network (DQN) algorithms with quantised action spaces to solve the problem. Experiments on scales between 100 and 220 products
show that these approaches perform better than other baseline algorithms. They are also able to transfer learning without retraining, when the number of products change."

Using Reinforcement Learning for a Large Variable-Dimensional Inventory Management Problem

Recent advances in artificial intelligence (AI) have drawn attention to the need for AI systems to be understandable to human users. The explainable AI (XAI) literature aims to enhance human understanding and human-AI team performance by providing users with necessary information about AI system behavior. Simultaneously, the human factors literature has long addressed important considerations that contribute to human performance, including how to determine human informational needs.
 Drawing from the human factors literature, we propose a three-level framework for the development and evaluation of explanations about AI system behavior. Our proposed levels of XAI are based on the informational needs of human users, which can be determined using the levels of situation awareness (SA) framework from the human factors literature. Based on our levels of XAI framework, we also propose a method for assessing the effectiveness of XAI systems.

Downloads

Next from AAMAS 2020

A Distributed Policy Iteration Scheme for Cooperative Multi-Agent Policy Approximation

Similar lecture

Encapsulating Reacting Behaviour In Goal-Based Plans For Programming BDI Agents

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2020

A Distributed Policy Iteration Scheme for Cooperative Multi-Agent Policy Approximation

Similar lecture

Encapsulating Reacting Behaviour In Goal-Based Plans For Programming BDI Agents

Downloads