New Zealand

Kunikazu Kobayashi, Koji Nakano, Takashi Kuremoto and Masanao Obayashi (February 1st 2010). Objective-based Reinforcement Learning System for Cooperative Behavior Acquisition, Application of Machine Learning, Yagang Zhang, IntechOpen, DOI: 10.5772/8615. 

Daniela P. Alves, Li Weigang and Bueno B. Souza (January 1st 2008). Reinforcement Learning to Support Meta-Level Control in Air Traffic Management, Reinforcement Learning, Cornelius Weber, Mark Elshaw and Norbert Michael Mayer, IntechOpen, DOI: 10.5772/5293. 

Yoshiyuki Tajima and Takehisa Onisawa (February 1st 2010). Model-based Reinforcement Learning with Model Error and Its Application, Application of Machine Learning, Yagang Zhang, IntechOpen, DOI: 10.5772/8607. 

Advertising has become a dominant source of revenue generation on the Internet. Billions of advertisement slots are sold via auctions. And there are many pricing methods ,e.g., CPM (cost-per-mille), CPC (cost-per-click), CPA (cost-per-action), OCPM (optimized cost-per-mille) and so on. In this paper, we study the OCPM method (i.e., advertisers bid for conversions while pay per mille) under VCG auction. However, automatically bid in each view to maximize advertisers&#39; conversions while still meet their target cost-per-conversion in feeds is difficult. To deal with these difficulties, we propose a reinforcement learning framework, i.e., RSDRL (ROI-sensitive distributional reinforcement learning). By making full use of the characteristics of auction rules which are missed by other methods, we design a reward function to surrogate conversion events and a bid generation method based on theoretical results. We also provide some theoretical results to guide hyperparameter tuning. Last, we validate RSDRL on a large industrial dataset with millions of auctions. Plenty of experiments (both online and offline) are used to evaluate the performance of our framework and RSDRL yields substantially better results than compared algorithms.


AAMAS 2020

Optimized Cost per Mille in Feeds Advertising

ocpm

reinforcement learning

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

AAAI 2021

Advertising has become a dominant source of revenue generation on the Internet. Billions of advertisement slots are sold via auctions. And there are many pricing methods ,e.g., CPM (cost-per-mille), CPC (cost-per-click), CPA (cost-per-action), OCPM (optimized cost-per-mille) and so on. In this paper, we study the OCPM method (i.e., advertisers bid for conversions while pay per mille) under VCG auction. However, automatically bid in each view to maximize advertisers' conversions while still meet their target cost-per-conversion in feeds is difficult. To deal with these difficulties, we propose a reinforcement learning framework, i.e., RSDRL (ROI-sensitive distributional reinforcement learning). By making full use of the characteristics of auction rules which are missed by other methods, we design a reward function to surrogate conversion events and a bid generation method based on theoretical results. We also provide some theoretical results to guide hyperparameter tuning. Last, we validate RSDRL on a large industrial dataset with millions of auctions. Plenty of experiments (both online and offline) are used to evaluate the performance of our framework and RSDRL yields substantially better results than compared algorithms.


technical paper

AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). The aim of the joint conference is to provide a single, high-profile, internationally-respected archival forum for scientific research in the theory and practice of autonomous agents and multi-agent systems.

Browse keynotes, discussions, panels and over 300 presentations.


AAMAS is the leading scientific conference for research in autonomous agents and multi-agent systems. The AAMAS conference series was initiated in 2002 as the merging of three respected scientific meetings: the International Conference on Multi-Agent Systems (ICMAS), the International Workshop on Agent Theories, Architectures, and Languages (ATAL), and the International Conference on Autonomous Agents (AA). 

The plan recognition task is to infer an actor’s plan and goal given observations about its behavior. We submit that in some cases, for plan recognition to be effective and complete, it must appeal to a notion of epistemics to i) recognize epistemic goals, where the actor is trying to achieve some state of knowledge or belief; and ii) model the observer, and its knowledge of the actor, as first class elements of the plan recognition process. To this end, we formalize the notion of Epistemic Plan Recognition, which builds on two growing areas of research: epistemic planning and plan recognition. Our epistemic plan recognition specification appeals to an epistemic logic framework to represent agent beliefs. To realize our specification, we cast the epistemic plan recognition problem as an epistemic planning problem, whose solutions can be generated using existing epistemic planning tools. Finally, we evaluate our approach by utilizing and comparing existing epistemic planners on a diverse set of epistemic plan recognition problems.


Epistemic Plan Recognition

This paper focuses on two-sided matching where one side (a hospital or firm) is matched to the other side (a doctor or worker) so as to maximize a cardinal objective under general feasibility constraints. In a standard model, even though multiple doctors can be matched to a single hospital, a hospital has a responsive preference and a maximum quota. However, in practical applications, a hospital has some complicated cardinal preference and constraints. With such preferences (e.g., submodular) and constraints (e.g., knapsack or matroid intersection), stable matchings may fail to exist. This paper first determines the complexity of checking and computing stable matchings based on preference class and constraint class. Second, we establish a framework to analyze this problem on packing problems, and the framework enables us to access the wealth of online packing algorithms so that we construct approximately stable algorithms as a variant of generalized deferred acceptance algorithm. We further provide some inapproximability results. 


Approximately Stable Matchings with General Constraints

Employing Models of Human Social Motor Behavior for Artifical Agent Trainers

We present a framework for verifying strategic behaviour in an unbounded multi-agent system. We introduce a novel probabilistic semantics for parameterised multi-agent systems and define the corresponding verification problem against two probabilistic variants of alternating-time temporal logic. We define a verification procedure using an abstract model construction. We show that the procedure is complete for one variant of our specification language, and partial for the other. We present an implementation and report experimental results. 


Parameterised Verification of Strategic Properties in Probabilistic Multi-Agent Systems

In computational social choice, shift bribery is the procedure of paying voters to shift the briber’s preferred candidate forward in their preferences so as to make this candidate an election winner; the more general swap bribery procedure also allows one to pay voters to swap other candidates in their preferences. The complexity of swap and shift bribery is well-understood for many voting rules; typically, finding a minimum-cost bribery is computationally hard. We have studied swap and shift bribery in the setting where voters’ preferences are known to be single-peaked or single-crossing. We obtain polynomial-time algorithms for several variants of these problems for classic voting rules, such as Plurality, Borda and Condorcet-consistent rules. In this talk we present a polynomial time algorithm for shift bribery in the setting where voters’ preferences are single-peaked. 
"


Algorithms for Swap and Shift Bribery in Structured Elections

Redistribution mechanisms have been proposed for more efficient resource allocation but not for profit. We consider redistribution mechanism design in a setting where participants are connected and the resource owner is only connected to some of them. In this setting, to make the resource allocation more efficient, the resource owner has to inform the others who are not her neighbours, but her neighbours do not want more participants to compete with them. Hence, the goal is to design a redistribution mechanism such that participants are incentivized to invite more participants and the resource owner does not earn or lose much money from the allocation. In this presentation, I will first show that existing redistribution mechanisms cannot be directly applied in the network setting. Then I will introduce a novel network-based redistribution mechanism such that all participants on the network are invited, the allocation is more efficient and the resource owner has no deficit.


Redistribution Mechanism on Networks

When planning for multi-robot navigation tasks under uncertainty, the presence of multiple robots in the same part of the environment simultaneously can have an adverse effect on navigation performance. We refer to this effect as congestion. Existing solutions fail to model congestion at planning time, which leads to inefficient behaviour. In this presentation, I will detail how to probabilistically model congestion, presenting a structure called the probabilistic reservation table that summarises the plans of robots using continuous-time Markov chains. I will then show how this structure can be used for planning by outlining an approach that, for each robot, sequentially builds and solves a Markov decision process where the transition probabilities are obtained by querying the probabilistic reservation table.


Multi-Robot Planning Under Uncertainty with Congestion-Aware Models

Sparse reward problems present a challenge for reinforcement learning (RL) agents. Previous work has shown that choosing start states according to a curriculum can significantly improve the learning performance. We observe that many existing curriculum generation algorithms rely on two key components: Performance measure estimation and a start selection policy. Therefore, we propose a unifying framework for performance-based start state curricula in RL, which allows to analyze and compare the performance influence of the two key components. Furthermore, a new start state selection policy using spatial performance measure gradients is introduced. We conduct extensive empirical evaluations to compare performance-based start state curricula and investigate the influence of performance measure model choice and estimation. Benchmarking on difficult robotic navigation tasks and a high-dimensional robotic manipulation task, we demonstrate state-of-the-art performance of our novel spatial gradient curriculum.

A Performance-Based Start State Curriculum Framework for Reinforcement Learning

By a combination of well-known results in judgment aggregation, it is essentially impossible to design an aggregation rule that simultaneously satisfies two crucial requirements: to always return an outcome that is logically consistent, and to be immune to strategic manipulation. To address this dilemma, we put forward a novel notion of strategyproofness, which requires immunity to strategic manipulation only in certain well-defined situations---namely when either the truthful profile of individual judgments or the profile a would-be manipulator is trying to reach are majority-consistent. We argue that this constitutes an attractive compromise for aggregation rules one may want to use in practice, and we prove that several important rules are strategyproof in this sense. This includes, in particular, all rules belonging to the family of additive majority rules, such as the Kemeny rule and the Slater rule.


Majority-Strategyproofness in Judgment Aggregation

We consider a requester who acquires a set of data (e.g. images) that is not owned by one party. In order to collect as many data as possible, crowdsourcing mechanisms have been widely used to seek help from the crowd. However, existing mechanisms rely on third-party platforms, and the workers from these platforms are not necessarily helpful and redundant data are also not properly handled. To combat this problem, we propose a novel crowdsourcing mechanism based on social networks, where the rewards of the workers are calculated by information entropy and a modified Shapley value. This mechanism incentivizes the workers from the network to not only provide all data they have but also further invite their neighbours to offer more data. Eventually, the mechanism is able to acquire all data from all workers on the network and the requester’s cost is no more than the value of the data acquired. The experiments show that our mechanism outperforms traditional crowdsourcing mechanisms.


Downloads

Next from AAMAS 2020

Epistemic Plan Recognition

Similar lecture

Coupon Design in Advertising Systems

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAMAS 2020

Epistemic Plan Recognition

Similar lecture

Coupon Design in Advertising Systems

Downloads