Reinforcement Learning

In this paper, autonomous learning of reward distri- bution in multi-agent reinforcement learning was ap- plied to the 4 player game named "not100". In this game, more shrewd tactics to cooperate with the other agents is required for each agent than the other tasks that the learning was applied previously. The reward distribution ratio after learning was varied among sim- ulation runs. However, the validity of the average non-uniform reward distribution ratio was examined in some ways. The three agents with higher win proba- bility after learning cooperated mutually, while strong cooperation was not observed in some cases when the agents learned with a fixed distribution ratio.

Petroleum industry production systems are highly automatized. In this industry, all functions (e.g., planning, scheduling and maintenance) are automated and in order to remain competitive researchers attempt to design an adaptive control system which optimizes the process, but also able to adapt to rapidly evolving demands at a fixed cost. In this paper, we present a multi-agent approach for the dynamic task scheduling in petroleum industry production system. Agents simultaneously insure effective production scheduling and the continuous improvement of the solution quality by means of reinforcement learning, using the SARSA algorithm. Reinforcement learning allows the agents to adapt, learning the best behaviors for their various roles without reducing the performance or reactivity. To demonstrate the innovation of our approach, we include a computer simulation of our model and the results of experimentation applying our model to an Algerian petroleum refinery.

This project proposes that language evolve through reinforcement learning where agents communicate with each other and provide rewards if communication is successful. The fundamental difference between the learning mechanisms that humans use to communicate with one another and how machines learn to communicate is that the system used by humans presupposes that the adult already knows the meanings associated with the human language. Languages evolve historically to be optimal communication systems where human language learning mechanisms have evolved in order to learn these systems more efficiently. Machines in their learning of natural language, have to start at a place that humans mastered thousands of years ago. Uttering previously unheard signals and collectively establishing meaning. The question that this paper deals with is how can a communication system evolve if none of the conspirators have mastered the system previously using evolutionary computation and reinforcement lear...

Victims of road traffic accidents face severe health problems on-site or after the event when they arrive at hospital lately in their emergency cycle. Road traffic accident has negative effect on the physical, social and emotional security of human lives which often lead to mortality, illness, pain, grief and even disability. This paper proposes a scheme that reduces the severity of road traffic accidents given its inevitable occurrence. The rational is to search for nearest hospitals to the accident location using Dijkstra algorithm and Fuzzy logic to recommend suitable hospitals out of list of nearest hospitals to timely attend to the emergency situation considering factors such as distance, severity of the accident, available facilities in the hospitals and other factors. The obtained results showed the practicability of the system to recommendation of quick solution to accident emergencies.

Obtaining an effective reward signal for dialogue management is a non trivial problem. Real user feedback is inconsistent and often even absent. This thesis investigates the use of intrinsic rewards for a reinforcement learning based dialogue manager in order to improve policy learning in an environment with sparse rewards and to move away from inefficient random ε-greedy exploration. In addition to rewards given by a user simulator for successful dialogues, intrinsic curiosity rewards are given in the form of belief-state prediction errors generated by an intrinsic curiosity module within the dialogue manager. We investigate two main settings for this method: (1) predicting the raw next belief-state, and (2) predicting belief-states in a learned feature space. In order to meet the right difficulty level for the system to be able to learn a feature space, the model is pre-trained on a small pool of dialogue transitions. For both settings, results comparable to and better than simple ε-greedy exploration are achieved. (1) is able to learn faster, but (2) achieves higher final results and has more potential for improvements and to be successful in larger state-action spaces, where feature encodings and generalization are beneficial.

In reference to methods analyzed recently by Sutton et al, and Konda & Tsitsiklis, we propose their modification called Randomized Policy Optimizer (RPO). The algorithm has a modular structure and is based on the value function rather than on the action-value function. The modules include neural approximators and a parameterized distribution of con- trol actions. The distribution must belong to

Abstract Next-generation wireless networks will integrate multiple wireless access technologies to provide seamless mobility to mobile users with high-speed wireless connectivity. This will give rise to a heterogeneous wireless access environment where ...

This work aims to show how an intelligent system based on reinforcement learning can benefit of classical financial indicators to overcome classic trading strategies in the stock market. Due to the non-linear, random and non-stationary nature of the financial markets, various classic strategies fail to benefit from all opportunities where they could be making profit. In order to achieve that, a system was built where only the buying strategy uses reinforcement learning SARSA algorithms while the sell is made through various classic strategies. To test it, a system that uses an identical selling strategy was constructed, where the purchase was based on a classic indicator. Results show that the intelligent algorithm achieved more stable returns, up to six times higher than the ones from tested classical strategies.

Abstract With the proliferation of smart meters in smart grids, new challenges have emerged in the energy sector and applications are continuously developed, mainly concerning data analytics to address those challenges. Traditionally, data analytics in smart grid systems is performed in server-side tier; however, it is necessary to process data analytics close to the smart meter to achieve better performance. In order to process data effectively, it is also necessary to implement methodologies to facilitate the integration of data analysis processes in the Advanced Metering Infrastructure (AMI). This paper presents a novel architecture for data analytics in Smart Metering Systems based on an edge-fog-cloud computing architecture that permits different types of data analytics in a multi-tier context. The proposed architecture has the capability of learning and adapting to different contexts in smart metering systems using a reinforcement learning approach. The architecture was tested with three different analytic applications: forecasting energy consumption, prediction of power quality and prediction of energy theft. The results indicate that the methodology can be feasible solution for direct implementation in Smart Metering Systems.

As a famous game in the domain of game theory, both pervasive empirical studies as well as intensive theoretical analysis have been conducted and performed worldwide to research different public goods game scenarios. At the same time, computer game simulators are utilized widely for better research of game theory by providing easy but powerful visualization and statistics functionalities. However, although solutions of public goods game have been widely discussed with empirical studies or theoretical approaches, no computational and automatic simulation approaches has been adopted. For this reason, we have implemented a computer simulator with reinforcement learning agents module for public goods game, and we have utilized this simulator to further study the characteristics of public goods game. Furthermore, in this article, we have also presented a bunch of interesting experimental results with respect to the strategies that agents used and the profits they earned.

Deregulated electricity markets are expected to provide affordable electricity for consumers through promoting competition. Yet, the results do not always fulfill the expectations. The regulator's market-clearing mechanism is a strategic choice that may affect the level of competition in the market. We conceive of the market-clearing mechanism as composed of two components: pricing rules and rationing policies. We investigate the strategic behavior of power generation companies under different market-clearing mechanisms using an agent-based simulation model which integrates a game-theoretical understanding of the auction mechanism in the electricity market and generation companies' learning mechanism. Results of our simulation experiments are presented using various case studies representing different market settings. The market in simulations is observed to converge to a Nash equilibrium of the stage game or to a similar state under most parameter combinations. Compared to pay-as-bid pricing, bid prices are closer to marginal costs on average under uniform pricing while GenCos' total profit is also higher. The random rationing policy of the ISO turns out to be more successful in achieving lower bid prices and lower GenCo profits. In minimizing GenCos' total profit, a combination of pay-as-bid pricing rule and random rationing policy is observed to be the most promising.

Reinforcement learning addresses the problem of learning to select actions in order to maximize an agent's performance in unknown environments. To scale reinforcement learning to complex real-world tasks, agent must be able to discover hierarchical structures within their learning and control systems. In this paper, the use of hierarchical reinforcement learning (HRL) to speed up the acquisition of cooperative multi-agent tasks is investigated, and a hierarchical multi-agent reinforcement learning (RL) framework and a hierarchical multi-agent RL algorithm called cooperative HRL are proposed. A fundamental property of the proposed approach is that it allows agents to learn coordination faster by sharing information at the level of cooperative subtasks, rather than attempting to learn coordination at the level of primitive actions. This approach can significantly speed up learning and make it more scalable with the number of agents.

The rapid production of mobile devices along with the wireless applications boom is continuing to evolve daily. This motivates the exploitation of wireless spectrum using multiple Radio Access Technologies (multi-RAT) and developing innovative network selection techniques to cope with such intensive demand while improving Quality of Service (QoS). Thus, we propose a distributed framework for dynamic network selection at the edge level, and resource allocation at the Radio Access Network (RAN) level, while taking into consideration diverse applications’ characteristics. In particular, our framework employs a deep Multi-Agent Reinforcement Learning (DMARL) algorithm, that aims to maximize the edge nodes’ quality of experience while extending the battery lifetime of the nodes and leveraging adaptive compression schemes. Indeed, our framework enables data transfer from the network’s edge nodes, with multi-RAT capabilities, to the cloud in a cost and energy-efficient manner, while mainta...

In recent years some researchers have explored the use of reinforcement learning (RL) algorithms as key components in the solution of various natural language processing tasks. For instance, some of these algorithms leveraging deep neural learning have found their way into conversational systems. This paper reviews the state of the art of RL methods for their possible use for different problems of natural language processing, focusing primarily on conversational systems, mainly due to their growing relevance. We provide detailed descriptions of the problems as well as discussions of why RL is well-suited to solve them. Also, we analyze the advantages and limitations of these methods. Finally, we elaborate on promising research directions in natural language processing that might benefit from reinforcement learning.

Real world multi-agent tasks often involve varying types and quantities of agents and non-agent entities. Agents frequently do not know a priori how many other agents and non-agent entities they will need to interact with in order to complete a given task, requiring agents to generalize across a combinatorial number of task configurations with each potentially requiring different strategies. In this work, we tackle the problem of multi-agent reinforcement learning (MARL) in such dynamic scenarios. We hypothesize that, while the optimal behaviors in these scenarios with varying quantities and types of agents/entities are diverse, they may share common patterns within sub-teams of agents that are combined to form team behavior. As such, we propose a method that can learn these sub-group relationships and how they can be combined, ultimately improving knowledge sharing and generalization across scenarios. This method, Attentive-Imaginative QMIX, extends QMIX for dynamic MARL in two way...

In our previous studies, we showed that the estimation of the rock-scissors-paper (RSP, janken) game strategy is effective for the prediction of a player's hand sign sequences. The purpose of this study is to propose a method to estimate the RSP game strategy in the basis of human personality in an RSP game. To estimate a player's strategy in the RSP game, it is effective to compare the player's hand sign sequence and the hand sign sequences given by various typical RSP strategies on the basis of similarity. In this study, we propose the method of using a homology search to calculate the similarity between sequences. The results show that our proposed method is effective for strategy estimation .

Advances in machine learning allow us to predict certain events before they happen. Diseases and deaths are one of the most painful of those events for people all around the world. There are huge amounts of health data available that can be used for machine learning to predict diseases that are going to be seen in a person. Sometimes it is possible to prevent diseases and even deaths if a patient takes precaution against it. So, it is possible to save millions of lives through predicting and preventing diseases and deaths using machine learning. In this paper, the concept of preventable diseases and deaths will be discussed. Then, studies that have been done in this field will be analyzed. In the end, future potential and enablers of disease prediction will be examined.

This paper proposes a novel optimal adaptive event-triggered
control algorithm for nonlinear continuous-time systems. The goal
is to reduce the controller updates, by sampling the state only
when an event is triggered to maintain stability and optimality.
The online algorithm is implemented based on an actor/critic
neural network structure. A critic neural network is used to
approximate the cost and an actor neural network is used to
approximate the optimal {event-triggered controller}. Since in the
algorithm proposed there are dynamics that exhibit continuous
evolutions described by ordinary differential equations and
instantaneous jumps or impulses, we will use an impulsive system
approach. A Lyapunov stability proof ensures that the closed-loop
system is asymptotically stable. Finally, we illustrate the effectiveness of the proposed solution compared to a time-triggered controller.

Citation: Kyriakos G. Vamvoudakis. Event-triggered optimal adaptive control algorithm for continuous-time nonlinear systems.  IEEE/CAA Journal of Automatica Sinica, 2014, 1(3): 282-293

This report focuses on the Access network implementation of a 5G small-cell communication system using Cognitive Radio. 5G systems are the next major wireless communications standard and Cognitive Radio is being heralded as a valid implementation method when the wireless communications industry finally concludes on a fifth generation standard. The project is concerned with implementing a Beyond-Next-Generation (5G) communications network based on Cognitive Radio. A functioning model is developed in matlab, the purpose of which is to show that Cognitive Radio based 5G systems will give acceptable Quality-of-Service and energy efficiency levels.
The Access network implementation of the system involves channel assignment for the access links which provide network access to mobile subscribers. Suitable channels to service call requests are determined based on cognition.
The system performance is evaluated by estimating the system blocking and dropping probabilities and the percentage of energy saved.

Wireless sensor devices are the backbone of the Internet of things (IoT), enabling real-world objects and human beings to be connected to the Internet and interact with each other to improve citizens’ living conditions. However, IoT devices are memory and power-constrained and do not allow high computational applications, whereas the routing task is what makes an object to be part of an IoT network despite of being a high power-consuming task. Therefore, energy efficiency is a crucial factor to consider when designing a routing protocol for IoT wireless networks. In this paper, we propose EER-RL, an energy-efficient routing protocol based on reinforcement learning. Reinforcement learning (RL) allows devices to adapt to network changes, such as mobility and energy level, and improve routing decisions. The performance of the proposed protocol is compared with other existing energy-efficient routing protocols, and the results show that the proposed protocol performs better in terms of ...

Computational modeling and brain imaging studies suggest that sensitivity to rewards and behaviorist learning principles partly explain smartphone engagement patterns and potentially smartphone dependence. Responses to a questionnaire, and observational measures of smartphone use were recorded for 121 university students. Each participant was also tested with a laboratory task of reward sensitivity and a test of verbal operant conditioning. Twenty-three percent of the sample had probable smartphone addiction. Using multivariate regression, smartphone use, particularly the number of instant messenger services employed, was shown to be significantly and independently predicted by reward sensitivity (a positive relationship), and by instrumental conditioning (a negative relationship). However, the latter association was driven by a subset of participants who developed declarative knowledge of the response-reinforcer contingency. This suggests a process of impression management driven by experimental demand characteristics, producing goal-directed instrumental behavior not habit-based learning. No other measures of smartphone use, including the self-report scale, were significantly associated with the experimental tasks. We conclude that stronger engagement with smartphones, in particular instant messenger services, may be linked to people being more sensitive to rewarding stimuli, suggestive of a motivational or learning mechanism. We propose that this mechanism could underly problem smartphone use and dependence. It also potentially explains why some aspects of smartphone use, such as habitual actions, appear to be poorly measured by technology-use questionnaires. A serendipitous secondary finding confirmed that smartphone use reflected active self-presentation. Our ‘conditioning’ task-induced this behavior in the laboratory and could be used in social-cognition experimental studies.

This paper describes an investigation into the refinement of context -based human behavior models through the use of experiential learning. Specifically, a tactical agent was endowed with a context -based control model developed through other means and tasked with a mission in a simulation. This simulation-based mission was employed to expose the agent to situations possibly not considered in the model’s original construction. Reinforcement learning was used to evaluate and refine the performance of this agent to improve its effectiveness and generality. Introduction and Background How one makes a decision when faced with a task can be described as that person’s behaviour. The Oxford dictionary [1], defines “behaviour” as ‘the actions or reactions of a person or animal in response to external or internal stimuli’. Human behaviours are, consequently the actions or reactions of a human in response to some external or internal stimuli. The external stimuli include touch, smell, sight, ...

Online reviews are a feedback to the product and play a key role in improving the product to cater to
consumers. Online reviews that rely heavily on manual categorization are time consuming and labor
intensive.The recurrent neural network in deep learning can process time series data, while the long and
short term memory network can process long time sequence data well. This has good experimental
verification support in natural language processing, machine translation, speech recognition and language
model.The merits of the extracted data features affect the classification results produced by the classification
model. The LDA topic model adds a priori a posteriori knowledge to classify the data so that the
characteristics of the data can be extracted efficiently.Applied to the classifier can improve accuracy and
efficiency. Two-way long-term and short-term memory networks are variants and extensions of cyclic neural
networks.The deep learning framework Keras uses Tensorflow as the backend to build a convenient two-way
long-term and short-term memory network model, which provides a strong technical support for the
experiment.Using the LDA topic model to extract the keywords needed to train the neural network and
increase the internal relationship between words can improve the learning efficiency of the model. The
experimental results in the same experimental environment are better than the traditional word frequency
features.

Internet-of-Things (IoT) generate large data that is processed, analysed
and filtered by cloud data centres. IoT is getting tremendously popular: the
number of IoT devices worldwide is expected to reach 50.1 billion by 2020 and
from this, 30.7% of IoT devices will be made available in Healthcare. Transmission
and analysis of this much amount of data will increase the response time
of cloud computing. The increase in response time will lead to high service
latency to the end-users. The main requirement of IoT is to have low latency to
transfer the data in real-time. Cloud cannot fulfill the QoS requirement in a
satisfactory manner. Both the volume of data as well as factors related to internet
connectivity may lead to high network latency in analyzing and acting upon the
data. The propose research work introduces a hybrid approach that combines
fuzzy and reinforcement learning to improve service and network latency in
healthcare IoT and cloud. This hybrid approach integrates healthcare IoT
devices with the cloud and uses fog services with Fuzzy Reinforcement
Learning Data Packet Allocation (FRLDPA) algorithm. The propose algorithm
performs batch workloads on IoT data to minimize latency and manages the
QoS of the latency-critical workloads. It has the potential to automate the reasoning
and decision making capability in fog computing nodes.

The Hint Factory is a method of automatic hint generation that has been used to augment hints in a number of educational systems. Although the previous implementations were done in domains with largely deterministic environments, the methods are inherently useful in stochastic environments with uncertainty. In this work, we explore the game Connect Four as a simple domain to give decision support under uncertainty. We speculate how the implementation created could be extended to other domains including simulated learning environments and advanced navigational tasks.