Source author record

Ajith Muralidharan

Ajith Muralidharan appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

Machine Learning math.DS Systems and Control

Catalog footprint

What is connected

4works

3topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2022arXiv

Multi-objective Optimization of Notifications Using Offline Reinforcement Learning

Mobile notification systems play a major role in a variety of applications to communicate, send alerts and reminders to the users to inform them about news, events or messages. In this paper, we formulate the near-real-time notification decision problem as a Markov Decision Process where we optimize for multiple objectives in the rewards. We propose an end-to-end offline reinforcement learning framework to optimize sequential notification decisions. We address the challenge of offline learning using a Double Deep Q-network method based on Conservative Q-learning that mitigates the distributional shift problem and Q-value overestimation. We illustrate our fully-deployed system and demonstrate the performance and benefits of the proposed approach through both offline and online experiments.

preprint2022arXiv

Offline Reinforcement Learning for Mobile Notifications

Mobile notification systems have taken a major role in driving and maintaining user engagement for online platforms. They are interesting recommender systems to machine learning practitioners with more sequential and long-term feedback considerations. Most machine learning applications in notification systems are built around response-prediction models, trying to attribute both short-term impact and long-term impact to a notification decision. However, a user's experience depends on a sequence of notifications and attributing impact to a single notification is not always accurate, if not impossible. In this paper, we argue that reinforcement learning is a better framework for notification systems in terms of performance and iteration speed. We propose an offline reinforcement learning framework to optimize sequential notification decisions for driving user engagement. We describe a state-marginalized importance sampling policy evaluation approach, which can be used to evaluate the policy offline and tune learning hyperparameters. Through simulations that approximate the notifications ecosystem, we demonstrate the performance and benefits of the offline evaluation approach as a part of the reinforcement learning modeling approach. Finally, we collect data through online exploration in the production system, train an offline Double Deep Q-Network and launch a successful policy online. We also discuss the practical considerations and results obtained by deploying these policies for a large-scale recommendation system use-case.

preprint2015arXiv

PointQ model of an arterial network: calibration and experiments

The calibration of a PointQ arterial microsimulation model is formulated as a quadratic programming problem (QP) whose decision variables are link flows, demands at entry links, and turn movements at intersections, subject to linear constraints imposed by flow conservation identities and field measurements of a subset of link flows (counts), demands and turn ratios. The quadratic objective function is the deviation of the decision variables from their measured values. The solution to the QP gives estimates of all unmeasured variables and thus yields a fully specified simulation model. Runs of this simulation model can then be compared with other field measurements, such as travel times along routes, to judge the reliability of the calibrated model. A section of the Huntington-Colorado arterial near I-210 in Los Angeles comprising 73 links and 16 intersections is used to illustrate the procedure. Two experiments are conducted with the calibrated model to determine the maximum traffic that can be diverted from the I-210 freeway to the arterial network, with and without permitting changes in the timing plans. The maximum diversion in both cases is obtained by solving a linear programming problem. A third experiment compares the delay and travel time using the existing fixed time control and a max pressure control. The fourth experiment compares two PointQ models: in the first model the freeway traffic follows a pre-specified route while the background traffic moves according to turn ratios, and in the second model turn ratios are modified in a single commodity model to match the link flows. The substantial modification of the turn ratios needed suggests that the use of a single-commodity model as frequently done in CTM models can be misleading...

preprint2014arXiv

Analysis of Fixed-Time Control

A network of signalized intersections is modeled as a queuing network. The intersections are regulated by fixed-time (FT) controls, all with the same cycle length or period, $T$. Vehicles arrive from outside the network at entry links in a deterministic periodic stream, also with period $T$, make turns at intersections in fixed proportions, and eventually leave the network. Vehicles take a fixed time to travel along each link, and at the end of the link they join a queue. There is a separate queue at each intersection for each movement. The storage capacity of the queues is infinite, so there is no spill back. The state of the network at time $t$ is the vector $x(t)$ of all queue lengths, together with the position of vehicles traveling along the links. The state evolves according to a delay-differential equation. Suppose the network is stable, that is, $x(t)$ is bounded. Then (1) there exists a unique periodic trajectory $x^*(t)$, with period $T$; (2) every trajectory converges to this periodic trajectory; (3) if vehicles do not follow loops, the convergence occurs in finite time. The periodic trajectory determines the performance of the entire network.