Source author record

Larry Rudolph

Larry Rudolph appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

Machine Learning Robotics cs.CY Neural and Evolutionary Computing Social and Information Networks

Catalog footprint

What is connected

3works

5topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2020arXiv

A Closer Look at Deep Policy Gradients

We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. To this end, we propose a fine-grained analysis of state-of-the-art methods based on key elements of this framework: gradient estimation, value prediction, and optimization landscapes. Our results show that the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict: the surrogate objective does not match the true reward landscape, learned value estimators fail to fit the true value function, and gradient estimates poorly correlate with the "true" gradient. The mismatch between predicted and empirical behavior we uncover highlights our poor understanding of current methods, and indicates the need to move beyond current benchmark-centric evaluation methods.

preprint2020arXiv

Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

preprint2016arXiv

Managing travel demand: Location recommendation for system efficiency based on mobile phone data

Growth in leisure travel has become increasingly significant economically, socially, and environmentally. However, flexible but uncoordinated travel behaviors exacerbate traffic congestion. Mobile phone records not only reveal human mobility patterns, but also enable us to manage travel demand for system efficiency. In this paper, we propose a location recommendation system that infers personal preferences while accounting for constraints imposed by road capacity in order to manage travel demand. We first infer unobserved preferences using a machine learning technique from phone records. We then formulate an optimization method to improve system efficiency. Coupling mobile phone data with traffic counts and road network infrastructures collected in Andorra, this study shows that uncoordinated travel behaviors lead to longer average travel delay, implying the opportunities in managing travel demand by collective decisions. The interplay between congestion relief and overall satisfied location preferences observed in extensive simulations indicate that moderate sacrifices of individual utility lead to significant travel time savings. Specifically, the results show that under full compliance rate, travel delay fell by 52% at a cost of 31% less satisfaction. Under 60% compliance rate, 41% travel delay is saved with a 17% reduction in satisfaction. This paper highlights the effectiveness of the synergy among collective behaviors in increasing system efficiency.