Source author record

C. Daniel Freeman

C. Daniel Freeman appears in the imported research catalog. Authorship, coauthor and topic links are available while profile ownership is still unclaimed.

ResearcherUnclaimed source record

Machine Learning Neural and Evolutionary Computing quant-ph cond-mat.stat-mech cond-mat.str-el hep-th math.OC

Catalog footprint

What is connected

6works

7topics

4close collaborators

Actions

Connect this record

Open graph Browse works

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.

preprint2022arXiv

Gradients are Not All You Need

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we discuss a common chaos based failure mode which appears in a variety of differentiable circumstances, ranging from recurrent neural networks and numerical physics simulation to training learned optimizers. We trace this failure to the spectrum of the Jacobian of the system under study, and provide criteria for when a practitioner might expect this failure to spoil their differentiation based optimization algorithms.

preprint2022arXiv

Practical tradeoffs between memory, compute, and performance in learned optimizers

Optimization plays a costly and crucial role in developing machine learning systems. In learned optimizers, the few hyperparameters of commonly used hand-designed optimizers, e.g. Adam or SGD, are replaced with flexible parametric functions. The parameters of these functions are then optimized so that the resulting learned optimizer minimizes a target loss on a chosen class of models. Learned optimizers can both reduce the number of required training steps and improve the final test loss. However, they can be expensive to train, and once trained can be expensive to use due to computational and memory overhead for the optimizer itself. In this work, we identify and quantify the design features governing the memory, compute, and performance trade-offs for many learned and hand-designed optimizers. We further leverage our analysis to construct a learned optimizer that is both faster and more memory efficient than previous work. Our model and training code are open source.

preprint2021arXiv

Training Learned Optimizers with Randomly Initialized Learned Optimizers

Learned optimizers are increasingly effective, with performance exceeding that of hand designed optimizers such as Adam~\citep{kingma2014adam} on specific tasks \citep{metz2019understanding}. Despite the potential gains available, in current work the meta-training (or `outer-training') of the learned optimizer is performed by a hand-designed optimizer, or by an optimizer trained by a hand-designed optimizer \citep{metz2020tasks}. We show that a population of randomly initialized learned optimizers can be used to train themselves from scratch in an online fashion, without resorting to a hand designed optimizer in any part of the process. A form of population based training is used to orchestrate this self-training. Although the randomly initialized optimizers initially make slow progress, as they improve they experience a positive feedback loop, and become rapidly more effective at training themselves. We believe feedback loops of this type, where an optimizer improves itself, will be important and powerful in the future of machine learning. These methods not only provide a path towards increased performance, but more importantly relieve research and engineering effort.

preprint2020arXiv

Using a thousand optimization tasks to learn hyperparameter search strategies

We present TaskSet, a dataset of tasks for use in training and evaluating optimizers. TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets. As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially. By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search. Next we use the diversity of the TaskSet and our method for learning hyperparameter lists to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. As part of this work we have opensourced code for all tasks, as well as ~29 million training curves for these problems and the corresponding hyperparameters.

preprint2016arXiv

Entanglement structure of non-equilibrium steady states

We study the problem of calculating transport properties of interacting quantum systems, specifically electrical and thermal conductivities, by computing the non-equilibrium steady state (NESS) of the system biased by contacts. Our approach is based on the structure of entanglement in the NESS. With reasonable physical assumptions, we show that a NESS close to local equilibrium is lightly entangled and can be represented via a computationally efficient tensor network. We further argue that the NESS may be found by dynamically evolving the system within a manifold of appropriate low entanglement states. A physically realistic law of dynamical evolution is Markovian open system dynamics, or the Lindblad equation. We explore this approach in a well-studied free fermion model where comparisons with the literature are possible. We study both electrical and thermal currents with and without disorder, and compute entropic quantities such as mutual information and conditional mutual information. We conclude with a discussion of the prospects of this approach for the challenging problem of transport in strongly interacting systems, especially those with disorder.

preprint2014arXiv

Relaxation dynamics of the toric code in contact with a thermal reservoir: Finite-size scaling in a low temperature regime

We present an analysis of the relaxation dynamics of finite-size topological qubits in contact with a thermal bath. Using a continuous-time Monte Carlo method, we explicitly compute the low-temperature nonequilibrium dynamics of the toric code on finite lattices. In contrast to the size-independent bound predicted for the toric code in the thermodynamic limit, we identify a low-temperature regime on finite lattices below a size-dependent crossover temperature with nontrivial finite-size and temperature scaling of the relaxation time. We demonstrate how this nontrivial finite-size scaling is governed by the scaling of topologically nontrivial two-dimensional classical random walks. The transition out of this low-temperature regime defines a dynamical finite-size crossover temperature that scales inversely with the log of the system size, in agreement with a crossover temperature defined from equilibrium properties. We find that both the finite-size and finite-temperature scaling are stronger in the low-temperature regime than above the crossover temperature. Since this finite-temperature scaling competes with the scaling of the robustness to unitary perturbations, this analysis may elucidate the scaling of memory lifetimes of possible physical realizations of topological qubits.