BZPEER
More

Discover

About BZPEERSearchGraph

Research tools

HomeCollectionsResearchCollaborationPublish

Network

ExpertsCommunities

Opportunities

Listings

Account

Log inCreate account
Log inCreate account

Source author record

Tuan Hang Nguyen

Explore this author’s publications, collaborators and research topics. This catalog record has not been claimed by its author.

ResearcherUnclaimed source record
Machine Learningmath.NAmath.OCNumerical Analysismath.DS

Catalog footprint

Research overview

2works
5topics
1close collaborators

Actions

Is this your work?

Log in to claim
Open graphBrowse papers

Published work

2 publications

preprint2020arXiv

Asymptotic behaviour of learning rates in Armijo's condition

Fix a constant 0<α<10<α<1. For a C1C^1 function f:Rk→Rf:\mathbb{R}^k\rightarrow \mathbb{R}, a point xx and a positive number δ>0δ>0, we say that Armijo's condition is satisfied if f(x−δ∇f(x))−f(x)≤−αδ∣∣∇f(x)∣∣2f(x-δ\nabla f(x))-f(x)\leq -αδ||\nabla f(x)||^2. It is a basis for the well known Backtracking Gradient Descent (Backtracking GD) algorithm. Consider a sequence {xn}\{x_n\} defined by xn+1=xn−δn∇f(xn)x_{n+1}=x_n-δ_n\nabla f(x_n), for positive numbers δnδ_n for which Armijo's condition is satisfied. We show that if {xn}\{x_n\} converges to a non-degenerate critical point, then {δn}\{δ_n\} must be bounded. Moreover this boundedness can be quantified in terms of the norms of the Hessian ∇2f\nabla ^2f and its inverse at the limit point. This complements the first author's results on Unbounded Backtracking GD, and shows that in case of convergence to a non-degenerate critical point the behaviour of Unbounded Backtracking GD is not too different from that of usual Backtracking GD. On the other hand, in case of convergence to a degenerate critical point the behaviours can be very much different. We run some experiments to illustrate that both scenrios can really happen. In another part of the paper, we argue that Backtracking GD has the correct unit (according to a definition by Zeiler in his Adadelta's paper). The main point is that since learning rate in Backtracking GD is bound by Armijo's condition, it is not unitless.

Open access
preprint2019arXiv

Backtracking gradient descent method for general C1C^1 functions, with applications to Deep Learning

While Standard gradient descent is one very popular optimisation method, its convergence cannot be proven beyond the class of functions whose gradient is globally Lipschitz continuous. As such, it is not actually applicable to realistic applications such as Deep Neural Networks. In this paper, we prove that its backtracking variant behaves very nicely, in particular convergence can be shown for all Morse functions. The main theoretical result of this paper is as follows. Theorem. Let f:Rk→Rf:\mathbb{R}^k\rightarrow \mathbb{R} be a C1C^1 function, and {zn}\{z_n\} a sequence constructed from the Backtracking gradient descent algorithm. (1) Either lim⁡n→∞∣∣zn∣∣=∞\lim _{n\rightarrow\infty}||z_n||=\infty or lim⁡n→∞∣∣zn+1−zn∣∣=0\lim _{n\rightarrow\infty}||z_{n+1}-z_n||=0. (2) Assume that ff has at most countably many critical points. Then either lim⁡n→∞∣∣zn∣∣=∞\lim _{n\rightarrow\infty}||z_n||=\infty or {zn}\{z_n\} converges to a critical point of ff. (3) More generally, assume that all connected components of the set of critical points of ff are compact. Then either lim⁡n→∞∣∣zn∣∣=∞\lim _{n\rightarrow\infty}||z_n||=\infty or {zn}\{z_n\} is bounded. Moreover, in the latter case the set of cluster points of {zn}\{z_n\} is connected. Some generalised versions of this result, including an inexact version, are included. Another result in this paper concerns the problem of saddle points. We then present a heuristic argument to explain why Standard gradient descent method works so well, and modifications of the backtracking versions of GD, MMT and NAG. Experiments with datasets CIFAR10 and CIFAR100 on various popular architectures verify the heuristic argument also for the mini-batch practice and show that our new algorithms, while automatically fine tuning learning rates, perform better than current state-of-the-art methods such as MMT, NAG, Adagrad, Adadelta, RMSProp, Adam and Adamax.

Open access

Research graph

See the researcher in context

Open full explorer

Inspect adjacent papers, topics, institutions and collaborators without losing the researcher page.

Building this map preview

BZPEER is loading the nearby papers, people, topics and institutions for this page.