Fundamentals of Unconstrained optimization

Introduction §

Optimization problems can be written as follows:

$$\underset{x\in {\mathbb{R}}^n}{\min }f(x)\text{subject to }\{\begin{array}{l}{c}_i(x)=0,i\in \mathcal{E},\\ c_i(x)\ge 0,i\in \mathcal{I},\end{array}$$

In the book (Numerical Optimization by Jorge Nocedal & Stephen J. Wright), we focus on continuous local deterministic optimization, and go over both constrained and unconstrained optimization.

Fundamentals of Unconstrained optimization §

We only consider the problem of

$$\underset{x}{\min }f(x)$$

here.

Talor’s Theorem §

This is not only Talor’s theorem. We also uses Lagrange theorem (mean value theorem) here.

Suppose that $f:{\mathbb{R}}^n\to \mathbb{R}$ is continuously differentiable and that $p\in {\mathbb{R}}^n$ . Then we have that

$$\begin{array}{rl}f(x+p)& =f(x)+\nabla f(x+tp{)}^{T}p\\ & =f(x)+\nabla f(x{)}^{T}p+\frac{1}{2}{p}^T{\nabla }^{2}f(x+tp)p\end{array}$$

FIrst Order Necessary Conditions §

If $x^{\ast }$ is a local minimizer and f is continuously differentiable in an open neighborhood of $x^{\ast }$ , then $\nabla f(x^{\ast })=0$.

Second Order Necessary Conditions §

If $x^{\ast }$ is a local minimizer of f and ${\nabla }^{2}f$ exists and is continuous in an open neighborhood of $x^{\ast }$ , then $\nabla f(x^{\ast })=0$ and ${\nabla }^{2}f(x^{\ast })$ is positive semidefinite.

Second-Order Sufficient Conditions §

Suppose that ${\nabla }^{2}f$ is continuous in an open neighborhood of $x^{\ast }$ and that $\nabla f(x^{\ast })=0$ and $\nabla f(x^{\ast })$ is positive definite. Then $x^{\ast }$ is a strict local minimizer of $f$ .