Optimization Theory Overview

Optimization techniques are used to find a set of design parameters, x = {x₁,x₂,...,x_n}, that can in some way be defined as optimal. In a simple case, this process might be the minimization or maximization of some system characteristic that is dependent on x. In a more advanced formulation, the objective function f(x), to be minimized or maximized, might be subject to constraints in one or more of these forms:

• Equality constraints, G_i(x) = 0 ( i = 1,...,m_e)

• Inequality constraints, G_i( x) ≤ 0 (i = m_e + 1,...,m)

• Parameter bounds, x_l, x_u, where x_l ≤ x ≤ x_u, some x_l can be –∞, and some x_u can be ∞

A General Problem (GP) description is stated as

$$\underset{x}{\min }f(x),$$

(1)

subject to

$$\begin{array}{l}\begin{array}{cc}{G}_i(x)=0& i=1,\mathrm{...},m_e\\ G_i(x)\le 0& i=m_e+1,\mathrm{...},m\end{array}\\ x_l\le x\le x_u,\end{array}$$

where x is the vector of length n design parameters, f(x) is the objective function (which returns a scalar value), and the vector function G(x) returns a vector of length m containing the values of the equality and inequality constraints evaluated at x.

An efficient and accurate solution to this problem depends not only on the size of the problem in terms of the number of constraints and design variables, but also on characteristics of the objective function and constraints. When both the objective function and the constraints are linear functions of the design variable, the problem is known as a Linear Programming (LP) problem. Quadratic Programming (QP) concerns the minimization or maximization of a quadratic objective function that is linearly constrained. For both the LP and QP problems, reliable solution procedures are readily available. More difficult to solve is the Nonlinear Programming (NP) problem in which the objective function and constraints can be nonlinear functions of the design variables. A solution of the NP problem generally requires an iterative procedure to establish a direction of search at each major iteration. This solution is usually achieved by the solution of an LP, QP, or unconstrained subproblem.

All optimization takes place in real numbers. However, unconstrained least-squares problems and equation solving can be formulated and solved using complex analytic functions. See Complex Numbers in Optimization Toolbox Solvers.

Related Topics

• Constrained Nonlinear Optimization Algorithms
• Constrained Optimization: Intuition behind the Lagrangian
• Solver-Based Optimization Problem Setup