optimization – lecture 1 review of differential calculus for functions of single...
TRANSCRIPT
Optimization – Lecture 1 Review of Differential Calculus for
Functions of Single Variable
http://users.encs.concordia.ca/~luisrod
Copyright © Luis Rodrigues, January 2014
Copyright © Luis Rodrigues
Outline
2
• Optimization Problems
• Real Numbers and Real Vectors
• Open, Closed and Bounded sets
• Real Functions
• Limit and Continuity
• Maximum and Minimum of Functions
• Derivative, L’Hopital’s Rule and Stationary Points
Optimization Problems • An optimization problem (OP) is a problem of the form
• This is a minimization (we can consider a maximization of f as a minimization of –f), f is a function to be minimized, s.t means subject to, is the set of feasible values of variable
• The obvious question is what kinds of functions, what kinds of sets and what kinds of variables can we handle
• The variable can be a single real variable or a vector, the function f will be assumed to be of class (twice differentiable), the set will be an open set or a closed set
• The next slides will explain what these concepts mean 3 Copyright © Luis Rodrigues
min f (x)s.t x ∈Ω
Ω x
xC2
Ω
Integer, Rational, Irrational Numbers • We assume familiarity with addition +, subtraction -,division /,
multiplication , less <, greater >, less or equal , greater or equal , (approximately ) equal =,different ,equivalent , implies ,or , and , union , intersection , (not) member ( ), for all , there exists one ,
• The integer numbers are the non-negative integers 0,1,2,… union with the negative integers -1,-2,-3,…
• A rational number is the ratio of two integers (e.g, ½). There are numbers that cannot be represented as the ratio of two integers and are called irrational numbers. An example is the length of the circumference divided by its diameter that yields the number
• Real numbers can be either rational or irrational • Example: The area of a rectangle with sides measuring
2cm and 1cm is centimeters squared
4 Copyright © Luis Rodrigues
b = 2cma =1cm
2cm2
≤≥
π = 3.14159...
≠
A = 2 ⋅1= 2
a ⋅b = 0⇔ a = 0∨b = 0
⇔∨ ∧
∉∈
∀
≈
∃
⋅
⇒
Real Numbers • The set of real numbers is denoted . In we can define
two closed operations (addition and multiplication) that have inverses (subtraction and division, respectively)
• This means that two real numbers and can be added to yield a real and multiplied to yield a real or
• Associative Property: • Commutative Property: • Existence of Identity: • Existence of Inverse: • Distributive Property: • A set with two operations verifying these properties is
called a field and is represented by the triad • All numbers form an open interval denoted (a,b) • Notation for large products, sums:
5 Copyright © Luis Rodrigues
ℜℜ
a+ (−a) = 0, b ⋅b−1 =1,b ≠ 0a+ 0 = a, a ⋅1= a
abbaabba .., =+=+
)..()..(),()( cbacbacbacba =++=++
a ⋅ (b+ c) = (a ⋅b)+ (a ⋅c)
ℜ,+, ⋅( )
aN = a ⋅aaN times , an
n=1
N
∑ = a1 + a2 ++ aN
a ⋅ba+ ba b
ab
a < x < b
Vectors and Vector Coordinates
6 Copyright © Luis Rodrigues
• A vector is a geometric object represented by a letter with an arrow, e.g., with magnitude and direction
• A vector has coordinates in a given frame. A vector is not the same as its coordinates. Coordinates of the same vector can be different in different frames.
• The coordinates of a vector in a frame E will be denoted by and will be represented as a column array of real numbers. The transpose of the coordinates of a vector will be a row array denoted by and the derivative of a vector with respect to t will be denoted by (more later)
• Example: For a 2D frame E with unit vectors
x
x E x[ ]
[ ]TE x
[ ]htxhtxx h ))()((lim 0 −+= →
x = te1 + t2e2,
E x[ ] =x1x2
!
"##
$
%&&=
tt2
!
"##
$
%&&, E x[ ]T = t t2!
"#$%&,E x!" $%=
12t
!
"#
$
%&
x = x12 + x2
2
x
( )21,ee
x
e1
e2
Open, Closed, Compact Sets • Let be the set of real numbers and the set of real
vectors with n coordinates • An open ball in with center and radius is the
set of vectors verifying
• An open set in is a set for which around each point one can find an open ball that is fully contained in the set. For example an open ball is an open set
• A closed set is a set for which its complementary which contains all elements in that are not in is open. For example
• A bounded set is a set that can be contained in an open ball with finite radius. For example
• A compact set is a set that is closed and bounded 7
ℜ ℜn
ℜn x0 r > 0xBr (x0 ) =
x : x − x0 < r{ }x0 r
ℜn
ℜn \Ωℜn Ωx : x − x0 ≤ r{ }
x : x ≤1{ }
Copyright © Luis Rodrigues
Real Functions • A real function is a mapping of a real vector into a real
value . If is a scalar, f is a function of a real variable • For each value of there can only be a value of f but
there could be more than one value of that have the same value of f. Example:
• When different values of correspond to different values of f the function is injective or 1-to-1. Example:
• The set of values of for which f exists is called the domain of f. The set of values f is called the image of f
• When the image of f is equal to then f is called surjective or onto. Example:
• A function that is both injective and surjective is called bijective. These functions can be inverted ( exists)
• A function f is linear if Example: 8 Copyright © Luis Rodrigues
f (x) = x2 ⇒ x = ± f (x)
ℜ
f (α x1 +βx2 ) =α f (
x1)+β f (x2 ), ∀α,β ∈ℜ
1)( += xxf
xxf =)(
xxf 2)( =
f −1
f (x)
x
x x
x
x
x
Real Variable Functions: Trigonometric
CC
AA
B
B
ca
b
C A
B
c
b
a
A
Right Triangle
Oblique Triangle
222 bac +=
AabBA
caBA
cbB
baA
cbA
caA
cottan,sincos,cossin
tan,cos,sin
======
===
Pythagorean Theorem:
Law of the sines:
Law of the cosines:
Cc
Bb
Aa
sinsinsin==
Cabbac cos2222 −+=Copyright © Luis Rodrigues
Straight lines can be used to form triangles between three points A, B, C
9
Useful Trigonometric Relations
10 Copyright © Luis Rodrigues
sin2θ + cos2θ =1sin2θ = 0.5− 0.5cos(2θ )sin(2θ ) = 2sin(θ )cos(θ )cos(2θ ) = cos2(θ )− sin2(θ )tan(2θ ) = 2 tan(θ ) / (1− tan2(θ ))sin(α +β) = sinα cosβ + cosα sinβsin(α −β) = sinα cosβ − cosα sinβcos(α +β) = cosα cosβ − sinα sinβcos(α −β) = cosα cosβ + sinα sinβ
sinα + sinβ = 2sinα +β2
cosα −β2
sinα − sinβ = 2cosα +β2
sinα −β2
cosα + cosβ = 2cosα +β2
cosα −β2
cosα − cosβ = −2sinα +β2
sinα −β2
Polynomials of a Real Variable • A polynomial function of order n is a function of the form
• If x is a real variable then we say that • When n=1 then f is a polynomial of first order
We say that f is affine if and linear if • If n=2 then f is second order. We say that f is quadratic • Examples of polynomials of first and second order:
11 Copyright © Luis Rodrigues
f (x) =α0 +α1x +α2x2 ++αnx
n,αi ∈ℜ
][xf ℜ∈
00 ≠α 00 =α
f (x) = x +1f (x) = 2xf (x) = (x +1)2 = x2 + 2x +1
Exponential & Logarithm Functions • The Euler’s number e is approximately equal to 2.71828
and is represented by the infinite sum
• The exponential function is defined by • The logarithm function is the inverse of the exponential
function, is defined by and verifies the following important properties
12 Copyright © Luis Rodrigues
e =1+11+11⋅2
+1
1⋅2 ⋅3+…
f (x) = ex
f (x) = ln(x)
ln(ex ) = x, eln(x ) = xln(xy) = ln(x)+ ln(y)ln(xa ) = a ln(x), a ∈ℜ
Inverse Function Property:
Limit of a Function and Continuity
13 Copyright © Luis Rodrigues x∞x1x 2x …
)(xf)( 1xf)( 2xf
)( ∞xf
• Suppose we have a sequence of numbers getting closer and closer to some value. For example
• The values get closer and closer to 1 as n increases. We say that converges and its limit is 1,i.e
• The difference also gets smaller for any p as n increases and we say that is a Cauchy sequence
• If f is a function of we sometimes want to determine if f also converges to a value when converges to a limit
• This value, if it exists, is called . If then we say that f is continuous at . If f is continuous
at all points we say that f is a continuous function Example:
limx→x∞f (x)
,2,1, =nxn
,999.0,99.0,9.0 321 === xxx
1lim =∞→
nn
x
x
x∞
x∞
0)()(lim,sin)(
==
==
→
∞
π
π
πfxfxxxf
x
xn
xn+p − xnxn
xn
limx→x∞f (x) = f (x∞)
The Dirac Delta Function • As an example of a function defined by a limit we will now
talk about the impulse function
• An impulse function, also called Dirac delta function , is the mathematical model of an impulsive force like, for example, the one exerted in a billard ball: strong force in a short period of time
• The function can be defined as the result of a limiting process as follows:
)(tδ
)(tδ
⎪⎩
⎪⎨⎧ Δ≤≤Δ== ΔΔ
→Δ otherwise
tifttt,0
0,1)(,)(lim)(0
δδδ
t
)(tΔδ
Δ
1
Δ
Area=1
14 Copyright © Luis Rodrigues
Maximum and Minimum of a Function • A value is called a local minimum (maximum) if ( ) • If the result is true in all the domain of then it is a
global minimum (maximum)
• Weierstrass Theorem: A continuous function defined on a compact set has a maximum and a minimum
15
f (x0 )f (x) ≤ f (x0 )∃δ > 0 : f (x) ≥ f (x0 ) ∀
x ≠ x0, x ∈ Bδ (x0 )D
xmaxxmin x
)(xf
Copyright © Luis Rodrigues
fD
Derivative of a Scalar Function • For a function f(x) represented by its graph one can
imagine connecting two points P, Q on the graph by a line. This line is called a secant.
• If we now make converge to a value such that Q converges to P then the secant converges to the tangent line at P. The derivative f’ of f at is the slope of the tangent and is defined by the limit in the box. A function that has a derivative at all points is called differentiable. The second derivative f’’ is the derivative of the derivative
Example:
16 Copyright © Luis Rodrigues
P
Q Q’
x
)(xf
1x2x0x
x 0x
0x
hxfhxf
xxxfxfxf
dxdf
hxxx
)()(lim)()(lim)(' 00
00
00
00
−+=
−
−==
→→
2)1)1((1)1(lim
1,1)(22
01
02
=+−++
=
=+=
→ hh
dxdf
xxxf
h
Useful Derivative Relations
17 Copyright © Luis Rodrigues
f (x) = xα, f '(x) =αxα−1
f (x) = sin(x), f '(x) = cos(x)f (x) = cos(x), f '(x) = −sin(x)f (x) = tan(x), f '(x) =1/ cos2(x)f (x) = cos(x)sin(x), f '(x) = cos(2x)f (x) = ex, f '(x) = ex
f (x) = ln(x), f '(x) =1/ x( f (x)+ g(x))' = f '(x)+ g '(x)( f (x)g(x))' = f '(x)g(x)+ f (x)g '(x)( f (x) / g(x))' = ( f '(x)g(x)− f (x)g '(x)) / g2 (x)
• Theorem (Chain Rule): If the derivatives g’(x) and f’(g(x)) exist and F(x)=f(g(x)) then
• Example: )2sin()cos()sin(2)(',1sin))(()(
sin)(,1)(2
2
xxxxFxxgfxFxxgxxf
==+==
=+=
)('))((')(' xgxgfxF =
L’Hopital’s Rule • As we saw a derivative is the limit of a ratio • Sometimes when computing the limit of a ratio of two
functions one can have an indeterminate result 0/0 • For these cases there exists a result called L’Hopital’s
rule stated in the following theorem • Theorem: Let f and g be two functions that have a
derivative at each point of an open interval (a,b). For a point in this interval let
If exists then
• Example:
18 Copyright © Luis Rodrigues
)('),(' xgxf0)(lim)(lim
00
==→→
xgxfxxxx0x
)(')('lim
0 xgxf
xx→ )(')('lim
)()(lim
00 xgxf
xgxf
xxxx →→=
111)cos(lim,
00)sin(lim)0sin()0sin(lim
0,sin)(
0000
0
0
=⇒===−+
=
==
→→→ dxdfh
hh
hh
dxdf
xxxf
hhh
Stationary Points of Functions • The derivative of a function at a point is the slope of the
tangent to the graph of the function at that point. If the derivative is zero then the slope of the tangent is zero
• Points for which the derivative is zero are thus called stationary points of a function. They can be a maximum or a minimum (see figure below)
19 Copyright © Luis Rodrigues
x
)(xf
xmaxxmin
f '(xmin ) = f '(xmax ) = 0Necessary Condition