[KOCW 선형대수] 11강. 벡터투영과 최소제곱법

제 11강. 벡터투영과 최소제곱법

(b−x̂ a)⊥a$\stackrel{}{}\mathrm{}$ = (b−x̂ A)⊥A$\stackrel{}{}\mathrm{}$

=> AT(b−x̂ A)=0${}^{}\stackrel{}{}\mathrm{}$

ATb−ATAx̂ =0${}^{}{}^{}\stackrel{}{}\mathrm{}$

ATb=ATAx̂ ${}^{}{}^{}\stackrel{}{}$

x̂ =(ATA)−1ATb$\stackrel{}{}{}^{}{}^{\mathrm{}}{}^{}$

ℙ=Ax̂ $\mathbb{}\stackrel{}{}$

=A(ATA)−1ATb${}^{}{}^{\mathrm{}}{}^{}$

투영의 개념과 최소제곱법의 개념이 한 번에 다 들어가있다.

Projections and Least Squares

• For more equations than the unknowns, there is usually no solution.

=> Overconstrained cases ( mxn matrix ( m > n ) )

• How to find an optimal Solution ? 

=> minimizing errors ||Ax−b||2=E2${}^{\mathrm{}}{}^{\mathrm{}}$ ( 거리의 개념으로 다가가기 )

Least square solution of 1 unknown -> projection onto the line a

• Orthogonality of a and e

-> The error vector e$$ connecting b$$ to p$$ must be perpendicular(수직) to a$$

aT(b−x̂ a)${}^{}\stackrel{}{}$

Least square problems with several variables

to project b onto a subspace ( m x n : A ( m > n ) )

E=||Ax=b||$$

x̂ $\stackrel{}{}$ : solution of least squares to minimize E$$

e=(b−Ax̂ )⊥space$\stackrel{}{}\mathrm{}$

(1) Since column space is perpendicular to left nullspace, b−Ax̂ $\stackrel{}{}$ is m the left nullspace

AT(b−Ax̂ )=0${}^{}\stackrel{}{}\mathrm{}$ or ATAx̂ =ATb${}^{}\stackrel{}{}{}^{}$

(2) The error vector must be orthogonal to each column vector

aTi(b−Ax̂ )=0${}_{}^{}\stackrel{}{}\mathrm{}$

⎡⎣⎢⎢⎢aTi⋮aTn⎤⎦⎥⎥⎥[b−Ax̂ ]=0$$\begin{array}{c}{}_{}^{}\\ \\ {}_{}^{}\end{array}\begin{array}{c}\stackrel{}{}\end{array}\mathrm{}$$

aTi${}_{}^{}$들을 AT${}^{}$로 표현할 수 있다.

AT[b−Ax̂ ]=ATb=ATAx̂ ${}^{}\stackrel{}{}{}^{}{}^{}\stackrel{}{}$

• Normal equation = ATAx̂ =ATb${}^{}\stackrel{}{}{}^{}$ ( 역행렬 있는지 모를 때 )

• Best estimate x̂ $\stackrel{}{}$ for invertibility of ATA${}^{}$ => x̂ =(ATA)−1ATb$\stackrel{}{}{}^{}{}^{\mathrm{}}{}^{}$

• Projection => p=Ax̂ =A(ATA)−1ATb$\stackrel{}{}{}^{}{}^{\mathrm{}}{}^{}$

Remark 1.

if b$$ is in ℂ(A)$\mathbb{}$ -> b=Ax$$

The projection of b$$ is still b$$

vector안에 존재해서 가까운게 자기 자신임

p=A(ATA)−1ATb=A(ATA)−1ATAx=Ax=b${}^{}{}^{\mathrm{}}{}^{}{}^{}{}^{\mathrm{}}{}^{}$

Remark 2.

If b$$ is perpendicular to every column

ATb=0${}^{}\mathrm{}$ -> b$$ in left nullspace

p=A(ATA)−1ATb=0${}^{}{}^{\mathrm{}}{}^{}\mathrm{}$

-> projected onto zero

Remark 3.

A$$ is square & invertible => p=b$$

p=A(ATA)−1ATb=A(A−1)(AT)−1ATb=b${}^{}{}^{\mathrm{}}{}^{}{}^{\mathrm{}}{}^{}{}^{\mathrm{}}{}^{}$

Least squares Fitting of Data

some observation errors -> not exactly fitted

[E2=∑e2i]${}^{\mathrm{}}{}_{}^{\mathrm{}}$ 을 minimize 하는 것이다.

y=ax+b$$

⎡⎣⎢⎢⎢⎢x1x2⋮xn111⎤⎦⎥⎥⎥⎥[ab]=⎡⎣⎢⎢⎢⎢y1y2⋮yn⎤⎦⎥⎥⎥⎥$$\begin{array}{cc}{}_{\mathrm{}}& \mathrm{}\\ {}_{\mathrm{}}& \mathrm{}\\ \\ {}_{}& \mathrm{}\end{array}\begin{array}{c}\\ \end{array}\begin{array}{c}{}_{\mathrm{}}\\ {}_{\mathrm{}}\\ \\ {}_{}\end{array}$$

이것이 결국 Ax=b$$ 로 표현이 되는 것이다.