convolutional dct image super-resolutionsignal.ee.psu.edu/research/ordsr_files/ordsr_cdct.pdf ·...
TRANSCRIPT
![Page 1: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/1.jpg)
Convolutional DCT Image Super-resolutionProgress Report & Next Step
Department of Electrical Engineering
Pennsylvania State University
2018
![Page 2: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/2.jpg)
Outline
1 Convolutional DCT Network
2D DCT and representation process2D DCT and 2D IDCT by transpose convolutional neural networkOrthogonality constrainsTraining process
2 Preliminary Results
SSIM, PSNR, IFCCDCT layer: threshold and orthogonality
3 Next Step
Complexity order constrainsDe-noising and SR
2
![Page 3: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/3.jpg)
2D DCT & Representation
Compute DCT coefficients block Xij for xij
Xij(k1, k2) =
N−1∑n2=0
N−1∑n1=0
xij(n1, n2)× fk1,k2(n1, n2)
Note that xij and Xij is the same size 8× 8.Coefficient Xij(k1, k2) represents the significance of basis fk1k2
embedded in xij . 3
![Page 4: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/4.jpg)
2D IDCT & Representation
Compute x(i,j) from DCT coefficients block Xij
xij(n1, n2) =
N−1∑k2=0
N−1∑k1=0
Xij(k1, k2)× fk1,k2(n1, n2)
Coefficient Xij(k1, k2) represents the significance of basis fk1k2
embedded in xij .
4
![Page 5: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/5.jpg)
2D DCT & IDCT
Process the whole image and inverse procedure:
Each block only contains its own spatial information.5
![Page 6: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/6.jpg)
DCT by neural network
Treat DCT basis functions as filters and organize in zig-zag order:
Re-index fk1,k2 to fk with zig-zag mapping function Zig:Zig{(k1, k2)} = k where(k1, k2) ∈ (0, . . . , N − 1)× (0, . . . , N − 1)→ k ∈ (1, . . . , N ×N)Zig{(k1, k2)} = k and Zig−1{k} = (k1, k2)Now here are {fk}N×N
k=1 DCT basis filters, each of size N ×N 6
![Page 7: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/7.jpg)
DCT by neural network
Convolve image x ∈ R(W×H) with {fk}N×Nk=1 DCT basis filters:
Convolve without overlapping, with shift of N .
For a fixed k, x ∗ fk gives DCT coefficients Xij(k) of all x’s xij
x ∗ fk = Xk, where Xk ∈ RWN ×H
N
As k increases, smaller details are captured by x ∗ f1
7
![Page 8: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/8.jpg)
DCT by neural network
8
![Page 9: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/9.jpg)
DCT by neural network
8
![Page 10: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/10.jpg)
DCT by neural network
8
![Page 11: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/11.jpg)
DCT by neural network
8
![Page 12: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/12.jpg)
DCT by neural network
8
![Page 13: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/13.jpg)
DCT by neural network: proofProve: DCT by neural network generates the same DCT coefficients Xij
for xij , ij denotes one 8× 8 blockFor a fixed block ij, k1, k2, n1, n2 = 1, . . . , NDCT:
Xij(k1, k2) =
N−1∑n2=0
N−1∑n1=0
xij(n1, n2)× fk1,k2(n1, n2)
DCT by neural network:
X(i, j)k =∑(i+1)×N
n2=i×N
∑(j+1)×Nn1=j×N x(n1, n2)× fk(n1, n2)
For fixed ij, the X(i, j)k can be re-indexed as:
Xij(k) =∑N
n2=0
∑N0 xij(n1, n2)× fk(n1, n2)
Since Zig−1(fk) = fk1,k2 , we have:
Zig−1(Xij(k)) =
N∑n2=0
N∑n1=0
xij(n1, n2)× Zig−1(fk(n1, n2))
=
N∑n2=0
N∑n1=0
xij(n1, n2)× fk1,k2(n1, n2) = Xij(k1, k2)
Thus, DCT by neural network generates zig-zag arranged blocked DCT.
![Page 14: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/14.jpg)
Inverse DCT by neural network
Transpose-convolve features X ∈ RN2×WN ×H
N with {fk}N×Nk=1 DCT basis:
Padding: X̄k(i, j) =
{Xk(k, l), if i = 8k and j = 8l
0, Otherwise,
k = 1, . . . WN , l = 1, . . . H
N ,
Transpose convolve: convolution with shifting 1
For a fixed k, X̄k ∗ fk gives all xij ’s k-th spatial component
X̄k ∗ fk = xk, where xk ∈ RW×H
The final results: x =∑N2
k=1 X̄k ∗ fk
10
![Page 15: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/15.jpg)
Inverse DCT by neural network
11
![Page 16: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/16.jpg)
Inverse DCT by neural network
11
![Page 17: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/17.jpg)
iDCT by neural network: proofProve: iDCT by neural network generates the same xij from DCTcoefficients Xij , ij denotes one 8× 8 blockFor a fixed block ij, k1, k2, n1, n2 = 1, . . . , NiDCT:
xij(n1, n2) =
N−1∑k2=0
N−1∑k1=0
Xij(k1, k2)× fk1,k2(n1, n2)
iDCT by neural network:
xij(n1, n2) =∑N×N
k=1
∑(i+1)×Nl=i×N
∑(j+1)×Nm=j×N X̄k(i × N + n1 − l, j × N +
n2 −m)× fk(l,m)
X̄k(i×N + n1 − l, j ×N + n2 −m) 6= 0 while n1 − l + i×N = i×N ,
reordering the index we have: xij(n1, n2) =∑N×N
k Xk(i, j)× fk(n1, n2)
![Page 18: Convolutional DCT Image Super-resolutionsignal.ee.psu.edu/research/ORDSR_files/ORDSR_CDCT.pdf · DCT by neural network Convolve image x 2R(W H) with ff kg N N k=1 DCT basis lters:](https://reader033.vdocuments.us/reader033/viewer/2022052002/6014e23854728a7df50cb740/html5/thumbnails/18.jpg)
iDCT by neural network: proofProve: iDCT by neural network generates the same xij from DCTcoefficients Xij , ij denotes one 8× 8 blockFor a fixed block ij, k1, k2, n1, n2 = 1, . . . , NiDCT:
xij(n1, n2) =
N−1∑k2=0
N−1∑k1=0
Xij(k1, k2)× fk1,k2(n1, n2)
iDCT by neural network:
Since Zig−1(fk) = fk1,k2and Zig−1(Xk) = Xk1,k2
, we have:
xij(n1, n2) =
N×N∑k
Xk(i, j)× fk(n1, n2)
=
N−1∑k1=0
N−1∑k2=0
Xk1,k2(i, j)× fk1,k2
(n1, n2)
=
N−1∑k1=0
N−1∑k2=0
Xij(k1, k2)× fk1,k2(n1, n2)
iDCT by neural network generates same xij(n1, n2) for a given block.