![Page 1: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/1.jpg)
Neural Inverse Knitting: From Images to Manufacturing Instruction
Alexandre Kaspar*, Tae-Hyun Oh*, Liane Makatura,Petr Kellnhofer and Wojciech Matusik
MIT CSAIL
Pacific Ballroom #137
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 2: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/2.jpg)
Industrial Knitting
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 3: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/3.jpg)
Industrial Knitting• Whole garments from scratch
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 4: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/4.jpg)
Industrial Knitting• Control of individual needles
• Whole garments from scratch
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 5: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/5.jpg)
Knitted Garment & Patterns
Many garments are knitted:
• Beanies, scarves
• Gloves, socks and underwear
• Sweaters, sweatpants
Current machines can create those garments seamlessly (no sewing needed).
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 6: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/6.jpg)
Knitted Garment & Patterns
Those garments have various types of surface patterns (knitting patterns).
These can be fully controlled by industrial knitting machine.
= User customization!
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 7: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/7.jpg)
Machine Knitting Programming
Low-level machine code requires skilled experts= knitting masters
Good news
• Many hand knitting patterns available online and in books
• Online communities of knitting enthusiasts sharing patterns
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 8: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/8.jpg)
Scenario
1.User takes picture of knitting pattern
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 9: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/9.jpg)
Scenario
1.User takes picture of knitting pattern
2.System creates knitting instructions
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
InverseNeuralKnitting
![Page 10: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/10.jpg)
Scenario
1.User takes picture of knitting pattern
2.System creates knitting instructions
3.User reuses pattern for new garment
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
MachineKnitting
![Page 11: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/11.jpg)
Dataset: DSL
Domain Specific Language (DSL) for regular knitting patterns
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Basic operations Cross operations
StackOrderMove operations
![Page 12: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/12.jpg)
Dataset: Capture
Capture setup with steel rods to normalize tension
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 13: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/13.jpg)
Dataset Content
• Paired instructions with real (2,088) and synthetic (14,440) images.
• Available on project page.
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 14: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/14.jpg)
Learning Problem
Mapping images to discrete instruction maps
= CE loss minimization
Using two domains of input data (one real, one synthetic)
= How to best combine both
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 15: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/15.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Generalization gap
With probability at least 1 − 𝛿
Ideal min.
![Page 16: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/16.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Generalization gap
With probability at least 1 − 𝛿
Empirical min.
Ideal min.
![Page 17: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/17.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
With probability at least 1 − 𝛿
![Page 18: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/18.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Parameter dependent term
With probability at least 1 − 𝛿
![Page 19: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/19.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Ideal error of the combined losses
With probability at least 1 − 𝛿
![Page 20: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/20.jpg)
Generalization Bound with Two Domains
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Discrepancy between distributions
With probability at least 1 − 𝛿
![Page 21: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/21.jpg)
Data distributions
• Two different distribution types
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Real data Synthetic data
![Page 22: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/22.jpg)
Data distributions
• Two different distribution types
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Real data Synthetic data
![Page 23: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/23.jpg)
From synthetic to real
• S+U Learning [Shrivastava’17]
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Real data Synthetic data
![Page 24: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/24.jpg)
From synthetic to real
• S+U Learning[Shrivastava’17]
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Real-looking data Synthetic data
![Page 25: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/25.jpg)
From synthetic to real
• One-to-many mapping!
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 26: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/26.jpg)
From synthetic to real
• One-to-many!
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
??
?Color
Tension YarnLighting
![Page 27: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/27.jpg)
From real to synthetic
• Many-to-one!
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Regular / Normalized
ColorTension YarnLighting
![Page 28: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/28.jpg)
Network composition
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
![Page 29: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/29.jpg)
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Gro
un
d Tru
th
Test Resu
lts
Ou
r Resu
lt
![Page 30: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/30.jpg)
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Gro
un
d Tru
th
Test Resu
lts
Ou
r Resu
lt
![Page 31: Neural Inverse Knitting: From Images to Manufacturing ...deepknitting.csail.mit.edu/slides.pdfNeural Inverse Knitting: From Images to Manufacturing Instruction Alexandre Kaspar*, Tae-Hyun](https://reader033.vdocuments.us/reader033/viewer/2022042016/5e74ac6ba6567428bc6a6514/html5/thumbnails/31.jpg)
Pacific Ballroom #137, http://deepknitting.csail.mit.edu
Pacific Ballroom #137http://deepknitting.csail.mit.edu
Pacific Ballroom #137http://deepknitting.csail.mit.edu