3d reconstruction for ar - arm architecture › cfs-file › __key › communityserver... ·...
TRANSCRIPT
![Page 1: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/1.jpg)
3D Reconstruction for AR
Prof Victor Adrian Prisacariu
Active Vision LabDepartment of Engineering Science, University of Oxford
6D.ai
![Page 2: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/2.jpg)
What do we want?
• in unstructured, real world environments;
• with little user intervention;
• in real time, on a mobile phone.
• without depth cameras.
Track and reconstruct the world:
![Page 3: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/3.jpg)
3D Models for Objects and Scenes
Full ScenesObjects
![Page 4: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/4.jpg)
3D Object Reconstruction and Tracking
![Page 5: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/5.jpg)
How would we do it?
Image IMU
Pose3D shape
3D tracking
Image statsback-projection
3D segmentation
Prob volume
Image stats
Pose
o
pti
mis
atio
nSh
ape
op
tim
isat
ion
![Page 6: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/6.jpg)
Pose Optimisation
𝐸 Φ = − log
𝑥𝜖𝐼
𝐻𝑒 Φ 𝑃𝑓 + 1 − 𝐻𝑒 Φ 𝑃𝑏
𝑃𝑏
𝑃𝑓
Φ Assumes known 3D shape and per-pixel image statistics.
𝑃𝑓, 𝑃𝑏 – image statistics (e.g. histograms)
Φ – contour embedding SDF.
𝐻𝑒 – Heaviside function
Φ 𝑝𝑜𝑠𝑒
𝛿𝐸 Φ, 𝑝𝑜𝑠𝑒
𝛿𝑝𝑜𝑠𝑒2D Segmentation and 3D pose:
![Page 7: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/7.jpg)
𝜕𝐸
𝜕pose= term wrt P𝑓 − 𝑃𝑏 ×
𝜕SDF
𝜕position×𝜕position
𝜕pose
Requires:
• Fast rendering of 3D shape
• Signed distance transform + derivative.
Pose Derivative
![Page 8: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/8.jpg)
Fast Rendering
• Model is stored as 3D volume: standard rendering (raycasting) is very slow.
• We use a hierarchical binary raycaster.– Alternate between image resizing and raycasting around the contour.
![Page 9: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/9.jpg)
SDF + Derivatives
• Computing full SDF + derivatives is very slow.
• We use per-contour-point local approximations.
![Page 10: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/10.jpg)
SDF + Derivatives
𝐱𝑖
𝐱𝑖+1
[𝑥 − 1, 𝑦] [𝑥, 𝑦]
• Values of the SDF are obtained by following the per point normal.
• The SDF derivatives are computed using finite differences.
![Page 11: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/11.jpg)
IMU Integration
• The mapping from shape to pose is ambiguous.• We use the mobile phone IMU to provide disambiguated rotation at each
frame.
𝑅𝑝𝑜Previous pose
𝑅𝑡0 𝑅𝑡1Inertial Sensor
𝑅𝑝1 Current pose
Preset calibration𝐶
![Page 12: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/12.jpg)
Tracking Results
We obtain speeds > 80 fps on a phone.
![Page 13: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/13.jpg)
Shape Optimisation
• I assume known 3D pose and per-pixel image likelihoods.• For a set of images, we build inside/outside membership functions.• These represent the probability of a voxel being:
– Inside of the shape (i.e. foreground).– Outside of the shape (i.e. background).
• Final shape obtained using a 3D segmentation optimisation.
![Page 14: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/14.jpg)
Reconstruction Results
![Page 15: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/15.jpg)
Reconstruction Results
![Page 16: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/16.jpg)
Reconstruction Results
![Page 17: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/17.jpg)
Reconstruction Results
![Page 18: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/18.jpg)
Conclusions
• I demoed an approach for 3D tracking and showed how you could connected it to a reconstruction stage.
• We can get :
– state of the art 3D tracking speed.
– state of the art space carving based reconstruction results.
• Processing is fast enough to run on a mobile phone at over 80fps.
![Page 19: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/19.jpg)
3D Models for Objects and Scenes
Full ScenesObjects
![Page 20: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/20.jpg)
Depth Fusion
Integration of depth images: KinectFusion [Newcombe et al, 2011]
![Page 21: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/21.jpg)
Our Depth Fusion
Integration of depth images: InfiniTAM
more than 10x speedup (up to kHz speeds).runs on mobile devices.
![Page 22: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/22.jpg)
InfiniTAM
DEMO
![Page 23: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/23.jpg)
Map Representation
the band holds a truncated SDFthe map only stores voxels close to the surfacesthe T-SDF is discretized into voxels
![Page 24: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/24.jpg)
Map Representation
![Page 25: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/25.jpg)
Main Processing Steps
Tracking
Localises camera relative to the world
model
Fusion
Fuses novel depth data into existing world
model
Rendering
Renders scene from current viewpoint
![Page 26: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/26.jpg)
Tracking Fusion Rendering
1.Allocation2.Visible List3.Data Fusion4.Swapping
![Page 27: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/27.jpg)
1.Allocation2.Visible List3.Data Fusion4.Swapping
Tracking Fusion Rendering
![Page 28: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/28.jpg)
1.Allocation2.Visible List3.Data Fusion4.Swapping
Tracking Fusion Rendering
![Page 29: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/29.jpg)
Tracking Fusion Rendering
1.Allocation2.Visible List3.Data Fusion4.Swapping
![Page 30: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/30.jpg)
1.Allocation2.Visible List3.Data Fusion4.Swapping
Tracking Fusion Rendering
![Page 31: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/31.jpg)
1.Allocation2.Visible List3.Data Fusion4.Swapping
Tracking Fusion Rendering
![Page 32: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/32.jpg)
1.Allocation2.Visible List3.Data Fusion4.Swapping
Tracking Fusion Rendering
![Page 33: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/33.jpg)
Rendering the map:• Cast ray for each pixel.• Take steps until D(X) = 0
Tracking Fusion Rendering
![Page 34: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/34.jpg)
How to get it faster?• Predict plausible depth range• Optimise read operations.• Draw only when needed.
Tracking Fusion Rendering
![Page 35: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/35.jpg)
How to get it faster?• Predict plausible depth range• Optimise read operations.• Draw only when needed.
The visible list allows us to predict depth range:• Forward project visible box • Bounding box rather than full polygon.• Low resolution projection
Tracking Fusion Rendering
![Page 36: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/36.jpg)
Tracking Fusion Rendering
How to get it faster?• Predict plausible depth range• Optimise read operations.• Draw only when needed.
![Page 37: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/37.jpg)
How to get it faster?• Predict plausible depth range• Optimise read operations.• Draw only when needed.‐ Full raycast every few frames.‐ Forward projection otherwise (much).
Tracking Fusion Rendering
![Page 38: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/38.jpg)
𝑝
𝑤𝑝 R𝑝 + t − 𝑉 ҧ𝑝𝑇𝑁 ҧ𝑝
2
𝑉 ҧ𝑝𝑝
[R, t]
Weighted ICP
Tracking Fusion Rendering
![Page 39: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/39.jpg)
Runtime Experiments
![Page 40: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/40.jpg)
Is it perfect? – No ☺
Biggest problem: tracking drift
loop closure
![Page 41: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/41.jpg)
Loop Closure
![Page 42: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/42.jpg)
Graph of Submaps
![Page 43: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/43.jpg)
Relative Constraints
• Track the same image in multiple submaps i and j: poses 𝐓t,i and 𝐓t,j.
• Pose between submaps: 𝐓t,i,j = 𝐓t,j−1𝐓t,i
• Robustly aggregate over time t to get final estimate 𝐓i,j.
• Stop tracking old scene on tracker failure.
• Also add constraints on relocalisation.
![Page 44: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/44.jpg)
Pose Graph Optimisation
Find global pose for each submap such that relative constraints are satisfied.
𝐏 pose of submap𝐓 relative constraint
𝐯 𝐓 = 𝐪 𝐓 , 𝐭 𝐓T
𝑖,𝑗
𝐯(𝐏i, 𝐏j, 𝐓i,j)
![Page 45: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/45.jpg)
Result: Drift is compensated …
![Page 46: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/46.jpg)
… and processing is still quick
• Processing time: 7.1 – 8.5 ms per frame
• Remains constant
![Page 47: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/47.jpg)
Depth cameras …
• Take space.
• Use lots of power.
• Do not work outside.
• We can do better ☺
![Page 48: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/48.jpg)
Live Meshing Demo
DEMO
![Page 49: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/49.jpg)
Underlying 3D Reconstruction
![Page 50: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/50.jpg)
Underlying 3D Reconstruction
DEMO
![Page 51: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/51.jpg)
Reconstructions can be very big …
DEMO
![Page 52: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/52.jpg)
How does it work?
![Page 53: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/53.jpg)
How do I use it?
• You’ll need:– iOS 11.4+
– Xcode 9.4.1 + ARKit 1.5+
– Unity 2018.2+
– iPhone 8 and higher.
• Sign-up and get username + SDK.
• Install the SDK in your app.– Standard drag and drop .framework on iOS.
– Developer keys need to be specified in Info.plist.
![Page 54: 3D Reconstruction for AR - ARM architecture › cfs-file › __key › communityserver... · 2018-08-19 · 3D Reconstruction for AR Prof Victor Adrian Prisacariu Active Vision Lab](https://reader033.vdocuments.us/reader033/viewer/2022052802/5f1d4fb2cbc3242a8669e017/html5/thumbnails/54.jpg)
How do I use it?
Init:
SixDegreesSDK_Initialize(EAGLContext*); // init with this
SixDegreesSDK_IsInitialized(); // wait until this returns true
Get pose:
float pose[16];
SixDegreesSDK_GetPose(pose, 16); // get the pose here!
Get mesh:
int blockBufferSize, vertexBufferSize, faceBufferSize;
SixDegreesSDK_GetMeshBlockInfo(&blockBufferSize, &vertexBufferSize, &faceBufferSize); // gets the live mesh info
int blockBuffer[blockBufferSize];
float vertexBuffer[vertexBufferSize];
int faceBuffer[faceBufferSize];
SixDegreesSDK_GetMeshBlocks(blockBuffer, vertexBuffer, faceBuffer, blockBufferSize, vertexBufferSize, faceBufferSize); // gets the live mesh