boolean searchable symmetric encryption with …...•from matryoshka filters (new bloom filter data...

Boolean Searchable Symmetric Encryption with Worst-Case Sub-Linear Complexity

Seny Kamara Tarik Moataz

2

Bob

2

Bob

2

Bob

I can’t search!

Many Approaches

• Stream ciphers [SWP00]

• Bucketing [HILM02]

• Structured and searchable encryption (STE/SSE) [CGKO06,CK10]

• Oblivious RAM (ORAM) [GO96]

• Functional encryption (e.g., PEKS) [BCOP06]

• Multi-party computation (MPC)

• Property-preserving encryption (PPE) [AKSX04,BBO06,BCLO09]

• Fully-homomorphic encryption [G09]

3

Efficiency

Security Expressiveness

4

Expressiveness

Efficiency

OXT

Blind Seer BOXT

Searchable Symmetric Encryption

5

RR Naïve RH Naive

Boolean

SNF

Expressiveness

Efficiency

OXT

Blind Seer BOXT

Searchable Symmetric Encryption

5

RR Naïve RH Naive This Work

Boolean

SNF

Related Work

• OXT [CJJKRS’13]

• Sub-linear for conjunctive queries

• Linear for disjunctive

• Linear for (arbitrary) Boolean queries

• Non-interactive

• Blind Seer [PKVKMCGKB’14]

• Sub-linear for arbitrary Boolean queries

• Interactive

• Logarithmic multiplicative overhead over the result set

6

Related Work

• OXT [CJJKRS’13]

• Sub-linear for conjunctive queries

• Linear for disjunctive

• Linear for (arbitrary) Boolean queries

• Non-interactive

• Blind Seer [PKVKMCGKB’14]

• Sub-linear for arbitrary Boolean queries

• Interactive

• Logarithmic multiplicative overhead over the result set

6

Related Work

• OXT [CJJKRS’13]

• Sub-linear for conjunctive queries

• Linear for disjunctive

• Linear for (arbitrary) Boolean queries

• Non-interactive

• Blind Seer [PKVKMCGKB’14]

• Sub-linear for arbitrary Boolean queries

• Interactive

• Logarithmic multiplicative overhead over the result set

6

Black-Box Constructions

• IEX: “purely” disjunctive SSE • from any single-keyword SSE

7

Black-Box Constructions

• IEX: “purely” disjunctive SSE • from any single-keyword SSE

• BIEX: Boolean SSE • from IEX

7

Black-Box Constructions

• IEX: “purely” disjunctive SSE • from any single-keyword SSE

• BIEX: Boolean SSE • from IEX

• DIEX: dynamic disjunctive SSE • from any dynamic single-keyword SSE

• Forward Secure

7

Concrete Constructions

• IEX-2Lev • from 2Lev [CJJJKRS14]

8

Concrete Constructions

• IEX-2Lev • from 2Lev [CJJJKRS14]

• BIEX-2Lev • from IEX-2Lev

8

Concrete Constructions

• IEX-2Lev • from 2Lev [CJJJKRS14]

• BIEX-2Lev • from IEX-2Lev

• ZMF: new single-keyword SSE • from Matryoshka filters (new Bloom filter data structure)

• Linear search complexity but very compact

8

Concrete Constructions

• IEX-2Lev • from 2Lev [CJJJKRS14]

• BIEX-2Lev • from IEX-2Lev

• ZMF: new single-keyword SSE • from Matryoshka filters (new Bloom filter data structure)

• Linear search complexity but very compact

• IEX-ZMF • from ZMF

8

Background: Data Structures

9

Background: Data Structures

• Dictionaries map labels to values

• Get: DX[w3] returns id2

9

w1

w2

w3

id1

id3

id2

Dictionary DX

Background: Data Structures

• Dictionaries map labels to values

• Get: DX[w3] returns id2

• Multi-maps map labels to tuples

• Get: MM[w3] returns (id2 , id4)

9

w1

w2

w3

id1

id3

id2

Dictionary DX

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

Background: Encrypted Data Structures [CK’10]

10

w1

l2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

Setup 1k, , w2

Background: Encrypted Data Structures [CK’10]

10

w1

l2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM w2

w1

id3

id3

Encrypted Multi-map EMM

w3

w1

id2

id4

w3 id4

w1 id1

Setup 1k, , w2

Background: Encrypted Data Structures [CK’10]

11

Token , w1

Background: Encrypted Data Structures [CK’10]

11

Token , w1 w1

Background: Encrypted Data Structures [CK’10]

12

Get , w1

w2

w1

id3

id3

Encrypted Multi-map EMM

w3

w1

id2

id4

w3 id4

w1 id1

Background: Encrypted Data Structures [CK’10]

12

Get , w1 id3 id4 id1

Response-hiding

w2

w1

id3

id3

Encrypted Multi-map EMM

w3

w1

id2

id4

w3 id4

w1 id1

Background: Encrypted Data Structures [CK’10]

13

Encrypted Multi-Map

Background: Encrypted Data Structures [CK’10]

13

Encrypted Multi-Map

Encrypted Inverted

Index

Background: Encrypted Data Structures [CK’10]

13

Single Keyword SSE

[SWP’00], [Goh’03], [CGKO’06], [CK10], [KPR’12], [KP’13], [CJJKRS’13], [CJJJKRS’14],

[Bost’16] …

Encrypted Multi-Map

Encrypted Inverted

Index

Adaptive Security

14

Adaptive Security

14

Real

Multi-map MM

Adaptive Security

14

Real

Multi-map MM

Encrypted Multi-map EMM

Adaptive Security

14

Real

Multi-map MM

Encrypted Multi-map EMM

wi

wi

Adaptive Security

14

Real

Multiple Time

Multi-map MM

Encrypted Multi-map EMM

wi

wi

Multi-map MM

Adaptive Security

14

Real Ideal

Multiple Time

Setup Leakage ℒ𝑆

Multi-map MM

Encrypted Multi-map EMM

wi

wi

Multi-map MM

Adaptive Security

14

Real Ideal

Multiple Time

Setup Leakage ℒ𝑆

Multi-map MM

Encrypted Multi-map EMM

wi

Encrypted Multi-map EMM Encrypted Multi-map EMM

wi

Multi-map MM

Adaptive Security

14

Real Ideal

Multiple Time

Setup Leakage ℒ𝑆

Query Leakage ℒ𝑄

Multi-map MM

Encrypted Multi-map EMM

wi wi

Encrypted Multi-map EMM Encrypted Multi-map EMM

wi

Multi-map MM

Adaptive Security

14

Real Ideal

Multiple Time

Setup Leakage ℒ𝑆

Query Leakage ℒ𝑄

Multi-map MM

Encrypted Multi-map EMM

wi wi

Encrypted Multi-map EMM Encrypted Multi-map EMM

wi wi wi

Multi-map MM

Adaptive Security

14

Real Ideal

Multiple Time

Setup Leakage ℒ𝑆

Query Leakage ℒ𝑄

Real ≈ Ideal

Multi-map MM

Encrypted Multi-map EMM

wi wi

Encrypted Multi-map EMM Encrypted Multi-map EMM

wi wi wi

Overview

• Multi-maps (indexes) can be viewed as collection of sets

15

Overview

• Multi-maps (indexes) can be viewed as collection of sets

• Disjunctive keyword queries can be viewed as set unions on those sets

15

Overview

• Multi-maps (indexes) can be viewed as collection of sets

• Disjunctive keyword queries can be viewed as set unions on those sets

• Naïve set union includes items with multiplicity (redundancy) • Implies sub-optimal communication complexity or heavy leakage

15

Overview

• Multi-maps (indexes) can be viewed as collection of sets

• Disjunctive keyword queries can be viewed as set unions on those sets

• Naïve set union includes items with multiplicity (redundancy) • Implies sub-optimal communication complexity or heavy leakage

• Inclusion/exclusion-based unions remove redundancy • Implies optimal communication complexity and less leakage

15

Overview

• Multi-maps (indexes) can be viewed as collection of sets

• Disjunctive keyword queries can be viewed as set unions on those sets

• Naïve set union includes items with multiplicity (redundancy) • Implies sub-optimal communication complexity or heavy leakage

• Inclusion/exclusion-based unions remove redundancy • Implies optimal communication complexity and less leakage

• New (plaintext) set structure with I/E-based union operations

15

Overview

• Multi-maps (indexes) can be viewed as collection of sets

• Disjunctive keyword queries can be viewed as set unions on those sets

• Naïve set union includes items with multiplicity (redundancy) • Implies sub-optimal communication complexity or heavy leakage

• Inclusion/exclusion-based unions remove redundancy • Implies optimal communication complexity and less leakage

• New (plaintext) set structure with I/E-based union operations

• Encrypted structure that supports I/E-based unions

15

Overview: Multi-Maps as Sets

16

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

Overview: Multi-Maps as Sets

16

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

id1

id3

Id4

Overview: Multi-Maps as Sets

16

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

id1

id3

Id4

id3

Overview: Multi-Maps as Sets

16

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

id1

id3

Id2

Id4

id3

Id4

Overview: Disjunctive Search as Set Union

17

Q = w1 w2 ∨ w3 ∨

Overview: Disjunctive Search as Set Union

17

id1 id3

Id2

Id4

Q = w1 w2 ∨ w3 ∨

Overview: Inclusion/Exclusion-based Union

18

id1 id3

Id2

Id4

Overview: Inclusion/Exclusion-based Union

18

id1 id3

Id2

Id4

Id2

Id4

id1 id3

Id4

id3

Overview: Inclusion/Exclusion-based Union

18

id1 id3

Id2

Id4

Id2

Id4

id3

Overview: Inclusion/Exclusion-based Union

18

id1 id3

Id2

Id4

Id2

Id4

Overview: Inclusion/Exclusion-based Union

18

id1 id3

Id2

Id4

Id2

Id4

𝑤𝑖

𝑛

𝑖=1

= (−1)𝑖+1 # 𝑀𝑀 𝑤𝑗1 ∩⋯∩𝑀𝑀 𝑤𝑗𝑖1≤𝑗1<⋯<𝑗𝑖≤𝑛

𝑛

𝑖=1

#Lookup

Overview: Set Structure with I/E-based Unions

19

id1 id3

Id2

Id4

Overview: Set Structure with I/E-based Unions

19

id1 id3

Id2

Id4

id1 id3

id4

id3

Id2

Id4

Pre-processing

Overview: Set Structure with I/E-based Unions

20

id1 id3

id4

id3

Id2

Id4

Overview: Set Structure with I/E-based Unions

20

id1 id3

id4

id3

Id2

Id4

w1

w2

w3

id1 id3 id4

id3

id2 id4

Global Multi-map MM

Overview: Set Structure with I/E-based Unions

20

id1 id3

id4

id3

Id2

Id4

w1

w2

w3

id1 id3 id4

id3

id2 id4

Global Multi-map MM

Overview: Set Structure with I/E-based Unions

20

id1 id3

id4

id3

Id2

Id4

w1

w2

w3

id1 id3 id4

id3

id2 id4

Global Multi-map MM

w1 ⋀ w2

w1 ⋀ w3

id3

id4

Local Multi-map MM1

w2 ⋀ w1 id3

Local Multi-map MM2

w3 ⋀ w1 id4

Local Multi-map MM3

IEX: Setup

21

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

SetupIEX 1k,

IEX: Setup

21

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

SetupIEX 1k,

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

w2 ⋀ w1 E(id3; w2)

Encrypted local Multi-map EMM1

w3 ⋀ w1 E(id3; w3)

Encrypted local Multi-map EMM2

,

IEX: Setup

22

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

SetupIEX 1k, ,

IEX: Setup

22

w1

w2

w3

id1 id3 id4

id3

id2 id4

Multi-map MM

SetupIEX 1k, ,

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

1

2

3

Encrypted Dictionary EDX

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

w2 ⋀ w1 E(id3; w2)

Encrypted local Multi-map EMM1

w3 ⋀ w1 E(id3; w3)

Encrypted local Multi-map EMM2

IEX: Token

23

TokenIEX , w1 w3 ∨

IEX: Token

23

TokenIEX , w1 w3 ∨ w1

Global sub-token

IEX: Token

23

TokenIEX , w1 w3 ∨ w1 w3

Global sub-token

Global sub-token

IEX: Token

23

TokenIEX , w1 w3 ∨ w1 w3 1

Global sub-token

Global sub-token

dictionary sub-token

IEX: Token

23

TokenIEX , w1 w3 ∨ w1 w3 1 w1 ⋀ w3

Global sub-token

Global sub-token

dictionary sub-token

Local sub-token

IEX: Get

24

GetIEX , w1 w3 1 w1 ⋀ w3

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

1

2

3

Encrypted Dictionary EDX

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

w2 ⋀ w1 E(id3; w2)

Encrypted local Multi-map EMM1

w3 ⋀ w1 E(id3; w3)

Encrypted local Multi-map EMM2

IEX: Get

25

Get , w1

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

IEX: Get

25

Get , E(id3; w1) E(id4; w1) E(id3; w1) w1

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

IEX: Get

25

Get , E(id3; w1) E(id4; w1) E(id3; w1)

Get ,

w1

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

w3

IEX: Get

25

Get , E(id3; w1) E(id4; w1) E(id3; w1)

Get , E(id2; w3) E(id4; w3)

w1

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

w2 E(id3; w1)

Encrypted Global Multi-map EMM

w1 E(id3; l2)

w3 E(id2; w3)

w1 E(id4; w1)

w3 E(id4; w3)

w1 E(id1; w1)

w3

IEX: Lookup

26

Get , 1

1

2

3

Encrypted Dictionary EDX

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

w2 ⋀ w1 E(id3; w2)

Encrypted local Multi-map EMM1

w3 ⋀ w1 E(id3; w3)

Encrypted local Multi-map EMM2

IEX: Lookup

26

Get , 1

1

2

3

Encrypted Dictionary EDX

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

w2 ⋀ w1 E(id3; w2)

Encrypted local Multi-map EMM1

w3 ⋀ w1 E(id3; w3)

Encrypted local Multi-map EMM2

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3

IEX: Lookup

27

Get , w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3 w1 ⋀ w3

IEX: Lookup

27

Get , E(id4; w1) w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3 w1 ⋀ w3

IEX: Lookup

27

Get , E(id4; w1) w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3 w1 ⋀ w3

E(id3; w1) E(id4; w1) E(id3; w1)

E(id2; w3) E(id4; w3)

IEX: Lookup

27

Get , E(id4; w1) w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3 w1 ⋀ w3

E(id3; w1) E(id4; w1) E(id3; w1)

E(id2; w3) E(id4; w3)

IEX: Lookup

27

Get , E(id4; w1)

Result sent to the client

w1 ⋀ w2 E(id3; w1)

Encrypted local Multi-map EMM1

E(id4; w1) w1 ⋀ w3 w1 ⋀ w3

E(id3; w1) E(id4; w1) E(id3; w1)

E(id2; w3) E(id4; w3)

E(id3; w1) E(id3; w1)

E(id2; w3) E(id4; w3)

IEX: Leakage

• Black-box setup leakage • Setup leakage of global EMM

• Setup leakage of EDX

• Black-box query leakage for disjunction • Query leakage of global EMM

• Query leakage of EDX

28

IEX: Leakage

• Black-box setup leakage • Setup leakage of global EMM

• Setup leakage of EDX

• Black-box query leakage for disjunction • Query leakage of global EMM

• Query leakage of EDX

• Concrete setup leakage • Size of global MM

• Total size of local MM

• Concrete query leakage • Search and access pattern of global MM

• Search pattern of accessed local MMs

• Access pattern of accessed local MMs

• Tags of accessed local MMs

• Setup leakage of local MMs

• Search and access pattern of DX

28

IEX: Leakage

• Black-box setup leakage • Setup leakage of global EMM

• Setup leakage of EDX

• Black-box query leakage for disjunction • Query leakage of global EMM

• Query leakage of EDX

• Concrete setup leakage • Size of global MM

• Total size of local MM

• Concrete query leakage • Search and access pattern of global MM

• Search pattern of accessed local MMs

• Access pattern of accessed local MMs

• Tags of accessed local MMs

• Setup leakage of local MMs

• Search and access pattern of DX

28

Less leakage than OXT

IEX: Asymptotics

• Communication complexity is optimal

29

IEX: Asymptotics

• Communication complexity is optimal

• Worst-case search complexity (q keywords) • Sub-linear in where

29

IEX: Asymptotics

• Communication complexity is optimal

• Worst-case search complexity (q keywords) • Sub-linear in where

• Storage

29

Improving IEX Storage Overhead

• Can we make IEX more compact? • Problem is local EMMs are too large

30

Improving IEX Storage Overhead

• Can we make IEX more compact? • Problem is local EMMs are too large

• Use Z-IDX [Goh03] as local EMM? • Linear search complexity is OK

• Very compact (based on Bloom filters)

• Not adaptively-secure!

30

Improving IEX Storage Overhead

• Can we make IEX more compact? • Problem is local EMMs are too large

• Use Z-IDX [Goh03] as local EMM? • Linear search complexity is OK

• Very compact (based on Bloom filters)

• Not adaptively-secure!

• Z-IDX can be made adaptively-secure • But token size too large (far from optimal)

30

Improving IEX Storage Overhead

• Matryoshka filters • New nested Bloom filters with variable size and fixed hash functions

31

Improving IEX Storage Overhead

• Matryoshka filters • New nested Bloom filters with variable size and fixed hash functions

• Encrypted Matryoshka filters • Based on online ciphers

• Adaptively-secure

• Compact structure

• Optimal token size

• Linear search complexity

31

Improving IEX Storage Overhead

• Matryoshka filters • New nested Bloom filters with variable size and fixed hash functions

• Encrypted Matryoshka filters • Based on online ciphers

• Adaptively-secure

• Compact structure

• Optimal token size

• Linear search complexity

31

Improving IEX Storage Overhead

• Matryoshka filters • New nested Bloom filters with variable size and fixed hash functions

• Encrypted Matryoshka filters • Based on online ciphers

• Adaptively-secure

• Compact structure

• Optimal token size

• Linear search complexity

31

Evaluation (up to 61M keyword/id pairs)

32

Evaluation (up to 61M keyword/id pairs)

32 OXT 200 ms

Evaluation (up to 61M keyword/id pairs)

32 OXT 200 ms

10×

Clusion

• Encrypted search library • Open source under GPLv3 • Java

33

Clusion

• Encrypted search library • Open source under GPLv3 • Java

• Currently implements • SSE: 2Lev & ZMF • Dynamic SSE: forward-secure 2Lev (new) • Disjuntive SSE: IEX-2Lev & IEX-ZMF • Boolean SSE: BIEX-2Lev & BIEX-ZMF

33

Clusion

• Encrypted search library • Open source under GPLv3 • Java

• Currently implements • SSE: 2Lev & ZMF • Dynamic SSE: forward-secure 2Lev (new) • Disjuntive SSE: IEX-2Lev & IEX-ZMF • Boolean SSE: BIEX-2Lev & BIEX-ZMF

• In progress • Dynamic SSE: forse-1, forse-2 • Graph encryption: LGX

33

Thank you!

34

https://github.com/encryptedsystems/Clusion