authorization in apache kafka - seattle kafka meetup - ashish singh

1©Cloudera,Inc.Allrightsreserved.

AshishSingh|So:wareEngineer,Cloudera

Authoriza>oninApacheKaAa

2©Cloudera,Inc.Allrightsreserved.

•  So:wareEngineer@Cloudera• ContributedtoKaAa,Sentry,HiveandParquet• UsedtoworkinHPC• @singhasdev

AboutMe

3©Cloudera,Inc.Allrightsreserved.

4©Cloudera,Inc.Allrightsreserved.

AboutKaAa

• Publish/SubscribeMessagingSystem• Highthroughput(100’sofkmessages/sec)•  Lowlatency(sub-secondtolowseconds)•  Fault-tolerant(ReplicatedandDistributed)•  Supportsagnos>cmessaging•  Standardizesformatanddelivery• Hugecommunity

5©Cloudera,Inc.Allrightsreserved.

ArchitectureProducer

Consumer Consumer

Producers

KaAaCluster

Consumers

Broker Broker Broker Broker

Producer

Zookeeper

6©Cloudera,Inc.Allrightsreserved.

Authoriza>on

7©Cloudera,Inc.Allrightsreserved.

Authoriza>on

Authoriza>onisthefunc>onofspecifyingaccessrightstoresourcesrelatedtoinforma>onsecurityandcomputersecurityingeneralandtoaccesscontrolinpar>cular.Moreformally,"toauthorize"istodefineanaccesspolicy.–Wikipedia

8©Cloudera,Inc.Allrightsreserved.

Authoriza>on

Authoriza>onisthefunc>onofspecifyingaccessrightstoresourcesrelatedtoinforma>onsecurityandcomputersecurityingeneralandtoaccesscontrolinpar>cular.Moreformally,"toauthorize"istodefineanaccesspolicy.–Wikipedia

9©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

10©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

11©Cloudera,Inc.Allrightsreserved.

Authoriza>oninApacheKaAa

12©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

13©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?

14©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• KaAausesabinaryprotocoloverTCP.• TheprotocoldefinesallAPIsasrequestresponsemessagepairs.• RequestsaresentthroughaRequestchannela:erasessionisestablished.•  SessioncontainsPrincipalandHost.• Principalisoftheform<PrincipalType>:<PrincipalName>,.e.g.,User:foo,Group:analyst,etc.• Hosthasinfoonwheretherequesthasoriginatedfrom.•  ProvidesIP-Filtering.

15©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• PermissionTypes.• Allow• Deny

16©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• Opera>onssupportedinKaAa.• Read• Write• Create• Delete• Alter• Describe• ClusterAc>on

17©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• ResourceTypesinKaAa.• Cluster• Topic• ConsumerGroup

18©Cloudera,Inc.Allrightsreserved.

AccessPolicy

• AccessPolicyinKaAaisrepresentedasACL,AccessControlList.• ACLinKaAaiscomposedofthefollowing.• SetofPrincipals.• Permissiontype,i.e.,AlloworDeny.• SetofHosts.• SetofOpera>ons.

19©Cloudera,Inc.Allrightsreserved.

KaAa’sDefaultAuthorizerSimpleAclAuthorizer

20©Cloudera,Inc.Allrightsreserved.

SimpleAclAuthorizer

• Outoftheboximplementa>onoftheKaAaAuthorizer.•  Selfcontainedandnodependencieswithanyothervendororproviders.•  Ituseszookeeperasthestoragelayerforacls.ACLsarestoredinJSONformatdescribedunder/kaAa-acls/resource-type/<resource-name>.• UsesCachingtoavoidgoingtoZKforeachrequest.• DenytakesprecedenceoverAllowincompe>ngACLs.• WhennoACLisaoachedtoaresource,useconfigallow.everyone.if.no.acl.found.• WhenanyACLisaoachedtoaresourceonlyusersthatareintheallowedlisthaveaccess.AlluserswithnoexplicitallowACLsaredeniedaccessbydefault.• READorWRITEpermission=>DESCRIBEOpera>on.

21©Cloudera,Inc.Allrightsreserved.

SimpleAclAuthorizer–NotsoSimple

• OnlysupportsUserprincipal• PRoutforGroupPrincipalforsome>me.

• Nowaytouseusergroupmappingfromexternalservices,like,LDAP,AD,etc.• VeryKaAaspecificimplementa>on.• Notscalable.• HaszNodesizelimita>ons,defaultandrecommendedinonly1MB.• Concurrencyissues.

• Notproduc>onready.

22©Cloudera,Inc.Allrightsreserved.

23©Cloudera,Inc.Allrightsreserved.

Sentry

• Providesunifiedrolebasedauthoriza>onforvariouscomponents.• Hive•  Impala• HDFS• Sqoop• KaAa(ohyea!)

24©Cloudera,Inc.Allrightsreserved.

RBAC

• RoleBasedAuthoriza>onControl,RBAC,isapowerfulmechanismtomanageauthoriza>onforalargesetofusersanddataobjectsinatypicalenterprise.• Assigningprivilegestoauser.• Privilege->User• Privilege->Group->User• Privilege->Role->Group->User

• UsergroupmappingisconfigurableandcancomefromShellorexternalsystems,like,LDAP,AD,etc.

25©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

26©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  DataEngineerscanREADfromCustomerData.•  DataEngineerscanWRITEtoEnrichedData.•  AnalystscanREADfromEnrichedData.•  AnalystscanWritetoSalesPredic>on.•  CFOcanREADfromSalesPredic>on.

27©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  Roles:•  ReadCustomerData=>READfromCustomerData•  WriteEnrichedData=>WRITEtoEnrichedData•  ReadEnrichedData=>READfromEnrichedData•  WriteSalesPredic>ons=>WRITEtoSalesPredic>ons•  ReadSalesPredic>ons=>READfromSalesPredic>ons•  AllRead=>READfromCustomer,EnrichedandSalesPredic>ons

•  RolestoGroups:•  DataEngineers=>ReadCustomerData,ReadEnrichedData,

WriteEnrichedData•  Analysts=>ReadEnrichedData,ReadSalesPredic>ons,

WriteSalesPredic>ons•  CFO=>ReadSalesPredic>ons

28©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  JoejustjoinedasDataEngineers•  JustaddJoetoDataEngineersgroup•  JoegetsallDataEngineersprivileges

29©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  Newteamaddedtomaintaindatalineage.•  Justaddrolestoit,noneedredefine

privileges.•  Auditors=>ReadAll

Auditors

30©Cloudera,Inc.Allrightsreserved.

31©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

• Bringsrolebasedauthoriza>oncontroltoKaAa.• Useusergroupmappingsfromexternalsystems,like,LDAP,AD,etc.•  Scalablearchitecture.• Unifiedauthoriza>oncontrolacrossvariousdatainfrastructurecomponents.

32©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

33©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

34©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

35©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

36©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

5.BrokersendsNo

tauthorizederror

codeiftherequest

isnotauthorized.

Otherwisesendsa

ppropriaterespons

e

fortherequest.

37©Cloudera,Inc.Allrightsreserved.

Demo

38©Cloudera,Inc.Allrightsreserved.

ThankyouAshishSinghasingh@cloudera.com@singhasdev