Transcript
Page 1: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

1©Cloudera,Inc.Allrightsreserved.

AshishSingh|So:wareEngineer,Cloudera

Authoriza>oninApacheKaAa

Page 2: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

2©Cloudera,Inc.Allrightsreserved.

•  So:wareEngineer@Cloudera• ContributedtoKaAa,Sentry,HiveandParquet• UsedtoworkinHPC• @singhasdev

AboutMe

Page 3: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

3©Cloudera,Inc.Allrightsreserved.

Page 4: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

4©Cloudera,Inc.Allrightsreserved.

AboutKaAa

• Publish/SubscribeMessagingSystem• Highthroughput(100’sofkmessages/sec)•  Lowlatency(sub-secondtolowseconds)•  Fault-tolerant(ReplicatedandDistributed)•  Supportsagnos>cmessaging•  Standardizesformatanddelivery• Hugecommunity

Page 5: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

5©Cloudera,Inc.Allrightsreserved.

ArchitectureProducer

Consumer Consumer

Producers

KaAaCluster

Consumers

Broker Broker Broker Broker

Producer

Zookeeper

Page 6: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

6©Cloudera,Inc.Allrightsreserved.

Authoriza>on

Page 7: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

7©Cloudera,Inc.Allrightsreserved.

Authoriza>on

Authoriza>onisthefunc>onofspecifyingaccessrightstoresourcesrelatedtoinforma>onsecurityandcomputersecurityingeneralandtoaccesscontrolinpar>cular.Moreformally,"toauthorize"istodefineanaccesspolicy.–Wikipedia

Page 8: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

8©Cloudera,Inc.Allrightsreserved.

Authoriza>on

Authoriza>onisthefunc>onofspecifyingaccessrightstoresourcesrelatedtoinforma>onsecurityandcomputersecurityingeneralandtoaccesscontrolinpar>cular.Moreformally,"toauthorize"istodefineanaccesspolicy.–Wikipedia

Page 9: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

9©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

Page 10: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

10©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

Page 11: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

11©Cloudera,Inc.Allrightsreserved.

Authoriza>oninApacheKaAa

Page 12: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

12©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanperformWHATac>ononaRESOURCE?

Page 13: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

13©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?

Page 14: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

14©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• KaAausesabinaryprotocoloverTCP.• TheprotocoldefinesallAPIsasrequestresponsemessagepairs.• RequestsaresentthroughaRequestchannela:erasessionisestablished.•  SessioncontainsPrincipalandHost.• Principalisoftheform<PrincipalType>:<PrincipalName>,.e.g.,User:foo,Group:analyst,etc.• Hosthasinfoonwheretherequesthasoriginatedfrom.•  ProvidesIP-Filtering.

Page 15: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

15©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• PermissionTypes.• Allow• Deny

Page 16: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

16©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• Opera>onssupportedinKaAa.• Read• Write• Create• Delete• Alter• Describe• ClusterAc>on

Page 17: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

17©Cloudera,Inc.Allrightsreserved.

AccessPolicy

WHOcanorcannotperformWHATac>ononaRESOURCEfromWHERE?• ResourceTypesinKaAa.• Cluster• Topic• ConsumerGroup

Page 18: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

18©Cloudera,Inc.Allrightsreserved.

AccessPolicy

• AccessPolicyinKaAaisrepresentedasACL,AccessControlList.• ACLinKaAaiscomposedofthefollowing.• SetofPrincipals.• Permissiontype,i.e.,AlloworDeny.• SetofHosts.• SetofOpera>ons.

Page 19: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

19©Cloudera,Inc.Allrightsreserved.

KaAa’sDefaultAuthorizerSimpleAclAuthorizer

Page 20: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

20©Cloudera,Inc.Allrightsreserved.

SimpleAclAuthorizer

• Outoftheboximplementa>onoftheKaAaAuthorizer.•  Selfcontainedandnodependencieswithanyothervendororproviders.•  Ituseszookeeperasthestoragelayerforacls.ACLsarestoredinJSONformatdescribedunder/kaAa-acls/resource-type/<resource-name>.• UsesCachingtoavoidgoingtoZKforeachrequest.• DenytakesprecedenceoverAllowincompe>ngACLs.• WhennoACLisaoachedtoaresource,useconfigallow.everyone.if.no.acl.found.• WhenanyACLisaoachedtoaresourceonlyusersthatareintheallowedlisthaveaccess.AlluserswithnoexplicitallowACLsaredeniedaccessbydefault.• READorWRITEpermission=>DESCRIBEOpera>on.

Page 21: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

21©Cloudera,Inc.Allrightsreserved.

SimpleAclAuthorizer–NotsoSimple

• OnlysupportsUserprincipal• PRoutforGroupPrincipalforsome>me.

• Nowaytouseusergroupmappingfromexternalservices,like,LDAP,AD,etc.• VeryKaAaspecificimplementa>on.• Notscalable.• HaszNodesizelimita>ons,defaultandrecommendedinonly1MB.• Concurrencyissues.

• Notproduc>onready.

Page 22: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

22©Cloudera,Inc.Allrightsreserved.

Page 23: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

23©Cloudera,Inc.Allrightsreserved.

Sentry

• Providesunifiedrolebasedauthoriza>onforvariouscomponents.• Hive•  Impala• HDFS• Sqoop• KaAa(ohyea!)

Page 24: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

24©Cloudera,Inc.Allrightsreserved.

RBAC

• RoleBasedAuthoriza>onControl,RBAC,isapowerfulmechanismtomanageauthoriza>onforalargesetofusersanddataobjectsinatypicalenterprise.• Assigningprivilegestoauser.• Privilege->User• Privilege->Group->User• Privilege->Role->Group->User

• UsergroupmappingisconfigurableandcancomefromShellorexternalsystems,like,LDAP,AD,etc.

Page 25: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

25©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

Page 26: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

26©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  DataEngineerscanREADfromCustomerData.•  DataEngineerscanWRITEtoEnrichedData.•  AnalystscanREADfromEnrichedData.•  AnalystscanWritetoSalesPredic>on.•  CFOcanREADfromSalesPredic>on.

Page 27: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

27©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  Roles:•  ReadCustomerData=>READfromCustomerData•  WriteEnrichedData=>WRITEtoEnrichedData•  ReadEnrichedData=>READfromEnrichedData•  WriteSalesPredic>ons=>WRITEtoSalesPredic>ons•  ReadSalesPredic>ons=>READfromSalesPredic>ons•  AllRead=>READfromCustomer,EnrichedandSalesPredic>ons

•  RolestoGroups:•  DataEngineers=>ReadCustomerData,ReadEnrichedData,

WriteEnrichedData•  Analysts=>ReadEnrichedData,ReadSalesPredic>ons,

WriteSalesPredic>ons•  CFO=>ReadSalesPredic>ons

Page 28: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

28©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  JoejustjoinedasDataEngineers•  JustaddJoetoDataEngineersgroup•  JoegetsallDataEngineersprivileges

Page 29: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

29©Cloudera,Inc.Allrightsreserved.

RBAC

CFO

CustomerDataDataEngineers

Analysts EnrichedData

SalesPredic>ons

•  Newteamaddedtomaintaindatalineage.•  Justaddrolestoit,noneedredefine

privileges.•  Auditors=>ReadAll

Auditors

Page 30: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

30©Cloudera,Inc.Allrightsreserved.

Page 31: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

31©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

• Bringsrolebasedauthoriza>oncontroltoKaAa.• Useusergroupmappingsfromexternalsystems,like,LDAP,AD,etc.•  Scalablearchitecture.• Unifiedauthoriza>oncontrolacrossvariousdatainfrastructurecomponents.

Page 32: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

32©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

Page 33: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

33©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

Page 34: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

34©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

Page 35: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

35©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

Page 36: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

36©Cloudera,Inc.Allrightsreserved.

SentryKaAaAuthorizer

Broker

Broker

Broker

Producer

Consumer

Sentry

1.  Clientauthen>c

ateswithBroker.

2.  Clientsendsreq

uesttoaBroker.

5.BrokersendsNo

tauthorizederror

codeiftherequest

isnotauthorized.

Otherwisesendsa

ppropriaterespons

e

fortherequest.

Page 37: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

37©Cloudera,Inc.Allrightsreserved.

Demo

Page 38: Authorization in Apache Kafka - Seattle Kafka Meetup - Ashish Singh

38©Cloudera,Inc.Allrightsreserved.

[email protected]@singhasdev


Top Related