hive at linkedin

©2013 LinkedIn Corporation. All Rights Reserved. ENGINEERING©2013 LinkedIn Corporation. All Rights Reserved.

Hive at LinkedInMohammad Islam, Mark Wagner, Karthik Ramasamy

©2013 LinkedIn Corporation. All Rights Reserved. ENGINEERING 3

Agenda

LinkedIn Data and its Ecosystem Performance Improvements – Avro User experiences


LinkedIn Data Sources

Event Data– Page Views– Clicks– Search queries

Database Data– Profile (Users & Companies)– Connections

External Data– Salesforce, DoubleClick


Data Ecosystem at LinkedInMember

Facing

Systems



Facing

Systems

Data

Data

Data



Facing

Systems

DataData



Facing

Systems

Data

Data



Facing

Systems

Data


Data in Hadoop

Almost all LinkedIn data is stored in Hadoop Tools used

– Hive/HCatalog– Pig– Java MapReduce– Azkaban


Hive Usage

Use-cases– Ad-hoc query– Reporting– Building Platforms

Segmentation Engine Experimentations Engine

Users– Data Scientist– Business Analytics– Security team– Product team


Hive Challenges

Performance– Faster query execution

Performance– Faster query execution

Efficient MR* execution plan– Effective resource usage– Ensure cluster stability


LinkedIn Hive Initiatives

Make HCatalog work and deploy [OnGoing] Hive Performance Improvement (Avro data reading) [On

Going] Stabilize Hive Server 2 at LI [About to Start] Expand the scope of HCatalog metadata [Planning]


HCatalog Initiatives

Expand scope of meta-data– Who creates this data?– What are the inputs?

Helpful to create data lineage

– Who is the maintainer of data?


What is the Problem?

Reading Avro record takes long time.– 52 micro-second/record

Found the hotspot using VisualVm


Improvement #1

Reduce the number of Schema.equals() calls Schema equality checks required primarily for evolved

schema. Solution includes caching to avoid unnecessary

expensive calls Results

– Trunk read overhead : 52 μs/record– After this patch read overhead : 32 μs/record


Improvement #2

Reduce extra data transformations Solution is to provide custom object inspectors Results

– Current read overhead : 52 μs/record– After this patch read overhead : 30 μs/record


Final Results

Trunk Improvement #1 Improvement #2 Combined0

10

20

30

40

50

6055

3230

11


56%Never Used Hive

44%Use Hive

27%Primarily use Hive

Out of all our Hadoop users:

Hive User Base at LinkedIn

of Hive jobs were from ad-hoc queries32%


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Who uses Hive and who doesn’t

Data Scientists

Engineers

Product Managers

Customer Support Specialists

Analysts

Hive adoption among Hadoop users by job title


Top concerns about Hive

Not friendly for long/complex workflows

Performance, especially for ad-hoc queries

Steep learning curve for tuning

Data/UDFs unavailability

hive at linkedin

Technology

linkedin corporation

agenda linkedin data

data ecosystem

linkedin hive initiatives

data lineage

maintainer of data

scope of metadata

member data profiles