analysis big data by use php with storm

PHP|

PaaS SAE

Top3 SuperAPP

Pla$orm Services Group

Sam

AWS S3

CDN ATS

Biwing

1k

30+10+

Monika

G o

Rambone

PHP Front-End

Services Panel

Skylee

C++ Go

Content Summary

"

# PHP

PHP

$ %

Case

Real 7me data collec7on

"

How to collect data in real 7me

?

…… ……

Ka<a —— a high-throughput distributed messaging system source from LinkedIn

Producer KaLa LinkedIn Apache

KaLa Producer

&

"

'

(

ModelTwo models: queuing and publish-subscribe. the la<er is more commonly used.

high-level abstracPonEach Topic corresponds to one or more log files. Each Topic has one or more par@@ons.

FeatureScalability & Durability & Reliability & Performance & Fault-tolerant

EcosystemStream processing systems, Hadoop integraPon, monitoring, and deployment tools.

Ka<a should be no7ced when using Ka<a

)

KaLa offsetConsumer

Buffer

KaLa gzip snappy lz4CPU

KaLa KaLa

KaLa

Topic Par@@onPar@@on

Topic Par@@on

ZookeeperKaLa Zookeeper

ZK ZK

Real-7me analysis

"

BackType nathanmarz Storm BackTypeTwiXer Storm TwiXer TwiXer

Apache

' Feature

Storm —— a distributed real-7me computa7on system source from TwiGer

Highly scalable

Fault-tolerant

Guarantees processing

Language agnosPc

* Topology of one Spout

* Topology of three Spout

* UI / Nimbus / ZK / Supervisor

Storm Storm cluster organiza7on structure

Storm UI ZookeeperStorm Nimbus

Storm Supervisor

Topology

Storm Storm mul7 language support

+

Python

PHP

JS

Shell

, Storm use STDIN & STDOUT

support multi language

PHPConnectPool

Storm PHPConnect Storm to PHP

+ PHP

STDIN

STDOUT

+PHP

PHP-FPMPHP

PHP

FastCGI

PHPphp analysis program

PHP

PHPPHP parse a real 7me log

*

-> Kafka -> Storm -> Spout Bolt -> Bolt FCGI

PHPPHP program for parse log

PHP

-

PHP-FPMdynamic PHP

pm.max_children

PHPDB Cache

Storm

web

RD

PHP

Input

PHP

PHP

Storm Bolt PHPStorm bolt send a request by FastCgi protocol

Architecture design

$

Overall architecture design

Performance data

CaseShow Case

%

Sta7s7cal service pla$orm

Sta7s7cal service pla$orm

Real-7me log analyse service

Q & A?