opensips and bigdataopensips.org/pub/...summit...opensips_and_bigdata.pdf · combination with...
TRANSCRIPT
![Page 1: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/1.jpg)
OpenSIPS and BigDataHow to integrate OpenSIPS with ElasticSearch
![Page 2: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/2.jpg)
Presenter
• Flavio E. Goncalves
• Owner of the VOffice Group in Brazil
• CTO of SipPulse Tecnologia Ltda.
• OpenSIPS New Book and OpenSIPS New Bootcamp
![Page 3: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/3.jpg)
SipPulse
• SipPulse is a Brazilian company dedicated to VoIP applications
• SipPulse Anti Fraud System - TFPS
• SipPulse Routing and Billing
• Session Border Controllers
• SIP-I/SIP-T Translators
• Media Gateway Controllers
• More than 50 small to medium Telcos running SipPulseand OpenSIPS
![Page 4: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/4.jpg)
The problems we were facing
• Logs from different sources
•Commands in the wrong console syndrome
• Time spent to get the information required to troubleshoot
• Logs stored only for a few days
Master Tool
![Page 5: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/5.jpg)
Our industry generates a lot of data
•Billions of CDRs
•Terabytes of Traces
•Gigabytes of Logs
•Not easy to capture, transmit, store and search
BIG
DATA
![Page 6: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/6.jpg)
How can we make this data valuable?
1. Reduce the time to troubleshoot problems centralizing logs
• 70% of the troubleshooting time is to collect data
2. Enhance customer service quickly solving billing issues
• Churn is a major problem in UCaaS and ITSPs
3. Decreasing the calls to invalid or disconnected numbers
• In some mailings more than 5% of the numbers are invalid
4. Search numbers and IPs used for fraud and block real time
• 20% of the numbers used in Toll Fraud are reused
5. Discover patterns with analytics and better serve your customers
![Page 7: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/7.jpg)
ELK
• Elastic Search – Search and Analyze Data
• Open Source Search Engine based on Apache Lucene
• Logstash – Process any data from any source
• Open Source Log Contextualizer
•Kibana – Explore and Visualize
• Open Source Analytics
![Page 8: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/8.jpg)
Case Study
•Using ELK for Anti-Fraud Information
•Problems
•Number formatting • 9011, 011, +, 901511…..
•Quick access to online information• Search Numbers and IPs in real time
•Provide easy access to information• Concerns regarding the delivery of information using
MySQL over the Internet
•Separate Databases for Online and Offline information
![Page 9: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/9.jpg)
Recipe for a HoneyPot
•OpenSIPS 2.1
•Apache
•Distributed DataCenter
• Frequent IP migrationsElasticSearch
Kibana
HoneyPotTokio HoneyPot
Virginia
HoneyPotSãoPauloHoneyPot
Frankfurt
OpenSIPS+LogStash
OpenSIPS+LogStash
OpenSIPS+LogStash
OpenSIPS+LogStash
![Page 10: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/10.jpg)
Data Flow DiagramOpenSIPS
SYSLOG(rsyslog,syslog-ng)
LogStash
ElasticSearch
Kibana
Generates Data
Logs Data to A File
Contextualize Data and Send in JSON to ElasticSearch
Index Data
Analyzes Data
![Page 11: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/11.jpg)
Logstash loves data!
https://www.elastic.co/guide/en/logstash/current/introduction.html
![Page 12: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/12.jpg)
200 Available Plugins, No Plugins for OpenSIPS
Parse arbitrary text and structure it.
Is your friend!GROK
![Page 13: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/13.jpg)
How GROK Works
filter {grok {match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{IP:honeypot_ip},%{IP:intruder_ip},%{WORD:sip_method},sip:%{WORD:ani}@%{HOSTNAME:ani_domain},sip:%{GREEDYDATA:dnis}@%{GREEDYDATA:dnis_domain},%{GREEDYDATA:user_agent},\[%{NUMBER:longitude},%{NUMBER:latitude}\]"
}}
}
http://grokconstructor.appspot.com/do/match
![Page 14: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/14.jpg)
ElasticSearch
{"name" : "Jonathan Richards","cluster_name" : "elasticsearch","version" : {
"number" : "2.3.1","build_hash" : "bd980929010aef404e7cb0843e61d0665269fc39","build_timestamp" : "2016-04-04T12:25:05Z","build_snapshot" : false,"lucene_version" : "5.5.0"
},"tagline" : "You Know, for Search"
}
http://w.x.y.z:5500
![Page 15: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/15.jpg)
Searching
http://w.x.y.z:5500/_search?q=972598294121
http://w.x.y.z:5500/_search?q=friendly-scanner
http://w.x.y.z:5500/_search?q=173.208.203.122
![Page 16: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/16.jpg)
Kibana - Analytics
![Page 17: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/17.jpg)
OpenSIPS Integration
• Logstash is based on Java and a bit slow
•We can bypass Logstash sending data straight from OpenSIPS
•We want also to consume data directly from Elastic Search
![Page 18: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/18.jpg)
New Data Flow
OpenSIPS
ElasticSearch
Kibana
Generates Data
Contextualize Data and Send in JSON to ElasticSearch
Index Data
Analyzes Data
![Page 19: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/19.jpg)
OpenSIPS Integration
if (is_method("INVITE")) {
#####Create crud json
$json(body) := "{}";
$json(body/time) = $time(%F %T-0300);
$json(body/sipRequest) = “INVITE”;
$json(body/ipIntruder) = $si;
$json(body/destNum) = $rU;
$json(body/userAgent) = $ua;
$json(body/country)=$var(city);
$json(body/location)=$var(latlon);
$json(body/ipHost) = $Ri;
async(rest_post("http://user:[email protected]:9200/opensips/1", "$json(body)", "$var(ctype)",
"$var(ct)", "$var(rcode)"),resume)
![Page 20: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/20.jpg)
Now OpenSIPS can go straight to the data!
if (rest_get("http://user:[email protected]:5500/_count?q=destNum:$rU&pretty",
"$var(body)", "$var(ctype)", "$var(rcode)")) {
$json(body) := $var(body);
if ($json(body/count) != 0) {
xlog(“Exists\n");
exit;
} else {
xlog(“Don’t Exist\n");
# ...
}
}
![Page 21: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/21.jpg)
What else?
•Quick Billing Log (RabbitMQ Plugin)
• INVITE REQUEST
• INVITE REPLY
• BYE REQUEST
• BYE REPLY
• DATA:• request_time,
• reply_time,
• caller_id,
• callee_id,
• call_id,
• microseconds,
• reply_code
•Purpose:
• Resolve billing discrepancies without sending a ton of data over the Internet
![Page 22: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/22.jpg)
Data FlowOpenSIPS
RabbitMQ
LogStash
ElasticSearch
Kibana
Generates Events
Queue Events
Reads the QueueContextualize Data and Send in JSON to ElasticSearch
Index Data
Analyzes Data
![Page 23: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/23.jpg)
Advantages of Elastic Search
• Free and Open Source
•Quick, easy and powerful search capabilities
•Unstructured and correlated data:• logs,
• cdrs
• and eventually traces (Homer can export)
•Control over the size of the data sent
• Less costly to store in AWS.
• Easy Analytics
![Page 24: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/24.jpg)
OpenSIPS and ElasticSearch
• Integration via Syslog
• Integration via REST_CLIENT
•Async Calls have low effect on SIP server performance
•Several use cases:
• Centralizing logs
• Anti-Fraud
• Do not call blacklists
![Page 25: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/25.jpg)
Scalability
•Vertical Scalability• More powerful hardware is not always the solution
•Horizontal Scalability• Cluster Ready
•Data Center Services• AWS ElasticSearch
•HipChat• 1.2 Billion messages
• 8 ElasticSearch Servers
• 60 messages per second
![Page 26: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/26.jpg)
Further Investigation
•SYSLOG-NG can be a good replacement for logstash
• Developed in C is probably much faster than logstash
• It is capable to send data straight to ElasticSearch
@module mod-java@include "scl.conf"
destination d_elastic {elasticsearch(
index("syslog-ng_${YEAR}.${MONTH}.${DAY}")type("test")
);};
![Page 27: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/27.jpg)
Conclusion
• ElasticSearch seems to be a viable platform for big data and to handle Logs and CDRs.
• ElasticSearch can be integrated with OpenSIPS using the REST_CLIENT directly, RABBITMQ and SYSLOG in combination with Logstash.
• This is a preliminary research, so we are not aware yet of scalability problems of the model. Horizontal scalability helps, but the cost/benefit has to be measured compared to SQL and NoSQL approaches
![Page 28: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/28.jpg)
![Page 29: OpenSIPS and BigDataopensips.org/pub/...Summit...OpenSIPS_and_BigData.pdf · combination with Logstash. •This is a preliminary research, so we are not aware yet of scalability problems](https://reader034.vdocuments.us/reader034/viewer/2022052202/5a9fa0447f8b9a6c178cf9d9/html5/thumbnails/29.jpg)
Contact Information
• E-mail: [email protected]
• Linkedin: https://br.linkedin.com/in/flavioegoncalves
• Twitter: #flagonc