crawler documentation - read the docs

Crawler DocumentationRelease 1.0

Arinze

Jan 16, 2019

Contents

1 Getting Started 31.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Architecture 15

3 Guides 17

4 Indices and tables 19

i

Crawler Documentation, Release 1.0

An enterprise call taking solution written in Python.

Contents:

Contents 1


2 Contents

CHAPTER 1

Getting Started

1.1 Installation

Note: CallTaker is built to work on CentOS 7 linux distribution. Make sure that development tools are installed:

# sudo yum update# sudo yum group install "Development Tools"

1.1.1 Prerequisites

Before you are ready to run CallTaker you will need additional software installed on your computer.

Python 2.7

CallTaker requires Python 2.7 or later. A compatible version comes preinstalled with most CentOS 7 Linux systems.If that is not the case consult your distribution for instructions on how to install python 2.7

PostgreSQL

CallTaker needs PostgreSQL version 9.4 or above to work. Use the PostgreSQL download page to get instructions foryour distribution OR follow the steps below:

# rpm -ivh https://yum.postgresql.org/9.6/redhat/rhel-7.3-x86_64/pgdg-centos96-9.6-3.→˓noarch.rpm# yum update# yum install postgresql96 postgresql96-server postgresql96-libs postgresql96-contrib→˓postgresql96-devel# /usr/pgsql-9.6/bin/postgresql96-setup initdb

(continues on next page)

3

https://www.postgresql.org/download/


(continued from previous page)

# systemctl enable postgresql-9.6.service# systemctl start postgresql-9.6.service

DAHDI(Digium Asterisk Hardware Device Interface)

DAHDI is a collection of open source drivers, for linux, that are used to interface with a variety of telephony relatedhardware. It consists of two parts.

• The DAHDI-Linux project contains the individual board drivers for the supported hardware.

• The DAHDI-Tools project contains an assortment of user space utilities that are used to setup and test the drivers.

# wget http://downloads.asterisk.org/pub/telephony/dahdi-linux-complete/dahdi-linux-→˓complete-2.8.0-rc4+2.8.0-rc4.tar.gz# tar -xvf dahdi-linux-complete-2.8.0-rc4+2.8.0-rc4.tar.gz# cd dahdi-linux-complete-2.8.0-rc4+2.8.0-rc4# make# make install# make config

Libpri

LibPRI is a library that adds support for ISDN (PRI and BRI). The use of LibPRI is optional, but since it takes verylittle time to install, doesn’t interfere with anything, and will come in handy if you ever want to add cards to a systemat a later point, we recommend that you install it now.

Warning: Before you can build libpri, you’ll need to Build and Install DAHDI.

# wget http://downloads.asterisk.org/pub/telephony/libpri/old/libpri-1.5.0.tar.gz# tar -xvf libpri-1.5.0.tar.gz# cd libpri-1.5.0# make# make install

Asterisk

Asterisk is an open source framework for building communications applications. To install asterisk, follow the follow-ing steps:

# wget http://downloads.asterisk.org/pub/telephony/certified-asterisk/releases/→˓asterisk-certified-11.6-cert18.tar.gz# tar -xvf asterisk-certified-11.6-cert18.tar.gz# cd asterisk-certified-11.6-cert18# cp contrib/init.d/rc.redhat.asterisk /etc/init.d/asterisk# ./configure

Note: During configuration, select format_mp3 and res_odbc. then,

4 Chapter 1. Getting Started

http://git.asterisk.org/gitweb/?p=dahdi/linux.git;/

http://git.asterisk.org/gitweb/?p=dahdi/tools.git;/

https://www.asterisk.org/get-started/


# make# make install# make samples

Kamailio

A typical use case is Kamailio as a SIP proxy router to scale Asterisk, by handling the user authentication and registra-tion, letting one or a farm of Asterisks to deal with call handling (e.g., IVR, transconding, gatewaying, prepaid billing,a.s.o.).

With such architecture, several other benefits can be achieved quickly:

• increase of security - Kamailio handling SIP signaling only, can absorb easier the flooding attacks, protectingAsterisk

• transport layer gatewaying - Kamailio has mature implementations for UDP, TCP, TLS and SCTP, therefore youcan use it in front of Asterisk to translate between these protocols

• load balancing - you can use several instances of Asterisk, Kamailio can do load balancing among them

• high availability - Kamailio can be configured to re-route the call if selected Asterisk box does not react in agiven period of time, e.g., if one Asterisk is not responsive in 2 sec, sent the call to another Asterisk

# cd /etc/yum.repos.d/# wget http://download.opensuse.org/repositories/home:/kamailio:/v4.4.x-rpms/CentOS_7/→˓home:kamailio:v4.4.x-rpms.repo# yum update# yum install -y kamailio kamailio-presence kamailio-ldap kamailio-postgres kamailio-→˓debuginfo kamailio-xmpp kamailio-unixodbc kamailio-utils kamailio-tls kamailio-→˓outbound kamailio-gzcompress# chkconfig kamailio on

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker

# yum install redis# systemctl enable redis# systemctl start redis

Psqlodbc

Psqlodbc is used for realtime connection between asterisk and postgresql database. To install psqlodbc, follow thefollowing steps:

# wget http://ftp.postgresql.org/pub/odbc/versions/src/psqlodbc-09.06.0310.tar.gz# tar -zxvf psqlodbc-xx.xx.xxxx.tar.gz# cd psqlodbc-xx.xx.xxxx# ./configure# make# make install

Note: Please take note of the location is psqlodbc because you would need it in the odbcinst.ini configuration.

1.1. Installation 5

https://redis.io/


To complete the prerequisites, install the following packages:

# yum install nginx# yum install python-devel libjpeg-devel zlib-devel# yum install postgresql-devel# yum install python-devel

1.1.2 Installation

1. Clone the repository (or use your own fork):

# git clone https://[email protected]/arinzecollins/anuri.git

2. Enter the directory:

$ cd anuri/

3. Install all dependencies:

We strongly recommend creating a virtual environment before installing any Python packages.

$ pip install -r requirements.txt

4. Set SECRET_KEY environment variable.

We try to provide usable default values for all of the settings. We’ve decided not to provide a default forSECRET_KEY as we fear someone would inevitably ship a project with the default value left in code.

$ export SECRET_KEY='<mysecretkey>'

Warning: Secret key should be a unique string only your team knows. Running code with a knownSECRET_KEY defeats many of Django’s security protections, and can lead to privilege escalation and remotecode execution vulnerabilities. Consult Django’s documentation for details.

1.2 Configuration

Now that we have done with the installation, it is time to commence configurations.

1.2.1 PostgreSQL

1. Edit pg_hba.conf:

# python manage.py migrate# cd /var/lib/pgsql/9.6/data# vi pg_hba.conf# change peer to trust# Add - host all all 0.0.0.0/0→˓md5

2. Edit postgresql.conf:


https://docs.python.org/3/tutorial/venv.html

https://docs.djangoproject.com/en/1.11/ref/settings/#secret-key


# cd /var/lib/pgsql/9.6/data# vi postgresql.conf# edit listen_addresses = '*'# systemctl restart postgresql-9.6.service

3. Create a password for default user:

See PostgreSQL’s createuser command for details.

# psql -U postgres# \password# Enter “bright1008” twice

4. Create a PostgreSQL database

See PostgreSQL’s createdb command for details.

# CREATE DATABASE creodb;# CREATE DATABASE kamailio;# psql creodb < creodb.sql;

5. Prepare the database:

$ python manage.py migrate

Warning: This command will need to be able to create database extensions. If you get an error related tothe CREATE EXTENSION command please review the notes from the user creation step.

1.2.2 Kamailio

This configuration file is an update of default Kamailio 4.0.x configuration file.

1. Create a .pgpass at the root directory:

Input localhost:*:*:username:password in the .pgpass and save

2. We then need to add some configurations in the /etc/kamailio/kamctlrc file:

• Your SIP domain

SIP_DOMAIN=IP address of the server

• Database read/write user

DBRWUSER="postgres"

• Password for database read/write user

DBRWPW="password"

• Database read only user

DBROUSER="username"

• Password for database read only user

DBROPW="password"

• Database access host (from where is kamctl used)

DBACCESSHOST=127.0.0.1

1.2. Configuration 7

https://www.postgresql.org/docs/current/static/app-createuser.html

https://www.postgresql.org/docs/current/static/app-createdb.html


3. Create kamailio tables:

# /usr/local/sbin/kamdbctl createor# kamdbctl create

4. Configuring /etc/kamailio/kamailio.cfg:

On the top of kamailio.cfg file, add

#!define WITH_MYSQL#!define WITH_AUTH#!define WITH_USRLOCDB#!define WITH_ASTERISK

#!define DBURL "postgres://username:password@localhost/kamailio"#!define DBASTURL "postgres://username:password@localhost/creodb"

Note: Remember to change the IP Address of the bind ip

asterisk.bindip = "xxxxxxxxxxx" desc "Asterisk IP Address"asterisk.bindport = "xxxx" desc "Asterisk Port"kamailio.bindip = "xxxxxxxxxxx" desc "Kamailio IP Address"kamailio.bindport = "xxxx" desc "Kamailio Port

add loadmodule "db_postgres.so"

5. Then, give permission:

1.2.3 Asterisk

1. Using database creodb create the following tables

• create table sipusers

CREATE TABLE public.sipusers(

id integer NOT NULL DEFAULT nextval('sipusers_id_seq'::regclass),name character varying(80) COLLATE pg_catalog."default" NOT NULL,ipaddr character varying(15) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,port integer NOT NULL DEFAULT 0,regseconds integer NOT NULL DEFAULT 0,defaultuser character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,fullcontact character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,regserver character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,useragent character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,lastms integer NOT NULL DEFAULT 0,host character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,type sip_type,





context character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

permit character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

deny character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

secret character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

md5secret character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

remotesecret character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

transport tsport,dtmfmode dtmf,directmedia directmedia,nat nat,callgroup character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,pickupgroup character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,language character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,disallow character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,allow character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,insecure character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,trustrpid decision,progressinband progress,promiscredir decision,useclientcode decision,accountcode character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,setvar character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,callerid character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,amaflags character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,callcounter decision,busylevel integer NOT NULL DEFAULT 0,allowoverlap decision,allowsubscribe decision,videosupport decision,maxcallbitrate integer NOT NULL DEFAULT 0,rfc2833compensate decision,mailbox character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,"session-timers" s_timers,"session-expires" integer NOT NULL DEFAULT 0,"session-minse" integer NOT NULL DEFAULT 0,"session-refresher" s_rfresher,t38pt_usertpsource character varying(255) COLLATE pg_catalog."default"

→˓DEFAULT NULL::character varying,regexten character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,(continues on next page)




fromdomain character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

fromuser character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

qualify character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

defaultip character varying(255) COLLATE pg_catalog."default" DEFAULT→˓NULL::character varying,

rtptimeout integer NOT NULL DEFAULT 0,rtpholdtimeout integer NOT NULL DEFAULT 0,sendrpid decision,outboundproxy character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,callbackextension character varying(255) COLLATE pg_catalog."default"

→˓DEFAULT NULL::character varying,timert1 integer NOT NULL DEFAULT 0,timerb integer NOT NULL DEFAULT 0,qualifyfreq integer NOT NULL DEFAULT 0,constantssrc decision,contactpermit character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,contactdeny character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,usereqphone decision,textsupport decision,faxdetect decision,buggymwi decision,auth character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,fullname character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,trunkname character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,cid_number character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,callingpres c_press,mohinterpret character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,mohsuggest character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,parkinglot character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,hasvoicemail decision,subscribemwi decision,vmexten character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,autoframing decision,rtpkeepalive integer NOT NULL DEFAULT 0,"call-limit" integer NOT NULL DEFAULT 0,g726nonstandard decision,ignoresdpversion decision,allowtransfer decision,dynamic decision,sippasswd character varying(255) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying)



CREATE TABLE public.sipregs(

id integer NOT NULL DEFAULT nextval('sipregs_id_seq'::regclass),name character varying(80) COLLATE pg_catalog."default" NOT NULL,fullcontact character varying(80) COLLATE pg_catalog."default" NOT NULL,ipaddr character varying(15) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,port integer NOT NULL DEFAULT 0,username character varying(80) COLLATE pg_catalog."default" NOT NULL,regserver character varying(15) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,regseconds integer NOT NULL DEFAULT 0,defaultuser character varying(80) COLLATE pg_catalog."default" NOT NULL,useragent character varying(15) COLLATE pg_catalog."default" DEFAULT

→˓NULL::character varying,lastms integer NOT NULL DEFAULT 0,CONSTRAINT sipregs_name_key UNIQUE (name)

)

2. UnixODBC Configuration Edit /etc/odbcinst.ini and add:

[PostgreSQL]Description = ODBC for PostgreSQLDriver = /usr/local/lib/psqlodbca.soSetup = /usr/lib64/libodbcpsqlS.soFileUsage = 1

Edit /etc/odbc.ini and add:

[asterisk-connector]Description = PostgreSQL connection to 'asterisk' databaseDriver = PostgreSQLDatabase = creodbServername = localhostUserName = postgresPassword = xxxxxxxxPort = 5432Protocol = 8.1ReadOnly = NoRowVersioning = NoShowSystemTables = NoShowOidColumn = NoFakeOidIndex = NoConnSettings =

3. Asterisk UnixODBC Configuration Edit /etc/asterisk/res_odbc.conf and set:

[creodb]enabled => yesdsn => asterisk-connectorusername => postgrespassword => xxxxxxxxpre-connect => yes

Edit /etc/asterisk/extconfig.conf and set:



[settings]sipusers => odbc,creodb,sipuserssippeers => odbc,creodb,sipuserssipregs => odbc,creodb,sipregsvoicemail => odbc,creodb,voicemailqueue_log => odbc,creodb,report_queue

4. Asterisk Configuration In case you need to cache the realtime users, then edit /etc/asterisk/sip.conf and set:

rtcachefriends=yes

Note: Be sure you update the listen IP and port as well if Asterisk is running on the same system withKamailio.

1.2.4 NGINX

Edit /etc/nginx/nginx.conf and set:

server {listen 8000 default_server;listen [::]:8000 default_server;server_name xxx.xxx.xxx.xxx;location / {

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header Host $http_host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-Proto $scheme;proxy_pass http://0.0.0.0:8001;

}location /stream {

proxy_pass http://0.0.0.0:9001;proxy_http_version 1.1;proxy_read_timeout 86400;proxy_redirect off;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection "upgrade";proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Host $server_name;

}location /static/ {

root /home/;}error_page 404 /404.html;

location = /40x.html {}

error_page 500 502 503 504 /50x.html;location = /50x.html {

}}



1.2.5 Gunicorn

Create /etc/systemd/system/gunicorn.service and set:

[Unit]Description=gunicornAfter=network.target[Service]PIDFile=/run/gunicorn/pidUser=xxxGroup=nginxWorkingDirectory=/home/anuriExecStart=/usr/bin/gunicorn anuri.wsgi --bind 0.0.0.0:8001 --log-level error→˓--log-file=- --workers 5 --preloadExecReload=/bin/kill -s HUP $MAINPIDExecStop=/bin/kill -s TERM $MAINPIDRestart=on-abortPrivateTmp=true[Install]WantedBy=multi-user.target

Note: xxx is the system user.

1.2.6 Daphne

Create /etc/systemd/system/daphne.service and set:

[Unit]Description=daphne daemonAfter=network.target[Service]PIDFile=/run/daphne/pidUser=xxxGroup=nginxWorkingDirectory=/home/anuriExecStart=/usr/bin/daphne -b 0.0.0.0 -p 9000 anuri.asgi:channel_layerExecReload=/bin/kill -s HUP $MAINPIDExecStop=/bin/kill -s TERM $MAINPIDRestart=on-abortPrivateTmp=true[Install]WantedBy=multi-user.target

1.2.7 Worker

Create /etc/systemd/system/worker.service and set:

[Unit]Description=worker daemonAfter=network.target[Service]User=xxxGroup=creo





WorkingDirectory=/home/anuriExecStart=/usr/bin/python /home/creo/anuri/manage.py runworker --threads 4ExecReload=/bin/kill -s HUP $MAINPIDExecStop=/bin/kill -s TERM $MAINPIDRestart=alwaysPrivateTmp=true[Install]WantedBy=multi-user.target

Note: Remember to put notifier in /etc/init.d/ and run chmod +x notifier


CHAPTER 2

Architecture

15


16 Chapter 2. Architecture

CHAPTER 3

Guides

17


18 Chapter 3. Guides

CHAPTER 4

Indices and tables

• genindex

• modindex

• search

19

crawler documentation - read the docs

Documents