#abbyysummit15 - training (1/6): flexicapture best practices of system configuration: scaling,...

TechnologySummit 2015

© Copyright 2015 ABBYY Technology Summit

BEST PRACTICES OF SYSTEM CONFIGURATION. SCALING, REDUNDANCY

Andrew Zyuzin, PM of DataCapture Dev Team

#ABBYYSummit15


How to setup a production system for desired performance?

● Performance

#ABBYYSummit15 2

● Scalability & Redundancy

● Bottlenecks

TechnologySummit 2015Choose a hardware

#ABBYYSummit15 3

BWpages per

day

GRAYpages per day

COLORpages per day

Verifiers

5K 1K 1K 3

1M 300К 200К 100

600K 300

Unlimited Unlimited Unlimited Unlimited

10Gb/s

ABBYY FlexiCapture 12


Environment

ABBYY FlexiCaptureServers

ABBYY FlexiCaptureStations

TechnologySummit 2015Calculate amount of cores

#ABBYYSummit15 5

10Gb/s0

100

200

300

400

500

600

700

800

900

1000

8 16 24 32 40 48 56 64 72 80

Tho

usn

ad o

f p

age

s p

er

24

ho

urs

Processing CPU cores

bw

TechnologySummit 2015ABBYY QA Farm Performance

#ABBYYSummit15 6

10Gb/s0

100

200

300

400

500

600

700

800

900

1000

8 16 24 32 40 48 56 64 72 80

Tho

usn

ad o

f p

age

s p

er

24

ho

urs


bw

gray

color

TechnologySummit 2015Approach

7#ABBYYSummit15

Test

Analyze

Design

Deliver

TechnologySummit 2015Explore the Business Process

How many documents? Seasonality? How many pages in documents?

#ABBYYSummit15 8

mount

ime

ource

ntegrity

erification

What is SLA about time to process,

time to store?

What is SLA about quality?

What are the sources of images?

What is the smallest

integral part to process?

TechnologySummit 2015Parameters of the System

#ABBYYSummit15 9

Metric UoM

Average, PeakPerformance

pages per 24 hours,pages per hour

Input flow MB/s

File Storage size GB

Amount of verifiers guys

Metric UoM

Document size pages

Batch size pages

Optimal

up to 100

10 - 1000

Limitations

1M pages / day,50K pages / hour

1 Gb/s [10 Gb/s]

Cost of Hardware

Up to 100 (300)

TechnologySummit 2015Architecture

#ABBYYSummit15 10

• Application Server• Licensing Server

• Processing Servers• Processing stations

• Database Server• FileStorage

• Clients• Network

TechnologySummit 2015Interaction of components

#ABBYYSummit15 11

TechnologySummit 2015Automatic Processing

#ABBYYSummit15 12


#ABBYYSummit15 13


#ABBYYSummit15 14

TechnologySummit 2015Manual Processing

#ABBYYSummit15 15

TechnologySummit 2015Export Results

#ABBYYSummit15 16


Network utilization

Application Server (Web Servie)

Image EnhancementInput Flow * upload times, Input Flow * download timesMAX ( )

vs. Network Capacity


File Storage


Input Flow * upload times

+Input Flow * download times

vs.

IOPS * bytes per operation

RedundancyRAID 10 from SATA2,

SAN, NAS

Volume

Input Flow * Time to Store


Application Server


Clients

Clients

Processing Stations

Critical resources:Network (2+ NIC)Connection to FileStorageProcessorRAM


Application Server


Clients

Clients

Processing Stations

Critical resources:Network (2+ NIC)Connection to FileStorageProcessorRAM

Network Load Balancing Cluster


Database ServerCritical resources:

RAMFast Physical DiskProcessorNICs

Fault Tolerance & High Availability

RecommendationsDisable MirroringUse Simple Recovery ModelStore database log at separate physical driveRebuild indexes


Processing, Licensing Servers

Microsoft Failover Cluster

Active Passive

Critical resources:RAM: 4GBProcessor: 2 coresNICs

Highly Availability


Processing Station

#ABBYYSummit15 24

10Gb/s

Critical resources:Processor: - frequency,- up to 12 coresFast Hard DriveRAM: 2 GB per core


Operators

#ABBYYSummit15 25

10Gb/s

● Scanning

● Verifiers

Tho

usn

ad o

f p

age

s p

er

24

h

ou

rsProcessing CPU cores

100 cores


Production - Starter


Production - Redundant

#ABBYYSummit15 27

10Gb/s


ABBYY FlexiCapture 12

#ABBYYSummit15 28

10Gb/s

Load Balancer


Performance

#ABBYYSummit15 29

10Gb/s0

100

200

300

400

500

600

700

800

900

1000

8 16 24 32 40 48 56 64 72 80

Tho

usn

ad o

f p

age

s p

er

24

ho

urs


bw

color

color*


Help & Support

#ABBYYSummit15 30

10Gb/sAlexander Timoshenko,Head of Large Projects Support Dept.

Guide about High Performance Configurations


LOOKING FOR BOTTLENECKSBest practices of system configuration

#ABBYYSummit15

TechnologySummit 2015Performance Monitor

#ABBYYSummit15 32

10Gb/s

● Web Performance Monitor (free)

● System Center Operations Manager

TechnologySummit 2015General System Parameters

#ABBYYSummit15 33

10Gb/s

● Memory● Memory\Available Bytes● Memory\Pages/sec

● Process(<All instances>)\Working Set

● Processor● Processor(_Total)\% Processor● Time System\Processor Queue Length

● Process(<All instances>)\% Processor Time

TechnologySummit 2015General System Parameters

#ABBYYSummit15 34

10Gb/s

● Disk● LogicalDisk(<All instances>)\Free Megabytes

● PhysicalDisk(<All instances>)\% Disk Time ● PhysicalDisk(<All instances>)\Avg. Disk Queue Length

● Network● Network Interface(<All instances>)\Output Queue Length ● Network Interface(<All instances>)\Packets Outbound Discarded● Network Interface(<All instances>)\Bytes Total/sec ● Network Interface(<All instances>)\Current Bandwidth

TechnologySummit 2015Application Server

#ABBYYSummit15 35

10Gb/s

● Latency of App Server● FlexiCapture(All instances)\ASCT

Latency --- 30 sec = Critical Mode

● Activity● W3SVC_W3WP(_Total)\Active

Requests

● W3SVC_W3WP(_Total)\Requests/

● Web Service(Default Web Site)\Current Connections

● Network● Web Service(Default Web Site)\Bytes

Received/sec

● Web Service(Default Web Site)\Bytes Sent/sec

● Threads● W3SVC_W3WP(_Total)\Active Threads

Count

● W3SVC_W3WP(_Total)\Maximum Threads Count

TechnologySummit 2015Processing Server

#ABBYYSummit15 36

10Gb/s

● Latency of Processing Server● FlexiCapture(Processing Server)\Primary Thread Latency

10 sec is critical

● Cores● FlexiCapure(Processing Server)\TotalCores

● FlexiCapture(Processing Server)\Free Cores


QUESTIONS

#ABBYYSummit15 37

#abbyysummit15 - training (1/6): flexicapture best practices of system configuration: scaling,...

Software