engageone deliver performance benchmarks engageone deliver 2

EngageOne Deliver Performance White Paper of 40

Performance Benchmark White Paper

EngageOne Deliver

Performance Benchmarks

EngageOne Deliver 2.5

June 2018

©2018 Pitney Bowe Software, Inc.



Contents Purpose & Scope ..................................................................................................................................................... 4

Architecture ............................................................................................................................................................ 4

Test Environment and Methodology ...................................................................................................................... 5

Hardware Specifications ..................................................................................................................................... 6

Testing Scenarios ................................................................................................................................................ 7

Email Performance Testing ..................................................................................................................................... 8

Email Test Scenarios (Batch processing) ............................................................................................................. 9

Email Performance for Different Batch Sizes .................................................................................................... 10

Email Performance for Different Attachment Sizes .......................................................................................... 11

Email Performance for Optional Features Turned ON ...................................................................................... 12

SMS Performance Testing (Batch Processing) ...................................................................................................... 13

SMS Test Scenarios ........................................................................................................................................... 13

SMS Performance for Different Batch Sizes ...................................................................................................... 14

SMS Performance with Archiving ON and OFF ................................................................................................. 15

REST API Testing ................................................................................................................................................... 16

Email Test Scenarios (REST API) ........................................................................................................................ 16

SMS Test Scenarios (REST API) .......................................................................................................................... 18

Inbound SMS Testing ............................................................................................................................................ 20

Inbound SMS Performance ............................................................................................................................... 21

Bounce Email Testing ............................................................................................................................................ 22

Bounce Email Performance ............................................................................................................................... 23

Inbound New Email Testing .................................................................................................................................. 24

Inbound New Email Performance ..................................................................................................................... 25

Mobile Push Notification Testing ......................................................................................................................... 26

Mobile Push Notification Performance ............................................................................................................ 27

System Load Results ............................................................................................................................................. 28

Scenario 1: Email, Async mode, 5 files, 5 threads ............................................................................................. 29



Scenario 2: Email, Async mode, 5 files, 5 threads, 120 KB attachment ............................................................ 31

Scenario 3: Email, Async mode, 5 files, 5 threads, 120 KB attachment, Digital Signature ............................... 32

Scenario 4: Email, Async mode, 5 files, 5 threads, 1 MB attachment .............................................................. 33

Scenario 5: SMS, 10 files, 5 threads .................................................................................................................. 34

Scenario 6: SMS, 10 files, 5 threads, Archival ON ............................................................................................. 36

EngageOne Deliver Performance Recommendations .......................................................................................... 37

Configuration considerations for optimal performance ...................................................................................... 39

Running email outbound processing in Async mode ........................................................................................ 39

Settings for parallel profile threads and gateway connections ........................................................................ 40



Purpose & Scope The purpose of this document is to provide performance results achieved after testing the EngageOne Deliver application in different user scenarios and test data. These results can be used by the customers to estimate their hardware and software requirements based on their organizational data to be processed.

Architecture The following diagram shows the EngageOne Deliver physical architecture.



Test Environment and Methodology The following sections of this document describe performance benchmarks for EngageOne Deliver running under Windows with a Tomcat server. Test results are available for both Batch (DIJ) processing as well as REST APIs. This performance benchmark includes testing for Email & SMS using outbound and inbound scenarios.

The testing environment consisted of three dedicated servers configured as follows:

• EngageOne Deliver application server (Tomcat) • Database Server (SQL Server) • Mail server (hMailServer) and SMS gateway (SMPP simulator).

The list below summarizes key information related to the environment and its configuration:-

• Latency between servers was around 1 to 3 ms.

• Before starting the test scenarios, the main table for outbound and inbound had around 5 million

and 1.5 million records respectively.

• The default setting was used for async.properties.

• In the outboundProcessor.properties file, the outbound.email.messageCountPerConnection=19

was set to 100

• The Vendor folder was kept local to Tomcat server machine.

• For REST API’s, JMeter /SOAPUI was deployed on the email server machine.

• Performance testing was carried out on a single node deployment of Deliver. However, Deliver can

be horizontally scaled using a multi-node clustering deployment.

• Email DIJ based processing can be run in sync/async mode. In async mode, email composition and

sending process becomes decoupled. As a result, a higher sending (email gateway and network

bandwidth) capacity can be utilized, if available. This may provide higher throughput although

fewer file processing threads in the outbound process were used. REST, by default runs in async

mode. SMS does not have an async mode as composition time in SMS is almost negligible.

• DIJ based performance tests were run with 1 processing thread and 5 processing threads

respectively, hence 5 input files were used for each test as a standard for most scenarios.



Hardware Specifications This section describes the hardware specifications of the machine(s) used for performance testing.

EngageOne Deliver Server

Operating System Windows Server 2012 R2 CPU Intel(R) Xeon(R) CPU E7-4820 v3 @ 1.90 GHz 1.90 GHz (4 Processor) Memory 8 GB Disk Drives 80 GB JVM Memory (Tomcat) 2 GB Apache Tomcat version Apache Tomcat 8.5.16 Java version Java 8

Database Server

Operating System Windows Server 2016 CPU Intel(R) Xeon(R) CPU X742 @ 2.67 GHz, 2.67 GHz (2 Processor) Memory 8 GB Disk Drives 80 GB SQL Server version SQL Server 2016

Mail / SMS Server & JMeter (for REST)

Operating System Windows Server 2012 R2 CPU Intel(R) Xeon(R) CPU X742 @ 2.67 GHz, 2.67 GHz (2 Processor) Memory 12 GB Disk Drives 120 GB Mail server version hMailServer 5.6.6



Testing Scenarios The following test scenarios were performed and tracked:

• Baseline functionality to provide baseline performance numbers • Benchmark tests to determine the impact on performance using different options • Customer(s) specific scenarios derived from the field • Load tests to check the threshold limit



Email Performance Testing This section presents performance testing results for email processing. DIJ based Batch Email processing can be run in both the legacy sync(multithreading) and the newly introduced async modes. Async mode allows for the sending process to be run independently of the email composition process; this provides flexibility in configuring sending threads & gateway connections. Async mode is the default and preferred option as it provides improved performance in most scenarios where higher network bandwidth/speed and email server capacity is available.



Email Test Scenarios (Batch processing) The following scenarios were executed to test the performance of email processing. For Async mode 10 sending threads were set up which effectively means 10 simultaneous connections to the email gateway.

Table 1: Email Test Results for Sync and Async mode

S. No. Number of Input

Files

Number of

Records / Input

File

Attachment Size/Type1

Feature Enabled

Messages Delivered/Sec with 1 thread

Messages Delivered/Sec with 5 threads

Sync Async Sync Async

1 5 2000 No Attachment NA 23 41 39 51

2 5 2000 120KB / (Personalized) NA 14 39 36 44

3 5 2000 120KB / (Personalized)

Digital Signature 9 31 31 34

4 5 2000 120KB / (Personalized) Archive 13 35 33 36

5 5 2000 1 MB / (Personalized) NA 5 17 18 36

6 1 2000 120KB / (Personalized) NA - - 20 37

7 3 2000 120KB / (Personalized) NA - - 28 42

8 5 50000 No Attachment NA 20 36 30 40

9 5 20000 120KB / (Personalized) NA 12 29 28 36

10 5 10000 1 MB / (Personalized) NA 4 14 15 27

1 Note that a personalized attachment type indicates that the attachment is personalized for the recipient of the Email



Email Performance for Different Batch Sizes This section presents performance results for a number of test cases that represent actual customer usage. The following graph shows performance numbers for test cases where there are a varying number of input DIJ files.

Graph 1: Email Performance for Different Batch Sizes

0

10

20

30

40

50

60

1 File of 2K records withAttachment

3 Files of 2K records eachwith Attachment

5 Files of 2K records eachwith Attachment

5 Files of 2K records eachwithout Attachment

Mes

sage

s per

sec

Email Performance (5 threads)

Sync Async



Email Performance for Different Attachment Sizes The following graph indicates the impact of increasing attachments size on performance.

Graph 2: Email Performance for Increasing Attachment Size

0

10

20

30

40

50

60

1 Thread 5 Thread 1 Thread 5 Thread

Mes

sage

s per

sec

Email Performance - Increasing Attachment Size

No Attachment 120KB / (Personalized) 1 MB / (Personalized)

Sync Mode Async Mode



Email Performance for Optional Features Turned ON EngageOne Deliver provides a number of optional features, which can be turned ON or OFF. This section explains how performance varies when you turn the features ON one-by-one.

Graph 3: Email Performance with Optional Features ON

0

10

20

30

40

50

60

No Attachment 120KB Attachment Digital Signature Archive 1 MB Attachment

Mes

sage

s per

sec

Email Performance - Optional Features ON

Sync Mode with 1 Thread Async Mode with 1 Thread Sync Mode with 5 Thread Async Mode with 5 Thread



SMS Performance Testing (Batch Processing) This section presents performance testing results for SMS processing. Note that SMS runs in sync mode only.

SMS Test Scenarios The following scenarios were executed to test the performance of SMS processing:

Table 2: SMS Test Data

S. No. Number of Input Files

Number of Records / Input

File

Feature Enabled

Messages Delivered/Sec with 1 thread

Messages Delivered/Sec with 5 threads

1 1 5000 NA 34 35

2 1 10000 NA 35 36

3 5 5000 NA 36 71

4 5 10000 NA 35 72

5 10 5000 NA 37 91

6 10 10000 NA 36 90

7 1 5000 Archive 28 34

8 5 5000 Archive 33 65

9 10 5000 Archive 27 74

10 5 50000 Archive 25 62

11 5 50000 NA 32 72



SMS Performance for Different Batch Sizes This section presents performance results for a number of test cases that represent actual customer usage. The following graph presents performance numbers for test cases where the number of input DIJ files varies.

Graph 4: SMS Performance for Different Batch Sizes

0

10

20

30

40

50

60

70

80

90

100

5K (1 Input fileof 5K records)

10K (1 Input fileof 10K records)

25K (5 Inputfiles of 5Krecords)

50K (5 Inputfiles of 10K

records)



records)


records)

Mes

sage

s per

sec

SMS Performance

1 Thread 5 Thread



SMS Performance with Archiving ON and OFF This section presents the performance numbers achieved after testing SMS processing where the archive feature is enabled and disabled.

Graph 5: SMS Performance with Archiving ON/OFF

0

10

20

30

40

50

60

70

80

90

100

5K (1 Inputfile of 5Krecords)




records)

5K (1 Inputfile of 5Krecords)




records)

Mes

sage

s per

sec

SMS Performance - Impact of Archival

1 Thread 5 Thread

Without Archival With Archival



REST API Testing This section presents performance testing results for Email and SMS processing through REST APIs. Note that the JMeter tool was used to measure user hits.

Email Test Scenarios (REST API) The following scenarios were executed to test the performance of email processing:

Table 1: Email Test Results for REST APIs

S. No. No. of Concurrent Users

No. of Hits/user

Attachment Size/Type

Feature Enabled

Average TPS (Message Delivered per sec)

Average Response Time (ms)

1 2 5000 No Attachment NA 41 47



4 2 5000 120KB / (Personalized) NA 29 54




8 2 5000 1MB / (Personalized) NA 11 124

9 5 5000 No Attachment Digital Sign 26 159

10 5 5000 No Attachment Archival 29 124

11 5 5000 No Attachment Digital Sign, Archival 24 183




REST API – Email Performance with different user scenarios

REST API – Email Performance with Optional Features enabled

05

1015202530354045

2 Users with5K hits each

withoutattachment


withoutattachment

10 Userswith 5K hits

each withoutattachment

2 Users with5K hits eachwith 120 Kbattachment


10 Userswith 5K hitseach with

120 Kbattachment




withoutattachment

Mes

sage

s per

Sec

REST API - Email Performance

0

5

10

15

20

25

30

35

40

No Attachment 120KB / (Personalized) Digital Sign Archival Digital Sign, Archival

Mes

sage

s per

sec

REST API Email Performance - Optional Features ON



SMS Test Scenarios (REST API) The following scenarios were executed to test the performance of SMS processing:

Table 1: SMS Test Results for REST APIs

S. No. No. of Concurrent Users

No. of Hits/user

Feature Enabled

Average TPS (Message Delivered per sec)

Average Response Time (ms)

1 2 10000 NA 63 33

2 5 10000 NA 60 81

3 10 10000 NA 57 164

4 2 10000 Archival 61 32

5 5 10000 Archival 58 79

6 10 10000 Archival 56 166

7 2 50000 NA 60 33

8 2 50000 Archival 58 34

9 2 100000 NA 61 33

10 2 100000 Archival 59 167

11 5 50000 NA 59 87

12 5 50000 Archival 57 85



REST API – SMS Performance with different user scenarios

REST API – SMS Performance showing impact of Archival

54

55

56

57

58

59

60

61

62

63

64

2 Users with 10K hits each 5 Users with 10K hits each 10 Users with 10K hits each 2 Users with 50K hits each

Mes

sage

s per

sec

REST API - SMS Performance

52

54

56

58

60

62

64

2 Users with 10K hits each 5 Users with 10K hits each 10 Users with 10K hits each 2 Users with 50K hits each

Mes

sage

s per

sec

REST API SMS Performance - Impact of Archival




Inbound SMS Testing This section presents performance testing results for Inbound SMS (MO). Tests were run with inbound running in existing sync mode and the newly introduced async mode.

These processing modes are controlled by the enable.InboundSmpp.ThreadPool property in SMPPService.properties file (valid values are true/false). When the value is set to true, async mode is in force. All other properties of Inbound SMS thread pool were kept at default values. Note that Async mode is the default and preferred option.

To simulate fresh MO messages, MO URL of SMPP simulator was hit from JMeter. These MOs were then read by Deliver through the Inbound SMS profile.

Inbound SMS Test Results

S. No. Inbound (sync / async)

Outbound SMS running in parallel

Feature Enabled

Average TPS (Message received per sec)

1 sync false NA 38

2 async false NA 80

3 sync true NA 34

4 async true NA 38

5 sync false Archival 8

6 async false Archival 36



Inbound SMS Performance

0102030405060708090

Sync data processing Async data processing Sync data processing Async data processing

Mes

sage

s per

sec

Inbound SMS - Performance

Without Outbound Processing in With Outbound Processing in Parallel



Bounce Email Testing This section presents performance testing results for Inbound Bounce email handling. Tests were run with inbound running in existing sync mode and the newly introduced async mode.

Outbound process for sending emails was also running in parallel to inbound bounce process. DIJ based outbound process was used for sending outbound messages.

Bounce Email Test Results

S. No. # DIJ files

# Records /

DIJ file

# Outbound

Threads Email

Attachment

Inbound process type

(Sync / Async)

Average Inbound TPS (Bounce

received per sec)

1 5 2000 5 NA S 5

2 5 2000 5 120 KB S 3

3 5 2000 5 500 KB S 1

4 5 10000 5 NA S 4

5 5 2000 5 NA A 13

6 5 2000 5 120 KB A 6

7 5 2000 5 500 KB A 2

8 5 10000 5 NA A 7



Bounce Email Performance

0

2

4

6

8

10

12

14

10K Records 50K Records

Mes

sage

s per

sec

Bounce Performance

Sync Async

0

2

4

6

8

10

12

14

No Attachment 120 KB 500 KB

Mes

sage

s per

sec

Bounce Performance - Impact of Attachment Size

Sync Async



Inbound New Email Testing This section presents performance testing results for inbound new email handling. Tests were run with inbound running in existing sync mode and the newly introduced async mode.

New emails were push to hMailServer through JMeter SMTP sampler, which were later read by EngageOne Deliver inbound profile

New Email Test Results

S. No. # JMeter threads

# Records /

thread Email

Attachment

Archival

(Original Parts only)

Inbound process type

(Sync / Async)

Average Inbound TPS (Email received

per sec)

1 5 5000 NA NA S 10

2 5 5000 NA NA A 25

3 5 5000 NA Y S 4

4 5 5000 NA Y A 21

5 5 5000 120 KB NA S 4

6 5 5000 120 KB NA A 9

7 5 5000 120 KB Y S 3

8 5 5000 120 KB Y A 8



Inbound New Email Performance

0

5

10

15

20

25

30

No Attachment 120 KB

Mes

sage

s per

sec

New Email Performance - Attachment Size

Sync Async

0

5

10

15

20

25

30

No Attachment 120 KB

Mes

sage

s per

sec

New Email Performance - Impact of Archival




Mobile Push Notification Testing This section presents performance testing results for Mobile Push Notifications. Tests were run for FCM, APNS and AWS gateways. JMeter was used for hitting web services.

Test Results

Gateway Type

Rest Service Max Thread Pool Size

# Users/Threads in JMeter # Hits/User Average TPS (Message

delivered per sec)

APNS 20 10 5000 50

APNS 30 10 5000 55

AWS APNS 20 10 5000 11

AWS APNS 30 10 5000 21

FCM 20 10 5000 40

FCM 30 10 5000 50

AWS FCM 20 10 5000 10

AWS FCM 30 10 5000 20



Mobile Push Notification Performance

0

10

20

30

40

50

60

APNS AWS APNS FCM AWS SNS

TPS

Mobile Push Notification Performance - Mobile Push Channels

Thread Pool Size = 30No. of Users = 5No. of Hits per User = 5000

0

10

20

30

40

50

60

APNS AWS APNS FCM AWS SNS

TPS

Mobile Push Notification Performance - Thread Pool Size

Thread Pool Size 20 Thread Pool Size 30

No. of Users = 5No. of Hits per User = 5000



System Load Results This section presents the system load in terms of memory and CPU utilization.

The following parameters were observed while processing a number of scenarios. Statistics were captured through the Windows Task Manager utility.

• CPU Utilization • Memory Consumption • Network IO

Screen shots were taken during the main processing time period of each particular scenario. The period selected represents the most frequent utilization on average for the relevant scenario. Note that results are captured for the EngageOne Deliver Application Server (Tomcat), Mail / SMS server and Database machine.



Scenario 1: Email, Async mode, 5 files, 5 threads Tomcat Server machine



hMailServer machine

Database machine



Scenario 2: Email, Async mode, 5 files, 5 threads, 120 KB attachment Tomcat Server machine

hMailServer machine



Scenario 3: Email, Async mode, 5 files, 5 threads, 120 KB attachment, Digital Signature Tomcat Server machine

hMailServer machine



Scenario 4: Email, Async mode, 5 files, 5 threads, 1 MB attachment Tomcat Server machine

hMailServer machine



Scenario 5: SMS, 10 files, 5 threads Tomcat Server machine



SMPP Simulator machine

Database machine



Scenario 6: SMS, 10 files, 5 threads, Archival ON Tomcat Server machine

SMPP Simulator machine



EngageOne Deliver Performance Recommendations Note: You can also review the recommendations in the EODeliver Installation Guide.

The primary factors for achieving maximum performance include the following:

1. Disk Drive IOPS: Persistent storage performance is vital for achieving optimum results. High Performance RAID storage with Serial Attached SCSI (SAS) drives, such as 15K RPM Hard Drives or High Performance SLC SSD is highly recommended.

2. CPU (quantity and speed): Due to the multi-threaded architecture of Deliver application, 8 processors (2 Quad cores) or more are recommended.

3. RAM (quantity and speed): To take advantage of file system caching and to provide ample memory for JVM to allow internal caching, 32 GB or more of high-speed RAM is recommended.

4. Multiple disks: If separate and independent high-performance drives are available, configure each of the following directories on a separate drive but on the same server. If only one high performance drive is available, keep all directories on that drive.

• Directories containing the DIJ, HTML, text, images, error, and vendor files

• Archive Folder (if archiving enabled)

5. Latency to the Email server and SMS gateways should be kept to a minimum; 10 ms maximum, zero latency when possible.

6. Bandwidth to the Email server and SMS gateways should be sufficient to handle the expected amount of traffic.

7. It is recommended to use a connection with a speed of at least double the expected traffic size since any binary data (such as PDF attachments) must be UUEncoded; this increases the size of the email. For example, if your email body is 15KB and your attachment is 120KB, your total email size will be 135KB. If you want to send 10 per second you would need at least 2.7MB/sec Internet connectivity (2 * 10 * 135KB/sec. = 2.7MB/sec). Note if emails and SMS will be sent and received simultaneously ensure your bandwidth requirements include both.

8. Firewalls can reduce performance and should be avoided wherever possible.

9. Ensure that the Deliver database purge scripts are set to run on a regular basis.



10. Regularly perform server Health Check and Database maintenance activities:

• Update Statistics, Purge SQL Logs and tempDB

• Application logs - Review logging levels

• Disk Defragmentation



Configuration considerations for optimal performance In order to achieve the above stated performance numbers, the following settings have been configured in accordance with the test environment used for this performance benchmark test. These values may not work for all environments. However, whenever a new environment is tuned, you should typically targeted these settings first:

• The number of threads configured in the outboundProcessor.properties file should be equal to the number of DIJs (Journal files) to be executed. For more information related to this property, refer EODeliver Reference Guide.

• While installing Deliver, you can set the Root Folder (vendor folder) locally or remotely depending on which provides a desirable latency time of 1 ms or less. For more information, please refer EODeliver Installation Guide.

DIJ based email outbound processing can be run in legacy sync mode and more advanced async mode. Async mode is the default and preferred method.

Running email outbound processing in Async mode With the release of 1.5.1.0, a new Async mode was introduced for email outbound processing. This effectively separated the email composition process from the email sending process. This helps in configuring sending threads (or effective email gateway connection) separately, irrespective of the processing threads setting. This may provide a considerable performance improvements where higher email gateway capacity is available. For example, the availability of high speed bandwidth and email gateway response time. Improved performance may be observed where more than one email gateway is available to EngageOne Deliver.

Note that the Async mode setting is valid for DIJ file based email processing only. REST APIs always work in async mode.

Async mode can be enabled through a property in asyncProcesses.properties file.

# define process type values sync and async outbound.process.type=async

Detailed configurations for sending thread and pool size can also be applied in this file. Refer to the EODeliver Reference guide for details.



Settings for parallel profile threads and gateway connections The following table presents a sample method for calculating the required Email/SMS connection pool settings based on the outbound profile and number of profile threads. Note that Email and SMS connection pool settings need to be adjusted based on the load and available gateway capacity.

Table 1: Parallel Thread Settings

No of active e-mail profiles [X]

E-mail parallel threads [A]

Pool requirement for e-mail [X*A]

No. of active SMS profiles [Y]

SMS parallel threads [B]

Pool requirements for SMS [Y*B]

Total pool size [(X*A)+(Y*B)]

No. of parallel connections allowed at EngageOne Deliver email gateway [1.2*(X*A)]

10 1 10 1 1 1 11 12

10 2 20 1 2 2 22 24

10 3 30 2 1 2 32 36

20 1 20 2 2 4 24 24

20 2 40 3 1 3 43 48

20 3 60 3 2 6 66 72

50 1 50 3 3 9 59 60

50 2 100 5 1 5 105 120

50 3 150 5 2 10 160 180

50 3 100 5 3 15 115 120

engageone deliver performance benchmarks engageone deliver 2

Documents