plone tuneup challenges

1

Plone TuneUp challenges

in search of enterprise intranet solution

Andrew Mleczko (redturtle.net)

European Plone Symposium 2009

Sorrento, Italy

2

PoznanPoznan FerraraFerrara

3

● Co-founder of ACC Poland Ltd., the first Polish

Plone company (past)

● Plone consultant at RedTurtle Technology -

based in Ferrara (present)

● Plone-based and Lotus-based web applications

4

RedTurtle Technology

“Sharing knowledge

about knowledge sharing”

5

So what can you expect...

● Plone is just a CMS?

● Case study – Intranet Provincia di Ferrara

● Loadtests environment

● Plone TuneUp challenges

6

How we can use

Plone?

7

Plone variety

● Community portals

● Document management system (DMS)

● Groupware - Collaboration

● Subsite management system

● Enterprise intranets

● ...

● Combination of all mantioned above

8

Plone as CMS

vs.

Plone as framework

9

Why?

“Plone is NOT a general web development framework.“

/Carlos de la Guardia/

10

Plone can be SLOW*

*when using the wrong way

11

Framework comparision

● Used hardware:

– CPU: AMD OpteronT Processor 146 (2 GHz)

– Memory: 2 GB

● Testing with Siege 2.68

● Read scenario

● Results for 50, 200, 300 Concurent Users (CU)

● More details: www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks

12

Comparision

Django RubyOnRails TurboGears0

0,5

1

1,5

2

2,5

50 CU200 CU300 CU

Response time, secs.

13

Comparision

Django RubyOnRails TurboGears0

5

10

15

20

25

30

35

50 CU200 CU300 CU

Longest transaction, secs.

14

So where is Plone?

15

Far, far away

Django RubyOnRails TurboGears Plone0

2

4

6

8

10

12

14

16

18

50 CU200 CU300 CU

Response time, secs.

Django RubyOnRails TurboGears Plone0

10

20

30

40

50

60

50 CU200 CU300 CU

Longest transaction, secs.

16

But comparing with Zope2

Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope20

2

4

6

8

10

12

14

16

18

50 CU200 CU300 CU

Response time, secs.

17

What can we do...

● Do not treat Plone as a framework

● For some use cases:

use ContentMirror + fast framework (django,

pylons, repoze)

● But what about enterprise intranet solutions?

18

Plone for enterprise intranet

● Large number of users (more than 500) and

documents (more than 100 000)

● A lot of simultaneous write requests (>20 CU/s)

● Using of caching limited

● Slowest acceptable resp. time <10 seconds

the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]

19

Intranet Provincia di Ferrara

Case study

20

Prerequisites

● 800 users stored in Domino Server (by PAS)

● 170.000 documents

● Yearly growth: ca 25.000 documents

● Peak traffic: 20 CU/s - write transactions

(based on Domino logs)

21

Objective

● User response time (mixed read/write):

– Slowest (5-6 seconds)

– Recommended (2-4 seconds) 90% of all request

– Fast (<2 seconds)

22

Test environment

● Hardware (current):

– 4 dual-core Intel Xeon 2.8 Ghz

– 32 GB RAM

● Amazon EC2 (future)

23

Test environment

● collective.loadtesting

– funkload

– collective.funkload

● PageTemplate Profiler

● mr.bent

● Munin + redturtle.munin

24

class Readonly(testcase.FLTestCase):        def setUp(self):        self.server_url = 'http://localhost:8080/plone'

    def test_ReadOnly(self):        server_url = self.server_url        self.get(server_url + "/plone/login_form",            description="Get /plone/login_form")

        self.post(server_url + "/plone/login_form", params=[            ['came_from', server_url + '/plone'],            ['form.submitted', '1'],            ['cookies_enabled', ''],            ['login_name', ''],            ['pwd_empty', '0'],            ['__ac_name', 'admin'],            ['__ac_password', 'admin'],            ['submit', 'Accedi']],            description="Post /plone/login_form")

collective.loadtesting

25

mr.bent

>>> from mr.bent.wrapper import mkwrapper>>> from mr.bent.tests.test_bent import callcounter>>> from Products.CMFPlone.CatalogTool import \    CatalogTool

>>> mkwrapper(CatalogTool.searchResults,  callcounter, "catalogsearches")

26

Test scenario

● Log in

● View random folder

● Create document (2 requests)

● View newly created document

● Publish it

● View home page

● Logout

27

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

5CU20CU60CU

Default clean Plone

Response time, secs.

28

Default clean Plone

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU

% Errors (ConflictErrors)

29

Let's start TuneUp and install

the basics:

Varnish + Pound (with sticky session) +

+ 6 ZEO Clients (18 threads) + ZEO Server

30

● Apache – proxy/rewriting urls

● Varnish – cache resources, add http headers

● Pound – balancing with sticky sessison

● 6 ZEO clients

● ZEO Server

31

+ Varnish + Pound + ZEO

Log in Folder view Create document Save document View Publish Home page Logout

0

5

10

15

20

25

30

35

40

5CU20CU60CU

Response time, secs.

32

+ Varnish + Pound + ZEO

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU

% Errors (ConflictErrors)

33

Comparing to default Plone

34

Log in Folder view Create document Save document View Publish Home page Logout

0

5

10

15

20

25

30

35

40

Default PloneImprovment

Comparision

Response time, secs., 20 CU

35

Where are the conflict errors

comming from?mainly portal_catalog

36

Let's try:mount portal_catalog to different ZODB

and increase zodb cache on it

37

● Install redturtle.catalogmount● Add zope-conf-additional:

<zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>

38

● And add zeo-conf-additional:

<filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs</filestorage>

39

+ redturtle.catalogmount

Log in Folder view Create document Save document View Publish Home page Logout

0

5

10

15

20

25

5CU20CU60CU

Response time, secs.

40

+ redturtle.catalogmount

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU

% Errors (ConflictErrors)

41

Maybe there is a more

stable solution:collective.indexing

experimental.catalogqueryplan

archetypes.schematuning

42

Log in Folder view Create document Save document View Publish Home page Logout

0

2

4

6

8

10

12

14

16

5CU20CU60CU

+ collective.indexing + archetypes.schematuning

Response time, secs.

43

+ collective.indexing + archetypes.schematuning

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

70

80

90

100

5CU20CU60CU

% Errors (ConflictErrors)

44

So what is the improvment?

45

Improvment

Log in Folder view Create document Save document View Publish Home page Logout

0

5

10

15

20

25

30

35

40

fresh Ploneafter TuneUp

Response time, secs., 20 CU

46

Let's try to scale it...

47

ZEO scalability problem

Log in Folder view Create document Save document View Publish Home page Logout

0

5

10

15

20

25

30

35

40

45

1:104:206:1810:30

Response time, secs. 20 CU

48

ZEO scalability problem

Log in Folder view Create document Save document View Publish Home page Logout

0

10

20

30

40

50

60

1:104:206:1810:30

% Errors (ConflictErrors), 20 CU

49

Plone intranet limits

● ZODB is slow when handling concurrent writes

transactions

● Scalability problem using flat ZEO

● Missing 'out of the box' performance

(comparing to other framworks, CMS)

50

How we can solve the problem?

51

Ideas

● RelStorage for scalability problem

● collective.solr

● others ?

● share knowledge at plone-enterprise mailing list