FroNtier: FroNtier: High Performance Database Access High Performance Database Access Using Standard Web Components in a Using Standard Web Components in a

Scalable Multi-tier ArchitectureScalable Multi-tier Architecture

Marc PaternoMarc Paterno

FermilabFermilab

CHEP 2004 CHEP 2004 Sept. 27-Oct. 1, 2004 Sept. 27-Oct. 1, 2004

Interlaken SwitzerlandInterlaken Switzerland

2

CreditsCredits

Fermilab, Batavia, IllinoisFermilab, Batavia, IllinoisJim Kowalkowski, Sergey Kosyakov, Dmitri Litvintsev, Lee Jim Kowalkowski, Sergey Kosyakov, Dmitri Litvintsev, Lee

Lueking, Marc Paterno, Stephen WhiteLueking, Marc Paterno, Stephen White

Johns Hopkins University, Baltimore, MarylandJohns Hopkins University, Baltimore, MarylandBarry Blumenfeld, Petar Maksimovic, Mark MathisBarry Blumenfeld, Petar Maksimovic, Mark Mathis

Rovaniemi Polytechnic, Rovaniemi, FinlandRovaniemi Polytechnic, Rovaniemi, FinlandLauri Autio Lauri Autio

(through collaboration with Fermilab(through collaboration with Fermilaband The University of Helsinki)and The University of Helsinki)

3

OutlineOutline

MotivationMotivation Overview Overview TestingTesting DeploymentDeployment Status and PlansStatus and Plans

4

MotivationMotivation

CDF wasCDF was– concerned about the scalability of their current database concerned about the scalability of their current database

access layeraccess layer

– looking for a straightforward path to move to a multi-looking for a straightforward path to move to a multi-tier data delivery architecture using commodity toolstier data delivery architecture using commodity tools

Why a multi-tier approach with caching?Why a multi-tier approach with caching?– Ideal for write once, read many data Ideal for write once, read many data

– Reliability and high performance w/o expensive DB Reliability and high performance w/o expensive DB serversservers

– Readily scalable to serve thousands of clientsReadily scalable to serve thousands of clients

– Decouples client and database development and Decouples client and database development and maintenancemaintenance

5

Transport ProtocolTransport Protocol

Several choices for transport protocol were Several choices for transport protocol were possiblepossible– many projects use home-grown protocols, based on many projects use home-grown protocols, based on

bare socketsbare sockets

The use of HTTP for the server-client protocol The use of HTTP for the server-client protocol provides a starting pointprovides a starting point– Opens the door to many well understood commodity Opens the door to many well understood commodity

tools and techniques used routinely on the internet.tools and techniques used routinely on the internet.

– These tools are easily deployed and have low These tools are easily deployed and have low administrative overheads.administrative overheads.

6

A Multi-tier ArchitectureA Multi-tier Architecture

Database (Oracle, MySQL, etc.)Database (Oracle, MySQL, etc.) Database Access LayerDatabase Access Layer

– Tomcat: servlet management engineTomcat: servlet management engine

– JDBC: Database connection JDBC: Database connection

– Database Connection pool management Database Connection pool management

Caching and proxy layer: SquidCaching and proxy layer: Squid– widely used, highly configurable, widely used, highly configurable,

caching proxy servercaching proxy server

ClientClient– Client API provides simple interface packaged in a Client API provides simple interface packaged in a

compact client librarycompact client library

7

FroNtier OverviewFroNtier Overview

CDF Persistent Object Templates(Java)

FroNtier components in yellow

Client

Caching

FroNtierServer

Database

FroNtier Client API Library

Squid Proxy/Caching Server

FroNtier Servlet running under Tomcat

Database (or other

persistency service)

XML Server Descriptors

DDL for Table Descriptions

C++ Header and Stubs

JDBC

HTTP

HTTP

8

FroNtier Servlet DesignFroNtier Servlet Design

1.1. Client sends request (URI)Client sends request (URI)

2.2. Command ParserCommand Parser translates translates URI into commands + URI into commands + valuesvalues

3.3. Servicer FactoryServicer Factory gets XSD gets XSD (XML Server Description) (XML Server Description) from database and from database and

4.4. Instantiate aInstantiate a Servicer Servicer

5.5. Servicer queries database Servicer queries database andand

6.6. Results sent for encodingResults sent for encoding

7.7. EncoderEncoder marshals marshals (serializes) the data to (serializes) the data to requestor clientrequestor client DatabaseDatabase

CommandCommandParserParser

ServicerServicerFactoryFactory

ServicerServicer

EncoderEncoder

ClientClient

1

2

3

4

5

6

7

9

FroNtier XML Server DescriptorFroNtier XML Server Descriptor

Object name and Object name and version informationversion information

Response Response descriptiondescription

The SQL mapping to The SQL mapping to the databasethe database

– Select statementSelect statement

– From statementFrom statement

– Where clauseWhere clause

– Special modifiers Special modifiers ((order byorder by, etc), etc)

<descriptor type="CalibRunLists“ version="1" xsdversion="1"><attribute position="1“   type="int“ field="calib_run" /> <attribute position="2"   type="int“     field="calib_version" /> <attribute position="3"   type="string“ field="data_status" /> <select> calib_run, calib_version, data_status </select><from> CalibRunLists </from> <where> <clause> cid = @param </clause>   <param position="1" type="int“ key="cid"/> </where><final> </final>

</descriptor>

10

FroNtier use of Squid CacheFroNtier use of Squid Cache

Squid (v 2.5STABLESquid (v 2.5STABLEnn) – Open source) – Open source– Well documented, widespread operational experienceWell documented, widespread operational experience– Easily installed and maintainedEasily installed and maintained– Highly configurable for access control, disk cache tuning, Highly configurable for access control, disk cache tuning,

distributed cache peer relationships,…distributed cache peer relationships,…– Monitoring built in through SNMP-2 interfaceMonitoring built in through SNMP-2 interface

Refreshing the cache (3 options)Refreshing the cache (3 options)– Driven by servlet: expiration time sent in HTTP headerDriven by servlet: expiration time sent in HTTP header– Driven by client: forced object refresh through requestDriven by client: forced object refresh through request– Delete each Squid’s cache files and rebuild the cacheDelete each Squid’s cache files and rebuild the cache

However, the objects being delivered are generally However, the objects being delivered are generally not changing, so a static cache matches the not changing, so a static cache matches the requirements.requirements.

11

FroNtier client API featuresFroNtier client API features

C and C++ interfacesC and C++ interfaces PortablePortable

– 32 and 64 bit systems tested32 and 64 bit systems tested

Transparent object accessTransparent object access– Type conversion detectionType conversion detection

– Preserves data integrityPreserves data integrity

Multi-object requestsMulti-object requests Easy runtime configurationEasy runtime configuration Extensive error reportingExtensive error reporting

– Adjustable logsAdjustable logs

FroNtier Service

User application

FroNtier API

12

General Deployment OverviewGeneral Deployment Overview

Dedicated Squid Node

Tomcat Node 1

Tomcat Node N

Tomcat Node 2

CAF 1Processor Farm

Database

Dedicated Squid Node

CAF NProcessor Farm

Dedicated Squid Node

CAF 2Processor Farm

Squid Node N

SquidNode 2

Cache Layer local to each CAFGbit ethernit

Intermediate Cache Layer

DB Access Layer

SquidNode 1

Load Balancing and Failover

LaunchpadWANWAN

14

Client Side: FNAL/SDSCClient Side: FNAL/SDSC

No CachingCaching at SDSC Squid

1000 Sec 1 Sec

16

FroNtier Status and PlansFroNtier Status and Plans

CDF Production readyCDF Production ready– Client library is included in CDF production code Client library is included in CDF production code

– Production executables built and tested with FroNtier.Production executables built and tested with FroNtier.

– Widespread use will begin as new releases are builtWidespread use will begin as new releases are built

Plan to extend the use in CDF beyond calibration Plan to extend the use in CDF beyond calibration data to trigger and other static data needed offline.data to trigger and other static data needed offline.

Exploring a prototype of a FroNtier “plug-in” for Exploring a prototype of a FroNtier “plug-in” for POOL (LCG). Some interest in CMS to use a POOL (LCG). Some interest in CMS to use a web-based approach for DB access. web-based approach for DB access.

17

SummarySummary

FroNtier is a multi-tier architecture providing FroNtier is a multi-tier architecture providing clients high throughput, low latency, scalable clients high throughput, low latency, scalable access to a persistent store, such as a database. access to a persistent store, such as a database.

The CDF DB access framework has been adapted The CDF DB access framework has been adapted to use the FroNtier approach. Extensive testing to use the FroNtier approach. Extensive testing was performed to confirm the integrity, was performed to confirm the integrity, performance, and reliability of the system.performance, and reliability of the system.

We are in the process of executing a deployment We are in the process of executing a deployment plan and will soon have real-world production plan and will soon have real-world production experience with the system.experience with the system.

18

ReferencesReferences

FroNtier Talks:FroNtier Talks:– http://whcdf03/ntier-wiki/Additional_20Documentationhttp://whcdf03/ntier-wiki/Additional_20Documentation

FroNtier working page:FroNtier working page:– http://whcdf03/ntier-wikihttp://whcdf03/ntier-wiki