© 2011 SAP AG. All rights reserved. / Page 2
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP.
SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice.
This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
© 2011 SAP AG. All rights reserved. / Page 3
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer
© 2011 SAP AG. All rights reserved. / Page 4
ERP
Architecture OverviewIn-Memory Computing Engine and Surroundings
LogERP DB
In-Memory Computing Engine
Clients (planned, e.g.) BI4 Explorer
Dashboard Design
SAP BI4 universes (WebI,...)
Request Processing / Execution Control
MS Excel
BI4 Analysis
SQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
In-Memory Computing Studio
Administration Modeling
Load Controller
Replication Agent
Replication Server
SAP Business Objects BI4
Data Services Designer
SBO BI4 servers
( program for client)
SBO BI4 Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
© 2011 SAP AG. All rights reserved. / Page 5
ERP
Architecture OverviewThe Engine
LogERP DB
Clients (planned, e.g.) SBOP Explorer 4.0
Xcelsius SAP BI universes (WebI,...)
MS Excel
SBOP Analysis
IMC Studio
Administration Modeling
Load Controller
Replication Agent
Business Objects Enterprise
Data Services Designer
SBO server programs for clients
SBO Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
Replication Server
© 2011 SAP AG. All rights reserved. / Page 6
ERP
Architecture OverviewLoading Data into SAP HANA
LogERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
In-Memory Computing Studio
Administration Modeling
Load Controller
Replication Agent
Replication Server
Business Objects Enterprise
Data Services Designer
SBO BI4 servers
( program for client)
SBO Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
Clients (planned, e.g.) BI4 Explorer
Dashboard Design
SAP BI4 universes (WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 7
ERP
Architecture OverviewData Modeling
LogERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
In-Memory Computing Studio
Administration Modeling
Load Controller
Replication Agent
Replication Server
Business Objects Enterprise
Data Services Designer
SBO BI4 servers
( program for client)
SBO Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
Clients (planned, e.g.) BI4 Explorer
Dashboard Design
SAP BI4 universes (WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 8
Clients (planned, e.g.)
ERP
Architecture OverviewReporting
LogERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
In-Memory Computing Studio
Administration Modeling
Load Controller
Replication Agent
Replication Server
Business Objects Enterprise
Data Services Designer
SBO BI4 servers
( program for client)
SBO Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
BI4 Explorer
Dashboard Design
SAP BI4 universes (WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 9
ERP
Architecture OverviewAdministration
LogERP DB
In-Memory Computing Engine
Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine
Transaction Manager
Session Management
Relational EnginesRow Store Column Store
Persistence LayerPage Management Logger
Disk StorageLog VolumesData Volumes
Authorization Manager
Metadata Manager
In-Memory Computing Studio
Administration Modeling
Load Controller
Replication Agent
Replication Server
Business Objects Enterprise
Data Services Designer
SBO BI4 servers
( program for client)
SBO Information Design Tool
Other Source Systems
SAP NetWeaver
BW3rd Party
Data Services
Clients (planned, e.g.) BI4 Explorer
Dashboard Design
SAP BI4 universes (WebI,...)
MS Excel
BI4 Analysis
© 2011 SAP AG. All rights reserved. / Page 10
DB Server
SAP High-Performance Analytic Appliance 1.0
SAP HANA
JDBC ODBC ODBO SQL DBC
SAP In-Memory Computing Engine
ReplicationServer
SAP In-Memory Computing Studio
SAP Business Application
ReplicationAgent
SAP BusinessObjects Data Services 4.0
Any source
SAP BusinessObjects
BI 4.0
Repository
SAP BusinessObjects BI clients
SQ
L
MD
X
BIC
S
Aut
hent
icat
ion
Con
tent
mgm
t
sync
Adm
in &
mod
el
load (optional)
(optional)
(optional)
(existing)
© 2011 SAP AG. All rights reserved. / Page 11
Request Processing and Execution ControlConceptual View
Standard SQL Processed directly by DB engine
SQL Script, MDX and planning engine interface
Domain-specific programming languages or modelsConverted into calculation models
Calc EngineCreate logical execution plan for calculation modelsExecute user defined functions
Relational EngineDB optimizer produces physical executing planAccess to row and column store
© 2011 SAP AG. All rights reserved. / Page 12
Calc Engine for Dummies
The easiest way to think of Calculation Models is to see them as dataflow graphs, where the modeler can define data sources as inputs and different operations (join, aggregation, projection,…) on top of them for data manipulations.
The Calculation Engine will break up a model, for example some SQL Script, into operations that can be processed in parallel (rule based model optimizer). Then these operations will be passed to the database optimizer which will determine the best plan for accessing row or column stores (algebraic transformations and cost based optimizations based on database statistics).
© 2011 SAP AG. All rights reserved. / Page 14
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer
© 2011 SAP AG. All rights reserved. / Page 15
In-Memory Computing EngineHigh Level Architecture
Row StoreOne of the relational enginesInterfaced from calculation / execution layerPure in-memory store
Persistence managed in persistence layer
SAP in-memorycomputing engine
HANA
© 2011 SAP AG. All rights reserved. / Page 16
Row Store ArchitectureRow Store Block Diagram
Row Store Block DiagramTransactional Version Memory
Contains temporary versionsNeeded for Multi-Version Concurrency Control (MVCC)
SegmentsContain the actual data (content of row-store tables) in pages
Page ManagerMemory allocationKeeping track of free/used pages
Version Memory ConsolidationThink ‘garbage collector for MVCC’
Persistence LayerInvoked in write operations (log)And in performing savepoints checkpoint writer
© 2011 SAP AG. All rights reserved. / Page 17
Row Store ArchitectureHighlights
Write OperationsMainly go into “Transactional Version Memory”“INSERT” also writes to Persisted Segment
Read Operations
Write Operations
Transactional Version Memory
Main Memory
Persisted Segment
Data that may be
seen by all active
transactions
Recent versions of changed records
Version Memory Consolidation
Version ConsolidationMoves “visible version” from Transaction Version Memory into Persisted Segment (based on Commit ID)Clears “outdated” record versions from Transactional Version Memory
Memory HandlingRow store tables are linked list of memory pagesPages are grouped in segmentsPage size: 16 KB
Persisted SegmentContains data that may be seen by any ongoing transaction Data that has been committed before any active transaction was started)
© 2011 SAP AG. All rights reserved. / Page 18
Indexes for Row Store TablesPrimary Index / Row ID / Index Persistence
Each row-store table has a primary indexPrimary index maps ROW ID primary key of table
ROW ID: a number specifying for each record its memory segment and page
How to find the memory page for a table record?A structure called “ROW ID” contains the segment and the page for the recordThe page can then be searched for the records based on primary keyROW ID is part of the primary index of the table
Secondary indexes can be created if needed
Persistence of indexes in row storeIndexes in row store only exist in memory
No persistence of index dataIndex definition stored with table metadataIndexes filled on-the-fly when system loads tables into memory on system start-up
© 2011 SAP AG. All rights reserved. / Page 19
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer
© 2011 SAP AG. All rights reserved. / Page 20
In-Memory Computing Engine High Level Architecture
Column StoreOne of the relational enginesInterfaced from calculation / execution layerPure in-memory store
Persistence managed in persistence layer
Optimized for high performance of read operationGood performance of write operationsEfficient data compression
SAP in-memorycomputing engine
HANA
© 2011 SAP AG. All rights reserved. / Page 21
Column Store ArchitectureColumn Store Block Diagram
Column Store Block DiagramOptimizer and Executor
Handles queries and execution plan
Main and Delta StorageCompressed data for fast read Delta data for fast writeAsynchronous delta merge
Consistent View Manager
Transaction Manager
Persistence Layer
© 2011 SAP AG. All rights reserved. / Page 22
Column StoreHighlights
Storage Separation (Main & Delta)Enables high compression and high write performance at the same time
Delta Merge OperationSee next slide
Read Operations
Write Operations
Main
Main Memory
Delta
Write optimized
Compressed and
Read optimized
Read OperationsAlways have to read from both main & delta storages and merge the results.Engine uses multi version concurrency control (MVCC) to ensure consistent read operations.
Data Compression in Main Storage
Compression by creating dictionary and applying further compression methodsSpeed up
Data load into CPU cacheEquality check Search
The compression is computed during delta merge operation.
Write OperationsOnly in delta storage because write optimized.The update is performed by inserting a new entry into the delta storage.
© 2011 SAP AG. All rights reserved. / Page 23
Column Store Delta Management
Delta Merge OperationPurpose
To move changes in delta storage into the compressed and read optimized main storageCharacteristics
Happens asynchronouslyEven during merge operation the columnar table will be still available for read and write operationsTo fulfil this requirement, a second delta and main storage are used internally
Read Operations
Write Operations
Main
Before Merge
Delta
Read Operations
Write Operations
MainNew
After Merge
DeltaNew
Read Operations
Write Operations
Main
During Merge
Main New
DeltaNewDelta
Merge Operations
© 2011 SAP AG. All rights reserved. / Page 24
Agenda
1. Architecture Overview2. Row Store3. Column Store4. Persistence Layer
© 2011 SAP AG. All rights reserved. / Page 25
Persistence Layer Purpose and Scope
Why Does An In-memory Database Need A Persistence Layer?Main Memory is volatile. What happens upon…
Database restart?Power outage?...
Data needs to be stored in a non-volatile wayBackup and restore
SAP in-memory computing engine offers one persistence layer which is used by row store and column store
Regular “savepoints” full persisted image of DB at time of savepoint
Logs capturing all DB transactions since last savepoint (redo logs and undo logs written)restore DB from latest savepoint onwards
Ability to create "snapshots"used for backups
© 2011 SAP AG. All rights reserved. / Page 26
Persistence Layer System Restart and Population of In-memory Stores
Actions During System RestartLast savepoint must be restored plus…
Undo logs must be read for uncommitted transactions saved with last savepointRedo logs for committed transactions since last savepoint
Complete content of row store is loaded into memory
Column store tables may be marked for preload or notOnly tables marked for preload are loaded into memory during startupIf table is marked for loading on demand, the restore procedure is invoked on first access
© 2011 SAP AG. All rights reserved. / Page 28
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc.JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, Clear Enterprise, SAP BusinessObjects Explorer and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP France in the United States and in other countries. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.
© 2011 SAP AG. All Rights Reserved