Download - Information Management Trends and Some History

© 2007 IBM Corporation

Information Management Trendsand Some History

C. Mohan, PhD IBM Fellow & IBM India Chief Scientist Member, IBM Software Group, Asset Architecture & Information Management Architecture Boards

http://www.almaden.ibm.com/u/mohan/[email protected]

DBA Mindshare, Kolkata 26 Apr 2007, C. MohanDBA Mindshare, Kolkata 26 Apr 2007, C. Mohan2

Key Customer Pain Points

Can’t Find Information – Discovery

Can’t combine Information – Integration

Can’t extract value from Information – Insight

Can’t consume Information – Dissemination


Today’s business challenges mandate a fresh approach to

managing information

Managing information in silos has

become obsolete

The Information Challenge Information is in Silos… Trusted Information is Not Available

Multiple Versions of the Truth

Inaccurate, Untimely

Inconsistent

Incomplete, Inaccessible

Out of Context…

Globalization, M&As

Risk & Compliance,

Eroding Customer Loyalty,

Supply Chain Complexity,

Industry Transformations,

Cost Cutting…

70% of people’s time can be spent searching for

relevant information

60%+ of CEOs: Need to do a better job leveraging

informationSources: IBM Attributes & Capabilities Study, 2005; Client Interviews 2004; IBM CFO Study, 2006

5X More Value creation by organizations effective at

using Information as an Asset

Information Must Become a

Strategic Asset


Information Management Trends

Information Intensive Applications Shift from transaction-centric to information-intensive applications

Information Diversity Delivering insight over increasingly diverse sources of information

New Business & Delivery Models Information as a Service, Outsourcing, New Licensing Models

Democratization of Information Changing User Expectations & the “Parent Test”

Massive Collaboration & Societal Intelligence Collaboration over shared information to creating business insight


Presentation Services

EDW

Legacy LegacyPortals, Browsers, and or Devices

StrategicAPPL

EventProcessing

TacticalAPPL

TxAPPL

AppServer

DiscoveryAPPL

MasterDataAPPLProcess

Services

Information Integration Services Analytic Services

Master Data Services

Transaction Application Services Analytic Application Services

Business Process Management

Federation

Discovery Services

ECW

Content ServicesCollaboration Services

Notes

Email

Enterprise Service Bus

Metadata Services

Master data Hubs

Product Customer

Supplier Location

Transaction Services

OLTP2OLTP1

OLTP

BusinessRules

BusinessMonitoring

StreamingBatch

Metadata

Information as a Strategic Asset


Compliance & Risk Mgmt. Sales and

Marketing – Closed Loop

Campaign Mgmt.

CustomerService

Data Stewardship& Administration

Compliance

Marketing

AccountAdministration

Privacy Management

Web Self-Service

WirelessSelf-Service

Distributor

IVRSelf-Service

Branch / Sales Office

Call CenterBrowser-based

UnlimitedAttributes

MultipleCategorizations

Multi-enterprise

Standards-based

Security andAudit

NewBusiness

Processing

Privacyand Data

Mgmt.

MarketingInsight

CustomerFacing Channels Internal Users

Customer

Master Data

Master Data Integration


Data Services– Databases, Warehouses, Tools…

Content & Discovery Services– Content Mgmt. & Integration Services– Discovery Services…

Information Integration Services– Quality Services– Transformation Services– Federation Services– Metadata Services…

Information Accelerators– Master Data Management– Entity Analytics– Information Warehousing– Customizable Dashboards– Industry Data Models…

Information Delivered On DemandBased on Services Oriented Architecture

IBM Information Management SoftwareDelivering Value Beyond Traditional Repositories


XML Developer “I see a sophisticated XML repository that also supports SQL."

SQL Developer"I see a sophisticated

RDBMS that also supports XML."

Familiar Programming Models

OptimizedStorage Models

MatureServices

Familiar Tooling

OptimizedPerformance &

Scale

DB2 9 – A Pure XML, Relational Hybrid


Integration of XML & Relational Capabilities

DB2 SERVER

CLIENT SQL/XML

XQuery

DB2 Engine

XMLInterface

RelationalInterface Relational

XML

DB2 Storage:

DB2 Client /Customer Client Application

– Applications combine XML & relational data

– Native XML data type (server & client side)

– XML Capabilities in all DB2 components


XQuerySQL/XML

APIs/ClientXML Indexes

XML Schemasupport Native

Storage

XML Load

Import/Export

Native XML support in DB2 with more to comeSeamless integration with the relational world

New XML

Join Methods

Tools

And all the

relational stuff

DB2 V9 pureXML support


DB2 V9 pureXML support

XML as a native data type

Pure XML storage and indexing

XQuery and SQL/XML support

XML Schema Repository

Schema validation

Application Support (Java, C/C++, .NET, PHP, etc.)

Visual Tooling, Control Center Enhancements

Annotated schema shredding

DB2 Utilities: Import/Export, HADR, etc.

…and more

Secure and Resilient

Infrastructure for a New

Breed of Agile

Applications

DB2

9


Some of Our Info Mgmt Research Legacy

Invention of Relational Model/Technology & SQL Research prototypes

ƒ System R ƒ R* Distributed DBMSƒ Starburst Extensible Object-Relational DBMS ƒ Garlic Heterogeneous DBMS

Product Contributionsƒ Data sharing on DB2 390 Sysplex ƒ DB2 UDB Query Processor ƒ Intelligent Minerƒ Lotus Notes R5 Recoveryƒ Discovery Link & DB2 Information Integrator

6 IBM Fellows from team of < 50


Why We Have Experience with Customers

Over 2 decades of partnership with SWG Toronto & SVL– Incorporation of Starburst prototype into DB2– Component Owners of DB2 for LUW’s Query Compiler– Versions 2 – 5 (1992-1997)– Dealt with customer APARs, Visits, & Presentations

Responsible for many DB2 innovations– Query Graph Model (internal query representation, key to extensibility)

– Query ReWrite and Optimizer technology

– ARIES recovery and locking methods

– Triggers and Constraints

– Star Join and Hash Join

– Object-relational features

– Automatic Summary Tables (materialized views)

– Visual Explain

– Index Advisor

Respected for our vision– World-class publications in leading database conferences– Cognizant of industry trends


Leveraging Technology and People

IMS

Development

DB2

Development

IDS / U2

Development

Customer

Requirements

IBM

Products

IBM

Research


SVL DB2 UDB for z/OS & OS/390IMSBusiness IntelligenceContent ManagementDB2 EveryplaceRed BrickIcingTraditional AD Languages

Boeblingen DB2 Text ExtendersSAP/R3 EnablementIntelligent Miner for DataIntelligent Miner for Text

Somers

HawthorneAdvanced Technology

AlmadenAdvanced Technology

Menlo Park & OaklandIDSXPSJDBCVisionaryCloudscapeDatabladesObject Connect & TranslatorContent Management

India DB2 UDB ServiceBusiness IntelligenceIDS

AustinGBIS

Portland XPS & DB2

LenexaIDS

Boulder & DenverContent ManagementU2

Datablades

Boca Raton & MiamiEMMSLA Informix Support

Rochester DB2 UDB for AS/400

Toronto DB2 UDB for UNIX, Windows, & OS/2

IBM Information Management Teams

Beijing Information IntegrationDB2 for zOSContent Management DB2 and IMS tools

Las VegasEntity Analytics

Over 6000 employees worldwide

Yamato High Speed Inverted Index SearchBusiness IntelligenceContent Management

Hursley Enterprise Master DataSolutions

India Software Lab– 3000 employees– Broad range of skills – all SWG Brands– Linux Competency Center

DB2 Lab within ISL– 100+ developers – Lab based services teams – DB2, CM, BI

Other Resources– India Research Lab– Solution Porting Center– Education Center for IBM Software– IBM Academic Initiative


A Spectrum of Info Serving Requirements

Platform: Mobile Desktop Small Servers Large Servers

Data Size: Micro Compact Large Extremely Large

Workload: Batch Online Transactions Real-time Analysis Data Mining

Structure: Hierarchical Relational Multi-Value XML

OS: Symbian PalmOS Windows Linux Unix i5/OS z/OS

Scope: Embedded Intra-application Single application Multi-application

Support: None Web/E-mail Business hours 24x7


Products to Match the Spectrum of Data Serving Needs

DB2 Everyplace

OLTP

Relational

MobileEmbedded

LinuxPalmOSSymbian

Cloudscape

OLTP

Relational

Intra-App / Single-App

Java

IDS

OLTP

Relational


AIX, etc.Linux

Windows

DB2

OLTP &Analysis

Relational & XML

Single / Multi-App

z/OSI5/OS

AIX, etc.Linux

Windows

IMS

OLTP

Hierarchical

Single / Multi-App

z/OS

U2

OLTP

Multi-Value


AIX, etc.Linux

Windows

Superior capabilities across the spectrum of requirements


DB2 for z/OS

The power and function of an open, industry standard data server with zSeries’ industry leading availability, performance, and security

What it takes to be the industry’s most extreme data server

Continuous application availability measured in years Ability to process over 1B SQL transactions per hour Uninterrupted growth from 1 byte to over a peta-byte Serving 100s of applications for 100,000s of users US Government’s highest security classification (zSeries) Support for industry standards: XML, Web services, Java, C, COBOL Support for complex business applications: SAP, PeopleSoft, Siebel

Extreme qualities of service XML and Relational data server


Technology Evolution with Mainframe Specialty Engines

Internal Coupling Facility (ICF) 1997

Integrated Facility for Linux (IFL) 2001

IBM System z9 Integrated Information Processor (IBM zIIP) planned for 2006

System z9 Application Assist Processor (zAAP) 2004

Building on a strong track record of technology innovation with specialty engines, IBM intends to introduce the System z9 Integrated Information Processor

Support for new workloads and open standards

Designed to help improve resource optimization for eligible data workloads within the enterprise

Centralized data sharing across mainframes

Incorporation of JAVA into existing mainframe solutions


Data ChallengesVariety, Velocity, and Volume New composite applications

need data from multiple sources Consumers expect holistic,

personalized, and value-addedcontent

Relational, XML, packaged applications, content repositories, file systems all contain critical business information

Increasing emphasis on current data Real-time analytics

Business activity monitoring

Petabytes will be the measure ofavailable online data

All client interactions are important ( e.g., instant messages, audio records, web traffic,…)

Internet and intranet content

The world produces 250MB of information every year for every

man, woman and child on earth.

10-100GB100s GB - 1TB

1 - 20 GBs100s MB100s KB

1999

1s TB1s TB

100s TB100s TB

1s TB1s TB

10s GB10s GB

1s GB1s GB

2004

10X

100X

100X

1,000X

10,000X

Common Database SizesCommon Database Sizes

Transactions

Warehouses

Marts

Mobile

Pervasive37% CGR DiskGrowth ’96-’07

70,000 TB of TV and Radio contentin 2002 alone; 30% growth/year


Addressing the Changing Characteristics of Data

Actionability

Heterogeneity

Scale

Query

CCGAGTACCCAC

Satellite & Surveillance Images and Video

Gene Sequences

Transactions

Text and Web

Increasing need to manage and analyze new data types

Protein Folding


Research in Information and Interaction

Drive our leadership technologies for search, structured and unstructured information processing and analytics, natural language processing, and conversational and multimodal interaction, across multiple tiers of business activities in SWG products and solutions. Foster the exploitation of components with these leading research

technologies in IGS services offerings.

Conversational and Multimodal Interactions

UnstructuredInformation

Management

InformationManagement

Database

Synthesis

Information Integration

Metadata

Speech Recognition

CM

InformationRetrieval

NLP

Analytics

Video Analysis


Worlds of Structured & Unstructured Data Come Together

Analytical

Complexity

Collect

Store

Retrieve

Drill

Mine

ETL

Warehouse

SQL

OLAP

Cluster, Classify, ..

Crawl

ECM

Search

Navigate

Cluster, Classify, ..

Solutions

II

Structured Data Unstructured Data


Need for Business Intelligence

HIPAAHIPAA

Basel IIBasel IIPatriot ActPatriot Act

Sarbanes-OxleySarbanes-Oxley

Loyalty Profitability Buyer Behavior Targeted Offers

Loyalty Profitability Buyer Behavior Targeted Offers

Homeland SecurityHomeland Security

Internet Buzz Anti-Money

Laundering Border Control Crime Information

Internet Buzz Anti-Money

Laundering Border Control Crime Information

Globalization Business Controls Mergers and Acquisitions Supply Chain Efficiencies

Globalization Business Controls Mergers and Acquisitions Supply Chain Efficiencies

Capitalism and Its Troubles: A Survey of International Finance -May 24, 2002

Capitalism and Its Troubles: A Survey of International Finance -May 24, 2002

Accountability and ComplianceAccountability and Compliance Customer KnowledgeCustomer Knowledge

Preparing for terrorHow scared should you be?

Nov 28th 2002 From The Economist print edition

Preparing for terrorHow scared should you be?

Nov 28th 2002 From The Economist print edition

Business PerformanceBusiness Performance

Risk Management Fraud and Abuse Public Protection

Risk Management Fraud and Abuse Public Protection


SOA Reference Architecture

Business Innovation & Optimization Services

Dev

elo

pm

ent

Ser

vice

s

Integrated environment for design

and creation of solution

assets

Manage and secure services,

applications &

resources

Facilitates better decision-making with real-time business information

IT S

ervi

ceM

anag

emen

t

Infrastructure Services

Optimizes throughput, availability and performance

ESBFacilitates communication between services

Ap

ps

&

Info

As

setsPartner Services Business App Services Access Services

Connect with trading partners

Build on a robust, scaleable, and secure services environment

Facilitates interactions with existing information and application assets

Interaction Services Process Services Information Services

Enables collaboration between people,

processes & information

Orchestrate and automate business

processes

Manages diverse data and content in a

unified manner


Understand Information Assets and Link to Business Context Discover information

metadata Map information to

business processes Develop data &

content models

Compose Information Services Across Heterogeneous Sources Extract, federate & transform

heterogeneous information

Service Information Requests Deliver unified data

& content Deliver business

context Discover

relationships

Ensure Performance, Availability & Security Meet Service Levels

Define & Refine Information Management Rules & Policies Monitor information usage over time

Information as a Service The SOA Lifecycle Mapped to Information Needs


and more…

abc…DB2

IBM ContentManager Oraclexyz…

Heterogeneous Applications & Information

Insight

Information as a ServiceOptimize, Virtualize, Integrate, Accelerate

Data & Content

BusinessContext

InsightfulRelationships

Master Data, Entity Analytics, Decision Portals, Executive Dashboards,Industry Data Models

Extracted or Real-time

Standards-based

e.g., XQuery, JSR170, JDBC, Web Services...

Information as a ServiceMoving From a Project-Based to a Flexible Architecture (SOA)

Processes PeopleTools & Applications


Information Services for SOAUnprecedented Business Flexibility

Store Information DB2 Viper

Optimized XML storage

Virtualize Information Access WebSphere Information Server

Integrate Information WebSphere Information Server

Accelerate Master Information WebSphere Customer Center

WebSphere Product Center

IBM Entity Analytics

Industry Models


Industry Solutions Deliver Insight On Demand

Law Enforcement Crime Information

Warehouse Entity Resolution Anti Money

Laundering

Banking

Basel II and Banking Data Warehouse

Entity Resolution

Health Care

Aligned Clinical Environment

Retail

RFID

Retail Data Model

Telco

Telco Data Warehouse

Insurance

Customer Insight

IIW

Automotive

Quality Insight Early Warning

Life Sciences

Drug Discovery


OmniFind Key Technologies

ContentContentCrawling Scalable Web crawler Data Source crawlers Content Push

Parsing/Tokenizing

HTML/XML 200+ Doc Filters Advance Linguistic

SearchCollections

Categorization Taxonomy Rule-based

Annotation Text Analytics Plug-in

Indexing Global Analysis Static Ranking Store

Dynamic Ranking Fielded Search Dynamic Summary Parametric Search Spell Checking

Searching

Security


Content Management Portfolio Strategy

Capture, store, and manage all forms of content

Complete and scalable, content management functionality

Document management

Image management

Digital asset management

Report management

Web content management

Records management

Digital rights management

Email/Messaging archiving and management

Collaboration tools

…

Enterprise-scale business process management

Cross-portfolio, out-of-the-box integration

Rich, common client platform


IBM Content Management Platform Roadmap

4Q20041Q2005

20052006

…and Beyond

WebSphere Portal V5.1Embeds DB2 Content Manager Runtime Edition (JCR)

Records Manager V4.1.1A Dynamic RM Infrastructure

Workplace Web Content Management V2.0 Leveraging DB2 Content Manager and WebSphere Portal Framework

DB2 Content Manager V8.3Enhance Doc RoutingEnable BPMExtend Integration CapabilitiesSeamless RM

DB2 Document Manager V8.3Compliance/RMExtending Native Language Support

DB2 CommonStore V8.3Full-Text SearchSeamless RM

First Step ECM Unified ClientNew PortletsJ2EE Web ComponentsExtend to DPMExtend Document ManagementEmail/Messaging Archiving and Management EnhancementsPhysical Records ManagementVirtual Records ManagementWCM Leveraging Workplace and DB2 Content Manager Runtime (JCR)

Common Content RepositoryWorkplace Unified End-User Experience (Client)Event FrameworkIntegrated / Interoperable DPM/BPMExtended ECM Capabilities as Add-On FeaturesEnterprise JCRIBM CM SDKEnterprise Content Integration – JSR170DB2 Content Manager Runtime in ISV ApplicationsLDDM* Fully Supports JSR170

Autonomic CapabilitiesContent PreservationContent IntelligencePervasive Enablement…and More

* Lotus Domino Document Manager


Query Optimization

Industry-Leading Optimization Extensible – SQL to XQuery! Optimizes for Parallel

I/O accesses Within a node (SMP) Between nodes (MPP)

Powerful for complex OLAP & BI queries Industry-Strength Engineering Portable

Across HW & SW platforms Databases of 1 GB to > 300 TB

Continuing "technology pump" of improvements from Research


Unstructured Information Management Architecture

Common Research infrastructure for advancing Text Analysis and NLP capability Promotes re-use of best-of-breed components Promotes combination hypothesis through ease of integration

Unstructured Information

Application Libraries

Specialized Application Libraries

Provide basic functions common to a broad class of application libraries & applications (e.g. Glossary Extraction Taxonomy Generation, Classification, Translation, etc.)

Question Answering

e-Commerce

Semantic Search EngineToken and Concept Indexing

Query Key words, concepts, spans, ranges -> Ranked Hit List

National & Intelligence Business

Bioinformatics

Technical Support

Document & Meta Data StoreDocuments with meta data based on key-value pairs

Enables view & collection management

(Text) Analysis Engine (TAEs)Combination of analysis engines employing a variety of analytical techniques and strategies

Structured Knowledge AccessKnowledge Source Adapters - (KSAs) deliver content from many structured knowledge sources according to central ontologies

Collection

Processing Manager

KSA Directory Service

Dynamic query & delivery of KSAs

TAE Directory Service

Dynamic query & delivery of TAEs

UIMA Standard Application Libraries

Relevant Application Knowledge

Structured Data

UIM

So

luti

on

s


Analytics bridge the Unstructured & Structured worlds

UnstructuredInformation

UnstructuredInformation UIMAUIMA

High-ValueMost Current ContentFastest GrowingBUT ...

Buried in Huge Volumes – Lots of NoiseImplicit SemanticsInefficient Search

Explicit StructureExplicit SemanticsEfficient SearchFocused Content

Text, Chat, Email, Audio,

Video

Text, Chat, Email, Audio,

Video

IndicesIndices

DBsDBs

KBsKBs

Identify Semantic Entities, Induce StructureChats, Phone Calls, Transfers People, Places, Org, Events Times, Topics, Opinions, RelationshipsThreats, Plots, etc.

Identify Semantic Entities, Induce StructureChats, Phone Calls, Transfers People, Places, Org, Events Times, Topics, Opinions, RelationshipsThreats, Plots, etc.

UIMA - The Big Picture

StructuredInformation


Evolution of Metadata

Hierarchical Data Model Rigid MetadataSingle Application

Domain Specific OntologiesFlexible MetadataCross Industry Integration

Increased Business Value of Metadata

Syntactic annotation of

data: what this data

represents

Semantic annotations of data: what this

data means

Relational Data ModelRigid MetadataIntegration Within Enterprise

Extensible Data Model (XML)Flexible MetadataIntegration Within Industry

1970 1990 2000 20101980


Data-driven analysis, reporting, monitoring, data rule & integration

specification

Data Analysts

Business context mapped to information

technology assets

Subject Matter Experts, Data

Stewards

Simplify integration

Metadata and data-driven data modeling

and management

Architects

Increase trust and confidence in information

Increase compliance to standards

Facilitate change management & reuse

Database application and transformation

development

ImplementersData

Administrators

Development Data Modeling Data Stewardship

Metadata Server

Integrated Metadata Enables Shared Understanding

Business Glossary

Data Architect Source System AnalysisInformation Analyzer

DataStage

QualityStage

Information Server


How Does Metadata Make Information Services Different?

Getcustomer

Getcustomer

OtherData Sources

ContentRepositories

?

WSDL WSDL

Information Services provide a basis for trust in information – providing visibility into lineage, relationships to other systems, and business definition

Traditional Service Information Service

• Where does the information come from?• What happens to it along the way?• How does this fit into how the business defines things?• How do I know I’m using the right service?


Metadata-driven Design for Integration

40% of IT budgets may be spent on integration

30% of people’s time is searching for relevant information

30% of development time is copy management

Remember ItRemember relationships and dependencies

Find ItFind and visualize related information

Connect ItGenerate the integration glue

WebService

Build These

Using These

New Business Process

New Integrated View

Legacy and packaged apps

Relational databases

XML documents

New DataFlow

WBI II ETL


Metadata Will Be Used to Facilitate Information and Application IntegrationToday – manual

integration, custom hard-wired integration

Tomorrow – semi-automated integration by using tools and connectors

Future – automated integration through metadata standards and tools

Dictionaries

Taxonomies

Ontologies

Download - Information Management Trends and Some History

Top Related