project omniglean team members: kenny trytek derek woods abby birkett joe briggie advisor: simanta...
TRANSCRIPT
PROJECTOMNIGLEANTeam Members:Kenny TrytekDerek WoodsAbby BirkettJoe Briggie
Advisor:Simanta Mitra
Client:Kingland Systems
PROBLEM STATEMENT
• Large companies have many layers of corporate hierarchy.
• Financial and data records sometimes conflict between
various layers/entities.
• Accurate and comprehensive company records are needed.
• There is a need for “Data Mastering”, to take multiple
conflicting sources of data and
• determine the reality of the matter in conflict.
CONCEPT SKETCH
Client User
Analyst User
OmnigleanInternet
Harvest & store
Figure 2.1.1: Concept Sketch
FUNCTIONAL REQUIREMENTS
• System shall autonomously traverse publicly available websites
• System shall parse information from downloaded file in
portable document format (pdf)
• System shall store parsed information in a flat file
• System shall allow user the ability to create, modify, and delete
records
• System shall maintain a normalized database
• System shall expose functionality through web services
NON-FUNCTIONAL REQUIREMENTS
• System shall support up to 250 concurrent users
• A single run of system shall complete execution in less
than six hours
• System shall be easily extensible to include more
websites than originally specified
• System shall be completed by May of 2011
CONSTRAINTS AND TECHNICAL CONSIDERATIONS
• System shall interact with a third party library to facilitate
database interaction
• The database may not be available at all times
• Using SVN to manage code
• Using the spiral design process
MARKET SURVEY
• Omniglean provides a unique combination of access to freely available FDIC and FFIEC data through a data mastering suite.
• Omniglean provides access to the mastering capability through web services, to enable a rapid delivery of functionality to customers of Kingland Systems, as well as analysts located in different geographical areas.
POTENTIAL RISKS AND MITIGATION
• External format change
• External availability
• New technologies
• Web services
• Not enough time for testing and debugging
• Team members
COST AND RESOURCE ESTIMATE
Item Cost
Reporting
Poster Materials $50.00
Report Materials $50.00
Labor @ $20.00/hr
Kenny Trytek $3,560
Abby Birkett $3,220
Joe Briggie $3,160
Derek Woods $3,360
Total $13,100
PROJECT MILESTONES
• Complete modules related to harvesting and
transforming the data.
• Complete web services and user interface modules
• Integrate all modules successfully
FUNCTIONAL DECOMPOSITION
• Harvester – Gathers data
• ETL – Transforms data
• DAL – Database access layer
• Web services – Exposes data to external users
• User interfaces
SYSTEM DIAGRAM
Flat FileDatabase
ETL Tool
Normalized
Kingland Data Analyst UI
DAL
No Conflicts?
External Client UI
Web Svcs.
WWW Data Scraper Tool
HTML ParserPDF Parser
CreateRead
UpdateDelete
HARVESTERScraper
Flat File
(XML)
World Wide Web
Parser
PDF Parser
HTML ParserstartGatheringData() - Returns the XML
document populated with data from the site.
getLogFile() - Returns the log file that is either being written to or has been written to this session.
stopGatheringData() - Stops all current harvester operations and writes an error to the log file.
ETL TOOL ETL Tool DALFlat File(XML)
• loadFFIEC() - This method will load the data from the XML file into the FFIEC table.
• loadFDIC() - This method will load the data from the XML file into the FFDIC table.
• createORGANIZATION() - This method will take the information from both the FFIEC table and the FDIC table and put it into the ORGANIZATION table.
DATA ACCESS LAYER
DAL
Database
User InterfaceETLTool
Add()Find()Update()Delete()
Organization- A class that creates and maintains a connection to the ORGANIZATION table
GetConnection()CloseConnection()
OrganizationService- This class will allow CRUD functionality with an Organization object.
Find(String organizationId)Add(...)Delete(String organizationId)Update(String organizationId, ...)
WEBSERVICES
Unauthenticated
Authenticated
Read()
LogOut()
Write()
Update()
Delete()
Read()
LogIn()
• Create()• Read() • Update()• Delete()• login()• logout()
Allows remote users to access the database throughthe Internet
TECHNOLOGY PLATFORM
• SQL Server 2008
• Visual Studio 2010 development environment
• Microsoft Windows operating system
• WSDL and SOAP for web services
TEST PLAN
The team will be testing the system in three phases.
The first phase is testing the individual modules, the
second phase will be testing the integration of the
modules, and the final phase will be testing the
system as a whole.
PROTOTYPINGWe have begun prototyping. The harvester is able to
traverse the necessary websites easily. The ETL can read in
XML files. The user interface has been mocked up.
CURRENT PROJECT STATUS
Activity Start Date
End Date
Sept 13-19
Sept 20-26
Sept 27-Oct 3
Oct 4-10
Oct 11-17
Oct 18-24
Oct 25-31
Nov 1-7
Nov 8-14
Nov 15-21
Nov 22-28
Nov 29-Dec 5
Dec 6-12
Dec 13-19
Dec 20-26
Dec 27-Jan 2
Jan 3-9
Jan 10-16
Jan 17-23
Jan 24-30
Jan 31- Feb 6
Feb 7-13
Feb 14-20
Feb 21 -27
Feb 28 – Mar 6
Mar 7-13
Mar 14-20
Mar 21-27
Mar 28 – Apr 3
Apr 4-10
Apr 11-17
Apr 18-24
Apr 25 – May 1
Project Plan Presentaion
09/21/10 09/27/10
Project Plan Rough Draft
09/21/10 10/05/10
Project Plan Final Draft
10/05/10 10/12/10
Prototyping 10/05/10 12/10/10
Design Document Rough Draft
10/05/10 11/15/10
Design Document Final Draft
11/15/10 12/03/10
Testing Phase 01/31/11 04/04/11
System Completion
12/10/10 04/04/11
End Product Documentation
03/21/11 04/04/11
Project Poster 03/14/11 03/28/11
Project Presentation
04/25/11 04/25/11
Buffer 04/11/11 4/24/11
TASK RESPONSIBILITY
Kenny Trytek – Team Leader, responsible for the
harvester
Derek Woods – Developer, responsible for the ETL and
UI
Abby Birkett – Developer, responsible for database
and DAL
Joe Briggie – Developer, responsible for web services
PLAN FOR NEXT SEMESTER
• Continue prototyping
• Continue to meet with client to be sure we are
meeting expectations
• Develop a more thorough test plan