introduction to grid computing
DESCRIPTION
Deployment of a Language Detector Grid Service. Introduction to Grid Computing. Felix Hageloh Roberto Valenti. University of Amsterdam, 02-11-2005. Overview. Introduction Required Steps Our Service Introduction The basic idea Use Case Interface Implementation Problems Encountered - PowerPoint PPT PresentationTRANSCRIPT
Introduction to Grid Introduction to Grid ComputingComputing
Felix HagelohFelix HagelohRoberto Roberto ValentiValenti
Deployment of a Language Detector Deployment of a Language Detector Grid ServiceGrid Service
University of Amsterdam, 02-11-University of Amsterdam, 02-11-20052005
OverviewOverview IntroductionIntroduction Required StepsRequired Steps Our Service Our Service
IntroductionIntroduction The basic ideaThe basic idea Use CaseUse Case InterfaceInterface ImplementationImplementation
Problems EncounteredProblems Encountered Future WorkFuture Work ConclusionsConclusions QuestionsQuestions
IntroductionIntroduction
Our chosen task: Grid ServicesOur chosen task: Grid Services
Task Goals:Task Goals: Build a grid service. Build a grid service. Aggregate the service with another to Aggregate the service with another to
provide additional, higher-level provide additional, higher-level servicesservices
StepsSteps Get access to the systemsGet access to the systems
AuthenticationAuthentication Security issuesSecurity issues
Obtain User CertificateObtain User Certificate Obtain Host CertificateObtain Host Certificate
Implement the serviceImplement the service Create required filesCreate required files
WSDLWSDL QNamesQNames WSDDWSDD JNDIJNDI
Compile and create GAR fileCompile and create GAR file As Globus user:As Globus user:
Deploy service Deploy service Start containerStart container
But you all know this…But you all know this…
So… we jump to our service.So… we jump to our service.
Our ServiceOur Service
Our Service: IntroductionOur Service: Introduction
We were requested to implement a We were requested to implement a useful service which could be useful service which could be integrated on other servicesintegrated on other services
We are AI students so…We are AI students so…
Let’s Merge AI and Grid Computing!!Let’s Merge AI and Grid Computing!!
Our Service: The Basic IdeaOur Service: The Basic Idea
Idea: Language Detection Is a Idea: Language Detection Is a necessary first step in a multitude of necessary first step in a multitude of applicationsapplications
Useful Web Service Examples:Useful Web Service Examples: Email filteringEmail filtering Information retrievalInformation retrieval Spell checkersSpell checkers
Can also be component of an Can also be component of an aggregated grid serviceaggregated grid service
Our Service: The Basic IdeaOur Service: The Basic Idea
What about creating a Language What about creating a Language Detector on the Grid?Detector on the Grid? Training and Testing can be extremely Training and Testing can be extremely
time consuming running on a single time consuming running on a single machinemachine
Data difficult to obtain -> can be Data difficult to obtain -> can be shared on the Gridshared on the Grid Duplicate data for parallel computingDuplicate data for parallel computing
Our Service: Use CaseOur Service: Use Case
Simple Interface:Simple Interface: Receives a Receives a
piece of textpiece of text Returns a string Returns a string
indicating the indicating the languagelanguage
Our Service: Adding StatesOur Service: Adding States
Grid services can have states (as Grid services can have states (as opposed to web services)opposed to web services)
Not necessary for our service but Not necessary for our service but for the learning factorfor the learning factor
Added “dummy” states to our Added “dummy” states to our service:service: Last OperationLast Operation Times UsedTimes Used
Our Service: Statefull Use Our Service: Statefull Use CaseCase
Our Service: InterfaceOur Service: Interface
Requests and ResponsesRequests and Responses
<xsd:element name="detect" type="xsd:string" /> <xsd:element name="detectResponse“ type="xsd:string“ /> <xsd:element name="getLanguageRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getLanguageRPResponse" type="xsd:string" /> <xsd:element name="getLastOpRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getLastOpRPResponse" type="xsd:string" /> <xsd:element name="getTimesUsedRP"> <xsd:complexType /> </xsd:element> <xsd:element name="getTimesUsedRPResponse" type="xsd:int" />
Our Service: InterfaceOur Service: Interface
Port TypesPort Types<portType name="LanguageDetectorPortType" … > <operation name="detect"> <input message="tns:DetectInputMessage" /> <output message="tns:DetectOutputMessage" /> </operation> <operation name="getLanguageRP"> <input message="tns:GetLanguageRPInputMessage" /> <output message="tns:GetLanguageRPOutputMessage" /> </operation> <operation name="getLastOpRP"> <input message="tns:GetLastOpRPInputMessage" /> <output message="tns:GetLastOpRPOutputMessage" /> </operation> <operation name="getTimesUsedRP"> <input message="tns:GetTimesUsedRPInputMessage" /> <output message="tns:GetTimesUsedRPOutputMessage" /> </operation></portType>
Our ServiceOur ServiceImplementationImplementation
Language Detection: Basic Language Detection: Basic IdeaIdea
Essentially based on probabilities of Essentially based on probabilities of character combinationscharacter combinations
Every language has typical character Every language has typical character combinations that are very frequent in combinations that are very frequent in that languagethat language ““th” in englishth” in english ““ij” in dutchij” in dutch
Easy for humans to detect a language Easy for humans to detect a language even when we don’t know that specific even when we don’t know that specific languagelanguage
Language Learning: Standard Language Learning: Standard ProcessProcess
Standard machine learning processStandard machine learning process
Language Learning: Markov Language Learning: Markov ModelsModels
Basic Markov ModelBasic Markov Model
kkthth order Markov Model order Markov Model
n
iiinssssssS
21121
|PP,...,,PP
n
iikiiknssssssssS
21121
,...,|P,...,P,...,,PP
Language Detection: Language Detection: ClassificationClassification
Transitional probabilities estimated Transitional probabilities estimated asas
ClassificationClassification
21
21
21 ,#
,,#,|P
is
iii
iii ss
ssssss
)||P(max|Pmax2,1
3cii
n
ii
cCC
LsssLSC
Language Detection: Language Detection: ExampleExample
The training text for a The training text for a language consists of language consists of the stringthe string
Learned model:Learned model:
test text
( ^^, t, 1.0 )( ^t, e, 1.0 )( te, s, 0.5 )( es, t, 1.0 )( st, , 1.0 )( te, x, 0.5 )( ex, t, 1.0 )( xt, , 1.0 )
the probability of the the probability of the stringstring
would be:would be:
test
P(test|L) = P(t|^^)*P(e|^t)*P(s|te) *P(t|es)*P(_|st) = 1*1*0.5*1*1=0.5
Language Detection: Language Detection: PerformancePerformance
60%
70%
80%
90%
100%
number of words
% c
orr
ec
tly
cla
ss
ifie
d
German 78,91 92,91 96,84 98,15 98,81
English 73,89 90,55 95,07 97,18 98,10
Dutch 68,32 88,66 94,70 97,12 98,09
one two three four five
Problems EncounteredProblems Encountered
Necessary tools had to be installed Necessary tools had to be installed (ANT)(ANT)
Problems on our machine (GRAM)Problems on our machine (GRAM) Conflicts with other teamConflicts with other team Buggy shell script to build gar fileBuggy shell script to build gar file
Sensitive to path lengths/ namesSensitive to path lengths/ names
Future WorkFuture Work
Connect with other servicesConnect with other services Make training and evaluation a grid Make training and evaluation a grid
serviceservice Make it part of a multi lingual retrieval Make it part of a multi lingual retrieval
engineengine
Web interface (interactive)Web interface (interactive)
ConclusionsConclusions
Successfully managed to create Successfully managed to create and deploy our own web serviceand deploy our own web service Broke loose from the tutorial web Broke loose from the tutorial web
service structureservice structure Merged Grid Computing with AIMerged Grid Computing with AI
Got hands on experience with Grid Got hands on experience with Grid applications and structureapplications and structure
A lot of possibilities to integrate A lot of possibilities to integrate and/or extend the implemented and/or extend the implemented serviceservice
QuestionsQuestions
??