remote procedure calling

Remote Procedure Calling

Dr. Andrew C.R. [email protected]://www.bioinf.org.uk/

Aims and objectivesUnderstand the concepts of

remote procedure calling and web services

To be able to describe different methods of remote procedure calls

Understand the problems of ‘screen scraping’

Know how to write code using LWP and SOAP

What is RPC?

A network accessible interface to applicationfunctionality using standard Internet technologies

NetworkRPC

WebService

Why do RPC?distribute the load between

computers

access to other people's methods

access to the latest data

Ways of performing RPCscreen scraping

simple cgi scripts (REST)

custom code to work across networks

standardized methods (e.g. CORBA, SOAP, XML-RPC)

Web servicesRPC methods which work across

the internet are often called “Web Services”

Web Services can alsobe self-describing (WSDL)provide methods for discovery (UDDI)

Screen scraping

Web service

NetworkWeb

service

Screen scraping

NetworkScreenscraper

Webserver

Extracting content from a web page

Fragile procedure...

DataWeb page

Extractdata

Data

Partial dataSemantics lost

Errors indata extraction

Extractdata

Provider

Consumer Visual markupPartial (error-prone) data

Fragile procedure...Trying to interpret semantics

from display-based markup

If the presentation changes,the screen scraper breaks

CGIScript

Web servers…

Send request for page to web server

Pages

ExternalPrograms

RDBMS

Web browserWeb server

Screen scraping

Straightforward in Perl

Perl LWP moduleeasy to write a web client

Pattern matching and string handling routines

Example scraper…A program for

secondary structure prediction

Want a program that: specifies an amino acid sequence

provides a secondary structure prediction

Example scraper...

#!/usr/bin/perl -wuse LWP::UserAgent;use strict;

my($seq, $ss);

$seq = "KVFGRCELAAAMKRHGLDNYRGYSLGNWVCAAKFESNFNTQATNRNTDGSTDYGILQINSRWWCNDGRTPGSRNLCNIPCSALLSSDITASVNCAKKIVSDGNGMNAWVAWRNRCKGTDVQAWIRGCRL";

if(($ss = PredictSS($seq)) ne ""){ print "$seq\n"; print "$ss\n";}

Example scraper…NNPREDICT web server at

http://alexander.compbio.ucsf.edu/~nomi/nnpredict.html

http://alexander.compbio.ucsf.edu/cgi-bin/nnpredict.pl

Example scraper…Program must:

connect to web server

submit the sequence

obtain the results and extract data

Examine the source for the page…

<form method="POST”action="http://alexander.compbio.ucsf.edu/cgi-bin/nnpredict.pl">

<b>Tertiary structure class:</b>

<input TYPE="radio" NAME="option" VALUE="none" CHECKED> none <input TYPE="radio" NAME="option" VALUE="all-alpha"> all-alpha <input TYPE="radio" NAME="option" VALUE="all-beta"> all-beta <input TYPE="radio" NAME="option" VALUE="alpha/beta"> alpha/beta

<b>Name of sequence</b> <input name="name" size="70">

<b>Sequence</b><textarea name="text" rows=14 cols=70></textarea>

</form>

Example scraper…option

'none', 'all-alpha', 'all-beta', or 'alpha/beta’

name

optional name for the sequence

text

the sequence

Example server...sub PredictSS{ my($seq) = @_; my($url, $post, $webproxy, $ua, $req, $result, $ss);

# $webproxy = 'http://user:[email protected]:8080'; $webproxy = "";

$url = "http://alexander.compbio.ucsf.edu/cgi-bin/nnpredict.pl";

$post = "option=none&name=&text=$seq";

$ua = CreateUserAgent($webproxy); $req = CreatePostRequest($url, $post); $result = GetContent($ua, $req); if(defined($result)) { $ss = GetSS($result); return($ss); } else { print STDERR "connection failed\n"; }

return("");}

If behinda firewall

CGI scriptto access

Values passedto CGI script

Create a LWP-based connection; Create the post request;

Connect and get the returned page

<HTML><HEAD>

<TITLE>NNPREDICT RESULTS</TITLE>

</HEAD>

<BODY bgcolor="F0F0F0">

<h1 align=center>Results of nnpredict query</h1>

<p><b>Tertiary structure class:</b> alpha/beta

<p><b>Sequence</b>:<br>

<tt>

MRSLLILVLCFLPLAALGKVFGRCELAAAMKRHGLDNYRGYSLGNWVCAAKFESNFNTQA<br>

TNRNTDGSTDYGILQINSRWWCNDGRTPGSRNLCNIPCSALLSSDITASVNCAKKIVSDG<br>

NGMNAWVAWRNRCKGTDVQAWIRGCRL<br>

</tt>

<p><b>Secondary structure prediction

<i>(H = helix, E = strand, - = no prediction)</i>:<br></b>

<tt>

----EEEEEEE-H---H--EE-HHHHHHHHHH--------------HHHHHH--------<br>

------------HHHHE-------------------------------HH-----EE---<br>

---HHHHHHH--------HHHHHHH--<br>

</tt>

</body></html>

Example server…sub GetSS{ my($html) = @_; my($ss);

$html =~ s/\n//g;

$html =~ /^.*<tt>(.*)<\/tt>.*$/;

$ss = $1;

$ss =~ s/\<br\>//g;

return($ss);}

Remove returncharactersMatch the last

<tt>...</tt>Grab the text

within <tt> tagsRemove the<br> tags

If authors changed presentation of results, this might break!

Wrappers to LWPCreateUserAgent()

CreatePostRequest()

GetContent()

CreateGetRequest()

Pros and cons

Advantages 'service provider' doesn’t do anything

special

Disadvantagesscreen scraper will break if format

changes

may be difficult to determine semantic content

Simple CGI scripts

REST:Representational State Transfer

http://en.wikipedia.org/wiki/REST

Simple CGI scripts

Extension of screen scrapingrelies on service provider to provide a

script designed specifically for remote access

Client identical to screen scraperbut guaranteed that the data will be

parsable (plain text or XML)

Simple CGI scripts

Server's point of viewprovide a modified CGI script which

returns plain text

May be an option given to the CGI script

Simple CGI scripts'Entrez programming utilities'

http://eutils.ncbi.nlm.nih.gov/entrez/query/static/eutils_help.html

I have provided a script you can try to extract papers from PubMed

Simple CGI scriptsSearch using EUtils is performed in

2 stages:

specified search string returns a set of PubMed Ids

fetch the results for each of these PubMed IDs in turn.

Custom code

Custom codeGenerally used to distribute tasks

on a local network

Code is complexlow-level OS calls

sample on the web

Custom codeLink relies on IP address and a

'port’

Ports tied to a particular serviceport 80 : HTTP

port 22 : ssh

See /etc/services

Custom codeGenerally a client/server model:

server listens for messages

client makes requests of the server

Client Serverrequest message

response message

Custom code: serverServer creates a 'socket' and

'binds' it to a port

Listens for a connection from clients

Responds to messages received

Replies with information sent back to the client

Custom code: clientClient creates a socket

Binds it to a port and the IP address of the server

Sends data to the server

Waits for a response

Does something with the returned data

StandardizedMethods

Standardized methodsVarious methods. e.g.

CORBA

XML-RPC

SOAP

Will now concentrate on SOAP...

Advantages of SOAP

Applicationclient

Webservice

Applicationcode

Platform andlanguage

specific code

Platform andlanguage

independentcode

Advantages of SOAP

Application XML message Application

Information encoded in XML Language independentAll data are transmitted as simple

text

Advantages of SOAPHTTP post SOAP request

HTTP response SOAP response

Normally uses HTTP for transport Firewalls allow access to the HTTP

protocolSame systems will allow SOAP access

Advantages of SOAP

W3C standard

Libraries available for many programming languages

XML encoding

Which of these is correct?

<phoneNumber>01234 567890</phoneNumber>

<phoneNumber> <areaCode>01234</areaCode> <number>567890</number></phoneNumber>

<phoneNumber areaCode='01234' number='567890' />

<phoneNumber areaCode='01234'>567890</phoneNumber>

SOAP XML encoding

Must define a standard way of encoding data:

Type of data being exchangedHow it will be expressed in XMLHow the information will be exchanged

Defined byvarious transport

protocols

How the information will be exchanged

Defined bySOAP message

format

Type of data being exchangedHow it will be expressed in XML

SOAP messages

SOAP Envelope

SOAP Header (optional)

SOAP Body

Header block

Header block

Message body

SOAP Envelope

SOAP HeaderHeader block

SOAP BodyMessage body

<s:Envelope xmlns:s=”http://www.w3.org/2001/06/soap-envelope”>

<s:Header>

<m:transaction xmlns:m=”soap-transaction”

s:mustUnderstand=”true”>

<transactionID>1234</transactionID>

</m:transaction>

</s:Header>

<s:Body>

<n:predictSS xmlns:n=”urn:SequenceData”>

<sequence id='P01234'>

SARTASCWIPLKNMNTYTRSFGHSGHRPLKMNSGDGAAREST

</sequence>

</n:predictSS>

</s:Body>

</s:Envelope>

Example SOAP message

Header blockSpecifies data must be handled as a

single 'transaction’

Message bodycontains a sequence simply encoded

in XMLPerfectly legal, but more common to

use special RPC encoding

The RPC ideal

Ideal situation:

$ss = PredictSS($id, $sequence);

Client Serverrequest message

response message

Subroutine calls

Only important factorsthe type of the variables

the order in which they are handed to the subroutine

SOAP type encoding

SOAP provides standard encoding for variable types:integersfloatsstringsarrays hashesstructures…

<s:Envelope

xmlns:s=”http://www.w3.org/2001/06/soap-envelope”>

<s:Body>

<n:predictSS xmlns:n=”urn:SequenceData”>

<id xsi:type='xsd:string'>

P01234

</id>

<sequence xsi:type='xsd:string'>

SARTASCWIPLKNMNTYTRSFGHSGHRPLKMNSGDGAAREST

</sequence>

</n:predictSS>

</s:Body>

</s:Envelope>

Encoded SOAP message

Response

<s:Envelope

xmlns:s=”http://www.w3.org/2001/06/soap-envelope”>

<s:Body>

<n:predictSSResponse

xmlns:n=”urn:SequenceData”>

<ss xsi:type='xsd:string'>

---HHHHHHH-----EEEEEEEE----EEEEEEE--------

</ss>

</n:predictSSResponse>

</s:Body>

</s:Envelope>

SOAP transport

SOAP is a packaging protocol

Layered on networking and transport

SOAP doesn't care what these are

SOAP transport

Generally uses HTTP, but may also use:FTP, raw TCP, SMTP, POP3, Jabber, etc.

HTTP is pervasive across the Internet.

Request-response model of RPC matches HTTP

Web service components

Applicationspecific

code

ServiceProxy

ServiceListener

Web application server

SOAP::LiteYou need to know very little of

this!

Simply need a good SOAP librarySOAP::Lite for Perl

Apache SOAP

Toolkits available for many languagesJava, C#, C++, C, PHP, Python, …

A simple SOAP server

Applicationspecific

code

ServiceProxy

ServiceListener

Web application server

HTTPDSimple

SOAP-specificCGI script

Applicationcode

use SOAP::Transport::HTTP;

SOAP::Transport::HTTP::CGI

->dispatch_to('/home/httpd/cgi-bin/SOAPTEST')

->handle;

SOAP-specific CGI script

Directory is where application-specific

code is stored

Any Perl module stored there will be accessible via SOAP

(Can limit to individual modules and routines)

Application code

Lives in a Perl module:Filename with extension .pm

Starts with a package statement:

package mymodule;

Filename must match package name

mymodule.pm

Must return 1. Generally end file with

1;

Application code

package hello;

sub sayHello {

my($class, $user) = @_;

return "Hello $user from the SOAP server";

}

1;

Simply place this file (hello.pm) in the directory specified in the SOAP proxy

A simple SOAP client#!/usr/bin/perl

use SOAP::Lite;

my $name = shift;

print "\nCalling the SOAP server...\n";

print "The SOAP server says:\n";

$s = SOAP::Lite

->uri('urn:hello')

->proxy('http://localhost/cgi-bin/SOAPTEST.pl');

print $s->sayHello($name)->result;

print "\n\n";

Calling the SOAP server...

The SOAP server says:

Hello Andrew from the SOAP server

SOAP::Lite

The code is very simple!None of the hard work to package or

unpack the request in XML

All the hard work is hidden…

Query<s:Envelope

xmlns:s=”http://schemas.xmlsoap.org/soap/envelope”

xmlns:xsi=”http://www.w3.org/1999/XMLSchema-instance”

xmlns:xsd=”http://www.w3.org/1999/XMLSchema”>

<s:Body>

<m:sayHello xmlns:m=”urn:hello”>

<name xsi:type='xsd:string'>Andrew</name>

</m:sayHello>

</s:Body>

</s:Envelope>

Response<s:Envelope

xmlns:s=”http://schemas.xmlsoap.org/soap/envelope”

xmlns:xsi=”http://www.w3.org/1999/XMLSchema-instance”

xmlns:xsd=”http://www.w3.org/1999/XMLSchema”>

<s:Body>

<n:sayHelloResponse xmlns:n=”urn:hello”>

<return xsi:type='xsd:string'>Hello Andrew from

the SOAP server</return>

</n:sayHelloResponse>

</s:Body>

</s:Envelope>

An even simpler SOAP client

#!/usr/bin/perl

use SOAP::Lite +autodispatch=>

uri=>"urn:hello",

proxy=>"http://localhost/cgi-bin/SOAPTEST.pl";

my $name = shift;

print "\nCalling the SOAP server...\n";

print "The SOAP server says:\n";

print sayHello($name);

print "\n\n";

Summary - RPCRPC allows access to methods and

data on remote computers

Four main ways of achieving thisScreen scrapingSpecial CGI scriptsCustom codeStandardized methods (SOAP, etc.)

Summary - SOAPPlatform and language independentUses XML to wrap RPC data and

requestsVarious transport methods

generally use HTTPGood toolkits make coding VERY easy

all complexity hiddenRelated technologies allow

service discovery (UDDI)self-describing services (WSDL)

remote procedure calling

Documents