developing applications that harness the agricultural and natural resource grid (and linking your...
TRANSCRIPT
![Page 1: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/1.jpg)
Developing applications that harness the Agricultural and Natural Resource Grid
(and linking your own data)
Matthew Laurenson and Seishi NinomiyaNational Agricultural Research Center
TsukubaJapan
![Page 2: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/2.jpg)
Contents•Concepts
–Remote Procedure Calls (RPC)
–Brokers
•Using a broker–Metadata requests
–Data requests
–Issues
–Linking your own data
•SOAP•New brokers and future enhancements
![Page 3: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/3.jpg)
Overall Goal: To help you make “portable” software applications (DSS).
![Page 4: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/4.jpg)
Current Situation: Web-based but…
• Applications closely coupled to databases
• Limited range of applications at each site
• Duplicated development at each site
![Page 5: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/5.jpg)
Soils
Ag and NR Grid – a “Hotel Room” for Agricultural and Natural Resource DSS
Room
Weather Elevation
Maps
Connectors for local data
DSSDSS DSSDSS
User interface in national language
![Page 6: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/6.jpg)
New ZealandNew Zealand
Soils
Weather Elevation
Maps
![Page 7: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/7.jpg)
地
気象 地面の高さ
地図
日本
![Page 8: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/8.jpg)
中国中国
土壤
气象 海拔
交通图
![Page 9: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/9.jpg)
Goal of Ag and NR Grid:To provide this “hotel room” in your
country, so you can use models developed by others (and they can use your models).
![Page 10: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/10.jpg)
Distributed Computing Concepts
• Remote Procedure Calls
• Brokers to handle heterogeneous databases
![Page 11: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/11.jpg)
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on a remote computer as if it were local.
![Page 12: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/12.jpg)
Remote Procedure Call
My Computer
remote=findRemote()
A=5
B=6
C=remote.add(A,B)
Print C
...
Remote Computer
procedure add(x,y)
return x+y
}
...
![Page 13: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/13.jpg)
Remote Procedure Calls (RPC)
• RPCs let a program execute a procedure on a remote computer as if it were local.
• Newer RPC mechanisms let programs send and receive data objects.
• RPC mechanisms: CORBA, .NET, Java RMI, SOAP...
• RMI – only Java programs can use• SOAP – used by other languages as well
![Page 14: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/14.jpg)
RPC vs Local Calls
• Latency (delay)– RPCs much slower than local calls– difficult to predict how long call will take
• May fail– network problems– problems with other computer (“server”)
• Security issues– firewalls...
![Page 15: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/15.jpg)
But...
• Can distribute processing and data over multiple machines
• Great when multiple users must share the same dynamic data item or communicate with each other.
![Page 16: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/16.jpg)
Mediated Architecture – “Brokers”
Broker
DomainDatabases
(Heterogeneous) Applications
Consistent data access(using RPC mechanism)
![Page 17: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/17.jpg)
MetBroker
MetBroker
Korea
Japan
New Zealand
WeatherDatabases
Data Retrieval
Applications
South Africa
USA
Data RetrievalServlet
Weather Risk
Crop Growth
Disease Risk
Web Browsers
![Page 18: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/18.jpg)
Contents•Concepts
–Remote Procedure Calls (RPC)
–Brokers
•Using a broker–Metadata requests
–Data requests
–Issues
–Linking your own data
•SOAP•New brokers and future enhancements
![Page 19: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/19.jpg)
Data and Metadata Requests
Metadatarequest
Metadataresult
Broker Database(various)
Application
Standardrequest
DB-specificquery
DB-specificresult
Standardresult
Metadata Data
Interaction between applications and brokers uses objects – data packages
![Page 20: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/20.jpg)
Example - MetBroker Services
• Weather Data– Single station– All stations in a region
• Metadata (data about data)– Database details (coverage of globe, login req.)– Regions within databases– Station details (latitude, longitude, start date,
end date, elements recorded, resolution...)
![Page 21: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/21.jpg)
Elements of a Station Data Request
Weather station
Required weather elements
Time interval
Username/password
Resolution
Allow summarizing
![Page 22: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/22.jpg)
Send Request to MetBroker and Receive Result
Data ResultTsuchiura
Operated 1976-present1999/1/3 06:00-1999/2/1 00:00
10.1, -0.2, 012.6, 3.8, 0
11.3, -1.5, 0.3...
To MetBroker
Data Requestamedas40341
rain, air temperature1999/1/1-1999/2/1
dailyOK to summarize
![Page 23: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/23.jpg)
Query Results - Single Station
Station& query details
rain air temp. solar rad.
Station details anddata sequencesare encapsulatedin a singleresult object
![Page 24: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/24.jpg)
Broker
Temperature
Rainfall
Radiation
Client
Spatial Request
![Page 25: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/25.jpg)
WakayamaRainfall DB
AMeDASDaily max/min temperatures & rainfall for all stations in region
Spatial Request Background map dynamically retrieved from http://www.mapblast.com
![Page 26: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/26.jpg)
Query results - multiple stations
![Page 27: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/27.jpg)
Using Brokers – Implementation Issues
![Page 28: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/28.jpg)
RMI Blocked by Firewalls
ClientApplication
Java Broker
Server MachineClientApplication
Java
Wrapper Servlet
HTTP
RMI
RMI
Client-side firewalls
![Page 29: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/29.jpg)
Applet Using Multiple Brokers
Spatial Risk Applet
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text
![Page 30: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/30.jpg)
Without Multiple Threads - Sequential
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text 12 seconds
Spatial Risk Applet
![Page 31: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/31.jpg)
With Multiple Threads – Parallel
MetBroker
DEMBroker
ChizuBroker
Background maps
Elevation data
Meteorological data
Resource Server
Application text 5 seconds
Spatial Risk Applet
![Page 32: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/32.jpg)
JavaBeans for Java GUI programs
• For rapid GUI development in Java can use existing MetBeans
• Transparently handle firewalls, latency...
• Making a simple data retrieval application takes about 30 minutes
• Used in many MetBroker demo applications
![Page 33: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/33.jpg)
Easy Development of New Applications
![Page 34: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/34.jpg)
Databases Linked Using “Drivers”
DatabasesApplications
Broker
![Page 35: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/35.jpg)
Linking a Database to a Broker
• Need to write one Java class (a “driver”) with two or three main methods
• Typically use Java Database Connectivity (JDBC) drivers to query relational databases
• Can reuse existing classes for Oracle, Postgres, and MySQL JDBC.
• Drivers can read data from files via HTTP, or parse HTML tables.
• “Test harness” available for driver development
![Page 36: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/36.jpg)
How much work?
• Add a new database to MetBroker– roughly working - 3 days– tested more thoroughly – 1 week– (assuming that database is readily accessible)
• Make a MetBroker applet– working prototype in one day– (assuming basic science is “done”)
![Page 37: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/37.jpg)
Developer Resources
• Demos, documentation, downloads, pictures– http://www.agmodel.net
• Mailing lists – subscribe at web site
• Brokers are open source under the GNU Lesser General Public License.– Can be used in commercial applications– Any improvements must be released under same
license
![Page 38: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/38.jpg)
• Linking your data to the grid lets you use grid tools developed by others.
• It doesn’t mean you must share your data with everyone (password protection).
• The key to interoperability is the interface
Reminders
![Page 39: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/39.jpg)
Contents•Concepts
–Remote Procedure Calls (RPC)
–Brokers
•Using a broker–Metadata requests
–Data requests
–Issues
–Linking your own data
•SOAP•New brokers and future enhancements
![Page 40: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/40.jpg)
SOAP
• An XML-based RPC mechanism
• Independent of programming language
• SOAP toolkits for C++, Visual Basic, Delphi, Java, ASP, PHP, Perl, Python…
• Firewall-friendly
• Basis of Microsoft .NET architecture
• A SOAP Web service is described in standard format using a .WSDL file(Web Services Description Language)
![Page 41: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/41.jpg)
Visual Basic client application
Applicationcode
MS SOAPToolkit
VB function calls
VB objects
MetSOAP
SOAP over HTTP
MetBroker
Delphi client application
Applicationcode
Borland SOAPToolkit
Delphi function calls
Delphi objects
SOAP over HTTP
RMI
Example - MetSOAP
![Page 42: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/42.jpg)
SOAP Summary
• Positives:– Opens brokers to a range of programming languages
– Already the standard for Web-based RPC
– Avoids most firewall issues
• Negatives– Can’t reuse Java classes - must “mirror” in each language
– Interoperability is still imperfect (but improving fast)
– Inefficient for large data sets
![Page 43: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/43.jpg)
Typical Array in SOAP
• ... • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">23.43</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">21.79</item> • <item xsi:type="xsd:float">23.3</item> • <item xsi:type="xsd:float">21.11</item> • <item xsi:type="xsd:float">19.49</item> • <item xsi:type="xsd:float">18.48</item> • <item xsi:type="xsd:float">16.52</item> • <item xsi:type="xsd:float">14.73</item> • <item xsi:type="xsd:float">14.73</item> • ...
![Page 44: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/44.jpg)
Model Deployment Options - Summary
• Java applet– flexible, responsive once loaded but requires plug-in
• Java servlet– simple browsers, rapid load, but clunky interface
• SOAP– direct access for Delphi, VB…
• Existing models (FORTAN,C,BASIC)– Use simple Java app. to retrieve data and write to a file
![Page 45: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/45.jpg)
Contents
•Concepts–Brokers–Remote Procedure Calls (RPC)
•Using a broker–Metadata requests–Data requests–Issues–Linking your own data
•SOAP•New brokers and future enhancements
![Page 46: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/46.jpg)
New Brokers
• DEM Broker - Digital Elevation Model data– Japan 50m resolution– World 1km resolution
• ResourceServer– Internet-based service for software localization
• ChizuBroker – Web map services– www.mapfan.com (Japan)– www.mapzone.co.nz (New Zealand)
• SoilBroker – soils data
![Page 47: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/47.jpg)
Future enhancements
• Broker-side interpolation (using DEMBroker)– temporal– spatial
![Page 48: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/48.jpg)
Server
Temperature
Rainfall
Radiation
Client
“Virtual station”Broker-side interpolation at single site
![Page 49: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/49.jpg)
Future enhancements
• Broker-side interpolation (using DEMBroker)– temporal– spatial
• Other brokers– Place names– Crops and crop models (prototyped)– Agrichemicals
![Page 50: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/50.jpg)
Final Points
• The Grid provides a stable, powerful foundation for Ag and NR software
• You can now develop grid applications in your favourite programming language.
• NARO is keen to help you to both develop new applications and link new databases.
• Database owners around the world will be keen to evaluate your applications.
![Page 51: Developing applications that harness the Agricultural and Natural Resource Grid (and linking your own data) Matthew Laurenson and Seishi Ninomiya National](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e575503460f94b4f04a/html5/thumbnails/51.jpg)
Thank you