food and agriculture organization of the united nations software tools what we built and building...

FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS

Software toolswhat we built

andbuilding community.

Vilnius, LithuaniaPC-Axis Conference, September 2009

Yves Jaques ESSS, FAO of the UN


2 September 2009PC-Axis Conference, Vilnius Lithuania

Outline

• what we built– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX

• building community– Best practices– Tools– Administration



Outline





PX-Web customization - intro

• Over the last few years PX-Web code has been modified to fit CountrySTAT needs.

• The core query engine and exporting / graphing / mapping components have remained as is while,

• Many other features are not used.

what we built



PX-Web customization - Rationale• CountrySTAT represents a unique use-case of PX-

Web– Almost twenty countries running under one server in a

shared environment– A flexible, configurable CMS was needed.

Solution• Build a site around PX-Web query engine,

– borrow what was useful from PX-Web,– make use of other available JavaScript and ASP libraries,– write bespoke code.

what we built



PX-Web customization - Differences• Houses as many sites as desired in one instance of

PX-Web• An easy to use admin area• Rich text editing of pages• Menu tree a separate process not using PX-Web

with some new visual features• A flexible RSS feature (whole site, or any folder)• Custom icons in the query area• Support for confidentiality (public, internal private)• Support for a validity period

what we built



PX-Web customization - Issues

• Perhaps not compatible with newer versions of PX-Web?

• Not a community project, so who will maintain over the long-term?

• How to handle and prioritize new feature requests?

what we built



PX-Web customization - Conclusions

• Should investigate if an off-the-shelf CMS could be used to house PX-Web.– Think of PX-Web more as a plug-in, e.g. within Umbraco.– Benefit from a huge community of users.– Other countries seem to be doing so (e.g. Switzerland)

• Find it at: http://www.countrystat.org

what we built



PX-Publisher

• What is it?– A web-based .px file uploader.– A data tree manager.– A .px file publisher.

• Implementation?– VB.NET to conform to the upcoming PX-Web.NET.

what we built



PX-Publisher - Rationale

• Again, CountrySTAT represents a unique use-case of PX-Web– Almost twenty countries running under one server in a

shared environment– Within countries, data uploaded by numerous sibling

institutions

• PX-Web not designed to upload PX files– FTP– Logging– Roles– Versioning & Synchronization

what we built



PX-Publisher - Solution

• A new component: PX-Publisher– define users and their roles– manage data tree– role-based upload and management of .px files – role-based publication of .px files and folders– definition of data validity date range– define minimum required metadata to permit file

upload– multi-lingual– maintain previous versions of PX files

what we built


PX Publisher – New keywords

• VALIDFROM– taking the format DD/MM/YYYY, this field references the date from which the data is

valid.– Used by CountrySTAT CMS to disable table listing of a PX until the data passes the

criteria.

• VALIDTO– taking the format DD/MM/YYYY, this field references the date until which the data is

valid.– Used by CountrySTAT CMS to disable table listing of a PX until the data passes the

criteria.

• OWNER– A field containing the ID of the person responsible for the PX file.– Used by PX-Publisher to assign user rights to PX files.




PX-Publisher - Status

• Version 1.0 deployed with basic needed functionality:– user management– file upload– folder association– Publishing

• http://devel.si-vs506.com/upload

what we built



PX-Publisher - Next steps• Training

– CountrySTAT users at FAO advanced training next week

• Next iteration(s)– data tree manipulation

– enforce user roles on data tree

– minimum metadata requirements

– group actions zip archives publish whole branches

– restore files

– metadata editing

– PX-Edit integration?

– Reconfigure with PAXIOM class?

what we built



PX-Publisher - Want to try?

• You can always download the latest build here:– http://si-vs506.com:81/svn/countryNet/trunk/uploader/

(login: pc-axis, pw: reader)– Ask us for a JIRA account if you want to report issues

• Interested in developing or extending?– Just ask and we can set you up in the source control

repository on a branch for later integration.

what we built



CountrySTAT Installer

• Simplify local installations for training purposes.– Written using open-source INNO

• Contains:– PX-Edit 2.4– PC-Axis 2008– PX-Web 2007 w/ CountrySTAT data– PX-Map 1– FileZilla FTP

• Checks for:– OS type– presence/absence IIS (performs silent install)– problematic Anti-virus products (Norton, MacAfee)– Skype (often blocks port 80)– previous data and backs up if necessary

what we built



CountrySTAT Virtual Image• What is it?

– A stock Windows XP/IIS5.1/IE6 with– PC-Axis– PX-Edit– Excel– CountrySTAT website with PX-Web2007, PX-Graph and PX-Map1.

• How does it work?– User installs Virtual Box, a free virtual machine from SUN.– User imports the CountrySTAT snapshot and starts it.

• Advantages– Runs on Windows XP/Vista, Mac OSX, Linux and OpenSolaris– Resolves many installation problems by providing a common running

environment for training sessions.

• Disadvantages– Can be confusing to users since they are running two operating systems.

what we built



SDMX - CountrySTAT vision

• Promote it as a way to harmonize national data at regional and international level.

• Offer a standard with which to disseminate national/regional data

• Disseminate FAO classification systems metadata in a standard format.

• Streamline data input to FAO statistical working system

what we built



SDMX - Timeline (June-November 2009)

• Create Data Structure (DSD) and Metadata Structure (MSD) Definitions for several FAOSTAT domains– e.g.. production, commodities.

• Install a metadata registry on CountrySTAT server and train FAO personnel

• Define a mapping between PX and SDMX

• Develop prototype deployable within CountrySTAT capable of– Using the registry, perform round-trip PX–SDMX conversion for a

limited set of structures

– Aggregate from national to regional level.

what we built



SDMX - Issues

• Not possible to automatically map between PX and SDMX.

• Requires a lot of groundwork and training to really benefit.

• Very sophisticated, how to hide complexity from most users?

• Not clear how to simplify the interface between PX and SDMX– depends a lot on the data structure and

– we have MANY different data structures coming from our national partners.

what we built



SDMX - sharing

• Registry is open for you to use– just ask and we can create you as an organization with

user rights for your schema.

what we built



Outline





Outline• what we built

– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX




Building community - Successes• PC-Axis is a vibrant community of some 50 countries

• Dissemination website contains much useful information

• Forum is helpful in resolving issues

• Member countries provide valuable plug-ins

• Components function quite well

but,

Could it be time to take things one step further?

building community

Highlights



Building community - Status

• At a crossroads?– The community’s success has caused it to grow

beyond anyone’s best guess.– It will become increasingly difficult to maintain

coherency as time passes.

• The community’s present size suggests a more formal approach to dealing with development.

building community



Building community – Components

building community

Process

Methods

Tools



Building community – Best practices• Clarify development process(es)

– Waterfall, RAD, Agile, etc.?– Essential for collaboration

• Use accepted development methods– How are requirements developed and negotiated?– Make spec and design docs available– Publish timelines and indications of state of progress– Involve the community in V&V– Create standardized API documentation– Create standardized end-user documentation– Release source code to community

building community



Building community - Tools

• Make use of development process tools– Add bug tracking to allow for systematic

documentation of problems and their resolution

– Add issue tracking to help organize what we built

– Create a community code repository

– Publish a WIKI for community to organize documentation

building community



Building community - AdministrationTreat PC-Axis as a “closed” open-source community

• Create a simple charter with a clear set of roles & responsibilities– Community admin– Repository admin– WIKI admin

• Agree on a minimum level of software artifacts– Requirements specification– API documentation– etc.

• Alignment– existing work added to the community as possible– what we built conform to the charter

building community



Building community – Next steps (a proposal)

• Adopt a cautious, iterative approach– As they teach all medical students,

Primum non nocere!

• Start with the two most essential tools– A documentation WIKI divided by component– A code repository

• Add existing artifacts– Fill in WIKI with requirements, design docs, technical docs, etc.– Baseline code in the shared repository

• Assess the situation and decide how best to continue

building community



Conclusion• PC-Axis continues to meet CountrySTAT requirements

– Easy installation– Powerful conversion tools– Simple human-readable file format– Excellent query interface– Flexible display– Great plugins

• CountrySTAT promotes the use of PC-Axis.– presentation of the tools to interested countries– development and delivery of training materials,– definition and tracking of unmet requirements and

• We plan to keep developing new and existing tools– CountrySTAT CMS,– PX-Publisher,– PC-Axis package installer and– training materials.

• We will continue embracing a closed open-source model– standard processes, methods and tools,– community access,– open code,

Thank you for your time.

food and agriculture organization of the united nations software tools what we built and building...

Documents

px web

instance of px

vilnius lithuania pxpublisher

pxweb query engine

years pxweb code

lithuania pcaxis conference

px file publisher

agriculture organization