wso2con eu 2015: open source journey at ordnance survey
TRANSCRIPT
The Open Source Journey at OS
Hillary Corney Principal Architect
Ordnance Survey
An introduc9on to Ordnance Survey
• Ordnance Survey over 220 years old. Founded in 1791 • Started as a military mapping unit now the UK’s na9onal mapping agency.
• No longer military, now a UK Government owned company a “GovCo”
• Famous for the best paper maps in the world • What you may not know is that OS is now a mostly digital organisa9on and that it hosts the largest Geospa9al database in the world called “MasterMap”.
• We also hold the Guinness Book of Records world record for the largest MinecraT map ever made with an astonishing 83 billion blocks. (And you thought big Government compu9ng projects always failed!)
Digital Maps of the UK accurate to a 1cm updated 10,000 9mes a day
Produc9on IT soTware at Ordnance Survey • Ini9al experiments using IT to support map making started in late 1960s • First digital product made available in 1971 (1:125,000 coastline) • Increasing ramping up of digi9sa9on (1970s – 1990s) • Move from automa9ng map produc9on to digital data (1990s onwards) • In the early years much of the Mapping soTware we used was bespoke • Over 9me increasing use made of proprietary off-‐the-‐shelf soTware products (as
capability aligned with our needs) • In the last 10 years, open-‐source op9ons have developed and matured (as
capability has improved)
Bespoke Proprietary
Open source
1970 Mid-‐1980s Mid-‐2000s Today 1970 Mid-‐1980s Mid-‐2000s Today
The state of Open Source SoTware adop9on at OS Pre-‐2011
• Managers and Architects shied away from open source.
• Its use was organic, developer led, no enterprise level involvement or strategy
• Open Source soTware used in ad-‐hoc way • OTen used “under the radar” • SoTware repositories and version control available but seldom used.
• Very licle maintenance, Patching, upgrades etc. • Separate discrete code bases for the same soTware
• Almost no hand-‐over into produc9on and licle documenta9on
A short history of open source at Ordnance Survey Apache, Tomcat, J2EE & MySQL for initial web systems
2002
Linux operating system for web-facing applications
2004
Linux operating system for internal production systems
2006
Using Open Layers for OS OpenSpace
2008
PostGIS, GeoNetwork, GeoServer, INSPIRE
2010
PostGIS, Web Services Consolidation
2012 2013
Magento, Solr for Map Shop, Apache Jena for linked data
Open source
First
Push
The Open Source in the Enterprise “revolu9on” at OS
• In 2010 the outgoing Treasury Minister Liam Burne, leT a now infamous note, “there's no cash le.”. He has recently said it was a foolish thing to do and not a day goes by without him feeling guilty for wri9ng it.
• But don't despair Liam, there was an unexpected upside......
• The incoming government, worried, and looking for savings, sent out the “Open Source first” direc9ve.
The key points of the Government’s new directive were:
1) The Government will ac=vely and fairly consider open source solu=ons alongside proprietary ones in making procurement decisions,
2) Procurement decisions will be made on the basis on the best value for money solu9on to the business requirement, taking account of total life9me cost of ownership of the solu9on, including exit and transi9on costs, aTer ensuring that solu9ons fulfill minimum and essen9al capability, security, scalability, transferability, support and manageability requirements.
3) The Government will expect those pujng forward IT solu9ons to develop where necessary a suitable mix of open source and proprietary products to ensure that the best possible overall solu9on can be considered.
4) Where there is no significant overall cost difference between open and non-‐open source products, open source will be selected on the basis of its addi=onal inherent flexibility.
How do you “ac=vely and fairly consider open source solu=ons alongside proprietary ones”?
• What are the barriers to the adop=on of Open Source So.ware?
• The lack of a Vendor is a factor in low OSS uptake.
• Because there is no Vendor: • No one promotes and markets the soTware • No one pays for reviews of the soTware • No one champions the soTware • No one sponsors the soTware
• So OS so.ware is o.en invisible, even good OS so.ware is o.en a well kept secret.
Barriers to the adop9on of Open Source Software:
• Myths and FUD taken as truth. • MicrosoT and others spent large amounts of effort in the first half of the last decade debunking Linux/OSS by spreading fear and rumours which they now admit actually backfired and increased awareness and adop9on of Linux/OSS but much of the misinforma9on from those days s9ll lingers on.
• If you are interested Google “MicrosoT The Halloween document”
• Lack of a “Vendor” driving any tender response • Lack of external support Lack of documenta9on • Lack of appropriate in-‐house coding skills • Lack of fair and honest evalua=ons comparisons and reviews
More Myths -‐ Open Source SoTware is “poor quality”
• A Coverity 2011 survey, found Open source was as good as proprietary soTware
• “Both open source code quality and proprietary code quality, as measured by defect density, is becer than the average for the soTware industry, which is a defect density of 1.0.”
• “Linux 2.6, PHP 5.3, and PostgreSQL 9.1 are recognized as open source projects with superior code quality and can be used as industry benchmarks, achieving defect densi9es of .62, .20, and .21 respec9vely.”
• Open source code quality is on par with proprietary code quality, par9cularly in cases where codebases are of similar size. For instance, Linux 2.6, a project with nearly 7 million lines of code, has a defect density of .62 which is roughly iden9cal to that of its proprietary codebase counterparts.
Coverity's 2013 Scan concluded that the defect density of Open Source code was now be-er than proprietary code!
• In 2013, the defect density rate of open source code was lower than that of proprietary code. According to recent reports sponsored by Black Duck SoTware and North Bridge Venture Partners, “quality concerns are no longer a barrier to open source adop9on in the enterprise. In fact, the quality of the open source code for Coverity Scan par9cipants can be higher than the proprietary code included in an enterprise product.”
So if quality is not a problem how do you “ac=vely and fairly consider open source solu=ons alongside proprietary ones”?
• Appoint an internal Open Source Champion. Someone who will take the place of the missing vendor and “market” the product to the stakeholders.
• Use proof of concept builds to evaluate the suitability of OSS.
• Think crea9vely about how you can find equivalents to the normal Vendor procurement process.
Evaluate OSS like any COTS package but….
• Where you would evaluate the vendor subs9tute the User community.
• Where you would evaluate help desk support look at the availability of of OSS Specialist Books, the community and internal knowledge.
• Where you would review the release roadmap Look at the release frequency and access to nightly builds.
• Ask the user community for reference cases and recommenda9ons.
Understand Licensing
• There are over 70 licenses approved by the open source ini=a=ve, so it is definitely not an easy task to understand all of the licenses out there.
• Fortunately the 10 most popular licenses make up about 80% of the open source available, so understanding the first 10 licenses is a preXy reasonable start.
• Create an open source review board consis9ng of both legal counsel and SoTware engineering governance, who can understand the implica9ons of the various licenses.
• Create a summary of current OS licences keep it up to date.
• Lead developers and Architects will need to review the summary and be aware of the legal traps.
Working with Open Source
• As Laurie Wurster of the Gartner Group said: “Just because something is free, doesn’t mean it has no cost.” “Companies must have a policy for procuring OSS, deciding which applications will be supported by OSS, and identifying the intellectual property risk or supportability risk associated with using OSS. Once a policy is in place, then there must be a governance process to enforce it.”
• There must be a policy and enforcing governance for procuring Open source.
• There must also be a policy and governance for implementing, developing and supporting Open Source Software.
Recommenda9ons
• We incorporate the Government Open Source Policy into our procurement policy
• Ensure that when Evalua9ng OSS there is a quick route to financing proof of Concept Evalua9on Environments.
• We adopt a structured approach to evalua9ng Open Source SoTware.
• We educate developers to be aware of the risks inherent in Open Source licences. We encourage open source skills in house and we create a library of open source books for developers to use.
• We appoint OSS Champions to each Procurement ini9a9ve to make sure that OSS is considered fairly and is “Marketed” to the Procurement Stakeholders in a proac9ve way.
• We ensure good legal and technical governance of our use of Open Source.
• We create our own Ordnance Survey Open Source Licence
Devise a “Procurement decision chart”
Open Source first
Based on our experiences, we have adopted an “open source first” enterprise level policy for new soTware When looking for new soTware we will evaluate open source packages ahead of proprietary ones We will adopt an open source solu9on if:
• The licence is suitable for our needs • The project is well supported with a lively and responsive user
base • The documenta9on is good • We have the appropriate skills in-‐house or training is readily
available • We can support it (or we can buy support) • It works equally well as available proprietary soTware.
If not, we will go through standard COTS selec9on / procurement processes
What have we learned
• Start small, download the soTware and try stuff out.
• Involve and listen to your developers • Broaden technical knowledge and skills • invest in training • Change the COTS only culture • “Open Source” is free is a myth the costs are just in different places.
• Be rigorous about evalua9ng licenses
What about support?
• Just because it’s Open Source does not necessarily mean you are on your own, but the support model is different. • You can do it in house (but be aware of the effort involved and don’t scrimp)
• Or you can buy support from vendors such as WSO2 which is an increasing sign that Open Source is entering the mainstream.
How did we apply what we have learned to our API delivery plarorm project?
• We reviewed the Open Source packages first. • We evaluated WSO2 against other Open Source and COTS solu9ons
• We loaded it onto our developer sandboxes and “kicked the 9res”
• We engaged WSO2 and ini9ated a very successful proof of concept….
POC Solu9on hosted on two servers in AWS and built with a Joint OS/WSO2 team in just two weeks
Integra9on with Magento
• Another benefit of the “Open Source First” approach is that we are able to closely couple and integrate another Open Source program “Magento Enterprise” a leading eCommerce package with the WSO2 SoTware
• As we have access to the source code of both plarorms we can 9ghtly integrate the two in a way that would not be possible, or very much more difficult with COTTS soTware.
Logical Architecture PRODUCTION environment
Authen9ca9on Architecture Integra9on between Magento and WSO2 Iden9ty Server via SAML and SCIM
The outcome of the Open Source First process:
• We purchased support and finally having sa9sfied ourselves that WSO2 can meet our needs….
• We engaged WSO2 and are building not one but two API delivery produc9on environments with WSO2 soTware. U9lising:
• WSO2 API manager • WSO2 Iden9ty Manager • WSO2 Business ac9vity Monitor • WSO2 Complex event processor • Magento Enterprise Open Source Shopping cart.
The best of both worlds?
• It is my opinion that organisa9ons like WSO2 and Magento offer the best of both worlds.
• A vibrant user community feeding ideas and fixes back to the user base
• With Enterprise level support and SoTware roadmap and strategy from the Parent organisa9on.
• WSO2 are the “Vendor” providing Enterprise level support oTen missing from tradi9onal Open Source products, eBay Enterprise provide similar Support for Magento.
Ian James OS Chief Architect said:
“Open Source allows us to deliver more effec9vely”
“Open Source is mainstream”
“Open Source is here to stay at Ordnance Survey”
Thank You
Ques9ons?