business information – sources and search techniques
TRANSCRIPT
Business Information – sources and search
techniques
Scottish Law Librarians Group, Edinburgh, 5th March 2015Edinburgh Training and Conference Venue
Karen BlakemanRBA Information Services
http://www.rba.co.uk/
twitter.com/karenblakeman
Presentation available for a short time at: http://www.rba.co.uk/bi/
This work is licensed under a Creative Commons Attribution-Share-Alike License.
Everything is changing very quickly – not all of the slides are printed here. Download the full set from www.rba.co.uk/bi/[Also slides for Google and other advanced search workshops available at www.rba.co.uk/as/
Some advanced search techniques, lots of sources
Practical sessions - exercises provided but feel free to do your own stuff
Ask questions as we go along – the slides are just a framework and we can explore topics in more depth
“Top Tips” from you at the end of the day
Share with colleagues
09/03/2015 www.rba.co.uk 2
What’s new
EU “right to be forgotten”
More government information and documents supposedly being
moved to www.gov.uk- increasing amount of information going missing
More official data being made available as open data including
Companies House- possible quality and accessibility issues
Google knows best!- major changes to how it analyses and runs your search
- rewriting your search is standard
- results increasingly device dependent
09/03/2015 3www.rba.co.uk
EU - so called “right to be forgotten” ruling
09/03/2015 www.rba.co.uk 4
Mario Costeja Gonzalez
Edition of Monday, January 19, 1998, page 23 -
Newspaper - Lavanguardia.es
http://hemeroteca.lavanguardia.com/preview/19
98/01/19/pagina-23/33842001/pdf.html
EU Court of Justice ruled that Google is a “data
controller” under Data Protection legislation
and must remove, if requested, links to
information that is “inadequate, irrelevant .... or
excessive” from search results on a person’s
name.
Information is NOT removed from the web, only the link to
information in the search results
Not automatic – subject has to apply to have links that point to
specific information about themselves removed from the
results
Request will be assessed to see if the information is
“inadequate, irrelevant or no longer relevant, or excessive in
relation to the purposes for which they were processed.”
Not just Google – all search engines with an EU presence
Only applies to searches conducted in the EU + Norway,
Switzerland, Iceland and Lichtenstein
09/03/2015 www.rba.co.uk 5
EU - so called “right to be forgotten” ruling
Google adds removal statement from all results for searches
on personal names even if nothing has been removed. (Does
not apply to famous people or celebrities!)
Use non-European Google to see all results
e.g. Google.com, Google.ca
09/03/2015 www.rba.co.uk 6
EU - so called “right to be forgotten” ruling
Remember - Google knows best
Google very kindly....
1. Goes to great lengths to personalise your results according
to your search history, contacts, location, device, phase of
the moon, the train, bus or tram you take to work and
anything else it can think of
2. Rewrites your search for you by leaving out some of your
terms and looking for weird and wonderful alternatives
3. Doesn’t bother you with everything that might be relevant
4. Changes its algorithms on a regular basis to keep you on
your toes
5. Constantly conducts experiments on you to ensure that you
don’t feel forgotten
09/03/2015 www.rba.co.uk 7
Private browsing - quickest way “un-personalise”search
Chrome - New Incognito window Ctrl+Shift+N
FireFox Ctrl+Shift+P
Internet Explorer Ctrl+Shift+P
Opera Ctrl+Shift+N
Will not remove country personalisation
Not search engine specific, built into the browser
09/03/2015 8www.rba.co.uk
Google rewrites your search
– Google now tells you which terms it has ignored (some of the
time)
– “..” around terms, phrases, names, titles of documents does not
always work
– To force an exact match and inclusion of a term in a search prefix
it with ‘intext:’
intext:agricultural occupational asthma
– Use Verbatim for an exact match search
09/03/2015 9www.rba.co.uk
09/03/2015 www.rba.co.uk 10
Let’s throw it into the soup
and see if the cat licks it up
Google – missing terms
09/03/2015 www.rba.co.uk 11
Google Verbatim
09/03/2015 12www.rba.co.uk
Choosing your search terms
09/03/2015 www.rba.co.uk 13
Google does not search everything it has in its database
– two indexes: main, default index and the supplemental index
– supplemental index may contain less popular, unusual, specialist
material
– supplemental index comes into play when Google thinks your
search has returned too few results
– Verbatim and some advanced search commands seems to
trigger a search in the supplemental index
09/03/2015 14www.rba.co.uk
09/03/2015 www.rba.co.uk 15
“Normal search”
1,555,500
Search after Verbatim is applied
35,500,000
Google changes its algorithms several hundred times a year
How Google makes improvements to its search algorithm -
YouTube http://youtu.be/J5RZOU6vK4Q
09/03/2015 16www.rba.co.uk
We are all Google’s lab rats
Just Testing: Google Users May See Up To A Dozen
Experiments
http://searchengineland.com/just-testing-google-searchers-may-
see-up-to-a-dozen-experiments-141570
Mostly minor effects on search but sometimes totally bizarre
results
Google decides that coots are really lions
http://www.rba.co.uk/wordpress/2011/02/12/
google-decides-that-coots-are-really-lions/
Update on coots vs. lions
http://www.rba.co.uk/wordpress/2011/02/21/
update-on-coots-vs-lions/
09/03/2015 17www.rba.co.uk
Hummingbird
Not just an update but a completely new way of analysing your search
No longer concentrating on keywords in isolation
Tries to make “sense” of your search and put it into context, natural language queries, uses what others have searched and clicked on
Announced 26th September 2013 but had already been implemented for about a month
Constantly changing – all bets are off when it comes to predicting what your results will look like
09/03/2015 www.rba.co.uk 18
09/03/2015 www.rba.co.uk 19
Google wants to rank websites based on facts not links - 28 February 2015 -New Scientist http://www.newscientist.com/article/mg22530102.600-google-wants-to-rank-websites-based-on-facts-not-links.html
09/03/2015 www.rba.co.uk 20
09/03/2015 www.rba.co.uk 21
And then on another day...
http://googlesystem.blogspot.co.uk/2013/11/google-
knowledge-graph-gets-confused.html
09/03/2015 www.rba.co.uk 22
And one of many wrong Quick Answers submitted to me by
delegates at a recent conference
09/03/2015 www.rba.co.uk 23
Many thanks to Philip Stirups for the example. About 24 hours after taking this screen shot Google corrected the error.
Waitrose Caversham opening times New Year’s Day
09/03/2015 www.rba.co.uk 24
Google used the standard opening times in its answer, not the seasonal opening times
Essential search commands especially
when looking for government
information and data sets
09/03/2015 www.rba.co.uk 25
Think file format
– PDF for research documents, government reports, industry papers
– ppt or pptx for presentations, tracking down an expert on a topic
– xls or xlsx for spreadsheets containing data
Use the advanced search screen or the filetype: command
UK waste vegetable oil energy generation
regulations filetype:pdf
UK waste vegetable oil energy generation
regulations filetype:ppt
UK waste vegetable oil energy generation
regulations filetype:pptx
UK waste vegetable oil energy generation
filetype:xls
UK waste vegetable oil energy generation
filetype:xlsx
09/03/2015 www.rba.co.uk 26
Google site:
site: to search within a site or type of site
waste vegetable oil energy generation Glasgow site:gov.uk
waste vegetable oil energy generation Glasgow
site:www.gov.uk
waste vegetable oil energy generation Glasgow site:ac.uk
Can exclude sites using -site:
agricultural occupational asthma UK site:gov.uk –
site:hse.gov.uk
Does NOT search inside databases or protected areas
Can combine with filetype:
waste vegetable oil energy generation Glasgow site:gov.uk
filetype:pdf
09/03/2015 www.rba.co.uk 27
The best-laid plans of researchers often go awry
09/03/2015 www.rba.co.uk 28
http://www.apho.org.uk/resource/view.aspx?RID=116446
09/03/2015 www.rba.co.uk 29
Date/Any time
Restrict your results to information that has been published
within the last hour, day, week, month, year or your own date
range
Search tools, Any time and select an option [does NOT work
with Verbatim]
09/03/2015 www.rba.co.uk 30
daterange:
Date/Any time menu option can not be used with Verbatim
Use daterange: command instead
Uses Julian date format (fractions omitted)
Julian Date Converter, and information on Julian date format) is at http://aa.usno.navy.mil/data/docs/JulianDate.php/Convert your dates to the Julian format and then, omitting the fractions, copy and paste them into your search
Syntax– for example articles between September 1st and November 1st
2014 on the roll out of universal credituniversal credit announcement
daterange:2456901-2456962
Then apply Verbatim to the results page
09/03/2015 www.rba.co.uk 31
09/03/2015 www.rba.co.uk 32
09/03/2015 www.rba.co.uk 33
Run your search and then apply Verbatim
The Northern Ireland Assembly
http://www.niassembly.gov.uk/
09/03/2015 www.rba.co.uk 37
Oh joy - NOT!
09/03/2015 www.rba.co.uk 38
More UK information vanishes into GOV.UK http://www.rba.co.uk/wordpress/2015/02/28/more-uk-information-vanishes-into-gov-uk/
Where’s the information gone to?
List of departments, agencies and public bodies at https://www.gov.uk/government/organisations
“Home pages” on GOV.UK
Data and information may still be on the old websites
Data may have been moved to http://data.gov.uk/
Information may have been sent to http://www.nationalarchives.gov.uk/webarchive/
Or information may have been “lost”
Don’t rely on just GOV.UK search – use Google/Bing site: command combined with filetype: if appropriate
09/03/2015 www.rba.co.uk 39
"Yes Minister" The Skeleton in the Cupboard (TV Episode 1982) -
Quotes - IMDb http://www.imdb.com/title/tt0751825/quotes
09/03/2015 40www.rba.co.uk
James Hacker: [reads memo] This file contains the complete set of papers, except for a number of secret documents, a few others which are part of still active files, some correspondence lost in the floods of 1967...James Hacker: Was 1967 a particularly bad winter?Sir Humphrey Appleby: No, a marvellous winter. We lost no end of embarrassing files.James Hacker: [reads] Some records which went astray in the move to London and others when the War Office was incorporated in the Ministry of Defence, and the normal withdrawal of papers whose publication could give grounds for an action for libel or breach of confidence or cause embarrassment to friendly governments.James Hacker: That's pretty comprehensive. How many does that normally leave for them to look at?James Hacker: How many does it actually leave? About a hundred?... Fifty?... Ten?... Five?... Four?... Three?... Two?... One?... *Zero?*Sir Humphrey Appleby: Yes, Minister.
[Add “transfer to GOV.UK” to the list of excuses]
09/03/2015 www.rba.co.uk 41
VAT Mini One Stop Shop – Service Assessment | Data at GDS https://gdsdata.blog.gov.uk/vat-mini-one-stop-shop-service-assessment/
An example of the
rigorous nature of
GOV.UK page
development
VAT MOSS pages are
now live without any
of the concerns and
criticisms having
been addressed
09/03/2015 www.rba.co.uk 42
https://www.gov.uk/search-house-prices
All is not well with
some of the pages on
GOV.UK
09/03/2015 www.rba.co.uk 43
Land Registry open data http://landregistry.data.gov.uk/
Land Registry database moved to the Open Data web site
http://www.rightmove.co.uk/
09/03/2015 www.rba.co.uk 44
Land Registry data often goes missing. I know that 10 months ago the sold price for number 90 was listed as £185,000 and for 2012.
09/03/2015 www.rba.co.uk 45
http://landregistry.data.gov.uk/app/ppd/search
Data doesn’t show up via the Land Registry Open Data interface either.
Title document
09/03/2015 www.rba.co.uk 46
Data IS present in the title document.
Missing data
09/03/2015 www.rba.co.uk 47
Error report filed with the Land Registry - still waiting for a response
Why might a property/price paid not appear in the data?
Seems not that uncommon according to discussion boards – usually data entry error (but the above example was in the open data sets until a few months ago)
Absence of price – gift of property or purchase of a share
Impractical to calculate price e.g. bulk purchase of properties
Commercial transactions
https://www.gov.uk/about-the-price-paid-data#data-excluded-from-the-house-price-index-and-price-paid-data
Raw data files downloaded and searched and data for number 90 is missing
Why some data goes AWOL
09/03/2015 www.rba.co.uk 48
Variable Pitch http://www.variablepitch.co.uk/stations/1310/
Uses public electricity microgeneration data
09/03/2015 www.rba.co.uk 49
Variable Pitch http://www.variablepitch.co.uk/stations/2580/
Virginia Station is the hydroelectric installation at Windsor Castle – no data!
FoI request generation data for Virginia Station
09/03/2015 www.rba.co.uk 50
https://www.whatdotheyknow.com/request/request_electricity_output_of_ro
http://www.independent.co.uk/news/uk/home-news/royal-family-granted-new-right-of-secrecy-2179148.html
http://www.legislation.gov.uk/
Launched in 2010
Not all legislation has been updated
Pending updates are flagged
09/03/2015 www.rba.co.uk 51
Monitoring progress of legislation
http://services.parliament.uk/bills/2014-15/infrastructure.html
09/03/2015 www.rba.co.uk 53
They work for you http://www.theyworkforyou.com/
09/03/2015 www.rba.co.uk 54
WhatDoTheyKnow - Freedom of Information (FOI)
requests https://www.whatdotheyknow.com/
09/03/2015 www.rba.co.uk 55
http://www.nationalarchives.gov.uk/webarchive/
09/03/2015 www.rba.co.uk 56
Company information
Ownership, directors, structure, share price, accounts, activities, news
Availability depends on
– type of company
• sole trader
• partnership
• private
• limited liability e.g. Ltd, LLP
• companies with publicly traded shares e.g. PLC
• CIC (Community Interest Company), CIO (Charitable
Incorporated Organisation – England and Wales)
– business size (large, medium, small)
– country
state or region within the country
09/03/2015 www.rba.co.uk 58
UK Companies Act 2006
Some of the main changes
– memorandum of association now only needs to state that the
founders of the company wish to form a limited company
– directors no longer need to apply to withhold their private
address, for example if they believe they may be targeted by
activists. Can now choose to supply a “service address”
– requirement to have a company secretary removed
UK Companies House
– https://www.gov.uk/government/organisations/companies-house
– http://wck2.companieshouse.gov.uk/
– details of registration and filing requirements
– search for companies for free, some information free, accounts
and other documents priced
– free Disqualified Directors search
09/03/2015 www.rba.co.uk 59
Exemptions
For UK small and medium sized enterprises
May prepare and file abbreviated accounts
Must fulfil at least two of the following conditions to qualify
09/03/2015 www.rba.co.uk 60
Exemptions not available to public companies, banking,
insurance or shipping companies
Small Medium
Turnover not exceeding £6.5m £25.9m
Assets/balance sheet not
exceeding
£3.26m £12.9m
Average number of employees 50 250
UK filing requirements
Small
companies
Medium
companies
Large
companies
Balance sheet Abbreviated
balance sheet
with notes
Full balance
sheet
Full balance
sheet signed by a
Director
Profit and loss Not required Abbreviated profit
and loss account
Full profit and
loss account
Turnover Not required Not required Required
Director report Not required Required Required and
signed by an
officer of the
company
Notes to the
accounts
Not required Required Required
09/03/2015 www.rba.co.uk 61
Legal Forms & Company Extensions
A Guide to Legal Forms for Social Enterprise
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/
31677/11-1400-guide-legal-forms-for-social-enterprise.pdf
A Guide to Legal Forms for Business
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/
31676/11-1399-guide-legal-forms-for-business.pdf
CorporateInformation.com - Company Extensions and Security Identifiers
http://www.corporateinformation.com/Company-Extensions-Security-
Identifiers.aspx [does not appear to be updated]
Types of business entity http://en.wikipedia.org/wiki/Types_of_business_entity
Company Formation Survey http://www.offshoreinvestment.com/
Doing Business - Measuring Business Regulations - World Bank Group
http://www.doingbusiness.org/
09/03/2015 www.rba.co.uk 62
Office of the Regulator of Community Interest Companies
https://www.gov.uk/government/organisations/office-of-the-
regulator-of-community-interest-companies
09/03/2015 www.rba.co.uk 63
Mutual societies - Financial Conduct Authority
http://www.fca.org.uk/firms/firm-types/mutual-societies
09/03/2015 www.rba.co.uk 64
UK Companies Beneficial Ownership Register
http://www.macfarlanes.com/media/382444/uk-companies-
beneficial-ownership-register2.pdf
09/03/2015 www.rba.co.uk 65
Will be implemented in the Small Business, Enterprise and Employment billhttp://services.parliament.uk/bills/2014-15/smallbusinessenterpriseandemployment.html
Charity Commission – does not include Scotland
09/03/2015 www.rba.co.uk 66
https://www.gov.uk/government/organisations/charity-commission
http://apps.charitycommission.gov.uk/Showcharity/RegisterOfCharities/registerhomepage.aspx
The Charity Commission for Northern Ireland
https://www.charitycommissionni.org.uk/
09/03/2015 www.rba.co.uk 67
OSCR Scottish Charity Regulator http://www.oscr.org.uk/
09/03/2015 www.rba.co.uk 68
Scotland's charity regulator (OSCR) plans to publish charity accounts http://www.fundraising.co.uk/2015/02/09/scotlands-charity-regulator-oscr-plans-publish-charity-accounts/
CAF Charity Trends http://www.charitytrends.org/Default.aspx
09/03/2015 www.rba.co.uk 69
Many thanks to Andrew Oakes for the information and link to this siteOnly covers England and Wales
UK Companies House
– https://www.gov.uk/government/organisations/companies-house
– Database still at http://wck2.companieshouse.gov.uk/
– documents £1 each
09/03/2015 www.rba.co.uk 70
Company documents
Companies House charges £1 per document and there can be
lots of them - decide which ones you really need
The list of company documents may be all you need to see
09/03/2015 www.rba.co.uk 71
09/03/2015 www.rba.co.uk 72
Free Companies House data to boost UK economy - Press releases - GOV.UK https://www.gov.uk/government/news/free-companies-house-data-to-boost-uk-economy
Companies House free data
http://download.companieshouse.gov.uk/en_accountsdata.html
Bulk data – all or nothing
Large daily files available as zipped files
No support provided – you’re on your own!
09/03/2015 www.rba.co.uk 73
Companies House free data
09/03/2015 www.rba.co.uk 74
Each file within the zipped file is a separate document. Note the uninformative file names!
Company Check
09/03/2015 www.rba.co.uk 75
Company Check
09/03/2015
www.rba.co.uk 76
Company Check Dashboard
09/03/2015 www.rba.co.uk 77
DUEDIL http://www.duedil.com/
09/03/2015 www.rba.co.uk 78
Sign-up required (free of charge), some information priced
DUEDIL http://www.duedil.com/ group chart
09/03/2015 www.rba.co.uk 80
Now only available as part of a subscription
Official company information – other countries
Lists of official company registers
– Official Company Registers
http://www.rba.co.uk/sources/registers.htm
– Company registration around the world
http://www.commercial-register.sg.ch/home/worldwide.html
– Companies House Overseas registries links
http://www.gov.uk/government/publications/overseas-registries#reg
US – listed companies
– SEC Edgar IDEA – Interactive Date Electronic Applications
http://www.sec.gov/edgar.shtml
Canada - listed companies
– SEDAR http://www.sedar.com/
09/03/2015 www.rba.co.uk 81
Examples of problem areas for the researcher
Switzerland
– only banks, insurance companies and companies traded on the
Swiss stock exchange are required to provide financial
statements
Cayman Islands
– “The Registrar of Companies can only release the name and
type of company, its date of registration, the address of the
registered office and the company’s status. Disclosing any other
information is prohibited unless requested by a law enforcement
agency.”
British Virgin Islands
– “A company is not required to file its register of directors, register
of members, register of charges or an annual return with the BVI
Registrar of Corporate Affairs.”
09/03/2015 www.rba.co.uk 82
European Business Register
Gateway to official company information in 28 European countries
– http://www.ebr.org
UK distributer is GBRDirect – priced service
– http://www.gbrdirect.co.uk
09/03/2015 www.rba.co.uk 85
7side http://www.7side.co.uk/
Priced service
UK companies and documents, directors information,
insolvency reports
Credit reports
International registry and documents service
– 1 hour registry and documents service
www.7side.co.uk/moreinfo/International_Registry_and_Doc
uments_Service.pdf
– translation service available
– have agents "on the ground", can quickly tell you what is
available in that country and what may not be available
09/03/2015 www.rba.co.uk 86
Several of the workshop delegates use this service and had very positive feedback on it.
ICIJ Offshore Leaks Database
http://offshoreleaks.icij.org/
09/03/2015 www.rba.co.uk 87
ICIJ Releases Offshore Leaks Database Revealing Names Behind Secret Companies, Trusts | International Consortium of Investigative Journalists http://www.icij.org/offshore/icij-releases-offshore-leaks-database-revealing-names-behind-secret-companies-trusts
https://www.youtube.com/watch?v=GHmiomMNfLQ&feature=youtu.be
VAT number validation and EU VAT rates
09/03/2015 www.rba.co.uk 88
VAT number validationVIES http://ec.europa.eu/taxation_customs/vies/vieshome.do
EU VAT rates - essential if a seller is subject to the new VAT MOSS ruleshttp://ec.europa.eu/taxation_customs/resources/documents/taxation/vat/how_vat_works/rates/vat_rates_en.pdf
Google street view
As well as for personal use, useful for assessing location of a
business (but remember out of date images)
09/03/2015 www.rba.co.uk 89
Social media & professional networks
09/03/2015 www.rba.co.uk 90
09/03/2015 www.rba.co.uk 91
Official statistics
OFFSTATS http://www.offstats.auckland.ac.nz/
UK National Statistics Publication Hub http://www.statistics.gov.uk/ [now redirects to https://www.gov.uk/government/statistics/announcements ]
Office for National Statistics http://www.ons.gov.uk/
Official Statistics in Scotland http://www.gov.scot/Topics/Statistics
Welsh Government Statistics & Research http://gov.wales/statistics-and-research/?lang=en
Welsh Assembly Government StatsWales http://statswales.wales.gov.uk/
data.gov.uk http://data.gov.uk/
Eurostat http://ec.europa.eu/eurostat/
European Union Open Data Portal http://open-data.europa.eu/en/data/
09/03/2015 www.rba.co.uk 92
Google Public Data Explorer
http://www.google.com/publicdata/
One of Google's best kept secrets!
Public data sets made available by Eurostat, World Bank, IMF,
CSO Ireland, OECD, ITU, some national statistics offices (but
not ONS), and many more.
Source and date updated given.
Charts and charting options can highlight oddities and missing
data
Look at the charts to see if there is a sudden change in the
trends.
09/03/2015 www.rba.co.uk 94
Google Public Data Explorer Minimum Wage
09/03/2015 www.rba.co.uk 95
Some countries are missing e.g. Germany
Eurostat - Minimum Wage
09/03/2015 www.rba.co.uk 96
09/03/2015 www.rba.co.uk 97
Datamarket http://datamarket.com/
Open portal to datasets worldwide and market research
Creates visualisations of the data
09/03/2015 www.rba.co.uk 98
No longer available. DataMarket has been
acquired by Qlik
Statista http://www.statista.com/
“The Statistics Portal for Market Data, Market Research and Market
Studies”
– 60,000 topics from over 18,000 sources
– http://www.statista.com/topics/
– some information free, registration (free) required
– Chart of the day
09/03/2015 www.rba.co.uk 99
Zanran
http://www.zanran.com/
Zanran – great for data in tables, charts and graphs
http://www.rba.co.uk/wordpress/2013/01/25/zanran-great-for-data-in-
tables-charts-and-graphs/
09/03/2015 www.rba.co.uk 100
09/03/2015 www.rba.co.uk 101
The Guardian http://www.theguardian.com/data
And finally.....
Per capita consumption of mozzarella cheese (US) correlates with Civil
engineering doctorates awarded (US)
http://tylervigen.com/view_correlation?id=3890
09/03/2015 www.rba.co.uk 102