october 1 niso training thursday: using alerting systems to ensure oa policy compliance
TRANSCRIPT
SHARE Phase IIJudy Ruttenberg, Association of Research Libraries
Erin Braswell, Center for Open ScienceFabian von Feilitzsch, Center for Open Science
NISO Virtual Conference: October 1, 2015Using Alerting Systems to Ensure OA Policy Compliance
Founded by Academic Leaders, Built with Open Technology
Research universities are long-lived and are mission-driven to generate, make accessible, and preserve over time new
knowledge and understanding.
What is SHARE?
SHARE is building a free, open data set about research and scholarly activities across their life cycle.
Research Lifecycle
Open Science Frameworkhttp://osf.io
Using SHARE’s Search API● API is currently a slightly restricted Elasticsearch
instance● You can hit the API with any valid Elasticsearch query● Going to go over some quick and sort of interesting
aggregations that are available
What it looks like: The query$ curl -XPOST "https://osf.io/api/v1/share/search/" -H 'content-type: application/json' -d '{
"query": {"match_all": {}},
"size": 0,
"aggs": {
"top tags": {
"terms": {
"field": "tags"
}
}
}
}'
What it looks like: The response{ "count": 2137806, "time": 0.1, "results": [], "aggs": null, "aggregations": { "top tags": { "buckets": [ { "key": "ecological", "doc_count": 20347 }, { "key": "long", "doc_count": 20179 }, { "key": "term", "doc_count": 20021 }, { "key": "lter", "doc_count": 18862 }, { "key": "data", "doc_count": 17086 }, ....
...
{ "key": "research", "doc_count": 16539 }, { "key": "earth", "doc_count": 16395 }, { "key": "water", "doc_count": 16150 }, { "key": "program", "doc_count": 16098 }, { "key": "remote", "doc_count": 15963 } ], "sum_other_doc_count": 1049093, "doc_count_error_upper_bound": 3000 } }}
Kind of a pain
● We have an experimental python library to help cut down the verbosity a bit
Same Example>> from sharepa import ShareSearch>> from sharepa.analysis import bucket_to_dataframe
>> search = ShareSearch()>> search.aggs.bucket('top tags', 'significant_terms', field='tags')
Internal structure is:
{ "query": { "match_all": {} }, "aggs": { "top tags": { "significant_terms": { "field": "tags" } } }}
Now we send the JSON blob to the SHARE search API
>> results = search.execute()
And we get back the same response.
We can then use some of our utilities to convert
the Elasticsearch response to a dataframe
(basically just a table)
>> df = bucket_to_dataframe( 'top tags', results.aggregations['top tags']['buckets'] ).sort('top tags', ascending=False)
and plot it as well:
>> df.plot(kind='bar', x='key', y=['bg_count', 'top tags'])
Cancer
Flu
Influenza
Vaccine
Phase II
