Upload File

introduction to codalab competitions · introduction in a shared task / challenge / competition /...

  • Home
  • Documents
  • Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
63
Introduction to CodaLab Competitions Tristan Miller Presented at: School of Data Analysis and Artificial Intelligence National Research University – Higher School of Economics 25 May 2017 25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 1

Upload: others

Post on 07-Oct-2020

8 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction to CodaLab Competitions

Tristan Miller

Presented at:School of Data Analysis and Artificial IntelligenceNational Research University – Higher School of Economics25 May 2017

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 1

Page 2: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Overview

What are shared tasks?

What is CodaLab Competitions?

Organizing CodaLab Competitions

Student competitions

Caveats

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 2

Page 3: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

What are shared tasks?

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 3

Page 4: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)

I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 5: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performance

I produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 6: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 7: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithmsI applying the evaluation metrics, score the algorithms’ output against the gold

standard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 8: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 9: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test data

I implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 10: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test dataI implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 11: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Introduction

In a shared task / challenge / competition / evaluation campaign / evaluationexercise, the organizers

I define a data processing task (classification, segmentation, ranking, etc.)I define evaluation metrics to measure performanceI produce test data and the gold standard answers

I produce trial data for demonstration purposesI produce or provide ancillary resources (knowledge bases, etc.)I produce training data for use by supervised algorithms

I solicit participants to write algorithms to process the test dataI implement various baseline algorithms

I applying the evaluation metrics, score the algorithms’ output against the goldstandard

I compare and analyze the participants’ and baseline algorithms

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 4

Page 12: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Shared tasks: pros and cons

Pros:I stimulate methodological research on unsolved problemsI provide standardized data sets, resources, and evaluation metricsI facilitate reproducibility of research resultsI centralize publication and discussion of research results

Cons:I everything must be planned in advanceI large organizational overhead (data distribution, publicity, communication with

participants, etc.)I encourages “teaching to the test”

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 5

Page 13: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Shared tasks: pros and cons

Pros:I stimulate methodological research on unsolved problemsI provide standardized data sets, resources, and evaluation metricsI facilitate reproducibility of research resultsI centralize publication and discussion of research results

Cons:I everything must be planned in advanceI large organizational overhead (data distribution, publicity, communication with

participants, etc.)I encourages “teaching to the test”

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 5

Page 14: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

What is CodaLab Competitions?

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 6

Page 15: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

CodaLab Competitions

I Web-based platform for running online data-based competitionsI Developed by Microsoft, Stanford University, and othersI Free hosted implementation: https://competitions.codalab.org/I Free software (Apache License 2.0):

https://github.com/codalab/codalab-competitions/

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 7

https://competitions.codalab.org/
https://github.com/codalab/codalab-competitions/
Page 16: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Microsoft COCO Image Captioning Challenge

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 8

Page 17: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

ChaLearn Looking at People Challenges

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 9

Page 18: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

SemEval-2017 Multilingual and Cross-lingualSemantic Word Similarity

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 10

Page 19: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

CodaLab features

I Hosts public task websiteI Hosts private gold-standard data and scoring softwareI Manages participant registrationI Enforces submission deadlinesI Runs scoring softwareI Tabulates, stores, and publishes resultsI Handles communication between organizers/participants (e-mail, forums)I Provides some publicity

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 11

Page 20: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 21: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 22: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 23: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 24: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 25: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 26: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 27: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 28: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Organizing CodaLab Competitions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 20

Page 29: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 30: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 31: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 32: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 33: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Competition bundle file structure

Competitions are defined by a zipped file archive (a bundle) containing:I a logo imageI HTML files for the competition websiteI scoring software (ZIP archive)I gold-standard (“reference”) data (ZIP archive)I competition.yaml

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 25

Page 34: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

competition.yaml settings

I Competition title, description, and logoI Whether registration is requiredI Filenames for standard and option web pagesI Configuration of competition phasesI Format of the leaderboard

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 26

Page 35: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

competition.yaml: basic settings

title: SemEval -2017 Task 7, Subtask 3description: Interpretation of English Punsimage: semeval2017 -task7 -logo.pnghas_registration: Trueallow_teams: Trueend_date: 2017 -01 -30html:

overview: overview.htmlevaluation: evaluation.htmlterms: terms_and_conditions.htmldata: data.html...

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 27

Page 36: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 37: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

HTML files

<p>Data sets for this subtask are described in detail on the<a href="http :// alt.qcri.org/semeval2017/task7/index.php

?id=data -and -resources">SemEval -2017 Task 7 website </a>.</p>

<h4>Download </h4><ul><li><a href="data/uploads/semeval2017_pun_task.tar.xz">

Trial data</a></li><li>Test data will not be released until the evaluation

begins </li></ul>

<p>Note that , due to the difficulty in amassing a largenumber of pun examples per word or per sense , there is <strong >no training data</strong > for this task.</p>

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 29

Page 38: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Competition phases

I Phases break your competition into optional subtasksI Each phase has its own:

I titleI start dateI scoring programI dataI submission limit

I Phases can be run in parallel, staggered, or (with some difficulty) in sequenceI It’s common to have a “trial” phase for sandbox testing:

I earlier start dateI toy dataI unlimited submissions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 30

Page 39: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

competition.yaml: defining phases

phases:1:

phasenumber: 1label: "Trial"start_date: 2016 -12 -05max_submissions: 999scoring_program: scorer.zipreference_data: data_trial.zipleaderboard_management_mode: hide_resultscolor: white

2:phasenumber: 2label: "Test␣(Homographic)"start_date: 2017 -01 -23...

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 31

Page 40: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 41: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Leaderboards

I A leaderboard is a dynamically updated results tableI Each phase has 0 or more leaderboards, public or hiddenI The columns are the metrics output by your scoring softwareI The rows are the participants’ submissionsI You define the column labels and numeric formatI You can also rank the metrics by priority

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 33

Page 42: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

competition.yaml: defining the leaderboards

leaderboard:leaderboards:

RESULTS: &RESULTSlabel: Resultsrank: 1

columns:coverage:

leaderboard: *RESULTSlabel: coveragerank: 4numeric_format: 4

precision:leaderboard: *RESULTSlabel: precision...

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 34

Page 43: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 44: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Scoring programs and reference data

I You provide a ZIP file containing arbitrary gold-standard reference dataI You provide another ZIP file containing

I an executable scorerI a metadata file that describes how to run the scorer, using the following variables:

$program scorer directory$input/ref reference data directory$input/res submission data directory

$output output directory

description: Scoring program for Subtask 3 of SemEval-2017 Task 7 (pun interpretation)

command: java -classpath $program de.tudarmstadt.ukp.semeval2017.task7.scorer.PunScorer -i $input/ref/truth.txt $input/res/answer.txt $output/scores.txt

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 36

Page 45: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Scoring programs and reference data

I Scorers can be written in any language (Python, Java, Perl, etc.)I The scorer must produce a key–value file $output/scores.txtI Each key corresponds to a leaderboard column key:

coverage: 1.000precision: 0.825recall: 0.775f1: 0.799

I stderr is captured and reported to the submitterI stderr and stdout are captured and stored for the competition organizer

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 37

Page 46: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 47: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 48: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 49: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Upon uploading a competition

I You will (usually) be warned if there is a problemI You optionally “publish” your competition:

Unpublished: accessible only if you know the URL (default)Published: listed on CodaLab Competitions home page

I Competitions can be edited via the web interface to:I Publish/unpublishI Change competition settingsI Edit web pages (via rich text editor)I Add or remove reference data and scoring programs

I Editing the original YAML is no longer possible!

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 41

Page 50: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 51: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 52: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 53: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Viewing and downloading submissions

The “Submissions” page allows you to:I see a list of all successful and failed submissionsI download the original system outputI download a CSV of all system scoresI re-run the scorer on one or all submissionsI delete a submissionI (un)hide a submission from the leaderboard

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 45

Page 54: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 55: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Student competitions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 47

Page 56: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing
Page 57: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Student competitions

I skill development for student participants/organizersI experimentation, problem formulation/solvingI writing/fulfilling technical specificationsI collaboration with peersI presentation of research results

I benefits for student participantsI unrestricted choice of tools/methodsI instant feedback to studentsI freedom to experiment

I benefits for teachersI automatic enforcement of submission deadlinesI instant tabulation of scoresI consistent packaging of submissions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 49

Page 58: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Student competitions

I skill development for student participants/organizersI experimentation, problem formulation/solvingI writing/fulfilling technical specificationsI collaboration with peersI presentation of research results

I benefits for student participantsI unrestricted choice of tools/methodsI instant feedback to studentsI freedom to experiment

I benefits for teachersI automatic enforcement of submission deadlinesI instant tabulation of scoresI consistent packaging of submissions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 49

Page 59: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Student competitions

I skill development for student participants/organizersI experimentation, problem formulation/solvingI writing/fulfilling technical specificationsI collaboration with peersI presentation of research results

I benefits for student participantsI unrestricted choice of tools/methodsI instant feedback to studentsI freedom to experiment

I benefits for teachersI automatic enforcement of submission deadlinesI instant tabulation of scoresI consistent packaging of submissions

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 49

Page 60: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Caveats

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 50

Page 61: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Caveats

CodaLab Competitions is . . .I underdocumentedI unpredictableI unintuitiveI unstable

But it’s . . .I free (as in beer)I free (as in speech)I popularI under active developmentI probably a time-saver, in the long run

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 51

Page 62: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Caveats

CodaLab Competitions is . . .I underdocumentedI unpredictableI unintuitiveI unstable

But it’s . . .I free (as in beer)I free (as in speech)I popularI under active developmentI probably a time-saver, in the long run

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 51

Page 63: Introduction to CodaLab Competitions · Introduction In a shared task / challenge / competition / evaluation campaign / evaluation exercise, the organizers I define a data processing

Thank you!

Questions?

25 May 2017 | Ubiquitous Knowledge Processing Lab | Department of Computer Science | Tristan Miller | 52


Define the major features of Cisco intrusion protection ...ptgmedia.pearsoncmg.com/images/chap04_1587051443/elementLinks/...• Define the major features of Cisco intrusion protection

HLA and autoantibodies define scleroderma …HLA and autoantibodies define scleroderma subtypes and risk in African and European Americans and suggest a role for molecular mimicry

Successor Features for Transfer in Reinforcement Learning · Successor Features for Transfer in Reinforcement Learning ... policy evaluation and policy improvement, define the basic

An Ontology Design Pattern to Define ExplanationsAn Ontology Design Pattern to Define Explanations Ilaria Tiddi, Mathieu d’Aquin, Enrico Motta Knowledge Media Institute, The Open

Designing for Extensibility and Planning for Conflict ...djg/theses/BenLerner_thesis.pdfHTML to define the structure or content of the UI, CSS to define the appearance, and JavaScript

Identify properties of definite integrals Define odd and even …math121/Notes/Annotated_Online/... · 2012-06-01 · • Identify properties of definite integrals • Define

Semantic e-Science: From Microformats to Models · {Invoke external packages - e.g., to facilitate interaction with databases} {Accept input (if necessary)} {Define variables} {Define

Mikko Laamanen - UEF · Q4: Evaluation: Does the complete design meet the requirements? The means to define if the platform is suitable for its purpose. 1.2 Methodology My research

A Systematic Approach to Define the Domain of Information

zowe → cics → define...zowe → cics Interact with IBM CICS programs and transactions. zowe → cics → define Define new resources (for example, programs) to CICS through

INSTITUT D ESTUDIS CATALANS REPORTS DE LA … · 5.1. Patents ... CODALAB Control, ... ICC Institut Cartogràfic de Catalunya ICRA Fundació Institut Català de Recerca de l’Aigua

Taking VOC to the Next Level: Define,Differentiate,Disruptevents.roundtable.com/Event_Center/VOC04/VOC2004.pdfTaking VOC to the Next Level: Define, Differentiate, Disrupt, November

Quantitative Evaluation of Color Image Segmentation … · mentation algorithm which is based on regions. As an example, the edge information can be used to define the seed points

TrackMLthroughput challenge on CodaLab...•non-uniform magnetic fieldsimilar toATLAS solenoid •detailed simulation of particle interactions withdetector material •three types

Seismic performance evaluation of staggered wall system ... · factors, to define inelastic behavior of the structure are not specified. To evaluate the seismic performance of SWS

A Partisan Solution to Partisan Gerrymandering: The Define

CodaLab E velyne Viegas, Microsoft Research

messaging strategy How to define a customer Masterclass 4 · 2020. 6. 24. · Masterclass 4: How to define a customer messaging strategy @BuildingaSubBiz #SubBiz . 2 ... This chart

Design, Implementation and Evaluation of Merging ... · Structured peer-to-peer overlays define a robust and scalable alternative to the centralized server archi-tecture. Hence those

Speex Reference Manual · 5.1.2.5 #define SPEEX_GET_DTX 35 Get DTX status (1 for on, 0 for off) 5.1.2.6 #define SPEEX_GET_ENH 1 Get enhancement state (decoder only) 5.1.2.7 #define

How to Define Property Rights? A Social Documentation of

LinguisticAnnotationin/forCorpus Linguistics · LinguisticAnnotationin/forCorpus Linguistics ... Thisarticlesurveyslinguisticannotationincorporaandcorpuslinguistics.Wefirst define

An ontology-based approach to define and manage B2B

Daniel Blanchard Michael Heilman Nitin Madnani · In the Output section, we first define the additional evaluation metrics we want to compute in addition to the tuning objective

Dependable Systems Reliability Prediction · • Procedure of system reliability prediction [Misra] • Define system and its operating conditions • Define system performance

Evaluation: define MiroslavHruškaphoebe.inf.upol.cz/~hruskam/materials/define.pdf · Outline 1 define 2 (definex0) 3 (definex(+11)) 4 (define0x) 5 (definexx) 6 (defineyx)

define database(ddl) define database(ddl) - Firebird · define database(ddl) define database(ddl) NAME define database −create a database SYNTAX define database quoted-filespec[

How to define your CUSTOMER HEALTH SCORE

Writting you first linux kernel module - uni-hamburg.de · • Error number defini$ons at ... • #define ENOEXEC 8 /* Exec format error */ • #define

Direct observation of incommensurate structure in Mo3Si › Research › ...ActaCryst.pdf · one can define two vectors in reciprocal space which can span the whole space and define

Studies in Comtrace Monoidsledt/papers/thesis.pdfand comtrace monoids. We also define and study the canonical representations of these monoids. Our second contribution is to define

CD15, CD24, and CD29 Define a Surface Biomarker Code for ...neuroregeneration.org/CD15Pruszak.pdf · EMBRYONIC STEM CELLS/INDUCED PLURIPOTENT STEM CELLS CD15, CD24, and CD29 Define

The Yin Yang of Design and Anthropology · Learning intentions Define Yin Yang Define Anthropology, as itself and Yin energy Define Design, as itself and Yang energy Enumerate

Define Your Petrochemical Project Execution Plan … Your...Define Your Petrochemical Project Execution Plan Introduction ... Modularization, ... Define Your Petrochemical Project

Query shredding: Efficient relational evaluation of queries ... · normalisation phase in that we can define translations on types and terms independently. Section 5 shows how to