inria, nice. december 7 th -8 th 2006 evaluation protocol evaluation process

INRIA, Nice. December 7th-8th 2006

EvaluationEvaluation protocolprotocol

Evaluation processEvaluation process


EvaluationEvaluation protocolprotocol

ScheduleSchedule

AgreementAgreement

ResultsResults submissionsubmission

File format transfer toolFile format transfer tool

FeedbackFeedback


Results collection

AprilJan July Oct

Participants test

Corpus diffusion

First evaluationSeminar

DecJune

Results analysis

Evaluation review

CYCLE 1

Corpus, metrics & Gt review

Results analysis

Evaluation review

Data Collection

CYCLE 2 Seminar

Results collection

Participants test

Corpus diffusion

Etiseo evaluation

ScheduleSchedule


ETISEO ProjectETISEO Project

ScheduleSchedule

AgreementAgreement

Results submissionResults submission

FileFile formatformat transfertransfer tooltool

FeedbackFeedback


AgreementAgreement

- Actors :

The participant & the coordinator (Silogic)- To participate to the ETISEO project evaluation cycle, the participant will receive specific resources containing video sequences and meta-data files.

The agreement describes the terms of the exploitation of the resources, accepted by the participant.

The coordinator will transmit the resources to the participant after the signature of the agreement.


AgreementAgreement

- The participant agrees : - To use the resources only for research and development of video surveillance evaluation purpose.- Not to use the resources for commercial purposes.

- Not to diffused the resources to others outside the participant’s research group.

- To carry out the tests of its algorithms on the resources. - To send to the coordinator the results of the tests, in due time, in a suitable XML format using the defined ontology.


AgreementAgreement

- The participant is authorised :

- To make scientific publications based on the exploitation of the resources, under the condition of mentioning the ETISEO project.

- Small part of the resources can be displayed in a research environment, under the condition of mentioning the ETISEO project.


AgreementAgreement

- The participant authorises : - The coordinator to exploit the results in comparison with the ETISEO ground-truth to compute the evaluation. - The coordinator to diffused the evaluation results in an anonymous way.

- The coordinator to diffuse the evaluation results to INRIA-ORION team, which is the ETISEO scientific leader, for the evaluation analysis.

- The coordinator has to send back to the participant the results of the evaluation concerning its own results.



ScheduleSchedule

AgreementAgreement

ResultsResults submissionsubmission

FileFile formatformat transfertransfer tooltool

FeedbackFeedback


Results Results format -1format -1

Result Submission Format :

• Based on PETS-metrics XML schema, with the addition of events.

• Some attributes are not processed (segmentation = shape).

• Documentation on this format is available on the web site also with examples.



Result Submission Format :

Extended PETS metrics

developper details

algorithm details

run date

camera used

results

frame

tracked_target

info_2D

segmentation

row_seg

object

info_3D

parent target

event_target

physical_object

contextual_object

speed

Must occur once and only once in the parent element.

Must occur at least once.

Optional but can occur once and only once in the parent element.

Optional to occur but can occur numerously.



Based on VIPER using ETISEO attributes.<?xml version="1.0" encoding="UTF-8"?><viper xmlns="http://lamp.cfar.umd.edu/viper#" xmlns:data="http://lamp.cfar.umd.edu/viperdata#">

<config> <…></config><data>

<sourcefile filename="file:ETI-VS1-CR-4-C1.avi"><object framespan="1:100" id="4" name="physical_object">

<attribute name="nature"><data:svalue value="physical_object_of_interest"/>

</attribute><attribute name="type">

<data:svalue value="person"/></attribute><attribute name="info2D">

<data:bbox framespan="1:1" height="182" width="86" x="138" y="59"/><data:bbox framespan="2:2" height="183" width="86" x="139" y="59"/>

<data:bbox framespan="3:3" height="185" width="86" x="141" y="60"/> <data:bbox framespan="4:17" height="187" width="87" x="143" y="60"/> <…>

<data:bbox framespan="40:100" height="212" width="10" x="3" y="10"/></attribute><…>

</sourcefile></data>

</viper>



• Id• Start time• Type (ex: person,

vehicle…)• Subtype (ex: car,

van…)• Info2D• Info3D

• Links• Speed• GT Nature• GT Status• GT Occluded (partially or

completely)• GT Connected with

Physical object

GT : specific to annotation purpose



• Id• Name• Start and end time• Status• Involved objects (physical object of interest

and contextual object)

Event


Results Results contentcontent

Submission content :

- For each sequence you processed, its xml results file using the naming convention:

sequence-name.xml

- The questionnaire describing how your system proceeded on all sequences.



Submission method :

- by email (<10Mo) to: [email protected]

- on ftp www.silogic.fr/etiseo. Connection parameters will be provide to all participants. Send also a notification to [email protected]. Warning : other participants can have access to your results.- by mail to:

Silogic, ETISEO project.Attention of Luc Barthelemy6, rue Camboulives BP 113331036 Toulouse Cedex 1France



Submission method :

In any case, you received an acknowledgement to your submission.



ScheduleSchedule

AgreementAgreement


File format transfer toolFile format transfer tool

Evaluation feed-backEvaluation feed-back


Evaluation feedbackEvaluation feedback

Evaluation feedback :

Metrics results are sent back to participants:• res file•Example: xls file1, xls file2, xls file3

• Multiple submission is allowed.

• Statistics on metrics for each sequence available on the web-site.

• Global feedback to all participants.

inria, nice. december 7 th -8 th 2006 evaluation protocol evaluation process

Documents