custom models in no time: how active learning is changing ai€¦ · custom models in no time: how...

Kfir BarChief Scientist

Custom Models in No Time:

How Active Learning is Changing AI

HLTCon2018

2

Supervised machine learning

➔ Teaching a machine to classify items,

by showing it pairs of items and their

true labels

Apple

Karambola

Grapefruit

Cherry

3

Peach

Fruit detector

4

Supervised machine learning

GrapesBanana

Apple

Orange

Training set

Training

5

What is this?

6

Supervised machine learning

Sample

?

It’s a cherry

Prediction

7

How training sets get created?

➔ We need domain experts

➔ We need domain experts

➔ and we need to pay them...

8

How training sets get created?

9

Standard (static) annotation process

Collect unlabeled data (e.g., fruits)1

Select an item, randomly2

Assign a label 3

Train and evaluate a classifier4

Not happy with the results?

10

Did I see this item before?

➔ Randomly selecting an item for annotation

might be suboptimal

➔ The dataset could be unbalanced

➔ For example, looking for tweets that mention the word

“move” in the sense of “moving to a new apartment”

11

Unbalanced datasets - document classification

10%

90%New apartment

Other

mood: wanna move to another city and start a new life

Might move to another house next month !

➔ For example, looking for tweets that mention the word

“move” in the sense of “moving to a new apartment”

12

Unbalanced datasets - document classification

10%

90%New apartment

Other

mood: wanna move to another city and start a new life

Might move to another house next month !

Nice/bad move

Move on

13

Introducing Active Learning

➔ We train a classifier as we annotate

➔ We keep training the classifier with

every new annotated item

➔ The classifier helps us select the most important item to annotate next, the one

that has the biggest impact on the

classifier

14

Introducing Active Learning

➔ Instead of picking up many apples, the classifier will select

the most difficult items

15

Introducing Active Learning

➔ Given 400 random points, distributed in a 2-D space. Half are green squares and half are red triangles

➔ We want to train a classifier to distinguish between the two groups

➔ Let’s say we train the classifier with only 30 points (out of the 400)

16

Introducing Active Learning

Random selection of 30 points

Classifier selects the 30 most difficult points

70% Accuracy 90% Accuracy

➔ Builds a classifier behind the scene

➔ Selects the most difficult items for annotation

➔ Serves items with suggested labels, so you only need to correct them, when needed, and submit

➔ Provides a dashboard, where you can see performance metrics change as you annotate more items

17

New Active-Learning annotation tool

For now, it supports Named Entity Recognition (NER).

Next on the list: Document Classification and Sentiment Analysis

18

Automatically find names of people,

organizations, locations, and more in text

across many languages.

First NLP task: Named entity recognition (NER)

According to Elon Musk, Mars rocket will

fly ‘short flights’ next year.

19

20

Example

21

NER Experiment w/ and w/o Active Learning

80%

200K 700K

Accuracy

# Words

Random Selection

Active Learning

22

How does it work?

Upload unlabeled documents

1

23

How does it work?

Annotate a few sentences.

When there are enough annotated sentences, the system trains a classifier

2

24

How does it work?

Then, the classifier predicts labels for the remaining unlabeled sentences.

Each sentence is assigned with a confidence score

30.7 0.9 0.3

0.5 0.50.5

0.5 0.5

0.2

0.1

0.00.8

0.3

0.50.7

0.1

0.3 0.4

0.60.5

25

How does it work?

Next sentence to annotate will be the one with minimum confidence

40.7 0.9 0.3

0.5 0.50.5

0.5 0.5

0.2

0.1

0.00.8

0.3

0.50.7

0.1

0.3 0.4

0.60.5

26

How does it work?

0.7 0.9 0.3

0.5 0.50.5

0.5 0.5

0.2

0.1

0.00.8

0.3

0.50.7

0.1

0.3 0.4

0.60.5

27

Active deep learning

+

LSTM

Washington

LSTM

+

LSTM

said

LSTM

+

LSTM

in

LSTM

+

LSTM

Chicago

LSTM

+

LSTM

last

LSTM

...

OTHER OTHER B-LOC OTHER0.1, 0.5, 0.01, …, 0.08

B-PER

28

Active deep learning

+

LSTM

Washington

LSTM

+

LSTM

said

LSTM

+

LSTM

in

LSTM

+

LSTM

Chicago

LSTM

+

LSTM

last

LSTM

...

0.1, 0.5, 0.01, …, 0.08

B-PER

0.01, 0.2, …, 0.2 0.01, 0.2, …, 0.2 0.6, 0.2, …, 0.2 0.01, 0.2, …, 0.2

OTHER B-LOC

➔ We can use the new annotation tool to adjust an existing model for a specific task

➔ For example, start with our English NER model and adjust it to work on legal contracts

29

Adjusting existing models

Questions?

Thank you!

kfir@basistech.com

@kfirbar