argonne leadership computing facility · 2018. 8. 17. · argonne leadership computing facility ♦...

Argonne LeadershipComputing Facility


L’intelligenza artificiale e’ intelligente come pensa di essere?


Is Artificial Intelligence As Intelligent As It Thinks?Why start a talk in Italian?


♦ It helps get your attention even while you’re eating dinner.

♦ It emphasizes we make assumptions (like an English talk), and base predictions on them, but we aren’t always accurate.§ Most people expect a talk at ATPESC to be

given in English. It is highly likely, but not certain.§ Helps us consider how humans make decisions.

Why start a talk in Italian?


♦ What can someone trained in econ and finance possibly say about AI that is useful to tech experts like you? Social scientists:§ Have an inclination to ask how human beings

think/act, and question the impacts on the models we create (like AI).

§ Analyze methods and their foundations for strengths and weaknesses.

Why an Economist?


♦ The opening question about AI’s intelligence is useful as a way to focus on qualities used to formulate AI.§ How well do humans model & predict?§ Can AI algorithms do better and do they

have potential shortcomings?But First: what do we mean by AI?

Our Goals Tonight


♦Many varying definitions.♦ Tonight’s view geared toward

machine abilities to analyze data, and make predictions. (decisions?)§ With a focus toward the underlying

algorithms.

AI

Graham School of General StudiesPrinciples of Economics

.


♦Consider briefly how humans analyze data and make decisions.§ How logical/rational are we?♦Do/can machines do better than

humans?

Our Starting Point


♦ Human theories, analyses, and decisions draw upon beliefs, assumptions, and historic data to attempt to predict the future.§ In many instances people analyze well

and outcomes are good.§ In some situations, use of heuristics (simple

time-saving rules) can lead to poor results.

How Humans Decide


♦ Some methods humans use (like heuristics) are prone to biases. § The term bias refers to incongruities.§ Many of these can be predicted. vPsychology and behavioral science.

• Richard Thaler’s 2017 Nobel Prize.

How Humans Decide


♦ It is hoped that by looking at predictable human biases we may better understand their impact on the underlying frameworks for the models we create.

How Humans Decide


Predictable Human Biases

Which table is longer and which is wider?


♦Most people choose the first table as longer and the second wider.§ Both are the same size, but many

people are certain they differ.§ The bias can be avoided by

measuring both. vAutomated measuring avoids the bias.




X

X

Y

Y


♦ There are many instances where humans can be fooled, but unbiased algorithms consistently applied to data are not.♦ The following shows another case

where people are often fooled.

Predictable Biases


Labor Participation Rate: Ages 25-54

0.010.020.030.040.050.060.070.080.090.0

Jan-

90Ja

n-91

Jan-

92Ja

n-93

Jan-

94Ja

n-95

Jan-

96Ja

n-97

Jan-

98Ja

n-99

Jan-

00Ja

n-01

Jan-

02Ja

n-03

Jan-

04Ja

n-05

Jan-

06Ja

n-07

Jan-

08Ja

n-09

Jan-

10Ja

n-11

Jan-

12Ja

n-13

Jan-

14Ja

n-15

Jan-

16Ja

n-17

Jan-

18

Predictable Biases

80.080.581.081.582.082.583.083.584.084.585.0

Jan-

90Ja

n-91

Jan-

92Ja

n-93

Jan-

94Ja

n-95

Jan-

96Ja

n-97

Jan-

98Ja

n-99

Jan-

00Ja

n-01

Jan-

02Ja

n-03

Jan-

04Ja

n-05

Jan-

06Ja

n-07

Jan-

08Ja

n-09

Jan-

10Ja

n-11

Jan-

12Ja

n-13

Jan-

14Ja

n-15

Jan-

16Ja

n-17

Jan-

18

Same data, different scalesData from Bureau of Labor Statistics website.


♦ Given our biases, how good are human forecasts?§ Experts predict worse than simple models.

v Overconfident, inconsistent, and expect more exceptions than observed (overconfidence bias).

v Not good at revising forecasts in light of changing data (status quo bias).• A big problem in project management.

Biases & Forecasting


¨ P. Tetlock (Penn) interviewed 284 political-economic experts (80,000 predictions).§ Performed worse than if equal probabilities

assigned to the three outcomes requested.§ Most experts had difficulty admitting error.§ The most famous experts were more

overconfident with less accurate predictions.

Political Forecasting Evidence


♦ Models based upon expert’s rules. § Predict better than most experts upon whom

they are based.v The algorithms are consistent, adapt better to

incremental changes in data, and reflect collective judgments of the best predicting experts.• Averaging probabilistic thinking by the experts in algorithms

gives the best results. The algorithms do not provide the probabilities upon which they are based. How might AI derive probabilities for instances where there is limited or no data?

Biases & Forecasting


♦ How good are our investment decisions?§ Most people have poor performance.

v Most investment returns are worse than the “market”.• Over optimism, fear, and greed lower overall results.

v Terrance Odean’s (Berkeley) analysis. • 10,000 individual brokerage accounts, ~163,000 trades.• On average after trading costs, shares sold did better than those

bought by 3.2% per year (with random results and no bias they would be about equal).

Biases & Financial Decisions


♦And the experts?§ Professional traders do better than

individual investors, but 70% - 80% still underperform the “market”*.

§ Top experts using algorithms with HPC have also failed.v Long Term Capital Management.

* averages represented by indices.



♦ How About Using AI?§ A recent article examines the impact of using

machine learning with decision trees and neural nets to predict stock prices*.v The authors conclude “that machine learning offers

an improved description of asset price behavior [stocks] relative to traditional methods”.

* Shihao Gu, Bryan T. Kelly, and Dacheng Xiu, “Empirical Asset Pricing via Machine Learning,” Working paper, April 2018. Examines 30,000 stocks traded between 1957 and 2016.



♦ Humans have biases that affect their decisions in predictable ways.

♦ Machines can improve results. § Able to avoid some common human biases.§ Predictable biases are left out of the process.

Human v. Machine


♦ Does that mean AI is free from bias?§ It is reasonable to ask:

v Are the assumptions free from bias?v Is the model’s framework free from bias?v Is the data free of bias?v Do the programs ignore all human biases?

§ And reasonable to note:v The algorithms and the models come from humans.

Human v. Machine


♦ Estimate the future value of a stream of payments.§ Affects everyone with a defined

contribution or defined benefit retirement plan.vA complex problem where the answer has

significant impact on retiree well-being .

Finance Example


♦ Most models start by looking at data for past returns and extrapolating forward.§ Which data is best as a basis for modeling?

v The accuracy and consistency of the data can change in different time periods for many reasons. • Measurement methods. • Company changes in the indices. • Other factors.

Estimating Future Value


♦ Projections change significantly depending on the data selected as the basis for estimates.vAverage returns and variations for returns can

differ markedly.vDifferent periods are influenced by many

factors, including starting and ending points of the data set relative to business cycles and relative asset prices.



♦ What outcomes are possible?§ Using different historic data as the basis for

modeling the future can show significantly different retirement savings outcomes* (hundreds of thousands of dollars or more) depending on the time period selected.v *For a worker with an average income, who saves

8% of salary per year (including employer contributions) over 30 years.



♦ Can AI help?§ It can operate without emotions or biases

born from fear, greed, over optimism, ...§ As you know, AI can analyze a lot of data,

look for correlations, test fits of data between time periods, assign probabilities, use simulations, adjust as new data becomes available, etc. But . . .

Using AI to Estimate


♦ The starting point of data for past returns can still introduce bias.§ Fundamental changes in the economy can

occur where the past is not a good indicator of the future (like disclaimers).v Trends not yet recognized or easily predicted.

• Technological changes, devastating war, etc.

v Black swan events.• Events and non events that are difficult to anticipate.



♦ The data itself can introduce bias.§ Quantitative Bias

v The ability to accurately measure returns can change over time and skew results.

v Spurious correlations occur. How to distinguish?v Focusing on measurable variables, quantitative v.

qualitative data. A bias?• How much is needed for the “proper” retirement lifestyle?

v Can impact all analysis including AI.


Can Quantitative Measures Tell Which is More Beautiful?


♦ The general modeling and data issues described are not unique.§ They are applicable to natural

science and other applications.

What Does It All Mean?


♦ There are many advantages of AI.§ The ability to analyze more data, more

accurately, and more efficiently.§ Minimizing human biases in

implementing models and forecasts.vNot as prone to optimism bias, status quo

bias, etc.

Conclusions


♦And some shortcomings.§ Fundamental structural changes may

not be captured well by the models.v“Revolutions” in scientific thought/theory.

§ Data problems can still exist.vMeasurement errors, spurious correlation, etc.

§ Algorithm constraints and biases.

Conclusions


♦ Is artificial intelligence as intelligent as it thinks it is?§ I conclude it is not. Why?§ Simply because it is still far from perfect due to

the human element. Humans:v Write the algorithms upon which AI is based.v Are susceptible to biases for the methods we create,

including AI.

Back to Our Opening Question


Special Thank You

Marta Garcia Martinez Carlo Graziani

Haritha Siddabathuni Som

Doug KotheVenkat Vishwanath

Beth CernyRichard Coffey Mike Papka

Nick Romero Donna Lingle

Thanks to the following people for helping me think about and refine tonight’s presentation. The biases and shortcomings are mine alone.


This presentation is supported by the Argonne Leadership Computing Facility at Argonne National Laboratory, which is in turn supported by the Office of Science of the U.S. Department of Energy, Office of Science, under contract number DE-AC02-06CH11357.

Acknowledgements


Questions?

argonne leadership computing facility · 2018. 8. 17. · argonne leadership computing facility ♦...

Documents