uncertainty estimates of psychoacoustic thresholds ...€¦ · bootstrap: nonparametric 82.6...
TRANSCRIPT
![Page 1: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/1.jpg)
1
Uncertainty estimates of psychoacoustic thresholds obtained from group tests
National Aeronautics and Space Administration
Spring 2016 Meeting of the Acoustical Society of America
Salt Lake City, UT
May 24, 2016
Jonathan Rathsam
Andrew Christian
https://ntrs.nasa.gov/search.jsp?R=20160009124 2020-06-22T18:46:13+00:00Z
![Page 2: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/2.jpg)
Acknowledgments
• NASA Commercial Supersonic Technology Project– Jacob Klos, Alexandra Loubeau, Jerry Rouse, Kevin Shepherd
• NASA Environmentally Responsible Aviation Project– Steve Rizzi, Russ Thomas, Casey Burley
2
![Page 3: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/3.jpg)
Outline
1. Research motivation
2. Confidence interval estimation methods
a. Bayesian Posterior Estimation
b. Bootstrap (Parametric and Non-Parametric)
c. Delta Method
3. Results
4. Conclusion
3
![Page 4: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/4.jpg)
Research motivation
• Find threshold for binary outcome:– Defaulting on credit card debt (yes/no)
• based on monthly balance
– Projectile pierces armor (yes/no)• based on projectile velocity
– Subjects find test signal more annoying than reference signal• based on test signal level
• Two research groups with same question
4
![Page 5: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/5.jpg)
Aircraft Auralizations
Reference
5
Test
![Page 6: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/6.jpg)
Test Method
6
![Page 7: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/7.jpg)
Test Method
7
Pr 𝑦𝑖 = 1 =1
1 + 𝑒−(𝛽0+𝛽1𝑥)
![Page 8: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/8.jpg)
Test Method
8
Pr 𝑦𝑖 = 1 =1
1 + 𝑒−(𝛽0+𝛽1𝑥)
![Page 9: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/9.jpg)
Test Method
9
Point of Subjective Equality
(PSE)
PSE =−𝛽0𝛽1
Pr 𝑦𝑖 = 1 =1
1 + 𝑒−(𝛽0+𝛽1𝑥)
![Page 10: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/10.jpg)
Test Method
10
ConfidenceInterval
Point of Subjective Equality
(PSE)
![Page 11: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/11.jpg)
Research Question
• What is most appropriate interval estimation technique?
a. Bayesian Posterior Estimation
b. Bootstrap: non-parametric
c. Bootstrap: parametric
d. Delta Method
11
![Page 12: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/12.jpg)
Bayesian Posterior Estimation
12
• Begin with data and best fit…
![Page 13: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/13.jpg)
Bayesian Posterior Estimation
13
Blue line fit is poorer than black line,but still reasonable
![Page 14: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/14.jpg)
Bayesian Posterior Estimation
14
![Page 15: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/15.jpg)
BPE numerically samples likelihood function…
allowing confidence intervals to be constructed
anywhere along logistic probability curve
Bayesian Posterior Estimation
15
![Page 16: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/16.jpg)
BPE numerically samples likelihood function…
allowing confidence intervals to be constructed
anywhere along logistic probability curve
Bayesian Posterior Estimation
16
![Page 17: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/17.jpg)
Bayesian Posterior Estimation
17
All possible parameter combinations with corresponding goodness-of-fit yield “likelihood function”
![Page 18: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/18.jpg)
Bayesian Posterior Estimation
• BPE can include background knowledge (if known) in the form of “prior distributions”
• Previously posterior could only be evaluated when likelihood and prior known analytically
• MCMC methods enable numerical evaluation of arbitrary likelihoods/priors
18
𝑝 𝛽0, 𝛽1|𝐷𝑎𝑡𝑎 ∝ 𝐿 𝐷𝑎𝑡𝑎 𝛽0, 𝛽1 ∗ 𝑝 𝛽0, 𝛽1
Posterior Likelihood Prior
![Page 19: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/19.jpg)
b. Bootstrap Analysis: Nonparametric
19
![Page 20: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/20.jpg)
20
Bootstrap Analysis: Non-parametric
• What if we ran this experiment 10,000 times?
![Page 21: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/21.jpg)
21
Bootstrap Analysis: Non-parametric• Resample dataset with replacement
• Each resample uses slightly less than entire dataset
![Page 22: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/22.jpg)
22
Bootstrap Analysis: Non-parametric
![Page 23: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/23.jpg)
23
Bootstrap Analysis: Non-parametric
![Page 24: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/24.jpg)
24
Bootstrap Analysis: Non-parametric
![Page 25: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/25.jpg)
c. Bootstrap Analysis: Parametric
25
![Page 26: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/26.jpg)
Bootstrap Analysis: Parametric
26
1) Fit data using maximum likelihood method (output is 𝛽0, 𝛽1, and Cov 𝛽0, 𝛽1 )
2) Use output to construct multivariate distribution
![Page 27: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/27.jpg)
27
Bootstrap Analysis: Parametric1) Fit data using maximum likelihood method (output
is 𝛽0, 𝛽1, and Cov 𝛽0, 𝛽1 )
2) Use output to construct multivariate distribution
3) Sample from multivariate distribution
![Page 28: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/28.jpg)
Bootstrap Analysis: Parametric
28
![Page 29: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/29.jpg)
Bootstrap Analysis: Parametric
29
![Page 30: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/30.jpg)
d. Delta Method
30
![Page 31: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/31.jpg)
Delta Method: Theory
31
Taylor Series Approximation to Variance of PSE [Morgan 1992]
Var PSE =1
𝛽12 Var 𝛽0 + PSE2 ∗ Var 𝛽1 + 2 ∗ PSE ∗ Cov 𝛽0, 𝛽1
Delta Method Confidence Interval
PSE ± 𝑧1−
𝛼
2
Var 𝑃𝑆𝐸
The GLM logistic regression model returns:
• 𝛽0, 𝛽1 -- maximum likelihood estimators of logistic regression parameters
• Cov 𝛽0, 𝛽1 -- Covariance of parameters
![Page 32: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/32.jpg)
Results
• All methods gave the same results!
• PSE = -2.44 dB
• 95% CI [-3.26, -1.62] dB
32Bayesian Posterior Estimation Parametric Bootstrap
![Page 33: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/33.jpg)
Results: Guidance Table
33
Method Notes
Bayesian Posterior Estimation
•Most flexible (can include prior information)•Uses all data for calculating likelihood•Diagnostics needed to ensure proper numeric performance
Bootstrap:Nonparametric
•Takes longest to calculate (10,000x as long as Delta Method)•Most affected by low-N binomial data
Bootstrap:Parametric
•Observable failure modes (e.g. negative slope)
Delta Method •Closed form•Assumes confidence interval is symmetric about PSE•Unobservable failure modes
![Page 34: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/34.jpg)
Conclusions
• Bayesian and Frequentist concepts yield same results
• What is most appropriate interval estimation technique among four standard solutions?
-All methods yield equivalent results
-Delta Method is fastest to calculate
-BPE is most complex (pros and cons)
34
![Page 35: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/35.jpg)
Thank You
35
Reference:
• Morgan, B.J.T. Analysis of Quantal Response Data London: Chapman & Hall (1992).
![Page 36: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/36.jpg)
Backup Slides
36
![Page 37: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/37.jpg)
Bayesian Posterior Estimation
37
![Page 38: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/38.jpg)
Results: Guidance Table
38
Method PSEPSE Intervalmin—max
Longest Operation
Notes
Delta 82.6 81.3—83.9 1 GLM fit(fastest)
•Closed form•Unknown failure modes
Bootstrap:Parametric
82.6 81.2—83.9 Sorting N resampled PSEs(2nd fastest)
•Resamples are normally distributed•Observable failure modes (e.g. negative slope)
Bootstrap:Nonparametric
82.6 81.3—83.9 N GLM fits (slowest)
•Fewest assumptions•Not suitable for low-N binomial data
Bayesian Posterior Estimation
82.6 81.4—83.9 N likelihood evaluations (2nd slowest)
•Most flexible (can include prior information)•Diagnostics needed to ensure proper MCMC performance
![Page 39: Uncertainty estimates of psychoacoustic thresholds ...€¦ · Bootstrap: Nonparametric 82.6 81.3—83.9 N GLM fits (slowest) •Fewest assumptions •Not suitable for low-N binomial](https://reader034.vdocuments.us/reader034/viewer/2022042318/5f0709197e708231d41af9eb/html5/thumbnails/39.jpg)
39
Bootstrap Analysis: Non-parametric