Programming Abstractions for Approximate Computing
Michael Carbin
with Sasa Misailovic, Hank Hoffmann, Deokhwan Kim, Stelios Sidiroglou, Martin
Rinard
MIT
Challenges for Programming
• Expression (specifying approximations)
• Reasoning (verifying resulting program)
• Debugging (reproducing failures)
• Deployment (changes in assumptions)
Challenges for Programming
• Expression (specifying approximations)
• Reasoning (verifying resulting program)
• Debugging (reproducing failures)
• Deployment (changes in assumptions)
Fundamental Questions
• Scope• What do we approximate and how?
• Specifications of “Correctness” • What is correctness?
• Reasoning• How do we reason about correctness?
What do we approximate?
• Domains have inherent uncertainty
• Execution time dominated by a fraction of code
Benchmark
Domain LOC LOC (Kernel)
Time in Kernel (%)
sor numerical 173 23 82.30%
blackscholes
finance 494 88 65.11%
dctimage processing
532 62 99.20%
idct 532 93 98.86%
scale 218 88 93.43%
[Misailovic, Carbin, Achour, Qi, Rinard MIT-TR 14]
Approximation Transformations
• Approximate HardwarePCMOS, Palem et al. 2005; Narayanan et al., DATE ’10; Liu et al. ASPLOS ’11; Sampson et al, PLDI ’11; Esmaeilzadeh et al. , ASPLOS ’12, MICRO’ 12
• Function SubstitutionHoffman et al., APLOS ’11; Ansel et al., CGO ’11; Zhu et al., POPL ‘12
• Approximate MemoizationAlvarez et al., IEEE TOC ’05; Chaudhuri et al., FSE ’12; Samadi et al., ASPLOS ’14
• Relaxed Synchronization (Lock Elision)Renganarayana et al., RACES ’12; Rinard, HotPar ‘13; Misailovic, et al., RACES ’12
• Code PerforationRinard, ICS ‘06; Baek et al., PLDI 10; Misailovic et al., ICSE ’10; Sidiroglou et al., FSE ‘11; Misailovic et al., SAS ‘11; Zhu et al., POPL ‘12; Carbin et al. PEPM ’13;
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;
Example: Loop Perforation
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;float avg = (sum * 2) / n;
Example: Loop Perforation
i += 2
• Skip iterations (or truncate or random subset)
• Potentially add extrapolations to reduce bias
TraditionalTransformation
≡
.c .c
What is correctness?
ApproximateTransformation
.c .c
What is correctness?
What is correctness?
• Safety: satisfies standard unary assertions (type safety, memory safety, partial functionality)
• Accuracy: program satisfies relational assertions that constrain difference in results
Relational Assertions
• Contribution: program logic and verification system that supports verifying relationships between implementations
• x<o>: value of x in original implementation
• x<a>: value of x in approximate implementation
relate |x<o> - x<a>| / x<o> <= .1;
[Carbin, Kim, Misailovic, Rinard PLDI’12, PEPM ‘13]
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;float avg = (sum * 2) / n;
Example: Loop Perforation
i += 2
• Worst-case error:
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;float avg = (sum * 2) / n;
Example: Loop Perforation
i += 2
Novel Safety Verification Concept
• Assume validity of assertions in original program
• Use relations to prove that approximation does not change validity of assertions
• Lower verification complexity than verifying an assertion outright
assert (safe(*p))
p<o> == p<a>
safe(*p<o>)
safe(*p<a>)
∧ ⊨
Specifications and Reasoning
• Worst-case (for all inputs)• Non-interference [Sampson et al., PLDI ‘11]• Assertions [Carbin et al., PLDI ‘12; PEPM ‘13]• Accuracy [Carbin et al., PLDI ‘12]
• Statistical (with some probability)• Assertions [Sampson et al., PLDI ‘14]• Reliability [Carbin, Misailovic, and Rinard,
OOPSLA ‘13]• Expected Error [Zhu, Misailovic et al., POPL
‘13]• Probabilistic Error Bounds [Misailovic et al.,
SAS ‘13]
Questions from Computer Science
Question #1:
“Programmers will never do this.”- Unnamed Systems and PL Researchers
ChallengeR
easo
nin
g
Com
ple
xit
y
Types
Partial Specifications
Full FunctionalCorrectness
Expressivity
ChallengeR
easo
nin
g
Com
ple
xit
y
Types
Partial Specifications
Full FunctionalCorrectness
Expressivity
• Ordering of unary assertions still true (improved by relational verification)
Challenge
Worst-case AccuracyReliability
Probabilistic Accuracy BoundsProbabilistic Assertions
Error DistributionsExpected Error
Reaso
nin
g
Com
ple
xit
y
Expressivity
Performance/Energy Benefit
Assertions
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;float avg = (sum * 2) / n;
Example: Loop Perforation (Misailovic et al., SAS ‘11)
i += 2
• Worst-case error:
• Probabilistic Error Bound (.95):
float sum = 0;for (int i = 0; i < n; i += 1) { sum = sum + a[i];} float avg = sum / n;float avg = (sum * 2) / n;
Example: Loop Perforation (Misailovic et al., SAS ‘11)
i += 2
Question #2:
“There’s no hope for building approximate hardware.”- Unnamed Computer Architect
Challenge
• Approx. PL community must work with hardware community to collaborate on models (not the PL community’s expertise)
• Approx. hardware community faces major challenge in publishing deep architectural changes: simulation widely panned
• Moving forward may require large joint effort
Question #3:
“I believe your techniques are fundamentally flawed.”- Unnamed Numerical Analyst
Challenge
• Numerical analysts have been wronged• Programming systems have failed to
provide support for making the statements they desire
• Approximate computing community risks repeating work done by numerical analysis• Opportunity for collaboration
• New research opportunities• It’s no longer just about floating-point
Broadly Accessible Motivations• Programming with uncertainty
(uncertain operations and data)
• Programming unrealizable computation (large scale numerical simulations)
• Useful computation from non-digital fabrics(analog, quantum, biological, and human)Opportunity to build reliable and
resilient computing systems built upon
anything
End
Conclusion
• Adoption hinges on tradeoff between expressivity, complexity and benefit
• Opportunity/necessity for tighter integration of PL and hardware communities
Open Challenges and Directions
Problem: benefits (performance/energy) may vary between different types of guarantees
Worst-case Accuracy
Reliability
Probabilistic Accuracy Bounds
Probabilistic Assertions
Error DistributionsExpected AccuracyReaso
nin
g
Com
ple
xit
y
Expressivity
Open Challenges and Directions
Expressivity and Reasoning Complexity
Type Checking
Partial Specifications
Full FunctionalCorrectness
Experimental Results
Benchmark LOC LOC (Kernel) Time in Kernel(%)
sor 23 173 82.30%
blackscholes 88 494 65.11%
dct 62 532 99.20%
idct 93 532 98.86%
scale 88 218 93.43%
Open Challenges Directions
• Traditional Tradeoff
Reasoning Complexity
Performance/Energy
Type Checking
Partial Specifications
Dynamic Analysis
Full FunctionalCorrectness