elicitation and utilization of application-level utility functions paul de grandis and giuseppe...
TRANSCRIPT
Elicitation and Utilization of Application-level Utility Functions
Paul de Grandis and Giuseppe ValettoDrexel University - Dept. of Computer Science
Outline
Motivation Goal Approach Engineering considerations Experiments Conclusions
Motivation
Autonomic provisions are often costly and complex
Many times, those provisions target only specific classes of problems expected or likely conditions
It’s not only a matter of requirements … … it’s also hard to develop complete system models
that enable general-purpose adaptations
The challenge
Unexpected conditions are often the most critical Typically occur only on the field By definition, they have not been accounted for by
designers, programmer, testers etc. Potentially devastating for the system
One could argue that autonomic provisions fulfill their promise when they can handle the unexpected
Designing for the unexpected
Our goal
Enable non-analytic self-assessment of SW applications
Self-assessment: map the state space of the application to some abstraction of value Recognize where the system on in that map
Non-analytic: do not require a complete model of the system in order to enable self-assessment
Two-steps strategy
Self-assessment
Self-adaptation
State space
“bad” state“good” states
Approach: application-based utility
Utility of an application
What is the app. value for, e.g., its client? A single scalar: Uapp [0,1]
Utility function: Uapp = f(V[s0,…,sn]) s0,…,sn are state variables
Network measures System measures Application specific measures User-observable measures
Our vision
Elicitation: construct Uapp for a given application Often, for a given task or feature
Embedding: insert runtime code to monitor state variables and compute Uapp
Utilization: continuously evaluate Uapp
We are interested in a method that can handle legacy applications as well as new ones
Our process
1. Identify utility from the client perspective
2. Instrument application and environment
3. Run experiments and observe V[s0, …,sn]
4. Synthesize Uapp(V[])
5. Substitute Uapp in the instrumented code
1. Manual, expert knowledge
2. Automated
3. Automated
4. Off-line, statistical correlation
5. Automated
Observation and synthesis We operate at multiple
distinct levels Typically, application-
vs. environment-level
Collect lots of lower-level monitoring data from the running environment …
… and relate it statistically to some externally visible manifestation of application goals
Environment
Application
Statistical correlation
Example – FTP service
Simple protocol-based application Utility is straightforward:
For an FTP client, utility = client throughput
Relevant environment (socket-level) parameters: Latency Bandwidth Loss rate …
Eliciting application utility
Environment level
Application level
U
t
Embedding application utility
Environment level
Application level
U
t
Engineering approach
Elicitation and embedding framework
Aupy: “Application Utility in Python” Framework leveraging meta-programming
capabilities of Python to instrument applications
Two flavors: Python annotations (legacy code-friendly) Python contexts (granular and flexible)
Open-sourced at: http://www.aupy.org
Aupy annotations
Added to method definitions @monitor(foo) bar triggers evaluation of utility function
“foo” when “bar” is executed
u.f.method declarations: @pre-utility foo @post-utility foo @utility foo @concurrent-utility foo
foo() can contain monitoring code (elicitation) or code computing Uapp (embedding)
Aupy contexts
Contexts are Python scoping constructs “with” keyword establishes a custom scope
Aupy uses them to associate utility functions to arbitrary blocks of code Finer-grained control
You can pass to a context utility function, arguments, configuration data, callback information …
Benefits of contexts
Contexts (and hence u.f.) can be composed
Multi-step u.f. Utility of a u.f. Support of selection logic among multiple
alternative u.f.
Experiments
FTP experiment set-up
In-the-lab data collection 125 test runs using dummynet to throttle network create a uniform map of environmental conditions
Two download scenarios: Bulk-ftp: transfer one large file Small-ftp: transfer multiple small files
Off-line data analysis and u.f. synthesis
FTP results
X=inter-arrival time Y= socket throughput Very different u.f. for the 2
scenarios We plugged in elicited u.f.
kept monitoring client throughput online and compared
€
Ubulk
= a1X1+ a2X2+ a4X4+ a5X5+ b1Y1+ b2Y2+ b5Y5
Usmall
= a4X4+ a5X5+ b1Y1+ b2Y2+ b4Y4+ b5Y5
Bulk-FTP Small-FTP
a1 -175150.7079 a1 0
a2 -145163.8736 a2 0
a3 0 a3 0
a4 -579.7485 a4 1028.3839
a5 2.0136 a5 -157.4976
b1 0.9241 b1 0.5368
b2 -0.2341 b2 -0.42717
b3 0 b3 0
b4 0 b4 612.90247
b5 -0.17755 b5 15.18621
FTP-bulk results
Estimated Actual
Another experiment – log explosion
Real-world case study Logging component of a Python-powered
popular WWW site gone awry Explosive growth of log would block the WWW
site in a few seconds Even prevent remote login from sysadmins
Could we apply non-analytic self-assessment to this problem?
Log explosion experiment
Phase 1: catching log explosions Utility of WWW site drops almost immediately to 0 upon
log explosion need to quickly capture acceleration peaks in disk writes
per process before disk resource is exhausted
Phase 2: preventing log explosions: hypothesis: utility is Uapp ≈ f(1-pn
boom)
pboom : probability that a log explosion is going to happen “some time soon”
Log explosion results
We were able to catch log explosions Captured log explosion dynamics for disk writes Write speed raises 2 orders of magnitude each second Uapp looks like a step function
Collapse occurs in a few seconds But we could detect the front and turn off logging in time
The prevention problem is more complex Would require very frequent sampling of disk write speed That interferes too much with normal operation We cannot really “see” preliminary signs and estimate “pboom”
Conclusions
Lessons learned
We have understood the elicitation process Repeatable Requires only a modicum of domain
knowledge
We have acquired know-how on the mechanisms for elicitation and embedding Can be made minimally intrusive Can be applied to legacy systems Layered approach is generally valid
Lessons learned
We have insight on the task-based nature of application utility
Need an array of u.f. that can be selected opportunistically On a per-feature basis Also based on different tasks
Idea applies especially well to: Protocol-based applications Infrastructure-level services
Research directions
Towards in-the-field data collection Deploy Aupy-instrumented application
Multiple features are exercised Assign clusters in the monitored space to
exercised features Elicit all feature-specific u.f. from the clustered
areas
Research directions Towards automated utility definition
Use cooperative negotiation and dependency discovery to figure which application characteristics to consider as utility
Leverage utility map to guide adaptations Reinforcement learning to compute trajectories
back from bad utility to good utility states Actionable trajectories: can be traversed by
tuning some state variables Utility delta as RL “reward”
Questions?