ats post-mortem debugging
TRANSCRIPT
![Page 1: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/1.jpg)
ATS Post-Mortem DebuggingAbel Mathew
https://backtrace.io; @0xCD03
![Page 2: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/2.jpg)
Debugging
● In-situ
● Logging
● Tracing / Profiling
● Post-Mortem
● Flexibility
● Verbosity
● Overhead (run-time, space,
post-processing)
● Type of failure
○ Explicit / Implicit
○ Fatal / Non-Fatal
![Page 3: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/3.jpg)
Instrumentation
● Dump mem info periodically
○ CONFIG proxy.config.dump_mem_info_frequency INT
<value>
■ dump mem info to traffic.out every
<value> seconds
● Debug Tags output to traffic.out
○ CONFIG proxy.config.diags.debug.enabled INT 1
○ CONFIG proxy.config.diags.debug.tags STRING
<tag-name>
○ -T<tag-name>
![Page 4: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/4.jpg)
Post-Mortem Debugging
Pros:
● Rich data set
● Robust data capture
● Overhead only at the time of error
● Allows for powerful tooling
Cons:
● State at a single point in time
● Large data artifacts
● Lack of useful tooling, documentation
![Page 5: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/5.jpg)
![Page 6: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/6.jpg)
Post-Mortem Debugging for ATS
Detection, Capture
Triage, Diagnosis
Fix
Release
![Page 7: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/7.jpg)
CONFIG proxy.config.crash_log_helper
Default: traffic_crashlog, if
remote_unwinding enabled
Forks child process configured application
at startup.
Child waits and wakes up (SIGCONT) when
parent receives SIGBUS, SIGSEGV, SIGILL,
SIGTRAP, SIGFPE, SIGABRT. traffic_server
then pauses
By default logs to
crash-%Y-%m-%d-%H%M%S.log
Enable coredumps: CONFIG proxy.config.core_limit
INT -1
Detection, Capture
![Page 8: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/8.jpg)
CONFIG proxy.config.crash_log_helper
Default: traffic_crashlog, if
remote_unwinding enabled
1. Startup Death Spiral? Zombie children
a. prctl(PR_SET_PDEATHSIG, signum, 0, 0,
0)
2. libunwind, remote unwinding,
waitpid()
3. Unwinding, capture in ATS
4. Variables? Deeper Analysis?
Detection, Capture
![Page 9: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/9.jpg)
Triage, Diagnosis
Fix, Release
![Page 10: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/10.jpg)
Post-Mortem Debugging for ATS?
Detection, Capture
Triage, Diagnosis
Fix
Release
![Page 11: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/11.jpg)
Error Management:CaptureNotificationAggregationAnalysis (in-depth & at-large)
![Page 12: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/12.jpg)
CONFIG proxy.config.crash_log_helper
New: backtrace-invoker
https://github.com/backtrace-labs/invoker
Drop-in replacement for traffic_crashlog
Generic invoker, allows to configure
multiple tracers , customer args
Used to invoke Backrace’s snapshot
generator
● Everything traffic_crashlog does○ sans records
● Local + Automatic Variables
● System Stats
● Extensible
Detection, Capture
![Page 13: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/13.jpg)
● Deduplication
● Aggregation
● Tracking, regressions
● Integrate data into third party
systems
● Convenient investigation,
collaboration
Detection, CaptureTriage, Diagnosis
![Page 14: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/14.jpg)
![Page 15: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/15.jpg)
Future Work: ATS● Write an extension for Backtrace snapshot generator to capture ATS “state
machine” at the time of fault/error.
![Page 16: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/16.jpg)
Resourceshttps://github.com/backtrace-labs/invoker
https://docs.trafficserver.apache.org/en/latest/developer-guide/debugging/index.en.html
https://backtrace.io/blog
![Page 17: ATS Post-Mortem Debugging](https://reader031.vdocuments.us/reader031/viewer/2022012510/61884458aaecf079665a4505/html5/thumbnails/17.jpg)
ATS Post-Mortem DebuggingAbel Mathew
https://backtrace.io; @0xCD03