![Page 1: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/1.jpg)
| Log management as a service Simplify Log Management
Apache Storm
To Log or Not to Log: Proven Best Practices for Instrumentation
Jon Gifford Founder and Chief Search Officer January 2014
![Page 2: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/2.jpg)
| Log management as a service Simplify Log Management
About Me 15+ years of search experience
§ Lots of different search products § Lots and lots of machines § Lots and lots of logs...
Bias: § Lucene, Java, *nix, Distributed, Realtime, § Measure, Measure, Measure
![Page 3: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/3.jpg)
| Log management as a service Simplify Log Management
§ Centralized logging and archival
§ Real-time processing, analysis and visualization
§ Monitoring, alerting and troubleshooting
Loggly: Addressing the first big data problem every company faces
![Page 4: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/4.jpg)
| Log management as a service Simplify Log Management
To Log or Not to Log?
Lots of data to solve operational problems faster!
Incredibly painful to grep and inspect yourself + -
![Page 5: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/5.jpg)
| Log management as a service Simplify Log Management
Instrumentation: Your Most Valuable Log Data For us, instrumentation is the act of gathering performance data about your application and surfacing it via logs.
§ Provides a low-impact mechanism for gathering production system performance data
§ Visualization is key to understanding this data
![Page 6: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/6.jpg)
| Log management as a service Simplify Log Management
Best Practices for Instrumentation
![Page 7: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/7.jpg)
| Log management as a service Simplify Log Management
Best Practice #1 Treat instrumentation as an ongoing, iterative process.
§ Start by logging everything at a high level § Instrument deeper every time you ask, “What
happened here?” § Instrument deeper in areas that are more variable
or expensive than you thought
![Page 8: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/8.jpg)
| Log management as a service Simplify Log Management
Best Practice #2 Always instrument anything that goes out of process.
§ Can be indicative of larger problems, like the canary in the coal mine § Example: Instrument calls to
your database, even though they should always be fast
§ Distributed system problems are not well behaved – by definition
![Page 9: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/9.jpg)
| Log management as a service Simplify Log Management
Best Practice #3 Always log unacceptable performance.
§ Log “soft exceptions” (e.g. slow queries) § “Trust but verify”
![Page 10: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/10.jpg)
| Log management as a service Simplify Log Management
Best Practice #4 If possible, log enough context so you can get the complete picture of what happened from a single log event.
Process
Log this … not this!
Elapsed time End
End Start
![Page 11: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/11.jpg)
| Log management as a service Simplify Log Management
Best Practice #5 View machines, not people, as your logs’ consumer.
![Page 12: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/12.jpg)
| Log management as a service Simplify Log Management
Best Practice #6 Trends tell the story better than data points.
§ Graph performance over time to see daily fluctuations
§ Use rollups where it’s not practical to log every single event
![Page 13: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/13.jpg)
| Log management as a service Simplify Log Management
Best Practice #7 Instrumentation is NOT a substitute for profiling, and vice versa.
Instrumentation Profiling
Your production environment is not the same as development!
![Page 14: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/14.jpg)
| Log management as a service Simplify Log Management
Best Practice #8 Flying slower is better than flying blind.
§ “Speed advantage” is illusory if you can’t understand system performance
§ Our debate: not whether to instrument but how much
Phot
o by
Eri
c Ki
lby
![Page 15: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/15.jpg)
| Log management as a service Simplify Log Management
Implement Instrumentation Best Practices Faster and Better with Loggly.
About Us: Loggly is the world’s most popular cloud-based log management solution, used by more than 5,000 happy customers to effortlessly spot problems in real-time, easily pinpoint root causes and resolve issues faster to ensure application success.
Try Loggly for Free! → http://bit.ly/BestLoggingPractices
Visit us at loggly.com or follow @loggly on Twitter.
![Page 16: To Log or Not to Log: Proven Best Practices for Instrumentation - via @Loggly](https://reader034.vdocuments.us/reader034/viewer/2022051818/549f69c8ac795938768b49c9/html5/thumbnails/16.jpg)
| Log management as a service Simplify Log Management
Proven and Trusted by 5,000+ your peers…
“If we didn’t have Loggly, there would be days where our revenue would be affected by as much as 70 percent.”
– Albert Ho, Executive Producer/
Platform Product Manager, Rumble Entertainment