application logging for large systems
DESCRIPTION
Logging what is taking place in your application provides important insights into applications at runtime, and can be extremely helpful to support analysts when something goes wrong. We'll discuss best practices about what, why, and how to log and what to avoid when logging, as well as where write your logs and how to decide what logging framework to use. Time permitting, we'll talk about log analysis tools to help make sense of large log files as well.TRANSCRIPT
![Page 1: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/1.jpg)
©2010 Improving Enterprises, Inc.
Logging For Fun and Profit
Jane Prusakova
[email protected]://softwareandotherthings.blogspot.com
Improving Enterprises
Houston JUG
2014
![Page 2: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/2.jpg)
©2010 Improving Enterprises, Inc.
Jane Prusakova
Hard-core developer
Data science geek
Dancer and photographer
![Page 3: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/3.jpg)
©2010 Improving Enterprises, Inc.
Why
How-to
Using the results
So lets talk about logging…
![Page 4: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/4.jpg)
©2010 Improving Enterprises, Inc.
Peek into the soul
Real setup
Real data
Real users
Real workDev environment
Happy path
Minimum load
Limited data
![Page 5: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/5.jpg)
©2010 Improving Enterprises, Inc.
Inside the cloud
No programming-by-coincidence
Replaces debugging
Works on server
![Page 6: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/6.jpg)
©2010 Improving Enterprises, Inc.
Logging [vs Debugging]
Set up once, always ready to use
Minimum disruption to the flow
Works locally and in the cloud
Faster data collection
Collected data is persisted
![Page 7: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/7.jpg)
©2010 Improving Enterprises, Inc.
Logging helps with…
Integration
Multithreading
High load
![Page 8: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/8.jpg)
©2010 Improving Enterprises, Inc.
What else?
Trace what the users do
Features
Sequences
Data flow
![Page 9: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/9.jpg)
©2010 Improving Enterprises, Inc.
When to use logging?
New systems
New features
Bug fixing
![Page 10: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/10.jpg)
©2010 Improving Enterprises, Inc.
Are logs forever?
Permanent
Errors
Extraordinary situations
![Page 11: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/11.jpg)
©2010 Improving Enterprises, Inc.
Some are seasonal
Very temporary
Debugging info
Trace bug fixes
![Page 12: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/12.jpg)
©2010 Improving Enterprises, Inc.
Others last awhile
Medium term
Data flow
Resource utilization
Optimization points
![Page 13: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/13.jpg)
©2010 Improving Enterprises, Inc.
How-to
Using the results
![Page 14: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/14.jpg)
©2010 Improving Enterprises, Inc.
Yes, I want to setup logging!
Pick a framework
Configure logging
Storage and access
Getting value from logs
![Page 15: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/15.jpg)
©2010 Improving Enterprises, Inc.
Log4J SLF4J
Apache MIT license
Logging framework Logging facade
Configuration, log levels and categories,
rotate log files, thread-safe logging.
Support: online documentation, tutorials,
online forums.
![Page 16: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/16.jpg)
©2010 Improving Enterprises, Inc.
Database Files
Easy to parse,
hard to evolve
Very hard to parse,
easy to change
Uses DB
connections
Requires FS
access
Can cause app
crash, loss of logs,
slow down app
Can fill up space
![Page 17: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/17.jpg)
©2010 Improving Enterprises, Inc.
Where to write logs?
Console
File system
DB
Queue mechanisms
Combination
![Page 18: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/18.jpg)
©2010 Improving Enterprises, Inc.
Logging configuration
Java properties-style
XML
Programmatically
Hard-coded
Takes preference
![Page 19: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/19.jpg)
©2010 Improving Enterprises, Inc.
Log Message
What to log with the message
Importance level
Timestamp
Where in the code
Data being processed
System state info
![Page 20: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/20.jpg)
©2010 Improving Enterprises, Inc.
Example layout
log4j.appender.stdout.layout.ConversionPattern=
%d %5p [%t] (%F:%L) - %m%n
- Date
- Log Level
- Thread name
- File and line (slow to retrieve)
![Page 21: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/21.jpg)
©2010 Improving Enterprises, Inc.
Good log messages
Support code
Correctness
Readability
Performance
![Page 22: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/22.jpg)
©2010 Improving Enterprises, Inc.
Better log messages
Relate to generating point
Code
Data
![Page 23: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/23.jpg)
©2010 Improving Enterprises, Inc.
Best log messages
Are formatted for
Readability
Aggregation
![Page 24: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/24.jpg)
©2010 Improving Enterprises, Inc.
Logging objects
.toString()
org.apache.commons.lang.builder.ToStringBuilder
log.info(myDataObject);
![Page 25: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/25.jpg)
©2010 Improving Enterprises, Inc.
What to log
Input and output
Errors and exceptions
Computation results
Integration points
Thread rendezvous
![Page 26: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/26.jpg)
©2010 Improving Enterprises, Inc.
Input and output
public int calculateCost(String user, Flight flight)
{
log.debug(“calculateCost: “ + user + “: “ +
flight);
int cost = … ;
… // cost calculation
log.debug((“calculateCost: cost for “ + user
+ “: “ + flight + “ = “ + cost);
return cost;
}
![Page 27: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/27.jpg)
©2010 Improving Enterprises, Inc.
Errors and exceptions
// do dangerous work
result = DoDangerousWork(user, flight);}
catch (Exception e)
{
log.error(“methodName: exception in
DoDangerousWork for “ + user + “: “ + flight, e);
// recover or re-throw
}
![Page 28: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/28.jpg)
©2010 Improving Enterprises, Inc.
More
Log destination, parameters, outcomesDB calls
SOAP calls
Wait time
Thread joins
Available memory
Memory-intensive operations
Intermediate calculation results
For complicated calculations
![Page 29: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/29.jpg)
©2010 Improving Enterprises, Inc.
Avoid
Little information
… // step 1
logger.info(“did something - XXXX”);
… // step 2
logger.info(“did more work - YYYY”);
foreach (…) {
… // useful work
logger.info(“working hard”);
}
![Page 30: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/30.jpg)
©2010 Improving Enterprises, Inc.
Avoid
Code cluttering
if (logger.isDebugEnabled()) {
logger.debug(“Show Views");
logger.debug("Address:\n" +
view.getAddress());
logger.debug("Email:\n" +
view.getEmail());
}
![Page 31: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/31.jpg)
©2010 Improving Enterprises, Inc.
Watch for logging-caused bugs
Side effects
Logger.info(count++ + “: did something”);
Errors and exceptions
Logger.info(m.GetValue());
![Page 32: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/32.jpg)
©2010 Improving Enterprises, Inc.
DRY principle applies
![Page 33: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/33.jpg)
©2010 Improving Enterprises, Inc.
Growing, growing…
Naming conventions
Rotation
by size
by time
![Page 34: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/34.jpg)
©2010 Improving Enterprises, Inc.
You’ve got logs!
Real-time access
Deal with space limitations
Warning: large datasets
![Page 35: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/35.jpg)
©2010 Improving Enterprises, Inc.
Security
Absolutely
No Sensitive Data
![Page 36: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/36.jpg)
©2010 Improving Enterprises, Inc.
Using the results
![Page 37: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/37.jpg)
©2010 Improving Enterprises, Inc.
Learning from the logs
Never rely on eye-balling
WYS is not WYG
Distinguish common and
rare
![Page 38: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/38.jpg)
©2010 Improving Enterprises, Inc.
Code and logs
![Page 39: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/39.jpg)
©2010 Improving Enterprises, Inc.
Process log
Line by line
No need to load entire file
Aggregate events
Relate to time
![Page 40: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/40.jpg)
©2010 Improving Enterprises, Inc.
Tools
Utilities
grep
sort
uniq
Scripting
Perl
Awk
sed
![Page 41: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/41.jpg)
©2010 Improving Enterprises, Inc.
Learning from the logs
> grep userName myApplication-MMDDYYYY.log
> grep methodName myApplication-MMDDYYYY.log
> grep FATAL myApplication-MMDDYYYY.log
> tail -2000 myApplication-MMDDYYYY.log | grep Exception
![Page 42: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/42.jpg)
©2010 Improving Enterprises, Inc.
Querying logs using Awk
GNU Awk www.gnu.org/software/gawk
Create histograms
Wait times
Number of calls
Exceptions per time period or per user
Compare and relate events
Exception stacks traces
Outcomes by caller or thread
![Page 43: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/43.jpg)
©2010 Improving Enterprises, Inc.
AWK demo
Logs for the demo generously
provided by
http://42graphy.org/
![Page 44: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/44.jpg)
©2010 Improving Enterprises, Inc.
Jane.Prusakova at ImprovingEnterprises.com
SoftwareAndOtherThings.blogspot.com
@jprusakova
Jane Prusakova
http://www.slideshare.net/jprusakova
![Page 45: Application Logging for large systems](https://reader034.vdocuments.us/reader034/viewer/2022042813/548299f9b07959420c8b4841/html5/thumbnails/45.jpg)
©2010 Improving Enterprises, Inc.
Logging For Fun and Profit
Jane Prusakova
[email protected]://softwareandotherthings.blogspot.com
Improving Enterprises
Houston JUG
2014