![Page 1: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/1.jpg)
Robust and Scalable Concurrent Programming: Lessons from the Trenches
Sangjin Lee, Mahesh Somani, & Debashis Saha eBay Inc.
![Page 2: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/2.jpg)
2
Agenda
> Why Does It Matter? > Patterns: the Good, the Bad, and the $#&%@! > Lessons Learned
![Page 3: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/3.jpg)
3
Why Does It Matter?
> Concurrency is becoming only more important Multi-core systems are becoming the norm of
enterprise servers Your enterprise software needs to run under high
concurrency to take advantage of hardware There are other ways to scale (virtualization or running
multiple processes), but it is not always doable
![Page 4: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/4.jpg)
4
Why Does It Matter?
> Writing safe and concurrent code is hard Thread safety is hard for an average Java
developer Many still don’t fully understand the Java Memory Model Thread safety is often an afterthought Shared data is still the primary mechanism of handling
concurrency in Java Other approaches: Actor-based share-nothing concurrency (Scala),
STM, etc. Thread safety concerns are not easily isolated
![Page 5: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/5.jpg)
5
Why Does It Matter?
> Writing safe and concurrent code is hard Writing safe and scalable code is even harder
It requires understanding of your software under concurrency Will this lock become hot under our production traffic usage? You should tune only if it is shown to be a real hot spot
It requires knowledge of what scales and what doesn’t Synchronized HashMap vs. ConcurrentHashMap? When does ReadWriteLock make sense? Atomic variables?
![Page 6: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/6.jpg)
6
Why Does It Matter?
> Yet, the penalty for incorrect or non-scalable code is severe They are often the most difficult bugs: hard to
reproduce, and happen only under load (a.k.a. your production peak time)
The code works just fine under light load, but it blows up in your face as the traffic peaks
![Page 7: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/7.jpg)
7
Patterns
> Lazy Initialization
![Page 8: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/8.jpg)
8
Patterns [1] Lazy Initialization > What’s wrong with this picture?
public class Singleton { private static Singleton instance;
public static Singleton getInstance() { if (instance == null) { instance = new Singleton(); } return instance; }
private Singleton() {} }
![Page 9: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/9.jpg)
9
Patterns [1] Lazy Initialization > Not thread safe of course! Let’s fix it.
public class Singleton { private static Singleton instance;
public static Singleton getInstance() { if (instance == null) { instance = new Singleton(); } return instance; }
private Singleton() {} }
![Page 10: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/10.jpg)
10
Patterns [1] Lazy Initialization > Fix #1: synchronize!
public class Singleton { private static Singleton instance;
public static synchronized Singleton getInstance() { if (instance == null) { instance = new Singleton(); } return instance; }
private Singleton() {} }
![Page 11: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/11.jpg)
11
Patterns [1] Lazy Initialization > It is now correct (thread safe) but ... > We have just introduced a potential lock
contention > It can pose a serious scalability issue if it is in the
critical path “The principal threat to scalability in concurrent
applications is the exclusive resource lock.” – Brian Goetz, Java Concurrency in Practice
![Page 12: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/12.jpg)
12
Patterns [1] Lazy Initialization > Impact of a lock contention
0
20
40
60
80
100
0 20 40 60 80 100
Rel
ativ
e TP
S (%
)
CPU (%)
Lock Contention Fixed
![Page 13: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/13.jpg)
13
Patterns [1] Lazy Initialization > Fix #2 (bad): synchronize only when we allocate public class Singleton { private static Singleton instance;
public static Singleton getInstance() { if (instance == null) { synchronized (Singleton.class) { if (instance == null) { instance = new Singleton(); } } } return instance; }
private Singleton() {} }
![Page 14: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/14.jpg)
14
Patterns [1] Lazy Initialization > There is a name for this pattern > Optimization gone awry: it appears correct and
efficient, but it is not thread safe
![Page 15: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/15.jpg)
15
Patterns [1] Lazy Initialization > Fix #3: $#&%@!
public class Singleton { private static Singleton instance; private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
public static Singleton getInstance() { lock.readLock().lock(); try { if (instance == null) { lock.readLock().unlock(); lock.writeLock().lock(); try { if (instance == null) { instance = new Singleton(); } } finally { lock.readLock().lock(); lock.writeLock().unlock(); } } return instance; } finally { lock.readLock().unlock(); } }
private Singleton() {} }
![Page 16: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/16.jpg)
16
Patterns [1] Lazy Initialization > None of the fixes quite works: they are either
unsafe, or don’t scale if in the critical path > Then what is a better fix?
![Page 17: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/17.jpg)
17
Patterns [1] Lazy Initialization > Eager Initialization!
No reason to initialize lazily: allocations are cheap Lazy initialization was popular when allocations
used to be expensive Today we do it simply by habit for no good reason Static initialization takes care of thread safety, and
is also lazy For singletons, you gain nothing by using an explicit
lazy initialization
![Page 18: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/18.jpg)
18
Patterns [1] Lazy Initialization > Good fix #1: eager initialization
public class Singleton { private static final Singleton instance = new Singleton();
public static Singleton getInstance() { return instance; }
private Singleton() {} }
![Page 19: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/19.jpg)
19
Patterns [1] Lazy Initialization > Then when does lazy initialization make sense?
(For a non-singleton) If instantiation is truly expensive, and it has a chance of not being needed
If you want to retry if it fails the first time If you want to unload and reload the object To reduce confusion if there was an exception
during eager initialization ExceptionInInitializerError for the first call NoClassDefFoundError for subsequent calls
![Page 20: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/20.jpg)
20
Patterns [1] Lazy Initialization > What are the right lazy initialization patterns?
![Page 21: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/21.jpg)
21
Patterns [1] Lazy Initialization > Good fix #2: holder pattern (for non-singletons)
public class Foo { public static Bar getBar() { return BarHolder.getInstance(); // lazily initialized }
private static class BarHolder { private static final Bar instance = new Bar();
static Bar getInstance() { return instance; } } }
![Page 22: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/22.jpg)
22
Patterns [1] Lazy Initialization > Good fix #3: corrected double-checked locking
(only on Java 5 or later) public class Singleton { private static volatile Singleton instance;
public static Singleton getInstance() { if (instance == null) { synchronized (Singleton.class) { if (instance == null) { instance = new Singleton(); } } } return instance; }
private Singleton() {} }
![Page 23: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/23.jpg)
23
Patterns
> Lazy Initialization > Map & Compound Operation
![Page 24: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/24.jpg)
24
Patterns [2] Map & Compound Operation > What’s wrong with this picture?
public class KeyedCounter { private Map<String,Integer> map = new HashMap<String,Integer>();
public void increment(String key) { Integer old = map.get(key); int value = (old == null) ? 1 : old.intValue()+1; map.put(key, value); }
public Integer getCount(String key) { return map.get(key); } }
![Page 25: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/25.jpg)
25
Patterns [2] Map & Compound Operation > It may be blazingly fast but ... it’s not thread safe!
public class KeyedCounter { private Map<String,Integer> map = new HashMap<String,Integer>();
public void increment(String key) { Integer old = map.get(key); int value = (old == null) ? 1 : old.intValue()+1; map.put(key, value); }
public Integer getCount(String key) { return map.get(key); } }
![Page 26: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/26.jpg)
26
Patterns [2] Map & Compound Operation > “How bad can it get?” Really bad.
ConcurrentModificationException may be your best outcome
Worse, you might end up in an infinite loop Worse yet, it may simply give you a wrong result
without any errors; data corruption, etc. > Don’t play with fire: never modify an
unsynchronized collection concurrently
![Page 27: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/27.jpg)
27
Patterns [2] Map & Compound Operation > Fix #1 (bad): synchronize the map
public class KeyedCounter { private Map<String,Integer> map = Collections.synchronizedMap(new HashMap<String,Integer>());
public void increment(String key) { Integer old = map.get(key); int value = (old == null) ? 1 : old.intValue()+1; map.put(key, value); }
public Integer getCount(String key) { return map.get(key); } }
![Page 28: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/28.jpg)
28
Patterns [2] Map & Compound Operation > Fix #1 (bad): synchronize the map
It is still not thread safe! Even though individual operations (get() and put()) are thread safe, the compound operation (KeyedCounter.increment()) is not
![Page 29: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/29.jpg)
29
Patterns [2] Map & Compound Operation > Fix #2: synchronize the operations
public class KeyedCounter { private Map<String,Integer> map = new HashMap<String,Integer>();
public synchronized void increment(String key) { Integer old = map.get(key); int value = (old == null) ? 1 : old.intValue()+1; map.put(key, value); }
public synchronized Integer getCount(String key) { return map.get(key); } }
![Page 30: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/30.jpg)
30
Patterns [2] Map & Compound Operation > Fix #2: synchronize the operations
It is now correct But does it scale? As increment() and getCount() get called from
multiple threads, they may become a source of lock contention
![Page 31: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/31.jpg)
31
Patterns [2] Map & Compound Operation > ConcurrentHashMap comes to the rescue!
In general it scales significantly better than a synchronized HashMap under concurrency Full reader concurrency Some writer concurrency with finer-grained locking
The ConcurrentMap interface supports additional thread safe methods; e.g. putIfAbsent()
![Page 32: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/32.jpg)
32
Patterns [2] Map & Compound Operation > Good fix: ConcurrentHashMap with atomic
variables public class KeyedCounter { private final ConcurrentMap<String,AtomicInteger> map = new ConcurrentHashMap<String,AtomicInteger>();
public void increment(String key) { AtomicInteger value = new AtomicInteger(0); AtomicInteger old = map.putIfAbsent(key, value); if (old != null) { value = old; } value.incrementAndGet(); // increment the value atomically }
public Integer getCount(String key) { AtomicInteger value = map.get(key); return (value == null) ? null : value.get(); } }
![Page 33: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/33.jpg)
33
Patterns [2] Map & Compound Operation > Even better...
public class KeyedCounter { private final ConcurrentMap<String,AtomicInteger> map = new ConcurrentHashMap<String,AtomicInteger>();
public void increment(String key) { AtomicInteger value = map.get(key); if (value == null) { value = new AtomicInteger(0); AtomicInteger old = map.putIfAbsent(key, value); if (old != null) { value = old; } } value.incrementAndGet(); // increment the value atomically }
public Integer getCount(String key) { AtomicInteger value = map.get(key); return (value == null) ? null : value.get(); } }
![Page 34: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/34.jpg)
34
Patterns
> Lazy Initialization > Map & Compound Operation > Other People’s Money Classes
Unsafe classes
![Page 35: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/35.jpg)
35
Patterns [3] Other People’s Classes #1 > What’s wrong with this picture?
public class DateFormatter { private static final DateFormat df = DateFormat.getInstance();
public static String formatDate(Date d) { return df.format(d); } }
![Page 36: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/36.jpg)
36
Patterns [3] Other People’s Classes #1 > DateFormat is not thread safe! > Some JDK classes that are not thread safe
DateFormat: declared in javadoc as not thread safe MessageDigest: no mention of thread safety CharsetEncoder/CharsetDecoder: declared in
javadoc as not thread safe > Lessons
Do not assume thread safety of others’ classes Even javadoc may not have a full disclosure
![Page 37: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/37.jpg)
37
Patterns [3] Other People’s Classes #1 > Fix #1: synchronize!
It is now safe, but has a potential of turning into a lock contention
public class DateFormatter { private static final DateFormat df = DateFormat.getInstance();
public static String formatDate(Date d) { synchronized (df) { return df.format(d); } } }
![Page 38: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/38.jpg)
38
Patterns [3] Other People’s Classes #1 > Good fix #2: allocate it every time
It is completely safe (stack confinement via local variable)
Allocations are not as expensive as you might think
public class DateFormatter {
public static String formatDate(Date d) { DateFormat df = DateFormat.getInstance(); return df.format(d); } }
![Page 39: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/39.jpg)
39
Patterns [3] Other People’s Classes #1 > Good fix #3: use ThreadLocal
Use it if truly concerned about allocation cost
public class DateFormatter { private static final ThreadLocal<DateFormat> tl = new ThreadLocal<DateFormat>() { @Override protected DateFormat initialValue() { return DateFormat.getInstance(); } };
public static String formatDate(Date d) { return tl.get().format(d); } }
![Page 40: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/40.jpg)
40
Patterns
> Lazy Initialization > Map & Compound Operation > Other People’s Classes
Unsafe classes Hidden costs
![Page 41: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/41.jpg)
41
Patterns [3] Other People’s Classes #2 > What’s wrong with this picture?
public class ForceInit {
public static void init(String className) { try {
// Force initialization Class.forName(className); } catch (ClassNotFoundException e) {
// May happen. Do nothing. }
} }
![Page 42: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/42.jpg)
42
Patterns [3] Other People’s Classes #2 > Thread safe … but potential lock contention when
missing class
public class ForceInit {
public static void init(String className) { try {
// Force initialization Class.forName(className); } catch (ClassNotFoundException e) {
// May happen. Do nothing. }
} }
![Page 43: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/43.jpg)
43
Patterns [3] Other People’s Classes #2 > Class.forName()
Positive results cached but not negative ones When class not found, (expensive) search in
classpath Class loading deals with different locks: repeated
invocation => lock contention and CPU hot spot! > Lessons
Measure contention: positive and negative tests Beware as javadoc may not have disclosure on
locks or inner workings
![Page 44: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/44.jpg)
44
Patterns [3] Other People’s Classes #2 > Fix: trading memory for speed. Works when
limited range of class names public class ForceInit { private static final ConcurrentMap<String,String> cache = new ConcurrentHashMap<String,String>();
public static void init(String className) { try { if (cache.putIfAbsent(className, className) == null) { // Force initialization Class.forName(className); } } catch (ClassNotFoundException e) { // May happen. Do nothing. } } }
![Page 45: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/45.jpg)
45
Patterns
> Lazy Initialization > Map & Compound Operation > Other People’s Classes
Unsafe classes Hidden costs Hidden costs #2
![Page 46: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/46.jpg)
46
Patterns [3] Other People’s Classes #3 > What’s wrong with this picture?
public class XMLParser {
public void parseXML(InputStream is, DefaultHandler handler) throws Exception { SAXParser parser = SAXParserFactory.newInstance().newSAXParser(); parser.parse(is, handler); } }
![Page 47: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/47.jpg)
47
Patterns [3] Other People’s Classes #3 > Depends … newSAXParser() could have a
potentially significant lock contention
public class XMLParser {
public void parseXML(InputStream is, DefaultHandler handler) throws Exception { SAXParser parser = SAXParserFactory.newInstance().newSAXParser(); parser.parse(is, handler); } }
![Page 48: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/48.jpg)
48
Patterns [3] Other People’s Classes #3 > But why?
Implementation class determined in following order System property lib/jaxp.properties META-INF/
services/jaxp… in class path Platform default META-INF not cached! Possible lock contention
> Also applies to DOM & XML Reader > JAXP implementations (e.g., Xerces) may have
additional configurations read from classpath > Lessons
Measure contention Read documentation!
![Page 49: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/49.jpg)
49
Patterns [3] Other People’s Classes #3 > Good fix #1: Set up environment (system
properties)
$ java ‐Djavax.xml.parsers...=IMPL_CLASS ... ... MAIN_CLASS [args]
![Page 50: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/50.jpg)
50
Patterns [3] Other People’s Classes #3 > Good fix #2: ThreadLocal
Beware reuse issues however
public class XMLParser { private static final ThreadLocal<SAXParser> tlSax = new ThreadLocal<SAXParser>();
private static SAXParser getParser() throws Exception { SAXParser parser = tlSax.get(); if (parser == null) { parser = SAXParserFactory.newInstance().newSAXParser(); tlSax.set(parser); } return parser; }
public void parseXML(InputStream is, DefaultHandler handler) throws Exception { getParser().parse(is, handler);
} }
![Page 51: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/51.jpg)
51
Patterns
> Lazy Initialization > Map & Compound Operation > Other People’s Classes
Unsafe classes Hidden costs Hidden costs #2
![Page 52: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/52.jpg)
52
Lessons Learned: Best Practices
> Patterns & anti-patterns Common problems and solutions “Anti-patterns” are a good way of educating Useful way to analyze solutions Surprisingly repeated use and misuse
![Page 53: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/53.jpg)
53
Lessons Learned: Best Practices
> Correctness first: thread safety is non-negotiable Premature optimization for scalability could risk
correctness Concurrency issues are insidious, hard to
reproduce and debug Concurrency issues increase nonlinearly with load
![Page 54: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/54.jpg)
54
Lessons: Not All Code is Created Equal
> Scalability next Scalability and lock contention
A few locks cause most of contention
TPS should increase linearly with utilization Lock contention possible cause if
nonlinear behavior
Ways to determine contended locks Lock analyzers, CPU profiling
(oprofile, tprof) Thread dumps a few seconds apart
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 3 5 7 9 11 13 15 17 19
% o
f Con
tent
ion
Contended lock
![Page 55: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/55.jpg)
55
Lessons: Best Practices
> Multi-pronged attack: proactive Regular training and documentation Static code analysis Runtime code analysis Ongoing performance monitoring and measurements
> Multi-pronged attack: reactive Periodic tuning Complimentary to programming discipline Be disciplined: tune only the top hot spots
![Page 56: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/56.jpg)
56
Lessons: Best Practices
> eBay’s Systems Lab Lab environment to put applications under microscope with
different profiling tools On-going tests to systemically identify and fix bottlenecks Yielded substantial scalability improvements and server
savings
![Page 57: Robust and Scalable Concurrent Programming: Lesson from the Trenches](https://reader034.vdocuments.us/reader034/viewer/2022052618/55492552b4c905a54c8ba4ad/html5/thumbnails/57.jpg)
57
Q & A
> Questions?