• 1

Better Living Through New Releases

by Richard E. Baum

Overview

•  Server performance-related functionality •  Upgrades contain much more than bug fixes •  This is a high level view •  Close to chronological order

•  By topic where appropriate •  Not an exhaustive list •  Release notes are your friend

• 2

Automatic labels (2006.2)

•  Create labels based on changelist and path

•  No ‘p4 labelsync’ needed •  Label ‘Revision:’ field applied to view lines

Automatic labels (2006.2)

•  Changelist alias:

•  Files in a single changelist:

Label: nightly_build_01-Apr-2009 Owner: buildmeister Descr: Nightly build process View: //depot/... Revision: @4567

Label: patch-78901 Owner: mr-patchit Descr: Patch for bug 8471 View: //depot/... Revision: @7890,7890

• 3

p4 license (2006.2)

•  Add licensed users without server restart •  From command line •  From Perforce Administration GUI

•  IP change requires a restart

+Sn filetypes (2007.2)

•  Only store most recent n revisions •  Older revisions purged at submit •  Older metadata remains intact •  Values for n:

•  Improves handling of large binary objects

1-10, 16, 32, 64, 128, 256, 512

• 4

Journal file buffering (2006.2)

•  Journal buffer flushed periodically •  No longer after every transaction

•  Reduces writes •  Reduces locks taken •  Improves overall performance

Faster journal / checkpoint rotation (2007.3)

•  Rotation can now be a rename •  UNIX / Linux

•  Rotation can be instant on UNIX / Linux

•  Server copies and truncates if •  -z flag used for compression •  Rename fails •  Non UNIX / Linux server used

p4d -jc or p4d -jj

• 5

Checkpoint / Journal locking (2007.3)

•  Only read locks taken during:

•  Most read operations can continue p4d –jc p4d –jj

TCP/IP buffering (2008.1)

•  Auto-discover / adjust buffering levels •  Make better use of buffering space •  Improved network performance

•  Particularly over high latency connections

• 6

Proxy caches on submit (2007.3)

•  Files immediately ready for sync via proxy •  No additional sync required

•  MD5 hash values used for validation •  Timestamps used only as a fallback

Improved Proxy cache load (2007.3)

•  Update cache without sending files to client:

•  Intended to be used with dedicated client •  Have list assists with file selection •  Client workspace remains empty

p4 –Zproxyload sync

• 7

Sensible proxy compression (2008.1)

•  When syncing through Perforce Proxy •  Eliminates proxy decompress/compress •  Less overall work

•  Saves CPU on proxy •  Speeds up overall processing

•  Start server/proxy compression earlier

Obliterate improvements (2008.1)

•  We now compare: •  Number of lazy copies •  Number of revisions purged

•  If equal, purge the archive file! •  Fewer scans of database •  Mapping optimizations

• 8

Locking – (various releases)

•  Overall huge improvements •  Incremental changes every release •  Targeted rewrites for better concurrency •  New locking algorithm (2008.1)

MaxLockTime (2006.2)

•  p4 group field •  Max time an operation can lock a db file •  Server request fails when limit is reached •  Group spec:

•  User output:

•  Log output:

Group: all_users MaxLockTime: 10000 Users: jerry

Operation took too long (over 10.00 seconds); see 'p4 help maxlocktime'.

--- killed by MaxLockTime

• 9

Group ‘unlimited’ values (2008.1)

•  p4 group limit maximums •  MaxScanRows, MaxResults, MaxLockTime, Timeout

•  Allows use of finer granularity •  These both set no limit:

•  ‘unlimited’– other group settings don’t apply •  ‘unset’ – other group settings apply

•  Timeout of ‘0’ no longer allowed (use ‘unset’)

•  Old behavior maintained on upgrade •  ‘unlimited’ becomes ‘unset’

New locking methodology (2008.1)

•  Old, blocking methodology: •  Take locks as needed •  If needed lock is unavailable, wait until it is •  Perform operation •  Release locks

•  Locks held during wait •  Blocks other processes

• 10

Old methodology locking log fragment: 'user-client -d build_client' -- db.domain -- locks wait+held read/write 0ms+0ms/0ms+252838ms -- db.view -- locks wait+held read/write 0ms+1ms/0ms+252838ms -- db.have -- locks wait+held read/write 115ms+10ms/3942ms+252837ms -- db.label -- locks wait+held read/write 0ms+0ms/0ms+252836ms -- db.resolve -- locks wait+held read/write 0ms+0ms/0ms+252836ms -- db.locks -- locks wait+held read/write 0ms+0ms/0ms+252835ms -- db.working -- locks wait+held read/write 0ms+0ms/0ms+252835ms -- db.change -- locks wait+held read/write 0ms+0ms/252831ms+4ms

•  7 tables blocked 4+ minutes… •  …for 4ms of access to blocking table

New locking methodology (2008.1)

•  New methodology: •  Locks requested until a request blocks •  Release acquired locks •  Attempt to acquire blocking lock •  When granted, drop and start again •  After third try, use old behavior

•  Only one lock held while waiting •  Does not block additional processes

• 11

Batched updates (various)

•  Many database updates now batched •  Rather than updating revisions as changed •  Changes buffered to reduce I/O, locking •  Table update only every 100 records

•  More commands added each release (sync, resolve, edit, revert, integ, submit, etc.)

Server tunables (2008.2)

•  Don’t try this at home! •  Configure server limits/values

•  Previously hard-coded values

•  See documentation for more details: •  p4 help tunables •  p4 admin tunables

db.isalive 10K Rows scanned before maxLockTime check db.trylock 3 Attempts to avoid locks that block dm.flushtry 100 update buffer for sync, resolve, edit

• 12

Enhanced ‘p4d -xv’ (2008.2)

•  Newly built db files include data checksums •  For db files built with 2008.2 and later server •  Reported by ‘p4 admin dbstat’

•  Now, ‘p4d -xv’ checks the checksum •  Previously only table structure was checked •  ‘p4d -xr’ sets new checksums during repair

•  Only repair with assistance from support!

Enhanced ‘p4d -xv’ (2008.2)

•  Example: %p4 admin dbstat db.rev db.rev internal+leaf 0+1 page size 8k end page 1 generation 5 levels 1 fanout 0 Checksum 325666531

%p4d -xv Validating db.counters . . .

Validating db.rev **** Checksum mismatch **** possible data corruption

• 13

Summary

•  Performance-related changes every release

•  Release notes detail many modern marvels!

•  Modernize your depot – upgrade! •  Need help? Call!

•  Perforce Technical Support •  Perforce Consulting Services

Q & A

• 14

The End