Better Living Through New Releases

Better Living
         Through New Releases

                      by
               Richard E. Baum

Overview

•   Server performance-related functionality
•   Upgrades contain much more than bug fixes
•   This is a high level view
•   Close to chronological order
    • By topic where appropriate
• Not an exhaustive list
• Release notes are your friend

                                                •1

Automatic labels (2006.2)

• Create labels based on changelist and
    path
•   No ‘p4 labelsync’ needed
•   Label ‘Revision:’ field applied to view lines

Automatic labels (2006.2)

• Changelist alias:
    Label:      nightly_build_01-Apr-2009
    Owner:      buildmeister
    Descr:      Nightly build process
    View:             //depot/...
    Revision:   @4567

• Files in a single changelist:
    Label:      patch-78901
    Owner:      mr-patchit
    Descr:      Patch for bug 8471
    View:             //depot/...
    Revision:   @7890,7890

                                                    •2

p4 license (2006.2)

• Add licensed users without server restart
    • From command line
    • From Perforce Administration GUI
• IP change requires a restart

+Sn filetypes                (2007.2)

•   Only store most recent n revisions
•   Older revisions purged at submit
•   Older metadata remains intact
•   Values for n:
    1-10, 16, 32, 64, 128, 256, 512

• Improves handling of large binary objects

                                              •3

Journal file buffering (2006.2)

• Journal buffer flushed periodically
    • No longer after every transaction
• Reduces writes
• Reduces locks taken
• Improves overall performance

Faster journal / checkpoint rotation (2007.3)

• Rotation can now be a rename
    • UNIX / Linux
• Rotation can be instant on UNIX / Linux
    p4d -jc   or   p4d -jj

• Server copies and truncates if
       • -z flag used for compression
       • Rename fails
       • Non UNIX / Linux server used

                                                •4

Checkpoint / Journal locking (2007.3)

• Only read locks taken during:
    p4d –jc   p4d –jj

• Most read operations can continue

TCP/IP buffering (2008.1)

• Auto-discover / adjust buffering levels
• Make better use of buffering space
• Improved network performance
    • Particularly over high latency connections

                                                   •5

Proxy caches on submit (2007.3)

• Files immediately ready for sync via proxy
    • No additional sync required
• MD5 hash values used for validation
    • Timestamps used only as a fallback

Improved Proxy cache load (2007.3)

• Update cache without sending files to
   client:
    p4 –Zproxyload   sync

• Intended to be used with dedicated client
• Have list assists with file selection
• Client workspace remains empty

                                               •6

Sensible proxy compression (2008.1)

• When syncing through Perforce Proxy
• Eliminates proxy decompress/compress
• Less overall work
    • Saves CPU on proxy
    • Speeds up overall processing

• Start server/proxy compression earlier

Obliterate improvements       (2008.1)

• We now compare:
    • Number of lazy copies
    • Number of revisions purged
• If equal, purge the archive file!
• Fewer scans of database
• Mapping optimizations

                                           •7

Locking – (various releases)

•   Overall huge improvements
•   Incremental changes every release
•   Targeted rewrites for better concurrency
•   New locking algorithm (2008.1)

MaxLockTime                (2006.2)

•   p4 group field
•   Max time an operation can lock a db file
•   Server request fails when limit is reached
•   Group spec:
        Group:       all_users
        MaxLockTime: 10000
        Users:       jerry

• User output:    Operation took too long (over 10.00
                  seconds); see 'p4 help maxlocktime'.

• Log output:     --- killed by MaxLockTime

                                                         •8

Group ‘unlimited’ values (2008.1)

• p4 group limit maximums
    •   MaxScanRows, MaxResults, MaxLockTime, Timeout
• Allows use of finer granularity
• These both set no limit:
    • ‘unlimited’– other group settings don’t apply
    • ‘unset’     – other group settings apply
• Timeout of ‘0’ no longer allowed (use
    ‘unset’)
•   Old behavior maintained on upgrade
    • ‘unlimited’ becomes ‘unset’

New locking methodology             (2008.1)

• Old, blocking methodology:
    •   Take locks as needed
    •   If needed lock is unavailable, wait until it is
    •   Perform operation
    •   Release locks

• Locks held during wait
• Blocks other processes

                                                          •9

Old methodology locking log fragment:
'user-client -d build_client'
-- db.domain
-- locks wait+held read/write   0ms+0ms/0ms+252838ms
-- db.view
-- locks wait+held read/write   0ms+1ms/0ms+252838ms
-- db.have
-- locks wait+held read/write   115ms+10ms/3942ms+252837ms
-- db.label
-- locks wait+held read/write   0ms+0ms/0ms+252836ms
-- db.resolve
-- locks wait+held read/write   0ms+0ms/0ms+252836ms
-- db.locks
-- locks wait+held read/write   0ms+0ms/0ms+252835ms
-- db.working
-- locks wait+held read/write   0ms+0ms/0ms+252835ms
-- db.change
-- locks wait+held read/write   0ms+0ms/252831ms+4ms

•   7 tables blocked 4+ minutes…
•   …for 4ms of access to blocking table

New locking methodology (2008.1)

• New methodology:
     •   Locks requested until a request blocks
     •   Release acquired locks
     •   Attempt to acquire blocking lock
     •   When granted, drop and start again
     •   After third try, use old behavior
• Only one lock held while waiting
• Does not block additional processes

                                                             •10

Batched updates           (various)

• Many database updates now batched
    • Rather than updating revisions as changed
    • Changes buffered to reduce I/O, locking
    • Table update only every 100 records
• More commands added each release
    (sync, resolve, edit, revert, integ, submit, etc.)

Server tunables (2008.2)

• Don’t try this at home!
• Configure server limits/values
    • Previously hard-coded values
db.isalive    10K   Rows scanned before maxLockTime check
db.trylock    3     Attempts to avoid locks that block
dm.flushtry   100   update buffer for sync, resolve, edit

• See documentation for more details:
    • p4 help tunables
    • p4 admin tunables

                                                            •11

Enhanced ‘p4d -xv’ (2008.2)

• Newly built db files include data checksums
    • For db files built with 2008.2 and later server
    • Reported by ‘p4 admin dbstat’
• Now, ‘p4d -xv’ checks the checksum
• Previously only table structure was checked
• ‘p4d -xr’ sets new checksums during repair
    • Only repair with assistance from support!

Enhanced ‘p4d -xv’ (2008.2)

• Example:
   %p4 admin dbstat db.rev
   db.rev
        internal+leaf 0+1
        page size 8k end page 1
        generation 5 levels 1 fanout 0
        Checksum 325666531
   %p4d -xv
   Validating db.counters
   ...

   Validating db.rev
       **** Checksum mismatch **** possible data corruption

                                                              •12

Summary

• Performance-related changes every
    release
•   Release notes detail many modern
    marvels!
•   Modernize your depot – upgrade!
•   Need help? Call!
    • Perforce Technical Support
    • Perforce Consulting Services

                 Q&A

                                       •13

The End

          •14