db2 performance primer baltimore/washington db2 users group december 10, 2003 michael murray senior...
TRANSCRIPT
DB2 Performance Primer
Baltimore/Washington DB2 Users Group
December 10, 2003
Michael MurraySenior Consultant
Presentation Overview
How Does DB2 Get Data? Index Considerations Tablespace Considerations Tune the SQL to Get the Biggest Bang
for your Buck Access Paths – What are the Choices?
DB2 SubsystemDatabase
Tablespace
Table
Page
Page
Row
Indexspace
Index
Page
Page
What is a Page?
4K page, maximum number of data bytes on the page 4,074. Maximum row size 4,056. Row size greater than 2,020 bytes will end up with just one row per page.
4K
8K
16K
32K
GETPAGE
Physical I/O
Your Data
Row Limits
PSID – page set identifier
DBM1 Address Space
SQL, DDL, DML, DCL Optimization Data Manager Relational Data System Buffer Manager Sort Pool EDM Pool RID Pool Bind/Rebind IRLM Interaction
Locking IRLM
Directory Catalog
WorkFile User Data
DSNDB01 DSNDB06
DSNDB07
Database ServicesDBM1
SQL
stored procedures
DB2 Stogroups
STOGROUP GPRVVOL
Volume B2RP03
Volume B2RP02
Volume B2RP01
CREATE STOGROUP GSMSVOL VOLUMES(“*”) VCAT S0;
CREATE STOGROUP GPRVVOL VOLUMES(B2RP01 ,B2RP02 ,B2RP03) VCAT P0;
Non-SMS User Managed Stogroup
SMS Managed Stogroup
SMS Allocations
indexspacevolume PBP2
Storage Group I
cache
indexspacevolume PBP8
cache
tablespacevolume PB34
Storage Group T
cache
tablespacevolume PBL8
cache
ACS Routines
(4th node is NOT X or I) (4th node is X or I)
Tablespace or Indexspace Allocations (Create, Extent Allocation, Load, Reorg)
DB2
SMS
Buffer Pool Strategy
DSNDB07DSNDB01DSNDB06
BP0 BP7
Default Tablespace
BP48
Default Indexspace
BP49
CRM Indexes
BP5
CRM Tablespaces
BP8
ERP Indexes
BP3
ERP Tablespaces
BP6
Critical Path Indexes
BP1
Critical Path Tablespace
BP2
DSNDB04
BP4
Other Indexes
BP9
Other Tablespaces
BP10
32K Tablespaces
BP32K
Random Tablespaces
BP12
Buffer Manager
indexspace
tablespace
Hiperpool
Group Buffer Pool
BUFFER POOL DATA MANAGER
Relational Data System
STAGE 1 PREDICATES
STAGE 2 PREDICATES
sargableindexableGet Page Request
Physical I/O
(page not found)
(page not found)
(page not found)
(page fou
nd
)
(page fou
nd
)
cache
cache
DB2’s Work File Explained
DATA MANAGER
Relational Data System
STAGE 1 PREDICATES
STAGE 2 PREDICATES
sargableindexable
Filtered Result Set
DSNDB07
SORT required
CREATE INDEX DEFER NO
•No matching index•DISTINCT•ORDER BY•UNION•GROUP BY•Some Joins
SORT POOL
3. sort columns
1. selected columns2. calc/derived values
key columns, RIDs
BP7
sort assist hardware
Sort assist hardware?
Index Considerations
Clustering Vs. Non Clustering Unique Vs. Duplicate Piecesize with NPI’s Uniqueness, Performance,
RI, Sorting Table Size Minimize I/O
1 2 5 7 1015
PRT B PRT L PRT M SG M/L SG S C
# Indexes
Index Limits based on # of pages? What's the I/O rate per page? Is this even realistic?
TS Type Number PagesPartitioned big > 5,000,000Partitioned large < 5,000,000
> 200,000Partitioned medium < 200,000
> 5,000Segmented medium - large > 5,000
< 500,000Segmented small < 5,000Code Tables –Segmented or Partitioned
< 200
Peaks > 13,999 Feet Located in US
Indexes: Clustered Index on: Elevation (desc), State, Peak Name Non-Clustered Index on: Peak Name Unique PK Index on: Latitude, Longitude
Peak Name State Elevation Latitude Longitude RatingMount McKinley AK 20320 630410N 1510013W VERY DIFFICULT
Mount Bear AK 14831 611702N 1410832W MODERATE
Mount Whitney CA 14494 363438N 1181733W MODERATE
Mount Harvard CO 14420 385528N 1061912W EASY
Mount Rainier WA 14410 465110N 1214531W DIFFICULT
Goal of indexes is to minimize I/O
Elevation (desc)
State Peak Name
ROOT Page
CLUSTERED INDEX Page 552
14831 AK Mount Bear
14420 CO Mount Harvard
p. 501, 2
p. 502, 2
LEAF & DATA Page
Page 553
14270 CO Grays Peak
14196 CO Mount Yale p. 504, 2
p. 503, 2
Page 554
14110 CO Pikes Peak
14059 CO Sunlight Peak
p. 505, 2
Page 555
14037 CO Little Bear Peak
14000 CA Thunderbolt Peak
p. 507, 2
NON-LEAF Page
Page 717
14420 CO Mount Harvard
14196 CO Mount Yale
14059 CO Sunlight Peak
14000 CA Thunderbolt Peak
p. 553, 2
p. 554, 2
p. 552, 2
p. 508, 2
20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CO Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14000 CA Thunderbolt Peak …
Page 501
14005 CO Mount o/t Holy Cross …
p. 555, 2
p. 506, 2
Peak Name p. 401, 2
Page 301
Little Bear Peak
Mount Bear p. 402, 2
ROOT Page
NON-LEAF Page
LEAF Page
NON-CLUSTERED INDEX 20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CO Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14005 CO Mount o/t Holy Cross …
14000 CA Thunderbolt Peak …
DATA Page
Page 501
p. 403, 2
Page 302
Mount Harvard
Mount o/t Holy Cross p. 404, 2
p. 405, 2
Page 303
Mount Whitney
Pikes Peak p. 406, 2
p. 407, 2
Page 304
Starlight Peak
Thunderbolt Peak p. 408, 2
p. 301, 2
Page 201
Mount Bear
Mount o/t Holy Cross p. 302, 2
p. 303, 2
Page 202
Pikes Peak
Thunderbolt Peak p. 304, 2
p. 201, 2
Page 101
Mount o/t Holy Cross
Thunderbolt Peak p. 202, 2
p. 503, 2
Page 401
Grays Peak
Little Bear Peak p. 507, 2
p. 504, 1
Page 402
Longs Peak
Mount Bear p. 501, 2
p. 505, 1
Page 403
Mount Bross
Mount Harvard p. 502, 2
p. 501, 1
Page 404
Mount McKinley
Mount o/t Holy Cross p. 508, 1
p. 503, 1
Page 405
Mount Rainier
Mount Whitney p. 502, 1
p. 504, 2
Page 406
Mount Yale
Pikes Peak p. 505, 2
p. 507, 1
Page 407
Split Mountain
Starlight Peak p. 506, 1
p. 506, 2
Page 408
Sunlight Peak
Thunderbolt Peak p. 508, 2NON-LEAF Page
p. 461, 2
Page 331
370116N 1182520W
370539N 1183124W p. 462, 2
ROOT Page
NON-LEAF Page
LEAF Page
PRIMARY NON-CLUSTERED INDEX 20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CO Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14005 CO Mount o/t Holy Cross …
14000 CA Thunderbolt Peak …
DATA Page
Page 501
LatitudeLongitude
p. 463, 2
Page 332
373738N 1073543W
385039N 1061848W p. 464, 2
p. 465, 2
Page 333
392007N 1060625W
393802N 1054901W p. 466, 2
p. 467, 2
Page 334
465110N 1214531W
630410N 1510013W p. 468, 2
p. 331, 2
Page 281
370539N 1183124W
385039N 1061848W p. 332, 2
p. 333, 2
Page 282
393802N 1054901W
630410N 1510013W p. 334, 2
p. 281, 2
Page 101
385039N 1061848W
630410N 1510013W p. 282, 2
p. 502, 1
Page 461
363438N 1181733W
370116N 1182520W p. 507, 1
p. 506, 1
Page 462
370539N 1183119W
370539N 1183124W p. 508, 2
p. 507, 2
Page 463
373400N 1052948W
373738N 1073543W p. 506, 2
p. 505, 2
Page 464
385026N 1050238W
385039N 1061848W p. 504, 2
p. 502, 2
Page 465
385528N 1061912W
392007N 1060625W p. 505, 1
p. 508, 1
Page 466
392805N 1062845W
393802N 1054901W p. 503, 2
p. 504, 1
Page 467
401517N 1053655W
465110N 1214531W p. 503, 1
p. 501, 2
Page 468
611702N 1410832W
630410N 1510013W p. 501, 1NON-LEAF Page
Adding Columns to an Index
Improves filtering Provides index only access Increases matched columns Provides a sequence to avoid SORTS Increases cluster ratio Helps in index screening Is the increased RID chain length negatively
impacting your non-read transactions?
Index Statistics
Indexes & Stats: PK Index on: Latitude, Longitude, Climber_ID, Strt_DT NPI: Climber_ID, Summit, O2, Guided RUNSTATS TABLESPACE dbname.tsname INDEX(ALL)
1st Column & Fullkey Cardinality on all indexesTop 10 most frequent values for 1st column of all indexes
RUNSTATS INDEX owner.ixname FREQVAL xx NUMCOLS xxGive me the top 15 climbed peaks (FREQVAL 15 NUMCOLS 2 for PK index)
LATITUDE LONGITUDE CLIMBER_ID STRT_DT END_DT TB_TIME SUMMIT O2 GUIDED630410N 1510013W CA000015 4/20/2001 5/28/2001 11:00 YES YES NO630410N 1510013W CA000015 2/11/2000 3/13/2000 10:00 NO NO YES630410N 1510013W CE000008 2/7/2000 3/22/2000 12:00 YES YES NO630410N 1510013W CE000008 3/3/1999 4/16/1999 13:00 YES YES YES630410N 1510013W CE000008 1/14/1998 3/14/1998 15:30 NO YES YES630410N 1510013W CJ000033 3/17/2002 5/13/2002 14:30 YES NO NO630410N 1510013W CJ000033 12/19/2000 2/14/2001 15:00 NO YES YES630410N 1510013W CR000039 2/14/2002 4/9/2002 13:30 YES NO NO630410N 1510013W CR000039 3/30/2001 5/12/2001 14:00 YES NO YES630410N 1510013W CR000039 1/7/1999 3/10/1999 13:00 NO NO NO
PEAKS CLIMBED TABLE
SELECT CLIMBER_ID, LATITUDE, LONGITUDE, O2, GUIDED FROM PEAKS_CLIMBEDWHERE CLIMBER_ID = ‘CE000008’ AND SUMMIT = ‘YES’
ORDER BY O2, GUIDED WITH UR;
SORT Required?
Colvalue CardF FrequencyFAK 2 0.1250CA 4 0.2500CO 9 0.5625WA 1 0.0625
SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CO’;
20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CO Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14005 CO Mount o/t Holy Cross …
14000 CA Thunderbolt Peak …
Page 501
p. 501, 1
Page 307
AK 20320
p. 501, 2AK 14831
p. 502, 1
Page 308
CA 14494
p. 506, 1CA 14080
p. 507, 1
Page 309
CA 14058
p. 508, 2CA 14000
p. 502, 2
Page 310
CO 14420
p. 503, 2CO 14270
p. 504, 1
Page 311
CO 14255
p. 504, 2CO 14196
p. 505, 1
Page 312
CO 14172
p. 505, 2CO 14110
p. 506, 2
Page 313
CO 14059
p. 507, 2CO 14037
p. 508, 1
Page 314
CO 14005
p. 503, 1WA 14410
Colvalue CardF FrequencyFAK 21 0.2308CA 15 0.1648CO 54 0.5934WA 1 0.0110
SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CO’ AND ELEVATION =‘14110’;
ColGroupColNo CardF FrequencyFAK 20320 1 0.0625AK 14831 1 0.0625CA xxxxx 1 0.0625CO xxxxx 1 0.0625WA xxxxx 1 0.0625
ColGroupColNo CardF FrequencyFAK xxxxx 1 0.0110CA xxxxx 1 0.0110CA 14162 2 0.0220CA 14000 2 0.0220CO xxxxx 1 0.0110CO 14265 2 0.0220CO 14162 2 0.0220CO 14042 2 0.0220CO 14014 2 0.0220CO 14005 2 0.0220CO 14197 3 0.0330WA 14410 1 0.0110
RUNSTATS INDEX DB2.STELEV FREQVAL 10 NUMCOLS 2
RUNSTATS TABLESPACE dbname.tsname INDEX(ALL)
DATA PageLEAF Page
CREATE INDEX DB2.STELEV ON DB2.PEAKS STATE, ELEVATION DEFER YES PIECESIZE 2G;
List Prefetch
20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CA Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14005 CO Mount o/t Holy Cross …
14000 CA Thunderbolt Peak …
Page 501
p. 502, 1
Page 461
363438N 1181733W
370116N 1182520W p. 507, 1
p. 506, 1
Page 462
370539N 1183119W
370539N 1183124W p. 508, 2
p. 507, 2
Page 463
373400N 1052948W
373738N 1073543W p. 506, 2
p. 505, 2
Page 464
385026N 1050238W
385039N 1061848W p. 504, 2
p. 502, 2
Page 465
385528N 1061912W
392007N 1060625W p. 505, 1
p. 508, 1
Page 466
392805N 1062845W
393802N 1054901W p. 503, 2
p. 504, 1
Page 467
401517N 1053655W
465110N 1214531W p. 503, 1
p. 501, 2
Page 468
611702N 1410832W
630410N 1510013W p. 501, 1
p. 502, 1
RID LIST
363438N 1181733W
370116N 1182520W p. 507, 1
p. 506, 1370539N 1183119W
370539N 1183124W p. 508, 2
373738N 1073543W p. 506, 2
p. 502, 1
Sorted RID LIST
363438N 1181733W
p. 506, 1370539N 1183119W
370539N 1183124W p. 508, 2
373738N 1073543W p. 506, 2
physical I/O order
370116N 1182520W p. 507, 1
(poor cluster ratio)
= unwanted disk head movement
SELECT ELEVATION, PEAKNAME FROM PEAKS WHERE LATITUDE IN (‘363438N’,’370116N’,’370539N’,’373738N’)
Tablespace TypesSegmented Tablespace - 3 table example
Segmented Tablespace - 1 table example
Spacemap keeps track of segments (pages) by table. Spacemap page in a segmented tablespace helps when a program deletes without a where clause by marking those segments logically deleted. Spacemap page also makes segments tied to dropped tables accessible. Segsize between 4-64 increasing in increments of 4. See recommendations in notes section.
Simple Tablespace - 1 table example
Simple Tablespaces can have multiple tables; however, rows can be intermingled. Use with caution on small static read only tables.
Partitioned Tablespace - NUMPARTS 18
Partitioned tablespaces allow only 1 table. Maximum number of partitions 254. Above 64 considered Large.
Table Space Scans
Sequential I/O
Random I/O
SELECT PEAKNAME FROM PEAKS ORDER BY RATING;
SELECT PEAKNAME FROM PEAKS WHERE STATE = ‘CA’ AND RATING = ‘MODERATE’;
AK
CA
CO
WA
LEGEND
Buffer Manager initiates a GETPAGE request for P1 through P16.
Buffer Manager initiates a GETPAGE request for P3 and P13.
Sequential Detection
20320 AK Mount McKinley …
14831 AK Mount Bear …
Page 502
14494 CA Mount Whitney …
14420 CO Mount Harvard …
Page 503
14410 WA Mount Rainier …
14270 CO Grays Peak …
Page 504
14255 CO Longs Peak …
14196 CO Mount Yale …
Page 505
14172 CO Mount Bross …
14110 CO Pikes Peak …
Page 506
14080 CA Starlight Peak …
14059 CO Sunlight Peak …
Page 507
14058 CA Split Mountain …
14037 CO Little Bear Peak …
Page 508
14000 CA Thunderbolt Peak …
Page 501
SELECT PEAKNAME FROM PEAKSWHERE STATE = ‘CO’;
14005 CO Mount o/t Holy Cross …
p. 507p. 502 p. 503 p. 504 p. 505 p. 506 p. 508 p. 532
SELECT PEAKNAME FROM PEAKSWHERE STATE = ‘AK’;
p. 564p. 501 p. 502 p. 522 p. 542 p. 562 p. 566 p. 583
ACTIVE6 out of 8
NOT ACTIVE3 out of 8
(data access sequential = 4 out of last 8 pages are page-sequential)
1 1 1 1 1 1 24
1 20 20 20 2 2 17
p. 570
p. 630
38
47
Minimize Logging
Default column placement from data model.
Blue represents static columns that do not change once the row is created
Purple represents column grouping updated associated to daily billing transactions
Teal represents column grouping updated associated to monthly inventory transactions
Dashed box represents variable data
Adjusted column placement taking logging considerations into play.
Tuning SQL
Tuning Steps Static and Dynamic SQL Get What You Need, Not What You Want Reoptimization Explain & Optimization Hints
WHAT is “Estimate & Select Access Path with Cheapest Estimated Cost ”
Proper statistics can reduce the amount of SQL tuning required
Lackluster statistics can be a Pandora's box!– REBIND DATE < RUNSTATS DATE– RUNSTATS performed yearly– EXPLAIN(NO)
Optimizer does not use Real Time Stats for access path selection
Better filtering = fewer rows returned
Get Only What You Need
data manager
relational data system
filtered result set
GETPAGE REQUEST
BP3
sort
DBM1 Address Space
buffer manger
DSNDB07
DASD
SELECT * FROM TBL
data manager
relational data system
filtered result set
GETPAGE REQUEST
BP3
sort
buffer manger
DSNDB07
DASD
SELECT C5,C6,C7 FROM TBL
Performance, Maintenance, Control
ROLE ROLE DESCRIPTION0001 Database Administrator0002 System Programmer0003 Application Programmer0004 Data Modeler
EMP_ID ROLE PROJECT_1ST PROJECT_2ND77009134 0001 CLIENT REWRITE AGENT COMPENSATION77009134 0003 CUST RETENTION93989203 0002 AGENT COMPENSATION CUST RETENTION99080622 0004 CLIENT REWRITE CUST RETENTION
DELETE RESTRICT
IT_ROLE (parent) PROJECT_ROLE (child)
EMP_ID ROLE PROJECT_1ST PROJECT_2ND77009134 Database Administrator CLIENT REWRITE AGENT COMPENSATION77009134 Application Programmer CUST RETENTION93989203 System Programmer AGENT COMPENSATION CUST RETENTION
PROJECT_ROLE (column constraint)
. DDL change to add a new role (data modeler)
. DBA involvement needed for the table change
. Better Performance, Tighter Control
. Insert new row to add a new role (data modeler)
. User, Programmer, or DBA can insert the row
. Slower performance
(DB2 RI)
Getting Your Way
REOPT(VARS) – – Useful when DB2’s estimate of qualifying
rows would benefit if the real host variable was available to the optimizer at execution
– Evaluate data values at runtime– Limit parts for partition scans– Influence join sequence
Reoptimization is done at open cursor Static SQL – package level. Consider
isolating/consolidating your reopt statements to a few static packages.
Dynamic SQL – statement level. More granular and easier to invoke on a smaller scale.
IFCID 0022 – Did you get a better access path?
Would static SQL benefit from being dynamic and taking advantage of dynamic cache?
Sounds Great! Difficult to implement and maintain.
DSNZPARM change to activate Programmers should add QUERYNO to their
code. How are you going to name and manage your
opthints? CHAR(8) After code has been bound with explain yes,
need to update plan_table rows to add a OPTHINT.
To return to the good access path you have previously established, you would rebind your package with OPTHINT(‘ohstring’).
Verify hint is in use! SQL code +394, HINT_USED column of PLAN_TABLE, or Query special register: CURRENT OPTIMIZATION HINT
REOPTIMIZATION OPTIMIZATION HINTS
Access Paths
More on QUERYNO = 103; DB2’s Choices Understanding Predicates Small Tables in Memory Plan_Table Considerations
ACCESSTYPE in Plan Table
RID = record identifier
ACCESNAME
R Table space scan, segmented tablespace
Table space scan, non-segmented tablespace
Table space scan, partition scan (scanning limited parts)
I mc=0
Non-matching index scan (scan of index leaf pages)
One fetch index access (min/max)
Perform 1 matching index scan for each value in (LIST)
I1
MATCHCOLS
PREFETCHN
M Multiple index scan (same index twice or two different indexes)
MX
Qualifying RIDs for each index used in multiple index scan
MI AND predicate, intersection of the qualifying RIDs from each index
MU OR predicate, union of the qualifying RIDs from each indexOPTHINT
INDEXONLY
PRIMARY_ACCESSTYPE
D Direct row access via ROWID (page#, rid). One getpage!
I mc>0
Matching index scan (using IX structure - root, non-leaf, leaf)
Predicates
Know Your Predicates– Indexable, matching predicates on index key columns
– Stage 1, not picked as matching but still refer to index columns (index screening)
– Stage 1, operating on a data page
– Stage 2, everything else
Application Programming and SQL GuideTable 68, page 629, DB2 for OS/390 and z/OS V7
Peak Name State Elevation Latitude Longitude Rating
Mount McKinley AK 20320 630410N 1510013W VERY DIFFICULTMount Bear AK 14831 611702N 1410832W MODERATEMount Whitney CA 14494 363438N 1181733W MODERATEMount Harvard CO 14420 385528N 1061912W EASYMount Rainier WA 14410 465110N 1214531W DIFFICULTGrays Peak CO 14270 393802N 1054901W EASYLongs Peak CO 14255 401517N 1053655W MODERATEMount Yale CO 14196 385039N 1061848W EASYMount Bross CO 14172 392007N 1060625W EASYPikes Peak CO 14110 385026N 1050238W EASYStarlight Peak CA 14080 370539N 1183119W EASYSunlight Peak CO 14059 373738N 1073543W MODERATESplit Mountain CA 14058 370116N 1182520W MODERATELittle Bear Peak CO 14037 373400N 1052948W MODERATEMount o/t Holy Cross CO 14005 392805N 1062845W EASY MODERATEThunderbolt Peak CA 14000 370542N 1183124W EASY
PEAKS Table
Questions...