amsaa maturity projection model (ampm) user guide€¦ · model (ampm) october 2009 . u.s. army...

USER GUIDE FOR THE AMSAA MATURITY PROJECTION MODEL (AMPM)

OCTOBER 2009

U.S. ARMY MATERIEL SYSTEMS ANALYSIS ACTIVITY ABERDEEN PROVING GROUND, MARYLAND 21005-5071

DISTRIBUTION LIMITED TO U.S. GOVERNMENT AGENCIES AND THEIR CONTRACTORS; ADMINISTRATIVE OR OPERATIONAL USE; JANUARY 2005. OTHER REQUESTS FOR THIS DOCUMENT SHALL BE REFERRED TO DIRECTOR, U.S. ARMY MATERIEL SYSTEMS ANALYSIS ACTIVITY, APG, MD 21005-5071.

AMSAA MATURITY PROJECTION MODEL (AMPM)

2

Contents 1 AMSAA Maturity Projection Model. ..................................................................................... 7

1.1 Individual B-mode Time Data. ....................................................................................... 8 1.2 Single Fix Effectiveness Factor (FEF) Method. ............................................................. 8 1.3 Gap Method. ................................................................................................................. 10 1.4 Segmented Fix Effectiveness Factor Method. .............................................................. 10

2 Option for Individual B-mode Time Data. ........................................................................... 11 2.1 Single Fix Effectiveness Factor Method. ...................................................................... 11

2.1.1 Input Phase. ....................................................................................................................... 11 2.1.2 Application of the Single FEF Method. ............................................................................ 13 2.1.3 Model Results. .................................................................................................................. 17 2.1.4 Reliability Plots. ................................................................................................................ 22 2.1.5 Goodness-of-Fit. ............................................................................................................... 28

2.2 Gap Method. ................................................................................................................. 30 2.2.1 Rationale for Using the Gap Method. ............................................................................... 30 2.2.2 Application of the Gap Method. ....................................................................................... 34 2.2.3 Model Results. .................................................................................................................. 39 2.2.4 Reliability Plots. ................................................................................................................ 40 2.2.5 Goodness-of-Fit. ............................................................................................................... 45

2.3 Restart Method. ............................................................................................................. 47 2.3.1 Rationale for using the Restart Method. ........................................................................... 47 2.3.2 Application of the Restart Method. ................................................................................... 47 2.3.3 Model Results. .................................................................................................................. 49 2.3.4 Reliability Plots. ................................................................................................................ 51 2.3.5 Goodness-of-fit. ................................................................................................................ 56

2.4 Segmented Fix Effectiveness Factor (FEF) Method. .................................................... 57 2.4.1 Rationale for Using the Segmented FEF Method. ............................................................ 57 2.4.2 Application of the Segmented FEF Method. .................................................................... 57 2.4.3 Model Results. .................................................................................................................. 60 2.4.4 Reliability Plots. ................................................................................................................ 62 2.4.5 Goodness-of-Fit. ............................................................................................................... 69 3 Option for Grouped Data. ..................................................................................................... 70

3.1 Input Phase. ................................................................................................................... 71 3.2 Model Results. .............................................................................................................. 75 3.3 Reliability Plots. ............................................................................................................ 77 3.4 Goodness-of-fit. ............................................................................................................ 81


3

Figure 1. AMPM Startup Window................................................................................................. 7 Figure 2. Main Inputs Window ....................................................................................................... 8 Figure 3. Input Window for AMPM Individual B-mode Time Data ........................................... 11 Figure 4. Input Window for AMPM Individual B-mode Time Data Following a Successful Input Sequence ....................................................................................................................................... 16 Figure 5. Model Results for AMPM Individual B-mode Time Data ........................................... 17 Figure 6. Prepare to View Plots for AMPM Individual B-mode Time Data ............................... 21 Figure 7. Plot of Expected versus Observed Number of B-modes for Individual B-mode Time Data ............................................................................................................................................... 22 Figure 8. Plot of Percent Surfaced of B-mode Initial Failure Intensity for AMPM .................... 23 Figure 9. Plot of Expected Rate of Occurrence of New B-modes for AMPM ............................ 24 Figure 10. Plot of Rate of Occurrence of B-modes Versus Moving Average at Intersection of Curve for AMPM .......................................................................................................................... 25 Figure 11. Entering Group Size for Plot of Moving Average for AMPM (Partial Window) ...... 26 Figure 12. Plot of Rate of Occurrence of B-modes Versus Moving Average at Midpoint of Group Interval for AMPM ............................................................................................................ 26 Figure 13. Preparing to Plot Projected Expected Number of B-modes (Partial Window) .......... 27 Figure 14. Plot of the Projected Expected Number of B-modes for AMPM ............................... 27 Figure 15. Plot of MTBF Projections for AMPM........................................................................ 28 Figure 16. Goodness-of-Fit Results (Chi-square Test) for AMPM ............................................. 29 Figure 17. Help Page for AMPM Reliability Plots ...................................................................... 30 Figure 18. Example Curve for Illustrating the Gap Method ........................................................ 31 Figure 19. Estimated Expected Rate of Occurrence of New B-modes ........................................ 32 Figure 20. MTBF Projection Curve ............................................................................................. 33 Figure 21. Input Window for AMPM Prior to Check Input Event .............................................. 34 Figure 22. Input Window for AMPM Individual B-mode Time Data > Gap Method Option Following a Successful Input Sequence ....................................................................................... 38 Figure 23. AMPM Results Using the Gap Method ..................................................................... 39 Figure 24. Tables of Model Results for AMPM – Gap Option ................................................... 40 Figure 25. Visual Goodness-of-Fit with AMPM (Gap Method, v = 250 Hours) ........................ 41 Figure 26. Entering the Total Test Time for Plotting the Percent Surfaced of the B-mode Initial Failure Intensity (Partial Window) ............................................................................................... 42 Figure 27. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity....................................................................................................................................................... 42 Figure 28. Entering the Ending Value and Starting Value for the B-mode Rate of Occurrence Curve (Partial Window) ................................................................................................................ 43 Figure 29. Plot of the Expected Rate of Occurrence of New B-modes ....................................... 43 Figure 30. Entering the Ending Value for the Projected Expected Number of B-modes Curve (Partial Window) ........................................................................................................................... 44 Figure 31. Plot of the Projected Expected Number of B-modes.................................................. 44 Figure 32. Plot of MTBF Projections for AMPM (Gap Option, v = 250 Hours) ........................ 45 Figure 33. Entering the Number of Groups for the Chi-square Test ........................................... 46 Figure 34. Goodness-of-Fit Results (Chi-square Test) for AMPM Gap Option ......................... 46 Figure 35. Gap Method Restart Method Main Window .............................................................. 48 Figure 36. Restart Method Gap Cases .......................................................................................... 48 Figure 37. Restart Method Gap Case 2 Window .......................................................................... 49


4

Figure 38. Individual B-Mode Time Data Tables for Restart Method Case 2. ............................ 50 Figure 39. Restart Method Goodness-of-fit graph. ...................................................................... 52 Figure 40. Entering the Total Test Time for Plotting the Percent Surfaced of the B-mode Initial Failure Intensity (Partial Window) ............................................................................................... 52 Figure 41. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity....................................................................................................................................................... 53 Figure 42. Entering the Ending Value and Starting Value for the B-mode Rate of Occurrence Curve (Partial Window) ................................................................................................................ 53 Figure 43. Plot of the Expected Rate of Occurrence of New B-modes ....................................... 54 Figure 44. Entering the Ending Value for the Projected Expected Number of B-modes Curve (Partial Window) ........................................................................................................................... 54 Figure 45. Plot of the Projected Expected Number of B-modes.................................................. 55 Figure 46. Input window for Restart Method. .............................................................................. 55 Figure 47. Plot of MTBF Projections for AMPM (Restart Method, v = 250 Hours) .................. 56 Figure 48. Restart Method Goodness-of-fit Case 2. ..................................................................... 57 Figure 49. Example Curve for Illustrating the Segmented FEF Method ..................................... 58 Figure 50. Input Window for AMPM Prior to Check Input Event .............................................. 59 Figure 51. Main Input Window for FEF Method. ........................................................................ 60 Figure 52. Model Results for AMPM using Segmented FEF Method ........................................ 61 Figure 53. Tables of Model Results for AMPM Segmented FEF Method .................................. 62 Figure 54. Visual Goodness-of-Fit with AMPM Segmented FEF Method ................................. 64 Figure 55. Entering the Total Test Time for Plotting the Percent Surfaced of the B-mode Initial Failure Intensity (Partial Window) ............................................................................................... 64 Figure 56. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity for AMPM Segmented FEF Method............................................................................................. 65 Figure 57. Entering the Ending and Starting value for Plotting the Estimated Expected B-mode Rate of Occurrence (Partial Window)........................................................................................... 66 Figure 58. Plot of the Estimated Expected Rate of Occurrence of New B-modes for AMPM Segmented FEF Method ............................................................................................................... 66 Figure 59. Entering the Ending Value for the Projected Expected Number of B-modes Curve (Partial Window) ........................................................................................................................... 66 Figure 60. Plot of the Projected Expected Number of B-modes for AMPM Segmented FEF Method .......................................................................................................................................... 67 Figure 61. Entering the Ending Value and Starting Value for MTBF Projections Plot (Partial Window) ....................................................................................................................................... 67 Figure 62. Plot of MTBF Projections for AMPM Segmented FEF Method ............................... 68 Figure 63. Goodness-of-Fit Results (Chi-square Test) for AMPM ............................................. 69 Figure 64. AMPM Options Selection (partial view) .................................................................... 70 Figure 65. Input Window for AMPM Grouped Data .................................................................. 71 Figure 66. Input Window for AMPM Grouped Data (Partially Completed) ............................... 72 Figure 67. B-Modes Data Input Window for AMPM Grouped Data ........................................... 72 Figure 68. AMPM Grouped Data Model Results ........................................................................ 75 Figure 69. Tables of Model Results for AMPM Grouped Data .................................................. 76 Figure 70. Visual Goodness-of-Fit for AMPM Grouped Data Method ....................................... 78 Figure 71. Entering an Ending Value for the Estimated Expected Percent Surfaced Curve for the Grouped Data Option (Partial Window) ....................................................................................... 78


5

Figure 72. Plot of the Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity......................................................................................................................................... 79 Figure 73. Plot of the Estimated Expected Rate of Occurrence of New B-modes for the AMPM Grouped Option ............................................................................................................................ 80 Figure 74. Entering an Ending and Starting Value for Plotting the MTBF Projections .............. 80 Figure 75. Plot of MTBF Projections for AMPM Grouped Data ................................................ 81 Figure 76. Goodness-of-Fit Results for AMPM Grouped Data ................................................... 81


6

THIS PAGE INTENTIONALLY LEFT BLANK


7

1 AMSAA Maturity Projection Model. The AMSAA Maturity Projection Model (AMPM) provides estimates for:

• The B-mode initial failure intensity • The expected number of B-modes surfaced • The expected percent surfaced of the B-mode initial failure intensity • The expected rate of occurrence of new B-modes • The projected failure intensity • The projected mean time between failures (MTBF)

AMPM does not require that all fixes be delayed until the end of the current test phase (as does the AMSAA-Crow Projection Model). It only assumes that corrective actions are implemented prior to the time at which projections are made. In addition, this model allows for projections at future milestones beyond the start of the follow-on test phase.

Figure 1. AMPM Startup Window

Select Go to Main Inputs Page


8

Figure 2. Main Inputs Window

There are two options for this model: (click on box for drop down list) 1. Individual B-mode Time Data - choose this option if the B-mode first occurrence times

are known 2. Grouped Data - choose this option if the actual B-mode first occurrence times are

unknown, but you are able to determine the time intervals and the number of new B-modes in each time interval.

1.1 Individual B-mode Time Data. Using this option, there are three further methods to choose from: (click on second box for drop down list) Single FEF Method, Gap Method, and Segmented FEF Method.

1.2 Single Fix Effectiveness Factor (FEF) Method. There will be two further options to choose from: (click on third box for drop down list)


9

• Option #1 - choose this option if ALL fixes were delayed (i.e. NO fixes were implemented during test). Then you must pick either Case A or Case B. (click on fourth box for drop down list)

o Case A - choose this case if there are a number of B-mode repeats (becomes

advantageous to use the additional information provided by the repeat data) o Case B - choose this case if there are no B-mode repeats (in this case, model

estimates will be based solely on B-mode first occurrence times) • Option #2 - choose this option if not all fixes were delayed (i.e. some or all fixes

were implemented during the test) (this option uses B-mode first occurrence times only)


10

1.3 Gap Method. Use this method to "jump the gap" when then initial rate of occurrence of B-modes is very steep due to start-up problems such as:

• Vendor problems; Initial assembly problems due to inadequate workmanship; Inferior parts selection; Excessive variation in the material; and Operator unfamiliarity with the system. A gap point, v, is chosen and all of the B-modes contained within the gap, [0, v], are excluded.

• If your gap size is too large, you will also then have the option of choosing the Restart

Method, in which all of the B-modes which occur after the gap point, v, are shifted to the left by the size of the gap.

1.4 Segmented Fix Effectiveness Factor Method. Use this method when the initial rate of occurrence of B-modes is very steep due to the start-up problems mentioned above, and engineering analysis determines that the early B-modes (those on or before a partition point v) are aggressively and effectively corrected, thus having a higher FEF of d1 than the B-modes beyond the partition point.


11

On the next screen, based upon which options you choose, inputs specific to those options will appear for you to enter.

2 Option for Individual B-mode Time Data. If the B-mode occurrence times are known, then use the first option for “Individual B-mode Time Data.” If the actual B-mode occurrence times are unknown but you are able to determine the time intervals and the number of new B-modes in each time interval, then use the second option for “Grouped Data.” For individual B-mode time data, there are three options:

• Single Fix Effectiveness Factor method • A “gap” method • A “segmented fix effectiveness factor” method

2.1 Single Fix Effectiveness Factor Method.

2.1.1 Input Phase. You need to first choose “Individual B-Mode Time Data” as the Data Category by clicking on the first box, selecting “Individual B-Mode Time Data” from the drop down list and then clicking the “NEXT” button (located to the right of the first box). For Individual B-Mode Time Data, the window will look like the following (after clicking the “NEXT” button):

Figure 3. Input Window for AMPM Individual B-mode Time Data

The input window for the Single FEF option requires the user to make further decisions about which option their data fits. To choose the Single FEF option, click on the second box, select “Single FEF Method” from the drop down list, and then click the “NEXT” button located to the right of second box. The following box now appears for the user to choose either Option #1 (all fixes delayed) or #2 (not all fixes delayed).


12

Recall that AMPM does not require that all fixes be delayed until the end of the current test phase. Therefore, the model provides two options with respect to fix implementation. If ALL fixes are implemented at the END of the test phase, then choose the first option. If not ALL of the fixes are implemented at the END of the test phase, then choose the second option. If there are any doubts about fix implementation, use option two. Click on the third box, select either Option #1 or Option #2 from the drop down list, and then click on the “NEXT” button to the right of this box. If Option #1 is selected, the following box now appears for the user to select either Case A (have B-mode repeats) or Case B (no B-mode repeats). A fourth box appears after selecting NEXT.

Click on the fourth box, select either Case A or Case B from the drop down list, and then click on the “NEXT” button (located to the right of the fourth box). If you choose Case A, the window changes to show the following:


13

If Case A is chosen, the required inputs are shown in the figure above. The user must enter the following:

1. Total Test Time, (T > 0) 2. # of Observed B-Modes 3. # of A-Mode Failures 4. # of Projections to Make 5. Finite K (assumed # of B-modes) 6. Total # of B-mode Failures (1st occurrences and repeats)

If Option #1 > Case B or Option #2 is chosen instead, the only difference in the required inputs is that the user does not enter the Total # of B-mode Failures.

2.1.2 Application of the Single FEF Method. Start with the first of the require inputs, the total amount of test time. For our test example, enter the following as a sample test:

a. enter 400 hours for total test time b. enter 16 for the number of observed B-modes c. enter 10 for the # of A-Mode Failures (this includes first occurrences plus repeats) d. enter six for the number of projections to make e. for the number of B-mode failures, which include first occurrences plus repeats, enter

32, indicating that there were 16 B-mode repeats Within the context of the AMPM, the symbol “K” represents the finite number of B-modes in the system. In practice, the value of K may not be known, but an assessment of K may be available


14

if a Failure Modes and Effects Criticality Analysis have been performed. For a complex system, it has been our observation that the choice of K ≥ 10M, where M is the number of observed B-modes by time T, results in AMPM projection quantities insensitive to K. Therefore, enter 160 in the cell for “Finite K.” Now that all the required input data have been entered, select the “Check Inputs” button to make sure that all of the inputs entered are correct. Next select the “B-Mode Data Inputs” button. This will take you to the B-Mode Data Inputs tab (shown below) where you can enter all of your first occurrence times for each of the observed B-modes. Also to be entered is a list of times where projected quantities are to be computed. These estimated projection quantities consist of –

• The expected number of B-modes • The expected percent surfaced of the B-mode initial failure intensity • The expected rate of occurrence of new B-modes • The projected failure intensity • The projected MTBF

Enter the B-Mode 1st Occurrence Times and Projection Times as shown above.


15

The user has two options with regard to the fix effectiveness factor used in the calculation of the model estimates.

1. The user can enter a fixed average fix effectiveness factor (useful in doing sensitivity analyses) in the FEF cell located on the Main Inputs tab, or

2. The user can enter a fix effectiveness factor for each individual observed B-mode, and then once on the Main Inputs tab, they can click the “Calculate Average FEF” button and let the program calculate the average FEF of all the individual FEFs entered.

Now select the “Check B-Mode Data Inputs” button to make sure all of the data entered is correct. (Note: If the user decides not to enter FEFs for each of the individual B-modes, then when the “Check B-Mode Data Inputs” button is clicked and the following message appears, just click “No”.)

In our example, we are not going to be entering FEFs for each of the individual B-modes, so click “No” when the above message appears. If the model finds no problems with the data, the following message will appear:

Click the “OK” button and then click the “Return to Main Inputs” button (located on the right side of the screen). This will return the user to the MAIN INPUTS tab where the user can then either enter their own average FEF or click the “Calculate Average FEF” button. For our example, enter an average FEF of 0.70 in red outlined cell shown below:


16

Figure 4. Input Window for AMPM Individual B-mode Time Data Following a Successful Input Sequence

Next click on the “Check FEF” button. The program will check your entered FEF and make sure it is in the interval of (0, 1) and that it is not a letter or other character. If there are no problems found with your chosen FEF, the following message will be displayed:

Click “OK”.


17

At this point, even there is still an opportunity to change individual inputs or clear all the inputs by either changing any of the inputs or selecting the “Clear All Inputs” button. Assume for the purposes of this example that no changes are necessary, and select the “Calculate!” button. The program does one final check of all of the input data (from both input tabs), and if a problem with the input data is found, a message will appear. In our example, we chose not to enter individual FEFs for each of the 16 observed B-modes, so the program will ask again if we intend to enter the individual FEFs. Again click on the “No” button and then click on the “OK” button when the program produces the message telling us that the program may proceed. This will take you to the Main Outputs tab where all of the model results (and some of the model inputs) are displayed.

2.1.3 Model Results.

Figure 5. Model Results for AMPM Individual B-mode Time Data


18

The first four lines of output display the entries made on the Main Input tab which are common to all four of the methods of this model: the total amount of test time, the total number of observed B-modes by the end of the test period T, the average fix effectiveness factor and the total number of observed A-mode failures. (Note: If Option #1 -> Case A is chosen, then also displayed in the top section is the total number of B-mode failures (1st occurrences + any repeats). The fifth line from the top is an estimate of the A-mode failure rate, which is the ratio of the total number of A-mode failures to the total amount of test time. The middle section displays the inputs and outputs unique to the Single FEF Method. The first line is the initial (assumed) number of B-modes, K which was chosen on the Main Inputs tab. Next two estimates for the MTBF Growth Potential are shown: one for the limiting case (K → ∞, or Infinite K) and one for the Finite K case. Recall that the MTBF growth potential represents an upper bound on the system MTBF under the assumption that all the B-modes in the system have been surfaced and fixed with effectiveness (in our case) equal to 0.7. For informational purposes, at the bottom of the middle section, the model displays the smallest integer for the initial finite number of B-modes in the system for which model estimates exist. This number is derived from the observed data and the underlying model assumption. Though not shown, the MTBF growth potential for the finite K case (where K = 23 is the smallest integer for the initial number of B-modes for which model estimates exist) is calculated to be 21.93. The three lines below the MTBF growth potential provide numerical estimates of key model parameters, beginning with the B-mode initial failure intensity, followed by the two parameters of the model, alpha and beta. Note the estimate for alpha for the limiting case, which is -1, is not displayed. Estimates for beta for the limiting and finite cases are shown and, for our choice of K = 160, are relatively close. One possible interpretation of the estimate for beta is the following: Given a mode occurs at or near the origin (at the beginning of the test) then beta approximates the expected failure rate of that mode. The box to the right of the middle section reflects the fix implementation choice made previously. This type of information is important to have displayed in close proximity to the model estimates because the choice of fix implementation has a direct effect upon the estimates.

The bottom section displays the number of projections inputted on the Main Inputs tab.


19

The right side of the Main Outputs tab is reserved for navigation and printing. At the top of the right side, the user can click to:

o Return to the Main Inputs tab o Go back to the B-Mode Data Inputs tab, or o Go to the Table Outputs tab and view any of the four provided output tables

(along with a summary of the B-mode 1st occurrence times entered) Click on the “Go to Table Outputs” button and the following screen will be shown (broken into three sections for display purposes):


20


21

The values in Tables 1-4 are indexed by the list of projection times entered in the “B-Mode Data Inputs” tab. Each table consists of a column of estimates for the Infinite case followed by a column of estimates for the Finite K case. Click on the “Return to Main Outputs” button to go back to the buttons for viewing the various plots.

Figure 6. Prepare to View Plots for AMPM Individual B-mode Time Data


22

2.1.4 Reliability Plots. Two of the buttons pertain to the finite case (Visual GOF for finite case and there is a section in the Chi-Square Goodness-Of-Fit Table for the finite case); the rest apply to the infinite case. Start with the two Visual GOF buttons on the right side (third and fourth buttons down) and select the “Visual GOF (Infinite Case). This provides a visual plot of the goodness-of-fit of the model. It shows how closely the model, as represented by the smooth curve, captures the overall trend of the observed data points. The shape of the smooth curve is also important as it is an indicator of the rate of occurrence of new B-modes. A concave shape (as shown by our situation) indicates that the rate of occurrence of new B-modes is decreasing with time, which means that the system is maturing. The model would represent a constant or increasing rate of occurrence of new B-modes as a straight line, in which case the model should not be used to make projections with such a dataset.

Figure 7. Plot of Expected versus Observed Number of B-modes for Individual B-mode Time Data


23

Next select the “Plot of Percent Surfaced of B-mode Initial Failure” button. On this tab you will have to input an ending value for the curve (see figure below). For our example, enter 500 and click “Create Plot.”

Figure 8. Plot of Percent Surfaced of B-mode Initial Failure Intensity for AMPM

This plot shows that, according to model estimates, approximately 70 percent of the initial B-mode failure intensity has already been surfaced by the end of the 400 hour test, and that with an additional 100 hours of test time (under the assumption that the rate of occurrence pattern of new B-modes continues to hold) an additional 5 percent of the initial B-mode failure intensity is expected to be surfaced. Now select the “Plot of Expected Rate of Occurrence of New B-modes” button. On this tab you will have to enter two inputs in order to generate this curve. The first input is an ending time for the rate of occurrence function (the curve), and the second input is a starting point. With regard to the ending value, the model can be used to make projections beyond the data if you are willing to accept the assumption that the rate of occurrence pattern that the model predicts will continue


24

to hold in the near future. Even with that assumption, projections should probably be made only slightly beyond the range of the data. With regard to the starting point for this model, the rate of occurrence of new B-modes at the origin is finite, so a starting point equal to zero is permissible. For our example, enter 500 for the ending value and 0 for the starting value, and click the “Create Plot” button. The resulting curve is shown.

Figure 9. Plot of Expected Rate of Occurrence of New B-modes for AMPM

Note that the estimated expected rate of occurrence of B-modes at the origin (approximately 0.075 from the graph) corresponds closely to the estimated initial B-mode failure intensity (0.07586161). The graph is clearly showing that the rate of occurrence of new B-modes is decreasing. The next two plots continue with this theme but they incorporate the notion of a moving average with respect to the observed data. Moving average in this context is a method for estimating the rate of occurrence based on groups of observed B-modes. Select the “Plot of Rate of Occurrence of B-modes vs Moving Avg. at Intersection of Curve” button. Plotting the moving average at the intersection means the moving average is plotted where it intersects the model curve. On this tab you will have to enter a group size, where the group size is the number of observed B-modes per group over which the moving average is calculated, and then click the


25

“Create Plot” button. A group size of five B-modes is a reasonable compromise for this type of analysis, but the use of different group sizes may provide additional insight. An advantage of the moving average method is that it does not assume any specific model or distribution – it is based solely on the observed data. It is useful for portraying a general trend in the observed B-mode occurrence rate with respect to time. The moving average curve (in red) is plotted along with the model curve (in blue) for comparison purposes.

Figure 10. Plot of Rate of Occurrence of B-modes Versus Moving Average at Intersection of Curve for

AMPM

The next figure is a plot of the moving average at the midpoint, which means the moving average is plotted at the midpoint of the group interval. On this tab you will have to enter a group size, where the group size is the number of observed B-modes per group over which the moving average is calculated, and then click the “Create Plot” button.


26

Figure 11. Entering Group Size for Plot of Moving Average for AMPM (Partial Window)

Figure 12. Plot of Rate of Occurrence of B-modes Versus Moving Average at Midpoint of Group Interval for

AMPM

Next select the “Plot of Projected Expected Number of B-modes” button. On this tab you will have to enter an ending value for the curve, so choose 500, and then click the “Create Plot” button. With regard to the ending point, even though the model may be used to project the expected number of B-modes beyond the total test time T, choose a value carefully as any point beyond the range of the data assumes that the pattern that the model predicts will continue to hold. This assumption may hold for a short period of time slightly beyond the range of the data but tends not to hold far beyond the observed data as any change in test environment or deviation from the mission profile can cause a change in the pattern of B-modes.


27

Figure 13. Preparing to Plot Projected Expected Number of B-modes (Partial Window)

Figure 14. Plot of the Projected Expected Number of B-modes for AMPM

If we accept the assumption that the rate of occurrence of new B-modes pattern continues to hold beyond the range of the data, then the model predicts that an additional two B-modes are expected to be surfaced by the end of the next 100 hours of test time. This type of information can be helpful to management for purposes of planning test resources. For the final plot, select the “Plot of MTBF Projections” button. On this tab you will have to enter two inputs in order to generate this curve. The first input is an ending time, and the second input is a starting time. With regard to the ending value, choose 500. With regard to the starting value, a value at the origin is permissible, so choose zero. Then click on the “Create Plot” button.


28

Figure 15. Plot of MTBF Projections for AMPM

Note that these projections take into account the impact of A-mode failures, B-mode failures and fix effectiveness, and recall that fixes may be delayed or incorporated during the test phase.

2.1.5 Goodness-of-Fit. Chi-square goodness-of-fit tests are available for both the limiting case and the finite case. The data are grouped so that there are at least three groups for the test. This ensures at least one degree of freedom for entering a standard table of critical values. More than three groups are possible, subject to the constraint that the frequency of failures per group is at least three. (The constraint requiring an expected frequency per interval of at least five, which was used for the continuous tracking model, has been relaxed for this model.) Since there are 16 B-modes in our dataset, this means that the test can conceivably be run with as many as 5 groups. It is often informative to run the test over the range of possible numbers of groups. Click the “Chi-Square Goodness-of-Fit Table” button. On this tab you can enter the number of groups for both the limiting (infinite) case and the finite case. First scroll over to the right to the Infinite Case and


29

enter 3 for the number of groups in the cell outlined in red. Then click on the “Do Infinite Chi-Square GOF” button. The results follow.

Figure 16. Goodness-of-Fit Results (Chi-square Test) for AMPM

Note that the model cannot be rejected at the 20 percent significance level for one degree of freedom. Further, though not displayed here, the model could not be rejected at the 20 percent significance level for two and three degrees of freedom as the chi-square statistics were 1.50 and 0.88, respectively. Based on these tests, the applicability of the model for this dataset is accepted. Finally, a “Help” button is available to provide further explanation of the plots and goodness-of-fit tests for this model.


30

Figure 17. Help Page for AMPM Reliability Plots

2.2 Gap Method.

2.2.1 Rationale for Using the Gap Method. From previous discussions with regard to the AMSAA-Crow Projection Model and the AMSAA Maturity Projection Model, it was stated that a projection is a reliability estimate that takes into account the contribution of A-mode failures, B-mode failures and the effectiveness of the corrective actions that are implemented to the latter class of failure modes. The statistical reliability projection based on this information is significantly influenced by the initial B-mode rate of occurrence. For some systems under development, the initial rate of occurrence of the correctable failure modes (B-modes) is steep. This can be due to problems associated with infant mortality, initial assembly procedures or operator unfamiliarity with the system. One way of excluding the impact of such start-up problems on the reliability projection is to utilize only the test data beyond an initial time period. This is referred to as “jumping the gap.” For the case


31

where the individual B-mode occurrence times are known, the gap method may be an appropriate strategy to use for a dataset where the initial rate of occurrence of B-modes is very steep due to these start-up problems. Such a situation is described below.

Figure 18. Example Curve for Illustrating the Gap Method

One of the first considerations when doing a reliability projection analysis is a plot of the estimated expected versus the observed number of B-modes surfaced as a function of cumulative test time; as such a plot will reveal many important aspects of the data as well as the model being applied to the data. Besides showing the overall trend of the data and the total number of observed B-modes during the test period, this plot has the following features. First, it provides a visual perspective of the goodness-of-fit of the model and shows (non-statistically) the degree to which the model represents the observed test data. (A statistical procedure for testing the goodness-of-fit of the projection model will be discussed later.) Second, the shape of the curve illustrates whether the rate of occurrence of new B-modes is diminishing with time, as the concavity of this particular curve indicates that the rate is beginning to flatten out. Finally, this curve shows (indirectly) the rate of occurrence of new B-modes at the origin, since for the example curve, nearly a third of the total number of B-modes have occurred within the first 10 percent of the total test duration. The steepness of this initial rate of occurrence indicates there could be start-up problems for which a gap method may be useful. This steepness is further revealed by considering the slope of the above curve, which is the rate of change of the estimated expected number of B-modes with respect to time. This is shown below.


32

Figure 19. Estimated Expected Rate of Occurrence of New B-modes

The above curve is referred to as the rate of occurrence, or h(t), curve. Note the downward trend and the steepness at the origin. It can be shown that the initial B-mode failure rate is equal to h(0). For the example curve, the statistical estimate for the initial B-mode failure rate is approximately 0.22, which can also be approximated directly from the h(t) curve by reading off the rate of occurrence value at t = 0. In addition to providing an estimate for the initial B-mode failure rate, the rate of occurrence function, h(t), plays a key role in projecting the system reliability. Incidentally, for our example, the total test time equals 1856, the number of observed B-modes equals 96, the number of A-mode failures equals 0, the number of projections to make is 1, the total number of B-mode failures (which includes 1st occurrences plus repeats) equals 104, the assumed number (K) of B-modes initially in the system equals 960, and the overall average fix effectiveness equals .7. The resulting MTBF projection curve follows.


33

Figure 20. MTBF Projection Curve

Typically, MTBF projections are of most interest at the end of the test period, T, and for milestones beyond T, provided the projections are not made too far beyond the range of the data. The curve above is plotted from the origin and over the entire range of the data simply to capture and display the trend with which the reliability is improving over time. At the end of the test, namely at T = 1856 hours, the projected MTBF is estimated to be approximately 12.8 hours. Given the steep rate of occurrence of B-modes, the gap method may be an appropriate strategy to use. By jumping the gap, this allows us to specify a segment of time at the beginning of the data so that the set of B-modes used in the analysis are those that have occurred after the gap. To use this approach, there ought to be some compelling reason, something peculiar about the initial test situation, that would justify using the gap method. In choosing a positive gap size, the underlying assumption is that there is a special group of B-modes occurring within the gap whose failure mechanisms are such that they can be assigned very high fix effectiveness factors (perhaps, close to 1), thus excluding them from consideration. It is equivalent to stating that there are two types of B-modes occurring within the gap: those with a collective failure rate λ1 due to start-up problems and those with a collective failure rate λ2 which are not due to start-up problems. Things that may contribute to start-up problems within the gap are:

• Vendor problems • Initial assembly problems due to inadequate workmanship • Inferior parts selection • Excessive variation in the material • Operator unfamiliarity with the system

In choosing potential gap sizes, it is important to use visual as well as statistical methods for assessing model goodness-of-fit. A good tool for visual purposes is a plot of the estimates for


34

the expected versus observed number of B-modes. A good statistical tool for assessing model applicability is the chi-square goodness-of-fit procedure that is shown later.

2.2.2 Application of the Gap Method. Suppose there were start-up problems that gave rise to the steep slope shown at the start of the example curve, and that it was determined by engineering analysis that it was reasonable to choose a gap size of v = 250 hours (v is a variable that represents the gap size). The following discussion shows the series of windows that one would see in applying the gap method. Begin on the Main Inputs tab by selecting from the first box, the option for “Individual B-mode Time Data.” Then click on the second box, select the “Gap Method” from the drop down list, and click the “NEXT” button located next to the second box. The total test time is 1856 hours, the number of observed B-modes is 96, the number of A-mode failures is 0, the number of projections to make is 1, and the gap point is 250.

Figure 21. Input Window for AMPM Prior to Check Input Event

After entering these inputs, you need to click on the “Check Inputs” button to make sure there are no problems with the inputs you just entered. If the program finds no problems, then click on the “Ok” button in the pop-up message and then click on the “B-Mode Data Inputs” button. You will be redirected to the “B-Mode Data Inputs” tab where you need to enter the following data:

o First Occurrence Time for Each Observed B-Mode o Individual FEF for Each Observed B-Mode (Optional) o Projection Time for each Projection selected

For our example enter the data as shown below:


35


36


37

Again we are not entering Individual FEFs for each observed B-mode, so after entering the above data, click on the “Check Inputs” button. Since we didn’t enter the Individual FEFs, the program will display a message asking if we intend to enter the Individual FEFs.

Click on “No” and then then following message will be displayed:

Click on “OK”, and then click on the “Return to Main Inputs” button which will return you to the Main Inputs window. The average FEF must then be entered in the cell outlined in red. For our example enter 0.70 for the average FEF. Next click on the “Check FEF” button. The program will check your entered FEF and make sure it is in the interval of (0, 1) and that it is not a letter or other character. If there are no problems found with your inputted FEF, the following message will be displayed:

Click “OK”. After entering and checking the average FEF, the Main Input tab will appear as follows:


38

Figure 22. Input Window for AMPM Individual B-mode Time Data > Gap Method Option Following a

Successful Input Sequence You are now ready to click on the “Calculate!” button and view the model results. Again at any time, you can clear all of the inputs by clicking on the “Clear All Inputs” button, or you can change individual inputs. The program does one final check of all of the input data (from both input tabs), and if a problem with the input data is found, a message will appear. In our example, we chose not to enter individual FEFs for each of the 96 observed B-modes, so the program will ask again if we intend to enter the individual FEFs. Again click on the “No” button and then click on the “OK” button when the program produces the message telling us that the program may proceed. This will take you to the Main Outputs tab where all of the model results (and some of the model inputs) are displayed.


39


Figure 23. AMPM Results Using the Gap Method

The first four lines of output display the entries made on the Main Inputs tab: the total amount of test time, the number of B-modes observed during the test, the total number of A-mode failures by T, and the average Fix Effectiveness Factor inputted. The next line provides an estimate for the A-mode failure rate, which is the ratio of the total number of A-mode failures to the total amount of test time. The first inputs/outputs specific to the Gap Method is the gap size v = 250. The next four lines of output provide model estimates. The estimate for beta (for the limiting case only) is shown. One possible interpretation of the estimate for beta is the following: Given a mode occurs at or near the origin (at the beginning of the test) then beta approximates the expected failure rate of that mode. Note that if we had run the model with this same dataset and a gap size of zero, the estimated initial B-mode failure rate has been reduced to approximately 0.07 from 0.22 for the original dataset. The next two lines of output display estimates for the failure rate growth potential and the MTBF growth potential, respectively. Recall that the failure rate growth


40

potential represents a lower bound on the system failure intensity and the MTBF growth potential represents an upper bound on the system MTBF under the assumption that all the B-modes in the system have been surfaced and fixed with average effectiveness equal to 0.7 (in our case). The last line for this section indicates that the first 37 B-modes are excluded from the analysis. The last section displays that we chose to make only one projection. Before looking at the available plots, select the “Go to Table Outputs” button. This will display a tab with four tables of important model results and also a summary of the first occurrence times of all of the observed B-modes entered (not shown in the figure below).

Figure 24. Tables of Model Results for AMPM – Gap Option

The values in the tables are indexed by the list of projection times entered in the “B-Mode Data Inputs” tab. Recall that we chose to compute estimates for a single projection time at t = 1856 hours. Table 1 provides estimates of the cumulative expected number of B-modes surfaced by the end of t, Table 2 provides estimates of the percent of the B-mode initial failure intensity surfaced by the end of t, Table 3 provides estimates of the expected rate of occurrence of B-modes by the end of t, and Table 4 displays the projected failure intensity and projected MTBF by the end of t. Now select the “Return to Main Outputs” button to return to the Main Outputs tab and select any of the five plot buttons to view available plots of reliability projection quantities or to get to the Chi-Square Goodness-Of-Fit Table.

2.2.4 Reliability Plots. Plots are available for the following:

• Visual goodness-of-fit • Estimated expected percent surfaced of the B-mode initial failure intensity • Estimated expected rate of occurrence of new B-modes • Projected expected number of B-modes • Projected MTBF • Statistical goodness-of-fit


41

Start at the top and select the “Visual Goodness-Of-Fit (Model vs. Observed B-modes) Plot” button.

Figure 25. Visual Goodness-of-Fit with AMPM (Gap Method, v = 250 Hours)

This plot provides a visual perspective of goodness-of-fit by showing how closely the model, as represented by the smooth curve, captures the trend of the observed data. Essentially, the model is fitted to the B-modes occurring beyond the gap, and the curve is interpolated back to the


42

origin. The model appears to represent the data very well, and the curve displays a nice, concave shape. Now select the “Plot of Percent Surfaced Initial B-mode Failure Intensity” button. Once on this tab, you will have to enter a test time input for the percent surfaced. For our example, enter 2300, as this number includes approximately 25 percent additional test time beyond the actual test time of 1856 hours, and then click on “Create Plot”. The input section and the resulting plot follow.

Figure 26. Entering the Total Test Time for Plotting the Percent Surfaced of the B-mode Initial Failure

Intensity (Partial Window)

Figure 27. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity

This plot shows that if the rate of occurrence of new B-modes were to continue at the same pace as expected through T = 1856 hours, then by the end of 2300 hours, approximately 70 percent of the B-mode initial failure intensity is expected to be surfaced. Now select the “Plot of Rate of Occurrence of New B-modes” button. You will have to enter two inputs for the curve. The first input is an ending value for the rate of occurrence function (the curve), and the second input is a starting value for the function. With regard to the ending


43

point, enter 2300. With regard to the starting point, the rate of occurrence of new B-modes at the origin is finite, so enter 0. Next click on the “Create Plot” button. These two input sections and the rate of occurrence curve follow.

Figure 28. Entering the Ending Value and Starting Value for the B-mode Rate of Occurrence Curve (Partial

Window)

Figure 29. Plot of the Expected Rate of Occurrence of New B-modes

Next select the “Plot of Projected Expected Number of B-modes” button. You will have to enter an ending value for the curve. With regard to the ending point, even though the model can project the expected number of B-modes beyond the total test time T, choose a value carefully as any point beyond the range of the data assumes that the pattern will continue to hold. For our example, enter 2300.


44

Figure 30. Entering the Ending Value for the Projected Expected Number of B-modes Curve (Partial

Window)

Figure 31. Plot of the Projected Expected Number of B-modes

If we accept the assumption that the rate of occurrence of new B-modes pattern continues to hold beyond the range of the actual data, then the model predicts that an additional 10 B-modes are expected to be surfaced with an additional 444 hours of testing. This type of information may be useful to management for purposes of planning test resources. Next select the “Plot of MTBF Projections” button. You will have to enter two inputs for the curve. The first input is an ending time, and the second input is a starting time. With regard to the ending value, enter 1856. With regard to the starting value, enter 0. Then click on the “Create Plot” button. The two input sections and the resulting plot follow.


45

Figure 32. Plot of MTBF Projections for AMPM (Gap Option, v = 250 Hours)

MTBF projections take into account the A-mode failure intensity (zero in our case), the B-mode failure intensity, and fix effectiveness. Recall that fixes may be delayed or incorporated during the test phase and that model estimates are based solely on B-mode first occurrence times. Note that without the gap, the projected MTBF after 1856 hours of test time was approximately 13 hours. With an initial gap size of 250 hours, the projected MTBF after 1856 hours of time is approximately 26 hours.

2.2.5 Goodness-of-Fit. A chi-square goodness-of-fit procedure is used to test the null hypothesis that the AMPM adequately represents the observed test data occurring beyond the gap. The procedure operates by partitioning the observed B-modes into N number of groups such that the number of B-modes per group is at least 3 (this is a relaxation of the recommended number of 5 but should be


46

sufficient for our purposes). Since the chi-square statistic is approximately distributed as a chi-square random variable with N-2 degrees of freedom, the number of groups must be at least 3. Therefore, there must be at least 9 B-modes beyond the gap to run the goodness-of-fit procedure. The null hypothesis is rejected if the chi-square statistic exceeds the critical value for a chosen significance level. The input box and the goodness-of-fit results are displayed below for three groups. Click the “Chi-Square Goodness-Of-Fit” button, enter a value of 3 in the cell outlined in red and then click “Do Infinite Chi-Square GOF.”

Figure 33. Entering the Number of Groups for the Chi-square Test

Figure 34. Goodness-of-Fit Results (Chi-square Test) for AMPM Gap Option

Note that the model cannot be rejected at the 20 percent significance level for one degree of freedom. Further, the model could not be rejected at the 20 percent significance level for 2, 3, 4 and 5 degrees of freedom as the chi-square statistics were 0.19, 1.08, 3.34 and 3.53, respectively. (These values were calculated by entering 4, 5, 6 and 7, successively, for the number of groups.) The model could not be rejected at the 15 percent significance level for 6 degrees of freedom as the chi-square statistic was 8.66. Based on these tests and the visual goodness-of-fit, there is not strong evidence to reject the model for this dataset. Therefore, we accept the applicability of the model and the resulting reliability metrics. In summary, some systems under development experience a very steep rate of occurrence of problem failure modes (B-modes) at the outset of testing. Some of these incipient B-modes are


47

due to design problems that are part of the normal corrective action process, which is dedicated toward mitigating their failure rates. Others are due to start-up problems, which as a class are generally more readily understood and capable of being eliminated. If the B-modes due to these start-up problems are included in the analysis, they could have a significant bearing upon the initial B-mode failure intensity, which in turn could have an appreciable negative impact upon the system reliability projection. One way of excluding the impact of these start-up problems on the projection is by jumping the gap. The effect of jumping the gap was illustrated by example.

2.3 Restart Method. Another approach to analyze the data would be to re-initialize the data beginning at the partition point, v. Thus, failure data prior to v would not be used in the analysis. Note that a repeat of a failure mode occurring prior to v that occurs after v may now be the first occurrence of that failure mode and thus included as a first occurrence.

2.3.1 Rationale for using the Restart Method. The rationale for implementing this approach versus the Gap Method may be due more for practical analysis or engineering concerns of the data such as significant changes in the systems configuration or possible differences in test conditions.

2.3.2 Application of the Restart Method. The basic difference between the Gap Method and the Restart Method is after determining a partition v, the Restart Method reinitializes test time to zero at v. And as noted above, first occurrences of modes prior to v having repeats after v would now, with the next occurrence of that mode, be considered first occurrences. There are two ways to access the Restart Method using this program. When running the program using the Gap Method with the inputted dataset, if the program finds that it cannot compute the model estimates, it will give the user the opportunity to try the inputted dataset with the Restart Method instead, using the same partition or gap point as the Gap Method used. The second way is also if using the Gap Method, on the Main Outputs tab, there is a button to run the Restart Method on the current dataset.


48

Figure 35. Gap Method Restart Method Main Window

To illustrate how the Restart Method works, go to the Main Inputs page, select “Individual B-Mode Time Data” and then select “Gap Method.” Scroll to the bottom of the screen and click on the “Sample Data Page” button. Find the Gap Method Data section and click on “Copy Gap Method Data.” This will automatically copy the dataset that was used previously in the Gap Method example, so all you have to do once on the Main Inputs page is to click on the “Calculate!” button. You will now see the above Main Outputs page. Click on the “Restart Method” button located on the far right of the Main Outputs page. Once you click the Restart Method button, the following page will appear:

Figure 36. Restart Method Gap Cases


49

With the Restart Method, you have two cases to choose from. In Case 1, you have at least one B-mode that falls within the Gap [0, v] that has at least one repeat outside of the Gap [0, v]. In this case is selected, then after clicking the “Case 1” button you will have to input on the next screen the repeat time(s) of the B-modes that are contained within the Gap [0, v]. In Case 2, you have no B-modes in the Gap [0, v] that have a repeat outside of the Gap [0, v]. Here, the B-modes either have no repeats at all or all of the repeats are contained within the Gap [0, v]. For Case 2, you have no further input to enter after clicking on the “Case 2” button. You also have the option of returning to the Main Inputs page or to rerun the original gap method data that was entered on the two input pages.

2.3.3 Model Results. For our example, we are going to assume that Case 2 is true in that none of our B-modes have repeats at all. Click on the “Case 2” button and the following screen will display the model results:

Figure 37. Restart Method Gap Case 2 Window

The first line is our new Total Test Time which is now 1606 hours instead of the original 1856 hours, since our partition/gap point was 250 hours (1856 – 250 = 1606). The number of observed B-modes also decreased from 96 to 59 since by shifting all of the B-mode first occurrence times to the right by 250 hours, we excluded the first 37 B-modes (same # of B-modes excluded as when we “jumped the gap” in the Gap Method). The number of A-mode failures remained the


50

same as the Gap Method, as did the Average Fix Effectiveness Factor (FEF) and the number of projections we chose to make, which was only one.

The first Restart Method output is the estimate of the model parameter beta. One possible interpretation of the estimate for beta is the following: Given a mode occurs at or near the origin then beta approximates the expected failure rate of that mode. The second output is the estimate of the initial B-mode failure intensity which can also be interpreted as the rate of occurrence of new B-modes at the origin (t = 0). If we compare this result to the estimated initial B-mode failure intensity for the Gap method, we can see that with the Restart Method the estimated initial B-mode failure intensity has decreased from 0.069 to 0.056. The next line of output provides as estimate for the Failure Rate Growth Potential, which represents a lower bound on the system failure intensity under the assumption that all the B-modes in the system have been surfaced and fixed with effectiveness equal to 0.70. The fourth line of output is the estimate of the MTBF growth potential which represents an upper bound on the system MTBF under the assumption that all the B-modes in the system have been seen and fixed with effectiveness equal to 0.70. It represents a best case MTBF estimate. The last line of output is the estimate of the A-mode failure rate which is the total number of A-mode failures divided by the total time, and for our example it is equal to zero since we had no A-mode failures.

Now click on the “Go to Table Outputs” button to view the available output tables. The values in the tables are indexed by the list of projection times chosen in the original dataset from the Gap Method (in our example, there is only one projection being made). Table 1 provides estimates of the cumulative expected number of B-modes surfaced, Table 2 provides estimates of the expected percent of the B-mode initial failure intensity surfaced, Table 3 provides estimated of the expected rate of occurrence of B-modes and Table 4 displays the projected failure intensity and projected MTBF (all shown below). Not shown in the figure below is the summary table of the B-modes in this smaller dataset and the new first occurrence times, which the model estimates are based off of.

Figure 38. Individual B-Mode Time Data Tables for Restart Method Case 2.

Now select the “< GO BACK” button to return to the Restart (Case 2) Output page and view the available plots.


51


• Visual Goodness-of-fit • Estimated expected percent surfaced of the B-mode initial failure intensity • Estimated expected rate of occurrence of new B-modes • Projected expected number of B-modes • Projected MTBF • Statistical goodness-of-fit

Start with the top button and select “Visual Goodness-Of-Fit (Model vs. Observed B-modes) Plot” button.


52

Figure 39. Restart Method Goodness-of-fit graph.

This plot provides a visual perspective of goodness-of-fit by showing how closely the model, as represented by the smooth curve, captures the trend of the observed data. Essentially, the model is fitted to the B-modes occurring beyond the gap, and the curve is interpolated back to the origin. The model appears to represent the data very well, and the curve displays a nice, concave shape. Now select the “Plot of Percent Surfaced Initial B-mode Failure Intensity” button. Once on this tab, you will have to enter a test time input for the percent surfaced. For our example, enter 2000, as this number includes approximately 25 percent additional test time beyond the actual test time of 1606 hours, and then click on “Create Plot”. The input section and the resulting plot follow.




53

Figure 41. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity

This plot shows that if the rate of occurrence of new B-modes were to continue at the same pace as expected through T = 1606 hours, then by the end of 2000 hours, approximately 60 percent of the B-mode initial failure intensity is expected to be surfaced. Now select the “Plot of Rate of Occurrence of New B-modes” button. You will have to enter two inputs for the curve. The first input is an ending value for the rate of occurrence function (the curve), and the second input is a starting value for the function. With regard to the ending point, enter 2000. With regard to the starting point, the rate of occurrence of new B-modes at the origin is finite, so enter 0. Next click on the “Create Plot” button. These two input sections and the rate of occurrence curve follow.

Figure 42. Entering the Ending Value and Starting Value for the B-mode Rate of Occurrence Curve (Partial

Window)


54

Figure 43. Plot of the Expected Rate of Occurrence of New B-modes

Next select the “Plot of Projected Expected Number of B-modes” button. You will have to enter an ending value for the curve. With regard to the ending point, even though the model can project the expected number of B-modes beyond the total test time T, choose a value carefully, as any point beyond the range of the data assumes that the pattern will continue to hold. For our example, enter 2000.


Window)


55

Figure 45. Plot of the Projected Expected Number of B-modes

If we accept the assumption that the rate of occurrence of new B-modes pattern continues to hold beyond the range of the actual data, then the model predicts that an additional 10 B-modes are expected to be surfaced with an additional 394 hours of testing. This type of information may be useful to management for purposes of planning test resources. Next select the “Plot of MTBF Projections” button. You will have to enter two inputs for the curve. The first input is an ending time, and the second input is a starting time. With regard to the ending value, enter 1606. With regard to the starting value, enter 0. Then click on the “Create Plot” button. The two input sections and the resulting plot follow.

Figure 46. Input window for Restart Method.


56

Figure 47. Plot of MTBF Projections for AMPM (Restart Method, v = 250 Hours)

MTBF projections take into account the A-mode failure intensity (zero in our case), the B-mode failure intensity, and fix effectiveness. Recall that fixes may be delayed or incorporated during the test phase and that model estimates are based solely on B-mode first occurrence times. With the Restart Method, the projected MTBF after 1606 hours of time is approximately 28 hours.

2.3.5 Goodness-of-fit. A chi-square goodness-of-fit procedure is used to test the null hypothesis that the AMPM adequately represents the observed test data occurring beyond the gap. The procedure operates by partitioning the observed B-modes into N number of groups such that the number of B-modes per group is at least 3 (this is a relaxation of the recommended number of 5 but should be sufficient for our purposes). Since the chi-square statistic is approximately distributed as a chi-square random variable with N-2 degrees of freedom, the number of groups must be at least 3. Therefore, there must be at least 9 B-modes beyond the gap to run the goodness-of-fit procedure. The null hypothesis is rejected if the chi-square statistic exceeds the critical value for a chosen significance level. The input box and the goodness-of-fit results are displayed below for three groups. Click the “Chi-Square Goodness-Of-Fit Table” button, enter a value of 3 in the cell outlined in red and then click “Do Infinite Chi-Square GOF Case 2.”


57

Figure 48. Restart Method Goodness-of-fit Case 2.

Note that the model cannot be rejected at the 20 percent significance level for one degree of freedom. Further, the model could not be rejected at the 20 percent significance level for 2, 3, 4 and 5 degrees of freedom as the chi-square statistics were 0.19, 1.08, 3.34 and 3.53, respectively. (These values were calculated by entering 4, 5, 6 and 7, successively, for the number of groups.) The model could not be rejected at the 15 percent significance level for 6 degrees of freedom as the chi-square statistic was 8.66. Based on these tests and the visual goodness-of-fit, there is not strong evidence to reject the model for this dataset. Therefore, we accept the applicability of the model and the resulting reliability metrics. The results of the Chi-Square Goodness-of-Fit test using the Restart Method are the same as when we used the Gap Method in the previous example.

2.4 Segmented Fix Effectiveness Factor (FEF) Method.

2.4.1 Rationale for Using the Segmented FEF Method. The segmented FEF method is another strategy that may be useful for a dataset where the initial rate of occurrence of B-modes is very steep due to start-up problems. With this method, a partition point v is chosen such that a relatively high average fix effectiveness factor d1 is applied to the B-modes occurring on or before v, and a more typical average fix effectiveness factor d2 is applied to the B-modes surfaced beyond v. Note that this method is justified only if the early B-modes (those occurring on or before v) are aggressively and effectively corrected. Engineering analysis should be the driving force in choosing the initial segment v. In using this method, the underlying assumption is that the B-modes occurring during the early segment are those whose failure mechanisms are so well understood that they can be assigned very high fix effectiveness factors.

2.4.2 Application of the Segmented FEF Method. Suppose there were start-up problems that gave rise to the steep slope shown in the following example curve.


58

Figure 49. Example Curve for Illustrating the Segmented FEF Method

Moreover, suppose it was determined by engineering analysis that the B-modes occurring on or before v = 250 hours were going to receive focused attention, which means they were going to be aggressively analyzed and corrected. The following discussion shows the series of windows that one would see in applying the segmented FEF method. Begin on the Main Inputs tab by selecting from the first box, the option for “Individual B-mode Time Data.” Then click on the second box, select the “Segmented FEF Method” from the drop down list, and click the “NEXT” button located next to the second box. We will use the same dataset that was used to illustrate the Gap Method. The inputs for that dataset are as follows. The total test time is 1856 hours, the number of observed B-modes is 96, the number of A-mode failures is 0, the number of projections to make is 1, and the partition point v is 250. After entering these inputs, you should be viewing the following window.


59

Figure 50. Input Window for AMPM Prior to Check Input Event

Select the “Check Inputs” button, which will check all of the data inputted and make sure that everything entered is okay. If no problems are found with the inputs a message box will pop up telling you to proceed. Select “OK”.

Since we are using the same dataset as the Gap Method, you don’t have to click on the “B-Mode Data Inputs” button. In the two cells outlined in red you must enter the first fix effectiveness factor d1 and the second fix effectiveness factor d2. For our example, enter 0.95 and 0.7, respectively. Next, click on the “Check FEF” button. The program will check your entered FEFs and make sure they are in the interval of (0, 1) and that they are not a letter or other character. If there are no problems found with your inputted FEFs, a message box will pop up telling you to proceed. Select “OK”.

After entering and checking the average FEF, the Main Input tab will appear as follows:


60

Figure 51. Main Input Window for FEF Method.

You are now ready to click on the “Calculate!” button and view the model results. Again at any time, you can clear all of the inputs by clicking on the “Clear All Inputs” button, or you can change individual inputs. The program does one final check of all of the input data (from both input tabs), and if a problem with the input data is found, a message will appear. In our example, we chose not to enter individual FEFs for each of the 96 observed B-modes since we are using the same dataset as the Gap Method, so the program will ask again if we intend to enter the individual FEFs. Again click on the “No” button and then click on the “OK” button when the program produces the message telling us that the program may proceed. This will take you to the Main Outputs tab where all of the model results (and some of the model inputs) are displayed.



61

Figure 52. Model Results for AMPM using Segmented FEF Method

The first three lines of output display the entries made on the input window: the total amount of test time, the total number of observed B-modes by the end of the test period T, and the total number of observed A-mode failures (NA, includes all repeats). The fourth line from the top is an estimate of the A-mode failure rate, which is the ratio of NA to T and since there were no A-mode failures, the estimate for the A-mode failure intensity is zero. The first line in the second section displays the chosen partition point. The second and third lines in this section are the two average fix effectiveness factors, d1 and d2 that were chosen. The fourth line in the second section provides the estimate for the B-mode initial failure intensity, and since this estimate is independent of any FEF segmentation, its value is the same as if we hadn’t done any segmentation at all. The same holds true for the estimate for the model parameter beta. The next line of output provides, at the partition point v, the rate of occurrence of new B-modes, namely h(v). Recall that the B-mode initial failure intensity is actually h(0) so it is clear that the rate of occurrence of new B-modes is decreasing with time. The last two lines of output provide growth potential estimates. The failure intensity growth potential represents a


62

lower bound on the system failure intensity and the MTBF growth potential represents an upper bound on the system MTBF, both under the assumption that all the B-modes in the system have been surfaced and fixed with effectiveness equal to 0.95 for those B-modes occurring on or before 250 hours and effectiveness equal to 0.7 for those B-modes occurring after 250 hours. Finally, by using the segmented FEF method, the MTBF growth potential has essentially been doubled from approximately 15 hours to approximately 30 hours. Before looking at available reliability plots, select the “Go to Table Outputs” button, which will display a window with four tables of results and one table which is a summary of the B-mode 1st occurrence times entered for this method (not shown below).

Figure 53. Tables of Model Results for AMPM Segmented FEF Method

The values in the tables are indexed by the list of projection times from the “B-Mode Data Input” tab. Recall that we chose to compute estimates for a single projection time at t = 1856 hours. Table 1 provides estimates of the cumulative expected number of B-modes surfaced by the end of t, Table 2 provides estimates of the percent of the B-mode initial failure intensity surfaced by the end of t, Table 3 provides estimates of the expected rate of occurrence of B-modes by the end of t, Table 4 displays the projected failure intensity and projected MTBF by the end of t, and the bottom command button provides a printout of a summary of the data in the first input data file. The use of the segmented FEF method has no impact upon the estimates provided in Tables 1-3 since these estimates are independent of fix effectiveness. The reliability projections provided in Table 4, however, are positively impacted by the segmented FEF method, as the projected MTBF at t = 1856 hours has gone from approximately 13 hours prior to segmentation to approximately 21 hours after the use of segmentation. At this point, select the “Return to Main Outputs” button to return to the Main Outputs tab and then select any of the five plot buttons or the “Chi-Square Goodness-Of-Fit Table” button.


• Visual goodness-of-fit • Estimated expected percent surfaced of the B-mode initial failure intensity • Estimated expected rate of occurrence of new B-modes • Projected expected number of B-modes


63

• Projected MTBF • Statistical goodness-of-fit

Start with the top button and select the “Visual Goodness-Of-Fit (Model vs. Observed B-modes) Plot” button.


64

Figure 54. Visual Goodness-of-Fit with AMPM Segmented FEF Method

This plot provides a visual perspective of the goodness-of-fit of the model. It shows how closely the model, as represented by the smooth curve, captures the overall trend of the observed data points. The use of the segmented FEF method has no bearing upon the shape of this plot since the pattern of first occurrence times for the B-modes is independent of any fix implementation policy. Next select the “Plot of Percent Surfaced Initial B-mode Failure Intensity” button. Once on this tab you need to enter a test time input for the percent surfaced. For our example, enter 2300, as this number includes approximately 25 percent additional test time beyond the actual test time of 1856 hours, and then click on the “Create Plot” button. Again, the use of the segmented FEF method has no impact upon the expected percent surfaced curve since it, too, is independent of fix effectiveness. The input section and the resulting plot follow.




65

Figure 56. Plot of Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity for AMPM

Segmented FEF Method This plot shows that if the rate of occurrence of new B-modes were to continue at the same pace as expected through T = 1856 hours, then by the end of 2300 hours, approximately 93 percent of the B-mode initial failure intensity is expected to be surfaced. Now select the “Plot of Rate of Occurrence of New B-modes” button. You will have to enter two inputs for the curve. The first input is an ending value for the rate of occurrence function (the curve), and the second input is a starting value for the function. With regard to the ending point, enter 2300. With regard to the starting point, the rate of occurrence of new B-modes at the origin is finite, so enter 0. Then click on the “Create Plot” button. These two inputs and the resulting plot follow.


66

Figure 57. Entering the Ending and Starting value for Plotting the Estimated Expected B-mode Rate of

Occurrence (Partial Window)

Figure 58. Plot of the Estimated Expected Rate of Occurrence of New B-modes for AMPM Segmented FEF

Method Again, fix implementation has no impact upon the rate of occurrence of new B-modes. Continue by selecting the “Plot of Projected Expected Number B-modes” button. You need to enter an ending value for the curve, and for our example, enter 2300. Then click on the “Create Plot” button. The input and the resulting plot follow.


Window)


67

Figure 60. Plot of the Projected Expected Number of B-modes for AMPM Segmented FEF Method

If we accept the assumption that the expected rate of occurrence of new B-modes pattern continues to hold beyond the range of the data, then the model predicts that approximately 8 additional B-modes are expected to be surfaced with the additional 444 hours of testing. This type of information may be useful to management for purposes of planning test resources. Again, the projected expected number of B-modes is independent of fix effectiveness. Next select the “Plot of MTBF Projections” button. You will need to enter two input for the curve. The first input is an ending time, and the second input is a starting time. With regard to the ending value, enter 1856. With regard to the starting value, enter 0. Then click on the “Create Plot” button. The two input sections and the resulting plot follow.

Figure 61. Entering the Ending Value and Starting Value for MTBF Projections Plot (Partial Window)


68

Figure 62. Plot of MTBF Projections for AMPM Segmented FEF Method

MTBF projections take into account the A-mode failure intensity (zero in our case), the B-mode failure intensity, and fix effectiveness. Recall that we chose the option where some or all fixes were incorporated during the test phase so that model estimates are based solely on B-mode first occurrence times. Without segmentation, the projected MTBF after 1856 hours was approximately 13 hours. Using the segmented FEF method with v = 250 hours, the projected MTBF after 1856 hours is approximately 21 hours (based on d1 = 0.95 and d2 = 0.7).


69

2.4.5 Goodness-of-Fit. A chi-square goodness-of-fit procedure is available to test the null hypothesis that the AMPM adequately represents the observed test data. The procedure operates by partitioning the observed B-modes into N number of groups such that the number of B-modes per group is at least 3 (this is a relaxation of the recommended number of 5 that was used for the continuous tracking model). Since the chi-square statistic is approximately distributed as a chi-square random variable with N-2 degrees of freedom, the number of groups must be at least 3. The maximum number of groups has been set to eight. The null hypothesis is rejected if the chi-square statistic exceeds the critical value for a chosen significance level. Click on the “Chi-Square Goodness-Of-Fit Table” button. For this example, enter 3 for the number of groups (in the cell outlined in red) and then click the “Do Infinite Chi-Square GOF” button. The input section and the goodness-of-fit results are displayed below for three groups.

Figure 63. Goodness-of-Fit Results (Chi-square Test) for AMPM

Note that the model cannot be rejected at the 15 percent significance level for one degree of freedom. Further, the model could not be rejected at the 20 percent significance level for 2, 4 and 5 degrees of freedom as the chi-square statistics were 3.08, 4.63 and 5.79, respectively. (Successively larger values were entered for the number of groups to obtain these results.) For 6 degrees of freedom, the model could not be rejected at the 15 percent significance level. However, for 3 degrees of freedom (5 groups), the model is rejected at the 5 percent significance level but not at the 1 percent significance level. Despite the lack of statistical fit for 5 groups, there is not strong evidence to reject the model for this dataset. Therefore, we accept the applicability of the model and the resulting reliability metrics. With regard to goodness-of-fit, it is not unusual (when running goodness-of-fit tests for multiple group sizes) to occasionally reject the model for some of the smaller significance levels. It is generally not a serious problem if it occurs once as it did in our example above. However, if the


70

model is rejected consistently for various group sizes, then it indicates that the model is not representative of the data and should not be used to make reliability projections. Finally, the use of the segmented FEF method has no bearing upon model goodness-of-fit since these statistical procedures work directly with B-mode first occurrence times and, as was noted previously with regard to visual goodness-of-fit, the pattern of first occurrence times for the B-modes is independent of any fix implementation policy.

3 Option for Grouped Data. Return to the Main Inputs page, click on the first box, select “Grouped Data” from the drop down list, and then click on the “NEXT” button (located next to the first box) to display the required inputs for the AMPM Grouped Data option.

Figure 64. AMPM Options Selection (partial view)


71

Figure 65. Input Window for AMPM Grouped Data

3.1 Input Phase. Scroll down to the Required Inputs section and enter 3618 for the total test time. Next enter 36 for the number of observed B-Modes and then enter 0 for the number of A-Mode Failures. The next required input is the number of projections to make, and for our example we want to make six projections. Next enter 9 for the number of groups. The last input in the Required Inputs section is a value of t > T (total test time) where the projected MTBF is to be computed. With regard to choosing a value for t, the model can be used to make projections beyond the data if you are willing to accept the assumption that the rate of occurrence pattern that the model predicts will continue to hold in the near future. Even with that assumption, projections should probably be made only slightly beyond the range of the data. For our example, enter 4500, as this number includes approximately 25 percent additional test time beyond the actual test time of 3618 hours. The input window should appear as follows.


72

Figure 66. Input Window for AMPM Grouped Data (Partially Completed)

Now that all the required input data have been entered, select the “Check Inputs” button to make sure that all of the inputs entered are correct. If all of the inputs entered are correct, a message box appears, click “OK” to continue. Next, click on the “B-Mode Data Inputs” button. This will take you to the B-Mode Data Inputs tab (shown below) where you can enter the amount of test time and the number of B-modes in each group. Also to be entered is a list of times where projected quantities are to be computed. For our example, on the “B-Mode Data Inputs” page, enter the data as shown below.

Figure 67. B-Modes Data Input Window for AMPM Grouped Data


73

The user has two options with regard to the fix effectiveness factor used in the calculation of the model estimates.

1. The user can enter a fixed average fix effectiveness factor (useful in doing sensitivity analyses) in the FEF cell located on the Main Inputs tab, or

2. The user can enter a fix effectiveness factor for each individual observed B-mode, and then once on the Main Inputs tab, they can click the “Calculate Average FEF” button and let the program calculate the average FEF of all the individual FEFs entered.

Now select the “Check B-Mode Data Inputs” button to make sure all of the data entered is correct. (Note: If the user decides not to enter FEFs for each of the individual B-modes, then when the “Check B-Mode Data Inputs” button is clicked and the following message appears, just click “No”.)

In our example, we are not going to be entering FEFs for each of the individual B-modes, so click “No” when the above message appears. If the model finds no problems, a message box pops up on screen, click “OK” to continue. Then click the “Return to Main Inputs” button (located on the right side of the screen). This will return the user to the Main Inputs tab where the user can either enter their own average FEF or click the “Calculate Average FEF” button. For our example, enter an average FEF of 0.85 in red outlined cell shown below:


74

Next click on the “Check FEF” button. The program will check your entered FEF and make sure it is in the interval of (0, 1) and that it is not a letter or other character. If there are no problems, you can click “OK” to continue. At this point, even there is still an opportunity to change individual inputs or clear all the inputs by either changing any of the inputs or selecting the “Clear All Inputs” button. Assume for the purposes of this example that no changes are necessary, and select the “Calculate!” button. The program does one final check of all of the input data (from both input tabs), and if a problem with the input data is found, a message will appear. In our example, we chose not to enter individual FEFs for each of the 36 observed B-modes, so the program will ask again if we intend to enter the individual FEFs. Again click on the “No” button and then click on the “OK” button when the program produces the message telling us that the program may proceed. This will take you to the Main Outputs tab where all of the model results (and some of the model inputs) are displayed.


75

3.2 Model Results.

Figure 68. AMPM Grouped Data Model Results

This first section of the Main Outputs page essentially writes back some of the information that was entered on the Main Inputs page, namely, the total amount of test time, the total number of observed B-modes, the total number of A-mode failures and the overall average fix effectiveness factor. The last line displayed in the first section of output is an estimate for the A-mode failure intensity, which is essentially the ratio of the total number of A-mode failures to the total amount of test time.

The second section of the Main Outputs page displays one input and 8 outputs that are specific to the Grouped Data option. The top line is the number of groups entered on the Main Inputs page. The second line of output provides the estimate for the model parameter beta (for the limiting


76

case). One possible interpretation of the estimate for beta is the following: Given a mode occurs at or near the origin (at the beginning of the test) then beta approximates the expected failure rate of that mode. The third line of output provides an estimate for the initial B-mode failure intensity, which can also be interpreted as the rate of occurrence of new B-modes at the origin (t = 0). Compare this value with the next line of output, which provides an estimate for the rate of occurrence of new B-modes at the end of the test, t = T, and note the decrease in the rate of occurrence of new B-modes from the beginning of the test to the end. The fifth line of output from the top displays the projected failure intensity at the end of the test. This estimate takes into account the contribution of the A-mode failures, the B-mode failures and the overall average fix effectiveness. The next line of output provides an estimate for the failure intensity growth potential, which represents a lower bound on the system failure intensity under the assumption that all the B-modes in the system have been surfaced and fixed with effectiveness equal to 0.85. The projected MTBF at T is the reciprocal of the projected failure intensity at T. The next estimate is the MTBF growth potential, which represents an upper bound on the system MTBF under the assumption that all the B-modes in the system have been seen and fixed with effectiveness equal to 0.85. It represents a best case MTBF estimate. The last line of output is an MTBF projection for any value of t > T, provided t is not too much greater than T. The value of t (t = 4500) was entered on the Main Inputs page. The resulting MTBF projection at 4500 hours is approximately 132 hours. Now select the “Go to Table Outputs” button to view the available output tables.

Figure 69. Tables of Model Results for AMPM Grouped Data

The values in the tables are indexed by the list of projection times inputted on the B-Mode Data Inputs page (3618, 3668, 3718, 3768, 3818 and 3868). Table 1 provides estimates of the cumulative expected number of B-modes surfaced, Table 2 provides estimates of the expected percent of the B-mode initial failure intensity surfaced, Table 3 provides estimates of the expected rate of occurrence of B-modes and Table 4 displays the projected failure intensity and


77

projected MTBF. Also found on the Table Outputs page but not shown above is a summary table of the B-mode Data Inputs. The table contains the number of B-modes and the test time for each group, and if you chose to enter an individual FEF for each B-mode, that will be displayed as well. Now select the “Return to Main Outputs” button to return to the Main Outputs page. The figure below shows all of the available plots of reliability projection quantities.

3.3 Reliability Plots. Plots are available for the following:

• Visual goodness-of-fit • Estimated expected percent surfaced of the B-mode initial failure intensity • Estimated expected rate of occurrence of new B-modes • Projected MTBF • Statistical goodness-of-fit

Start with the top button and select the “Visual Goodness-Of-Fit (Model vs. Observed B-modes) Plot” button.


78

Figure 70. Visual Goodness-of-Fit for AMPM Grouped Data Method

This is essentially a visual plot of the goodness-of-fit of the model. It shows how closely the model, as represented by the smooth curve, captures the overall trend of the observed data points. In addition, there is a slight concavity to the curve, which indicates that the rate of occurrence of new B-modes is decreasing. The model would represent a constant or increasing rate of occurrence of new B-modes as a straight line, in which case the model should not be used to make projections with such a dataset. Now select the “Plot of Percent Surfaced Initial B-mode Failure Intensity” button. You will need to enter an ending value for the curve. With regard to the ending point, the model can be used to make projections beyond the range of the data if one is willing to accept the assumption that the rate of occurrence of new B-modes pattern that the model predicts will continue to hold beyond the range of the data. For our example, enter 4500 for the ending point and then click on the “Create Plot” button.

Figure 71. Entering an Ending Value for the Estimated Expected Percent Surfaced Curve for the Grouped

Data Option (Partial Window) The resulting curve indicates that the model predicts that approximately 5 percent more of the B-mode initial failure intensity is expected to be surfaced with an additional 900 hours of test time beyond the already accumulated 3618 hours.


79

Figure 72. Plot of the Estimated Expected Percent Surfaced of the B-mode Initial Failure Intensity

Now select the “Plot of B-mode Rate of Occurrence” button. You will need to enter two inputs for the curve. The first input is an ending value for the curve, and the second input is a starting value for the curve. Enter 4500 for the ending value and 0 for the starting value and then click on the “Create Plot” button. Recall that at the origin for the AMPM, the rate of occurrence of new B-modes is finite, so a starting value equal to zero is permissible. Further, the rate of occurrence of new B-modes at the origin is the B-mode initial failure intensity, so an approximate estimate for the latter can be “read off” from the curve.


80

Figure 73. Plot of the Estimated Expected Rate of Occurrence of New B-modes for the AMPM Grouped

Option For the final plot, select the “Plot of MTBF Projections” button. You will need to enter two inputs for the curve. The first input is an ending value, and the second input is a starting value. Once again, enter 4500 for the ending value and zero for the starting value and then click on the “Create Plot” button. Note that these projections take into account the impact of any A-mode failures, B-mode failures and the effect of corrective actions. The two inputs and the projections follow.

Figure 74. Entering an Ending and Starting Value for Plotting the MTBF Projections


81

Figure 75. Plot of MTBF Projections for AMPM Grouped Data

3.4 Goodness-of-fit. Now select the “Chi-Square Goodness-Of-Fit Table” button.

Figure 76. Goodness-of-Fit Results for AMPM Grouped Data


82

Goodness-of-fit is used to determine model applicability for each dataset. A chi-square goodness-of-fit statistic is used to test the null hypothesis that the AMPM adequately represents a dataset consisting of grouped data (B-mode first occurrence times). The null hypothesis is rejected if the chi-square statistic exceeds the critical value for a chosen significance level. The method is based on obtaining an expected frequency of B-mode first occurrences per group of at least five, subject to the requirement of having at least three groups. Adjacent intervals are combined, if necessary, to meet this expectation (the recombination procedure is performed automatically without user intervention). The dataset for our example originally consisted of nine groups. The goodness-of-fit results indicate that recombination of a few of the groups was necessary to meet the expected frequency condition. The final number of intervals after recombination turned out to be four, and this gave rise to k – 2 = 2 degrees of freedom, where k represents the final number of groups after recombination. Note that the model could not be rejected, even for a significance level equal to 0.20. Therefore, the applicability of the model for our dataset is accepted. Incidentally, if the AMPM does not fit the dataset in question, not only are the projections invalid but the growth potential estimates are also invalid since the estimate for the B-mode initial failure intensity is based on the B-mode first occurrence times.


83

REFERENCES

Broemm, W., Ellner, P., and Woodworth, J., AMSAA Reliability Growth Guide, TR-652, September 2000.

Hall, J., and Ellner, P., Reliability Planning Testbed Coverage of the SSPLAN LCB on MTBF,

TR-740, February 2004

MIL-HDBK-189, Reliability Growth Management, 13 February 1981. Musa, J., Iannino, A., and Okumoto, K., Software Reliability Measurement, Prediction,

Application, McGraw-Hill Book Company, 1987.

amsaa maturity projection model (ampm) user guide€¦ · model (ampm) october 2009 . u.s. army...

Documents