Public /

Code Magus Limited. Reg. no. 4024745 (England)

Contributors

Hayward (talk) 10:02, 24 January 2022 (UTC) [1]

Summary

The following test(s) were included in this test session.

|LABEL         |Description |Start               |End                 |
|:-------------|:-----------|:-------------------|:-------------------|
|TermAppISONFT |TermAppISO  |2022-01-22 23:12:01 |2022-01-23 00:34:00 |

In the above graph, the items labelled Include and Exclude indicate, respectively, the intervals that should be included or excluded in the resource analysis. The resource analysis applies a linear regression of the resources consumed per server classification onto the total request arrival rate for that interval. Note that the analysis regresses to the arrival rate for the interval, but the determination of the outcome relates to the departures for that interval. The intervals should be long enough so that these numbers are reasonably close to each other.
The following table shows the full set of interval summary records obtained during the test. The Interval timestamp is the start time of the interval over which the metrics are summarised for that interval. A portion of the first metric record collected for each interval may relate to the previous interval. Long intervals or many intervals at the same load make this effect negligible.
The table of summary records provides the value of the load against which resource measurements are correlated. The correlation is determined using the least-squares linear module regressing the interval resource measurement onto the interval test loads.
Response time (Resp) in the table is the mean response time that an INSTANCE (device, customer, client, etc.) takes to complete a full cycle. A cycle is the set of states/actions that an INSTANCE undergoes from awakening from the idle state to the next time the same INSTANCE awakens from the idle state). The time for a cycle includes the idle time, the sum of all the individual response times of the functions/actions performed during the cycle of activity, and the think-times between the functions/actions performed during the cycle.
The Rate column shows the rate at which the INSTANCES exit the idle state, and hence, is the customer arrival rate measure in arrivals per second.
The summary table (or possibly a cut-down version of the summary table) is used to join against the corresponding resource usage tables, and over which the respective resource linear models are determined. The linear models include coefficients of the intercept (which is interpreted as resource usage unrelated to the applied test load) and the slope of the line (which is interpreted as the increment in the resource usage for each unit increment of the test load. This slope coefficient, therefore, gives us a measure of the resource usage for each additional customer arrival per second.

|   |Interval            |   Rate|    Resp|Analysis |
|:--|:-------------------|------:|-------:|:--------|
|1  |2022-01-22 23:14:00 |  0.750|  80.462|Include  |
|2  |2022-01-22 23:15:00 |  1.433|  57.014|Include  |
|3  |2022-01-22 23:16:00 |  1.767|  86.408|Include  |
|4  |2022-01-22 23:17:00 |  2.583|  72.404|Include  |
|5  |2022-01-22 23:18:00 |  2.450|  92.787|Include  |
|6  |2022-01-22 23:19:00 |  3.467|  82.260|Include  |
|7  |2022-01-22 23:20:00 |  3.350|  91.841|Include  |
|8  |2022-01-22 23:21:00 |  4.350|  86.846|Include  |
|9  |2022-01-22 23:22:00 |  4.167| 102.164|Include  |
|10 |2022-01-22 23:23:00 |  5.017|  90.615|Include  |
|11 |2022-01-22 23:24:00 |  5.067| 100.638|Include  |
|12 |2022-01-22 23:25:00 |  6.217|  95.556|Include  |
|13 |2022-01-22 23:26:00 |  5.933|  98.532|Include  |
|14 |2022-01-22 23:27:00 |  6.900|  97.590|Include  |
|15 |2022-01-22 23:28:00 |  6.783|  97.950|Include  |
|16 |2022-01-22 23:29:00 |  7.733|  98.773|Include  |
|17 |2022-01-22 23:30:00 |  7.483|  99.670|Include  |
|18 |2022-01-22 23:31:00 |  8.733|  98.573|Include  |
|19 |2022-01-22 23:32:00 |  8.200| 106.248|Include  |
|20 |2022-01-22 23:33:00 |  9.383| 105.506|Include  |
|21 |2022-01-22 23:34:00 |  9.617| 103.030|Include  |
|22 |2022-01-22 23:35:00 | 10.067| 103.664|Include  |
|23 |2022-01-22 23:36:00 | 10.167| 105.350|Include  |
|24 |2022-01-22 23:37:00 | 10.317| 104.162|Include  |
|25 |2022-01-22 23:38:00 | 11.383| 107.496|Include  |
|26 |2022-01-22 23:39:00 | 11.933| 110.256|Include  |
|27 |2022-01-22 23:40:00 | 12.233| 105.922|Include  |
|28 |2022-01-22 23:41:00 | 12.250| 104.737|Include  |
|29 |2022-01-22 23:42:00 | 13.367| 106.255|Include  |
|30 |2022-01-22 23:43:00 | 13.833|  99.027|Include  |
|31 |2022-01-22 23:44:00 | 13.917| 108.279|Include  |
|32 |2022-01-22 23:45:00 | 14.333| 104.284|Include  |
|33 |2022-01-22 23:46:00 | 14.833| 101.864|Include  |
|34 |2022-01-22 23:47:00 | 15.517| 105.885|Include  |
|35 |2022-01-22 23:48:00 | 15.933| 105.955|Include  |
|36 |2022-01-22 23:49:00 | 15.850| 102.450|Include  |
|37 |2022-01-22 23:50:00 | 16.100| 107.248|Include  |
|38 |2022-01-22 23:51:00 | 17.167| 106.939|Include  |
|39 |2022-01-22 23:52:00 | 17.250| 104.244|Include  |
|40 |2022-01-22 23:53:00 | 17.467| 105.120|Include  |
|41 |2022-01-22 23:54:00 | 17.583| 113.903|Include  |
|42 |2022-01-22 23:55:00 | 18.650| 109.172|Include  |
|43 |2022-01-22 23:56:00 | 18.717| 111.223|Include  |
|44 |2022-01-22 23:57:00 | 19.667| 105.346|Include  |
|45 |2022-01-22 23:58:00 | 20.483| 106.451|Include  |
|46 |2022-01-22 23:59:00 | 20.817| 106.539|Include  |
|47 |2022-01-23 00:00:00 | 21.067| 104.103|Include  |
|48 |2022-01-23 00:01:00 | 21.817| 101.319|Include  |
|49 |2022-01-23 00:02:00 | 21.233| 107.946|Include  |
|50 |2022-01-23 00:03:00 | 21.650| 104.254|Include  |
|51 |2022-01-23 00:04:00 | 22.333| 106.270|Include  |
|52 |2022-01-23 00:05:00 | 23.667| 110.511|Include  |
|53 |2022-01-23 00:06:00 | 23.950| 106.210|Include  |
|54 |2022-01-23 00:07:00 | 24.267| 104.861|Include  |
|55 |2022-01-23 00:08:00 | 23.900| 109.415|Include  |
|56 |2022-01-23 00:09:00 | 24.750| 105.875|Include  |
|57 |2022-01-23 00:10:00 | 25.617| 108.284|Include  |
|58 |2022-01-23 00:11:00 | 25.967| 104.487|Include  |
|59 |2022-01-23 00:12:00 | 26.367| 109.764|Include  |
|60 |2022-01-23 00:13:00 | 27.050| 107.046|Include  |
|61 |2022-01-23 00:14:00 | 26.233| 100.903|Include  |
|62 |2022-01-23 00:15:00 | 26.917| 109.822|Include  |
|63 |2022-01-23 00:16:00 | 28.650| 112.363|Include  |
|64 |2022-01-23 00:17:00 | 27.317| 103.297|Include  |
|65 |2022-01-23 00:18:00 | 28.917| 107.903|Include  |
|66 |2022-01-23 00:19:00 | 28.517| 104.127|Include  |
|67 |2022-01-23 00:20:00 | 29.867| 108.580|Include  |
|68 |2022-01-23 00:21:00 | 29.300| 107.553|Include  |
|69 |2022-01-23 00:22:00 | 30.383| 109.827|Include  |
|70 |2022-01-23 00:23:00 | 30.817| 106.727|Include  |
|71 |2022-01-23 00:24:00 | 31.033| 107.809|Include  |
|72 |2022-01-23 00:25:00 | 32.400| 107.382|Include  |
|73 |2022-01-23 00:26:00 | 32.817| 107.146|Include  |
|74 |2022-01-23 00:27:00 | 33.150| 108.135|Include  |
|75 |2022-01-23 00:28:00 | 33.000| 105.478|Include  |
|76 |2022-01-23 00:29:00 | 33.350| 106.077|Include  |
|77 |2022-01-23 00:30:00 | 33.650| 107.822|Include  |
|78 |2022-01-23 00:31:00 | 34.783| 107.883|Include  |
|79 |2022-01-23 00:32:00 | 34.567| 108.721|Include  |
|80 |2022-01-23 00:33:00 | 37.067| 106.754|Include  |
|81 |2022-01-23 00:34:00 | 34.833| 106.155|Include  |

The following tables show estimates and projections of CPU usage for each class (by hosted function) of server where some confidence in these numbers is reasonable. A coefficient is determined reasonable by the significance of the respective class coefficient's p-value.
Where there is some significant response of the resource utilisation to the applied load, and where there is some confidence in this correlation, an estimate of the resource usage can be determined for capacity planning and/or comparison purposes. Here, the column Est Use @ 30.00/s is the expected CPU usage at 30.00 requests per second. The column Estimate should be used as a factor to multiply by the transaction rate in order to estimate the resource usage required to support that transaction rate. The column Std. Error should be used to determine the confidence interval around the estimate:

|   |Class   |Test       |Measure   |Units                    |Coef | Estimate| Std. Error|  t value| p.value| Est Use @ 30.00/s|
|:--|:-------|:----------|:---------|:------------------------|:----|--------:|----------:|--------:|-------:|-----------------:|
|1  |LOAD    |TermAppISO |CPU Usage |Percent single processor |Rate |   0.9554|     0.0086| 110.9485|   0e+00|           28.6610|
|4  |MODEL   |TermAppISO |CPU Usage |Percent single processor |Rate |   0.5405|     0.0483|  11.1913|   0e+00|           16.2142|
|7  |NETWORK |TermAppISO |CPU Usage |Percent single processor |Rate |   1.0345|     0.2558|   4.0451|   1e-04|           31.0360|

In some cases, it is possible to estimate with confidence the background workload on the various servers and server groups, the resource usage which is independent of the applied test load. The column Estimate indicates this. The column Std. Error should be used to determine a confidence interval:

|   |Class   |Test       |Measure   |Units                    |Coef | Estimate| Std. Error| t value| p.value|
|:--|:-------|:----------|:---------|:------------------------|:----|--------:|----------:|-------:|-------:|
|1  |LOAD    |TermAppISO |CPU Usage |Percent single processor |Bias |   1.4977|     0.1803|  8.3051|       0|
|4  |MODEL   |TermAppISO |CPU Usage |Percent single processor |Bias |   8.7513|     1.0114|  8.6527|       0|
|7  |NETWORK |TermAppISO |CPU Usage |Percent single processor |Bias |  96.0863|     5.3561| 17.9396|       0|

The following tables show estimates and projections of Network usage for each class (by hosted function) of server where some confidence in these numbers is reasonable. A coefficient is determined reasonable by the significance of the respective class coefficient's p-value.
Where there is some significant response of the resource utilisation to the applied load, and where there is some confidence in this correlation, an estimate of the resource usage can be determined for capacity planning and/or comparison purposes. Here, the column Est Use @ 30.00/s is the expected Network usage at 30.00 requests per second. The column Estimate should be used as a factor to multiply by the transaction rate in order to estimate the resource usage required to support that transaction rate. The column Std. Error should be used to determine the confidence interval around the estimate:

|   |Class   |Test       |Measure            |Units           |Coef |   Estimate| Std. Error|  t value| p.value| Est Use @ 30.00/s|
|:--|:-------|:----------|:------------------|:---------------|:----|----------:|----------:|--------:|-------:|-----------------:|
|2  |LOAD    |TermAppISO |Network Usage recv |bits per second |Rate |  23417.502|   151.6781| 154.3895|       0|          702525.0|
|3  |LOAD    |TermAppISO |Network Usage sent |bits per second |Rate |  28163.931|   212.2305| 132.7044|       0|          844917.9|
|5  |MODEL   |TermAppISO |Network Usage recv |bits per second |Rate |  12517.495|   113.1614| 110.6163|       0|          375524.8|
|6  |MODEL   |TermAppISO |Network Usage sent |bits per second |Rate |  13038.938|   285.7243|  45.6347|       0|          391168.1|
|8  |NETWORK |TermAppISO |Network Usage recv |bits per second |Rate |   7152.579|   633.5699|  11.2893|       0|          214577.4|
|9  |NETWORK |TermAppISO |Network Usage sent |bits per second |Rate | 172032.924| 13421.4084|  12.8178|       0|         5160987.7|

In some cases, it is possible to estimate with confidence the background workload on the various servers and server groups, the resource usage which is independent of the applied test load. The column Estimate indicates this. The column Std. Error should be used to determine a confidence interval:

|   |Class   |Test       |Measure            |Units           |Coef |  Estimate| Std. Error| t value| p.value|
|:--|:-------|:----------|:------------------|:---------------|:----|---------:|----------:|-------:|-------:|
|2  |LOAD    |TermAppISO |Network Usage recv |bits per second |Bias | 147656.33|   3176.535| 46.4835|       0|
|3  |LOAD    |TermAppISO |Network Usage sent |bits per second |Bias | 396176.83|   4444.663| 89.1354|       0|
|5  |MODEL   |TermAppISO |Network Usage recv |bits per second |Bias |  78885.79|   2369.895| 33.2866|       0|
|6  |MODEL   |TermAppISO |Network Usage sent |bits per second |Bias | 151017.36|   5983.815| 25.2376|       0|
|8  |NETWORK |TermAppISO |Network Usage recv |bits per second |Bias | 630081.82|  13268.610| 47.4866|       0|

Change Control

Feedback

Record

Notes from the test

Landscape

The analysis is performed for each test within the test session and for each unique Classification of a server in the collected Server landscape/Classification list. For each Interval the resources are summed across all the servers in that class. This means that the resource usage coefficients and projections are for the total resource usage across the Classification.

|   |Classification |Server    |IP_Address     |OSType |Description   |Hardware_Support |Server_Support  |Application_Support |SOURCE_FILE                            |
|:--|:--------------|:---------|:--------------|:------|:-------------|:----------------|:---------------|:-------------------|:--------------------------------------|
|1  |LOAD           |LOAD0     |176.67.166.86  |Linux  |CML EcoSystem |Patrick Hayward  |Patrick Hayward |Patrick Hayward     |CMLEcoSystem_Server_Classification.csv |
|2  |LOAD           |LOAD1     |176.67.166.89  |Linux  |CML EcoSystem |Patrick Hayward  |Patrick Hayward |Patrick Hayward     |CMLEcoSystem_Server_Classification.csv |
|3  |MODEL          |MODELAPP0 |176.67.166.20  |Linux  |CML EcoSystem |Patrick Hayward  |Patrick Hayward |Patrick Hayward     |CMLEcoSystem_Server_Classification.csv |
|4  |MODEL          |MODELAPP1 |176.67.166.72  |Linux  |CML EcoSystem |Patrick Hayward  |Patrick Hayward |Patrick Hayward     |CMLEcoSystem_Server_Classification.csv |
|5  |NETWORK        |NETWORK   |109.123.111.17 |Linux  |CML EcoSystem |Patrick Hayward  |Patrick Hayward |Patrick Hayward     |CMLEcoSystem_Server_Classification.csv |

Analysis

Test: TermAppISO

This test started at 2022-01-22 23:12:01 and ended at 2022-01-23 00:34:00.

Resource usage for platform Linux

Server class LOAD CPU usage

CPU Usage - cpu.cpu.system.user

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.77375 -0.61468 -0.03656  0.50954  2.17412 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 1.497703   0.180335   8.305 2.22e-12 ***
Rate        0.955367   0.008611 110.948  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.789 on 79 degrees of freedom
Multiple R-squared:  0.9936,	Adjusted R-squared:  0.9935 
F-statistic: 1.231e+04 on 1 and 79 DF,  p-value: < 2.2e-16

Server class LOAD Network usage

Network Usage recv - recv

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
   Min     1Q Median     3Q    Max 
-36782  -8465    833   9729  29301 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 147656.3     3176.5   46.48   <2e-16 ***
Rate         23417.5      151.7  154.39   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 13900 on 79 degrees of freedom
Multiple R-squared:  0.9967,	Adjusted R-squared:  0.9967 
F-statistic: 2.384e+04 on 1 and 79 DF,  p-value: < 2.2e-16

Network Usage sent - sent

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
   Min     1Q Median     3Q    Max 
-51931 -10553   1278  13469  38324 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 396176.8     4444.7   89.14   <2e-16 ***
Rate         28163.9      212.2  132.70   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 19450 on 79 degrees of freedom
Multiple R-squared:  0.9955,	Adjusted R-squared:  0.9955 
F-statistic: 1.761e+04 on 1 and 79 DF,  p-value: < 2.2e-16

Server class MODEL CPU usage

CPU Usage - cpu.cpu.system.user

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
   Min     1Q Median     3Q    Max 
-6.185 -3.926 -1.839  4.553  9.593 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  8.75133    1.01140   8.653 4.64e-13 ***
Rate         0.54047    0.04829  11.191  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 4.425 on 79 degrees of freedom
Multiple R-squared:  0.6132,	Adjusted R-squared:  0.6083 
F-statistic: 125.2 on 1 and 79 DF,  p-value: < 2.2e-16

Server class MODEL Network usage

Network Usage recv - recv

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
     Min       1Q   Median       3Q      Max 
-29176.7  -7382.0     37.8   6655.4  20586.0 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  78885.8     2369.9   33.29   <2e-16 ***
Rate         12517.5      113.2  110.62   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 10370 on 79 degrees of freedom
Multiple R-squared:  0.9936,	Adjusted R-squared:  0.9935 
F-statistic: 1.224e+04 on 1 and 79 DF,  p-value: < 2.2e-16

Network Usage sent - sent

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
   Min     1Q Median     3Q    Max 
-63388 -19880  -3327  22576  53289 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 151017.4     5983.8   25.24   <2e-16 ***
Rate         13038.9      285.7   45.63   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 26180 on 79 degrees of freedom
Multiple R-squared:  0.9635,	Adjusted R-squared:  0.963 
F-statistic:  2083 on 1 and 79 DF,  p-value: < 2.2e-16

Server class NETWORK CPU usage

CPU Usage - cpu.cpu.system.user

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
    Min      1Q  Median      3Q     Max 
-18.323 -12.206  -7.605   0.092  94.063 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  96.0863     5.3561  17.940  < 2e-16 ***
Rate          1.0345     0.2558   4.045 0.000121 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 23.43 on 79 degrees of freedom
Multiple R-squared:  0.1716,	Adjusted R-squared:  0.1611 
F-statistic: 16.36 on 1 and 79 DF,  p-value: 0.0001209

Server class NETWORK Network usage

Network Usage recv - recv

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
    Min      1Q  Median      3Q     Max 
-138325  -35969   -3471   28540  133257 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 630081.8    13268.6   47.49   <2e-16 ***
Rate          7152.6      633.6   11.29   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 58050 on 79 degrees of freedom
Multiple R-squared:  0.6173,	Adjusted R-squared:  0.6125 
F-statistic: 127.4 on 1 and 79 DF,  p-value: < 2.2e-16

Network Usage sent - sent

Call:
lm(formula = Value.sum ~ Rate, data = summ.class.include)

Residuals:
     Min       1Q   Median       3Q      Max 
-2352417 -1029643    24162   932847  2844392 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -1518407     281079  -5.402 6.76e-07 ***
Rate          172033      13421  12.818  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1230000 on 79 degrees of freedom
Multiple R-squared:  0.6753,	Adjusted R-squared:  0.6712 
F-statistic: 164.3 on 1 and 79 DF,  p-value: < 2.2e-16

Performance

For test TermAppISO/TermAppISONFT (Test 1, from 2022-01-22 23:12:01 to 2022-01-23 00:34:00), summary of response times by operation/function-outcome over the range of loads where the arrival rate did not exceed the observed or required peak target rate of 15.00 arrivals per second (this rate is as opposed to the projected/future peak rate). This cut-off time occurred at 2022-01-22 23:46:00. The descriptive statistics mean, median and sd are calculated over the entire time-frame that the arrival rate did not exceed the target arrival rate, which includes any outliers caused by possible external influences (which are removed for the regression analysis).

Distribution of response times where rate does not exceed the target cutoff target rate for outcome AUTHORISATION_RESPONSE_1110_OK:

|   |Basename                   |Outcome                        | Count| Percent|  Resp| StdDev|
|:--|:--------------------------|:------------------------------|-----:|-------:|-----:|------:|
|1  |authorisation_request_1100 |AUTHORISATION_RESPONSE_1110_OK | 15599|     100| 0.208|  0.012|

Distribution of response times where rate does not exceed the target cutoff target rate for outcome TRANSACTION_ADVICE_RESPONSE_1230_OK:

|   |Basename                         |Outcome                             | Count| Percent|  Resp| StdDev|
|:--|:--------------------------------|:-----------------------------------|-----:|-------:|-----:|------:|
|2  |transaction_advice_response_1230 |TRANSACTION_ADVICE_RESPONSE_1230_OK | 15535|     100| 0.106|  0.012|