This report compares this build to production build 11821, as tested by product-dev-deploy/11821.
The release notes for the completed tickets should help the target audience understand what this change is about. Are there examples and / or links to the documentation? When you have any doubts talk to the person who worked on the ticket and / or review the commits.
Fixes the dreaded bug that would violate the users expectations.
59a7ff0 TICKET-2931 Finally fix the bug
8ae0846 TICKET-2931 Refactor a little
Fixes the internal server error that was preventing us from returning a response.
The tickets that are in play require extra attention. Could the ongoing work break existing functionality? When in doubt speak to the person who worked on it and review the commits.
48304e8 TICKET-2955: Create that feature
208c46c TICKET-2864 Removing waste
The load test replays production log data. The average response time as well as CPU load over the 20 minutes of the test should be simlar to the current productions version. To accomodate for changes in traffic we run a pattern from the previous as well as one from the current month. If there are major discrepancies review the console ouput of the respective builds or try a different build candidate.
production | candidate ============================ ============================ LOG REPLAY LOAD TEST RESULT LOG REPLAY LOAD TEST RESULT SAMPLE FROM --date 2015-07-01 SAMPLE FROM --date 2015-07-01 ============================ ============================ Total requests: 25280.00 Total requests: 25280.00 Average response time: 197.61 ms | Average response time: 195.97 ms timestamp average timestamp average --------- ------- --------- ------- 2015-07-17T10:05:00.000Z 35.25% | 2015-07-24T14:30:00.000Z 41.67% 2015-07-17T10:06:00.000Z 63.33% | 2015-07-24T14:31:00.000Z 61.31% 2015-07-17T10:07:00.000Z 60.00% | 2015-07-24T14:32:00.000Z 61.69% 2015-07-17T10:08:00.000Z 58.03% | 2015-07-24T14:33:00.000Z 52.46% 2015-07-17T10:09:00.000Z 49.83% | 2015-07-24T14:34:00.000Z 50.51% 2015-07-17T10:10:00.000Z 49.33% | 2015-07-24T14:35:00.000Z 47.21% 2015-07-17T10:11:00.000Z 49.00% | 2015-07-24T14:36:00.000Z 47.80% 2015-07-17T10:12:00.000Z 48.33% | 2015-07-24T14:37:00.000Z 50.00% 2015-07-17T10:13:00.000Z 50.00% | 2015-07-24T14:38:00.000Z 50.33% 2015-07-17T10:14:00.000Z 49.33% | 2015-07-24T14:39:00.000Z 48.00% 2015-07-17T10:15:00.000Z 50.00% | 2015-07-24T14:40:00.000Z 51.33% 2015-07-17T10:16:00.000Z 44.59% | 2015-07-24T14:41:00.000Z 43.00% 2015-07-17T10:17:00.000Z 45.08% | 2015-07-24T14:42:00.000Z 47.67% 2015-07-17T10:18:00.000Z 44.59% | 2015-07-24T14:43:00.000Z 42.00% 2015-07-17T10:19:00.000Z 42.37% | 2015-07-24T14:44:00.000Z 44.00% 2015-07-17T10:20:00.000Z 42.33% | 2015-07-24T14:45:00.000Z 42.67% 2015-07-17T10:21:00.000Z 44.92% | 2015-07-24T14:46:00.000Z 40.98% 2015-07-17T10:22:00.000Z 42.71% | 2015-07-24T14:47:00.000Z 45.76% 2015-07-17T10:23:00.000Z 48.20% | 2015-07-24T14:48:00.000Z 46.00% 2015-07-17T10:24:00.000Z 44.07% <
production | candidate ============================ ============================ LOG REPLAY LOAD TEST RESULT LOG REPLAY LOAD TEST RESULT SAMPLE FROM --date 2015-06-01 SAMPLE FROM --date 2015-06-01 ============================ ============================ Total requests: 20145.00 Total requests: 20145.00 Average response time: 262.98 ms | Average response time: 263.69 ms timestamp average timestamp average --------- ------- --------- ------- 2015-07-17T10:06:00.000Z 40.98% | 2015-07-24T14:30:00.000Z 40.67% 2015-07-17T10:07:00.000Z 55.59% | 2015-07-24T14:31:00.000Z 57.70% 2015-07-17T10:08:00.000Z 52.33% | 2015-07-24T14:32:00.000Z 52.88% 2015-07-17T10:09:00.000Z 44.92% | 2015-07-24T14:33:00.000Z 44.67% 2015-07-17T10:10:00.000Z 46.00% | 2015-07-24T14:34:00.000Z 44.92% 2015-07-17T10:11:00.000Z 45.76% | 2015-07-24T14:35:00.000Z 42.03% 2015-07-17T10:12:00.000Z 44.92% | 2015-07-24T14:36:00.000Z 42.33% 2015-07-17T10:13:00.000Z 43.39% | 2015-07-24T14:37:00.000Z 42.67% 2015-07-17T10:14:00.000Z 48.00% | 2015-07-24T14:38:00.000Z 47.00% 2015-07-17T10:15:00.000Z 47.21% | 2015-07-24T14:39:00.000Z 48.33% 2015-07-17T10:16:00.000Z 45.08% | 2015-07-24T14:40:00.000Z 41.64% 2015-07-17T10:17:00.000Z 43.33% | 2015-07-24T14:41:00.000Z 43.73% 2015-07-17T10:18:00.000Z 40.66% | 2015-07-24T14:42:00.000Z 44.33% 2015-07-17T10:19:00.000Z 47.80% | 2015-07-24T14:43:00.000Z 46.00% 2015-07-17T10:20:00.000Z 40.33% | 2015-07-24T14:44:00.000Z 39.00% 2015-07-17T10:21:00.000Z 40.67% | 2015-07-24T14:45:00.000Z 40.00% 2015-07-17T10:22:00.000Z 40.67% | 2015-07-24T14:46:00.000Z 39.67% 2015-07-17T10:23:00.000Z 41.67% | 2015-07-24T14:47:00.000Z 43.33% 2015-07-17T10:24:00.000Z 40.66% | 2015-07-24T14:48:00.000Z 41.33% 2015-07-17T10:25:00.000Z 40.34% | 2015-07-24T14:49:00.000Z 39.34%
The stress test tries to bring the system to the breaking point by increasing the request rate. The graphs show latency and error rate over request rate. Again the candidate shouldn't be too different from the current production system. Due to virtualisation we expect fluctuactions. The behaviour beyond the rated load is not as important as we ideally scale up before that point is reached.
This step compares the responses that production and candidate system give to randomly sampled user requests. The status codes should be the same, i.e. on the downward diagonal. Deviations need to be investigated, because we generally guarantee backwards compatibility. The json responses are all diffed and the summary is also printed. The changes should be individually reviewed. Ideally they are linked to the things you read it the release notes above.
Running replay of production logs against a: https://product.production.dev.company.com b: https://product.candidate.dev.company.com Number of requests: 2000 Status Code Statistics Exp\Act 200 302 200 1996 0 302 0 4 PASSED Response Differences Number of responses with json nodes that changed: 1 Number of responses with json nodes were added: 0 Number of responses with json nodes were removed: 1Review differences...
Review our public facing front page for layout problems such as misplaced or missing elements.
If you are still happy with the candidate should be suitable for release using the approval button: product-dev-deploy/11848