root / doc / design-multi-version-tests.rst @ 9110fb4a
History | View | Annotate | Download (7.1 kB)
1 | a8b1e9f8 | Hrvoje Ribicic | =================== |
---|---|---|---|
2 | a8b1e9f8 | Hrvoje Ribicic | Multi-version tests |
3 | a8b1e9f8 | Hrvoje Ribicic | =================== |
4 | a8b1e9f8 | Hrvoje Ribicic | |
5 | a8b1e9f8 | Hrvoje Ribicic | .. contents:: :depth: 4 |
6 | a8b1e9f8 | Hrvoje Ribicic | |
7 | a8b1e9f8 | Hrvoje Ribicic | This is a design document describing how tests which use multiple |
8 | a8b1e9f8 | Hrvoje Ribicic | versions of Ganeti can be introduced into the current build |
9 | a8b1e9f8 | Hrvoje Ribicic | infrastructure. |
10 | a8b1e9f8 | Hrvoje Ribicic | |
11 | a8b1e9f8 | Hrvoje Ribicic | Desired improvements |
12 | a8b1e9f8 | Hrvoje Ribicic | ==================== |
13 | a8b1e9f8 | Hrvoje Ribicic | |
14 | a8b1e9f8 | Hrvoje Ribicic | The testing of Ganeti is currently done by using two different |
15 | a8b1e9f8 | Hrvoje Ribicic | approaches - unit tests and QA. While the former are useful for ensuring |
16 | a8b1e9f8 | Hrvoje Ribicic | that the individual parts of the system work as expected, most errors |
17 | a8b1e9f8 | Hrvoje Ribicic | are discovered only when all the components of Ganeti interact during |
18 | a8b1e9f8 | Hrvoje Ribicic | QA. |
19 | a8b1e9f8 | Hrvoje Ribicic | |
20 | a8b1e9f8 | Hrvoje Ribicic | However useful otherwise, until now the QA has failed to provide support |
21 | a8b1e9f8 | Hrvoje Ribicic | for testing upgrades and version compatibility as it was limited to |
22 | a8b1e9f8 | Hrvoje Ribicic | using only one version of Ganeti. While these can be tested for every |
23 | a8b1e9f8 | Hrvoje Ribicic | release manually, a systematic approach is preferred and none can exist |
24 | a8b1e9f8 | Hrvoje Ribicic | with this restriction in place. To lift it, the buildbot scripts and QA |
25 | a8b1e9f8 | Hrvoje Ribicic | utilities must be extended to allow a way of specifying and using |
26 | a8b1e9f8 | Hrvoje Ribicic | diverse multi-version checks. |
27 | a8b1e9f8 | Hrvoje Ribicic | |
28 | a8b1e9f8 | Hrvoje Ribicic | Required use cases |
29 | a8b1e9f8 | Hrvoje Ribicic | ================== |
30 | a8b1e9f8 | Hrvoje Ribicic | |
31 | a8b1e9f8 | Hrvoje Ribicic | There are two classes of multi-version tests that are interesting in |
32 | a8b1e9f8 | Hrvoje Ribicic | Ganeti, and this chapter provides an example from each to highlight what |
33 | a8b1e9f8 | Hrvoje Ribicic | should be accounted for in the design. |
34 | a8b1e9f8 | Hrvoje Ribicic | |
35 | a8b1e9f8 | Hrvoje Ribicic | Compatibility tests |
36 | a8b1e9f8 | Hrvoje Ribicic | ------------------- |
37 | a8b1e9f8 | Hrvoje Ribicic | |
38 | a8b1e9f8 | Hrvoje Ribicic | One interface Ganeti exposes to clients interested in interacting with |
39 | a8b1e9f8 | Hrvoje Ribicic | it is the RAPI. Its stability has always been a design principle |
40 | a8b1e9f8 | Hrvoje Ribicic | followed during implementation, but whether it held true in practice was |
41 | a8b1e9f8 | Hrvoje Ribicic | not asserted through tests. |
42 | a8b1e9f8 | Hrvoje Ribicic | |
43 | a8b1e9f8 | Hrvoje Ribicic | An automatic test of RAPI compatibility would have to take a diverse set |
44 | a8b1e9f8 | Hrvoje Ribicic | of RAPI requests and perform them on two clusters of different versions, |
45 | a8b1e9f8 | Hrvoje Ribicic | one of which would be the reference version. If the clusters had been |
46 | a8b1e9f8 | Hrvoje Ribicic | identically configured, all of the commands successfully executed on the |
47 | a8b1e9f8 | Hrvoje Ribicic | reference version should succeed on the newer version as well. |
48 | a8b1e9f8 | Hrvoje Ribicic | |
49 | a8b1e9f8 | Hrvoje Ribicic | To achieve this, two versions of Ganeti can be run separately on a |
50 | a8b1e9f8 | Hrvoje Ribicic | cleanly setup cluster. With no guarantee that the versions can coexist, |
51 | a8b1e9f8 | Hrvoje Ribicic | the deployment of these has to be separate. A proxy placed between the |
52 | a8b1e9f8 | Hrvoje Ribicic | client and Ganeti records all the requests and responses. Using this |
53 | a8b1e9f8 | Hrvoje Ribicic | data, a testing utility can decide if the newer version is compatible or |
54 | a8b1e9f8 | Hrvoje Ribicic | not, and provide additional information to assist with debugging. |
55 | a8b1e9f8 | Hrvoje Ribicic | |
56 | a8b1e9f8 | Hrvoje Ribicic | Upgrade / downgrade tests |
57 | a8b1e9f8 | Hrvoje Ribicic | ------------------------- |
58 | a8b1e9f8 | Hrvoje Ribicic | |
59 | a8b1e9f8 | Hrvoje Ribicic | An upgrade / downgrade test serves to examine whether the state of the |
60 | a8b1e9f8 | Hrvoje Ribicic | cluster is unchanged after its configuration has been upgraded or |
61 | a8b1e9f8 | Hrvoje Ribicic | downgraded to another version of Ganeti. |
62 | a8b1e9f8 | Hrvoje Ribicic | |
63 | a8b1e9f8 | Hrvoje Ribicic | The test works with two consecutive versions of Ganeti, both installed |
64 | a8b1e9f8 | Hrvoje Ribicic | on the same machine. It examines whether the configuration data and |
65 | a8b1e9f8 | Hrvoje Ribicic | instances survive the downgrade and upgrade procedures. This is done by |
66 | a8b1e9f8 | Hrvoje Ribicic | creating a cluster with the newer version, downgrading it to the older |
67 | a8b1e9f8 | Hrvoje Ribicic | one, and upgrading it to the newer one again. After every step, the |
68 | a8b1e9f8 | Hrvoje Ribicic | integrity of the cluster is checked by running various operations and |
69 | a8b1e9f8 | Hrvoje Ribicic | ensuring everything still works. |
70 | a8b1e9f8 | Hrvoje Ribicic | |
71 | a8b1e9f8 | Hrvoje Ribicic | Design and implementation |
72 | a8b1e9f8 | Hrvoje Ribicic | ========================= |
73 | a8b1e9f8 | Hrvoje Ribicic | |
74 | a8b1e9f8 | Hrvoje Ribicic | Although the previous examples have not been selected to show use cases |
75 | a8b1e9f8 | Hrvoje Ribicic | as diverse as possible, they still show a number of dissimilarities: |
76 | a8b1e9f8 | Hrvoje Ribicic | |
77 | a8b1e9f8 | Hrvoje Ribicic | - Parallel installation vs sequential deployments |
78 | a8b1e9f8 | Hrvoje Ribicic | - Comparing with reference version vs comparing consecutive versions |
79 | a8b1e9f8 | Hrvoje Ribicic | - Examining result dumps vs trying a sequence of operations |
80 | a8b1e9f8 | Hrvoje Ribicic | |
81 | a8b1e9f8 | Hrvoje Ribicic | With the first two real use cases demonstrating such diversity, it does |
82 | a8b1e9f8 | Hrvoje Ribicic | not make sense to design multi-version test classes. Instead, the |
83 | a8b1e9f8 | Hrvoje Ribicic | programmability of buildbot's configuration files can be leveraged to |
84 | a8b1e9f8 | Hrvoje Ribicic | implement each test as a separate builder with a custom sequence of |
85 | a8b1e9f8 | Hrvoje Ribicic | steps. The individual steps such as checking out a given or previous |
86 | a8b1e9f8 | Hrvoje Ribicic | version, or installing and removing Ganeti, will be provided as utility |
87 | a8b1e9f8 | Hrvoje Ribicic | functions for any test writer to use. |
88 | a8b1e9f8 | Hrvoje Ribicic | |
89 | a8b1e9f8 | Hrvoje Ribicic | Current state |
90 | a8b1e9f8 | Hrvoje Ribicic | ------------- |
91 | a8b1e9f8 | Hrvoje Ribicic | |
92 | a8b1e9f8 | Hrvoje Ribicic | An upgrade / downgrade test is a part of the QA suite as of commit |
93 | a8b1e9f8 | Hrvoje Ribicic | aa104b5e. The test and the corresponding buildbot changes are a very |
94 | a8b1e9f8 | Hrvoje Ribicic | good first step, both by showing that multi-version tests can be done, |
95 | a8b1e9f8 | Hrvoje Ribicic | and by providing utilities needed for builds of multiple branches. |
96 | a8b1e9f8 | Hrvoje Ribicic | Previously, the same folder was used as the base directory of any build, |
97 | a8b1e9f8 | Hrvoje Ribicic | and now a directory structure more accommodating to multiple builds is |
98 | a8b1e9f8 | Hrvoje Ribicic | in place. |
99 | a8b1e9f8 | Hrvoje Ribicic | |
100 | a8b1e9f8 | Hrvoje Ribicic | The builder running the test has one flaw - regardless of the branch |
101 | a8b1e9f8 | Hrvoje Ribicic | submitted, it compares versions 2.10 and 2.11 (current master). This |
102 | a8b1e9f8 | Hrvoje Ribicic | behaviour is different from any of the other builders, which may |
103 | a8b1e9f8 | Hrvoje Ribicic | restrict the branches a test can be performed on, but do not |
104 | a8b1e9f8 | Hrvoje Ribicic | differentiate between them otherwise. While additional builders for |
105 | a8b1e9f8 | Hrvoje Ribicic | different versions pairs may be added, this is not a good long-term |
106 | a8b1e9f8 | Hrvoje Ribicic | solution. |
107 | a8b1e9f8 | Hrvoje Ribicic | |
108 | a8b1e9f8 | Hrvoje Ribicic | The test can be improved by making it compare the current and the |
109 | a8b1e9f8 | Hrvoje Ribicic | previous version. As the buildbot has no notion of what a previous |
110 | a8b1e9f8 | Hrvoje Ribicic | version is, additional utilities to handle this logic will have to be |
111 | a8b1e9f8 | Hrvoje Ribicic | introduced. |
112 | a8b1e9f8 | Hrvoje Ribicic | |
113 | a8b1e9f8 | Hrvoje Ribicic | Planned changes |
114 | a8b1e9f8 | Hrvoje Ribicic | --------------- |
115 | a8b1e9f8 | Hrvoje Ribicic | |
116 | a8b1e9f8 | Hrvoje Ribicic | The upgrade / downgrade test should be generalized to work for any |
117 | a8b1e9f8 | Hrvoje Ribicic | version which can be downgraded from and upgraded to automatically, |
118 | a8b1e9f8 | Hrvoje Ribicic | meaning versions from 2.11 onwards. This will be made challenging by |
119 | a8b1e9f8 | Hrvoje Ribicic | the fact that the previous version has to be checked out by reading the |
120 | a8b1e9f8 | Hrvoje Ribicic | version of the currently checked out code, identifying the previous |
121 | a8b1e9f8 | Hrvoje Ribicic | version, and then making yet another checkout. |
122 | a8b1e9f8 | Hrvoje Ribicic | |
123 | a8b1e9f8 | Hrvoje Ribicic | The major and minor version can be read from a Ganeti repository in |
124 | a8b1e9f8 | Hrvoje Ribicic | multiple ways. The two are present as constants defined in source files, |
125 | a8b1e9f8 | Hrvoje Ribicic | but due to refactorings shifting constants from the Python to the |
126 | a8b1e9f8 | Hrvoje Ribicic | Haskell side, their position varies across versions. A more reliable way |
127 | a8b1e9f8 | Hrvoje Ribicic | of fetching them is by examining the news file, as it obeys strict |
128 | a8b1e9f8 | Hrvoje Ribicic | formatting restrictions. |
129 | a8b1e9f8 | Hrvoje Ribicic | |
130 | a8b1e9f8 | Hrvoje Ribicic | With the version found, a script that acts as a previous version |
131 | a8b1e9f8 | Hrvoje Ribicic | lookup table can be invoked. This script can be constructed dynamically |
132 | a8b1e9f8 | Hrvoje Ribicic | upon buildbot startup, and specified as a build step. The checkout |
133 | a8b1e9f8 | Hrvoje Ribicic | following it proceeds as expected. |
134 | a8b1e9f8 | Hrvoje Ribicic | |
135 | a8b1e9f8 | Hrvoje Ribicic | The RAPI compatibility test should be added as a separate builder |
136 | a8b1e9f8 | Hrvoje Ribicic | afterwards. As the test requires additional comparison and proxy logic |
137 | a8b1e9f8 | Hrvoje Ribicic | to be used, it will be enabled only on 2.11 onwards, comparing the |
138 | a8b1e9f8 | Hrvoje Ribicic | versions to 2.6 - the reference version for the RAPI. Details on the |
139 | a8b1e9f8 | Hrvoje Ribicic | design of this test will be added in a separate document. |
140 | a8b1e9f8 | Hrvoje Ribicic | |
141 | a8b1e9f8 | Hrvoje Ribicic | Potential issues |
142 | a8b1e9f8 | Hrvoje Ribicic | ================ |
143 | a8b1e9f8 | Hrvoje Ribicic | |
144 | a8b1e9f8 | Hrvoje Ribicic | While there are many advantages to having a single builder representing |
145 | a8b1e9f8 | Hrvoje Ribicic | a multi-version test, working on every branch, there is at least one |
146 | a8b1e9f8 | Hrvoje Ribicic | disadvantage: the need to define a base or reference version, which is |
147 | a8b1e9f8 | Hrvoje Ribicic | the only version that can be used to trigger the test, and the only one |
148 | a8b1e9f8 | Hrvoje Ribicic | on which code changes can be tried. |
149 | a8b1e9f8 | Hrvoje Ribicic | |
150 | a8b1e9f8 | Hrvoje Ribicic | If an error is detected while running a test, and the issue lies with |
151 | a8b1e9f8 | Hrvoje Ribicic | a version other than the one used to invoke the test, the fix would |
152 | a8b1e9f8 | Hrvoje Ribicic | have to make it into the repository before the test could be tried |
153 | a8b1e9f8 | Hrvoje Ribicic | again. |
154 | a8b1e9f8 | Hrvoje Ribicic | |
155 | a8b1e9f8 | Hrvoje Ribicic | For simple tests, the issue might be mitigated by running them locally. |
156 | a8b1e9f8 | Hrvoje Ribicic | However, the multi-version tests are more likely to be complicated than |
157 | a8b1e9f8 | Hrvoje Ribicic | not, and it could be difficult to reproduce a test by hand. |
158 | a8b1e9f8 | Hrvoje Ribicic | |
159 | a8b1e9f8 | Hrvoje Ribicic | The situation can be made simpler by requiring that any multi-version |
160 | a8b1e9f8 | Hrvoje Ribicic | test can use only versions lower than the reference version. As errors |
161 | a8b1e9f8 | Hrvoje Ribicic | are more likely to be found in new rather than old code, this would at |
162 | a8b1e9f8 | Hrvoje Ribicic | least reduce the number of troublesome cases. |