root / doc / design-cmdlib-unittests.rst @ 5cbf7832
History | View | Annotate | Download (6.6 kB)
1 | 27eb699d | Thomas Thrainer | ===================================== |
---|---|---|---|
2 | 27eb699d | Thomas Thrainer | Unit tests for cmdlib / LogicalUnit's |
3 | 27eb699d | Thomas Thrainer | ===================================== |
4 | 27eb699d | Thomas Thrainer | |
5 | 27eb699d | Thomas Thrainer | .. contents:: :depth: 4 |
6 | 27eb699d | Thomas Thrainer | |
7 | 27eb699d | Thomas Thrainer | This is a design document describing unit tests for the cmdlib module. |
8 | 27eb699d | Thomas Thrainer | Other modules are deliberately omitted, as LU's contain the most complex |
9 | 27eb699d | Thomas Thrainer | logic and are only sparingly tested. |
10 | 27eb699d | Thomas Thrainer | |
11 | 27eb699d | Thomas Thrainer | Current state and shortcomings |
12 | 27eb699d | Thomas Thrainer | ============================== |
13 | 27eb699d | Thomas Thrainer | |
14 | 27eb699d | Thomas Thrainer | The current test coverage of the cmdlib module is at only ~14%. Given |
15 | 27eb699d | Thomas Thrainer | the complexity of the code this is clearly too little. |
16 | 27eb699d | Thomas Thrainer | |
17 | 27eb699d | Thomas Thrainer | The reasons for this low coverage are numerous. There are organisational |
18 | 27eb699d | Thomas Thrainer | reasons, like no strict requirements for unit tests for each feature. |
19 | 27eb699d | Thomas Thrainer | But there are also design and technical reasons, which this design |
20 | 27eb699d | Thomas Thrainer | document wants to address. First, it's not clear which parts of LU's |
21 | 27eb699d | Thomas Thrainer | should be tested by unit tests, i.e. the test boundaries are not clearly |
22 | 27eb699d | Thomas Thrainer | defined. And secondly, it's too hard to actually write unit tests for |
23 | 27eb699d | Thomas Thrainer | LU's. There exists no good framework or set of tools to write easy to |
24 | 27eb699d | Thomas Thrainer | understand and concise tests. |
25 | 27eb699d | Thomas Thrainer | |
26 | 27eb699d | Thomas Thrainer | Proposed changes |
27 | 27eb699d | Thomas Thrainer | ================ |
28 | 27eb699d | Thomas Thrainer | |
29 | 27eb699d | Thomas Thrainer | This design document consists of two parts. Initially, the test |
30 | 27eb699d | Thomas Thrainer | boundaries for cmdlib are laid out, and considerations about writing |
31 | 27eb699d | Thomas Thrainer | unit tests are given. Then the test framework is described, together |
32 | 27eb699d | Thomas Thrainer | with a rough overview of the individual parts and how they are meant |
33 | 27eb699d | Thomas Thrainer | to be used. |
34 | 27eb699d | Thomas Thrainer | |
35 | 27eb699d | Thomas Thrainer | Test boundaries |
36 | 27eb699d | Thomas Thrainer | --------------- |
37 | 27eb699d | Thomas Thrainer | |
38 | 27eb699d | Thomas Thrainer | For the cmdlib module, every LogicalUnit is seen as a unit for testing. |
39 | 27eb699d | Thomas Thrainer | Unit tests for LU's may only execute the LU but make sure that no side |
40 | 27eb699d | Thomas Thrainer | effect (like filesystem access, network access or the like) takes |
41 | 27eb699d | Thomas Thrainer | place. Smaller test units (like individual methods) are sensible and |
42 | 27eb699d | Thomas Thrainer | will be supported by the test framework. However, they are not the main |
43 | 27eb699d | Thomas Thrainer | scope of this document. |
44 | 27eb699d | Thomas Thrainer | |
45 | 27eb699d | Thomas Thrainer | LU's require the following environment to be provided by the test code |
46 | 27eb699d | Thomas Thrainer | in order to be executed: |
47 | 27eb699d | Thomas Thrainer | |
48 | 27eb699d | Thomas Thrainer | An input opcode |
49 | 27eb699d | Thomas Thrainer | LU's get all the user provided input and parameters from the opcode. |
50 | 27eb699d | Thomas Thrainer | The command processor |
51 | 27eb699d | Thomas Thrainer | Used to get the execution context id and to output logging messages. |
52 | 27eb699d | Thomas Thrainer | It also drives the execution of LU's by calling the appropriate |
53 | 27eb699d | Thomas Thrainer | methods in the right order. |
54 | 27eb699d | Thomas Thrainer | The Ganeti context |
55 | 27eb699d | Thomas Thrainer | Provides node-management methods and contains |
56 | 27eb699d | Thomas Thrainer | |
57 | 27eb699d | Thomas Thrainer | * The configuration. This gives access to the cluster configuration. |
58 | 27eb699d | Thomas Thrainer | * The Ganeti Lock Manager. Manages locks during the execution. |
59 | 27eb699d | Thomas Thrainer | |
60 | 27eb699d | Thomas Thrainer | The RPC runner |
61 | 27eb699d | Thomas Thrainer | Used to communicate with node daemons on other nodes and to perform |
62 | 27eb699d | Thomas Thrainer | operations on them. |
63 | 27eb699d | Thomas Thrainer | |
64 | 27eb699d | Thomas Thrainer | The IAllocator runner |
65 | 27eb699d | Thomas Thrainer | Calls the IAllocator with a given request. |
66 | 27eb699d | Thomas Thrainer | |
67 | 27eb699d | Thomas Thrainer | All of those components have to be replaced/adapted by the test |
68 | 27eb699d | Thomas Thrainer | framework. |
69 | 27eb699d | Thomas Thrainer | |
70 | 27eb699d | Thomas Thrainer | The goal of unit tests at the LU level is to exercise every possible |
71 | 27eb699d | Thomas Thrainer | code path in the LU at least once. Shared methods which are used by |
72 | 27eb699d | Thomas Thrainer | multiple LU's should be made testable by themselves and explicit unit |
73 | 27eb699d | Thomas Thrainer | tests should be written for them. |
74 | 27eb699d | Thomas Thrainer | |
75 | 27eb699d | Thomas Thrainer | Ultimately, the code coverage for the cmdlib module should be higher |
76 | 27eb699d | Thomas Thrainer | than 90%. As Python is a dynamic language, a portion of those tests |
77 | 27eb699d | Thomas Thrainer | only exists to exercise the code without actually asserting for |
78 | 27eb699d | Thomas Thrainer | anything in the test. They merely make sure that no type errors exist |
79 | 27eb699d | Thomas Thrainer | and that potential typos etc. are caught at unit test time. |
80 | 27eb699d | Thomas Thrainer | |
81 | 27eb699d | Thomas Thrainer | Test framework |
82 | 27eb699d | Thomas Thrainer | -------------- |
83 | 27eb699d | Thomas Thrainer | |
84 | 27eb699d | Thomas Thrainer | The test framework will it make possible to write short and concise |
85 | 27eb699d | Thomas Thrainer | tests for LU's. In the simplest case, only an opcode has to be provided |
86 | 27eb699d | Thomas Thrainer | by the test. The framework will then use default values, like an almost |
87 | 27eb699d | Thomas Thrainer | empty configuration with only the master node and no instances. |
88 | 27eb699d | Thomas Thrainer | |
89 | 27eb699d | Thomas Thrainer | All aspects of the test environment will be configurable by individual |
90 | 27eb699d | Thomas Thrainer | tests. |
91 | 27eb699d | Thomas Thrainer | |
92 | 27eb699d | Thomas Thrainer | MCPU mocking |
93 | 27eb699d | Thomas Thrainer | ************ |
94 | 27eb699d | Thomas Thrainer | |
95 | 27eb699d | Thomas Thrainer | The MCPU drives the execution of LU's. It has to perform its usual |
96 | 27eb699d | Thomas Thrainer | sequence of actions, but additionally it has to provide easy access to |
97 | 27eb699d | Thomas Thrainer | the log output of LU's. It will contain utility assertion methods on the |
98 | 27eb699d | Thomas Thrainer | output. |
99 | 27eb699d | Thomas Thrainer | |
100 | 27eb699d | Thomas Thrainer | The mock will be a sub-class of ``mcpu.Processor`` which overrides |
101 | 27eb699d | Thomas Thrainer | portions of it in order to support the additional functionality. The |
102 | 27eb699d | Thomas Thrainer | advantage of being a sub-class of the original processor is the |
103 | 27eb699d | Thomas Thrainer | automatic compatibility with the code running in real clusters. |
104 | 27eb699d | Thomas Thrainer | |
105 | 27eb699d | Thomas Thrainer | Configuration mocking |
106 | 27eb699d | Thomas Thrainer | ********************* |
107 | 27eb699d | Thomas Thrainer | |
108 | 27eb699d | Thomas Thrainer | Per default, the mocked configuration will contain only the master node, |
109 | 27eb699d | Thomas Thrainer | no instances and default parameters. However, convenience methods for |
110 | 27eb699d | Thomas Thrainer | the following use cases will be provided: |
111 | 27eb699d | Thomas Thrainer | |
112 | 27eb699d | Thomas Thrainer | - "Shortcut" methods to add objects to the configuration. |
113 | 27eb699d | Thomas Thrainer | - Helper methods to quickly create standard nodes/instances/etc. |
114 | 27eb699d | Thomas Thrainer | - Pre-populated default configurations for standard use-cases (i.e. |
115 | 27eb699d | Thomas Thrainer | cluster with three nodes, five instances, etc.). |
116 | 27eb699d | Thomas Thrainer | - Convenience assertion methods for checking the configuration. |
117 | 27eb699d | Thomas Thrainer | |
118 | 27eb699d | Thomas Thrainer | Lock mocking |
119 | 27eb699d | Thomas Thrainer | ************ |
120 | 27eb699d | Thomas Thrainer | |
121 | 27eb699d | Thomas Thrainer | Initially, the mocked lock manager always grants all locks. It performs |
122 | 27eb699d | Thomas Thrainer | the following tasks: |
123 | 27eb699d | Thomas Thrainer | |
124 | 27eb699d | Thomas Thrainer | - It keeps track of requested/released locks. |
125 | 27eb699d | Thomas Thrainer | - Provides utility assertion methods for checking locks (current and |
126 | 27eb699d | Thomas Thrainer | already released ones). |
127 | 27eb699d | Thomas Thrainer | |
128 | 27eb699d | Thomas Thrainer | In the future, this component might be extended to prevent locks from |
129 | 27eb699d | Thomas Thrainer | being granted. This could eventually be used to test optimistic locking. |
130 | 27eb699d | Thomas Thrainer | |
131 | 27eb699d | Thomas Thrainer | RPC mocking |
132 | 27eb699d | Thomas Thrainer | *********** |
133 | 27eb699d | Thomas Thrainer | |
134 | 27eb699d | Thomas Thrainer | No actual RPC can be made during unit tests. Therefore, those calls have |
135 | 27eb699d | Thomas Thrainer | to be replaced and their results mocked. As this will entail a large |
136 | 27eb699d | Thomas Thrainer | portion of work when writing tests, mocking RPC's will be made as easy as |
137 | 27eb699d | Thomas Thrainer | possible. This entails: |
138 | 27eb699d | Thomas Thrainer | |
139 | 27eb699d | Thomas Thrainer | - Easy construction of RPC results. |
140 | 27eb699d | Thomas Thrainer | - Easy mocking of RPC calls (also multiple ones of the same type during |
141 | 27eb699d | Thomas Thrainer | one LU execution). |
142 | 27eb699d | Thomas Thrainer | - Asserting for RPC calls (including arguments, affected nodes, etc.). |
143 | 27eb699d | Thomas Thrainer | |
144 | 27eb699d | Thomas Thrainer | IAllocator mocking |
145 | 27eb699d | Thomas Thrainer | ****************** |
146 | 27eb699d | Thomas Thrainer | |
147 | 27eb699d | Thomas Thrainer | Calls (also multiple ones during the execution of a LU) to the |
148 | 27eb699d | Thomas Thrainer | IAllocator interface have to be mocked. The framework will provide, |
149 | 27eb699d | Thomas Thrainer | similarly to the RPC mocking, provide means to specify the mocked result |
150 | 27eb699d | Thomas Thrainer | and to assert on the IAllocator requests. |
151 | 27eb699d | Thomas Thrainer | |
152 | 27eb699d | Thomas Thrainer | Future work |
153 | 27eb699d | Thomas Thrainer | =========== |
154 | 27eb699d | Thomas Thrainer | |
155 | 27eb699d | Thomas Thrainer | With unit tests for cmdlib in place, further unit testing for other |
156 | 27eb699d | Thomas Thrainer | modules can and should be added. The test boundaries therefore should be |
157 | 27eb699d | Thomas Thrainer | aligned with the boundaries from cmdlib. |
158 | 27eb699d | Thomas Thrainer | |
159 | 27eb699d | Thomas Thrainer | The mocked locking module can be extended to allow testing of optimistic |
160 | 27eb699d | Thomas Thrainer | locking in LU's. In this case, on all requested locks are actually |
161 | 27eb699d | Thomas Thrainer | granted to the LU, so it has to adapt for this situation correctly. |
162 | 27eb699d | Thomas Thrainer | |
163 | 27eb699d | Thomas Thrainer | A higher test coverage for LU's will increase confidence in our code and |
164 | 27eb699d | Thomas Thrainer | tests. Refactorings will be easier to make as more problems are caught |
165 | 27eb699d | Thomas Thrainer | during tests. |
166 | 27eb699d | Thomas Thrainer | |
167 | 27eb699d | Thomas Thrainer | After a baseline of unit tests is established for cmdlib, efficient |
168 | 27eb699d | Thomas Thrainer | testing guidelines could be put in place. For example, new code could be |
169 | 27eb699d | Thomas Thrainer | required to not lower the test coverage in cmdlib. Additionally, every |
170 | 27eb699d | Thomas Thrainer | bug fix could be required to include a test which triggered the bug |
171 | 27eb699d | Thomas Thrainer | before the fix is created. |
172 | 27eb699d | Thomas Thrainer | |
173 | 27eb699d | Thomas Thrainer | .. vim: set textwidth=72 : |
174 | 27eb699d | Thomas Thrainer | .. Local Variables: |
175 | 27eb699d | Thomas Thrainer | .. mode: rst |
176 | 27eb699d | Thomas Thrainer | .. fill-column: 72 |
177 | 27eb699d | Thomas Thrainer | .. End: |