root / doc / design-cmdlib-unittests.rst @ 47cce79a
History | View | Annotate | Download (6.6 kB)
1 |
===================================== |
---|---|
2 |
Unit tests for cmdlib / LogicalUnit's |
3 |
===================================== |
4 |
|
5 |
.. contents:: :depth: 4 |
6 |
|
7 |
This is a design document describing unit tests for the cmdlib module. |
8 |
Other modules are deliberately omitted, as LU's contain the most complex |
9 |
logic and are only sparingly tested. |
10 |
|
11 |
Current state and shortcomings |
12 |
============================== |
13 |
|
14 |
The current test coverage of the cmdlib module is at only ~14%. Given |
15 |
the complexity of the code this is clearly too little. |
16 |
|
17 |
The reasons for this low coverage are numerous. There are organisational |
18 |
reasons, like no strict requirements for unit tests for each feature. |
19 |
But there are also design and technical reasons, which this design |
20 |
document wants to address. First, it's not clear which parts of LU's |
21 |
should be tested by unit tests, i.e. the test boundaries are not clearly |
22 |
defined. And secondly, it's too hard to actually write unit tests for |
23 |
LU's. There exists no good framework or set of tools to write easy to |
24 |
understand and concise tests. |
25 |
|
26 |
Proposed changes |
27 |
================ |
28 |
|
29 |
This design document consists of two parts. Initially, the test |
30 |
boundaries for cmdlib are laid out, and considerations about writing |
31 |
unit tests are given. Then the test framework is described, together |
32 |
with a rough overview of the individual parts and how they are meant |
33 |
to be used. |
34 |
|
35 |
Test boundaries |
36 |
--------------- |
37 |
|
38 |
For the cmdlib module, every LogicalUnit is seen as a unit for testing. |
39 |
Unit tests for LU's may only execute the LU but make sure that no side |
40 |
effect (like filesystem access, network access or the like) takes |
41 |
place. Smaller test units (like individual methods) are sensible and |
42 |
will be supported by the test framework. However, they are not the main |
43 |
scope of this document. |
44 |
|
45 |
LU's require the following environment to be provided by the test code |
46 |
in order to be executed: |
47 |
|
48 |
An input opcode |
49 |
LU's get all the user provided input and parameters from the opcode. |
50 |
The command processor |
51 |
Used to get the execution context id and to output logging messages. |
52 |
It also drives the execution of LU's by calling the appropriate |
53 |
methods in the right order. |
54 |
The Ganeti context |
55 |
Provides node-management methods and contains |
56 |
|
57 |
* The configuration. This gives access to the cluster configuration. |
58 |
* The Ganeti Lock Manager. Manages locks during the execution. |
59 |
|
60 |
The RPC runner |
61 |
Used to communicate with node daemons on other nodes and to perform |
62 |
operations on them. |
63 |
|
64 |
The IAllocator runner |
65 |
Calls the IAllocator with a given request. |
66 |
|
67 |
All of those components have to be replaced/adapted by the test |
68 |
framework. |
69 |
|
70 |
The goal of unit tests at the LU level is to exercise every possible |
71 |
code path in the LU at least once. Shared methods which are used by |
72 |
multiple LU's should be made testable by themselves and explicit unit |
73 |
tests should be written for them. |
74 |
|
75 |
Ultimately, the code coverage for the cmdlib module should be higher |
76 |
than 90%. As Python is a dynamic language, a portion of those tests |
77 |
only exists to exercise the code without actually asserting for |
78 |
anything in the test. They merely make sure that no type errors exist |
79 |
and that potential typos etc. are caught at unit test time. |
80 |
|
81 |
Test framework |
82 |
-------------- |
83 |
|
84 |
The test framework will it make possible to write short and concise |
85 |
tests for LU's. In the simplest case, only an opcode has to be provided |
86 |
by the test. The framework will then use default values, like an almost |
87 |
empty configuration with only the master node and no instances. |
88 |
|
89 |
All aspects of the test environment will be configurable by individual |
90 |
tests. |
91 |
|
92 |
MCPU mocking |
93 |
************ |
94 |
|
95 |
The MCPU drives the execution of LU's. It has to perform its usual |
96 |
sequence of actions, but additionally it has to provide easy access to |
97 |
the log output of LU's. It will contain utility assertion methods on the |
98 |
output. |
99 |
|
100 |
The mock will be a sub-class of ``mcpu.Processor`` which overrides |
101 |
portions of it in order to support the additional functionality. The |
102 |
advantage of being a sub-class of the original processor is the |
103 |
automatic compatibility with the code running in real clusters. |
104 |
|
105 |
Configuration mocking |
106 |
********************* |
107 |
|
108 |
Per default, the mocked configuration will contain only the master node, |
109 |
no instances and default parameters. However, convenience methods for |
110 |
the following use cases will be provided: |
111 |
|
112 |
- "Shortcut" methods to add objects to the configuration. |
113 |
- Helper methods to quickly create standard nodes/instances/etc. |
114 |
- Pre-populated default configurations for standard use-cases (i.e. |
115 |
cluster with three nodes, five instances, etc.). |
116 |
- Convenience assertion methods for checking the configuration. |
117 |
|
118 |
Lock mocking |
119 |
************ |
120 |
|
121 |
Initially, the mocked lock manager always grants all locks. It performs |
122 |
the following tasks: |
123 |
|
124 |
- It keeps track of requested/released locks. |
125 |
- Provides utility assertion methods for checking locks (current and |
126 |
already released ones). |
127 |
|
128 |
In the future, this component might be extended to prevent locks from |
129 |
being granted. This could eventually be used to test optimistic locking. |
130 |
|
131 |
RPC mocking |
132 |
*********** |
133 |
|
134 |
No actual RPC can be made during unit tests. Therefore, those calls have |
135 |
to be replaced and their results mocked. As this will entail a large |
136 |
portion of work when writing tests, mocking RPC's will be made as easy as |
137 |
possible. This entails: |
138 |
|
139 |
- Easy construction of RPC results. |
140 |
- Easy mocking of RPC calls (also multiple ones of the same type during |
141 |
one LU execution). |
142 |
- Asserting for RPC calls (including arguments, affected nodes, etc.). |
143 |
|
144 |
IAllocator mocking |
145 |
****************** |
146 |
|
147 |
Calls (also multiple ones during the execution of a LU) to the |
148 |
IAllocator interface have to be mocked. The framework will provide, |
149 |
similarly to the RPC mocking, provide means to specify the mocked result |
150 |
and to assert on the IAllocator requests. |
151 |
|
152 |
Future work |
153 |
=========== |
154 |
|
155 |
With unit tests for cmdlib in place, further unit testing for other |
156 |
modules can and should be added. The test boundaries therefore should be |
157 |
aligned with the boundaries from cmdlib. |
158 |
|
159 |
The mocked locking module can be extended to allow testing of optimistic |
160 |
locking in LU's. In this case, on all requested locks are actually |
161 |
granted to the LU, so it has to adapt for this situation correctly. |
162 |
|
163 |
A higher test coverage for LU's will increase confidence in our code and |
164 |
tests. Refactorings will be easier to make as more problems are caught |
165 |
during tests. |
166 |
|
167 |
After a baseline of unit tests is established for cmdlib, efficient |
168 |
testing guidelines could be put in place. For example, new code could be |
169 |
required to not lower the test coverage in cmdlib. Additionally, every |
170 |
bug fix could be required to include a test which triggered the bug |
171 |
before the fix is created. |
172 |
|
173 |
.. vim: set textwidth=72 : |
174 |
.. Local Variables: |
175 |
.. mode: rst |
176 |
.. fill-column: 72 |
177 |
.. End: |