|
1 |
=====================================
|
|
2 |
Unit tests for cmdlib / LogicalUnit's
|
|
3 |
=====================================
|
|
4 |
|
|
5 |
.. contents:: :depth: 4
|
|
6 |
|
|
7 |
This is a design document describing unit tests for the cmdlib module.
|
|
8 |
Other modules are deliberately omitted, as LU's contain the most complex
|
|
9 |
logic and are only sparingly tested.
|
|
10 |
|
|
11 |
Current state and shortcomings
|
|
12 |
==============================
|
|
13 |
|
|
14 |
The current test coverage of the cmdlib module is at only ~14%. Given
|
|
15 |
the complexity of the code this is clearly too little.
|
|
16 |
|
|
17 |
The reasons for this low coverage are numerous. There are organisational
|
|
18 |
reasons, like no strict requirements for unit tests for each feature.
|
|
19 |
But there are also design and technical reasons, which this design
|
|
20 |
document wants to address. First, it's not clear which parts of LU's
|
|
21 |
should be tested by unit tests, i.e. the test boundaries are not clearly
|
|
22 |
defined. And secondly, it's too hard to actually write unit tests for
|
|
23 |
LU's. There exists no good framework or set of tools to write easy to
|
|
24 |
understand and concise tests.
|
|
25 |
|
|
26 |
Proposed changes
|
|
27 |
================
|
|
28 |
|
|
29 |
This design document consists of two parts. Initially, the test
|
|
30 |
boundaries for cmdlib are laid out, and considerations about writing
|
|
31 |
unit tests are given. Then the test framework is described, together
|
|
32 |
with a rough overview of the individual parts and how they are meant
|
|
33 |
to be used.
|
|
34 |
|
|
35 |
Test boundaries
|
|
36 |
---------------
|
|
37 |
|
|
38 |
For the cmdlib module, every LogicalUnit is seen as a unit for testing.
|
|
39 |
Unit tests for LU's may only execute the LU but make sure that no side
|
|
40 |
effect (like filesystem access, network access or the like) takes
|
|
41 |
place. Smaller test units (like individual methods) are sensible and
|
|
42 |
will be supported by the test framework. However, they are not the main
|
|
43 |
scope of this document.
|
|
44 |
|
|
45 |
LU's require the following environment to be provided by the test code
|
|
46 |
in order to be executed:
|
|
47 |
|
|
48 |
An input opcode
|
|
49 |
LU's get all the user provided input and parameters from the opcode.
|
|
50 |
The command processor
|
|
51 |
Used to get the execution context id and to output logging messages.
|
|
52 |
It also drives the execution of LU's by calling the appropriate
|
|
53 |
methods in the right order.
|
|
54 |
The Ganeti context
|
|
55 |
Provides node-management methods and contains
|
|
56 |
|
|
57 |
* The configuration. This gives access to the cluster configuration.
|
|
58 |
* The Ganeti Lock Manager. Manages locks during the execution.
|
|
59 |
|
|
60 |
The RPC runner
|
|
61 |
Used to communicate with node daemons on other nodes and to perform
|
|
62 |
operations on them.
|
|
63 |
|
|
64 |
The IAllocator runner
|
|
65 |
Calls the IAllocator with a given request.
|
|
66 |
|
|
67 |
All of those components have to be replaced/adapted by the test
|
|
68 |
framework.
|
|
69 |
|
|
70 |
The goal of unit tests at the LU level is to exercise every possible
|
|
71 |
code path in the LU at least once. Shared methods which are used by
|
|
72 |
multiple LU's should be made testable by themselves and explicit unit
|
|
73 |
tests should be written for them.
|
|
74 |
|
|
75 |
Ultimately, the code coverage for the cmdlib module should be higher
|
|
76 |
than 90%. As Python is a dynamic language, a portion of those tests
|
|
77 |
only exists to exercise the code without actually asserting for
|
|
78 |
anything in the test. They merely make sure that no type errors exist
|
|
79 |
and that potential typos etc. are caught at unit test time.
|
|
80 |
|
|
81 |
Test framework
|
|
82 |
--------------
|
|
83 |
|
|
84 |
The test framework will it make possible to write short and concise
|
|
85 |
tests for LU's. In the simplest case, only an opcode has to be provided
|
|
86 |
by the test. The framework will then use default values, like an almost
|
|
87 |
empty configuration with only the master node and no instances.
|
|
88 |
|
|
89 |
All aspects of the test environment will be configurable by individual
|
|
90 |
tests.
|
|
91 |
|
|
92 |
MCPU mocking
|
|
93 |
************
|
|
94 |
|
|
95 |
The MCPU drives the execution of LU's. It has to perform its usual
|
|
96 |
sequence of actions, but additionally it has to provide easy access to
|
|
97 |
the log output of LU's. It will contain utility assertion methods on the
|
|
98 |
output.
|
|
99 |
|
|
100 |
The mock will be a sub-class of ``mcpu.Processor`` which overrides
|
|
101 |
portions of it in order to support the additional functionality. The
|
|
102 |
advantage of being a sub-class of the original processor is the
|
|
103 |
automatic compatibility with the code running in real clusters.
|
|
104 |
|
|
105 |
Configuration mocking
|
|
106 |
*********************
|
|
107 |
|
|
108 |
Per default, the mocked configuration will contain only the master node,
|
|
109 |
no instances and default parameters. However, convenience methods for
|
|
110 |
the following use cases will be provided:
|
|
111 |
|
|
112 |
- "Shortcut" methods to add objects to the configuration.
|
|
113 |
- Helper methods to quickly create standard nodes/instances/etc.
|
|
114 |
- Pre-populated default configurations for standard use-cases (i.e.
|
|
115 |
cluster with three nodes, five instances, etc.).
|
|
116 |
- Convenience assertion methods for checking the configuration.
|
|
117 |
|
|
118 |
Lock mocking
|
|
119 |
************
|
|
120 |
|
|
121 |
Initially, the mocked lock manager always grants all locks. It performs
|
|
122 |
the following tasks:
|
|
123 |
|
|
124 |
- It keeps track of requested/released locks.
|
|
125 |
- Provides utility assertion methods for checking locks (current and
|
|
126 |
already released ones).
|
|
127 |
|
|
128 |
In the future, this component might be extended to prevent locks from
|
|
129 |
being granted. This could eventually be used to test optimistic locking.
|
|
130 |
|
|
131 |
RPC mocking
|
|
132 |
***********
|
|
133 |
|
|
134 |
No actual RPC can be made during unit tests. Therefore, those calls have
|
|
135 |
to be replaced and their results mocked. As this will entail a large
|
|
136 |
portion of work when writing tests, mocking RPC's will be made as easy as
|
|
137 |
possible. This entails:
|
|
138 |
|
|
139 |
- Easy construction of RPC results.
|
|
140 |
- Easy mocking of RPC calls (also multiple ones of the same type during
|
|
141 |
one LU execution).
|
|
142 |
- Asserting for RPC calls (including arguments, affected nodes, etc.).
|
|
143 |
|
|
144 |
IAllocator mocking
|
|
145 |
******************
|
|
146 |
|
|
147 |
Calls (also multiple ones during the execution of a LU) to the
|
|
148 |
IAllocator interface have to be mocked. The framework will provide,
|
|
149 |
similarly to the RPC mocking, provide means to specify the mocked result
|
|
150 |
and to assert on the IAllocator requests.
|
|
151 |
|
|
152 |
Future work
|
|
153 |
===========
|
|
154 |
|
|
155 |
With unit tests for cmdlib in place, further unit testing for other
|
|
156 |
modules can and should be added. The test boundaries therefore should be
|
|
157 |
aligned with the boundaries from cmdlib.
|
|
158 |
|
|
159 |
The mocked locking module can be extended to allow testing of optimistic
|
|
160 |
locking in LU's. In this case, on all requested locks are actually
|
|
161 |
granted to the LU, so it has to adapt for this situation correctly.
|
|
162 |
|
|
163 |
A higher test coverage for LU's will increase confidence in our code and
|
|
164 |
tests. Refactorings will be easier to make as more problems are caught
|
|
165 |
during tests.
|
|
166 |
|
|
167 |
After a baseline of unit tests is established for cmdlib, efficient
|
|
168 |
testing guidelines could be put in place. For example, new code could be
|
|
169 |
required to not lower the test coverage in cmdlib. Additionally, every
|
|
170 |
bug fix could be required to include a test which triggered the bug
|
|
171 |
before the fix is created.
|
|
172 |
|
|
173 |
.. vim: set textwidth=72 :
|
|
174 |
.. Local Variables:
|
|
175 |
.. mode: rst
|
|
176 |
.. fill-column: 72
|
|
177 |
.. End:
|