root / doc / design-storagetypes.rst @ 32f88ce7
History | View | Annotate | Download (13.7 kB)
1 |
============================================================================= |
---|---|
2 |
Management of storage types and disk templates, incl. storage space reporting |
3 |
============================================================================= |
4 |
|
5 |
.. contents:: :depth: 4 |
6 |
|
7 |
Background |
8 |
========== |
9 |
|
10 |
Currently, there is no consistent management of different variants of storage |
11 |
in Ganeti. One direct consequence is that storage space reporting is currently |
12 |
broken for all storage that is not based on lvm technolgy. This design looks at |
13 |
the root causes and proposes a way to fix it. |
14 |
|
15 |
Proposed changes |
16 |
================ |
17 |
|
18 |
We propose to streamline handling of different storage types and disk templates. |
19 |
Currently, there is no consistent implementation for dis/enabling of disk |
20 |
templates and/or storage types. |
21 |
|
22 |
Our idea is to introduce a list of enabled disk templates, which can be |
23 |
used by instances in the cluster. Based on this list, we want to provide |
24 |
storage reporting mechanisms for the available disk templates. Since some |
25 |
disk templates share the same underlying storage technology (for example |
26 |
``drbd`` and ``plain`` are based on ``lvm``), we map disk templates to storage |
27 |
types and implement storage space reporting for each storage type. |
28 |
|
29 |
Configuration changes |
30 |
--------------------- |
31 |
|
32 |
Add a new attribute "enabled_disk_templates" (type: list of strings) to the |
33 |
cluster config which holds disk templates, for example, "drbd", "file", |
34 |
or "ext". This attribute represents the list of disk templates that are enabled |
35 |
cluster-wide for usage by the instances. It will not be possible to create |
36 |
instances with a disk template that is not enabled, as well as it will not be |
37 |
possible to remove a disk template from the list if there are still instances |
38 |
using it. |
39 |
|
40 |
The list of enabled disk templates can contain any non-empty subset of |
41 |
the currently implemented disk templates: ``blockdev``, ``diskless``, ``drbd``, |
42 |
``ext``, ``file``, ``plain``, ``rbd``, and ``sharedfile``. See |
43 |
``DISK_TEMPLATES`` in ``constants.py``. |
44 |
|
45 |
Note that the abovementioned list of enabled disk types is just a "mechanism" |
46 |
parameter that defines which disk templates the cluster can use. Further |
47 |
filtering about what's allowed can go in the ipolicy, which is not covered in |
48 |
this design doc. Note that it is possible to force an instance to use a disk |
49 |
template that is not allowed by the ipolicy. This is not possible if the |
50 |
template is not enabled by the cluster. |
51 |
|
52 |
The ipolicy also contains a list of enabled disk templates. Since the cluster- |
53 |
wide enabled disk templates should be a stronger constraint, the list of |
54 |
enabled disk templates in the ipolicy should be a subset of those. In case the |
55 |
user tries to create an inconsistent situation here, gnt-cluster should emit |
56 |
a warning. |
57 |
|
58 |
We consider the first disk template in the list to be the default template for |
59 |
instance creation and storage reporting. This will remove the need to specify |
60 |
the disk template with ``-t`` on instance creation. |
61 |
|
62 |
Currently, cluster-wide dis/enabling of disk templates is not implemented |
63 |
consistently. ``lvm`` based disk templates are enabled by specifying a volume |
64 |
group name on cluster initialization and can only be disabled by explicitly |
65 |
using the option ``--no-lvm-storage``. This will be replaced by adding/removing |
66 |
``drbd`` and ``plain`` from the set of enabled disk templates. |
67 |
|
68 |
Up till now, file storage and shared file storage could be dis/enabled at |
69 |
``./configure`` time. This will also be replaced by adding/removing the |
70 |
respective disk templates from the set of enabled disk templates. |
71 |
|
72 |
There is currently no possibility to dis/enable the disk templates |
73 |
``diskless``, ``blockdev``, ``ext``, and ``rdb``. By introducing the set of |
74 |
enabled disk templates, we will require these disk templates to be explicitely |
75 |
enabled in order to be used. The idea is that the administrator of the cluster |
76 |
can tailor the cluster configuration to what is actually needed in the cluster. |
77 |
There is hope that this will lead to cleaner code, better performance and fewer |
78 |
bugs. |
79 |
|
80 |
When upgrading the configuration from a version that did not have the list |
81 |
of enabled disk templates, we have to decide which disk templates are enabled |
82 |
based on the current configuration of the cluster. We propose the following |
83 |
update logic to be implemented in the online update of the config in |
84 |
the ``Cluster`` class in ``objects.py``: |
85 |
- If a ``volume_group_name`` is existing, then enable ``drbd`` and ``plain``. |
86 |
(TODO: can we narrow that down further?) |
87 |
- If ``file`` or ``sharedfile`` was enabled at configure time, add the |
88 |
respective disk template to the list of enabled disk templates. |
89 |
- For disk templates ``diskless``, ``blockdev``, ``ext``, and ``rbd``, we |
90 |
inspect the current cluster configuration regarding whether or not there |
91 |
are instances that use one of those disk templates. We will add only those |
92 |
that are currently in use. |
93 |
The order in which the list of enabled disk templates is built up will be |
94 |
determined by a preference order based on when in the history of Ganeti the |
95 |
disk templates were introduced (thus being a heuristic for which are used |
96 |
more than others). |
97 |
|
98 |
The list of enabled disk templates can be specified on cluster initialization |
99 |
with ``gnt-cluster init`` using the optional parameter |
100 |
``--enabled-disk-templates``. If it is not set, it will be set to a default |
101 |
set of enabled disk templates, which includes the following disk templates: |
102 |
``drbd`` and ``plain``. The list can be shrunk or extended by |
103 |
``gnt-cluster modify`` using the same parameter. |
104 |
|
105 |
Storage reporting |
106 |
----------------- |
107 |
|
108 |
The storage reporting in ``gnt-node list`` will be the first user of the |
109 |
newly introduced list of enabled disk templates. Currently, storage reporting |
110 |
works only for lvm-based storage. We want to extend that and report storage |
111 |
for the enabled disk templates. The default of ``gnt-node list`` will only |
112 |
report on storage of the default disk template (the first in the list of enabled |
113 |
disk templates). One can explicitly ask for storage reporting on the other |
114 |
enabled disk templates with the ``-o`` option. |
115 |
|
116 |
Some of the currently implemented disk templates share the same base storage |
117 |
technology. Since the storage reporting is based on the underlying technology |
118 |
rather than on the user-facing disk templates, we introduce storage types to |
119 |
represent the underlying technology. There will be a mapping from disk templates |
120 |
to storage types, which will be used by the storage reporting backend to pick |
121 |
the right method for estimating the storage for the different disk templates. |
122 |
|
123 |
The proposed storage types are ``blockdev``, ``diskless``, ``ext``, ``file``, |
124 |
``lvm-pv``, ``lvm-vg``, ``rados``. |
125 |
|
126 |
The mapping from disk templates to storage types will be: ``drbd`` and ``plain`` |
127 |
to ``lvm-vg``, ``file`` and ``sharedfile`` to ``file``, and all others to their |
128 |
obvious counterparts. |
129 |
|
130 |
Note that there is no disk template mapping to ``lvm-pv``, because this storage |
131 |
type is currently only used to enable the user to mark it as (un)allocatable. |
132 |
(See ``man gnt-node``.) It is not possible to create an instance on a storage |
133 |
unit that is of type ``lvm-pv`` directly, therefore it is not included in the |
134 |
mapping. |
135 |
|
136 |
The storage reporting for file storage will report space on the file storage |
137 |
dir, which is currently limited to one directory. In the future, if we'll have |
138 |
support for more directories, or for per-nodegroup directories this can be |
139 |
changed. |
140 |
|
141 |
For now, we will implement only the storage reporting for non-shared storage, |
142 |
that is disk templates ``file``, ``lvm``, and ``drbd``. For disk template |
143 |
``diskless``, there is obviously nothing to report about. When implementing |
144 |
storage reporting for file, we can also use it for ``sharedfile``, since it |
145 |
uses the same file system mechanisms to determine the free space. In the |
146 |
future, we can optimize storage reporting for shared storage by not querying |
147 |
all nodes that use a common shared file for the same space information. |
148 |
|
149 |
In the future, we extend storage reporting for shared storage types like |
150 |
``rados`` and ``ext``. Note that it will not make sense to query each node for |
151 |
storage reporting on a storage unit that is used by several nodes. |
152 |
|
153 |
We will not implement storage reporting for the ``blockdev`` disk template, |
154 |
because block devices are always adopted after being provided by the system |
155 |
administrator, thus coming from outside Ganeti. There is no point in storage |
156 |
reporting for block devices, because Ganeti will never try to allocate storage |
157 |
inside a block device. |
158 |
|
159 |
RPC changes |
160 |
----------- |
161 |
|
162 |
The noded RPC call that reports node storage space will be changed to |
163 |
accept a list of <storage_type>,<key> string tuples. For each of them, it will |
164 |
report the free amount of storage space found on storage <key> as known |
165 |
by the requested storage_type. Depending on the storage_type, the key would |
166 |
be a volume group name in case of lvm, a directory name for the file-based |
167 |
storage, and a rados pool name for rados storage. |
168 |
|
169 |
Masterd will know through the mapping of storage types to storage calculation |
170 |
functions which storage type uses which mechanism for storage calculation |
171 |
and invoke only the needed ones. |
172 |
|
173 |
Note that for file and sharedfile the node knows which directories are allowed |
174 |
and won't allow any other directory to be queried for security reasons. The |
175 |
actual path still needs to be passed to distinguish the two, as the type will |
176 |
be the same for both. |
177 |
|
178 |
These calculations will be implemented in the node storage system |
179 |
(currently lib/storage.py) but querying will still happen through the |
180 |
``node info`` call, to avoid requiring an extra RPC each time. |
181 |
|
182 |
Ganeti reporting |
183 |
---------------- |
184 |
|
185 |
`gnt-node list`` can be queried for the different disk templates, if they |
186 |
are enabled. By default, it will just report information about the default |
187 |
disk template. Examples:: |
188 |
|
189 |
> gnt-node list |
190 |
Node DTotal DFree MTotal MNode MFree Pinst Sinst |
191 |
mynode1 3.6T 3.6T 64.0G 1023M 62.2G 1 0 |
192 |
mynode2 3.6T 3.6T 64.0G 1023M 62.0G 2 1 |
193 |
mynode3 3.6T 3.6T 64.0G 1023M 62.3G 0 2 |
194 |
|
195 |
> gnt-node list -o dtotal/drbd,dfree/file |
196 |
Node DTotal (drbd, myvg) DFree (file, mydir) |
197 |
mynode1 3.6T - |
198 |
mynode2 3.6T - |
199 |
|
200 |
Note that for drbd, we only report the space of the vg and only if it was not |
201 |
renamed to something different than the default volume group name. With this |
202 |
design, there is also no possibility to ask about the meta volume group. We |
203 |
restrict the design here to make the transition to storage pools easier (as it |
204 |
is an interim state only). It is the administrator's responsibility to ensure |
205 |
that there is enough space for the meta volume group. |
206 |
|
207 |
When storage pools are implemented, we switch from referencing the disk template |
208 |
to referencing the storage pool name. For that, of course, the pool names need |
209 |
to be unique over all storage types. For drbd, we will use the default 'drbd' |
210 |
storage pool and possibly a second lvm-based storage pool for the metavg. It |
211 |
will be possible to rename storage pools (thus also the default lvm storage |
212 |
pool). There will be new functionality to ask about what storage pools are |
213 |
available and of what type. Storage pools will have a storage pool type which is |
214 |
one of the disk templates. There can be more than one storage pool based on the |
215 |
same disk template, therefore we will then start referencing the storage pool |
216 |
name instead of the disk template. |
217 |
|
218 |
``gnt-cluster info`` will report which disk templates are enabled, i.e. |
219 |
which ones are supported according to the cluster configuration. Example |
220 |
output:: |
221 |
|
222 |
> gnt-cluster info |
223 |
[...] |
224 |
Cluster parameters: |
225 |
- [...] |
226 |
- enabled disk templates: plain, drbd, sharedfile, rados |
227 |
- [...] |
228 |
|
229 |
``gnt-node list-storage`` will not be affected by any changes, since this design |
230 |
is restricted only to free storage reporting for non-shared storage types. |
231 |
|
232 |
Allocator changes |
233 |
----------------- |
234 |
|
235 |
The iallocator protocol doesn't need to change: since we know which |
236 |
disk template an instance has, we'll pass only the "free" value for that |
237 |
disk template to the iallocator, when asking for an allocation to be |
238 |
made. Note that for DRBD nowadays we ignore the case when vg and metavg |
239 |
are different, and we only consider the main volume group. Fixing this is |
240 |
outside the scope of this design. |
241 |
|
242 |
With this design, we ensure forward-compatibility with respect to storage |
243 |
pools. For now, we'll report space for all available disk templates that |
244 |
are based on non-shared storage types, in the future, for all available |
245 |
storage pools. |
246 |
|
247 |
Rebalancing changes |
248 |
------------------- |
249 |
|
250 |
Hbal will not need changes, as it handles it already. We don't forecast |
251 |
any changes needed to it. |
252 |
|
253 |
Space reporting changes |
254 |
----------------------- |
255 |
|
256 |
Hspace will by default report by assuming the allocation will happen on |
257 |
the default disk template for the cluster/nodegroup. An option will be added |
258 |
to manually specify a different storage. |
259 |
|
260 |
Interactions with Partitioned Ganeti |
261 |
------------------------------------ |
262 |
|
263 |
Also the design for :doc:`Partitioned Ganeti <design-partitioned>` deals |
264 |
with reporting free space. Partitioned Ganeti has a different way to |
265 |
report free space for LVM on nodes where the ``exclusive_storage`` flag |
266 |
is set. That doesn't interact directly with this design, as the specifics |
267 |
of how the free space is computed is not in the scope of this design. |
268 |
But the ``node info`` call contains the value of the |
269 |
``exclusive_storage`` flag, which is currently only meaningful for the |
270 |
LVM storage type. Additional flags like the ``exclusive_storage`` flag |
271 |
for lvm might be useful for other disk templates / storage types as well. |
272 |
We therefore extend the RPC call with <storage_type>,<key> to |
273 |
<storage_type>,<key>,[<param>] to include any disk-template-specific |
274 |
(or storage-type specific) parameters in the RPC call. |
275 |
|
276 |
The reporting of free spindles, also part of Partitioned Ganeti, is not |
277 |
concerned with this design doc, as those are seen as a separate resource. |
278 |
|
279 |
.. vim: set textwidth=72 : |
280 |
.. Local Variables: |
281 |
.. mode: rst |
282 |
.. fill-column: 72 |
283 |
.. End: |