root / doc / design-2.0.rst @ 7eac4a4d
History | View | Annotate | Download (75.4 kB)
1 |
================= |
---|---|
2 |
Ganeti 2.0 design |
3 |
================= |
4 |
|
5 |
This document describes the major changes in Ganeti 2.0 compared to |
6 |
the 1.2 version. |
7 |
|
8 |
The 2.0 version will constitute a rewrite of the 'core' architecture, |
9 |
paving the way for additional features in future 2.x versions. |
10 |
|
11 |
.. contents:: :depth: 3 |
12 |
|
13 |
Objective |
14 |
========= |
15 |
|
16 |
Ganeti 1.2 has many scalability issues and restrictions due to its |
17 |
roots as software for managing small and 'static' clusters. |
18 |
|
19 |
Version 2.0 will attempt to remedy first the scalability issues and |
20 |
then the restrictions. |
21 |
|
22 |
Background |
23 |
========== |
24 |
|
25 |
While Ganeti 1.2 is usable, it severely limits the flexibility of the |
26 |
cluster administration and imposes a very rigid model. It has the |
27 |
following main scalability issues: |
28 |
|
29 |
- only one operation at a time on the cluster [#]_ |
30 |
- poor handling of node failures in the cluster |
31 |
- mixing hypervisors in a cluster not allowed |
32 |
|
33 |
It also has a number of artificial restrictions, due to historical |
34 |
design: |
35 |
|
36 |
- fixed number of disks (two) per instance |
37 |
- fixed number of NICs |
38 |
|
39 |
.. [#] Replace disks will release the lock, but this is an exception |
40 |
and not a recommended way to operate |
41 |
|
42 |
The 2.0 version is intended to address some of these problems, and |
43 |
create a more flexible code base for future developments. |
44 |
|
45 |
Among these problems, the single-operation at a time restriction is |
46 |
biggest issue with the current version of Ganeti. It is such a big |
47 |
impediment in operating bigger clusters that many times one is tempted |
48 |
to remove the lock just to do a simple operation like start instance |
49 |
while an OS installation is running. |
50 |
|
51 |
Scalability problems |
52 |
-------------------- |
53 |
|
54 |
Ganeti 1.2 has a single global lock, which is used for all cluster |
55 |
operations. This has been painful at various times, for example: |
56 |
|
57 |
- It is impossible for two people to efficiently interact with a cluster |
58 |
(for example for debugging) at the same time. |
59 |
- When batch jobs are running it's impossible to do other work (for |
60 |
example failovers/fixes) on a cluster. |
61 |
|
62 |
This poses scalability problems: as clusters grow in node and instance |
63 |
size it's a lot more likely that operations which one could conceive |
64 |
should run in parallel (for example because they happen on different |
65 |
nodes) are actually stalling each other while waiting for the global |
66 |
lock, without a real reason for that to happen. |
67 |
|
68 |
One of the main causes of this global lock (beside the higher |
69 |
difficulty of ensuring data consistency in a more granular lock model) |
70 |
is the fact that currently there is no long-lived process in Ganeti |
71 |
that can coordinate multiple operations. Each command tries to acquire |
72 |
the so called *cmd* lock and when it succeeds, it takes complete |
73 |
ownership of the cluster configuration and state. |
74 |
|
75 |
Other scalability problems are due the design of the DRBD device |
76 |
model, which assumed at its creation a low (one to four) number of |
77 |
instances per node, which is no longer true with today's hardware. |
78 |
|
79 |
Artificial restrictions |
80 |
----------------------- |
81 |
|
82 |
Ganeti 1.2 (and previous versions) have a fixed two-disks, one-NIC per |
83 |
instance model. This is a purely artificial restrictions, but it |
84 |
touches multiple areas (configuration, import/export, command line) |
85 |
that it's more fitted to a major release than a minor one. |
86 |
|
87 |
Architecture issues |
88 |
------------------- |
89 |
|
90 |
The fact that each command is a separate process that reads the |
91 |
cluster state, executes the command, and saves the new state is also |
92 |
an issue on big clusters where the configuration data for the cluster |
93 |
begins to be non-trivial in size. |
94 |
|
95 |
Overview |
96 |
======== |
97 |
|
98 |
In order to solve the scalability problems, a rewrite of the core |
99 |
design of Ganeti is required. While the cluster operations themselves |
100 |
won't change (e.g. start instance will do the same things, the way |
101 |
these operations are scheduled internally will change radically. |
102 |
|
103 |
The new design will change the cluster architecture to: |
104 |
|
105 |
.. image:: arch-2.0.png |
106 |
|
107 |
This differs from the 1.2 architecture by the addition of the master |
108 |
daemon, which will be the only entity to talk to the node daemons. |
109 |
|
110 |
|
111 |
Detailed design |
112 |
=============== |
113 |
|
114 |
The changes for 2.0 can be split into roughly three areas: |
115 |
|
116 |
- core changes that affect the design of the software |
117 |
- features (or restriction removals) but which do not have a wide |
118 |
impact on the design |
119 |
- user-level and API-level changes which translate into differences for |
120 |
the operation of the cluster |
121 |
|
122 |
Core changes |
123 |
------------ |
124 |
|
125 |
The main changes will be switching from a per-process model to a |
126 |
daemon based model, where the individual gnt-* commands will be |
127 |
clients that talk to this daemon (see `Master daemon`_). This will |
128 |
allow us to get rid of the global cluster lock for most operations, |
129 |
having instead a per-object lock (see `Granular locking`_). Also, the |
130 |
daemon will be able to queue jobs, and this will allow the individual |
131 |
clients to submit jobs without waiting for them to finish, and also |
132 |
see the result of old requests (see `Job Queue`_). |
133 |
|
134 |
Beside these major changes, another 'core' change but that will not be |
135 |
as visible to the users will be changing the model of object attribute |
136 |
storage, and separate that into name spaces (such that an Xen PVM |
137 |
instance will not have the Xen HVM parameters). This will allow future |
138 |
flexibility in defining additional parameters. For more details see |
139 |
`Object parameters`_. |
140 |
|
141 |
The various changes brought in by the master daemon model and the |
142 |
read-write RAPI will require changes to the cluster security; we move |
143 |
away from Twisted and use HTTP(s) for intra- and extra-cluster |
144 |
communications. For more details, see the security document in the |
145 |
doc/ directory. |
146 |
|
147 |
Master daemon |
148 |
~~~~~~~~~~~~~ |
149 |
|
150 |
In Ganeti 2.0, we will have the following *entities*: |
151 |
|
152 |
- the master daemon (on the master node) |
153 |
- the node daemon (on all nodes) |
154 |
- the command line tools (on the master node) |
155 |
- the RAPI daemon (on the master node) |
156 |
|
157 |
The master-daemon related interaction paths are: |
158 |
|
159 |
- (CLI tools/RAPI daemon) and the master daemon, via the so called |
160 |
*LUXI* API |
161 |
- the master daemon and the node daemons, via the node RPC |
162 |
|
163 |
There are also some additional interaction paths for exceptional cases: |
164 |
|
165 |
- CLI tools might access via SSH the nodes (for ``gnt-cluster copyfile`` |
166 |
and ``gnt-cluster command``) |
167 |
- master failover is a special case when a non-master node will SSH |
168 |
and do node-RPC calls to the current master |
169 |
|
170 |
The protocol between the master daemon and the node daemons will be |
171 |
changed from (Ganeti 1.2) Twisted PB (perspective broker) to HTTP(S), |
172 |
using a simple PUT/GET of JSON-encoded messages. This is done due to |
173 |
difficulties in working with the Twisted framework and its protocols |
174 |
in a multithreaded environment, which we can overcome by using a |
175 |
simpler stack (see the caveats section). |
176 |
|
177 |
The protocol between the CLI/RAPI and the master daemon will be a |
178 |
custom one (called *LUXI*): on a UNIX socket on the master node, with |
179 |
rights restricted by filesystem permissions, the CLI/RAPI will talk to |
180 |
the master daemon using JSON-encoded messages. |
181 |
|
182 |
The operations supported over this internal protocol will be encoded |
183 |
via a python library that will expose a simple API for its |
184 |
users. Internally, the protocol will simply encode all objects in JSON |
185 |
format and decode them on the receiver side. |
186 |
|
187 |
For more details about the RAPI daemon see `Remote API changes`_, and |
188 |
for the node daemon see `Node daemon changes`_. |
189 |
|
190 |
The LUXI protocol |
191 |
+++++++++++++++++ |
192 |
|
193 |
As described above, the protocol for making requests or queries to the |
194 |
master daemon will be a UNIX-socket based simple RPC of JSON-encoded |
195 |
messages. |
196 |
|
197 |
The choice of UNIX was in order to get rid of the need of |
198 |
authentication and authorisation inside Ganeti; for 2.0, the |
199 |
permissions on the Unix socket itself will determine the access |
200 |
rights. |
201 |
|
202 |
We will have two main classes of operations over this API: |
203 |
|
204 |
- cluster query functions |
205 |
- job related functions |
206 |
|
207 |
The cluster query functions are usually short-duration, and are the |
208 |
equivalent of the ``OP_QUERY_*`` opcodes in Ganeti 1.2 (and they are |
209 |
internally implemented still with these opcodes). The clients are |
210 |
guaranteed to receive the response in a reasonable time via a timeout. |
211 |
|
212 |
The job-related functions will be: |
213 |
|
214 |
- submit job |
215 |
- query job (which could also be categorized in the query-functions) |
216 |
- archive job (see the job queue design doc) |
217 |
- wait for job change, which allows a client to wait without polling |
218 |
|
219 |
For more details of the actual operation list, see the `Job Queue`_. |
220 |
|
221 |
Both requests and responses will consist of a JSON-encoded message |
222 |
followed by the ``ETX`` character (ASCII decimal 3), which is not a |
223 |
valid character in JSON messages and thus can serve as a message |
224 |
delimiter. The contents of the messages will be a dictionary with two |
225 |
fields: |
226 |
|
227 |
:method: |
228 |
the name of the method called |
229 |
:args: |
230 |
the arguments to the method, as a list (no keyword arguments allowed) |
231 |
|
232 |
Responses will follow the same format, with the two fields being: |
233 |
|
234 |
:success: |
235 |
a boolean denoting the success of the operation |
236 |
:result: |
237 |
the actual result, or error message in case of failure |
238 |
|
239 |
There are two special value for the result field: |
240 |
|
241 |
- in the case that the operation failed, and this field is a list of |
242 |
length two, the client library will try to interpret is as an |
243 |
exception, the first element being the exception type and the second |
244 |
one the actual exception arguments; this will allow a simple method of |
245 |
passing Ganeti-related exception across the interface |
246 |
- for the *WaitForChange* call (that waits on the server for a job to |
247 |
change status), if the result is equal to ``nochange`` instead of the |
248 |
usual result for this call (a list of changes), then the library will |
249 |
internally retry the call; this is done in order to differentiate |
250 |
internally between master daemon hung and job simply not changed |
251 |
|
252 |
Users of the API that don't use the provided python library should |
253 |
take care of the above two cases. |
254 |
|
255 |
|
256 |
Master daemon implementation |
257 |
++++++++++++++++++++++++++++ |
258 |
|
259 |
The daemon will be based around a main I/O thread that will wait for |
260 |
new requests from the clients, and that does the setup/shutdown of the |
261 |
other thread (pools). |
262 |
|
263 |
There will two other classes of threads in the daemon: |
264 |
|
265 |
- job processing threads, part of a thread pool, and which are |
266 |
long-lived, started at daemon startup and terminated only at shutdown |
267 |
time |
268 |
- client I/O threads, which are the ones that talk the local protocol |
269 |
(LUXI) to the clients, and are short-lived |
270 |
|
271 |
Master startup/failover |
272 |
+++++++++++++++++++++++ |
273 |
|
274 |
In Ganeti 1.x there is no protection against failing over the master |
275 |
to a node with stale configuration. In effect, the responsibility of |
276 |
correct failovers falls on the admin. This is true both for the new |
277 |
master and for when an old, offline master startup. |
278 |
|
279 |
Since in 2.x we are extending the cluster state to cover the job queue |
280 |
and have a daemon that will execute by itself the job queue, we want |
281 |
to have more resilience for the master role. |
282 |
|
283 |
The following algorithm will happen whenever a node is ready to |
284 |
transition to the master role, either at startup time or at node |
285 |
failover: |
286 |
|
287 |
#. read the configuration file and parse the node list |
288 |
contained within |
289 |
|
290 |
#. query all the nodes and make sure we obtain an agreement via |
291 |
a quorum of at least half plus one nodes for the following: |
292 |
|
293 |
- we have the latest configuration and job list (as |
294 |
determined by the serial number on the configuration and |
295 |
highest job ID on the job queue) |
296 |
|
297 |
- there is not even a single node having a newer |
298 |
configuration file |
299 |
|
300 |
- if we are not failing over (but just starting), the |
301 |
quorum agrees that we are the designated master |
302 |
|
303 |
- if any of the above is false, we prevent the current operation |
304 |
(i.e. we don't become the master) |
305 |
|
306 |
#. at this point, the node transitions to the master role |
307 |
|
308 |
#. for all the in-progress jobs, mark them as failed, with |
309 |
reason unknown or something similar (master failed, etc.) |
310 |
|
311 |
Since due to exceptional conditions we could have a situation in which |
312 |
no node can become the master due to inconsistent data, we will have |
313 |
an override switch for the master daemon startup that will assume the |
314 |
current node has the right data and will replicate all the |
315 |
configuration files to the other nodes. |
316 |
|
317 |
**Note**: the above algorithm is by no means an election algorithm; it |
318 |
is a *confirmation* of the master role currently held by a node. |
319 |
|
320 |
Logging |
321 |
+++++++ |
322 |
|
323 |
The logging system will be switched completely to the standard python |
324 |
logging module; currently it's logging-based, but exposes a different |
325 |
API, which is just overhead. As such, the code will be switched over |
326 |
to standard logging calls, and only the setup will be custom. |
327 |
|
328 |
With this change, we will remove the separate debug/info/error logs, |
329 |
and instead have always one logfile per daemon model: |
330 |
|
331 |
- master-daemon.log for the master daemon |
332 |
- node-daemon.log for the node daemon (this is the same as in 1.2) |
333 |
- rapi-daemon.log for the RAPI daemon logs |
334 |
- rapi-access.log, an additional log file for the RAPI that will be |
335 |
in the standard HTTP log format for possible parsing by other tools |
336 |
|
337 |
Since the :term:`watcher` will only submit jobs to the master for |
338 |
startup of the instances, its log file will contain less information |
339 |
than before, mainly that it will start the instance, but not the |
340 |
results. |
341 |
|
342 |
Node daemon changes |
343 |
+++++++++++++++++++ |
344 |
|
345 |
The only change to the node daemon is that, since we need better |
346 |
concurrency, we don't process the inter-node RPC calls in the node |
347 |
daemon itself, but we fork and process each request in a separate |
348 |
child. |
349 |
|
350 |
Since we don't have many calls, and we only fork (not exec), the |
351 |
overhead should be minimal. |
352 |
|
353 |
Caveats |
354 |
+++++++ |
355 |
|
356 |
A discussed alternative is to keep the current individual processes |
357 |
touching the cluster configuration model. The reasons we have not |
358 |
chosen this approach is: |
359 |
|
360 |
- the speed of reading and unserializing the cluster state |
361 |
today is not small enough that we can ignore it; the addition of |
362 |
the job queue will make the startup cost even higher. While this |
363 |
runtime cost is low, it can be on the order of a few seconds on |
364 |
bigger clusters, which for very quick commands is comparable to |
365 |
the actual duration of the computation itself |
366 |
|
367 |
- individual commands would make it harder to implement a |
368 |
fire-and-forget job request, along the lines "start this |
369 |
instance but do not wait for it to finish"; it would require a |
370 |
model of backgrounding the operation and other things that are |
371 |
much better served by a daemon-based model |
372 |
|
373 |
Another area of discussion is moving away from Twisted in this new |
374 |
implementation. While Twisted has its advantages, there are also many |
375 |
disadvantages to using it: |
376 |
|
377 |
- first and foremost, it's not a library, but a framework; thus, if |
378 |
you use twisted, all the code needs to be 'twiste-ized' and written |
379 |
in an asynchronous manner, using deferreds; while this method works, |
380 |
it's not a common way to code and it requires that the entire process |
381 |
workflow is based around a single *reactor* (Twisted name for a main |
382 |
loop) |
383 |
- the more advanced granular locking that we want to implement would |
384 |
require, if written in the async-manner, deep integration with the |
385 |
Twisted stack, to such an extend that business-logic is inseparable |
386 |
from the protocol coding; we felt that this is an unreasonable |
387 |
request, and that a good protocol library should allow complete |
388 |
separation of low-level protocol calls and business logic; by |
389 |
comparison, the threaded approach combined with HTTPs protocol |
390 |
required (for the first iteration) absolutely no changes from the 1.2 |
391 |
code, and later changes for optimizing the inter-node RPC calls |
392 |
required just syntactic changes (e.g. ``rpc.call_...`` to |
393 |
``self.rpc.call_...``) |
394 |
|
395 |
Another issue is with the Twisted API stability - during the Ganeti |
396 |
1.x lifetime, we had to to implement many times workarounds to changes |
397 |
in the Twisted version, so that for example 1.2 is able to use both |
398 |
Twisted 2.x and 8.x. |
399 |
|
400 |
In the end, since we already had an HTTP server library for the RAPI, |
401 |
we just reused that for inter-node communication. |
402 |
|
403 |
|
404 |
Granular locking |
405 |
~~~~~~~~~~~~~~~~ |
406 |
|
407 |
We want to make sure that multiple operations can run in parallel on a |
408 |
Ganeti Cluster. In order for this to happen we need to make sure |
409 |
concurrently run operations don't step on each other toes and break the |
410 |
cluster. |
411 |
|
412 |
This design addresses how we are going to deal with locking so that: |
413 |
|
414 |
- we preserve data coherency |
415 |
- we prevent deadlocks |
416 |
- we prevent job starvation |
417 |
|
418 |
Reaching the maximum possible parallelism is a Non-Goal. We have |
419 |
identified a set of operations that are currently bottlenecks and need |
420 |
to be parallelised and have worked on those. In the future it will be |
421 |
possible to address other needs, thus making the cluster more and more |
422 |
parallel one step at a time. |
423 |
|
424 |
This section only talks about parallelising Ganeti level operations, aka |
425 |
Logical Units, and the locking needed for that. Any other |
426 |
synchronization lock needed internally by the code is outside its scope. |
427 |
|
428 |
Library details |
429 |
+++++++++++++++ |
430 |
|
431 |
The proposed library has these features: |
432 |
|
433 |
- internally managing all the locks, making the implementation |
434 |
transparent from their usage |
435 |
- automatically grabbing multiple locks in the right order (avoid |
436 |
deadlock) |
437 |
- ability to transparently handle conversion to more granularity |
438 |
- support asynchronous operation (future goal) |
439 |
|
440 |
Locking will be valid only on the master node and will not be a |
441 |
distributed operation. Therefore, in case of master failure, the |
442 |
operations currently running will be aborted and the locks will be |
443 |
lost; it remains to the administrator to cleanup (if needed) the |
444 |
operation result (e.g. make sure an instance is either installed |
445 |
correctly or removed). |
446 |
|
447 |
A corollary of this is that a master-failover operation with both |
448 |
masters alive needs to happen while no operations are running, and |
449 |
therefore no locks are held. |
450 |
|
451 |
All the locks will be represented by objects (like |
452 |
``lockings.SharedLock``), and the individual locks for each object |
453 |
will be created at initialisation time, from the config file. |
454 |
|
455 |
The API will have a way to grab one or more than one locks at the same |
456 |
time. Any attempt to grab a lock while already holding one in the wrong |
457 |
order will be checked for, and fail. |
458 |
|
459 |
|
460 |
The Locks |
461 |
+++++++++ |
462 |
|
463 |
At the first stage we have decided to provide the following locks: |
464 |
|
465 |
- One "config file" lock |
466 |
- One lock per node in the cluster |
467 |
- One lock per instance in the cluster |
468 |
|
469 |
All the instance locks will need to be taken before the node locks, and |
470 |
the node locks before the config lock. Locks will need to be acquired at |
471 |
the same time for multiple instances and nodes, and internal ordering |
472 |
will be dealt within the locking library, which, for simplicity, will |
473 |
just use alphabetical order. |
474 |
|
475 |
Each lock has the following three possible statuses: |
476 |
|
477 |
- unlocked (anyone can grab the lock) |
478 |
- shared (anyone can grab/have the lock but only in shared mode) |
479 |
- exclusive (no one else can grab/have the lock) |
480 |
|
481 |
Handling conversion to more granularity |
482 |
+++++++++++++++++++++++++++++++++++++++ |
483 |
|
484 |
In order to convert to a more granular approach transparently each time |
485 |
we split a lock into more we'll create a "metalock", which will depend |
486 |
on those sub-locks and live for the time necessary for all the code to |
487 |
convert (or forever, in some conditions). When a metalock exists all |
488 |
converted code must acquire it in shared mode, so it can run |
489 |
concurrently, but still be exclusive with old code, which acquires it |
490 |
exclusively. |
491 |
|
492 |
In the beginning the only such lock will be what replaces the current |
493 |
"command" lock, and will acquire all the locks in the system, before |
494 |
proceeding. This lock will be called the "Big Ganeti Lock" because |
495 |
holding that one will avoid any other concurrent Ganeti operations. |
496 |
|
497 |
We might also want to devise more metalocks (eg. all nodes, all |
498 |
nodes+config) in order to make it easier for some parts of the code to |
499 |
acquire what it needs without specifying it explicitly. |
500 |
|
501 |
In the future things like the node locks could become metalocks, should |
502 |
we decide to split them into an even more fine grained approach, but |
503 |
this will probably be only after the first 2.0 version has been |
504 |
released. |
505 |
|
506 |
Adding/Removing locks |
507 |
+++++++++++++++++++++ |
508 |
|
509 |
When a new instance or a new node is created an associated lock must be |
510 |
added to the list. The relevant code will need to inform the locking |
511 |
library of such a change. |
512 |
|
513 |
This needs to be compatible with every other lock in the system, |
514 |
especially metalocks that guarantee to grab sets of resources without |
515 |
specifying them explicitly. The implementation of this will be handled |
516 |
in the locking library itself. |
517 |
|
518 |
When instances or nodes disappear from the cluster the relevant locks |
519 |
must be removed. This is easier than adding new elements, as the code |
520 |
which removes them must own them exclusively already, and thus deals |
521 |
with metalocks exactly as normal code acquiring those locks. Any |
522 |
operation queuing on a removed lock will fail after its removal. |
523 |
|
524 |
Asynchronous operations |
525 |
+++++++++++++++++++++++ |
526 |
|
527 |
For the first version the locking library will only export synchronous |
528 |
operations, which will block till the needed lock are held, and only |
529 |
fail if the request is impossible or somehow erroneous. |
530 |
|
531 |
In the future we may want to implement different types of asynchronous |
532 |
operations such as: |
533 |
|
534 |
- try to acquire this lock set and fail if not possible |
535 |
- try to acquire one of these lock sets and return the first one you |
536 |
were able to get (or after a timeout) (select/poll like) |
537 |
|
538 |
These operations can be used to prioritize operations based on available |
539 |
locks, rather than making them just blindly queue for acquiring them. |
540 |
The inherent risk, though, is that any code using the first operation, |
541 |
or setting a timeout for the second one, is susceptible to starvation |
542 |
and thus may never be able to get the required locks and complete |
543 |
certain tasks. Considering this providing/using these operations should |
544 |
not be among our first priorities. |
545 |
|
546 |
Locking granularity |
547 |
+++++++++++++++++++ |
548 |
|
549 |
For the first version of this code we'll convert each Logical Unit to |
550 |
acquire/release the locks it needs, so locking will be at the Logical |
551 |
Unit level. In the future we may want to split logical units in |
552 |
independent "tasklets" with their own locking requirements. A different |
553 |
design doc (or mini design doc) will cover the move from Logical Units |
554 |
to tasklets. |
555 |
|
556 |
Code examples |
557 |
+++++++++++++ |
558 |
|
559 |
In general when acquiring locks we should use a code path equivalent |
560 |
to:: |
561 |
|
562 |
lock.acquire() |
563 |
try: |
564 |
... |
565 |
# other code |
566 |
finally: |
567 |
lock.release() |
568 |
|
569 |
This makes sure we release all locks, and avoid possible deadlocks. Of |
570 |
course extra care must be used not to leave, if possible locked |
571 |
structures in an unusable state. Note that with Python 2.5 a simpler |
572 |
syntax will be possible, but we want to keep compatibility with Python |
573 |
2.4 so the new constructs should not be used. |
574 |
|
575 |
In order to avoid this extra indentation and code changes everywhere in |
576 |
the Logical Units code, we decided to allow LUs to declare locks, and |
577 |
then execute their code with their locks acquired. In the new world LUs |
578 |
are called like this:: |
579 |
|
580 |
# user passed names are expanded to the internal lock/resource name, |
581 |
# then known needed locks are declared |
582 |
lu.ExpandNames() |
583 |
... some locking/adding of locks may happen ... |
584 |
# late declaration of locks for one level: this is useful because sometimes |
585 |
# we can't know which resource we need before locking the previous level |
586 |
lu.DeclareLocks() # for each level (cluster, instance, node) |
587 |
... more locking/adding of locks can happen ... |
588 |
# these functions are called with the proper locks held |
589 |
lu.CheckPrereq() |
590 |
lu.Exec() |
591 |
... locks declared for removal are removed, all acquired locks released ... |
592 |
|
593 |
The Processor and the LogicalUnit class will contain exact documentation |
594 |
on how locks are supposed to be declared. |
595 |
|
596 |
Caveats |
597 |
+++++++ |
598 |
|
599 |
This library will provide an easy upgrade path to bring all the code to |
600 |
granular locking without breaking everything, and it will also guarantee |
601 |
against a lot of common errors. Code switching from the old "lock |
602 |
everything" lock to the new system, though, needs to be carefully |
603 |
scrutinised to be sure it is really acquiring all the necessary locks, |
604 |
and none has been overlooked or forgotten. |
605 |
|
606 |
The code can contain other locks outside of this library, to synchronise |
607 |
other threaded code (eg for the job queue) but in general these should |
608 |
be leaf locks or carefully structured non-leaf ones, to avoid deadlock |
609 |
race conditions. |
610 |
|
611 |
|
612 |
Job Queue |
613 |
~~~~~~~~~ |
614 |
|
615 |
Granular locking is not enough to speed up operations, we also need a |
616 |
queue to store these and to be able to process as many as possible in |
617 |
parallel. |
618 |
|
619 |
A Ganeti job will consist of multiple ``OpCodes`` which are the basic |
620 |
element of operation in Ganeti 1.2 (and will remain as such). Most |
621 |
command-level commands are equivalent to one OpCode, or in some cases |
622 |
to a sequence of opcodes, all of the same type (e.g. evacuating a node |
623 |
will generate N opcodes of type replace disks). |
624 |
|
625 |
|
626 |
Job executionโโLife of a Ganeti jobโ |
627 |
++++++++++++++++++++++++++++++++++++ |
628 |
|
629 |
#. Job gets submitted by the client. A new job identifier is generated |
630 |
and assigned to the job. The job is then automatically replicated |
631 |
[#replic]_ to all nodes in the cluster. The identifier is returned to |
632 |
the client. |
633 |
#. A pool of worker threads waits for new jobs. If all are busy, the job |
634 |
has to wait and the first worker finishing its work will grab it. |
635 |
Otherwise any of the waiting threads will pick up the new job. |
636 |
#. Client waits for job status updates by calling a waiting RPC |
637 |
function. Log message may be shown to the user. Until the job is |
638 |
started, it can also be canceled. |
639 |
#. As soon as the job is finished, its final result and status can be |
640 |
retrieved from the server. |
641 |
#. If the client archives the job, it gets moved to a history directory. |
642 |
There will be a method to archive all jobs older than a a given age. |
643 |
|
644 |
.. [#replic] We need replication in order to maintain the consistency |
645 |
across all nodes in the system; the master node only differs in the |
646 |
fact that now it is running the master daemon, but it if fails and we |
647 |
do a master failover, the jobs are still visible on the new master |
648 |
(though marked as failed). |
649 |
|
650 |
Failures to replicate a job to other nodes will be only flagged as |
651 |
errors in the master daemon log if more than half of the nodes failed, |
652 |
otherwise we ignore the failure, and rely on the fact that the next |
653 |
update (for still running jobs) will retry the update. For finished |
654 |
jobs, it is less of a problem. |
655 |
|
656 |
Future improvements will look into checking the consistency of the job |
657 |
list and jobs themselves at master daemon startup. |
658 |
|
659 |
|
660 |
Job storage |
661 |
+++++++++++ |
662 |
|
663 |
Jobs are stored in the filesystem as individual files, serialized |
664 |
using JSON (standard serialization mechanism in Ganeti). |
665 |
|
666 |
The choice of storing each job in its own file was made because: |
667 |
|
668 |
- a file can be atomically replaced |
669 |
- a file can easily be replicated to other nodes |
670 |
- checking consistency across nodes can be implemented very easily, |
671 |
since all job files should be (at a given moment in time) identical |
672 |
|
673 |
The other possible choices that were discussed and discounted were: |
674 |
|
675 |
- single big file with all job data: not feasible due to difficult |
676 |
updates |
677 |
- in-process databases: hard to replicate the entire database to the |
678 |
other nodes, and replicating individual operations does not mean wee |
679 |
keep consistency |
680 |
|
681 |
|
682 |
Queue structure |
683 |
+++++++++++++++ |
684 |
|
685 |
All file operations have to be done atomically by writing to a temporary |
686 |
file and subsequent renaming. Except for log messages, every change in a |
687 |
job is stored and replicated to other nodes. |
688 |
|
689 |
:: |
690 |
|
691 |
/var/lib/ganeti/queue/ |
692 |
job-1 (JSON encoded job description and status) |
693 |
[โฆ] |
694 |
job-37 |
695 |
job-38 |
696 |
job-39 |
697 |
lock (Queue managing process opens this file in exclusive mode) |
698 |
serial (Last job ID used) |
699 |
version (Queue format version) |
700 |
|
701 |
|
702 |
Locking |
703 |
+++++++ |
704 |
|
705 |
Locking in the job queue is a complicated topic. It is called from more |
706 |
than one thread and must be thread-safe. For simplicity, a single lock |
707 |
is used for the whole job queue. |
708 |
|
709 |
A more detailed description can be found in doc/locking.rst. |
710 |
|
711 |
|
712 |
Internal RPC |
713 |
++++++++++++ |
714 |
|
715 |
RPC calls available between Ganeti master and node daemons: |
716 |
|
717 |
jobqueue_update(file_name, content) |
718 |
Writes a file in the job queue directory. |
719 |
jobqueue_purge() |
720 |
Cleans the job queue directory completely, including archived job. |
721 |
jobqueue_rename(old, new) |
722 |
Renames a file in the job queue directory. |
723 |
|
724 |
|
725 |
Client RPC |
726 |
++++++++++ |
727 |
|
728 |
RPC between Ganeti clients and the Ganeti master daemon supports the |
729 |
following operations: |
730 |
|
731 |
SubmitJob(ops) |
732 |
Submits a list of opcodes and returns the job identifier. The |
733 |
identifier is guaranteed to be unique during the lifetime of a |
734 |
cluster. |
735 |
WaitForJobChange(job_id, fields, [โฆ], timeout) |
736 |
This function waits until a job changes or a timeout expires. The |
737 |
condition for when a job changed is defined by the fields passed and |
738 |
the last log message received. |
739 |
QueryJobs(job_ids, fields) |
740 |
Returns field values for the job identifiers passed. |
741 |
CancelJob(job_id) |
742 |
Cancels the job specified by identifier. This operation may fail if |
743 |
the job is already running, canceled or finished. |
744 |
ArchiveJob(job_id) |
745 |
Moves a job into the โฆ/archive/ directory. This operation will fail if |
746 |
the job has not been canceled or finished. |
747 |
|
748 |
|
749 |
Job and opcode status |
750 |
+++++++++++++++++++++ |
751 |
|
752 |
Each job and each opcode has, at any time, one of the following states: |
753 |
|
754 |
Queued |
755 |
The job/opcode was submitted, but did not yet start. |
756 |
Waiting |
757 |
The job/opcode is waiting for a lock to proceed. |
758 |
Running |
759 |
The job/opcode is running. |
760 |
Canceled |
761 |
The job/opcode was canceled before it started. |
762 |
Success |
763 |
The job/opcode ran and finished successfully. |
764 |
Error |
765 |
The job/opcode was aborted with an error. |
766 |
|
767 |
If the master is aborted while a job is running, the job will be set to |
768 |
the Error status once the master started again. |
769 |
|
770 |
|
771 |
History |
772 |
+++++++ |
773 |
|
774 |
Archived jobs are kept in a separate directory, |
775 |
``/var/lib/ganeti/queue/archive/``. This is done in order to speed up |
776 |
the queue handling: by default, the jobs in the archive are not |
777 |
touched by any functions. Only the current (unarchived) jobs are |
778 |
parsed, loaded, and verified (if implemented) by the master daemon. |
779 |
|
780 |
|
781 |
Ganeti updates |
782 |
++++++++++++++ |
783 |
|
784 |
The queue has to be completely empty for Ganeti updates with changes |
785 |
in the job queue structure. In order to allow this, there will be a |
786 |
way to prevent new jobs entering the queue. |
787 |
|
788 |
|
789 |
Object parameters |
790 |
~~~~~~~~~~~~~~~~~ |
791 |
|
792 |
Across all cluster configuration data, we have multiple classes of |
793 |
parameters: |
794 |
|
795 |
A. cluster-wide parameters (e.g. name of the cluster, the master); |
796 |
these are the ones that we have today, and are unchanged from the |
797 |
current model |
798 |
|
799 |
#. node parameters |
800 |
|
801 |
#. instance specific parameters, e.g. the name of disks (LV), that |
802 |
cannot be shared with other instances |
803 |
|
804 |
#. instance parameters, that are or can be the same for many |
805 |
instances, but are not hypervisor related; e.g. the number of VCPUs, |
806 |
or the size of memory |
807 |
|
808 |
#. instance parameters that are hypervisor specific (e.g. kernel_path |
809 |
or PAE mode) |
810 |
|
811 |
|
812 |
The following definitions for instance parameters will be used below: |
813 |
|
814 |
:hypervisor parameter: |
815 |
a hypervisor parameter (or hypervisor specific parameter) is defined |
816 |
as a parameter that is interpreted by the hypervisor support code in |
817 |
Ganeti and usually is specific to a particular hypervisor (like the |
818 |
kernel path for :term:`PVM` which makes no sense for :term:`HVM`). |
819 |
|
820 |
:backend parameter: |
821 |
a backend parameter is defined as an instance parameter that can be |
822 |
shared among a list of instances, and is either generic enough not |
823 |
to be tied to a given hypervisor or cannot influence at all the |
824 |
hypervisor behaviour. |
825 |
|
826 |
For example: memory, vcpus, auto_balance |
827 |
|
828 |
All these parameters will be encoded into constants.py with the prefix |
829 |
"BE\_" and the whole list of parameters will exist in the set |
830 |
"BES_PARAMETERS" |
831 |
|
832 |
:proper parameter: |
833 |
a parameter whose value is unique to the instance (e.g. the name of a |
834 |
LV, or the MAC of a NIC) |
835 |
|
836 |
As a general rule, for all kind of parameters, โNoneโ (or in |
837 |
JSON-speak, โnilโ) will no longer be a valid value for a parameter. As |
838 |
such, only non-default parameters will be saved as part of objects in |
839 |
the serialization step, reducing the size of the serialized format. |
840 |
|
841 |
Cluster parameters |
842 |
++++++++++++++++++ |
843 |
|
844 |
Cluster parameters remain as today, attributes at the top level of the |
845 |
Cluster object. In addition, two new attributes at this level will |
846 |
hold defaults for the instances: |
847 |
|
848 |
- hvparams, a dictionary indexed by hypervisor type, holding default |
849 |
values for hypervisor parameters that are not defined/overridden by |
850 |
the instances of this hypervisor type |
851 |
|
852 |
- beparams, a dictionary holding (for 2.0) a single element 'default', |
853 |
which holds the default value for backend parameters |
854 |
|
855 |
Node parameters |
856 |
+++++++++++++++ |
857 |
|
858 |
Node-related parameters are very few, and we will continue using the |
859 |
same model for these as previously (attributes on the Node object). |
860 |
|
861 |
There are three new node flags, described in a separate section "node |
862 |
flags" below. |
863 |
|
864 |
Instance parameters |
865 |
+++++++++++++++++++ |
866 |
|
867 |
As described before, the instance parameters are split in three: |
868 |
instance proper parameters, unique to each instance, instance |
869 |
hypervisor parameters and instance backend parameters. |
870 |
|
871 |
The โhvparamsโ and โbeparamsโ are kept in two dictionaries at instance |
872 |
level. Only non-default parameters are stored (but once customized, a |
873 |
parameter will be kept, even with the same value as the default one, |
874 |
until reset). |
875 |
|
876 |
The names for hypervisor parameters in the instance.hvparams subtree |
877 |
should be choosen as generic as possible, especially if specific |
878 |
parameters could conceivably be useful for more than one hypervisor, |
879 |
e.g. ``instance.hvparams.vnc_console_port`` instead of using both |
880 |
``instance.hvparams.hvm_vnc_console_port`` and |
881 |
``instance.hvparams.kvm_vnc_console_port``. |
882 |
|
883 |
There are some special cases related to disks and NICs (for example): |
884 |
a disk has both Ganeti-related parameters (e.g. the name of the LV) |
885 |
and hypervisor-related parameters (how the disk is presented to/named |
886 |
in the instance). The former parameters remain as proper-instance |
887 |
parameters, while the latter value are migrated to the hvparams |
888 |
structure. In 2.0, we will have only globally-per-instance such |
889 |
hypervisor parameters, and not per-disk ones (e.g. all NICs will be |
890 |
exported as of the same type). |
891 |
|
892 |
Starting from the 1.2 list of instance parameters, here is how they |
893 |
will be mapped to the three classes of parameters: |
894 |
|
895 |
- name (P) |
896 |
- primary_node (P) |
897 |
- os (P) |
898 |
- hypervisor (P) |
899 |
- status (P) |
900 |
- memory (BE) |
901 |
- vcpus (BE) |
902 |
- nics (P) |
903 |
- disks (P) |
904 |
- disk_template (P) |
905 |
- network_port (P) |
906 |
- kernel_path (HV) |
907 |
- initrd_path (HV) |
908 |
- hvm_boot_order (HV) |
909 |
- hvm_acpi (HV) |
910 |
- hvm_pae (HV) |
911 |
- hvm_cdrom_image_path (HV) |
912 |
- hvm_nic_type (HV) |
913 |
- hvm_disk_type (HV) |
914 |
- vnc_bind_address (HV) |
915 |
- serial_no (P) |
916 |
|
917 |
|
918 |
Parameter validation |
919 |
++++++++++++++++++++ |
920 |
|
921 |
To support the new cluster parameter design, additional features will |
922 |
be required from the hypervisor support implementations in Ganeti. |
923 |
|
924 |
The hypervisor support implementation API will be extended with the |
925 |
following features: |
926 |
|
927 |
:PARAMETERS: class-level attribute holding the list of valid parameters |
928 |
for this hypervisor |
929 |
:CheckParamSyntax(hvparams): checks that the given parameters are |
930 |
valid (as in the names are valid) for this hypervisor; usually just |
931 |
comparing ``hvparams.keys()`` and ``cls.PARAMETERS``; this is a class |
932 |
method that can be called from within master code (i.e. cmdlib) and |
933 |
should be safe to do so |
934 |
:ValidateParameters(hvparams): verifies the values of the provided |
935 |
parameters against this hypervisor; this is a method that will be |
936 |
called on the target node, from backend.py code, and as such can |
937 |
make node-specific checks (e.g. kernel_path checking) |
938 |
|
939 |
Default value application |
940 |
+++++++++++++++++++++++++ |
941 |
|
942 |
The application of defaults to an instance is done in the Cluster |
943 |
object, via two new methods as follows: |
944 |
|
945 |
- ``Cluster.FillHV(instance)``, returns 'filled' hvparams dict, based on |
946 |
instance's hvparams and cluster's ``hvparams[instance.hypervisor]`` |
947 |
|
948 |
- ``Cluster.FillBE(instance, be_type="default")``, which returns the |
949 |
beparams dict, based on the instance and cluster beparams |
950 |
|
951 |
The FillHV/BE transformations will be used, for example, in the |
952 |
RpcRunner when sending an instance for activation/stop, and the sent |
953 |
instance hvparams/beparams will have the final value (noded code doesn't |
954 |
know about defaults). |
955 |
|
956 |
LU code will need to self-call the transformation, if needed. |
957 |
|
958 |
Opcode changes |
959 |
++++++++++++++ |
960 |
|
961 |
The parameter changes will have impact on the OpCodes, especially on |
962 |
the following ones: |
963 |
|
964 |
- ``OpCreateInstance``, where the new hv and be parameters will be sent |
965 |
as dictionaries; note that all hv and be parameters are now optional, |
966 |
as the values can be instead taken from the cluster |
967 |
- ``OpQueryInstances``, where we have to be able to query these new |
968 |
parameters; the syntax for names will be ``hvparam/$NAME`` and |
969 |
``beparam/$NAME`` for querying an individual parameter out of one |
970 |
dictionary, and ``hvparams``, respectively ``beparams``, for the whole |
971 |
dictionaries |
972 |
- ``OpModifyInstance``, where the the modified parameters are sent as |
973 |
dictionaries |
974 |
|
975 |
Additionally, we will need new OpCodes to modify the cluster-level |
976 |
defaults for the be/hv sets of parameters. |
977 |
|
978 |
Caveats |
979 |
+++++++ |
980 |
|
981 |
One problem that might appear is that our classification is not |
982 |
complete or not good enough, and we'll need to change this model. As |
983 |
the last resort, we will need to rollback and keep 1.2 style. |
984 |
|
985 |
Another problem is that classification of one parameter is unclear |
986 |
(e.g. ``network_port``, is this BE or HV?); in this case we'll take |
987 |
the risk of having to move parameters later between classes. |
988 |
|
989 |
Security |
990 |
++++++++ |
991 |
|
992 |
The only security issue that we foresee is if some new parameters will |
993 |
have sensitive value. If so, we will need to have a way to export the |
994 |
config data while purging the sensitive value. |
995 |
|
996 |
E.g. for the drbd shared secrets, we could export these with the |
997 |
values replaced by an empty string. |
998 |
|
999 |
Node flags |
1000 |
~~~~~~~~~~ |
1001 |
|
1002 |
Ganeti 2.0 adds three node flags that change the way nodes are handled |
1003 |
within Ganeti and the related infrastructure (iallocator interaction, |
1004 |
RAPI data export). |
1005 |
|
1006 |
*master candidate* flag |
1007 |
+++++++++++++++++++++++ |
1008 |
|
1009 |
Ganeti 2.0 allows more scalability in operation by introducing |
1010 |
parallelization. However, a new bottleneck is reached that is the |
1011 |
synchronization and replication of cluster configuration to all nodes |
1012 |
in the cluster. |
1013 |
|
1014 |
This breaks scalability as the speed of the replication decreases |
1015 |
roughly with the size of the nodes in the cluster. The goal of the |
1016 |
master candidate flag is to change this O(n) into O(1) with respect to |
1017 |
job and configuration data propagation. |
1018 |
|
1019 |
Only nodes having this flag set (let's call this set of nodes the |
1020 |
*candidate pool*) will have jobs and configuration data replicated. |
1021 |
|
1022 |
The cluster will have a new parameter (runtime changeable) called |
1023 |
``candidate_pool_size`` which represents the number of candidates the |
1024 |
cluster tries to maintain (preferably automatically). |
1025 |
|
1026 |
This will impact the cluster operations as follows: |
1027 |
|
1028 |
- jobs and config data will be replicated only to a fixed set of nodes |
1029 |
- master fail-over will only be possible to a node in the candidate pool |
1030 |
- cluster verify needs changing to account for these two roles |
1031 |
- external scripts will no longer have access to the configuration |
1032 |
file (this is not recommended anyway) |
1033 |
|
1034 |
|
1035 |
The caveats of this change are: |
1036 |
|
1037 |
- if all candidates are lost (completely), cluster configuration is |
1038 |
lost (but it should be backed up external to the cluster anyway) |
1039 |
|
1040 |
- failed nodes which are candidate must be dealt with properly, so |
1041 |
that we don't lose too many candidates at the same time; this will be |
1042 |
reported in cluster verify |
1043 |
|
1044 |
- the 'all equal' concept of ganeti is no longer true |
1045 |
|
1046 |
- the partial distribution of config data means that all nodes will |
1047 |
have to revert to ssconf files for master info (as in 1.2) |
1048 |
|
1049 |
Advantages: |
1050 |
|
1051 |
- speed on a 100+ nodes simulated cluster is greatly enhanced, even |
1052 |
for a simple operation; ``gnt-instance remove`` on a diskless instance |
1053 |
remove goes from ~9seconds to ~2 seconds |
1054 |
|
1055 |
- node failure of non-candidates will be less impacting on the cluster |
1056 |
|
1057 |
The default value for the candidate pool size will be set to 10 but |
1058 |
this can be changed at cluster creation and modified any time later. |
1059 |
|
1060 |
Testing on simulated big clusters with sequential and parallel jobs |
1061 |
show that this value (10) is a sweet-spot from performance and load |
1062 |
point of view. |
1063 |
|
1064 |
*offline* flag |
1065 |
++++++++++++++ |
1066 |
|
1067 |
In order to support better the situation in which nodes are offline |
1068 |
(e.g. for repair) without altering the cluster configuration, Ganeti |
1069 |
needs to be told and needs to properly handle this state for nodes. |
1070 |
|
1071 |
This will result in simpler procedures, and less mistakes, when the |
1072 |
amount of node failures is high on an absolute scale (either due to |
1073 |
high failure rate or simply big clusters). |
1074 |
|
1075 |
Nodes having this attribute set will not be contacted for inter-node |
1076 |
RPC calls, will not be master candidates, and will not be able to host |
1077 |
instances as primaries. |
1078 |
|
1079 |
Setting this attribute on a node: |
1080 |
|
1081 |
- will not be allowed if the node is the master |
1082 |
- will not be allowed if the node has primary instances |
1083 |
- will cause the node to be demoted from the master candidate role (if |
1084 |
it was), possibly causing another node to be promoted to that role |
1085 |
|
1086 |
This attribute will impact the cluster operations as follows: |
1087 |
|
1088 |
- querying these nodes for anything will fail instantly in the RPC |
1089 |
library, with a specific RPC error (RpcResult.offline == True) |
1090 |
|
1091 |
- they will be listed in the Other section of cluster verify |
1092 |
|
1093 |
The code is changed in the following ways: |
1094 |
|
1095 |
- RPC calls were be converted to skip such nodes: |
1096 |
|
1097 |
- RpcRunner-instance-based RPC calls are easy to convert |
1098 |
|
1099 |
- static/classmethod RPC calls are harder to convert, and were left |
1100 |
alone |
1101 |
|
1102 |
- the RPC results were unified so that this new result state (offline) |
1103 |
can be differentiated |
1104 |
|
1105 |
- master voting still queries in repair nodes, as we need to ensure |
1106 |
consistency in case the (wrong) masters have old data, and nodes have |
1107 |
come back from repairs |
1108 |
|
1109 |
Caveats: |
1110 |
|
1111 |
- some operation semantics are less clear (e.g. what to do on instance |
1112 |
start with offline secondary?); for now, these will just fail as if |
1113 |
the flag is not set (but faster) |
1114 |
- 2-node cluster with one node offline needs manual startup of the |
1115 |
master with a special flag to skip voting (as the master can't get a |
1116 |
quorum there) |
1117 |
|
1118 |
One of the advantages of implementing this flag is that it will allow |
1119 |
in the future automation tools to automatically put the node in |
1120 |
repairs and recover from this state, and the code (should/will) handle |
1121 |
this much better than just timing out. So, future possible |
1122 |
improvements (for later versions): |
1123 |
|
1124 |
- watcher will detect nodes which fail RPC calls, will attempt to ssh |
1125 |
to them, if failure will put them offline |
1126 |
- watcher will try to ssh and query the offline nodes, if successful |
1127 |
will take them off the repair list |
1128 |
|
1129 |
Alternatives considered: The RPC call model in 2.0 is, by default, |
1130 |
much nicer - errors are logged in the background, and job/opcode |
1131 |
execution is clearer, so we could simply not introduce this. However, |
1132 |
having this state will make both the codepaths clearer (offline |
1133 |
vs. temporary failure) and the operational model (it's not a node with |
1134 |
errors, but an offline node). |
1135 |
|
1136 |
|
1137 |
*drained* flag |
1138 |
++++++++++++++ |
1139 |
|
1140 |
Due to parallel execution of jobs in Ganeti 2.0, we could have the |
1141 |
following situation: |
1142 |
|
1143 |
- gnt-node migrate + failover is run |
1144 |
- gnt-node evacuate is run, which schedules a long-running 6-opcode |
1145 |
job for the node |
1146 |
- partway through, a new job comes in that runs an iallocator script, |
1147 |
which finds the above node as empty and a very good candidate |
1148 |
- gnt-node evacuate has finished, but now it has to be run again, to |
1149 |
clean the above instance(s) |
1150 |
|
1151 |
In order to prevent this situation, and to be able to get nodes into |
1152 |
proper offline status easily, a new *drained* flag was added to the |
1153 |
nodes. |
1154 |
|
1155 |
This flag (which actually means "is being, or was drained, and is |
1156 |
expected to go offline"), will prevent allocations on the node, but |
1157 |
otherwise all other operations (start/stop instance, query, etc.) are |
1158 |
working without any restrictions. |
1159 |
|
1160 |
Interaction between flags |
1161 |
+++++++++++++++++++++++++ |
1162 |
|
1163 |
While these flags are implemented as separate flags, they are |
1164 |
mutually-exclusive and are acting together with the master node role |
1165 |
as a single *node status* value. In other words, a flag is only in one |
1166 |
of these roles at a given time. The lack of any of these flags denote |
1167 |
a regular node. |
1168 |
|
1169 |
The current node status is visible in the ``gnt-cluster verify`` |
1170 |
output, and the individual flags can be examined via separate flags in |
1171 |
the ``gnt-node list`` output. |
1172 |
|
1173 |
These new flags will be exported in both the iallocator input message |
1174 |
and via RAPI, see the respective man pages for the exact names. |
1175 |
|
1176 |
Feature changes |
1177 |
--------------- |
1178 |
|
1179 |
The main feature-level changes will be: |
1180 |
|
1181 |
- a number of disk related changes |
1182 |
- removal of fixed two-disk, one-nic per instance limitation |
1183 |
|
1184 |
Disk handling changes |
1185 |
~~~~~~~~~~~~~~~~~~~~~ |
1186 |
|
1187 |
The storage options available in Ganeti 1.x were introduced based on |
1188 |
then-current software (first DRBD 0.7 then later DRBD 8) and the |
1189 |
estimated usage patters. However, experience has later shown that some |
1190 |
assumptions made initially are not true and that more flexibility is |
1191 |
needed. |
1192 |
|
1193 |
One main assumption made was that disk failures should be treated as |
1194 |
'rare' events, and that each of them needs to be manually handled in |
1195 |
order to ensure data safety; however, both these assumptions are false: |
1196 |
|
1197 |
- disk failures can be a common occurrence, based on usage patterns or |
1198 |
cluster size |
1199 |
- our disk setup is robust enough (referring to DRBD8 + LVM) that we |
1200 |
could automate more of the recovery |
1201 |
|
1202 |
Note that we still don't have fully-automated disk recovery as a goal, |
1203 |
but our goal is to reduce the manual work needed. |
1204 |
|
1205 |
As such, we plan the following main changes: |
1206 |
|
1207 |
- DRBD8 is much more flexible and stable than its previous version |
1208 |
(0.7), such that removing the support for the ``remote_raid1`` |
1209 |
template and focusing only on DRBD8 is easier |
1210 |
|
1211 |
- dynamic discovery of DRBD devices is not actually needed in a cluster |
1212 |
that where the DRBD namespace is controlled by Ganeti; switching to a |
1213 |
static assignment (done at either instance creation time or change |
1214 |
secondary time) will change the disk activation time from O(n) to |
1215 |
O(1), which on big clusters is a significant gain |
1216 |
|
1217 |
- remove the hard dependency on LVM (currently all available storage |
1218 |
types are ultimately backed by LVM volumes) by introducing file-based |
1219 |
storage |
1220 |
|
1221 |
Additionally, a number of smaller enhancements are also planned: |
1222 |
- support variable number of disks |
1223 |
- support read-only disks |
1224 |
|
1225 |
Future enhancements in the 2.x series, which do not require base design |
1226 |
changes, might include: |
1227 |
|
1228 |
- enhancement of the LVM allocation method in order to try to keep |
1229 |
all of an instance's virtual disks on the same physical |
1230 |
disks |
1231 |
|
1232 |
- add support for DRBD8 authentication at handshake time in |
1233 |
order to ensure each device connects to the correct peer |
1234 |
|
1235 |
- remove the restrictions on failover only to the secondary |
1236 |
which creates very strict rules on cluster allocation |
1237 |
|
1238 |
DRBD minor allocation |
1239 |
+++++++++++++++++++++ |
1240 |
|
1241 |
Currently, when trying to identify or activate a new DRBD (or MD) |
1242 |
device, the code scans all in-use devices in order to see if we find |
1243 |
one that looks similar to our parameters and is already in the desired |
1244 |
state or not. Since this needs external commands to be run, it is very |
1245 |
slow when more than a few devices are already present. |
1246 |
|
1247 |
Therefore, we will change the discovery model from dynamic to |
1248 |
static. When a new device is logically created (added to the |
1249 |
configuration) a free minor number is computed from the list of |
1250 |
devices that should exist on that node and assigned to that |
1251 |
device. |
1252 |
|
1253 |
At device activation, if the minor is already in use, we check if |
1254 |
it has our parameters; if not so, we just destroy the device (if |
1255 |
possible, otherwise we abort) and start it with our own |
1256 |
parameters. |
1257 |
|
1258 |
This means that we in effect take ownership of the minor space for |
1259 |
that device type; if there's a user-created DRBD minor, it will be |
1260 |
automatically removed. |
1261 |
|
1262 |
The change will have the effect of reducing the number of external |
1263 |
commands run per device from a constant number times the index of the |
1264 |
first free DRBD minor to just a constant number. |
1265 |
|
1266 |
Removal of obsolete device types (MD, DRBD7) |
1267 |
++++++++++++++++++++++++++++++++++++++++++++ |
1268 |
|
1269 |
We need to remove these device types because of two issues. First, |
1270 |
DRBD7 has bad failure modes in case of dual failures (both network and |
1271 |
disk - it cannot propagate the error up the device stack and instead |
1272 |
just panics. Second, due to the asymmetry between primary and |
1273 |
secondary in MD+DRBD mode, we cannot do live failover (not even if we |
1274 |
had MD+DRBD8). |
1275 |
|
1276 |
File-based storage support |
1277 |
++++++++++++++++++++++++++ |
1278 |
|
1279 |
Using files instead of logical volumes for instance storage would |
1280 |
allow us to get rid of the hard requirement for volume groups for |
1281 |
testing clusters and it would also allow usage of SAN storage to do |
1282 |
live failover taking advantage of this storage solution. |
1283 |
|
1284 |
Better LVM allocation |
1285 |
+++++++++++++++++++++ |
1286 |
|
1287 |
Currently, the LV to PV allocation mechanism is a very simple one: at |
1288 |
each new request for a logical volume, tell LVM to allocate the volume |
1289 |
in order based on the amount of free space. This is good for |
1290 |
simplicity and for keeping the usage equally spread over the available |
1291 |
physical disks, however it introduces a problem that an instance could |
1292 |
end up with its (currently) two drives on two physical disks, or |
1293 |
(worse) that the data and metadata for a DRBD device end up on |
1294 |
different drives. |
1295 |
|
1296 |
This is bad because it causes unneeded ``replace-disks`` operations in |
1297 |
case of a physical failure. |
1298 |
|
1299 |
The solution is to batch allocations for an instance and make the LVM |
1300 |
handling code try to allocate as close as possible all the storage of |
1301 |
one instance. We will still allow the logical volumes to spill over to |
1302 |
additional disks as needed. |
1303 |
|
1304 |
Note that this clustered allocation can only be attempted at initial |
1305 |
instance creation, or at change secondary node time. At add disk time, |
1306 |
or at replacing individual disks, it's not easy enough to compute the |
1307 |
current disk map so we'll not attempt the clustering. |
1308 |
|
1309 |
DRBD8 peer authentication at handshake |
1310 |
++++++++++++++++++++++++++++++++++++++ |
1311 |
|
1312 |
DRBD8 has a new feature that allow authentication of the peer at |
1313 |
connect time. We can use this to prevent connecting to the wrong peer |
1314 |
more that securing the connection. Even though we never had issues |
1315 |
with wrong connections, it would be good to implement this. |
1316 |
|
1317 |
|
1318 |
LVM self-repair (optional) |
1319 |
++++++++++++++++++++++++++ |
1320 |
|
1321 |
The complete failure of a physical disk is very tedious to |
1322 |
troubleshoot, mainly because of the many failure modes and the many |
1323 |
steps needed. We can safely automate some of the steps, more |
1324 |
specifically the ``vgreduce --removemissing`` using the following |
1325 |
method: |
1326 |
|
1327 |
#. check if all nodes have consistent volume groups |
1328 |
#. if yes, and previous status was yes, do nothing |
1329 |
#. if yes, and previous status was no, save status and restart |
1330 |
#. if no, and previous status was no, do nothing |
1331 |
#. if no, and previous status was yes: |
1332 |
#. if more than one node is inconsistent, do nothing |
1333 |
#. if only one node is inconsistent: |
1334 |
#. run ``vgreduce --removemissing`` |
1335 |
#. log this occurrence in the Ganeti log in a form that |
1336 |
can be used for monitoring |
1337 |
#. [FUTURE] run ``replace-disks`` for all |
1338 |
instances affected |
1339 |
|
1340 |
Failover to any node |
1341 |
++++++++++++++++++++ |
1342 |
|
1343 |
With a modified disk activation sequence, we can implement the |
1344 |
*failover to any* functionality, removing many of the layout |
1345 |
restrictions of a cluster: |
1346 |
|
1347 |
- the need to reserve memory on the current secondary: this gets reduced |
1348 |
to a must to reserve memory anywhere on the cluster |
1349 |
|
1350 |
- the need to first failover and then replace secondary for an |
1351 |
instance: with failover-to-any, we can directly failover to |
1352 |
another node, which also does the replace disks at the same |
1353 |
step |
1354 |
|
1355 |
In the following, we denote the current primary by P1, the current |
1356 |
secondary by S1, and the new primary and secondaries by P2 and S2. P2 |
1357 |
is fixed to the node the user chooses, but the choice of S2 can be |
1358 |
made between P1 and S1. This choice can be constrained, depending on |
1359 |
which of P1 and S1 has failed. |
1360 |
|
1361 |
- if P1 has failed, then S1 must become S2, and live migration is not |
1362 |
possible |
1363 |
- if S1 has failed, then P1 must become S2, and live migration could be |
1364 |
possible (in theory, but this is not a design goal for 2.0) |
1365 |
|
1366 |
The algorithm for performing the failover is straightforward: |
1367 |
|
1368 |
- verify that S2 (the node the user has chosen to keep as secondary) has |
1369 |
valid data (is consistent) |
1370 |
|
1371 |
- tear down the current DRBD association and setup a DRBD pairing |
1372 |
between P2 (P2 is indicated by the user) and S2; since P2 has no data, |
1373 |
it will start re-syncing from S2 |
1374 |
|
1375 |
- as soon as P2 is in state SyncTarget (i.e. after the resync has |
1376 |
started but before it has finished), we can promote it to primary role |
1377 |
(r/w) and start the instance on P2 |
1378 |
|
1379 |
- as soon as the P2?S2 sync has finished, we can remove |
1380 |
the old data on the old node that has not been chosen for |
1381 |
S2 |
1382 |
|
1383 |
Caveats: during the P2?S2 sync, a (non-transient) network error |
1384 |
will cause I/O errors on the instance, so (if a longer instance |
1385 |
downtime is acceptable) we can postpone the restart of the instance |
1386 |
until the resync is done. However, disk I/O errors on S2 will cause |
1387 |
data loss, since we don't have a good copy of the data anymore, so in |
1388 |
this case waiting for the sync to complete is not an option. As such, |
1389 |
it is recommended that this feature is used only in conjunction with |
1390 |
proper disk monitoring. |
1391 |
|
1392 |
|
1393 |
Live migration note: While failover-to-any is possible for all choices |
1394 |
of S2, migration-to-any is possible only if we keep P1 as S2. |
1395 |
|
1396 |
Caveats |
1397 |
+++++++ |
1398 |
|
1399 |
The dynamic device model, while more complex, has an advantage: it |
1400 |
will not reuse by mistake the DRBD device of another instance, since |
1401 |
it always looks for either our own or a free one. |
1402 |
|
1403 |
The static one, in contrast, will assume that given a minor number N, |
1404 |
it's ours and we can take over. This needs careful implementation such |
1405 |
that if the minor is in use, either we are able to cleanly shut it |
1406 |
down, or we abort the startup. Otherwise, it could be that we start |
1407 |
syncing between two instance's disks, causing data loss. |
1408 |
|
1409 |
|
1410 |
Variable number of disk/NICs per instance |
1411 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
1412 |
|
1413 |
Variable number of disks |
1414 |
++++++++++++++++++++++++ |
1415 |
|
1416 |
In order to support high-security scenarios (for example read-only sda |
1417 |
and read-write sdb), we need to make a fully flexibly disk |
1418 |
definition. This has less impact that it might look at first sight: |
1419 |
only the instance creation has hard coded number of disks, not the disk |
1420 |
handling code. The block device handling and most of the instance |
1421 |
handling code is already working with "the instance's disks" as |
1422 |
opposed to "the two disks of the instance", but some pieces are not |
1423 |
(e.g. import/export) and the code needs a review to ensure safety. |
1424 |
|
1425 |
The objective is to be able to specify the number of disks at |
1426 |
instance creation, and to be able to toggle from read-only to |
1427 |
read-write a disk afterward. |
1428 |
|
1429 |
Variable number of NICs |
1430 |
+++++++++++++++++++++++ |
1431 |
|
1432 |
Similar to the disk change, we need to allow multiple network |
1433 |
interfaces per instance. This will affect the internal code (some |
1434 |
function will have to stop assuming that ``instance.nics`` is a list |
1435 |
of length one), the OS API which currently can export/import only one |
1436 |
instance, and the command line interface. |
1437 |
|
1438 |
Interface changes |
1439 |
----------------- |
1440 |
|
1441 |
There are two areas of interface changes: API-level changes (the OS |
1442 |
interface and the RAPI interface) and the command line interface |
1443 |
changes. |
1444 |
|
1445 |
OS interface |
1446 |
~~~~~~~~~~~~ |
1447 |
|
1448 |
The current Ganeti OS interface, version 5, is tailored for Ganeti 1.2. |
1449 |
The interface is composed by a series of scripts which get called with |
1450 |
certain parameters to perform OS-dependent operations on the cluster. |
1451 |
The current scripts are: |
1452 |
|
1453 |
create |
1454 |
called when a new instance is added to the cluster |
1455 |
export |
1456 |
called to export an instance disk to a stream |
1457 |
import |
1458 |
called to import from a stream to a new instance |
1459 |
rename |
1460 |
called to perform the os-specific operations necessary for renaming an |
1461 |
instance |
1462 |
|
1463 |
Currently these scripts suffer from the limitations of Ganeti 1.2: for |
1464 |
example they accept exactly one block and one swap devices to operate |
1465 |
on, rather than any amount of generic block devices, they blindly assume |
1466 |
that an instance will have just one network interface to operate, they |
1467 |
can not be configured to optimise the instance for a particular |
1468 |
hypervisor. |
1469 |
|
1470 |
Since in Ganeti 2.0 we want to support multiple hypervisors, and a |
1471 |
non-fixed number of network and disks the OS interface need to change to |
1472 |
transmit the appropriate amount of information about an instance to its |
1473 |
managing operating system, when operating on it. Moreover since some old |
1474 |
assumptions usually used in OS scripts are no longer valid we need to |
1475 |
re-establish a common knowledge on what can be assumed and what cannot |
1476 |
be regarding Ganeti environment. |
1477 |
|
1478 |
|
1479 |
When designing the new OS API our priorities are: |
1480 |
- ease of use |
1481 |
- future extensibility |
1482 |
- ease of porting from the old API |
1483 |
- modularity |
1484 |
|
1485 |
As such we want to limit the number of scripts that must be written to |
1486 |
support an OS, and make it easy to share code between them by uniforming |
1487 |
their input. We also will leave the current script structure unchanged, |
1488 |
as far as we can, and make a few of the scripts (import, export and |
1489 |
rename) optional. Most information will be passed to the script through |
1490 |
environment variables, for ease of access and at the same time ease of |
1491 |
using only the information a script needs. |
1492 |
|
1493 |
|
1494 |
The Scripts |
1495 |
+++++++++++ |
1496 |
|
1497 |
As in Ganeti 1.2, every OS which wants to be installed in Ganeti needs |
1498 |
to support the following functionality, through scripts: |
1499 |
|
1500 |
create: |
1501 |
used to create a new instance running that OS. This script should |
1502 |
prepare the block devices, and install them so that the new OS can |
1503 |
boot under the specified hypervisor. |
1504 |
export (optional): |
1505 |
used to export an installed instance using the given OS to a format |
1506 |
which can be used to import it back into a new instance. |
1507 |
import (optional): |
1508 |
used to import an exported instance into a new one. This script is |
1509 |
similar to create, but the new instance should have the content of the |
1510 |
export, rather than contain a pristine installation. |
1511 |
rename (optional): |
1512 |
used to perform the internal OS-specific operations needed to rename |
1513 |
an instance. |
1514 |
|
1515 |
If any optional script is not implemented Ganeti will refuse to perform |
1516 |
the given operation on instances using the non-implementing OS. Of |
1517 |
course the create script is mandatory, and it doesn't make sense to |
1518 |
support the either the export or the import operation but not both. |
1519 |
|
1520 |
Incompatibilities with 1.2 |
1521 |
__________________________ |
1522 |
|
1523 |
We expect the following incompatibilities between the OS scripts for 1.2 |
1524 |
and the ones for 2.0: |
1525 |
|
1526 |
- Input parameters: in 1.2 those were passed on the command line, in 2.0 |
1527 |
we'll use environment variables, as there will be a lot more |
1528 |
information and not all OSes may care about all of it. |
1529 |
- Number of calls: export scripts will be called once for each device |
1530 |
the instance has, and import scripts once for every exported disk. |
1531 |
Imported instances will be forced to have a number of disks greater or |
1532 |
equal to the one of the export. |
1533 |
- Some scripts are not compulsory: if such a script is missing the |
1534 |
relevant operations will be forbidden for instances of that OS. This |
1535 |
makes it easier to distinguish between unsupported operations and |
1536 |
no-op ones (if any). |
1537 |
|
1538 |
|
1539 |
Input |
1540 |
_____ |
1541 |
|
1542 |
Rather than using command line flags, as they do now, scripts will |
1543 |
accept inputs from environment variables. We expect the following input |
1544 |
values: |
1545 |
|
1546 |
OS_API_VERSION |
1547 |
The version of the OS API that the following parameters comply with; |
1548 |
this is used so that in the future we could have OSes supporting |
1549 |
multiple versions and thus Ganeti send the proper version in this |
1550 |
parameter |
1551 |
INSTANCE_NAME |
1552 |
Name of the instance acted on |
1553 |
HYPERVISOR |
1554 |
The hypervisor the instance should run on (e.g. 'xen-pvm', 'xen-hvm', |
1555 |
'kvm') |
1556 |
DISK_COUNT |
1557 |
The number of disks this instance will have |
1558 |
NIC_COUNT |
1559 |
The number of NICs this instance will have |
1560 |
DISK_<N>_PATH |
1561 |
Path to the Nth disk. |
1562 |
DISK_<N>_ACCESS |
1563 |
W if read/write, R if read only. OS scripts are not supposed to touch |
1564 |
read-only disks, but will be passed them to know. |
1565 |
DISK_<N>_FRONTEND_TYPE |
1566 |
Type of the disk as seen by the instance. Can be 'scsi', 'ide', |
1567 |
'virtio' |
1568 |
DISK_<N>_BACKEND_TYPE |
1569 |
Type of the disk as seen from the node. Can be 'block', 'file:loop' or |
1570 |
'file:blktap' |
1571 |
NIC_<N>_MAC |
1572 |
Mac address for the Nth network interface |
1573 |
NIC_<N>_IP |
1574 |
Ip address for the Nth network interface, if available |
1575 |
NIC_<N>_BRIDGE |
1576 |
Node bridge the Nth network interface will be connected to |
1577 |
NIC_<N>_FRONTEND_TYPE |
1578 |
Type of the Nth NIC as seen by the instance. For example 'virtio', |
1579 |
'rtl8139', etc. |
1580 |
DEBUG_LEVEL |
1581 |
Whether more out should be produced, for debugging purposes. Currently |
1582 |
the only valid values are 0 and 1. |
1583 |
|
1584 |
These are only the basic variables we are thinking of now, but more |
1585 |
may come during the implementation and they will be documented in the |
1586 |
:manpage:`ganeti-os-api` man page. All these variables will be |
1587 |
available to all scripts. |
1588 |
|
1589 |
Some scripts will need a few more information to work. These will have |
1590 |
per-script variables, such as for example: |
1591 |
|
1592 |
OLD_INSTANCE_NAME |
1593 |
rename: the name the instance should be renamed from. |
1594 |
EXPORT_DEVICE |
1595 |
export: device to be exported, a snapshot of the actual device. The |
1596 |
data must be exported to stdout. |
1597 |
EXPORT_INDEX |
1598 |
export: sequential number of the instance device targeted. |
1599 |
IMPORT_DEVICE |
1600 |
import: device to send the data to, part of the new instance. The data |
1601 |
must be imported from stdin. |
1602 |
IMPORT_INDEX |
1603 |
import: sequential number of the instance device targeted. |
1604 |
|
1605 |
(Rationale for INSTANCE_NAME as an environment variable: the instance |
1606 |
name is always needed and we could pass it on the command line. On the |
1607 |
other hand, though, this would force scripts to both access the |
1608 |
environment and parse the command line, so we'll move it for |
1609 |
uniformity.) |
1610 |
|
1611 |
|
1612 |
Output/Behaviour |
1613 |
________________ |
1614 |
|
1615 |
As discussed scripts should only send user-targeted information to |
1616 |
stderr. The create and import scripts are supposed to format/initialise |
1617 |
the given block devices and install the correct instance data. The |
1618 |
export script is supposed to export instance data to stdout in a format |
1619 |
understandable by the the import script. The data will be compressed by |
1620 |
Ganeti, so no compression should be done. The rename script should only |
1621 |
modify the instance's knowledge of what its name is. |
1622 |
|
1623 |
Other declarative style features |
1624 |
++++++++++++++++++++++++++++++++ |
1625 |
|
1626 |
Similar to Ganeti 1.2, OS specifications will need to provide a |
1627 |
'ganeti_api_version' containing list of numbers matching the |
1628 |
version(s) of the API they implement. Ganeti itself will always be |
1629 |
compatible with one version of the API and may maintain backwards |
1630 |
compatibility if it's feasible to do so. The numbers are one-per-line, |
1631 |
so an OS supporting both version 5 and version 20 will have a file |
1632 |
containing two lines. This is different from Ganeti 1.2, which only |
1633 |
supported one version number. |
1634 |
|
1635 |
In addition to that an OS will be able to declare that it does support |
1636 |
only a subset of the Ganeti hypervisors, by declaring them in the |
1637 |
'hypervisors' file. |
1638 |
|
1639 |
|
1640 |
Caveats/Notes |
1641 |
+++++++++++++ |
1642 |
|
1643 |
We might want to have a "default" import/export behaviour that just |
1644 |
dumps all disks and restores them. This can save work as most systems |
1645 |
will just do this, while allowing flexibility for different systems. |
1646 |
|
1647 |
Environment variables are limited in size, but we expect that there will |
1648 |
be enough space to store the information we need. If we discover that |
1649 |
this is not the case we may want to go to a more complex API such as |
1650 |
storing those information on the filesystem and providing the OS script |
1651 |
with the path to a file where they are encoded in some format. |
1652 |
|
1653 |
|
1654 |
|
1655 |
Remote API changes |
1656 |
~~~~~~~~~~~~~~~~~~ |
1657 |
|
1658 |
The first Ganeti remote API (RAPI) was designed and deployed with the |
1659 |
Ganeti 1.2.5 release. That version provide read-only access to the |
1660 |
cluster state. Fully functional read-write API demands significant |
1661 |
internal changes which will be implemented in version 2.0. |
1662 |
|
1663 |
We decided to go with implementing the Ganeti RAPI in a RESTful way, |
1664 |
which is aligned with key features we looking. It is simple, |
1665 |
stateless, scalable and extensible paradigm of API implementation. As |
1666 |
transport it uses HTTP over SSL, and we are implementing it with JSON |
1667 |
encoding, but in a way it possible to extend and provide any other |
1668 |
one. |
1669 |
|
1670 |
Design |
1671 |
++++++ |
1672 |
|
1673 |
The Ganeti RAPI is implemented as independent daemon, running on the |
1674 |
same node with the same permission level as Ganeti master |
1675 |
daemon. Communication is done through the LUXI library to the master |
1676 |
daemon. In order to keep communication asynchronous RAPI processes two |
1677 |
types of client requests: |
1678 |
|
1679 |
- queries: server is able to answer immediately |
1680 |
- job submission: some time is required for a useful response |
1681 |
|
1682 |
In the query case requested data send back to client in the HTTP |
1683 |
response body. Typical examples of queries would be: list of nodes, |
1684 |
instances, cluster info, etc. |
1685 |
|
1686 |
In the case of job submission, the client receive a job ID, the |
1687 |
identifier which allows to query the job progress in the job queue |
1688 |
(see `Job Queue`_). |
1689 |
|
1690 |
Internally, each exported object has an version identifier, which is |
1691 |
used as a state identifier in the HTTP header E-Tag field for |
1692 |
requests/responses to avoid race conditions. |
1693 |
|
1694 |
|
1695 |
Resource representation |
1696 |
+++++++++++++++++++++++ |
1697 |
|
1698 |
The key difference of using REST instead of others API is that REST |
1699 |
requires separation of services via resources with unique URIs. Each |
1700 |
of them should have limited amount of state and support standard HTTP |
1701 |
methods: GET, POST, DELETE, PUT. |
1702 |
|
1703 |
For example in Ganeti's case we can have a set of URI: |
1704 |
|
1705 |
- ``/{clustername}/instances`` |
1706 |
- ``/{clustername}/instances/{instancename}`` |
1707 |
- ``/{clustername}/instances/{instancename}/tag`` |
1708 |
- ``/{clustername}/tag`` |
1709 |
|
1710 |
A GET request to ``/{clustername}/instances`` will return the list of |
1711 |
instances, a POST to ``/{clustername}/instances`` should create a new |
1712 |
instance, a DELETE ``/{clustername}/instances/{instancename}`` should |
1713 |
delete the instance, a GET ``/{clustername}/tag`` should return get |
1714 |
cluster tags. |
1715 |
|
1716 |
Each resource URI will have a version prefix. The resource IDs are to |
1717 |
be determined. |
1718 |
|
1719 |
Internal encoding might be JSON, XML, or any other. The JSON encoding |
1720 |
fits nicely in Ganeti RAPI needs. The client can request a specific |
1721 |
representation via the Accept field in the HTTP header. |
1722 |
|
1723 |
REST uses HTTP as its transport and application protocol for resource |
1724 |
access. The set of possible responses is a subset of standard HTTP |
1725 |
responses. |
1726 |
|
1727 |
The statelessness model provides additional reliability and |
1728 |
transparency to operations (e.g. only one request needs to be analyzed |
1729 |
to understand the in-progress operation, not a sequence of multiple |
1730 |
requests/responses). |
1731 |
|
1732 |
|
1733 |
Security |
1734 |
++++++++ |
1735 |
|
1736 |
With the write functionality security becomes a much bigger an issue. |
1737 |
The Ganeti RAPI uses basic HTTP authentication on top of an |
1738 |
SSL-secured connection to grant access to an exported resource. The |
1739 |
password is stored locally in an Apache-style ``.htpasswd`` file. Only |
1740 |
one level of privileges is supported. |
1741 |
|
1742 |
Caveats |
1743 |
+++++++ |
1744 |
|
1745 |
The model detailed above for job submission requires the client to |
1746 |
poll periodically for updates to the job; an alternative would be to |
1747 |
allow the client to request a callback, or a 'wait for updates' call. |
1748 |
|
1749 |
The callback model was not considered due to the following two issues: |
1750 |
|
1751 |
- callbacks would require a new model of allowed callback URLs, |
1752 |
together with a method of managing these |
1753 |
- callbacks only work when the client and the master are in the same |
1754 |
security domain, and they fail in the other cases (e.g. when there is |
1755 |
a firewall between the client and the RAPI daemon that only allows |
1756 |
client-to-RAPI calls, which is usual in DMZ cases) |
1757 |
|
1758 |
The 'wait for updates' method is not suited to the HTTP protocol, |
1759 |
where requests are supposed to be short-lived. |
1760 |
|
1761 |
Command line changes |
1762 |
~~~~~~~~~~~~~~~~~~~~ |
1763 |
|
1764 |
Ganeti 2.0 introduces several new features as well as new ways to |
1765 |
handle instance resources like disks or network interfaces. This |
1766 |
requires some noticeable changes in the way command line arguments are |
1767 |
handled. |
1768 |
|
1769 |
- extend and modify command line syntax to support new features |
1770 |
- ensure consistent patterns in command line arguments to reduce |
1771 |
cognitive load |
1772 |
|
1773 |
The design changes that require these changes are, in no particular |
1774 |
order: |
1775 |
|
1776 |
- flexible instance disk handling: support a variable number of disks |
1777 |
with varying properties per instance, |
1778 |
- flexible instance network interface handling: support a variable |
1779 |
number of network interfaces with varying properties per instance |
1780 |
- multiple hypervisors: multiple hypervisors can be active on the same |
1781 |
cluster, each supporting different parameters, |
1782 |
- support for device type CDROM (via ISO image) |
1783 |
|
1784 |
As such, there are several areas of Ganeti where the command line |
1785 |
arguments will change: |
1786 |
|
1787 |
- Cluster configuration |
1788 |
|
1789 |
- cluster initialization |
1790 |
- cluster default configuration |
1791 |
|
1792 |
- Instance configuration |
1793 |
|
1794 |
- handling of network cards for instances, |
1795 |
- handling of disks for instances, |
1796 |
- handling of CDROM devices and |
1797 |
- handling of hypervisor specific options. |
1798 |
|
1799 |
There are several areas of Ganeti where the command line arguments |
1800 |
will change: |
1801 |
|
1802 |
- Cluster configuration |
1803 |
|
1804 |
- cluster initialization |
1805 |
- cluster default configuration |
1806 |
|
1807 |
- Instance configuration |
1808 |
|
1809 |
- handling of network cards for instances, |
1810 |
- handling of disks for instances, |
1811 |
- handling of CDROM devices and |
1812 |
- handling of hypervisor specific options. |
1813 |
|
1814 |
Notes about device removal/addition |
1815 |
+++++++++++++++++++++++++++++++++++ |
1816 |
|
1817 |
To avoid problems with device location changes (e.g. second network |
1818 |
interface of the instance becoming the first or third and the like) |
1819 |
the list of network/disk devices is treated as a stack, i.e. devices |
1820 |
can only be added/removed at the end of the list of devices of each |
1821 |
class (disk or network) for each instance. |
1822 |
|
1823 |
gnt-instance commands |
1824 |
+++++++++++++++++++++ |
1825 |
|
1826 |
The commands for gnt-instance will be modified and extended to allow |
1827 |
for the new functionality: |
1828 |
|
1829 |
- the add command will be extended to support the new device and |
1830 |
hypervisor options, |
1831 |
- the modify command continues to handle all modifications to |
1832 |
instances, but will be extended with new arguments for handling |
1833 |
devices. |
1834 |
|
1835 |
Network Device Options |
1836 |
++++++++++++++++++++++ |
1837 |
|
1838 |
The generic format of the network device option is: |
1839 |
|
1840 |
--net $DEVNUM[:$OPTION=$VALUE][,$OPTION=VALUE] |
1841 |
|
1842 |
:$DEVNUM: device number, unsigned integer, starting at 0, |
1843 |
:$OPTION: device option, string, |
1844 |
:$VALUE: device option value, string. |
1845 |
|
1846 |
Currently, the following device options will be defined (open to |
1847 |
further changes): |
1848 |
|
1849 |
:mac: MAC address of the network interface, accepts either a valid |
1850 |
MAC address or the string 'auto'. If 'auto' is specified, a new MAC |
1851 |
address will be generated randomly. If the mac device option is not |
1852 |
specified, the default value 'auto' is assumed. |
1853 |
:bridge: network bridge the network interface is connected |
1854 |
to. Accepts either a valid bridge name (the specified bridge must |
1855 |
exist on the node(s)) as string or the string 'auto'. If 'auto' is |
1856 |
specified, the default brigde is used. If the bridge option is not |
1857 |
specified, the default value 'auto' is assumed. |
1858 |
|
1859 |
Disk Device Options |
1860 |
+++++++++++++++++++ |
1861 |
|
1862 |
The generic format of the disk device option is: |
1863 |
|
1864 |
--disk $DEVNUM[:$OPTION=$VALUE][,$OPTION=VALUE] |
1865 |
|
1866 |
:$DEVNUM: device number, unsigned integer, starting at 0, |
1867 |
:$OPTION: device option, string, |
1868 |
:$VALUE: device option value, string. |
1869 |
|
1870 |
Currently, the following device options will be defined (open to |
1871 |
further changes): |
1872 |
|
1873 |
:size: size of the disk device, either a positive number, specifying |
1874 |
the disk size in mebibytes, or a number followed by a magnitude suffix |
1875 |
(M for mebibytes, G for gibibytes). Also accepts the string 'auto' in |
1876 |
which case the default disk size will be used. If the size option is |
1877 |
not specified, 'auto' is assumed. This option is not valid for all |
1878 |
disk layout types. |
1879 |
:access: access mode of the disk device, a single letter, valid values |
1880 |
are: |
1881 |
|
1882 |
- *w*: read/write access to the disk device or |
1883 |
- *r*: read-only access to the disk device. |
1884 |
|
1885 |
If the access mode is not specified, the default mode of read/write |
1886 |
access will be configured. |
1887 |
:path: path to the image file for the disk device, string. No default |
1888 |
exists. This option is not valid for all disk layout types. |
1889 |
|
1890 |
Adding devices |
1891 |
++++++++++++++ |
1892 |
|
1893 |
To add devices to an already existing instance, use the device type |
1894 |
specific option to gnt-instance modify. Currently, there are two |
1895 |
device type specific options supported: |
1896 |
|
1897 |
:--net: for network interface cards |
1898 |
:--disk: for disk devices |
1899 |
|
1900 |
The syntax to the device specific options is similar to the generic |
1901 |
device options, but instead of specifying a device number like for |
1902 |
gnt-instance add, you specify the magic string add. The new device |
1903 |
will always be appended at the end of the list of devices of this type |
1904 |
for the specified instance, e.g. if the instance has disk devices 0,1 |
1905 |
and 2, the newly added disk device will be disk device 3. |
1906 |
|
1907 |
Example: gnt-instance modify --net add:mac=auto test-instance |
1908 |
|
1909 |
Removing devices |
1910 |
++++++++++++++++ |
1911 |
|
1912 |
Removing devices from and instance is done via gnt-instance |
1913 |
modify. The same device specific options as for adding instances are |
1914 |
used. Instead of a device number and further device options, only the |
1915 |
magic string remove is specified. It will always remove the last |
1916 |
device in the list of devices of this type for the instance specified, |
1917 |
e.g. if the instance has disk devices 0, 1, 2 and 3, the disk device |
1918 |
number 3 will be removed. |
1919 |
|
1920 |
Example: gnt-instance modify --net remove test-instance |
1921 |
|
1922 |
Modifying devices |
1923 |
+++++++++++++++++ |
1924 |
|
1925 |
Modifying devices is also done with device type specific options to |
1926 |
the gnt-instance modify command. There are currently two device type |
1927 |
options supported: |
1928 |
|
1929 |
:--net: for network interface cards |
1930 |
:--disk: for disk devices |
1931 |
|
1932 |
The syntax to the device specific options is similar to the generic |
1933 |
device options. The device number you specify identifies the device to |
1934 |
be modified. |
1935 |
|
1936 |
Example:: |
1937 |
|
1938 |
gnt-instance modify --disk 2:access=r |
1939 |
|
1940 |
Hypervisor Options |
1941 |
++++++++++++++++++ |
1942 |
|
1943 |
Ganeti 2.0 will support more than one hypervisor. Different |
1944 |
hypervisors have various options that only apply to a specific |
1945 |
hypervisor. Those hypervisor specific options are treated specially |
1946 |
via the ``--hypervisor`` option. The generic syntax of the hypervisor |
1947 |
option is as follows:: |
1948 |
|
1949 |
--hypervisor $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE] |
1950 |
|
1951 |
:$HYPERVISOR: symbolic name of the hypervisor to use, string, |
1952 |
has to match the supported hypervisors. Example: xen-pvm |
1953 |
|
1954 |
:$OPTION: hypervisor option name, string |
1955 |
:$VALUE: hypervisor option value, string |
1956 |
|
1957 |
The hypervisor option for an instance can be set on instance creation |
1958 |
time via the ``gnt-instance add`` command. If the hypervisor for an |
1959 |
instance is not specified upon instance creation, the default |
1960 |
hypervisor will be used. |
1961 |
|
1962 |
Modifying hypervisor parameters |
1963 |
+++++++++++++++++++++++++++++++ |
1964 |
|
1965 |
The hypervisor parameters of an existing instance can be modified |
1966 |
using ``--hypervisor`` option of the ``gnt-instance modify`` |
1967 |
command. However, the hypervisor type of an existing instance can not |
1968 |
be changed, only the particular hypervisor specific option can be |
1969 |
changed. Therefore, the format of the option parameters has been |
1970 |
simplified to omit the hypervisor name and only contain the comma |
1971 |
separated list of option-value pairs. |
1972 |
|
1973 |
Example:: |
1974 |
|
1975 |
gnt-instance modify --hypervisor cdrom=/srv/boot.iso,boot_order=cdrom:network test-instance |
1976 |
|
1977 |
gnt-cluster commands |
1978 |
++++++++++++++++++++ |
1979 |
|
1980 |
The command for gnt-cluster will be extended to allow setting and |
1981 |
changing the default parameters of the cluster: |
1982 |
|
1983 |
- The init command will be extend to support the defaults option to |
1984 |
set the cluster defaults upon cluster initialization. |
1985 |
- The modify command will be added to modify the cluster |
1986 |
parameters. It will support the --defaults option to change the |
1987 |
cluster defaults. |
1988 |
|
1989 |
Cluster defaults |
1990 |
|
1991 |
The generic format of the cluster default setting option is: |
1992 |
|
1993 |
--defaults $OPTION=$VALUE[,$OPTION=$VALUE] |
1994 |
|
1995 |
:$OPTION: cluster default option, string, |
1996 |
:$VALUE: cluster default option value, string. |
1997 |
|
1998 |
Currently, the following cluster default options are defined (open to |
1999 |
further changes): |
2000 |
|
2001 |
:hypervisor: the default hypervisor to use for new instances, |
2002 |
string. Must be a valid hypervisor known to and supported by the |
2003 |
cluster. |
2004 |
:disksize: the disksize for newly created instance disks, where |
2005 |
applicable. Must be either a positive number, in which case the unit |
2006 |
of megabyte is assumed, or a positive number followed by a supported |
2007 |
magnitude symbol (M for megabyte or G for gigabyte). |
2008 |
:bridge: the default network bridge to use for newly created instance |
2009 |
network interfaces, string. Must be a valid bridge name of a bridge |
2010 |
existing on the node(s). |
2011 |
|
2012 |
Hypervisor cluster defaults |
2013 |
+++++++++++++++++++++++++++ |
2014 |
|
2015 |
The generic format of the hypervisor cluster wide default setting |
2016 |
option is:: |
2017 |
|
2018 |
--hypervisor-defaults $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE] |
2019 |
|
2020 |
:$HYPERVISOR: symbolic name of the hypervisor whose defaults you want |
2021 |
to set, string |
2022 |
:$OPTION: cluster default option, string, |
2023 |
:$VALUE: cluster default option value, string. |
2024 |
|
2025 |
.. vim: set textwidth=72 : |