5 This document describes the major changes in Ganeti 2.0 compared to
8 The 2.0 version will constitute a rewrite of the 'core' architecture,
9 paving the way for additional features in future 2.x versions.
16 Ganeti 1.2 has many scalability issues and restrictions due to its
17 roots as software for managing small and 'static' clusters.
19 Version 2.0 will attempt to remedy first the scalability issues and
20 then the restrictions.
25 While Ganeti 1.2 is usable, it severly limits the flexibility of the
26 cluster administration and imposes a very rigid model. It has the
27 following main scalability issues:
29 - only one operation at a time on the cluster [#]_
30 - poor handling of node failures in the cluster
31 - mixing hypervisors in a cluster not allowed
33 It also has a number of artificial restrictions, due to historical design:
35 - fixed number of disks (two) per instance
36 - fixed number of nics
38 .. [#] Replace disks will release the lock, but this is an exception
39 and not a recommended way to operate
41 The 2.0 version is intended to address some of these problems, and
42 create a more flexible codebase for future developments.
47 Ganeti 1.2 has a single global lock, which is used for all cluster
48 operations. This has been painful at various times, for example:
50 - It is impossible for two people to efficiently interact with a cluster
51 (for example for debugging) at the same time.
52 - When batch jobs are running it's impossible to do other work (for example
53 failovers/fixes) on a cluster.
55 This poses scalability problems: as clusters grow in node and instance
56 size it's a lot more likely that operations which one could conceive
57 should run in parallel (for example because they happen on different
58 nodes) are actually stalling each other while waiting for the global
59 lock, without a real reason for that to happen.
61 One of the main causes of this global lock (beside the higher
62 difficulty of ensuring data consistency in a more granular lock model)
63 is the fact that currently there is no "master" daemon in Ganeti. Each
64 command tries to acquire the so called *cmd* lock and when it
65 succeeds, it takes complete ownership of the cluster configuration and
68 Other scalability problems are due the design of the DRBD device
69 model, which assumed at its creation a low (one to four) number of
70 instances per node, which is no longer true with today's hardware.
72 Artificial restrictions
73 -----------------------
75 Ganeti 1.2 (and previous versions) have a fixed two-disks, one-NIC per
76 instance model. This is a purely artificial restrictions, but it
77 touches multiple areas (configuration, import/export, command line)
78 that it's more fitted to a major release than a minor one.
83 In order to solve the scalability problems, a rewrite of the core
84 design of Ganeti is required. While the cluster operations themselves
85 won't change (e.g. start instance will do the same things, the way
86 these operations are scheduled internally will change radically.
88 The new design will change the cluster architecture to:
90 .. image:: arch-2.0.png
92 This differs from the 1.2 architecture by the addition of the master
93 daemon, which will be the only entity to talk to the node daemons.
99 The changes for 2.0 can be split into roughly three areas:
101 - core changes that affect the design of the software
102 - features (or restriction removals) but which do not have a wide
104 - user-level and API-level changes which translate into differences for
105 the operation of the cluster
110 The main changes will be switching from a per-process model to a
111 daemon based model, where the individual gnt-* commands will be
112 clients that talk to this daemon (see the design-2.0-master-daemon
113 document). This will allow us to get rid of the global cluster lock
114 for most operations, having instead a per-object lock (see
115 design-2.0-granular-locking). Also, the daemon will be able to queue
116 jobs, and this will allow the invidual clients to submit jobs without
117 waiting for them to finish, and also see the result of old requests
118 (see design-2.0-job-queue).
120 Beside these major changes, another 'core' change but that will not be
121 as visible to the users will be changing the model of object attribute
122 storage, and separate that into namespaces (such that an Xen PVM
123 instance will not have the Xen HVM parameters). This will allow future
124 flexibility in defining additional parameters. More details in the
125 design-2.0-cluster-parameters document.
127 The various changes brought in by the master daemon model and the
128 read-write RAPI will require changes to the cluster security; we move
129 away from Twisted and use http(s) for intra- and extra-cluster
130 communications. For more details, see the security document in the
136 In Ganeti 2.0, we will have the following *entities*:
138 - the master daemon (on the master node)
139 - the node daemon (on all nodes)
140 - the command line tools (on the master node)
141 - the RAPI daemon (on the master node)
143 Interaction paths are between:
145 - (CLI tools/RAPI daemon) and the master daemon, via the so called *luxi* API
146 - the master daemon and the node daemons, via the node RPC
148 The protocol between the master daemon and the node daemons will be
149 changed to HTTP(S), using a simple PUT/GET of JSON-encoded
150 messages. This is done due to difficulties in working with the Twisted
151 framework and its protocols in a multithreaded environment, which we
152 can overcome by using a simpler stack (see the caveats section). The
153 protocol between the CLI/RAPI and the master daemon will be a custom
154 one (called *luxi*): on a UNIX socket on the master node, with rights
155 restricted by filesystem permissions, the CLI/RAPI will talk to the
156 master daemon using JSON-encoded messages.
158 The operations supported over this internal protocol will be encoded
159 via a python library that will expose a simple API for its
160 users. Internally, the protocol will simply encode all objects in JSON
161 format and decode them on the receiver side.
166 We will have two main classes of operations over the master daemon API:
168 - cluster query functions
169 - job related functions
171 The cluster query functions are usually short-duration, and are the
172 equivalent of the OP_QUERY_* opcodes in ganeti 1.2 (and they are
173 internally implemented still with these opcodes). The clients are
174 guaranteed to receive the response in a reasonable time via a timeout.
176 The job-related functions will be:
179 - query job (which could also be categorized in the query-functions)
180 - archive job (see the job queue design doc)
181 - wait for job change, which allows a client to wait without polling
183 For more details, see the job queue design document.
185 Daemon implementation
186 +++++++++++++++++++++
188 The daemon will be based around a main I/O thread that will wait for
189 new requests from the clients, and that does the setup/shutdown of the
190 other thread (pools).
192 There will two other classes of threads in the daemon:
194 - job processing threads, part of a thread pool, and which are
195 long-lived, started at daemon startup and terminated only at shutdown
197 - client I/O threads, which are the ones that talk the local protocol
200 Master startup/failover
201 +++++++++++++++++++++++
203 In Ganeti 1.x there is no protection against failing over the master
204 to a node with stale configuration. In effect, the responsibility of
205 correct failovers falls on the admin. This is true both for the new
206 master and for when an old, offline master startup.
208 Since in 2.x we are extending the cluster state to cover the job queue
209 and have a daemon that will execute by itself the job queue, we want
210 to have more resilience for the master role.
212 The following algorithm will happen whenever a node is ready to
213 transition to the master role, either at startup time or at node
216 #. read the configuration file and parse the node list
219 #. query all the nodes and make sure we obtain an agreement via
220 a quorum of at least half plus one nodes for the following:
222 - we have the latest configuration and job list (as
223 determined by the serial number on the configuration and
224 highest job ID on the job queue)
226 - there is not even a single node having a newer
229 - if we are not failing over (but just starting), the
230 quorum agrees that we are the designated master
232 #. at this point, the node transitions to the master role
234 #. for all the in-progress jobs, mark them as failed, with
235 reason unknown or something similar (master failed, etc.)
241 The logging system will be switched completely to the logging module;
242 currently it's logging-based, but exposes a different API, which is
243 just overhead. As such, the code will be switched over to standard
244 logging calls, and only the setup will be custom.
246 With this change, we will remove the separate debug/info/error logs,
247 and instead have always one logfile per daemon model:
249 - master-daemon.log for the master daemon
250 - node-daemon.log for the node daemon (this is the same as in 1.2)
251 - rapi-daemon.log for the RAPI daemon logs
252 - rapi-access.log, an additional log file for the RAPI that will be
253 in the standard http log format for possible parsing by other tools
255 Since the watcher will only submit jobs to the master for startup of
256 the instances, its log file will contain less information than before,
257 mainly that it will start the instance, but not the results.
262 A discussed alternative is to keep the current individual processes
263 touching the cluster configuration model. The reasons we have not
264 chosen this approach is:
266 - the speed of reading and unserializing the cluster state
267 today is not small enough that we can ignore it; the addition of
268 the job queue will make the startup cost even higher. While this
269 runtime cost is low, it can be on the order of a few seconds on
270 bigger clusters, which for very quick commands is comparable to
271 the actual duration of the computation itself
273 - individual commands would make it harder to implement a
274 fire-and-forget job request, along the lines "start this
275 instance but do not wait for it to finish"; it would require a
276 model of backgrounding the operation and other things that are
277 much better served by a daemon-based model
279 Another area of discussion is moving away from Twisted in this new
280 implementation. While Twisted hase its advantages, there are also many
281 disatvantanges to using it:
283 - first and foremost, it's not a library, but a framework; thus, if
284 you use twisted, all the code needs to be 'twiste-ized'; we were able
285 to keep the 1.x code clean by hacking around twisted in an
286 unsupported, unrecommended way, and the only alternative would have
287 been to make all the code be written for twisted
288 - it has some weaknesses in working with multiple threads, since its base
289 model is designed to replace thread usage by using deferred calls, so while
290 it can use threads, it's not less flexible in doing so
292 And, since we already have an HTTP server library for the RAPI, we
293 can just reuse that for inter-node communication.
299 We want to make sure that multiple operations can run in parallel on a Ganeti
300 Cluster. In order for this to happen we need to make sure concurrently run
301 operations don't step on each other toes and break the cluster.
303 This design addresses how we are going to deal with locking so that:
305 - high urgency operations are not stopped by long length ones
306 - long length operations can run in parallel
307 - we preserve safety (data coherency) and liveness (no deadlock, no work
308 postponed indefinitely) on the cluster
310 Reaching the maximum possible parallelism is a Non-Goal. We have identified a
311 set of operations that are currently bottlenecks and need to be parallelised
312 and have worked on those. In the future it will be possible to address other
313 needs, thus making the cluster more and more parallel one step at a time.
315 This document only talks about parallelising Ganeti level operations, aka
316 Logical Units, and the locking needed for that. Any other synchronisation lock
317 needed internally by the code is outside its scope.
322 We intend to implement a Ganeti locking library, which can be used by the
323 various ganeti code components in order to easily, efficiently and correctly
324 grab the locks they need to perform their function.
326 The proposed library has these features:
328 - Internally managing all the locks, making the implementation transparent
330 - Automatically grabbing multiple locks in the right order (avoid deadlock)
331 - Ability to transparently handle conversion to more granularity
332 - Support asynchronous operation (future goal)
334 Locking will be valid only on the master node and will not be a distributed
335 operation. In case of master failure, though, if some locks were held it means
336 some opcodes were in progress, so when recovery of the job queue is done it
337 will be possible to determine by the interrupted opcodes which operations could
338 have been left half way through and thus which locks could have been held. It
339 is then the responsibility either of the master failover code, of the cluster
340 verification code, or of the admin to do what's necessary to make sure that any
341 leftover state is dealt with. This is not an issue from a locking point of view
342 because the fact that the previous master has failed means that it cannot do
345 A corollary of this is that a master-failover operation with both masters alive
346 needs to happen while no other locks are held.
351 At the first stage we have decided to provide the following locks:
353 - One "config file" lock
354 - One lock per node in the cluster
355 - One lock per instance in the cluster
357 All the instance locks will need to be taken before the node locks, and the
358 node locks before the config lock. Locks will need to be acquired at the same
359 time for multiple instances and nodes, and internal ordering will be dealt
360 within the locking library, which, for simplicity, will just use alphabetical
363 Handling conversion to more granularity
364 +++++++++++++++++++++++++++++++++++++++
366 In order to convert to a more granular approach transparently each time we
367 split a lock into more we'll create a "metalock", which will depend on those
368 sublocks and live for the time necessary for all the code to convert (or
369 forever, in some conditions). When a metalock exists all converted code must
370 acquire it in shared mode, so it can run concurrently, but still be exclusive
371 with old code, which acquires it exclusively.
373 In the beginning the only such lock will be what replaces the current "command"
374 lock, and will acquire all the locks in the system, before proceeding. This
375 lock will be called the "Big Ganeti Lock" because holding that one will avoid
376 any other concurrent ganeti operations.
378 We might also want to devise more metalocks (eg. all nodes, all nodes+config)
379 in order to make it easier for some parts of the code to acquire what it needs
380 without specifying it explicitly.
382 In the future things like the node locks could become metalocks, should we
383 decide to split them into an even more fine grained approach, but this will
384 probably be only after the first 2.0 version has been released.
389 All the locking will be its own class, and the locks will be created at
390 initialisation time, from the config file.
392 The API will have a way to grab one or more than one locks at the same time.
393 Any attempt to grab a lock while already holding one in the wrong order will be
394 checked for, and fail.
396 Adding/Removing locks
397 +++++++++++++++++++++
399 When a new instance or a new node is created an associated lock must be added
400 to the list. The relevant code will need to inform the locking library of such
403 This needs to be compatible with every other lock in the system, especially
404 metalocks that guarantee to grab sets of resources without specifying them
405 explicitly. The implementation of this will be handled in the locking library
408 Of course when instances or nodes disappear from the cluster the relevant locks
409 must be removed. This is easier than adding new elements, as the code which
410 removes them must own them exclusively or can queue for their ownership, and
411 thus deals with metalocks exactly as normal code acquiring those locks. Any
412 operation queueing on a removed lock will fail after its removal.
414 Asynchronous operations
415 +++++++++++++++++++++++
417 For the first version the locking library will only export synchronous
418 operations, which will block till the needed lock are held, and only fail if
419 the request is impossible or somehow erroneous.
421 In the future we may want to implement different types of asynchronous
424 - Try to acquire this lock set and fail if not possible
425 - Try to acquire one of these lock sets and return the first one you were
426 able to get (or after a timeout) (select/poll like)
428 These operations can be used to prioritize operations based on available locks,
429 rather than making them just blindly queue for acquiring them. The inherent
430 risk, though, is that any code using the first operation, or setting a timeout
431 for the second one, is susceptible to starvation and thus may never be able to
432 get the required locks and complete certain tasks. Considering this
433 providing/using these operations should not be among our first priorities.
438 For the first version of this code we'll convert each Logical Unit to
439 acquire/release the locks it needs, so locking will be at the Logical Unit
440 level. In the future we may want to split logical units in independent
441 "tasklets" with their own locking requirements. A different design doc (or mini
442 design doc) will cover the move from Logical Units to tasklets.
444 Lock acquisition code path
445 ++++++++++++++++++++++++++
447 In general when acquiring locks we should use a code path equivalent to::
456 This makes sure we release all locks, and avoid possible deadlocks. Of course
457 extra care must be used not to leave, if possible locked structures in an
460 In order to avoid this extra indentation and code changes everywhere in the
461 Logical Units code, we decided to allow LUs to declare locks, and then execute
462 their code with their locks acquired. In the new world LUs are called like
465 # user passed names are expanded to the internal lock/resource name,
466 # then known needed locks are declared
468 ... some locking/adding of locks may happen ...
469 # late declaration of locks for one level: this is useful because sometimes
470 # we can't know which resource we need before locking the previous level
471 lu.DeclareLocks() # for each level (cluster, instance, node)
472 ... more locking/adding of locks can happen ...
473 # these functions are called with the proper locks held
476 ... locks declared for removal are removed, all acquired locks released ...
478 The Processor and the LogicalUnit class will contain exact documentation on how
479 locks are supposed to be declared.
484 This library will provide an easy upgrade path to bring all the code to
485 granular locking without breaking everything, and it will also guarantee
486 against a lot of common errors. Code switching from the old "lock everything"
487 lock to the new system, though, needs to be carefully scrutinised to be sure it
488 is really acquiring all the necessary locks, and none has been overlooked or
491 The code can contain other locks outside of this library, to synchronise other
492 threaded code (eg for the job queue) but in general these should be leaf locks
493 or carefully structured non-leaf ones, to avoid deadlock race conditions.
499 Granular locking is not enough to speed up operations, we also need a
500 queue to store these and to be able to process as many as possible in
503 A ganeti job will consist of multiple ``OpCodes`` which are the basic
504 element of operation in Ganeti 1.2 (and will remain as such). Most
505 command-level commands are equivalent to one OpCode, or in some cases
506 to a sequence of opcodes, all of the same type (e.g. evacuating a node
507 will generate N opcodes of type replace disks).
510 Job execution—“Life of a Ganeti job”
511 ++++++++++++++++++++++++++++++++++++
513 #. Job gets submitted by the client. A new job identifier is generated and
514 assigned to the job. The job is then automatically replicated [#replic]_
515 to all nodes in the cluster. The identifier is returned to the client.
516 #. A pool of worker threads waits for new jobs. If all are busy, the job has
517 to wait and the first worker finishing its work will grab it. Otherwise any
518 of the waiting threads will pick up the new job.
519 #. Client waits for job status updates by calling a waiting RPC function.
520 Log message may be shown to the user. Until the job is started, it can also
522 #. As soon as the job is finished, its final result and status can be retrieved
524 #. If the client archives the job, it gets moved to a history directory.
525 There will be a method to archive all jobs older than a a given age.
527 .. [#replic] We need replication in order to maintain the consistency across
528 all nodes in the system; the master node only differs in the fact that
529 now it is running the master daemon, but it if fails and we do a master
530 failover, the jobs are still visible on the new master (though marked as
533 Failures to replicate a job to other nodes will be only flagged as
534 errors in the master daemon log if more than half of the nodes failed,
535 otherwise we ignore the failure, and rely on the fact that the next
536 update (for still running jobs) will retry the update. For finished
537 jobs, it is less of a problem.
539 Future improvements will look into checking the consistency of the job
540 list and jobs themselves at master daemon startup.
546 Jobs are stored in the filesystem as individual files, serialized
547 using JSON (standard serialization mechanism in Ganeti).
549 The choice of storing each job in its own file was made because:
551 - a file can be atomically replaced
552 - a file can easily be replicated to other nodes
553 - checking consistency across nodes can be implemented very easily, since
554 all job files should be (at a given moment in time) identical
556 The other possible choices that were discussed and discounted were:
558 - single big file with all job data: not feasible due to difficult updates
559 - in-process databases: hard to replicate the entire database to the
560 other nodes, and replicating individual operations does not mean wee keep
567 All file operations have to be done atomically by writing to a temporary file
568 and subsequent renaming. Except for log messages, every change in a job is
569 stored and replicated to other nodes.
573 /var/lib/ganeti/queue/
574 job-1 (JSON encoded job description and status)
579 lock (Queue managing process opens this file in exclusive mode)
580 serial (Last job ID used)
581 version (Queue format version)
587 Locking in the job queue is a complicated topic. It is called from more than
588 one thread and must be thread-safe. For simplicity, a single lock is used for
591 A more detailed description can be found in doc/locking.txt.
597 RPC calls available between Ganeti master and node daemons:
599 jobqueue_update(file_name, content)
600 Writes a file in the job queue directory.
602 Cleans the job queue directory completely, including archived job.
603 jobqueue_rename(old, new)
604 Renames a file in the job queue directory.
610 RPC between Ganeti clients and the Ganeti master daemon supports the following
614 Submits a list of opcodes and returns the job identifier. The identifier is
615 guaranteed to be unique during the lifetime of a cluster.
616 WaitForJobChange(job_id, fields, […], timeout)
617 This function waits until a job changes or a timeout expires. The condition
618 for when a job changed is defined by the fields passed and the last log
620 QueryJobs(job_ids, fields)
621 Returns field values for the job identifiers passed.
623 Cancels the job specified by identifier. This operation may fail if the job
624 is already running, canceled or finished.
626 Moves a job into the …/archive/ directory. This operation will fail if the
627 job has not been canceled or finished.
630 Job and opcode status
631 +++++++++++++++++++++
633 Each job and each opcode has, at any time, one of the following states:
636 The job/opcode was submitted, but did not yet start.
638 The job/opcode is waiting for a lock to proceed.
640 The job/opcode is running.
642 The job/opcode was canceled before it started.
644 The job/opcode ran and finished successfully.
646 The job/opcode was aborted with an error.
648 If the master is aborted while a job is running, the job will be set to the
649 Error status once the master started again.
655 Archived jobs are kept in a separate directory,
656 /var/lib/ganeti/queue/archive/. This is done in order to speed up the
657 queue handling: by default, the jobs in the archive are not touched by
658 any functions. Only the current (unarchived) jobs are parsed, loaded,
659 and verified (if implemented) by the master daemon.
665 The queue has to be completely empty for Ganeti updates with changes
666 in the job queue structure. In order to allow this, there will be a
667 way to prevent new jobs entering the queue.
674 Across all cluster configuration data, we have multiple classes of
677 A. cluster-wide parameters (e.g. name of the cluster, the master);
678 these are the ones that we have today, and are unchanged from the
683 #. instance specific parameters, e.g. the name of disks (LV), that
684 cannot be shared with other instances
686 #. instance parameters, that are or can be the same for many
687 instances, but are not hypervisor related; e.g. the number of VCPUs,
688 or the size of memory
690 #. instance parameters that are hypervisor specific (e.g. kernel_path
694 The following definitions for instance parameters will be used below:
696 :hypervisor parameter:
697 a hypervisor parameter (or hypervisor specific parameter) is defined
698 as a parameter that is interpreted by the hypervisor support code in
699 Ganeti and usually is specific to a particular hypervisor (like the
700 kernel path for PVM which makes no sense for HVM).
703 a backend parameter is defined as an instance parameter that can be
704 shared among a list of instances, and is either generic enough not
705 to be tied to a given hypervisor or cannot influence at all the
706 hypervisor behaviour.
708 For example: memory, vcpus, auto_balance
710 All these parameters will be encoded into constants.py with the prefix "BE\_"
711 and the whole list of parameters will exist in the set "BES_PARAMETERS"
714 a parameter whose value is unique to the instance (e.g. the name of a LV,
717 As a general rule, for all kind of parameters, “None” (or in
718 JSON-speak, “nil”) will no longer be a valid value for a parameter. As
719 such, only non-default parameters will be saved as part of objects in
720 the serialization step, reducing the size of the serialized format.
725 Cluster parameters remain as today, attributes at the top level of the
726 Cluster object. In addition, two new attributes at this level will
727 hold defaults for the instances:
729 - hvparams, a dictionary indexed by hypervisor type, holding default
730 values for hypervisor parameters that are not defined/overrided by
731 the instances of this hypervisor type
733 - beparams, a dictionary holding (for 2.0) a single element 'default',
734 which holds the default value for backend parameters
739 Node-related parameters are very few, and we will continue using the
740 same model for these as previously (attributes on the Node object).
745 As described before, the instance parameters are split in three:
746 instance proper parameters, unique to each instance, instance
747 hypervisor parameters and instance backend parameters.
749 The “hvparams” and “beparams” are kept in two dictionaries at instance
750 level. Only non-default parameters are stored (but once customized, a
751 parameter will be kept, even with the same value as the default one,
754 The names for hypervisor parameters in the instance.hvparams subtree
755 should be choosen as generic as possible, especially if specific
756 parameters could conceivably be useful for more than one hypervisor,
757 e.g. instance.hvparams.vnc_console_port instead of using both
758 instance.hvparams.hvm_vnc_console_port and
759 instance.hvparams.kvm_vnc_console_port.
761 There are some special cases related to disks and NICs (for example):
762 a disk has both ganeti-related parameters (e.g. the name of the LV)
763 and hypervisor-related parameters (how the disk is presented to/named
764 in the instance). The former parameters remain as proper-instance
765 parameters, while the latter value are migrated to the hvparams
766 structure. In 2.0, we will have only globally-per-instance such
767 hypervisor parameters, and not per-disk ones (e.g. all NICs will be
768 exported as of the same type).
770 Starting from the 1.2 list of instance parameters, here is how they
771 will be mapped to the three classes of parameters:
786 - hvm_boot_order (HV)
789 - hvm_cdrom_image_path (HV)
792 - vnc_bind_address (HV)
799 To support the new cluster parameter design, additional features will
800 be required from the hypervisor support implementations in Ganeti.
802 The hypervisor support implementation API will be extended with the
805 :PARAMETERS: class-level attribute holding the list of valid parameters
807 :CheckParamSyntax(hvparams): checks that the given parameters are
808 valid (as in the names are valid) for this hypervisor; usually just
809 comparing hvparams.keys() and cls.PARAMETERS; this is a class method
810 that can be called from within master code (i.e. cmdlib) and should
812 :ValidateParameters(hvparams): verifies the values of the provided
813 parameters against this hypervisor; this is a method that will be
814 called on the target node, from backend.py code, and as such can
815 make node-specific checks (e.g. kernel_path checking)
817 Default value application
818 +++++++++++++++++++++++++
820 The application of defaults to an instance is done in the Cluster
821 object, via two new methods as follows:
823 - ``Cluster.FillHV(instance)``, returns 'filled' hvparams dict, based on
824 instance's hvparams and cluster's ``hvparams[instance.hypervisor]``
826 - ``Cluster.FillBE(instance, be_type="default")``, which returns the
827 beparams dict, based on the instance and cluster beparams
829 The FillHV/BE transformations will be used, for example, in the RpcRunner
830 when sending an instance for activation/stop, and the sent instance
831 hvparams/beparams will have the final value (noded code doesn't know
834 LU code will need to self-call the transformation, if needed.
839 The parameter changes will have impact on the OpCodes, especially on
842 - OpCreateInstance, where the new hv and be parameters will be sent as
843 dictionaries; note that all hv and be parameters are now optional, as
844 the values can be instead taken from the cluster
845 - OpQueryInstances, where we have to be able to query these new
846 parameters; the syntax for names will be ``hvparam/$NAME`` and
847 ``beparam/$NAME`` for querying an individual parameter out of one
848 dictionary, and ``hvparams``, respectively ``beparams``, for the whole
850 - OpModifyInstance, where the the modified parameters are sent as
853 Additionally, we will need new OpCodes to modify the cluster-level
854 defaults for the be/hv sets of parameters.
859 One problem that might appear is that our classification is not
860 complete or not good enough, and we'll need to change this model. As
861 the last resort, we will need to rollback and keep 1.2 style.
863 Another problem is that classification of one parameter is unclear
864 (e.g. ``network_port``, is this BE or HV?); in this case we'll take
865 the risk of having to move parameters later between classes.
870 The only security issue that we foresee is if some new parameters will
871 have sensitive value. If so, we will need to have a way to export the
872 config data while purging the sensitive value.
874 E.g. for the drbd shared secrets, we could export these with the
875 values replaced by an empty string.
880 The main feature-level changes will be:
882 - a number of disk related changes
883 - removal of fixed two-disk, one-nic per instance limitation
885 Disk handling changes
886 ~~~~~~~~~~~~~~~~~~~~~
888 The storage options available in Ganeti 1.x were introduced based on
889 then-current software (first DRBD 0.7 then later DRBD 8) and the
890 estimated usage patters. However, experience has later shown that some
891 assumptions made initially are not true and that more flexibility is
894 One main assupmtion made was that disk failures should be treated as 'rare'
895 events, and that each of them needs to be manually handled in order to ensure
896 data safety; however, both these assumptions are false:
898 - disk failures can be a common occurence, based on usage patterns or cluster
900 - our disk setup is robust enough (referring to DRBD8 + LVM) that we could
901 automate more of the recovery
903 Note that we still don't have fully-automated disk recovery as a goal, but our
904 goal is to reduce the manual work needed.
906 As such, we plan the following main changes:
908 - DRBD8 is much more flexible and stable than its previous version (0.7),
909 such that removing the support for the ``remote_raid1`` template and
910 focusing only on DRBD8 is easier
912 - dynamic discovery of DRBD devices is not actually needed in a cluster that
913 where the DRBD namespace is controlled by Ganeti; switching to a static
914 assignment (done at either instance creation time or change secondary time)
915 will change the disk activation time from O(n) to O(1), which on big
916 clusters is a significant gain
918 - remove the hard dependency on LVM (currently all available storage types are
919 ultimately backed by LVM volumes) by introducing file-based storage
921 Additionally, a number of smaller enhancements are also planned:
922 - support variable number of disks
923 - support read-only disks
925 Future enhancements in the 2.x series, which do not require base design
926 changes, might include:
928 - enhancement of the LVM allocation method in order to try to keep
929 all of an instance's virtual disks on the same physical
932 - add support for DRBD8 authentication at handshake time in
933 order to ensure each device connects to the correct peer
935 - remove the restrictions on failover only to the secondary
936 which creates very strict rules on cluster allocation
938 DRBD minor allocation
939 +++++++++++++++++++++
941 Currently, when trying to identify or activate a new DRBD (or MD)
942 device, the code scans all in-use devices in order to see if we find
943 one that looks similar to our parameters and is already in the desired
944 state or not. Since this needs external commands to be run, it is very
945 slow when more than a few devices are already present.
947 Therefore, we will change the discovery model from dynamic to
948 static. When a new device is logically created (added to the
949 configuration) a free minor number is computed from the list of
950 devices that should exist on that node and assigned to that
953 At device activation, if the minor is already in use, we check if
954 it has our parameters; if not so, we just destroy the device (if
955 possible, otherwise we abort) and start it with our own
958 This means that we in effect take ownership of the minor space for
959 that device type; if there's a user-created drbd minor, it will be
960 automatically removed.
962 The change will have the effect of reducing the number of external
963 commands run per device from a constant number times the index of the
964 first free DRBD minor to just a constant number.
966 Removal of obsolete device types (md, drbd7)
967 ++++++++++++++++++++++++++++++++++++++++++++
969 We need to remove these device types because of two issues. First,
970 drbd7 has bad failure modes in case of dual failures (both network and
971 disk - it cannot propagate the error up the device stack and instead
972 just panics. Second, due to the assymetry between primary and
973 secondary in md+drbd mode, we cannot do live failover (not even if we
976 File-based storage support
977 ++++++++++++++++++++++++++
979 This is covered by a separate design doc (<em>Vinales</em>) and
980 would allow us to get rid of the hard requirement for testing
981 clusters; it would also allow people who have SAN storage to do live
982 failover taking advantage of their storage solution.
984 Better LVM allocation
985 +++++++++++++++++++++
987 Currently, the LV to PV allocation mechanism is a very simple one: at
988 each new request for a logical volume, tell LVM to allocate the volume
989 in order based on the amount of free space. This is good for
990 simplicity and for keeping the usage equally spread over the available
991 physical disks, however it introduces a problem that an instance could
992 end up with its (currently) two drives on two physical disks, or
993 (worse) that the data and metadata for a DRBD device end up on
996 This is bad because it causes unneeded ``replace-disks`` operations in
997 case of a physical failure.
999 The solution is to batch allocations for an instance and make the LVM
1000 handling code try to allocate as close as possible all the storage of
1001 one instance. We will still allow the logical volumes to spill over to
1002 additional disks as needed.
1004 Note that this clustered allocation can only be attempted at initial
1005 instance creation, or at change secondary node time. At add disk time,
1006 or at replacing individual disks, it's not easy enough to compute the
1007 current disk map so we'll not attempt the clustering.
1009 DRBD8 peer authentication at handshake
1010 ++++++++++++++++++++++++++++++++++++++
1012 DRBD8 has a new feature that allow authentication of the peer at
1013 connect time. We can use this to prevent connecting to the wrong peer
1014 more that securing the connection. Even though we never had issues
1015 with wrong connections, it would be good to implement this.
1018 LVM self-repair (optional)
1019 ++++++++++++++++++++++++++
1021 The complete failure of a physical disk is very tedious to
1022 troubleshoot, mainly because of the many failure modes and the many
1023 steps needed. We can safely automate some of the steps, more
1024 specifically the ``vgreduce --removemissing`` using the following
1027 #. check if all nodes have consistent volume groups
1028 #. if yes, and previous status was yes, do nothing
1029 #. if yes, and previous status was no, save status and restart
1030 #. if no, and previous status was no, do nothing
1031 #. if no, and previous status was yes:
1032 #. if more than one node is inconsistent, do nothing
1033 #. if only one node is incosistent:
1034 #. run ``vgreduce --removemissing``
1035 #. log this occurence in the ganeti log in a form that
1036 can be used for monitoring
1037 #. [FUTURE] run ``replace-disks`` for all
1040 Failover to any node
1041 ++++++++++++++++++++
1043 With a modified disk activation sequence, we can implement the
1044 *failover to any* functionality, removing many of the layout
1045 restrictions of a cluster:
1047 - the need to reserve memory on the current secondary: this gets reduced to
1048 a must to reserve memory anywhere on the cluster
1050 - the need to first failover and then replace secondary for an
1051 instance: with failover-to-any, we can directly failover to
1052 another node, which also does the replace disks at the same
1055 In the following, we denote the current primary by P1, the current
1056 secondary by S1, and the new primary and secondaries by P2 and S2. P2
1057 is fixed to the node the user chooses, but the choice of S2 can be
1058 made between P1 and S1. This choice can be constrained, depending on
1059 which of P1 and S1 has failed.
1061 - if P1 has failed, then S1 must become S2, and live migration is not possible
1062 - if S1 has failed, then P1 must become S2, and live migration could be
1063 possible (in theory, but this is not a design goal for 2.0)
1065 The algorithm for performing the failover is straightforward:
1067 - verify that S2 (the node the user has chosen to keep as secondary) has
1068 valid data (is consistent)
1070 - tear down the current DRBD association and setup a drbd pairing between
1071 P2 (P2 is indicated by the user) and S2; since P2 has no data, it will
1072 start resyncing from S2
1074 - as soon as P2 is in state SyncTarget (i.e. after the resync has started
1075 but before it has finished), we can promote it to primary role (r/w)
1076 and start the instance on P2
1078 - as soon as the P2?S2 sync has finished, we can remove
1079 the old data on the old node that has not been chosen for
1082 Caveats: during the P2?S2 sync, a (non-transient) network error
1083 will cause I/O errors on the instance, so (if a longer instance
1084 downtime is acceptable) we can postpone the restart of the instance
1085 until the resync is done. However, disk I/O errors on S2 will cause
1086 dataloss, since we don't have a good copy of the data anymore, so in
1087 this case waiting for the sync to complete is not an option. As such,
1088 it is recommended that this feature is used only in conjunction with
1089 proper disk monitoring.
1092 Live migration note: While failover-to-any is possible for all choices
1093 of S2, migration-to-any is possible only if we keep P1 as S2.
1098 The dynamic device model, while more complex, has an advantage: it
1099 will not reuse by mistake another's instance DRBD device, since it
1100 always looks for either our own or a free one.
1102 The static one, in contrast, will assume that given a minor number N,
1103 it's ours and we can take over. This needs careful implementation such
1104 that if the minor is in use, either we are able to cleanly shut it
1105 down, or we abort the startup. Otherwise, it could be that we start
1106 syncing between two instance's disks, causing dataloss.
1109 Variable number of disk/NICs per instance
1110 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112 Variable number of disks
1113 ++++++++++++++++++++++++
1115 In order to support high-security scenarios (for example read-only sda
1116 and read-write sdb), we need to make a fully flexibly disk
1117 definition. This has less impact that it might look at first sight:
1118 only the instance creation has hardcoded number of disks, not the disk
1119 handling code. The block device handling and most of the instance
1120 handling code is already working with "the instance's disks" as
1121 opposed to "the two disks of the instance", but some pieces are not
1122 (e.g. import/export) and the code needs a review to ensure safety.
1124 The objective is to be able to specify the number of disks at
1125 instance creation, and to be able to toggle from read-only to
1126 read-write a disk afterwards.
1128 Variable number of NICs
1129 +++++++++++++++++++++++
1131 Similar to the disk change, we need to allow multiple network
1132 interfaces per instance. This will affect the internal code (some
1133 function will have to stop assuming that ``instance.nics`` is a list
1134 of length one), the OS api which currently can export/import only one
1135 instance, and the command line interface.
1140 There are two areas of interface changes: API-level changes (the OS
1141 interface and the RAPI interface) and the command line interface
1147 The current Ganeti OS interface, version 5, is tailored for Ganeti 1.2. The
1148 interface is composed by a series of scripts which get called with certain
1149 parameters to perform OS-dependent operations on the cluster. The current
1153 called when a new instance is added to the cluster
1155 called to export an instance disk to a stream
1157 called to import from a stream to a new instance
1159 called to perform the os-specific operations necessary for renaming an
1162 Currently these scripts suffer from the limitations of Ganeti 1.2: for example
1163 they accept exactly one block and one swap devices to operate on, rather than
1164 any amount of generic block devices, they blindly assume that an instance will
1165 have just one network interface to operate, they can not be configured to
1166 optimise the instance for a particular hypervisor.
1168 Since in Ganeti 2.0 we want to support multiple hypervisors, and a non-fixed
1169 number of network and disks the OS interface need to change to transmit the
1170 appropriate amount of information about an instance to its managing operating
1171 system, when operating on it. Moreover since some old assumptions usually used
1172 in OS scripts are no longer valid we need to re-establish a common knowledge on
1173 what can be assumed and what cannot be regarding Ganeti environment.
1176 When designing the new OS API our priorities are:
1178 - future extensibility
1179 - ease of porting from the old api
1182 As such we want to limit the number of scripts that must be written to support
1183 an OS, and make it easy to share code between them by uniforming their input.
1184 We also will leave the current script structure unchanged, as far as we can,
1185 and make a few of the scripts (import, export and rename) optional. Most
1186 information will be passed to the script through environment variables, for
1187 ease of access and at the same time ease of using only the information a script
1194 As in Ganeti 1.2, every OS which wants to be installed in Ganeti needs to
1195 support the following functionality, through scripts:
1198 used to create a new instance running that OS. This script should prepare the
1199 block devices, and install them so that the new OS can boot under the
1200 specified hypervisor.
1202 used to export an installed instance using the given OS to a format which can
1203 be used to import it back into a new instance.
1205 used to import an exported instance into a new one. This script is similar to
1206 create, but the new instance should have the content of the export, rather
1207 than contain a pristine installation.
1209 used to perform the internal OS-specific operations needed to rename an
1212 If any optional script is not implemented Ganeti will refuse to perform the
1213 given operation on instances using the non-implementing OS. Of course the
1214 create script is mandatory, and it doesn't make sense to support the either the
1215 export or the import operation but not both.
1217 Incompatibilities with 1.2
1218 __________________________
1220 We expect the following incompatibilities between the OS scripts for 1.2 and
1223 - Input parameters: in 1.2 those were passed on the command line, in 2.0 we'll
1224 use environment variables, as there will be a lot more information and not
1225 all OSes may care about all of it.
1226 - Number of calls: export scripts will be called once for each device the
1227 instance has, and import scripts once for every exported disk. Imported
1228 instances will be forced to have a number of disks greater or equal to the
1230 - Some scripts are not compulsory: if such a script is missing the relevant
1231 operations will be forbidden for instances of that os. This makes it easier
1232 to distinguish between unsupported operations and no-op ones (if any).
1238 Rather than using command line flags, as they do now, scripts will accept
1239 inputs from environment variables. We expect the following input values:
1242 The version of the OS api that the following parameters comply with;
1243 this is used so that in the future we could have OSes supporting
1244 multiple versions and thus Ganeti send the proper version in this
1247 Name of the instance acted on
1249 The hypervisor the instance should run on (eg. 'xen-pvm', 'xen-hvm', 'kvm')
1251 The number of disks this instance will have
1253 The number of nics this instance will have
1255 Path to the Nth disk.
1257 W if read/write, R if read only. OS scripts are not supposed to touch
1258 read-only disks, but will be passed them to know.
1259 DISK_<N>_FRONTEND_TYPE
1260 Type of the disk as seen by the instance. Can be 'scsi', 'ide', 'virtio'
1261 DISK_<N>_BACKEND_TYPE
1262 Type of the disk as seen from the node. Can be 'block', 'file:loop' or
1265 Mac address for the Nth network interface
1267 Ip address for the Nth network interface, if available
1269 Node bridge the Nth network interface will be connected to
1270 NIC_<N>_FRONTEND_TYPE
1271 Type of the Nth nic as seen by the instance. For example 'virtio', 'rtl8139', etc.
1273 Whether more out should be produced, for debugging purposes. Currently the
1274 only valid values are 0 and 1.
1276 These are only the basic variables we are thinking of now, but more may come
1277 during the implementation and they will be documented in the ganeti-os-api man
1278 page. All these variables will be available to all scripts.
1280 Some scripts will need a few more information to work. These will have
1281 per-script variables, such as for example:
1284 rename: the name the instance should be renamed from.
1286 export: device to be exported, a snapshot of the actual device. The data must be exported to stdout.
1288 export: sequential number of the instance device targeted.
1290 import: device to send the data to, part of the new instance. The data must be imported from stdin.
1292 import: sequential number of the instance device targeted.
1294 (Rationale for INSTANCE_NAME as an environment variable: the instance name is
1295 always needed and we could pass it on the command line. On the other hand,
1296 though, this would force scripts to both access the environment and parse the
1297 command line, so we'll move it for uniformity.)
1303 As discussed scripts should only send user-targeted information to stderr. The
1304 create and import scripts are supposed to format/initialise the given block
1305 devices and install the correct instance data. The export script is supposed to
1306 export instance data to stdout in a format understandable by the the import
1307 script. The data will be compressed by ganeti, so no compression should be
1308 done. The rename script should only modify the instance's knowledge of what
1311 Other declarative style features
1312 ++++++++++++++++++++++++++++++++
1314 Similar to Ganeti 1.2, OS specifications will need to provide a
1315 'ganeti_api_version' containing list of numbers matching the version(s) of the
1316 api they implement. Ganeti itself will always be compatible with one version of
1317 the API and may maintain retrocompatibility if it's feasible to do so. The
1318 numbers are one-per-line, so an OS supporting both version 5 and version 20
1319 will have a file containing two lines. This is different from Ganeti 1.2, which
1320 only supported one version number.
1322 In addition to that an OS will be able to declare that it does support only a
1323 subset of the ganeti hypervisors, by declaring them in the 'hypervisors' file.
1329 We might want to have a "default" import/export behaviour that just dumps all
1330 disks and restores them. This can save work as most systems will just do this,
1331 while allowing flexibility for different systems.
1333 Environment variables are limited in size, but we expect that there will be
1334 enough space to store the information we need. If we discover that this is not
1335 the case we may want to go to a more complex API such as storing those
1336 information on the filesystem and providing the OS script with the path to a
1337 file where they are encoded in some format.
1344 The first Ganeti RAPI was designed and deployed with the Ganeti 1.2.5 release.
1345 That version provide Read-Only access to a cluster state. Fully functional
1346 read-write API demand significant internal changes which are in a pipeline for
1349 We decided to go with implementing the Ganeti RAPI in a RESTful way, which is
1350 aligned with key features we looking. It is simple, stateless, scalable and
1351 extensible paradigm of API implementation. As transport it uses HTTP over SSL,
1352 and we are implementing it in JSON encoding, but in a way it possible to extend
1353 and provide any other one.
1358 The Ganeti API implemented as independent daemon, running on the same node
1359 with the same permission level as Ganeti master daemon. Communication done
1360 through unix socket protocol provided by Ganeti luxi library.
1361 In order to keep communication asynchronous RAPI process two types of client
1364 - queries: sever able to answer immediately
1365 - jobs: some time needed.
1367 In the query case requested data send back to client in http body. Typical
1368 examples of queries would be list of nodes, instances, cluster info, etc.
1369 Dealing with jobs client instead of waiting until job completes receive a job
1370 id, the identifier which allows to query the job progress in the job queue.
1371 (See job queue design doc for details)
1373 Internally, each exported object has an version identifier, which is used as a
1374 state stamp in the http header E-Tag field for request/response to avoid a race
1378 Resource representation
1379 +++++++++++++++++++++++
1381 The key difference of REST approach from others API is instead having one URI
1382 for all our requests, REST demand separate service by resources with unique
1383 URI. Each of them should have limited amount of stateless and standard HTTP
1384 methods: GET, POST, DELETE, PUT.
1386 For example in Ganeti case we can have a set of URI:
1387 - /{clustername}/instances
1388 - /{clustername}/instances/{instancename}
1389 - /{clustername}/instances/{instancename}/tag
1390 - /{clustername}/tag
1392 A GET request to /{clustername}/instances will return list of instances, a POST
1393 to /{clustername}/instances should create new instance, a DELETE
1394 /{clustername}/instances/{instancename} should delete instance, a GET
1395 /{clustername}/tag get cluster tag
1397 Each resource URI has a version prefix. The complete list of resources id TBD.
1399 Internal encoding might be JSON, XML, or any other. The JSON encoding fits
1400 nicely in Ganeti RAPI needs. Specific representation client can request with
1401 Accept field in the HTTP header.
1403 The REST uses standard HTTP as application protocol (not just as a transport)
1404 for resources access. Set of possible result codes is a subset of standard HTTP
1405 results. The stateless provide additional reliability and transparency to
1412 With the write functionality security becomes much bigger an issue. The Ganeti
1413 RAPI uses basic HTTP authentication on top of SSL connection to grant access to
1414 an exported resource. The password stores locally in Apache-style .htpasswd
1415 file. Only one level of privileges is supported.
1418 Command line changes
1419 ~~~~~~~~~~~~~~~~~~~~
1421 Ganeti 2.0 introduces several new features as well as new ways to
1422 handle instance resources like disks or network interfaces. This
1423 requires some noticable changes in the way commandline arguments are
1426 - extend and modify commandline syntax to support new features
1427 - ensure consistent patterns in commandline arguments to reduce cognitive load
1429 The design changes that require these changes are, in no particular
1432 - flexible instance disk handling: support a variable number of disks
1433 with varying properties per instance,
1434 - flexible instance network interface handling: support a variable
1435 number of network interfaces with varying properties per instance
1436 - multiple hypervisors: multiple hypervisors can be active on the same
1437 cluster, each supporting different parameters,
1438 - support for device type CDROM (via ISO image)
1440 As such, there are several areas of Ganeti where the commandline
1441 arguments will change:
1443 - Cluster configuration
1445 - cluster initialization
1446 - cluster default configuration
1448 - Instance configuration
1450 - handling of network cards for instances,
1451 - handling of disks for instances,
1452 - handling of CDROM devices and
1453 - handling of hypervisor specific options.
1455 There are several areas of Ganeti where the commandline arguments will change:
1457 - Cluster configuration
1459 - cluster initialization
1460 - cluster default configuration
1462 - Instance configuration
1464 - handling of network cards for instances,
1465 - handling of disks for instances,
1466 - handling of CDROM devices and
1467 - handling of hypervisor specific options.
1469 Notes about device removal/addition
1470 +++++++++++++++++++++++++++++++++++
1472 To avoid problems with device location changes (e.g. second network
1473 interface of the instance becoming the first or third and the like)
1474 the list of network/disk devices is treated as a stack, i.e. devices
1475 can only be added/removed at the end of the list of devices of each
1476 class (disk or network) for each instance.
1478 gnt-instance commands
1479 +++++++++++++++++++++
1481 The commands for gnt-instance will be modified and extended to allow
1482 for the new functionality:
1484 - the add command will be extended to support the new device and
1486 - the modify command continues to handle all modifications to
1487 instances, but will be extended with new arguments for handling
1490 Network Device Options
1491 ++++++++++++++++++++++
1493 The generic format of the network device option is:
1495 --net $DEVNUM[:$OPTION=$VALUE][,$OPTION=VALUE]
1497 :$DEVNUM: device number, unsigned integer, starting at 0,
1498 :$OPTION: device option, string,
1499 :$VALUE: device option value, string.
1501 Currently, the following device options will be defined (open to
1504 :mac: MAC address of the network interface, accepts either a valid
1505 MAC address or the string 'auto'. If 'auto' is specified, a new MAC
1506 address will be generated randomly. If the mac device option is not
1507 specified, the default value 'auto' is assumed.
1508 :bridge: network bridge the network interface is connected
1509 to. Accepts either a valid bridge name (the specified bridge must
1510 exist on the node(s)) as string or the string 'auto'. If 'auto' is
1511 specified, the default brigde is used. If the bridge option is not
1512 specified, the default value 'auto' is assumed.
1517 The generic format of the disk device option is:
1519 --disk $DEVNUM[:$OPTION=$VALUE][,$OPTION=VALUE]
1521 :$DEVNUM: device number, unsigned integer, starting at 0,
1522 :$OPTION: device option, string,
1523 :$VALUE: device option value, string.
1525 Currently, the following device options will be defined (open to
1528 :size: size of the disk device, either a positive number, specifying
1529 the disk size in mebibytes, or a number followed by a magnitude suffix
1530 (M for mebibytes, G for gibibytes). Also accepts the string 'auto' in
1531 which case the default disk size will be used. If the size option is
1532 not specified, 'auto' is assumed. This option is not valid for all
1534 :access: access mode of the disk device, a single letter, valid values
1537 - *w*: read/write access to the disk device or
1538 - *r*: read-only access to the disk device.
1540 If the access mode is not specified, the default mode of read/write
1541 access will be configured.
1542 :path: path to the image file for the disk device, string. No default
1543 exists. This option is not valid for all disk layout types.
1548 To add devices to an already existing instance, use the device type
1549 specific option to gnt-instance modify. Currently, there are two
1550 device type specific options supported:
1552 :--net: for network interface cards
1553 :--disk: for disk devices
1555 The syntax to the device specific options is similiar to the generic
1556 device options, but instead of specifying a device number like for
1557 gnt-instance add, you specify the magic string add. The new device
1558 will always be appended at the end of the list of devices of this type
1559 for the specified instance, e.g. if the instance has disk devices 0,1
1560 and 2, the newly added disk device will be disk device 3.
1562 Example: gnt-instance modify --net add:mac=auto test-instance
1567 Removing devices from and instance is done via gnt-instance
1568 modify. The same device specific options as for adding instances are
1569 used. Instead of a device number and further device options, only the
1570 magic string remove is specified. It will always remove the last
1571 device in the list of devices of this type for the instance specified,
1572 e.g. if the instance has disk devices 0, 1, 2 and 3, the disk device
1573 number 3 will be removed.
1575 Example: gnt-instance modify --net remove test-instance
1580 Modifying devices is also done with device type specific options to
1581 the gnt-instance modify command. There are currently two device type
1584 :--net: for network interface cards
1585 :--disk: for disk devices
1587 The syntax to the device specific options is similiar to the generic
1588 device options. The device number you specify identifies the device to
1591 Example: gnt-instance modify --disk 2:access=r
1596 Ganeti 2.0 will support more than one hypervisor. Different
1597 hypervisors have various options that only apply to a specific
1598 hypervisor. Those hypervisor specific options are treated specially
1599 via the --hypervisor option. The generic syntax of the hypervisor
1600 option is as follows:
1602 --hypervisor $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE]
1604 :$HYPERVISOR: symbolic name of the hypervisor to use, string,
1605 has to match the supported hypervisors. Example: xen-pvm
1607 :$OPTION: hypervisor option name, string
1608 :$VALUE: hypervisor option value, string
1610 The hypervisor option for an instance can be set on instance creation
1611 time via the gnt-instance add command. If the hypervisor for an
1612 instance is not specified upon instance creation, the default
1613 hypervisor will be used.
1615 Modifying hypervisor parameters
1616 +++++++++++++++++++++++++++++++
1618 The hypervisor parameters of an existing instance can be modified
1619 using --hypervisor option of the gnt-instance modify command. However,
1620 the hypervisor type of an existing instance can not be changed, only
1621 the particular hypervisor specific option can be changed. Therefore,
1622 the format of the option parameters has been simplified to omit the
1623 hypervisor name and only contain the comma separated list of
1626 Example: gnt-instance modify --hypervisor
1627 cdrom=/srv/boot.iso,boot_order=cdrom:network test-instance
1629 gnt-cluster commands
1630 ++++++++++++++++++++
1632 The command for gnt-cluster will be extended to allow setting and
1633 changing the default parameters of the cluster:
1635 - The init command will be extend to support the defaults option to
1636 set the cluster defaults upon cluster initialization.
1637 - The modify command will be added to modify the cluster
1638 parameters. It will support the --defaults option to change the
1643 The generic format of the cluster default setting option is:
1645 --defaults $OPTION=$VALUE[,$OPTION=$VALUE]
1647 :$OPTION: cluster default option, string,
1648 :$VALUE: cluster default option value, string.
1650 Currently, the following cluster default options are defined (open to
1653 :hypervisor: the default hypervisor to use for new instances,
1654 string. Must be a valid hypervisor known to and supported by the
1656 :disksize: the disksize for newly created instance disks, where
1657 applicable. Must be either a positive number, in which case the unit
1658 of megabyte is assumed, or a positive number followed by a supported
1659 magnitude symbol (M for megabyte or G for gigabyte).
1660 :bridge: the default network bridge to use for newly created instance
1661 network interfaces, string. Must be a valid bridge name of a bridge
1662 existing on the node(s).
1664 Hypervisor cluster defaults
1665 +++++++++++++++++++++++++++
1667 The generic format of the hypervisor clusterwide default setting option is:
1669 --hypervisor-defaults $HYPERVISOR:$OPTION=$VALUE[,$OPTION=$VALUE]
1671 :$HYPERVISOR: symbolic name of the hypervisor whose defaults you want
1673 :$OPTION: cluster default option, string,
1674 :$VALUE: cluster default option value, string.
1677 Functionality changes
1678 ---------------------
1680 The disk storage will receive some changes, and will also remove
1681 support for the drbd7 and md disk types. See the
1682 design-2.0-disk-changes document.
1684 The configuration storage will be changed, with the effect that more
1685 data will be available on the nodes for access from outside ganeti
1686 (e.g. from shell scripts) and that nodes will get slightly more
1687 awareness of the cluster configuration.
1689 The RAPI will enable modify operations (beside the read-only queries
1690 that are available today), so in effect almost all the operations
1691 available today via the ``gnt-*`` commands will be available via the
1694 A change in the hypervisor support area will be that we will support
1695 multiple hypervisors in parallel in the same cluster, so one could run
1696 Xen HVM side-by-side with Xen PVM on the same cluster.
1701 There will be a number of minor feature enhancements targeted to
1702 either 2.0 or subsequent 2.x releases:
1704 - multiple disks, with custom properties (read-only/read-write, exportable,
1708 These changes will require OS API changes, details are in the
1709 design-2.0-os-interface document. And they will also require many
1710 command line changes, see the design-2.0-commandline-parameters