Ensure that mcpu always works with lists of locks
Apparently sometimes unicode strings appear out of nowhere (maybe fromthe RPC calls) and then they are used as a list of characters (hailPython). This patch adds a function that ensures we always have a...
Upgrade Roman numerals support
Ganeti prouds itself of its really good retro-compatibility and API stability.Some of our users haven't upgraded their hardware in the last 2000 years (onecentury more, one century less) and their Xen-PVM (Parchment and VelluM) does...
Acquire config lock for RPC calls that check consistency
Since LUClusterVerifyGroup issues 3 RPC calls to get information fromnodes, and during every of these calls, the configuration can bemodified by somebody else, it's necessary to acquire a shared lock on...
Make ConfigWriter internal lock re-entrant
The ConfigWriter now allows nested locking calls, and acquires(releases) the config lock from WConfd only the first (last) time.
This is needed for blocks that acquire a lock on the configuration anduse synchronized configuration calls inside. While this should be...
Improve the heuristic of guessing masterd's livelock file
As, during startup, masterd uses livelock files for specialtasks, the lexicographically last livelock file for masterdis the masterd_check_agreement_ file. However, when debugging,we want the primary livelock file. Hence extend the prefix to...
Make gnt-debug wconfd listlocks use only job ids (again)
Now that the product has been replaced by the sum we can, again,identify jobs only by their job id.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Use thread name as identifier for non-job tasks
Non-jobs requesting resources need to identify themselves by a stringdescribing uniquely what they do. For the time being (while masterdstill exists), these non-jobs are client requests to masterd. Fortunately,...
Merge branch 'stable-2.11' into master
Merge branch 'stable-2.10' into stable-2.11
Merge branch 'stable-2.9' into stable-2.10
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Merge branch 'stable-2.8' into stable-2.9
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix configuration calls for LUBackupExport
Re-read the instance in-memory objects from WConfd after calls thatchange its state.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix configuration calls for LUInstanceRename
Re-read the instance in-memory objects from WConfd after callsthat change its state.
Fix configuration calls for LUInstanceReboot
Fix configuration calls for LUInstanceReinstall
Fix configuration calls for LUInstanceShutdown
Fix configuration calls for LUInstanceStartup
Instance storage: Mark functions that modify instances
.. in the configuration with comments.
Also re-read an instance object where needed.
When modifying an instance status, return the instance
This simplifies code that works with instance objects and needs to havetheir fresh values after performing such an operation.
Fix calls to Update in TLMigrateInstance
Instead of modifying the primary node on an object and calling 'Update',a new method was added to ConfigWriter.
Fix calls to Update in LUNodeSetParams
Add calls to Update after modifying the node's configuration and movesome of the modifications so that they don't conflict with other callsto ConfigWriter.
Fix calls to Update in LUInstanceSetParams
In particular, move the call to Update so that it doesn't conflict withcalls to ConfigWriter methods that change the configuration state.
Fix calls to Update in LUClusterSetParams
Since this LU intersperses calls ConfigWriter methods with modifyingobjects directly, it's necessary to call 'Update' and/or re-read them atappropriate places.
Also a new ConfigWriter method is created for modifying the instance...
Move master failover code that uses ConfigWriter
.. so that it's in the scope of a running WConfd daemon. This is nowrequired as read-only calls invoke WConfd as well.
Make configuration per job/thread
Previously there was one shared configuration object for all jobs,threads and other tasks. This patch creates separate ConfigWriteinstances for distinct jobs/threads.
All exported methods of ConfigWriter are now wrapped in calls that...
Use explicit configuration for nodes in GanetiContext
This way, each call to AddNode or RemoveNode uses a passed configurationobject that belongs to the appropriate job.
This is required to subsequently get rid of the single sharedConfigWriter object....
Remove GanetiContext reference from ConfigWriter
It was only required for RPC calls, which are now handled by WConfD.
Add _UnlockedDropECReservations to ConfigWriter
There was just a locking version, and to keep things consistent in_OpenConfig, we need also a non-locking one.
Save the configuration after releasing an exclusive lock
In most cases, this is what has been already implemented manually.In some cases, the configuration write is issued even if theconfiguration hasn't been modified, but since WConfd checks for suchcases, neither file write nor distribution to nodes is performed....
Expand the Python configuration synchronization decorator
.. so that it can be enhanced later.
Add a designated method for synchronizing the configuration
This will be needed later to add WConfd locking for the synchronization.
Refer to configuration using a method instead of a variable
This is a prerequisite for having thread-local configuration states.
Fix specification of TIDiskParams
Commit 580b1fdd incorrectly assumes that disk parameters arejust the standard ones, whereas the man page explicitly statesthat additional parameters can be passed as well, if they makesense for the chosen storage type. Fix this....
Add renaming of instance custom params
Much like the groups before, this patch allows custom_* params to besubmitted under the same name they can be retrieved as in the infocall.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Add renaming of group custom ndparams, ipolicy, diskparams
This patch adds the ability to set the group-specific parameters in thesame way they are described when returned by the info command - withthe "custom_" prefix.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Add aliases for nodes
This patch adds a single alias for the secondary_ip property of thenode.
Add aliases for cluster parameters
This patch adds aliases for two cluster parameters.
Add support for value aliases to RAPI
This patch extends the metaclass used to generate RAPI handlers toallow creating aliases of certain values returned by GET methods.
Add reason parameter to RAPI client functions
Only the functions for starting, stopping and rebooting a VM had a reasonparameter. Now, all the RAPI client functions generating opcodes do.
Also, one test is expanded to verify that a RAPI request with both body and...
Remove excess new line in mcpu.py
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Export VLAN nicparam to NIC configuration scripts
The vlan nicparam is used in openvswitch mode. This patchexports it as VLAN environment variable so that NICconfiguration scripts can later use it.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Centralize the creation of a WConfd context in Python code
This will allow easier modification of the structure of a clientidentity later.
Also add a helper method for creating a WConfd context from a context.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
kvm: Add migration capabilities as an hvparam
Latest QEMU versions support various migration capabilities. Eachcan be enabled/disabled with 'migrate_set_capability' monitorcommand.
Version 1.7.0 defines x-rdma-pin-all, auto-converge, zero-blocks,and xbzrle migration capabilities....
Make watcher submit queries low priority
Make the watcher collect its data using low-priority jobs,to avoid blocking user/admin jobs. Note that repair jobs arestill submitted normal priority. Fixes issue 772.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add a function for getting the list of candidate certs
.. to ConfigWriter as well.
Move utility functions for candidate certs. to ConfigWriter
In particular AddNodeToCandidateCerts and RemoveNodeFromCandidateCerts.
Calling 'cfg.Update(cluster)' causes problems in WConfd, asit doesn't operate on a shared configuration object any more....
Show OS variant information in gnt-os info
Currently, the non-standard/modified per-OS hypervisor parameters, orOS specific parameters can be listed only by the 'gnt-cluster info'command, which is a non-standard place to show them. Extend the'gnt-os info' command to display the available/supported OS variants...
Skip rename when OS scripts are absent
When an instance does not have OS scripts because, for example, ituses an OS image, do not rename the instance after an import.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Use raw disks in import/export when OS scripts are absent
When an instance does not have OS scripts because, for example, it hasan OS image, then the import/export should not try to run the OSscripts. Instead, it should use raw import/export.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>...
Add the thread identifier parameter to gnt-debug listlocks
.. to comply with the updated WConfd interface.
This makes the call less useful as the thread ID is often unknown tousers. This needs to be improved in the future.
Add a thread ID to the WConfd client id
This allows to distinguish threads that don't have a job id, which isneeded for answering queries.
Since Python thread IDs aren't guaranteed to be unique, in future it'dbe preferable to use a different, unique identifier....
Fix conflict between virtio + spice or soundhw
With regard to PCI slot occupied by a KVM instance we haveobserved the following:
1) Slot 0 will always be Host bridge.2) Slot 1 will always be ISA bridge.3) Slot 2 will always be VGA controller (even with -display none)....
Fix bitarray ops wrt PCI slots
Introduce new method `_GetFreeSlot()` responsible only for bitarrayoperations. It fixes search in case of bitarray is either '0000..'or '1111..'.
Use it instead of `_UpdatePCISlots()` and in `_GetFreePCISlot()`.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
Remove unused functions to check OS variants
... as this is now performed on the node.
Signed-off-by: Jose A. Lopes <jabolopes@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Remove unused RPC 'os_get'
... include the RPC post process hook.
Remove calls to 'CheckNodeHasOS'
... because 'CheckOSParams' already checks the OS variant.
Add 'force_variant' to RPC 'os_validate'
Move function 'CheckOSVariant' to the node and add parameter'force_variant' to RPC 'os_validate', thus making the node verify theOS variant together with the rest of the OS params.
Remove SSH copyfile from LU and assume the file exists
According to Ganeti design, files should not be copied from master tonodes and instead they are assumed to exist and it is the user'sresponsibility to assure that the file does exist.
Fix OS image detection on master
Extend '_DumpDevice' to enable/disable file truncation
... because when the data source is infinite, truncation is notnecessary, but when the data source is finite and is, for example,smaller than the device, truncation can reduce the disk size.
Fix disk truncation in download and dump OS images
Check if OS image exists on the node before dumping
Make mcpu acquire WConfD locks
So far, the mcpu acquires locks that live in memoryof masterd. This design does not fit with our jobs-as-processesgoal. So make mcpu acquire the corresponding locks in WConfDinstead.
Note that this implies changes in various other files that call...
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Rename compression option in gnt-backup export
The gnt-backup export option --compress did not do what most wouldexpect upon seeing the name: compress the exported image. Instead, itused compression to try and speed up the transfer of the image,decompressing it prior to writing the file. To reduce confusion, this...
Instance reinstall with OS images
OS images in 'LUInstanceCreate' and OS scripts optional
Instance create with OS image
Extend 'LUInstanceCreate' to image the instance's first disk if an OSimage is specified via the OS params. If the OS image is a file, itwill be copied to the node via SSH. If ths OS image is a URL, it willbe passed directly to the node, which will then download the file....
Function to check if the OS image parameter is valid
Add helper function to check if the OS image parameter contained inthe OS parameters of an opcode is valid.
Function to image disks while ensuring that disks are paused
Function to remove instance if disks are degraded
RPC 'blockdev_image' to image devices
Add RPC 'blockdev_image' that uses 'ganeti.backend.BlockdevImage' todump an image to an instance's disk device, optionally downloadingthat image.
Helper function to image a device by downloading or dumping
Add 'BlockdevImage' which downloads a file and dumps it to aninstance's disk if the path is a URL, otherwise it dumps the filedirectly to the instance's disk.
Helper function that downloads an image and dumps it to disk
Generalize 'WipeDevice' to 'DumpDevice'
Helper functions to get and update OS image from OSParams
Add 'GetOSImage' and 'PutOSImage' which handle the OS image key in theOS parameters dict.
Fix export order according to definition order
Fix docstrings
Fix several docstrings.
Reuse method to parse name from OS 'name+variant' string
Fix error introduced during merge
A parameter was lost while resolving a conflict in the signature of a function.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
gnt-cluster copyfile: accept relative paths
If, on the command line, the argument to gnt-cluster copyfile isa relative path, consider this a shorthand for the correspondingabsolute path.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Merge branch 'origin/stable-2.8' into stable-2.9
Let WConfd distribute SSConf to nodes
.. and remove the corresponding code from lib/config.py.
Let the SSConf RPC server side handle lists
Since on Haskell side we represent SSConf as list of lines, let thePython side understand it as well.
Let WConfd distribute the configuration to MCs
.. and remove the distribution from lib/config.py
Add a new RPC server call for uploading a single file
The server side processes the request exactly the same as for"upload_file".
Unlike "upload_file", the new call "upload_file_single" declares allrequired fields without requiring additional preprocessing....
Add more meaningful error messages to asserts in vcluster
.. to simplify debugging of RPC calls.
Improve RAPI detection of the watcher
If the watcher is not allowed to access RAPI, it doesn't mean that it is deadand needs to be restarted.
Fixes Issue 752.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Enable a timeout for instance shutdown
Add the timeout parameter to the StopInstance function of the hypervisor baseclass and to all its implementations.
Also, change the tests as required by this change.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Allow KVM commands to have a timeout
Modify the function that sends commands to the KVM monitor so that it ispossible to specify an optional timeout after which the command is killed.
Allow xen commands to have a timeout
Modify the function that runs Xen commands so that it is possible to specify anoptional timeout after which the command is killed.
Fix wrong docstring
Fields must be the final elements in an epytext string.
Use correct lockfile for gnt-debug wconfd
As jobs are currently running in masterd, use the masterd livelockfile.
Add utility to guess livelock file for an owner
As livelock files are constructed in a systematic manner,we can guess what the livelock file for a given owner is.While this will not necessarily work perfectly, it will beuseful to simplify direct debugging of WConfD....