backend: Check paths and always write CA file for import/export daemon
Once the import/export daemon uses separate users, the node daemon file (whichis used for intra-cluster transfers) might not be readable anymore. Alwayswriting it to a daemon-specific file will make this easier....
Add optional “salt” argument to utils.{,Verify}Sha1Hmac
We're using salted hashes all over the place.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Use utility functions for HMAC
HMAC will be used in more places. Centralizing some parts can't hurt.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
LUExportInstance: Move exporting code to helper class
This will simplify the implementation of intra-cluster instanceexports and reduces the number of local variables inLUExportInstance.Exec.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.1'
verify cluster: check /etc/hosts consistency
If we are told to modify /etc/hosts, then verify its consistency duringcluster verify.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Cleanup /etc/hosts during node removal
It seems that commit d8470559 dropped the cleanup of /etc/hosts when anode is removed from the cluster. I don't know for sure, but it seemsaccidental. As long as we add it to /etc/hosts, we should clean it uptoo....
Move cmdlib._VerifyCertificate to utils
This function will also be useful for inter-cluster instancemoves for verifying certificates.
KVM: make tags available in KVM_NET_SCRIPT
Make instance tags available as a space-separated list during the execution ofthe network setup script. This allows tag-based control of things like firewallrules and/or networking setup.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Remove "ssconf.CheckMasterCandidate"
This function is not used anymore, so there's no point in keeping itaround.
This reverts commit 3f71b464ad5cdd1f1b53f2a31a4eef4e2a5550cc, apart froma one empty line conflict in ssconf.py
Add wrapper class for signal.set_wrapper_fd
Managing file descriptors is always a bit tricky. Having this in a separateclass is better.
Conflicts: lib/utils.py: Trivial
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Remove two unused RPC functions
Both of these functions, “snapshot_export” and “instance_os_import”,have been replaced by the instance import/export daemon.
cmdlib: Convert instance import to new style
Use instance data transfer utility to import instances.
cmdlib: Convert instance export to new style
With this change, exports of several disks will occur in parallel. Errorhandling has also been improved.
cmdlib: Add utility to transfer instance data within the cluster
This is yet another wrapper around the instance import/export utilityclasses, providing an even simpler API for instance imports/exports withinthe same cluster.
cmdlib: Add utility for instance data import/export
Interpreting the backend's import/export daemon status is a bit tricky.This utility code keeps track of multiple transfers at the same time.Users can supply callback functions to react to events.
Timeouts are currently hardcoded. Intra-cluster instance moves will likely...
Factorize LUExportInstance
Apart from moving parts of the code to separate functions, error handlingis also improved by making sure snapshots are always removed.
Use ints instead of strings to represent user-ids
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Complete what was begun in commit ef40fbfb4
The X509 certificate RPC functions were also renamed, but Iforgot to rename them in rpc.py. Fortunately they aren't usedyet.
Add RPC call to send SIGTERM to import/export daemon
This will be used to stop the daemon without doing complete cleanup (yet).
Rename import/export RPC calls to match others
Fix broken commit 9e302a8
Commit 9e302a8 split the StopInstance function in two without properlyduplicating the local variables.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
ssh: make quiet configurable
Currently both CopyFileToNode and BuildCmd hardcode "-q" in the options.This patch moves this setting as a configurable option to_BuildSshOptions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
SSH: do not check IPs
Since we use the cluster name for the SSH known_hosts file, ssh willalways detect a changed IP (since we never connect to the cluster mastername, but the node names), and will complain about it/try to update theuser known hosts file (since that is /dev/null, it doesn't matter, but...
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
hv_chroot: move unmount to CleanupInstance()
This allows cleanup to be done properly if the "instance" wasn't runningat all (based on the CleanupInstance framework, instead of the retry inhypervisor).
LUSetClusterParams: initialize needed parameters
… since the opcode doesn't auto-initialize to None.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Add a hypervisor constant for migration support
This variable can be used by other tools to determine in a generic waywhether a given hypervisor supports migration or not.
Fix indentation error
Commit 9cf4321fc39ec36359d9c90b22b36d33b6adc2f4 indented some lines by4 spaces rather than 2, and was git-amed without noticing. Fixing.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
Add -usbdevice tablet to KVM when using vnc
When using VNC, it is recommended to use a tablet-styleinput device instead of a mouse. This allows most VNC viewersto send proper mouse coordinates to the virtual machine's desktopresulting in perfectly aligned guest and host mouse pointers....
Only use boot=on on non-ide disks only (KVM)
boot=on implies that KVM boots using extboot. This is only requiredto boot non-IDE disks and has the side-effect that there is at mostone bootable device. This behaviour breaks some operating systems,most notably the windows installer that tries to chainload the hard-disk...
KVM: fix a bug in _TryReadUidFile
If the uid pool is not in use _TryReadUidFile will try to return "uid" even if it hasn't tried to read it at all.
utils: Add function to read locked PID file
This is useful in combination with utils.StartDaemon and will be used forreading the import/export daemon's PID file.
Add ganeti.masterd module
This will be used to keep masterd-specific code.
Fix RAPI client methods return values
Ensure that all RAPI client methods that should return job ids actuallyreturn job ids.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>
Conflicts: doc/security.rst trivial lib/cli.py trivial
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
KVM: implement the HT_SM_POOL security model
In order not to complicate to much the _ExecuteKVMRuntime function withnested try/except/finally/else constructs we move the actualruncmd+check call in a separate _RunKVMCmd function.
KVM: add an instance uid file concept
If this file exists, for an instance, we release the uid containedinside back to the uid pool, at instance shutdown.
ReleaseUid: accept an integer as uid
We can't guarantee to keep the LockedObject around, when ReleaseUid iscalled (because that might happen, for example, after a ganetishutdown/restart). So we'll support releasing on just an integer uid.
LockedUid: add GetUid() method
uidpool users that get passed a LockedUid by the uidpool need to knowwhich one the uid is. Since it's not nice to make them access a privatemember, and we shouldn't make "uid" public either, we'll add a gettermethod.
KVM: move post-shutdown cleanup operations
Currently putting the cleanup just after the actual shutdown ensuresthat it never gets called, due to how the retry/shutdown cycle works.So we move those cleanups to their new dedicated place.
Add CleanupInstance hypervisor call
Currently some hypervisors (namely kvm) need to do some cleanup aftermaking sure an instance is stopped. With the moving of the retry cyclein backend those cleanups were never done. In order to solve this we adda new optional hypervisor function, CleanupInstance, which gets called...
Fix changing the list of enabled hypervisors
When enabling a new hypervisor, we must ensure that we have itsparameters in the cluster (global) hvparams dictionary.
Furthermore, we add a verify check for this case (this should be betterreplaced with an auto-fix?)....
Fix heading to the one of epydoc
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix a couple of pydoc syntax problems
Readd node: allow changed primary IP
The primary IP is not used for DRBD disks, only for the job queue andinternal commands. As such, it's trivial (with a quiet job queue) tore-ip the node.
The patch just relaxes the checks in LUAddNode, and manual testing shows...
Add RPC calls to import and export instance data
These RPC calls can be used to start, monitor and stop the instance dataimport/export daemon.
Add daemon for instance import and export
This backend daemon for instance import and export will be used totransfer instance data to other machines. It is implemented in a genericway to support different ways of data input and output. The third-partyprogram “socat”, which is already used by the KVM hypervisor abstraction,...
Add ExecuteWithUnusedUid() to uidpool.py
Manage the assignment of uids from the uid pool
Add uid_pool to ssconf
Add --uid-pool option to gnt-cluster init
Add printing of uid_pool to gnt-cluster info
Add --add-uids/--remove-uids to gnt-cluster modify
Add --uid-pool option to gnt-cluster modify
Add lib/uidpool.py module
Add uid_pool cluster parameter
backend: Consolidate code opening real block device
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
Fix cluster behaviour with disabled file storage
There are a few issues with disabled file storage:- cluster initialization is broken by default, as it uses the 'no' setting which is not a valid path- some other parts of the code require the file storage dir to be a...
Fix cfgupgrade with non-default DATA_DIR
Commit 43575108 added bootstrap.GenerateclusterCrypto and commit7506a7f1 changed cfgupgrade to use it. However, this lost thefunctionality of upgrading in non-default DATA_DIR.
To fix this, we enhance bootstrap.GenerateclusterCrypto to accept custom...
Improving the RAPI documentation
Merge remote branch 'devel-2.1'
Add a identify-defaults options for import
When importing an instance, all the saved valued will be used asexplicitly specified values, overriding the cluster defaults. This meansexport+import will change the status (from default to explicitlyspecified) of parameters....
Fix create/import verification of hvparams
Currently the instance creation checks the cluster hv defaults + the newparameters for validity, ignoring the os-specific hvparams (this was anoversight during the implementation of the os hvp). This patch uses the...
objects.Cluster: add method to get hv defaults
Currently the FillHV method is the one that does the cluster hvparams +os hvparams merger. However, in some cases we need to do just this,without adding the instance hvparams on top.
This patch adds a function to compute just this (hv + os hvp...
Reuse NIC information from export
If the user doesn't pass any nics in import, do not use a defaultone-nic, but instead read the nics from the export file as is.
Fortunately the export and the way nics are read from the command lineare compatible…
Signed-off-by: Iustin Pop <iustin@google.com>...
Reuse backend parameters from export
Similar to the previous patches, if we're missing some parameters andthe export has them (either in the new style or old-style), we reusethem.
Reuse disk information from export
If the user doesn't pass the disk information on import, automaticallyreuse the number and size of disks. This loses the iv_name attribute,but that is only cosmetic and cannot be changed by the user.
Reuse hypervisor parameters in import
If available, we reuse the parameters from the export info.
Read disk template from export info
This patch changes the instance import to read the disk templateautomatically from the export info, if the opcode doesn't alreadyspecify a disk template.
To do this, we have a couple of additional changes:
- change from required parameter to optional one for disk_template...
CreateInstance: separate the reading of the export
We move the reading of the export to a separate function, to simplifyCheckPrepreq and also read it earlier. This will allow building themissing opcode parameters from the export information, instead of...
Move code from ExpandNames to CheckPrereq
This is needed since only in CheckPrereq we have the nodes locked, andfuture import enhancements will need to have access to the export infoduring the parameter build.
CreateInstance: Move some code to CheckArguments
ExpandNames holds too much non-locking code (first LU to be converted toExpandNames, and we didn't have CheckArguments at that poin), and thispatch moves the checks that are lock-independent to CheckArguments....
Export more instance parameters in instance export
Currently the backend parameters are not exported automatically, butonly a few directly in the '[instance]' section. Hypervisor type andhypervisor parameters are not exported at all.
This patch creates two separate sections for the be and hv parameters,...
Export the nicparams too during instance export
The patch tries to export all params (based on the dict defined inconstants), using None for missing keys.
Handle errors better for wrong nic_count in export
This fixes an old 'FIXME' entry.
Fix new pylint errors
ConfdClient.SendRequest: allow max coverage
This patch changes the coverage parameter to allow specification of maxcoverage (via -1), versus auto-computation (default, 0) and manualspecification.
Unittests are updated for this case too.
Export the maintain_node_health option in ssconf
Add a new cluster parameter maintain_node_health
This will be used to conditionally enable the watcher node maintenancefeature.
Add a new confd callback (StoreResultCallback)
This new callback simply stores (without calling any lower-levelcallback) the last result; coupled with the filtering callback, thisensures that it has the 'best' response after all have been received.
The result can then be retrieved via the GetResponse method....
ConfdClient: add synchronous wait for replies mode
Currently, there is no way for a user of the confd client library toknow how many replies there should be, whether all have been received,etc. This is bad since we can't reliably detect the consistency of the...
ConfdClient: unify some internal variables
Currently the requests are tracked in _request and in _expire_requests.This is conventient, but it restricts the ability to extend the requesttracking, e.g. via packet stats and/or extension of expiration time....
Fix consistency checks in ConfdFilterCallback
Commit 49b3fda added consistency checks, but these are wrongly triggeredfor old responses - we need to make sure to check that we have the sameserial.
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
Fix bug introduced in 76e5f8b54: mkdir mode
After commit 76e5f8b54, mkdir_mode in utils.RenameFile isno longer passed to Makedirs. This is fixed by this patch.
utils: Move wrapper code around os.makedirs into separate function
Adding RAPI client library.
Signed-off-by: David Knowles <dknowles@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Signed-off-by: Iustin Pop <iustin@google.com>(modified slightly the unittest to account for missing httplib2 library)
Extend ConfdFilterCallback with consistency checks
Note that users of the callback will have to manually check theattribute.