lvmstrap: also test sysfs holders
If a device has entries in its holder directory(/sys/block/$name/holders), it means that some kernel system "uses" that device, and hence should not be considered available.
This patch adds a new 'in-use' check based on this sysfs test, and...
lvmstrap: add support for non-partitioned md disks
This patch, originally written by Marc Schmitt <mschmitt@google.com>,adds support for MD devices (used in a non-partitioned mode). Iabstracted all the original startswith('md') checks into separatefunctions, and also moved the supported disk types to a list....
Merge branch 'devel-2.3'
cfgupgrade: Remove unused “program” variable
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
cfgupgrade: Check master name, clarify question
- Check hostname and abort if it doesn't match contents of “ssconf_master_node”, can be overridden using “--ignore-hostname” parameter.- Clarify confirmation question and don't mention instances anymore....
Fix build errors with ganeti-listrunner
- Remove non-ASCII character from manpage- Reformat docstring for epydoc in script
These caused build breakage on some but not all distributions.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Initial import of listrunner
This tool was used and worked on internally for quite a long time. Wedecided to include it in Ganeti.
Known issues:- Code doesn't match rest of Ganeti (e.g. using “print” all over the place, hardcoded calls to sys.exit deep in functions)...
Remove editor-specific settings from cfgupgrade*
Move “rapi_users” file into separate directory
This reduces the number of notifications in “ganeti-rapi”. Until now itwas notified for every change in …/lib/ganeti and had to check whetherthe users file was affected. A symlink is always created in cfgupgrade...
Bump version for Ganeti 2.3
Also update cfgupgrade and NEWS.
setup-ssh: Better error reporting
Together with Michael we refactored the code to make it better andeasier error reporting. Without printing backtraces for authenticationand verification issues.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Adjust the error message of setup-ssh if join check fails
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Merge branch 'devel-2.2'
move-instance: Allow overriding instance parameters
When moving a single instance within the same cluster, the NICis not allowed to re-use an existing MAC address. To avoid this,NIC parameters must be overridden. BE, HV, OS and NIC parameterscan be overridden after applying this patch....
Change behaviour of OpDiagnoseOS w.r.t. 'valid'
This patch changes the behaviour of OpDiagnoseOS with regards to the'valid' field to be similar to the one for the hidden/blacklistedfields: unless this field is requested, invalid OSes are filtered out....
Merge branch 'stable-2.2' into devel-2.2
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Adding functionality to check feasability of joining the host
This checks if the host might possibily belong already to anothercluster. If this is the case we abort without any further actionunless we are forced by --force-join.
Signed-off-by: René Nussbaumer <rn@google.com>...
If we had any errors in setup in one of the hosts, exit with non-zero
Fix the output of the key fingerprint from binary to hex
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Revert "Add 2.2->2.3 cluster upgrade function"
This reverts commit 2c9cf6bb05e14ea606638fce9680994cf92e8e24.The change is better done in objects.py
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix some epydoc warnings
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
move-instance: Fix parameter order in _CreateInstance
This patch fixes a bug, which caused the primary and secondarydestination nodes to be revered on the destination cluster after aninstance move.
Signed-off-by: Manuel Franceschini <livewire@google.com>...
Add 2.2->2.3 cluster upgrade function
Revert behaviour for calling SetupNodeDaemon
This was previously done in setup-ssh which was the wrong place
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Adding host key verification to setup-ssh
setup-ssh: fix updating of authorized_keys
Due to what seems like a bug (or inconsistency) in paramiko, filesopened with a+ over SFTP need a seek() in order for the user to be ableto read data from them. We implement this, and rely on the fact that wedo iterate over all lines before writing and that the file is opened in...
setup-ssh: Also use keys from the ssh-agent
Currently, setup-ssh only uses one disk-based key. This means that anysetup where we use keys from ssh-agent (which do not necessarily existon disk) will break when moving from the old method to setup-ssh.
This patch moves the SSH key handling to separate functions, and uses...
setup-ssh: fix the logging error message
This took a bit of interesting debugging, but the issue seems to be thaton older python/logging/paramiko (I'm not sure which is at fault here),the other threads are finishing later than the main thread, so thelogging module closes the logging file (but doesn't unregister it)...
setup-ssh: only read the ssh port once
Then reuse it. With certain setups, this could be slow when pre-seedingan entire cluster.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
setup-ssh: redo the logging levels
There is not enough logged by default on stderr, and way too muchinformation in the log file.
Since we don't want to debug paramiko itself, we filter only warningsand above.
Otherwise, we make WARNINGS the default level, with verbose meaning INFO...
setup-ssh: try to use key auth first
This patch changes the setup-ssh workflow to try key authenticationfirst, and then fall-back to password authentication. The password isalso read lazily, with no prompts if we can authenticate via keys.
Signed-off-by: Iustin Pop <iustin@google.com>...
move-instance: Use constants for parameters
move-instance: Pass OS parameters to new instance
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
burning: fix handling of empty job sets
If we call burning with only existing instance, then it will fail tocreate any of them, and thus in the removal phase it won't have anythingto remove. Since calling luxi.SUBMIT_MULTIPLE_JOBS with an empty job set...
workerpool: Change signature of AddTask function to not use *args
By changing it to a normal parameter, which must be a sequence, we canstart using keyword parameters.
Before this patch all arguments to “AddTask(self, *args)” were passed asarguments to the worker's “RunTask” method. Priorities, which should be...
Fix burnin and live migration
This is breakage from the original 'live' parameter changes.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Disable 'invalid name' pylint warning for tools/setup-ssh
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Adding tool to setup SSH on a remote host
This prepares the remote node to be joined into a cluster
Add function to format all job log messages
Just calling utils.SafeEncode on the log message failed when itwasn't of the type ELOG_MESSAGE and not a string. Now non-messagelog entries are formatted using repr().
Merge branch 'devel-2.1'
Conflicts: qa/qa-sample.json (trivial) qa/qa_cluster.py (trivial)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
QA, burnin: allow selection of reboot types
After some more investigation, only the soft reboot type fails for Xen3.4 (due to the reboot/uptime time counter). As such, it's better toallow selective testing, since we do want to test in general theseopcodes/the command line script....
RAPI client: Switch to pycURL
Currently the RAPI client uses the urllib2 and httplib modules fromPython's standard library. They're used with pyOpenSSL in a very fragileway, and there are known issues when receiving large responses from a RAPIserver....
cfgupgrade: Add option to override config verification
This is needed in a new test script for cfgupgrade where it's betterto only use the minimum parts necessary for testing cfgupgrade.Including full configuration data to ensure verification doesn't fail...
cfgupgrade: Fix bug when checking configuration directory
In the condition for checking the configuration directory, one “or” should havebeen an “and”. This bug was in cfgupgrade since commit 95e4a8142 (June 2008).Found thanks to a test script.
cfgupgrade: Add support for Ganeti 2.2
Since configuration changes are now usually done in the configurationwriter class, cfgupgrade's only task is to change the config version.This means we can easily support upgrading from Ganeti 2.0 and 2.1 to2.2....
Remove dead code from cfgupgrade
Three constants were no longer user after commit 11c31f5cb5, whichremoved support for upgrading from Ganeti 1.2.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Fix burnin's export test
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add support for OS parameters during instance add
This is not yet complete, as it lacks proper support for instanceimport.
cfgupgrade: Local variable for cluster-domain-secret filename
This is necessary to allow cfgupgrade to work on a non-standard directory.
move-instance: Use error message instead of multiple state variables
Until now, move-instance used different status variables: “success”,“abort” and “error_message”. With this patch, everything is changedto use “error_message” only. This simplifies the code a bit....
Add tool to move instances between clusters
This is a first version of the instance move tool and it supports moving1..N instances from one cluster to another. When moving a single instance,the instance can be renamed, allowing for moves within the same cluster...
ganeti-cleaner: Remove expired X509 certs
Importing/exporting an instance to a remote machine creates X509certificates which expire after some time. They need to be removed fromthe nodes as they become useless.
cfgupgrade: allow working with relative paths
Since utils.WriteFile requires an absolute path, cfgupgrade needs tonormalize the directory it is given.
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
burnin: skip migration based on hypervisor support
The patch adds selection of hypervisor in burnin (on multi-hypervisorclusters, we always used the default) and optional hypervisorparameters, and skips migration if the hypervisor doesn't support it.
Fix cfgupgrade's use of GenerateSelfSignedSslCert
In master, the function GenerateSelfSignedSslCert has been moved toutils.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Conflicts: doc/security.rst trivial lib/cli.py trivial
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Using the added stop-all functionality from daemon-utils in cluster-merge
Additionally also move to the RunWhileStopped method for the configuration merger.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>...
Update docstrings in tools/ and enable epydoc
This patch updates the docstrings in tools/ (mostly in lvmstrap, whichis very very old code-base) and then enabled the tools in this directoryfor 'make apidoc' too.
Forward-port the ganeti 2.0 cfgupgrade
2.1's cfgupgrade doesn't support upgrading from 1.2, only from 2.0.However, it's trivial to forward-port the 2.0 cfgupgrade to work with2.1, thus providing an upgrade path for 1.2 users directly to 2.1,without the intermediate step of installing 2.0....
Add a new tool: sanitize-config
This can be used for two purposes:
- safety copy of the config file, with just the secrets changed- cleanup of the config file (full randomization), so that (e.g.) users could send a broken config file to the devel-list...
Fix cfgupgrade with non-default DATA_DIR
Commit 43575108 added bootstrap.GenerateclusterCrypto and commit7506a7f1 changed cfgupgrade to use it. However, this lost thefunctionality of upgrading in non-default DATA_DIR.
To fix this, we enhance bootstrap.GenerateclusterCrypto to accept custom...
Mark cluster-merge as executable
burnin.ExecOrQueue: remove variable argument list
In order to later add an optional parameter we transform the variableops argument list in an explicit list.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
burnin.ExecOrQueue: add post-process function
If a post-process function is passed to ExecOrQueue it is executed ifand only if the job is successful. This happens immediately if we'reproceding iteratively, and at the end, when we collect all job results,...
burnin: only remove instances we actually added
Currently burnin, if proceding in parallel, will remove all instanceswhich were passed, even if they failed to add. This is bad because itwill also remove instances which existed before burnin started. By...
Merge remote branch 'devel-2.1'
Abstract the confd client creation
Most creation of confd clients will do the same steps: read MC file,parse it, read HMAC key, etc. We abstract this functionality so thatwe don't duplicate the code.
Fix burnin error when trying to grow a file volume
Abstract the growable disk types in a ganeti constants, and only rundisk grow, from burnin, on them.
burnin: implement basic confd testing
Just a few queries are checked, but this should give us confidence thatat least the basic confd framework is working properly.
Burnin: don't add/remove routed nics
Add cluster domain secret
Information exchanged between different clusters via untrustedthird parties (e.g. for remote instance import/export) must besigned with a secret shared between all involved clusters toensure the third party doesn't modify the information....
Rightname confd's HMAC key
Currently, the ganeti-confd's HMAC key is called “cluster HMAC key” orsimply “HMAC key” everywhere. With the implementation of inter-clusterinstance moves, another HMAC key will be introduced for signing criticaldata. They can not be the same, so this patch clarifies the purpose of the...
cfgupgrade: Use new bootstrap function for certs and keys
Switch burnin to cli.JobExecutor
Burnin has a custom job executor, because of its need to retry some jobseries.
While we cannot replace all of it, at least the execution we can switchto cli.JobExecutor, to take advantage of the recently-introducedout-of-order waiting....
Improve burnin's Log function
This makes the Log function able to take multiple args for simplifiedmessage construction, similar to the ToStdout one.
Switch from os.path.join to utils.PathJoin
This passes a full burnin with lots of instances, and should be safe aswe mostly to join a known root (various constants) to a run-timevariable.
Make pylint happy
I was using a too old version which doesn't got all those. Thispatch is fixing the new lint errors.
Adding tool for automated cluster-merger
This is the implementation of docs/design-cluster-merger.rst. It allowsthe automatic merging of one or more clusters into the invoking cluster.
While this version is tested and working it still needs some tweaking...
Add debug mode to burnin
There are two entry points to job execution in burnin, ExecOp andExecOrQueue, and these are modified to call the new _SetDebug method onthe opcodes.
Fix dumpers/loaders after slots cleanup
Commit 154b958 changed (correctly) the slots usage, but this brokedumpers/loaders since we relied directly on the own class slotsfield.
To compensate, we introduce a simple function for computing the slots...
Add an early release lock/storage for disk replace
This patch adds an early_release parameter in the OpReplaceDisks andOpEvacuateNode opcodes, allowing earlier release of storage and moreimportantly of internal Ganeti locks.
The behaviour of the early release is that any locks and storage on all...
Fix unused imports or add silences where needed
In some cases pylint doesn't parse the import correctly, so we addsilences; but there are also many cases of unused imports, which wesimply remove.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Further pylint disables, mostly for Unused args
Many of our functions have to follow a given API, and thus we have tokeep a given signature, but pylint doesn't understand this. Therefore,we silence this warning.
The patch does a few other cleanups.
Convert to static methods (where appropriate)
Many methods are simple pure functions, and not depending on the objectstate. We convert these to staticmethods.
Remove more unused variables
This removes unused variables in the rest of the code (outside lib/).
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
burnin: move decorators out of classes
Similar to commit c881c5, we move the decorators out of classes, suchthat they become simple functions instead of methods. This more clean,since only the wrapped functions need to be methods/have access to‘self’....
Fix use of the logging functions
The logging functions expand the arguments themselves, thus it's saferto let them do it rather than manual string formatting.
Also re-wraps one comment.