QA: Run instance tests with different cluster configurations
Instance tests are run with or without the exclusive-storage flag set. Moreconfigurations can be added easily.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
QA: Instance tests check the configuration before running
Instance test functions check the current configuration, and they will runthe actual tests only if the configuration supports them. This will beused for refactoring in following patches.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
QA: Use lists of nodes as argument to instance tests
Some instance test functions took two node arguments, some took one, andsome took two but the second argument could be None. This patch makes suchfunctions uniform by using a list of nodes as an argument. This simplifies...
QA: Refactored launching of the various instance tests
The code launching instance tests for "plain" and "drbd" templates has beenunified. In this way it's easier to add more templates, and to re-use the sametests with different configurations.
QA: Run cluster-verify after instance tests
We want to make sure that instance operations don't leave the cluster in abad state. The "instance-remove-drbd-offline" test leaves some debrisbehind, so it's been moved to the last position.
QA: Added functions to allocate/free N nodes at once
These make possible to simplify the logic of tests requiring morenodes. Used in the following patches.
QA: Added exclusive_storage to qa_config
QA configuration now tracks the state of the exclusive_storage flag atcluster level. This will be used to selectively enable tests according tothe configuration.
Also, it's now possible to specify the initial value of the...
QA fix: TestInstanceExportWithRemove() updates configuration
When the instance is removed from the cluster by this test, it's also freedin the QA configuration.
QA cleanup: Removed instance-disk-failure test
The test was broken, out of sync with the rest of the code, and prone tocrashes. Until someone does a better job, it's better to remove thetest. This simplifies further refactoring.
QA: Added disk template to config
The disk templates used by the instances are tracked in the QAconfiguration. This will be used to selectively enable tests according tothe configuration.
QA: Better initialization of a global dictionary
The global 'cfg' variable is initialized to an empty dictionary, so thereis no more need to disable some pylint checks.
QA: Cluster-verify reports shared PVs with exclusive storage
Exclusive storage forbids sharing PVs between unrelated LVs. This is a testthat cluster-verify correctly report such cases.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>...
QA: Added constants for LVM volumes
The LVM volume group becomes a configurable parameter, and a prefix forlogical volume names used in tests is introduced as a constant.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
QA: Templates incompatible with exclusive storage reported
Cluster-verify should report instances whose disk template is notcompatible with the exlusive storage setting.
QA: Uniformity check for exclusive_storage in cluster-verify
Cluster-verify should report nodes in the same nodegroup with differentvalues of the exclusive_storage flag.
QA: Test for basic features of exclusive storage
Check that creation of instances doesn't trigger any error.
QA: Added function to read cluster configuration fields
This function will be used in following patches.
QA: Added function to check cluster-verify result
Now it's possible to check that cluster-verify reports all the expectederrors, instead of simply checking that it fails.
Document better gnt-network
Specify clearly that the nic parameters depend on thenetwork-to-nodegroup connection.
Clarify in gnt-network how instance nics are connected to a network.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Fix the gnt-instance section in the htools manpage
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Update 2.7 design doc
Move two designs that have been implemented in 2.7 to design-2.7 and outof drafts.
Other designs were partly implemented: explain what part was, and leavethem as drafts pending their completion.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
update 2.7 NEWS file
Include a handful of important changes that weren't mentioned.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Disable live-RPC queries under split query
Currently, the node listing RPC is very slow due to missingparallelisation. For the 2.7 release, we reset these back to masterd,hoping to revert them by the time 2.8 is ready.
There are a number of queries that I've left pointing to confd, as...
hv_kvm: Original error message, keyword parameter
- Include original error message when creating TAP interface failed- Pass keyword parameter as such
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
CLI.hs: fix double spaces in option help strings
Some help strings with continuation backslashes ('\') were providing aspace both before and after the backslash, resulting in double spaces inhelp output. Provide it only after the backslash, which fixes the issue and...
Add new NIC parameter to man page of 'gnt-instance'
According to doc/design-network.rst, there is a new NICparameter 'network', which can be used to link an instanceto a network. This was so far undocumented in the manpage of 'gnt-instance'.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Fix spelling issue in hroller.rst
'explicitly' was spelled the wrong way
Signed-off-by: Benjamin Kerensa <bkerensa@ubuntu.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
kvm: fix bug while fetching -device list
_GetKVMOutput expects the command to succeed, but unfortunately on someversions of kvm "-device ?" will output a correct list of devices, whileexiting with an error code.
To fix this we accept failure in that case (note that this doesn't...
hv_xen: Remove config after shutdown was successful
If stopping an instance failed, the configuration would already be goneand other operations depending on it (e.g. migration) would no longerwork. With this patch the configuration file is only removed once the...
hv_*: Always return from Verify, style fixes
Change all “Verify” methods in hypervisor abstractions to explicitelyreturn None if no problem was detected. Remove punctuation from errormessages. Update docstrings with “@return” and some small mistakes.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
_VerifyErrors()._Error() and _ErrorIf() are now consistent
_Error() didn't contain the logic for demoting errors to warnings and formarking an operation as failed. Now _ErrorIf() is just a minimal wrapperfor _Error().
Unit tests included.
Merge branch 'master' into devel-2.7
Handle the result of QueryGroups() correctly
If no group is given for the “gnt-network connect“/“… disconnect”commands, the client uses the result of “QueryGroups()” which is a listof lists. Use “itertools.chain()” to handle the return value correctly....
hv_xen: Compose file name outside error handling
In _ReadConfigFile, the filename should be prepared outside thetry/except block. Fixes bad code formatting, too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
testutils: Module-level method for reading test data
This patch moves the functions “_ReadTestData” and “_TestDataFilename”to module level. They do not depend on a class instance at all.
Some cases of “utils.ReadFile(self._TestDataFilename(…))” are also...
hv_base: Remove empty constructor
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Add test for backend._GetBlockDevSymlinkPath
Add a unit test for the trivial “_GetBlockDevSymlinkPath” function inbackend (small changes in the function were required).
Fix format string of KVM output
This fixes a missing 's' in the format string andthe wrong quotes. Those bugs were introduced incommit 6e043e60.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Renames and cleanup of variable names in confd
The current names are quite confusing; this patch cleans up theconfusion by making sure we use different terms for the two threads,etc.
No actual code changes besides the renames.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add a function to get the current time in microseconds
In some cases we need higher resolution that seconds; I've settled onmicroseconds as that is what 'threadDelay' wants, for exactly, so it'seasier if we keep the same units.
Increase Haskell log file resolution
Currently the Haskell logging goes with the default time formatting,which lacks sub-second resolution. This is not good, as it's toocoarse.
The patch adds picoseconds to the log; they will contain usually toomany zeroes, but the other option is to use a log format that (for...
First part of confd timer changes
This patch changes the resolution of the timers: the watcher timergoes from 60s to 17s, and the polling-mode timer goes from 2 secondsto 250ms. The code changes a bit more due to the changes in the unitsof the various constants....
Change the polling thread sleep behaviour
Currently, the polling thread sleeps at the end of its body; theoriginal reason was that when the thread initially starts, we want toload the config immediately (in the old-style config load model).
However, now we explicitly load the config, so we can make this thread...
Start confd in notify mode if we can
This patch changes the default model—where confd is always started inpolling mode—to a model where if possible we enable inotify modebefore starting any of the background threads.
There are some tricky details here: if we enable inotify, we should...
Explicitly load the config at confd startup
Currently, confd main function doesn't load the config, but simplyforks the individual timer threads, which will—as a side effect-alsoload the configuration as soon as they start.
However, this makes codes to the startup more complex, so let's...
Fix type of 'node_whitelist' request parameter
If opportunistic_locking is used, then 'node_whitelist' parameter passedto the allocator is set to the LU's owned node locks. However, LU owned_lockshas type of 'set' while IReqInstanceAlloc expects type of...
hv_xen: Add test for CPU pinning configuration
Add a unittest for a function formatting CPU pinning information forXen's configuration.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
kvm: deduplicate 'get output' code
We had the same code twice, and were about to add a third time. Betterto collapse it into just one function.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
kvm: extract a regexp matching out of a for loop
kvm: remove last version-based feature detection
This was left behind because it required a different kvm invocation.Now that we can add new ones cheaply (two constants) it's easy to getrid of it. Differently than in other cases we support old version which...
Make Xen config path a build-time option
Stop hardcoding the path in “hv_xen.py”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Makefile.am: Protect placeholders for sed
If the placeholder has exactly the same name as the variable used inautoconf it would be replaced as well. In the case of“ENABLE_SPLIT_QUERY” this led to the sed command “s#False#False#g”.Inserting additional quotes after the first @ sign prevents this....
ENABLE_SPLIT_QUERY
burnin: Don't keep hypervisor class around
Just determine whether it can migrate and keep that value instead of thefull hypervisor class.
Text wrap in Haskell code
This adds two functions in Utils.hs: 'wrap' and 'trim' along with somesimple tests. 'wrap' is used in Common.hs for wrapping commanddescriptions to a fixed line length, e.g. useful for the "--help" option.
Signed-off-by: Yiannis Tsiouris <gtsiour@softlab.ntua.gr>...
Run pre-migrate hooks on primary node too
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Reviewed-by: Guido Trotter <ultrotter@google.com>
Makefile: Allow tarball creation when man-in-doc is active
Documentation with included man pages should not be distributed as itcontains fixed paths. Since the documentation with man pages is actuallywritten to a separate directory, tarball creation is still possible....
Check if KVM machine version is supported
If machine version is passed as an hv param, a check is madein target node whether this version is included in the supportedones derived from kvm -M ? command.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
Revert "run-in-tempdir: Improve speed"
This reverts commit dd60179dfd8 as it broke coverage for Python.
Verify that templates are compatible with exclusive storage
cluster-verify reports instances with disk templates not compatible withexclusive storage but that are running on nodes with the exclusive storageflag set.
cluster-verify checks uniformity of PV sizes
When exclusive_storage is set, cluster-verify complains if the size of PVsis not uniform across a node group.
New RPC call for node verify in dns-only RPC client
This will be used in node-add.
add-node checks PVs
add-node now performs a few checks on LVM before adding a node to thecluster.
Moved checks within LUClusterVerifyGroup
Almost all instance-specific checks have been moved from the Exec method tothe _VerifyInstance method. This cleans up Exec, which was becoming too bigeven for pylint…
bdev.GetPVInfo() returns list of LVs
This will be used for checks related to exclusive storage.
cluster-verify checks that PVs are not shared
When exclusive_storage is set, cluster-verify complains if PVs are sharedamong unrelated LVs.
Small cleanup in constants.py
- compat.UniqueFrozenset used in one more place, for consistency.- The order of NV_XXX constants is fixed so that all values are now sortedby name.
cluster-verify checks uniformity of exclusive_storage flag
The value of the flag should be the same for nodes within the same group.
utils.LvmExclusiveCheckNodePvs() returns more info
utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest andbiggest PVs of a node. They will be used for cross-node consistency checks.
run-in-tempdir: Improve speed
By creating symlinks instead of actually creating files we still get thebenefits of using a temporary directory, but the preparation is aboutthree times as fast. Test used:
$ time bash -c 'for ((i=0; i < 100; ++i)); do autotools/run-in-tempdir true; done'...
run-in-tempdir: Copy doc/* only if needed
To preserve disk space on copying all files to a temporary directory,the “doc” directory is only copied when building documentation. Acomment describing the need is also added to Makefile.am, as well as asuperfluous direct invocation of Bash removed....
Integrate DRBD with the report generation infrastructure
With this commit, the DRBD data collector does not only print the datait extracts, but includes them in the proper JSON structure common toall the data collectors, as prescribed by the design document....
Add reporting infrastructure for data collectors
This commit adds the part of the JSON report generation code that will becommon to all the data collectors, according to the format specified in thedesign document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Move getCurrentTime from Confd.Utils to Utils
The getCurrentTime function is of wide interest, so its moved to a moreappropriate position.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
burnin: Add unittest for supported disk templates
When new a new disk template is added, burnin should be updated.
kvm: make kvm path configurable as an hv parameter
This fixes Issue 225.
make _{Generate,Execute}KVMRuntime accept kvm help
Currently kvm --help is called twice at instance startup, this makessure it's called only once.
kvm: make auxiliary functions accept a kvm path
This will be used to be able to customize the path in an hv parameter.
Also fix two comments that still mentioned version-based checking.
Add KVM machine version as hv param
This is needed in case an instance must be booted with an exact type ofmachine version (due to e.g. outdated drivers). If this hv paramexplicitly defined (during instance creation/modification) then the kvmprocess will be started having this version for the -M option. Otherwise...
Move burnin to ganeti.tools.burnin
Similar to other scripts, “tools/burnin” just becomes a wrapper script.
Remove special-case of "0" in niceSort
Since now niceSort is more consistently behaved, we don't have thespecial casing. This should have been removed in commit a7f0953a, butI didn't realise it.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
burnin: Enable DT_EXT/DT_RBD for single node clusters
As reported by Constantinos Venetsanopoulos, DT_RBD and DT_EXT work onsingle node clusters.
test/Types.hs: add an HLint ANN for ignoring camelCase
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Link man pages in documentation
This patch depends on “Option to include man pages in documentation”. Inthe documentation build including man pages, all “:manpage:`…`”references are converted to links. For man pages not provided by Ganeti,Sphinx' standard formatting is used....
Make niceSort a wrapper over niceSortKey id
Currently, niceSort behaves differently than niceSortKey in caseswhere the parsed groups are equal but the original not, as it usesboth members of the tuple as opposed to just the first one. BothniceSortKey and the Python code use only the first one, so in corner...
Add version check for Sphinx
Sphinx 1.0 and above can check version by setting “needs_sphinx” in theconfiguration.
Option to include man pages in documentation
Before this patch, HTML versions of man pages (man/*.rst) were alreadybuilt. However, since they are separate from the normal documentation,their content is not indexed for Sphinx' search functionality.Additionally it would simply be nice to have everything in one place....
design-autorepair.rst: separate list of jobs with plus sign
Commas are not valid characters in tags, hence they can't be use toseparate the different job IDs; plus signs (+) are available, and not toobad.
burnin: Disk template option cleanup
Burnin doesn't support all disk templates (specifically it doesn'tsupport “blockdev”). The code, however, used manually composed lists insome places, constants.DISK_TEMPLATES in others, and an internal list ofsupported templates in some....
burnin: Factorize disk template lists
Keep the lists at the top instead of having them embedded in the code.Use frozenset, too.
Replace nodegroups' PartialNic by PartialNicParams
This fixes a bug that corrups the config when one adds anetwork to a node group. I wil soon extend QA by some commandsthat would have detected this.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Loader.hs: ignore expired ArSuspended policies
At the moment, because 'mergeData' is pure, it may set instance auto-repairpolicies that are of the form `ArSuspended $ Until timestamp_in_the_past`.If later on the auto-repair tool notices this, it has lost access to what...
Loader.hs: rewrite extractExTags to use chompPrefix
design-autorepair.rst: clarify tag precedence and conflict
This commit clarifies one particular point of the auto-repair workflow:what to do when multiple, conflicting administrator-set tags exist in anobject; and how tags at different levels (cluster, node group and instance)...
Loader.hs: set instance auto-repair policy in mergeData
'getArPolicy' and 'setArPolicy' follow the precedence rules introduced inb1eb71c: within an object, the most restrictive tag wins; across object,the nearest tag wins.
Signed-off-by: Dato Simó <dato@google.com>...
Instance.hs: add an 'arPolicy' field for auto-repair policy
Utils.hs: add a clockTimeToString function
This function allows to easily convert a ClockTime object to a stringrepresentation of its timestamp (seconds-only).
Utils.hs: function to chomp prefix + separator from a string
Some auto-repair tags are composed of a prefix and then an optionalargument; the new "chompPrefix" function in Utils.hs allows to strip aprefix, allowing the last character (the separator) to be absent if there...
design-autorepair.rst: add a graph with state transitions
The graph added here follows the data types introduced in 7ec2f76, by which"Suspended" is not a state, but a possible value of the auto-repair policy.
HTools/Types.hs: more auto-repair types
AutoRepairPolicy, AutoRepairStatus, and other auxiliary types are added.These are used only internally by the auto-repair tool, and parsed from thevarious object tags as defined in the design doc.