Expand Haddock to run over test files as well
This patch does multiple enhancements to the way we build the Haddockdocs, motivated by the fact that if we don't run Haddock over testfiles as well, bad formatting can be submitted and will accumulateover time (without any checks)....
Add export lists for files which didn't have them
Two test helper files didn't have export lists (lost during the splitof tests). This patch finally reintroduces them, to hopefully make theexport list saner and with fewer changes for purely-internal changes....
Fix Haskell compatibility tests with disabled file storage
When file storage is disabled at ./configure time, we shouldn't passopcodes containing DTFile/DTSharedFile/StorageFile to Python forvalidation, as they will fail.
This patch implements this by simply tweaking the Arbitrary instances...
Allow rpc.MakeLegacyNodeInfo to parse non-LVM results
'MakeLegacyNodeInfo' is not the best place for this, but we'd have toduplicate it if we wanted a LVM-less version, so the easiest is to addan optional parameter that allows it to accept/skip LVM-less results....
Add IQ_NETWORKS in query tests
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Rename lib/objectutils to outils.py
Back when this was introduced, I mentioned that it breaks heavily tabcompletion (ob<TAB> doesn't work anymore), but at that moment I didn'thave a suggestion what to name it. I think outils is good and shortenough, and doesn't conflict with anything else, so here it goes....
Add functions to parse CLI-level format of priorities
The current serialisation format for submit priorities isinteger-based, same as the opcode json serialisation. But for CLIlevel, we need to support a string-based format, so we add functionsto parse and format this representation....
Enable use of the priority option in hbal
This patch adds the option to hbal, and uses it to tweak the submittedjobs. There are also two small shelltests for testing the parsing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Finish the remote→restricted commands rename
The documentation still points to /etc/ganeti/remote-commands,although the code is already using restricted-command. Update thedocumentation and a few docstrings accordingly.
Signed-off-by: Iustin Pop <iustin@google.com>...
Removes check for conflicts from NetworkDisconnect
This removes the check for conflicts from the Haskellversion of the OpCode NetworkDisconnect. This alignesthe Haskell code with the patch"Force conflicts check in LUNetworkDisconnect" (whichis currently under review). I will submit these patches...
Add unit tests for RADOSBLockDevice
Add unit tests for the RADOSBlockDevice JSON output parsing function,and modify the unit tests for the plain output parsing function towork with the new code.
Signed-off-by: Stratos Psomadakis <psomas@grnet.gr>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove network_type slot (Issue 363)
This slot was not used by Ganeti so the same info can beprovided via tags. In order not to break configuration datawe add a FromDict() method in Network config object thatremoves the deprecated network_type (if found) and then invoke...
Moved uniformity check for exclusive_storage flag
Cluster-verify used to check that the value of exclusive_storage is uniformwithin node groups. Now, it's impossible to change the flag for a singlenode, so that check has been removed and an equivalent one has been added...
"exclusive_storage" cannot be changed on single nodes
There's never been support for a configuration where nodes in the same nodegroup have different values of the exclusive_storage flag. This patchdisables the possibility to change the flag for individual nodes....
Upgrades made on loading the configuration are always saved
Before, only some upgrades were written back to the configuration file. Alittle refactoring of _UpgradeConfig() has been done to write unit tests.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Remove use of 'head' and add hlint warning for it
Since 'head' is unsafe to use in most cases, this patch removes itsuse from most of the code, adds a lint warning for it (and for tail aswell), and adds override annotations in the few cases where it'sactually OK to use it (mainly when using head over the result of...
Add some more Haskell/Python equivalence tests
This would have caught the log file problem fixed in the previouspatch (9411474b), for example. Also we test user/group equivalence,name only.
Add a helper function for hypervisor verification
This will allow easier multi-error results from hypervisors; rightnow, we only report the first error, which is not nice.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
_VerifyErrors()._Error() and _ErrorIf() are now consistent
_Error() didn't contain the logic for demoting errors to warnings and formarking an operation as failed. Now _ErrorIf() is just a minimal wrapperfor _Error().
Unit tests included.
testutils: Module-level method for reading test data
This patch moves the functions “_ReadTestData” and “_TestDataFilename”to module level. They do not depend on a class instance at all.
Some cases of “utils.ReadFile(self._TestDataFilename(…))” are also...
Add test for backend._GetBlockDevSymlinkPath
Add a unit test for the trivial “_GetBlockDevSymlinkPath” function inbackend (small changes in the function were required).
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
hv_xen: Add test for CPU pinning configuration
Add a unittest for a function formatting CPU pinning information forXen's configuration.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Text wrap in Haskell code
This adds two functions in Utils.hs: 'wrap' and 'trim' along with somesimple tests. 'wrap' is used in Common.hs for wrapping commanddescriptions to a fixed line length, e.g. useful for the "--help" option.
Signed-off-by: Yiannis Tsiouris <gtsiour@softlab.ntua.gr>...
utils.LvmExclusiveCheckNodePvs() returns more info
utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest andbiggest PVs of a node. They will be used for cross-node consistency checks.
burnin: Add unittest for supported disk templates
When new a new disk template is added, burnin should be updated.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove special-case of "0" in niceSort
Since now niceSort is more consistently behaved, we don't have thespecial casing. This should have been removed in commit a7f0953a, butI didn't realise it.
test/Types.hs: add an HLint ANN for ignoring camelCase
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Replace nodegroups' PartialNic by PartialNicParams
This fixes a bug that corrups the config when one adds anetwork to a node group. I wil soon extend QA by some commandsthat would have detected this.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Loader.hs: ignore expired ArSuspended policies
At the moment, because 'mergeData' is pure, it may set instance auto-repairpolicies that are of the form `ArSuspended $ Until timestamp_in_the_past`.If later on the auto-repair tool notices this, it has lost access to what...
Utils.hs: add a clockTimeToString function
This function allows to easily convert a ClockTime object to a stringrepresentation of its timestamp (seconds-only).
Utils.hs: function to chomp prefix + separator from a string
Some auto-repair tags are composed of a prefix and then an optionalargument; the new "chompPrefix" function in Utils.hs allows to strip aprefix, allowing the last character (the separator) to be absent if there...
Add initial constants and Haskell ADTs for auto repair
In this commit, the AutoRepairType and AutoRepairResult types are defined,with the possible values specified in doc/design-autorepair.rst.
kvm: add two help dump files
One is the --help output for kvm 1.1.2 and the second one is the same as0.9.1 but with a fake option added between -drive and its boot=on|offoption: this tests that if boot=on|off appears in another option after-drive it is not accepted by our regexp....
kvm: use help based feature detection
As discussed on the list some people backport features to different kvmversions, making the version based detection system unreliable. Wechange wherever we can to detection on the help output.
This also fixes Issue 250...
test/Utils.hs: fix capitalization in docstring comment
Unit tests for bdev and utils.lvm
Also amended a comment of a tested method.The tests are for methods introduced in commit 63c73073 (LVM disk creationuses dedicated PVs).
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Support instance-minor pairing in the DRBD collector
This commits enables the DRBD data collector to use the Confd client togather information about the pairing between DRBD minors and instances.
For testing purposes, the DRBD data collector now requires either zero...
Support integrating instance information in the DRBD parser
This commit modifies the DRBD parser and its data structures to includeinformation about the instance a DRBD minor belongs to.
Test files have been updated as well, to support the new field in the data...
Add "use_locking" parameter to network query opcode
This was extracted from a patch by Dimitris Aragiorgis with the subject“Add locking to _NetworkQuery”. The rest of the patch did no longerapply and will require more work.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Move src/Ganeti/HTools/Program.hs to Program/Main.hs
This removes one more tab conflict; this is the last module in ourcode where we have both x.hs and x/.
Furthermore, we collapse all actual code into the new Main.hs module,leaving the htools.hs basically empty (will allow better testing in...
Remove duplicate test file functions
This patch unifies the functions that generated the treereorganisation discussion in the first place :)
Rename test/hs/test to htest
The current base name of 'test' means that this binary will generateprofiling, coverage, etc. files with the name 'test.*', result intab-completion conflicts with the test directory. Let's slightlychange its base name for typing easiness....
Move htest/ files under the test/ tree
htest/data becomes test/data/htools (basically reverting commit8feabc89), and htest/* becomes test/hs/*.
Most changes beside the rename are trivial s/…/…, with the exceptionof autotools/run-in-tempdir, which needed some more changes now that...
Move python test files to test/py
This is the first step of the test files reorganisation: moving test/*(except test/data) to new directory test/py/.
Add the gnt-storage client
Add a new client called 'gnt-storage'.The client interacts with the ExtStorage interface, similarly tothe way gnt-os interacts with the OS interface.
For now, only two commands are supported: 'info' and 'diagnose'.
'diagnose' calculates the node status of each provider on each node,...
gnt-* list-fields: Show friendly field type
For people writing query filters (documented in ganeti(7)) knowing a field'stype can be useful.
$ gnt-instance list-fields name be/memoryName Type Title Descriptionname Text Instance Instance name...
Add utility to format dictionary as key=value strings
This will be used in QA to format network interface parameters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
cmdlib: Additional opportunistic locking check
If opportunistic locking is requested in the opcode, but no iallocatoris used (not specified or no default), an error will be raised.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but ratheronly replaces “frozenset” where appropriate. Most of the places are“static” information that doesn't change after the module has beenloaded....
Add exclusive_storage node parameter
Unit tests updated and expanded with an inheritance check.
The flag has no effect yet.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix error during cluster initialization due to hv_kvm
Commit 141d148 was a bit too enthusiastic. The three parameters added tothe list of parameters to be checked default to a value not evaluatingto false, leading to a failure on cluster initialization....
Add test for SPICE parameter list, add missing ones
“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list ofSPICE-related KVM hypervisor parameters with the exception of“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with“HV_KVM_SPICE_*” are included. Three previously missing parameters are...
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exactinput values; this makes it harder to check consistency againstHaskell code.
To compensate for this, we add a way to override the formatting of the...
Add RPC for setting watcher pause
The watcher pause file should be set/unset on all nodes at once, notonly the master node. For that a new RPC is needed.
Rename test for backend._CommonRestrictedCmdCheck
“TestWriteFile” was not renamed when adding the file based on“ganeti.utils.io_unittest-runasroot.py”.
jqueue: Improve inotify error reporting
This addresses issue 218. When the number of inotify watches isexhausted, for example by being set too low from the beginning or byother programs, waiting for a job to change would just report a lost job(e.g. “Error checking job status: Job with id 7817 lost”)....
Improve test for tools.ensure_dirs
- Add more checks, some of them are deliberately redundant- Descriptive error messages- Add comment describing order to “tools.ensure_dirs”- Avoid copying a list in an assertion in “tools.ensure_dirs”
Remove checks wrt IDISK_PARAMS from OpCode level
Change the "--disks" option validation, to just check the formatof the dict and do not check whether the keys are included in theIDISK_PARAMS constant at OpCode level. This allows the passing ofarbitrary parameters at the CLI, which will then be logically...
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would haveavoided the issue fixed in commit e2dd6ec. The new function is locatedin the “compat” module as it will be used at module load time in most...
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI clientmodule. A newly added unit test ensures all error codes are contained in“ECODE_ALL”, as well as ensuring consistency between the RAPI client and...
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists areprovided to exclude LUs from the two tests.
Fix node-daemon-setup test with older pyOpenSSL
Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of“-----BEGIN PRIVATE KEY-----”.
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
jqueue: Don't modify input opcode when changing priority
Commit 4679547 implemented the ability to change job's priority after itwas submitted. The code contained a bug whereby it would modify theinput data for an opcode, something the job queue shouldn't do (logical...
iallocator: Add node whitelist
In the future instance creations might have a lock on all nodes as wasthe case until the implementation of opportunistic locking. Nodes forwhich the lock is not held will be shown to the iallocator plugin as ifthey were marked offline....
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the nodedaemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” areverified, too.
locking: Implement opportunistic locking in LockSet
This patch adds a new parameter to “LockSet.acquire” named“opportunistic”. When enabled the lockset will try to acquire as manylocks as possible, but it won't wait for them (with the exception of thelockset-internal lock in case the whole set is acquired). This is...
Add ssconf function to read all files
Configuring a node daemon on a newly added node will need all ssconfvalues.
ssconf: Add dry-run support for writing files
A new utility for configuring the node daemon will support a dry-runmode. This patch adds the necessary functionality to“ssconf.SimpleStore” and provides comprehensive tests for“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter...
ssconf: Add function to verify keys
The new utility for configuring the node daemon will have to checkwhether it received valid ssconf names.
Add test for mutable default values in opcode parameters
This is not comprehensive, since in Python one can't determine what isand what is not mutable; but I've added a few base cases (list, dict,set).
The patch also improves (makes more uniform) the error messages in the...
Introduce ht.TMaybeValueNone and ht.TValueNone
TValueNone checks if a value is "none" and TMaybeValueNone is a wrapperof TOr(TValueNone, x). This is used by OpNetworkSetParam in order toreset a network value (e.g. mac_prefix, gateway, etc.)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
opcodes: Replace manual loop with map
Also remove a superfluous empty line in test file.
Fix type descriptions in RAPI documentation
This patch adds descriptors to the “_CheckCIDR*” functions in opcodesand improves the descriptions generated by “ht.TInstanceOf”, therebyindirectly fixing bad type descriptions in the RAPI documentation.
Before this patch:...
Move cluster verification out of prepare-node-join
A new tool for configuring the node daemon will also have to verify thecluster name, so it's better to have this function in a central place.In the process of moving it to ssconf it is also changed to use...
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Factorize code to load and verify JSON
A new tool to configure the node daemon will also have to load andverify JSON data.
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
locking: Method to check if LockSet is fully acquired
A new method is added to check whether the LockSet-internal lock isheld. This is the case after LockSet.acquire was called withlocking.ALL_SET.
Unit tests are updated, including one where the list of names must be...
Add new lock level for node allocations
The new lock is similar to the BGL in the sense that it has its ownlevel and there is only one. It is called “node allocation lock”.Logical units will use it to synchronize with instance creations, whichin turn will start using opportunistic locks on nodes....
Introduce a TMaybe combinator
We have many cases in the code where we write TOr(TNone, a), so let'sintroduce a combinator that simplifies this case.
Beside replacing the above with TMaybe(a), I did a few other parameterfixes:
- noop change TOr(TNone, TDict) to TMaybeDict...
Replace dict() with {}
The network patches and an existing test added function-call baseddict construction as opposed to literal sintax.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Rename leftovers from remote to restricted commands
As per Iustin Pop's suggestion in <20121115131730.GX824@google.com> on<ganeti-devel@googlegroups.com>.
backend: Rename RunRemoteCommand to RunRestrictedCmd
Add tests for repr in locking classes
“locking.PipeCondition” and “locking.SharedLock” define “__repr__”,which until now was not tested at all.
test/*.py: s/'/"/
Now that 2.6 is essentially finished and 2.7 going to be branchedsoon-ish, I thought it would be a good moment to replace some singlequotes in test/*.py. Merge pains should be limited.
In one place in test/ganeti.locking_unittest.py, spaces are added for...
Add unit test for default parameter default values
Fails if the default value of an opcode parameter doesn't verify.
locking: Add test for downgrade without names
Until now there was no test for calling “LockSet.downgrade” withoutspecifying any names.
Fixes to pass unittests (make check)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD parser unit tests
This adds tests that existing test files can be parsed by the Haskellparser as well, plus one new test file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Cleanup ht's use of positive/strictpositive
Currently, ht.py uses a bad terminology for positive/non-negativenumbers. Per http://en.wikipedia.org/wiki/Positive_number, this is thecorrect terminology:
- A number is positive if it is greater than zero.- A number is negative if it is less than zero....
Add opcode for running commands remotely
The opcode doesn't pay attention to the build-time flag to enable ordisable restricted commands. In a cluster different nodes could havedifferent settings.
Node locks are acquired in shared mode by default, but the use of an...
backend: Implement remote commands
As per design document (doc/design-remote-commands.rst), a number ofrather strict tests is applied to any incoming request, a delay isinserted upon errors and returned error messages are very generic(unless it's the actual command that failed). There are unit tests for...
Add unit test for RAPI handler access definitions
- Ensure query-related resources have the same access permissions (specifically “/2/query/*” and “/2/*/console”)- Check access permission consistency (write implies read)
rapi: Add new user option for querying
This was requested in issue 301. Before this patch, requests to“/2/query/*” and “/2/instances/*/console” would require authenticationwith a user with write access. Since that is not strictly necessary, anew user option named “read” is added....
Don't check for remote command directory as file storage
This test does not work properly if localstatedir is not “/etc”.
pathutils: Add directory for remote commands
Also add tests to ensure it's never allowed as a file storage path. Aconstant for the lock file is also added.
Add previously missing node daemon GID to getent mock
The UID is there, the GID wasn't.
Add test utility to count calls to function
In some cases it's nice to verify a function has been called exactly Ntimes. This is going to be used in tests for remote commands.