Forbid the compression tools from being set over RAPI
Although we impose restrictions on what can be input as a compressiontool, someone with RAPI credentials could choose and execute a numberof commands by first setting the custom tools and then executing them....
Add the compression tools parameter
This patch makes the myriad of changes necessary for the compressiontool parameter to be added. The filtering of compression tools forsuspicious entries has been added for this exact purpose.
Signed-off-by: Hrvoje Ribicic <riba@google.com>...
Make arbitrary compression tools work
We assume that the compression tools the user supplies use stdin andstdout for handling data, and that a switch is used to distinguishcompression from decompression. This patch introduces these constraintsby adding the invocation of these tools to the import-export daemon....
Disable protections against unknown compression types
Ganeti took care to restrict all possible compression invocations tothe few options that were available. This patch strips away all ofthose, but does not allow any interesting and dangerous commands...
Extend offered compression types
This patch adds a few new types of compression supported by Ganeti:gzip-fast (gzip -1), gzip-slow (ordinary gzip), and lzop. gzip nowbecomes a shorthand for gzip-fast kept for compatibility.
Extend tests in preparation for new compression types
To ensure that later modifications are also noted in tests, this patchupdates the test to account for all compression types known.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Remove the unit test for verifying node's DRBD reservations
We'll need to reconstruct a similar test later when we add functions formanipulating the configuration directly to WConfd.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Fix the mocked ConfigWriter to pass DRBD minors methods
.. since for unit tests, we don't have access to WConfd, which nowimplements the functionality.
Test the 'findFirst' function
Also add 'genSublist' which picks a random sublist from a given list.
Add DiskParams to Disk object
The 'DiskParams' slot was missing from Haskell's Disk objects.Since Wconfd is now responsible for writting the config file this wascausing the 'params' slot to not be written in the config file.
Signed-off-by: Ilias Tsitsimpis <iliastsi@grnet.gr>...
Merge branch 'stable-2.11' into stable-2.12
Merge branch 'stable-2.10' into stable-2.11
Add a basic test for --restricted-migration
Essentially verify that, in the given example, a solution is stillfound and that the original present failover is dropped.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Hrvoje Ribicic <riba@google.com>
Support restricted migration
Make hbal support an option to disallow ReplacePrimary movesand restrict ReplaceAndFailover to instances where the primarynode is drained. If used in evacuation mode, the only migrationmoves will be off the drained nodes....
Add an example for node evacuation
The configuration shows an unbalanced cluster witha node being drained. The natural evacuation strategyincludes frf-moves.
Upgrade config to support top-level citizen disks
Implement upgrade/downgrade of the config file to support disks astop-level citizens. During downgrade, disks that are not attached to anyinstance will be removed from config file.
Eliminate local variable
Signed-off-by: Ilias Tsitsimpis <iliastsi@grnet.gr>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Fix the broken unittests
Fix the unittest that have been broken when we lifted the Disk objectsfrom the Instances.
Implement getDisks in Confd
Add 'ReqInstanceDisks' request type and allow Confd to query for thedisks of an instance. The helper function 'getInstanceDisks' returns thelist of instances on the given node along with their disks and is usedby the function 'addInstNameToLv'....
Add 'disks_info' Instance slot
Instance's 'disks' slot will contain a list of disk UUIDs. Create a newslot named 'disks_info' which will be annotated with the disk objects'during the RPC.
Get instance's nodes from config
Use 'GetInstanceNodes' to get the nodes of an instance, instead of the'all_nodes' Instance property.
Get instance secondary nodes from config
Use 'GetInstanceSecondaryNodes' to get the secondary nodes of aninstance, instead of the 'secondary_nodes'.Annotate 'secondary_nodes' in rpc so that backend has access to it.
Change TestInstance in objects unittest
Now that 'all_nodes' and 'secondary_nodes' of an instance are computedfrom the 'config' module, move the corresponding tests from'objects_unittest' to 'config_unittest'.
Fix '_create_instance' in config_unittest
Previously '_create_instance' would initialize the config object everytime it would be called. This causes '_WriteConfig' to fail withConfigurationError: The configuration file has been modified since thelast write. Fix this by passing the config object to '_create_instance'...
Add timestamp/serial_no slot to disk objects
Now that disks are top level citizens in config,they need a timestamp and a serial_no slot.
Add disks entry to config.data
Add disks entry to config.data.
kvm: use a dedicated QMP socket for kvmd
The KVM daemon keeps a persistent connection to the instances' QMPsockets, listening for asynchronous events. As each monitor socket(either human, or QMP) can handle only one client at a time, this hasthe side-effect that QMP cannot be used for regular instance operations....
Add systemd unittests
Test the consistency between daemon-util and the systemd unit files:
- Each service started/stopped by daemon-util must have a respective unit file. - The daemon user/group in the service file must be the same as the one reported by daemon-util....
Add a test for parsing version strings
...even in the presence of patch levels.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Petr Pudlak <pudlak@google.com>
Avoid assertIn
...as it is only present in python version 2.7 and higher.
Remove left-over debug code
This also avoids writing to hard-coded paths under /tmp
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
daemon-util: support for daemons loaded on demand
We add a second category of daemons in daemon-util, those that arestarted on demand and should not be started by the start_all action, butshould be stopped during service shutdown. This category currently...
Merge branch 'stable-2.11' into master
Make GlusterFS tests dual-stack-aware
Resolving "localhost" may either return "127.0.0.1" or "::1",so we have to check for both cases in the relevant tests.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add basic test for CPU speed
In this example, the cluster has two nodes and four instances, two withprimary on each of the nodes. The most scarce resource on this clusterare (virtual) CPUs and the second node has 3 times the CPU speed of thefirst one. So distributing the instances 1 and 3 gives a more balanced...
Add a new node parameter cpu_speed
This parameter will describe the speed of the CPU relativeto the speed of a "normal" node in this node group.
Add VTypeFloat
...in order not to have to declare floating pointvalues as VTypeInt and rely on the sloppiness ofthe JSON specification to not distinguish betweenintegers and floating point numbers.
Add optional fields for job livelocks and process IDs
This will allow to check if a particular job is alive, and send signalsto it when it's running.
The fields aren't serialized, if missing, for backwards compatibility.
Signed-off-by: Petr Pudlak <pudlak@google.com>...
Add tests for RAPI forbidden parameters
This patch introduces a number of tests ensuring that forbiddenparameters cannot be used. This is done by introducing a fake opcodeand a fake RAPI handler for it. Forbidden parameters, specificforbidden values, and renamings to forbidden parameters were all...
Minor preparations for forbidden params in RAPI handlers
This patch prepares the way for defining forbidden parameters in RAPIhandler classes as is done with renames and the like. No implementationof the logic is present here, just the expansion of expected...
Make RAPI opcode attributes use a class and not a tuple
The tuple approach to storing opcode attributes might have worked whenthere were four possible modifiers per method, but with aliases thisnumber has been pushed to five, and with forbidden parameters it will...
Clean up from LockAllocation what is no longer used
With the change from LockAllocations to LockWaitings, severalmanipulation operations had to be implemented for LockWaitingsand became unused in LockAllocation. Remove these functions thatare no longer used....
Verify the answer of opportunistic lock union
Verify that the result set of the opportunistic lockunion is correct.- If a lock is mentioned in the result set, the request must have been granted.- If it is not mentioned, the owner state must be unchanged....
Verify that opportunistic lock union is monotone
Verify that a request to opportunistically allocate locks neverreduces the amount of locks held.
Use intersection and union for arbitrary waitings
Allow intersection and opportunistic lock union in the constructionof the arbitrary element of a lock waiting. While definable fromupdateLocks, they are typical interface functions and can helpget a more natural distribution in arbitrary....
Refactor rlib2 unit tests
This huge patch removes a lot of cruft that has been accumulated in therlib2 test file. A new class, RAPITestCase, provides a common base forall the tests and unifies checks that were repeated on a per-test-casebasis.
Verify: liftM extRepr . readJSON . showJSON = Ok . extRepr
In other words, verify that for LockWaiting, the JSON encodingcan be decoded correctly up to observable indistinguishability.
Verify readJSON . showJSON = Ok for LockRequest
Verify that the JSON encoding of a LockRequest can correctlybe decoded.
Verify extRepr-equal states are equal on updateLocksWaiting
Add a test that verifies that extRepr-equal states cannot bedistinguished by updateLocksWaiting-transitions. To obtainextRepr-equal states, we compare each state to the one computedfrom its extensional representation....
Verify that extRepr-equal states behave equal on updateLocks
Add a test that verifies that extRepr-equal states cannot bedistinguished by updateLocks-transitions. To obtain extRepr-equalstates, we compare each state to the one computed from itsextensional representation....
Verify decoding from extensional representation is sound
Verify that, on the image of extRepr, it holds extRepr . fromExtrRepr = id.In other words, verify that when obtain a lock waiting from an extensionalrepresentation, it has the same extensional representation....
Verify that all pending requests are justified
If a request is pending in a lock waiting structure, it has to fulfilltwo properties:- It must be a valid request, not violating any lock order requirements, and- it must not be possible to fulfill it immediately....
Verify soundness of notifications
Verify that, if a lock owner gets notified, he- had a pending request, and- the request is fulfilled now.
Strengthen progress property in lock waiting
Not only verify that after the blockers for one pending requesthave gone, some request was honored, but also verify that it wasnot a request by one of the blockers.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Fix non-running test and remove custom_nicparams rename
Due to a missing base class, the test that was supposed to check ifRAPI instance parameter renames during modifications are working didnot execute. As the instances are not covered by the QA tests, the...
Add process id to lock-owner description
...so that we can notify owners when their pending request got granted.
Verify that some progress is made after lock releases
Add a basic sanity check for progress on pending requests. If arequest is pending and all owners it is blocked on release theirlocks, there is at least one pending request that can be granted(the said one). So verify that at least one owner gets notified...
Verify that an owner gets notified
Add a basic sanity check on notification: if a request is blocked,and all old pending owners and blockers give up their resources,the owner gets notified of his lock being granted.
Verify that pending requests get fulfilled eventually
Add a basic sanity check for the lock waiting mechanism. If a requestget added to pending, and all blockers and pending owners give up theirresources it is automatically handled.
Export requestSucceeded
...so that this predicate can be used for testing the LockWaitingstructure as well.
Verify that unfulfilled requests lead to pending requests
Verify that an owner has a pending request after a waiting requestnot fullfilled immediately.
Verify locks can't be updated while having a pending request
Add a verify first sanity check for lock waiting: a user that has a pendingrequest cannot modify his locks. This patch also brings in the necessaryinfra structure for having arbitrary waiting structures....
Export the test locks for testing
As lock waiting is also based on locking, export thetest lock/owner structure, so that the tests for lockwaiting can use them.
Fix typos in comments
QMP: make QmpConnection act as a context manager
This will allow automatic connection and socket cleanup on commandcompletion.
We also repeat the Qmp tests using the context manager. For this to befeasible, we move the test scenario to class variables and modify...
Add zeroing timeout params to opcode
This patch adds two parameters controlling the zeroing timeout - onethat is fixed and another that depends on the amount of data (size ofdisks) to zero.
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Jose A. Lopes <jabolopes@google.com>
Add the zeroing-image option
This patch adds the zeroing-image option to gnt-cluster and theOpBackupExport params. The many changes are all minor, yet necessary.
Add the zero-free-space option
Although unused for now, add the zero-free-space option to the backupexport opcode, and add a rudimentary check for whether it should beused.
QMP: always return the command result
According to the QEMU Machine Protocol Specification, the messages sentby QMP as a response to a command can be of two types: either an errormessage (identified by the "error" key), or a success message(identified by the "return" key)....
QMP: test supported_commands
Add test for supported_commands.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
KVM: move tap control functions to a submodule
Move all tap-related functionality to the hv_kvm.netdev submodule. Werename _OpenTap to OpenTap, since it will now be used as a publicfunction.
Also, change the hv_kvm tests to import the new code.
Signed-off-by: Apollon Oikonomopoulos <apoikos@gmail.com>...
Verify readJSON . showJSON = Ok for ClientType
Verify that the JSON encoding of the ClientType can correctlybe decoded.
Verify soundness of listAllLocksOwners
Verify that every owner mentioned in the output of listAllLocksOwnersactually owns the locks claimed.
Verify that all locks are mentioned in listAllLocksOwners
Add a test verifying that whenever an owner owns a lock, itis also mentioned in the output of listAllLocksOwners.
Verify consistency between listAllLocks{,Owners}
As listAllLocks is supposed to list all locks currently in use, this shouldbe a superset listAllLocksOwners, i.e., of all locks actively owned by someuser.
Verify that the list of all locks is complete
Verify that all locks owned by someone are included in thelist of all (active) locks.
Fix lint errors after moving Luxi code to UDSServer
Remove unused imports and update the 'buildCall' call in tests.
Update 'daemons/daemon-util' to stop 'ganeti-metad'
Note that 'ganeti-metad' is not automatically started together withall the other daemons. Instead, 'ganeti-metad' is only started when...
Identify lock owners by either string or job id
Most requests asking for resources are jobs. However, in exceptionalcases, other requests (like currently requests to masterd requiring theconfiguration) need to ask for resources. They identify themselves by...
Merge branch 'stable-2.9' into stable-2.10
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>...
Merge branch 'stable-2.8' into stable-2.9
Signed-off-by: Hrvoje Ribicic <riba@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Remove VerifyConfig tests
These tests don't work with WConfd, because the configuration isverified automatically before each operation and it's not possible tomodify the configuration state directly.
It'll be necessary to re-create these tests for WConfd once...
Update ConfigWriter test for _UpgradeConfig
.. to work with the updated version.
Remove desctructive ConfigWriter test
This test relies on bringing a configuration into an inconsistent state,and doesn't work with WConfd any more.
Update ConfigWriter unit tests
This involves:
- Updating how a mocked configuration object is set up.- Adding calls to refresh in-memory objects from the configuration (because they're not shared any more).- Adding calls to Update to save modified in-memory objects so that the...
Add unittests for instance modify parameter renaming
The get/put QA test could be used for instances as well, but unlikeother objects, instances have a very large number of discrepancies andmissing fields. To avoid enumerating every discrepancy and testing...
Add test for group custom parameter renaming
This patch introduces a simple test checking if a custom_* parameter isrenamed and passed to the opcode correctly.
Fix failing cluster query test
This patch makes a mock use a dictionary instead of an object, allowingalias finding to work on it.
Add support for value aliases to RAPI
This patch extends the metaclass used to generate RAPI handlers toallow creating aliases of certain values returned by GET methods.
Add reason parameter to RAPI client functions
Only the functions for starting, stopping and rebooting a VM had a reasonparameter. Now, all the RAPI client functions generating opcodes do.
Also, one test is expanded to verify that a RAPI request with both body and...
Add a method for WConfd context to GanetiContextMock
Move the test for adding/removing certs to config. tests
.. and update it to work with ConfigWriter instead of the 'security'module.
Signed-off-by: Petr Pudlak <pudlak@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Extend unittest for gnt-os info changes
Extend unit test ganeti.query_unittest.py, to correspond to thechanges made in the 'gnt-os info' output.
Signed-off-by: Dimitris Bliablias <bl.dimitris@gmail.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Fix WConfd mock client after updating the lock
.. because instead of a job ID and a lock file, we now pass a single,client ID object.
Add tests for (de)serialization of ClientId
.. to test if `readJSON . showJSON = Ok`.