Add hvparams to RPC call 'node_info'
This patch adds the hvparams parameter to the RPC call'node_info'. It also adjusts the related code in noded.pyand Query/Node.hs
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
New tests for hspace and exclusive storage
hspace is run on clusters with exclusive storage enabled.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Spindles become part of htools resource spec
Spindles are now part of resource spec. Instances get created with spindlesspecified (which are just ignored when exclusive storage is disabled).
htools cluster score takes spindles into account
When exclusive storage is enabled, spindles are used instead of disk spaceto compute the cluster score.
Comments and variable names in computePDsk has been changed to match theactual code.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Update spindles when moving instances in htools
Spindles get updated, and errors raised when not enough free spindlesexist. No new error is raised when exclusive storage is disabled.
Unit tests included.
Unit tests for htools and exclusive storage
The existing tests are run also on nodes with exclusive storage enabled. Thevalues for spindles and exclusive storage are set in a consistent way, forboth nodes and instances.
Load node spindles data in htools
The data structure for nodes gets a new field for free spindles, and theexisting field for total spindles gets renamed to avoid identifying it withthe node parameter that had the same name. These fields get filled with...
Fix shadowing of library function
The "reads" field shadows a library function from Prelude. This commitfixes the problem.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Add shelltests for diskstats
Check that the behavior of running the diskstats data collector is asexpected.
Add unit tests for the Diskstats parser
Add one test parsing an actual /proc/diskstats file and one QuickChecktest.
Add function for generating arbitrary non-negative numbers
In many tests it is useful to have the possibility to easily generatenon-negative integer numbers.
Factor out and rename testFile function
The testFile function used in the DRBD parser test file can actually beuseful in any case where a parser has to be tested.
This patch moves it to the TestCommon file, and renames it to a more meaningfultestParser....
Fix erroneous docstring
Unit tests for spindles and exclusive storage in htools
Everything should work the same when exclusive storage is active, asspindles are not yet handled as a resource. A new test has been added forthe interactions between spindles and policies.
Check real spindles in ipolicies
When exclusive storage is enabled, the spindles in instance disks are usedto check the instance policies (as outlined in design-partitioned.rst).
Add spindles to instance disks in htools
A new data type is introduced for disks to store both size and spindles.When available, spindles are filled with input data. Except for loading andstoring, spindles are ignored.
Load exclusive_storage in htools
The node parameter is loaded into the data structures. No behavior is yetmodified.
New function to load JSON arrays of optional values
This will be needed to load spindles in some htools backends. Unit testsprovided.
Add unit test for text backend + fix bug
Test serialization and deserialization of instances. Fix check of secondarynode.
Test with non-redundant instances and several node-groups
The example cluster contains two node groups. The first contains three nodes,with two non-redundant instances each. As each node has only capacity for threeinstances, the nodes have to be rebooted individually. Additionally, there is...
Merge branch 'stable-2.8' into master
Add tests for network-aware allocation
hail-alloc-invalid-network defines a cluster with two nodegroups and anallocation request which does not fit on any of the groups. Group 1 hasinvalid disk-templates while Group 2 is not connected to the rightnetworks....
Parse NIC data from allocation request in hail
Add a NIC type and extend the Instance type by a list of NIC's. Parsethe NIC's in allocation requests and store them for now. Later patcheswill make use of this field in order to ensure that the requestedinstance is only placed in node groups wich are connected to those...
Parse node group networks
Extend the Group by the network ids it is connected to. Adaptthe IAlloc backend such that the networks are parsed correctly.This also required the adaption of test data.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add disks_active to configuration
This flag tracks if the disks of an instace are supposed to be active.That's the case when an instance is running or when its disks gotactivated explicitly (and in a couple of other cases).It will be used by watcher to re-activate disks after a node reboot....
Add spindles field to disk object
The field is filled with the value provided on the command line.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Extend hroller tests by options for non-redundant instances
The cluster now consists of 3 nodes, with drbd instances betweennodes 1 and 2, and 2 and 3. Additionally, nodes 1 and 3 each containa non-redundant instance, but node 2 cannot hold two additional...
Test for hroller taking non-redundant instances into account
The example cluster consists of 6 nodes, each hosting 2 instances andhaving capacity for 3. So, while the drbd-induced graph consists ofonly insulated nodes, no more than two nodes can be rebooted at the...
Compatibility test for instances
This patch introduces a test to check the compatibilityof the Haskell and the Python representation of instances.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Instance generators
This patch introduces and enhances generators forinstances:- 'genInstWithNets' is split into the generation of an arbitrary instance and enhancing an instance with nets- 'genInst' calls 'genInstWithNets' with an empty set of initial networks to provide a reasonable default...
Annotate every arbitrary instance field
The Arbitrary instance of the 'Instance' object is writtenusing the <*> syntax. Since it often uses the 'arbitrary'generator for the instance's fields it is hard to figureout which 'arbitrary' fills which instance field. This...
Generators for disks
This patch adds generators for Disk instances to the Haskelltest code. It uses somewhat more reasonable generators tofill the fields instead of just arbitrary values.'genDiskWithChildren' is a generator that generates a diskwith a specified number of disk children. To avoid shooting...
Extend hroller test to also verify tag-based node selection
While the multiple-tags test was added to verify that coloring is doneonly after node selection (otherwise it wouldn't be possible to get inboth cases a single reboot group), it can easily be extended to also...
Add a test for online rolling reboot scheduling
In the example configuration, the graph constructed by just connectingprimary and secondary instances is two-colorable. However, when takingconflicting locations of secondary nodes into account, three reboot...
Fix expectation in hroller test
Regular expressions are not shell globs. So "any symbol" is expressedby a dot, not a question mark. In this case, the confusion lead to atoo liberal expectation, hence the test passed. Fix it nevertheless.
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Add a test demonstrating the --node-tags option of hroller
The example is a cluster of 6 nodes, paired into 3 group by threeinstances. So the whole cluster would need two reboot groups. The twotags select, in two different ways, one node of each group. So, when...
Add tests for the -O option of hroller
In hroller, the option -O can be used to mark certain nodes as offline.These nodes should then not be part of any reboot group. Add teststo verify this behavior.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge remote-tracking branch 'origin/stable-2.8'
Conflicts: lib/bdev.py lib/cmdlib.py
Add reason trail to haskell opcode format
The haskell type definition of opcodes should remain aligned with the pythonone.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add test for DRBD 8.4 /proc/drbd parsing
The current parser already supports the output of DRBD 8.4 /proc/drbd,so add tests which actually verify this behaviour.Note that the empty lines in proc_drbd84.txt are actually present in theoutput of DRBD 8.4, they always appear when minors are not contiguous....
Add test for empty version in /proc/drbd
The previous patch removed a (rather implicit) test for /proc/drbd notcontaining a version line. This adds an explicit test for this case.
Signed-off-by: Thomas Thrainer <thomasth@google.com>Signed-off-by: Michele Tartara <mtartara@google.com>...
Extract /proc/drbd parsing code into DRBD8Info
As the DRBD8 class got bigger due to the previous merge of BaseDRBD, nowparts of it are ripped out into DRBD8Info. This new class parses/proc/drbd and exposes the information in an easily accessible way. This...
Add shelltests for hspace allocation
Both tiered and standard allocations are tested, with a single and a doublemin/max instance specification.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add multiple min/max specs in instance policy
Now instance policies can contain more than one min/max specs. This is themain element of the "Constrained instance sizes" section in the"Partitioned Ganeti" design doc.
This is a big patch, but changing the type of a configuration item requires...
Limit specs in instance policies are always complete
Specs used to specify limits in instance policies are always complete, evenat group level (only the whole limit set can be overridden). This is inpreparation for introducing multiple limits.
Extend the simulation backend to also simulate a master node
In a simulated cluster as created by the simulation backend tothe htools, make the first node of the first node group the masternode. In this way, hools (like hroller) that require a master node...
Add unit tests for hroller
Add a unit test for hroller in a situation where the orderis completely determined. There are 3 nodes where node 2 has,for each of the other nodes, an instanced shared with it. Sonode 2 has to be a reboot group of its own. There are no more...
Remove old "reason" implementation
Remove the useless parts of the old, partial, implementation of the support fortracking the reason of instances state change, before implementing the newreason trail support, as per the design document.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Make hroller insist on finding precisely one master node
As people rely on the master node being the last node of the lastgroup, make hroller fail, if no master node could be found in thecluster. This happens, e.g., if a backend format is used that does not...
Add nic/disk name and UUID fields in Haskell code
Add name and uuid fields to Disk and nic Haskell objects. Also,since they now have an UUID, make them instances of UuidObject.
Signed-off-by: Christor Stavrakakis <cstavr@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Add name to INicParams and IDiskParams
This commits adds name to NIC and Disk modification definition.
Signed-off-by: Chris Stavrakakis <cstavr@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Merge branch 'devel-2.7'
Removes obsolete 'enabled storage types'
Since managing of different storage units is now done usingdisk templates and not storage types, we remove the obsoleteenabled storage types.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtarara@google.com>
gnt-cluster modify: modify enabled disk templates
Adds setting of the list of enabled disk templates to 'gnt-cluster modify'.Note that this does not yet include any verification regarding disk templatescurrently used by instances.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Add shelltests verifying hail applies disk ipolicy per disk
We test on two minimal examples, a positive and a negative one, wherethe possibility to allocate a node depends on whether the disk policyis checked on a per-disk level or on the total amount of disk space...
Make the disks parameter available to the constructor
In that way, tools building on Instance will benefit from the correctedverification semantics of the instance policy on disk space.
Properly export errors while reading job list
In case of problems while reading the job list from disk (such as permissionerrors) confd would silently fail, writing a warning on the log file butsending an empty list and no error message to the client....
Fix a bug in the Runtime tests
A group name was erroneusly inserted in the list of user names.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add parameter pnode to OpInstanceSetParams
This parameter will be used to set the new primary nodeof an instance, assuming that the disks have been movedby other means (outside ganeti's control).
Refactor ispecs in ipolicy structures
Minimum and maximum instance specs are put together into a single elementof the instance policy. This is in preparation for introducing multiplemin/max specs.
gnt-cluster modify: dis/enabling storage types
This patch extends the 'gnt-cluster modify' command to manipulate the listof enabled storage types. Note that this currenlty does no validationwith respect to whether or not there are instances currently using a storage...
Add Mond to the list of possible daemons
Also, add its logfiles and extra log files.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Expand TestHelper to allow non-underscore prefixes
HLint 1.8.28 requires us to always add "ignore CamelCase", which canbe problematic sometimes (e.g. when using OverloadedStrings). Let'sexpand TestHelper to also support less-standard 'caseFooBar' names, so...
Change to CamelCase format in Test/Ganeti/Objects.hs
This is in preparation for using OverloadedString extensions in thisfile, which conflicts with hlint 1.8.28's handling of annotations.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add simple Ip4Address/Ip4Network types
This patch adds some very simple IPv4 address/network types, and usesthem in the 'Network' config object.
We need these in order to properly compute the reserved IP addresses,without depending on an external library (which I haven't found, by...
Conflicts: Makefile.am (curl changes and new hs directories)...
Expand Haddock to run over test files as well
This patch does multiple enhancements to the way we build the Haddockdocs, motivated by the fact that if we don't run Haddock over testfiles as well, bad formatting can be submitted and will accumulateover time (without any checks)....
Switch LispConfig double comparison to relative error
This further improves the comparison for "non-trivial" numbers. Without this patch, there are still cases where the absoluteerror is too big, and we need to switch to relative error.
Concept has been taken from...
Make the XmParser config test runtime more consistent
Currently, the test uses a frequency of 5 string/5 double/1 list forgenerating Arbitrary instances of ListConfig. However, the list casehas simply a "choose (1, 20)" `vectorOf` arbitrary, which means it...
Improve output of the XmParser config test
Currently, this tests and its helper function 'isAlmostEqual' usesplain booleans to signify failures, which means you can't really debuga failed test. The patch changes the call chain to use annotatedproperties all through, which results in messages like:...
Fix node partial name matching in Haskell code
This implements QffHostname and fixes the node listing (as well asexport listing when filtering on node name).
This bug was hidden by the fact that node listing with "gnt-node listaa" works if you don't have live queries (as it was originally), as...
Fix bug in group queries related to node/instance fields
Since we use the primitive string type for group UUIDs, the groupfields have a bug where we pass the group name as filter for nodetests, whereas the nodes themselves use the group UUID. This results...
Rename/make uniform the other query entities
Following the new naming style introduced in Exports.hs, this patchrenames the other resources to export non-qualified names (fieldMap asopposed to nodeFieldMap), and to use qualified module imports.
Also fixes a haddock issue in a docstring....
Add export lists for files which didn't have them
Two test helper files didn't have export lists (lost during the splitof tests). This patch finally reintroduces them, to hopefully make theexport list saner and with fewer changes for purely-internal changes....
Fix Haskell compatibility tests with disabled file storage
When file storage is disabled at ./configure time, we shouldn't passopcodes containing DTFile/DTSharedFile/StorageFile to Python forvalidation, as they will fail.
This patch implements this by simply tweaking the Arbitrary instances...
Remove test of obsolete getNetworkUuid method
This will remove the tests for the getNetworkUuid method.The method will be obsolete after the changes regardingidentification of networks by UUID and not name are mergedinto master. Will send another patch for removing the...
Simplify RPC error cases
This patch removes the node from the RPC error constructursCurlLayerError and OfflineNodeError. The rationale is that we anywayreturn tuples (node, result), and removing this duplication allowssimplified signatures/calls in the execution of RPC calls....
Add two utility functions for handling Either lists
These two functions permit operating in bulk on only the Left or Rightvalues in the original list, then reassembling the list back in theoriginal order.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Status change reason support for Reboot
Add support to the Reboot command for specifying the reason for the laststatus change.
Some features are implemented as functions, even if used only once, becausethey will be used by the future patches introducing reason support for all...
Infrastructure for specifying instance status change reason
This patch introduces some infrastructural modifications that will be used bythe following commits to implement the support for specifying the reason forthe last status change of an instance....
Unit tests for Query/Network.hs
This patch adds a couple of unit tests for Query/Network.hs.Note that they'll need to be adapted, once issue 362 is addressed.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add Haskell parser for "xm uptime"
In order to fetch precise information about the uptime of the VMsrunning in Xen, we need to analyze the output of the "xm uptime" command.
This commit adds the parser to do that, and its tests.
Add Haskell parser for "xm list --long"
In order to fetch precise information about the status of the VMs running inXen, we need to analyze the output of the "xm list --long" command.
Add functions to parse CLI-level format of priorities
The current serialisation format for submit priorities isinteger-based, same as the opcode json serialisation. But for CLIlevel, we need to support a string-based format, so we add functionsto parse and format this representation....
Enable use of the priority option in hbal
This patch adds the option to hbal, and uses it to tweak the submittedjobs. There are also two small shelltests for testing the parsing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Removes check for conflicts from NetworkDisconnect
This removes the check for conflicts from the Haskellversion of the OpCode NetworkDisconnect. This alignesthe Haskell code with the patch"Force conflicts check in LUNetworkDisconnect" (whichis currently under review). I will submit these patches...
Remove network_type slot (Issue 363)
This slot was not used by Ganeti so the same info can beprovided via tags. In order not to break configuration datawe add a FromDict() method in Network config object thatremoves the deprecated network_type (if found) and then invoke...
Remove use of 'head' and add hlint warning for it
Since 'head' is unsafe to use in most cases, this patch removes itsuse from most of the code, adds a lint warning for it (and for tail aswell), and adds override annotations in the few cases where it'sactually OK to use it (mainly when using head over the result of...
Add some more Haskell/Python equivalence tests
This would have caught the log file problem fixed in the previouspatch (9411474b), for example. Also we test user/group equivalence,name only.
Program/Harep.hs: add skeleton for the new auto-repair tool
harep(1) detects certain kind of problems with instances and applies theallowed set of solutions. See doc/design-autorepair.rst.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Extend config by networks and networks by UUIDs
For network queries to work, we need to extend the generalconfig type to include all available networks. Additionally,we add UUIDs to the network type itself.
Text wrap in Haskell code
This adds two functions in Utils.hs: 'wrap' and 'trim' along with somesimple tests. 'wrap' is used in Common.hs for wrapping commanddescriptions to a fixed line length, e.g. useful for the "--help" option.
Signed-off-by: Yiannis Tsiouris <gtsiour@softlab.ntua.gr>...
Remove special-case of "0" in niceSort
Since now niceSort is more consistently behaved, we don't have thespecial casing. This should have been removed in commit a7f0953a, butI didn't realise it.