Add two more unittests for Node add/remove
These just test that add/remove are idempotent. This is not perfect,as we use unsorted lists for some values (instead of sets), so whenusing non-empty nodes this would break (but for empty nodes, a1-element list is sorted, so it's fine)....
Use the spindles metric in cluster scores
This makes balancing use the new metric.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Track instance-used spindles in node set/add/remove
This simply tracks the instance-used spindles (using hard-coded '1'per instance), and additionally prevents additions in soft-mode whenwe go over the limit.
Note: there's an assymetry between addSec and removeSec (basically...
Shorten some function names
Since we use the functions always module-qualified, let's drop the`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Simplify logic in Node operations
This patch adds two helper functions and uses them for a somewhatsimplified logic in the add/remove pri/sec node functions.
htools: support spindles in simu backend
htools: support spindles in text load/save
If missing, we'll default to 1 spindle.
This also updates the text file format with the ipolicy, which wasforgotten.
htools: add support for listing spindles for nodes
htools: add spindle-related attribute to nodes
This adds the spindleCount, hiSpindles and instSpindles attributes.
The spindleCount is equivalent to spindle_count on the ganeti side (anode parameter). hiSpindles is the maximum instance-used spindles, and...
Add spindle ratio to htool's IPolicy
Just the usual data/type declarations, read/save in the text backend,etc. Also does a bit of unit-test cleanup (the way we buildipolicies).
hinfo: Reorganize the code to separate functions
Also rename some functions as they don't reflect their actions anymore
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
hinfo: Adding basic skeleton based on hbal
It prints the information which hbal prints in a nice formatted manner and ofcourse without doing any actions. An example output looks like this:
$ hinfo -m xen.example.comLoaded 4 nodes, 2 instancesCluster has 1 node group(s)...
htools: allow rbd disk template
Notes: - it'd be nice if there was a way to automatically generate the DiskTemplate list instead of manually specifying it. After all we have C.diskTemplates - Of course this actually does nothing, as for other...
Use the ipolicy constants for key names
Sorry, I missed this when I wrote the code originally.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix integer overflow in Node unittest
When using a really random "extra cpus" variable, this can overflow(especially on 32 bit) and thus the final value can be < 0, thus thetest will fail to fail.
It doesn't make sense to test with huge values, so let's just generate...
Add vcpu_ratio definition to the IPolicy type
This will be "naturally" read in the JSON-based backends (Luxi/Rapi),and we add explicit code for handling it in the Text backend.
Remove Node.mCpu, replaced with the ipol vcpu_ratio
Since the iPolicy contains the vcpu_ratio now, we can remove theper-node mCpu value (which was the initial way to set this attribute).
Furthermore, we need to re-set the hiCpu value when changing thepolicy....
Temporary fix for hspace's NPU calculation
hspace computes the NPU (normalised CPU units) via the(used|pool|avail) VCPU divided by the (currently cluster-wide) maxvcpu_ratio.
However, in the future we'll have variable vcpu_ratios, so we can'tthe calculation as such. We temporarily change to computing the...
Change meaning of the --max-cpu option
Instead of always overriding, we should update the nodes's maxcpusetting only when the option is specified. Otherwise this will be readfrom the cluster or from the Ganeti constant for non-live backends.
Signed-off-by: Iustin Pop <iustin@google.com>...
Remove obsolete constant for vcpu_ratio
This is superseded now.
Re-indent code for readability
Otherwise the docstrings for members take too many lines.
Track 'normalised' used CPUs in Cluster.CStats
This will be used in hspace for proper tracking of used/pool/unavailnpus.
Add a new type for holding allocation statistics
The current RSpec type doesn't cut it anymore (and it's used in otherplaces), so let's introduce a separate type for holding thesestatistics.
Switch over some hspace stats from RSpec to AllocInfo
This patch changes the allocation delta stats from RSpec toAllocInfo. There's nothing interesting about it, but it should allowmore correct computations of Npus in hspace.
I also moved the AllocStats type alias from Cluster.hs to Types.hs...
Expand the tiered alloc unittest to check allocation stats
This was missing before, so with the new metric (NCpus), it's a goodtime to test this.
Remove two extraneous imports
Another small optimisation in test data generation
It seems that QuickCheck will happily generate huge data structures,if that's what one requests. For this test, where we ask for [[Char]],it could happen that we get a huge number of chars, so let's limit...
Generate tags using the helper in Node tests
Otherwise we test very big lists (slow) without any advantage in doingso.
Combine two Node unittests
These two tests are very similar, so let's run them together insteadof generating the data twice. Also improve the test a bit.
Add unit-tests for --help/--version as common options
After reorganizing the way we process options, we can finally add atest that all program personalities accept --help and --version andthat they exit early for these two.
This trivial patch has a higher than expected impact on the coverage,...
Split the list of program personalities into a module
… a mini-module, actually. This is only done for unit-testing, as wedon't want to duplicate the personalities list in the tests.
Parse options in htools.hs, not in each program
This patch moves the call to getArgs/parseOpts into htools.hs, asopposed to the current situation where htools.hs only calls the mainfile of each program, which in turn calls these two functions.
By doing it only once we remove some duplicate code and we also export...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Split CLI.parseOpts into IO/non-IO parts
This will allow unittesting the CLI options much more easily. Thepatch also fixes an inconsistency: usage help was displayed with anextra new line in the error case (but not in --help).
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Remove some dead code
While wondering why Cluster.tryReloc is not unit-tested, I realisedthat this is no longer uses. Apparently, last summer's new IAllocatormodes and related changes have silently deprecated this function.
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Fix a couple of bugs in handling offline instances
The addition of the offline state for instances has introduced a fewbugs related to their handling. The current code adds a helperfunction for making the check whether an instances uses reservedmemory on the secondary (and uses it instead of just autoBalance),...
Add a helper function for 'not offline' instances
Unittests for the instance policy checks on allocation
Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Obey instance policy in allocation routines
This currently checks the minimum/maximum sizes and the disktemplate. Unittests will come later.
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
A trivial optimisation in Loader.mergeData
During loading, we can eliminate some intermediate datastructures. This only matters for big clusters, and even there it'sless than 1% less memory allocated, but still, it's a no-brainer.
Trivial performance optimisation
This is not in the hot-path, but still, no sense in doing O(n) insteadof O(1).
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Enable Eq instances for TH-built objects
This is needed for the next patch.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Read the disk templates part of the ipolicy
The default value is badly defined (hardcoded defaults)…
Move DiskTemplate definition around
This is needed since we'll need the DiskTemplate definition in theIPolicy one.
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Add a helper function converting ispecs to rspecs
The rspec type is simpler than the ispec one; most likely it should bedeprecated later.
Abstract creation of instance from a spec
Load cluster ipolicy via Rapi
This requires changing from querying the /tags resource to the /inforesource.
Update memory/maxmem reading in Rapi backend
Recent changes to the instance beparams have replaced memory withmaxmem in Rapi bulk queries. Until this is either reverted (forbackwards compat) or we decide to go ahead with only maxmem, we changethe backend to read both; this only affects the "instance down" code...
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Add default ipolicy declarations
Add object definitions for the ispec and ipolicy
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
htools: split parts of HTools/Types.hs into BasicTypes.hs
The 'Result' type is common and:
- might be used outside of HTools-specific code too- is better split as we need these basic types for building the more complex ones in Types.hs
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Abstract tiered specs command line parsing
This will be used in the future for the standard specs too, so let'sabstract it away.
Add infrastructure for, and two extra hlint rules
This will allow us to easily use add new lint rules that westandardise on, if any.
Some small test simplifications
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Abstract some common hspace code into a function
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Abstract some common Cluster.hs code into a function
Convert opcode TH code to the use of Field type
This makes more explicit the field behaviour - previously an optionalfield was detected via a "Maybe" constructor, and an optional one viaa "Just defval" one. With this, field behaviour become more explicit...
htools: small change in error message in THH.hs
We should also display the value we can't parse, otherwise debuggingis very hard.
htools: improvements to JSON deserialisation
This fixes two problems:
- first, when we deserialise a big object, showing its value is not useful, as it will hide the actual error message- second, we shouldn't deserialise a container at once, because then...
htools: add new template haskell system
This system based on explicit types instead of ad-hoc rules(e.g. instead of deducing from "Maybe Int" an optional field, we nowcan say explicitly OptionalField ''Int). In the first phase, this willbe used for the equivalent of lib/objects.py, which has slightly...
htools: adjust imports for newer compilers
While testing with ghc 7.2, I saw that some imports we are using arevery old (from ghc 6.8 time), even though current libraries are usingdifferent names.
We fix this and bump minimum documented version to ghc 6.12, as I...
Merge branch 'devel-2.5'
Merge branch 'stable-2.5' into devel-2.5
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's freememory reaches 0, it will refuse to add any instances (offline ornot). I think this is a safe safeguard (I don't expect nodes to runwithout at least 1MB of free memory), so rather than change this...
htools: finish re-indenting Cluster.hs
htools: re-indent Node.hs
htools: re-indent QC.hs
htools: reindent CLI.hs
htools: reindent hbal
htools: reindent hspace
htools: re-indent IAlloc.hs
htools: reindent the rest of the files