Add vcpu_ratio definition to the IPolicy type
This will be "naturally" read in the JSON-based backends (Luxi/Rapi),and we add explicit code for handling it in the Text backend.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Remove Node.mCpu, replaced with the ipol vcpu_ratio
Since the iPolicy contains the vcpu_ratio now, we can remove theper-node mCpu value (which was the initial way to set this attribute).
Furthermore, we need to re-set the hiCpu value when changing thepolicy....
Temporary fix for hspace's NPU calculation
hspace computes the NPU (normalised CPU units) via the(used|pool|avail) VCPU divided by the (currently cluster-wide) maxvcpu_ratio.
However, in the future we'll have variable vcpu_ratios, so we can'tthe calculation as such. We temporarily change to computing the...
Change meaning of the --max-cpu option
Instead of always overriding, we should update the nodes's maxcpusetting only when the option is specified. Otherwise this will be readfrom the cluster or from the Ganeti constant for non-live backends.
Signed-off-by: Iustin Pop <iustin@google.com>...
Remove obsolete constant for vcpu_ratio
This is superseded now.
Re-indent code for readability
Otherwise the docstrings for members take too many lines.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Track 'normalised' used CPUs in Cluster.CStats
This will be used in hspace for proper tracking of used/pool/unavailnpus.
Add a new type for holding allocation statistics
The current RSpec type doesn't cut it anymore (and it's used in otherplaces), so let's introduce a separate type for holding thesestatistics.
Remove two extraneous imports
Another small optimisation in test data generation
It seems that QuickCheck will happily generate huge data structures,if that's what one requests. For this test, where we ask for [[Char]],it could happen that we get a huge number of chars, so let's limit...
Generate tags using the helper in Node tests
Otherwise we test very big lists (slow) without any advantage in doingso.
Combine two Node unittests
These two tests are very similar, so let's run them together insteadof generating the data twice. Also improve the test a bit.
Add unit-tests for --help/--version as common options
After reorganizing the way we process options, we can finally add atest that all program personalities accept --help and --version andthat they exit early for these two.
This trivial patch has a higher than expected impact on the coverage,...
Split the list of program personalities into a module
… a mini-module, actually. This is only done for unit-testing, as wedon't want to duplicate the personalities list in the tests.
Parse options in htools.hs, not in each program
This patch moves the call to getArgs/parseOpts into htools.hs, asopposed to the current situation where htools.hs only calls the mainfile of each program, which in turn calls these two functions.
By doing it only once we remove some duplicate code and we also export...
Add some very basic options test
Just tests that passing a string argument returns it correctly. We'renot interested in the flags themselves that much just that thefunction itself parses some options at least correctly.
Split CLI.parseOpts into IO/non-IO parts
This will allow unittesting the CLI options much more easily. Thepatch also fixes an inconsistency: usage help was displayed with anextra new line in the error case (but not in --help).
Add test for CLI.parseYesNo
Also fix a tiny typo in the docstring.
Fix bug in CLI.parseISpecString and add test
'zip' finishes on the shortest list, so parseISpecString accepted anystring of 3 values or more, which is not good. Fix this and add testfor both successful and failed parsing.
Add a unittest for the 'Simu' backend
This only test successful parsing, but for this case it testsreasonably well the resulting cluster properties, I think (coverageis >70% for expressions).
Further optimise instance test data generation
Following on the lessons from this week (commit 7806125e), this patchreplaces generation of instances `suchThat` they are smaller than onnode with direct generation of instances with upper bounds onmemory/disk/cpu....
Fix very slow unit-test data generation in some cases
Learning, learning. It turns out that for objects with even just fivefields, generating them and filtering can be costly, especially whenthe filtering is nested (as it was in this case, arbitrary1 →...
Fix one bad unit-test data generation case
It could happen that the node is generated with 0 disk available, inwhich case the operation will fail due to disk issues, not CPU. Fixthis by using the recently-introduced genOnlineNode helper.
Further JSON unit-tests for various Types.hs types
Add a unit-test helper function
… instead of always using "printTestCase … False".
Extend node evacuation unittests
Currently they only check for change secondary, via iallocator'srelocate method. We add two separate unittests that checks all thenode evacuate modes (ChangeAll, ChangePrimary) and also the groupchange request. All of these are done only for DRBD instances....
Fix a few issues in unittest data generation
This patch fixes two ugly but non-critical issues:
- the getFQDN wrongly generated only the first component as limited-length name, the rest could be any length, and that makes the generated name bad for display purposes...
Remove some dead code
While wondering why Cluster.tryReloc is not unit-tested, I realisedthat this is no longer uses. Apparently, last summer's new IAllocatormodes and related changes have silently deprecated this function.
Fix another data generation issue in tests
Another "good" seed value manages to generate nodes with identicalnames, which of course makes the sets be of smaller value than thelists. Avoid this issue by generating unique node names.
Add a help function for generating test nodes
This abstracts the concept of "give an sane, online node" in a singleplace, rather than spread around the tests. We also change thesignature for isNodeBig for easier usage.
Generate clusters with unique node names in tests
Nothing breaks otherwise in the cluster routines, but serialisationto/from text gets confused otherwise.
Add a complex allocation/serialisation/load test
This is unfortunately a complex test that checks if:
- starting from a empty cluster- allocating a few instances on it- serialising it via the Text backend- loading it back into internal data structures...
Implement saving/restoring of policies in Text backend
This adds a new optional (so we're backwards-compat) section to thecluster data so that we can reliably save/restore the policy data.
Note that different from the other sections, we save all policies...
Further unittests for ipolicy/ispec text serialisation
Fix a couple of bugs in handling offline instances
The addition of the offline state for instances has introduced a fewbugs related to their handling. The current code adds a helperfunction for making the check whether an instances uses reservedmemory on the secondary (and uses it instead of just autoBalance),...
Add a helper function for 'not offline' instances
Unittests for the instance policy checks on allocation
Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Obey instance policy in allocation routines
This currently checks the minimum/maximum sizes and the disktemplate. Unittests will come later.
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
A trivial optimisation in Loader.mergeData
During loading, we can eliminate some intermediate datastructures. This only matters for big clusters, and even there it'sless than 1% less memory allocated, but still, it's a no-brainer.
Trivial performance optimisation
This is not in the hot-path, but still, no sense in doing O(n) insteadof O(1).
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Enable Eq instances for TH-built objects
This is needed for the next patch.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Read the disk templates part of the ipolicy
The default value is badly defined (hardcoded defaults)…
Move DiskTemplate definition around
This is needed since we'll need the DiskTemplate definition in theIPolicy one.
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Add a helper function converting ispecs to rspecs
The rspec type is simpler than the ispec one; most likely it should bedeprecated later.
Abstract creation of instance from a spec
Load cluster ipolicy via Rapi
This requires changing from querying the /tags resource to the /inforesource.
Update memory/maxmem reading in Rapi backend
Recent changes to the instance beparams have replaced memory withmaxmem in Rapi bulk queries. Until this is either reverted (forbackwards compat) or we decide to go ahead with only maxmem, we changethe backend to read both; this only affects the "instance down" code...
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Add default ipolicy declarations
Add object definitions for the ispec and ipolicy
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
htools: split parts of HTools/Types.hs into BasicTypes.hs
The 'Result' type is common and:
- might be used outside of HTools-specific code too- is better split as we need these basic types for building the more complex ones in Types.hs
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Abstract tiered specs command line parsing
This will be used in the future for the standard specs too, so let'sabstract it away.
Some small test simplifications
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Abstract some common hspace code into a function
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Abstract some common Cluster.hs code into a function
Convert opcode TH code to the use of Field type
This makes more explicit the field behaviour - previously an optionalfield was detected via a "Maybe" constructor, and an optional one viaa "Just defval" one. With this, field behaviour become more explicit...
htools: small change in error message in THH.hs
We should also display the value we can't parse, otherwise debuggingis very hard.
htools: improvements to JSON deserialisation
This fixes two problems:
- first, when we deserialise a big object, showing its value is not useful, as it will hide the actual error message- second, we shouldn't deserialise a container at once, because then...
htools: add new template haskell system
This system based on explicit types instead of ad-hoc rules(e.g. instead of deducing from "Maybe Int" an optional field, we nowcan say explicitly OptionalField ''Int). In the first phase, this willbe used for the equivalent of lib/objects.py, which has slightly...
htools: adjust imports for newer compilers
While testing with ghc 7.2, I saw that some imports we are using arevery old (from ghc 6.8 time), even though current libraries are usingdifferent names.
We fix this and bump minimum documented version to ghc 6.12, as I...
Merge branch 'devel-2.5'
Merge branch 'stable-2.5' into devel-2.5
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's freememory reaches 0, it will refuse to add any instances (offline ornot). I think this is a safe safeguard (I don't expect nodes to runwithout at least 1MB of free memory), so rather than change this...
htools: finish re-indenting Cluster.hs
htools: re-indent Node.hs
htools: re-indent QC.hs
htools: reindent CLI.hs
htools: reindent hbal
htools: reindent hspace
htools: re-indent IAlloc.hs
htools: reindent the rest of the files
htools: rework message display construction
While diagnosing some (unrelated) memory usage in htools, I'vestumbled upon some very bad behaviour in checkData: mapAccum isnon-strict, and the tuple we use also, so that results in the list oflist of messages being very bad space-wise (hundreds of MB of memory...
hbal: handle empty node groups
This patch changes an internal assert (which can only be triggeredwhen a node group is empty) into properly handling this case (andreturning empty node/instance lists).
While we could handle this in the backend (Cluster.splitNodeGroup)...
Start converting the indentation to 2-chars
… also do some other small style cleanups.
htools: Nicer arbitrary instantiation
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: change behaviour for offline instances
htools: remove 'running' from instance options
htools: change runSt type to InstanceStatus
htools: create InstanceStatus ADT
Generalize the generation of ADT from raw types
Luxi support for Query status in htools
hail: don't select the primary as new secondary
This just adds the primary node of the instance as 'non-allocable'during the choosing of the new secondary.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(cherry picked from commit 7073b3a86856bcd8d8a62c0b72f82deaabb8d8f1)...
hail: add an extra safety check in relocate
If we select the primary as new secondary, better to fail than returnwrong data to Ganeti.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(cherry picked from commit f25508bef4e85032f0468e5a6f0f8930ff154e66)...
Further cleanup in hspace
This moves the checking of results from the allocation functions to aseparate function, so that we have less code duplication. It also doesa bit of simplification in the printing functions.
A bit of cleanup in hspace
The node offline/mcpu is identical to hbal's setNodesStatus, so let'smove that to CLI.hs and reuse it in hspace (also, rename it and dropone 's').
Also, the check for the number of nodes is obsolete, as we computethat from the disk template....
htools: Simplify Luxi query results parsing
The logic is not entirely correct—the new Query interface exports thefield status, and we don't use that yet. But the new code should bemore readable.
Add a type synonym for the allocation function sig
Both iterateAlloc and tieredAlloc share the same signature, but it'snot documented nor exported (needed for refactoring).
Adjust htools code to new Luxi argument format
This partially undoes commit 92678b3, more specifically it removes theStore data type and the associated code, since all Luxi arguments arenow lists.
Furthermore, since the qfilter field on Query is complex (it's...
Signed-off-by: Michael Hanselmann <hansmi@google.com>...