Do proper name lookup for the -O option
hspace and hbal treat -O differently, and use aliases for short names(although hbal succeeds in that, and hspace doesn't). Uniform this witha name lookup, using the same functions we used for instanceselection/exclusion....
htools: add a machine-readable CLI flag
This will be used in hspace to toggle between "human" readableand machine readable output formats.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
htools: switch hspace to the generic binary
This is the last patch of the binaries conversion.
As information, we now have a single binary that is approx. 5.4MiB insize, compared to 4 binaries that were approx. 5.1-5.2MiB in size;this will result in a smaller package and install size, and the single...
htools: import the program modules in QC.hs
This adds the binaries code to the coverage, and thus the coveragefinally shows the real coverage over all logic code (except for thehtools.hs code, which is not logic code related to the algorithms, soit doesn't matter — plus it's also very small)....
htools: move tiered spec map helper to Hspace.hs
This is used just in hspace, so let's help in making Cluster.hssmaller. We also split the function in two, as computing the spec mapand formatting it are two different tasks.
Signed-off-by: Iustin Pop <iustin@google.com>...
htools: switch hscan to the generic binary
htools: switch hbal to the generic binary
In addition, the patch adds a separate Makefile variable for holdingthe binary roles to make it more clear what we symlink.
htools: switch hail to the generic binary
This converts the first binary to the generic 'htools' binary.
htools: add a compatibility module
When compiling with the parallel-3.x library, we get a deprecationwarning, which makes understanding any other error messagesharder. This patch adds a compatibility module that will hold suchcode for transitioning libraries....
htools: remove no-longer-needed tryMG* functionality
… which was deprecated by the previous patch.
htools: remove ialloc/relocate and multi-evacuate
Since the new node-evacuate mode does both their work and alsosupports better multi-group clusters (including handling splitinstances).
htools: fix potential bug in ialloc/change-group
Currently, the ChangeAll mode of nodeEvac computes the primary groupof the instance and then uses the resulting group index for computingthe group score. However, during the change-group operation (which...
htools: run IAllocator input through checkData
As the IAllocator backend is using a different data path than theothers, it doesn't get the full functionality that loadExternalDatadoes. This results in the current situation where checkData is not runon the input cluster state, which means the node memory properties are...
htools: abstract a function for displaying warnings
This will make it possible to reuse this in IAllocator too.
htools: return the final instance map in ialloc
Similar to the previous patch, this returns the final instance mapfrom the iallocator run, which will allow saving the cluster state forfurther examination/post-processing.
htools: implement post-alloc cluster status display
This patch changes the IAllocator result formatting workflow to returnthe final node list, which can be then used to display the final nodestatus too—currently only the initial status can be shown, which is...
htools: return new state from new IAllocator modes
The old modes already return the node list (as part of AllocSolution),this patch makes the new modes provide this new information.
htools: update the node-evac/change-group result
… per the new design changes. This also requires changes toupdateEvacSolution, as passing in the original instance is nothelpful; we just pass the instance index now and look up the actualinstance in the modified instance map....
htools: add function for getting all nodes of an instance
htools: add change-group IAllocator mode
This builds a lot upon the node-evacuation mode, and itsimplementation is therefore quite easy (once node-evac was added).
htools: add group restrictions to findBestAllocGroup
While a normal multi-group allocation doesn't need to restrict thelist of target groups beyond the unallocable status of some groups,when we relocate instances from one group to the other we need tolimit the subset of searched groups....
htools: generalise and rename availableLocalNodes
Currently this function works in the context of an instance, computingits local group nodes. We can use generalise it to take a group indexargument, so that it's not bound to an instance, and we rename it...
htools: add ChangeGroup to IAllocator types/loader
This patch updates the types and functions in IAllocator.hs andLoader.hs to the new design changes (elimination of multi-relocatemode and replacement with change-group).
It also removes an extra re-export of EvacMode from Loader.hs....
htools: split tryMGAlloc in two
Currently, this function does both identify the best group for aninstance, and format the results (partially). The former functionalityis useful in other contexts too, so let's split the function in two.
htools: add node-evacuation of DRBD all nodes
This is much more complicated than either change primary (which istrivial) and change secondary (which is also simple).
The main strategy is:
- choose a new secondary (that will become the final primary) and...
htools: small optimisation in Cluster.collapseFailures
This makes fewer temporary data structures, and thus brings a bit ofimprovement in total runtime.
htools: add optional limit to allocation count
Currently, the tieredAlloc/iterateAlloc functions will not returnuntil the allocation fails; this means unit-testing theirfunctionality (e.g. that an instance can be allocated) is slow, sincethey will allocate all possible instances....
htools: add node-evacuation of DRBD secondaries
htools: rework compCV so that we can use [Node] too
In some cases, we have a [Node.Node] list, instead of a Node.Liststructure. Instead of converting [Node.Node] to the latter, let'srework compCV and add a compCVNodes such that we skip this(potentially expensive) step....
Fix htool unittest failure due to invalid prereqs
Currently, the test prop_Node_rMem does some reserved memory checks onnodes, and for the test to work correctly (can add more instances) werequire that the node free memory is greater than zero via aprerequisite....
htools: add parseUnit function
This is similar, but not identical, to Utils.ParseUnit. The biggestdifference is that we don't round up/down; as we only use integraltypes, the result will always be rounded down.
Moreover, since (real-world) disk sizes come in SI units, the function...
hspace: add support for units in the options
This patch converts hspace to allow units in all disk and memoryspecifications, and updates the man page accordingly. It also correcta mistake in the man page for the --max-cpu option.
htools: abstract and simplify a multigroup function
In the future, this 'split/associated instance idx list' functionalitywill be used by other code, so let's move it to its own code.
The new method of computing the split also requires exporting one more...
htools: rename an internal constant
'emptySolution' is fine while we have only one Solution type(AllocSolution), but in the future we'll introduce another one andthus we should rename it.
htools: add a function for computing evacuated nodes
The new IAllocator interface is based on instances, not nodes, so weneed to backtrack and compute on which nodes we can't allocateinstances during the current operation. This patch adds a function for...
htools: add two monad-conversion utility functions
For monadic operations, these function will make it easier to convertother monads (or monad-like structure) to our 'Result' type.
htools: add more IAlloc data types and functions
These move the 'not supported' message deeper into the workflow;assuming that the workflow as added by this patch is correct, all thatremains is:
- for disk template DRBD8, plug in the currently existing mechanisms...
htools: add node-evacuate of DRBD8 primaries
htools: simplify select tests
Rather than running all possible tests in one go, split them out inthree cases: "normal", undefined default value, undefined list values(both of which require a valid response before them).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
htools: lookup instance names in select/exclude
Currently we just match the passed name with fqdns. With this change wedo a proper search for either the fqdn or the initial name component, asin Ganeti itself.
htools: Add if' and select as functions
if' is just a functional version of the if-then-else construct. Using itwe can manipulate ifs as functions, and also build other functionality.
select allows to choose a result depending on truth value of statements,...
htools: infrastructure for looking up names
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: remove obsolete option INodes
hspace no longer uses it, hence it can be removed.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: add an utility function for disk templates
In the future we'll use this instead of requiring the number of nodesfrom the user.
htools: define the --disk-template option
Also sorts the export list correctly.
htools: read/save the disk template in Text backend
This requires that we change the signature of loadInst in order toproperly annotate the error messages, which in turn requires moreunittest changes.
Also, this invalidates yet again saved data files…...
htools: read the disk_template in the Luxi backend
We need to request the disk template in the Luxi query and then use itwhen parsing the response.
htools: read the disk template in Luxi and Rapi
Both these backends already have the data, we just need to use it.
htools: add DiskTemplate to instance definition
Currently always fixed to DRBD8, as that is what we assume already.
htools: add some unittests for Types.hs
htools: add a DiskTemplate type
This will be used later in the Instance type.
htools: small simplification to parseUtilisation
We eliminate the local variable as it's used only once.
htools: rename Instance.auto_balance
… to autoBalance, per the recommended style.
htools: simplify some JSON-related code
Instead of manually encoding the JSValue(s), we can use showJSON aslong as we use data types that can be auto-converted.
htools: allow different result types
This patch extends the previous patch changes to allow that eachCluster function returns a different result type, as long as we havean auxiliary function that processes that into the standardIAllocResult type.
htools: start reorganising the IAllocator workflow
Currently, all IAllocator requests return the same result type: a listof strings (usually nodes, but can also be instance:nodes). This won'tbe true in the future, with the new request types that will return...
htools: split more code from hail/main
The logic of request processing is not separated into its ownfunction, and (beside CLI interaction, e.g. verbosity handling) alllogic is now in IAllocator.hs.
htools: move code from hail.hs to IAllocator.hs
This will make it easier to unittest the code, and keeps all login in asingle module.
htools: add parsing of input node-evac messages
This is a lot of duplication with modeMreloc, which will get sortedout once we convert mreloc to change-group.
htools: remove pre-2.4 compatibility from RAPI
This was potentially hiding issues in the group query, so let's removeit since we're now bound to the current Ganeti version.
htools: introduce new data type for node-evac
The evac-mode goes into the Types.hs module since it needs to beavailable to Cluster.hs too.
htools: docstring fixes and improvements
No code changes (except one definition being moved around in QC.hs)are contained in this patch.
One Haskell and integer sizes fix
Haskell has two main integer types:
- Int, which is a native-type, and is guaranteed to have at least [-2²⁹, 2²⁹-1] range; on 64-bit platforms, it has much higher range- Integer, which is a software type (implemented using libgmp), and...
htools: Rework some unittests
The new scaffolding which replaced the batch driver of QuickCheck 1 nowshows how many passes we have for incomplete tests. Some tests show verylow pass counts, so we rework them to have more actually valid testcases.
htools: more fixes to unsatisfiable tests
Currently the way we generate nodes in some cases is by creating atotally random node, then restricting the test based on whether the node'size' (as defined by multiples of base unit) satifies some high/lowrules. This results in hard-to-satisfy conditions, so we change this...
htools: improve more unittests
Using new functionality in QuickCheck 2 (the suchThat function), wegenerate now better test cases, such that (heh) we have no longerincomplete tests.
htools: allow unittest to be replayed
This just adds glue to allow replaying of tests using a given RNG stateand test size (both are needed for exact replayability).
htools: fix corner case in prop_Text_Load_Instance
This unittest had a corner case where it could fail if the sameprimary/secondary node names were generated.
htools: fix prop_Node_rMem corner case
This patch fixes a bug in the test specification where we allowed nodeswith zero free memory (hence no instance can be added, at all) and addsa simple labeling of the way this test can fail.
htools: Switch to QuickCheck 2.x
Since current distros don't package anymore QuickCheck 1.x, let's moveto 2.x.
This requires also a few changes to the code:
- Test.QuickCheck.Batch doesn't exist anymore, so we need to write some scaffolding code to replace it...
htools: some lint fixes
Removal of duplicate parantheses, removal of extra 'do', conversion fromnested if to guards, use hierarchical imports. All per hlint.
htools: rewrite Cluster.filterMGResults
filterMGResults was built using a sequence of map and filter calls;while this was logically correct, it used some incomplete patternmatching which with the new GHC 7 triggers a warning.
The patch rewrites it using a single foldl that does both the filtering...
htools: use the new Group.isAllocable
… instead of the hardcoded test against AllocUnallocable.
htools: add a helper function
… that checks if a group is allocable.
Check that the selected instances are known
Loader.updateMovable: evaluate selected instances
This also adds docstrings for the function arguments and renames exinstto exinsts, which is how it is called in other functions, since it's alist.
Add instance selection list to Loader.mergeData
Add --select-instances hbal flag
Remove double whitespace in help string
htools: introduce a type alias for JSON objects
This makes the type definitions a bit more readable/simpler.
hail: stop using old-style 'nodes' key
hail: add parsing of multi-relocate request
This is not handled yet, this patch just adds parsing of the incomingrequest.
hail: add new data types for the multi-reloc mode
Implement balancing with no instance moves
Note that --no-disk-moves and --no-instance-moves are not incompatible,but if both are used no solution can possibly exist.
Add --no-instance-moves cli htools option
This option doesn't currently do anything.
IAllocator.hs: replace a few strings with constants
Replace instance states hardcoded with constants
Convert group policies to constants
Convert job status strings to constants
Misc other conversions
Implement conversion of Python constants to Haskell
With the merge of the repositories, we can now auto-generate the codefor Haskell constants from the Python code.
Currently this only handles the basic types (strings andintegers). Handling containers such as lists and dictionaries is only...
htools: make some error messages more explicit
Add some tests for the auto_balance attribute
It tests node add/remove secondary, rather than cluster-level N+1checks, but it's better than nothing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Node operations: take into account auto_balance
This patch changes the add to secondary/remove from secondary code tonot deduct/add the instance's memory if the instance is notauto_balanced.
Read/write auto_balance via Text
This also means another change in the text format; we really shouldmove to json…
The unittests are also update for the new 9-column layout andadditionally a bit of improvement is done.
Read auto_balance via Rapi
Read auto_balance via Luxi
Show the auto_balance flag in the instance listing
Add a new attribute to Instance.Instance
This will mirror Ganeti's be/auto_balance one, which we need to use toproperly match N+1 computations.
htools: Make opcode naming consistent with Ganeti codebase
This patch just cleans up the htools codebase to make it more consistentwith the naming of the Ganeti codebase.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>