htools: switch hspace to the generic binary
This is the last patch of the binaries conversion.
As information, we now have a single binary that is approx. 5.4MiB insize, compared to 4 binaries that were approx. 5.1-5.2MiB in size;this will result in a smaller package and install size, and the single...
htools: import the program modules in QC.hs
This adds the binaries code to the coverage, and thus the coveragefinally shows the real coverage over all logic code (except for thehtools.hs code, which is not logic code related to the algorithms, soit doesn't matter — plus it's also very small)....
htools: switch hscan to the generic binary
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
htools: switch hbal to the generic binary
In addition, the patch adds a separate Makefile variable for holdingthe binary roles to make it more clear what we symlink.
htools: switch hail to the generic binary
This converts the first binary to the generic 'htools' binary.
htools: add a generic binary
This is the start of a series of patches that will unify all thebinaries currently in use in a single one, which can perform differentroles based on the name it is installed as.
Signed-off-by: Iustin Pop <iustin@google.com>...
htools: add a compatibility module
When compiling with the parallel-3.x library, we get a deprecationwarning, which makes understanding any other error messagesharder. This patch adds a compatibility module that will hold suchcode for transitioning libraries....
htools: remove no-longer-needed tryMG* functionality
… which was deprecated by the previous patch.
htools: remove ialloc/relocate and multi-evacuate
Since the new node-evacuate mode does both their work and alsosupports better multi-group clusters (including handling splitinstances).
htools: fix potential bug in ialloc/change-group
Currently, the ChangeAll mode of nodeEvac computes the primary groupof the instance and then uses the resulting group index for computingthe group score. However, during the change-group operation (which...
htools: run IAllocator input through checkData
As the IAllocator backend is using a different data path than theothers, it doesn't get the full functionality that loadExternalDatadoes. This results in the current situation where checkData is not runon the input cluster state, which means the node memory properties are...
htools: abstract a function for displaying warnings
This will make it possible to reuse this in IAllocator too.
htools: use maybePrintNodes in hail.hs
This eliminates duplication of codes (and was forgotten back whenmaybePrintNodes was added).
htools: add cluster state saving support to hail
This add support for saving the cluster state (both pre- andpost-iallocator run) to a text file such that it can be fed back intoany of the htools commands.
htools: return the final instance map in ialloc
Similar to the previous patch, this returns the final instance mapfrom the iallocator run, which will allow saving the cluster state forfurther examination/post-processing.
htools: implement post-alloc cluster status display
This patch changes the IAllocator result formatting workflow to returnthe final node list, which can be then used to display the final nodestatus too—currently only the initial status can be shown, which is...
htools: return new state from new IAllocator modes
The old modes already return the node list (as part of AllocSolution),this patch makes the new modes provide this new information.
htools: update the node-evac/change-group result
… per the new design changes. This also requires changes toupdateEvacSolution, as passing in the original instance is nothelpful; we just pass the instance index now and look up the actualinstance in the modified instance map....
htools: add function for getting all nodes of an instance
htools: add change-group IAllocator mode
This builds a lot upon the node-evacuation mode, and itsimplementation is therefore quite easy (once node-evac was added).
htools: add group restrictions to findBestAllocGroup
While a normal multi-group allocation doesn't need to restrict thelist of target groups beyond the unallocable status of some groups,when we relocate instances from one group to the other we need tolimit the subset of searched groups....
htools: generalise and rename availableLocalNodes
Currently this function works in the context of an instance, computingits local group nodes. We can use generalise it to take a group indexargument, so that it's not bound to an instance, and we rename it...
htools: add ChangeGroup to IAllocator types/loader
This patch updates the types and functions in IAllocator.hs andLoader.hs to the new design changes (elimination of multi-relocatemode and replacement with change-group).
It also removes an extra re-export of EvacMode from Loader.hs....
htools: split tryMGAlloc in two
Currently, this function does both identify the best group for aninstance, and format the results (partially). The former functionalityis useful in other contexts too, so let's split the function in two.
htools: add node-evacuation of DRBD all nodes
This is much more complicated than either change primary (which istrivial) and change secondary (which is also simple).
The main strategy is:
- choose a new secondary (that will become the final primary) and...
htools: small optimisation in Cluster.collapseFailures
This makes fewer temporary data structures, and thus brings a bit ofimprovement in total runtime.
htools: add optional limit to allocation count
Currently, the tieredAlloc/iterateAlloc functions will not returnuntil the allocation fails; this means unit-testing theirfunctionality (e.g. that an instance can be allocated) is slow, sincethey will allocate all possible instances....
htools: add node-evacuation of DRBD secondaries
htools: rework compCV so that we can use [Node] too
In some cases, we have a [Node.Node] list, instead of a Node.Liststructure. Instead of converting [Node.Node] to the latter, let'srework compCV and add a compCVNodes such that we skip this(potentially expensive) step....
Mark htools/README as obsolete
And point to the main INSTALL file for up-to-date information.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix htool unittest failure due to invalid prereqs
Currently, the test prop_Node_rMem does some reserved memory checks onnodes, and for the test to work correctly (can add more instances) werequire that the node free memory is greater than zero via aprerequisite....
htools: add parseUnit function
This is similar, but not identical, to Utils.ParseUnit. The biggestdifference is that we don't round up/down; as we only use integraltypes, the result will always be rounded down.
Moreover, since (real-world) disk sizes come in SI units, the function...
hspace: add support for units in the options
This patch converts hspace to allow units in all disk and memoryspecifications, and updates the man page accordingly. It also correcta mistake in the man page for the --max-cpu option.
hspace: also display the starting tiered specs
These are useful for checking what instance specifications we startwith (as they are not necessarily the first entry in the TSPEC map).
htools: abstract and simplify a multigroup function
In the future, this 'split/associated instance idx list' functionalitywill be used by other code, so let's move it to its own code.
The new method of computing the split also requires exporting one more...
htools: rename an internal constant
'emptySolution' is fine while we have only one Solution type(AllocSolution), but in the future we'll introduce another one andthus we should rename it.
htools: add a function for computing evacuated nodes
The new IAllocator interface is based on instances, not nodes, so weneed to backtrack and compute on which nodes we can't allocateinstances during the current operation. This patch adds a function for...
htools: add two monad-conversion utility functions
For monadic operations, these function will make it easier to convertother monads (or monad-like structure) to our 'Result' type.
htools: add more IAlloc data types and functions
These move the 'not supported' message deeper into the workflow;assuming that the workflow as added by this patch is correct, all thatremains is:
- for disk template DRBD8, plug in the currently existing mechanisms...
htools: add node-evacuate of DRBD8 primaries
htools: simplify select tests
Rather than running all possible tests in one go, split them out inthree cases: "normal", undefined default value, undefined list values(both of which require a valid response before them).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
htools: lookup instance names in select/exclude
Currently we just match the passed name with fqdns. With this change wedo a proper search for either the fqdn or the initial name component, asin Ganeti itself.
htools: Add if' and select as functions
if' is just a functional version of the if-then-else construct. Using itwe can manipulate ifs as functions, and also build other functionality.
select allows to choose a result depending on truth value of statements,...
htools: infrastructure for looking up names
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: remove obsolete option INodes
hspace no longer uses it, hence it can be removed.
hspace: also report the disk template
hspace: ask for the disk_template not number of nodes
This makes more sense, as the disk template is more meaningful thanthe number of nodes.
htools: add an utility function for disk templates
In the future we'll use this instead of requiring the number of nodesfrom the user.
htools: define the --disk-template option
Also sorts the export list correctly.
htools: read/save the disk template in Text backend
This requires that we change the signature of loadInst in order toproperly annotate the error messages, which in turn requires moreunittest changes.
Also, this invalidates yet again saved data files…...
htools: read the disk_template in the Luxi backend
We need to request the disk template in the Luxi query and then use itwhen parsing the response.
htools: read the disk template in Luxi and Rapi
Both these backends already have the data, we just need to use it.
htools: add DiskTemplate to instance definition
Currently always fixed to DRBD8, as that is what we assume already.
htools: add some unittests for Types.hs
htools: add a DiskTemplate type
This will be used later in the Instance type.
htools: small simplification to parseUtilisation
We eliminate the local variable as it's used only once.
htools: rename Instance.auto_balance
… to autoBalance, per the recommended style.
htools: simplify some JSON-related code
Instead of manually encoding the JSValue(s), we can use showJSON aslong as we use data types that can be auto-converted.
htools: allow different result types
This patch extends the previous patch changes to allow that eachCluster function returns a different result type, as long as we havean auxiliary function that processes that into the standardIAllocResult type.
htools: start reorganising the IAllocator workflow
Currently, all IAllocator requests return the same result type: a listof strings (usually nodes, but can also be instance:nodes). This won'tbe true in the future, with the new request types that will return...
htools: split more code from hail/main
The logic of request processing is not separated into its ownfunction, and (beside CLI interaction, e.g. verbosity handling) alllogic is now in IAllocator.hs.
htools: move code from hail.hs to IAllocator.hs
This will make it easier to unittest the code, and keeps all login in asingle module.
htools: add parsing of input node-evac messages
This is a lot of duplication with modeMreloc, which will get sortedout once we convert mreloc to change-group.
htools: remove pre-2.4 compatibility from RAPI
This was potentially hiding issues in the group query, so let's removeit since we're now bound to the current Ganeti version.
htools: introduce new data type for node-evac
The evac-mode goes into the Types.hs module since it needs to beavailable to Cluster.hs too.
htools: docstring fixes and improvements
No code changes (except one definition being moved around in QC.hs)are contained in this patch.
One Haskell and integer sizes fix
Haskell has two main integer types:
- Int, which is a native-type, and is guaranteed to have at least [-2²⁹, 2²⁹-1] range; on 64-bit platforms, it has much higher range- Integer, which is a software type (implemented using libgmp), and...
htools: Rework some unittests
The new scaffolding which replaced the batch driver of QuickCheck 1 nowshows how many passes we have for incomplete tests. Some tests show verylow pass counts, so we rework them to have more actually valid testcases.
htools: more fixes to unsatisfiable tests
Currently the way we generate nodes in some cases is by creating atotally random node, then restricting the test based on whether the node'size' (as defined by multiples of base unit) satifies some high/lowrules. This results in hard-to-satisfy conditions, so we change this...
htools: improve more unittests
Using new functionality in QuickCheck 2 (the suchThat function), wegenerate now better test cases, such that (heh) we have no longerincomplete tests.
htools: allow unittest to be replayed
This just adds glue to allow replaying of tests using a given RNG stateand test size (both are needed for exact replayability).
htools: fix corner case in prop_Text_Load_Instance
This unittest had a corner case where it could fail if the sameprimary/secondary node names were generated.
htools: fix prop_Node_rMem corner case
This patch fixes a bug in the test specification where we allowed nodeswith zero free memory (hence no instance can be added, at all) and addsa simple labeling of the way this test can fail.
htools: Switch to QuickCheck 2.x
Since current distros don't package anymore QuickCheck 1.x, let's moveto 2.x.
This requires also a few changes to the code:
- Test.QuickCheck.Batch doesn't exist anymore, so we need to write some scaffolding code to replace it...
htools: some lint fixes
Removal of duplicate parantheses, removal of extra 'do', conversion fromnested if to guards, use hierarchical imports. All per hlint.
htools: rewrite Cluster.filterMGResults
filterMGResults was built using a sequence of map and filter calls;while this was logically correct, it used some incomplete patternmatching which with the new GHC 7 triggers a warning.
The patch rewrites it using a single foldl that does both the filtering...
htools: use the new Group.isAllocable
… instead of the hardcoded test against AllocUnallocable.
htools: add a helper function
… that checks if a group is allocable.
htools live test: test instance selection as well
Check that the selected instances are known
Loader.updateMovable: evaluate selected instances
This also adds docstrings for the function arguments and renames exinstto exinsts, which is how it is called in other functions, since it's alist.
Add instance selection list to Loader.mergeData
Add --select-instances hbal flag
Remove double whitespace in help string
htools: introduce a type alias for JSON objects
This makes the type definitions a bit more readable/simpler.
hail: stop using old-style 'nodes' key
hail: add parsing of multi-relocate request
This is not handled yet, this patch just adds parsing of the incomingrequest.
hail: add option for displaying the parsed request
This can be used for debugging.
hail: add new data types for the multi-reloc mode
Add --no-instance-moves to the htools live tests
Implement balancing with no instance moves
Note that --no-disk-moves and --no-instance-moves are not incompatible,but if both are used no solution can possibly exist.
Pass the instance moves option in hbal
While still being ignored, now it gets passed down to the iterationfunction.
Add --no-instance-moves cli htools option
This option doesn't currently do anything.
IAllocator.hs: replace a few strings with constants
Replace instance states hardcoded with constants
Convert group policies to constants
Convert job status strings to constants
Misc other conversions
Implement conversion of Python constants to Haskell
With the merge of the repositories, we can now auto-generate the codefor Haskell constants from the Python code.
Currently this only handles the basic types (strings andintegers). Handling containers such as lists and dictionaries is only...
htools: make some error messages more explicit