Add a type synonym for the allocation function sig
Both iterateAlloc and tieredAlloc share the same signature, but it'snot documented nor exported (needed for refactoring).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: Simplify Luxi query results parsing
The logic is not entirely correct—the new Query interface exports thefield status, and we don't use that yet. But the new code should bemore readable.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Adjust htools code to new Luxi argument format
This partially undoes commit 92678b3, more specifically it removes theStore data type and the associated code, since all Luxi arguments arenow lists.
Furthermore, since the qfilter field on Query is complex (it's...
Merge branch 'devel-2.5'
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Merge branch 'stable-2.5' into devel-2.5
Rename filter and filter_ to qfilter
We currently use 'filter' as the OpCode, QueryRequest and RAPI fieldname for representing a query filter. However, since 'filter' is abuilt-in function, we actually have to use filter_ throughout the codein order to not override the built-in function....
Haskell support for generic Query in Luxi
Untill now htools did not have support for generic Query in Luxi. Thispatch introduces Query as a supported Luxi operation and replacesQueryNodes, QueryInstances and QueryGroups with Query.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
TH simplification for Luxi
This patch simplifies the generation of save constructors for LuxiOpby always using showJSON over an array of JSValues, instead of havingto pass showJSON in most cases, except the 5-tuple case.
Dots in docstings and hlint error fixes for htools
Signed-off-by: Agata Murawska <agatamurawska@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove the oneline output option in hbal
This was, AFAIK, never used, and complicates the output code enoughthat it's better to remove it.
Rework/split hbal's main function
This is just moving code around. A subsequent patch will do a bit morecleanup and changing the output.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Skip application of 'id' in TH code
This is just beautification when dumping splices to stdout, as ghcwill optimise the 'id' away anyway.
Original generate code:
opToArgs QueryTags kind name = J.showJSON (id kind, id name)
Afterwards:
opToArgs QueryTags kind name = J.showJSON (kind, name)...
hail: Fix result for node evacuation
According to the iallocator documentation the “node-evacuate” call needsto return a list of jobs, not a list of lists of jobs.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Use TemplateHaskell to create LUXI operations
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Some TH simplifications
Now that the basic code works, let's use some aliases for simpler codeand less ))))))))).
A few minor test improvements
This patch adds a few niceties to the test suite:
- allows matching test groups case insensitive and emit warnings when we give test group names that don't match anything- add a new operator that is similar to assertEqual in Python: it...
Use TemplateHaskell to decorate tests with names
This makes error message change from "Test 4 failed …" to "Testprop_Loader_mergeData failed", which is much more readable. It alsoremoves the duplication of test suite names in the test.hs file.
Signed-off-by: Iustin Pop <iustin@google.com>...
Use TemplateHaskell to generate opcode serialisation
This replaces the hand-coded opcode serialisation code withauto-generation based on TemplateHaskell.
Use TemplateHaskell to build the opID function
This replaces the hand-coded opID with one automatically generatedfrom the constructor names, similar to the way Python does it, exceptit's done at compilation time as opposed to runtime.
Again, the code line delta does not favour this patch, but this...
Use TemplateHaskell instead of hand-coded instances
This patch replaces the current hard-coded JSON instances (all alike,just manual conversion to/from string) with auto-generated code basedon Template Haskell(http://www.haskell.org/haskellwiki/Template_Haskell)....
Rename some helper functions for consistency
This changes the names for some helper functions so that futurepatches are touching less unrelated code. The change replacesshortened prefixes with the full type name.
Split part of Utils.hs into JSON.hs
Utils is a bit big, let's split the JSON stuff (not all of it) into aseparate module that doesn't have any other dependencies.
Add an allocation limit to hspace
This is very useful for testing/benchmarking.
Small simplification in tryAlloc
Change how node pairs are generated/used
Currently, the node pairs used for allocation are a simple [(primary,secondary)] list of tuples, as this is how they were used before theprevious patch. However, for that patch, we use them separately perprimary node, and we have to unpack this list right after generation....
Parallelise instance allocation/capacity computation
This patch finally enables parallelisation in instance placement.
My original try for enabling this didn't work well, but it took awhile (and liberal use of threadscope) to understand why. The attempt...
Abstract comparison of AllocElements
This is moved outside of the concatAllocs as it will be needed inanother place in the future.
Change type of Cluster.AllocSolution
Originally, this data type was used both by instance allocation (1result), and by instance relocation (many results, one perinstance). As such, the field 'asSolutions' was a list, and thevarious code paths checked whether the length of the list matches the...
hspace: add short forms for the group policy
This adds a shortened versions of the allocation policies, as writingout the whole name in the command line can become tedious.
htools: add a MonadPlus instance for Result
This will be used to implement more easily 'choice' parsing of inputdata, without resorting to syntax (case … of Bad _ -> …).
htools: remove dead code
The tryEvac/evacuateInstance functions are no longer used in the newmulti-group world order, so we remove them and change the unit-test totest the actual IAllocator function.
hail: don't select the primary as new secondary
This just adds the primary node of the instance as 'non-allocable'during the choosing of the new secondary.
hail: add an extra safety check in relocate
If we select the primary as new secondary, better to fail than returnwrong data to Ganeti.
Revert deprecation of evacuate mode in hail
As discussed offline, the new node-change mode could be used forevacuation, but it's not directly useful as it returns a list ofopcodes; therefore, we need to partially revert commits fbe5fcf and5b53ca7 that removed it (and multi-evacuate, which remains removed)....
htools: rework the algorithm for ChangeAll mode
I think I've identified the problem with the current ChangeAllmode. The current algorithm works as follows:
- identify a new primary by choosing the node which gives best score as new secondary- failover to it...
htools: replace two hardcoded uses of pri+sec nodes
These two cases use explicit uses of primary and secondary nodes withInstance.allNodes, which means the code is more flexible if theinternal layout of the instance changes.
I've verified that the output of involvedNodes is not required to be...
htools: add target_node member to migrate opcode
… and failover too. Not many changes otherwise except forserialisation and unittests.
htools: do not change node disk for non-local storage
htools: add more functions for local disk storage
These will be used in Node.hs for proper add/remove instancecode. Furthermore, we restrict the movable status to the right disktemplates only, so that we don't attempt to move the 'wrong' instancetypes....
htools: add two more small unittests
This adds tests for the opToResult and eitherToResult functions fromTypes.hs, and changes two other tests for the same module to test JSONserialisation (which automatically also tests the lower-level to/fromstring conversion functions)....
htools: a few more hlint fixes
Tested only on GHC 7.x, will test on 6.1x too before commit.
htools: further docstring fixes
This adds parameter documentation for Cluster.iMoveToJob (I think itwas not clear if the new or old node list is needed) and fixes otherdocstring style issues.
After this patch, all modules except for CLI.hs (which has many...
htools: add JSON instance for EvacMode
This abstracts the JSON parsing of the type EvacMode near itsdefinition, and simplifies its conversion in IAlloc.parseData.
htools: add human-readable output to hspace
Currently, hspace can only output a machine-readable format that(while detailed) is hard to parse quickly by people. This patch adds(and enables by default) a human-readable output that shows the mostimportant metrics in a simple format....
Fix job constants use in htools
Commit 56c094b4 added use of job constants, but I didn't payattention and ended up mixing things: job constants were used foropcode ones, and the job ones didn't get converted.
This patch corrects it and uses only C.* constants throughout the Jobs...
Rename *_STATUS_WAITLOCK to …_WAITING
This patch renames the {JOB,OP}_STATUS_WAITLOCK constants to {JOB,OP}_STATUS_WAITING, as per design document for chained jobs.
Do proper name lookup for the -O option
hspace and hbal treat -O differently, and use aliases for short names(although hbal succeeds in that, and hspace doesn't). Uniform this witha name lookup, using the same functions we used for instanceselection/exclusion....
htools: add a machine-readable CLI flag
This will be used in hspace to toggle between "human" readableand machine readable output formats.
htools: move tiered spec map helper to Hspace.hs
This is used just in hspace, so let's help in making Cluster.hssmaller. We also split the function in two, as computing the spec mapand formatting it are two different tasks.
htools: import the program modules in QC.hs
This adds the binaries code to the coverage, and thus the coveragefinally shows the real coverage over all logic code (except for thehtools.hs code, which is not logic code related to the algorithms, soit doesn't matter — plus it's also very small)....
htools: switch hspace to the generic binary
This is the last patch of the binaries conversion.
As information, we now have a single binary that is approx. 5.4MiB insize, compared to 4 binaries that were approx. 5.1-5.2MiB in size;this will result in a smaller package and install size, and the single...
htools: switch hscan to the generic binary
htools: switch hbal to the generic binary
In addition, the patch adds a separate Makefile variable for holdingthe binary roles to make it more clear what we symlink.
htools: switch hail to the generic binary
This converts the first binary to the generic 'htools' binary.
htools: add a generic binary
This is the start of a series of patches that will unify all thebinaries currently in use in a single one, which can perform differentroles based on the name it is installed as.
htools: add a compatibility module
When compiling with the parallel-3.x library, we get a deprecationwarning, which makes understanding any other error messagesharder. This patch adds a compatibility module that will hold suchcode for transitioning libraries....
htools: remove no-longer-needed tryMG* functionality
… which was deprecated by the previous patch.
htools: remove ialloc/relocate and multi-evacuate
Since the new node-evacuate mode does both their work and alsosupports better multi-group clusters (including handling splitinstances).
htools: fix potential bug in ialloc/change-group
Currently, the ChangeAll mode of nodeEvac computes the primary groupof the instance and then uses the resulting group index for computingthe group score. However, during the change-group operation (which...
htools: run IAllocator input through checkData
As the IAllocator backend is using a different data path than theothers, it doesn't get the full functionality that loadExternalDatadoes. This results in the current situation where checkData is not runon the input cluster state, which means the node memory properties are...
htools: abstract a function for displaying warnings
This will make it possible to reuse this in IAllocator too.
htools: use maybePrintNodes in hail.hs
This eliminates duplication of codes (and was forgotten back whenmaybePrintNodes was added).
htools: add cluster state saving support to hail
This add support for saving the cluster state (both pre- andpost-iallocator run) to a text file such that it can be fed back intoany of the htools commands.
htools: return the final instance map in ialloc
Similar to the previous patch, this returns the final instance mapfrom the iallocator run, which will allow saving the cluster state forfurther examination/post-processing.
htools: implement post-alloc cluster status display
This patch changes the IAllocator result formatting workflow to returnthe final node list, which can be then used to display the final nodestatus too—currently only the initial status can be shown, which is...
htools: return new state from new IAllocator modes
The old modes already return the node list (as part of AllocSolution),this patch makes the new modes provide this new information.
htools: update the node-evac/change-group result
… per the new design changes. This also requires changes toupdateEvacSolution, as passing in the original instance is nothelpful; we just pass the instance index now and look up the actualinstance in the modified instance map....
htools: add function for getting all nodes of an instance
htools: add change-group IAllocator mode
This builds a lot upon the node-evacuation mode, and itsimplementation is therefore quite easy (once node-evac was added).
htools: add group restrictions to findBestAllocGroup
While a normal multi-group allocation doesn't need to restrict thelist of target groups beyond the unallocable status of some groups,when we relocate instances from one group to the other we need tolimit the subset of searched groups....
htools: generalise and rename availableLocalNodes
Currently this function works in the context of an instance, computingits local group nodes. We can use generalise it to take a group indexargument, so that it's not bound to an instance, and we rename it...
htools: add ChangeGroup to IAllocator types/loader
This patch updates the types and functions in IAllocator.hs andLoader.hs to the new design changes (elimination of multi-relocatemode and replacement with change-group).
It also removes an extra re-export of EvacMode from Loader.hs....
htools: split tryMGAlloc in two
Currently, this function does both identify the best group for aninstance, and format the results (partially). The former functionalityis useful in other contexts too, so let's split the function in two.
htools: add node-evacuation of DRBD all nodes
This is much more complicated than either change primary (which istrivial) and change secondary (which is also simple).
The main strategy is:
- choose a new secondary (that will become the final primary) and...
htools: small optimisation in Cluster.collapseFailures
This makes fewer temporary data structures, and thus brings a bit ofimprovement in total runtime.
htools: add optional limit to allocation count
Currently, the tieredAlloc/iterateAlloc functions will not returnuntil the allocation fails; this means unit-testing theirfunctionality (e.g. that an instance can be allocated) is slow, sincethey will allocate all possible instances....
htools: add node-evacuation of DRBD secondaries
htools: rework compCV so that we can use [Node] too
In some cases, we have a [Node.Node] list, instead of a Node.Liststructure. Instead of converting [Node.Node] to the latter, let'srework compCV and add a compCVNodes such that we skip this(potentially expensive) step....
Mark htools/README as obsolete
And point to the main INSTALL file for up-to-date information.
Fix htool unittest failure due to invalid prereqs
Currently, the test prop_Node_rMem does some reserved memory checks onnodes, and for the test to work correctly (can add more instances) werequire that the node free memory is greater than zero via aprerequisite....
htools: add node-evacuate of DRBD8 primaries
htools: add more IAlloc data types and functions
These move the 'not supported' message deeper into the workflow;assuming that the workflow as added by this patch is correct, all thatremains is:
- for disk template DRBD8, plug in the currently existing mechanisms...
htools: add two monad-conversion utility functions
For monadic operations, these function will make it easier to convertother monads (or monad-like structure) to our 'Result' type.
htools: add a function for computing evacuated nodes
The new IAllocator interface is based on instances, not nodes, so weneed to backtrack and compute on which nodes we can't allocateinstances during the current operation. This patch adds a function for...
htools: rename an internal constant
'emptySolution' is fine while we have only one Solution type(AllocSolution), but in the future we'll introduce another one andthus we should rename it.
htools: abstract and simplify a multigroup function
In the future, this 'split/associated instance idx list' functionalitywill be used by other code, so let's move it to its own code.
The new method of computing the split also requires exporting one more...
hspace: also display the starting tiered specs
These are useful for checking what instance specifications we startwith (as they are not necessarily the first entry in the TSPEC map).
hspace: add support for units in the options
This patch converts hspace to allow units in all disk and memoryspecifications, and updates the man page accordingly. It also correcta mistake in the man page for the --max-cpu option.
htools: add parseUnit function
This is similar, but not identical, to Utils.ParseUnit. The biggestdifference is that we don't round up/down; as we only use integraltypes, the result will always be rounded down.
Moreover, since (real-world) disk sizes come in SI units, the function...
htools: simplify select tests
Rather than running all possible tests in one go, split them out inthree cases: "normal", undefined default value, undefined list values(both of which require a valid response before them).
Signed-off-by: Guido Trotter <ultrotter@google.com>...
htools: lookup instance names in select/exclude
Currently we just match the passed name with fqdns. With this change wedo a proper search for either the fqdn or the initial name component, asin Ganeti itself.
htools: infrastructure for looking up names
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Add if' and select as functions
if' is just a functional version of the if-then-else construct. Using itwe can manipulate ifs as functions, and also build other functionality.
select allows to choose a result depending on truth value of statements,...
htools: remove obsolete option INodes
hspace no longer uses it, hence it can be removed.
hspace: also report the disk template
hspace: ask for the disk_template not number of nodes
This makes more sense, as the disk template is more meaningful thanthe number of nodes.
htools: add an utility function for disk templates
In the future we'll use this instead of requiring the number of nodesfrom the user.
htools: define the --disk-template option
Also sorts the export list correctly.
htools: read/save the disk template in Text backend
This requires that we change the signature of loadInst in order toproperly annotate the error messages, which in turn requires moreunittest changes.
Also, this invalidates yet again saved data files…...
htools: read the disk_template in the Luxi backend
We need to request the disk template in the Luxi query and then use itwhen parsing the response.