Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Signed-off-by: Iustin Pop <iustin@google.com>...
Obey instance policy in allocation routines
This currently checks the minimum/maximum sizes and the disktemplate. Unittests will come later.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Set group policy on nodes
In order to be able to use the group policy during allocation, we havea few choices:
- pass the group policy as a separate argument to all involved functions, or- add the policy as an attribute to nodes, or- add the group (not only the group index) as an attribute to nodes...
A trivial optimisation in Loader.mergeData
During loading, we can eliminate some intermediate datastructures. This only matters for big clusters, and even there it'sless than 1% less memory allocated, but still, it's a no-brainer.
Trivial performance optimisation
This is not in the hot-path, but still, no sense in doing O(n) insteadof O(1).
Add ipolicy attribute to group objects
This is straightforward; as before, so far the text and simu backendshave no way to use a non-default ipolicy.
Furthermore, we do some unit-test preparation for future changes(these seem a no-op for now, but they are needed for later patches)....
Enable Eq instances for TH-built objects
This is needed for the next patch.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Read the disk templates part of the ipolicy
The default value is badly defined (hardcoded defaults)…
Move DiskTemplate definition around
This is needed since we'll need the DiskTemplate definition in theIPolicy one.
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Add a helper function converting ispecs to rspecs
The rspec type is simpler than the ispec one; most likely it should bedeprecated later.
Abstract creation of instance from a spec
Load cluster ipolicy via Rapi
This requires changing from querying the /tags resource to the /inforesource.
Update memory/maxmem reading in Rapi backend
Recent changes to the instance beparams have replaced memory withmaxmem in Rapi bulk queries. Until this is either reverted (forbackwards compat) or we decide to go ahead with only maxmem, we changethe backend to read both; this only affects the "instance down" code...
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Add default ipolicy declarations
Add object definitions for the ispec and ipolicy
Stop exporting JSON functionality from Utils.hs
This completes the Utils/JSON split started in commit f047f90f. Theimport graph should be cleaner now.
More reshuffling of code
Following the split Types/BasicTypes, we can remove the lastJSON-related stuff from Utils.hs, and do some more cleanup.
htools: split parts of HTools/Types.hs into BasicTypes.hs
The 'Result' type is common and:
- might be used outside of HTools-specific code too- is better split as we need these basic types for building the more complex ones in Types.hs
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Abstract tiered specs command line parsing
This will be used in the future for the standard specs too, so let'sabstract it away.
Some small test simplifications
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Abstract some common hspace code into a function
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Abstract some common Cluster.hs code into a function
Convert opcode TH code to the use of Field type
This makes more explicit the field behaviour - previously an optionalfield was detected via a "Maybe" constructor, and an optional one viaa "Just defval" one. With this, field behaviour become more explicit...
htools: small change in error message in THH.hs
We should also display the value we can't parse, otherwise debuggingis very hard.
htools: improvements to JSON deserialisation
This fixes two problems:
- first, when we deserialise a big object, showing its value is not useful, as it will hide the actual error message- second, we shouldn't deserialise a container at once, because then...
htools: add new template haskell system
This system based on explicit types instead of ad-hoc rules(e.g. instead of deducing from "Maybe Int" an optional field, we nowcan say explicitly OptionalField ''Int). In the first phase, this willbe used for the equivalent of lib/objects.py, which has slightly...
htools: adjust imports for newer compilers
While testing with ghc 7.2, I saw that some imports we are using arevery old (from ghc 6.8 time), even though current libraries are usingdifferent names.
We fix this and bump minimum documented version to ghc 6.12, as I...
Merge branch 'devel-2.5'
Merge branch 'stable-2.5' into devel-2.5
Fix unittest bug related to offline instances
Currently, the code in Node.hs is overly strict: once a node's freememory reaches 0, it will refuse to add any instances (offline ornot). I think this is a safe safeguard (I don't expect nodes to runwithout at least 1MB of free memory), so rather than change this...
htools: reindent the rest of the files
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
htools: re-indent IAlloc.hs
htools: reindent hspace
htools: reindent hbal
htools: reindent CLI.hs
htools: re-indent QC.hs
htools: re-indent Node.hs
htools: finish re-indenting Cluster.hs
htools: rework message display construction
While diagnosing some (unrelated) memory usage in htools, I'vestumbled upon some very bad behaviour in checkData: mapAccum isnon-strict, and the tuple we use also, so that results in the list oflist of messages being very bad space-wise (hundreds of MB of memory...
hbal: handle empty node groups
This patch changes an internal assert (which can only be triggeredwhen a node group is empty) into properly handling this case (andreturning empty node/instance lists).
While we could handle this in the backend (Cluster.splitNodeGroup)...
Start converting the indentation to 2-chars
… also do some other small style cleanups.
htools: Nicer arbitrary instantiation
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: change behaviour for offline instances
htools: remove 'running' from instance options
htools: change runSt type to InstanceStatus
htools: create InstanceStatus ADT
Generalize the generation of ADT from raw types
Luxi support for Query status in htools
hail: don't select the primary as new secondary
This just adds the primary node of the instance as 'non-allocable'during the choosing of the new secondary.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(cherry picked from commit 7073b3a86856bcd8d8a62c0b72f82deaabb8d8f1)...
hail: add an extra safety check in relocate
If we select the primary as new secondary, better to fail than returnwrong data to Ganeti.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>(cherry picked from commit f25508bef4e85032f0468e5a6f0f8930ff154e66)...
Further cleanup in hspace
This moves the checking of results from the allocation functions to aseparate function, so that we have less code duplication. It also doesa bit of simplification in the printing functions.
A bit of cleanup in hspace
The node offline/mcpu is identical to hbal's setNodesStatus, so let'smove that to CLI.hs and reuse it in hspace (also, rename it and dropone 's').
Also, the check for the number of nodes is obsolete, as we computethat from the disk template....
Add a type synonym for the allocation function sig
Both iterateAlloc and tieredAlloc share the same signature, but it'snot documented nor exported (needed for refactoring).
htools: Simplify Luxi query results parsing
The logic is not entirely correct—the new Query interface exports thefield status, and we don't use that yet. But the new code should bemore readable.
Adjust htools code to new Luxi argument format
This partially undoes commit 92678b3, more specifically it removes theStore data type and the associated code, since all Luxi arguments arenow lists.
Furthermore, since the qfilter field on Query is complex (it's...
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Rename filter and filter_ to qfilter
We currently use 'filter' as the OpCode, QueryRequest and RAPI fieldname for representing a query filter. However, since 'filter' is abuilt-in function, we actually have to use filter_ throughout the codein order to not override the built-in function....
Haskell support for generic Query in Luxi
Untill now htools did not have support for generic Query in Luxi. Thispatch introduces Query as a supported Luxi operation and replacesQueryNodes, QueryInstances and QueryGroups with Query.
Signed-off-by: Agata Murawska <agatamurawska@google.com>...
TH simplification for Luxi
This patch simplifies the generation of save constructors for LuxiOpby always using showJSON over an array of JSValues, instead of havingto pass showJSON in most cases, except the 5-tuple case.
Dots in docstings and hlint error fixes for htools
Signed-off-by: Agata Murawska <agatamurawska@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove the oneline output option in hbal
This was, AFAIK, never used, and complicates the output code enoughthat it's better to remove it.
Rework/split hbal's main function
This is just moving code around. A subsequent patch will do a bit morecleanup and changing the output.
Skip application of 'id' in TH code
This is just beautification when dumping splices to stdout, as ghcwill optimise the 'id' away anyway.
Original generate code:
opToArgs QueryTags kind name = J.showJSON (id kind, id name)
Afterwards:
opToArgs QueryTags kind name = J.showJSON (kind, name)...
hail: Fix result for node evacuation
According to the iallocator documentation the “node-evacuate” call needsto return a list of jobs, not a list of lists of jobs.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Use TemplateHaskell to create LUXI operations
Some TH simplifications
Now that the basic code works, let's use some aliases for simpler codeand less ))))))))).
A few minor test improvements
This patch adds a few niceties to the test suite:
- allows matching test groups case insensitive and emit warnings when we give test group names that don't match anything- add a new operator that is similar to assertEqual in Python: it...
Use TemplateHaskell to decorate tests with names
This makes error message change from "Test 4 failed …" to "Testprop_Loader_mergeData failed", which is much more readable. It alsoremoves the duplication of test suite names in the test.hs file.
Use TemplateHaskell to generate opcode serialisation
This replaces the hand-coded opcode serialisation code withauto-generation based on TemplateHaskell.
Use TemplateHaskell to build the opID function
This replaces the hand-coded opID with one automatically generatedfrom the constructor names, similar to the way Python does it, exceptit's done at compilation time as opposed to runtime.
Again, the code line delta does not favour this patch, but this...
Use TemplateHaskell instead of hand-coded instances
This patch replaces the current hard-coded JSON instances (all alike,just manual conversion to/from string) with auto-generated code basedon Template Haskell(http://www.haskell.org/haskellwiki/Template_Haskell)....
Rename some helper functions for consistency
This changes the names for some helper functions so that futurepatches are touching less unrelated code. The change replacesshortened prefixes with the full type name.
Split part of Utils.hs into JSON.hs
Utils is a bit big, let's split the JSON stuff (not all of it) into aseparate module that doesn't have any other dependencies.
Add an allocation limit to hspace
This is very useful for testing/benchmarking.
Small simplification in tryAlloc
Change how node pairs are generated/used
Currently, the node pairs used for allocation are a simple [(primary,secondary)] list of tuples, as this is how they were used before theprevious patch. However, for that patch, we use them separately perprimary node, and we have to unpack this list right after generation....
Parallelise instance allocation/capacity computation
This patch finally enables parallelisation in instance placement.
My original try for enabling this didn't work well, but it took awhile (and liberal use of threadscope) to understand why. The attempt...
Abstract comparison of AllocElements
This is moved outside of the concatAllocs as it will be needed inanother place in the future.
Change type of Cluster.AllocSolution
Originally, this data type was used both by instance allocation (1result), and by instance relocation (many results, one perinstance). As such, the field 'asSolutions' was a list, and thevarious code paths checked whether the length of the list matches the...
hspace: add short forms for the group policy
This adds a shortened versions of the allocation policies, as writingout the whole name in the command line can become tedious.
htools: add a MonadPlus instance for Result
This will be used to implement more easily 'choice' parsing of inputdata, without resorting to syntax (case … of Bad _ -> …).
htools: remove dead code
The tryEvac/evacuateInstance functions are no longer used in the newmulti-group world order, so we remove them and change the unit-test totest the actual IAllocator function.
Revert deprecation of evacuate mode in hail
As discussed offline, the new node-change mode could be used forevacuation, but it's not directly useful as it returns a list ofopcodes; therefore, we need to partially revert commits fbe5fcf and5b53ca7 that removed it (and multi-evacuate, which remains removed)....
htools: rework the algorithm for ChangeAll mode
I think I've identified the problem with the current ChangeAllmode. The current algorithm works as follows:
- identify a new primary by choosing the node which gives best score as new secondary- failover to it...
htools: replace two hardcoded uses of pri+sec nodes
These two cases use explicit uses of primary and secondary nodes withInstance.allNodes, which means the code is more flexible if theinternal layout of the instance changes.
I've verified that the output of involvedNodes is not required to be...
htools: add target_node member to migrate opcode
… and failover too. Not many changes otherwise except forserialisation and unittests.
htools: do not change node disk for non-local storage
htools: add more functions for local disk storage
These will be used in Node.hs for proper add/remove instancecode. Furthermore, we restrict the movable status to the right disktemplates only, so that we don't attempt to move the 'wrong' instancetypes....
htools: add two more small unittests
This adds tests for the opToResult and eitherToResult functions fromTypes.hs, and changes two other tests for the same module to test JSONserialisation (which automatically also tests the lower-level to/fromstring conversion functions)....
htools: a few more hlint fixes
Tested only on GHC 7.x, will test on 6.1x too before commit.