Add a new type ClusterData
This will be used to hold all the disparate uses of the cluster data:we have either tuples with these four elements, or functions takingthese four arguments, etc.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Ajust hspace manpage for the new simulation syntax
Simulation backend: read the allocation policy too
This patch moves the allocation policy from hardcoded to be read fromthe given specification, and extends the error message for invalidspecifications.
Simulation backend: allow multiple node groups
This patch changes the behaviour of the --simulation option to be anincremental option, where each new use defines a new node group. Thisallows simulation of more complex clusters.
Signed-off-by: Iustin Pop <iustin@google.com>...
Merge branch 'stable-0.2'
Conflicts (all removed): man/hail.1 man/hbal.1 man/hscan.1 man/hspace.1
Move man files to man/ subdirectory
This is just change on the 0.2 branch to synchronize with the masterbranch. It allows automated builds to work better across the twoversions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Update NEWS file for 0.2.8 release
hbal: return meaningful exit code for job failures
Currently, LUXI job failures only display a warning message, whilestill returning a success exit code. We change hbal to returntrue/false from within execJobSet/runJobSet, and add a wrapper forsimpler code....
Change the balancing function
Currently the balancing function is a modified version of the standarddeviation (stddev divided by list length), due to historical reasons.
While this works fine for small clusters, for big clusters it makesthe balancing effect too "weak", and in some cases it refuses to...
Move some tiered spec functionality to Cluster.hs
This splits out a bit of code from hspace.hs and moves it into its ownfunction in Cluster.hs.
Change the Node.group attribute
Currently, the Node.group attribute is the UUID of the group, as untilrecently Ganeti didn't export the node group properties. Since it doesso now, we make the following changes (again apologies for a bigpatch):
- we change the group attribute to be an index, similar to the way an...
Text.hs: also save the group data when serialising
This should have been in the previous patches, but sent separate forclarity.
The live-test script is updated to read the first node from thecluster, now that the text files don't start anymore with the node...
hbal: display the group name in the multi-group case
hail: display group names in info messages
This patch switches from the group index to the group name for theinformational messages in the hail results.
Generalise the sepSplit function
Currently it works on splitting strings by individual chars, but wecan generalise it to split lists by list elements, which means we canreuse it later in the Text module for splitting both lists of chars by'|' or lists of lines by empty newlines. The change also makes the...
Text.hs: change to use sepSplit
The new sepSplit function can split based on empty lines, so we removethe hackish text splitting from before and simply use sepSplit. Thisis needed as the addition of extra sections would have increased thecode linearly, which we don't want :)...
Text.hs: also read cluster tags from the data file
This means that a file with the correct information is as accurate asthe other backends (Luxi, Rapi). Serialization of tags is in the nextpatch.
Text.hs: serialize cluster tags when writing data
This is the complement to the reading part. Now the live-test workscorrectly against clusters with configured exclusion tags.
live-test: support multi-group clusters
Since currently hbal can only work on single groups at a time, we needto be able to specify the target group when running the live test.
Implement a JSON instance for AllocPolicy
This will allow reading this attribute via the Rapi/Luxi backends.
Rapi: read the allocation policy from the cluster
Luxi: read the allocation policy from the cluster
Text: read/write the allocation policy
hail: allow overriding cluster data from requests
Currently, it's not easy to generate “fake” IAllocator request filesfor hail. As such, testing on simulated clusters is hard to do.
To workaround this, we change hail to also take the ‘-t’ and‘--simulate’ options, so that we can override the cluster data read...
IAllocator: respect the alloc_policy for groups
This patch changes the allocate mode to respect the alloc_policy forgroups. It does this by changing the sort key from simply the solutionscore, to a tuple with two elements: the alloc policy (which is now an...
Rework the data loader pipelines to read groups
This (invasive) patch changes all the loader pipelines to read the nodegroups data from the cluster, via the various backends. It is invasiveas it needs coordinated changes across all the loaders.
Note that the new group data is not used, just returned....
Add lookupGroup utility function
This will be used in the various backends similar to the lookupNodefunction.
Add a new Group.hs module describing node groups
This is not yet used by the rest of the code.
Add the new OpQueryGroups opcode definition
Update hscolour usage
This patch fixes two issues related to coloured sources generation.First, recent hscolour has changed the css file (and we need to updateit), but it also can output it at runtime, so there's no need to storeit anymore in the source tree....
Improve error reporting for small clusters
When doing a two-node allocation on a cluster/group in which only onenode is online, or a one-node allocation without any online nodes, wedidn't show a valid error mesage. The patch changes tryAlloc to "failhard" in this case, to make the failure explicit....
hail/allocate: implement multi-group support
This is a bit hackish. We add a new function that takes the input data,splits it into groups, runs the original tryAlloc for each group, andthen chooses the best solution, but adds the log messages from all the...
hail: remove the custom info message generation
Since the solutions are "self-annotated", we can remove the custom codefrom hail, and just keep a very small processResults function.
After this change, allocation/failure shows the new detailedinformation....
Add a 'log' attribute to allocation solutions
And also a couple of functions for describing a given solution; thesewill be used in the future instead of the ones currently in hail.
The patch also enhances the description of failure messages.
Change AllocSolution from tuple to its own type
Tuples are good for two, three, at most four elements. Beyond that, thecontinuous pattern matching and construction/deconstruction becomestedious.
Since in the future we'll probably keep more information in the...
Cleanup AllocSolution after AllocElement changes
Since we added the score to AllocElement, we don't need to wrapAllocElement in yet another tuple, just to attach the cluster score. Sowe simplify the AllocSolution type.
AllocElement: extend with the cluster score
AllocElement, a type used as a result of allocations, holds the statusof the nodes after the allocation. In most cases, we'll compare thisallocation result with others, to see which allocation decision makesthe most sense. This comparison is done via the cluster score....
Add two utility functions for the Result type
Actually, this just moves the functions from the QC module to Types, andremoves a duplicate entry from Cluster.
Rework the types used during data loading
This improves on the previous change. Currently, the node and instancelists shipped around during data loading are (again) association lists.For instances it's not a big issue, but the node list is rewrittencontinuously while we assign instances to nodes, and that is very slow....
Loader functions: move from assoc lists to maps
When loading big clusters, the association lists become a bit slow, sowe'll replace this with a simple Map String Int; the change is trivialand can be reverted easily, while it brings up a good speedup in the...
Convert some leftovers to NameAssoc
The type alias NameAssoc has been introduced a long time ago, but thereare some few not-yet-converted cases. In preparation for changes to thattype, let's make sure we use it consistently.
hbal: implement handling of multi-group clusters
On a single-group cluster, we proceed as before. On multi-groupclusters, we require selection of the desired group (currently via UUIDonly).
Add Cluster.splitCluster for node groups
This splits a top-level cluster information into the component nodegroups. Instance go to the group of their primary node, but otherwise wedon't disallow split instances.
Add the man html files to gitignore
Rework Container.hs and improve test coverage
Since some of the functions we export from Container.hs are 1:1identical to IntMap, we can just export the originals and remove thewrappers. This reduces the code we need to unittest.
Furthermore, we add two simple unittest for the two non-trivial...
Add new command-line option for group selection
Add two functions for checking cluster consistency
For now, we don't support instances allocated across two groups, and wewill reject such clusters. The isClusterConsistent function will returna list of inconsistent instances, potentially allowing operation without...
Add function for nodes to (nodgroup, nodes) split
Unittests included. The function will be needed for consistency checksin the algorithms.
Add a type alias for UUIDs
This is to pottentially allow easier changes later.
Also build HTML versions of man pages
RAPI: read the group UUID from the server
This depends on future support from Ganeti (2.4+).
IAlloc: read group uuid from the input message
This makes the code incompatible with JSON files from Ganeti pre-2.4.
Text: read/save the node group UUID
Compatibility with old text files is kept by using the default UUID ifthe file (or even some records) don't have a UUID.
Luxi: read the node uuid from the cluster
This makes the code incompatible with Ganeti pre-2.4.
Node: add the node group's UUID
This is not used anywhere yet, and the backend are all just adding thedefault UUID, not the real one.
The patch also allows displaying the group UUID in the node list.
Utils: add a default UUID
This will be used as a placeholder for the cases when we need a UUID(any UUID), but we don't have one handy.
Merge branch 'devel-0.2' into master
Improve the standard deviation computation
This does just two passes, instead of three, over the list. This reducesthe overall runtime well enough (~25%) in some tests, but it's notreproducible using profiling, so I don't know how much the functionitself is being sped-up....
hbal: change handling of signal
Currently, hbal does a one-two signal handling, where the first signalcauses graceful termination, and the second one an immediate on (eitherSIGINT or SIGTERM can be used, interchangeably). However, this poses atiming problem: if two programs want to send a graceful termination...
Simu loader: move the loading to non-IO code
While we don't actually have IO code in the Simu loader, we do have thesame interface. So we move the code again to a separate parseDatafunction which is exported.
Luxi loader: split parsing from loading
Rapi loader: split parsing from loading
The change is similar to the text loader change.
Text loader: split parsing from loadData
This change, which will be followed by similar changes in the otherloaders, splits the parsing of the data from the actual loading fromdisk. Since the parsing doesn't usually involve IO actions, we will beable to better test the parsing. The loading becomes a smaller part of...
Ignore nodes which are not vm_capable
This break compatibility with Ganeti pre-2.3.
Merge branch 'devel-0.2'
Fix tag exclusion weight
Currently, the tag exclusion metric has a weight of one, which meansthere might be cases where we won't move instances around because itupsets the cluster metrics. However, we do want to make a higher effortfor cleaning up tag collisions, so we increase the weight to an...
Force UTF-8 locale for pandoc invocation
Pandoc 1.5.x uses the locale information to parse its input files (only1.5, pre and post version use always UTF-8). Hence we need to enforce aUTF-8 locale for proper parsing of input files.
Move from hand-written man pages to RST/pandoc
This simplifies the maintenance of the man pages, and unifies the rst-to-*converter to pandoc.
Add design for htools/Ganeti 2.3 sync
This is a work in progress, will be modified along with the progressof Ganeti 2.3.
Update NEWS file for 0.2.7 release
Fix some warnings in unittests
Add a hack for normalized CPU values in hspace
Currently, the key metrics/tiered spec computations show the virtual cpucount. However, since we do have a maximum ration Vcpu/Pcpu, we can alsoshow the “normalized” cpu count, i.e. the equivalent physical cpu count...
Improve the error message for tiered alloc option
hbal: implement user-friendly termination requests
Currently, hbal will abort immediately when requested (^C, or SIGINT,etc.). This is not nice, since then the already started jobs need to betracked manually.
This patch adds a signal handler for SIGINT and SIGTERM, which will, the...
Document the gain options in hbal's manpage
Use the mingain options in the balancing algorithm
Also adds them in hbal.
Add new CLI options for min gain during balancing
Recent hbal seems to run many steps for small improvements (< 1e-3), sowe should stop early in this case.
We add a new option (-g), that will be used for the minimum gain duringbalancing. This check will only become active when the cluster score is...
Makefile: make the rst2html converter more strict
This will make the automated builds flag any problems.
Update NEWS file
Fix ReplaceSecondary moves for offline nodes
The addition of a new secondary on a node is doing two memory tests:- in strict mode, reject if we get into N+1 failure- reject if the new instance memory is greater than the free memory (not available memory) on the node...
Add some more debugging functions
These are just variations of the standard debug, but are provided forsimpler code, since lazyness is something causing non-computation ofdebug statements.
Implement cluster state saving in hbal
Also move the LUXI execution (-X) to the end, after all the outputmessages are printed. No good in waiting for the messages for a longwhile, especially as they are not up-to-date stats after the jobexecution, just an estimation of what the state will be.
Change iterateAlloc to return the instance list
The Cluster.iterateAlloc and tieredAlloc functions are changed to alsoreturn the updated instance list, since it is needed to have a “full”cluster view.
Implement cluster state saving in hspace
This also uncovered a few issues with the allocation model (instancesnot being marked up, etc.).
Compared to hbal, hspace will generate either one or two files (for boththe standard and the tiered allocation mode), depending on the input...
hspace: mark new instances as running
Otherwise the saved cluster state and the in-memory one are wrong.
Update man pages for the new -S option
Abstract the cluster serialization from hscan.hs
This is currently hardcoded in an internal function in hscan.hs, and wemove it to Text.hs for later use.
Add a new option --save-cluster
This option will in the future be used to serialize the cluster state inhbal and hspace after the rebalance/allocation steps.
Add unittest for Node text serialization
This checks that the Node text serialization and deserializationoperations are idempotent when combined other.
Switch unittest to custom hostnames
Currently, the hostnames are almost fully arbitrary chars, which breaksthe assumption that nodes/instances will be normal DNS hostnames.
This patch adds some custom generators for these hostnames, that willallow better testing of text loader serialization/deserialization.
Move text serialization functions to Text.hs
Currently these are in hscan, and cannot be reused easily.
Fix a couple of typos in the manpages
Again, thanks to lintian.
hail: fix error message for failed multi-evac
Currently we show the instance index, but this makes no sense outsidethe current running program. Instead, we show the instance name.
NEWS: Add double blank lines before headers
This looks better for text-only viewing…
Update NEWS file for the 0.2.6 release
hscan: return exit code 2 for RAPI failures
If some clusters failed during RAPI collection, exit with exit code 2 sothat tests can detect this failure.
More enhancements to live-test.sh
Fix another haddock issue
Remove an obsolete function and add Utils tests