Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
Signed-off-by: Iustin Pop <iustin@google.com>...
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Remove unsafePerformIO usage
We need to change a few things, most importantly CLI options defaults,but otherwise we already used the path to files from functions whichwere already in the I/O monad, so we don't have to change much of thecode flow.
Additionally, Path.hs now has an explicit export list, to stop...
Change type of program options to 'IO [Options]'
Some options have defaults that depend on the environment, and wecould handle these in two ways:
- use a place-holder value (e.g. data X a = Default | Custom a) that is later read from the environment- move the options list to IO monad, where it can read the...
Convert Luxi results to Ganeti errors
This a bit too complex patch converts the result of Luxi calls(submitJob, query*, etc.) from Result to ErrorResult. It thenimmediately revers this in the HTools/Backend/Luxi module, where wedon't need necessarily the full error type (just a nice error...
Move htools backends to a separate directory
Five modules under the HTools/ directories are backendimplementations, so let's move them to a separate directory, to moreclearly show the hierarchy. I wanted to do this for a while, butmerging between branches is always an issue, so let's do it know since...
Fix a few issues found by newer hlint
Testing with a newer hlint found a few minor issues; but all are real,valid recommendations:
- don't use "if cond then f x else f y", but "f (if cond then x else y)" - "if a then b else True" is equivalent to the simpler "not a || b"...
A few unittests improvements
Small simplifications of other unit tests using the (==?)operator when possible, and typo fixes.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Use exitErr instead of explicit error message and exitWith
Furthermore, a few messages have their capitalisation changed (fixed).
Signed-off-by: Helga Velroyen <helgav@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Cleanup HTools.Types/BasicTypes imports
Before we reorganised the source tree, the 'Result' type was exportedfrom HTools/Types.hs. This changed during the reorg, but at that timewe didn't change the exports; instead, we kept re-exporting it fromthe old module for compatibility....
Define the actual arguments that are supported by programs
This defines the arguments supported and then modifies the--help-completion output to include them too.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Rename Ganeti/HTools/Utils.hs to Ganeti/Utils.hs
This is, I believe, the last non-htools specific file that still livedin the htools directory; it's already widely used in non-htools code,so let's move it before we add more functionality to this module....
Remove generic options from individual programs
Currently, we test and require that each individual program (hbal,etc.) defines/supports the generic options (currently --help and--version). Even with the test, this is not optimal, since it requireschanges in many places whenever we modify the list of generic options,...
Add printing of group score in hinfo
The global cluster score is less interesting than individual groupscores, for multi-group allocation purposes.
Replace a big tuple with a custom data type
This will make it easier to add new parameters.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Use autoconf-based paths from Haskell instead of constants
Future changes will change Path.hs to use an environment variable.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Further hlint fixes
Commit 2cdaf22, “Re-enable standard hlint warnings”, got it almostright. The only problem is that (confusingly) the default set of hintsis not in HLint.Default, but in HLint.HLint (it includes Default andsome built-ins).
After changing the lint file to correctly include the defaults, we had...
Re-enable standard hlint warnings
Commit 5a1e31b4 (Add infrastructure for, and two extra hlint rules)was intended to add two extra hlint rules, but I didn't realise atthat time that "--hint" when first used overrides the built-inlints. As such, since then we were basically running with just those...
Switch job IDs to numeric
This has been a long-standing cleanup item, which we've alwaysrefrained from doing due to the high estimated effort needed.
In reality, it turned out that after some infrastructure improvements(the previous patches), the actual job queue-related changes are quite...
Introduce a type for the ganeti job type
This will be used for easier change later.
Merge branch 'stable-2.6'
hbal: return exit status 0 in case of early exit
This derives from an internal bug, but the story is consistent acrossboth internal and external usage of hbal.
Basically right now, hbal returns exit code 1 if requested to exitearly, even if all jobs are successful. This is counter-intuitive due...
hcheck: add two simple type aliases for readability
The same types are reused a couple of times, so let's add a couple oftype aliases for easier change later and readability.
hcheck: rework output mode
Looking at the output of hcheck, in human readable mode, it looks likeit always starts with a blank line. This is not nice, so I wanted toredo this to start cleanly.
However, looking at the code, I realised that we need some internal...
hcheck: reword and fix typo in instance/offline msg
Add hlint warning for wrong use of 'length'
While removing split instances check from hcheck, I saw a wrong use of'length': this should only be used when one needs the actual length ofthe list, and not when one wants to check whether the list is empty or...
Remove can_rebalance functionality from hcheck
Since hbal can now run rebalances even in the presence of splitinstances, we can remove this check and always run the rebalance.
hbal: relax restrictions on split instances
The recent patch series on hcheck made me realise how much pain wehave due to hbal not being able to work (at all) with split instances.
This is sub-optimal; ideally, hbal would degrade its behaviour, butnot completely refuse to work. This patch does this, by marking any...
Fix a few style issue in hcheck
This fixes a couple of issue I've seen while fixing the List import:
- removes over-use of printf with putStr/show- wraps some lines- removes superfluous parentheses in constructs like 'IO (a)'- makes sure that in type signatures, the '->' are placed at the...
Fix old-style import
Commit 1213f9d6 (re)added an old-style import, which fails with recentcompilers.
Style fix in hcheck
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Simplify stats printing
PrintGroupStats and printClusterStats had very similar codestructure. This patch abstracts those two into printStats, making ita little more generic. Also, printStats has its arguments documentedas requested in previous patch series....
Cleaner simulation of rebalance
This replaces the 'if .. then .. else ..' with a large block of code in'else' branch by separating this code into a separate function.
Small reordering and renaming in hcheck
Reordered functions and shortened one function name.
Move 'need rebalance' check from cluster check
As suggested by Rene, we should keep the check separate from printing.
Always print all variables in machine readable version
As discussed in cover letter of patch series introducing hcheck, inmachine readable version we should always print values of all variables.When hbal simulation was not run, FINAL values are equal to INIT values....
Print idx <-> uuid mappings in machine readable mode
As per Iustin Pop's suggestion, we are printing group mappingsfrom group idx (known only to htools) to group uuid, which identifiesthe group uniquely. This is required because group uuid and namemay contain characters such as '-' which are not valid parts of...
Human readable group name improved
Instead of printing group id in human readable version of hcheck,we now print group name.
Print info about ability to rebalance
As suggested during offline discussion, this patch adds a new bashvariable and description field in hcheck, which informs if therebalance was possible. This is to determine if there are splitinstances on the cluster, in which case rebalance is impossible....
Improve readability for hcheck-related changes
As suggested by Rene, I added comments to types for printKeysand printFinal. Also, realigned some lines in Hspace.
Machine readable hcheck functionality
Machine readable output for hcheck.
Human readable hcheck functionality
Full hcheck functionality is implemented in this patch. Hcheck runsper-group checks, provides summary for the entire cluster and simulatesrebalance if required, the printing the modified metrics.In this patch, only the human-readable output is available....
Allowing rebalance to run silently
Part of hcheck's simulation required us to silently run hbal. For thatpurpose we expose iterateDepth and allow it to run silently.
Generalize functions used by both hspace and hcheck
As hspace and hcheck both use machine readable options, they requiresimilar functions - which are now generalized and moved to CLI andUtils.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
hspace: Make use of the spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
htools: Refactor some of IAlloc.readRequest to Hail
This makes it possible to reuse that function when we want to use IAllocas a data source backend
hinfo: Adjust the verbosity
hinfo: Gather and print group statistics
hinfo: Reorganize the code to separate functions
Also rename some functions as they don't reflect their actions anymore
hinfo: Adding basic skeleton based on hbal
It prints the information which hbal prints in a nice formatted manner and ofcourse without doing any actions. An example output looks like this:
$ hinfo -m xen.example.comLoaded 4 nodes, 2 instancesCluster has 1 node group(s)...
Temporary fix for hspace's NPU calculation
hspace computes the NPU (normalised CPU units) via the(used|pool|avail) VCPU divided by the (currently cluster-wide) maxvcpu_ratio.
However, in the future we'll have variable vcpu_ratios, so we can'tthe calculation as such. We temporarily change to computing the...
Switch over some hspace stats from RSpec to AllocInfo
This patch changes the allocation delta stats from RSpec toAllocInfo. There's nothing interesting about it, but it should allowmore correct computations of Npus in hspace.
I also moved the AllocStats type alias from Cluster.hs to Types.hs...
Remove two extraneous imports
Parse options in htools.hs, not in each program
This patch moves the call to getArgs/parseOpts into htools.hs, asopposed to the current situation where htools.hs only calls the mainfile of each program, which in turn calls these two functions.
By doing it only once we remove some duplicate code and we also export...
Show explicit message when hspace fails to alloc
Currently, in the human-readable mode, hspace doesn't show anymessages when no instances can be allocated in tiered-alloc mode. Thisis confusing, so let's add an explicit message.
Switch hspace to read the disk template from the cluster
The default disk template is the first enabled template.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Switch hspace defaults to the cluster policy
This changes from the current hardcoded defaults to the clusterpolicy. The command line options now override the defaults from thecluster, and the tiered spec mode is always enabled.
Also fixes a tiny typo in the man page (together with the man page...
Abstract creation of instance from a spec
Load cluster ipolicy via Luxi
Also show it in hbal's verbose output (helpful for debugging).
Extend ClusterData with the cluster instance policy
This attribute is always initialised to the default, and is not (yet)read/saved in the various backends.
Rename optISpec to optStdSpec
More consistency with the tiered allocation mode.
Change how hspace reads cmdline args for std specs
For the standard specs, hspace currently takes separate options forthe memory, disk and VCPUs. For the tiered specs, which wereintroduced later, it takes a single option with all three valuescombined....
Cleanup hlint errors
First, we update the recommended hlint version to what I used to get aclean output (1.8.15). Most of the changes are:
- remove unneeded parentheses- some simplifications (intercalate " " → unwords, maybe … id → fromMaybe, etc.)- removal of some duplicate code (in previous patches)...
Abstract some common hspace code into a function
htools: adjust imports for newer compilers
While testing with ghc 7.2, I saw that some imports we are using arevery old (from ghc 6.8 time), even though current libraries are usingdifferent names.
We fix this and bump minimum documented version to ghc 6.12, as I...
Merge branch 'devel-2.5'
htools: reindent the rest of the files
htools: reindent hspace
htools: reindent hbal
hbal: handle empty node groups
This patch changes an internal assert (which can only be triggeredwhen a node group is empty) into properly handling this case (andreturning empty node/instance lists).
While we could handle this in the backend (Cluster.splitNodeGroup)...
htools: change runSt type to InstanceStatus
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Generalize the generation of ADT from raw types
Further cleanup in hspace
This moves the checking of results from the allocation functions to aseparate function, so that we have less code duplication. It also doesa bit of simplification in the printing functions.
A bit of cleanup in hspace
The node offline/mcpu is identical to hbal's setNodesStatus, so let'smove that to CLI.hs and reuse it in hspace (also, rename it and dropone 's').
Also, the check for the number of nodes is obsolete, as we computethat from the disk template....
Remove the oneline output option in hbal
This was, AFAIK, never used, and complicates the output code enoughthat it's better to remove it.
Rework/split hbal's main function
This is just moving code around. A subsequent patch will do a bit morecleanup and changing the output.
Rename some helper functions for consistency
This changes the names for some helper functions so that futurepatches are touching less unrelated code. The change replacesshortened prefixes with the full type name.
Add an allocation limit to hspace
This is very useful for testing/benchmarking.
htools: a few more hlint fixes
Tested only on GHC 7.x, will test on 6.1x too before commit.
htools: further docstring fixes
This adds parameter documentation for Cluster.iMoveToJob (I think itwas not clear if the new or old node list is needed) and fixes otherdocstring style issues.
After this patch, all modules except for CLI.hs (which has many...
htools: add human-readable output to hspace
Currently, hspace can only output a machine-readable format that(while detailed) is hard to parse quickly by people. This patch adds(and enables by default) a human-readable output that shows the mostimportant metrics in a simple format....
Do proper name lookup for the -O option
hspace and hbal treat -O differently, and use aliases for short names(although hbal succeeds in that, and hspace doesn't). Uniform this witha name lookup, using the same functions we used for instanceselection/exclusion....
htools: move tiered spec map helper to Hspace.hs
This is used just in hspace, so let's help in making Cluster.hssmaller. We also split the function in two, as computing the spec mapand formatting it are two different tasks.
htools: switch hspace to the generic binary
This is the last patch of the binaries conversion.
As information, we now have a single binary that is approx. 5.4MiB insize, compared to 4 binaries that were approx. 5.1-5.2MiB in size;this will result in a smaller package and install size, and the single...
htools: switch hscan to the generic binary
htools: switch hbal to the generic binary
In addition, the patch adds a separate Makefile variable for holdingthe binary roles to make it more clear what we symlink.
htools: switch hail to the generic binary
This converts the first binary to the generic 'htools' binary.