Introduce a type for the ganeti job type
This will be used for easier change later.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Merge branch 'stable-2.6'
Reorganise the lookup functions
Currently, the LookupResult, MatchPriority and related functions arelocate in Loader.hs, since (so far) only hbal needs them in theselection of instances. However, with the new functionality on confdside, we need these functions there too, but we don't want to import...
Replace a few explicit case expressions
Since we're just talking about converting Maybe into another monad, wecan do that via the maybe function, instead of explicit casing.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Fix Haskell coverage results
There are two current issues with the coverage values:
- we don't import all modules, thus leading to incomplete coverage results (too optimistic);- we use hpc in its default mode (intersection), which means that even modules which do have coverage results but are not used in all...
Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
hbal: return exit status 0 in case of early exit
This derives from an internal bug, but the story is consistent acrossboth internal and external usage of hbal.
Basically right now, hbal returns exit code 1 if requested to exitearly, even if all jobs are successful. This is counter-intuitive due...
hcheck: add two simple type aliases for readability
The same types are reused a couple of times, so let's add a couple oftype aliases for easier change later and readability.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
hcheck: rework output mode
Looking at the output of hcheck, in human readable mode, it looks likeit always starts with a blank line. This is not nice, so I wanted toredo this to start cleanly.
However, looking at the code, I realised that we need some internal...
hcheck: reword and fix typo in instance/offline msg
Add hlint warning for wrong use of 'length'
While removing split instances check from hcheck, I saw a wrong use of'length': this should only be used when one needs the actual length ofthe list, and not when one wants to check whether the list is empty or...
Remove can_rebalance functionality from hcheck
Since hbal can now run rebalances even in the presence of splitinstances, we can remove this check and always run the rebalance.
Correct a wrong --help option for --simulate
This was not update when the alloc policy was added to the parameter.
hbal: relax restrictions on split instances
The recent patch series on hcheck made me realise how much pain wehave due to hbal not being able to work (at all) with split instances.
This is sub-optimal; ideally, hbal would degrade its behaviour, butnot completely refuse to work. This patch does this, by marking any...
Fix a few style issue in hcheck
This fixes a couple of issue I've seen while fixing the List import:
- removes over-use of printf with putStr/show- wraps some lines- removes superfluous parentheses in constructs like 'IO (a)'- makes sure that in type signatures, the '->' are placed at the...
Fix old-style import
Commit 1213f9d6 (re)added an old-style import, which fails with recentcompilers.
Style fix in hcheck
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Simplify stats printing
PrintGroupStats and printClusterStats had very similar codestructure. This patch abstracts those two into printStats, making ita little more generic. Also, printStats has its arguments documentedas requested in previous patch series....
Cleaner simulation of rebalance
This replaces the 'if .. then .. else ..' with a large block of code in'else' branch by separating this code into a separate function.
Small reordering and renaming in hcheck
Reordered functions and shortened one function name.
Move 'need rebalance' check from cluster check
As suggested by Rene, we should keep the check separate from printing.
Always print all variables in machine readable version
As discussed in cover letter of patch series introducing hcheck, inmachine readable version we should always print values of all variables.When hbal simulation was not run, FINAL values are equal to INIT values....
Print idx <-> uuid mappings in machine readable mode
As per Iustin Pop's suggestion, we are printing group mappingsfrom group idx (known only to htools) to group uuid, which identifiesthe group uniquely. This is required because group uuid and namemay contain characters such as '-' which are not valid parts of...
Human readable group name improved
Instead of printing group id in human readable version of hcheck,we now print group name.
Print info about ability to rebalance
As suggested during offline discussion, this patch adds a new bashvariable and description field in hcheck, which informs if therebalance was possible. This is to determine if there are splitinstances on the cluster, in which case rebalance is impossible....
Improve readability for hcheck-related changes
As suggested by Rene, I added comments to types for printKeysand printFinal. Also, realigned some lines in Hspace.
Machine readable hcheck functionality
Machine readable output for hcheck.
Human readable hcheck functionality
Full hcheck functionality is implemented in this patch. Hcheck runsper-group checks, provides summary for the entire cluster and simulatesrebalance if required, the printing the modified metrics.In this patch, only the human-readable output is available....
Allowing rebalance to run silently
Part of hcheck's simulation required us to silently run hbal. For thatpurpose we expose iterateDepth and allow it to run silently.
Generalize functions used by both hspace and hcheck
As hspace and hcheck both use machine readable options, they requiresimilar functions - which are now generalized and moved to CLI andUtils.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Fix corner case in quick-check generated tests
Thanks to Agata Murawska, a random seed that generates bad test caseswas found ("seed was 1768143307 2118231514, test size 74"). This isdue to the fact that some nodes were generated with tMem=0, fMem=0,which means that pMem was NaN and thus the node was not comparing...
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Add a new JSON function
And its associated unittests.
Switch to new-style exception handling
Currently, we're using Prelude.catch to handle I/O errors inhtools. This style of error handling has been deprecated for a while,but it still used to work without warnings.
However, the GHC release 7.4 has started to emit deprecation warnings...
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unixconvention. This will help with testing.
hspace: Make use of the spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Fix some compiler warnings
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actualcurl part), also in the case where we didn't compile against curl.
htools: Reflect the rename of spindle_usage to spindle_use
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to workon shared storage instances. Diskless instances are handled the same(as in, None is the same everywhere :).
Signed-off-by: Iustin Pop <iustin@google.com>...
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just useeither with const functions (not really readable, but…).
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improvethe docstrings.
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) thatit will be used for other replace modes too. So all we need to do isto abstract it to a separate function (that can be called from...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work(with the known caveats related to disk space calculation) for sharedstorage disk templates.
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
htools: Convert all the other backends as this is trivial
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only viasecondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we checkwhether we're internally mirrored, externally mirrored, or not...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Rework evacDrbdSecondaryInner to take the imove
Instead of hard-coding the instance move operation, let's accept is asa parameter from the caller. This is the single thing that makes thisDRBD-specific, so removing it will help for futurepatches. Furthermore, this abstract the imove op type into a single...
Add a simple type alias for nicer type signatures
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes orhard-coding it, compute it from the disk template of theinstance. Otherwise tests will fail when the disk template is notDRBD8 and we ask for allocation on 2 nodes....
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance asprimary and secondary to a given node. This won't work for instancetemplates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates...
Use the mirror type in Cluster.possibleMoves
Currently the function possibleMoves always presumes DtDrbd, which iswrong for shared storage. To make it work with all disk templates, weadditionally pass the mirror type to it (we assume that the movebehaviour only depends on the mirror type, and not the individual...
iPolicy: enable by default all storage templates
… otherwise we can't run tests or simulations using non-defaulttemplates. Tests still pass.
Fix printSolutionLine to handle big scores
Currently we don't set a boundary on the length of the score field,which breaks all our nicely-aligned output efforts.
Fix printSolutionLine to handle non-DRBD steps
Currently Cluster.printSolutionLine function always wants to print theold and new secondary, and as such it fails (ugly) for moves ofnon-DRBD instances. Change it so that it looks explicitly fornoSecondary nodes, and format the output different in that case....
Add a helper type and function for the mirror type
Currently we track the mirroring type as simply yes/no, with "yes" meaning DRBD. This is not enough for handling shared storage, so weneed to add a type that gives the exact mirroring type (none,internal, external), and a function that ties the disk template type...
Small simplification in tryBalance
Instead of computing the bad/good node lists separately (in effectiterating twice over the all_nodes list and also creating that listtwice from the container), let's use 'partition' and do a smallimprovement.
Mark Rbd as a movable disk template
Do not rely on no-secondary for movable tests
Currently, the code in Loader.hs marks as un-movable any instanceswhich don't have a secondary. This is not valid for any shared storagetemplates, so let's remove that check completely. The initial state of...
Handle nodes without local storage
Our current cluster model is based on per-node storage, and as such wedon't handle shared disk templates. Until we can improve the model,let's handle nodes without local storage better: in order to stillcompute a valid cluster score, we need to handle nodes with tDsk == 0....
htools/ExtLoader: Bring the pieces together
This make it possible to load from an Ialloc file
htools/CLI: Adding new options to use ialloc as source
htools/IAlloc: Adding loadData method for use as backend
htools: Refactor some of IAlloc.readRequest to Hail
This makes it possible to reuse that function when we want to use IAllocas a data source backend
hinfo: Adjust the verbosity
hinfo: Gather and print group statistics
htools: Adding printTable helper
This helper abstract away all the unlines and unwords call you've to doafter you got a formatTable back.
Fixup a Node unittest
This fixes the prop_Node_rMem test, which cares about memory only;otherwise we could fail due to other problems (e.g. not enough disk).
Allow overriding the target test count
This is not perfect, as the override applies identical target testvalues to both the 'fast' and 'slow' tests (making the slow tests takea long time), but it allows a quick override for manual runs.
Add two more unittests for Node add/remove
These just test that add/remove are idempotent. This is not perfect,as we use unsorted lists for some values (instead of sets), so whenusing non-empty nodes this would break (but for empty nodes, a1-element list is sorted, so it's fine)....
Use the spindles metric in cluster scores
This makes balancing use the new metric.
Track instance-used spindles in node set/add/remove
This simply tracks the instance-used spindles (using hard-coded '1'per instance), and additionally prevents additions in soft-mode whenwe go over the limit.
Note: there's an assymetry between addSec and removeSec (basically...
Shorten some function names
Since we use the functions always module-qualified, let's drop the`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Simplify logic in Node operations
This patch adds two helper functions and uses them for a somewhatsimplified logic in the add/remove pri/sec node functions.
htools: support spindles in simu backend
htools: support spindles in text load/save
If missing, we'll default to 1 spindle.
This also updates the text file format with the ipolicy, which wasforgotten.
htools: add support for listing spindles for nodes
htools: add spindle-related attribute to nodes
This adds the spindleCount, hiSpindles and instSpindles attributes.
The spindleCount is equivalent to spindle_count on the ganeti side (anode parameter). hiSpindles is the maximum instance-used spindles, and...
Add spindle ratio to htool's IPolicy
Just the usual data/type declarations, read/save in the text backend,etc. Also does a bit of unit-test cleanup (the way we buildipolicies).
hinfo: Reorganize the code to separate functions
Also rename some functions as they don't reflect their actions anymore
hinfo: Adding basic skeleton based on hbal
It prints the information which hbal prints in a nice formatted manner and ofcourse without doing any actions. An example output looks like this:
$ hinfo -m xen.example.comLoaded 4 nodes, 2 instancesCluster has 1 node group(s)...
htools: allow rbd disk template
Notes: - it'd be nice if there was a way to automatically generate the DiskTemplate list instead of manually specifying it. After all we have C.diskTemplates - Of course this actually does nothing, as for other...
Use the ipolicy constants for key names
Sorry, I missed this when I wrote the code originally.