Change hbal handling of !auto_balance instances
Note: I'll add tests for this on the master branch, but not here.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix PID file writing in Haskell daemons
Currently, the code uses createFile, which has the effect of alwaystruncating the file. This is bad, as the content of the PID file iswiped even when we wouldn't be able to lock it!
We switch to openFd (createFile is just a wrapper over that), and we...
Fix compatibility with newer Haskell libraries
This small patch fixes compatibility with a few newer Haskell libraries:
- base 4.6, included with ghc 7.6, removed the deprecated 'catch' function from Prelude, so our "import Prelude hiding (catch)" is now...
Fix bug in non-mirrored instance allocation
The function `allocateOnSingle' has a bug in the calculation of thecluster score used for deciding which of the many target nodes to usein placing the instance: it uses the original node list for the scorecalculation....
Fix 'explicitely' common typo
It seems that 'explicitely' is wrong, and that the right form is'explicitly'. This is just fixing the typo plus adjusting affectedparagraphs.
Implement a node to drbd minors query function
This can be queried remotely since it's a pure configuration query.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Add support for computing instance all/secondary nodes
This fixes an old FIXME. Since we now how actual DRBD configurationdata, we can finally compute the instance's secondary nodes.
Add disk logical ID support in Objects.hs
This is a complex field, so we have to do a lot of manual work for now.
The complexity arises from the fact that the contents of the field,and the way to parse it, depends on the disk type field, so we don'thave a single, static way of parsing it. Hence we needed the...
Extend the Template Haskell loadFn model
Currently, we only allow field-by-field de-serialisation. Since wehave cases where information about how to un-serialise a field issplit across two JSON fields (e.g. disk type and disk logical_id,hypervisor and hvparams, etc.), we need to pass the entire object to...
Change how customFields are built
Instead of passing an expression (which cannot come from the currentmodule), we pass a name, which is allowed to reference functions fromthe module we're in. Since we currently don't have custom fields, wedon't need to modify any callers....
Implement lookup of partial names in Config.hs
This uses the recently-moved functions to implement partial lookup ofnames on getNode and getInstance, similar to the Python codebase.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Reorganise the lookup functions
Currently, the LookupResult, MatchPriority and related functions arelocate in Loader.hs, since (so far) only hbal needs them in theselection of instances. However, with the new functionality on confdside, we need these functions there too, but we don't want to import...
Remove an unused function
This is not used, as we need a more complex serialisation, which isdone in the saveObjectField function.
hcheck: add two simple type aliases for readability
The same types are reused a couple of times, so let's add a couple oftype aliases for easier change later and readability.
hcheck: rework output mode
Looking at the output of hcheck, in human readable mode, it looks likeit always starts with a blank line. This is not nice, so I wanted toredo this to start cleanly.
However, looking at the code, I realised that we need some internal...
hcheck: reword and fix typo in instance/offline msg
Add hlint warning for wrong use of 'length'
While removing split instances check from hcheck, I saw a wrong use of'length': this should only be used when one needs the actual length ofthe list, and not when one wants to check whether the list is empty or...
Remove can_rebalance functionality from hcheck
Since hbal can now run rebalances even in the presence of splitinstances, we can remove this check and always run the rebalance.
Correct a wrong --help option for --simulate
This was not update when the alloc policy was added to the parameter.
hbal: relax restrictions on split instances
The recent patch series on hcheck made me realise how much pain wehave due to hbal not being able to work (at all) with split instances.
This is sub-optimal; ideally, hbal would degrade its behaviour, butnot completely refuse to work. This patch does this, by marking any...
Fix a few style issue in hcheck
This fixes a couple of issue I've seen while fixing the List import:
- removes over-use of printf with putStr/show- wraps some lines- removes superfluous parentheses in constructs like 'IO (a)'- makes sure that in type signatures, the '->' are placed at the...
Fix old-style import
Commit 1213f9d6 (re)added an old-style import, which fails with recentcompilers.
Style fix in hcheck
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Simplify stats printing
PrintGroupStats and printClusterStats had very similar codestructure. This patch abstracts those two into printStats, making ita little more generic. Also, printStats has its arguments documentedas requested in previous patch series....
Cleaner simulation of rebalance
This replaces the 'if .. then .. else ..' with a large block of code in'else' branch by separating this code into a separate function.
Small reordering and renaming in hcheck
Reordered functions and shortened one function name.
Move 'need rebalance' check from cluster check
As suggested by Rene, we should keep the check separate from printing.
Always print all variables in machine readable version
As discussed in cover letter of patch series introducing hcheck, inmachine readable version we should always print values of all variables.When hbal simulation was not run, FINAL values are equal to INIT values....
Print idx <-> uuid mappings in machine readable mode
As per Iustin Pop's suggestion, we are printing group mappingsfrom group idx (known only to htools) to group uuid, which identifiesthe group uniquely. This is required because group uuid and namemay contain characters such as '-' which are not valid parts of...
Human readable group name improved
Instead of printing group id in human readable version of hcheck,we now print group name.
Print info about ability to rebalance
As suggested during offline discussion, this patch adds a new bashvariable and description field in hcheck, which informs if therebalance was possible. This is to determine if there are splitinstances on the cluster, in which case rebalance is impossible....
Improve readability for hcheck-related changes
As suggested by Rene, I added comments to types for printKeysand printFinal. Also, realigned some lines in Hspace.
Machine readable hcheck functionality
Machine readable output for hcheck.
Human readable hcheck functionality
Full hcheck functionality is implemented in this patch. Hcheck runsper-group checks, provides summary for the entire cluster and simulatesrebalance if required, the printing the modified metrics.In this patch, only the human-readable output is available....
Allowing rebalance to run silently
Part of hcheck's simulation required us to silently run hbal. For thatpurpose we expose iterateDepth and allow it to run silently.
Generalize functions used by both hspace and hcheck
As hspace and hcheck both use machine readable options, they requiresimilar functions - which are now generalized and moved to CLI andUtils.
Initial commit for introducting hcheck tool
Introduce infrastructure required to add Hcheck and build itsuccessfuly.
Fix corner case in quick-check generated tests
Thanks to Agata Murawska, a random seed that generates bad test caseswas found ("seed was 1768143307 2118231514, test size 74"). This isdue to the fact that some nodes were generated with tMem=0, fMem=0,which means that pMem was NaN and thus the node was not comparing...
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add support for SIGHUP handling in Haskell daemons
This patch adds support for handling SIGHUP and reopening thestdout/stderr logfile in daemon mode (but not in foreground mode).
Rework logging setup for Haskell daemons
This fixes a missing functionality: closing of stdout/stderr whenforking daemons. Without this, starting the daemons from the commandline seems to work, but doing it from utils.RunCmd breaks, sincewithout closing the standard descriptors, RunCmd never finishes....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Auto-define a LuxiReq data type
We currently auto-generate a LuxiOp data type, which holds the entireoperation (including parameters). However, having a data type just forthe method call would be useful, so let's change THH to alsodefineSADT for the Luxi constructors....
Replace hardcoded constants with Constants.hs names
Now that luxi.py constants are exported, we can use them for moreconsistency.
Add a new JSON function
And its associated unittests.
Further fixes for new-style exception handling
Commit 30d25dd8 moved the htools code to new-style exception handling,but the hconfd code hasn't been, which fails when compiling on newerGHC versions.
This patch does the rest of the move; however, the situation is not...
Switch to new-style exception handling
Currently, we're using Prelude.catch to handle I/O errors inhtools. This style of error handling has been deprecated for a while,but it still used to work without warnings.
However, the GHC release 7.4 has started to emit deprecation warnings...
Change a type computation for compatibility with 6.12
This is the last warning related to TemplateHaskell that was 6.12specific; for some reason, it doesn't "see" that traw/tname were used.
The patch just replaces the quoting syntax with an explicit type...
Fix compatibility with TemplateHaskell from GHC 7.4
GHC 7.4 has updated the TemplateHaskell library, and it turns out thatthe way we built the JSON instance implementation for showJSON was notgood (probably this is why GHC 6.12 was generating some warnings)....
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
Add command line option for controlling syslog use
… and enable it in hconfd.
Add support for syslog logging to Ganeti.Logging
Currently this is initialised to no from Daemon.hs, but will in thefuture allow command-line options for controlling it.
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unixconvention. This will help with testing.
Update hconfd bind address handling
Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to useeither the any network for the configured cluster address family(ipv4/ipv6), or whatever the user passes in via the --bind option.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add skeleton ssconf module
This currently has only one export function in it, which will be usedfor future bind address functionality in daemons.
Add two utility functions
These both are work with/on the Result type, so we add them toBasicTypes. The functions will be used as more generic versions ofsome more specialised functions that are right now spread across themodules.
Add the bind-address option
This implements the same logic as the Python code: if the option isnot used, use the default appropriate for the cluster, otherwise tryto parse and use whatever was passed in.
htools: fix long version of --port for daemons
The dashes do not need to be in, if they are then the resulting optionis:
-p PORT ----port=PORT Network port (default: 1814)
hspace: Make use of the spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Fix some compiler warnings
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actualcurl part), also in the case where we didn't compile against curl.
htools: Reflect the rename of spindle_usage to spindle_use
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to workon shared storage instances. Diskless instances are handled the same(as in, None is the same everywhere :).
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just useeither with const functions (not really readable, but…).
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improvethe docstrings.
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) thatit will be used for other replace modes too. So all we need to do isto abstract it to a separate function (that can be called from...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work(with the known caveats related to disk space calculation) for sharedstorage disk templates.
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
htools: Convert all the other backends as this is trivial
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
htools: add confd server module
This contains a more-or-less complete implementation for theserver-side confd.
Note that most of the code is behaving identical to the Python code,with a notable exception: the asyncore/inotify code was changed (sinceHaskell doesn't have asyncore) to 3-thread system. While otherwise I...
htools: add basic daemon-related functionality
This is not complete for now, just the basic functionality has beenimplemented:
- daemonize- check we're running under the correct user- call setup logging
htools: add logging functionality
This adds a dependency on hslogger 1.1+, which is not available inDebian Squeeze, only in later versions.
htools: add runtime functionality
This duplicates the lib/runtime.py functionality, allowing us to checkfor runtime users and groups consistency.
htools: add a small module for crypto functions
This adds a dependency on the following new libraries:
- Data.HMAC module from the 'crypto' library- Data.Text from the 'text' library- Data.ByteString from the 'bytestring' library
In Debian, the bytestring library comes with the ghc compiler, whereas...
htools: add definitions for confd types
While we have some of these as plain types in Constants.hs, we addproper ADT definitions for them in a new file. Furthermore, we add theConfdRequest and ConfdReply types here (in Python they are inobjects.py).
htools: add a very basic lib/config.py equivalent
This only is able to load the configuration data for now.
htools: add partial implementation of lib/objects.py
This is partial since not all object types can be easily converted fornow (will need some changes on the Python side for this).
Most importantly, the *Params types do not have a good solution now:the Python code, due to its dynamic typing, hides the fact that we...
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only viasecondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we checkwhether we're internally mirrored, externally mirrored, or not...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Rework evacDrbdSecondaryInner to take the imove
Instead of hard-coding the instance move operation, let's accept is asa parameter from the caller. This is the single thing that makes thisDRBD-specific, so removing it will help for futurepatches. Furthermore, this abstract the imove op type into a single...
Add a simple type alias for nicer type signatures
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes orhard-coding it, compute it from the disk template of theinstance. Otherwise tests will fail when the disk template is notDRBD8 and we ask for allocation on 2 nodes....
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance asprimary and secondary to a given node. This won't work for instancetemplates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates...
Use the mirror type in Cluster.possibleMoves
Currently the function possibleMoves always presumes DtDrbd, which iswrong for shared storage. To make it work with all disk templates, weadditionally pass the mirror type to it (we assume that the movebehaviour only depends on the mirror type, and not the individual...
iPolicy: enable by default all storage templates
… otherwise we can't run tests or simulations using non-defaulttemplates. Tests still pass.
Fix printSolutionLine to handle big scores
Currently we don't set a boundary on the length of the score field,which breaks all our nicely-aligned output efforts.
Fix printSolutionLine to handle non-DRBD steps
Currently Cluster.printSolutionLine function always wants to print theold and new secondary, and as such it fails (ugly) for moves ofnon-DRBD instances. Change it so that it looks explicitly fornoSecondary nodes, and format the output different in that case....
Add a helper type and function for the mirror type
Currently we track the mirroring type as simply yes/no, with "yes" meaning DRBD. This is not enough for handling shared storage, so weneed to add a type that gives the exact mirroring type (none,internal, external), and a function that ties the disk template type...