Cleanup the QC.hs file w.r.t. compiler options
Currently, we build the Haskell unittests with custom GHC flags,because we (I) were quite lazy when initially writing theunittests. This is not a problem for the tests themselves, but itcreates problem when (for example) one would want to pass all...
Fix Query2.hs to compile without warnings
Since this is not yet used by any targets, we didn't detect yet thecompilation warnings. Just trivial exports/imports cleanup.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
hbal: return exit status 0 in case of early exit
This derives from an internal bug, but the story is consistent acrossboth internal and external usage of hbal.
Basically right now, hbal returns exit code 1 if requested to exitearly, even if all jobs are successful. This is counter-intuitive due...
Add query2 numeric comparison operators
These were recently added to the python version.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Haskell data types for query2
This contains the types to express query2 queries, filters and results.
Fix prefix bug in Haskell ssconf implementation
This patch fixes a (brown-bag) bug in the ssconf implementation wherethe looked-for filenames were not prefixed with the ssconf fileprefix.
Additionally, a test for this is added (which fails without the fix)....
Add support for SIGHUP handling in Haskell daemons
This patch adds support for handling SIGHUP and reopening thestdout/stderr logfile in daemon mode (but not in foreground mode).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Rework logging setup for Haskell daemons
This fixes a missing functionality: closing of stdout/stderr whenforking daemons. Without this, starting the daemons from the commandline seems to work, but doing it from utils.RunCmd breaks, sincewithout closing the standard descriptors, RunCmd never finishes....
Add decoding of Luxi calls and unittests for LuxiOp
This patch adds a hand-coded decoder for LuxiCall arguments, as thedata-structure is not uniform enough for automated generation (evenfor the serialisation, we had to add hints for some fields,de-serialisation is even harder)....
Some unit tests improvements
In preparation for unit-testing the Luxi definitions, this improvesthe auto-generation of OpCodes: currently this uses unbounded plainStrings for names and fields, and this creates too big test sizes(e.g. when automatically generating job sets)....
Auto-define a LuxiReq data type
We currently auto-generate a LuxiOp data type, which holds the entireoperation (including parameters). However, having a data type just forthe method call would be useful, so let's change THH to alsodefineSADT for the Luxi constructors....
Replace hardcoded constants with Constants.hs names
Now that luxi.py constants are exported, we can use them for moreconsistency.
Add a new JSON function
And its associated unittests.
Further fixes for new-style exception handling
Commit 30d25dd8 moved the htools code to new-style exception handling,but the hconfd code hasn't been, which fails when compiling on newerGHC versions.
This patch does the rest of the move; however, the situation is not...
Switch to new-style exception handling
Currently, we're using Prelude.catch to handle I/O errors inhtools. This style of error handling has been deprecated for a while,but it still used to work without warnings.
However, the GHC release 7.4 has started to emit deprecation warnings...
Change a type computation for compatibility with 6.12
This is the last warning related to TemplateHaskell that was 6.12specific; for some reason, it doesn't "see" that traw/tname were used.
The patch just replaces the quoting syntax with an explicit type...
Fix compatibility with TemplateHaskell from GHC 7.4
GHC 7.4 has updated the TemplateHaskell library, and it turns out thatthe way we built the JSON instance implementation for showJSON was notgood (probably this is why GHC 6.12 was generating some warnings)....
Add trivial tests for gnt-* cli
While testing some other stuff, I realised that the gnt-* commandscould be broken (as in, the script fails with syntax errors), but makecheck doesn't detect it. Since we have shelltest, we can now addtrivial tests for this case....
Rework exit model
While updating the confd code, I realised that we have lots ofduplication in the exit model for the various programs.
So this patch attempts to abstract all the exits via a couple of newfunctions; sorry for the somewhat big patch, but I hope the payoff is...
Fix out-of-tree builds
The new shell tests do not succeed out-of-tree, due to static pathsand other issues. This trivial patch fixes these issue, make distchecknow passes.
Convert manual shell tests to shelltestrunner
This is more of a RFC… Basically most of the shell-based tests areconverted from exec+grep to shelltestrunner.
Things are not all fine and nice though:
- we have dependencies between tests, as some generate some data files...
Add command line option for controlling syslog use
… and enable it in hconfd.
Add support for syslog logging to Ganeti.Logging
Currently this is initialised to no from Daemon.hs, but will in thefuture allow command-line options for controlling it.
Allow hail to read data from stdin
This patch makes hail treat '-' as denoting stdin, per the usual Unixconvention. This will help with testing.
Update hconfd bind address handling
Instead of hardcoded IPv4 INADDR_ANY, this patch changes hconfd to useeither the any network for the configured cluster address family(ipv4/ipv6), or whatever the user passes in via the --bind option.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add skeleton ssconf module
This currently has only one export function in it, which will be usedfor future bind address functionality in daemons.
Add two utility functions
These both are work with/on the Result type, so we add them toBasicTypes. The functions will be used as more generic versions ofsome more specialised functions that are right now spread across themodules.
Add the bind-address option
This implements the same logic as the Python code: if the option isnot used, use the default appropriate for the cluster, otherwise tryto parse and use whatever was passed in.
Rename htools-hpc to hpc-tools to fix laziness
This is a stupid/trivial patch. I didn't realise when addinghtools-hpc that this will "break" tab completion on htools; as in, Iwill have to add / manually always when completing files on thecommand line....
htools: fix long version of --port for daemons
The dashes do not need to be in, if they are then the resulting optionis:
-p PORT ----port=PORT Network port (default: 1814)
hspace: Make use of the spindle_use
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Fix some compiler warnings
Add tests for custom text files
Add utilisation and instance selection tests
Enable pipefail mode for shell tests
These allow more accurate checks.
Add RAPI shell-level unittest
Enhance the RAPI backend to accept file:// URLs
This will allow offline testing of this backend (except for the actualcurl part), also in the case where we didn't compile against curl.
Add new hail (shell) tests
Using some hand-crafted files (manually modified from a real clusterrunning master branch), we test that hail behaves as we expect:doesn't load obvious wrong data, can do relocations/allocations, etc.
htools: Reflect the rename of spindle_usage to spindle_use
Add a helper function for mirrorType computations
We most always use `templateMirrorType . diskTemplate`, so let's add ahelper function for this.
Enable all disk templates in unit-tests
After the many patches implementing the shared storage functionalityin iallocator, balances, etc., and after the patches preparing theunit-tests to do the right thing for shared storage, we can finallyflip the bits that make instances have "random" disk templates as...
Enable evacuation/relocation for shared storage
This patch toggles the final bit and enables nodeEvacInstance to workon shared storage instances. Diskless instances are handled the same(as in, None is the same everywhere :).
Replace explicit case with 'either'
Instead of manually case-ing on the Either contents, let's just useeither with const functions (not really readable, but…).
Rename evacDrbdSecondaryInner to evacOneNodeInner
The function is now generic, so let's rename it and cleanup/improvethe docstrings.
Abstract the secondary evac into a separate function
This is generic enough (no longer depending on the replace type) thatit will be used for other replace modes too. So all we need to do isto abstract it to a separate function (that can be called from...
Fix Node_addPriFD test for arbitrary disk templates
Currently, this test assumes localStorageTemplates, so let's make surewe enforce that on the given instance
Extend the possible move types with FailoverToAny
This brings together all previous pieces and allows balancing to work(with the known caveats related to disk space calculation) for sharedstorage disk templates.
htools: Add the new ipolicy field
Also add the spindle usage to QC
htools: Use the spindle usage in the algorithms
In the same transition also do a simple refactor of the caculation
htools: Convert all the other backends as this is trivial
htools: Adapt the Text backend to spindle usage
The old text files remain compatible with this change.
htools: Adding spindle usage to Instance
For now we just pass in "1" as spindle usage and adapt the backendlater.
Rework unit parsing
Due to how conversions were implemented previously, 1TB failed toparse on 32-bit, as we were overflowing during computation, eventhough the final result would fit easily.
This patch moves the parsing of the scaling factor to a separate...
Add more rebalance offline tests
These test that we can actually rebalance (and that it results inreplace-disks and failovers/migrations), and that we don'tdouble-rebalance, etc.
Enhance the offline tests
This adds new offline tests:
- checks that files generated by hspace can be read by hbal/hinfo- checks that hbal handles node groups as expected- checks that enabling multiple backends results in failure- checks hspace machine readable code and higher verbosity...
Build a coverage-enabled version of the tools
This allows the offline tests to also generate coverage data, and thehs-coverage target is changed to show both unit-test and offline-testscoverage.
The downside is that now we build yet-another-binary, which makes the...
Add a simple binary tester for htools
This only tests the command line basic options. In the future, it canbe expanded to check other things.
Add a helper for htools CLI tests
… and update live-test.sh to use it, instead of hard-coding theprogram paths.
A few additional style fixes are also done (indentation).
htools: add confd daemon binary
This is currently not installed anywhere, just built.
htools: add confd server module
This contains a more-or-less complete implementation for theserver-side confd.
Note that most of the code is behaving identical to the Python code,with a notable exception: the asyncore/inotify code was changed (sinceHaskell doesn't have asyncore) to 3-thread system. While otherwise I...
htools: add basic daemon-related functionality
This is not complete for now, just the basic functionality has beenimplemented:
- daemonize- check we're running under the correct user- call setup logging
htools: add logging functionality
This adds a dependency on hslogger 1.1+, which is not available inDebian Squeeze, only in later versions.
htools: add runtime functionality
This duplicates the lib/runtime.py functionality, allowing us to checkfor runtime users and groups consistency.
htools: add a small module for crypto functions
This adds a dependency on the following new libraries:
- Data.HMAC module from the 'crypto' library- Data.Text from the 'text' library- Data.ByteString from the 'bytestring' library
In Debian, the bytestring library comes with the ghc compiler, whereas...
htools: add definitions for confd types
While we have some of these as plain types in Constants.hs, we addproper ADT definitions for them in a new file. Furthermore, we add theConfdRequest and ConfdReply types here (in Python they are inobjects.py).
htools: add a very basic lib/config.py equivalent
This only is able to load the configuration data for now.
htools: add partial implementation of lib/objects.py
This is partial since not all object types can be easily converted fornow (will need some changes on the Python side for this).
Most importantly, the *Params types do not have a good solution now:the Python code, due to its dynamic typing, hides the fact that we...
htools: Make it possible to load an IAlloc request
This is only useful in the stand-alone command line tools:
- hbal- hspace- hinfo
This change also documents the flag in all man-pages.
Fix relocation test ClusterAllocRelocate
As for the evacuate tests, we require a mirrored disk template, butotherwise the test should work for both mirror types.
Additionally, we perform a simplification that was left as TODO.
Fix IAlloc.processRelocate to support shared storage
Currently, this function expects (but doesn't really check! only viasecondary node being defined) a DRBD8 template type.
This patch (not very readable, sorry) changes it so that we checkwhether we're internally mirrored, externally mirrored, or not...
Fix up Cluster nodeEvac unittests for shared storage
Currently, these presume that the instance being generated always isDRBD/relocatable (but won't be in the future), and also presume thatall relocation modes can be used (hint: not ☺).
So we add a couple of helper functions to restrict the used disk...
Rework evacDrbdSecondaryInner to take the imove
Instead of hard-coding the instance move operation, let's accept is asa parameter from the caller. This is the single thing that makes thisDRBD-specific, so removing it will help for futurepatches. Furthermore, this abstract the imove op type into a single...
Add a simple type alias for nicer type signatures
Fix a few tests that hard-coded the number of nodes
Instead of randomly generating the number of required nodes orhard-coding it, compute it from the disk template of theinstance. Otherwise tests will fail when the disk template is notDRBD8 and we ask for allocation on 2 nodes....
Fix test Node_addSec_idempotent
Another test that only works for DRBD8 instances.
Fix the Node_rMem test to specify DRBD8 templates
… since memory reservation only is used for this template.
Split the Node_addOffline test into pri/sec
Currently, this (single) test checks whether we can add an instance asprimary and secondary to a given node. This won't work for instancetemplates other than DRBD8, so we split it into:
- an add to primary test, which should work for all disk templates...
Use the mirror type in Cluster.possibleMoves
Currently the function possibleMoves always presumes DtDrbd, which iswrong for shared storage. To make it work with all disk templates, weadditionally pass the mirror type to it (we assume that the movebehaviour only depends on the mirror type, and not the individual...
iPolicy: enable by default all storage templates
… otherwise we can't run tests or simulations using non-defaulttemplates. Tests still pass.
Fix printSolutionLine to handle big scores
Currently we don't set a boundary on the length of the score field,which breaks all our nicely-aligned output efforts.
Fix printSolutionLine to handle non-DRBD steps
Currently Cluster.printSolutionLine function always wants to print theold and new secondary, and as such it fails (ugly) for moves ofnon-DRBD instances. Change it so that it looks explicitly fornoSecondary nodes, and format the output different in that case....
Add a helper type and function for the mirror type
Currently we track the mirroring type as simply yes/no, with "yes" meaning DRBD. This is not enough for handling shared storage, so weneed to add a type that gives the exact mirroring type (none,internal, external), and a function that ties the disk template type...
Small simplification in tryBalance
Instead of computing the bad/good node lists separately (in effectiterating twice over the all_nodes list and also creating that listtwice from the container), let's use 'partition' and do a smallimprovement.
Mark Rbd as a movable disk template
Do not rely on no-secondary for movable tests
Currently, the code in Loader.hs marks as un-movable any instanceswhich don't have a secondary. This is not valid for any shared storagetemplates, so let's remove that check completely. The initial state of...
Handle nodes without local storage
Our current cluster model is based on per-node storage, and as such wedon't handle shared disk templates. Until we can improve the model,let's handle nodes without local storage better: in order to stillcompute a valid cluster score, we need to handle nodes with tDsk == 0....
htools/ExtLoader: Bring the pieces together
This make it possible to load from an Ialloc file
htools/CLI: Adding new options to use ialloc as source
htools/IAlloc: Adding loadData method for use as backend
htools: Refactor some of IAlloc.readRequest to Hail
This makes it possible to reuse that function when we want to use IAllocas a data source backend
hinfo: Adjust the verbosity
hinfo: Gather and print group statistics
htools: Adding printTable helper
This helper abstract away all the unlines and unwords call you've to doafter you got a formatTable back.
Fixup a Node unittest
This fixes the prop_Node_rMem test, which cares about memory only;otherwise we could fail due to other problems (e.g. not enough disk).
Allow overriding the target test count
This is not perfect, as the override applies identical target testvalues to both the 'fast' and 'slow' tests (making the slow tests takea long time), but it allows a quick override for manual runs.
Add two more unittests for Node add/remove
These just test that add/remove are idempotent. This is not perfect,as we use unsorted lists for some values (instead of sets), so whenusing non-empty nodes this would break (but for empty nodes, a1-element list is sorted, so it's fine)....
Use the spindles metric in cluster scores
This makes balancing use the new metric.
Track instance-used spindles in node set/add/remove
This simply tracks the instance-used spindles (using hard-coded '1'per instance), and additionally prevents additions in soft-mode whenwe go over the limit.
Note: there's an assymetry between addSec and removeSec (basically...
Shorten some function names
Since we use the functions always module-qualified, let's drop the`instance` prefix and use a shorter one.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Simplify logic in Node operations
This patch adds two helper functions and uses them for a somewhatsimplified logic in the add/remove pri/sec node functions.