ExtStorage support in htools
This just adds new constructors for the disk template and relatedtypes, enabling the Ext storage.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Move rstripSpace from Ssconf to Utils
Also renames it in the process and adds a few tests for it.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
confd: strip newlines before logging the request
The Python confd client sends newlines in messages (due to use ofDumpSignedJson/DumpJson), so that results in ugly debug output.
Make node groups' networks field a dict of nicparams
The networks field of the node groups class is a dictionaryof nic params and not a list of networks. This should fixthe broken QA.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Display command description in htools usage message
By abstracting the command/personality formatting, we can easilyswitch htools to the same output as the rest of the commands.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Add hroller htools personality
This is a new personality that for the moment doesn't do anything.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Use hroller to print nodes to reboot by groups
This is highly incomplete, as per the manpage BUGS section.
Add a network fields to NIC params and node group type
This adds a network field to the NIC params and a list of networksto the node group type. It adjusts the nodegroup's unit tests accordingly.
Move Network definition ahead of depending definitions
Networks will be properties of NicParams and NodeGroups. Thereforethey need to be defined before those.
Enable job queries via confd in gnt-node and RAPI
This patch is enabling split queries for jobs for gnt-node and rapiaccess (only for job listing, not job waiting).
Move a node-specific function to Query.Node
This shouldn't live in the main Query module, since it's too nodespecific.
Add support for job queries in hconfd
This adds support for job queries, including (basic) unit-tests.
I've tested this for memory and cpu usage as follows:
- 3600 jobs (live queue): - via masterd, default: ~1.1s (masterd: ~60MB ram) - via confd, default: ~1.1s (hconfd: ~25MB ram)...
Change makeSimpleFilter accept NumericValues as well
This is required for QueryJobs implementation; we changemakeSimpleFilter to support both string and integers as names.
Add a read-only job queue module
This patch adds implementation for a read-only job queue module,together with "full" test (as full as can be in a lazy language…).
One note about the behaviour of the job queue is the handling ofopcodes that fail validation: the 'input' opcode actually is a...
Implement support for QFF_SPLIT_TIMESTAMP
I am very very unhappy with this patch. We have to do this, in orderto be compatible with the Python code and behaviour, but there's lotsof duplication; both the Python and the Haskell code will need to becleaned up and simplified (we don't need QFF_* at all)....
Add exclusive_storage node parameter
Unit tests updated and expanded with an inheritance check.
The flag has no effect yet.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add a few tests for OpParams types
As OpParams definitions are used at Template Haskell type, they don'tget any coverage (although the values defined by them are actuallyused in unittests). However, we can at least test some of thefunctions defined in the module....
Improve mon-collector drbd CLI handling
Currently, this doesn't abort if multiple arguments are passed, andthe error message for errors during file read has a duplicate "Error".
Remove two hlint overrides
"Use comparing" was not needed, and "Use on" can be solved by actuallyusing on in the single case we have (the patch also reorders importsin that file).
Add a custom CLI module for DataCollectors
Currently, the mon-collector binary uses the HTools/CLI module, whichis OK but mean it links in lots of htools code. By copying that moduleto DataCollectors/CLI and removing the unneeded code, we reduce thenumber of modules it depends on fro 20 to 12, meaning both a shorter...
Add a type alias for the personality lists
The signature of the personality definitions is so ugly that, atleast, we should hide it a bit behind a type alias.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Introduce generic multi-command binary handling
Currently, the mon-collector binary is the only multi-command binarywe have in Haskell. Not surprisingly therefore, its command linehandling is not as robust as we need, not being able to supportstandard "--help" commands....
Add description to personality definitions
This allows usage information to display nicer help (like in Python).
Implement fake completion support for multi-cmd binaries
This will allow build-bash-completion to understand what is the listof commands that each binary supports, and then provide recursivelycompletion support for each command.
Note: by "fake", I mean we use hand-built output and hardcoded...
Allow shutting down offline instance
If an instance is offline we definitely shouldn't start it up.But shutting it down, should it be up by mistake is not "that" bad.Still, we only allow it with --force, as it still performs an action onan instance we shouldn't touch. This should make everybody happy....
Switch Luxi sendMsg from strict to lazy ByteStrings
Commit e821050d (“Switch the Luxi interface from Strings toByteStrings”) was designed to optimise the receive interface, but hasan unfortunate side-effect: when sending non-trivial messages, itmeans that both the entire String and the ByteString versions must be...
Add NFData instances for query results
This allows us to ensure that query results are strict as we buildthem, instead of being lazy and only evaluated when the reply is sentover the Luxi interface.
Introduce NFData instances for JSON types
The JSValue/JSObject types don't come with a NFData instance, so let'sadd one ourselves, so that we can force evaluation of JSValues (eitherwhen building or when reading them).
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix parsing of absolute job dependencies
It seems that Python code generates (sometimes) absolute jobdependencies which are strings, instead of integers, so we should beable to parse these as well. We simply change from explicit int-basedparsing (makeJobId) to the generic one (parseJobId)....
Make two common opcode parameters always serialised
This changes deps and comment fields to always be shown, to match thePython behaviour for (at least) job listing/ops field.
Fix disk_template OpInstSetParams parameter
This is not a required field, but rather an optional one; we add a newparameter and use it instead.
Support 'null' in Luxi QueryJobs names field
Python code sometimes sends this, so let's support it even though it'snon-standard.
Abstract code related to job ID creation
Having makeJobIdS as a separate function will allow us to use itoutside of json encoding.
The patch also exports one more function from the Types module.
Export more paths from Path.hs
Also adds a small helper for building the paths.
Simplify a bit the FilterRegex JSON instance
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposedto Python, we always use custom code for formatting, since we don'twant to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,...
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix small typo in OpTestDelay/duration definition
The extra space results in inaccessible names; currently GHC doesn'tflag this as an error, but I've filled an upstream bug for that(http://hackage.haskell.org/trac/ghc/ticket/7484).
Add constants for two values which we re-evaluate
Two constants which we use as Integer are defined as Int inConstants.hs (coming from constants.py), so we do the conversion everytime we use it. Let's move them to top-level names, so that theconversion is only done once....
Very very very basic openvswitch support
This is a "better than nothing" support, just for kvm and just joiningthe machine to the opevswitch bridge with the right command.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
remove duplicate code in Graph.hs
Also update the docstring of a function.
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxireplies. This is not a good idea; the logging library operates onstrings, and as such it will use huge amounts of memory: without debug...
Jobs.hs: make L.Client the last argument for all functions
Having the Luxi client be the last argument of the functions allows foreasier use with `Exception.bracket L.getClient L.closeClient ...`.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,the name of the actual data collector to be run.This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD...
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
htools/Ganeti/Errors: Add ECodeTempNoRes
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” touse opportunistic locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix typo in comment
There was a typo in the docstring of the parseMessage function of theconfd server.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Hbal.hs: move job execution functions to Jobs.hs
Ganeti.Jobs now holds functions that can be used to submit and monitor thestatus of jobs. In particular, execJobsWait and waitForJobs are factoredout of Hbal.hs.
Signed-off-by: Dato Simó <dato@google.com>...
Hbal.hs: use Result (), not Bool, as return value for exec*
Previously, functions in Hbal.hs related to execution of jobsets werereturning only IO Bool, and printing any errors they found directly tostderr on their own.
I'm going to be moving some of these functions to a library module in...
Hbal.hs: rename functions that deal with early termination
In Hbal.hs, it's easy to get lost in the flow of exec* functionsbecause their names are similar and don't convey their differentpurposes (e.g. runJobSet, execJobSet, execWrapper).
This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'...
Hbal.hs: fix typo in signal handler names (hangle -> handle)
'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and'handleSigTerm', respectively.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're thetarget of a future move of some functions, so we leave them around, anddon't delete them, to avoid unnecessary delete/create diffs....
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstractsthe neighColors function into two subfunctions that this algorithm canreuse.
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the caseonly if there are no loops and the graph is undirected.
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.It can transform a list of nodes plus a list of instances into a graphwhich uses the nodes as vertices, and instances as edges connecting them(as long as they have both a primary and a secondary node)...
Fix Dsatur and add Dcolor
Our Dsatur implementation was incorrect: while the paper defined thedegree of saturation of a vertex as the number of different colors it isadjacent to, we were using the number of colors, without consideringuniqueness. This effectively implemented a different algorithm, which is...
Add "proper coloring" unittest check
We have to check that for each edge its vertices have different colors.
This is very easy to do with a vertex-to-color map, but not so easy witha color-to-vertex one. Since all our coloring algorithms created avertex-to-color map behind the scenes and then converted it, we flip...
Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes thesignatures much nicer.
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directlya JSValue; in contrast, the object (single-constructor) types generatea 'toDict' function, and then `showJSON = makeObj . toDict`. This is...
Remove unused THH function 'saveLuxiField'
This was deprecated via commit 88609f00, “Switch Luxi TH code fromsimple to custom fields”.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since ingeneral we use custom fields to aggregate multiple entries in the JSONobject into a safer data-type, we should also have a way to declarewhich extra entries this field covers (so that in the future we can...
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines allfields of a given OP_ID. Not very clean, since for an invalid OP_ID wereturn empty list, but since it will only be used in tests it shouldbe good enough....
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix variable names in Loader.assignIndices
This function can be applied to any type of class "Element" but itsinternal variables used to refer to a list of tuples containing theseelements and their names as "nodes". Moreover it built a list of name toindices tuples referring to it as "na" (which is not very clear)....
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Add two support functions for building JSON objects
The new support functions in module Ganeti.JSON will be used by the nextcommits to build JSON objects with optional (Maybe) fields, excludingthem when they contain Nothing.
Change -G option description
The new one is more consistent with the rest of the nodegroupterminology and more generic for use in different htools.
Optimise recursive Query filters
Currently, the And and Or filters use very nice code, e.g. in case ofOrFilter:
any id <$> mapM evaluateFilter flts
However, looking at the memory profiles shows that application ofany/id to monadic values via '<$>' does not work nicely, losing the...
Make Query operators enforce strictness
Currently, the query operators (binop, etc.) create thunks, instead offorcing the evaluation of the simple boolean results. This results inhigher than needed memory use.
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Replace hand-coded 'live' field of OpInstanceMigrate
This is the last inline-written field; I've done it separately fromthe last patch due to changes being needed in HTools (field changedtype).
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Small improvements to the DRBD parser and tests