Make node groups' networks field a dict of nicparams
The networks field of the node groups class is a dictionaryof nic params and not a list of networks. This should fixthe broken QA.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Update the shared storage design document
Update the shared storage design document to reflect the currentchanges, after the implementation of the ExtStorage interface.
Signed-off-by: Constantinos Venetsanopoulos <cven@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
netutils: Normalize all FQDNs
Addresses issue 147. Only the hostnames resolved using“netutils.Hostname($name)” were normalized, but those returned by theclass method “GetSysName” weren't.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Display command description in htools usage message
By abstracting the command/personality formatting, we can easilyswitch htools to the same output as the rest of the commands.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
More style updates for network-related code
Error message formatting, else/if merging to elif, docstrings.
NEWS: Opportunistic locking, clarifications
Some rewording, some clarifications, and a new item on opportunisticlocking.
cmdlib: Additional opportunistic locking check
If opportunistic locking is requested in the opcode, but no iallocatoris used (not specified or no default), an error will be raised.
Detect wrong Python module structure
If, for one reason or another, a module named "ganeti.ganeti" can besuccesfully imported (e.g. through a symlink in the wrong place), errorwhich are diffcult to debug can occur. This piece of code tests for theunlikely case of that happening....
Use hroller to print nodes to reboot by groups
This is highly incomplete, as per the manpage BUGS section.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add hroller htools personality
This is a new personality that for the moment doesn't do anything.
Fix QA for --offline semantics modification
The previous patch series made it simpler to deal with ERROR_up offlineinstances, but didn't fix qa, that had the old expectations. This fixesthat, and adds more testing about the expected behavior.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
One more offline+shutdown fix
Currently gnt-instance shutdown --force of an offline instanceresets the offline status. Which of course is bad and completelyunintended. Fix. (caught by the new qa tests)
Fix network LUs use of error code in OpPrereqError
Currently, most network LUs simply use EINVAL instead of correct errorcodes; this patch tries to adjust the obvious cases.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
_NetworkQuery: Simplify statistics computation
The code can loop directly over “self.wanted”, saving a lookup. Anexplicit loop is also replaced with a generator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
cmdlib: Factorize network conflict detection
LUNetworkConnect and LUNetworkDisconnect had very similar code to detectconflicts between instance's network interfaces and networks. This codefactorizes the common part and does some cleanup:
- Remove single-letter variable for lambda (“l”)...
gnt-network: Fix import for textwrap
The style guide says to use “import foo”-style imports, not “from fooimport bar” unless it's a Ganeti module. There are some places withexceptions, but this one certainly isn't warranted. Also fix the importorder....
gnt-network add: "--network" is required
Also do some minor code re-formatting.
cmdlib: Replace period with colon in error message
query: Factorize code for getting statistics value
This was not only copied for the networking fields in commit 306bed0e,but commit cfcea7ef fixed wrongly ordered parameters and didn't fix theoriginal. Either way, this patch merges the two cases again. The newly...
gnt-network(8): s/ipv6/IPv6/
gnt-network add: Network mode bash completion
This makes entering the command easier.
OpNetworkConnect: Check for network mode
Use generated field list in gnt-network.8
The list of query fields can be automatically generated.
Improve descriptions of network query fields
They should be in the same style as other descriptions.
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but ratheronly replaces “frozenset” where appropriate. Most of the places are“static” information that doesn't change after the module has beenloaded....
Renaming fam -> net_family on test network generation
Small renaming to make the test more readable.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Serialization compatibility test for node groups
This creates a number of node groups with up to 3 networks,serializes them and compares them with their python equvalents.
Add a network fields to NIC params and node group type
This adds a network field to the NIC params and a list of networksto the node group type. It adjusts the nodegroup's unit tests accordingly.
Move Network definition ahead of depending definitions
Networks will be properties of NicParams and NodeGroups. Thereforethey need to be defined before those.
Remove some unused Python code
This patch removes code which is no longer used due to refactoring:
- http.InitSsl, last usage removed in commit 33231500 (“Convert RPC client to PycURL”)- rapi.baserlib.MakeParamsDict, last usage remove in commit 4e5a68f8...
Enable job queries via confd in gnt-node and RAPI
This patch is enabling split queries for jobs for gnt-node and rapiaccess (only for job listing, not job waiting).
Move a node-specific function to Query.Node
This shouldn't live in the main Query module, since it's too nodespecific.
Add support for job queries in hconfd
This adds support for job queries, including (basic) unit-tests.
I've tested this for memory and cpu usage as follows:
- 3600 jobs (live queue): - via masterd, default: ~1.1s (masterd: ~60MB ram) - via confd, default: ~1.1s (hconfd: ~25MB ram)...
Change makeSimpleFilter accept NumericValues as well
This is required for QueryJobs implementation; we changemakeSimpleFilter to support both string and integers as names.
Add a read-only job queue module
This patch adds implementation for a read-only job queue module,together with "full" test (as full as can be in a lazy language…).
One note about the behaviour of the job queue is the handling ofopcodes that fail validation: the 'input' opcode actually is a...
Implement support for QFF_SPLIT_TIMESTAMP
I am very very unhappy with this patch. We have to do this, in orderto be compatible with the Python code and behaviour, but there's lotsof duplication; both the Python and the Haskell code will need to becleaned up and simplified (we don't need QFF_* at all)....
Add exclusive_storage node parameter
Unit tests updated and expanded with an inheritance check.
The flag has no effect yet.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Update design doc for partitioned Ganeti
PVs don't need to be marked as unallocatable, as the code will ignorepartially used PVs. exclusive_storage made a node parameter.
Remove superfluous typing to Int
Forgot to fix that after the review of patch"Haskell/python compatibility test for networks".
Extend job ID parsing tests
Since this is a type that behaves non-standard, let's add some moretest for "wrong" input values.
Add a few tests for OpParams types
As OpParams definitions are used at Template Haskell type, they don'tget any coverage (although the values defined by them are actuallyused in unittests). However, we can at least test some of thefunctions defined in the module....
Remove two hlint overrides
"Use comparing" was not needed, and "Use on" can be solved by actuallyusing on in the single case we have (the patch also reorders importsin that file).
Add more mon-collector shell tests
These test the newly-changed behaviour, and fix an inconsistency inthe hs-check target (versus hs-coverage).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Improve mon-collector drbd CLI handling
Currently, this doesn't abort if multiple arguments are passed, andthe error message for errors during file read has a duplicate "Error".
Enable bash completion for mon-collector
This just ties in all the pieces so far and enables bash completionfor it.
bash-completion: add support for multi-cmd Haskell binaries
This patch adds support for parsing the command list out of a binary(very strict format), and then iterating over that and building thesub-commands options/arguments.
It also does a bit of general cleanup in the script....
Implement fake completion support for multi-cmd binaries
This will allow build-bash-completion to understand what is the listof commands that each binary supports, and then provide recursivelycompletion support for each command.
Note: by "fake", I mean we use hand-built output and hardcoded...
Simplify htools.hs personality selection
No need to do a boolean-based lookup, we can use the regularassociation list lookup.
Add description to personality definitions
This allows usage information to display nicer help (like in Python).
Introduce generic multi-command binary handling
Currently, the mon-collector binary is the only multi-command binarywe have in Haskell. Not surprisingly therefore, its command linehandling is not as robust as we need, not being able to supportstandard "--help" commands....
Add a type alias for the personality lists
The signature of the personality definitions is so ugly that, atleast, we should hide it a bit behind a type alias.
Add a custom CLI module for DataCollectors
Currently, the mon-collector binary uses the HTools/CLI module, whichis OK but mean it links in lots of htools code. By copying that moduleto DataCollectors/CLI and removing the unneeded code, we reduce thenumber of modules it depends on fro 20 to 12, meaning both a shorter...
Fix profiling targets
As noted by Guido, there are problems when using the hs-prof andhs-prof-quick targets in the default configuration (compiling allprograms). The errors manifest in the form of wrong symbols duringcompilation.
I knew that the hs-prof targets, which compiled multiple objects with...
Allow shutting down offline instance
If an instance is offline we definitely shouldn't start it up.But shutting it down, should it be up by mistake is not "that" bad.Still, we only allow it with --force, as it still performs an action onan instance we shouldn't touch. This should make everybody happy....
Allow running instances to be put online
If an instance is running (eg. ERROR_up) and at the same time offline,there's no way to either shut it down or reonline it. This allowsonlining it. Offlining is still disabled for running instances.
Fixes and improvements to comments
Some fixes, added more information in a few points, removed a stale (5+year old) TODO comment.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Explain how to use GHCI to see spliced TH code
This can be more useful than the "make" version.
Fix error during cluster initialization due to hv_kvm
Commit 141d148 was a bit too enthusiastic. The three parameters added tothe list of parameters to be checked default to a value not evaluatingto false, leading to a failure on cluster initialization....
Make LUNetworkAdd pass _VerifyLocks()
LEVEL_NODE_ALLOC should be aquired too if LEVEL_NODE is ALL_SET.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Haskell/python compatibility test for networks
This patch contains the HUnit test that checks the compatibilityof Haskell-generated networks with the python code. For that thegeneration of test instances of networks was enhanced to meetthe validation steps of the python implementation. Also, so far...
Add test for SPICE parameter list, add missing ones
“_SPICE_ADDITIONAL_PARAMS” is supposed to be the full list ofSPICE-related KVM hypervisor parameters with the exception of“HV_KVM_SPICE_BIND”. The new test checks if all parameters starting with“HV_KVM_SPICE_*” are included. Three previously missing parameters are...
hv_kvm: List of SPICE parameters should be module-global
This list is static at runtime and doesn't need to be recreated everytime.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
_DeclareLocksForMigration: Fix non-DRBD locking issue
When non-DRBD disks are used for an instance,“lu.needed_locks[locking.LEVEL_NODE]” is set to “locking.ALL_SET” (whichis None). The assertion will then fail as None evaluates to False.
Reported by Constantinos Venetsanopoulos....
LUOobCommand: Always declare NAL in shared mode
Doing so avoids triggering an assertion in mcpu which cross-checks thenode lock and node allocation lock acquisition mode.
Fix two logging messages in TLReplaceDisks
Commit f0f8d060 (“Show old primary/secondary node on diskreplacement”) added two wrong uses of feedback_fn, which results inlog entries such like these:
"log": [ [ 7, [1351258326, 466214], "message", "Replacing disk(s) 0 for instance 'instance1.example.com'"...
Split and extend Confd types tests
Some confd types were not tested for serialisation, so let's movethese (type-related) tests to their own module and extend them.
Switch Luxi sendMsg from strict to lazy ByteStrings
Commit e821050d (“Switch the Luxi interface from Strings toByteStrings”) was designed to optimise the receive interface, but hasan unfortunate side-effect: when sending non-trivial messages, itmeans that both the entire String and the ByteString versions must be...
Add NFData instances for query results
This allows us to ensure that query results are strict as we buildthem, instead of being lazy and only evaluated when the reply is sentover the Luxi interface.
Introduce NFData instances for JSON types
The JSValue/JSObject types don't come with a NFData instance, so let'sadd one ourselves, so that we can force evaluation of JSValues (eitherwhen building or when reading them).
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix parsing of absolute job dependencies
It seems that Python code generates (sometimes) absolute jobdependencies which are strings, instead of integers, so we should beable to parse these as well. We simply change from explicit int-basedparsing (makeJobId) to the generic one (parseJobId)....
Make two common opcode parameters always serialised
This changes deps and comment fields to always be shown, to match thePython behaviour for (at least) job listing/ops field.
Fix disk_template OpInstSetParams parameter
This is not a required field, but rather an optional one; we add a newparameter and use it instead.
Support 'null' in Luxi QueryJobs names field
Python code sometimes sends this, so let's support it even though it'snon-standard.
Abstract code related to job ID creation
Having makeJobIdS as a separate function will allow us to use itoutside of json encoding.
The patch also exports one more function from the Types module.
Add new Haskell library dependency for tests
The 'temporary' library add support for create temporarydirectories/files beyond what is offered in the base library (which issomewhat limited, and hard to use in tests).
Reduce logging level in tests
This allows tests to execute IO actions which have logging as a sideeffect, without polluting the stderr too much.
A better solution would be that we have fine-grained control overloggers, so that tests can run with their own logging, etc. etc.....
Export jstore constants as well to Constants.hs
This single constant could be as well moved to constants.py. For now Ichoose to just export the module as well.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Generalise the test helper 'genUniquesList'
Instead of always using "arbitrary", make it require a generator (thatcan then use more complex rules for building the list elements), andalso convert it to use a temporary set instead of list membership.
Export more paths from Path.hs
Also adds a small helper for building the paths.
Simplify a bit the FilterRegex JSON instance
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposedto Python, we always use custom code for formatting, since we don'twant to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,...
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exactinput values; this makes it harder to check consistency againstHaskell code.
To compensate for this, we add a way to override the formatting of the...
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Fix small typo in OpTestDelay/duration definition
The extra space results in inaccessible names; currently GHC doesn'tflag this as an error, but I've filled an upstream bug for that(http://hackage.haskell.org/trac/ghc/ticket/7484).
Add constants for two values which we re-evaluate
Two constants which we use as Integer are defined as Int inConstants.hs (coming from constants.py), so we do the conversion everytime we use it. Let's move them to top-level names, so that theconversion is only done once....
Fix an small but quite nasty typo
Introduced in commit d4752785.
update gnt-instance manpage about self-rename
burnin: rename instances to same name
this (a) allows us to test renaming in burnin even if we don't have anextra instance name and (b) is a valid operation, as it allows to "fix" an instance who went name-astray
Very very very basic openvswitch support
This is a "better than nothing" support, just for kvm and just joiningthe machine to the opevswitch bridge with the right command.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Improve a few Graph test properties
Return type is changed from Property to Bool, and the ==? True at theend is dropped.
remove duplicate code in Graph.hs
Also update the docstring of a function.
Add profiling dependencies note
As discussed on the list, these are needed to do a make hs-prof.
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxireplies. This is not a good idea; the logging library operates onstrings, and as such it will use huge amounts of memory: without debug...
Pass check_ip and not hostname.ip to _ComputeNics
This should be done because in the case of --no-name-check thereis no 'hostname.ip' attribute, causing an execution error.'check_ip' is always set (in CheckArguments) even if --no-name-checkis passed in the command line....
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment thatneeds updating for a new version is more likely going to trigger a checkof the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all testshave been completed.
Move four designs from draft to 2.7
These are the ones I implemented and for which I'm certain they'recomplete.
RAPI documentation fix and update
- Mention user capabilities in security.rst- Replace “query” with “read” in RAPI documentation
Add hpc-mon-collector to .gitignore
It is a binary file and it should never be added to the git repository.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Read watcher pause using RPC, not directly
The master daemon should not directly read files written by the nodedaemon. This patch adds a new RPC to read the watcher pause file andchanges the master code to use it.