Introduce NFData instances for JSON types
The JSValue/JSObject types don't come with a NFData instance, so let'sadd one ourselves, so that we can force evaluation of JSValues (eitherwhen building or when reading them).
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix parsing of absolute job dependencies
It seems that Python code generates (sometimes) absolute jobdependencies which are strings, instead of integers, so we should beable to parse these as well. We simply change from explicit int-basedparsing (makeJobId) to the generic one (parseJobId)....
Make two common opcode parameters always serialised
This changes deps and comment fields to always be shown, to match thePython behaviour for (at least) job listing/ops field.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Fix disk_template OpInstSetParams parameter
This is not a required field, but rather an optional one; we add a newparameter and use it instead.
Support 'null' in Luxi QueryJobs names field
Python code sometimes sends this, so let's support it even though it'snon-standard.
Abstract code related to job ID creation
Having makeJobIdS as a separate function will allow us to use itoutside of json encoding.
The patch also exports one more function from the Types module.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add new Haskell library dependency for tests
The 'temporary' library add support for create temporarydirectories/files beyond what is offered in the base library (which issomewhat limited, and hard to use in tests).
Reduce logging level in tests
This allows tests to execute IO actions which have logging as a sideeffect, without polluting the stderr too much.
A better solution would be that we have fine-grained control overloggers, so that tests can run with their own logging, etc. etc.....
Export jstore constants as well to Constants.hs
This single constant could be as well moved to constants.py. For now Ichoose to just export the module as well.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Generalise the test helper 'genUniquesList'
Instead of always using "arbitrary", make it require a generator (thatcan then use more complex rules for building the list elements), andalso convert it to use a temporary set instead of list membership.
Export more paths from Path.hs
Also adds a small helper for building the paths.
Simplify a bit the FilterRegex JSON instance
Implement opcode summary support
This implements in the Haskell codebase the opcode summary. As opposedto Python, we always use custom code for formatting, since we don'twant to use dynamic attribute lookup.
To test this properly, we need to change MetaOpCode to record-syntax,...
Add optional formatting for OP_DSC_FIELD
For some opcodes, the output is not "stable", and depends on the exactinput values; this makes it harder to check consistency againstHaskell code.
To compensate for this, we add a way to override the formatting of the...
Add a type for the log type in job/opcode logs
Also fixes indentation for OP_STATUS.
Fix small typo in OpTestDelay/duration definition
The extra space results in inaccessible names; currently GHC doesn'tflag this as an error, but I've filled an upstream bug for that(http://hackage.haskell.org/trac/ghc/ticket/7484).
Add constants for two values which we re-evaluate
Two constants which we use as Integer are defined as Int inConstants.hs (coming from constants.py), so we do the conversion everytime we use it. Let's move them to top-level names, so that theconversion is only done once....
Fix an small but quite nasty typo
Introduced in commit d4752785.
update gnt-instance manpage about self-rename
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
burnin: rename instances to same name
this (a) allows us to test renaming in burnin even if we don't have anextra instance name and (b) is a valid operation, as it allows to "fix" an instance who went name-astray
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Very very very basic openvswitch support
This is a "better than nothing" support, just for kvm and just joiningthe machine to the opevswitch bridge with the right command.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Add profiling dependencies note
As discussed on the list, these are needed to do a make hs-prof.
remove duplicate code in Graph.hs
Also update the docstring of a function.
Improve a few Graph test properties
Return type is changed from Property to Bool, and the ==? True at theend is dropped.
Log only partial response in Luxi when in debug mode
Currently, we log the entire response (at debug level) in the Luxireplies. This is not a good idea; the logging library operates onstrings, and as such it will use huge amounts of memory: without debug...
Pass check_ip and not hostname.ip to _ComputeNics
This should be done because in the case of --no-name-check thereis no 'hostname.ip' attribute, causing an execution error.'check_ip' is always set (in CheckArguments) even if --no-name-checkis passed in the command line....
Add version check for doc/design-draft.rst
This is similar to README or “doc/security.rst”. Having a comment thatneeds updating for a new version is more likely going to trigger a checkof the lists of design drafts.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Makefile/check-local: List all errors before failing
Use the “error” variable in multiple places and fail only once all testshave been completed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Move four designs from draft to 2.7
These are the ones I implemented and for which I'm certain they'recomplete.
RAPI documentation fix and update
- Mention user capabilities in security.rst- Replace “query” with “read” in RAPI documentation
Add hpc-mon-collector to .gitignore
It is a binary file and it should never be added to the git repository.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Read watcher pause using RPC, not directly
The master daemon should not directly read files written by the nodedaemon. This patch adds a new RPC to read the watcher pause file andchanges the master code to use it.
Set watcher pause on all nodes
Instead of just setting the watcher pause file, which isn't replicated,RPC is used to set it on all nodes (where possible). This means thatafter an outage of the master node and a following master-failover, thewatcher will still be paused....
Add RPC for setting watcher pause
The watcher pause file should be set/unset on all nodes at once, notonly the master node. For that a new RPC is needed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Generated RPC code: Don't write whitespace after paren
There is no obvious need for this space--the line would wrap at the nextspace, which comes soon after.
Rename test for backend._CommonRestrictedCmdCheck
“TestWriteFile” was not renamed when adding the file based on“ganeti.utils.io_unittest-runasroot.py”.
jqueue: Improve inotify error reporting
This addresses issue 218. When the number of inotify watches isexhausted, for example by being set too low from the beginning or byother programs, waiting for a job to change would just report a lost job(e.g. “Error checking job status: Job with id 7817 lost”)....
Improve test for tools.ensure_dirs
- Add more checks, some of them are deliberately redundant- Descriptive error messages- Add comment describing order to “tools.ensure_dirs”- Avoid copying a list in an assertion in “tools.ensure_dirs”
Jobs.hs: make L.Client the last argument for all functions
Having the Luxi client be the last argument of the functions allows foreasier use with `Exception.bracket L.getClient L.closeClient ...`.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix a bug in the hs-coverage rule of Makefile.am
The coverage information for mon-collector was not merged correctly.
uidpool: Remove roman number support
Doing so simplifies to code a bit and never had a practical use.
Remove checks wrt IDISK_PARAMS from OpCode level
Change the "--disks" option validation, to just check the formatof the dict and do not check whether the keys are included in theIDISK_PARAMS constant at OpCode level. This allows the passing ofarbitrary parameters at the CLI, which will then be logically...
Add shelltests for the mon-collector
This commit adds shelltests for the mon-collector binary and for the DRBDdata collector.
Also, it fixes a small bug in the DRBD parser found thanks to the tests.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add the stand-alone data collector for DRBD
It is implemented as a single binary receiving, as its first parameter,the name of the actual data collector to be run.This way, its structure can be used for all the future data collectors.
Also, factored out of bdev.py into constants.py the location of the DRBD...
Move the path of the DRBD status file to the Constants file
It will be needed by the DRBD data collector, that will be added shortly.
Make configure check for haskell vector library
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Remove usage of reserved key word 'family'
An old hlint version (1.8.28) was complaining about theword 'family' being used in the network tests. Thanks toan epiphany of Iustin this was due to that it is a reservedkey word.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Add documentation about new dependency to vector lib
Updates the INSTALL file to add the haskell librarylibghc-vector-dev.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
Fix ordering of entries in tools.ensure_dirs
Commit ebd437a added two new entries to tools.ensure_dirs, but did so inthe wrong order. Patch forthcoming to improve the unittest's errormessage.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Set owner on watcher pause and queue drain files
If the files were created by a user different, e.g. due to a switch fromrunning masterd as root to running it as a dedicated user, they couldn'tbe modified/removed anymore.
lib/tools/ensure_dirs.py: Code formatting
Wrap lines in a consistent manner (uid/gid on the second line) ifwrapping is necessary at all. “git diff --color-words” shows nodifference at all as only whitespace changed.
Replicate queue drain flag across all master candidates
Until now, the flag was unset on a master failover unless the“$localstatedir/lib/ganeti/queue/drain” file existed.
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would haveavoided the issue fixed in commit e2dd6ec. The new function is locatedin the “compat” module as it will be used at module load time in most...
constants: Remove duplicate DRBD barrier option
doc/rapi.rst: Add ECODE_TEMP_NORES
The assertion was failing as ECODE_TEMP_NORES was not documented.
Change value for ECODE_TEMP_NORES
Unfortunately there was a bug in commit 518a45e whereby ECODE_TEMP_NOREShad the same value as ECODE_NORES, leading to failures in a Haskelltest. Of course this would also have affected other users of the constant.
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
masterd: Remove duplicate code
Improve error message for when adding inotify watcher fails
Explicitely mention the fs.inotify.max_user_watches sysctl value.
htools/Ganeti/Errors: Add ECodeTempNoRes
Update pyinotify website
It has moved to github.com.
htest/Types.hs: check ordering of some ADTs
In some ADT, ordering of constructors is important because a certainmeaning is attached to ordinality (e.g., "clusters with lesser AllocPolicyare preferred").
To ensure constructor order is not accidentally changed, this commit...
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add error code for temporary lack of resources
When an instance creation uses opportunistic locks, the iallocator mightnot be able to find an allocation solution if not enough node locks (ora suboptimal subset thereof) were acquired. As per the design document...
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI clientmodule. A newly added unit test ensures all error codes are contained in“ECODE_ALL”, as well as ensuring consistency between the RAPI client and...
cmdlib: Use locked nodes as node whitelist
Also actually start using opportunistic locks (if requested).
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” touse opportunistic locks.
cmdlib: Node whitelist support for allocation request
Forward the node whitelist to the iallocator plugin.
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists areprovided to exclude LUs from the two tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
_ExportQuery: Use node allocation lock
Block instance allocations when all node locks will be acquired.
LUBackupExport: Use node allocation lock
LUBackupRemove: Use node allocation lock
LUInstanceMultiAlloc: Use node allocation lock
Avoid conflicts between instance allocations.
LUInstanceRecreateDisks: Use node allocation lock
LUNodeSetParams: Use node allocation lock
LUNodeQueryvols: Use node allocation lock
LUOobCommand: Use node allocation lock
If no node names are given, all node locks are acquired.
Fix typo in comment
There was a typo in the docstring of the parseMessage function of theconfd server.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix node-daemon-setup test with older pyOpenSSL
Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of“-----BEGIN PRIVATE KEY-----”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add tool to clean up node
Sometimes a node is not removed properly from a cluster (especiallyduring development). This new tool stops all daemons and removes (aftermaking copies) the most critical files.
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
Switch from scp/ssh to node daemon setup utility
This patch does away with many calls to scp and by means of using“tools/node-daemon-setup”, verifies most of the values before writingthem to files.
make genMaybe more Just
There is a common conception that Just something is more worth thanNothing. So we're biasing our tests towards that. As such let's generateNothing fewer times, and Just subgen more times. The values were copiedfrom the "official" maybe generator....
Locking related fixes for networks
Use GetNetwork() only when having already aquired the lock,i.e. in CheckPrereq().
In LUNetworkConnect/Disconnect do not include Network info in Hooksenvironment, so that network locking can be avoided if conflictsare not checked....
Fix build breakage in Jobs.hs test code
Commit 3bdbe4b3 (“Jobs.hs: move OpStatus and JobStatus ADTs toTypes.hs”) removed the TemplateHaskell language pragma fromhtest/Test/Ganeti/Jobs.hs due to a hlint warning, but that is bad: itmeans the testSuite call is no longer interpreted as a splice, so it results in:...
configure.ac: fix reading of HLINT environment variable
configure.ac advertised that the HLINT environment variable could be set tospecify the path to the 'hlint' tool. However, HLINT was being initializedto "no" inside configure.ac, which broke this usage....
Hbal.hs: move job execution functions to Jobs.hs
Ganeti.Jobs now holds functions that can be used to submit and monitor thestatus of jobs. In particular, execJobsWait and waitForJobs are factoredout of Hbal.hs.
Signed-off-by: Dato Simó <dato@google.com>...
Hbal.hs: use Result (), not Bool, as return value for exec*
Previously, functions in Hbal.hs related to execution of jobsets werereturning only IO Bool, and printing any errors they found directly tostderr on their own.
I'm going to be moving some of these functions to a library module in...
Hbal.hs: rename functions that deal with early termination
In Hbal.hs, it's easy to get lost in the flow of exec* functionsbecause their names are similar and don't convey their differentpurposes (e.g. runJobSet, execJobSet, execWrapper).
This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'...
Hbal.hs: fix typo in signal handler names (hangle -> handle)
'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and'handleSigTerm', respectively.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're thetarget of a future move of some functions, so we leave them around, anddon't delete them, to avoid unnecessary delete/create diffs....
Makefile: Reset environment for epydoc
epydoc allows overriding configuration values via environment variables.While this might be useful in certain use cases, but as no prefixwhatsoever is used, conflicts are easily created. Some people have theenvironment variable “NAME” set, effectively overriding the project name...
jqueue: Don't modify input opcode when changing priority
Commit 4679547 implemented the ability to change job's priority after itwas submitted. The code contained a bug whereby it would modify theinput data for an opcode, something the job queue shouldn't do (logical...
Use new util function for mac_prefix validation
Use new NormalizeAndValidateThreeOctetMacPrefix() util function inLUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.Additionally, move the check in CheckArguments() in the case ofLUNetworkAdd....
LUClusterRedistConf: Use node allocation lock
All node locks are acquired.
LUClusterRepairDiskSizes: Use node allocation lock
This opcode acquires all node resource locks, which conflicts withinstance allocations.
LUGroupVerifyDisks: Use node allocation lock
See comment in code.