Move the path of the DRBD status file to the Constants file
It will be needed by the DRBD data collector, that will be added shortly.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Make configure check for haskell vector library
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Remove usage of reserved key word 'family'
An old hlint version (1.8.28) was complaining about theword 'family' being used in the network tests. Thanks toan epiphany of Iustin this was due to that it is a reservedkey word.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Add documentation about new dependency to vector lib
Updates the INSTALL file to add the haskell librarylibghc-vector-dev.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Network and address pool (again)
Implementation of the network objects and address pool. Functionalityas in the reverted commit b9a616e132af9bba718d2da8c94eeec7af886814, butnow using only the vector library and not the bit-vector library.Tested with vector library version 0.9 and 0.10., which are...
Fix ordering of entries in tools.ensure_dirs
Commit ebd437a added two new entries to tools.ensure_dirs, but did so inthe wrong order. Patch forthcoming to improve the unittest's errormessage.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Set owner on watcher pause and queue drain files
If the files were created by a user different, e.g. due to a switch fromrunning masterd as root to running it as a dedicated user, they couldn'tbe modified/removed anymore.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
lib/tools/ensure_dirs.py: Code formatting
Wrap lines in a consistent manner (uid/gid on the second line) ifwrapping is necessary at all. “git diff --color-words” shows nodifference at all as only whitespace changed.
Replicate queue drain flag across all master candidates
Until now, the flag was unset on a master failover unless the“$localstatedir/lib/ganeti/queue/drain” file existed.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add utility function to create frozenset with unique values
When used instead of a plain call to “frozenset”, this would haveavoided the issue fixed in commit e2dd6ec. The new function is locatedin the “compat” module as it will be used at module load time in most...
constants: Remove duplicate DRBD barrier option
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
doc/rapi.rst: Add ECODE_TEMP_NORES
The assertion was failing as ECODE_TEMP_NORES was not documented.
Change value for ECODE_TEMP_NORES
Unfortunately there was a bug in commit 518a45e whereby ECODE_TEMP_NOREShad the same value as ECODE_NORES, leading to failures in a Haskelltest. Of course this would also have affected other users of the constant.
Make htools/Ganeti/Constants.hs depend on errors.py
It also uses values from “lib/errors.py”.
masterd: Remove duplicate code
Improve error message for when adding inotify watcher fails
Explicitely mention the fs.inotify.max_user_watches sysctl value.
htools/Ganeti/Errors: Add ECodeTempNoRes
Update pyinotify website
It has moved to github.com.
htest/Types.hs: check ordering of some ADTs
In some ADT, ordering of constructors is important because a certainmeaning is attached to ordinality (e.g., "clusters with lesser AllocPolicyare preferred").
To ensure constructor order is not accidentally changed, this commit...
htools: Add missing parameter to OpInstanceMultiAlloc
Somehow this went missing in commit 1f1188c.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add error code for temporary lack of resources
When an instance creation uses opportunistic locks, the iallocator mightnot be able to find an allocation solution if not enough node locks (ora suboptimal subset thereof) were acquired. As per the design document...
Export error codes from RAPI client module
Until now the error codes were not available from the RAPI clientmodule. A newly added unit test ensures all error codes are contained in“ECODE_ALL”, as well as ensuring consistency between the RAPI client and...
cmdlib: Use locked nodes as node whitelist
Also actually start using opportunistic locks (if requested).
cmdlib: Opportunistic locking on instance creation
Adds a new parameter to “OpInstanceCreate” and “OpInstanceMultiAlloc” touse opportunistic locks.
cmdlib: Node whitelist support for allocation request
Forward the node whitelist to the iallocator plugin.
mcpu: Verify node allocation lock mode
Add verification code to mcpu to check an LU's locks. Two whitelists areprovided to exclude LUs from the two tests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
_ExportQuery: Use node allocation lock
Block instance allocations when all node locks will be acquired.
LUBackupExport: Use node allocation lock
LUBackupRemove: Use node allocation lock
LUInstanceMultiAlloc: Use node allocation lock
Avoid conflicts between instance allocations.
LUInstanceRecreateDisks: Use node allocation lock
LUNodeSetParams: Use node allocation lock
LUNodeQueryvols: Use node allocation lock
LUOobCommand: Use node allocation lock
If no node names are given, all node locks are acquired.
Fix typo in comment
There was a typo in the docstring of the parseMessage function of theconfd server.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "Network and address pool"
This reverts commit b9a616e132af9bba718d2da8c94eeec7af886814, whichdepends on the "bitvec" library. We need to investigate a bit further thatdependency, as it in turns depends on "vector" 0.9.1 or below, since0.10.* removed support for Data.Vector.Unboxed.Safe which it uses....
Network and address pool
Implementation of the network and address pool class inHaskell. Not complete yet. Includes unit tests that coverall functionality that is so far implemented.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix node-daemon-setup test with older pyOpenSSL
Older versions use “-----BEGIN RSA PRIVATE KEY-----” instead of“-----BEGIN PRIVATE KEY-----”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add tool to clean up node
Sometimes a node is not removed properly from a cluster (especiallyduring development). This new tool stops all daemons and removes (aftermaking copies) the most critical files.
Add tool to configure node daemon
The design for this is in “doc/design-node-add.rst”. The tool receives aJSON data structure on stdin and configures the node's daemon afterverifying the received values.
Switch from scp/ssh to node daemon setup utility
This patch does away with many calls to scp and by means of using“tools/node-daemon-setup”, verifies most of the values before writingthem to files.
make genMaybe more Just
There is a common conception that Just something is more worth thanNothing. So we're biasing our tests towards that. As such let's generateNothing fewer times, and Just subgen more times. The values were copiedfrom the "official" maybe generator....
Locking related fixes for networks
Use GetNetwork() only when having already aquired the lock,i.e. in CheckPrereq().
In LUNetworkConnect/Disconnect do not include Network info in Hooksenvironment, so that network locking can be avoided if conflictsare not checked....
Fix build breakage in Jobs.hs test code
Commit 3bdbe4b3 (“Jobs.hs: move OpStatus and JobStatus ADTs toTypes.hs”) removed the TemplateHaskell language pragma fromhtest/Test/Ganeti/Jobs.hs due to a hlint warning, but that is bad: itmeans the testSuite call is no longer interpreted as a splice, so it results in:...
configure.ac: fix reading of HLINT environment variable
configure.ac advertised that the HLINT environment variable could be set tospecify the path to the 'hlint' tool. However, HLINT was being initializedto "no" inside configure.ac, which broke this usage....
Hbal.hs: move job execution functions to Jobs.hs
Ganeti.Jobs now holds functions that can be used to submit and monitor thestatus of jobs. In particular, execJobsWait and waitForJobs are factoredout of Hbal.hs.
Signed-off-by: Dato Simó <dato@google.com>...
Hbal.hs: use Result (), not Bool, as return value for exec*
Previously, functions in Hbal.hs related to execution of jobsets werereturning only IO Bool, and printing any errors they found directly tostderr on their own.
I'm going to be moving some of these functions to a library module in...
Hbal.hs: rename functions that deal with early termination
In Hbal.hs, it's easy to get lost in the flow of exec* functionsbecause their names are similar and don't convey their differentpurposes (e.g. runJobSet, execJobSet, execWrapper).
This patch renames 'runJobSet' to 'execWithCancel', and 'execWrapper'...
Hbal.hs: fix typo in signal handler names (hangle -> handle)
'hangleSigInt' and 'hangleSigTerm' are renamed to 'handleSigInt' and'handleSigTerm', respectively.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Jobs.hs: move OpStatus and JobStatus ADTs to Types.hs
This leaves Ganeti/Jobs.hs and Test/Ganeti/Jobs.hs empty, but they're thetarget of a future move of some functions, so we leave them around, anddon't delete them, to avoid unnecessary delete/create diffs....
Makefile: Reset environment for epydoc
epydoc allows overriding configuration values via environment variables.While this might be useful in certain use cases, but as no prefixwhatsoever is used, conflicts are easily created. Some people have theenvironment variable “NAME” set, effectively overriding the project name...
jqueue: Don't modify input opcode when changing priority
Commit 4679547 implemented the ability to change job's priority after itwas submitted. The code contained a bug whereby it would modify theinput data for an opcode, something the job queue shouldn't do (logical...
Use new util function for mac_prefix validation
Use new NormalizeAndValidateThreeOctetMacPrefix() util function inLUNetworkAdd/LUNetworkSetParams to validate network's MAC prefix.Additionally, move the check in CheckArguments() in the case ofLUNetworkAdd....
LUClusterRedistConf: Use node allocation lock
All node locks are acquired.
LUClusterRepairDiskSizes: Use node allocation lock
This opcode acquires all node resource locks, which conflicts withinstance allocations.
LUGroupVerifyDisks: Use node allocation lock
See comment in code.
LUClusterVerifyGroup: Use node allocation lock
LUInstanceReplaceDisks: Acquire node allocation lock
If the lock was acquired in the first place (only when an iallocator isused), it is released as soon as possible.
LUInstanceChangeGroup: Acquire node allocation lock
Changing instances' groups shouldn't conflict with instance allocations.
Acquire node allocation lock during node query
If locking is used (usually by ganeti-watcher), node allocations must betemporarily blocked.
Mention dsh/cssh in man page for "gnt-cluster command"
Until now they are only mentioned in “doc/admin.rst” and the man pagefor “ganeti-listrunner”.
iallocator: Add node whitelist
In the future instance creations might have a lock on all nodes as wasthe case until the implementation of opportunistic locking. Nodes forwhich the lock is not held will be shown to the iallocator plugin as ifthey were marked offline....
Allow ignoring successful commands in "gnt-cluster command"
In some cases it is useful to ignore the output of and avoid mentioningsuccessful commands. One would be when looking for a certain string ina file:
$ gnt-cluster command egrep -q '^testing$' /etc/......
Add Ganeti.HTools.Graph
This module implements some algorithms on Data.Graph data structures.At the moment its main functionality is an LF-color implementation(greedy coloring in descending order of degree). There are also a fewextra functions to calculate the degree order, and convert the node to...
Add Dsatur implementation
Implement the Dsatur algorithm for Graph coloring. This also abstractsthe neighColors function into two subfunctions that this algorithm canreuse.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Ganeti/HTools/Graph Add isColorable
Check whether coloring on a given graph makes sense. This is the caseonly if there are no loops and the graph is undirected.
Add tests for verticesByDegree{Asc,Desc}
This brings our coverage of Graph.hs to 100%
HTools/Node: add mkNodeGraph function
This function helps treating node node problems as graph problems.It can transform a list of nodes plus a list of instances into a graphwhich uses the nodes as vertices, and instances as edges connecting them(as long as they have both a primary and a secondary node)...
Fix Dsatur and add Dcolor
Our Dsatur implementation was incorrect: while the paper defined thedegree of saturation of a vertex as the number of different colors it isadjacent to, we were using the number of colors, without consideringuniqueness. This effectively implemented a different algorithm, which is...
Add "proper coloring" unittest check
We have to check that for each edge its vertices have different colors.
This is very easy to do with a vertex-to-color map, but not so easy witha color-to-vertex one. Since all our coloring algorithms created avertex-to-color map behind the scenes and then converted it, we flip...
errors: Show error descriptions in API documentation
Comments with a colon after the hash sign (“#:”) show up in theepydoc output.
Small improvement to rpc-test
This will show smaller/better error messages: full node dumps are nolonger included.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix locking mistake introduced in commit 5cc1f88
The node resource locks were not set correctly on instance import.
Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
Signed-off-by: Iustin Pop <iustin@google.com>...
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes thesignatures much nicer.
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directlya JSValue; in contrast, the object (single-constructor) types generatea 'toDict' function, and then `showJSON = makeObj . toDict`. This is...
Remove unused THH function 'saveLuxiField'
This was deprecated via commit 88609f00, “Switch Luxi TH code fromsimple to custom fields”.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Add safety check on job dependency/TIsLength
If TIsLength is applied to a non-container item, it will fail (typeerror) due to invalid application of len(). Since this can happen onuser-supplied data, we add an explicit TList/TTuple check (the TTupletest is a new one)....
Small Makefile.am cleanups
Indentation within rules is fixed to always be 2 spaces (as opposed tomixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,which was more complex than necessary, has been simplified: movedauto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the...
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Two small haskell-related Makefile improvements
Just some cleanup:
- don't run hlint over hpc-htools, since it's just a symlink to htools.hs (so we'd get duplicate warnings)- build rpc-test in make really-all, via HS_ALL_PROGS variable
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since ingeneral we use custom fields to aggregate multiple entries in the JSONobject into a safer data-type, we should also have a way to declarewhich extra entries this field covers (so that in the future we can...
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines allfields of a given OP_ID. Not very clean, since for an invalid OP_ID wereturn empty list, but since it will only be used in tests it shouldbe good enough....
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add test for opcode fields equivalence
As opposed to the existing test, which tests the type/serialisation offields, this one simply tests the equivalence of the list of fieldsfor each opcode.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix generation of clusters via 'genEmptyCluster'
The current uniqueness algorithm (generate random node names, suffixthem with node index) is actually wrong: a node named "21" at index 5will end up with the same name as a node named "2" at position 15....
LUClusterSetParams: Use node allocation lock
All resources are acquired and opportunistic instance creations wouldfail. Also add a TODO.
LUInstanceCreate: Acquire node allocation lock
Opportunistic locks are not yet used. This patch changesLUInstanceCreate to acquire the node allocation lock to avoid conflictswith other opcodes acquiring many node locks.
Acquire node allocation lock for failover/migration
See code for an explanatory comment. The lock is released as soon aspossible.
Use GetMultiInstanceInfo in LUNetwork* opcodes
LUNetworkConnect/Disconnect looks up a nodegroup's instances forconflicting IPs. To do so, use GetNodeGroupInstances() andGetMulitInstanceInfo().
Additionally, check if the correct locks were acquired.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Improve the Haskell api doc directory creation system
Instead of having two different lists of directories, now there is only one.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>