Switch luxi submit job calls to use MetaOpCode
This patch changes the luxi submit job calls to use wrapped opcodes,and therefore it changes Hbal to submit actual meta opcodes. Fornicety, hbal also submits a comment now, showing who generated thejob.
Signed-off-by: Iustin Pop <iustin@google.com>...
hbal: restore nice job ID display
This fixes:
Got job IDs JobId {fromJobId = 1052613}
And restores to:
Got job IDs 1052624
Other improvements could be done here, of course.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add a negative type
This mirrors the positive one, and will be needed for relative jobIDs.
Add types and parameters for common opcode implementation
This will go into a separate type; this patch adds the neededunderlying types and parameters.
Add a type alias for simpler THH signatures
This is reused in more than just a few places, so adding it makes thesignatures much nicer.
Change opcode/luxi showJSON generation in THH
Currently, the opcode and luxi "showJSON" functions generate directlya JSValue; in contrast, the object (single-constructor) types generatea 'toDict' function, and then `showJSON = makeObj . toDict`. This is...
Remove unused THH function 'saveLuxiField'
This was deprecated via commit 88609f00, “Switch Luxi TH code fromsimple to custom fields”.
Add CommonOpParams and MetaOpCode types
This patch adds the "meta" opcode type and the common opparams. Compatibility tests with Python are changed to pass Metaopcodes.
Add safety check on job dependency/TIsLength
If TIsLength is applied to a non-container item, it will fail (typeerror) due to invalid application of len(). Since this can happen onuser-supplied data, we add an explicit TList/TTuple check (the TTupletest is a new one)....
Small Makefile.am cleanups
Indentation within rules is fixed to always be 2 spaces (as opposed tomixed tabs/2 spaces/4 spaces). Additionally, the check-dirs rule,which was more complex than necessary, has been simplified: movedauto4mte.cache to DIRCHECK_EXCLUDE, and simplify the find call and the...
Fix long lines in Ganeti.HTools.Types and add Make check
Since hlint doesn't check style issues but rather code only issues,let's add a simple Makefile check for too-long lines, and fix thesingle bad case we have currently.
Two small haskell-related Makefile improvements
Just some cleanup:
- don't run hlint over hpc-htools, since it's just a symlink to htools.hs (so we'd get duplicate warnings)- build rpc-test in make really-all, via HS_ALL_PROGS variable
Rework custom fields handling
This patch changes a bit the handling of custom fields. Since ingeneral we use custom fields to aggregate multiple entries in the JSONobject into a safer data-type, we should also have a way to declarewhich extra entries this field covers (so that in the future we can...
Add a function for all fields of a given OP_ID
This patch changes THH to export a new function which defines allfields of a given OP_ID. Not very clean, since for an invalid OP_ID wereturn empty list, but since it will only be used in tests it shouldbe good enough....
Rework/enhance original htools opcodes
The original htools opcodes were minimalistic and not 1:1 equivalentwith the Python ones. Let's add all missing fields and, since wechanged the order, switch to more readable record syntax for buildingthe opcodes....
Add a few missing fields from opcodes
Due to manual conversion, a few fields were missing from theconversion, but as they were optional our type equivalence checkingdidn't detect this.
Fix Haskell OpNetworkAdd
Commit 213076f (“Fix locking in networks”) changed Python OpNetworkAddwithout corresponding Haskell definition changes.
Add test for opcode fields equivalence
As opposed to the existing test, which tests the type/serialisation offields, this one simply tests the equivalence of the list of fieldsfor each opcode.
Add type for finalised job statuses
For now, we don't need a pending job status type as well, so we'lldelay adding that until later.
Add a 'real' type for JobIds
Currently, the job ID is a simple type alias. This is suboptimal, asit means we can't use a custom JSON (or Arbitrary) instance for it.
The patch changes it into a newtype, and then a) simplifies somedeserialisation code and b) changes some more fields to this new type...
Fix generation of clusters via 'genEmptyCluster'
The current uniqueness algorithm (generate random node names, suffixthem with node index) is actually wrong: a node named "21" at index 5will end up with the same name as a node named "2" at position 15....
LUClusterSetParams: Use node allocation lock
All resources are acquired and opportunistic instance creations wouldfail. Also add a TODO.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
LUInstanceCreate: Acquire node allocation lock
Opportunistic locks are not yet used. This patch changesLUInstanceCreate to acquire the node allocation lock to avoid conflictswith other opcodes acquiring many node locks.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Acquire node allocation lock for failover/migration
See code for an explanatory comment. The lock is released as soon aspossible.
Use GetMultiInstanceInfo in LUNetwork* opcodes
LUNetworkConnect/Disconnect looks up a nodegroup's instances forconflicting IPs. To do so, use GetNodeGroupInstances() andGetMulitInstanceInfo().
Additionally, check if the correct locks were acquired.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Improve the Haskell api doc directory creation system
Instead of having two different lists of directories, now there is only one.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the nodedaemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” areverified, too.
Support opportunistic locks in mcpu/LUs
Similar to “share_locks”, a new dictionary containing booleans for eachlocking level is added to “cmdlib.LogicalUnit”. Logical units wanting tomake use of opportunistic locks will be able to configure thisdictionary accordingly....
Add opportunistic locking to GanetiLockManager
Just forwarding the parameter, nothing more.
locking: Implement opportunistic locking in LockSet
This patch adds a new parameter to “LockSet.acquire” named“opportunistic”. When enabled the lockset will try to acquire as manylocks as possible, but it won't wait for them (with the exception of thelockset-internal lock in case the whole set is acquired). This is...
Add ssconf function to read all files
Configuring a node daemon on a newly added node will need all ssconfvalues.
bootstrap.RunNodeSetupCmd: Add IPv6 support
Commit 224ff0f modified the node SSH setup to use the system's SSHclient. Before that Paramiko was used. It's not entirely clear whehterthe latter ever supported IPv6 properly, but with this patch“bootstrap.RunNodeSetupCmd” is changed to use it if configured. The code...
Factorize running node setup command
Part of the code used for running “prepare-node-join” can be re-used forrunning a tool to configure the node daemon.
ssconf: Add dry-run support for writing files
A new utility for configuring the node daemon will support a dry-runmode. This patch adds the necessary functionality to“ssconf.SimpleStore” and provides comprehensive tests for“SimpleStore.WriteFiles”. To enable the latter, a testing-only parameter...
ssconf: Add function to verify keys
The new utility for configuring the node daemon will have to checkwhether it received valid ssconf names.
Design for opportunistic locking
Add the design document for opportunistic locking.
Fix variable names in Loader.assignIndices
This function can be applied to any type of class "Element" but itsinternal variables used to refer to a list of tuples containing theseelements and their names as "nodes". Moreover it built a list of name toindices tuples referring to it as "na" (which is not very clear)....
LUNetworkAdd: Log warning when needed
In case conflicts are checked, log warnings if nodes' IPs cannotbe reserved.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Use constants.IP4_VERSION in LUNetworkAdd
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix locking in networks
Ensure that locks are held only if needed.
Add conflicts_check in OpNetworkAdd. This is needed if we want tocheck whether nodes/master IPs are included in network.
Depending on conflicts_check value, we have to hold node/instance locks...
Add two support functions for building JSON objects
The new support functions in module Ganeti.JSON will be used by the nextcommits to build JSON objects with optional (Maybe) fields, excludingthem when they contain Nothing.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Add JSON serialization capabilities for DRBD status
The serialized JSON is not a 1:1 dump of the data structurespopulated by the parser. This is done intentionally, with theaim of producing a more stable and more meaningful output tobe used by the (future) monitoring agent and stand-alone data...
Add unit tests for the JSON serialization of DRBD status
The serialization itself is done by Text.JSON, so the tests deal with checkingthat Text.JSON objects are created correctly from the DRBD parser datastructures.
Reduce hostname length in tests and opcode py_compat count
This improves Issue 325 - new runtime and memory consumption is about1/10 compared to before.
Add generator for an instance on a node list
Given a node list in input, we get an instance that had nodes in it.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add generator for list of arbitrary instances
This takes an instance generator and produces a possibly empty list ofinstances.
Add generator for list of arbitrary nodes
This generates a minimum of one node, because legal clusters never havezero nodes.
Fix options in hbal manpage
hbal doesn't have any "-o" option, but on the other hand supports -S(described below, but not present in the short summary)
Change -G option description
The new one is more consistent with the rest of the nodegroupterminology and more generic for use in different htools.
Create a genInstanceMaybeBiggerThan function
This is used only once when testing Cluster.hs, but having it abstractedclarifies there what that call is about, makes that test shorter, andallows us to better do refactoring of the main genInstanceSmallerThan...
Make Query operators enforce strictness
Currently, the query operators (binop, etc.) create thunks, instead offorcing the evaluation of the simple boolean results. This results inhigher than needed memory use.
Optimise recursive Query filters
Currently, the And and Or filters use very nice code, e.g. in case ofOrFilter:
any id <$> mapM evaluateFilter flts
However, looking at the memory profiles shows that application ofany/id to monadic values via '<$>' does not work nicely, losing the...
Fix Haskell profiling targets
It turns out that in newer GHC versions, the suffix shouldn't beprefixed with a dot; otherwise name mangling is broken.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Fix htools/rpc-test after recent changes
Since this program is not built normally, a few recent commits brokeit:
- 707cd3d (“Use exitErr instead of explicit error message and exitWith”) introduced wrong syntax (missing '$')- 29a3053 (“Remove unsafePerformIO usage”) changed the signature of...
Add two last missing opcodes
OpInstanceQuery was missing accidentally, whereas OpRestrictedCommandwas just recently added without Haskell definitions.
The patch also slightly improves the OpNodeQuery arbitrary generation.
Enable equivalence checks of opcode list
This patch enhances the opcode list checks - instead of spawning aPython interpreter to display the opcode list, we export it staticallyin Constants.hs via a slight convert-constants change.
Furthermore, since we now have opcode parity, we enable full opcode...
Make QuickCheck generator names more uniform
Due to lack of attention, we have two styles for generators ofarbitrary values: get* and gen* (e.g. getFQDN and genDiskIndices). Inorder to make this more obvious that we deal with a function in theGen monad, let's rename all get* functions to gen*....
Abstract query common opcode definitions
It would be even better if the opcodes would actually have all thesame definitions, until then we have two sets of definitions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Remove remaining in-line parameters
… except one, and replace them with separately-defined ones inOpParams.
Replace hand-coded 'live' field of OpInstanceMigrate
This is the last inline-written field; I've done it separately fromthe last patch due to changes being needed in HTools (field changedtype).
Remove read instances from our Haskell code
It turns out that optimising 'read' derived instances (via -O) forcomplex data types (like OpCode, or the various objects) can be slowto very slow. Disabling such instances results in (time make$all_our_haskell_binaries) large compile-time savings and also smaller...
Add explicit test for wrong OpCode arbitrary instances
I've been bitten a couple of times with arbitrary opcodes working onUTF-8 locale, but failing on buildbot (ASCII). So let's add anexplicit test that checks always (even with UTF-8) for correctarbitrary values, showing explicitly which opcodes fail....
Add test for mutable default values in opcode parameters
This is not comprehensive, since in Python one can't determine what isand what is not mutable; but I've added a few base cases (list, dict,set).
The patch also improves (makes more uniform) the error messages in the...
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Add a helper for the "iallocator" opcode field
This field is used with just changed description in about 10 opcodes,so unifying it makes things simpler for future potential changes tothe field type.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Rename OpTestAllocator.allocator to iallocator
This makes the OpCode more consistent with the other opcodes. Thedownside is incompatibility when upgrading from 2.6, but since this isa test opcode it shouldn't be problematic.
Complete the Test opcodes
This adds the OpTestAllocator, OpTestJqueue and OpTestDummyopcodes. The OpTestAllocator seems to need some cleanup (on the Pythonside), for now we implement it as is. As for the other two, while notused in production, we should have full coverage for them as well....
Move NICMode from Objects.hs to Types.hs
Also add some unittests for this type.
Add types, parameters and the opcodes for networks
This completes the last missing opcode group. The only difficulty waswith the ip addresses, where we used simple strings to represent themand (for IPv4) a few helpers to generate arbitrary instances;otherwise, the patch is trivial....
Fix empty list as default value in OpInstanceMultiAlloc
Commit 12e62af5 (“Adding the new opcode for multi-allocation”)introduced a "bad" default value; while porting this to Haskell, Irealised this is wrong.
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Fix breakage introduced in commit a8b3b09
The order of the calls to “ctx.use_privatekey” and “ctx.use_certificate”was wrong, leading to an exception being thrown.
Factorize SSL context setup for certificate check
This code will also be used by the node daemon setup utility.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Introduce ht.TMaybeValueNone and ht.TValueNone
TValueNone checks if a value is "none" and TMaybeValueNone is a wrapperof TOr(TValueNone, x). This is used by OpNetworkSetParam in order toreset a network value (e.g. mac_prefix, gateway, etc.)
opcodes: Replace manual loop with map
Also remove a superfluous empty line in test file.
Fix type descriptions in RAPI documentation
This patch adds descriptors to the “_CheckCIDR*” functions in opcodesand improves the descriptions generated by “ht.TInstanceOf”, therebyindirectly fixing bad type descriptions in the RAPI documentation.
Before this patch:...
Fixing crash when removing disks
This bug was introduced by my commit "Check ispecs against ipolicy oninstance modify", because I did not filter for the 'add' action. Thus itwould try to read the size parameter also when removing disks.
Signed-off-by: Helga Velroyen <helgav@google.com>...
locking: Change locking order, move NAL after instances
Some opcodes, for example LUInstanceFailover and LUInstanceMigrate,can't know whether they need to acquire all nodes until they have a lockon the instance. In turn they would have to acquire the node allocation...
Fix Attoparsec test coverage check compatibility
The hpc tool, on squeeze, does not handle utf8 encoded files correctly.The Attoparsec test is just needed to verify whether the parser handlesuft8 correctly, so the solution is to exclude it from coverage checks....
Add constant for node certificate mode
A new utility for configuring the node daemon will have to write thenode certificate as well. To not split information about the certificatefile even more, the constant is added to “pathutils”.
Move cluster verification out of prepare-node-join
A new tool for configuring the node daemon will also have to verify thecluster name, so it's better to have this function in a central place.In the process of moving it to ssconf it is also changed to use...
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Fix pylint error introduced in commit 9675661
Missing spaces around operator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
ssconf: Small error message fixes
- Include filename in error message- s/ip/IP/- Stop using superfluous variable
Remove ssconf.SimpleConfigReader
Commit b0dcdc1 removed the last user of this class.
ssconf: Move class-level constants to module
There is no good reason why these two constants should be inside theclass. This patch moves them to the module so they can be used without“self”.
Fix network query for extra stats fields
Extend _GetNetworkStatsField() to accept 4 arguments.This fixes Issue 323.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Export UUID and serial no in network queries
Add uuid and serial_no to _NETWORK_SIMPLE_FIELDS in order to exportthem in network queries. Modify gnt-network to list them too.
ssconf: Remove unused regular expression
RE_VALID_SSCONF_NAME is not used anywhere.
Fix build after commit 9b7e05a
Commit 9b7e05a didn't quite work as expected: glm.is_owned expects alocking level, not a lock name.
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
Factorize code to load and verify JSON
A new tool to configure the node daemon will also have to load andverify JSON data.
Small improvements to the DRBD parser and tests
LUNodeQueryStorage: Use node allocation lock
Block instance allocations when all node locks will be acquired.
cmdlib: Don't pass processor to _ComputeNics
The execution context ID can be passed right away.
cmdlib.LUInstanceRename: Another assertion for BGL
Check whether the BGL is actually held instead of just testing“REQ_BGL”.