Add test for mutable default values in opcode parameters
This is not comprehensive, since in Python one can't determine what isand what is not mutable; but I've added a few base cases (list, dict,set).
The patch also improves (makes more uniform) the error messages in the...
Complete the instance OpCodes and parameters
Only the original instance opcodes (used in htools) are leftnon-converted to only parameter style; they'll be cleaned up later,once the htools codebase itself migrates to safer types.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add Group, OS and Backup opcodes
This also corrects a docstring in OpBackupExport on the Python side.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Complete the Tag opcodes and fix OpTestDelay missing param
The regexp in OpTagsSearch is loaded as is, without testing forvalidity; the rest of the patch is trivial.
Add a helper for the "iallocator" opcode field
This field is used with just changed description in about 10 opcodes,so unifying it makes things simpler for future potential changes tothe field type.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Rename OpTestAllocator.allocator to iallocator
This makes the OpCode more consistent with the other opcodes. Thedownside is incompatibility when upgrading from 2.6, but since this isa test opcode it shouldn't be problematic.
Fix empty list as default value in OpInstanceMultiAlloc
Commit 12e62af5 (“Adding the new opcode for multi-allocation”)introduced a "bad" default value; while porting this to Haskell, Irealised this is wrong.
Add types and parameters for OpInstanceCreate
This is a "big" opcode, so sending it separately.
A few types needed changing, and a few parameters were renamed to makeit more clear which are cluster-level and which are instance-levelparameters.
Move FileDriver from Objects to Types
Another base type that we need in opcodes as well.
Fix breakage introduced in commit a8b3b09
The order of the calls to “ctx.use_privatekey” and “ctx.use_certificate”was wrong, leading to an exception being thrown.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Factorize SSL context setup for certificate check
This code will also be used by the node daemon setup utility.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Introduce ht.TMaybeValueNone and ht.TValueNone
TValueNone checks if a value is "none" and TMaybeValueNone is a wrapperof TOr(TValueNone, x). This is used by OpNetworkSetParam in order toreset a network value (e.g. mac_prefix, gateway, etc.)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>...
opcodes: Replace manual loop with map
Also remove a superfluous empty line in test file.
Fix type descriptions in RAPI documentation
This patch adds descriptors to the “_CheckCIDR*” functions in opcodesand improves the descriptions generated by “ht.TInstanceOf”, therebyindirectly fixing bad type descriptions in the RAPI documentation.
Before this patch:...
Fixing crash when removing disks
This bug was introduced by my commit "Check ispecs against ipolicy oninstance modify", because I did not filter for the 'add' action. Thus itwould try to read the size parameter also when removing disks.
Signed-off-by: Helga Velroyen <helgav@google.com>...
locking: Change locking order, move NAL after instances
Some opcodes, for example LUInstanceFailover and LUInstanceMigrate,can't know whether they need to acquire all nodes until they have a lockon the instance. In turn they would have to acquire the node allocation...
Fix Attoparsec test coverage check compatibility
The hpc tool, on squeeze, does not handle utf8 encoded files correctly.The Attoparsec test is just needed to verify whether the parser handlesuft8 correctly, so the solution is to exclude it from coverage checks....
Add constant for node certificate mode
A new utility for configuring the node daemon will have to write thenode certificate as well. To not split information about the certificatefile even more, the constant is added to “pathutils”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Move cluster verification out of prepare-node-join
A new tool for configuring the node daemon will also have to verify thecluster name, so it's better to have this function in a central place.In the process of moving it to ssconf it is also changed to use...
ssconf: Verify file size when reading, add some tests
Until now ssconf would limit the amount read from files to 128 KiB andsilently ignored files larger than that. With this patch a check isadded by using fstat(2) on the file descriptor while it's being read....
Fix pylint error introduced in commit 9675661
Missing spaces around operator.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
ssconf: Small error message fixes
- Include filename in error message- s/ip/IP/- Stop using superfluous variable
Remove ssconf.SimpleConfigReader
Commit b0dcdc1 removed the last user of this class.
ssconf: Move class-level constants to module
There is no good reason why these two constants should be inside theclass. This patch moves them to the module so they can be used without“self”.
Fix network query for extra stats fields
Extend _GetNetworkStatsField() to accept 4 arguments.This fixes Issue 323.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Helga Velroyen <helgav@google.com>
Export UUID and serial no in network queries
Add uuid and serial_no to _NETWORK_SIMPLE_FIELDS in order to exportthem in network queries. Modify gnt-network to list them too.
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Michael Hanselmann <hansmi@google.com>
ssconf: Remove unused regular expression
RE_VALID_SSCONF_NAME is not used anywhere.
Fix build after commit 9b7e05a
Commit 9b7e05a didn't quite work as expected: glm.is_owned expects alocking level, not a lock name.
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
Factorize code to load and verify JSON
A new tool to configure the node daemon will also have to load andverify JSON data.
Small improvements to the DRBD parser and tests
Signed-off-by: Michele Tartara <mtartara@google.com>...
LUNodeQueryStorage: Use node allocation lock
Block instance allocations when all node locks will be acquired.
cmdlib: Don't pass processor to _ComputeNics
The execution context ID can be passed right away.
cmdlib.LUInstanceRename: Another assertion for BGL
Check whether the BGL is actually held instead of just testing“REQ_BGL”.
cmdlib: Stop using proc.Log*
The Log* functions are re-exported through the LU. Inline use of stringformatting (“%”) was converted to additional arguments to Log* (unlike“feedback_fn”, these functions support it). Some punctuation has beenremoved from messages, as well as some small re-wordings....
cmdlib: Use CommaJoin where appropriate
Don't use inline versions of CommaJoin.
LUNetwork*: Build dictionaries right away
… instead of constructing an object, only to fill it separately.
locking: Method to check if LockSet is fully acquired
A new method is added to check whether the LockSet-internal lock isheld. This is the case after LockSet.acquire was called withlocking.ALL_SET.
Unit tests are updated, including one where the list of names must be...
mcpu: Start locking at correct level
Commit 8716b1d added a new lock level, LEVEL_NODE_ALLOC. It is ahead ofLEVEL_INSTANCE. The latter was hardcoded in mcpu to be locked rightafter the BGL, effectively ignoring LEVEL_NODE_ALLOC.
Fix wrong edge case in a QuickChek Drbd property
The bug was in the test itself, not in the tested code.
Also, fixed a line longer than 80 characters in the same file.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add new lock level for node allocations
The new lock is similar to the BGL in the sense that it has its ownlevel and there is only one. It is called “node allocation lock”.Logical units will use it to synchronize with instance creations, whichin turn will start using opportunistic locks on nodes....
QuickCheck Tests for the commaIntParser, part of DRBD Parser
These new tests substitute the HTests, as by the suggestionreceived after submitting the previous patch.
Add gnt-network to .gitignore
Corrected network design doc regarding user interface
This patch fixes some inconsistencies betwen the networkdesign doc and the actual user interface. Adding reservedips works with option --add-reserved-ips and removing themwith --remove-reserved-ips. There is no toggling of the...
Fix network opcode parameters
Commit 32e3d8b1 (“opcodes: Network parameter improvements and fixes”)changed a few parameters in the network add, connect and set paramsopcodes, but some of the changes are buggy. The patch changed the typeto TMaybe(), whereas the default should have been changed from None to...
Introduce a TMaybe combinator
We have many cases in the code where we write TOr(TNone, a), so let'sintroduce a combinator that simplifies this case.
Beside replacing the above with TMaybe(a), I did a few other parameterfixes:
- noop change TOr(TNone, TDict) to TMaybeDict...
Replace dict() with {}
The network patches and an existing test added function-call baseddict construction as opposed to literal sintax.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Export network in FinalizeExport()
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix bug in OpNetworkQuery result check
Fix bug in network module
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>[iustin@google.com: slightly improved code flow]Reviewed-by: Iustin Pop <iustin@google.com>
Add design for simplified node-add process
Instead of initiating many SSH connections to copy files using “scp”, aJSON structure is passed to a program running on the node to be added.The design is similar to the one used for SSH setup.
Implement the correct handling of numbers without commas
commaInt now recognizes only the first 3 digits for numbers without commas.
It was erroneously recognizing numbers of any size before the first comma.
Make ipolicy violations a warning
So far, when running 'gnt-cluster verify' on a cluster whichhas instances that violate the instance policy, thoseviolations were shown as an error. This patch makes them awarning.
Small style fixes (' vs ") in network code
This is just trivial fixes; I think I caught all of them.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
A few more hlint fixes
I'm ignoring the use of Control.Arrow.*** since it's a non-obviousfunction, and we're not widely using Arrows in the code base, so Ithink switching to it would make the code less readable to mostpeople.
Fix OpCode arbitrary generation w.r.t. non-ASCII chars
Since we generate arbitrary strings and then we send them over a pipeto Python, we can (and will) generate non-ascii chars, which meanswhen run in a non-UTF8 locale, this will break with ghc 6.12 and...
Bulk add of remaining cluster/node opcodes
These are tested for equivalence with the Python code using existingtests. These tests are not perfect - for many "TDict" we simply sendempty dicts, as we don't have a way to generate proper data, but thetests did catch many trivial errors during writing of this patch...
Convert a few existing opcode parameters to safer types
This does a partial conversion of existing opcodes to the newnon-empty string type, which is needed so that we don't add even moreopcodes without this safety net (and to have consistent "target_node",...
Add many more opcode parameters
This is a bulk add of all node parameters needed for node and clusteropcodes. The parameters are defined with a few helper functions inthis module, and, opposite from opcodes.py, there won't be any (new)opcode attributes declared in-line (without a separate field...
Add two more basic types
These are needed for completing the node opcode parameters.
Add an arbitrary set generator helper
This makes it easy to generate sets of "things" that don't even needto have arbitrary instances, but which have bounded/enum instances.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Adeodato Simo <dato@google.com>
Move StorageType from RPC.hs to Types.hs
This fixes a FIXME actually recommending this move :)
Split 'Query.Language.ItemType' in two sub-types
The QR_VIA_OP/QR_VIA_LUXI types in Python are using yet anothervalidation mode: QR_VIA_OP is the base type, and QR_VIA_LUXI extendsit (when doing luxi queries). But on the wire they have the samerepresentation....
Move the 'Hypervisor' type from Objects to Types
This is a very basic type and 'Objects' is a heavy-weight module. Bymoving it to 'types' we simplify (in the future) the import chains.
Generalise the JSON "Container" type
Currently, we have some types that we kept as dictionaries in theobjects representation (due to inconsistent behaviour, or othertechnical reasons). This should be improved in the future, but in themeantime we can improve the Container type by allowing its keys to be...
Add more basic Ganeti types
These are needed both in opcodes and (potentially) outside, so we addthem in the main Types module.
Rename leftovers from remote to restricted commands
As per Iustin Pop's suggestion in <20121115131730.GX824@google.com> on<ganeti-devel@googlegroups.com>.
backend: Rename RunRemoteCommand to RunRestrictedCmd
Rename constants and directory for restricted commands
Rename configure option for restricted commands
It is no longer “--enable-remote-commands”, but rather“--enable-restricted-commands”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add command line interface for running commands remotely
This patch adds a new command, “gnt-node restricted-command”. Since thesemantics are different from “gnt-cluster command”, the same subcommandshouldn't be re-used.
The included man page update also includes a small description of how to...
Remove delayed iallocator call in TLReplaceDisks
By setting the “delay_iallocator” parameter one could delay the call tothe instance allocator. This was not used for some time now, but thecode was still there.
cmdlib: Merge duplicated code for TLMigrateInstance
LUInstanceFailover and LUInstanceMigrate use TLMigrateInstance and hadthe essentially same code for expanding names and declaring locks. InLUInstanceMigrate.ExpandNames there was a mistake which led to node...
Fix lint error about too many variables
Lint complains about too many variables in the CheckPrereqmethod. While I think that this method/class would needsome refactoring in general, at least this patch makes lintshut up for now.
Check ispecs against ipolicy on instance modify
When modifying an instance, so far the specs were not checked againstthe ipolicy. This patch fixes this issue.
Note that for backend parameters which have a minimum and a maximumvalue (currently only memory), it checks both limits against the...
locking.LockSet: Replace boolean parameter with constants
Upcoming changes will add opportunistic locking to “locking.LockSet”.Doing so will require additional code in “LockSet.__acquire_inner”, atwhich point the existing “want_all” parameter does not always apply....
RAPI: Clarify non-exhaustive nature of examples
The examples given in the RAPI documentation are not exhaustive. Thispatch tries to clarify this by adding ellipses where appropriate. Oneexample's indentation is also fixed.
Add tests for repr in locking classes
“locking.PipeCondition” and “locking.SharedLock” define “__repr__”,which until now was not tested at all.
test/*.py: s/'/"/
Now that 2.6 is essentially finished and 2.7 going to be branchedsoon-ish, I thought it would be a good moment to replace some singlequotes in test/*.py. Merge pains should be limited.
In one place in test/ganeti.locking_unittest.py, spaces are added for...
network: Style updates
- Quote characters- Docstrings
Document requirement for ipaddr and bitarray modules
Also update configure.ac.
ht: Complain if TNone isn't listed first for TOr/TAnd
Some type descriptions are rather long. If "None" is listed at the endor somewhere in between it is easily missed. Therefore it should be atthe beginning, e.g. "None or (long description)".
Add unit test for default parameter default values
Fails if the default value of an opcode parameter doesn't verify.
opcodes: Network parameter improvements and fixes
- Use variables for checks for IP addresses (makes them more consistent, too)- Fix checks to accept default values- Small description improvements (e.g. s/IP/IP address/)
Merge branch 'devel-2.6'
Merge branch 'stable-2.6' into devel-2.6
locking: Use frozenset instead of utils.UniqueSequence
In this case “frozenset” is good enough as the result's order doesn'tmatter--it is the input to “sorted” (“utils.UniqueSequence” preservesthe order). “frozenset” is ca. 25% faster for this use-case....
locking: Add test for downgrade without names
Until now there was no test for calling “LockSet.downgrade” withoutspecifying any names.
Fix some more hlint warnings
Sorry…
Fix tools/move-instance after networks addition
Commits d4117a72 (“Modify LUInstanceCreate to support networks”) and09a3d21f (“Pass detailed network info in hooks“) expanded the nicinformation as computed by _NICToTuple, which means that the format ofthe result value from LUInstanceQueryData has changed, and...
Move _BuildNetworkEnv() at top-level
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
Use bitarray 0.8 interface
Minor fix in query.py
Minor fix in ovf
Fix a few docstrings
Simplify GenerateFree in network module
GenerateFree now returns the first available IP in the networkor raises AddressPoolError if it is full.
Replace string values with proper constants
_UnlockedCommitIp is used either for releasing or reserving anIP inside a network. New constants RELEASE_ACTION/RESERVE_ACTIONare used to decide which is the case.
Add docstring to the GetECReserved config method
Simplify mac generation funtions
Introduce new config method _UnlockedGetNetworkMACPrefix and refactororiginal _GenerateOneMAC so that it takes prefix as an optionalargument and returns a function that actually generates a MAC. Ifprefix is not given or None it uses the cluster level default....
Fix _ComputeNics to include network slot
This bug was introduced after rebasing master branch.