Don't allow optional node parameters
Ganeti does not support optional fields in parameters(hypervisor-params, disk-params, etc.). OpenVSwitch related nodeparameters were the exception to this rule, which caused numerousproblems related to import/export and (de-)serialization....
Show the key in "'None' is not a valid Maybe value"
Currently the error message doesn't say which key is affected, whichmakes it kind of useless.
Signed-off-by: Petr Pudlak <pudlak@google.com>Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Streamlining handling of spindles and default templates
This rather lenghy patch comprises a couple of refactoringsto achieve the following goals:- 'gnt-node info' should only report space information about spindles, when exclusive storage is enabled on the node...
Use 'DTS_LVM' when possible
This patch replaces all usages of the utility function'GetLvmDiskTemplate' by the new 'DTS_LVM' constantto make it consistant with the usage of other DTS_*constants.
Additionally, it provides a unit tests to ensureconsistancy between DTS_LVM and the mapping of disk...
utils/storage.py: storage info lookup by disk template
This patch improves the handling of storage informationbefore and after a call to the RPC 'node_info'. Itadds a function to not only call for all storageinformation on the cluster (as it is used right now),...
Provide utility function to check a configuration version
In order for 'gnt-cluster upgrade --resume' to determine whether the configurationis already upgraded, it needs to compare whether a configuration version is compatiblewith a Ganeti target version. Provide a utility function for this....
Fix typo in doc string
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Provide an inverse to UnescapeAndSplit
With utils.UnescapeAndSplit, we have a function to parsearbitrary non-empty string lists encoded as strings. Alsoprovide the appropriate encoding function.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Add predicate whether to call cfgupgrade --downgrade
Provide a predicate that, given the current version and theversion to go to, tells whether it is appropriate to callcfgupgrade --downgrade.
Signed-off-by: Klaus Aehlig <aehlig@google.com>Reviewed-by: Jose Lopes <jabolopes@google.com>
Add utility function to recognize upgrade ranges
Upgrading is possible within the same major version to any equalor higher minor version. Downgrading is possible within the samemajor version to the previous minor version. Moreover, automaticupgrades are only supported from version 2.10 onwards. Add a utility...
Add utility function to parse version strings
The new 'gnt-cluster upgrade' command will get a Ganeti version asargument. So provide a function able to parse it.
Merge branch 'stable-2.9' into master
Merge branch 'stable-2.8' into stable-2.9
Signed-off-by: Klaus Aehlig <aehlig@google.com>...
Move 'BuildVersion' to 'lib/utils/version.py'
Functions 'BuildVersion' and 'SplitVersion' are no longer needed bythe constants and, given that they are not constants, they should bemoved elsewhere. Since they are only used by 'cfgupgrade' and tests,these functions are moved to 'lib/utils/version.py' and references to...
Fix corner-case in handling of remaining retry time
Consider a remaining time of 0 as already timed out. Otherwise,there is no guarantee that calling utils.Retry with timeout 0will call the function precisely once; it might run in timeshorter than the resolution of timer....
Fix strings invalid with newer lint versions
Generating ASCII characters via a supported but not official escape sequenceleads to a "Anomalous backslash in string" error in newer pylint versions. Thispatch fixes the issue.
Signed-off-by: Michele Tartara <mtartara@google.com>...
Lint improvements to regexps
Fix some regular expressions so that they pass lint checks with newer versionsof pylint.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Merge branch 'stable-2.9'
gnt-cluster modify --shared-file-storage-dir
This patch introduces to 'gnt-cluster modify' the option'--shared-file-storage-dir' to change the default directoryfor instances using shared file storage at cluster runtime.
Signed-off-by: Helga Velroyen <helgav@google.com>...
Remove obsolete autoconf variable from remaining files
This removes the obsolete autoconf variable'ENABLE_SHARED_FILE_STORAGE' from all remaining files.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Thomas Thrainer <thomasth@google.com>
Fix usage of mapping from disk templates to storage types
This fixes the currently broken tests due to a slippedoccurrence of the mapping from disk templates to storagetypes due to a merge.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Rename VALID_STORAGE_TYPES to STORAGE_TYPES
For consistency, this patch renames 'VALID_STORAGE_TYPES'to 'STORAGE_TYPES', because the set of valid disktemplates is also just names 'DISK_TEMPLATES'.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Rename disk_template/storage_type map + cleanup
This patch renames the mapping of disk templates to storagetypes to MAP_DISK_TEMPLATE_STORAGE_TYPE, which is a moreexpressive name since it indicates that it is a mapping anddoes not have the inconsistent usage of singular and plural...
Move space reporting constant to constants.py
This patch moves the constant which is used to determinewhether a storage type provides storage space reportingfrom the utils package to the constants. This way, wecan also use it in haskell and it fits there semantically...
Warn instead of crashing on preexisting wrong data type
Using None in a VTYPE_MAYBE_STRING is wrong, because it cannot reallybe serialized to a Haskel Nothing, but given that this can (and does)influence external components already using None, it is turned to a...
Support proper encoding of Nothing in constants file
The Nothing value of a Haskell Maybe needs to be properly encoded.If a Python "None" is used, it is not going to be handled properly when theresulting JSON is decoded by Haskell.
This patch adds support for proper "Nothing" encoding....
Merge branch 'stable-2.8' into master
storage utils: read file storage dir from cluster config
This patch make the storage utils read the file storagedirectory from the cluster config instead of the constants.
Signed-off-by: Helga Velroyen <helgav@google.com>Reviewed-by: Klaus Aehlig <aehlig@google.com>
Utility functions to check if a disk template is enabed
This patch adds some helper functions to the config andstorage utils which check whether a disk template isenabled or not. The functions themselves are quite smallbut they will be used quite often and therefore should...
Add function for checking file access permissions
The CanRead function checks whether a user of the local machine (specifiedby name) can access a given file.
IsUserInGroup is a helper function for CanRead, but might also be usedindependently, so its name does not begin with an underscore....
Utility functions for storage types
Handling various storage types for the free space reportingrequires some utility functions. They will be invoked inlater patches. Unit tests are included.
Utility functions for storage
This patch add a couple of utility functions dealing withstorage types and disk templates.
Add function for getting the timestamp in nanoseconds
The timestamp is returned as an integer number of nanoseconds since the Unixepoch.
Check that device names are unique and valid
Extend the CheckArguments phase of LUInstanceCreate and CheckPrereqphase of LUInstanceSetParams to also check if the name parameters ofdisks and NICs are unique and valid.
Signed-off-by: Christos Stavrakakis <cstavr@grnet.gr>...
Document that OpTestDelay/TestDelay take duration in seconds
Also, fix @rtype and @return elements of utils.TestDelay(), which nowreturns a tuple but this wasn't being indicated.
Signed-off-by: Dato Simó <dato@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
utils.LvmExclusiveCheckNodePvs() returns more info
utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest andbiggest PVs of a node. They will be used for cross-node consistency checks.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
LVM disk creation uses dedicated PVs
When exclusive_storage is set, PVs are allocated according to thedesign-partitioned design doc.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Added class to contain information about a PV
This makes the code more readable and easier to upgrade.bdev.LogicalVolume.GetPVInfo and the code that depends on it have beenrefactored to use the new class.
utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more...
Add function for generating UUIDs in the Haskell codebase
Its first use will be to generate the salt for ConfD requests of the Haskellclient, as in the Python client.
Unit test added as well.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add utility to format dictionary as key=value strings
This will be used in QA to format network interface parameters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the nodedaemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” areverified, too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Fix breakage introduced in commit a8b3b09
The order of the calls to “ctx.use_privatekey” and “ctx.use_certificate”was wrong, leading to an exception being thrown.
Factorize SSL context setup for certificate check
This code will also be used by the node daemon setup utility.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
Fixes to pass pep8 (make lint)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
utils.process.RunResult: Always set "fail_reason" attribute
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RunCmd: Expose "postfork" callback
The “_postfork_fn” parameter was only used for tests until now. Toimplement a good locking scheme, remote commands must also make use ofthis callback to release a lock when the command was successfullystarted (but did not yet finish)....
Add utility to check if file is executable
This replaces direct calls to “os.access” and“os.path.exists”/“os.path.isfile”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.io: Improve handling of double and single slashes
Up until now “IsBelowDir("/", …)” would never return True. The reasonwas that an additional slash was added to the root path resulting in“//", which is “implementation-defined” in posix and treated specially...
RunCmd: Support standard input file descriptor
This patch changes “utils.RunCmd” to accept a file-like object or anumeric file descriptor which will be used as the command's standardinput. One use-case will be to pass all necessary data to“prepare-node-join”....
utils.x509: Factorize code to extract X509 certificate
This will be useful in “gnt-node add”.
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
Compare significant fields only for simple SSH keys
For simple SSH keys, that is those without options such as“command="…"”, only the first two parts need to be compared. The thirdfield is a free-form comment.
This patch changes the comparison used in...
Move constant for /etc/hosts to pathutils
Needed for coming patches.
utils.FilterEmptyLinesAndComments: Return list
We don't use generators often and lists are easier to re-use.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Factorize removing comments and empty lines from string
This will also be used for verifying the file storage directory.
utils.filelock: Remove executable bit from lock files
There's no need for lock files to be executable.
ShellWriter: Add parameter to disable indentation
This will be used to write a more compact bash completion script.
Migrate lib/utils/*.py from constants to pathutils
File system paths moved from constants to pathutils.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Verify user supplied dicts against defaults
This verifies the user (especially in nested dicts) does notprovide a key which is not seen in the defaults dict for that dict.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Allow clock skews in certificate verification
Currently we allow for up to NODE_MAX_CLOCK_SKEW time differencebetween nodes in some operations, but not everywhere: SSL certificateverification (import/export, both intra and inter-cluster) has a zerolimit (downwards), and a week upwards. This can cause even...
Log all external commands execution
This logs all external commands in normal (non-debug) mode. This willleak the DRBD secrets, but in any case we do log failed commands, soit's not a significant change.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.algo: Use str.isdigit instead of regular expression
str.isdigit is about 4x faster than using a regular expression ("\d+").This is in the inner sorting code so speed matters.
$ python -m timeit -s 'import re; s = re.compile("^\d+$")' \'s.match(""); s.match("Hello World"); s.match("1234")'...
Merge cli.FormatTimestamp and utils.FormatTime
… to some degree at least. Unittests are included.
Fix lint error in commit 035b33e2
Commit 035b33e2 forgot one blank space, and current pylint in DebianSid doesn't run on our code… so I didn't realise this until aftercommit, sorry.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix tempfile reset code & test on newer Python
Python 2.7.3 (rc status) and 3.2.3/3.3 (rc, respectively alpha status)have fixed http://bugs.python.org/issue12856 which we worked aroundourselves.
This means two things:
- we don't need to manually reset the module...
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
utils.text: Add function to truncate string
The function adds an ellipse if the string was actually truncated. Alsostart using it in mcpu for result checks (where the message is alsoslightly changed to use a colon).
Merge branch 'devel-2.5'
Fix a bug in command line option parsing code
Fix bug affecting command line options of "keyval" type. Althoughescaping commands with \ is supported, it is is not applied to theinput recursively.
Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
utils.ResetTempfileModule: Improve performance
This function is called for after every fork (e.g. for handling an RPCrequest). With the changes in this patch generating the next randomfilename is about 30% faster.
Move helper class from watcher to utils.io
“FileStatHelper” can be used together with “ReadFile” to a file's statuswhile it's opened. This avoids certain race conditions.
noded: Fix /etc/hosts group ownership
When Ganeti was compiled with the option of running as a differentuser/group, the group ownership of /etc/hosts was set to gnt-daemons. Nowpermissions for /etc/hosts are preserved (or set correctly).
utils.WriteFile: Add new parameter to preserve file permissions
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.io.WritePidFile: Improve error reporting
If the PID file is already locked by another process, try to readthe content and report it as part of the error message.
Move ErrnoOnStr backend function to utils
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.ListVisibleFiles: Hide “lost+found” directories
If a “lost+found” directory is found at a filesystem's root path it isignored. In all other cases directory entries named “lost+found” aretreated normally. Unittests are included. Fixes issue 153.
algo: Make a dict from an flat list
This is in preparation to take deeper dict constructs from the commandline. You can feed the optionslist directly constructed of type"identkeyval" to it and it returns a fully deflated dict.
This is mainly needed for the resource model changes where we have to...
utils.algo: Add utility to convert sequence to dictionary
Useful for converting list of query fields to a dictionary and toconvert RPC definitions. Includes duplicate detection.
utils.algo: Add a function to insert a list into a list
Move RenameFile to the new functions
ensure_dirs: Move some useful functions into utils.
With this change we can easily reuse this functionality where it makessense on other parts of Ganeti.
Add the JoinDisjointDicts function to utils.algo
Add a function that joins two dictionaries, enforcing the constraintthat the two key sets should be disjoint. Also, add unit tests for thisfunction.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>...
Un-revert comments in utils.mlock
These updates and clarifications are still useful.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
utils.ShellWriter: Don't indent empty lines
Empty lines shouldn't get indented. Unittest included.
Merge branch 'stable-2.5' into devel-2.5
Merge branch 'devel-2.4' into stable-2.5
Fix queue archive creation with wrong permissions
On a master failover some of the archive dirs might have wrongpermissions in the non-root model. This is due to the nature of nodedstill running as root and the job queue is synced that way. This patchwill fix this behaviour by setting the permissions accordingly....
Revert "Rename utils.mlock to utils.cfunc"
The rename is not needed either, since we're not adding more code as ofnow.This reverts commit 57ca011e1cd2681948969724e2646edaac22da28.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "utils.cfunc: Cleanup, more flexibility"
We discussed that this is not needed right now, and it breaks existingfunctionality and unittests.
This reverts commit 6915fe26da8dce41fc967d761f005390aa956161.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Revert "Implementation of TLS-protected SPICE connections"
This reverts commit b6267745ede04b3c943bc02e004bdb9347e0f564.This commit will be readded on master.
utils.cfunc: Cleanup, more flexibility
- Split code using ctypes directly into a helper class- Don't load “libc.so.6”, but use handle for main program instead (see comment in code)- Clarify comment on errno with older ctypes versions- Rename unittest since it can't be used for other functions (modifies...