utils.LvmExclusiveCheckNodePvs() returns more info
utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest andbiggest PVs of a node. They will be used for cross-node consistency checks.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
LVM disk creation uses dedicated PVs
When exclusive_storage is set, PVs are allocated according to thedesign-partitioned design doc.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Added class to contain information about a PV
This makes the code more readable and easier to upgrade.bdev.LogicalVolume.GetPVInfo and the code that depends on it have beenrefactored to use the new class.
utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more...
Add function for generating UUIDs in the Haskell codebase
Its first use will be to generate the salt for ConfD requests of the Haskellclient, as in the Python client.
Unit test added as well.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add utility to format dictionary as key=value strings
This will be used in QA to format network interface parameters.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
utils.text: Function to verify MAC address prefix
The network management code needs to verify a MAC address prefix.Instead of (ab)using NormalizeAndValidateMac, clean code should be used.Unit tests for NormalizeAndValidateMac are updated and new ones for...
Factorize code for checking node daemon certificate
This code is going to be used by a new utility for setting up the nodedaemon. Unit tests are updated/added.
Additionally, the certificate and key stored in “server.pem” areverified, too.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
Fix breakage introduced in commit a8b3b09
The order of the calls to “ctx.use_privatekey” and “ctx.use_certificate”was wrong, leading to an exception being thrown.
Factorize SSL context setup for certificate check
This code will also be used by the node daemon setup utility.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Helga Velroyen <helgav@google.com>
Factorize logging setup in tools
Most tools had their own “SetupLogging” function, but they were allessentially the same. This patch adds a generic version to “utils.log”and provides unit tests.
Fixes to pass pep8 (make lint)
Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>Reviewed-by: Iustin Pop <iustin@google.com>
utils.process.RunResult: Always set "fail_reason" attribute
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
RunCmd: Expose "postfork" callback
The “_postfork_fn” parameter was only used for tests until now. Toimplement a good locking scheme, remote commands must also make use ofthis callback to release a lock when the command was successfullystarted (but did not yet finish)....
Add utility to check if file is executable
This replaces direct calls to “os.access” and“os.path.exists”/“os.path.isfile”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.io: Improve handling of double and single slashes
Up until now “IsBelowDir("/", …)” would never return True. The reasonwas that an additional slash was added to the root path resulting in“//", which is “implementation-defined” in posix and treated specially...
RunCmd: Support standard input file descriptor
This patch changes “utils.RunCmd” to accept a file-like object or anumeric file descriptor which will be used as the command's standardinput. One use-case will be to pass all necessary data to“prepare-node-join”....
utils.x509: Factorize code to extract X509 certificate
This will be useful in “gnt-node add”.
Add initial implementation of prepare-node-join
This is a new tool as per the design document “design-ssh-setup”. Itreceives a JSON data structure on its standard input and configures theSSH daemon and root's SSH keys accordingly. Unit tests are included....
Compare significant fields only for simple SSH keys
For simple SSH keys, that is those without options such as“command="…"”, only the first two parts need to be compared. The thirdfield is a free-form comment.
This patch changes the comparison used in...
Move constant for /etc/hosts to pathutils
Needed for coming patches.
utils.FilterEmptyLinesAndComments: Return list
We don't use generators often and lists are easier to re-use.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
Factorize removing comments and empty lines from string
This will also be used for verifying the file storage directory.
utils.filelock: Remove executable bit from lock files
There's no need for lock files to be executable.
ShellWriter: Add parameter to disable indentation
This will be used to write a more compact bash completion script.
Migrate lib/utils/*.py from constants to pathutils
File system paths moved from constants to pathutils.
constants: Move most paths to separate module
This is inpreparation for the implementation of virtual clusters. Manypaths will change based on an environment variable and are no longerconstant and should no longer be in “constants.py”. Since “constants.py”...
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Verify user supplied dicts against defaults
This verifies the user (especially in nested dicts) does notprovide a key which is not seen in the defaults dict for that dict.
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Allow clock skews in certificate verification
Currently we allow for up to NODE_MAX_CLOCK_SKEW time differencebetween nodes in some operations, but not everywhere: SSL certificateverification (import/export, both intra and inter-cluster) has a zerolimit (downwards), and a week upwards. This can cause even...
Log all external commands execution
This logs all external commands in normal (non-debug) mode. This willleak the DRBD secrets, but in any case we do log failed commands, soit's not a significant change.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.algo: Use str.isdigit instead of regular expression
str.isdigit is about 4x faster than using a regular expression ("\d+").This is in the inner sorting code so speed matters.
$ python -m timeit -s 'import re; s = re.compile("^\d+$")' \'s.match(""); s.match("Hello World"); s.match("1234")'...
Merge cli.FormatTimestamp and utils.FormatTime
… to some degree at least. Unittests are included.
Fix lint error in commit 035b33e2
Commit 035b33e2 forgot one blank space, and current pylint in DebianSid doesn't run on our code… so I didn't realise this until aftercommit, sorry.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix tempfile reset code & test on newer Python
Python 2.7.3 (rc status) and 3.2.3/3.3 (rc, respectively alpha status)have fixed http://bugs.python.org/issue12856 which we worked aroundourselves.
This means two things:
- we don't need to manually reset the module...
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
utils.text: Add function to truncate string
The function adds an ellipse if the string was actually truncated. Alsostart using it in mcpu for result checks (where the message is alsoslightly changed to use a colon).
Merge branch 'devel-2.5'
Fix a bug in command line option parsing code
Fix bug affecting command line options of "keyval" type. Althoughescaping commands with \ is supported, it is is not applied to theinput recursively.
Signed-off-by: Nikos Skalkotos <skalkoto@grnet.gr>Signed-off-by: Iustin Pop <iustin@google.com>...
utils.ResetTempfileModule: Improve performance
This function is called for after every fork (e.g. for handling an RPCrequest). With the changes in this patch generating the next randomfilename is about 30% faster.
Move helper class from watcher to utils.io
“FileStatHelper” can be used together with “ReadFile” to a file's statuswhile it's opened. This avoids certain race conditions.
noded: Fix /etc/hosts group ownership
When Ganeti was compiled with the option of running as a differentuser/group, the group ownership of /etc/hosts was set to gnt-daemons. Nowpermissions for /etc/hosts are preserved (or set correctly).
utils.WriteFile: Add new parameter to preserve file permissions
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.io.WritePidFile: Improve error reporting
If the PID file is already locked by another process, try to readthe content and report it as part of the error message.
Move ErrnoOnStr backend function to utils
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.ListVisibleFiles: Hide “lost+found” directories
If a “lost+found” directory is found at a filesystem's root path it isignored. In all other cases directory entries named “lost+found” aretreated normally. Unittests are included. Fixes issue 153.
algo: Make a dict from an flat list
This is in preparation to take deeper dict constructs from the commandline. You can feed the optionslist directly constructed of type"identkeyval" to it and it returns a fully deflated dict.
This is mainly needed for the resource model changes where we have to...
utils.algo: Add utility to convert sequence to dictionary
Useful for converting list of query fields to a dictionary and toconvert RPC definitions. Includes duplicate detection.
utils.algo: Add a function to insert a list into a list
Move RenameFile to the new functions
ensure_dirs: Move some useful functions into utils.
With this change we can easily reuse this functionality where it makessense on other parts of Ganeti.
Add the JoinDisjointDicts function to utils.algo
Add a function that joins two dictionaries, enforcing the constraintthat the two key sets should be disjoint. Also, add unit tests for thisfunction.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>...
Un-revert comments in utils.mlock
These updates and clarifications are still useful.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
utils.ShellWriter: Don't indent empty lines
Empty lines shouldn't get indented. Unittest included.
Merge branch 'stable-2.5' into devel-2.5
Merge branch 'devel-2.4' into stable-2.5
Fix queue archive creation with wrong permissions
On a master failover some of the archive dirs might have wrongpermissions in the non-root model. This is due to the nature of nodedstill running as root and the job queue is synced that way. This patchwill fix this behaviour by setting the permissions accordingly....
Revert "Rename utils.mlock to utils.cfunc"
The rename is not needed either, since we're not adding more code as ofnow.This reverts commit 57ca011e1cd2681948969724e2646edaac22da28.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Revert "utils.cfunc: Cleanup, more flexibility"
We discussed that this is not needed right now, and it breaks existingfunctionality and unittests.
This reverts commit 6915fe26da8dce41fc967d761f005390aa956161.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Revert "Implementation of TLS-protected SPICE connections"
This reverts commit b6267745ede04b3c943bc02e004bdb9347e0f564.This commit will be readded on master.
utils.cfunc: Cleanup, more flexibility
- Split code using ctypes directly into a helper class- Don't load “libc.so.6”, but use handle for main program instead (see comment in code)- Clarify comment on errno with older ctypes versions- Rename unittest since it can't be used for other functions (modifies...
Rename utils.mlock to utils.cfunc
Renaming so that more code using ctypes could be added to the same file.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Revert "utils.log: Write error messages to stderr"
This reverts commit 34aa8b7c4bb6f5e2e788108e024c9cd70bdb3431. Writingerror messages to stderr would also include backtraces, something wetried to avoid in the past.
utils.log: Write error messages to stderr
When “gnt-cluster copyfile” failed it would only print “Copy of file …to node … failed”. A detailed message is written using logging.error.Writing error messages to stderr can be helpful in figuring out whatwent wrong (the messages also go to the log file, but not everyone might...
Move _TimeoutExpired to utils
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
TemporaryFilesManager implementation
Signed-off-by: Agata Murawska <agatamurawska@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Implementation of TLS-protected SPICE connections
Added support for TLS-protected SPICE connections:
Switch other commonprefix to IsBelowDir
utils: Introduce IsBelowDir
This is mainly a wrapper to overcome the limitation of commonprefixwhich makes a string by string comparisation and reports the commonprefix in both strings. However this is bad for directory handling.
IsBelowDir works around this limitation and should be used in favour of...
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
Merge branch 'devel-2.4' into devel-2.5
Conflicts: NEWS (trivial) configure.ac (trivial) daemons/ensure-dirs.in (deleted)
utils: Fix UnescapeAndSplit parsing bug
If a value passed to UnescapeAndSplit ended with a backslash anexception would be raised:
$ gnt-instance modify -H mem=x\\ inst1.example.com[…] e2 = slist.pop(0)IndexError: pop from empty list
PEP8 style fixes
Identified using the “pep8” utility.
Added function for parsing multiple CPU pinning mask
Signed-off-by: Tsachy Shacham <tsachy@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.ReadFile: Add pre-read callback
This will be used by the watcher to store the file's fstat(2). It mustbe done from the filehandle.
Merge branch 'stable-2.4'
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fixed a typo in utils/process.py
Signed-off-by: Agata Murawska <agatamurawska@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Fix small typo in docstring
Most boring patch ever
s/'/"/ in (hopefully) the right places.
Merge branch 'devel-2.4'
Reopen daemon's stdio on SIGHUP
Before this patch daemons would continue to refer to an old logfile fortheir standard I/O if they had been asked to reopen the log (SIGHUP).
Reopen log file only once after SIGHUP
Commit b6fa9a44 added a re-openable log handler. The log file isreopened when a daemon is sent a HUP signal. Due to a bug in the code,fixed by this patch, the log file would be reopened for every single logmessage thereafter....
Don't leak file descriptors when setting up daemon output
When a daemon's output is configured using “utils.SetupDaemonFDs”, thefunction must use dup2(2). Unfortunately the code didn't close theoriginal file descriptors, leaking them in the process.
utils.algo: Add InvertDict to invert a dict
mlock: fail gracefully if libc.so.6 cannot be loaded
This allows noded to continue instead of blowing up if the libc majornumber changes.
Fix 'unused import' lint error
Sorry!
SetEtcHostsEntry: maintain existing ordering
Currently RemoveEtcHostsEntry keeps the ordering, but SetEtcHostsEntrynot, as it will always write the new entry at the end of file. Ipersonally dislike this as it "uglifies" my custom host files, so thispatch makes it update the record instead in-place so to say instead of...
Convert utils.nodesetup to utils.WriteFile(data=…)
It makes no sense to iteratively write the new etc/hosts file, as wecan pre-compute the desired contents (neither the old nor the newversions are safe against concurrent changes anyway).
Signed-off-by: Iustin Pop <iustin@google.com>...
Fix WriteFile with unicode data
Unicode is fun, indeed:
len(buffer("abc"))
3
len(buffer(u"abc"))
12
So we can't pass unicode data to buffer(), as the result will be towrite the in-memory (usually UTF-32) representation to disk.
Fix potential data-loss in utils.WriteFile
os.write can do incomplete writes, as long as at least some bytes havebeen written (like write(2)):
os.write(fd, " " * 1300)
1300
os.write(fd, " " * 1300)...