History | View | Annotate | Download (84.6 kB)
Fix IgnoreSignals on socket.error
Some confusion arose handling EINTR on this function: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
In 2.4 and 2.5 it's not, and so its errno variable must be extracted...
Make utils.EnsureDirs() ignore umask
EnsureDirs() should create directories with the exact mode requestedin the arguments, but it currently applies the umask.This patch makes it independent from the umask.
Signed-off-by: Balazs Lecz <leczb@google.com>...
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix some pylint warnings
Disable warnings for:- except Exception,- use of __errno_location,- redeclaration of handleError()
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Fix logging string format
Fix this pylint warning:[W6501, Mlockall] Specify string format arguments as logging function parameters
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Fix Filehandler / FileHandler typo
Fix typo spotted by pylint:E1101:2095:LogFileHandler.handleError: Module 'logging' has no 'Filehandler' member
Use LogFileHandler conditionally in SetupLogging
Add a parameter to SetupLogging to use LogFileHandler (default is off)
Introduce LogFileHandler class
Add a log handler class which logs to /dev/console in case of logging errors.
Introduce Mlockall()
Add Mlockall() utility to lock current process' virtual adress space into RAM.
Convert some ReadFile calls to ReadOneLineFile
For passwords we require strict oneliners, we're a bit more lax with pidand uid files.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils.ReadOneLineFile()
Read the first non-empty file line. When strict is set, abort if morethan one line is non-empty. Some unittests inspired by the reverted onesfrom commit b774bb106cc28d008e790ad2666eb64c76866fa0, and some new ones.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Remove oneline= parameter from utils.ReadFile
This partially reverts commit b774bb106cc28d008e790ad2666eb64c76866fa0.Unittests unrelated to that particular functionality but introduced inthat commit are left untouched. Since the temporary directory is now...
utils.ReadFile: Add parameter to read only one line
Reading only one line is useful for reading passwords from files. Thisalso adds unittests for utils.ReadFile.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Retry{Again,Timeout}: explain reraising
IsProcessAlive: retry stat() a few times
On multiprocessor dom0 stat() on /proc can sometimes return EINVAL.Before giving up, we try a few times to get a consistent answer.
utils.Retry: pass up timeout arguments
If Retry has to fail with RetryTimeout, it might be useful to pass theRetry argument to RetryTimeout, to help debugging outside the Retrycycle. We also define a RetryTimeout.RaiseInner() helper, to re-raise anexception passed to RetryAgain. All served with a side of unit tests....
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
Fix bug introduced in 76e5f8b54: mkdir mode
After commit 76e5f8b54, mkdir_mode in utils.RenameFile isno longer passed to Makedirs. This is fixed by this patch.
utils: Move wrapper code around os.makedirs into separate function
Revert "Only override any and all if not defined"
This reverts commit bd5617020a50bcd08269330638d64078c1b30b71.
Turns out our and python's any/all are not compatible.
move http.WaitForSocketCondition to utils
WaitForSocketCondition: rename, handle EINTR
- Rename WaitForSocketCondition to SingleWaitForFdCondition - Avoid potentially infinite loop, if we continue to get interrupted - Handle eintr correctly - Avoid the poller try/finally, as the poller object gets destroyed...
Only override any and all if not defined
If any or all are already defined (because we're using a new version ofpython) just link them inside "utils" rather than redefining them.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>...
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Add a partition function to split a list in two
This is similar to the Haskell function, except that the signature isreverse to match the 'any' and 'all' Python functions.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Support passing in file object in utils.FileLock
This way we can re-use file objects opened in other places. Also add moreunittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support arguments in utils.RunInSeparateProcess
Convert utils.FileLock to utils.Retry
Add a function to validate and normalize hostnames
This differs slightly from the specification, by allowing names to startwith digits, not checking the length of individual components, andallowing underscores.
Signed-off-by: Iustin Pop <iustin@google.com>...
ListVisibleFiles: require normalized path names
This patch changes ListVisibleFiles to raise ProgrammerError if it'spassed a non-absolute/non-normalized path name, and adds unittests forthis behaviour.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Replace os.path.sep.join(seq) with utils.PathJoin
This is a no-op change, but at least we concentrate the calls to pathjoins into a single function.
A use in utils.FindFile is left as-is (don't want to raise exceptionsthere, at least for now).
Switch from os.path.join to utils.PathJoin
This passes a full burnin with lots of instances, and should be safe aswe mostly to join a known root (various constants) to a run-timevariable.
utils: Add a PathJoin function
This will replace os.path.join since it is not safe for directorytraversal issues.
Move watcher's EnsureDaemon function to utils
This is going to be used from the nbma repository, to ensure that thenld daemon is running.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Reset tempfile module after fork where useful
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add function to reset tempfile module after fork
On fork, the tempfile module's pseudo random generator isnot reset. If several processes (e.g. two children or parentand child) try to create a temporary file, they'll conflict.This function can be used to reset the name generator which...
Change the debug CLI option to integer/count
This changes from boolean to integer/count (for a future differentiationbased on the actual debug level). All the uses of the code only testit's boolean status, so it still works as an integer value.
Merge branch 'stable-2.1' into devel-2.1
utils: Fix retry delay calculator
Before this patch, it would always sleep for at leastthe time specified as the upper limit. Now it actuallylimits the sleep time.
Add capability to use syslog for logging
This patch adds a configure-time parameter that will set the defaultsused by all programs, and command-line parameters in the daemons thatallow overriding it.
Syslog 'yes' enables syslog in addition to file-based logging, 'only'...
utils.FileLock: handle init errors properly
If the open of the lock file fails (due to whatever reason), 'self'won't have the 'fd' attribute, and thus we fail in Close/__del__, whichwill ruin proper error reporting:
IOError: [Errno 30] Read-only file system: '/var/lib/ganeti/queue/lock'...
Merge branch 'devel-2.0' into devel-2.1
Conflicts: lib/backend.py - trivial merge...
Ensure all int/float conversions are handled right
int()/float() can raise either ValueError (in case of int("a")), orTypeError (in case of int(None)). We had many bugs over time due tothis, and a recent one was just diagnosed, so we go over the codebase...
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
Normalize MAC addresses to all lower.
This change will normalize the MAC to all lower after validation.
Signed-off-by: René Nussbaumer <rn@google.com>Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Further pylint disables, mostly for Unused args
Many of our functions have to follow a given API, and thus we have tokeep a given signature, but pylint doesn't understand this. Therefore,we silence this warning.
The patch does a few other cleanups.
Add targeted pylint disables
This patch should have only:
- pylint disables- docstring changes- whitespace changes
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Partial cherry-pick of 6c881c5 from the 2.1 branch
This cherry-picks the utils.FieldSet.Matches changes and the significantjqueue.py change. These are stable in the 2.1 branch and therefore makesense to backport to 2.0 (are basically cleanups).
Merge branch 'stable-2.0' into stable-2.1
Security issue: add validation of script names
This patch unifies the search for external script to always go throughutils.FindFile and implements in that function a restriction on validchars in file names and (additionally) that the passed name is the...
Simplify utils.ReadFile
The documentation for file objects' read method states that if the sizeis "negative or ommitted", all data is read; thus we can simplify it tohave size=-1 by default and not have the if test.
Remove quotes from CommaJoin and convert to it
This patch removes the quotes from CommaJoin and converts most of thecallers (that I could find) to it. Since CommaJoin does str(i) for i inparam, we can remove these, thus simplifying slightly a few calls....
Revert "Get rid of utils.CommaJoin"
This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread onganeti-devel.
Conflicts:
lib/cmdlib.py (due to the error code classification, trivial)
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
Fix pylint 'E' (error) codes
This patch adds some silences and tweaks the code slightly so that“pylint --rcfile pylintrc -e ganeti” doesn't give any errors.
The biggest change is in jqueue.py, the move of _RequireOpenQueue out ofthe JobQueue class. Since that is actually a function and not a method...
Introduce a wrapper for hostname resolving
Currently a few of the LU's CheckPrereq use utils.HostInfo which raisesa resolver error in case of failure. This is an exception from thestandard that CheckPrereq should raise an OpPrereqError if the error isin the 'pre' phase (so that it can be retried)....
Another round of pylint-related style fixes
A newer version of pylint, more warnings…
utils: Convert to utils.Retry
Add generic retry loop function
There are quite a few retry loops with timeouts in Ganeti'scode. Duplicating code is not good, so this patch introducesa new function named “utils.Retry” to remedy this situation.
Unify the query fields for the storage framework
This patch unifies the query fields in the storage framework for alltypes. Note that the information is still computed on-demand, so if e.g.the used disk space is not requested for the ‘file’ type, it won't be...
Get rid of utils.CommaJoin
- We never remember to use it (5 uses vs 21 " ,".join())- It's longer to write than " ,".join()- The added value of the apostrophe in the string is not very much
Add case_sensitive keyword to MatchNameComponent
Now featuring unit testing, and more deterministic results on somecorner cases.
Fix utils.MatchNameComponent for full matches
While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,exact match and we should return it if that is the case.
Move LoadModule function to ganeti.build
It should only be used at build-time.
Even more style fixes
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Move LoadModule function to utils
It can be used by unittests for daemons/* or scripts/*.
utils.CalculateDirectorySize: Don't redefine builtin
Use ReadFile/WriteFile in more places
This survived QA, burnin and unittests.
Remove watcher pause file 1h past end time
Add file to pause watcher for a certain duration
This can be used during maintenance work.
GenerateSecret: specify the length of the secret
Add a parameter to specify the length of the bytes sequence represented by thereturned hex string. Defaulting to 20 so it's compatible with previous SHA1hash.
Signed-off-by: Luca Bigliardi <shammash@google.com>...
Add a new SignalHandled decorator
This decorator can be used to call a function holding with protectionfrom a given signal. The function should accept a dict ofutils.SignalHandler, indexed by signal number, and make use of it.
SignalHandler, only accept list of signals
utils.SignalHandler can accept either a list of signals or a singlesignal, but it's always called with a list. Removing the single signaloption.
Add utils.FormatTime and a simple unittest
We don't format with subsecond-precision.
utils: Add functions to calc directory size and free space on filesystem
These will be used by the new storage unit framework.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Change GetNodeDaemonPort to GetDaemonPort in utils
GetNodeDaemonPort is used to lookup the node daemon port in the servicesfile, and if not found to return the default one. We make it a genericfunction, which accepts the daemon name in input, so that it can be used...
Merge branch 'next' into branch-2.1
Fix pylint warnings
Fix some typos
check_ident_key_val, handle no_ and - prefixes
If an ident member of an IdentKeyVal relationship starts with no_ or -,handle it the same way we do for a key. Some unittests are added tocheck that check_ident_key_val behaves as expected.
This patch also changes ForceDictType to, for now, fail on such an...
Fix various pylint warnings
There were multiple issues: - copy-paste resulted in wrong indentation - wrong function name - missing spaces around assignment - overriding built-in names (type, dir) or already defines ones (errors, hypervisor)
Convert test_delay rpc to new style result
Remove utils.CheckDict since it's no longer used
Only the custom handling in rpc.py for call_node_info used this, so nowit can be removed.
IsNormAbsPath and users, use "normalized" term
We used to refer to normalized paths as "normal" which might beconfusing. This fixes the syntax in all current IsNormAbsPath users andin the docstring.
Add utils.IsNormAbsPath function
Currently most of the time we check for absolute path, but that doesn'tprotect us from some invalid paths. In some places we should be morestrict, and this function should help us to.
Fix the SafeEncoding behaviour
Currently we have bad behaviour in SafeEncode: - binary strings are actually not handled correctly (ahem) - the encoding is not stable, due to use of string_escape
For this reason, we replace the use of string_escape with part of the...
Move more hypervisor strings into constants
This patch adds constants for the mouse and boot order strings; whilethere are still some issues remaining, we're trying to cleanup hardcodedstrings from the hypervisors.
Since the formatting of frozensets is currently wrong, we also add an...
Avoid DeprecationWarning on Python >= 2.6
Python 2.6 complains about module 'sha' being deprecated. It makesexecution of Ganeti commands a bit annoying, and when you run'ganeti-watcher' in cron jobs, you get a mail message after everyexecution.
Tests pass under under Python 2.6 and Python 2.4....
RunCmd: log command line for missing cmd case
In case of missing programs, currently utils.RunCmd doesn't show anyinformation to help debugging, only 'No such file or directory'. Thispatch adds error handling for the ENOENT case such that at least we have...
Small optimisation in utils.WriteFile
Currently we always try to remove the new file, even if the renamesucceeded. This patch tracks the existence of the new file and doesn'ttry to remove it if we managed to rename it.