History | View | Annotate | Download (78.1 kB)
Add new parameter type “maybe string”
Before strict checking was implemented, NIC IP addresses could be setto “None”. Commit bd061c35 added more strict checking, includingenforcing the IP address to be a string. With this new type, itcan again be set to None....
Add ParseCpuMask() utility function
Also adds a generic ParseError exception.
Signed-off-by: Balazs Lecz <leczb@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Move ShellWriter class to utils
Also add unittest.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Rename test for utils.IgnoreProcessNotFound
Usually our tests are named “Test…”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Manuel Franceschini <livewire@google.com>
Use reserved documentation IPs and domains
Use RFC 5737 IP addresses and RFC 2606 domain names in allunittests, docs, qa and docstrings.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
Add utils.GetMounts()
IPv6 support for utils.TcpPing()
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Add function to retrieve family of an ip address
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Introduce utils.IsValidIP{4,6}()
This patch introduces functions to check for valid IPv4 and IPv6addresses and converts IsValidIP() to return True if it is either a IPv4or a IPv6 address.
For now we do not change the functional behavior and replace IsValidIP...
User assertFalse instead of assert_(not ...)
Rename some constants to facilitate IPv6 support
ListVisibleFiles: do not sort output
Among all users, turns out just one may need the output to be sorted.All the others can cope without.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
utils: Add function to validate service name
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Handle ESRCH when sending signals
Upon sending signals, ESRCH can be reported when the target nolonger exists.
ListVisibleFiles: do optional sorting
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Add function to format seconds
utils: Add function to check whether process handles a signal
This will be used to avoid a race condition between starting a program (ddfor import/export) and sending signals to it.
Merge branch 'devel-2.1'
Fix IgnoreSignals on socket.error
Some confusion arose handling EINTR on this function: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
In 2.4 and 2.5 it's not, and so its errno variable must be extracted...
Move hash functions to the compat module
Since the hash functions' changed their module name between python 2.4and 2.6, and we have to do an try/import/except trick, we'll do it justonce, for both hash functions, and in compat.py. This also fixes a use...
Fix {Ignore, RetryOn}Signals on socket.error
Some confusion arose handling EINTR on those functions: in python 2.6socket.error is an IOError, and thus: - It's an EnvironmentError - It has an .errno member
Make utils.EnsureDirs() ignore umask
EnsureDirs() should create directories with the exact mode requestedin the arguments, but it currently applies the umask.This patch makes it independent from the umask.
Signed-off-by: Balazs Lecz <leczb@google.com>...
Add checks for master IP in cluster verify
This also updates a comment in the unittest for utils.py. We unittestthe new function for two things: correct reporting on real case (forlocalhost), and correct reporting with a mocked-out TcpPing that returns...
utils.IgnoreSignals
Remove duplicate code between a couple of asyncore related function byhaving a function in charge of handling EINTR errors. Unittests included.
utils.ReadOneLineFile()
Read the first non-empty file line. When strict is set, abort if morethan one line is non-empty. Some unittests inspired by the reverted onesfrom commit b774bb106cc28d008e790ad2666eb64c76866fa0, and some new ones.
Signed-off-by: Guido Trotter <ultrotter@google.com>...
Remove oneline= parameter from utils.ReadFile
This partially reverts commit b774bb106cc28d008e790ad2666eb64c76866fa0.Unittests unrelated to that particular functionality but introduced inthat commit are left untouched. Since the temporary directory is now...
utils.ReadFile: Add parameter to read only one line
Reading only one line is useful for reading passwords from files. Thisalso adds unittests for utils.ReadFile.
Fix a unittest name and docstring
The docstring has been copied by the name, but the name reports amisspell of the function it is actually testing.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Balazs Lecz <leczb@google.com>
Fix a unittest docstring
utils.Retry: pass up timeout arguments
If Retry has to fail with RetryTimeout, it might be useful to pass theRetry argument to RetryTimeout, to help debugging outside the Retrycycle. We also define a RetryTimeout.RaiseInner() helper, to re-raise anexception passed to RetryAgain. All served with a side of unit tests....
Add a few Retry unittests
Check that Retry actually times out, and actually retries.
Add optional “salt” argument to utils.{,Verify}Sha1Hmac
We're using salted hashes all over the place.
Use utility functions for HMAC
HMAC will be used in more places. Centralizing some parts can't hurt.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: René Nussbaumer <rn@google.com>
Move cmdlib._VerifyCertificate to utils
This function will also be useful for inter-cluster instancemoves for verifying certificates.
Conflicts: lib/utils.py: Trivial
Signed-off-by: Michael Hanselmann <hansmi@google.com>...
masterd: Log PID, UID and GID of connected client
This can be very useful if client programs run as non-root.
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
utils: Add function for partial application of function arguments
The function's code was mostly copied from Python's documentationand it's equivalent to “functools.partial” in Python 2.5 and above.
utils: Add function to read locked PID file
This is useful in combination with utils.StartDaemon and will be used forreading the import/export daemon's PID file.
utils: Add class to split string stream into lines
This will be used by the new import/export daemon to splitoutput by its child processes into lines.
Fix utils.WaitForFdCondition inner retry loop
Commit dfdc4060 added WaitForFdCondition which uses utils.Retry withouthandling timeout exceptions. This breaks any nested retry loops.
This patch fixes the above function, and also changes utils.Retry todetect and warn future similar cases. In addition, we add a few small...
utils: Move wrapper code around os.makedirs into separate function
Merge remote branch 'devel-2.1'
Remove unused import from test file
utils: Add functions to sign and verify X509 certs using HMAC
Certificates exchanged via an untrusted third party should besigned to ensure they haven't been modified.
Merge remote branch 'origin/devel-2.1'
utils.RunCmd: Test case with reset_env set and setting variables
utils.CreateBackup: Use human-readable instead of seconds since Epoch
Seconds since the Epoch are not easily readable by a human. Using aformatted timestamp makes it easier (e.g.“….backup-2010-03-12_14_02_43.…”). This patch also makes OS logfiles usethis formatted timestamp....
utils: Add function to extract X509 cert validity
X509 uses ASN1 GENERALIZEDTIME or UTCTIME to store certificate validity.pyOpenSSL 0.7 and above allow us to retrieve both “notBefore” and“notAfter” as strings. Parsing them turned out to be a challenge since...
Support passing in file object in utils.FileLock
This way we can re-use file objects opened in other places. Also add moreunittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Support arguments in utils.RunInSeparateProcess
Add a function to validate and normalize hostnames
This differs slightly from the specification, by allowing names to startwith digits, not checking the length of individual components, andallowing underscores.
Signed-off-by: Iustin Pop <iustin@google.com>...
ListVisibleFiles: require normalized path names
This patch changes ListVisibleFiles to raise ProgrammerError if it'spassed a non-absolute/non-normalized path name, and adds unittests forthis behaviour.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils: Add a PathJoin function
This will replace os.path.join since it is not safe for directorytraversal issues.
Implement utils.RunParts and use it for hooks
This function is a generic pythonic version of runparts. We currentlyuse it in the backend HooksRunner, but we'll use it for runningdifferent directories as well.
Add reset_env option to RunCmd
This allows to run a command with only the passed in environment, ratherthan just updating the default one with it.
Now with unit testing.
Add utility function to start daemon
The currently available functions in this direction (utils.RunCmd andutils.Daemonize) both can not be used to start an external daemonprocess. This function takes many steps to ensure the daemon is startedproperly and does careful error checking. Unittests are included....
Use OpenSSL module instead of binary to generate certs
This saves us one dependency and saves us from complicated handling ofexternal files if we need key and certificate separated from each other.
At the same time, the number of bits used for RSA keys is increased from...
utils: Add function to set O_NONBLOCK on file descriptor
utils: Add function to set FD_CLOEXEC on file descriptor
This will be used by another patch adding a utility function to start a daemon.
Move RunInSeparateProcess to ganeti.utils
This function could be useful in other places and thisway we can easily unittest it.
Add unittest for utils._FingerprintFile
Move function generating SSL certs into utils
Add an UnescapeAndSplit function
In many cases, where we accept (usually from the command line) a list ofparameters, we remove the use of the separator as an component of any ofthe elements.
This patch adds a new function that can split strings of the form...
Handle EEXIST in utils.RenameFile
This should fix an issue I've seen exactly once during testing. It might havebeen caused by parallel RPC calls to archive jobs.
[…] ganeti-noded:112 ERROR Error in RPC call […] File "/usr/lib/python2.4/site-packages/ganeti/backend.py", line 2365, in JobQueueRename...
Ignore log messages in unittests
Add case_sensitive keyword to MatchNameComponent
Now featuring unit testing, and more deterministic results on somecorner cases.
Fix utils.MatchNameComponent for full matches
While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,exact match and we should return it if that is the case.
Wrap lines over 80 characters
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Add utils.FormatTime and a simple unittest
We don't format with subsecond-precision.
Merge branch 'next' into branch-2.1
Remove utils.CheckDict since it's no longer used
Only the custom handling in rpc.py for call_node_info used this, so nowit can be removed.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
IsNormAbsPath and users, use "normalized" term
We used to refer to normalized paths as "normal" which might beconfusing. This fixes the syntax in all current IsNormAbsPath users andin the docstring.
Add utils.IsNormAbsPath function
Currently most of the time we check for absolute path, but that doesn'tprotect us from some invalid paths. In some places we should be morestrict, and this function should help us to.
Fix the SafeEncoding behaviour
Currently we have bad behaviour in SafeEncode: - binary strings are actually not handled correctly (ahem) - the encoding is not stable, due to use of string_escape
For this reason, we replace the use of string_escape with part of the...
Apply the right permissions to /etc/hosts
In the current Ganeti version when modifying /etc/hosts we mistakenlygive it the permissions of the temporary file we create to define itscontent, which is by default 0600. This breaks most non-rootapplications, and thus must be corrected. This patch forces the mode to...
Instance parameters: force typing
We want all the hv/be parameters to have a known type, rather than arandom mix of empty string, boolean values, and None, so we declare thetype of each variable and we enforce/convert it.
- Add some new constants for enforceable value types...
Add a TailFile function
This patch adds a tail file function, to be used for parsing and returning inthe job log OS installation failures.
Reviewed-by: ultrotter
Unify some unittest functions
This patch adds unified temporary file handling to thetestutils.GanetiTestCase class, which adds easy creation and automatedcleanup of temporary files.
The patch allows a simpler handling in a couple of test cases butrequires all child classes to call the parent setUp and tearDown...
Add rename function automatically creating directories if needed
Unfortunately, os.makedirs in Python 2.4 is not safe against multipleprocesses creating the same directory tree at the same time. This isonly fixed in Python 2.5 and up. Adding more checks in our code doesn't...
Enable auto-unit formatting in script output
This patch enables by default the old 'human-readable' option, but in aslightly different model.
The option is now called "units" and takes either: - 'h' for automatic formatting - 'm', 'g' or 't' for mebi/gibi/tebibytes...
Move FieldSet class to utils.py
Since we can use the FieldSet class in cli.py to nicely format disksizes and such, we move it to utils.py and also move its associatedunittest. I didn't remove the cmdlib.py unittest file as that's not thegood direction :)...
Remove zombie special case from IsProcessAlive
Based on the discussion on the mailing list, we remove the specialcasing of zombie processes.
Note we don't use kill -0 since that has a different meaning than "checkprocess is alive", so this function is still linux-specific....
Add cleanup of processes to utils.KillProcess
In case the process we want to signal is our own process, and the signalwe send is a deadly one, we should also cleanup after the process.
This patch adds a new parameter waitpid to this function that does this,...
Implement working directory for utils.RunCmd
In order to replace usage of the shell just for 'cd %s; ...' we need tobe able to specify the working directory for childs.
The patch also changes the default working directory (when notspecified) to "/", as opposed to the current dir; since the daemos run...
Modify utils.RunCmd to write output to file
Currently we launch processes via the shell in a few places only toredirect standard output and error to a log file ("&> $file"). It isbetter to do such redirection from within RunCmd itself.
This patch splits RunCmd in two parts, the setup and the execution part,...
Really fix with the zombie test (hopefully)
So, instead of any timeouts for synchronization, we move to actualevents. We still have one (huge) timeout for pathological cases (just tocleanup), but we don't sleep at all in normal usage.
The patch moves the zombie/cleaned up child creation out from setUp(),...
Abstract checking own address into a function
Currently, we check if we have a given ip address (i.e. it's alive onone of our interfaces) but manually calling TcpPing(source=localhost).This works, but having it spread all over the code makes it hard to...
utils.SplitTime: More rounding fixes
SplitTime didn't round the same on different platforms. This patch changesit to use microseconds and not care about rounding.
Reviewed-by: iustinp
utils.SplitTime: Fix rounding of milliseconds
Reported by Iustin.
It used to return this:
utils.SplitTime(1234.999999999999)
(1234, 1000)
while it should've returned this:
(1235, 0)