Statistics
| Branch: | Tag: | Revision:

root / lib / utils.py @ 38e250ba

History | View | Annotate | Download (52.7 kB)

# Date Author Comment
d21d09d6 01/20/2009 07:19 pm Iustin Pop

Update the logging output of job processing

(this is related to the master daemon log)

Currently it's not possible to follow (in the non-debug runs) the
logical execution thread of jobs. This is due to the fact that we don't
log the thread name (so we lose the association of log messages to jobs)...

ae59efea 01/20/2009 06:47 pm Michael Hanselmann

.gitignore: Don't exclude whole /autotools/ dir, but only files

This way newly added files will be not be excluded by default. Fixes
also a small whitespace error in utils.py.

Reviewed-by: iustinp

f65f63ef 01/20/2009 01:18 pm Iustin Pop

Add a TailFile function

This patch adds a tail file function, to be used for parsing and returning in
the job log OS installation failures.

Reviewed-by: ultrotter

7d88772a 01/09/2009 02:52 pm Iustin Pop

Rework the daemonization sequence

The current fork+close fds sequence has deficiencies which are hard to
work around:
- logging can start logging before we fork (e.g. if we need to emit
messages related to master checking), and thus use FDs which we...

6e797216 12/18/2008 06:23 pm Michael Hanselmann

Add rename function automatically creating directories if needed

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't...

7167159a 12/17/2008 01:24 pm Michael Hanselmann

utils.KillProcess: Use waitpid() to wait for child processes

Sometimes the proc filesystem doesn't reflect the current status of
a process. By calling waitpid(), we make sure to get the current
information, at least for child processes. The timeout is still...

bf988c29 12/16/2008 06:24 pm Guido Trotter

_RunCmdPipe: handle EINTR in poller.poll()

poll() can be interrupted. rather than failing we retry until it
returns.

Reviewed-by: iustinp

0b5ad33e 12/14/2008 02:03 pm Iustin Pop

cleanup: TcpPing, wrong variable name

The default value of 'False' wasn't initialized properly. It doesn't
require initialization, but it's cleaner this way.

Reviewed-by: amishchenko

cd34e7bf 12/14/2008 02:03 pm Iustin Pop

cleanup: SetEtcHostsEntry unused var

Reviewed-by: amishchenko

90eb468c 12/03/2008 01:12 pm Guido Trotter

CheckBEParams handle a bool BE_AUTO_BALANCE

This only happens at cluster init, if the value is not user-specified.

Reviewed-by: imsnah

42ce2e13 12/02/2008 12:53 pm Guido Trotter

Add utils.CheckBEParams

This function will be used in LUCreateInstance, LUSetInstanceParams,
LUSetClusterParams and InitCluster to check the backend parameters
validity and convert the relevant values to integer, without duplicating
code. It lives in utils as bootstrap.py is calling it too....

9fbfbb7b 11/23/2008 05:34 pm Iustin Pop

Enable auto-unit formatting in script output

This patch enables by default the old 'human-readable' option, but in a
slightly different model.

The option is now called "units" and takes either:
- 'h' for automatic formatting
- 'm', 'g' or 't' for mebi/gibi/tebibytes...

a2d2e1a7 11/21/2008 05:35 pm Iustin Pop

Move FieldSet class to utils.py

Since we can use the FieldSet class in cli.py to nicely format disk
sizes and such, we move it to utils.py and also move its associated
unittest. I didn't remove the cmdlib.py unittest file as that's not the
good direction :)...

ec17d09c 11/21/2008 12:46 pm Michael Hanselmann

Get rid of node daemon password

With the new SSL client certificate stuff it's no longer needed.

Reviewed-by: iustinp

6346a9e5 11/12/2008 02:51 pm Michael Hanselmann

utils.SetupLogging: Remove previously setup handlers

If a logging function is called before the logging module is setup
using utils.SetupLogging, it calls logging.basicConfig, which adds
a StreamHandler, on its own. If we leave it in, all log output will...

e1bd0072 11/12/2008 05:43 am Iustin Pop

Fix utils.KillProcess

Rev 1978 introduced a breakage on the SIGKILL finall signal to the
process, due to mistyped variable.

Reviewed-by: francis.perron

44bf25ff 11/11/2008 12:58 pm Iustin Pop

Remove zombie special case from IsProcessAlive

Based on the discussion on the mailing list, we remove the special
casing of zombie processes.

Note we don't use kill -0 since that has a different meaning than "check
process is alive", so this function is still linux-specific....

ff5251bc 11/11/2008 12:58 pm Iustin Pop

Add cleanup of processes to utils.KillProcess

In case the process we want to signal is our own process, and the signal
we send is a deadly one, we should also cleanup after the process.

This patch adds a new parameter waitpid to this function that does this,...

8797df43 11/11/2008 05:12 am Iustin Pop

Implement working directory for utils.RunCmd

In order to replace usage of the shell just for 'cd %s; ...' we need to
be able to specify the working directory for childs.

The patch also changes the default working directory (when not
specified) to "/", as opposed to the current dir; since the daemos run...

58885d79 10/28/2008 01:20 am Iustin Pop

Documentation updates for utils.py

Reviewed-by: imsnah

36117c2b 10/24/2008 05:43 pm Iustin Pop

Modify utils.RunCmd to write output to file

Currently we launch processes via the shell in a few places only to
redirect standard output and error to a log file ("&> $file"). It is
better to do such redirection from within RunCmd itself.

This patch splits RunCmd in two parts, the setup and the execution part,...

82d9caef 10/20/2008 03:50 pm Iustin Pop

Remove the logger.py module

Since now we use only one function from the logger module
(SetupLogging), we move it to utils.py (which is already imported by all
users of this function), and we remove the module.

Reviewed-by: imsnah

caad16e2 10/12/2008 11:40 pm Iustin Pop

Abstract checking own address into a function

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to...

4a8b186a 10/01/2008 08:33 pm Michael Hanselmann

Move functions from ssconf.py elsewhere

These functions will be used to access config values instead of using
ssconf.

Reviewed-by: iustinp

33081d90 09/28/2008 05:44 pm Iustin Pop

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

ca0aa6d0 09/17/2008 07:07 pm Michael Hanselmann

Add utils.ReadFile function

It abstracts exception handling and is like a complement to
utils.WriteFile.

Reviewed-by: iustinp

aa74b828 09/05/2008 06:38 pm Michael Hanselmann

utils.FileLock: Implement timeout

The timeout can be used in ganeti-noded to be more robust against
deadlocks.

Reviewed-by: iustinp

45bc5e4a 09/05/2008 01:57 pm Michael Hanselmann

utils.SplitTime: More rounding fixes

SplitTime didn't round the same on different platforms. This patch changes
it to use microseconds and not care about rounding.

Reviewed-by: iustinp

b77ba978 08/29/2008 06:04 pm Michael Hanselmann

utils.SplitTime: Fix rounding of milliseconds

Reported by Iustin.

It used to return this:

utils.SplitTime(1234.999999999999)

(1234, 1000)

while it should've returned this:

utils.SplitTime(1234.999999999999)

(1235, 0)

Reviewed-by: ultrotter

e67bd559 08/26/2008 06:44 pm Michael Hanselmann

Add simple lock debug output

Currently it can only be enabled by modifying utils.py, but we can
add a command line parameter later if needed.

Reviewed-by: schreiberal

2557ff82 08/14/2008 01:26 pm Guido Trotter

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

e326d4e5 08/13/2008 05:23 pm Guido Trotter

Convert RunCmd to an epydoc docstring

Reviewed-by: imsnah

739be818 08/11/2008 07:26 pm Michael Hanselmann

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

aa65ed72 08/08/2008 01:02 pm Michael Hanselmann

Log only unexpected errors in utils.FileLock

Otherwise users might be confused by errors in log files.

Reviewed-by: iustinp

a87b4824 07/31/2008 05:33 pm Michael Hanselmann

Add FileLock utility class

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

8161a646 07/30/2008 06:00 pm Iustin Pop

Fix a misuse of exc_info in logging.info

This is my fault, sorry.

Reviewed-by: imsnah

38206f3c 07/30/2008 05:04 pm Iustin Pop

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

53beffbb 07/30/2008 11:34 am Iustin Pop

Expose utils.DaemonPidFileName

Since we need to compute this from outside utils.py, we change this to a
public function.

Reviewed-by: ultrotter

b2a1f511 07/29/2008 11:49 am Iustin Pop

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

d9f311d7 07/29/2008 11:49 am Iustin Pop

Change IsPidFileAlive into ReadPidFile

We already have a function to test if a PID is alive, so it makes more
sense to use function composition that force calling (since we need to
read PIDs from files in other places too). Now IsProcessAlive returns
False for PIDs <= 0, since this is the error return from ReadPidFile....

533bb4b1 07/28/2008 12:16 pm Michael Hanselmann

Fix exception class name in utils.WritePidFile

Reviewed-by: iustinp

b330ac0b 07/23/2008 05:23 pm Guido Trotter

Add utils.{Write,Remove}PidFile

WritePidFile is a helper function that writes the current pid in a
pidfile within the ganeti run directory. RemovePidFile tries to delete
it.

Reviewed-by: iustinp

fee80e90 07/23/2008 05:23 pm Guido Trotter

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp

de499029 07/21/2008 06:32 pm Michael Hanselmann

Add signal handler class

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

eb0f0ce0 07/10/2008 03:38 pm Michael Hanselmann

Move watcher's LockFile function to utils

Reviewed-by: iustinp

7996a135 07/10/2008 03:17 pm Iustin Pop

Add a simple decorator for instance methods

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

bb698c1f 07/09/2008 01:41 pm Iustin Pop

Convert utils.py to the logging module

The patch also logs all commands executed from RunCmd when we are at
debug level.

Reviewed-by: imsnah

d4fa5c23 07/09/2008 01:41 pm Iustin Pop

Remove the old locking functions

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah

7b4126b7 07/01/2008 02:44 pm Iustin Pop

Add a FirstFree function to utils.py

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

04a8d789 06/23/2008 12:52 pm Michael Hanselmann

utils.WriteFile: Remove optional check_abspath parameter

cfgupgrade will not work with relative paths at all, but rather get them
from constants.py.

Reviewed-by: iustinp

71714516 06/18/2008 03:30 pm Michael Hanselmann

Add more parameters to utils.WriteFile

- Make closing file optional: Required by ganeti-watcher to keep
file open after writing it. Changes return value of utils.WriteFile
if "close" parameter evaluates to True.
- Pre- and post-write functions: Can be used to lock files. This...

8d1a2a64 06/12/2008 04:04 pm Michael Hanselmann

Move cmdlib._HasValidVG to utils.CheckVolumeGroupSize

This is required for splitting the cluster initialization code.

Reviewed-by: iustinp

d9c02ca6 06/12/2008 04:04 pm Michael Hanselmann

Move {Set,Remove}EtcHostsEntry wrappers to utils.py

This is required for the split of the cluster initialization code.

Reviewed-by: iustinp, ultrotter

e3e66f02 05/09/2008 01:12 pm Michael Hanselmann

Remove utils.CheckDaemonAlive and use “xm info” instead

There are a couple of reasons for doing so:
- /proc is not OS independent, it's only supported by Linux (there are
emulations on other systems, but those might differ from the way
Linux represents data)....

b74159ee 04/29/2008 10:37 am Iustin Pop

Disable forking in the master daemon

This patch adds a mechanism to disable utils.RunCmd in selected
programs. This is needed in the master daemon unless we confirm
threading doesn't pose any problems.

This makes cluster init fail, but creating new trunk clusters is anyway...

70f4497c 04/25/2008 04:37 pm Michael Hanselmann

Extend utils.WriteFile

Add parameters to not check the path for absoluteness, implement a dry
run mode and automatically create a backup if requested. This will be
used by the cfgupgrade utility.

Reviewed-by: schreiberal

57c177af 04/10/2008 04:41 pm Iustin Pop

Move the OS search code into an abstract function

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

8ff612c2 04/04/2008 06:29 pm Iustin Pop

Allow utils.Daemonize() to not close some fds

This patch implements an optional parameter to utils.Daemonize() which
allows that function to not close some file descriptors.

This will allow the master daemon to open the listening socket before
fork - in order to be able to notify errors and return a meaningful exit...

8f765069 04/04/2008 03:02 pm Iustin Pop

Move the daemonize function to utils.py

Currently, in ganeti-noded we have the createDaemon function. Since
we'll need the same in other daemons, we move this function to utils.py

With the move, a few changes were also done:
- change the name to Daemonize()...

b15d625f 03/11/2008 09:44 am Iustin Pop

Modify utils.TcpPing to make source address optional

This patch modifies TcpPing and its callers to make the source address
selection optional. Usually, the kernel will know better what
source address to use, just in some cases we want to enforce a given...

7c0d6283 03/05/2008 04:39 pm Michael Hanselmann

Codestyle fixes: adding a few empty lines

Reviewed-by: ultrotter

06009e27 02/05/2008 04:12 pm Iustin Pop

Add a test opcode that sleeps for a given duration

This can be used for testing purposes.

Reviewed-by: ultrotter,imsnah

1862d460 12/27/2007 05:31 pm Alexander Schreiber

Allow instance MAC address to be set.

Allow the MAC address of an instance to be specified optionally during
instance creation and later to be changed via instance modify.

Reviewed-by: iustinp

9c233417 12/19/2007 01:11 pm Iustin Pop

Make utils.RunCmd() deal with interleaved stdout/stderr

Currently, RunCmd is written with the assumption that programs will have
a small stderr output, therefore we read the child's stdout (which can
be big, so we don't want to block the child) and then the stderr (which...

3aecd2c7 11/29/2007 06:52 pm Iustin Pop

Replace hardcoded lock dir

This patch replaces the hardcoded ‘/var/lock/’ directory with one based on
LOCALSTATEDIR.

Reviewed-by: imsnah

f362096f 11/29/2007 06:03 pm Iustin Pop

Make utils.RunCmd log failures when using debug

This patch adds logging of command failures to the debug log in case the
user either started the command (gnt-*) or the node daemon with the
debug flag.

Reviewed-by: imsnah

f3299a07 11/22/2007 04:10 pm Michael Hanselmann

Sort list returned by ListVisibleFiles.

If we didn't sort this list, files would be in arbitrary order.

Reviewed-by: iustinp

7fbb1f65 11/20/2007 11:18 am Michael Hanselmann

Make sure aliases written to /etc/hosts are unique.

Reviewed-by: iustinp

f7414041 11/20/2007 11:17 am Michael Hanselmann

Add function to return list with unique elements.

Reviewed-by: ultrotter

7e3dbb94 11/13/2007 01:11 am Iustin Pop

Fix another breakage in SetEtcHostsEntry

The code assumes all lines have at least two elements which are
whitespace separated - i.e. it does not deal with empty lines or comment
lines with no spaces. This patch fixes this.

Also, the patch replaces the blank between IP and the canonical name...

2e3e75b7 11/12/2007 07:10 pm Michael Hanselmann

Call fsync() after modifying /etc/hosts.

This is a critical file. Breaking it would be very bad. Thus, if the system
crashes before the data is synched to the disk, it should have the complete
file afterwards.

Reviewed-by: iustinp

9440aeab 11/12/2007 07:09 pm Michael Hanselmann

Fix functions for /etc/hosts.

- Combine hostname and aliases on one line
- Fix bug with wrongfully removed newline characters
- Use wrapper for SetEtcHostsEntry in cmdlib

Reviewed-by: iustin

424908c1 11/12/2007 07:03 pm Michael Hanselmann

Rollback commit 356.

2d6cfa31 11/12/2007 06:44 pm Michael Hanselmann

Add functions for known_hosts files.

Reviewed-by: TODO

3e1cdf9f 11/12/2007 12:57 pm Michael Hanselmann

Add missing descriptions to {Add,Remove}EtcHostsEntry.

Reviewed-by: schreiberal

c8a0948f 11/09/2007 04:54 pm Michael Hanselmann

Use new functions to modify /etc/hosts.

Reviewed-by: schreiberal

899d2a81 11/09/2007 04:53 pm Michael Hanselmann

Add functions to modify /etc/hosts.

Reviewed-by: schreiberal

59f82e3f 11/09/2007 04:52 pm Michael Hanselmann

Cleanup temporary file in RemoveAuthorizedKey.

Reviewed-by: schreiberal

65fe4693 11/05/2007 05:06 pm Iustin Pop

Miscellaneous style fixes

This patch fixes some minor pylint warnings (unused variables, wrong
indentation, etc.) and a real bug in the recovery for drbd8 rename
procedure.

Reviewed-by: imsnah

24818e8f 11/05/2007 02:14 pm Michael Hanselmann

Rename utils.GetUUID to utils.NewUUID.

Reviewed-by: schreiberal

78feb6fb 10/28/2007 04:32 pm Guido Trotter

Import two itertools recipes

The two function 'any' and 'all' are copied as-is from the python 2.4
documentation for the itertools module. They are useful (and are already
builtin function in python 2.5).

Reviewed-by: iustinp

087b34fe 10/12/2007 04:02 pm Iustin Pop

Add a generic write file function

Currently there are a few version of "write a file safely" in the code.
This patch adds a generic function that should be able to replace all
the other versions.

The "take attributes from the target file" option, while useful, is not...

257f4c0a 10/12/2007 03:27 pm Iustin Pop

Enhance GetHomeDir to accept either names or UIDs

Currently GetHomeDir accepts UIDs only. Enhance it to accept either a
user name or a user id, to allow for nicer usage.

Reviewed-by: imsnah

59072e7e 10/11/2007 03:04 pm Michael Hanselmann

Use the kernel's ability to generate UUIDs.

This removes the dependency on either the uuid module or e2fsprogs' uuidgen.

Reviewed-by: iustinp

2f8b60b3 10/11/2007 11:02 am Iustin Pop

Add small function to read the homedir of a user

This can be used to replace hardcoded "/root/" paths.

Reviewed-by: imsnah

16abfbc2 10/10/2007 08:20 pm Alexander Schreiber

Remove fping as a dependency for Ganeti.

This patch completely gets rid of fping
- replace all fping invocations with TcpPing calls
- update documentation accordingly.
- associated cleanups (use constant for localhost IP, use more sensible
defaults for TcpPing and use those)...

2f31098c 10/10/2007 01:00 pm Iustin Pop

Remove the shebang from modules

Since modules are not directly executables, remove the shebang from
them. This helps with lintian warnings.

Also make the autogenerated _autoconf.py contain two comment lines at
the beginning, like the other modules.

Reviewed-by: ultrotter

eedbda4b 10/08/2007 11:59 am Michael Hanselmann

Add function to list files in a directory, excluding those beginning with a
dot.

Reviewed-by: iustinp

081b1e69 09/27/2007 04:03 pm Michael Hanselmann

Prevent race condition in CreateBackup().

Reviewed-by: ultrotter

89e1fc26 09/21/2007 04:37 pm Iustin Pop

Remove requirement that host names are FQDN

We currently require that hostnames are FQDN not short names
(node1.example.com instead of node1). We can allow short names as long
as:
- we always resolve the names as returned by socket.gethostname()
- we rely on having a working resolver...

bcf043c9 09/19/2007 09:56 am Iustin Pop

Change resolved hostname from dict to a class

The current result of utils.LookupHostname() is a dict, but this does
not allow static checkers to check the correctness of the code. This
patch introduces a new class names HostInfo and changes LookupHostname...

4ca1b175 09/17/2007 05:04 pm Alexander Schreiber

Cleanup up selective imports for utils and utils unit test.

Reviewed-by: iustinp, imsnah

2c30e9d7 09/17/2007 02:18 pm Alexander Schreiber

Added TcpPing to do ping-alike via TCP connect(2) with defined source address.
To be used to replace the currently fping(8) based reachability test.

Reviewed-by: imsnah

23f41a3e 09/12/2007 03:02 pm Michael Hanselmann

Change the way how locales are overridden:
- Set only LC_ALL, it overrides any other variable
- Restore environment after unittest

Reviewed-by: iustinp

f6441c7c 09/11/2007 03:18 pm Iustin Pop

Filters out LANG and LC_* on RunCmd

This filters out the LANG and LC_ variables from the environment of programs
executed by RunCmd, in order to get consistent output of execute programs like
lvs, etc.

Reviewed-by: imsnah

88d14415 09/05/2007 08:14 pm Michael Hanselmann

Add function to get all local IP addresses, will be used in cmdlib.py.

Reviewed-by: schreiberal

113b55aa 09/03/2007 02:22 pm Iustin Pop

Switch utils.RunCmd from popen2 to subprocess

This changes the implementation of RunCmd from using the popen2 module
to using the (new in 2.4) subprocess module.

This is helpful because the subprocess module has more advanced features
than popen2, the most important ones being the ability to run code in the...

3ecf6786 08/14/2007 06:17 pm Iustin Pop

Style changes for pep-8 and python-3000 compliance.

This changes the raising of exceptions from:
raise Exception, value
to
raise Exception(value)

as the first form will be removed in python-3000 and the second form is
preferred now.

The changes also involve a few cases of changing from raising standard...

098c0958 07/26/2007 02:40 pm Michael Hanselmann

Comment formatting updates.

Reviewed-by: iustinp

a8083063 07/16/2007 04:39 pm Iustin Pop

Initial commit.