Statistics
| Branch: | Tag: | Revision:

root / lib / utils.py @ 5af3da74

History | View | Annotate | Download (55.9 kB)

# Date Author Comment
17c61836 05/22/2009 06:03 pm Guido Trotter

IsNormAbsPath and users, use "normalized" term

We used to refer to normalized paths as "normal" which might be
confusing. This fixes the syntax in all current IsNormAbsPath users and
in the docstring.

Signed-off-by: Guido Trotter <>
Reviewed-by: Iustin Pop <>

da961187 05/22/2009 03:58 pm Guido Trotter

Add utils.IsNormAbsPath function

Currently most of the time we check for absolute path, but that doesn't
protect us from some invalid paths. In some places we should be more
strict, and this function should help us to.

Signed-off-by: Guido Trotter <>...

835528af 05/19/2009 04:21 pm Iustin Pop

Move more hypervisor strings into constants

This patch adds constants for the mouse and boot order strings; while
there are still some issues remaining, we're trying to cleanup hardcoded
strings from the hypervisors.

Since the formatting of frozensets is currently wrong, we also add an...

7ffe8fba 05/05/2009 06:06 pm Carlos Valiente

Avoid DeprecationWarning on Python >= 2.6

Python 2.6 complains about module 'sha' being deprecated. It makes
execution of Ganeti commands a bit annoying, and when you run
'ganeti-watcher' in cron jobs, you get a mail message after every
execution.

Tests pass under under Python 2.6 and Python 2.4....

c803b052 05/05/2009 02:00 pm Iustin Pop

RunCmd: log command line for missing cmd case

In case of missing programs, currently utils.RunCmd doesn't show any
information to help debugging, only 'No such file or directory'. This
patch adds error handling for the ENOENT case such that at least we have...

81b7354c 05/04/2009 04:51 pm Iustin Pop

Small optimisation in utils.WriteFile

Currently we always try to remove the new file, even if the rename
succeeded. This patch tracks the existence of the new file and doesn't
try to remove it if we managed to rename it.

Signed-off-by: Iustin Pop <>...

1b2c8f85 03/02/2009 02:18 pm Iustin Pop

Fix a bug in utils.EnsureDirs

This fixes a bug introduced in rev 2562 and also fixes the indentation.

Reviewed-by: ultrotter

9dae41ad 02/27/2009 07:08 pm Guido Trotter

Create runtime dir in bootstrap

Some hypervisors (KVM) need RUN_GANETI_DIR to exist even at cluster init
time. This patch creates it in InitCluster just before hv parameter
checking. Since the code to make list of directories is already repeated
twice in the code, and this would be the third time, we abstract it into...

5fcc718f 02/27/2009 03:06 pm Iustin Pop

Fix some epydoc style issues

99% of the epydoc return tags are "@return:", but each of the modified files
had one "@returns:" line. We fix this for consistency.

Reviewed-by: imsnah

69efe319 02/26/2009 02:32 pm Michael Hanselmann

Fix typos in utils.WriteFile's docstring

Reviewed-by: iustinp

9b977740 02/12/2009 11:15 am Guido Trotter

Apply the right permissions to /etc/hosts

In the current Ganeti version when modifying /etc/hosts we mistakenly
give it the permissions of the temporary file we create to define its
content, which is by default 0600. This breaks most non-root
applications, and thus must be corrected. This patch forces the mode to...

a5728081 02/10/2009 05:06 pm Guido Trotter

Instance parameters: force typing

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types...

1de62f37 02/03/2009 06:04 pm Guido Trotter

Correct a typo in ReadPidFile's docstring

Reviewed-by: iustinp

26f15862 02/03/2009 04:45 pm Iustin Pop

An attempt at fixing some encoding issues

This patch unifies the hardcoded re-encoding attempts into a single
function in utils.py. This function is used to take either an unicode or
str object and convert it to a ASCII-only str object which can be safely...

9939547b 01/23/2009 03:33 pm Iustin Pop

Fix batcher for 2.0-style disks and nics

This patch fixes the gnt-instance batch-create command, and in doing so
also slightly changes two other functions:
- we change utils.ParseUnit so that it accepts integer values also
(both ParseUnit(5) and ParseUnit("5") return the same value)...

d21d09d6 01/20/2009 07:19 pm Iustin Pop

Update the logging output of job processing

(this is related to the master daemon log)

Currently it's not possible to follow (in the non-debug runs) the
logical execution thread of jobs. This is due to the fact that we don't
log the thread name (so we lose the association of log messages to jobs)...

ae59efea 01/20/2009 06:47 pm Michael Hanselmann

.gitignore: Don't exclude whole /autotools/ dir, but only files

This way newly added files will be not be excluded by default. Fixes
also a small whitespace error in utils.py.

Reviewed-by: iustinp

f65f63ef 01/20/2009 01:18 pm Iustin Pop

Add a TailFile function

This patch adds a tail file function, to be used for parsing and returning in
the job log OS installation failures.

Reviewed-by: ultrotter

7d88772a 01/09/2009 02:52 pm Iustin Pop

Rework the daemonization sequence

The current fork+close fds sequence has deficiencies which are hard to
work around:
- logging can start logging before we fork (e.g. if we need to emit
messages related to master checking), and thus use FDs which we...

6e797216 12/18/2008 06:23 pm Michael Hanselmann

Add rename function automatically creating directories if needed

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't...

7167159a 12/17/2008 01:24 pm Michael Hanselmann

utils.KillProcess: Use waitpid() to wait for child processes

Sometimes the proc filesystem doesn't reflect the current status of
a process. By calling waitpid(), we make sure to get the current
information, at least for child processes. The timeout is still...

bf988c29 12/16/2008 06:24 pm Guido Trotter

_RunCmdPipe: handle EINTR in poller.poll()

poll() can be interrupted. rather than failing we retry until it
returns.

Reviewed-by: iustinp

0b5ad33e 12/14/2008 02:03 pm Iustin Pop

cleanup: TcpPing, wrong variable name

The default value of 'False' wasn't initialized properly. It doesn't
require initialization, but it's cleaner this way.

Reviewed-by: amishchenko

cd34e7bf 12/14/2008 02:03 pm Iustin Pop

cleanup: SetEtcHostsEntry unused var

Reviewed-by: amishchenko

90eb468c 12/03/2008 01:12 pm Guido Trotter

CheckBEParams handle a bool BE_AUTO_BALANCE

This only happens at cluster init, if the value is not user-specified.

Reviewed-by: imsnah

42ce2e13 12/02/2008 12:53 pm Guido Trotter

Add utils.CheckBEParams

This function will be used in LUCreateInstance, LUSetInstanceParams,
LUSetClusterParams and InitCluster to check the backend parameters
validity and convert the relevant values to integer, without duplicating
code. It lives in utils as bootstrap.py is calling it too....

9fbfbb7b 11/23/2008 05:34 pm Iustin Pop

Enable auto-unit formatting in script output

This patch enables by default the old 'human-readable' option, but in a
slightly different model.

The option is now called "units" and takes either:
- 'h' for automatic formatting
- 'm', 'g' or 't' for mebi/gibi/tebibytes...

a2d2e1a7 11/21/2008 05:35 pm Iustin Pop

Move FieldSet class to utils.py

Since we can use the FieldSet class in cli.py to nicely format disk
sizes and such, we move it to utils.py and also move its associated
unittest. I didn't remove the cmdlib.py unittest file as that's not the
good direction :)...

ec17d09c 11/21/2008 12:46 pm Michael Hanselmann

Get rid of node daemon password

With the new SSL client certificate stuff it's no longer needed.

Reviewed-by: iustinp

6346a9e5 11/12/2008 02:51 pm Michael Hanselmann

utils.SetupLogging: Remove previously setup handlers

If a logging function is called before the logging module is setup
using utils.SetupLogging, it calls logging.basicConfig, which adds
a StreamHandler, on its own. If we leave it in, all log output will...

e1bd0072 11/12/2008 05:43 am Iustin Pop

Fix utils.KillProcess

Rev 1978 introduced a breakage on the SIGKILL finall signal to the
process, due to mistyped variable.

Reviewed-by: francis.perron

44bf25ff 11/11/2008 12:58 pm Iustin Pop

Remove zombie special case from IsProcessAlive

Based on the discussion on the mailing list, we remove the special
casing of zombie processes.

Note we don't use kill -0 since that has a different meaning than "check
process is alive", so this function is still linux-specific....

ff5251bc 11/11/2008 12:58 pm Iustin Pop

Add cleanup of processes to utils.KillProcess

In case the process we want to signal is our own process, and the signal
we send is a deadly one, we should also cleanup after the process.

This patch adds a new parameter waitpid to this function that does this,...

8797df43 11/11/2008 05:12 am Iustin Pop

Implement working directory for utils.RunCmd

In order to replace usage of the shell just for 'cd %s; ...' we need to
be able to specify the working directory for childs.

The patch also changes the default working directory (when not
specified) to "/", as opposed to the current dir; since the daemos run...

58885d79 10/28/2008 01:20 am Iustin Pop

Documentation updates for utils.py

Reviewed-by: imsnah

36117c2b 10/24/2008 05:43 pm Iustin Pop

Modify utils.RunCmd to write output to file

Currently we launch processes via the shell in a few places only to
redirect standard output and error to a log file ("&> $file"). It is
better to do such redirection from within RunCmd itself.

This patch splits RunCmd in two parts, the setup and the execution part,...

82d9caef 10/20/2008 03:50 pm Iustin Pop

Remove the logger.py module

Since now we use only one function from the logger module
(SetupLogging), we move it to utils.py (which is already imported by all
users of this function), and we remove the module.

Reviewed-by: imsnah

caad16e2 10/12/2008 11:40 pm Iustin Pop

Abstract checking own address into a function

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to...

4a8b186a 10/01/2008 08:33 pm Michael Hanselmann

Move functions from ssconf.py elsewhere

These functions will be used to access config values instead of using
ssconf.

Reviewed-by: iustinp

33081d90 09/28/2008 05:44 pm Iustin Pop

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

ca0aa6d0 09/17/2008 07:07 pm Michael Hanselmann

Add utils.ReadFile function

It abstracts exception handling and is like a complement to
utils.WriteFile.

Reviewed-by: iustinp

aa74b828 09/05/2008 06:38 pm Michael Hanselmann

utils.FileLock: Implement timeout

The timeout can be used in ganeti-noded to be more robust against
deadlocks.

Reviewed-by: iustinp

45bc5e4a 09/05/2008 01:57 pm Michael Hanselmann

utils.SplitTime: More rounding fixes

SplitTime didn't round the same on different platforms. This patch changes
it to use microseconds and not care about rounding.

Reviewed-by: iustinp

b77ba978 08/29/2008 06:04 pm Michael Hanselmann

utils.SplitTime: Fix rounding of milliseconds

Reported by Iustin.

It used to return this:

utils.SplitTime(1234.999999999999)

(1234, 1000)

while it should've returned this:

utils.SplitTime(1234.999999999999)

(1235, 0)

Reviewed-by: ultrotter

e67bd559 08/26/2008 06:44 pm Michael Hanselmann

Add simple lock debug output

Currently it can only be enabled by modifying utils.py, but we can
add a command line parameter later if needed.

Reviewed-by: schreiberal

2557ff82 08/14/2008 01:26 pm Guido Trotter

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

e326d4e5 08/13/2008 05:23 pm Guido Trotter

Convert RunCmd to an epydoc docstring

Reviewed-by: imsnah

739be818 08/11/2008 07:26 pm Michael Hanselmann

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

aa65ed72 08/08/2008 01:02 pm Michael Hanselmann

Log only unexpected errors in utils.FileLock

Otherwise users might be confused by errors in log files.

Reviewed-by: iustinp

a87b4824 07/31/2008 05:33 pm Michael Hanselmann

Add FileLock utility class

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

8161a646 07/30/2008 06:00 pm Iustin Pop

Fix a misuse of exc_info in logging.info

This is my fault, sorry.

Reviewed-by: imsnah

38206f3c 07/30/2008 05:04 pm Iustin Pop

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

53beffbb 07/30/2008 11:34 am Iustin Pop

Expose utils.DaemonPidFileName

Since we need to compute this from outside utils.py, we change this to a
public function.

Reviewed-by: ultrotter

b2a1f511 07/29/2008 11:49 am Iustin Pop

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

d9f311d7 07/29/2008 11:49 am Iustin Pop

Change IsPidFileAlive into ReadPidFile

We already have a function to test if a PID is alive, so it makes more
sense to use function composition that force calling (since we need to
read PIDs from files in other places too). Now IsProcessAlive returns
False for PIDs <= 0, since this is the error return from ReadPidFile....

533bb4b1 07/28/2008 12:16 pm Michael Hanselmann

Fix exception class name in utils.WritePidFile

Reviewed-by: iustinp

b330ac0b 07/23/2008 05:23 pm Guido Trotter

Add utils.{Write,Remove}PidFile

WritePidFile is a helper function that writes the current pid in a
pidfile within the ganeti run directory. RemovePidFile tries to delete
it.

Reviewed-by: iustinp

fee80e90 07/23/2008 05:23 pm Guido Trotter

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp

de499029 07/21/2008 06:32 pm Michael Hanselmann

Add signal handler class

This signal handler class abstracts some of the code previously
used in other places. It also uninstalls its handler when Reset()
is called or the class is destructed, thereby restoring the
previous behaviour.

Reviewed-by: iustinp

eb0f0ce0 07/10/2008 03:38 pm Michael Hanselmann

Move watcher's LockFile function to utils

Reviewed-by: iustinp

7996a135 07/10/2008 03:17 pm Iustin Pop

Add a simple decorator for instance methods

This is just a simple, hardcoded decorator for object methods needing
synchronization on the _lock instance attribute.

Reviewed-by: ultrotter

bb698c1f 07/09/2008 01:41 pm Iustin Pop

Convert utils.py to the logging module

The patch also logs all commands executed from RunCmd when we are at
debug level.

Reviewed-by: imsnah

d4fa5c23 07/09/2008 01:41 pm Iustin Pop

Remove the old locking functions

This removes (hopefully) all traces of the old locking functions and
uses.

Reviewed-by: imsnah

7b4126b7 07/01/2008 02:44 pm Iustin Pop

Add a FirstFree function to utils.py

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

04a8d789 06/23/2008 12:52 pm Michael Hanselmann

utils.WriteFile: Remove optional check_abspath parameter

cfgupgrade will not work with relative paths at all, but rather get them
from constants.py.

Reviewed-by: iustinp

71714516 06/18/2008 03:30 pm Michael Hanselmann

Add more parameters to utils.WriteFile

- Make closing file optional: Required by ganeti-watcher to keep
file open after writing it. Changes return value of utils.WriteFile
if "close" parameter evaluates to True.
- Pre- and post-write functions: Can be used to lock files. This...

8d1a2a64 06/12/2008 04:04 pm Michael Hanselmann

Move cmdlib._HasValidVG to utils.CheckVolumeGroupSize

This is required for splitting the cluster initialization code.

Reviewed-by: iustinp

d9c02ca6 06/12/2008 04:04 pm Michael Hanselmann

Move {Set,Remove}EtcHostsEntry wrappers to utils.py

This is required for the split of the cluster initialization code.

Reviewed-by: iustinp, ultrotter

e3e66f02 05/09/2008 01:12 pm Michael Hanselmann

Remove utils.CheckDaemonAlive and use “xm info” instead

There are a couple of reasons for doing so:
- /proc is not OS independent, it's only supported by Linux (there are
emulations on other systems, but those might differ from the way
Linux represents data)....

b74159ee 04/29/2008 10:37 am Iustin Pop

Disable forking in the master daemon

This patch adds a mechanism to disable utils.RunCmd in selected
programs. This is needed in the master daemon unless we confirm
threading doesn't pose any problems.

This makes cluster init fail, but creating new trunk clusters is anyway...

70f4497c 04/25/2008 04:37 pm Michael Hanselmann

Extend utils.WriteFile

Add parameters to not check the path for absoluteness, implement a dry
run mode and automatically create a backup if requested. This will be
used by the cfgupgrade utility.

Reviewed-by: schreiberal

57c177af 04/10/2008 04:41 pm Iustin Pop

Move the OS search code into an abstract function

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

8ff612c2 04/04/2008 06:29 pm Iustin Pop

Allow utils.Daemonize() to not close some fds

This patch implements an optional parameter to utils.Daemonize() which
allows that function to not close some file descriptors.

This will allow the master daemon to open the listening socket before
fork - in order to be able to notify errors and return a meaningful exit...

8f765069 04/04/2008 03:02 pm Iustin Pop

Move the daemonize function to utils.py

Currently, in ganeti-noded we have the createDaemon function. Since
we'll need the same in other daemons, we move this function to utils.py

With the move, a few changes were also done:
- change the name to Daemonize()...

b15d625f 03/11/2008 09:44 am Iustin Pop

Modify utils.TcpPing to make source address optional

This patch modifies TcpPing and its callers to make the source address
selection optional. Usually, the kernel will know better what
source address to use, just in some cases we want to enforce a given...

7c0d6283 03/05/2008 04:39 pm Michael Hanselmann

Codestyle fixes: adding a few empty lines

Reviewed-by: ultrotter

06009e27 02/05/2008 04:12 pm Iustin Pop

Add a test opcode that sleeps for a given duration

This can be used for testing purposes.

Reviewed-by: ultrotter,imsnah

1862d460 12/27/2007 05:31 pm Alexander Schreiber

Allow instance MAC address to be set.

Allow the MAC address of an instance to be specified optionally during
instance creation and later to be changed via instance modify.

Reviewed-by: iustinp

9c233417 12/19/2007 01:11 pm Iustin Pop

Make utils.RunCmd() deal with interleaved stdout/stderr

Currently, RunCmd is written with the assumption that programs will have
a small stderr output, therefore we read the child's stdout (which can
be big, so we don't want to block the child) and then the stderr (which...

3aecd2c7 11/29/2007 06:52 pm Iustin Pop

Replace hardcoded lock dir

This patch replaces the hardcoded ‘/var/lock/’ directory with one based on
LOCALSTATEDIR.

Reviewed-by: imsnah

f362096f 11/29/2007 06:03 pm Iustin Pop

Make utils.RunCmd log failures when using debug

This patch adds logging of command failures to the debug log in case the
user either started the command (gnt-*) or the node daemon with the
debug flag.

Reviewed-by: imsnah

f3299a07 11/22/2007 04:10 pm Michael Hanselmann

Sort list returned by ListVisibleFiles.

If we didn't sort this list, files would be in arbitrary order.

Reviewed-by: iustinp

7fbb1f65 11/20/2007 11:18 am Michael Hanselmann

Make sure aliases written to /etc/hosts are unique.

Reviewed-by: iustinp

f7414041 11/20/2007 11:17 am Michael Hanselmann

Add function to return list with unique elements.

Reviewed-by: ultrotter

7e3dbb94 11/13/2007 01:11 am Iustin Pop

Fix another breakage in SetEtcHostsEntry

The code assumes all lines have at least two elements which are
whitespace separated - i.e. it does not deal with empty lines or comment
lines with no spaces. This patch fixes this.

Also, the patch replaces the blank between IP and the canonical name...

2e3e75b7 11/12/2007 07:10 pm Michael Hanselmann

Call fsync() after modifying /etc/hosts.

This is a critical file. Breaking it would be very bad. Thus, if the system
crashes before the data is synched to the disk, it should have the complete
file afterwards.

Reviewed-by: iustinp

9440aeab 11/12/2007 07:09 pm Michael Hanselmann

Fix functions for /etc/hosts.

- Combine hostname and aliases on one line
- Fix bug with wrongfully removed newline characters
- Use wrapper for SetEtcHostsEntry in cmdlib

Reviewed-by: iustin

424908c1 11/12/2007 07:03 pm Michael Hanselmann

Rollback commit 356.

2d6cfa31 11/12/2007 06:44 pm Michael Hanselmann

Add functions for known_hosts files.

Reviewed-by: TODO

3e1cdf9f 11/12/2007 12:57 pm Michael Hanselmann

Add missing descriptions to {Add,Remove}EtcHostsEntry.

Reviewed-by: schreiberal

c8a0948f 11/09/2007 04:54 pm Michael Hanselmann

Use new functions to modify /etc/hosts.

Reviewed-by: schreiberal

899d2a81 11/09/2007 04:53 pm Michael Hanselmann

Add functions to modify /etc/hosts.

Reviewed-by: schreiberal

59f82e3f 11/09/2007 04:52 pm Michael Hanselmann

Cleanup temporary file in RemoveAuthorizedKey.

Reviewed-by: schreiberal

65fe4693 11/05/2007 05:06 pm Iustin Pop

Miscellaneous style fixes

This patch fixes some minor pylint warnings (unused variables, wrong
indentation, etc.) and a real bug in the recovery for drbd8 rename
procedure.

Reviewed-by: imsnah

24818e8f 11/05/2007 02:14 pm Michael Hanselmann

Rename utils.GetUUID to utils.NewUUID.

Reviewed-by: schreiberal

78feb6fb 10/28/2007 04:32 pm Guido Trotter

Import two itertools recipes

The two function 'any' and 'all' are copied as-is from the python 2.4
documentation for the itertools module. They are useful (and are already
builtin function in python 2.5).

Reviewed-by: iustinp

087b34fe 10/12/2007 04:02 pm Iustin Pop

Add a generic write file function

Currently there are a few version of "write a file safely" in the code.
This patch adds a generic function that should be able to replace all
the other versions.

The "take attributes from the target file" option, while useful, is not...

257f4c0a 10/12/2007 03:27 pm Iustin Pop

Enhance GetHomeDir to accept either names or UIDs

Currently GetHomeDir accepts UIDs only. Enhance it to accept either a
user name or a user id, to allow for nicer usage.

Reviewed-by: imsnah

59072e7e 10/11/2007 03:04 pm Michael Hanselmann

Use the kernel's ability to generate UUIDs.

This removes the dependency on either the uuid module or e2fsprogs' uuidgen.

Reviewed-by: iustinp

2f8b60b3 10/11/2007 11:02 am Iustin Pop

Add small function to read the homedir of a user

This can be used to replace hardcoded "/root/" paths.

Reviewed-by: imsnah