Synnefo » snf-ganeti

Fix utils.MatchNameComponent for full matches

While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,
exact match and we should return it if that is the case.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Even more style fixes

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

utils.CalculateDirectorySize: Don't redefine builtin

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

Remove watcher pause file 1h past end time

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

GenerateSecret: specify the length of the secret

Add a parameter to specify the length of the bytes sequence represented by the
returned hex string. Defaulting to 20 so it's compatible with previous SHA1
hash.

Signed-off-by: Luca Bigliardi <shammash@google.com>...

SignalHandler, only accept list of signals

utils.SignalHandler can accept either a list of signals or a single
signal, but it's always called with a list. Removing the single signal
option.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Add functions to calc directory size and free space on filesystem

These will be used by the new storage unit framework.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Merge branch 'next' into branch-2.1

• next:
  Create a new --no-voting option for masterfailover
  ganeti-masterd: allow non-interactive --no-voting
  Fix pylint warnings
  Add custom pylintrc
  bootstrap: Don't leak file descriptor when generating SSL certificate...

Fix some typos

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix various pylint warnings

There were multiple issues:
- copy-paste resulted in wrong indentation
- wrong function name
- missing spaces around assignment
- overriding built-in names (type, dir) or already defines ones
(errors, hypervisor)

Signed-off-by: Iustin Pop <iustin@google.com>...

Convert test_delay rpc to new style result

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

IsNormAbsPath and users, use "normalized" term

We used to refer to normalized paths as "normal" which might be
confusing. This fixes the syntax in all current IsNormAbsPath users and
in the docstring.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix the SafeEncoding behaviour

Currently we have bad behaviour in SafeEncode:
- binary strings are actually not handled correctly (ahem)
- the encoding is not stable, due to use of string_escape

For this reason, we replace the use of string_escape with part of the...

Avoid DeprecationWarning on Python >= 2.6

Python 2.6 complains about module 'sha' being deprecated. It makes
execution of Ganeti commands a bit annoying, and when you run
'ganeti-watcher' in cron jobs, you get a mail message after every
execution.

Tests pass under under Python 2.6 and Python 2.4....

Small optimisation in utils.WriteFile

Currently we always try to remove the new file, even if the rename
succeeded. This patch tracks the existence of the new file and doesn't
try to remove it if we managed to rename it.

Signed-off-by: Iustin Pop <iustin@google.com>...

Create runtime dir in bootstrap

Some hypervisors (KVM) need RUN_GANETI_DIR to exist even at cluster init
time. This patch creates it in InitCluster just before hv parameter
checking. Since the code to make list of directories is already repeated
twice in the code, and this would be the third time, we abstract it into...

Fix typos in utils.WriteFile's docstring

Reviewed-by: iustinp

Instance parameters: force typing

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types...

An attempt at fixing some encoding issues

This patch unifies the hardcoded re-encoding attempts into a single
function in utils.py. This function is used to take either an unicode or
str object and convert it to a ASCII-only str object which can be safely...

Update the logging output of job processing

(this is related to the master daemon log)

Currently it's not possible to follow (in the non-debug runs) the
logical execution thread of jobs. This is due to the fact that we don't
log the thread name (so we lose the association of log messages to jobs)...

Add a TailFile function

This patch adds a tail file function, to be used for parsing and returning in
the job log OS installation failures.

Reviewed-by: ultrotter

Add rename function automatically creating directories if needed

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't...

_RunCmdPipe: handle EINTR in poller.poll()

poll() can be interrupted. rather than failing we retry until it
returns.

Reviewed-by: iustinp

cleanup: SetEtcHostsEntry unused var

Reviewed-by: amishchenko

Add utils.CheckBEParams

This function will be used in LUCreateInstance, LUSetInstanceParams,
LUSetClusterParams and InitCluster to check the backend parameters
validity and convert the relevant values to integer, without duplicating
code. It lives in utils as bootstrap.py is calling it too....

Move FieldSet class to utils.py

Since we can use the FieldSet class in cli.py to nicely format disk
sizes and such, we move it to utils.py and also move its associated
unittest. I didn't remove the cmdlib.py unittest file as that's not the
good direction :)...

utils.SetupLogging: Remove previously setup handlers

If a logging function is called before the logging module is setup
using utils.SetupLogging, it calls logging.basicConfig, which adds
a StreamHandler, on its own. If we leave it in, all log output will...

Remove zombie special case from IsProcessAlive

Based on the discussion on the mailing list, we remove the special
casing of zombie processes.

Note we don't use kill -0 since that has a different meaning than "check
process is alive", so this function is still linux-specific....

Implement working directory for utils.RunCmd

In order to replace usage of the shell just for 'cd %s; ...' we need to
be able to specify the working directory for childs.

The patch also changes the default working directory (when not
specified) to "/", as opposed to the current dir; since the daemos run...

Modify utils.RunCmd to write output to file

Currently we launch processes via the shell in a few places only to
redirect standard output and error to a log file ("&> $file"). It is
better to do such redirection from within RunCmd itself.

This patch splits RunCmd in two parts, the setup and the execution part,...

Abstract checking own address into a function

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to...

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

utils.FileLock: Implement timeout

The timeout can be used in ganeti-noded to be more robust against
deadlocks.

Reviewed-by: iustinp

utils.SplitTime: Fix rounding of milliseconds

Reported by Iustin.

It used to return this:

utils.SplitTime(1234.999999999999)

(1234, 1000)

while it should've returned this:

utils.SplitTime(1234.999999999999)

(1235, 0)

Reviewed-by: ultrotter

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

Add FileLock utility class

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

Fix exception class name in utils.WritePidFile

Reviewed-by: iustinp

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp

Move watcher's LockFile function to utils

Reviewed-by: iustinp

Convert utils.py to the logging module

The patch also logs all commands executed from RunCmd when we are at
debug level.

Reviewed-by: imsnah

Add a FirstFree function to utils.py

This function will return the first unused integer based on a list of
used integers (e.g. [0, 1, 3] will return 2).

Reviewed-by: imsnah

Add more parameters to utils.WriteFile

- Make closing file optional: Required by ganeti-watcher to keep
file open after writing it. Changes return value of utils.WriteFile
if "close" parameter evaluates to True.
- Pre- and post-write functions: Can be used to lock files. This...

Move {Set,Remove}EtcHostsEntry wrappers to utils.py

This is required for the split of the cluster initialization code.

Reviewed-by: iustinp, ultrotter

Disable forking in the master daemon

This patch adds a mechanism to disable utils.RunCmd in selected
programs. This is needed in the master daemon unless we confirm
threading doesn't pose any problems.

This makes cluster init fail, but creating new trunk clusters is anyway...

Move the OS search code into an abstract function

Based on the previous OS search code changes, we can now move the OS
search code into a generic look-for-file function in utils.py. This
means that the allocator code can use the same function.

Reviewed-by: ultrotter

Move the daemonize function to utils.py

Currently, in ganeti-noded we have the createDaemon function. Since
we'll need the same in other daemons, we move this function to utils.py

With the move, a few changes were also done:
- change the name to Daemonize()...

Codestyle fixes: adding a few empty lines

Reviewed-by: ultrotter