Synnefo » snf-ganeti

Normalize MAC addresses to all lower.

This change will normalize the MAC to all lower after validation.

Signed-off-by: René Nussbaumer <rn@google.com>
Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Add targeted pylint disables

This patch should have only:

- pylint disables
- docstring changes
- whitespace changes

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Olivier Tharan <olive@google.com>

Merge branch 'stable-2.0' into stable-2.1

• stable-2.0:
  Update NEWS file and release Ganeti 2.0.5
  Security issue: add validation of script names
  Move the hooks file mask into constants.py
  Improve LUQueryNodes for lockless case
  Ship rapi.rst/rapi.html in the dist archive...

Simplify utils.ReadFile

The documentation for file objects' read method states that if the size
is "negative or ommitted", all data is read; thus we can simplify it to
have size=-1 by default and not have the if test.

Signed-off-by: Iustin Pop <iustin@google.com>...

Revert "Get rid of utils.CommaJoin"

This reverts commit 6915bc28fe053e92aa16cf2d974d205f1140219c based on thread on
ganeti-devel.

Conflicts:

lib/cmdlib.py (due to the error code classification, trivial)

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fix pylint 'E' (error) codes

This patch adds some silences and tweaks the code slightly so that
“pylint --rcfile pylintrc -e ganeti” doesn't give any errors.

The biggest change is in jqueue.py, the move of _RequireOpenQueue out of
the JobQueue class. Since that is actually a function and not a method...

Another round of pylint-related style fixes

A newer version of pylint, more warnings…

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Add generic retry loop function

There are quite a few retry loops with timeouts in Ganeti's
code. Duplicating code is not good, so this patch introduces
a new function named “utils.Retry” to remedy this situation.

Signed-off-by: Michael Hanselmann <hansmi@google.com>...

Get rid of utils.CommaJoin

- We never remember to use it (5 uses vs 21 " ,".join())
- It's longer to write than " ,".join()
- The added value of the apostrophe in the string is not very much

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Fix utils.MatchNameComponent for full matches

While ‘test1’ matches both ‘test1’ and ‘test1.example’, it has a full,
exact match and we should return it if that is the case.

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Michael Hanselmann <hansmi@google.com>

Even more style fixes

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

utils.CalculateDirectorySize: Don't redefine builtin

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Luca Bigliardi <shammash@google.com>

Remove watcher pause file 1h past end time

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

GenerateSecret: specify the length of the secret

Add a parameter to specify the length of the bytes sequence represented by the
returned hex string. Defaulting to 20 so it's compatible with previous SHA1
hash.

Signed-off-by: Luca Bigliardi <shammash@google.com>...

SignalHandler, only accept list of signals

utils.SignalHandler can accept either a list of signals or a single
signal, but it's always called with a list. Removing the single signal
option.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

utils: Add functions to calc directory size and free space on filesystem

These will be used by the new storage unit framework.

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

Merge branch 'next' into branch-2.1

• next:
  Create a new --no-voting option for masterfailover
  ganeti-masterd: allow non-interactive --no-voting
  Fix pylint warnings
  Add custom pylintrc
  bootstrap: Don't leak file descriptor when generating SSL certificate...

Fix some typos

Signed-off-by: Michael Hanselmann <hansmi@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix various pylint warnings

There were multiple issues:
- copy-paste resulted in wrong indentation
- wrong function name
- missing spaces around assignment
- overriding built-in names (type, dir) or already defines ones
(errors, hypervisor)

Signed-off-by: Iustin Pop <iustin@google.com>...

Convert test_delay rpc to new style result

Signed-off-by: Iustin Pop <iustin@google.com>
Reviewed-by: Guido Trotter <ultrotter@google.com>

IsNormAbsPath and users, use "normalized" term

We used to refer to normalized paths as "normal" which might be
confusing. This fixes the syntax in all current IsNormAbsPath users and
in the docstring.

Signed-off-by: Guido Trotter <ultrotter@google.com>
Reviewed-by: Iustin Pop <iustin@google.com>

Fix the SafeEncoding behaviour

Currently we have bad behaviour in SafeEncode:
- binary strings are actually not handled correctly (ahem)
- the encoding is not stable, due to use of string_escape

For this reason, we replace the use of string_escape with part of the...

Avoid DeprecationWarning on Python >= 2.6

Python 2.6 complains about module 'sha' being deprecated. It makes
execution of Ganeti commands a bit annoying, and when you run
'ganeti-watcher' in cron jobs, you get a mail message after every
execution.

Tests pass under under Python 2.6 and Python 2.4....

Small optimisation in utils.WriteFile

Currently we always try to remove the new file, even if the rename
succeeded. This patch tracks the existence of the new file and doesn't
try to remove it if we managed to rename it.

Signed-off-by: Iustin Pop <iustin@google.com>...

Create runtime dir in bootstrap

Some hypervisors (KVM) need RUN_GANETI_DIR to exist even at cluster init
time. This patch creates it in InitCluster just before hv parameter
checking. Since the code to make list of directories is already repeated
twice in the code, and this would be the third time, we abstract it into...

Fix typos in utils.WriteFile's docstring

Reviewed-by: iustinp

Instance parameters: force typing

We want all the hv/be parameters to have a known type, rather than a
random mix of empty string, boolean values, and None, so we declare the
type of each variable and we enforce/convert it.

- Add some new constants for enforceable value types...

An attempt at fixing some encoding issues

This patch unifies the hardcoded re-encoding attempts into a single
function in utils.py. This function is used to take either an unicode or
str object and convert it to a ASCII-only str object which can be safely...

Update the logging output of job processing

(this is related to the master daemon log)

Currently it's not possible to follow (in the non-debug runs) the
logical execution thread of jobs. This is due to the fact that we don't
log the thread name (so we lose the association of log messages to jobs)...

Add a TailFile function

This patch adds a tail file function, to be used for parsing and returning in
the job log OS installation failures.

Reviewed-by: ultrotter

Add rename function automatically creating directories if needed

Unfortunately, os.makedirs in Python 2.4 is not safe against multiple
processes creating the same directory tree at the same time. This is
only fixed in Python 2.5 and up. Adding more checks in our code doesn't...

_RunCmdPipe: handle EINTR in poller.poll()

poll() can be interrupted. rather than failing we retry until it
returns.

Reviewed-by: iustinp

cleanup: SetEtcHostsEntry unused var

Reviewed-by: amishchenko

Add utils.CheckBEParams

This function will be used in LUCreateInstance, LUSetInstanceParams,
LUSetClusterParams and InitCluster to check the backend parameters
validity and convert the relevant values to integer, without duplicating
code. It lives in utils as bootstrap.py is calling it too....

Move FieldSet class to utils.py

Since we can use the FieldSet class in cli.py to nicely format disk
sizes and such, we move it to utils.py and also move its associated
unittest. I didn't remove the cmdlib.py unittest file as that's not the
good direction :)...

utils.SetupLogging: Remove previously setup handlers

If a logging function is called before the logging module is setup
using utils.SetupLogging, it calls logging.basicConfig, which adds
a StreamHandler, on its own. If we leave it in, all log output will...

Remove zombie special case from IsProcessAlive

Based on the discussion on the mailing list, we remove the special
casing of zombie processes.

Note we don't use kill -0 since that has a different meaning than "check
process is alive", so this function is still linux-specific....

Implement working directory for utils.RunCmd

In order to replace usage of the shell just for 'cd %s; ...' we need to
be able to specify the working directory for childs.

The patch also changes the default working directory (when not
specified) to "/", as opposed to the current dir; since the daemos run...

Modify utils.RunCmd to write output to file

Currently we launch processes via the shell in a few places only to
redirect standard output and error to a log file ("&> $file"). It is
better to do such redirection from within RunCmd itself.

This patch splits RunCmd in two parts, the setup and the execution part,...

Abstract checking own address into a function

Currently, we check if we have a given ip address (i.e. it's alive on
one of our interfaces) but manually calling TcpPing(source=localhost).
This works, but having it spread all over the code makes it hard to...

Move the pseudo-secret generation to utils.py

The bootstrap code needs a pseudo-secret and this is currently generated
inside the InitGanetiServerSetup function. Since more users will need
this, move it to utils.py

Reviewed-by: ultrotter

utils.FileLock: Implement timeout

The timeout can be used in ganeti-noded to be more robust against
deadlocks.

Reviewed-by: iustinp

utils.SplitTime: Fix rounding of milliseconds

Reported by Iustin.

It used to return this:

utils.SplitTime(1234.999999999999)

(1234, 1000)

while it should've returned this:

utils.SplitTime(1234.999999999999)

(1235, 0)

Reviewed-by: ultrotter

RunCmd: add optional environment overriding

If the user passes an env dict to RunCmd we'll override the environment
passed to the to-be-executed command with the values in the dict. This
allows us to pass arbitrary environment values to commands we run.
...

Add functions to split time into tuple and merge it back

These will be used for job logs.

Reviewed-by: ultrotter

Add FileLock utility class

This class is a wrapper around fcntl.flock and abstracts opening and
closing the lockfile. It'll used for the job queue.

(The patch also removes a duplicate import of tempfile into the unittest)

Reviewed-by: iustinp

Fix pylint-detected issues

This is mostly:
- whitespace fix (space at EOL in some files, not all, broken
indentation, etc)
- variable names overriding others (one is a real bug in there)
- too-long-lines
- cleanup of most unused imports (not all)...

Add a KillProcess function

We cannot depend on all environments to have a start-stop-daemon or
similar tool. We instead implement a KillProcess function that behaves
similar to “start-stop-daemon --retry”.

Note that the attached unittest can hang in foreground if the child...

Fix exception class name in utils.WritePidFile

Reviewed-by: iustinp

Add utils.IsPidFileAlive function

This helper function reads a pid from a file containing it and checks
whether it refers to a live process.

Reviewed-by: iustinp