History | View | Annotate | Download (109.1 kB)
Fix small typo in a docstring
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
Fix rbd showmapped output parsing
'rbd showmapped' output formatting differs between older and newer versions ofthe ceph tools. Try to use json output formatting if available (currentlyavailable only in the ceph master branch). For bobtail, argonaut and older...
bdev.GetPVInfo() returns list of LVs
This will be used for checks related to exclusive storage.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
utils.LvmExclusiveCheckNodePvs() returns more info
utils.LvmExclusiveCheckNodePvs() now returns the size of the smallest andbiggest PVs of a node. They will be used for cross-node consistency checks.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>...
Unit tests for bdev and utils.lvm
Also amended a comment of a tested method.The tests are for methods introduced in commit 63c73073 (LVM disk creationuses dedicated PVs).
Free disk space reporting follows exclusive storage rules
When exclusive_storage is set, nodes report only the space that could beused to create new disk according to the rules set in thedesign-partitioned doc (i.e., non-empty disks are ignored and the space...
Add exclusive_storage parameter to node_info RPC
The flag is passed to the low-level functions that need it, but it's notyet used. Also, RPCs get just a default value for now.
Signed-off-by: Bernardo Dal Seno <bdalseno@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
LVM disk creation uses dedicated PVs
When exclusive_storage is set, PVs are allocated according to thedesign-partitioned design doc.
Added class to contain information about a PV
This makes the code more readable and easier to upgrade.bdev.LogicalVolume.GetPVInfo and the code that depends on it have beenrefactored to use the new class.
utils.CheckVolumeGroupSize() has been moved to lib/utils/lvm.py, where more...
Add exclusive_storage parameter to blockdev_create RPC
The flag is passed to the low-level functions that need it, but it's notyet used.
Multiple ExtStorage Providers and ext-params
Add support for passing parameters to the ext template (ext-params).Take advantage of disk-params, that don't seem to make much sense inthis template (ExtStorage Providers are not predefined and we don'tknow their needs) and use them to pass the ext-params dynamically to...
Implement the External Storage Interface
With this commit we introduce the External Storage Interfaceto Ganeti, abbreviated: ExtStorage Interface.
The ExtStorage Interface provides Ganeti with the ability to interactwith externally connected shared storage pools, visible by all...
Replace frozenset with compat.UniqueFrozenset
This is not a trivial s/frozenset/compat.UniqueFrozenset/, but ratheronly replaces “frozenset” where appropriate. Most of the places are“static” information that doesn't change after the module has beenloaded....
Move the path of the DRBD status file to the Constants file
It will be needed by the DRBD data collector, that will be added shortly.
Signed-off-by: Michele Tartara <mtartara@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
backend: Switch to new file storage directory verification
The configuration is no longer used for verifying file storage paths.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
bdev: Remove unused import of itertools
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Michele Tartara <mtartara@google.com>
bdev: Add verification for file storage paths
An earlier version of this patch series verified all paths in cmdlib inthe master daemon. With this change all that verification code is movedto bdev to run inside the node daemon. The checks are much stricter...
LVM: remove old tags when adding new ones
This patch adds a small helper function to clear an LV's tags, andcalls it at SetInfo time. We need this to be able to correctly trackinstance renames, once we will call SetInfo at such times.
Signed-off-by: Iustin Pop <iustin@google.com>...
Add a small bdev helper function
I wanted to write that snippet the third time, which is too much :)
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
bdev: Remove unnecessary empty line
My local pylint didn't complain.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Bernardo Dal Seno <bdalseno@google.com>
bdev: Add functions to verify file storage paths
- LoadAllowedFileStoragePaths: Loads a list of allowed file storage paths from a file- CheckFileStoragePath: Checks a path against the list of allowed paths
The unit test for “utils.IsBelowDir” is updated with cases which weren't...
Bump pep8 version to 1.2
Debian Wheezy will ship with this version, and it has many improved checks compared to 0.6, so let's:
- bump version in the docs- silence some new checks that are wrong due to our indent=2 instead of 4- fix lots of errors in the code where the indentation was wrong by 1...
Merge branch 'stable-2.6'
Fix a docstring in bdev's DRBD8 class
It seems this was not updated since the move to static minors…
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Agata Murawska <agatamurawska@google.com>
Fix DRBD resize code
There are two bugs in the current resize code, affecting mostly DRBD.
First, due to bugs in old DRBD versions (pre 8.0.14), the code currentlycalls `drbdsetup resize' on both the primary or secondary. However,this is actually wrong per current DRBD (from drbdsetup(8)):...
bdev: Fix DRBD error reporting for syncer rate
The function is supposed to return a list, not a string. Without thispatch the error message ends up as “C, a, n, ', t, , c, h, a, n, g, e,…”.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Iustin Pop <iustin@google.com>...
Fixed typos, formatting, etc and added a comment
This patch only touches comments and man pages.
bdev: Add a guard to verify disk parameters are set
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
bdev: Take the params directly from disk without refill
Merge branch 'devel-2.5'
Fix LV status parsing to accept newer LVM
LVM version 2.02.93 (or at least, sometimes after .88) has extend thelv_attr field with two more flag; we only care about the first digit,so let's change the "!= 6" check to "< 6".
Thanks to Robin H Johnson <robbat2@gentoo.org> for finding this issue....
Replace single- with double-quotes
In at least two cases "%s" is replaced with str(), too.
Implement rbd disk template
Introduce the rbd disk template, which handles provisioning andmanagement of instance disks as block devices mapped to rbd volumeson a RADOS cluster.
The rbd template is of type DTS_EXT_MIRROR, meaning rbd-basedinstances can failover or migrate to any node, without...
Add the remaining DRBD dynamic sync disk params
Add the c-plan-ahead, c-fill-target, c-delay-target, c-max-rate,c-min-rate parameters; report errors while setting thesynchronization speed.
Change the error signaling model of SetSyncParams. Instead of returning...
Add the dynamic-resync DRBD disk parameter
constants.py, cmdlib.py:- add the dynamic-resync parameter, both at DT and LD levels;
lib/bdev.py:- change SetSyncSpeed to SetSyncParams, and _SetMinorSyncSpeed to _SetMinorSyncParams;- use the dynamic-resync parameter....
Add net-custom and disk-custom DRBD parameters
Those parameters can be used to pass options directly to drbdsetup diskand drbdsetup net.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Iustin Pop <iustin@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Move LD parameters constants to the LDP_ namespace
Add the LDP_ prefix to the LD parameters-related constants, in order toavoid pollution in the global constants namespace.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Add DRBD barriers disk parameters
Add the disk-barriers and meta-barriers parameters described in thedesign doc.
LV stripes parameters for plain and drbd
Add DRBD8 static resync speed disk parameter
Use disk parameters in noded
Set DRBD sync speed in DRBD8.Assemble
Instead of relying on clients of the class for setting the device speed(and, in general, the DRBD parameters), move this responsibility insidethe Assemble method.
Signed-off-by: Andrea Spadaccini <spadaccio@google.com>...
DeprecationWarning fixes for pylint
In version 0.21, pylint unified all the disable-* (and enable-*)directives to disable (resp. enable). This leads to a lot ofDeprecationWarning being emitted even if one uses the recommendedversion of pylint (0.21.1, as stated in devnotes.rst)....
Merge branch 'devel-2.4' into devel-2.5
Conflicts: NEWS (trivial) configure.ac (trivial) daemons/ensure-dirs.in (deleted)
Signed-off-by: René Nussbaumer <rn@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
PEP8 style fixes
Identified using the “pep8” utility.
Fix a parsing issue with DRBD 8.3.11 in the Linux Kernel
In the Linux kernel commit 4b0715f096 introduced a display bug into/proc/drbd which broke our regex.
The bug was first introduced into Linux 2.6.39-rc1. This bug is stillunfixed as of today.
This patch adapt the regular expression to workaround this bug for the...
Zero DRBD metadata before creation
The docstring of the DRBD8 class says:
… The meta device is checked for valid size and is zeroed on create.
which is not done today, hence we havehttp://code.google.com/p/ganeti/issues/detail?id=182:
node1# mkreiserfs -f /dev/xenvg/t8...
Most boring patch ever
s/'/"/ in (hopefully) the right places.
Implement dryrun mode for BlockDev.Grow()
This is always called with False from backend for now.
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Shared block storage support
This patch introduces basic shared block storage support.
It introduces a new storage backend, bdev.PersistentBlockDevice, touse as a backend for shared block storage. The new bdev requires a newBLOCKDEV_DRIVER_MANUAL constant with the value "manual" and uses it as...
Core shared file storage support
This patch introduces core file storage support, consisting of the following:
A configure-time switch for enabling/disabling shared file storagesupport and controlling the shared file storage location:--with-shared-file-storage-dir=. Shared file storage configuration is then...
Make it possible to pause/resume sync of mirrored dev
Make snapshots multi-vg aware
Currently, the Snapshot() function of LogicalVolume returns only thelogical volume path, with the assumption that we only have one VG. Butwith the recent changes, it makes more sense to return the full data (vgand lv) from it, so as to not require computing it in the master....
Migrate call from backend._GetVGInfo to bdev.LogicalVolume.GetVGInfo
This patch removes duplicate code found in backend which also needs toget VG infos. To make it simpler we moved to bdev.LogicalVolume.GetVGInfo.
Signed-off-by: René Nussbaumer <rn@google.com>...
Use free space in vg instead of biggest free pv space for a snapshot
Even for snapshot we looked at the biggest free pv space even thoughthe vg might have fit the snapshot we aborted if one of the pvs was toosmall. This patch fixes this by looking at the vg size instead of the pv...
Introduce new IPAddress classes
This patch unifies the netutils functions dealing with IP addresses tothree classes:- IPAddress: Common IP address functionality- IPv4Address: IPv4 specific functionality- IPv6address: IPv6-specific functionality
Furthermore it adds methods to check whether an address is a loopback...
Fix bug in bdev when drbd version format is x.x.x.x
This patch fixes a bug reported in [0]. Newer drbd versions can haveanother digit beyond the regular major, minor and point release digits.We modify the regex used to match that with an optional parts which is...
Introduce lib/netutils.py
This patch moves network utility functions to a dedicated module.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
BaseDRBD: provide a way to query usermode_helper parameter
Signed-off-by: Luca Bigliardi <shammash@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
DRBD IPv6 support
Support IPv6 configuration for 'drbdsetup show' parser and add unittestsconcerning that case. Renames some data files to use consistent namesclarifying their usage.
Signed-off-by: Manuel Franceschini <livewire@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Remove pred from compat.any/all
This makes it compatible with the python builtin, and we can even usethe builtin when running under the right version of python. The all andany functions are renamed to _all and _any, so that they can be tested,and (non)existing unittests are updated (translation: there are no unit...
Fix warnings with Python 2.6
'format' is a new built-in function, and 'bytes' is a new builtin type.We rename this to make pylint happy (and remove potential bugs).
Add separate module for backported language functionality
utils.py, where they were before, is already huge.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Guido Trotter <ultrotter@google.com>
Write grow support for file storage
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Iustin Pop <iustin@google.com>
Remove race condition in FileStorage.Create
Rather than checking that the file doesn't exist, and then creating it,we create it with O_CREAT | O_EXCL, making sure the checking/creation isatomic.
Signed-off-by: Guido Trotter <ultrotter@google.com>Reviewed-by: Michael Hanselmann <hansmi@google.com>
Clarify the error message for ':' in PV names
As described in issue 93, just saying ':' is not a valid char can beconfusing.
Add strict name validation for the LVM backend
Currently we don't enforce name validation for the LVM backend, on theidea that LVM itself will reject invalid names and we catch thoseerrors.
However, recent LVM documents the accepted VG/LV name space, so it's...
Implement disabling of file-based storage
Rationale: the file-based storage backend can add/remove files under acertain directory. However, the master node is also controlling thesetting of the file-based root directory, so basically it means we can'tprevent arbitrary modifications by the master of the node's filesystem....
Unify a few re.compile calls in DRBD
These are both cleanups and, in the case of _MassageProcData, switchingfrom a weaker RE to a stronger one (we now need cs: in the line,previosuly any line starting with \d+: was accepted).
Merge branch 'devel-2.0' into devel-2.1
Conflicts: lib/backend.py - trivial merge...
Add a crude disable for DRBD barriers
Ideally we want to/will have per-device DRBD controls of disk/metadataflushes. In the meantime, we want at least a disable of the barrierfunctionality for cases where one has battery-backed caches.
Background: DRBD has four mechanism of handling ordered disk-writes....
Ensure all int/float conversions are handled right
int()/float() can raise either ValueError (in case of int("a")), orTypeError (in case of int(None)). We had many bugs over time due tothis, and a recent one was just diagnosed, so we go over the codebase...
bdev: Add a TODO and a pylint silence
A piece of old code in bdev.py uses a for loop over a single variablebecause we can 'break' out of the loop or exit on the 'else' path. Thisis not a nice usage of the for loop, it should be converted to astandard if...elif...else structure....
Add some stubs to bdev.FileStorage
This patch adds explicit errors (instead of notimplemented) inFileStorage (and the associated TODOs).
Signed-off-by: Iustin Pop <iustin@google.com>Reviewed-by: Olivier Tharan <olive@google.com>
Remove many 'Unused variable' warnings
Note there are some cases left which need extra cleanup.
Add targetted pylint disables
This patch adds targeted pylint disables, where it makes sense (eitherdue to limitations in pylint or due to historical usage), and also a fewblanket ones in rapi where all the names are… “different”.
DRBD: ignore unreadable meta devices
The DRBD driver can ignore dead disks but not dead meta devices (forwhich it refuses to configure the minor). To handle this case, we checkthat the meta device is readable and if not we ignore it (the same aswhen backend._RecursiveAssembleBD didn't find it)....
Another round of pylint-related style fixes
A newer version of pylint, more warnings…
Throw specific error when ':' exists in PV names
While ':' is not actually a supporte character in PV names (it has aspecial meaning for commands like lvcreate), we should throw specificerrors for this case instead of generic “Can't create LV”.
This patch does two things:...
Change bdev.LogicalVolume.GetPVInfo usage
We will need to enumerate selectively the PVs of (possible) many VGs andnot only the allocatable ones. For this we make the VG selection and theallocatable filtering optional. The two callers are modified for this...
bdev: Convert to utils.Retry
Also replaces a hardcoded limit of 15 seconds with 1/4of NET_RECONFIG_TIMEOUT.
Merge remote branch 'origin/master' into mogu
Merge commit 'origin/next'
Use ReadFile/WriteFile in more places
This survived QA, burnin and unittests.
Signed-off-by: Michael Hanselmann <hansmi@google.com>Reviewed-by: Luca Bigliardi <shammash@google.com>
Fix /proc/drbd parsing in presence of gaps
In case there are gaps in /proc/drbd related to the minors sequence,this can lead to empty lines (see the attached data file). In this case,we currently fail to parse the file correctly.
The patch fixes this by skipping empty lines in...
Merge commit 'origin/next' into branch-2.1
Merge branch 'master' into next
Convert ldisk_degraded to tri-state value
This allows us to report “uncertain” states (LDS_UNKNOWN) for caseswhere the code can't easily detect or report what's wrong with ablock device.
Use object for blockdev_find RPC call result
This patch changes the return type for backend.BlockdevFind to an object(objects.BlockDevStatus). Before a tuple was used. Adding more values tothis tuple causes a lot of work. Converting the result to an object with...
bdev: Add function for reading actual disk size
This patch adds a GetActualSize for block devices that returns theactual disk size. It is done using blockdev (and stat for file storage).
While this could be done via reading /sys/block/N/size, that is not as...
bdev: allow ignoring of size in Assemble()
This patch changes the DRBD8 class (the only one to use the size inAssemble) to ignore the size in Assemble when a zero size is passed.This will allow activation of disks even when the size recorded in theconfiguration is wrong....
Use full-stripe size in LVM growth
LVM has issues when growing stripped volumes, so it's best to specifythe growth in exact multiples of the full stripe size (as precise aspossible). For this we need to do a couple of changes: - in LVM Attach(), we query additionally the VG extent size and the LV...
Fix some typos
Increase maximum accepted size for a DRBD meta dev
With the change to stripped LVs, the actual size of a meta device (whichis small) can be more than we expected (for non-stripped LVs). Thispatch increases from 160MB to 1GB the accepted size, and updates the...
Enable stripped LVs
This patch enables stripped LVs, falling back to non-stripped if thestripped creation fails. If the configure-time lvm-stripecount is 1,this patch becomes a noop (with an insignificant python-level overhead,but no extra lvm calls)....
Add more constants for DRBD and change sync tests
This patch adds constants for the connection status, peer roles and diskstatus, and it changes the rules for when the disk is considered as“resyncing” - previously it was only for syncsource/synctarget, but...
Assemble DRBD using the known size
This patch changes DRBD disk attachment to force the wanted size, as opposed toletting the device auto-discover its size.
This should make the disks more resilient with regard to small differences insize (e.g. due to LVM rounding). This still works with regard to disk...