From f9a1f3eb8544ca066f3a282cbe6ee2b55c603c30 Mon Sep 17 00:00:00 2001 From: Constantinos Venetsanopoulos Date: Mon, 2 Apr 2012 20:05:51 +0300 Subject: [PATCH] Update the shared storage design document Update the shared storage design document to reflect the current changes, after the implementation of the ExtStorage interface. Signed-off-by: Constantinos Venetsanopoulos --- doc/design-shared-storage.rst | 204 ++++++++++++++++++++++------------------- 1 file changed, 112 insertions(+), 92 deletions(-) diff --git a/doc/design-shared-storage.rst b/doc/design-shared-storage.rst index c175476..7080182 100644 --- a/doc/design-shared-storage.rst +++ b/doc/design-shared-storage.rst @@ -64,15 +64,11 @@ The design addresses the following procedures: filesystems. - Introduction of shared block device disk template with device adoption. +- Introduction of an External Storage Interface. Additionally, mid- to long-term goals include: - Support for external “storage pools”. -- Introduction of an interface for communicating with external scripts, - providing methods for the various stages of a block device's and - instance's life-cycle. In order to provide storage provisioning - capabilities for various SAN appliances, external helpers in the form - of a “storage driver” will be possibly introduced as well. Refactoring of all code referring to constants.DTS_NET_MIRROR ============================================================= @@ -159,6 +155,104 @@ The shared block device template will make the following assumptions: - The device will be available with the same path under all nodes in the node group. +Introduction of an External Storage Interface +============================================== +Overview +-------- + +To extend the shared block storage template and give Ganeti the ability +to control and manipulate external storage (provisioning, removal, +growing, etc.) we need a more generic approach. The generic method for +supporting external shared storage in Ganeti will be to have an +ExtStorage provider for each external shared storage hardware type. The +ExtStorage provider will be a set of files (executable scripts and text +files), contained inside a directory which will be named after the +provider. This directory must be present across all nodes of a nodegroup +(Ganeti doesn't replicate it), in order for the provider to be usable by +Ganeti for this nodegroup (valid). The external shared storage hardware +should also be accessible by all nodes of this nodegroup too. + +An “ExtStorage provider” will have to provide the following methods: + +- Create a disk +- Remove a disk +- Grow a disk +- Attach a disk to a given node +- Detach a disk from a given node +- Verify its supported parameters + +The proposed ExtStorage interface borrows heavily from the OS +interface and follows a one-script-per-function approach. An ExtStorage +provider is expected to provide the following scripts: + +- `create` +- `remove` +- `grow` +- `attach` +- `detach` +- `verify` + +All scripts will be called with no arguments and get their input via +environment variables. A common set of variables will be exported for +all commands, and some of them might have extra ones. + +- `VOL_NAME`: The name of the volume. This is unique for Ganeti and it + uses it to refer to a specific volume inside the external storage. +- `VOL_SIZE`: The volume's size in mebibytes. +- `VOL_NEW_SIZE`: Available only to the `grow` script. It declares the + new size of the volume after grow (in mebibytes). +- `EXTP_name`: ExtStorage parameter, where `name` is the parameter in + upper-case (same as OS interface's `OSP_*` parameters). + +All scripts except `attach` should return 0 on success and non-zero on +error, accompanied by an appropriate error message on stderr. The +`attach` script should return a string on stdout on success, which is +the block device's full path, after it has been successfully attached to +the host node. On error it should return non-zero. + +Implementation +-------------- + +To support the ExtStorage interface, we will introduce a new disk +template called `ext`. This template will implement the existing Ganeti +disk interface in `lib/bdev.py` (create, remove, attach, assemble, +shutdown, grow), and will simultaneously pass control to the external +scripts to actually handle the above actions. The `ext` disk template +will act as a translation layer between the current Ganeti disk +interface and the ExtStorage providers. + +We will also introduce a new IDISK_PARAM called `IDISK_PROVIDER = +provider`, which will be used at the command line to select the desired +ExtStorage provider. This parameter will be valid only for template +`ext` e.g.:: + + gnt-instance add -t ext --disk=0:size=2G,provider=sample_provider1 + +The Extstorage interface will support different disks to be created by +different providers. e.g.:: + + gnt-instance add -t ext --disk=0:size=2G,provider=sample_provider1 + --disk=1:size=1G,provider=sample_provider2 + --disk=2:size=3G,provider=sample_provider1 + +Finally, the ExtStorage interface will support passing of parameters to +the ExtStorage provider. This will also be done per disk, from the +command line:: + + gnt-instance add -t ext --disk=0:size=1G,provider=sample_provider1, + param1=value1,param2=value2 + +The above parameters will be exported to the ExtStorage provider's +scripts as the enviromental variables: + +- `EXTP_PARAM1 = str(value1)` +- `EXTP_PARAM2 = str(value2)` + +We will also introduce a new Ganeti client called `gnt-storage` which +will be used to diagnose ExtStorage providers and show information about +them, similarly to the way `gnt-os diagose` and `gnt-os info` handle OS +definitions. + Long-term shared storage goals ============================== Storage pool handling @@ -166,7 +260,7 @@ Storage pool handling A new cluster configuration attribute will be introduced, named “storage_pools”, modeled as a dictionary mapping storage pools to -external storage drivers (see below), e.g.:: +external storage providers (see below), e.g.:: { "nas1": "foostore", @@ -188,93 +282,19 @@ Furthermore, the storage pools will be used to indicate the availability of storage pools to different node groups, thus specifying the instances' “mobility domain”. -New disk templates will also be necessary to facilitate the use of external -storage. The proposed addition is a whole template namespace created by -prefixing the pool names with a fixed string, e.g. “ext:”, forming names -like “ext:nas1”, “ext:foo”. - -Interface to the external storage drivers ------------------------------------------ - -In addition to external storage pools, a new interface will be -introduced to allow external scripts to provision and manipulate shared -storage. - -In order to provide storage provisioning and manipulation (e.g. growing, -renaming) capabilities, each instance's disk template can possibly be -associated with an external “storage driver” which, based on the -instance's configuration and tags, will perform all supported storage -operations using auxiliary means (e.g. XML-RPC, ssh, etc.). - -A “storage driver” will have to provide the following methods: - -- Create a disk -- Remove a disk -- Rename a disk -- Resize a disk -- Attach a disk to a given node -- Detach a disk from a given node - -The proposed storage driver architecture borrows heavily from the OS -interface and follows a one-script-per-function approach. A storage -driver is expected to provide the following scripts: - -- `create` -- `resize` -- `rename` -- `remove` -- `attach` -- `detach` - -These executables will be called once for each disk with no arguments -and all required information will be passed through environment -variables. The following environment variables will always be present on -each invocation: - -- `INSTANCE_NAME`: The instance's name -- `INSTANCE_UUID`: The instance's UUID -- `INSTANCE_TAGS`: The instance's tags -- `DISK_INDEX`: The current disk index. -- `LOGICAL_ID`: The disk's logical id (if existing) -- `POOL`: The storage pool the instance belongs to. - -Additional variables may be available in a per-script context (see -below). - -Of particular importance is the disk's logical ID, which will act as -glue between Ganeti and the external storage drivers; there are two -possible ways of using a disk's logical ID in a storage driver: - -1. Simply use it as a unique identifier (e.g. UUID) and keep a separate, - external database linking it to the actual storage. -2. Encode all useful storage information in the logical ID and have the - driver decode it at runtime. - -All scripts should return 0 on success and non-zero on error accompanied by -an appropriate error message on stderr. Furthermore, the following -special cases are defined: - -1. `create` In case of success, a string representing the disk's logical - id must be returned on stdout, which will be saved in the instance's - configuration and can be later used by the other scripts of the same - storage driver. The logical id may be based on instance name, - instance uuid and/or disk index. - - Additional environment variables present: - - `DISK_SIZE`: The requested disk size in MiB - -2. `resize` In case of success, output the new disk size. - - Additional environment variables present: - - `DISK_SIZE`: The requested disk size in MiB - -3. `rename` On success, a new logical id should be returned, which will - replace the old one. This script is meant to rename the instance's - backing store and update the disk's logical ID in case one of them is - bound to the instance name. +The pool, in which to put the new instance's disk, will be defined at +the command line during `instance add`. This will become possible by +replacing the IDISK_PROVIDER parameter with a new one, called `IDISK_POOL += pool`. The cmdlib logic will then look at the cluster-level mapping +dictionary to determine the ExtStorage provider for the given pool. - Additional environment variables present: - - `NEW_INSTANCE_NAME`: The instance's new name. +gnt-storage +----------- +The `gnt-storage` client can be extended to support pool management +(creation/modification/deletion of pools, connection/disconnection of +pools to nodegroups, etc.). It can also be extended to diagnose and +provide information for internal disk templates too, such as lvm and +drbd. .. vim: set textwidth=72 : -- 1.7.10.4