Synnefo » snf-ganeti

========================================

Automatized Upgrade Procedure for Ganeti

========================================

.. contents:: :depth: 4

This is a design document detailing the proposed changes to the

upgrade process, in order to allow it to be more automatic.

Current state and shortcomings

==============================

Ganeti requires to run the same version of Ganeti to be run on all

nodes of a cluster and this requirement is unlikely to go away in the

foreseeable future. Also, the configuration may change between minor

versions (and in the past has proven to do so). This requires a quite

involved manual upgrade process of draining the queue, stopping

ganeti, changing the binaries, upgrading the configuration, starting

ganeti, distributing the configuration, and undraining the queue.

Proposed changes

================

While we will not remove the requirement of the same Ganeti

version running on all nodes, the transition from one version

to the other will be made more automatic. It will be possible

to install new binaries ahead of time, and the actual switch

between versions will be a single command.

While changing the file layout anyway, we install the python

code, which is architecture independent, under ``${prefix}/share``,

in a way that properly separates the Ganeti libraries of the

various versions. 

Path changes to allow multiple versions installed

-------------------------------------------------

Currently, Ganeti installs to ``${PREFIX}/bin``, ``${PREFIX}/sbin``,

and so on, as well as to ``${pythondir}/ganeti``.

These paths will be changed in the following way.

- The python package will be installed to

  ``${PREFIX}/share/ganeti/${VERSION}/ganeti``.

  Here ${VERSION} is, depending on configure options, either the full qualified

  version number, consisting of major, minor, revision, and suffix, or it is

  just a major.minor pair. All python executables will be installed under

  ``${PREFIX}/share/ganeti/${VERSION}`` so that they see their respective

  Ganeti library. ``${PREFIX}/share/ganeti/default`` is a symbolic link to

  ``${sysconfdir}/ganeti/share`` which, in turn, is a symbolic link to

  ``${PREFIX}/share/ganeti/${VERSION}``. For all python executatables (like

  ``gnt-cluster``, ``gnt-node``, etc) symbolic links going through

  ``${PREFIX}/share/ganeti/default`` are added under ``${PREFIX}/sbin``.

- All other files will be installed to the corresponding path under

  ``${libdir}/ganeti/${VERSION}`` instead of under ``${PREFIX}``

  directly, where ``${libdir}`` defaults to ``${PREFIX}/lib``.

  ``${libdir}/ganeti/default`` will be a symlink to ``${sysconfdir}/ganeti/lib``

  which, in turn, is a symlink to ``${libdir}/ganeti/${VERSION}``.

  Symbolic links to the files installed under ``${libdir}/ganeti/${VERSION}``

  will be added under ``${PREFIX}/bin``, ``${PREFIX}/sbin``, and so on. These

  symbolic links will go through ``${libdir}/ganeti/default`` so that the

  version can easily be changed by updating the symbolic link in

  ``${sysconfdir}``.

The set of links for ganeti binaries might change between the versions.

However, as the file structure under ``${libdir}/ganeti/${VERSION}`` reflects

that of ``/``, two links of differnt versions will never conflict. Similarly,

the symbolic links for the python executables will never conflict, as they

always point to a file with the same basename directly under

``${PREFIX}/share/ganeti/default``. Therefore, each version will make sure that

enough symbolic links are present in ``${PREFIX}/bin``, ``${PREFIX}/sbin`` and

so on, even though some might be dangling, if a differnt version of ganeti is

currently active.

The extra indirection through ``${sysconfdir}`` allows installations that choose

to have ``${sysconfdir}`` and ``${localstatedir}`` outside ``${PREFIX}`` to

mount ``${PREFIX}`` read-only. The latter is important for systems that choose

``/usr`` as ``${PREFIX}`` and are following the Filesystem Hierarchy Standard.

For example, choosing ``/usr`` as ``${PREFIX}`` and ``/etc`` as ``${sysconfdir}``,

the layout for version 2.10 will look as follows.

::

   /

   |

   +-- etc

   |   |

   |   +-- ganeti 

   |         |

   |         +-- lib -> /usr/lib/ganeti/2.10

   |         |

   |         +-- share  -> /usr/share/ganeti/2.10

   +-- usr

        |

        +-- bin

        |   |

        |   +-- harep -> /usr/lib/ganeti/default/usr/bin/harep

        |   |

        |   ...  

        |

        +-- sbin

        |   |

        |   +-- gnt-cluster -> /usr/share/ganeti/default/gnt-cluster

        |   |

        |   ...  

        |

        +-- ...

        |

        +-- lib

        |   |

        |   +-- ganeti

        |       |

        |       +-- default -> /etc/ganeti/lib

        |       |

        |       +-- 2.10

        |           |

        |           +-- usr

        |               |

        |               +-- bin

        |               |    |

        |               |    +-- htools

        |               |    |

        |               |    +-- harep -> htools

        |               |    |

        |               |    ...

        |               ...

        |

        +-- share

             |

             +-- ganeti

                 |

                 +-- default -> /etc/ganeti/share

                 |

                 +-- 2.10

                     |

                     + -- gnt-cluster

                     |

                     + -- gnt-node

                     |

                     + -- ...

                     |

                     + -- ganeti

                          |

                          +-- backend.py

                          |

                          +-- ...

                          |

                          +-- cmdlib

                          |   |

                          |   ...

                          ...

gnt-cluster upgrade

-------------------

The actual upgrade process will be done by a new command ``upgrade`` to

``gnt-cluster``. If called with the option ``--to`` which take precisely

one argument, the version to

upgrade (or downgrade) to, given as full string with major, minor, suffix,

and suffix. To be compatible with current configuration upgrade and downgrade

procedures, the new version must be of the same major version and

either an equal or higher minor version, or precisely the previous

minor version.

When executed, ``gnt-cluster upgrade --to=<version>`` will perform the

following actions.

- It verifies that the version to change to is installed on all nodes

  of the cluster that are not marked as offline. If this is not the

  case it aborts with an error. This initial testing is an

  optimization to allow for early feedback.

- An intent-to-upgrade file is created that contains the current

  version of ganeti, the version to change to, and the process ID of

  the ``gnt-cluster upgrade`` process. The latter is not used automatically,

  but allows manual detection if the upgrade process died

  unintentionally. The intend-to-upgrade file is persisted to disk

  before continuing.

- The Ganeti job queue is drained, and the executable waits till there

  are no more jobs in the queue. Once :doc:`design-optables` is

  implemented, for upgrades, and only for upgrades, all jobs are paused

  instead (in the sense that the currently running opcode continues,

  but the next opcode is not started) and it is continued once all

  jobs are fully paused.

- All ganeti daemons on the master node are stopped.

- It is verified again that all nodes at this moment not marked as

  offline have the new version installed. If this is not the case,

  then all changes so far (stopping ganeti daemons and draining the

  queue) are undone and failure is reported. This second verification

  is necessary, as the set of online nodes might have changed during

  the draining period.

- All ganeti daemons on all remaining (non-offline) nodes are stopped.

- A backup of all Ganeti-related status information is created for

  manual rollbacks. While the normal way of rolling back after an

  upgrade should be calling ``gnt-clsuter upgrade`` from the newer version

  with the older version as argument, a full backup provides an

  additional safety net, especially for jump-upgrades (skipping

  intermediate minor versions).

- If the action is a downgrade to the previous minor version, the

  configuration is downgraded now, using ``cfgupgrade --downgrade``.

- If the action is downgrade, any version-specific additional downgrade

  actions are carried out.

- The ``${sysconfdir}/ganeti/lib`` and ``${sysconfdir}/ganeti/share``

  symbolic links are updated.

- If the action is an upgrade to a higher minor version, the configuration

  is upgraded now, using ``cfgupgrade``.

- ``ensure-dirs --full-run`` is run on all nodes.

- All daemons are started on all nodes.

- ``gnt-cluster redist-conf`` is run on the master node. 

- All daemons are restarted on all nodes.

- The Ganeti job queue is undrained.

- The intent-to-upgrade file is removed.

- ``post-upgrade`` is run with the original version as argument.

- ``gnt-cluster verify`` is run and the result reported.

Considerations on unintended reboots of the master node

=======================================================

During the upgrade procedure, the only ganeti process still running is

the one instance of ``gnt-cluster upgrade``. This process is also responsible

for eventually removing the queue drain. Therefore, we have to provide

means to resume this process, if it dies unintentionally. The process

itself will handle SIGTERM gracefully by either undoing all changes

done so far, or by ignoring the signal all together and continuing to

the end; the choice between these behaviors depends on whether change

of the configuration has already started (in which case it goes

through to the end), or not (in which case the actions done so far are

rolled back).

To achieve this, ``gnt-cluster upgrade`` will support a ``--resume``

option. It is recommended

to have ``gnt-cluster upgrade --resume`` as an at-reboot task in the crontab.

The ``gnt-cluster upgrade --resume`` comand first verifies that

it is running on the master node, using the same requirement as for

starting the master daemon, i.e., confirmed by a majority of all

nodes. If it is not the master node, it will remove any possibly

existing intend-to-upgrade file and exit. If it is running on the

master node, it will check for the existence of an intend-to-upgrade

file. If no such file is found, it will simply exit. If found, it will

resume at the appropriate stage.

- If the configuration file still is at the initial version,

  ``gnt-cluster upgrade`` is resumed at the step immediately following the

  writing of the intend-to-upgrade file. It should be noted that

  all steps before changing the configuration are idempotent, so

  redoing them does not do any harm.

- If the configuration is already at the new version, all daemons on

  all nodes are stopped (as they might have been started again due

  to a reboot) and then it is resumed at the step immediately

  following the configuration change. All actions following the

  configuration change can be repeated without bringing the cluster

  into a worse state.

Caveats

=======

Since ``gnt-cluster upgrade`` drains the queue and undrains it later, so any

information about a previous drain gets lost. This problem will

disappear, once :doc:`design-optables` is implemented, as then the

undrain will then be restricted to filters by gnt-upgrade.

Requirement of job queue update

===============================

Since for upgrades we only pause jobs and do not fully drain the

queue, we need to be able to transform the job queue into a queue for

the new version. The preferred way to obtain this is to keep the

serialization format backwards compatible, i.e., only adding new

opcodes and new optional fields.

However, even with soft drain, no job is running at the moment `cfgupgrade`

is running. So, if we change the queue representation, including the

representation of individual opcodes in any way, `cfgupgrade` will also

modify the queue accordingly. In a jobs-as-processes world, pausing a job

will be implemented in such a way that the corresponding process stops after

finishing the current opcode, and a new process is created if and when the

job is unpaused again.