Synnefo » snf-ganeti

====================

Linux HA integration

====================

.. contents:: :depth: 4

This is a design document detailing the integration of Ganeti and Linux HA.

Current state and shortcomings

==============================

Ganeti doesn't currently support any self-healing or self-monitoring.

We are now working on trying to improve the situation in this regard:

- The :doc:`autorepair system <design-autorepair>` will take care

  of self repairing a cluster in the presence of offline nodes.

- The :doc:`monitoring agent <design-monitoring-agent>` will take care

  of exporting data to monitoring.

What is still missing is a way to self-detect "obvious" failures rapidly

and to:

- Maintain the master role active.

- Offline resource that are obviously faulty so that the autorepair

  system can perform its work.

Proposed changes

================

Linux-HA provides software that can be used to provide high availability

of services through automatic failover of resources. In particular

Pacemaker can be used together with Heartbeat or Corosync to make sure a

resource is kept active on a self-monitoring cluster.

Ganeti OCF agents

-----------------

The Ganeti agents will be slightly special in the HA world. The

following will apply:

- The agents will be able to be configured cluster-wise by tags (which

  will be read on the nodes via ssconf_cluster_tags) and locally by

  files on the filesystem that will allow them to "simulate" a

  particular condition (eg. simulate a failure even if none is

  detected).

- The agents will be able to run in "full" or "partial" mode: in

  "partial" mode they will always succeed, and thus never fail a

  resource as long as a node is online, is running the linux HA software

  and is responding to the network. In "full" mode they will also check

  resources like the cluster master ip or master daemon, and act if they

  are missing

Note that for what Ganeti does OCF agents are needed: simply relying on

the LSB scripts will not work for the Ganeti service.

Master role agent

-----------------

This agent will manage the Ganeti master role. It needs to be configured

as a sticky resource (you don't want to flap the master role around, do

you?) that is active on only one node. You can require quorum or fencing

to protect your cluster from multiple masters.

The agent will implement a stateless resource that considers itself

"started" only the master node, "stopped" on all master candidates and

in error mode for all other nodes.

Note that if not all your nodes are master candidates this resource

might have problems:

- if all nodes are configured to run the resource, heartbeat may decide

  to "fence" (aka stonith) all your non-master-candidate nodes if told

  to do so. This might not be what you want.

- if only master candidates are configured as nodes for the resource,

  beware of promotions and demotions, as nothing will update

  automatically pacemaker should a change happen at the Ganeti level.

Other solutions, such as reporting the resource just as "stopped" on non

master candidates as well might mean that pacemaker would choose the

"wrong" node to promote to master, which is also a bad idea.

Future improvements

+++++++++++++++++++

- Ability to work better with non-master-candidate nodes

- Stateful resource that can "safely" transfer the master role between

  online nodes (with queue drain and such)

- Implement "full" mode, with detection of the cluster IP and the master

  node daemon.

Node role agent

---------------

This agent will manage the Ganeti node role. It needs to be configured

as a cloned resource that is active on all nodes.

In partial mode it will always return success (and thus trigger a

failure only upon an HA level or network failure). Full mode, which

initially will not be implemented, couls also check for the node daemon

being unresponsive or other local conditions (TBD).

When a failure happens the HA notification system will trigger on all

other nodes, including the master. The master will then be able to

offline the node. Any other work to restore instance availability should

then be done by the autorepair system.

The following cluster tags are supported:

- ``ocf:node-offline:use-powercycle``: Try to powercycle a node using

  ``gnt-node powercycle`` when offlining.

- ``ocf:node-offline:use-poweroff``: Try to power off a node using

  ``gnt-node power off`` when offlining (requires OOB support).

Future improvements

+++++++++++++++++++

- Handle draining differently than offlining

- Handle different modes of "stopping" the service

- Implement "full" mode

Risks

-----

Running Ganeti with Pacemaker increases the risk of stability for your

Ganeti Cluster. Events like:

- stopping heartbeat or corosync on a node

- corosync or heartbeat being killed for any reason

- temporary failure in a node's networking

will trigger potentially dangerous operations such as node offlining or

master role failover. Moreover if the autorepair system will be working

they will be able to also trigger instance failovers or migrations, and

disk replaces.

Also note that operations like: master-failover, or manual node-modify

might interact badly with this setup depending on the way your HA system

is configured (see below).

This of course is an inherent problem with any Linux-HA installation,

but is probably more visible with Ganeti given that our resources tend

to be more heavyweight than many others managed in HA clusters (eg. an

IP address).

Code status

-----------

This code is heavily experimental, and Linux-HA is a very complex

subsystem. *We might not be able to help you* if you decide to run this

code: please make sure you understand fully high availability on your

production machines. Ganeti only ships this code as an example but it

might need customization or complex configurations on your side for it

to run properly.

*Ganeti does not automate HA configuration for your cluster*. You need

to do this job by hand. Good luck, don't get it wrong.

Future work

===========

- Integrate the agents better with the ganeti monitoring

- Add hooks for managing HA at node add/remove/modify/master-failover

  operations

- Provide a stonith system through Ganeti's OOB system

- Provide an OOB system that does "shunning" of offline nodes, for

  emulating a real OOB, at least on all nodes

.. vim: set textwidth=72 :

.. Local Variables:

.. mode: rst

.. fill-column: 72

.. End: