Synnefo » snf-ganeti

Job Queue

=========

.. contents::

Overview

--------

In Ganeti 1.2, operations in a cluster have to be done in a serialized way.

Virtually any operation locks the whole cluster by grabbing the global lock.

Other commands can't return before all work has been done.

By implementing a job queue and granular locking, we can lower the latency of

command execution inside a Ganeti cluster.

Detailed Design

---------------

Job execution—“Life of a Ganeti job”

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#. Job gets submitted by the client. A new job identifier is generated and

   assigned to the job. The job is then automatically replicated to all nodes

   in the cluster. The identifier is returned to the client.

#. A pool of worker threads waits for new jobs. If all are busy, the job has

   to wait and the first worker finishing its work will grab it. Otherwise any

   of the waiting threads will pick up the new job.

#. Client waits for job status updates by calling a waiting RPC function.

   Log message may be shown to the user. Until the job is started, it can also

   be cancelled.

#. As soon as the job is finished, its final result and status can be retrieved

   from the server.

#. If the client archives the job, it gets moved to a history directory.

   This could also be done regularily using a cron script.

Queue structure

~~~~~~~~~~~~~~~

All file operations have to be done atomically by writing to a temporary file

and subsequent renaming. Except for log messages, every change in a job is

stored and replicated to other nodes.

::

  /var/lib/ganeti/queue/

    job-1 (JSON encoded job description and status)

    […]

    job-37

    job-38

    job-39

    lock (Queue managing process opens this file in exclusive mode)

    serial (Last job ID used)

    version (Queue format version)

Locking

~~~~~~~

Locking in the job queue is a complicated topic. It is called from more than

one thread and must be thread-safe. For simplicity, a single lock is used for

the whole job queue.

A more detailed description can be found in doc/locking.txt.

Internal RPC

~~~~~~~~~~~~

RPC calls available between Ganeti master and node daemons:

jobqueue_update(file_name, content)

  Writes a file in the job queue directory.

jobqueue_purge()

  Cleans the job queue directory completely, including archived job.

jobqueue_rename(old, new)

  Renames a file in the job queue directory.

Client RPC

~~~~~~~~~~

RPC between Ganeti clients and the Ganeti master daemon supports the following

operations:

SubmitJob(ops)

  Submits a list of opcodes and returns the job identifier. The identifier is

  guaranteed to be unique during the lifetime of a cluster.

WaitForJobChange(job_id, fields, […], timeout)

  This function waits until a job changes or a timeout expires. The condition

  for when a job changed is defined by the fields passed and the last log

  message received.

QueryJobs(job_ids, fields)

  Returns field values for the job identifiers passed.

CancelJob(job_id)

  Cancels the job specified by identifier. This operation may fail if the job

  is already running, canceled or finished.

ArchiveJob(job_id)

  Moves a job into the …/archive/ directory. This operation will fail if the

  job has not been canceled or finished.

Job and opcode status

~~~~~~~~~~~~~~~~~~~~~

Each job and each opcode has, at any time, one of the following states:

Queued

  The job/opcode was submitted, but did not yet start.

Running

  The job/opcode is running.

Canceled

  The job/opcode was canceled before it started.

Success

  The job/opcode ran and finished successfully.

Error

  The job/opcode was aborted with an error.

If the master is aborted while a job is running, the job will be set to the

Error status once the master started again.

History

~~~~~~~

Archived jobs are kept in a separate directory, /var/lib/ganeti/queue/archive/.

The idea is to speed up the queue handling.

Ganeti updates

~~~~~~~~~~~~~~

The queue has to be completely empty for Ganeti updates with changes in the job

queue structure.