Bug #1029

Non-transactional processing of requests

Added by Vangelis Koukis about 8 years ago. Updated over 5 years ago.

Status:Closed Start date:08/31/2011
Priority:High Due date:11/11/2011
Assignee:Christos Stavrakakis % Done:

90%

Category:logic Spent time: -
Target version:v0.9.0

Description

It seems there are requests which do not get processed inside a transaction properly.
The following should be investigated:
  • If sending an invitation fails, e.g. the MQ is not available, an exception is raised.
    The invitation should not have to be removed manually, inserting it into the DB should happen inside a transaction,
    which should be rolled back.
  • API request processing should happen inside a transaction.
    What happens if a RAPI call fails? DB fields have already been updated [e.g., the action field of VirtualMachine],
    these changes should be rolled back if the RAPI call cannot complete.

Related issues

related to Synnefo - Bug #993: Transaction deadlock, exception in callbacks.py Closed 08/03/2011 11/11/2011

History

#1 Updated by Giorgos Gousios about 8 years ago

General comment: there is no way to do ensure proper transaction handling across two independent systems, except if they participate in a distributed transaction protocol (usually implemented with the two phase commit algorithm) and the underlying systems have ACID properties.

Regarding the issues you raised:

  • This has been sort of fixed with ecb4680f. Not a sound solution, but better than what we have now.
  • What happens if a RAPI call succeeds and the DB has been updated by another transaction (therefore, the current one deadlocks as per bug: #993)?

I agree that we have to do manual transaction management (see branch transactions), but neither of the issues you reported would be resolved using manual transactions.

#2 Updated by Vangelis Koukis about 8 years ago

  • Target version changed from v0.5.5 to v0.6

#3 Updated by Giorgos Gousios about 8 years ago

  • % Done changed from 0 to 90

The following changes were made as of 628d859941.

  • Manual transaction handling has been enabled on the dispatcher (each message is processed in its own transaction) and the API (each request is handled in a separate transaction). For the latter to work, the DJango transaction middleware has been enabled in the configuration.
  • AMQP messages that cannot be processed for whatever reason are declared rejected instead of remaining unacknowledged on the queue.

The changes have at least the following effects

  • RAPI request failures leave the VMs state unchanged in the DB
  • Invitations are added one invitation per transaction. If some error occurs, the transaction is rollbacked
  • Messages that could be processed once will not be processed again upon dispatcher reconnect, which means that stale messages cannot affect the status of the system when the dispatcher is restarted.

Warning: Transaction deadlocks are not yet handled appropriately. The only case we are aware that this might actually be a problem is when a VM is created and the following sequence of events takes place:

1. The RAPI request is sent
2. A message is queued by the ganeti-eventd daemon as a result
3. The message is processed by the dispatcher before the transaction that created the VM is fully committed. The message processing by the dispatcher will fail.

#4 Updated by Vangelis Koukis about 8 years ago

  • Target version changed from v0.6 to v0.6.1

#5 Updated by Vangelis Koukis about 8 years ago

  • Target version changed from v0.6.1 to v0.6.2

#6 Updated by Vangelis Koukis about 8 years ago

  • Target version changed from v0.6.2 to 67

#7 Updated by Vangelis Koukis about 8 years ago

  • Status changed from New to Assigned
  • Target version changed from 67 to v0.8.0

#8 Updated by Vangelis Koukis almost 8 years ago

  • Due date set to 11/11/2011

#9 Updated by Vangelis Koukis over 7 years ago

  • Assignee changed from Giorgos Gousios to Christos Stavrakakis

Moving to v0.9, must be re-evaluated.

#10 Updated by Vangelis Koukis over 7 years ago

  • Target version changed from v0.8.0 to v0.9.0

#11 Updated by Christos Stavrakakis over 5 years ago

  • Status changed from Assigned to Closed

Also available in: Atom PDF