Bug #3560

Accepting commissions sometimes fail with deadlock in astakos DB

Added by Georgios Tsoukalas over 7 years ago. Updated almost 7 years ago.

Status:Resolved Start date:04/08/2013
Priority:High Due date:
Assignee:Giorgos Korfiatis % Done:

0%

Category:Astakos Spent time: -
Target version:-

Description

For example,

2013-04-06 08:25:35 EEST ERROR: deadlock detected
2013-04-06 08:25:35 EEST DETAIL: Process 30576 waits for ShareLock on transaction 512051; blocked by process 30580.
Process 30580 waits for ShareLock on transaction 512050; blocked by process 30576.
Process 30576: SELECT "quotaholder_app_holding"."id", "quotaholder_app_holding"."entity_id", "quotaholder_app_holding"."resource", "quotaholder_app_holding"."policy_id", "quotaholder_app_holding"."flags", "quotaholder_app_holding"."imported", "quotaholder_app_holding"."importing", "quotaholder_app_holding"."exported", "quotaholder_app_holding"."exporting", "quotaholder_app_holding"."returned", "quotaholder_app_holding"."returning", "quotaholder_app_holding"."released", "quotaholder_app_holding"."releasing" FROM "quotaholder_app_holding" WHERE ("quotaholder_app_holding"."resource" = E'cyclades.ram' AND "quotaholder_app_holding"."entity_id" = E'cyclades' ) FOR UPDATE
Process 30580: SELECT "quotaholder_app_holding"."id", "quotaholder_app_holding"."entity_id", "quotaholder_app_holding"."resource", "quotaholder_app_holding"."policy_id", "quotaholder_app_holding"."flags", "quotaholder_app_holding"."imported", "quotaholder_app_holding"."importing", "quotaholder_app_holding"."exported", "quotaholder_app_holding"."exporting", "quotaholder_app_holding"."returned", "quotaholder_app_holding"."returning", "quotaholder_app_holding"."released", "quotaholder_app_holding"."releasing" FROM "quotaholder_app_holding" WHERE ("quotaholder_app_holding"."resource" = E'cyclades.vm' AND "quotaholder_app_holding"."entity_id" = E'cyclades' ) FOR UPDATE

It is suspected that the deadlock is generated in this code in accept_commission():

provisions = db_filter_provision(serial=serial, for_update=True)
for pv in provisions:
   try:
       h = db_get_holding(entity=pv.entity.entity,
                          resource=pv.resource, for_update=True)
       th = db_get_holding(entity=t, resource=pv.resource,
                           for_update=True)

the provisions returned by the query are not sorted,
therefore different commissions on the same subset of provisions
(e.g. VM commissions include vm, ram, disk, network resources)
may attempt FOR UPDATE lock for the same set of rows in different order.

Associated revisions

Revision e0223f54
Added by Giorgos Korfiatis over 7 years ago

quotaholder: Impose ordering on locking holdings

Prefetch all holdings required for update with a specified order
(by entity/resource pair).

Refs #3560

History

#1 Updated by Giorgos Korfiatis almost 7 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF