Synnefo

I don't think that deleting data is a good idea, for various reasons, 2 of which might be the following

-What happens if we want to trace back some strange behavior?
-What happens if the user wants to see his wallet history?

These are all valid questions.
If there is no garbage collection mechanism, does that mean the DB gets to contain information on every VM, ever created in the system, for ever? This does not scale. There has to be a configurable data retention period, after which terminated VMs will be collected.

• Tracing strange behavior will have to happen using logs, if this period has expired. The retention period of logs is completely unrelated and may be much longer.
• The wallet history, especially searchable, detailed information on debit and credit operations will be able to go up to a certain point back in the past. Even banking institutions only allow the user to view detailed transaction lists for the past one or two months.
• The retention policy for instances of the Debit and Credit or similar models in the DB may be entirely different altogether. The user may be able to see a charge for VM 12345, without the VM instance with id=12345 actually being there any more.

I think we are optimising prematurely here.

If there is no garbage collection mechanism, does that mean the DB gets to contain information on every VM, ever created in the system, for ever? This does not scale.

How can you be sure about that? Let's say you have 30000 users, each one creating 3 VMs every day for 3 years. How many rows do you get in db_virtualmachine? 32.400.000. Of which, only the last few days' worth of VMs (100000?) will need to be kept hot in memory-based indices. Most serious DBMSs should laugh in the face of such numbers.

There has to be a configurable data retention period, after which terminated VMs will be collected.

True, but after we face scalability issues. At the moment, our problem is to get users to use the service. Speaking of data retention, are there any laws affecting our service on that front?

• Tracing strange behavior will have to happen using logs, if this period has expired. The retention period of logs is completely unrelated and may be much longer.

So you effectively exchange high quality, structured, interconnected data, with low quality unconnected data...

• The wallet history, especially searchable, detailed information on debit and credit operations will be able to go up to a certain point back in the past. Even banking institutions only allow the user to view detailed transaction lists for the past one or two months.

6 months, but banks do face scaling problems with the millions of transactions they run per day (+they keep copies of all data in offline DBs). We don't.

• The retention policy for instances of the Debit and Credit or similar models in the DB may be entirely different altogether. The user may be able to see a charge for VM 12345, without the VM instance with id=12345 actually being there any more.

This is not true in a correctly structured db schema. If we delete a VM, referential integrity mechanisms should delete all related rows.