A distributed lock manager for OpenStack? [LWN.net]

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 18:43 UTC (Mon) by harlowja (guest, #97846)
In reply to: A distributed lock manager for OpenStack? by isotopp
Parent article: A distributed lock manager for OpenStack?

Understood, but then what really has survived jepsen (besides zookeeper)?

I'm more along the line of listing the various capabilities (strengths and weaknesses) of the underlying implementations and letting the smart user select one that is appropriate, jepsen testing is like ensuring you have a battleship, but sometimes you only need a powerboat and/or canoe (for lack of better analogy)...

to post comments

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 21:36 UTC (Mon) by isotopp (subscriber, #99763) [Link] (1 responses)

ZK, etcd and consul have (consul on 2nd try). And that's actually quite the point here - these things are built to manage a cluster properly by building the inner cluster (ZK calls it the "ensemble") and once that is stable, to provide a platform and service for the outer cluster.

Services provided usually include at least discovery (things such as hypervisors, running VMs, and *-api instances could check in with the ensemble and leave their endpoint data with it, making things such as keystone.endpoints and similar tables superfluous and much more accurate for degraded states. Also, HA for free). Other services that are common are locking (the original reason for the project), and with locking global order/sequence numbers/counters. Also, a kind of highly available, distributed, small key/value store, in consuls case with config file generator attached.

In the end, rigorous application of the tool to the entire openstack project would greatly benefit the project in handling degraded cases, HA, locking, and turning from a web system into a proper control system/state engine. That is, it would become much more reliable. For this the openstack project would have had to commit itself to a single of these system in order to be able to reliably use the feature set of that system.

With an abstraction layer inbetween you will only get what the weakest of these systems can give you (hence the Redis needs to die), and you would need to Jepsen the shit of out every possible configuration and service provided in order to make sure the abstraction on top of the underlying consensus system doesn't break stuff.

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 23:49 UTC (Mon) by harlowja (guest, #97846) [Link]

Totally agree with the one solution; it was what I (and others) was thinking was going to be an outcome of that session (which I was the other driver of) but it didn't turn out this way in the end (even though I tried to say we should really be opinionated). As far as the other drivers, yes, the point is well taken, and jepsen is a good base case and it might be the way we should handle new drivers (if they have not been proven jepsen safe, then they aren't allowed); redis would then get kicked out due to this (although for small enough deployments perhaps having a single redis that uses persistent storage and can *manually* failover to a secondary master will be fine). Overall we (as a tooz group) are figuring out the way to go here and there seems to be a few ideas (like this) that are being bounced around.