Mutex helper library 🎉


#1

Inspired by @Patrick_Yi here and here, I decided to create a small helper library for working with Mutex locks in Graphcool.

https://www.npmjs.com/package/@agartha/graphcool-mutex

It’s very easy to use. It uses a subscription query to wait for a Mutex lock to be released, so there’s no polling, and no delay.

I’m looking forward to your feedback!

graphcool-mutex

Mutex helper library for Graphcool

Background

If you execute multiple GraphQL mutations, they are processed by the server in parallel.
Sometimes however, you need to perform an operation on these mutations that needs to happen sequentially.

For example, you want to update an auto-increment field in a pipeline hook, or you want to update a totals field on a parent node.

In these cases, you need a mechanism to make sure these operations are executed sequentially. This library implements the Mutex pattern to achieve this.

Mutex pattern

Each of the parallel operations (threads) requests a Mutex lock. When the lock does not exist yet, it is acquired immediately. If a lock already exists, the acquire method will wait for the lock to be released by the currently active operation. This ensures all operations are executed sequentially.

The Mutex is not a database lock. It does not prevent any mutations against your data. It only halts execution of the operation that tries to acquire a lock.

Installation

Using this library requires a Type on your Graphcool project with the following schema:

type Mutex @model {
  id: ID! @isUnique
  name: String! @isUnique
}

Usage

Import the library into your Graphcool function.

const { withMutex } = require('@agartha/graphcool-mutex')

Wrap your graphcool-lib initialization with withMutex

const graphcool = await withMutex(fromEvent(event))

Optionally, you can specify your project region manually, to avoid the async call:

const graphcool = withMutex(fromEvent(event), 'EU_WEST_1')

Use the following syntax to acquire a Mutex lock:

await graphcool.mutex.acquire('__MUTEX_NAME__')

Use the following syntax to release the lock:

graphcool.mutex.release('__MUTEX_NAME__')

How concurrency and rollback works?
How to go about concurrent mutations
How to implement integer sequence IDs in Graphcool Framework?
#2

I really like the library. :+1:

Since you already check for a duplicate error, I think you can skip the query part to reduce delay.

Current flow
Query Mutex (Subscribe if exist) -> Create Mutex (Subscribe if exist, checked by a duplicate error)

Possible optimized flow
Create Mutex (Subscribe if exist, checked by a error)

Also, if the code that was using the mutex forgets to release the lock, or throws an error and terminate early, I think the mutex gets locked indefinitely. Adding a feature for mutex expiration might be a great addon.


#3

@Patrick_Yi Thank you very much for your feedback!

I had to change the workflow a bit, because the initial version didn’t work with 2+ simultaneous requests. After I receive a delete message from the subscription, I have to recreate the mutex for the current ‘leader’, otherwise all waiting consumers continue at the same time.

I agree that creating it directly and handling the error saves a roundtrip. I was brought up with ‘don’t use exception handling for errors you expect, because exception handling is expensive’. I haven’t done any measurements, but I am afraid that executing a mutation, and have the graphql-request library throw an error, might be more expensive than executing the query.

However, in nine out of ten cases with multiple threads, I will run into the edge case that two threads try to create the new mutex at the same time, and the error still occurs.

So while writing this, it seems you actually do have a good point there for the optimization. I’ll try to update and test if I have time.

You are right that the mutex might lock indefinitely if something goes wrong, and the mutex is not released. I’ll try to see what I can do with regards to a timestamp or something. In that case I do need the query actually, because directly creating will just throw the error, regardless of how old the mutex lock is.

Another option would be a cron job that does the cleanup.

I have created issues here for the things you mentioned.


#4

This is my PHP version of the mutex api.

public function acquire($name, $timeoutSeconds = 10, $expireSeconds = 10)
{
	// Use a global lock to prevent multiple requests racing for their own lock. Make them wait in line
	if ($this->mutex->acquire("GLOBAL-LOCK-" . $name, $timeoutSeconds, expireSeconds)) {
		$this->mutex->releaseIfExpired($name);		
		if ($this->mutex->acquire($name, $timeoutSeconds, $expireSeconds)) {
			return true;
		} else {
			// Timeout while acquiring the mutex
			return false;
		}
		$this->mutex->release("GLOBAL-LOCK-" . $name);
	} 
	// Timeout while acquiring the global mutex
	return false;
}

#5

It’s definitely easier when you can use an external webserver. :smile:


#6

The library looks like a very interesting building block.

On consistency: I’m not sure if it works as intended. What happens if a mutex in thread A is released exactly before the subscription is created in thread B? Am I missing something?
I might be able to code a test for this specific case later.

On exceptions for program flow (as you mentioned): Go ahead - given that you’re doing HTTP calls, any exception handling overhead should be negligible. I would not worry about optimising too much right now.


#7

thread B will be able to get the mutex once thread C releases it. (under the condition, another thread C is executed within a small time period)
It wouldn’t be problem as long as execution order matters.


#9

@ejoebstl There is a delay between checking if the mutex exists, and setting up the subscription to wait for it to be released. If the mutex is released within this delay, thread B would seem to wait indefinitely.

It might be good to create the subscription right away, and then try to create the mutex. If it fails, it’s already there, and the subscription will fire. If it succeeds, the mutex was either not there, or just released, and the subscription would still fire.

Thank you for this feedback, I will make this improvement.


#10

Sorry, I misread your post at first.

Yes, you are correct if a thread C exists, but there might be cases where no C exists and you end up with a deadlock.

@agartha is right, you need to create the subscription first for classic mutex behaviour.

(An alternative would be to try to acquire a second time after creating the subscription. I think @agartha’s proposal is better though)


#11

A - creates the Mutex
B - checks for the Mutex
A - releases Mutex
B - sets up subscription and waits for the mutex to be deleted

This last event will never happen. Now the proposed change:

A - creates the Mutex
B - sets up the subscription

Now either this happens:

A - releases the Mutex
B - creates the Mutex succesfully

In this case B doesn’t have to wait for the subscription, or:

B - can’t create the Mutex
A - releases the Mutex
B - will get the subscription message

The only slight twist is that in the first case, B has already set up the subscription, and is still able to create the Mutex because it was already released by A. In that case, the subscription needs to be closed.


#12

@ejoebstl @Patrick_Yi Thank you again for your feedback. Based on it, I have just published a new version (0.2.3), with the new and improved flow:

  1. Set up the subscription
  2. Try to create the Mutex
  3. If it already exists, wait for the subscription message
  4. Receive subscription message when the Mutex is released by another thread
  5. Try to create the Mutex
  6. If not, just keep waiting and repeat from step 4

This means the subscription is only set up once, and there’s no more race conditions I can think of.


#13

Great, thanks! I’d say from the point of program flow, this looks correct.

It might be worthwhile to point out that even if the initial acquire succeeds and the subscription fires simultaneously (because some other thread acquired and released the mutex very fast), the mutex still behaves properly.
Also the case where another thread quickly acquires the mutex before we can react on the subscription works fine.

You might want to include the steps for disposal of the object in your flow above. It’s very important that the subscription is closed after the successful acquisition of the mutex.


#14

@ejoebstl The subscription is closed immediately after the Mutex can be succesfully created:

I don’t believe there’s a timing issue possible there. As soon as the Mutex is created, no other threads can create it, so the subscription event will never fire anymore for any thread until the Mutex is released by the current thread, so I believe this is safe.


#15

Works great! Discovered this today… and is a lifesaver! Thanks @agartha :heart_eyes:


#16

The library resembles an extremely fascinating building square.

On consistency: I don’t know whether it fills in as proposed. What occurs if a mutex in string An is discharged precisely before the membership is made in string B? Am I missing something?

I may have the capacity to code a test for this particular case later.

On special cases for program stream (as you referenced): Go ahead - given that you’re doing HTTP calls, any exemption taking care of overhead ought to be irrelevant. I would not stress over improving excessively at this moment.