One shared work queue in a Laravel multi-tenant application

Question

I am creating a Laravel application with several tenants (on Laravel 5.3), which allows each tenant to have their own set of configurations for any supported Laravel settings. This is currently achieved by overriding the standard Laravel Application with my own implementation, which provides a custom configuration loader (overrides the default Illuminate\Foundation\Bootstrap\LoadConfiguration ). The application detects the current tenant from the environment (either PHP $_ENV , or the .env file) in the bootstrap, and then loads the appropriate configuration files for the detected tenant.

This approach is great for both HTTP cores and consoles, where each request / command has a limited life cycle, but I'm not sure how to approach the queue worker. I would like to have one queue employee for all tenants, and I have already implemented a special queue connector to add additional metadata when the queue task is scheduled so that the tenant can be identified when the employee received it.

The part I'm looking for your help with is how to run each queue job in an isolated environment that I can upload with my custom configuration.

A few possible solutions that I see will be as follows:

• to start a personalized desktop that starts as a daemon and receives a task from the queue, but performs the task in a separate PHP process (created via exec() ); after the task is completed, the worker collects the results (status, exceptions, etc.) and ends the work in the parent process (for example, deletes the task, etc.).

• similar to the above, but run the job in a separate PHP thread instead of a separate process using RunKit Sandbox

• implement a solution that "reloads" the application after receiving the job on the queue (for example, reloads the configuration for the current tenant, resets any allowed dependencies, etc.)

The important thing is that I would like the implementation of this multi-user work to be transparent to the work itself, so that jobs that are not designed to work in a multi-tenant environment (for example, jobs from third-party packages such as Laravel Scout) can be processed without either changes.

Any suggestions on how to approach this?

Answer 1

We have almost the same situation. Here is our approach:

Service provider

We have a ServiceProvider called BootTenantServiceProvider that loads the tenant in a regular HTTP / Console request. He expects an environment variable to exist under the name TENANT_ID . At the same time, it will download all the relevant configurations and configure a specific tenant.

Value with __sleep () and __wakeup ()

We have a BootsTenant trait that we will use in our queue tasks that looks like this:

 trait BootsTenant { protected $tenantId; /** * Prepare the instance for serialization. * * @return array */ public function __sleep() { $this->tenantId = env('TENANT_ID'); return array_keys(get_object_vars($this)); } /** * Restore the ENV, and run the service provider */ public function __wakeup() { // We need to set the TENANT_ID env, and also force the BootTenantServiceProvider again \Dotenv::makeMutable(); \Dotenv::setEnvironmentVariable('TENANT_ID', this->tenantId); app()->register(BootTenantServiceProvider::class, [], true); } } 

Now we can write a queue job that uses this feature. When a job is serialized in a queue, the __sleep() method will store the tenantId locator locally. When unesterized, the __wakeup() method will restore the environment variable and start the service provider.

Queue Jobs

Our jobs in the queue just have to use this attribute:

 class MyJob implements SelfHandling, ShouldQueue { use BootsTenant; protected $userId; public function __construct($userId) { $this->userId = $userId; } public function handle() { // At this point the job has been unserialized from the queue, // the trait __wakeup() method has restored the TENANT_ID // and the service provider has set us all up! $user = User::find($this->userId); // Do something with $user } } 

Conflict with SerializesModels

The SerializesModels attribute that Laravel includes includes its own __sleep and __wakeup . I did not quite understand how both of these traits work together or even if it is possible.

At the moment, I am sure that I never provide a full-fledged Eloquent model in the constructor. You can see in my example the work above. I only store identifiers as attributes of a class, not complete models. I have a handle() method for fetching models at run time. Then I don't need the SerializesModels trait at all.

Use queue: listen instead of --daemon

You need to start your queue employees using queue:listen instead of queue:work --daemon . The former loads the framework for each job into the queue, the latter stores the boot infrastructure loaded into memory.

At the very least, you need to do this if your tenant's boot process needs a new frame load. If you can load multiple tenants in a row by reading the rewrite of the configurations for each, then you can quit queue:work --daemon just fine.