Advanced Synchronized – Queueing Process Groups

This article focuses on a specific situation where the Synchronized function can help ensure groups of processes running multi-processed by RunProcess execute serially. This becomes necessary when one group of processes depend on the changes made by another to be fully committed before executing, and it is desired that they run in the same execution. Whether that be a process executing multiple master processes, or a chore doing the same. Basic use and functionality of process synchronization using the Synchronized function is covered by this article.

A quick example of this situation is having a process that runs two master processes that leverage multi-processing using RunProcess. The purpose of the first master process could be to clear data in a cube, while the second master process may increment data in that same cube. Both master processes could be leveraging RunProcess to break the executions into smaller, faster processes. For this to work as intended, the subprocesses of the data increment step cannot start until the associated data clear subprocesses have fully committed their changes.

A way to alleviate this issue could be to implement file communication for the master processes to make them wait until all their subprocesses have finished processing their epilog tab as described in this article: <!Link KB – File Communication>. A prerequisite to implementing this functionality is the permission to create and write to files on the Planning Analytics server. A Sleep function could also be added between the ExecuteProcess functions for the master processes to add a buffer. However, if the commit time for one of the subprocesses ever exceeds the wait time defined in the Sleep function, then data issues may arise. Adding the file communication is still advised if tracking process statistics, such as timing, for the master processes.

By leveraging the file system and using unique Synchronized flags, this issue can be resolved for all situations. To do this it must be defined which processes need to be fully committed before continuing, and at which point in the main execution that the wait needs to occur. In some situations, there may be multiple of these defined. For this example, it is assumed that the subprocesses of the Master Zero process all need to be fully committed before continuing to the Master Load process, and this wait must occur before the Master Load process starts (replacing the Sleep function seen above).

To implement this functionality in this situation, the following is needed (note: all example ExecuteCommand functions are using Linux syntax):

  • Master Zero Process needs to create a file / remove an old one if it exists
    • This file is unique to the master process
    • Will be populated when running subprocesses
    • Will be leveraged by the new wait process

  • Master Zero must export each unique Synchronized flag that will be used by the subprocesses to the file created previously
    • This flag must be passed to the subprocess so that it is guaranteed to use the same one
    • This example uses the command for file communication because it is guaranteed to be unique per subprocess

  • Subprocesses of Master Zero must use the Synchronized function to create a synchronization flag unique to each subprocess
    • With unique flags, it is ensured that the subprocesses will not wait on each other

  • A new process must be created to check all Synchronized flags added to the file
    • The new process will replace the Sleep function in the example above
    • The new process will use the created file as a source and call the Synchronized function on all flags added by the mater process
    • The details of this new process are defined below

The process to wait for the defined Synchronized flags needs to be set to use a file as the data source with one string variable. This variable will be the Synchronized flags created by the Master process earlier.

One Parameter must be added to the process so that it knows the name of the file to be used as a source. This will allow this one process to be reused for other processes if needed.

In the Prolog of the process, define the data source using the provided parameter.

In the data tab of the process, simply call the Synchronized function on each line of the file.

In the epilog tab of the process, delete the file used if desired. This isn’t necessary as the Master Zero process would delete the file before recreating it if it still exists.

At this point, the process can be saved and added to the process that executes the master processes, directly after executing the Master Zero Process.

With this implemented, the Master Load process will not be able to start until it is guaranteed that all Master Zero subprocesses have finished committing.