Using Files for TI Process Communication

Inter-process communication has limited support within IBM Planning Analytics, and often needs a custom solution when supporting complex process executions. A common one of these situations is when multiprocessing TI Processes. Multiprocessing can be very useful to improve TI process performance, but the master process used to multi-process will complete after it has initiated all its subprocesses. This can be a detriment when it is desirable for the master process to wait until all subprocesses are complete before completing itself. Situations where this is useful include when recording process timing statistics or when executing multiple processes in a row, where subsequent processes rely on the data from their predecessors.

One way to support this functionality is by utilizing the file system on the Planning Analytics server to create a system of communication between the master and sub processes. A prerequisite to implementing this functionality is the permission to create and write to files on the Planning Analytics server. To achieve this the following must occur:

  • Decide on a folder to use for the file communication
    • Best to create a folder specifically for this function
    • Personally, I use a folder named “semaphores” adjacent to the Planning Analytics Data folder
  • The master process needs to be aware of all running subprocesses
    • A file must be created by the master process for each subprocess that it executes
    • The master process must wait for all associated files to be deleted before finishing
  • Subprocesses need to handle their associated files
    • Parameter must be added so each subprocess know its associated file
    • Subprocesses to create error files if needed
  • Special care must be taken to communicate and handle errors
    • Master process must be provided maximum amount of time to wait to avoid waiting forever in case of errors
    • Error files can be used to communicate errors to the master process

After creating a folder to use for the file communication, the processes can be edited to utilize the folder. First, define the file naming convention so that they are guaranteed to be unique. Personally, I use the name of the subprocess appended with any non-unique parameters and finally a number that will be unique for each process.

In the prolog of the master process, code is added to remove any existing files that may have inadvertently been left over from a previous run. To do this, an ExecuteCommand function is used to remove any files matching a wildcard search. Linux examples are included below, and would need to be changed to Windows equivalents when running Planning Analytics servers on Windows. Ensure the correct command is used based on the  Planning Analytics server operating system. In addition, a file count variable is initialized that will be used to track the number of subprocesses that are executed.

Wherever the master process executes its subprocesses, code needs to be added to increment the file count variable and create the command to generate the file. After the command is created and the file is generated, the command must also be passed into the subprocess via a parameter so that each subprocess is aware of its associated file.

In the epilog of the master process code must be added to ensure the master process waits until all subprocesses have completed. To do this, a while loop is created that will search for any associated files. Once all associated files have been deleted, the master process will continue. An example of this loop is included below. A few things need to be considered in the loop:

  • Ensure the loop is searching for the correct file names
  • Determine how long each loop will wait before checking for existing files again
  • Determine the total number of loops / total wait time before the loop assumes there is a major error within the subprocesses
    • Needs to be set so that it will not finish before normal process execution completes
    • Should not be set to be so long that too much time is spent waiting in case of a major error
    • Should cleanup all associated files if the loop does time out

Optionally, if the subprocess is set up to create error files, search for any existing error files to communicate or raise errors.

Code changes are also required within the subprocess to handle the files. A parameter must be added to the subprocess to receive the command parameter. This allows the subprocess to easily remove its associated file once complete. Also, at the end of the subprocess epilog, add an ExecuteCommand function to delete its associated file.

In addition, code can be added to the epilog to better communicate errors to the master file if checking for them; with the sProcess variable below being the name of the sub process.

That is what is needed to create a version of rudimentary communication between processes. In more complicated situations where a process is executing multiple master processes with their own sets of subprocesses that rely on process execution order, a more comprehensive implementation including the Synchronized function may be necessary. This is because these manual semaphores only exist until the end of the epilog tab is reached, not necessarily until the process has finished committing. For more on the Synchronized function and its application in this situation see: https://quebit.com/askquebit/planning-analytics-process-synchronization/ and: https://quebit.com/askquebit/advanced-synchronized-queueing-process-groups/