Process Maintenance Plan
Why Do I Need to Implement a Process Maintenance Plan?
The Clarity Process Engine is a powerful and versatile tool, but it also requires regular maintenance by administrators to keep it in a healthy state. Otherwise, it can potentially stall, become unresponsive, or slow down. Broadcom recommends regular well-planned and continuous maintenance of your process instances, log details, and completed jobs.
Further Information
If you want to find out more, you can read the following article for the official recommendations. Note: This guide is intended for Clarity administrators who have a good understanding of job scheduling and use the Clarity administration options in Classic UI. The aim is to enable you as a customer to take control of your processes and keep your Clarity instance health
Quick Wrap-up
If you keep the these tasks in mind, your process engine will stay healthy and will keep processing your process instances efficiently:
- Keep the BPM_ERRORS table small by keeping a low log level in the GEL script of the process and deleting process instances. We recommend not to keep more than 500k records in BPM_ERRORS. Anything over 1M needs a cleanup
- Use the non-prod environments if the GEL script log level needs to be increased for debugging purposes
- Keep the number of process instances small (do not exceed a total amount of 150.000) by scheduling the ‘Delete Process Instance’ clarity job accordingly
Process Instance Clean-Up
Why Do I Need to Clean Up Process Instances Regularly?
If you allow too many completed processes to accumulate on your system, they can cause performance problems on the Initiated Processes tab and eventually slow down your processes in general.
Additional Recommendations
In addition, your processes that are in an error state will be loaded each time you restart your process engine. This adds to the process load time after the restart and can cause process performance issues.
Every message in a process instance, every run, and every log entry creates entries in the BPM_ERRORS table and various other tables.
Clarity will perform a full table scan under specific conditions which will in turn cause performance issues. Keeping the BPM_ERRORS table entries small ensures a smooth-running process engine.
Process Engine Health
What Can You Do to Keep Your Process Engine Healthy?
The following recommendations are based on the Broadcom article “Process engine – keeping it healthy”. For more details see advice from Jeanne Gaskill of CA Clarity Support.
Clarity Process Maintenance Recommendations
- Delete old process instances in the done or aborted state.
- Retry or abort and delete old process instances in the error state.
- Remove process instances stuck in aborting state.
- Use only the minimum required log level in your GEL scripts in processes.
- Keep the BPM_ERRORS table small when it comes to the number of entries, which in turn depends on the actions outlined in steps 1-4.
- We recommend setting a policy/ schedule for how often this should be done so this doesn’t become a recurring and cumbersome manual task.
- Name the scheduled jobs by including the process name.
- If you deploy new versions of processes, you will have to configure a new set of maintenance jobs for that and make this part of your standard operating procedure.
- The first thing that you need to do is to decide and set a policy for how long you want to retain records of old completed and aborted processes instances.
- Process Instances are not designed as long-term storage. You can create a custom object and store the results of process instances in these for audit/ archiving purposes.
Process vs. Process Instance
Process
Defines the process with conditions and steps
Process Instance
Number of instances of the processes defined in the process definition
You may also refer to the overview of Clarity standard maintenance jobs and their options: “Clarity Jobs Reference”. Please note, that this requires a Broadcom Login. Please reach out to itdesign support to obtain an export of the Techdocs for your respective version of Clarity.
Settings for Delete Process Instance Job
How to Determine the Settings for the Delete Process Instance Job to Stay Within the Limits Given by Broadcom?
To determine a sensible setting for how many instances of a process can be deleted in each run, you need to evaluate how many instances you create in each run, and how often that process runs each day. The aim is to calculate how many instances you have in a relative period of time of each particular process you wish to delete. Note: This job cannot run concurrently with other instances of this job.
- You will need to run the job separately to clean up completed and aborted processes. This means that for each date range, you must run the job with Done selected in the Process Status field and a second time with Aborted selected and no date selected.
- Once you have chosen how long you want to keep your completed process instances in Done status, you will need to clean up anything older than the time period you have chosen. You should use the Delete Process Instance job to do this. Learn more about the functionality and options here: “Delete Process Instance Job”.
Note: it is important to ensure that you select specific dates so that you delete five thousand or fewer processes each time the job runs. Trying to delete too many processes at once will not only slow down the performance of the job itself but can also affect the performance for your users.
For example, if you have one year’s worth of process instances for a particular process, and you now have twenty thousand process instances to delete, try deleting two months’ worth at a time, starting with the oldest processes. You can also run the job on aborted processes only, or on completed processes only, to reduce the number of processes you delete at once. Make sure you schedule the purge outside of production hours and we recommend that you try this in your TEST environment first until you get a better feel for it.
This is an example, but please note that it depends on how many process instances you have in each state at any one time, so you need to look at the data carefully.
Cleaning Up Old Processes in the Error and Aborted State
Processes in the status ‘Error’ and ‘Aborted’ require regular intervention. These processes are loaded by the process engine during each restart and if you have a lot of them, they can eventually cause performance degradation with your processes. Customers who run very few processes may only need to do this once a month.
You can learn more from the official Knowledge Base Article:
Step-by-Step
To determine your need to clean up process instances, go to the Initiated Processes tab and search for processes in the error state.
When you are ready to clean up your errored processes, look at each process and determine the following:
- Determine whether this process instance is still needed. If it is not needed, abort the process.
- Check for any processes that you see repetitively in an error state. If all or many instances of the process are failing, you need to look at why this is happening. In many cases this indicates a process that has a problem with its logic. It is a good idea to go back to the process developer and ask them to fix the issue. If it is a problem with the process logic, put the process on hold until it can be fixed. You may need to ask the developer to give you a second version of the fixed process that continues from where the problem occurred so that you can run it on all of the object instances where it failed so that the process can be completed.
- If the process is still needed, double-click the error icon under the Messages column to see what the error says. If it gives you a checkbox next to the step, it means the problem is likely to be something you can fix. Read the error message and look at the process details page (go back to the list view and click on the process id) to determine what the problem is. Then go to the object instance indicated in the list view and fix the problem. Once the problem is fixed, you can click back into the error icon, click the checkbox beside the step and click the retry button.
Hint
Sometimes it is a good idea to retry the step even if you don’t understand how to fix the cause. If it was a temporary error, the step will succeed, and the process will continue. If you cannot determine what is causing the problem with a process or how to fix what looks to be an error you can retry, contact support for assistance. Please be aware that custom processes are out of scope of support agreements, but we may of course assist with a brief check or offer additional support via our other services
Retry or Abort and Delete Old Process Instances in the Error State
Once you have cleaned up the Processes in Error state, you should remove Aborted Process Instances. To learn more please refer to these two articles by Broadcom:
Remove Processes Stuck in Aborting
If you see process instances that are stuck in aborting, reach out to the Support Team to initiate a restart of the Process Engines. After a restart, wait for at least an hour to allow the process engines to catch up if you have many running processes and old instances.
If a restart does not resolve the process instances being stuck in aborting, you may refer to the following article to directly update the status via an approved query: “Processes stuck in Aborting status for long time”
Code
UPDATE bpm_run_processes
SET status_code = ‘BPM_PIS_ABORTED’
WHERE status_code =’BPM_PIS_ABORTING’
commit
Then Run the Job: Delete Process Instance Again
- Process Instance Status: Aborted
- Finish Date From: select the Specific Date radio button and leave the date blank
- Finish Date To: select the Specific Date radio button and leave the date blank
Recommendations on Process Design
Using GEL if GEL based Processes Seem to Be Stuck
You Might Encounter the Following
- Processes are stuck, there is process contention, the processes that have just 1 GEL script which takes 100 ms will stay stuck in Running and not move to Complete until 8-10 minutes later.
- Sometimes the entire system gets overloaded with processes because of this, and we have to restart BG (sometimes a few times) to get the processes to move along again
- We have many different processes with GEL scripts, on multiple objects and tables.
- Some of the GEL scripts are large and run in over 10 minutes.
To Resolve the Problem
- In Clarity, we have a maximum of 15 threads that can run Custom Action Execution Scripts (GEL). This is a setting that is separate from the pipelines and is not exposed in the database or the configuration.
- This means that if those 15 threads in BG are running the Custom Script Execution for 10 minutes each for each GEL, no other process can run and they will wait appearing stuck.
- The threads will just run the action until they are completed, then pick up the next actions.
- If by chance another one of the slow processes comes up, the bottleneck may continue.
- Restarting BG may change the process order and allow the other processes to be picked up first.
Our Recommendation
Redesigning the slow GEL scripts that take >10 min or making sure there are just few running at once, never over 5 at a time, ideally 1 slow running process at a given time.
Use Only the Minimum Required Log Level in Your Gel Scrips in Processes
The Process Engine Persistence Log Level indicates the level of log activity that you see if you include the <gel:log> tag in your process GEL script. You can learn more about the official definition in this part of the Techdocs:
Select from the following values to configure what type of messages you see on the Home, Organizer, Initiated – Processes page:
Error. The default value indicates that only messages with <gel:log level=ERROR> display as messages on the UI. We recommend this setting to keep the size of the BPM_ERRORS table small.
Warn. The value indicates that messages with display as messages on the UI.
Info. The value indicates that all messages, including the ones without any log level display as messages on the UI.
Our Recommendation
- The tag ''< gel:log level="INFO" >'' should only be used during development in the DEV environment of your Clarity instance.
- When you deploy a process in your production or keep it running on your non-prod environment, make sure to review your GEL scripts and reduce the log level to the required minimum.
- The “< gel:log> ” tag uses the BPM logger and does not offer a DEBUG log level. The official recommendation by Broadcom is the use of the ERROR Log Level.
Not Found What You're Looking For?
If you couldn't locate the information you need, feel free to contact us.
Our support team is ready to assist you.
