Queue processor FAQ

A queue processor rule is an internal background process that you configure for queue management and asynchronous message processing. Use standard queue processor rules for simple queue management or low-throughput scenarios, or use dedicated queue processor rules for higher-scaling throughput, and customized or delayed processing of messages.

Best practice is to configure queue processors instead of using standard agents to manage message queuing in Pega Platform. For details, see Background processing best practices.

Frequently Asked Questions about queue processors include the following:

How do I queue a message to a queue processor?

You can queue a message to a queue processor by performing either of the following actions:

• Use the Queue-for-Processing activity method.
• Use the Run in Background SmartShape when you create a case.

For more information, see Queue-For-Processing method and Running a background process.

Can I queue a message to be processed at a scheduled time?

Yes. When you define a queue processor, set the When to process parameter to Delayed. Then, when you queue the message, you can specify the time to process the message.

For more information, see Queue-For-Processing method.

Do I have to create a queue processor to use this feature?

No. By default, Pega Platform includes standard queue processor rules. For more information, see Queue Processor rules.

How does a queue processor work, and where do I find information about how it runs?

The queue processor automatically generates a stream data set and a corresponding data flow. The stream data set sends messages to, and receives messages from, the stream service. The data flow manages the subscription of messages to ensure message processing.

Queue processor logical flow

You can view data flows that correspond with the queue processor rules in your system on the QueueProcessors landing page in Admin Studio. For more information see, Managing queue processors.

How do I configure the scaling requirements for my queue processor rules?

Queue processor rules can scale both horizontally and vertically. You can achieve horizontal scaling of queue processor rules by adding more nodes, and vertical scaling of queue processor rules by increasing the number of threads per node (on the queue processor rule form).

For more information, see Creating a Queue Processor rule.

Why is my new queue processor not working?

If your queue processor does not work, consider the following:

• Queue processor rules rely on the DSM service. In Pega Cloud Services and client-managed cloud environments, a stream node is automatically configured and ready to use. For on-premises environments, ensure that you define at least one node in a cluster to be a Stream node using the setting: -DNodeType=Stream. For more information, see Stream node configurations for a Queue Processor rule.
• One problem may be that Pega Platform is not be able to properly resolve your queue processor rule. For more information on how queue processor rules work, see System Runtime Context and Automating the runtime context management of background processes.
• If further troubleshooting is necessary, see Troubleshooting background processing.

What happens when I queue messages with no stream node running?

If a stream node is not available when the system queues a message, the system saves the message in a database, and then pushes the message to Kafka when the stream node becomes available again.

For more information, see Stream node configurations for a Queue Processor rule.

When I create a queue processor, what happens if I enter a large number of threads per node?

The maximum number of Kafka partitions is 20 for one Pega Platform cluster, which means that the maximum number of queue processor threads that can process messages across all nodes in a cluster is 20. This is not the global limit, but it is the limit for each queue processor.

How do I monitor and diagnose issues with queue processor rules?

In Admin Studio, the QueueProcessors landing page lists the queue processor rules that currently run on your system. From this landing page, you can open the DataFlow associated with every queue processor rule, with which you can monitor and diagnose the background processes. Also on this landing page, you can trace, enable, and disable your queue processor rules, or you can perform these same functions using REST APIs.

For more information, see Tracing a queue processor.

How do I increase the performance of my queue processor rules?

Performance has two dimensions: the time to process a message and the total message throughput. You can increase performance by completing one or both of the following actions:

• The time to process a message depends on the amount of work done by the processing activity. Optimize the activity to reduce the time needed to process a message.
• Enhance total message throughput in the following ways:
  • Scale out by increasing the number of processing nodes. This setting applies only to on-premises users. For Pega Cloud, a larger sandbox is required.
  • Scale up by increasing the number of threads per node, up to 20 threads per cluster.
  For more information, see Queue Processor rules.

How do I enable diagnostic logging for queue processor rules?

You can use background processing log categories to identify log entries that are relevant to and specific to background processing: job scheduler, queue processor, and agent log entries. For more information, see Details provided in background processing logs.

If the number of threads for a queue processor is adjusted, does it need a restart to take effect?

No. Just save the queue processor rule, and the change will take effect immediately.