Grafy News
Try Grafy free

Study finds trace-level AI fails at process-level forecasting

Researchers found that aggregating individual trace predictions leads to compounding errors, suggesting a need for dedicated process-level AI models.

Illustration for the news story: Study finds trace-level AI fails at process-level forecastingSource: Grafy Research Lab

Predictive process monitoring (PPM) research has historically focused on forecasting the future of individual process instances, but new research suggests this approach fails when applied to broader managerial decisions. According to a benchmarking study by Michael Grohs, Lukas Kirchdorfer, and Jana-Rebecca Rehse, aggregating individual "trace-level" predictions to estimate process-wide metrics results in significant error accumulation.

Why does trace-level prediction fail at the process level?

In typical predictive process monitoring, models operate at the trace level, meaning they attempt to predict the next steps or the remaining time for a single specific instance of a process. While this is useful for individual interventions, such as escalating a specific customer case, it does not necessarily translate to the needs of process managers.

Managers often require process-level insights to make high-level decisions, such as adjusting staffing levels to handle workloads or modifying process designs to improve throughput time. These decisions rely on Key Performance Indicators (KPIs) that represent the aggregate behavior of the entire system.

The study by Grohs et al. investigated whether existing trace-level models could be repurposed for these tasks by simply aggregating their outputs. The researchers found that this method consistently underperforms. Specifically, the authors noted that trace-level approaches tend to underestimate long-running traces, which causes errors to compound when those predictions are summed or averaged to reach a process-level conclusion. This suggests that process-level prediction is a distinct mathematical and modeling challenge rather than a simple extension of trace-level tasks.

What prediction tasks were tested in the study?

To evaluate the effectiveness of different modeling paradigms, the researchers focused on two specific process-level prediction tasks:

  1. Concurrent Trace Prediction (CTP): This task involves forecasting the number of traces that are simultaneously active within a process at any given time.
  2. Throughput Time Prediction (TTP): This task involves forecasting the average completion time for all traces in a process.

The benchmarking study utilized both synthetic and real-life event logs to compare a wide variety of methods. These included traditional statistical and machine learning (ML) methods, recently emerged tabular and time series foundation models, business process simulation approaches, and established trace-level PPM models that were aggregated to the process level.

Which modeling approaches were compared?

The researchers compared four distinct modeling paradigms to determine which is most suitable for aggregate forecasting:

  • Statistical and ML methods: Approaches traditionally used for forecasting process-level KPIs.
  • Foundation Models: Recently emerged models designed for tabular and time series data.
  • Process Simulation: Methods that use discrete-event or agent-based simulation to project future behavior by simulating the arrival and evolution of traces.
  • Aggregated Trace-level PPM: Existing state-of-the-art models (such as GLSTM, ProcessTransformer, or PGT) that were trained on individual traces and then mathematically aggregated to provide a process-level estimate.

The results indicated substantial variation in performance across these groups. While models that directly target process-level KPIs generally performed better than aggregated trace-level models, they were still often inaccurate, highlighting a significant gap in current predictive capabilities.

What are the implications for business process management?

The findings suggest that the current trend in PPM research—which heavily emphasizes improving the accuracy of individual trace predictions—may not be sufficient for supporting high-level organizational management. If a company relies on aggregating individual AI predictions to plan its workforce or capacity, it may face significant inaccuracies due to the compounding error effect identified in the study.

Instead, the authors argue that process-level prediction warrants its own dedicated research track. Rather than trying to "fix" trace-level models to work at scale, the industry may need to focus on developing models specifically designed to interpret aggregate signals, such as workload trends and system-wide throughput, as primary inputs.

How can researchers explore these findings?

For researchers and data scientists looking to dive deeper into the technical nuances of this study, including the mathematical definitions of the prediction tasks and the specific architectures used in the benchmark, the full paper provides an extensive breakdown of the experimental setup and results.

If you are currently reviewing similar academic literature or technical papers on process mining and machine learning, you can use Grafy Reader to upload the PDF. This allows you to ask specific questions about the error compounding mechanics or the performance of the foundation models mentioned in the study, helping you synthesize the research more efficiently.

Frequently asked questions

Does aggregating trace-level predictions work for process managers?

No, aggregating trace-level predictions is generally ineffective for process-level management. According to the study, these aggregated predictions cause errors to compound, particularly because trace-level models often underestimate long-running traces. This leads to inaccurate forecasts for aggregate metrics like workload or average throughput time.

What are the two main prediction tasks identified in the research?

The research focuses on Concurrent Trace Prediction (CTP), which forecasts the number of traces active at the same time, and Throughput Time Prediction (TTP), which forecasts the average time it takes for traces to complete. Both tasks represent aggregate process-level behaviors rather than individual instance behaviors.

Why is process-level prediction considered a distinct task?

Process-level prediction is considered distinct because it requires forecasting aggregate KPIs rather than individual trace suffixes or remaining times. Because errors from individual traces accumulate when combined, models must be specifically designed to handle the complexities of aggregate signals rather than being repurposed from trace-level models.

Source: Michael Grohs, Lukas Kirchdorfer, and Jana-Rebecca Rehse, "Predicting Process-Level Behavior: A Benchmarking Study"


Source: Grafy Research Lab — Predicting Process-Level Behavior: A Benchmarking Study