Application Timeouts

Over the course of the report execution cycle several watchdog timers are started to make sure that the process has a definite ending. If for some reason requests cannot be completed in a timely fashion, the timeouts will stop the process from hanging. This topic will define each timeout and describe the relationship between them and the entire report execution process.

Timeouts that can affect report execution can be found in the Admin Console, in the Scheduler Service configuration file and in the Application Settings.

Admin Console

These timeouts are defined in the Admin Console (and therefore in the Exago XML configuration file):

• General > Other Settings > Max Report Execution Time (minutes)
• General > Other Settings > Maximum Age for Temp Files
• General > Database Settings > Database Timeout and Roles > General > Database Timeout

Max Report Execution Time (minutes)

This timer begins at the start of report execution by the Web Application.

If a report takes longer to fully execute (including data queries, filtering, sorting, formula evaluation, aggregating and chart, map or gauge rendering) than this number of minutes, report execution will be canceled.

This timer operates asynchronously and can interrupt the report execution thread on its own. Report execution will progress until the timeout is reached at which time the thread will be stopped. The user interface will display an error dialog, and a FATAL message will be written to the log:

[Api.Execute.ActiveLocalExecution.IndicatePhaseException] System.Threading.ThreadAbortException: Thread was being aborted.
[Api.Execute.ActiveLocalExecution+<>c__DisplayClass14_0.<ExecuteAsync>b__0] General Exception in ExecuteAsync()
Type: System.Threading.ThreadAbortException
Message: Thread was being aborted.

The default value is 240 minutes.

Reports that are executed on a schedule, or by Remote Execution are not affected by this setting, but instead by the corresponding setting in the Scheduler Service’s configuration.

Maximum Age for Temp Files

As reports are executed, temporary support files are generated by the execution engine. These temporary files need to remain on the system for the duration of the execution cycle and for a reasonable amount of time after execution ends, as they will be referenced by the Report Viewer.

The Maximum Age for Temp Files determines the number of minutes before Exago may begin removing temporary files from the cache. If this timeout is too short, temporary files will be removed before report execution is completed or before the report is reviewed by the end user. If this occurs, users may notice content missing or error messages in the report outputs.

The minimum value is 30 minutes and the default value is 1440 minutes which translates to 24 hours. This value should always be greater than Max Report Execution Time (minutes).

Reports that are executed on a schedule, or by Remote Execution are not affected by this setting, but instead by the corresponding setting in the Scheduler Service’s configuration.

Database Timeout

Database Timeout is not a timer per se, but is a value that is passed to the data provider’s CommandTimeout property as Exago queries the business intelligence data source. It is up to the data provider what to do with this value. This timeout has no effect on connections to Folder Management or Storage Management databases, if applicable.

It is important to note that each individual data provider may implement the handling of this value differently. Therefore, its consequences in Exago may vary. This value is passed to the methods that open connections to Exago data sources except for .NET Assemblies and file system data sources (that is, Microsoft Excel or XML files).

For example, when a MySQL data source using the Devart.Data.MySql data provider hits the Database Timeout, an error dialog appears in the user interface and an ERROR message is written to the log:

ERROR [74cac7c2][cejppicx] [Api.Data.DbConnect.RegisterException] Lost connection to MySQL server during query
ERROR [74cac7c2][cejppicx] [Api.Execute.ExecuteReport.Process] Exception during report processing
Type: WebReports.Api.Common.WrDbConnectException
Message: DbConnect Exception Lost connection to MySQL server during query
Source: WebReportsApi

As another example, when a report executed by a Scheduler Service hits the Database Timeout on a Microsoft SQL Server source using the System.Data.SqlClient database provider, a different error dialog appears.

As the report has been executed by the Scheduler Service, the Scheduler Service’s log will contain an ERROR message:

[Api.Data.DbConnect.RegisterException] System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out

Database Timeout may be set in general by changing the value of General > Database Settings > Database Timeout or may be configured on a role-by-role basis by providing a value in Roles > General > Database Timeout. These values will also be used by reports that are executed by a Scheduler Service.

The default value is 600 seconds which translates to 5 minutes.

Scheduler Service Configuration

Scheduler Services used for both scheduled report execution and instantaneous remote execution of reports operate mostly independently of the Admin Console’s settings. The Scheduler Service references its own timeout values. These timeouts are defined the Scheduler Service’s configuration file:

• <max_job_execution_minutes>
• <default_job_timeout>
• <max_temp_file_age>
• <email_retry_time>
• <sleep_time>
• <flush_time>
• <sync_flush_time>
• <smtp_timeout>v2019.2.31+v2020.1.14+v2021.1.2+

max_job_execution_minutes and default_job_timeout

For versions pre-v2019.2.31, pre-v2020.1.14 and pre-v2020.1.2, both of these timers are started at the beginning of report execution and run simultaneously. They both perform the same function as Max Report Execution Time (minutes) but only in the context of reports executed by a Scheduler Service.

<default_job_timeout> also controls the SMTP server timeout. If the Scheduler Service is unable to send the job’s email out before <default_job_timeout> expires, it will try again after <email_retry_time> minutes (see below).

After the versions listed above, only max_job_execution_minutes applies, as default_job_timeout is no longer checked. In these versions, the SMTP timeout is set with the smtp_timeout setting.

Reports that take longer to execute than these timeouts will be stopped and marked abended. A Report has abended, Thread was being aborted or similar error dialog appears in the user interface for Remote Execution reports. The error message text may vary based on where in the execution cycle the thread is aborted. An ERROR message is written to the log file:

[Api.Execute.ActiveLocalExecution.IndicatePhaseException] System.Exception: The execution thread has been aborted, probably due to report execution timeout.
Caught exception running WebReportJob
System.Threading.ThreadAbortException: Thread was being aborted.

For both remote execution and scheduled reports, an email will be sent to the <error_report_to> address if configured.

Exago includes a default value for <default_job_timeout> of 3600 seconds in the configuration file. When configuring the Scheduler Service, the <max_job_execution_minutes>setting should be added to the file. Both of these values should be set to the same timeout period. It is important to note that <max_job_execution_minutes> is set in minutes and <default_job_timeout> is set in seconds. If the values are not the same, then the timeout that expires first will stop execution.

smtp_timeout

In versions of Exago v2019.2.31+, v2020.1.1.14+ and v2021.1.2+, this timeout replaces the behavior of <default_job_timeout> with respect to the SMTP server connection timeout.

max_temp_file_age

Just like Maximum Age for Temp Files this setting controls the deletion of temporary files created by Scheduler Services.

The minimum recommended value is 60 minutes. The default value is 1440 minutes which translates to 24 hours. This value should always be greater than max_job_execution_minutes and default_job_timeout.

email_retry_time

For Scheduler Service jobs that will be emailed, the service will wait this number of minutes if it cannot send the message before trying again. The Scheduler Service will make five attempts to send the email. If after five attempts the email still cannot go through, the job is marked Aborted.

The default value is 10 minutes. Therefore, email delivery could be delayed up to five times <email_retry_time> number of minutes (50 minutes by default).

sleep_time

The Scheduler Service will check every this number of seconds to see if there is work to do. This value pertains only to scheduled reports and not jobs for remote execution. While not directly attached to report execution, a job could be delayed by this number of seconds before execution begins.

The default value is 15 seconds.

flush_time and sync_flush_time

The <flush_time> and <sync_flush_time> timers are not connected to report execution, but are worth mentioning here. When Scheduler Service jobs are in a state where they will never run again (that is in a completed, deleted, aborted or abended (v2021.1.6+) state), they can be flushed from the job list. These two timers set the number of hours between automatic job flushes. Scheduled report jobs (including those created with the Schedule Wizard or email Report Immediately feature are removed every <flush_time> hours. Remote execution jobs are removed every <sync_flush_time> hours.

A value of 0 flushes the jobs as soon as they enter a completed, deleted, aborted or abended (v2021.1.6+) state. A value of -1 disables automatic flushing. The default values for <flush_time> and <sync_flush_time> are -1 and 0 respectively. Since remote execution (synchronous) jobs do not appear in the Schedule Manager, it is unlikely <sync_flush_time> will ever need to be changed from its default, as it is an internal queue.

Interaction Between Timeouts

Since there are some settings that affect report execution exclusively by the Web Application and some that affect execution exclusively by the Scheduler Service they will be considered separately in this section.

Report Execution by Web Application

The diagram below shows how the different timeouts interact when Remote Execution is not enabled. The narrative that follows will step through a general overview of the report execution cycle, highlighting the action of the timeouts.

1. A user requests the execution of a report. Report execution can be started one of two ways:
   • An interaction with the embedding host application, which in turn makes a call to Exago with one of the GetExecute() methods available in one of the APIs. This is illustrated by 1a and 2 in the figure.
   • Directly in the Web Application user interface, user clicks Run or Export to start report execution. This is illustrated by 1b in the figure.
2. Report execution begins and the Max Report Execution Time timer is started. If the time it takes for all of the proceeding steps to complete is longer than Max Report Execution Time, report execution will be canceled.
3. The Exago report execution engine will now query the business intelligence data sources needed for the report. For each query, the value of Database Timeout will be passed to the corresponding database provider. It is up to the database provider to handle this value.
4. The data sources return a data set to the execution engine for processing into the report. Depending on the database providers included, the time between steps 3 and 4 must be less than the Database Timeout (indicated by marker A in the figure); or only a partial data set may be returned or no data set at all. The handling of Database Timeout is up to the database provider, not Exago. Once the data set has been returned to Exago, this timeout becomes irrelevant.
5. If this report execution will result in a PDF, Excel workbook or RTF file, there are charts, maps or gauges to be rendered in the report and the Puppeteer rendering library is in use (available in v2019.1.5+, default in v2020.1+) then Exago will request one or more resources from Puppeteer to render them with. If no resource is available, Exago will wait maxWaitForPuppeteerResource number of seconds for one to become available. Puppeteer will immediately notify the rendering engine when it becomes available. If one does not become available before maxWaitForPuppeteerResource elapses (indicated by marker B in the figure), the chart, map or gauge will not be rendered and an “Unable to acquire a browser page. Ran out of time waiting for a resource to become available” error message will be displayed in its place. Once all widgets have been rendered, this timeout becomes irrelevant.
6. As the report is executed (with or without Puppeteer rendered widgets), Exago will create several temporary files. These temporary files are referenced by the execution engine, and by the Report Viewer after execution ends. A separate thread periodically deletes these temporary files, if the files are more than Maximum Age for Temp Files  number of minutes old.
7. The lifetime of temporary files are indicated by marker C in the figure. Exago should be able to read all of the temporary files needed to properly execute the report. Therefore, the Maximum Age for Temp Files should be set at the very minimum slightly above Max Report Execution Time (minutes) but reasonably longer allowing the files to remain for a satisfactory report viewing experience.
8. Finally, the completed report is returned to the user. All of the previous steps must occur within the Max Report Execution Time (minutes) timeout which was started in step 2, indicated by marker D in the figure. If they do not, an error message will be displayed to the end user and report execution stops.

Report Execution by Scheduler Service

The diagram below shows a basic interaction of the different timeouts for reports that are executed by a Scheduler Service. This could be either for a user requested report being executed as part of Remote Execution, or a scheduled report execution. The narrative that follows will step through a general overview of Remote Execution by a Scheduler Service, highlighting the action of the timeouts.

1. A user requests the execution of a report one of two ways:
   • An interaction with the embedding host application, which in turn makes a call to Exago with one of the GetExecute() methods available in one of the APIs. This is illustrated by 1a and 2a in the figure.
   • Directly in the Web Application user interface, user clicks Run or Export to start report execution. This is illustrated by 1b in the figure.
2. Report execution is forwarded to a Scheduler Service and marked as a synchronous job for immediate processing. Execution begins and the max_job_execution_timeout and default_job_timeout timers are started. If the time it takes for all of the proceeding steps to complete is longer than max_job_execution_timeout and default_job_timeout*, report execution will be canceled. * only in versions described above
3. The Scheduler Service’s report execution engine will now query the business intelligence data sources needed for the report. For each query, the value of Database Timeout will be passed to the corresponding database provider. It is up to the database provider to handle this value.
4. The data sources return a data set to the execution engine for processing into the report. Depending on the database provider(s) included, the time between steps 3 and 4 must be less than the Database Timeout (indicated by marker A in the figure); or only a partial data set may be returned. The handling of Database Timeout is up to the database provider, not Exago. Once the data set has been returned to Exago, this timeout becomes irrelevant.
5. If this report execution will result in a PDF, Excel or RTF file and there are charts, maps or gauges to be rendered in the report and the Puppeteer rendering library is in use (available in v2019.1.5+, default in v2020.1+) then the Scheduler Service will request one or more “pages” from Puppeteer to render them with. If no page is available, Exago will wait maxWaitForPuppeteerResource number seconds for one to become available. Puppeteer will immediately notify the rendering engine when a page becomes available. If one does not become available before maxWaitForPuppeteerResource elapses (indicated by marker B in the figure) the chart, map or gauge will not be rendered and an “Unable to acquire a browser page. Ran out of time waiting for a resource to become available” error message will be displayed in its place.
6. As the report is executed (with or without Puppeteer rendered widgets), the Scheduler Service will create several temporary files. A separate thread periodically deletes these temporary files, if the file is more than max_temp_file_age number of minutes old.
7. The lifetime of temporary files are indicated by marker C in the figure. Exago should be able to read all of the temporary files needed to properly execute the report. Therefore, the max_temp_file_age should be set at least slightly above max_job_execution_timeout and default_job_timeout.
8. Finally, the completed report is returned to the user. All of the previous steps must occur within the max_job_execution_timeout and default_job_timeout timer which was started in step 2, indicated by marker D in the figure.
9. Once the job is completed and will not run again, it will be removed from the Scheduler Service’s job list after the sync_flush_time timeout elapses.

Scheduled Jobs

For reports that weren’t triggered by a user action but are executed on a scheduled basis by a Scheduler Service, the execution steps generally remain the same with these differences:

1. The Scheduler Service will check for work every sleep_time number of seconds (indicated by marker G in the figure). If there is work to do, the job is loaded into the Scheduler Service and then execution resumes from step 2 above until step 8.
2. Resuming from step 8 above, the Scheduler Service now needs to deliver the report output. The file is either saved to a file repository or emailed. Assuming the file will be emailed, Exago will try to contact the configured SMTP server to send the message.
3. If Exago is unable to contact the SMTP server in default_job_timeout or smtp_timeout number of seconds depending on Exago version, it will stop and wait email_retry_time number of minutes before it tries again. It will try four more times to send the message otherwise it gives up and marks the job as Abended. This is not shown in the above figure.
4. Once the report output has successfully been emailed, the job will be re-queued or marked as Completed accordingly.
5. Every flush_time number of hours, the Scheduler Service will purge its job list of jobs that will never run again.

A report could be delayed by as much as minutes.