Scheduled Jobs

 

TopBraid EDG and TBL include a mechanism for running SWP scripts on a schedule.

Example use cases. This can be used in EDG customizations and TBL applications to automate recurring tasks, such as:

  • Checking an external source for updates and importing the new version
  • Pre-computing data for expensive dashboard charts
  • Archiving off old data every night
  • Re-training the AutoClassifier once a week

Limitations. But the mechanism is not intended for some other things:

  • Background tasks triggered immediately after a user action (e.g., long-running imports, inference or validation tasks)
  • Execution intervals shorter than a minute or so (due to the overhead of starting up an SWP engine)
  • Jobs that require recovery or re-tries in case of job failures or server downtime (see section on server restarts and server downtime below)
  • Jobs written in Java

All jobs run with administrator privileges, so it is up to the SWP programmer to implement access control if needed.

Creating a scheduled job through TBCME

  1. Create a new .ui.ttlx file (or use an existing one)
  2. Import server.topbraidlive.org/web/2018/scheduler.ttl
  3. Create a new subclass of scheduler:ScheduledJobs; it will automatically also become an instance of scheduler:ScheduledJob
  4. Add the property scheduler:cronSchedule to specify the schedule; 0 * * * * ? (“every minute”) can be a good value for testing; see further below for syntax
  5. Write SWP code in ui:prototype
  6. TBCME menu: System > Refresh TopBraid system registries
  7. Check TBCME Error Log for any reported errors/warnings
  8. If there are no errors, the job should be active!
  9. If the job is long-running (more than a minute or so), make sure to read the section below on long-running jobs

Deploying a scheduled job to EDG

Simply deploy a project containing the .ui.ttlx files with scheduled jobs to EDG. The jobs should become active immediately. The scheduler logs to the TBS Log.

Reviewing and troubleshooting scheduled jobs

In the EDG Server Administration there is an item “Scheduled Jobs” that opens a page with a table of all scheduler:ScheduledJob instances, with details about the job’s status.

  1. If a job is not listed there, then verify that it is indeed an instance of scheduler:ScheduledJob, and that it is part of the UI graph (in a .ui.ttlx file).
  2. Refresh registries to ensure that the scheduler has picked up any changes.
  3. Check the log for any errors or warnings about problems with the job definition.
  4. Check the next execution time in the table to ensure the schedule is correct.
  5. When a job is triggered, its start and completion are logged as an Info message.
  6. If a job aborts with an exception, a Warning with the stack trace will be logged.
  7. While a job is running, its runtime an other details will be shown in the table. This can be used to identify stuck or slow jobs.
  8. Sending debug information to the log from SWP: <ui:log ui:debug="message"/>

Issues with long-running jobs

  • TBCME shutdown and EDG server restarts are blocked while any jobs are executing. The system will wait for all jobs to finish, unless the job is written to be cancelable. Therefore, any jobs that take longer than a minute or so should be written to be cancelable, see below.
  • Job execution uses a thread pool whose size (3) is hardcoded. There cannot be more than this number of jobs executing in parallel. If all threads are busy, job executions are delayed until a thread is free.
  • If a job execution takes longer than the schedule interval (e.g., an hourly job takes two hours), then the next execution will be delayed until the current execution has finished.

Making jobs cancelable and reporting progress with ui:task and ui:subTask

Long-running jobs (longer than a minute or so) should be written using <ui:task> and <ui:subTask> to break the job down into smaller independent chunks. This has two benefits:

  1. It makes the job cancelable. If the system shuts down while the job is being executed, the current ui:task and ui:subTask will finish, but all subsequent tasks and subtasks will be skipped.
  2. It allows reporting of the job’s progress. Progress can be checked in a number of ways:
    • On the Scheduled Jobs page in the EDG Server Administration
    • From SWP, using the ui:progressMonitorStatus magic property
    • From JavaScript, with swa.openProgressMonitorDialog

Manually cancelling job execution

There is no UI for cancelling running jobs, but it can be done manually:

  1. In the table on the Scheduled Jobs page, in the “Currently running?” column, there is a link to the URI that identifies the current job execution. Copy it.
  2. Run the following SPARQL query, with the right job execution URI: (in the TBCME SPARQL pane or EDG SPARQL endpoint form):
SELECT ui:cancelProgress(<urn:x-job-execution:1527863743785>) {}

Note: A running job may not stop immediately if cancelled. That is because certain SWP commands/actions are “atomic” and need to finish before the SWP engine can stop the running script. If the command/action takes a long time, it may seem like the cancellation doesn’t do anything. If the cancel is not working, you can restart the server and this will cancel it. 

Server restarts and server downtime

The system cannot guarantee that scheduled jobs are actually executed! In particular:

  • If an exception occurs during job execution, the execution will be aborted. The system will try again at the next scheduled time.
  • If a job execution is in progress when the server is shut down or restarted, the current ui:taskand ui:subTask will finish, but all subsequent tasks and subtasks will be skipped. The system will not try to recover or continue when the server comes back up. It will execute the job again at the next scheduled time.
  • If the scheduler is not running at the scheduled execution time (e.g., because it is in the middle of a server restart), then that execution is skipped completely.

An example job

This is the ui:prototype of a job that simply sleeps 100x for 10 seconds each. It reports its progress to the Scheduled Jobs admin page, and supports cancellation after each ten-second sleep chunk.


<ui:group>
    <ui:log ui:info="At start of sleep job"/>
    <ui:task ui:taskName="Sleeping" ui:totalWork="100">
        <ui:forEach ui:resultSet="{#
                SELECT ?i
                WHERE {
                    ?i tops:for ( 1 100 ) .
                } }">
            <ui:subTask ui:subTaskName="chunk {= ?i }" ui:work="1">
                <scheduler:Sleep arg:ms="{= 10000 }"/>
            </ui:subTask>
        </ui:forEach>
    </ui:task>
    <ui:log ui:info="At end of sleep job"/>
</ui:group>

Expressing schedules with Quartz cron expressions

Cron expressions are a syntax for expressing recurring events. We use the Quartz flavour of cron expressions. The full syntax is described in this tutorial, but the basics are:

sec min hour day month day-of-week [year]

In the simplest case, each position is either a number, or * to indicate “every”. The day-of-weekslot is ? to indicate “whatever the day happens to fall on”. The year slot is absent. Examples:

  • 0 * * * * ? – At the start of every minute
  • 0 15 3 * * ? – 3:15am every night
  • 0 0 12 ? * MON-FRI – Every weekday at noon
  • 0 */10 * * * ? – Every ten minutes
  • 0 0 6,9,12,15,18 * * ? – Every three hours from 6am to 6pm