Spring @Scheduled job stopped? Know when it does

Your Spring Boot application is up, its health endpoint says UP, and a @Scheduled job has not run since Tuesday. Nothing crashed, nothing was logged. Here is how that happens, shown on a real run, and how to find out the day it starts instead of the day someone notices.

A job that throws is the easy case

Start with what does not stop a job. Take a method that always fails:

@Scheduled(fixedRate = 1000)
void failing() {
  throw new IllegalStateException("boom");
}

Run on Spring Boot 3.5.7, it logs this every second, and keeps being called:

ERROR 748 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task
java.lang.IllegalStateException: boom

Spring logs the exception of a repeating task and schedules the next run anyway. That is useful, and it is also why a failing job can go on failing for weeks: the error is in a log, and nobody reads that log until something downstream breaks.

The jobs that really go quiet are the next ones.

How a job stops without a trace

1. One stuck job freezes all the others

By default, Spring Boot runs every @Scheduled method on a single thread (spring.task.scheduling.pool.size is 1). If one of them never returns, because it waits on a socket without a timeout, a lock or a queue, none of the others runs again.

I reproduced it with three jobs: a heartbeat every second, the failing job above, and one that blocks forever from the fourth second:

17:39:40.449  heartbeat job ran
17:39:41.449  heartbeat job ran
17:39:41.451  stuck job starts and never returns
              (nothing after this line: no error, no warning, the process keeps running)

The heartbeat job and the failing job both stop at that moment. The application is alive, so a liveness probe and a health check are happy.

2. Scheduling is switched off

Without @EnableScheduling, a @Scheduled annotation is only an annotation: no error, no run. It is easy to lose in a refactoring of the configuration classes, and the application still starts and serves requests.

3. The schedule is disabled by a property

@Scheduled(cron = "${report.cron}") accepts the special value -, which Spring documents as a disabled trigger, meant to switch a job off from the outside. A wrong value in one environment’s configuration, and that job is no longer scheduled there.

4. The application is not running where you think

A job that runs in a single instance dies with it: a scaled-down deployment, a crash loop, an instance that was never started in that environment. Nothing in the application can report it, because the application is gone.

All four have the same signature: nothing happens, so nothing is written.

What does not catch it

• Logs. There is no line to search for. You can alert on an error; you cannot alert on a line that is not there.
• Health checks and liveness probes. They tell you the process answers, not that its jobs run.
• Metrics. A timer on each job helps, but catching a job that stopped means writing an alert on absence, for each job, and keeping those rules in step with the code.
• Reading the list of scheduled tasks. Actuator’s scheduledtasks endpoint shows what is registered. It is a list of intentions.

Look for the missing run: a dead man’s switch

The reliable signal is the one that fails in the right direction: the job reports every successful run to an outside service, which expects it on a schedule and alerts when it does not come. That is a dead man’s switch. A stuck thread, a missing annotation, a disabled cron and a dead instance all raise the same alert: the heartbeat did not arrive.

With the SilenceWatch Spring Boot starter, the setup is one dependency and one key:

pom.xml

<dependency>
  <groupId>com.silencewatch</groupId>
  <artifactId>silencewatch-spring-boot-starter</artifactId>
  <version>0.2.0</version>
</dependency>

application.yml

silencewatch:
  api-key: ${SILENCEWATCH_API_KEY}

At startup it finds every @Scheduled method and every Quartz job, declares them with their real schedule, and sends a heartbeat around each run, with its duration and any exception. The log says so:

SilenceWatch discovered 2 Spring scheduled task(s)
SilenceWatch is watching 2 job(s) in environment 'production'

The start of a run is reported separately from its end, and a start alone does not count as a success. A run that begins and never returns therefore leaves the check late, then down once its grace period has passed, and your alert channels are told. See the quick start to try it in five minutes, and check states and schedules for the exact rules.

The starter is built not to hurt the application: it never fails or blocks a job, and with no key or no network the application runs as without it. The details are on the Spring Boot page.

Fix the cause too

Being alerted is the first half. For the stuck-thread case, the usual remedies are:

• put a timeout on every outgoing call a job makes (HTTP client, database, queue): a job that can wait forever eventually will;
• give the scheduler more than one thread, so one slow job cannot hold the others back: spring.task.scheduling.pool.size: 5;
• keep long jobs out of the scheduler thread, by handing the work to an executor and returning.

Spring Boot quick start Get alerted when a job stops

Frequently asked questions

Why did my @Scheduled method stop running without an error?

The usual causes are another scheduled method that never returns (Spring Boot runs them all on one thread by default), scheduling not enabled, a cron disabled with the value "-", or an application instance that is no longer running. None of them writes a log line.

What happens when a @Scheduled method throws an exception?

For a repeating task, Spring logs the exception at ERROR level and runs the method again at the next time. The job does not stop, but it can fail on every run without anyone being told.

How many threads does Spring use for @Scheduled methods?

One by default in Spring Boot. Raise spring.task.scheduling.pool.size so that a slow job does not delay the others.

How can I be alerted when a scheduled job stops?

Have the job report each successful run to a service that expects it on a schedule and alerts when it is missing, a dead man’s switch. The SilenceWatch starter does it for @Scheduled and Quartz jobs with one dependency and one API key.

See also

• Java Spring Boot monitoring
• Spring Boot starter documentation
• Dead man’s switch
• Cron job not running