Blog
Spring @Scheduled job stopped? Know when it does
· updated
A Spring Boot @Scheduled job that stops without an error is usually waiting behind another job stuck on the scheduler’s single thread; scheduling switched off, a cron set to - or a missing instance do the same. None of them logs anything, and the health check stays UP. Here is how to confirm each cause, fix it, and be alerted the day it starts.
Does an exception stop a @Scheduled job?
Section titled “Does an exception stop a @Scheduled job?”Start with what does not stop a job. Take a method that always fails:
@Scheduled(fixedRate = 1000)void failing() { throw new IllegalStateException("boom");}Run on Spring Boot 3.5.7, it logs this every second, and keeps being called:
ERROR 748 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled taskjava.lang.IllegalStateException: boomSpring logs the exception of a repeating task and schedules the next run anyway. That is useful, and it is also why a failing job can go on failing for weeks: the error is in a log, and nobody reads that log until something downstream breaks.
The jobs that really go quiet are the next ones.
Why a @Scheduled job stops without an error
Section titled “Why a @Scheduled job stops without an error”1. One stuck job freezes all the others
Section titled “1. One stuck job freezes all the others”By default, Spring Boot runs every @Scheduled method on a single thread (spring.task.scheduling.pool.size is 1). If one of them never returns, because it waits on a socket without a timeout, a lock or a queue, none of the others runs again.
I reproduced it with three jobs: a heartbeat every second, the failing job above, and one that blocks forever from the fourth second:
17:39:40.449 heartbeat job ran17:39:41.449 heartbeat job ran17:39:41.451 stuck job starts and never returns (nothing after this line: no error, no warning, the process keeps running)The heartbeat job and the failing job both stop at that moment. The application is alive, so a liveness probe and a health check are happy.
To confirm, take a thread dump (jstack <pid> or Actuator’s threaddump endpoint; only health is exposed over HTTP by default, so add management.endpoints.web.exposure.include: scheduledtasks,threaddump,env for the checks below): scheduling-1 sits in the stuck method, in a socket read or waiting on a lock.
2. Scheduling is switched off
Section titled “2. Scheduling is switched off”Without @EnableScheduling, a @Scheduled annotation is only an annotation: no error, no run. It is easy to lose in a refactoring of the configuration classes, and the application still starts and serves requests. To confirm, look at Actuator’s scheduledtasks endpoint: it lists no task.
3. The schedule is disabled by a property
Section titled “3. The schedule is disabled by a property”@Scheduled(cron = "${report.cron}") accepts the special value -, which Spring documents as a disabled trigger, meant to switch a job off from the outside. A - left in one environment’s configuration (copied from a test profile, say), and that job is never scheduled there. Any other invalid value fails at startup instead. To confirm, note that the task is missing from scheduledtasks, and look for report.cron in that environment’s configuration. /actuator/env/report.cron shows where the property comes from, but Spring Boot 3 masks its value unless management.endpoint.env.show-values is set.
4. The application is not running where you think
Section titled “4. The application is not running where you think”A job that runs in a single instance dies with it: a scaled-down deployment, a crash loop, an instance that was never started in that environment. Nothing in the application can report it, because the application is gone. To confirm, check the replica count and restarts (kubectl get deploy,pods).
All four have the same signature: nothing happens, so nothing is written.
Why logs and health checks miss it
Section titled “Why logs and health checks miss it”- Logs. There is no line to search for. You can alert on an error; you cannot alert on a line that is not there.
- Health checks and liveness probes. They tell you the process answers, not that its jobs run.
- Metrics. A timer on each job helps, but catching a job that stopped means writing an alert on absence, for each job, and keeping those rules in step with the code.
- Reading the list of scheduled tasks. Actuator’s
scheduledtasksendpoint shows each task’s next and last execution, so a stuck run is visible there if someone looks. Nothing alerts on it.
How to be alerted when a scheduled job stops
Section titled “How to be alerted when a scheduled job stops”The reliable signal is the one that fails in the right direction: the job reports every successful run to an outside service, which expects it on a schedule and alerts when it does not come. That is a dead man’s switch. A stuck thread, a missing annotation, a disabled cron and a dead instance all raise the same alert: the heartbeat did not arrive.
With the SilenceWatch Spring Boot starter, the setup is one dependency and one key:
<dependency> <groupId>com.silencewatch</groupId> <artifactId>silencewatch-spring-boot-starter</artifactId> <version>0.2.0</version></dependency>silencewatch: api-key: ${SILENCEWATCH_API_KEY}At startup it finds every @Scheduled method and every Quartz job, declares them with their real schedule, and sends a heartbeat around each run, with its duration and whether it returned or threw; a run that throws turns the check down. The log says so:
SilenceWatch discovered 2 Spring scheduled task(s)SilenceWatch is watching 2 job(s) in environment 'production'The starter declares the jobs Spring actually scheduled. A job switched off by -, or an application that lost @EnableScheduling, is caught if its check was declared before: the check stays and goes down. A job that has never run in an environment, or an instance never started there, has no check yet; create that one by hand.
The start of a run is reported separately from its end, and a start alone does not count as a success. A run that begins and never returns therefore leaves the check late, then down once its grace period has passed, and your alert channels are told. One exception: when several instances run the same job without a lock, they share one check, and a scheduler stuck on one instance is hidden by the others. List those jobs under silencewatch.per-instance-jobs, with a stable instance-id, to give each instance its own check (several instances). See the quick start to try it, and check states and schedules for the exact rules.
The starter is built not to hurt the application: it never fails or blocks a job, and with no key or no network the application runs as without it. The details are on the Spring Boot page.
How to fix a stuck scheduler
Section titled “How to fix a stuck scheduler”Being alerted is the first half. For the stuck-thread case, the usual remedies are:
- put a timeout on every outgoing call a job makes (HTTP client, database, queue): a job that can wait forever eventually will;
- give the scheduler more than one thread, so one slow job cannot hold the others back:
spring.task.scheduling.pool.size: 5; - keep long jobs out of the scheduler thread, by handing the work to an executor and returning; the heartbeat then marks the hand-off, not the work, so give the work itself a check (a ping at its end) or prefer the two fixes above.
Frequently asked questions
Section titled “Frequently asked questions”Why did my @Scheduled method stop running without an error?
The usual causes are another scheduled method that never returns (Spring Boot runs them all on one thread by default), scheduling not enabled, a cron disabled with the value "-", or an application instance that is no longer running. None of them writes a log line.
What happens when a @Scheduled method throws an exception?
For a repeating task, Spring logs the exception at ERROR level and runs the method again at its next scheduled time. The job does not stop, but it can fail on every run without anyone being told.
How many threads does Spring use for @Scheduled methods?
One by default in Spring Boot. Raise spring.task.scheduling.pool.size so that a slow job does not delay the others.
How can I be alerted when a scheduled job stops?
Have the job report each successful run to a service that expects it on a schedule and alerts when it is missing, a dead man’s switch. The SilenceWatch starter does it for @Scheduled and Quartz jobs with one dependency and one API key.