At a minimum, launching a batch job requires two things: theJob to be launched and aJobLauncher. Both can be contained within the samecontext or different contexts. For example, if you launch jobs from thecommand line, a new JVM is instantiated for each Job. Thus, everyjob has its own JobLauncher. However, ifyou run from within a web container that is within the scope of anHttpRequest, there is usually oneJobLauncher (configured for asynchronous joblaunching) that multiple requests invoke to launch their jobs.
Spring Batch: CommandLineJobRunner to Run Multiple Jobs
We have multiple Spring Batch jobs each running in their own java instance using the CommandLineJobRunner. All of the jobs are started simultaneously, only read/write flat files and update the same Spring Batch metadata hosted in SQL Server. The only database involved is the Spring Batch metadata database.
When the multiple jobs are started simultaneously we get SQL deadlock exceptions. A more detailed stack trace can be found below. From the database perspective we can see that the deadlock victims were doing one of the following: Insert into BATCH_JOB_SEQ default values or Delete from BATCH_JOB_SEQ where ID
In my investigation I think the issue is due to how the default incrementer class, org.springframework.jdbc.support.incrementer.SqlServerMaxValueIncrementer, increments the job and step instance ids in conjunction with how the SQL Server database tables are constructed. The code in SqlServerMaxValueIncrementer is synchronized so if we were running all the jobs in the same Java instance this would not be an issue.
I think that mentioning these settings in the Spring Batch documentation regarding launching multiple jobs at a given time when using SQL Server as your dataSource would be quite helpful to others. Then again, I guess not many people are stuck with SQL Server.
At a minimum, launching a batch job requires two things: the Job to be launched and a JobLauncher. Both can be contained within the same context or different contexts. For example, if launching a job from the command line, a new JVM will be instantiated for each Job, and thus every job will have its own JobLauncher. However, if running from within a web container within the scope of an HttpRequest, there will usually be one JobLauncher, configured for asynchronous job launching, that multiple requests will invoke to launch their jobs.
A batch process is typically encapsulated by a Job consisting of multiple steps, as shown in our batch processing example. A single ItemReader, ItemProcessor, and ItemWriter are usually present in each Step. A JobLauncher executes a Job and a JobRepository stores metadata about configured and executed jobs.
At a minimum, launching a batch job requires two things: the Job to be launched and a JobLauncher. Both can be contained within the same context or different contexts. For example, if launching a job from the command line, a new JVM will be instantiated for each Job, and thus every job will have its own JobLauncher. However, if running from within a web container within the scope of an HttpRequest, there will usually be one JobLauncher, configured for asynchronous job launching, that multiple requests will invoke to launch their jobs. 2ff7e9595c
Comments