A Benchmarking Suite for Flexible Job Shop Scheduling Problems with Worker Flexibility under Uncertainty
Pith reviewed 2026-05-23 04:49 UTC · model grok-4.3
The pith
A new benchmarking suite extends 402 standard flexible job shop instances to include worker flexibility and uncertainty.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper creates a hitherto unique collection of ready-to-use worker flexibility instances by systematically extending 402 standardized Flexible Job Shop Scheduling Problem instances. The resulting benchmark suite supplies several metrics for algorithm performance assessment, visualization of algorithmic results, and state-of-the-art baseline results, thereby enabling rigorous, reproducible, and comparable performance analysis between solvers and scheduling problem subdomains. Through simulation-based integration of uncertainties in processing times and resource availabilities, the environment supports development and evaluation of robust optimization strategies.
What carries the argument
The benchmarking suite that systematically extends 402 FJSP instances with worker flexibility and uncertainty simulation.
If this is right
- Solvers from mathematical programming, constraint programming, and simulation-based optimization can be evaluated on identical worker-flexibility problems.
- Robust strategies can be tested against simulated changes in processing times and resource availability.
- Algorithm results can be assessed with standardized metrics and visualized for direct comparison.
- New algorithms for production scheduling can be validated against provided state-of-the-art baselines.
Where Pith is reading between the lines
- The suite could be used to identify which solver domains handle worker flexibility most efficiently under uncertainty.
- Manufacturers might adopt the instances as standard test cases when evaluating scheduling software for flexible workforces.
- Future extensions could add more uncertainty types while keeping the same base instances for longitudinal comparisons.
Load-bearing premise
The 402 standardized FJSP instances can be systematically extended to include worker flexibility while remaining representative and useful for cross-domain solver comparison and uncertainty simulation.
What would settle it
Running the same set of solvers on the new worker-flexibility instances and finding that performance rankings differ sharply from their rankings on the original FJSP instances without worker flexibility would indicate the extensions do not preserve comparability.
Figures
read the original abstract
This paper addresses the Flexible Job Shop Scheduling Problem and its extension with Worker Flexibility, which integrates workforce assignment into machine-operation scheduling. Diverse solvers have been proposed across multiple optimization domains including Mathematical Programming, Constraint Programming, and Simulation-Based Optimization, or Simulation-based Optimization. These are often tailored to narrow use cases and validated on limited test problem sets, hindering cross-domain comparison. To overcome this, a comprehensive benchmarking environment built on 402 standardized Flexible Job Shop Scheduling Problem instances is introduced and systematically extended to include worker flexibility. This creates a hitherto unique collection of ready-to-use worker flexibility instances. The benchmark suite features several metrics for algorithm performance assessment, the visualization of algorithmic results, as well as state-of-the-art baseline results. This enables rigorous, reproducible, and comparable performance analysis between solvers and scheduling problem subdomains. Through the simulation-based integration of uncertainties in processing times as well as resource availabilities, the environment supports the development and evaluation of robust optimization strategies. The present work lays a foundation for targeted algorithm development and consistent performance evaluation in production scheduling research.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces a benchmarking suite for the Flexible Job Shop Scheduling Problem (FJSP) with worker flexibility under uncertainty. It extends 402 standardized FJSP instances to include worker flexibility, supplies performance metrics, visualizations of results, state-of-the-art baselines, and simulation-based support for uncertainties in processing times and resource availabilities, with the goal of enabling rigorous, reproducible, and comparable evaluations across solvers from different optimization domains.
Significance. If the extension procedure is sound and the instances remain representative, the work supplies a useful infrastructure contribution by creating a previously unavailable collection of ready-to-use worker-flexibility instances together with standardized evaluation tools. This could facilitate cross-domain comparisons and robust optimization research in production scheduling.
major comments (1)
- [Abstract] Abstract: the description of benchmark construction supplies no details on instance generation, validation procedures, or baseline computation methods, preventing verification that the systematic extension of the 402 instances remains representative for cross-domain solver comparison and uncertainty simulation.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback and positive assessment of the work's potential contribution. We address the single major comment below.
read point-by-point responses
-
Referee: [Abstract] Abstract: the description of benchmark construction supplies no details on instance generation, validation procedures, or baseline computation methods, preventing verification that the systematic extension of the 402 instances remains representative for cross-domain solver comparison and uncertainty simulation.
Authors: We agree the abstract is high-level and omits these specifics. The full manuscript details the systematic extension of the 402 instances in Section 3 (including how worker flexibility and uncertainty parameters were added while preserving original problem characteristics), validation via statistical property checks in Section 3.3, and baseline solver computations (CP, MIP, and metaheuristics) in Sections 5 and 6. To improve accessibility, we will revise the abstract to briefly reference the extension method and validation steps. revision: yes
Circularity Check
No significant circularity
full rationale
The paper presents an infrastructural contribution: the systematic extension of 402 existing standardized FJSP instances to worker flexibility, plus metrics, visualizations, baselines, and uncertainty simulation support. No derivation, parameter fitting, or theorem is claimed; the central claim is the creation of a ready-to-use benchmark collection. No self-citation load-bearing steps, no self-definitional reductions, and no fitted inputs presented as predictions appear in the provided text. The work is self-contained against external standardized instances.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
J. Xie, L. Gao, K. Peng, X. Li, H. Li, Review on flexible job shop scheduling, IET collaborative intelligent manufacturing 1 (3) (2019) 67–77
work page 2019
-
[3]
S. Dauz` ere-P´ er` es, J. Ding, L. Shen, K. Tamssaouet, The flexible job shop scheduling problem: A review, European Journal of Operational Research 314 (2) (2024) 409–432. doi:10.1016/j.ejor.2023.05.017. URL https://www.sciencedirect.com/science/article/pii/ S037722172300382X
-
[4]
D. Behnke, M. J. Geiger, Test Instances for the Flexible Job Shop Scheduling Problem with Work Centers, Working Paper, accepted: 2017-10-24T14:10:24Z (2012). doi: 10.24405/436. URL https://openhsu.ub.hsu-hh.de/handle/10.24405/436 25
-
[5]
D. S. Johnson, Experimental analysis of algorithms, Data Structures, Near Neighbor Searches, and Methodology: 5th and 6th DIMACS Implementation Challenges: Pa- pers Related to the DIMACS Challenge on Dictionaries and Priority Queues (1995/96) and the DIMACS Challenge on Near Neighbor Searches (1998/99) 59 (2002) 215
work page 1995
-
[6]
R. L. Rardin, R. Uzsoy, Experimental evaluation of heuristic optimization algorithms: A tutorial, Journal of Heuristics 7 (2001) 261–304
work page 2001
-
[7]
O. Mersmann, M. Preuss, H. Trautmann, B. Bischl, C. Weihs, Analyzing the bbob results by means of benchmarking concepts, Evolutionary Computation 23 (1) (2015) 161–185
work page 2015
-
[8]
D. Whitley, S. Rana, J. Dzubera, K. E. Mathias, Evaluating evolutionary algorithms, Artificial intelligence 85 (1-2) (1996) 245–276
work page 1996
-
[9]
J. J. Mor´ e, S. M. Wild, Benchmarking derivative-free optimization algorithms, SIAM Journal on Optimization 20 (1) (2009) 172–191
work page 2009
-
[10]
M. Hellwig, H.-G. Beyer, Benchmarking evolutionary algorithms for single objective real-valued constrained optimization – a critical review, Swarm and Evolutionary Computation 44 (2019) 927–944. doi:https://doi.org/10.1016/j.swevo.2018. 10.002. URL https://www.sciencedirect.com/science/article/pii/ S2210650218305406
- [11]
-
[12]
doi:10.1109/MCI.2014.2326101
-
[13]
R. Reijnen, I. G. Smit, H. Zhang, Y. Wu, Z. Bukhsh, Y. Zhang, Job Shop Scheduling Benchmark: Environments and Instances for Learning and Non-learning Methods, arxivArXiv:2308.12794 [cs] (Mar. 2025). doi:10.48550/arXiv.2308.12794. URL http://arxiv.org/abs/2308.12794
- [14]
-
[15]
SchedulingLab/fjsp-instances, original-date: 2022-08-28T19:33:39Z (Mar. 2025). URL https://github.com/SchedulingLab/fjsp-instances
work page 2022
-
[16]
R. Reijnen, K. van Straaten, Z. Bukhsh, Y. Zhang, Job Shop Scheduling Benchmark Environments and Instances (Mar. 2025). 26 URL https://github.com/ai-for-decision-making-tue/Job_Shop_Scheduling_ Benchmark_Environments_and_Instances
work page 2025
-
[17]
Leo, Lei-Kun/FJSP-benchmarks, original-date: 2022-09-01T05:43:14Z (Mar. 2025). URL https://github.com/Lei-Kun/FJSP-benchmarks
work page 2022
-
[18]
D. Trentesaux, C. Pach, A. Bekrar, Y. Sallez, T. Berger, T. Bonte, P. Leit˜ ao, J. Bar- bosa, Benchmarking flexible job-shop scheduling and control systems, Control Engi- neering Practice 21 (9) (2013) 1204–1225.doi:10.1016/j.conengprac.2013.05.004. URL https://www.sciencedirect.com/science/article/pii/ S0967066113000889
-
[19]
A. Ghasemi, F. Farajzadeh, C. Heavey, J. Fowler, C. T. Papadopoulos, Simulation optimization applied to production scheduling in the era of industry 4.0: A review and future roadmap, Journal of Industrial Information Integration 39 (2024) 100599. doi:10.1016/j.jii.2024.100599. URL https://www.sciencedirect.com/science/article/pii/ S2452414X24000438
-
[20]
S. Peng, T. Li, J. Zhao, Y. Guo, S. Lv, G. Z. Tan, H. Zhang, Petri net-based scheduling strategy and energy modeling for the cylinder block remanufacturing under uncertainty, Robotics and Computer-Integrated Manufacturing 58 (2019) 208–219. doi:10.1016/j.rcim.2019.03.004. URL https://www.sciencedirect.com/science/article/pii/ S0736584518303806
-
[21]
S. Cavalieri, S. Terzi, M. Macchi, A Benchmarking Service for the evaluation and comparison of scheduling techniques, Computers in Industry 58 (7) (2007) 656–666. doi:10.1016/j.compind.2007.05.004. URL https://www.sciencedirect.com/science/article/pii/ S016636150700070X
-
[22]
A. T¨ urkyilmaz, O. Senvar, I.¨Unal, S. Bulkan, A research survey: heuristic approaches for solving multi objective flexible job shop problems, Journal of Intelligent Manufac- turing 31 (8) (2020) 1949–1983. doi:10.1007/s10845-020-01547-4. URL http://link.springer.com/10.1007/s10845-020-01547-4
-
[23]
O. Hazir, M. Haouari, E. Erel, Robust scheduling and robustness measures for the discrete time/cost trade-off problem, European Journal of Operational Research 207 (2) (2010) 633–643. doi:10.1016/j.ejor.2010.05.046. URL https://www.sciencedirect.com/science/article/pii/ S0377221710004029 27
-
[24]
A. Gnanavelbabu, R. H. Caldeira, T. Vaidyanathan, A simulation-based modified backtracking search algorithm for multi-objective stochastic flexible job shop schedul- ing problem with worker flexibility, Applied Soft Computing 113 (2021) 107960. doi:10.1016/j.asoc.2021.107960
-
[25]
Q. Luo, Q. Deng, G. Gong, X. Guo, X. Liu, A distributed flexible job shop schedul- ing problem considering worker arrangement using an improved memetic algorithm, Expert Systems with Applications 207 (2022) 117984. doi:10.1016/j.eswa.2022. 117984
-
[26]
G. Gong, R. Chiong, Q. Deng, X. Gong, A hybrid artificial bee colony algorithm for flexible job shop scheduling with worker flexibility, International journal of production research 58 (14) (2020) 4406–4420
work page 2020
-
[27]
G. Gong, R. Chiong, Q. Deng, W. Han, L. Zhang, W. Lin, K. Li, Energy-efficient flexible flow shop scheduling with worker flexibility, Expert Systems with Applications 141 (2020) 112902
work page 2020
-
[28]
Gurobi Optimization, LLC, Gurobi Optimizer Reference Manual (2023). URL https://www.gurobi.com
work page 2023
- [29]
-
[30]
URL https://www.ibm.com/products/ilog-cplex-optimization-studio/ cplex-cp-optimizer
Constraint program solvers - IBM CPLEX (May 2024). URL https://www.ibm.com/products/ilog-cplex-optimization-studio/ cplex-cp-optimizer
work page 2024
-
[31]
URL https://developers.google.com/optimization
OR-Tools | Google for Developers (2024). URL https://developers.google.com/optimization
work page 2024
-
[32]
M. R. Garey, D. S. Johnson, Computers and Intractability; A Guide to the Theory of NP-Completeness, W. H. Freeman & Co., USA, 1990
work page 1990
-
[33]
Critical current of a Josephson junction containing a conical magnet
D. Hutter, T. Steinberger, M. Hellwig, An Interior-point Genetic Algorithm with Restarts for Flexible Job Shop Scheduling Problems, in: 2024 IEEE Congress on Evolutionary Computation (CEC), 2024, pp. 01–09. doi:10.1109/CEC60901.2024. 10611934
work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/cec60901.2024 2024
-
[34]
E. G. Birgin, P. Feofiloff, C. G. Fernandes, E. L. de Melo, M. T. I. Oshiro, D. P. Ronconi, A MILP model for an extended version of the Flexible Job Shop Problem, Optimization Letters 8 (4) (2014) 1417–1431. doi:10.1007/s11590-013-0669-7
-
[35]
E. Demirkol, S. Mehta, R. Uzsoy, Benchmarks for shop scheduling problems, Eu- ropean Journal of Operational Research 109 (1) (1998) 137–141. doi:10.1016/ S0377-2217(97)00019-2. 28
work page 1998
-
[36]
P. Brandimarte, Routing and scheduling in a flexible job shop by tabu search, Annals of Operations Research 41 (3) (1993) 157–183. doi:10.1007/BF02023073. URL https://doi.org/10.1007/BF02023073
-
[37]
J. W. BARNES, J. B. CHAMBERS, Solving the job shop scheduling prob- lem with tabu search, IIE Transactions 27 (2) (1995) 257–263. doi:10.1080/ 07408179508936739. URL https://doi.org/10.1080/07408179508936739
-
[38]
S. Dauz` ere-P´ er` es, J. Paulli, An integrated approach for modeling and solving the general multiprocessor job-shop scheduling problem using tabu search, Annals of Op- erations Research 70 (0) (1997) 281–306. doi:10.1023/A:1018930406487. URL https://doi.org/10.1023/A:1018930406487
-
[39]
P. Fattahi, M. Saidi Mehrabad, F. Jolai, Mathematical modeling and heuristic ap- proaches to flexible job shop scheduling problems, Journal of Intelligent Manufactur- ing 18 (3) (2007) 331–342. doi:10.1007/s10845-007-0026-8. URL https://doi.org/10.1007/s10845-007-0026-8
-
[40]
I. Kacem, S. Hammadi, P. Borne, Pareto-optimality approach for flexible job-shop scheduling problems: hybridization of evolutionary algorithms and fuzzy logic, Mathematics and Computers in Simulation 60 (3) (2002) 245–276. doi:10.1016/S0378-4754(02)00019-8. URL https://www.sciencedirect.com/science/article/pii/ S0378475402000198
-
[41]
J. Hurink, B. Jurisch, M. Thole, Tabu search for the job-shop scheduling problem with multi-purpose machines, Operations-Research-Spektrum 15 (4) (1994) 205–215. doi:10.1007/BF01719451. URL https://doi.org/10.1007/BF01719451
-
[42]
Anonymous Authors of this paper, Benchmarking and Benchmark Creation for FJSSP-W Experiments (2025)
a. Anonymous Authors of this paper, Benchmarking and Benchmark Creation for FJSSP-W Experiments (2025). URL https://anonymous.4open.science/r/FJSSP-W-Benchmarking-1510
work page 2025
-
[43]
N. Escamilla-Serna, J. Seck Tuoh Mora, J. Medina, H. Romero, et al., A Global-local Neighborhood Search Algorithm and Tabu Search for Flexible Job Shop Scheduling Problem, PeerJ Computer Science 7 (2021) e574. doi:10.7717/peerj-cs.574
- [44]
-
[45]
S. Usman, C. Lu, Job-shop scheduling with limited flexible workers considering er- gonomic factors using an improved multi-objective discrete Jaya algorithm, Computers & Operations Research 162 (2024) 106456. doi:10.1016/j.cor.2023.106456
- [46]
-
[47]
H. E. Nouri, O. Belkahla Driss, K. Gh´ edira, Solving the flexible job shop problem by hybrid metaheuristics-based multiagent model, Journal of Industrial Engineering International 14 (1) (2018) 1–14. doi:10.1007/s40092-017-0204-z. URL http://link.springer.com/10.1007/s40092-017-0204-z
-
[48]
H. E. Nouri, O. Belkahla Driss, K. Gh´ edira, Benchmark data instances for the Multi- Objective Flexible Job shop Scheduling Problem with Worker flexibility 1, publisher: Mendeley Data (Sep. 2018). doi:10.17632/hpp82wtxfr.1
-
[49]
D. Hutter, T. Steinberger, M. Hellwig, A Parameter Adaptive Genetic Algorithm With Restarts for Flexible Job Shop Scheduling Problems with Worker Flexibility, in: Proceedings of the 2025 ACM Conference on Genetic and Evolutionary Computation (GECCO’25), 2025, pp. 1–4, (accepted)
work page 2025
-
[50]
N. Veˇ cek, M. ˇCrepinˇ sek, M. Mernik, On the influence of the number of algorithms, problems, and independent runs in the comparison of evolutionary algorithms, Applied Soft Computing 54 (01 2017). doi:10.1016/j.asoc.2017.01.011
-
[51]
P. J. Stuckey, T. Feydy, A. Schutt, G. Tack, J. Fischer, The MiniZinc Challenge 2008–2013, AI Mag. 35 (2) (2014) 55–60. doi:10.1609/aimag.v35i2.2539. URL https://doi.org/10.1609/aimag.v35i2.2539
-
[52]
The MiniZinc Team, The MiniZinc Challenge, online, https://www.minizinc.org/ challenge (2025)
work page 2025
-
[53]
P. J. Stuckey, R. Becket, J. Fischer, Philosophy of the minizinc challenge, Constraints 15 (3) (2010) 307–316. doi:10.1007/s10601-010-9093-0
-
[54]
M. Hollander, D. A. Wolfe, E. Chicken, Nonparametric statistical methods, John Wiley & Sons, 2013
work page 2013
-
[55]
Anonymous Authors of this paper, Benchmark Instances and Results for FJSSP-W Experiments (2025)
b. Anonymous Authors of this paper, Benchmark Instances and Results for FJSSP-W Experiments (2025). URL https://anonymous.4open.science/r/FJSSPW-GA-D609/ 30 Supplementary Material By describing three different problem representations for the FJSSP(-W) instances, this supplementary section establishes the foundation to utilize the benchmarking environment ...
work page 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.