crabs-exploration-test-data/ml-runs/317777717624044570/40b1688a76d94bd08175cb380d0a6e0e/artifacts/slurm_array.5313275-0.gpu-sr670-22.err

  Running command git clone --filter=blob:none --quiet https://github.com/SainsburyWellcomeCentre/crabs-exploration.git /tmp/pip-req-build-bazt9owa
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
`Trainer(limit_train_batches=1.0)` was configured so 100% of the batches per epoch will be used..
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [MIG-GPU-242780f9-d85c-a72f-1605-da1bd3f1cb87/4/0]

  | Name  | Type       | Params | Mode 
---------------------------------------------
0 | model | FasterRCNN | 43.3 M | train
---------------------------------------------
43.0 M    Trainable params
225 K     Non-trainable params
43.3 M    Total params
173.025   Total estimated model params size (MB)
SLURM auto-requeueing enabled. Setting signal handlers.
INFO:root:Average Precision (val): 0.3862,Average Recall (val): 0.3210
INFO:root:Average Precision (val): 0.8594,Average Recall (val): 0.5508
INFO:root:Average Precision (val): 0.9418,Average Recall (val): 0.5881
INFO:root:Average Precision (val): 0.9596,Average Recall (val): 0.6205
INFO:root:Average Precision (val): 0.9660,Average Recall (val): 0.6354
INFO:root:Average Precision (val): 0.9677,Average Recall (val): 0.6408
INFO:root:Average Precision (val): 0.9662,Average Recall (val): 0.6762
INFO:root:Average Precision (val): 0.9750,Average Recall (val): 0.6951
INFO:root:Average Precision (val): 0.9724,Average Recall (val): 0.6841
INFO:root:Average Precision (val): 0.9715,Average Recall (val): 0.7111
INFO:root:Average Precision (val): 0.9675,Average Recall (val): 0.7425
INFO:root:Average Precision (val): 0.9723,Average Recall (val): 0.7383
INFO:root:Average Precision (val): 0.9761,Average Recall (val): 0.7173
INFO:root:Average Precision (val): 0.9670,Average Recall (val): 0.7653
INFO:root:Average Precision (val): 0.9682,Average Recall (val): 0.7459
INFO:root:Average Precision (val): 0.9644,Average Recall (val): 0.7772
INFO:root:Average Precision (val): 0.9653,Average Recall (val): 0.7661
INFO:root:Average Precision (val): 0.9690,Average Recall (val): 0.7675
INFO:root:Average Precision (val): 0.9674,Average Recall (val): 0.7576
INFO:root:Average Precision (val): 0.9656,Average Recall (val): 0.7634
INFO:root:Average Precision (val): 0.9626,Average Recall (val): 0.7918
INFO:root:Average Precision (val): 0.9641,Average Recall (val): 0.7926
INFO:root:Average Precision (val): 0.9591,Average Recall (val): 0.7938
INFO:root:Average Precision (val): 0.9721,Average Recall (val): 0.7809
INFO:root:Average Precision (val): 0.9598,Average Recall (val): 0.7647
INFO:root:Average Precision (val): 0.9601,Average Recall (val): 0.7866
INFO:root:Average Precision (val): 0.9690,Average Recall (val): 0.7898
INFO:root:Average Precision (val): 0.9712,Average Recall (val): 0.7865
INFO:root:Average Precision (val): 0.9681,Average Recall (val): 0.8182
INFO:root:Average Precision (val): 0.9590,Average Recall (val): 0.8123
INFO:root:Average Precision (val): 0.9666,Average Recall (val): 0.8025
INFO:root:Average Precision (val): 0.9663,Average Recall (val): 0.7917
INFO:root:Average Precision (val): 0.9522,Average Recall (val): 0.8025
INFO:root:Average Precision (val): 0.9642,Average Recall (val): 0.7837
INFO:root:Average Precision (val): 0.9660,Average Recall (val): 0.7992
INFO:root:Average Precision (val): 0.9570,Average Recall (val): 0.8000
INFO:root:Average Precision (val): 0.9673,Average Recall (val): 0.8054
INFO:root:Average Precision (val): 0.9663,Average Recall (val): 0.8122
INFO:root:Average Precision (val): 0.9483,Average Recall (val): 0.8084
INFO:root:Average Precision (val): 0.9684,Average Recall (val): 0.7895
INFO:root:Average Precision (val): 0.9690,Average Recall (val): 0.8100
INFO:root:Average Precision (val): 0.9697,Average Recall (val): 0.8234
INFO:root:Average Precision (val): 0.9489,Average Recall (val): 0.8197
INFO:root:Average Precision (val): 0.9716,Average Recall (val): 0.8166
INFO:root:Average Precision (val): 0.9284,Average Recall (val): 0.8121
INFO:root:Average Precision (val): 0.9647,Average Recall (val): 0.7981
INFO:root:Average Precision (val): 0.9577,Average Recall (val): 0.8101
INFO:root:Average Precision (val): 0.9615,Average Recall (val): 0.8219
INFO:root:Average Precision (val): 0.9673,Average Recall (val): 0.8264
INFO:root:Average Precision (val): 0.9601,Average Recall (val): 0.8217
INFO:root:Average Precision (val): 0.9722,Average Recall (val): 0.8283
INFO:root:Average Precision (val): 0.9644,Average Recall (val): 0.8101
INFO:root:Average Precision (val): 0.9463,Average Recall (val): 0.8309
INFO:root:Average Precision (val): 0.9728,Average Recall (val): 0.8254
INFO:root:Average Precision (val): 0.9724,Average Recall (val): 0.8262
INFO:root:Average Precision (val): 0.9766,Average Recall (val): 0.8270
INFO:root:Average Precision (val): 0.9763,Average Recall (val): 0.8183
INFO:root:Average Precision (val): 0.9739,Average Recall (val): 0.8105
INFO:root:Average Precision (val): 0.9761,Average Recall (val): 0.8229
INFO:root:Average Precision (val): 0.9770,Average Recall (val): 0.8092
INFO:root:Average Precision (val): 0.9634,Average Recall (val): 0.8290
INFO:root:Average Precision (val): 0.9696,Average Recall (val): 0.8325
INFO:root:Average Precision (val): 0.9762,Average Recall (val): 0.8116
INFO:root:Average Precision (val): 0.9656,Average Recall (val): 0.8218
INFO:root:Average Precision (val): 0.9587,Average Recall (val): 0.8277
INFO:root:Average Precision (val): 0.9728,Average Recall (val): 0.8330
INFO:root:Average Precision (val): 0.9711,Average Recall (val): 0.8265
INFO:root:Average Precision (val): 0.9747,Average Recall (val): 0.8215
INFO:root:Average Precision (val): 0.9706,Average Recall (val): 0.8102
INFO:root:Average Precision (val): 0.9732,Average Recall (val): 0.8313
INFO:root:Average Precision (val): 0.9725,Average Recall (val): 0.8195
INFO:root:Average Precision (val): 0.9541,Average Recall (val): 0.8343
INFO:root:Average Precision (val): 0.9611,Average Recall (val): 0.8225
INFO:root:Average Precision (val): 0.9652,Average Recall (val): 0.8197
INFO:root:Average Precision (val): 0.9661,Average Recall (val): 0.8350
INFO:root:Average Precision (val): 0.9770,Average Recall (val): 0.8335
INFO:root:Average Precision (val): 0.9665,Average Recall (val): 0.8373
INFO:root:Average Precision (val): 0.9738,Average Recall (val): 0.8254
INFO:root:Average Precision (val): 0.9712,Average Recall (val): 0.8386
INFO:root:Average Precision (val): 0.9779,Average Recall (val): 0.8243
INFO:root:Average Precision (val): 0.9447,Average Recall (val): 0.8380
INFO:root:Average Precision (val): 0.9776,Average Recall (val): 0.8322
INFO:root:Average Precision (val): 0.9663,Average Recall (val): 0.8424
INFO:root:Average Precision (val): 0.9670,Average Recall (val): 0.8220
INFO:root:Average Precision (val): 0.9619,Average Recall (val): 0.8385
INFO:root:Average Precision (val): 0.9761,Average Recall (val): 0.8203
INFO:root:Average Precision (val): 0.9728,Average Recall (val): 0.8224
INFO:root:Average Precision (val): 0.9630,Average Recall (val): 0.8293
INFO:root:Average Precision (val): 0.9691,Average Recall (val): 0.8285
INFO:root:Average Precision (val): 0.9571,Average Recall (val): 0.8408
INFO:root:Average Precision (val): 0.9661,Average Recall (val): 0.8251
INFO:root:Average Precision (val): 0.9485,Average Recall (val): 0.8155
INFO:root:Average Precision (val): 0.9562,Average Recall (val): 0.7958
INFO:root:Average Precision (val): 0.9554,Average Recall (val): 0.8396
INFO:root:Average Precision (val): 0.9541,Average Recall (val): 0.8232
INFO:root:Average Precision (val): 0.9570,Average Recall (val): 0.8436
INFO:root:Average Precision (val): 0.9396,Average Recall (val): 0.8444
INFO:root:Average Precision (val): 0.9654,Average Recall (val): 0.8385
INFO:root:Average Precision (val): 0.9506,Average Recall (val): 0.8328
INFO:root:Average Precision (val): 0.9544,Average Recall (val): 0.8269
INFO:root:Average Precision (val): 0.9734,Average Recall (val): 0.8226
INFO:root:Average Precision (val): 0.9529,Average Recall (val): 0.8249
INFO:root:Average Precision (val): 0.9633,Average Recall (val): 0.8366
INFO:root:Average Precision (val): 0.9594,Average Recall (val): 0.8465
INFO:root:Average Precision (val): 0.9633,Average Recall (val): 0.8381
INFO:root:Average Precision (val): 0.9743,Average Recall (val): 0.8215
INFO:root:Average Precision (val): 0.9533,Average Recall (val): 0.8429
INFO:root:Average Precision (val): 0.9502,Average Recall (val): 0.8373
INFO:root:Average Precision (val): 0.9623,Average Recall (val): 0.8243
INFO:root:Average Precision (val): 0.9715,Average Recall (val): 0.8395
INFO:root:Average Precision (val): 0.9613,Average Recall (val): 0.8358
INFO:root:Average Precision (val): 0.9694,Average Recall (val): 0.8338
INFO:root:Average Precision (val): 0.9579,Average Recall (val): 0.8314
INFO:root:Average Precision (val): 0.9683,Average Recall (val): 0.8392
INFO:root:Average Precision (val): 0.9573,Average Recall (val): 0.8389
INFO:root:Average Precision (val): 0.9712,Average Recall (val): 0.8447
INFO:root:Average Precision (val): 0.9743,Average Recall (val): 0.8412
INFO:root:Average Precision (val): 0.9648,Average Recall (val): 0.8376
INFO:root:Average Precision (val): 0.9728,Average Recall (val): 0.8456
INFO:root:Average Precision (val): 0.9699,Average Recall (val): 0.8417
INFO:root:Average Precision (val): 0.9709,Average Recall (val): 0.8396
INFO:root:Average Precision (val): 0.9638,Average Recall (val): 0.8191
INFO:root:Average Precision (val): 0.9579,Average Recall (val): 0.8307
INFO:root:Average Precision (val): 0.9682,Average Recall (val): 0.8204
INFO:root:Average Precision (val): 0.9667,Average Recall (val): 0.8351
INFO:root:Average Precision (val): 0.9777,Average Recall (val): 0.8321
INFO:root:Average Precision (val): 0.9655,Average Recall (val): 0.8413
INFO:root:Average Precision (val): 0.9654,Average Recall (val): 0.8369
INFO:root:Average Precision (val): 0.9704,Average Recall (val): 0.8365
INFO:root:Average Precision (val): 0.9647,Average Recall (val): 0.8249
INFO:root:Average Precision (val): 0.9565,Average Recall (val): 0.8373
INFO:root:Average Precision (val): 0.9551,Average Recall (val): 0.8373
INFO:root:Average Precision (val): 0.9661,Average Recall (val): 0.8388
INFO:root:Average Precision (val): 0.9625,Average Recall (val): 0.8540
INFO:root:Average Precision (val): 0.9649,Average Recall (val): 0.8396
INFO:root:Average Precision (val): 0.9643,Average Recall (val): 0.8391
INFO:root:Average Precision (val): 0.9661,Average Recall (val): 0.8441
INFO:root:Average Precision (val): 0.9680,Average Recall (val): 0.8525
INFO:root:Average Precision (val): 0.9774,Average Recall (val): 0.8476
INFO:root:Average Precision (val): 0.9776,Average Recall (val): 0.8372
INFO:root:Average Precision (val): 0.9780,Average Recall (val): 0.8436
INFO:root:Average Precision (val): 0.9697,Average Recall (val): 0.8537
INFO:root:Average Precision (val): 0.9703,Average Recall (val): 0.8350
INFO:root:Average Precision (val): 0.9785,Average Recall (val): 0.8472
INFO:root:Average Precision (val): 0.9735,Average Recall (val): 0.8514
INFO:root:Average Precision (val): 0.9831,Average Recall (val): 0.8359
INFO:root:Average Precision (val): 0.9755,Average Recall (val): 0.8327
INFO:root:Average Precision (val): 0.9695,Average Recall (val): 0.8415
INFO:root:Average Precision (val): 0.9670,Average Recall (val): 0.8447
INFO:root:Average Precision (val): 0.9797,Average Recall (val): 0.8379
INFO:root:Average Precision (val): 0.9764,Average Recall (val): 0.8356
INFO:root:Average Precision (val): 0.9595,Average Recall (val): 0.8427
INFO:root:Average Precision (val): 0.9698,Average Recall (val): 0.8454
INFO:root:Average Precision (val): 0.9547,Average Recall (val): 0.8345
INFO:root:Average Precision (val): 0.9795,Average Recall (val): 0.8230
INFO:root:Average Precision (val): 0.9787,Average Recall (val): 0.8458
INFO:root:Average Precision (val): 0.9655,Average Recall (val): 0.8477
INFO:root:Average Precision (val): 0.9803,Average Recall (val): 0.8426
INFO:root:Average Precision (val): 0.9294,Average Recall (val): 0.8327
INFO:root:Average Precision (val): 0.9728,Average Recall (val): 0.8316
INFO:root:Average Precision (val): 0.9686,Average Recall (val): 0.8406
INFO:root:Average Precision (val): 0.9503,Average Recall (val): 0.8372
INFO:root:Average Precision (val): 0.9771,Average Recall (val): 0.8288
INFO:root:Average Precision (val): 0.9642,Average Recall (val): 0.8412
INFO:root:Average Precision (val): 0.9646,Average Recall (val): 0.8369
INFO:root:Average Precision (val): 0.9741,Average Recall (val): 0.8325
INFO:root:Average Precision (val): 0.9703,Average Recall (val): 0.8355
INFO:root:Average Precision (val): 0.9724,Average Recall (val): 0.8358
INFO:root:Average Precision (val): 0.9750,Average Recall (val): 0.8426
INFO:root:Average Precision (val): 0.9600,Average Recall (val): 0.8308
INFO:root:Average Precision (val): 0.9741,Average Recall (val): 0.8354
INFO:root:Average Precision (val): 0.9803,Average Recall (val): 0.8365
INFO:root:Average Precision (val): 0.9775,Average Recall (val): 0.8358
INFO:root:Average Precision (val): 0.9767,Average Recall (val): 0.8428
INFO:root:Average Precision (val): 0.9808,Average Recall (val): 0.8465
INFO:root:Average Precision (val): 0.9788,Average Recall (val): 0.8320
INFO:root:Average Precision (val): 0.9704,Average Recall (val): 0.8480
INFO:root:Average Precision (val): 0.9731,Average Recall (val): 0.8404
INFO:root:Average Precision (val): 0.9689,Average Recall (val): 0.8454
INFO:root:Average Precision (val): 0.9760,Average Recall (val): 0.8324
INFO:root:Average Precision (val): 0.9821,Average Recall (val): 0.8340
INFO:root:Average Precision (val): 0.9835,Average Recall (val): 0.8308
INFO:root:Average Precision (val): 0.9774,Average Recall (val): 0.8498
INFO:root:Average Precision (val): 0.9693,Average Recall (val): 0.8422
INFO:root:Average Precision (val): 0.9796,Average Recall (val): 0.8323
INFO:root:Average Precision (val): 0.9750,Average Recall (val): 0.8428
INFO:root:Average Precision (val): 0.9760,Average Recall (val): 0.8406
INFO:root:Average Precision (val): 0.9767,Average Recall (val): 0.8417
INFO:root:Average Precision (val): 0.9766,Average Recall (val): 0.8375
INFO:root:Average Precision (val): 0.9740,Average Recall (val): 0.8275
INFO:root:Average Precision (val): 0.9617,Average Recall (val): 0.8468
INFO:root:Average Precision (val): 0.9689,Average Recall (val): 0.8426
INFO:root:Average Precision (val): 0.9747,Average Recall (val): 0.8422
INFO:root:Average Precision (val): 0.9739,Average Recall (val): 0.8398
INFO:root:Average Precision (val): 0.9780,Average Recall (val): 0.8420
INFO:root:Average Precision (val): 0.9729,Average Recall (val): 0.8419
INFO:root:Average Precision (val): 0.9807,Average Recall (val): 0.8474
INFO:root:Average Precision (val): 0.9828,Average Recall (val): 0.8493
INFO:root:Average Precision (val): 0.9781,Average Recall (val): 0.8452
INFO:root:Average Precision (val): 0.9742,Average Recall (val): 0.8421
INFO:root:Average Precision (val): 0.9804,Average Recall (val): 0.8331
INFO:root:Average Precision (val): 0.9758,Average Recall (val): 0.8451
INFO:root:Average Precision (val): 0.9687,Average Recall (val): 0.8353
INFO:root:Average Precision (val): 0.9659,Average Recall (val): 0.8364
INFO:root:Average Precision (val): 0.9726,Average Recall (val): 0.8321
INFO:root:Average Precision (val): 0.9716,Average Recall (val): 0.8317
INFO:root:Average Precision (val): 0.9638,Average Recall (val): 0.8505
INFO:root:Average Precision (val): 0.9661,Average Recall (val): 0.8484
INFO:root:Average Precision (val): 0.9589,Average Recall (val): 0.8399
INFO:root:Average Precision (val): 0.9719,Average Recall (val): 0.8493
INFO:root:Average Precision (val): 0.9735,Average Recall (val): 0.8464
INFO:root:Average Precision (val): 0.9742,Average Recall (val): 0.8414
INFO:root:Average Precision (val): 0.9710,Average Recall (val): 0.8428
INFO:root:Average Precision (val): 0.9711,Average Recall (val): 0.8473
INFO:root:Average Precision (val): 0.9742,Average Recall (val): 0.8292
INFO:root:Average Precision (val): 0.9730,Average Recall (val): 0.8474
INFO:root:Average Precision (val): 0.9663,Average Recall (val): 0.8444
INFO:root:Average Precision (val): 0.9748,Average Recall (val): 0.8507
INFO:root:Average Precision (val): 0.9698,Average Recall (val): 0.8461
INFO:root:Average Precision (val): 0.9768,Average Recall (val): 0.8469
INFO:root:Average Precision (val): 0.9727,Average Recall (val): 0.8498
INFO:root:Average Precision (val): 0.9704,Average Recall (val): 0.8367
INFO:root:Average Precision (val): 0.9841,Average Recall (val): 0.8336
INFO:root:Average Precision (val): 0.9745,Average Recall (val): 0.8472
INFO:root:Average Precision (val): 0.9750,Average Recall (val): 0.8454
INFO:root:Average Precision (val): 0.9820,Average Recall (val): 0.8439
INFO:root:Average Precision (val): 0.9831,Average Recall (val): 0.8459
INFO:root:Average Precision (val): 0.9790,Average Recall (val): 0.8394
INFO:root:Average Precision (val): 0.9804,Average Recall (val): 0.8292
INFO:root:Average Precision (val): 0.9712,Average Recall (val): 0.8412
INFO:root:Average Precision (val): 0.9768,Average Recall (val): 0.8354
INFO:root:Average Precision (val): 0.9662,Average Recall (val): 0.8416
INFO:root:Average Precision (val): 0.9764,Average Recall (val): 0.8428
INFO:root:Average Precision (val): 0.9609,Average Recall (val): 0.8440
INFO:root:Average Precision (val): 0.9782,Average Recall (val): 0.8409
INFO:root:Average Precision (val): 0.9771,Average Recall (val): 0.8396
INFO:root:Average Precision (val): 0.9766,Average Recall (val): 0.8320
INFO:root:Average Precision (val): 0.9790,Average Recall (val): 0.8415
INFO:root:Average Precision (val): 0.9808,Average Recall (val): 0.8560
INFO:root:Average Precision (val): 0.9734,Average Recall (val): 0.8336
INFO:root:Average Precision (val): 0.9848,Average Recall (val): 0.8436
INFO:root:Average Precision (val): 0.9815,Average Recall (val): 0.8276
INFO:root:Average Precision (val): 0.9810,Average Recall (val): 0.8390
INFO:root:Average Precision (val): 0.9784,Average Recall (val): 0.8328
INFO:root:Average Precision (val): 0.9799,Average Recall (val): 0.8224
INFO:root:Average Precision (val): 0.9795,Average Recall (val): 0.8446
INFO:root:Average Precision (val): 0.9742,Average Recall (val): 0.8471
INFO:root:Average Precision (val): 0.9726,Average Recall (val): 0.8374
INFO:root:Average Precision (val): 0.9798,Average Recall (val): 0.8456
INFO:root:Average Precision (val): 0.9779,Average Recall (val): 0.8248
INFO:root:Average Precision (val): 0.9779,Average Recall (val): 0.8428
`Trainer.fit` stopped: `max_epochs=250` reached.