Troubleshooting: Master Curator fails or is unavailable and must be reassigned

Issue

Web images are not accessible.

Details

If the master Curator machine fails or is unavailable for an extended period of time, you may have to configure a secondary or slave Curator to take the role of the master Curator.

Solution

Reassign the master Curator so that a secondary or slave Curator takes ownership of the web cache and Curator job queue. (Using a load balancer with appropriate configuration may also help mitigate the risk of this happening.)

Note: If the web cache volumes are on the local drive of the master Curator and this machine is down, the web cache is not accessible if the master Curator is then reassigned. (In most cases, the web cache is located on a network drive.)

To reassign the master Curator

1. In the Job Manager, halt the Curator job queue.

   If wavelet images are needed immediately, reassign a new master Curator without waiting until all jobs have completed.

   or

   Wait until all jobs have completed and are no longer in the queue.

2. In CLUI, to change the ownership of the web cache, type

   UPDATE osr_volume SET ae_ref = AE_reference_of_slave_Curator WHERE volume_type = 'W' AND ae_ref = (SELECT ms.ae_ref FROM map_queue mq, map_service ms WHERE mq.service_ref = ms.service_ref AND mq.queue_ref = (SELECT MIN(queue_ref) FROM map_queue WHERE queue_title = 'CURATOR'))

3. To change the ownership of the Curator job queue, type

   UPDATE map_queue SET service_ref = (SELECT sc.service_ref FROM map_service mc, map_service sc, map_queue mq WHERE mc.type = sc.type AND mc.protocol = sc.protocol AND mc.port = sc.port AND sc.ae_ref = AE_reference_of_slave_Curator AND mc.service_ref = mq.service_ref AND mq.queue_ref = (SELECT MIN(queue_ref) FROM map_queue WHERE queue_title = 'CURATOR')) WHERE queue_ref = (SELECT MIN(queue_ref) FROM map_queue WHERE queue_title = 'CURATOR')

4. To signal to all machines in the cluster that the master Curator has been reassigned, type

   UPDATE map_ref_count SET ref_count = ref_count + 1 WHERE name = 'DATABASE_UPDATE_REF'

5. In the Job Manager, restart the Curator job queue.

   The Autopilot running on the new master Curator is now responsible for managing the web cache.