RabbitMQ broker force start¶

When to use¶

When the PR environments are no longer being created, and the lagoon-core-broker-<n> pods are missing or not running, and the container logs contain errors like Error while waiting for Mnesia tables: {timeout_waiting_for_tables.

This situation is caused by the RabbitMQ broker not starting correctly.

Prerequisites¶

A dplsh session with DPLPLAT_ENV exported .

Procedure¶

You are going to exec into the pod and stop the RabbitMQ application, and then start it with the force_boot feature, so that it can perform its Mnesia sync correctly.

Exec into the pod:

dplsh:~/host_mount$ kubectl -n lagoon-core exec -ti pod/lagoon-core-broker-0 -- sh

Stop RabbitMQ:

/ $ rabbitmqctl stop_app
Stopping rabbit application on node rabbit@lagoon-core-broker-0.lagoon-
core-broker-headless.lagoon-core.svc.cluster.local ...

Start it immediately after using the force_boot flag:

/ $ rabbitmqctl force_boot

Then exit the shell and check the container logs for one of the broker pods. It should start without errors.