2018-03-28 22:16:42.084 25550 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries (currently set to 1500) exhausted.  The amphora is unavailable. Reason: HTTPSConnectionPool(host='172.16.0.7', port=9443): Max retries exceeded with url: /0.5/plug/vip/10.10.200.47 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb7203e7810>: Failed to establish a new connection: [Errno 111] Connection refused',))

                 |                                                                 |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb721823210>}, 'provides': None}

                                                     |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb721823210>}, 'provides': None}

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 220, in execute

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker     amphora, loadbalancer, amphorae_network_config)

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 136, in post_vip_plug

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker     net_info)

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 375, in plug_vip

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker     json=net_info)

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 299, in request

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker     raise driver_except.TimeOutException()

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker TimeOutException: contacting the amphora timed out

2018-03-28 22:16:42.107 25550 ERROR octavia.controller.worker.controller_worker

2018-03-28 22:16:43.431 25550 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (55e66f2d-5140-4432-bce0-ec4a9b7575f1) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/dist-packages/taskflow/listeners/logging.py:194

2018-03-28 22:16:43.439 25550 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (55e66f2d-5140-4432-bce0-ec4a9b7575f1) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'

2018-03-28 22:16:43.449 25550 ERROR octavia.controller.worker.controller_worker [-] Failover exception: contacting the amphora timed out

2018-03-28 23:37:40.030 25550 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries (currently set to 1500) exhausted.  The amphora is unavailable. Reason: HTTPSConnectionPool(host='172.16.0.9', port=9443): Max retries exceeded with url: /0.5/plug/vip/10.10.200.47 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb720508850>: Failed to establish a new connection: [Errno 113] No route to host',))

                 |                                   |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb7291a6f90>}, 'provides': None}

                                                                                   |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb7291a6f90>}, 'provides': None}

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 220, in execute

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker     amphora, loadbalancer, amphorae_network_config)

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 136, in post_vip_plug

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker     net_info)

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 375, in plug_vip

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker     json=net_info)

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 299, in request

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker     raise driver_except.TimeOutException()

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker TimeOutException: contacting the amphora timed out

2018-03-28 23:37:40.056 25550 ERROR octavia.controller.worker.controller_worker

2018-03-28 23:37:41.115 25550 ERROR octavia.controller.worker.tasks.database_tasks [-] Failed to delete amphora 7186464c-a2c2-4462-99d5-2578411e93aa in the database due to: No row was found for one()

2018-03-28 23:37:41.197 25550 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (6e689a0a-b921-4e04-acff-f095402514bb) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/dist-packages/taskflow/listeners/logging.py:194

2018-03-28 23:37:41.205 25550 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (6e689a0a-b921-4e04-acff-f095402514bb) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'

2018-03-28 23:37:41.214 25550 ERROR octavia.controller.worker.controller_worker [-] Failover exception: contacting the amphora timed out

2018-03-28 23:41:50.988 25550 ERROR octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection retries (currently set to 1500) exhausted.  The amphora is unavailable. Reason: HTTPSConnectionPool(host='172.16.0.11', port=9443): Max retries exceeded with url: /0.5/plug/vip/10.10.200.47 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fb720537d10>: Failed to establish a new connection: [Errno 113] No route to host',))

                 |                                                                 |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb723c5e3d0>}, 'provides': None}

                                                     |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'amphora': <octavia.common.data_models.Amphora object at 0x7fb723c5e3d0>}, 'provides': None}

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last):

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker     result = task.execute(**arguments)

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 220, in execute

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker     amphora, loadbalancer, amphorae_network_config)

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 136, in post_vip_plug

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker     net_info)

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 375, in plug_vip

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker     json=net_info)

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker   File "/usr/local/lib/python2.7/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 299, in request

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker     raise driver_except.TimeOutException()

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker TimeOutException: contacting the amphora timed out

2018-03-28 23:41:51.007 25550 ERROR octavia.controller.worker.controller_worker

2018-03-28 23:41:51.306 25550 ERROR octavia.controller.worker.tasks.database_tasks [-] Failed to delete amphora 9a37cae2-44ac-421e-90d4-8cdad684511d in the database due to: No row was found for one()

2018-03-28 23:41:51.358 25550 DEBUG octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (026964a7-9082-4e2a-8107-5718ac63cc94) transitioned into state 'REVERTING' from state 'SUCCESS' _task_receiver /usr/lib/python2.7/dist-packages/taskflow/listeners/logging.py:194

2018-03-28 23:41:51.364 25550 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.lifecycle_tasks.AmphoraToErrorOnRevertTask' (026964a7-9082-4e2a-8107-5718ac63cc94) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'

2018-03-28 23:41:51.369 25550 ERROR octavia.controller.worker.controller_worker [-] Failover exception: contacting the amphora timed out