Issue: After upgrading to Advanced Authentication 6.4 Service Pack 2, the dashboard occasionally fails to load the data and displays the following error:
Transport Error(503, 'Search Guard not initialized (SG11). See https://docs.search-guard.com/latest/sgadmin')(Internal Server Error)
Reason: This is due to the corrupted ElasticSearch configuration on an Advanced Authentication appliance.
Workaround: Repair the corrupted Elasticsearch configuration.
IMPORTANT:It will reset the Elasticsearch container to its initial configuration and result in the loss of all previously gathered event data.
To repair a corrupted Elasticsearch installation on the AA appliance, perform the following steps:
(Optional) Run the following command to connect to the appliance using ssh:
systemctl start sshd
Run the following command on the command prompt to connect to the searchd container:
docker exec -it aaf-searchd-1 /bin/bash
Run the following commands to open the elasticsearch.yaml file:
vi /.backup/config/elasticsearch.yml
Add the following properties at end of the elasticsearch.yaml file:
action.auto_create_index: ".watches,.triggered_watches,.watcher[1]history-*"
Save the file and exit the docker container.
Run the following command to stop the Elasticsearch container from the Advanced Authentication appliance command prompt:
docker stop aaf-searchd-1
Validate the delete path before removing the data and configuration info from the containers that are used by the aaf-searchd-1 container at the AA appliance command prompt. To remove the data and configuration info, run the below command:
rm -r /var/lib/docker/volumes/aaf_searchd-config/_data/*
rm -r /var/lib/docker/volumes/aaf_searchd-data/_data/*
Run the following command to restart the aaf-searchd-1 container:
docker start aaf-searchd-1
Run the following command to connect to the aaf-aucore-1 container:
docker exec -it aaf-aucore-1 /bin/bash
Change the directory for the aaf-aucore-1 container at the command prompt to the scripts by using the below command:
cd /opt/AuCore/aucore/scripts
To set the module search path for python by exporting the PYTHONPATH environment variable, run the below command:
export PYTHONPATH=/opt/AuCore:/opt/penv/lib/python3.9/site[1]packages
Run the below command to execute the au_setup_reporting.pyc script to reconfigure elastic search:
python au_setup_reporting.pyc ../../data/production.ini
Verify the data on the Advanced Authentication dashboard to make sure the Elasticsearch container is configured properly and is recording event data.
NOTE:During the au_setup_reporting.pyc script execution, you may see several connection errors and warnings stating that elasticsearch is not ready. It should eventually print out the following lines:
a2023-11-27 22:20:27 INFO [elasticsearch] GET https://127.0.0.1:9200/_cluster/health?wait_for_status=yellow[status:200 request:0.275s] 2023-11-27 22:20:28 INFO [elasticsearch] PUT https://127.0.0.1:9200/aucore-stats-2 [status:200 request:0.321s]