Search Engine

Setting up your data portal begins with a search engine: Elasticsearch or OpenSearch. Maestro indexes your metadata into whichever engine you choose, and Arranger generates the search API and UI components on top of it.

Below are the steps for both engines; follow the tab for the one you are running.

Supported search engines

Arranger supports OpenSearch 1.x or higher and Elasticsearch 7.x (minimum 7.0, licensed or default distribution only; ES OSS and ES 8.x are not supported; the bundled client is @elastic/elasticsearch v7). OpenSearch maintains API compatibility with Elasticsearch 7.x, so the index templates, mappings, and queries in this guide apply to both engines unchanged.

Elasticsearch

1. Run Elasticsearch: Use the following command to pull and run the Elasticsearch docker container

   docker run -d --name elasticsearch \
   -p 9200:9200 \
   -e discovery.type=single-node \
   -e cluster.name=workflow.elasticsearch \
   -e ES_JAVA_OPTS="-Xms512m -Xmx2048m" \
   -e ELASTIC_PASSWORD=myelasticpassword \
   -e xpack.security.enabled=true \
   -e MANAGE_INDEX_TEMPLATES=true \
   -e NETWORK_HOST=http://localhost:9200 \
   docker.elastic.co/elasticsearch/elasticsearch:7.17.1

   Click here for a detailed breakdown

   • -p 9200:9200 maps port 9200 of the host to port 9200 of the container

   • -e discovery.type=single-node configures Elasticsearch to run in single-node mode, this bypasses the need for cluster discovery and formation protocols, making Elasticsearch start up as a standalone node, ideal for development, testing, or small-scale deployments where clustering is not necessary

   • -e cluster.name=workflow.elasticsearch names the Elasticsearch cluster, this is good practice in case you choose to run multiple clusters or nodes in the future

   • -e ES_JAVA_OPTS=-Xms512m -Xmx2048m sets the initial and maximum heap size for the Java Virtual Machine (JVM) running Elasticsearch. -Xms512m sets the initial heap size to 512 MB. -Xmx2048m sets the maximum heap size to 2048 MB (2 GB). Properly setting these values ensures that Elasticsearch has enough memory to handle its operations efficiently, but not so much that it starves other processes on the host machine.

   • -e xpack.security.enabled=true activates security features such as authentication, authorization, encryption, and audit logging

   • -e MANAGE_INDEX_TEMPLATES=true ensures Elasticsearch manages index templates, when true, the system expects to manage the index templates as part of its operations. In the next step we will create a client services to set up the default configurations for new indices

   • -e ELASTIC_PASSWORD=myelasticpassword Sets the password for the elastic user

   We use Elasticsearch 7

   Our search platform is built on and compatible with version 7.x of Elasticsearch. Applications and queries need to follow Elasticsearch 7 syntax and conventions.

2. Supply an index template: Create a folder titled elasticsearchConfigs

   Download and place the following quickstart_index_template.json within your elasticsearchConfigs folder. This file specifies settings, mappings, and configurations that will be applied automatically to new indices that match the template's pattern

   Learn More

   If you'd like to learn more about creating an index mapping for your own data see our administration guide on configuring the index mapping.

3. Initialize your index: Run the following scripts to set up your Elasticsearch cluster

   Update Elasticsearch with your index template using the following curl command:

   curl -u elastic:myelasticpassword -X PUT 'http://localhost:9200/_template/index_template' -H 'Content-Type: application/json' -d ./elasticsearchConfigs/quickstart_index_template.json

   Create a new alias in Elasticsearch using the following curl command:

   curl -u elastic:myelasticpassword -X PUT 'http://localhost:9200/overture-quickstart-index'

   If successful you should be able to view the updated index in your browser from http://localhost:9200/overture-quickstart-index with the username elastic and password myelasticpassword.

   How this works

   Any index alias that starts with overture- will use the mapping of the index template we initially provided. This is defined on line two of our quickstart_index_template.

OpenSearch

1. Run OpenSearch: Use the following command to pull and run the OpenSearch docker container

   docker run -d --name opensearch \
   -p 9200:9200 \
   -e discovery.type=single-node \
   -e cluster.name=workflow.opensearch \
   -e OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx2048m" \
   -e DISABLE_SECURITY_PLUGIN=true \
   -e DISABLE_INSTALL_DEMO_CONFIG=true \
   opensearchproject/opensearch:2.17.1

   Click here for a detailed breakdown

   • -p 9200:9200 maps port 9200 of the host to port 9200 of the container

   • -e discovery.type=single-node runs OpenSearch as a standalone node, bypassing cluster discovery and formation, ideal for development, testing, or small-scale deployments where clustering is not necessary

   • -e cluster.name=workflow.opensearch names the cluster, good practice in case you choose to run multiple clusters or nodes in the future

   • -e OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx2048m sets the initial (-Xms512m, 512 MB) and maximum (-Xmx2048m, 2 GB) heap size for the JVM running OpenSearch. This is the OpenSearch equivalent of Elasticsearch's ES_JAVA_OPTS

   • -e DISABLE_SECURITY_PLUGIN=true turns off the OpenSearch security plugin so the REST API is served over plain HTTP with no TLS and no admin password. This keeps the curl, Maestro, and Arranger configuration in this guide identical to the Elasticsearch path

   • -e DISABLE_INSTALL_DEMO_CONFIG=true skips installing the demo security certificates and users, which are not needed when the security plugin is disabled

   OpenSearch configuration notes

   • OpenSearch 2.x serves the Elasticsearch 7.x style index templates and queries used throughout this guide, so the index template and mapping files work unchanged.

   • Do not set compatibility.override_main_response_version. It hides version.distribution from the GET / response, which causes Arranger to auto-detect the Elasticsearch client instead of OpenSearch and fail to initialize.

   • Disabling the security plugin is only appropriate for a localhost-bound development instance. On any network-exposed deployment, keep the security plugin enabled and connect over HTTPS with credentials.

2. Supply an index template: Create a folder titled elasticsearchConfigs

   Download and place the following quickstart_index_template.json within your elasticsearchConfigs folder. This is the same template used for Elasticsearch; OpenSearch applies it without changes.

   Learn More

   If you'd like to learn more about creating an index mapping for your own data see our administration guide on configuring the index mapping.

3. Initialize your index: Run the following scripts to set up your OpenSearch cluster

   Update OpenSearch with your index template using the following curl command:

   curl -X PUT 'http://localhost:9200/_template/index_template' -H 'Content-Type: application/json' -d ./elasticsearchConfigs/quickstart_index_template.json

   Create a new alias in OpenSearch using the following curl command:

   curl -X PUT 'http://localhost:9200/overture-quickstart-index'

   With the security plugin disabled these requests need no credentials. If successful you should be able to view the updated index in your browser from http://localhost:9200/overture-quickstart-index.

   How this works

   Any index alias that starts with overture- will use the mapping of the index template we initially provided. This is defined on line two of our quickstart_index_template.

Continue to Maestro once your search engine is running and indexed.