Usage

The following usage docs provide instructions for common tasks and workflows in Maestro:

📄️ Indexing Data

Maestro offers flexible indexing at the repository, study, or individual document (analysis file) levels alongside a RESTful API for interacting with its core functionalities. There are two methods to interact with the Maestro API:

📄️ Exclusion Rules

Maestro supports data publication controls by providing configurable exclusion rules. These rules allow you to omit specific analyses from being indexed based on metadata tags assigned by Song. You can use Study, Analysis, File, Sample, Specimen, and Donor IDs to exclude data from indexing.

📄️ Index Mappings

An index mapping defines how documents and their fields are stored and indexed in Elasticsearch. Maestro is responsible for taking published Song metadata and translating it into Elasticsearch documents.

📄️ Kafka Topics

Maestro can be configured to listen to Kafka topics for various operations. This page explains how to set up Kafka integration and use different message types.

📄️ Slack Notifications

Maestro can be integrated with Slack to send real-time notifications about errors, warnings, and progress updates throughout the indexing process.

📄️ API Reference

Maestro's API is a RESTful API that uses JSON for request and response bodies. It follows standard HTTP methods and status codes, providing a robust interface for managing Maestro-related data in the Overture platform.

Maestro Terminology

Whether you're new to these tools or looking to refresh your understanding, this glossary is here as a reference of key terms and concepts.

• Analysis: A structured JSON file containing metadata that is submitted to Song and subsequently indexed by Maestro.

• Study: An organized collection of analysis files, tagged with a common study_ID.

• Repository: A collection of studies and their associated analyses.

• Schema: A description of the structure and constraints of data in a database.

• Index Mapping: A definition of how documents and their fields are stored and indexed in Elasticsearch.

• File Centric Index Mapping: A type of index mapping that organizes data by file Id.

• Analysis Centric Index Mapping: A type of index mapping that organizes data by analysis Id.

• Exclusion Rules: Configurable settings in Maestro that allow specific analyses to be omitted from indexing based on metadata tags.

• Alias: In the context of Elasticsearch, an alias is a secondary name for a group of indices.