Amazon OpenSearch Service FAQs

Amazon OpenSearch Service is a managed service that makes it easy for you to perform interactive log analytics, real-time application monitoring, website search, and more. OpenSearch is an open source, distributed search and analytics suite derived from Elasticsearch. Amazon OpenSearch Service offers the latest versions of OpenSearch, support for 19 versions of Elasticsearch (1.5 to 7.10 versions), as well as visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions). Amazon OpenSearch Service currently has tens of thousands of active customers with hundreds of thousands of clusters under management processing trillions of requests per month. See the Amazon OpenSearch Service FAQ for more information.

Amazon OpenSearch Service offers the latest versions of OpenSearch and support for several legacy open-source Elasticsearch versions (up to version 7.10). For more details refer our documentation.

Amazon OpenSearch Service domains are Elasticsearch (1.5 to 7.10) or OpenSearch clusters created using the Amazon OpenSearch Service console, CLI, or API. Each domain is an OpenSearch or Elasticsearch cluster in the cloud with the compute and storage resources you specify. You can create and delete domains, define infrastructure attributes, and control access and security. You can run one or more Amazon OpenSearch Service domains.

Amazon OpenSearch Service manages the work involved in setting up a domain, from provisioning infrastructure capacity in the network environment you request to installing the OpenSearch or Elasticsearch software. Once your domain is running, Amazon OpenSearch Service automates common administrative tasks, such as performing backups, monitoring instances and patching software. Amazon OpenSearch Service integrates with Amazon CloudWatch to produce metrics that provide information about the state of the domains. Amazon OpenSearch Service also offers options to modify your domain instance and storage settings to simplify the task of tailoring your domain based to your application needs.

Amazon OpenSearch Service supports most of the commonly used OpenSearch and Elasticsearch APIs, so the code, applications, and popular tools that you're already using with your Elasticsearch (up to version 7.10) or OpenSearch environments work seamlessly. For a full list of supported operations, see our documentation.

Amazon OpenSearch Service offers customers the option to deploy their instances across one, two, or three AZs. Customers running development or test workloads can pick the single AZ option. Those running production-grade workloads should use two or three AZs. Three AZ deployments are strongly recommended for workloads with higher availability requirements.

Note: The three AZ option is only available in regions where there are three or more AZs.

Amazon OpenSearch Service supports three AZ deployments in all regions in which the service is available, except US West (N. California), where we support two AZs only.

Amazon OpenSearch Service is a fully managed service that lets you run and scale OpenSearch clusters without having to worry about managing, monitoring, and maintaining your infrastructure, or having to build in-depth expertise in operating OpenSearch clusters. As a fully managed service, Amazon OpenSearch Service currently runs on AWS. However, OpenSearch is a distributed, community-driven, Apache 2.0-licensed, 100% open-source search and analytics suite that you can run on premises or in hybrid and multicloud environments. For example, there are partners who provide OpenSearch on other cloud platforms, or use OpenSearch in their applications. OpenSearch helps you more easily ingest, secure, search, aggregate, view, and analyze data for a number of use cases such as log analytics, application search, enterprise search, and more. OpenSearch provides a highly scalable system for providing fast access and response to large volumes of data with an integrated visualization tool—OpenSearch Dashboards—that makes it easier for users to explore their data. OpenSearch is powered by the Apache Lucene search library, and it supports a number of search and analytics capabilities such as k-nearest neighbors (KNN) search, SQL, Anomaly Detection, Machine Learning Commons, Trace Analytics, full-text search, and more.

No, Opensearch Reserved instance are not flexible; they only apply to the exact instance type that you reserve.

Yes. You can create a new Amazon OpenSearch Service domain with the Domain Creation Wizard in the console with just a few clicks. While creating a new domain you can specify the number of instances, instance types, and EBS volumes you want allocated to your domain. You can also modify or delete existing Amazon OpenSearch Service domains using the console.

Yes, Amazon OpenSearch Service is integrated with Amazon VPC. When choosing VPC access, IP addresses from your VPC are attached to your Amazon OpenSearch Service domain and all network traffic stays within the AWS network and is not accessible to the Internet. Moreover, you can use security groups and IAM policies to restrict access to your Amazon OpenSearch Service domains.

Yes. AWS CloudFormation supports Amazon OpenSearch Service. For more information, see the CloudFormation Template Reference documentation.

Yes. You can configure dedicated master nodes for your domains. When choosing a dedicated master configuration, you can specify the instance type and instance count.

Yes. You can create multiple Elasticsearch or OpenSearch indices within the same Amazon OpenSearch Service domain. Elasticsearch and OpenSearch automatically distribute the indices and any associated replicas between the instances allocated to the domain.

Amazon OpenSearch Service supports three options for data ingestion:

• For large data volumes, we recommend Amazon Kinesis Data Firehose, a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also transform, batch and compress the data before loading it.
• Amazon OpenSearch Service supports integration with Logstash. You can configure your Amazon OpenSearch Service domain as the data store for all logs arriving from your Logstash implementation.
• You can use native Elasticsearch (up to version 7.10) or OpenSearch APIs, such as the index and bulk APIs, to load data into your domain.

Yes. Amazon OpenSearch Service supports integration with Logstash. You can set up your Amazon OpenSearch Service domain as the backend store for all logs coming through your Logstash implementation. You can set up access control on your Amazon OpenSearch Service domain to either use request signing to authenticate calls from your Logstash implementation, or use resource based IAM policies to include IP addresses of instances running your Logstash implementation.

Yes. Amazon OpenSearch Service offers visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5 to 7.10 versions).

You can choose between local on-instance storage or EBS volumes. During domain creation, if you select EBS storage, you can increase and decrease the size of the storage volume as necessary.

You can choose between Magnetic, General Purpose, and Provisioned IOPS EBS volumes.

Yes. Amazon OpenSearch Service deploys storage based the choice of instance and/or the size of the associated EBS volume. The maximum storage per node is 24 TB with R6g.12xlarge instances with EBS gp3 storage. With the default maximum of 80 data nodes allowed per Amazon OpenSearch Service domain, you can allocate about 1920 TB of storage to a single domain. You can request a service limit increase up to 200 instances per domain by creating a case with the AWS Support Center. With 200 instances, you can allocate about 3 PB of storage to a single domain.

If you deploy your data instances in a single AZ, your dedicated master instances are also deployed in the same AZ. However, if you deploy your data instances across two or three AZs, Amazon OpenSearch Service automatically distributes the dedicated master instances across three AZs. The exception to this rule occurs if a region only has two AZs or if you select an older-generation instance type for the master instances that is not available in all AZs. For more details, refer our documentation.

For production workloads, we recommend deploying your data instances across three AZs since it offers better availability. Also, we recommend provisioning instances in multiples of three for equal distribution across AZs. In regions where three AZs are not available, we recommend using a two AZ deployment with an even number of data instances. In all cases, we recommend provisioning three dedicated master instances.

You can enable three AZ deployment for both existing and new domains using the AWS console, CLI or SDKs. For more details, refer our documentation.

No. Amazon OpenSearch Service does not charge anything for enabling three AZ deployment. You only pay for the number of instances in your domain, not the number of AZs to which they are deployed.

All domains configured for multiple AZs will have zone awareness enabled to ensure shards are distributed across Availability Zones. In the console, you can now explicitly choose two or three AZ deployments. Domains previously configured with “Zone Awareness” will continue to be deployed across two AZs unless they are reconfigured. For more details, refer our documentation.

If one or more instances in an AZ are unreachable or not functional, Amazon OpenSearch Service automatically tries to bring up new instances in the same AZ to replace the affected instances. In the rare event that new instances cannot be brought up in the AZ, Amazon OpenSearch Service brings up new instances in the other available AZs if the domain has been configured to deploy instances across multiple AZs. Once the AZ issue resolves, Amazon OpenSearch Service rebalances the instances such that they are equally distributed across the AZs configured for the domain. For more details refer our documentation.

Even when you configure one replica, we recommend three AZs. If an AZ disruption occurs in a three AZ domain, you only lose one-third of your capacity but if the disruption occurs in a two AZ domain, you lose half your capacity, which can be more disruptive. Also, in a three AZ domain, when an AZ is disrupted, Amazon OpenSearch Service can fall back to the two remaining AZs, and still support cross-AZ replication . In a two AZ domain, you lose cross-AZ replication if one AZ is disrupted, which can further reduce availability. For more details refer our documentation.

The number of AZs your domain is deployed to corresponds to the number of subnets you have configured for your VPC domain. You need to configure at least three subnets in your VPC domain to enable three AZ deployment. For more details on configuring VPC, refer our documentation.

Yes. The programs with public Internet access can access Amazon OpenSearch Service domains through a public endpoint. If your data center is already connected to Amazon VPC through Direct Connect or SSH tunneling, you can also use VPC access. In both cases, you can configure IAM policies and security groups to allow programs running on servers outside of AWS to access your Amazon OpenSearch Service domains. Click here for more information about signed requests.

To migrate data from an existing Elasticsearch or OpenSearch cluster, you should create a snapshot of an existing cluster, and store the snapshot in your Amazon S3 bucket. Then you can create a new Amazon OpenSearch Service domain and load data from the snapshot into the newly created Amazon OpenSearch Service domain using the restore API.

Amazon OpenSearch Service allows you to control the scaling of your Amazon OpenSearch Service domains using the console, API, and CLI. You can scale your Amazon OpenSearch Service domain by adding, removing, or modifying instances or storage volumes depending on your application needs. Amazon OpenSearch Service is integrated with Amazon CloudWatch to provide metrics about the state of your Amazon OpenSearch Service domains to enable you to make appropriate scaling decisions for your domains.

No. Scaling your Amazon OpenSearch Service domain by adding or modifying instances, and storage volumes is an online operation that does not require any downtime.

Yes. If you enable replicas for your OpenSearch/Elasticsearch indices and use multiple Availability Zones, Amazon OpenSearch Service automatically distributes your primary and replica shards across instances in different AZs.

Yes. Amazon OpenSearch Service exposes several performance metrics through Amazon CloudWatch including number of nodes, cluster health, searchable documents, EBS metrics (if applicable), CPU, memory and disk utilization for data and master nodes. Please refer to the service documentation for a full listing of available CloudWatch metrics.

Yes. AWS CloudTrail is a web service that records AWS API calls for your account and delivers log files to you. The AWS API call history produced by AWS CloudTrail enables security analysis, resource change tracking, and compliance auditing. Learn more about AWS CloudTrail at the AWS CloudTrail detail page, and turn it on via CloudTrail's AWS Management Console home page.

A snapshot is a copy of your Amazon OpenSearch Service domain at a moment in time.

Creating snapshots can be useful in case of data loss caused by node failure, as well as the unlikely event of a hardware failure. You can use snapshots to recover your Amazon OpenSearch Service domain with preloaded data or to create a new Amazon OpenSearch Service domain with preloaded data. Another common reason to use backups is for archiving purposes. Snapshots are stored in Amazon S3.

Yes. By default, Amazon OpenSearch Service automatically creates hourly snapshots of each Amazon OpenSearch Service domain and retains them for 14 days.

Amazon OpenSearch Service will retain the last 14 days worth of automated hourly snapshots.

There is no additional charge for the automated hourly snapshots. The snapshots are stored for free in an Amazon OpenSearch Service S3 bucket and will be made available for node recovery purposes.

Yes. You can use the snapshot API to create additional manual snapshots in addition to the daily-automated snapshots created by Amazon OpenSearch Service. The manual snapshots are stored in your S3 bucket and will incur relevant Amazon S3 usage charges.

Yes. Customers can create a new Amazon OpenSearch Service domain and load data from the snapshot into the newly created Amazon OpenSearch Service domain using the OpenSearch/Elasticsearch restore API.

The daily snapshots retained by Amazon OpenSearch Service will be deleted as part of domain deletion. Before deleting a domain, you should consider creating a snapshot of the domain in your own S3 buckets using the manual snapshot process. The snapshots stored in your S3 bucket will not be affected if you delete your Amazon OpenSearch Service domain.

Amazon OpenSearch Service exposes three Elasticsearch or OpenSearch logs through Amazon CloudWatch Logs: error logs, search slow logs, and index slow logs. These logs are useful for troubleshooting performance and stability issues with one’s domain.

Slow logs are log files that help track the performance of various stages in an operation. OpenSearch and Elasticsearch exposes two kinds of slow logs:

• Index Slow Logs – These logs provide insights into the indexing process and can be used to fine-tune the index setup.
• Search Slow Logs – These logs provide insights into how fast or slow queries and fetches are performing. These logs help fine tune the performance of any kind of search operation on OpenSearch or Elasticsearch.

For complete details on slow logs, please refer to OpenSearch documentation.

Slows logs can be enabled via the click of a button from the Console or via our CLI and APIs. For more details please refer to our documentation .

Yes. You can update the settings for a specific index to enable or disable slow logs for it. For more details refer to our documentation.

No. Turning on slow logs in Amazon OpenSearch Service enables the option to publish the generated logs to Amazon CloudWatch Logs for indices in the given domain. However, in order to generate the logs you have to update the settings for one or more indices to start the logging process. For more details on setting the index configuration for enabling slow logs, please refer to our documentation.

No. The generation of log files are dependent on the index settings. To turn off generation of the log files you have to update the index configuration. For more details on setting the index configuration for enabling slow logs, see our documentation.

You can only change the granularity of logging for slow logs. OpenSearch and Elasticsearch expose multiple levels of logging for slow logs. You need to set the appropriate level in the configuration of your index. For more details on setting the index configuration for enabling slow logs, please refer to OpenSearch documentation.

When slow logs or error logs are enabled, Amazon OpenSearch Service starts publishing the generated logs to CloudWatch Logs. Amazon OpenSearch Service does not charge anything for enabling the logs. However, standard CloudWatch charges will apply.

OpenSearch uses Apache Log4j 2 and its built-in log levels (from least to most severe) of TRACE, DEBUG, INFO, WARN, ERROR, and FATAL. If you enable error logs, Amazon OpenSearch Service publishes log lines of WARN, ERROR, and FATAL, and select errors from the DEBUG level to CloudWatch. For more details, refer our documentation.

Error logs can be enabled via the click of a button from the AWS Console or via our CLI and APIs. For more details please refer to our documentation.

No, error logs are exposed for the entire domain. That is, once enabled, log entries from all indices in the domain will be made available.

No, error logs are available only for Elasticsearch versions 5.x and above.

Yes. Each log entry made into CloudWatch will be limited to 255,000 characters. If your log entry is bigger than that, it will be truncated to 255,000 characters.

Slow logs are only needed when you want to troubleshoot your indexes or fine-tune performance. The recommended approach is to only enable logging for those indexes for which you need additional performance insights. Also, once the investigation is done, you should turn off logging so that you don’t incur any additional costs on account of it. For more details, see our documentation.

CloudWatch offers multiple ways to consume logs. You can view log data, export it to S3, or process it in real time. To learn more, see the CloudWatch Logs developer guide.

Yes, slow logs can be enabled for all versions of OpenSearch and Elasticsearch supported by Amazon OpenSearch Service. However, there are slight differences in the way log settings can be specified for each version of Elasticsearch. Please refer to our documentation for more details.

No. There will not be any down-time. Every time the log status is updated, we will deploy a new cluster in the background and replace the existing cluster with the new one. This process will not cause any down time. However, since a new cluster is deployed the update to the log status will not be instantaneous.

Amazon OpenSearch Service currently supports in-place version upgrade for domains with any OpenSearch version or Elasticsearch versions 5.x and above. The target versions that we support for the upgrade are 5.6, 6.3, 6.4, 6.5, 6.7, 6.8, 7.1, 7.4, 7.7, 7.8, 7.9, and 7.10. For more details refer our documentation.

Please refer to our documentation for details on migrating from various Elasticsearch versions.

No. Your domain remains available throughout the upgrade process. However, part of the upgrade process involves relocating shards, which can impact domain performance. We recommend upgrading when the load on your domain is low.

In-place version upgrade is available only for domains running Elasticsearch 5.x and above. If your domain is of version 5.x or above, you can run the upgrade eligibility check to validate whether your domain can be upgraded to the desired version. Please refer to our documentation to learn more.

For detailed list of the tests we run to validate upgrade eligibility, please refer to our documentation.

No. Once the in-place version upgrade has been triggered, you cannot make changes to your domain configuration until the upgrade completes or fails. You can continue reading and writing data while the upgrade is in progress. Also, you can delete the domain, in which case the upgrade is terminated and the domain deleted.

The version upgrade process automatically takes a snapshot of the system and only starts the actual upgrade if the snapshot succeeds. If the upgrade is in progress when the automated snapshot’s start time is reached, the automated snapshot is skipped for that day and continued on the next day.

Amazon OpenSearch Service runs a set of tests before triggering the upgrade to check for known issues that can block the upgrade. If no issues are encountered, the service takes a snapshot of the domain and starts the upgrade process if the snapshot is successful. The upgrade is not triggered if there are any issues encountered with any of the steps.

If encountered issues are minor and fixable, Amazon OpenSearch Service automatically tries to address them and unblock the upgrade. However, if an issue blocks the upgrade, the service reverts back to the snapshot that was taken before the upgrade and logs the error. For more details on viewing the logs from the upgrade progress, please refer to our documentation.

Yes. You can view the upgrade logs from the AWS console or request them using the CLI or SDKs. Please refer to our documentation for more details.

No. After the upgrade has been triggered, it cannot be paused or cancelled until it either completes or fails.

Yes. However, if you want to keep all of your domains on the same version, we recommend running the upgrade eligibility check on all domains before upgrading them. This extra step can help catch issues with one domain that might not be present on others.

Depending on the amount of data and the size of the cluster, upgrades can take anywhere from a few minutes to a few hours to complete.

No. With in-place version upgrade, all the data in your cluster is also restored as part of the upgrade process. If you only wish to upgrade the domain alone, you can take a snapshot of your data, delete all your indexes from the domain and then trigger an in-place version upgrade. Alternatively, you can create a separate domain with the newer version and then restore your data to that domain.

No. If you need to downgrade to an older version, contact AWS Support to restore the automatic, pre-upgrade snapshot on a new domain. If you took a manual snapshot of the original domain, you can perform this step yourself.

Multi-AZ with Standby is a new deployment option for Amazon OpenSearch Service that enables high-availability and consistent performance for business-critical workloads. With Multi-AZ with Standby, OpenSearch Service managed clusters are resilient to infrastructure failures like node drops, or a single Availability Zone failure, assuring no impact to performance or availability, even in the event of a single Availability Zone failure. Multi-AZ with Standby provides the added benefit of simplifying cluster configuration and management by enforcing best practices and reducing complexity.

To enable Multi-AZ with Standby, managed clusters need to meet the following conditions:

• Run OpenSearch 1.3 or more recent version.
• Deploy in AWS Regions with 3-AZ. Currently, AWS North California region does not support 3-AZ and are therefore not suitable for Multi-AZ with Standby.
• Number of data nodes needs to be in multiples of three.
• Number of data copies (primary + replica) should be in multiples of three.
• Follow sizing guidelines for the leader (recommended size based on number of nodes,  number of shards, and number of mappings in your cluster).

With Multi-AZ with Standby, Amazon OpenSearch Service detects and automatically recovers from some of the infrastructure failures. Amazon OpenSearch Service automatically fails over from active to standby nodes in under a minute when any of the following events occur:

• Loss of one active AZ or all nodes in an active AZ
• Loss of connectivity to one active AZ
• Instance hardware failure in the active AZ
• Storage failure on a node in the active AZ

Currently Multi-AZ with Standby does not cover the following events:

• Loss of master Quorum, as recovery from this event can take several minutes
• Loss of multiple Availability Zones
• Loss of connectivity to a Region
• Loss of more than 50% nodes in more than one AZ
• Downtime caused due to insufficient compute or storage as a result of change in workload characteristics
• Downtime caused due to rouge queries
• Loss of one or more services that Amazon OpenSearch Service depends on like ARPS and ALB
• Downtime of OpenSearch Dashboard during version upgrades

No. In principle, the sizing guidelines remain the same. Multi-AZ with Standby has pre-requisites that simplify the mental model required to size the cluster. The way to think of sizing for managed cluster is that you should look at the capacity needed to service your workload and then add 50% for redundancy. The primary difference between the current ‘Zone Awareness’ option and the Multi-AZ with Standby option is how redundant or additional capacity is handled to maintain availability. Multi-AZ with Standby requires you have at least one copy of data in each AZ, so that Multi-AZ with Standby can explicitly reserve capacity in one AZ as standby. This standby capacity acts as a failover target during AZ disruption or instance failure. The existing model requires that you maintain an optimum levels of resources to serve your workload. You would continue to monitor your cluster for sizing issues and take corrective actions as the workload characteristics changes.

No. Amazon OpenSearch Service works on a shared responsibility model. You are responsible for ensuring that your cluster is sized in accordance with your workload. Multi-AZ with Standby make the mental model of setting up your cluster simple. You should continue to monitor the error and latency metrics along with storage, CPU and RAM utilization for signals that the cluster is overloaded and may need to be scaled.

No. The feature Multi-AZ with Standby is available at no extra cost. You continue to pay for the resources deployed on the cluster to serve your workload. If your cluster already follows the best practices and has at least three copies of data for a 3-AZ cluster, you are unlikely to incur additional cost in moving to Multi-AZ with Standby. However, if your cluster is undersized or does not have enough redundant capacity to serve your workload, you will need to add capacity to move to Multi-AZ with Standby for enhanced availability and performance. The standby capacity is reserved from the configured total capacity.

Our Amazon OpenSearch Service SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Amazon OpenSearch Service.

You are eligible for a SLA credit for Amazon OpenSearch Service under the Amazon OpenSearch Service SLA if multi-AZ domains on Amazon OpenSearch Service have a Monthly Uptime Percentage of less than 99.9% during any monthly billing cycle.

For full details on all of the terms and conditions of the SLA, as well as details on how to submit a claim, please see the Amazon OpenSearch Service SLA details page.

Cross-cluster search is an Elasticsearch and OpenSearch feature that enables performing queries and aggregation across two connected clusters. It works by setting up a lightweight unidirectional connection between participating clusters.

Domains participating in a cross-cluster search needs to meet the following criteria:

• Participating domains should be on OpenSearch or Elasticsearch version 6.8 and above
• Participating domains need to have encryption in transit enabled
• Participating domains need to have Fine Grained Access Control (FGAC) enabled
• Participating domains versions should adhere to the same rules as rolling version upgrade

Cross-cluster search is currently supported on the following instance types

• i2, i3 family
• r3, r4, r5 family
• m4, m5 family
• c4, c5, family
• Graviton family

Cross-cluster search is not supported on the t2 and m3 family instances due to technical limitation.

Yes. Participating domains can belong to two different AWS accounts.

Yes.

To get started with cross-cluster search, follow the documentation here.

Cross-cluster replication, a new capability that allows Amazon OpenSearch Service customers to automate copying and synchronizing indices from one cluster to another at low latency in same or different AWS Regions.

Domains participating in a cross-cluster replications needs to meet the following criteria:

• Participating domains should be on Elasticsearch version 7.10
• Participating domains need to have encryption in transit enabled
• Participating domains need to have Fine Grained Access Control (FGAC) enabled
• Participating domains versions should adhere to the same rules as rolling version upgrade

Yes. domains in two different AWS Regions can participate in cross-cluster replication.

No. Current implementation of cross-cluster replication does not support Ultrawarm or Cold Storage.

Yes. You need to pay standard AWS data transfer charges for the data transferred in and out of Amazon OpenSearch Service.

We announced the OpenSearch project, a community-driven, open source fork of Elasticsearch and Kibana, on April 12, 2021. We committed to making a long-term investment in OpenSearch to ensure users continue to have a secure, high-quality, fully open source search and analytics suite with a rich roadmap of new and innovative functionality. This project includes OpenSearch (derived from Elasticsearch 7.10.2) and OpenSearch Dashboards (derived from Kibana 7.10.2). We launched version 1.0 of OpenSearch on July 12, 2021. As part of our long-term commitment to OpenSearch, we added support for OpenSearch 1.0 on the managed service on September 7, 2021 and changed the name from Amazon Elasticsearch Service to Amazon OpenSearch Service. Along with OpenSearch 1.0, we continue to support legacy Elasticsearch versions until 7.10 on the service. Aside from the name change, you can rest assured that we will continue to deliver the same great experience without any impact to ongoing operations, development methodology, or business use. Learn more about OpenSearch here: https://opensearch.org

We have strived to make this name change as seamless as possible for you. There are aspects, such as the new SDK/configuration APIs, that require your action to ensure you derive the best benefits from the service. While the existing SDK will continue to work from a compatibility perspective, any new functionality that requires new configuration APIs will only be implemented in the new SDK. Hence, we recommend that you move to the new SDK. In addition, irrespective of the new SDK, we strongly recommend that you move your existing IAM policies to use the renamed configuration APIs. As of now, your existing IAM policies will continue to work with the old API definition. However, we will move over to the new API-based permission validation and we will eventually require you to use the new APIs in your policies (specifically for the APIs where there is a name change; e.g. CreateElasticsearchDomain to CreateDomain). Please see the documentation for more details.

No. From a backward compatibility perspective, we will ensure that your existing setup continues to work with OpenSearch 1.0. However, we recommend that you eventually move to the latest SDK for a cleaner and up-to-date experience, as mentioned above.

No. There are no changes to pricing.

OpenSearch includes certain Apache-licensed Elasticsearch code from Elasticsearch B.V. and other source code. Elasticsearch B.V. is not the source of that other source code. ELASTICSEARCH is a registered trademark of Elasticsearch B.V.

Upgrading to OpenSearch 1.x ensures your search infrastructure is built on a growing and dynamic Apache-Licensed open source project and gives you access to a wealth of innovative improvements and features available in OpenSearch 1.2 (as of this writing).  Features such as enterprise-grade security, alerting, data-lifecycle management, observability, ML-based anomaly detection, and more are all part of  OpenSearch Service, with no additional licensing fees.

We use a blue/green (BG) deployment process during upgrade. During a BG, the service adds nodes to the OpenSearch Service cluster in the new configuration and version, migrates data from the old nodes, and drops the old nodes when the data migration is complete. During a BG, search and indexing APIs are available and function normally. Although BG is designed not to interfere with query and indexing requests, some changes (especially those involving changes to security-related settings) can cause dashboards to be unavailable during the period of change.

OpenSearch supports several versions of Elasticsearch and OpenSearch, some of which already have end of Standard and Extended Support dates announced. For a complete list of engine versions, and corresponding support dates, please see the documentation. For more information about Extended Support, please see the Extended Support. 

AWS maintains 19 versions of Apache-2.0-licensed Elasticsearch. None of these versions are deprecated or planned for deprecation at this time.

Yes, the upgrade will trigger a BG deployment process. Please review the upgrade preparation and steps here.

Please work with your AWS account team for information based on your specific situation with RIs.

The OpenSearch project 1.0 is a fork of open source Elasticsearch 7.10.2. It is wire-compatible with Elasticsearch 7.10--you don’t need to change your usage. To migrate, you can upgrade your domain to version Elasticsearch 7.10 from any prior version in the 6.x and 7.x series, take a snapshot, and restore that snapshot to a domain running OpenSearch Service 1.x. Some clients or tools include version checks that may cause the client or tool to not work with OpenSearch Service. When you upgrade, enable compatibility mode to work around these version checks.

In most cases, you can continue to use your existing clients. The APIs and core search functionality are compatible with Elasticsearch version 7.10.2. If you have older clients, the clients perform version checking, or the clients leverage functionality that is targeted to older versions of Elasticsearch such as major versions 5 or 6, then we suggest that you try to bring your clients up to a minimum standard of support on 7.10.2 to ensure a smooth transition.

The OpenSearch project supports a broad range of clients that are specifically built to address the OpenSearch versions of the engine on Amazon OpenSearch Service. Check your client against the list of the latest OpenSearch clients and supported programming languages for those clients.

You can enable the compatibility mode feature to interoperate with clients from other vendors, but remember to check the version that OpenSearch reports. Enable this setting to ensure the service responds with version 7.10.2 to clients developed prior to the introduction of the OpenSearch Service engine.

Elasticsearch 5.x indices are not compatible with Elasticsearch 7.10 or OpenSearch 1.x. You must create a new index and load data from your source. If you are running a log analytics workload, you can evaluate whether your data retention strategy supports running in parallel while you build up a full data set on the new domain.

Yes, please contact opensearchmigration-si-support@amazon.com to request a list of partners for your region, industry, and project complexity. AWS Partner Network (APN) partners are trained and have the experience to assist you with upgrades. 

OpenSearch 1.0 is a fork of Elasticsearch 7.10.2. OpenSearch and Elasticsearch are compatible. If you enable compatibility mode, Elasticsearch clients are also compatible with OpenSearch 1.0.

Amazon OpenSearch Service does not and will not offer versions of the Elasticsearch engine after version 7.10.2.

As AWS announced when we forked Elasticsearch, we intended to build, and have built a thriving community around OpenSearch. We have published a roadmap for OpenSearch, complete with community input and consensus on the feature priorities. We will make every reasonable effort to maintain compatibility with Elasticsearch. Our goal is to grow with our community and Amazon OpenSearch Service customers.

You can directly upgrade to OpenSearch Service 1.0 from Elasticsearch and Kibana versions 6.8.0 to 7.10.2, and open distro for Elasticsearch (ODFE) 1.x. For rolling upgrades from ODFE to OpenSearch, we recommend first upgrading to ODFE 1.13, then upgrading to OpenSearch 1.0.

Migration resources are here:

Analytics Migrations

Migrating to Amazon OpenSearch Service

Every engine version that is launched in OpenSearch Service is by default covered by Standard Support. As part of Standard Support, AWS provides regular bug fixes and security updates. When Standard Support ends for a version, AWS provides an Extended Support period of at least 12 months after the Standard Support end date. During the Extended Support period, AWS will provide critical security fixes and operating system patches. This gives you more time to plan your upgrade to a more recent supported engine version. When you are running a version that is in Extended Support, you will be charged a flat fee/Normalized instance hour (NIH), in addition to the standard instance and storage costs. Please see the documentation for more information on Extended Support, and the schedule for various versions. For pricing information, please see the pricing page.

No. Domains that are running versions that have reached the end of Standard Support will automatically be covered under Extended Support, and will be priced accordingly. Once you upgrade your domain to a newer version under Standard Support, you will stop being billed for Extended Support.

Domains that are running Extended Support will be charged an additional flat fee/Normalized instance hour (NIH), in addition to the standard instance and storage pricing. Please see pricing page for the exact pricing by region. Your domain will be charged for Extended Support automatically from the next day after end of Standard Support. If your domain is running a version that has Standard and Extended Support dates published (see here for details), we will send you a notification on your Personal Health Dashboard, on the OpenSearch Service console, and through EventBridge events, 3-months prior to the end of Standard Support date. For more information on monitoring notifications in OpenSearch Service, please see the documentation here.

Domains running versions under Extended Support will be charged an additional flat fee/Normalized Instance Hour (NIH). NIH is computed as a factor of the instance size (e.g. medium, large), and the number of instance hours. For example, if you are running an m7g.medium.search instance for 24 hours in the US East (North Virginia), which is priced at $0.068/hour (on-demand), you will typically pay $1.632 ($0.068x24) as the instance cost. If you are running a version that is in Extended Support, you will pay an additional $0.0065/NIH, which is computed as $0.0065 x 24 (number of instance hours) x 2 (size normalization factor; 2 for medium-sized instances), which comes to $0.312 for Extended Support for 24 hours. The total amount you will pay for 24 hours will be a sum of the standard instance usage cost (excluding storage) and the Extended Support cost, which is $1.944 ($1.632+$0.312). Please see documentation for more information.

You can upgrade your domain to an engine version that is covered by Standard Support. A version is covered under Standard Support until the published end of Standard Support date, or if no end of Standard Support dates have been announced for the version.

No. We recommend that you upgrade to a version that is covered by standard or Extended Support or to a version for which end of support has not been announced. Once Extended Support ends for a version, domains running the specific version will not receive bug fixes or security updates.

Once Extended Support ends for a version, we will no longer release bug fixes or security updates for that specific version. Domains running unsupported versions will be isolated and data preserved for at least 30 days. Isolated domains can no longer be accessed, and you will have to restore data into a new domain running a supported version. We strongly recommend that you update your domain to a supported version before the end of Extended Support for a specific version. If you need further assistance, please contact AWS Support.

Yes. As long the engine version that you wish to use is covered by Extended Support, you can continue to operate the service as usual, without any restrictions.

As part of Extended Support, AWS provides critical security fixes, and operating system patches as required.

Yes, depending the version you are upgrading from and upgrading to. Please see the documentation here for a list of supported upgrade paths. When you are moving from legacy versions like ES 1.5 or ES 2.3, we do not support in-place upgrades. Please refer to the documentation here for instructions on upgrading your domains running legacy versions.

This zero-ETL integration with Amazon DynamoDB abstracts away the operational complexity in orchestrating the replication of data from an operational datastore to a search datastore. Data pipelines that are used to keep different datastores in sync can be challenging and costly to build and manage, and suffer from intermittent errors which are difficult to track. This integration enables Amazon DynamoDB customers to obtain near real-time search results from their transactional data by offering a fully managed solution for making operational data from Amazon DynamoDB available in Amazon OpenSearch Service within seconds of being written.

The zero-ETL integration of Amazon OpenSearch Service with Amazon DynamoDB uses Amazon OpenSearch Ingestion to seamlessly move operational data from Amazon DynamoDB to Amazon OpenSearch Service. To enable an integration, customers first choose the Amazon DynamoDB table whose data needs to be replicated. The zero-ETL integration feature sets up an Amazon OpenSearch Ingestion pipeline in the customer’s account that takes care of replicating the data to an Amazon OpenSearch Service managed cluster or serverless collection. Amazon OpenSearch Ingestion understands the structure of the Amazon DynamoDB tables and then bootstraps an Amazon OpenSearch Service managed cluster or serverless collection with the existing data from the DynamoDB tables. Optionally, customers can specify a schema for the indices that will be created in Amazon OpenSearch Service. Any updates to the DynamoDB table are also replicated to Amazon OpenSearch Service without any manual intervention by customers.

This zero-ETL features uses Amazon OpenSearch Ingestion to move the data from Amazon DynamoDB to Amazon OpenSearch Service and leverages the native data transformational capabilities of Amazon OpenSearch Ingestion pipelines to aggregate and filter the data while it is in motion. When moving the data from an Amazon DynamoDB table, customers may want to drop a few fields or create new fields based on aggregations across existing fields. Optionally, customers can also write custom logic for Amazon OpenSearch Ingestion to achieve bespoke transformational capability. For other users, who just want to move their entire data from source to sink, Amazon OpenSearch Ingestion provides out-of-the box blueprints so that they can perform the integrations with just a few button clicks.

In order to ensure that Amazon OpenSearch Ingestion has the necessary permissions to replicate data across both these systems, the zero-ETL integration feature creates an IAM role with the necessary permissions to read data from Amazon DynamoDB tables and write to an Amazon OpenSearch domain or collection. This role is then assumed by Amazon OpenSearch Ingestion pipelines to ensure that the right security posture is always maintained when moving the data from source to destination.

You can view all the metrics related to your zero-ETL integration with Amazon DynamoDB on the dashboards provided by Amazon OpenSearch Ingestion along with real-time logs in Amazon CloudWatch. This enables customers to set up custom alerting that is triggered when user-defined thresholds are breached.

The OpenSearch Service query engine has been rearchitected to support analysis of operational data stored in cloud object stores, such as Amazon S3 and S3-based data lakes. It does so without duplicating data. When seconds make a difference, customers can boost the performance of their queries as well as build fast-loading dashboards by using the new integration’s built-in query acceleration capabilities.

To get started from the AWS Management Console, customers set up a new data source from an existing OpenSearch Service domain running OpenSearch Service version 2.11 or greater. When setting up the new Direct Query data source, customers will need to provide read/write access to Amazon S3 and AWS Glue Data Catalog to facilitate querying data in Amazon S3 from OpenSearch Service. Customers can customize IAM policies to restrict access to specific buckets in Amazon S3 or resources in AWS Glue Data Catalog. After configuring the new data source in the console, customers will go to OpenSearch Service to configure role-based access controls, accelerations to speed up query performance, and optionally out-of-the-box dashboards for popular log type templates such as VPC Flow Logs, Elastic Load Balancer Logs, and NGINX Logs. Customers are billed for the compute resources consumed in the form of Direct Query OpenSearch Compute Units (OCUs; usage rates will apply). After configuring the new data source, customers can start querying their data directly from the OpenSearch API or OpenSearch Dashboards.

Customers only pay for the resources consumed by their workload. OpenSearch Service charges for only the compute needed to directly query external data as well as maintain optional indexes in OpenSearch Service. The compute capacity is measured in OpenSearch Compute Units (OCUs) which is the same units used by Amazon OpenSearch Serverless and Amazon OpenSearch Ingestion. The number of OCUs corresponds directly to the vCPU and memory required to query or maintain indexes based on the data. Customers will see one entry for compute in OCU-hours with the label for direct query. OCUs are billed on an hourly basis with per-minute granularity. If no queries or indexing activities are active, no OCUs are consumed. Costs for Amazon S3 or AWS Glue Data Catalog will be billed separately in the customer’s account. For more pricing details visit the Amazon OpenSearch Service pricing page.

The integration between Amazon OpenSearch Service and Amazon Security Lake provides a streamlined experience for directly searching, gaining insights from, and analyzing data stored in Security Lake, all within the OpenSearch Service. There are two ways you can integrate Security Lake and OpenSearch Service: query-access and data-access.

With the query-access integration, security analysts can use the OpenSearch Dashboards console to run direct queries on the data in Security Lake and leverage on-demand indexing and pre-built content to get started with security-related queries. For the data-access integration, the OpenSearch ingestion service provides a preconfigured Security Lake blueprint, which includes a default configuration for ingesting Open Cybersecurity Schema Framework (OCSF) parquet files from Security Lake. The zero-ETL integration with Security Lake is only offered on collections with OpenSearch UI as opposed to on clusters.

To get started, you first need to have an existing Security Lake setup in your AWS environment. This will provide the centralized storage and access to your enterprise security data.

You can then enable the integration between the two services through the AWS Management Console or by using AWS CloudFormation templates. This involves configuring the necessary permissions and access controls to allow Amazon OpenSearch Service to securely access and query the data in your Security Lake.

You can then explore the pre-built queries and integrations available through OCSF to quickly get started with common security analytics use cases. You also have the option to configure on-demand indexing of specific data sets from your Security Lake into OpenSearch Service for advanced analytics and visualization needs.