Making AWS SigV4 Authenticated Requests to Amazon OpenSearch

Amazon OpenSearch and Amazon OpenSearch Serverless use AWS SigV4 for authentication. We’ve made it dead easy to make authenticated requests across all OpenSearch clients in opensearch-clients#22.

Command Line

curl

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...

curl \
  --verbose \
  --request GET "https://...us-west-2.es.amazonaws.com" \
  --aws-sigv4 "aws:amz:us-west-2:es" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  -H "x-amz-security-token:$AWS_SESSION_TOKEN"

If you want to PUT a document with curl you need some data, and the x-amz-content-sha256 header for Amazon OpenSearch Serverless. See this gist for a full example that inserts some vectors and perform an approximate nearest neighbor search.

awscurl

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...

awscurl \
  "https://search...us-west-2.es.amazonaws.com" \
  --region us-west-2 \
  --service es

See this gist for a full example that inserts some vectors and perform an approximate nearest neighbor search.

aws-es-curl

aws-es-curl \
  "https://search...us-west-2.es.amazonaws.com" \
  --region us-west-2

Java

opensearch-java

Use AwsSdk2Transport introduced in opensearch-java 2.1.0. This is the latest recommended approach.

SdkHttpClient httpClient = ApacheHttpClient.builder().build();

OpenSearchClient client = new OpenSearchClient(
    new AwsSdk2Transport(
        httpClient,
        "search-...us-west-2.es.amazonaws.com",
        Region.US_WEST_2,
        AwsSdk2TransportOptions.builder().build()
    )
);

InfoResponse info = client.info();
System.out.println(info.version().distribution() + ": " + info.version().number());

httpClient.close();

Working demo in Java in opensearch-java-client-demo, and another one written in Kotlin in opensearch-kotlin-client-demo.

aws-request-signing-apache-interceptor

Use an interceptor and any Apache REST client, including RestHighLevelClient.

HttpRequestInterceptor interceptor = new AwsRequestSigningApacheInterceptor(
    "es",
    Aws4Signer.create(), 
    DefaultCredentialsProvider.create(), 
    Region.US_WEST_2
);

CloseableHttpClient client = HttpClients.custom()
    .addInterceptorLast(interceptor)
    .build();

HttpGet httpGet = new HttpGet("https://...");
CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
System.out.println(httpResponse.getStatusLine());
System.out.println(IoUtils.toUtf8String(response.getEntity().getContent()));

You can see a working demo in the interceptor code. For an example that uses OpenSearch RestHighLevelClient see 1.x or 2.x depending on your version.

Ruby

opensearch-ruby

Use opensearch-aws-sigv4 1.0 or newer.

signer = Aws::Sigv4::Signer.new(
  service: 'es',
  region: 'us-west-2',
  access_key_id: ENV['AWS_ACCESS_KEY_ID'],
  secret_access_key: ENV['AWS_SECRET_ACCESS_KEY'],
  session_token: ENV['AWS_SESSION_TOKEN']
)

client = OpenSearch::Aws::Sigv4Client.new({
  host: 'https://...'
}, signer)

info = client.info
puts info['version']['distribution'] + ': ' + info['version']['number']

Working demo in opensearch-ruby-client-demo.

Node.js

opensearch-js

Use @opensearch-project/opensearch 2.x.

const client = new Client({
  ...AwsSigv4Signer({
    region: process.env.AWS_REGION || 'us-east-1',
    getCredentials: () => {
      const credentialsProvider = defaultProvider();
      return credentialsProvider();
    },
  }),
  node: process.env.OPENSEARCH_ENDPOINT
});

var info = await client.info();
var version = info.body.version
console.log(version.distribution + ": " + version.number);

Working demo in opensearch-node-client-demo.

Python

opensearch-py

url = urlparse(environ['OPENSEARCH_ENDPOINT'])
region = environ.get('AWS_REGION', 'us-east-1')

credentials = Session().get_credentials()

auth = AWSV4SignerAuth(credentials, region)

client = OpenSearch(
  hosts=[{
    'host': url.netloc,
    'port': url.port or 443
  }],
  http_auth=auth,
  use_ssl=True,
  verify_certs=True,
  connection_class=RequestsHttpConnection
)

info = client.info()
print(f"{info['version']['distribution']}: {info['version']['number']}")

Working demo in opensearch-python-client-demo.

DotNet

opensearch-net

Use OpenSearch.Client 1.2.0 or newer.

var endpoint = new Uri(Environment.GetEnvironmentVariable("OPENSEARCH_ENDPOINT") ?? throw new ArgumentNullException("Missing OPENSEARCH_ENDPOINT."));
var region = Amazon.RegionEndpoint.GetBySystemName(Environment.GetEnvironmentVariable("AWS_REGION") ?? "us-east-1");
var connection = new AwsSigV4HttpConnection(region);
var config = new ConnectionSettings(endpoint, connection);
var client = new OpenSearchClient(config);

Console.WriteLine($"{client.RootNodeInfo().Version.Distribution}: {client.RootNodeInfo().Version.Number}");

Working demo in opensearch-dotnet-client-demo.

Rust

opensearch-rs

let url = Url::parse(&env::var("OPENSEARCH_ENDPOINT").expect("Missing OPENSEARCH_ENDPOINT"));
let conn_pool = SingleNodeConnectionPool::new(url?);
let aws_config = aws_config::load_from_env().await.clone();
let transport = TransportBuilder::new(conn_pool).auth(aws_config.clone().try_into()?).build()?;
let client = OpenSearch::new(transport);

let info: Value = client.info().send().await?.json().await?;
println!("{}: {}", info["version"]["distribution"].as_str().unwrap(), info["version"]["number"].as_str().unwrap());

Working demo in opensearch-rust-client-demo.

PHP

opensearch-php

$client = (new \OpenSearch\ClientBuilder())
  ->setHosts([getenv("OPENSEARCH_ENDPOINT")])
  ->setSigV4Region(getenv("AWS_REGION"))    
  ->setSigV4CredentialProvider(true)
  ->build();

$info = $client->info();

echo "{$info['version']['distribution']}: {$info['version']['number']}\n";

Working demo in opensearch-php-client-demo.

Go

opensearch-go

ctx := context.Background()
cfg, _ := config.LoadDefaultConfig(ctx)
signer, _ := requestsigner.NewSigner(cfg)

endpoint, _ := os.LookupEnv("OPENSEARCH_ENDPOINT")

client, _ := opensearch.NewClient(opensearch.Config{
  Addresses: []string{endpoint},
  Signer:    signer,
})

if info, err := client.Info(); err != nil {
  log.Fatal("info", err)
} else {
  var r map[string]interface{}
  json.NewDecoder(info.Body).Decode(&r)
  version := r["version"].(map[string]interface{})
  fmt.Printf("%s: %s\n", version["distribution"], version["number"])
}

Working demo in opensearch-go-client-demo.

Making AWS SigV4 Authenticated Requests to Amazon OpenSearch opensearch | aws