Making Raw JSON REST Requests to OpenSearch

OpenSearch clients implement various high-level REST DSLs to invoke OpenSearch APIs. Efforts such as opensearch-clients#19 aim at generating these from spec in order to always be up-to-date with the default distribution, including plugins. However this is a game that cannot be won. Clients will always lag behind, and users often find themselves in a situation that requires them to invoke an API that is not supported by the client. Thus, in opensearch-clients#62 I proposed we level up all OpenSearch language clients in their capability to make raw JSON REST requests. You help on these issues would be very much appreciated.

In this post I’ll keep current state with links to working samples, similar to Making AWS SigV4 Authenticated Requests to Amazon OpenSearch. For all these I am running a local copy of OpenSearch 2.9 in docker.

docker run \
  -p 9200:9200 \
  -p 9600:9600 \
  -e "discovery.type=single-node" \
  opensearchproject/opensearch:latest

Command Line

We’ll be looking for the equivalent of the four GET, POST, PUT and DELETE operations.

curl

curl -k -u admin:admin https://localhost:9200

{
  "name" : "5d98546c8098",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "Hu0dA0iYREiBVPqEuHqYaA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.9.0",
    "build_type" : "tar",
    "build_hash" : "1164221ee2b8ba3560f0ff492309867beea28433",
    "build_date" : "2023-07-18T21:22:48.164885046Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

curl -k -u admin:admin \
  -X POST \
  -H "Content-type:application/json" \
  --data '{"director":"Bennett Miller","title":"Moneyball","year":2011}' \
  https://localhost:9200/movies/_doc/1 | jq

{
  "_index": "movies",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 1,
  "_primary_term": 1
}

curl -k -u admin:admin \
  -X GET \
  https://localhost:9200/movies/_doc/1 | jq

{
  "_index": "movies",
  "_id": "1",
  "_version": 1,
  "_seq_no": 2,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "director": "Bennett Miller",
    "title": "Moneyball",
    "year": 2011
  }
}

curl -k -u admin:admin \
  -X PUT \
  -H "Content-type:application/json" \
  --data '{"director":"Bennett Miller","title":"Moneyball","year":2011}' \
  https://localhost:9200/movies/_doc/1 | jq

{
  "_index": "movies",
  "_id": "1",
  "_version": 3,
  "result": "updated",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 2,
  "_primary_term": 1
}

curl -k -u admin:admin \
  -X DELETE \
  https://localhost:9200/movies/_doc/1 | jq

{
  "_index": "movies",
  "_id": "1",
  "_version": 4,
  "result": "deleted",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 3,
  "_primary_term": 1
}

Java

opensearch-java

Feature request, opensearch-java#257.

Ruby

opensearch-ruby

Feature request, opensearch-ruby#209. Should also be possible via client.perform_request.

Node.js

opensearch-js

Feature request, opensearch-js#631. Should also be possible via transport.request.

Python

opensearch-py

The Python client exposes client.transport.perform_request.

info = client.transport.perform_request('GET', '/')
print(f"Welcome to {info['version']['distribution']} {info['version']['number']}!")

document = {
  'title': 'Moneyball',
  'director': 'Bennett Miller',
  'year': '2011'
}

client.transport.perform_request("PUT", "/movies/_doc/1?refresh=true", body = document)

query = {
  'size': 5,
  'query': {
    'multi_match': {
      'query': 'miller',
      'fields': ['title^2', 'director']
    }
  }
}

client.transport.perform_request("POST", "/movies/_search", body = query)

client.transport.perform_request("DELETE", "/movies")

See the updated documentation and working demo for more information. I also made a PR for a higher level DSL.

DotNet

opensearch-net

Feature request, opensearch-net#403.

Rust

opensearch-rs

The rust client directly supports JsonBody<_> on request, and .json() on response.

let info: Value = client
    .send::<(), ()>(Method::Get, "/", HeaderMap::new(), None, None, None)
    .await?
    .json()
    .await?;
    
println!(
    "{}: {}",
    info["version"]["distribution"].as_str().unwrap(),
    info["version"]["number"].as_str().unwrap()
);

let document: JsonBody<_> = json!({
    "title": "Moneyball",
    "director": "Bennett Miller",
    "year": "2011"
}).into();

client.send(
    Method::Put,
    "movies/_doc/1",
    HeaderMap::new(),
    Some(&[("refresh", "true")]),
    Some(document),
    None,
).await?;

let query: JsonBody<_> = json!({
  "size": 5,
  "query": {
      "multi_match": {
          "query": "miller",
          "fields": ["title^2", "director"]
      }
  }
}).into();

let search_response = client.send(
    Method::Post,
    &"/movies/_search",
    HeaderMap::new(),
    Option::<&()>::None,
    Some(query),
    None,
)
.await?;

let search_result = search_response.json::<Value>().await?;

println!("Hits: {:#?}", search_result["hits"]["hits"].as_array().unwrap());

client.send::<(), ()>(
  Method::Delete,
  "/movies",
  HeaderMap::new(),
  None,
  None,
  None,
)
.await?;

See the updated user guide, a working demo and a API vs. raw JSON diff for more information.

Daniel Doubrovkine