OpenSearch clients implement various high-level REST DSLs to invoke OpenSearch APIs. Efforts such as opensearch-clients#19 aim at generating these from spec in order to always be up-to-date with the default distribution, including plugins. However this is a game that cannot be won. Clients will always lag behind, and users often find themselves in a situation that requires them to invoke an API that is not supported by the client. Thus, in opensearch-clients#62 I proposed we level up all OpenSearch language clients in their capability to make raw JSON REST requests. I am happy to report that six months later we have support for sending raw JSON to OpenSearch in all language clients!
In this post I’ll keep current state with links to working samples, similar to Making AWS SigV4 Authenticated Requests to Amazon OpenSearch. For many of these I am running a local copy of OpenSearch 2.9 in docker.
docker run \
-p 9200:9200 \
-p 9600:9600 \
-e "discovery.type=single-node" \
opensearchproject/opensearch:latest
Command Line
We’ll be looking for the equivalent of the four GET, POST, PUT and DELETE operations.
curl
curl -k -u admin:admin https://localhost:9200
{
"name" : "5d98546c8098",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "Hu0dA0iYREiBVPqEuHqYaA",
"version" : {
"distribution" : "opensearch",
"number" : "2.9.0",
"build_type" : "tar",
"build_hash" : "1164221ee2b8ba3560f0ff492309867beea28433",
"build_date" : "2023-07-18T21:22:48.164885046Z",
"build_snapshot" : false,
"lucene_version" : "9.7.0",
"minimum_wire_compatibility_version" : "7.10.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
curl -k -u admin:admin \
-X POST \
-H "Content-type:application/json" \
--data '{"director":"Bennett Miller","title":"Moneyball","year":2011}' \
https://localhost:9200/movies/_doc/1 | jq
{
"_index": "movies",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 1,
"_primary_term": 1
}
curl -k -u admin:admin \
-X GET \
https://localhost:9200/movies/_doc/1 | jq
{
"_index": "movies",
"_id": "1",
"_version": 1,
"_seq_no": 2,
"_primary_term": 1,
"found": true,
"_source": {
"director": "Bennett Miller",
"title": "Moneyball",
"year": 2011
}
}
curl -k -u admin:admin \
-X PUT \
-H "Content-type:application/json" \
--data '{"director":"Bennett Miller","title":"Moneyball","year":2011}' \
https://localhost:9200/movies/_doc/1 | jq
{
"_index": "movies",
"_id": "1",
"_version": 3,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 2,
"_primary_term": 1
}
curl -k -u admin:admin \
-X DELETE \
https://localhost:9200/movies/_doc/1 | jq
{
"_index": "movies",
"_id": "1",
"_version": 4,
"result": "deleted",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 3,
"_primary_term": 1
}
Java
opensearch-java
The Java client added .generic() that returns a OpenSearchGenericClient in 2.10.0 and fixed the implementation for AWS transport options in 2.10.3.
OpenSearchClient client = new OpenSearchClient(...)
OpenSearchGenericClient genericClient = client
.generic()
.withClientOptions(ClientOptions.throwOnHttpErrors());
The client can be used to execute a simple GET request/response.
Response response = genericClient.execute(
Requests.builder()
.endpoint("/")
.method("GET")
.build());
System.out.println(response.getBody().get().bodyAsString());
Sending JSON data is similar.
Requests.builder()
.endpoint(index + "/_doc/1")
.method("POST")
.json("{\"director\":\"Bennett Miller\",\"title\":\"Moneyball\",\"year\":2011}")
.build();
You can parse responses as generic JSON as well. Here’s a search example.
Response searchResponse = genericClient.execute(
Requests.builder().endpoint(index + "/_search").method("POST")
.json("{"
+ " \"query\": {"
+ " \"match\": {"
+ " \"title\": {"
+ " \"query\": \"Moneyball 2\""
+ " }"
+ " }"
+ " }"
+ "}")
.build());
JsonNode json = searchResponse.getBody()
.map(b -> Bodies.json(b, JsonNode.class, client._transport().jsonpMapper()))
.orElse(null);
JsonNode hits = json.get("hits").get("hits");
for (int i = 0; i < hits.size(); i++) {
System.out.println(hits.get(i).get("_source").toString());
}
See the updated documentation and working demo for more information.
Ruby
opensearch-ruby
The Ruby client added .http in 3.1.0.
A simple GET.
client = OpenSearch::Client.new(...)
info = client.http.get('/')
puts info
Create a document.
document = { title: 'Moneyball', director: 'Bennett Miller', year: 2011 }
client.http.post("/movies/_doc/1", body: document)
Search for a document.
results = client.http.post(
"/movies/_search",
body: { query: { match: { director: 'miller' } } }
)
results['hits']['hits'].each do |hit|
puts hit
end
Raw JSON also works with bulk by automatically transforming arrays into nd-json.
body = [
{ index: { _index: 'books', _id: 1 } },
{ title: 'The Lion King', year: 1994 },
{ index: { _index: 'books', _id: 2 } },
{ title: 'Beauty and the Beast', year: 1991 }
]
client.http.post('_bulk', body: body)
See the updated documentation and working demo for more information.
Node.js
opensearch-js
The Node.js client has long supported client.transport.perform_request and wrapped it up in the http namespace in 2.5.0.
info = client.http.get("/")
print(f"Welcome to {info["version"]["distribution"]} {info["version"]["number"]}!")
Use body to send JSON data.
q = "miller"
query = {
"size": 5,
"query": {
"multi_match": {
"query": q,
"fields": ["title^2", "director"]
}
}
}
client.http.post("/movies/_search", body = query)
See the updated documentation and working demo for more information.
Python
opensearch-py
The Python client has long exposed client.transport.perform_request and wrapped it up in an http namespace in 2.4.0.
info = client.http.get('/')
print(f"Welcome to {info['version']['distribution']} {info['version']['number']}!")
Create a document.
document = {
'title': 'Moneyball',
'director': 'Bennett Miller',
'year': '2011'
}
client.http.put("/movies/_doc/1?refresh=true", body = document)
Search for a document.
query = {
'size': 5,
'query': {
'multi_match': {
'query': 'miller',
'fields': ['title^2', 'director']
}
}
}
client.http.post("/movies/_search", body = query)
Delete an index.
client.http.delete("/movies")
See the updated documentation and working demo for more information.
DotNet
opensearch-net
The .NET client added a high level DSL in 1.6.0.
var info = await client.Http.GetAsync<DynamicResponse>("/");
Console.WriteLine($"Welcome to {info.Body.version.distribution} {info.Body.version.number}!");
Search for a document.
const string q = "miller";
var query = new
{
size = 5,
query = new {
multi_match = new {
query = q,
fields = new[] {
"title^2", "director"
}
}
}
};
var search = await client.Http.PostAsync<DynamicResponse>(
"/movies/_search",
d => d.SerializableBody(query)
);
foreach (var hit in search.Body.hits.hits) {
Console.WriteLine($"Search Hit: {hit["_source"]["title"]}");
}
See the updated documentation and working demo for more information.
Rust
opensearch-rs
The rust client directly supports JsonBody<_> on request, and .json() on response.
let info: Value = client
.send::<(), ()>(Method::Get, "/", HeaderMap::new(), None, None, None)
.await?
.json()
.await?;
println!(
"{}: {}",
info["version"]["distribution"].as_str().unwrap(),
info["version"]["number"].as_str().unwrap()
);
let document: JsonBody<_> = json!({
"title": "Moneyball",
"director": "Bennett Miller",
"year": "2011"
}).into();
client.send(
Method::Put,
"movies/_doc/1",
HeaderMap::new(),
Some(&[("refresh", "true")]),
Some(document),
None,
).await?;
let query: JsonBody<_> = json!({
"size": 5,
"query": {
"multi_match": {
"query": "miller",
"fields": ["title^2", "director"]
}
}
}).into();
let search_response = client.send(
Method::Post,
&"/movies/_search",
HeaderMap::new(),
Option::<&()>::None,
Some(query),
None,
)
.await?;
let search_result = search_response.json::<Value>().await?;
println!("Hits: {:#?}", search_result["hits"]["hits"].as_array().unwrap());
client.send::<(), ()>(
Method::Delete,
"/movies",
HeaderMap::new(),
None,
None,
None,
)
.await?;
See the updated user guide, a working demo and a API vs. raw JSON diff for more information.
PHP
opensearch-php
The PHP client has added a request() wrapper in 2.3.0.
$info = $client->request('GET', '/');
echo "{$info['version']['distribution']}: {$info['version']['number']}\n";
$indexName = "movies";
$client->request('POST', "/$indexName/_doc/1", [
'body' => [
'title' => 'Moneyball',
'director' => 'Bennett Miller',
'year' => 2011
]
]);
$result = $client->request('POST', "/$indexName/_search", [
'body' => [
'query' => [
'multi_match' => [
'query' => 'miller',
'fields' => ['title^2', 'director']
]
]
]
]);
print_r($result['hits']['hits'][0], false);
$client->request('DELETE', "/$indexName/_doc/1");
$client->request('DELETE', "/$indexName");
See the updated documentation and a working demo for more information. A higher level DSL is a feature request, opensearch-php#192.
Go
opensearch-go
The go client has long supported Client.NewRequest and Perform.
infoRequest, _ := http.NewRequest("GET", "/", nil)
infoResponse, _ := client.Client.Perform(infoRequest)
resBody, _ := io.ReadAll(infoResponse.Body)
fmt.Printf("client info: %s\n", resBody)
Sending data is similar.
query := strings.NewReader(`{
"size": 5,
"query": {
"multi_match": {
"query": "miller",
"fields": ["title^2", "director"]
}
}
}`)
searchRequest, _ := http.NewRequest("POST", "/movies/_search", query)
searchRequest.Header["Content-Type"] = []string{"application/json"}
searchResp, _ := client.Client.Perform(searchRequest)
searchRespBody, _ := io.ReadAll(searchResp.Body)
fmt.Println("search: ", string(searchRespBody))
See the updated documentation for more information, and please contribute a working demo to the project or opensearch-go-client-demo as I am too lazy to write all the error handlers.