Import / Index JSON File in Elasticsearch

Question

I am new to Elasticsearch and until now I entered data manually. For example, I did something like this:

$ curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{ "user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elastic Search" }' 

Now I have a .json file and I want to index it in Elasticsearch. I also tried something similar, but to no avail:

 curl -XPOST 'http://jfblouvmlxecs01:9200/test/test/1' -d lane.json 

How to import a .json file? Are there any steps I need to take to make sure the matching is correct?

Answer 1

The correct command if you want to use a file with curl is:

 curl -XPOST 'http://jfblouvmlxecs01:9200/test/_doc/1' -d @lane.json 

Elasticsearch does not have a schema, so you do not need to display. If you send json as is and use the default mapping, each field will be indexed and parsed using a standard parser .

If you want to interact with Elasticsearch through the command line, you can take a look at the elastic shell, which should be a little more convenient than curl.

2019-07-10: It should be noted that custom match types are outdated and should not be used. I updated the type in the above URL to make it easier to see which index was and which type, since both names with the name "test" were confusing.

Answer 2

In the current docs http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html :

If you provide a text file input for curling, you should use the --data-binary flag instead of plain -d. The latter does not save newline characters.

Example:

 $ curl -s -XPOST localhost:9200/_bulk --data-binary @requests 

Answer 3

We made a small tool for this type of thing https://github.com/taskrabbit/elasticsearch-dump

Answer 4

I am the author of rubbersearch_loader
I wrote an ESL for this particular problem.

You can download it using pip:

 pip install elasticsearch-loader 

And then you can upload the json files to elasticsearch by doing:

 elasticsearch_loader --index incidents --type incident json file1.json file2.json 

Answer 5

Adding to KenH's answer

 $ curl -s -XPOST localhost:9200/_bulk --data-binary @requests 

You can replace @requests with @complete_path_to_json_file

Note: @ important to the file path

Answer 6

I just made sure that I was in the same directory as the json file and then just ran this

 curl -s -H "Content-Type: application/json" -XPOST localhost:9200/product/default/_bulk?pretty --data-binary @product.json 

So, if you also make sure that you are in the same directory and run it this way. Note: the product / default / command in the command is specific to my environment. You can omit it or replace it with something that is relevant to you.

Answer 7

just get the postman from https://www.getpostman.com/docs/environments to indicate the location of the file with the command / test / test / 1 / _bulk? pretty.

Answer 8

One thing that I didn’t mention anyone about is that the JSON file should have one line with the index to which the next line belongs, for each line of a "clean" JSON file.

those.

 {"index":{"_index":"shakespeare","_type":"act","_id":0}} {"line_id":1,"play_name":"Henry IV","speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"} 

Without this, nothing works and he won’t tell you why

Answer 9

You use

 $ curl -s -XPOST localhost:9200/_bulk --data-binary @requests 

If the 'requests' is a JSON file, then you should change this to

 $ curl -s -XPOST localhost:9200/_bulk --data-binary @requests.json 

Now before that, if your json file is not indexed, you should insert an index line before each line inside the json file. You can do it with JQ. Link below: http://kevinmarsh.com/2014/10/23/using-jq-to-import-json-into-elasticsearch.html

Go to the training materials for elasticsearch (for example, the Shakespeare tutorial) and download the sample json file you use and look at it. Before each json object (each separate line) there is an index line. This is what you are looking for after using the jq command. This format is required to use the bulk API, regular json files will not work.

Answer 10

if you use VirtualBox and UBUNTU in it or you just use UBUNTU, then this may be useful

 wget https://github.com/andrewvc/ee-datasets/archive/master.zip sudo apt-get install unzip (only if unzip module is not installed) unzip master.zip cd ee-datasets java -jar elastic-loader.jar http://localhost:9200 datasets/movie_db.eloader 

Answer 11

I wrote some code to provide the Elasticsearch API through the file system API.

It is a good idea to clearly export / import data, for example.

I created a prototype of an elastic drive . It is based on FUSE.