Logstach: jdbc_page_size does not dump all my data into elastic search

Question

I would like to export the tom_test2 postgresql table to elastic search. The table has 176805 rows:

=> select count(*) from tom_test2;
 count  
--------
 176805
(1 row)

The following confstore logstach file correctly imports my data into elastic search:

input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "xxx"
        # The user we wish to execute our statement as
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "xxx"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "org.postgresql.Driver"
        # our query
        statement => "select * from tom_test2"
    }
}


output {
    elasticsearch {
        hosts => ["xxx"]
        index => "tom"
        document_type => "tom_test"
    }
}

In elastic search:

GET tom/tom_test/_search

  "hits": {
    "total": 176805,
    "max_score": 1,
}

I delete the index in the elastic search:

delete tom

And now I would like to perform the same operation using jdbc_page_size, in case my data gets bigger, now my confstach logstach file:

input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "xxx"
        # The user we wish to execute our statement as
        jdbc_user => "xxx"
        jdbc_password => "xxx"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "xxx"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "org.postgresql.Driver"
        # our query
        statement => "select * from tom_test2"

        jdbc_page_size => 1000
        jdbc_paging_enabled => true
    }
}


output {
    elasticsearch {
        hosts => ["xxx"]
        index => "tom"
        document_type => "tom_test"
    }
}

Now my score is incorrect:

GET tom/tom_test/_search

  "hits": {
    "total": 106174,
    "max_score": 1,
}

since 176805-106174 = 70631 lines are missing

Answer 1

, : : , , postgresql , : , , :( , , , .

SELECT * FROM tom_test2 ORDER BY id, . : elasticsearch . , , .. 1 10000, . 10001 20000, ... .

, logstash..., jdbc_fetch_size: .. SELECT * FROM tom_test2. , "" , "" : .

Answer 2

jdbc_page_size WARNED jdbc_paging_enabled.

jdbc_fetch_size jdbc_page_size , .

PS: ;) http://discuss.elastic.co