I am trying to set up a working example of presto and redis on my local machine according to the (limited) preliminary documentation .
Short description of the problem:
When using redis.key-prefix-schema-table=true and the redis key prefix with dev:simple_table: (as indicated on the redis pre-code page), all key columns are NULL and the inner column _key_corrupt is true. All value columns are parsed correctly.
Using redis.key-prefix-schema-table=false and using a clean JSON key and string values both work as expected. (note: I tried and could not use csv dataFormat)
What works:
I have an example of working with a redis server and a pre-server running on my local machine.
presto etc/catalog/redis.properties file:
connector.name=redis redis.table-names=simple_table redis.default-schema=dev redis.nodes=127.0.0.1:6379 redis.table-description-dir=/Users/acarson/var/redis/tables redis.key-delimiter=: redis.key-prefix-schema-table=false redis.hide-internal-columns=false
I have a table definition file in /Users/acarson/var/redis/tables/simple_table.json
{ "tableName": "simple_table", "schemaName": "dev", "key": { "dataFormat": "json", "fields": [ { "name": "id", "mapping": "id", "type": "BIGINT" }, { "name": "word", "mapping": "word", "type": "VARCHAR" } ] }, "value": { "dataFormat": "json", "fields": [ { "name": "name", "mapping": "name", "type": "VARCHAR" }, { "name": "number", "mapping": "number", "type": "BIGINT" }, { "name": "boolean", "mapping": "boolean", "type": "BOOLEAN" } ] } }
using redis-cli, I insert the value with this command:
SET '{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'
And I can then query the data using preo cli, and the resulting columns are displayed as expected:
presto:dev> SELECT * FROM simple_table; id | word | name | number | boolean | _key | _value | _key_length | _value_length | _key_corrupt | _value_corrupt ----+------+------+--------+---------+---------------------------+--------------------------------------------------+-------------+---------------+--------------+---------------- 42 | foo | bar | 3 | false | {"id": 42, "word": "foo"} | {"name": "bar", "number": 3, "boolean": "false"} | 25 | 48 | false | false
What fails:
I switch the value of redis.properties redis.key-prefix-schema-table to true, restart the preo, delete all the keys in redis-cli using FLUSHALL , and then insert a new row using the schema table prefix:
SET 'dev:simple_table:{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'
A request from preo cli shows _key_corrupt=true , and the key values are zero, although the values still work.
presto:dev> SELECT * FROM simple_table; id | word | name | number | boolean | _key | _value | _key_length | _value_length | _key_corrupt | _value_corrupt ------+------+--------+--------+---------+-----------------------------------------------+-----------------------------------------------------+-------------+---------------+--------------+---------------- NULL | NULL | bar | 3 | false | dev:simple_table:{"id": 42, "word": "foo"} | {"name": "bar", "number": 3, "boolean": "false"} | 42 | 48 | true | false
I have increased the redis and presto log levels for hints, but nothing that shows me any errors or reasons why the key is damaged. I have a feeling that my syntax for redis keys might be wrong , but I follow the exact method described in the presto redis documentation and even when reading through a redis pre-source that indicates that the "schema:table:*" format is used with this flag "schema:table:*" .
Here's a preliminary log at the debugging level when the query is run:
2016-10-20T17:09:55.048-0700 INFO main com.facebook.presto.server.PrestoServer ======== SERVER STARTED ======== 2016-10-20T17:10:24.785-0700 DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is PLANNING 2016-10-20T17:10:24.802-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Considering files: [/Users/acarson/var/redis/tables/simple_table.json] 2016-10-20T17:10:24.849-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Redis table dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}} 2016-10-20T17:10:24.850-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Loaded table definitions: [dev.simple_table] 2016-10-20T17:10:24.850-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Found Table definition for dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}} 2016-10-20T17:10:25.020-0700 DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is STARTING 2016-10-20T17:10:25.027-0700 DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULING 2016-10-20T17:10:25.064-0700 DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULED 2016-10-20T17:10:25.065-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is RUNNING 2016-10-20T17:10:25.119-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is RUNNING 2016-10-20T17:10:25.165-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULING 2016-10-20T17:10:25.174-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULED 2016-10-20T17:10:25.179-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is RUNNING 2016-10-20T17:10:25.278-0700 INFO 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisJedisManager Creating new JedisPool for 127.0.0.1:6379 2016-10-20T17:10:25.313-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisRecordCursor Scanning new Redis keys from cursor 0 . 0 values read so far 2016-10-20T17:10:25.326-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisRecordCursor Read a total of 1 values with 48 bytes. 2016-10-20T17:10:25.330-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.execution.TaskExecutor Split 20161021_001024_00000_qx72p.1.0-0 RedisSplit{connectorId=redis, schemaName=dev, tableName=simple_table, keyDataFormat=json, valueDataFormat=json, keyName=null, start=0, end=-1, nodes=[127.0.0.1:6379]} (start = 1477008625258, wall = 72 ms, cpu = 56 ms, calls = 1) is finished 2016-10-20T17:10:25.350-0700 DEBUG http-worker-77 com.facebook.presto.execution.SqlTask Aborting task 20161021_001024_00000_qx72p.1.0 output 0 2016-10-20T17:10:25.352-0700 DEBUG task-notification-1 com.facebook.presto.execution.TaskStateMachine Task 20161021_001024_00000_qx72p.1.0 is FINISHED 2016-10-20T17:10:25.357-0700 DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is FINISHED 2016-10-20T17:10:25.367-0700 DEBUG 20161021_001024_00000_qx72p.0.0-0-59 com.facebook.presto.execution.TaskExecutor Split 20161021_001024_00000_qx72p.0.0-0 (start = 1477008625257, wall = 110 ms, cpu = 9 ms, calls = 4) is finished 2016-10-20T17:10:25.369-0700 DEBUG http-worker-88 com.facebook.presto.execution.SqlTask Aborting task 20161021_001024_00000_qx72p.0.0 output 0 2016-10-20T17:10:25.372-0700 DEBUG task-notification-0 com.facebook.presto.execution.TaskStateMachine Task 20161021_001024_00000_qx72p.0.0 is FINISHED 2016-10-20T17:10:25.379-0700 DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is FINISHED 2016-10-20T17:10:25.380-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHING 2016-10-20T17:10:25.383-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHED 2016-10-20T17:10:25.420-0700 INFO query-execution-2 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20161021_001024_00000_qx72p :: Transaction:[c54dc7fe-8159-434d-b4cc-cb13ad41a5d7] :: elapsed 610ms :: planning 247ms :: scheduling 248ms :: running 0ms :: finishing 363ms :: begin 2016-10-20T17:10:24.773-07:00 :: end 2016-10-20T17:10:25.383-07:00