All geek questions in one place

Export custom delimited HDFS file to Mysql via Sqoop

  • I have a file like this:

    1^%~binod^*~1^%~ritesh^*~1^%~shisir^*~1^%~budhdha^*~1^%~romika^*~1^%~rubeena^*~

  • Where - input fields - completed '^% ~' - input lines - completed '^ * ~'.

  • I tried to export by command:

     sqoop export --connect jdbc:mysql://localhost/sqoop --username root -P --table makalu --export-dir sqoop/makalu --input-fields-terminated-by '^%~' --input-lines-terminated-by '^*~'

  • But he writes one line:

  id | name
 1 |% ~ binod

in MySQL.

  • Now my question is which team could solve this scenario.
+5
source share
2 answers

Arguments - input fields - completed and - I / O lines accept a single character

when you execute the export command in the terminal, you can see the warnings below.

 02/03 18:55:21 WARN sqoop.SqoopOptions: Character argument ^%~ has multiple characters; only the first will be used. 15/02/03 18:55:21 WARN sqoop.SqoopOptions: Character argument ^*~ has multiple characters; only the first will be used.

Update:

One thing I observed is that rows end differently than \ n or a new line, and then only one insert of one record into the database during export. I tried a couple of combinations with the end of the field and the line, but only in the case when the line ends with \ n data, it is exported to the RDBMS table

Even with arguments - input-lines-with completion and - lines with completion , but not luck.

I tested this on

 Cloudera CDH-5.0.0-1 Sqoop version: 1.4.4-cdh5.0.0

The only thing I can find is this .

+4
source

The argument works - input-fields-terminated-by 'any separator in single quotes'. Also, do not use -direct in this case, since then it overrides the delimiter. Obviously, you will need a table created in mysql that defaults to ","

0
source

Source: https://habr.com/ru/post/1212532/

More articles:


All Articles