Reading data with unusual characters

Question

I'm having problems with reading in a file that contains an unusual character, in this case, the arrow symbol: . We tried to specify the input file formats, for example:

> scan('SMKA121212' , what="", sep="\n", blank.lines.skip=T, fileEncoding="UTF-8")
Read 13 items
Warning message:
In scan(file, what, nmax, sep, dec, quote, skip, nlines, na.strings,  :
  invalid input found on input connection 'SMKA121212'

> scan('SMKA121212', what="", sep="\n", blank.lines.skip=FALSE, encoding="UTF-8")
Read 1724 items

(in fact, there are more than 10 thousand lines, and reading is interrupted by an arrow symbol)

I am a bit unclear about the difference between encodingand fileEncodingin terms of how R responds to characters that it does not expect. An explanation may be helpful.

I am grateful for any advice on how to get R to read the full documents and, possibly, just ignore characters that do not match the system.

Answer 1

, , - "|" , "\n", 1724 - :

kibbled [ìGrtzeî or ìgruttenî], pearl...     

, , , Grtze grutten, , , .

Mac

read.table("~/Downloads/lines/1720-1730.txt", sep="|")

:

[\x93Gr\032tze\x94 or \x93grutten\x94]

, , , \032. , , "" R. - ?Quotes, , 32 26 . , , , :

x <- read.table("yourpath/filename.txt", sep="|", stringsAsFactors=FALSE, allowEscapes = TRUE)

, "latin1", "UTF-8", "UTF-16", , Windows, .

, , ( "#" ). : quote="", comment.char="". , :

 table(count.fields("yourpath/filename.txt", sep="|", stringsAsFactors=FALSE, 
         allowEscapes = TRUE, quote="", comment.char=""))

, , :

 which(count.fields("yourpath/filename.txt", sep="|", stringsAsFactors=FALSE, 
         allowEscapes = TRUE, quote="", comment.char="") == 28)

. sessionInfo()

, , "CP1252", "Latin2" ( ISO-8859-2), , , :

 iconvlist()  # 419 encodings

, , ?

ZIP , "" zip , , reutl count.fields:

table( count.fields("~/Downloads/SMKA12_2012archive/SMKA121212", quote="", 
      sep="|",comment.char="") )
#------------
   15    27    28 
    1 10228     1 
which( count.fields("~/Downloads/SMKA12_2012archive/SMKA121212", quote="", sep="|",comment.char="") ==15)
#[1] 1
which( count.fields("~/Downloads/SMKA12_2012archive/SMKA121212", quote="", sep="|",comment.char="") ==28)
#[1] 10230

Mac R 3.0.1 TextEdit.app. , -, , , :

000000000 ||||||||||||||||||||||||||| HMCUSTOMS CONTROL DATA | 2012 | 12

, , . 999999999 | | | | | | | | | | | | | | | | | | | | | | | | | | | 0010228

, skip = 1 fill = TRUE .

dat <- read.table("~/Downloads/SMKA12_2012archive/SMKA121212", quote="", sep="|",comment.char="", fill=TRUE, skip=1 , colClasses=c( rep("integer", 2), rep("character", 4), rep("integer", 24-7+1), rep("character", 3)))
> str(dat)
'data.frame':   10230 obs. of  27 variables:
 $ V1 : int  10110100 10110900 10121000 10129100 10129900 10130000 10190000 10190110 10190190 10190300 ...
 $ V2 : int  0 0 0 0 0 0 0 0 0 0 ...
 $ V3 : chr  "00/00" "00/00" "01/12" "01/12" ...
 $ V4 : chr  "12/11" "12/11" "00/00" "00/00" ...
 $ V5 : chr  "00/00" "00/00" "01/12" "01/12" ...
 $ V6 : chr  "12/11" "12/11" "00/00" "00/00" ...
 $ V7 : int  0 0 0 0 0 0 0 0 0 0 ...
 $ V8 : int  150 150 150 150 150 150 150 150 150 150 ...
 $ V9 : int  2 2 2 2 2 2 2 2 2 2 ...
 $ V10: int  13 13 13 13 13 13 13 13 13 13 ...
 $ V11: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V12: int  200 200 200 200 200 200 200 200 200 200 ...
 $ V13: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V14: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V15: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V16: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V17: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V18: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V19: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V20: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V21: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V22: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V23: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V24: int  0 0 0 0 0 0 0 0 0 0 ...
 $ V25: chr  "KG " "KG " "KG " "KG " ...
 $ V26: chr  "NO " "NO " "NO " "NO " ...
 $ V27: chr  "Pure-bred breeding horses                                                                                                      "| __truncated__ "Pure-bred breeding asses                                                                                                       "| __truncated__ "Pure-bred breeding horses                                                                                                      "| __truncated__ "Horses for slaughter                                                                                                           "| __truncated__ ...

, :

Encoding (readLines("~/Downloads/SMKA12_2012archive/SMKA121212", n=1))
#[1] "unknown"