Spark using PySpark

Question

Hi, I have many images (lower millions) that I need to do for classification. I use Spark and can read all images in the format (filename1, content1), (filename2, content2) ... in a large RDD.

 images = sc.wholeTextFiles("hdfs:///user/myuser/images/image/00*") 

However, I am very confused about what to do with the image representation in Unicode.

Here is an example of a single image / file:

 (u'hdfs://NameService/user/myuser/images/image/00product.jpg', u'\ufffd\ufffd\ufffd\ufffd\x00\x10JFIF\x00\x01\x01\x01\x00`\x00`\x00\x00\ufffd\ufffd\x01\x1eExif\x00\x00II*\x00\x08\x00\x00\x00\x08\x00\x12\x01\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00\x1a\x01\x05\x00\x01\x00\x00\x00n\x00\x00\x00\x1b\x01\x05\x00\x01\x00\x00\x00v\x00\x00\x00(\x01\x03\x00\x01\x00\x00\x00\x02\x00\x00\x001\x01\x02\x00\x0b\x00\x00\x00~\x00\x00\x002\x01\x02\x00\x14\x00\x00\x00\ufffd\x00\x00\x00\x13\x02\x03\x00\x01\x00\x00\x00\x01\x00\x00\x00i\ufffd\x04\x00\x01\x00\x00\x00\ufffd\x00\x00\x00\x00\x00\x00\x00`\x00\x00\x00\x01\x00\x00\x00`\x00\x00\x00\x01\x00\x00\x00GIMP 2.8.2\x00\x002013:07:29 10:41:35\x00\x07\x00\x00\ufffd\x07\x00\x04\x00\x00\x000220\ufffd\ufffd\x02\x00\x04\x00\x00\x00407\x00\x00\ufffd\x07\x00\x04\x00\x00\x000100\x01\ufffd\x03\x00\x01\x00\x00\x00\ufffd\ufffd\x00\x00\x02\ufffd\x04\x00\x01\x00\x00\x00\x04\x04\x00\x00\x03\ufffd\x04\x00\x01\x00\x00\x00X\x01\x00\x00\x05\ufffd\x04\x00\x01\x00\x00\x00\ufffd\x00\x00\x00\x00\x00\x00\x00\x02\x00\x01\x00\x02\x00\x04\x00\x00\x00R98\x00\x02\x00\x07\x00\x04\x00\x00\x000100\x00\x00\x00\x00\ufffd\ufffd\x04_http://ns.adobe.com/xap/1.0/\x00<?xpacket begin=\'\ufeff\' id=\'W5M0MpCehiHzreSzNTczkc9d\'?>\n<x:xmpmeta xmlns:x=\'adobe:ns:meta/\'>\n<rdf:RDF xmlns:rdf=\'http://www.w3.org/1999/02/22-rdf-syntax-ns#\'>\n\n <rdf:Description xmlns:exif=\'http://ns.adobe.com/exif/1.0/\'>\n <exif:Orientation>Top-left</exif:Orientation>\n <exif:XResolution>96</exif:XResolution>\n <exif:YResolution>96</exif:YResolution>\n <exif:ResolutionUnit>Inch</exif:ResolutionUnit>\n <exif:Software>ACD Systems Digital Imaging</exif:Software>\n <exif:DateTime>2013:07:29 10:37:00</exif:DateTime>\n <exif:YCbCrPositioning>Centered</exif:YCbCrPositioning>\n <exif:ExifVersion>Exif Version 2.2</exif:ExifVersion>\n <exif:SubsecTime>407</exif:SubsecTime>\n <exif:FlashPixVersion>FlashPix Version 1.0</exif:FlashPixVersion>\n <exif:ColorSpace>Uncalibrated</exif:ColorSpace>\n 

Looking closer, some characters are actually similar to metadata like

 ... <x:xmpmeta xmlns:x=\'adobe:ns:meta/\'>\n<rdf:RDF xmlns:rdf=\'http://www.w3.org/1999/02/22-rdf-syntax-ns#\'>\n\n <rdf:Description xmlns:exif=\'http://ns.adobe.com/exif/1.0/\'>\n <exif:Orientation>Top-left</exif:Orientation>\n <exif:XResolution>96</exif:XResolution>\n <exif:YResolution>96</exif:YResolution>\n ... 

My previous experience was to use the scipy package and related functions like "imread" ... and the input is usually the file name. Now I'm really lost what this unicode means and what I can do to convert it to a format that I know.

Can someone share with me how I can read these unicode in a scipy image (ndarray)?

Answer 1

Your data looks like raw bytes from a real image file (jpg?). The problem with your data is that it should be bytes, not unicode. You need to figure out how to convert from unicode to bytes. There is a whole can of worms full of coding traps that you have to deal with, but you can get lucky with img.encode('iso-8859-1') . I do not know, and I will not deal with this in my answer.

The raw data for the PNG image is as follows:

 rawdata = '\x89PNG\r\n\x1a\n\x00\x00...\x00\x00IEND\xaeB`\x82' 

Once you get it in bytes, you can create a PIL image from the raw data and read it as nparray:

 >>> from StringIO import StringIO >>> from PIL import Image >>> import numpy as np >>> np.asarray(Image.open(StringIO(rawdata))) array([[[255, 255, 255, 0], [255, 255, 255, 0], [255, 255, 255, 0], ..., [255, 255, 255, 0], [255, 255, 255, 0], [255, 255, 255, 0]]], dtype=uint8) 

All you need to get it working on Spark is SparkContext.binaryFiles :

 >>> images = sc.binaryFiles("path/to/images/") >>> image_to_array = lambda rawdata: np.asarray(Image.open(StringIO(rawdata))) >>> images.values().map(image_to_array)