Numpy / pandas: How to convert a sequence of rows of zeros and ones to a matrix

Question

I have data that comes in this format:

[ (1, "000010101001010101011101010101110101", "aaa", ... ), (0, "111101010100101010101110101010111010", "bb", ... ), (0, "100010110100010101001010101011101010", "ccc", ... ), (1, "000010101001010101011101010101110101", "ddd", ... ), (1, "110100010101001010101011101010111101", "eeee", ... ), ... ] 

In a tuple format, it looks like this:

 (Y, X, other_info, ... ) 

At the end of the day I need to train a classifier (e.g. sklearn.linear_model.logistic.LogisticRegression) with Y and X.

What is the easiest way to turn a string of ones and zeros into something like np.array so that I can run it through a classifier? It seems like there should be a simple answer here, but I couldn't even think of it / google.

A few notes:

• I already use numpy / pandas / sklearn, so everything in these libraries is fair game.
• For many, what I'm doing is convenient to have other_info columns together in a DataFrame
• The rows are quite long (~ 20,000 columns), but the overall data frame is not very high (~ 500 rows).

Answer 1

Since you asked primarily for a way to convert a string of them and zeros into a numpy array, I offer my solution as follows:

 d = '0101010000' * 2000 # create a 20,000 long string of 1s and 0s d_array = np.fromstring(d, 'int8') - 48 # 48 is ascii 0. ascii 1 is 49 

This compares to @DSM in terms of speed:

 In [21]: timeit numpy.fromstring(d, dtype='int8') - 48 10000 loops, best of 3: 35.8 us per loop In [22]: timeit numpy.fromiter(d, dtype='int', count=20000) 100 loops, best of 3: 8.57 ms per loop 

Answer 2

How about something like this:

Make a data frame:

 In [82]: v = [ ....: (1, "000010101001010101011101010101110101", "aaa"), ....: (0, "111101010100101010101110101010111010", "bb"), ....: (0, "100010110100010101001010101011101010", "ccc"), ....: (1, "000010101001010101011101010101110101", "ddd"), ....: (1, "110100010101001010101011101010111101", "eeee"), ....: ] In [83]: In [83]: df = pandas.DataFrame(v) 

We can use fromiter or array to get ndarray :

 In [84]: d ="000010101001010101011101010101110101" In [85]: np.fromiter(d, int) # better: np.fromiter(d, int, count=len(d)) Out[85]: array([0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1]) In [86]: np.array(list(d), int) Out[86]: array([0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1]) 

It might be a slippery vector way of doing this, but I would just apply the obvious input function to the values ​​and continue my day:

 In [87]: df[1] Out[87]: 0 000010101001010101011101010101110101 1 111101010100101010101110101010111010 2 100010110100010101001010101011101010 3 000010101001010101011101010101110101 4 110100010101001010101011101010111101 Name: 1 In [88]: df[1] = df[1].apply(lambda x: np.fromiter(x, int)) # better with count=len(x) In [89]: df Out[89]: 0 1 2 0 1 [0 0 0 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 aaa 1 0 [1 1 1 1 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 1 1 0 bb 2 0 [1 0 0 0 1 0 1 1 0 1 0 0 0 1 0 1 0 1 0 0 1 0 1 0 ccc 3 1 [0 0 0 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 1 1 0 1 ddd 4 1 [1 1 0 1 0 0 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0 1 1 eeee In [90]: df[1][0] Out[90]: array([0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1])