Geek Answers Handbook

Error from memory while reading csv file in piece

I am processing a 2.5 GB csv file. Table 2.5 GB looks like this:

 columns=[ka,kb_1,kb_2,timeofEvent,timeInterval] 0:'3M' '2345' '2345' '2014-10-5',3000 1:'3M' '2958' '2152' '2015-3-22',5000 2:'GE' '2183' '2183' '2012-12-31',515 3:'3M' '2958' '2958' '2015-3-10',395 4:'GE' '2183' '2285' '2015-4-19',1925 5:'GE' '2598' '2598' '2015-3-17',1915

And I want to group ka and kb_1 to get the result like this:

 columns=[ka,kb,errorNum,errorRate,totalNum of records] '3M','2345',0,0%,1 '3M','2958',1,50%,2 'GE','2183',1,50%,2 'GE','2598',0,0%,1

(error definition Record: when kb_1 != kb_2 , the corresponding record is considered as an abnormal record)

My computer, which is ubuntu 12.04, has 16 GB of memory and free -m returns

  total used free shared buffers cached Mem: 112809 14476 98333 0 128 10823 -/+ buffers/cache: 3524 109285 Swap: 0 0 0

My python file is called bigData.py

 import pandas as pd import numpy as np import sys,traceback,os cksize=98333 # or 1024, either chunk size didn't work at all try: dfs = pd.DataFrame() reader=pd.read_table('data/petaJoined.csv', chunksize=cksize) for chunk in reader:#when executed this line,error occur! pass #temp=tb_createTopRankTable(chunk) #dfs.append(temp) #df=tb_createTopRankTable(dfs) except: traceback.print_exc(file=sys.stdout) 
 ipdb> pd.__version__ '0.16.0'

I use the following command to track memory usage:

 top ps -C python -o %cpu,%mem,cmd

Since it takes about 2 seconds to crash, I see that mem usage reached 90% for some time, and CPU reached 100%

When I excecute python bigData.py , the following error occurs:

 /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path from pkg_resources import resource_stream /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path from pkg_resources import resource_stream Traceback (most recent call last): File "bigData.py", line 10, in <module> for chunk in reader: File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__ yield self.read(self.chunksize) File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read ret = self._engine.read(nrows) File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read data = self._reader.read(nrows) File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411) File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819) File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268) File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142) File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728) CParserError: Error tokenizing data. C error: out of memory Segmentation fault (core dumped)

or

  /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path from pkg_resources import resource_stream /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path from pkg_resources import resource_stream Traceback (most recent call last): File "bigData.py", line 10, in <module> for chunk in reader: File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__ yield self.read(self.chunksize) File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read ret = self._engine.read(nrows) File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read data = self._reader.read(nrows) File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411) File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819) File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268) File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142) File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728) CParserError: Error tokenizing data. C error: out of memory *** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e *** ====== Backtrace: ======== /lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039] /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e] .... python(PyDict_SetItem+0x49)[0x577749] python(_PyModule_Clear+0x149)[0x4cafb9] python(PyImport_Cleanup+0x477)[0x4cb4f7] python(Py_Finalize+0x18e)[0x549f0e] python(Py_Main+0x3bc)[0x56b56c] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d] python[0x41bb11] ======= Memory map: ======== 00400000-00670000 r-xp 00000000 08:01 26612 /usr/bin/python2.7 0086f000-00870000 r--p 0026f000 08:01 26612 /usr/b....... 008d9000-008eb000 rw-p 00000000 00:00 0 01ddb000-036f7000 rw-p 00000000 00:00 0 [heap] 7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0 7f7504000000-7f7504021000 rw-p 00000000 00:00 0 7f7504021000-7f7508000000 ---p 00000000 00:00 0 7f750bf83000-7f750c285000 rw-p 00000000 00:00 0 7f750c285000-7f750c586000 rw-p 00000000 00:00 0 7f750c586000-7f750c707000 rw-p 00000000 00:00 0 7f750c707000-7f750c711000 r-xp 00000000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c711000-7f750c911000 ---p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c911000-7f750c912000 r--p 0000a000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205 /usr/local/lib/python2.7/dist-packages/pandas/_testing.so 7f750c913000-7f750c914000 rw-p 00000000 00:00 0 7f750c914000-7f750c918000 r-xp 00000000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750c918000-7f750cb17000 ---p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331 /lib/x86_64-linux-gnu/libuuid.so.1.3.0 7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071 /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so 7f750cd38000-7f750d039000 rw-p 00000000 00:00 0 7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070 /usr/local/lib/python2.7/dist-packages/pandas/json.so 7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270 /usr/local/lib/python2.7/dist-packages/pandas/parser.so 7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d591000-7f750d790000 ---p 000e2000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d790000-7f750d798000 r--p 000e1000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0 7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477 /usr/lib/pyshared/python2.7/matplotlib/_path.so 7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269 /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so 7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447 /usr/local/lib/python2.7/dist-packages/pandas/_period.so 7f750de86000-7f750de87000 rw-p 00000000 00:00 0 7f750de87000-7f750debc000 r-xp 00000000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203 /usr/local/lib/python2.7/dist-packages/pandas/index.so 7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e295000-7f750e494000 ---p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e494000-7f750e495000 r--p 001d4000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278 /usr/local/lib/python2.7/dist-packages/pandas/algos.so 7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0 7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831 /usr/lib/python2.7/lib-dynload/_csv.so 7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e782000-7f750e981000 ---p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e981000-7f750e982000 r--p 000cd000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449 /usr/local/lib/python2.7/dist-packages/pandas/lib.so 7f750e990000-7f750e992000 rw-p 00000000 00:00 0 7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271 /usr/local/lib/python2.7/dist-packages/pandas/tslib.so 7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0 7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837 /usr/lib/python2.7/lib-dynload/_ctypes.so 7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0 7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f123000-7f750f124000 r--p 00059000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046 /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so 7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0 7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f166000-7f750f365000 ---p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f365000-7f750f366000 r--p 00008000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f366000-7f750f367000 rw-p 00009000 08:01 532085 /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so 7f750f367000-7f750f368000 r-xp 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f368000-7f750f567000 ---p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f567000-7f750f568000 r--p 00000000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f568000-7f750f569000 rw-p 00001000 08:01 48818 /usr/lib/python2.7/lib-dynload/future_builtins.so 7f750f569000-7f750f588000 r-xp 00000000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f588000-7f750f787000 ---p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f787000-7f750f788000 r--p 0001e000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815 /usr/lib/python2.7/lib-dynload/_io.so 7f750f791000-7f750f907000 r-xp 00000000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750f907000-7f750fb06000 ---p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132 /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so 7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0 7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128 /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so 7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0 7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067 /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so 7f7510265000-7f751028f000 r-xp 00000000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751028f000-7f751048e000 ---p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751048e000-7f751048f000 r--p 00029000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108 /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so 7f7510491000-7f75104d2000 rw-p 00000000 00:00 0 7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833 /usr/lib/python2.7/lib-dynload/_heapq.so 7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751073e000-7f751093d000 ---p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751093d000-7f751093e000 r--p 00066000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f751093e000-7f7510942000 rw-p 00067000 08:01 532118 /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so 7f7510942000-7f7510944000 rw-p 00000000 00:00 0 7f7510944000-7f7510958000 r-xp 00000000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510958000-7f7510b57000 ---p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804 /usr/lib/python2.7/lib-dynload/datetime.so 7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106 /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so 7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0 7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7510f08000-7f7511107000 ---p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7511107000-7f7511108000 r--p 00038000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f7511108000-7f751110c000 rw-p 00039000 08:01 533450 /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so 7f751110c000-7f751110d000 rw-p 00000000 00:00 0 7f751110d000-7f7511296000 r--p 00000000 08:01 58562 /usr/lib/locale/locale-archive 7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511660000-7f751185f000 ---p 001b4000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f751185f000-7f7511863000 r--p 001b3000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327 /lib/x86_64-linux-gnu/libc-2.15.so 7f7511865000-7f751186a000 rw-p 00000000 00:00 0 7f751186a000-7f7511965000 r-xp 00000000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400 /lib/x86_64-linux-gnu/libm-2.15.so 7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f751212e000-7f7512149000 r--p 001b1000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f7512154000-7f7512158000 rw-p 00000000 00:00 0 7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123ac000-7f75123af000 r--p 00054000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283 /lib/x86_64-linux-gnu/libutil-2.15.so 7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406 /lib/x86_64-linux-gnu/ld-2.15.so 7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0 7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0 7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0 7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0 7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0 7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so 7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260 /lib/x86_64-linux-gnu/ld-2.15.so 7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0 [stack] 7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted (core dumped)

with the code below, there is no memory problem, but what can the code do below, I mean performing grouping and data aggregation

 with open("data/petaJoined.csv", "r") as content: for line in content: #print line pass #do stuff with line` content.close()

Does anyone know what is going on?

Actually I want to achieve the result shown in Pandas read csv from memory

Maybe there will be a solution?

Note. I already use csv reading in a piece, but there is still a memory error.

Then I resized the chunk so that another bigData.py file is

 import pandas as pd import numpy as np import sys, traceback, os import etl2 # my self processing flow reload(etl2) def iter_chunks(n,df): while True: try: yield df.get_chunk(n) except StopIteration: break cksize=5 try: dfs = pd.DataFrame() reader=pd.read_table( 'data/petaJoined.csv', chunksize = cksize, low_memory = False, iterator = True ) # choose as appropriate for chunk in iter_chunks(cksize,reader): temp=etl2.tb_createTopRankTable(chunk) dfs.append(temp) df=tb_createTopRankTable(dfs) # # for chunk in reader: # pass # temp=tb_createTopRankTable(chunk) # dfs.append(temp) # df=tb_createTopRankTable(dfs) except: traceback.print_exc(file=sys.stdout)

However, after execution for some time there will be a segmentation error

 def tb_createTopRankTable(df): try: key='name1' key2='name2' df2 = df.groupby([key,key2])['isError'].agg({ 'errorNum': 'sum','totalParcel': 'count' }) df2['errorRate'] = df2['errorNum'] / df2['totalParcel'] return df2
+7
python chunks out-of-memory csv mapreduce
source share
2 answers

Based on your snippet when reading line by line.

I assume kb_2 is an indicator of error,

 groups={} with open("data/petaJoined.csv", "r") as large_file: for line in large_file: arr=line.split('\t') #assuming this structure: ka,kb_1,kb_2,timeofEvent,timeInterval k=arr[0]+','+arr[1] if not (k in groups.keys()) groups[k]={'record_count':0, 'error_sum': 0} groups[k]['record_count']=groups[k]['record_count']+1 groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2]) for k,v in groups.items: print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))

This piece of code saves all the groups in the dictionary and calculates the error rate after reading the entire file.

He will meet an exception in his memory if there are too many combinations of groups.

+2
source share

Q: Does anyone know what is going on?

A: Yes. The sum of all the service data data memory for RAM objects! <RAM

This is a natural part of any formal abstraction for adding extra overhead if some additional functions need to be implemented at a higher (more abstract) level. This means that the more abstract / more fully functional representation of any data set that has been selected, the more memory and processing overhead would be expected.

 ITEMasINT = 32345 ITEMasTUPLE = ( 32345, ) ITEMasLIST = [ 32345, ] ITEMasARRAY = np.array( [ 32345, ] ) ITEMasDICT = { 0: 32345, } ######## .__sizeof__() -> int\nsize of object in memory, in bytes' ITEMasINT.__sizeof__() -> 12 #_____ 100% _ trivial INT ITEMasTUPLE.__sizeof__() -> 16 # 133% _ en-tuple-d ITEMasLIST.__sizeof__() -> 24 # 200% _ list-ed ITEMasARRAY.__sizeof__() -> 40 # 333% _ numpy-wrapped ITEMasDICT.__sizeof__() -> 124 # 1033% _ hash-associated asDict

If personal experience is not enough, check the "costs" of repackaging the input (not already small) data into the pandas service data:

 CParserError: Error tokenizing data. C error: out of memory Segmentation fault (core dumped)

and

 CParserError: Error tokenizing data. C error: out of memory *** glibc detected *** python: free(): ... ... .. . Aborted (core dumped)

Q: Perhaps there will be a solution?

A: Yes.

Just follow the computing strategy and deploy efficient and fast processing of csv input data (it is still fileIO with 8-15 ms access time and low flow-flow data, even if you use SSD devices with a maximum data transfer rate of 960 MB / s, your blocking fact is the limit of memory allocation ... so rather be patient with the input stream and do not crash into the main memory barrier for any super-object in RAM (which would be introduced to be finally asked and (if it did not crash during its creation ...) to calculate a simple sum/nROWs ).

Step-by-step or block readings allow calculating results on the fly and using a case-based strategy (asDict and similar for intermediate storage of results) for calculating a sliding window as fast and with memory support . ( Uri provided an example for this)

This basic approach is used for use both in systems with a limited lifetime and for systems on a chip that have been used to process large data streams for more than the last half century, so there was nothing new in the sun.

If the size of the results cannot be placed in RAM, then it makes no sense to even start processing any input file, right?

BigData processing is neither about super-scaling COTS-dataObjects objects and about finding the best or sexiest "single-line" ...

BigData requires a great understanding of how to handle both fast and smart in order to avoid excessive costs even for small overheads that forgive you to make major mistakes on only a few GB small-bigData files, but will kill any budget and effort by trying the same most on a larger site.

+1
source share

More articles:


All Articles