Geek Answers Handbook

How to read data only between two lines in python using numpy or pandas?

I have a data file similar to this:

# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.300000  0.01     0.999920
      14.2000     0.301000  0.02     0.999960
      14.2000     0.302000  0.03     0.999980
      14.2000     0.303000  0.04     0.999980
      14.2000     0.304000  0.06     0.999980
      14.2000     0.305000  0.08     0.999970
      14.2000     0.306000  0.2     0.999950
      14.2000     0.307000  0.4     0.999910
      14.2000     0.308000  0.8     0.999860
      14.2000     0.309000  0.9     0.999960
      14.2000     0.310000  0.8     0.999990
      14.2000     0.311000  0.4     0.999980
      14.2000     0.312000  0.2     0.999960
      14.2000     0.313000  0.06     0.999940
      14.2000     0.314000  0.03     0.999930
      14.2000     0.315000  0.02     1.00000
      14.2000     0.316000  0.01     1.00000

Required output file output.csv:

# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.304000  0.06     0.999980
      14.2000     0.305000  0.08     0.999970
      14.2000     0.306000  0.2     0.999950
      14.2000     0.307000  0.4     0.999910
      14.2000     0.308000  0.8     0.999860
      14.2000     0.309000  0.9     0.999960
      14.2000     0.310000  0.8     0.999990
      14.2000     0.311000  0.4     0.999980
      14.2000     0.312000  0.2     0.999960
      14.2000     0.313000  0.06     0.999940
      14.2000     0.314000  0.03     0.999930


      # conditions are: 
      # output first element of column3 >= 0.05   i.e. 0.06
      # output last  element of column3  < 0.05   i.e. 0.03

      # for the second may be we need to get the index of second 0.06 and 
      #     get the value of next index.

How to do it in python pandas or numpy?

My initial attempt:

#!/usr/bin/env python
# -*- coding: utf-8 -*- 
# Author    : Bhishan Poudel 
# Date      : June 16, 2016 


# Imports
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#==============================================================================
# read in a file
infile = 'filter_2.txt'
colnames = ['angle', 'wave','trans', 'refl']
print('{} {} {} {}'.format('\nreading file : ', infile, '','' ))
df = pd.read_csv(infile,sep='\s+', header = None,skiprows = 0,
                 comment='#',names=colnames,usecols=(0,1,2,3))

print(df)

# find value of wavelength just above 0.05
print("\n")
df         = df[(df['trans'] >=  0.05) ]
print(df)

Some related links are as follows:
How to read between two specific lines in python

+4
source share
3 answers

I would skip pandas or numpyether

fo = open('filter_3.txt', 'w')
with open('filter_2.txt', 'r') as fi:
    line = fi.readline()
    while line:
        split = line.split()
        if (split[0] == '#') or (float(split[2]) >= 0.027):
            print line,
            fo.write(line)

        line = fi.readline()

fo.close()

# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.302000  0.028     0.999980
      14.2000     0.303000  0.030     0.999980
      14.2000     0.304000  0.032     0.999980
      14.2000     0.305000  0.030     0.999970
      14.2000     0.306000  0.028     0.999950

New code to include another line

fo = open('filter_3.txt', 'w')
with open('filter_2.txt', 'r') as fi:
    new_line = fi.readline()
    old_line = None
    while new_line:
        split_new = new_line.split()
        if old_line is not None:
            split_old = old_line.split()

        cond0 = False if old_line is None else (split_old[0] == '#')
        cond1 = split_new[0] == '#'
        cond2 = float(split_new[2]) >= 0.05
        cond3 = False if old_line is None else (float(split_old[2]) >= 0.05)

        if (cond1 or cond2) or (cond3 and not cond0):
            print new_line,
            fo.write(new_line)
            printed_old = True

        old_line = new_line
        new_line = fi.readline()

fo.close()

# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
      14.2000     0.304000  0.06     0.999980
      14.2000     0.305000  0.08     0.999970
      14.2000     0.306000  0.2     0.999950
      14.2000     0.307000  0.4     0.999910
      14.2000     0.308000  0.8     0.999860
      14.2000     0.309000  0.9     0.999960
      14.2000     0.310000  0.8     0.999990
      14.2000     0.311000  0.4     0.999980
      14.2000     0.312000  0.2     0.999960
      14.2000     0.313000  0.06     0.999940
      14.2000     0.314000  0.03     0.999930
+4
source

IIUC, you can do it like this:

In [51]: df[df.loc[df.trans >= 0.05, 'trans'].index.min() : df.loc[df.trans >= 0.05, 'trans'].index.max() + 1]
Out[51]:
    angle   wave  trans     refl
4    14.2  0.304   0.06  0.99998
5    14.2  0.305   0.08  0.99997
6    14.2  0.306   0.20  0.99995
7    14.2  0.307   0.40  0.99991
8    14.2  0.308   0.80  0.99986
9    14.2  0.309   0.90  0.99996
10   14.2  0.310   0.80  0.99999
11   14.2  0.311   0.40  0.99998
12   14.2  0.312   0.20  0.99996
13   14.2  0.313   0.06  0.99994

:, Pandas 0.20.1 . index indexer , . iloc .loc.

+4

If your goal is to preserve the appearance of the recorded file (i.e., line spacing is the same), then you probably need to keep the original contents of the file.

from io import StringIO
contents = open(infile).read()

df = pd.read_csv(StringIO(contents), sep='\s+', header = None,skiprows = 0,
                 comment='#',names=colnames,usecols=(0,1,2,3))
allowed_indices = df.query('trans >= 0.027').index.values

content_lines = np.array(contents.split('\n'))
num_comments =  len([l for l in contents_lines if l.startswith('#')])
comment_and_allowed_indices = np.append(np.array(range(num_comments)),
                                        allowed_indices + num_comments)

Then you just need to write the original content to a file. They can be indexed via:

content_lines[comment_and_allowed_indices]
+3
source

More articles:


All Articles