Geek Answers Handbook

Python xarray.concat, then xarray.to_netcdf generates a huge new file size

So, I have 3 netcdf4 files (each about 90 MB) that I would like to combine using the xarray package. Each file has one variable (dis), presented with a resolution of 0.5 degrees (lat, lon) for 365 days (time). My goal is to combine the three files so that we have time periods of 1095 days (3 years).

Each file (for 2007, 2008, 2009) has: 1 variable: dis 3 coordinates: time, lat, lon ... as such

<xarray.Dataset>
Dimensions:  (lat: 360, lon: 720, time: 365)
Coordinates:
  * lon      (lon) float32 -179.75 -179.25 -178.75 -178.25 -177.75 -177.25    ...
  * lat      (lat) float32 89.75 89.25 88.75 88.25 87.75 87.25 86.75 86.25 ...
  * time     (time) datetime64[ns] 2007-01-01 2007-01-02 2007-01-03 ...
Data variables:
    dis      (time, lat, lon) float64 nan nan nan nan nan nan nan nan nan ...

I import them and use the concat module to concatenate, I think it is successful. In this case, the module reads 3 netcdf file names from filestrF file

flist1 = [1,2,3]
ds_new = xr.concat([xr.open_dataset(filestrF[0,1,1,f]) for f in flist1],dim='time')

Now the new data of the new dataset will be shown:

Dimensions:  (lat: 360, lon: 720, time: 1095)

, . , netcdf, , 1 , -, 700 .

ds_new.to_netcdf('saved_on_disk1.nc')
  • 2 ~ 1,5
  • 3,, 2.2
  • 4,, 2.9

3 x 90 MB = 270 MB - (3x) (). , dis lat lon .

, , ? .

+4
2

netCDF, , , , netCDF4.

, xarray . reset. , -, , .

netCDF , encoding xarray:

ds_new.to_netcdf('saved_on_disk1.nc', encoding={'dis': {'zlib': True}})

, chunksizes .

, , encoding, , xr.open_dataset(filestrF[0,1,1,1]).dis.encoding.

+5

, time , NCO ncrcat, , .

ncrcat file1.nc file2.nc file3.nc -O concat.nc

+2

More articles:


All Articles