Commit 55c873625b71e11d93137f823078ee6c4a096ba8 - netcdf4-python

New upstream version 1.3.1 Bas Couwenberg 6 years ago

18 changed file(s) with 601 addition(s) and 127 deletion(s). Raw diff Collapse all Expand all

+25

-2

.travis.yml less more

10	10
11	11	env:
12	12	global:
13		- DEPENDS="numpy cython setuptools==18.0.1"
	13	- DEPENDS="numpy>=1.9.0 cython>=0.21 setuptools>=18.0"
14	14	- NO_NET=1
	15	- MPI=0
15	16
16	17	python:
17	18	- "2.7"

25	26	# Absolute minimum dependencies.
26	27	- python: 2.7
27	28	env:
28		- DEPENDS="numpy==1.9.0 cython==0.19 ordereddict==1.1 setuptools==18.0"
	29	- DEPENDS="numpy==1.9.0 cython==0.21 ordereddict==1.1 setuptools==18.0"
	30	# test MPI
	31	- python: 2.7
	32	env:
	33	- MPI=1
	34	- CC=mpicc
	35	- DEPENDS="numpy>=1.9.0 cython>=0.21 setuptools>=18.0 mpi4py>=1.3.1"
	36	- NETCDF_VERSION=4.4.1.1
	37	- NETCDF_DIR=$HOME
	38	- PATH=${NETCDF_DIR}/bin:${PATH} # pick up nc-config here
	39	addons:
	40	apt:
	41	packages:
	42	- openmpi-bin
	43	- libopenmpi-dev
	44	- libhdf5-openmpi-dev
29	45
30	46	notifications:
31	47	email: false

34	50	- pip install $DEPENDS
35	51
36	52	install:
	53	- if [ $MPI -eq 1 ] ; then ci/travis/build-parallel-netcdf.sh; fi
37	54	- python setup.py build
38	55	- python setup.py install
39	56
40	57	script:
	58	- \|
	59	if [ $MPI -eq 1 ] ; then
	60	cd examples
	61	mpirun -np 4 python mpi_example.py
	62	cd ..
	63	fi
41	64	- cd test
42	65	- python run_all.py

+15

-0

Changelog less more

	0	version 1.3.1 (tag v1.3.1rel)
	1	=============================
	2	* add parallel IO capabilities. netcdf-c and hdf5 must be compiled with MPI
	3	support, and mpi4py must be installed. To open a file for parallel access,
	4	use `parallel=True` in `Dataset.__init__` and optionally pass the mpi4py Comm instance
	5	using the `comm` kwarg and the mpi4py Info instance using the `info` kwarg.
	6	IO can be toggled between collective and independent using `Variable.set_collective`.
	7	See `examples/mpi_example.py`. Issue #717, pull request #716.
	8	Minimum cython dependency bumped from 0.19 to 0.21.
	9	* Add optional `MFTime` calendar overload to use across all files, for example,
	10	`'standard'` or `'gregorian'`. If `None` (the default), check that the calendar
	11	attribute is present on each variable and values are unique across files raising
	12	a `ValueError` otherwise.
	13	* Allow _FillValue to be set for vlen string variables (issue #730).
	14
0	15	version 1.3.0 (tag v1.3.0rel)
1	16	==============================
2	17	* always search for HDF5 headers when building, even when nc-config is used

-1

PKG-INFO less more

0	0	Metadata-Version: 1.1
1	1	Name: netCDF4
2		Version: 1.3.0
	2	Version: 1.3.1
3	3	Author: Jeff Whitaker
4	4	Author-email: jeffrey s whitaker at noaa gov
5	5	Home-page: https://github.com/Unidata/netcdf4-python

-0

README.md less more

6	6
7	7	## News
8	8	For details on the latest updates, see the [Changelog](https://github.com/Unidata/netcdf4-python/blob/master/Changelog).
	9
	10	11/01/2017: Version 1.3.1 released. Parallel IO support with MPI!
	11	Requires that netcdf-c and hdf5 be built with MPI support, and [mpi4py](http://mpi4py.readthedocs.io/en/stable).
	12	To open a file for parallel access in a program running in an MPI environment
	13	using mpi4py, just use `parallel=True` when creating
	14	the `Dataset` instance. See [`examples/mpi_example.py`](https://github.com/Unidata/netcdf4-python/blob/master/examples/mpi_example.py)
	15	for a demonstration. For more info, see the tutorial [section](http://unidata.github.io/netcdf4-python/#section13).
9	16
10	17	9/25/2017: Version [1.3.0](https://pypi.python.org/pypi/netCDF4/1.3.0) released. Bug fixes
11	18	for `netcdftime` and optimizations for reading strided slices. `encoding` kwarg added to

+13

-14

appveyor.yml less more

33	33
34	34	# Add path, activate `conda` and update conda.
35	35	- cmd: set "PATH=%CONDA_INSTALL_LOCN%\\Scripts;%CONDA_INSTALL_LOCN%\\Library\\bin;%PATH%"
36		- cmd: conda update --yes --quiet conda
	36	- cmd: set PYTHONUNBUFFERED=1
37	37	- cmd: call %CONDA_INSTALL_LOCN%\Scripts\activate.bat
38
39		- cmd: set PYTHONUNBUFFERED=1
40
41		# Ensure defaults and conda-forge channels are present.
42		- cmd: conda config --set show_channel_urls true
43		- cmd: conda config --remove channels defaults
44		- cmd: conda config --add channels defaults
45		- cmd: conda config --add channels conda-forge
46
47		# Conda build tools.
48		- cmd: conda install -n root --quiet --yes obvious-ci
49		- cmd: obvci_install_conda_build_tools.py
50		- cmd: conda info
	38	# for obvci_appveyor_python_build_env.cmd
	39	- cmd: conda update --all --yes
	40	- cmd: conda install anaconda-client=1.6.3 --yes
	41	- cmd: conda install -c conda-forge --yes obvious-ci
	42	# for msinttypes and newer stuff
	43	- cmd: conda config --prepend channels conda-forge
	44	- cmd: conda config --set show_channel_urls yes
	45	- cmd: conda config --set always_yes true
	46	# For building conda packages
	47	- cmd: conda install --yes conda-build jinja2 anaconda-client
	48	# this is now the downloaded conda...
	49	- cmd: conda info -a
51	50
52	51	# Skip .NET project specific build phase.
53	52	build: off

+13

-0

ci/travis/build-parallel-netcdf.sh less more

	0	#!/bin/bash
	1
	2	set -e
	3
	4	echo "Using downloaded netCDF version ${NETCDF_VERSION} with parallel capabilities enabled"
	5	pushd /tmp
	6	wget ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-${NETCDF_VERSION}.tar.gz
	7	tar -xzvf netcdf-${NETCDF_VERSION}.tar.gz
	8	pushd netcdf-${NETCDF_VERSION}
	9	./configure --prefix $NETCDF_DIR --enable-netcdf-4 --enable-shared --disable-dap --enable-parallel
	10	make -j 2
	11	make install
	12	popd

+105

-15

docs/netCDF4/index.html less more

3	3	<meta name="viewport" content="width=device-width, initial-scale=1, minimum-scale=1" />
4	4
5	5	<title>netCDF4 API documentation</title>
6		<meta name="description" content="Version 1.3.0
	6	<meta name="description" content="Version 1.3.1
7	7	-------------
8	8	- - -
9	9

1245	1245	<li class="mono"><a href="#netCDF4.Variable.set_auto_mask">set_auto_mask</a></li>
1246	1246	<li class="mono"><a href="#netCDF4.Variable.set_auto_maskandscale">set_auto_maskandscale</a></li>
1247	1247	<li class="mono"><a href="#netCDF4.Variable.set_auto_scale">set_auto_scale</a></li>
	1248	<li class="mono"><a href="#netCDF4.Variable.set_collective">set_collective</a></li>
1248	1249	<li class="mono"><a href="#netCDF4.Variable.set_var_chunk_cache">set_var_chunk_cache</a></li>
1249	1250	<li class="mono"><a href="#netCDF4.Variable.setncattr">setncattr</a></li>
1250	1251	<li class="mono"><a href="#netCDF4.Variable.setncattr_string">setncattr_string</a></li>

1268	1269
1269	1270	<header id="section-intro">
1270	1271	<h1 class="title"><span class="name">netCDF4</span> module</h1>
1271		<h2>Version 1.3.0</h2>
	1272	<h2>Version 1.3.1</h2>
1272	1273	<hr />
1273	1274	<h1>Introduction</h1>
1274	1275	<p>netcdf4-python is a Python interface to the netCDF C library. </p>

1298	1299	<ul>
1299	1300	<li>Python 2.7 or later (python 3 works too).</li>
1300	1301	<li><a href="http://numpy.scipy.org">numpy array module</a>, version 1.9.0 or later.</li>
1301		<li><a href="http://cython.org">Cython</a>, version 0.19 or later.</li>
	1302	<li><a href="http://cython.org">Cython</a>, version 0.21 or later.</li>
1302	1303	<li><a href="https://pypi.python.org/pypi/setuptools">setuptools</a>, version 18.0 or
1303	1304	later.</li>
1304	1305	<li>The HDF5 C library version 1.8.4-patch1 or higher (1.8.x recommended)

1320	1321	If you want <a href="http://opendap.org">OPeNDAP</a> support, add <code>--enable-dap</code>.
1321	1322	If you want HDF4 SD support, add <code>--enable-hdf4</code> and add
1322	1323	the location of the HDF4 headers and library to <code>$CPPFLAGS</code> and <code>$LDFLAGS</code>.</li>
	1324	<li>for MPI parallel IO support, MPI-enabled versions of the HDF5 and netcdf
	1325	libraries are required, as is the <a href="http://mpi4py.scipy.org">mpi4py</a> python
	1326	module.</li>
1323	1327	</ul>
1324	1328	<h1>Install</h1>
1325	1329	<ul>

1336	1340	<li>run <code>python setup.py build</code>, then <code>python setup.py install</code> (as root if
1337	1341	necessary).</li>
1338	1342	<li><a href="https://pip.pypa.io/en/latest/reference/pip_install.html"><code>pip install</code></a> can
1339		also be used, with library paths set with environment variables. To make
1340		this work, the <code>USE_SETUPCFG</code> environment variable must be used to tell
1341		setup.py not to use <code>setup.cfg</code>.
1342		For example, <code>USE_SETUPCFG=0 HDF5_INCDIR=/usr/include/hdf5/serial
1343		HDF5_LIBDIR=/usr/lib/x86_64-linux-gnu/hdf5/serial pip install</code> has been
1344		shown to work on an Ubuntu/Debian linux system. Similarly, environment variables
1345		(all capitalized) can be used to set the include and library paths for
1346		<code>hdf5</code>, <code>netCDF4</code>, <code>hdf4</code>, <code>szip</code>, <code>jpeg</code>, <code>curl</code> and <code>zlib</code>. If the
1347		libraries are installed in standard places (e.g. <code>/usr</code> or <code>/usr/local</code>),
1348		the environment variables do not need to be set.</li>
	1343	also be used, with library paths set with environment variables. To make
	1344	this work, the <code>USE_SETUPCFG</code> environment variable must be used to tell
	1345	setup.py not to use <code>setup.cfg</code>.
	1346	For example, <code>USE_SETUPCFG=0 HDF5_INCDIR=/usr/include/hdf5/serial
	1347	HDF5_LIBDIR=/usr/lib/x86_64-linux-gnu/hdf5/serial pip install</code> has been
	1348	shown to work on an Ubuntu/Debian linux system. Similarly, environment variables
	1349	(all capitalized) can be used to set the include and library paths for
	1350	<code>hdf5</code>, <code>netCDF4</code>, <code>hdf4</code>, <code>szip</code>, <code>jpeg</code>, <code>curl</code> and <code>zlib</code>. If the
	1351	libraries are installed in standard places (e.g. <code>/usr</code> or <code>/usr/local</code>),
	1352	the environment variables do not need to be set.</li>
1349	1353	<li>run the tests in the 'test' directory by running <code>python run_all.py</code>.</li>
1350	1354	</ul>
1351	1355	<h1>Tutorial</h1>

1362	1366	<li><a href="#section10">Beyond homogeneous arrays of a fixed type - compound data types.</a></li>
1363	1367	<li><a href="#section11">Variable-length (vlen) data types.</a></li>
1364	1368	<li><a href="#section12">Enum data type.</a></li>
	1369	<li><a href="#section13">Parallel IO.</a></li>
1365	1370	</ol>
1366	1371	<h2><div id='section1'>1) Creating/Opening/Closing a netCDF file.</h2>
1367	1372	<p>To create a netCDF file from python, you simply call the <a href="#netCDF4.Dataset"><code>Dataset</code></a>

2116	2121	</pre></div>
2117	2122
2118	2123
2119		<p>All of the code in this tutorial is available in <code>examples/tutorial.py</code>,
	2124	<h2><div id='section13'>13) Parallel IO.</h2>
	2125	<p>If MPI parallel enabled versions of netcdf and hdf5 are detected, and
	2126	<a href="https://mpi4py.scipy.org">mpi4py</a> is installed, netcdf4-python will
	2127	be built with parallel IO capabilities enabled. To use parallel IO,
	2128	your program must be running in an MPI environment using
	2129	<a href="https://mpi4py.scipy.org">mpi4py</a>.</p>
	2130	<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="kn">from</span> <span class="nn">mpi4py</span> <span class="kn">import</span> <span class="n">MPI</span>
	2131	<span class="o">>>></span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="kn">as</span> <span class="nn">np</span>
	2132	<span class="o">>>></span> <span class="kn">from</span> <span class="nn">netCDF4</span> <span class="kn">import</span> <span class="n">Dataset</span>
	2133	<span class="o">>>></span> <span class="n">rank</span> <span class="o">=</span> <span class="n">MPI</span><span class="o">.</span><span class="n">COMM_WORLD</span><span class="o">.</span><span class="n">rank</span> <span class="c1"># The process ID (integer 0-3 for 4-process run)</span>
	2134	</pre></div>
	2135
	2136
	2137	<p>To run an MPI-based parallel program like this, you must use <code>mpiexec</code> to launch several
	2138	parallel instances of Python (for example, using <code>mpiexec -np 4 python mpi_example.py</code>).
	2139	The parallel features of netcdf4-python are mostly transparent -
	2140	when a new dataset is created or an existing dataset is opened,
	2141	use the <code>parallel</code> keyword to enable parallel access.</p>
	2142	<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">nc</span> <span class="o">=</span> <span class="n">Dataset</span><span class="p">(</span><span class="s1">'parallel_tst.nc'</span><span class="p">,</span><span class="s1">'w'</span><span class="p">,</span><span class="n">parallel</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
	2143	</pre></div>
	2144
	2145
	2146	<p>The optional <code>comm</code> keyword may be used to specify a particular
	2147	MPI communicator (<code>MPI_COMM_WORLD</code> is used by default). Each process (or rank)
	2148	can now write to the file indepedently. In this example the process rank is
	2149	written to a different variable index on each task</p>
	2150	<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">d</span> <span class="o">=</span> <span class="n">nc</span><span class="o">.</span><span class="n">createDimension</span><span class="p">(</span><span class="s1">'dim'</span><span class="p">,</span><span class="mi">4</span><span class="p">)</span>
	2151	<span class="o">>>></span> <span class="n">v</span> <span class="o">=</span> <span class="n">nc</span><span class="o">.</span><span class="n">createVariable</span><span class="p">(</span><span class="s1">'var'</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">int</span><span class="p">,</span> <span class="s1">'dim'</span><span class="p">)</span>
	2152	<span class="o">>>></span> <span class="n">v</span><span class="p">[</span><span class="n">rank</span><span class="p">]</span> <span class="o">=</span> <span class="n">rank</span>
	2153	<span class="o">>>></span> <span class="n">nc</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
	2154
	2155	<span class="o">%</span> <span class="n">ncdump</span> <span class="n">parallel_test</span><span class="o">.</span><span class="n">nc</span>
	2156	<span class="n">netcdf</span> <span class="n">parallel_test</span> <span class="p">{</span>
	2157	<span class="n">dimensions</span><span class="p">:</span>
	2158	<span class="n">dim</span> <span class="o">=</span> <span class="mi">4</span> <span class="p">;</span>
	2159	<span class="n">variables</span><span class="p">:</span>
	2160	<span class="n">int64</span> <span class="n">var</span><span class="p">(</span><span class="n">dim</span><span class="p">)</span> <span class="p">;</span>
	2161	<span class="n">data</span><span class="p">:</span>
	2162
	2163	<span class="n">var</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span> <span class="p">;</span>
	2164	<span class="p">}</span>
	2165	</pre></div>
	2166
	2167
	2168	<p>There are two types of parallel IO, independent (the default) and collective.
	2169	Independent IO means that each process can do IO independently. It should not
	2170	depend on or be affected by other processes. Collective IO is a way of doing
	2171	IO defined in the MPI-IO standard; unlike independent IO, all processes must
	2172	participate in doing IO. To toggle back and forth between
	2173	the two types of IO, use the <a href="#netCDF4.Variable.set_collective"><code>set_collective</code></a>
	2174	<a href="#netCDF4.Variable"><code>Variable</code></a>method. All metadata
	2175	operations (such as creation of groups, types, variables, dimensions, or attributes)
	2176	are collective. There are a couple of important limitatons of parallel IO:</p>
	2177	<ul>
	2178	<li>If a variable has an unlimited dimension, appending data must be done in collective mode.
	2179	If the write is done in independent mode, the operation will fail with a
	2180	a generic "HDF Error".</li>
	2181	<li>You cannot write compressed data in parallel (although
	2182	you can read it).</li>
	2183	<li>You cannot use variable-length (VLEN) data types. </li>
	2184	</ul>
	2185	<p>All of the code in this tutorial is available in <code>examples/tutorial.py</code>, except
	2186	the parallel IO example, which is in <code>examples/mpi_example.py</code>.
2120	2187	Unit tests are in the <code>test</code> directory.</p>
2121	2188	<p><strong>contact</strong>: Jeffrey Whitaker <a href="mailto:jeffrey.s.whitaker@noaa.gov">jeffrey.s.whitaker@noaa.gov</a></p>
2122	2189	<p><strong>copyright</strong>: 2008 by Jeffrey Whitaker.</p>

2699	2766	<p><strong><code>memory</code></strong>: if not <code>None</code>, open file with contents taken from this block of memory.
2700	2767	Must be a sequence of bytes. Note this only works with "r" mode.</p>
2701	2768	<p><strong><code>encoding</code></strong>: encoding used to encode filename string into bytes.
2702		Default is None (<code>sys.getdefaultfileencoding()</code> is used).</p></div>
	2769	Default is None (<code>sys.getdefaultfileencoding()</code> is used).</p>
	2770	<p><strong><code>parallel</code></strong>: open for parallel access using MPI (requires mpi4py and
	2771	parallel-enabled netcdf-c and hdf5 libraries). Default is <code>False</code>. If
	2772	<code>True</code>, <code>comm</code> and <code>info</code> kwargs may also be specified.</p>
	2773	<p><strong><code>comm</code></strong>: MPI_Comm object for parallel access. Default <code>None</code>, which
	2774	means MPI_COMM_WORLD will be used. Ignored if <code>parallel=False</code>.</p>
	2775	<p><strong><code>info</code></strong>: MPI_Info object for parallel access. Default <code>None</code>, which
	2776	means MPI_INFO_NULL will be used. Ignored if <code>parallel=False</code>.</p></div>
2703	2777	<div class="source_cont">
2704	2778	</div>
2705	2779

6296	6370
6297	6371
6298	6372	<div class="item">
	6373	<div class="name def" id="netCDF4.Variable.set_collective">
	6374	<p>def <span class="ident">set_collective</span>(</p><p>self,True_or_False)</p>
	6375	</div>
	6376
	6377
	6378
	6379
	6380	<div class="desc"><p>turn on or off collective parallel IO access. Ignored if file is not
	6381	open for parallel access.</p></div>
	6382	<div class="source_cont">
	6383	</div>
	6384
	6385	</div>
	6386
	6387
	6388	<div class="item">
6299	6389	<div class="name def" id="netCDF4.Variable.set_var_chunk_cache">
6300	6390	<p>def <span class="ident">set_var_chunk_cache</span>(</p><p>self,size=None,nelems=None,preemption=None)</p>
6301	6391	</div>

+35

-0

examples/mpi_example.py less more

	0	# to run: mpirun -np 4 python mpi_example.py
	1	from mpi4py import MPI
	2	import numpy as np
	3	from netCDF4 import Dataset
	4	rank = MPI.COMM_WORLD.rank # The process ID (integer 0-3 for 4-process run)
	5	nc = Dataset('parallel_test.nc', 'w', parallel=True, comm=MPI.COMM_WORLD,
	6	info=MPI.Info())
	7	# below should work also - MPI_COMM_WORLD and MPI_INFO_NULL will be used.
	8	#nc = Dataset('parallel_test.nc', 'w', parallel=True)
	9	d = nc.createDimension('dim',4)
	10	v = nc.createVariable('var', np.int, 'dim')
	11	v[rank] = rank
	12	# switch to collective mode, rewrite the data.
	13	v.set_collective(True)
	14	v[rank] = rank
	15	nc.close()
	16	# reopen the file read-only, check the data
	17	nc = Dataset('parallel_test.nc', parallel=True, comm=MPI.COMM_WORLD,
	18	info=MPI.Info())
	19	assert rank==nc['var'][rank]
	20	nc.close()
	21	# reopen the file in append mode, modify the data on the last rank.
	22	nc = Dataset('parallel_test.nc', 'a',parallel=True, comm=MPI.COMM_WORLD,
	23	info=MPI.Info())
	24	if rank == 3: v[rank] = 2*rank
	25	nc.close()
	26	# reopen the file read-only again, check the data.
	27	# leave out the comm and info kwargs to check that the defaults
	28	# (MPI_COMM_WORLD and MPI_INFO_NULL) work.
	29	nc = Dataset('parallel_test.nc', parallel=True)
	30	if rank == 3:
	31	assert 2*rank==nc['var'][rank]
	32	else:
	33	assert rank==nc['var'][rank]
	34	nc.close()

+14

-0

include/mpi-compat.h less more

	0	/* Author: Lisandro Dalcin */
	1	/* Contact: dalcinl@gmail.com */
	2
	3	#ifndef MPI_COMPAT_H
	4	#define MPI_COMPAT_H
	5
	6	#include <mpi.h>
	7
	8	#if (MPI_VERSION < 3) && !defined(PyMPI_HAVE_MPI_Message)
	9	typedef void *PyMPI_MPI_Message;
	10	#define MPI_Message PyMPI_MPI_Message
	11	#endif
	12
	13	#endif/MPI_COMPAT_H/

+16

-0

include/netCDF4.pxi less more

695	695	cdef extern from "netcdf_mem.h":
696	696	int nc_open_mem(const char path, int mode, size_t size, void memory, int *ncidp)
697	697
	698	IF HAS_NC_PAR:
	699	cdef extern from "mpi-compat.h": pass
	700	cdef extern from "netcdf_par.h":
	701	ctypedef int MPI_Comm
	702	ctypedef int MPI_Info
	703	int nc_create_par(char path, int cmode, MPI_Comm comm, MPI_Info info, int ncidp);
	704	int nc_open_par(char path, int mode, MPI_Comm comm, MPI_Info info, int ncidp);
	705	int nc_var_par_access(int ncid, int varid, int par_access);
	706	cdef enum:
	707	NC_COLLECTIVE
	708	NC_INDEPENDENT
	709	cdef extern from "netcdf.h":
	710	cdef enum:
	711	NC_MPIIO
	712	NC_PNETCDF
	713
698	714	# taken from numpy.pxi in numpy 1.0rc2.
699	715	cdef extern from "numpy/arrayobject.h":
700	716	ctypedef int npy_intp

-1

netCDF4/__init__.py less more

4	4	from ._netCDF4 import __doc__, __pdoc__
5	5	from ._netCDF4 import (__version__, __netcdf4libversion__, __hdf5libversion__,
6	6	__has_rename_grp__, __has_nc_inq_path__,
7		__has_nc_inq_format_extended__, __has_nc_open_mem__)
	7	__has_nc_inq_format_extended__, __has_nc_open_mem__,
	8	__has_cdf5_format__,__has_nc_par__)
8	9	__all__ =\
9	10	['Dataset','Variable','Dimension','Group','MFDataset','MFTime','CompoundType','VLType','date2num','num2date','date2index','stringtochar','chartostring','stringtoarr','getlibversion','EnumType']

+251

-73

netCDF4/_netCDF4.pyx less more

0	0	"""
1		Version 1.3.0
	1	Version 1.3.1
2	2	-------------
3	3	- - -
4	4

37	37
38	38	- Python 2.7 or later (python 3 works too).
39	39	- [numpy array module](http://numpy.scipy.org), version 1.9.0 or later.
40		- [Cython](http://cython.org), version 0.19 or later.
	40	- [Cython](http://cython.org), version 0.21 or later.
41	41	- [setuptools](https://pypi.python.org/pypi/setuptools), version 18.0 or
42	42	later.
43	43	- The HDF5 C library version 1.8.4-patch1 or higher (1.8.x recommended)

59	59	If you want [OPeNDAP](http://opendap.org) support, add `--enable-dap`.
60	60	If you want HDF4 SD support, add `--enable-hdf4` and add
61	61	the location of the HDF4 headers and library to `$CPPFLAGS` and `$LDFLAGS`.
	62	- for MPI parallel IO support, MPI-enabled versions of the HDF5 and netcdf
	63	libraries are required, as is the [mpi4py](http://mpi4py.scipy.org) python
	64	module.
62	65
63	66
64	67	Install

77	80	- run `python setup.py build`, then `python setup.py install` (as root if
78	81	necessary).
79	82	- [`pip install`](https://pip.pypa.io/en/latest/reference/pip_install.html) can
80		also be used, with library paths set with environment variables. To make
81		this work, the `USE_SETUPCFG` environment variable must be used to tell
82		setup.py not to use `setup.cfg`.
83		For example, `USE_SETUPCFG=0 HDF5_INCDIR=/usr/include/hdf5/serial
84		HDF5_LIBDIR=/usr/lib/x86_64-linux-gnu/hdf5/serial pip install` has been
85		shown to work on an Ubuntu/Debian linux system. Similarly, environment variables
86		(all capitalized) can be used to set the include and library paths for
87		`hdf5`, `netCDF4`, `hdf4`, `szip`, `jpeg`, `curl` and `zlib`. If the
88		libraries are installed in standard places (e.g. `/usr` or `/usr/local`),
89		the environment variables do not need to be set.
	83	also be used, with library paths set with environment variables. To make
	84	this work, the `USE_SETUPCFG` environment variable must be used to tell
	85	setup.py not to use `setup.cfg`.
	86	For example, `USE_SETUPCFG=0 HDF5_INCDIR=/usr/include/hdf5/serial
	87	HDF5_LIBDIR=/usr/lib/x86_64-linux-gnu/hdf5/serial pip install` has been
	88	shown to work on an Ubuntu/Debian linux system. Similarly, environment variables
	89	(all capitalized) can be used to set the include and library paths for
	90	`hdf5`, `netCDF4`, `hdf4`, `szip`, `jpeg`, `curl` and `zlib`. If the
	91	libraries are installed in standard places (e.g. `/usr` or `/usr/local`),
	92	the environment variables do not need to be set.
90	93	- run the tests in the 'test' directory by running `python run_all.py`.
91	94
92	95	Tutorial

104	107	10. [Beyond homogeneous arrays of a fixed type - compound data types.](#section10)
105	108	11. [Variable-length (vlen) data types.](#section11)
106	109	12. [Enum data type.](#section12)
	110	13. [Parallel IO.](#section13)
107	111
108	112
109	113	## <div id='section1'>1) Creating/Opening/Closing a netCDF file.

892	896	[0 2 4 -- 1]
893	897	>>> nc.close()
894	898
895		All of the code in this tutorial is available in `examples/tutorial.py`,
	899	## <div id='section13'>13) Parallel IO.
	900
	901	If MPI parallel enabled versions of netcdf and hdf5 are detected, and
	902	[mpi4py](https://mpi4py.scipy.org) is installed, netcdf4-python will
	903	be built with parallel IO capabilities enabled. To use parallel IO,
	904	your program must be running in an MPI environment using
	905	[mpi4py](https://mpi4py.scipy.org).
	906
	907	:::python
	908	>>> from mpi4py import MPI
	909	>>> import numpy as np
	910	>>> from netCDF4 import Dataset
	911	>>> rank = MPI.COMM_WORLD.rank # The process ID (integer 0-3 for 4-process run)
	912
	913	To run an MPI-based parallel program like this, you must use `mpiexec` to launch several
	914	parallel instances of Python (for example, using `mpiexec -np 4 python mpi_example.py`).
	915	The parallel features of netcdf4-python are mostly transparent -
	916	when a new dataset is created or an existing dataset is opened,
	917	use the `parallel` keyword to enable parallel access.
	918
	919	:::python
	920	>>> nc = Dataset('parallel_tst.nc','w',parallel=True)
	921
	922	The optional `comm` keyword may be used to specify a particular
	923	MPI communicator (`MPI_COMM_WORLD` is used by default). Each process (or rank)
	924	can now write to the file indepedently. In this example the process rank is
	925	written to a different variable index on each task
	926
	927	:::python
	928	>>> d = nc.createDimension('dim',4)
	929	>>> v = nc.createVariable('var', np.int, 'dim')
	930	>>> v[rank] = rank
	931	>>> nc.close()
	932
	933	% ncdump parallel_test.nc
	934	netcdf parallel_test {
	935	dimensions:
	936	dim = 4 ;
	937	variables:
	938	int64 var(dim) ;
	939	data:
	940
	941	var = 0, 1, 2, 3 ;
	942	}
	943
	944	There are two types of parallel IO, independent (the default) and collective.
	945	Independent IO means that each process can do IO independently. It should not
	946	depend on or be affected by other processes. Collective IO is a way of doing
	947	IO defined in the MPI-IO standard; unlike independent IO, all processes must
	948	participate in doing IO. To toggle back and forth between
	949	the two types of IO, use the `netCDF4.Variable.set_collective`
	950	`netCDF4.Variable`method. All metadata
	951	operations (such as creation of groups, types, variables, dimensions, or attributes)
	952	are collective. There are a couple of important limitatons of parallel IO:
	953
	954	- If a variable has an unlimited dimension, appending data must be done in collective mode.
	955	If the write is done in independent mode, the operation will fail with a
	956	a generic "HDF Error".
	957	- You cannot write compressed data in parallel (although
	958	you can read it).
	959	- You cannot use variable-length (VLEN) data types.
	960
	961	All of the code in this tutorial is available in `examples/tutorial.py`, except
	962	the parallel IO example, which is in `examples/mpi_example.py`.
896	963	Unit tests are in the `test` directory.
897	964
898	965	contact: Jeffrey Whitaker <jeffrey.s.whitaker@noaa.gov>

935	1002	# python3: zip is already python2's itertools.izip
936	1003	pass
937	1004
938		__version__ = "1.3.0"
	1005	__version__ = "1.3.1"
939	1006
940	1007	# Initialize numpy
941	1008	import posixpath

951	1018	import_array()
952	1019	include "constants.pyx"
953	1020	include "netCDF4.pxi"
	1021	IF HAS_NC_PAR:
	1022	cimport mpi4py.MPI as MPI
	1023	from mpi4py.libmpi cimport MPI_Comm, MPI_Info, MPI_Comm_dup, MPI_Info_dup, \
	1024	MPI_Comm_free, MPI_Info_free, MPI_INFO_NULL,\
	1025	MPI_COMM_WORLD
	1026	ctypedef MPI.Comm Comm
	1027	ctypedef MPI.Info Info
	1028	ELSE:
	1029	ctypedef object Comm
	1030	ctypedef object Info
954	1031
955	1032	# check for required version of netcdf-4 and hdf5.
956	1033

976	1053	__has_rename_grp__ = HAS_RENAME_GRP
977	1054	__has_nc_inq_path__ = HAS_NC_INQ_PATH
978	1055	__has_nc_inq_format_extended__ = HAS_NC_INQ_FORMAT_EXTENDED
979		__has_cdf5__ = HAS_CDF5_FORMAT
	1056	__has_cdf5_format__ = HAS_CDF5_FORMAT
980	1057	__has_nc_open_mem__ = HAS_NC_OPEN_MEM
	1058	__has_nc_par__ = HAS_NC_PAR
981	1059	_needsworkaround_issue485 = __netcdf4libversion__ < "4.4.0" or \
982	1060	(__netcdf4libversion__.startswith("4.4.0") and \
983	1061	"-development" in __netcdf4libversion__)

1549	1627	free(varids) # free pointer holding variable ids.
1550	1628	return variables
1551	1629
1552		cdef _ensure_nc_success(ierr, err_cls=RuntimeError):
	1630	cdef _ensure_nc_success(ierr, err_cls=RuntimeError, filename=None):
1553	1631	# print netcdf error message, raise error.
1554	1632	if ierr != NC_NOERR:
1555		raise err_cls((<char *>nc_strerror(ierr)).decode('ascii'))
	1633	err_str = (<char *>nc_strerror(ierr)).decode('ascii')
	1634	if issubclass(err_cls, EnvironmentError):
	1635	raise err_cls(ierr, err_str, filename)
	1636	else:
	1637	raise err_cls(err_str)
1556	1638
1557	1639	# these are class attributes that
1558	1640	# only exist at the python level (not in the netCDF file).

1690	1772	the parent Dataset or Group."""
1691	1773
1692	1774	def __init__(self, filename, mode='r', clobber=True, format='NETCDF4',
1693		diskless=False, persist=False, keepweakref=False,
1694		memory=None, encoding=None, **kwargs):
	1775	diskless=False, persist=False, keepweakref=False,
	1776	memory=None, encoding=None, parallel=False,
	1777	Comm comm=None, Info info=None, **kwargs):
1695	1778	"""
1696	1779	**`__init__(self, filename, mode="r", clobber=True, diskless=False,
1697	1780	persist=False, keepweakref=False, format='NETCDF4')`**

1762	1845
1763	1846	`encoding`: encoding used to encode filename string into bytes.
1764	1847	Default is None (`sys.getdefaultfileencoding()` is used).
	1848
	1849	`parallel`: open for parallel access using MPI (requires mpi4py and
	1850	parallel-enabled netcdf-c and hdf5 libraries). Default is `False`. If
	1851	`True`, `comm` and `info` kwargs may also be specified.
	1852
	1853	`comm`: MPI_Comm object for parallel access. Default `None`, which
	1854	means MPI_COMM_WORLD will be used. Ignored if `parallel=False`.
	1855
	1856	`info`: MPI_Info object for parallel access. Default `None`, which
	1857	means MPI_INFO_NULL will be used. Ignored if `parallel=False`.
1765	1858	"""
1766	1859	cdef int grpid, ierr, numgrps, numdims, numvars
1767	1860	cdef char *path
1768	1861	cdef char namstring[NC_MAX_NAME+1]
	1862	IF HAS_NC_PAR:
	1863	cdef MPI_Comm mpicomm
	1864	cdef MPI_Info mpiinfo
1769	1865
1770	1866	memset(&self._buffer, 0, sizeof(self._buffer))
1771	1867

1783	1879
1784	1880	if memory is not None and (mode != 'r' or type(memory) != bytes):
1785	1881	raise ValueError('memory mode only works with \'r\' modes and must be `bytes`')
	1882	if parallel:
	1883	IF HAS_NC_PAR != 1:
	1884	msg='parallel mode requires MPI enabled netcdf-c'
	1885	raise ValueError(msg)
	1886	if format != 'NETCDF4':
	1887	msg='parallel mode only works with format=NETCDF4'
	1888	raise ValueError(msg)
	1889	if comm is not None:
	1890	mpicomm = comm.ob_mpi
	1891	else:
	1892	mpicomm = MPI_COMM_WORLD
	1893	if info is not None:
	1894	mpiinfo = info.ob_mpi
	1895	else:
	1896	mpiinfo = MPI_INFO_NULL
1786	1897
1787	1898	if mode == 'w':
1788	1899	_set_default_format(format=format)
1789	1900	if clobber:
1790		if diskless:
	1901	if parallel:
	1902	IF HAS_NC_PAR:
	1903	ierr = nc_create_par(path, NC_CLOBBER \| NC_MPIIO, \
	1904	mpicomm, mpiinfo, &grpid)
	1905	ELSE:
	1906	pass
	1907	elif diskless:
1791	1908	if persist:
1792	1909	ierr = nc_create(path, NC_WRITE \| NC_CLOBBER \| NC_DISKLESS , &grpid)
1793	1910	else:

1795	1912	else:
1796	1913	ierr = nc_create(path, NC_CLOBBER, &grpid)
1797	1914	else:
1798		if diskless:
	1915	if parallel:
	1916	IF HAS_NC_PAR:
	1917	ierr = nc_create_par(path, NC_NOCLOBBER \| NC_MPIIO, \
	1918	mpicomm, mpiinfo, &grpid)
	1919	ELSE:
	1920	pass
	1921	elif diskless:
1799	1922	if persist:
1800	1923	ierr = nc_create(path, NC_WRITE \| NC_NOCLOBBER \| NC_DISKLESS , &grpid)
1801	1924	else:

1821	1944	nc_open_mem method not enabled. To enable, install Cython, make sure you have
1822	1945	version 4.4.1 or higher of the netcdf C lib, and rebuild netcdf4-python."""
1823	1946	raise ValueError(msg)
	1947	elif parallel:
	1948	IF HAS_NC_PAR:
	1949	ierr = nc_open_par(path, NC_NOWRITE \| NC_MPIIO, \
	1950	mpicomm, mpiinfo, &grpid)
	1951	ELSE:
	1952	pass
1824	1953	elif diskless:
1825	1954	ierr = nc_open(path, NC_NOWRITE \| NC_DISKLESS, &grpid)
1826	1955	else:
1827	1956	ierr = nc_open(path, NC_NOWRITE, &grpid)
1828	1957	elif mode == 'r+' or mode == 'a':
1829		if diskless:
	1958	if parallel:
	1959	IF HAS_NC_PAR:
	1960	ierr = nc_open_par(path, NC_WRITE \| NC_MPIIO, \
	1961	mpicomm, mpiinfo, &grpid)
	1962	ELSE:
	1963	pass
	1964	elif diskless:
1830	1965	ierr = nc_open(path, NC_WRITE \| NC_DISKLESS, &grpid)
1831	1966	else:
1832	1967	ierr = nc_open(path, NC_WRITE, &grpid)
1833	1968	elif mode == 'as' or mode == 'r+s':
1834		if diskless:
	1969	if parallel:
	1970	# NC_SHARE ignored
	1971	IF HAS_NC_PAR:
	1972	ierr = nc_open_par(path, NC_WRITE \| NC_MPIIO, \
	1973	mpicomm, mpiinfo, &grpid)
	1974	ELSE:
	1975	pass
	1976	elif diskless:
1835	1977	ierr = nc_open(path, NC_SHARE \| NC_DISKLESS, &grpid)
1836	1978	else:
1837	1979	ierr = nc_open(path, NC_SHARE, &grpid)
1838	1980	elif mode == 'ws':
1839	1981	if clobber:
1840		if diskless:
	1982	if parallel:
	1983	# NC_SHARE ignored
	1984	IF HAS_NC_PAR:
	1985	ierr = nc_create_par(path, NC_CLOBBER \| NC_MPIIO, \
	1986	mpicomm, mpiinfo, &grpid)
	1987	ELSE:
	1988	pass
	1989	elif diskless:
1841	1990	if persist:
1842	1991	ierr = nc_create(path, NC_WRITE \| NC_SHARE \| NC_CLOBBER \| NC_DISKLESS , &grpid)
1843	1992	else:

1845	1994	else:
1846	1995	ierr = nc_create(path, NC_SHARE \| NC_CLOBBER, &grpid)
1847	1996	else:
1848		if diskless:
	1997	if parallel:
	1998	# NC_SHARE ignored
	1999	IF HAS_NC_PAR:
	2000	ierr = nc_create_par(path, NC_NOCLOBBER \| NC_MPIIO, \
	2001	mpicomm, mpiinfo, &grpid)
	2002	ELSE:
	2003	pass
	2004	elif diskless:
1849	2005	if persist:
1850	2006	ierr = nc_create(path, NC_WRITE \| NC_SHARE \| NC_NOCLOBBER \| NC_DISKLESS , &grpid)
1851	2007	else:

1855	2011	else:
1856	2012	raise ValueError("mode must be 'w', 'r', 'a' or 'r+', got '%s'" % mode)
1857	2013
1858		_ensure_nc_success(ierr, IOError)
	2014	_ensure_nc_success(ierr, err_cls=IOError, filename=path)
1859	2015
1860	2016	# data model and file format attributes
1861	2017	self.data_model = _get_format(grpid)

3312	3468	if grp.data_model != 'NETCDF4': grp._enddef()
3313	3469	_ensure_nc_success(ierr)
3314	3470	else:
3315		# cast fill_value to type of variable.
3316		# also make sure it is written in native byte order
3317		# (the same as the data)
3318		if self._isprimitive or self._isenum:
3319		fillval = numpy.array(fill_value, self.dtype)
3320		if not fillval.dtype.isnative: fillval.byteswap(True)
3321		_set_att(self._grp, self._varid, '_FillValue',\
3322		fillval, xtype=xtype)
	3471	if self._isprimitive or self._isenum or \
	3472	(self._isvlen and self.dtype == str):
	3473	if self._isvlen and self.dtype == str:
	3474	_set_att(self._grp, self._varid, '_FillValue',\
	3475	_tostr(fill_value), xtype=xtype, force_ncstring=True)
	3476	else:
	3477	fillval = numpy.array(fill_value, self.dtype)
	3478	if not fillval.dtype.isnative: fillval.byteswap(True)
	3479	_set_att(self._grp, self._varid, '_FillValue',\
	3480	fillval, xtype=xtype)
3323	3481	else:
3324	3482	raise AttributeError("cannot set _FillValue attribute for VLEN or compound variable")
3325	3483	if least_significant_digit is not None:

4319	4477	The default value of `chartostring` is `True`
4320	4478	(automatic conversions are performed).
4321	4479	"""
4322		if chartostring:
4323		self.chartostring = True
4324		else:
4325		self.chartostring = False
	4480	self.chartostring = bool(chartostring)
4326	4481
4327	4482	def use_nc_get_vars(self,use_nc_get_vars):
4328	4483	"""

4333	4488	`nc_get_vars` not used since it slower than multiple calls
4334	4489	to the unstrided read routine `nc_get_vara` in most cases.
4335	4490	"""
4336		if not use_nc_get_vars:
4337		self._no_get_vars = True
4338		else:
4339		self._no_get_vars = False
4340
	4491	self._no_get_vars = not bool(use_nc_get_vars)
	4492
4341	4493	def set_auto_maskandscale(self,maskandscale):
4342	4494	"""
4343	4495	`set_auto_maskandscale(self,maskandscale)`

4389	4541	The default value of `maskandscale` is `True`
4390	4542	(automatic conversions are performed).
4391	4543	"""
4392		if maskandscale:
4393		self.scale = True
4394		self.mask = True
4395		else:
4396		self.scale = False
4397		self.mask = False
	4544	self.scale = self.mask = bool(maskandscale)
4398	4545
4399	4546	def set_auto_scale(self,scale):
4400	4547	"""

4433	4580	The default value of `scale` is `True`
4434	4581	(automatic conversions are performed).
4435	4582	"""
4436		if scale:
4437		self.scale = True
4438		else:
4439		self.scale = False
4440
	4583	self.scale = bool(scale)
	4584
4441	4585	def set_auto_mask(self,mask):
4442	4586	"""
4443	4587	`set_auto_mask(self,mask)`

4462	4606	The default value of `mask` is `True`
4463	4607	(automatic conversions are performed).
4464	4608	"""
4465		if mask:
4466		self.mask = True
4467		else:
4468		self.mask = False
4469
	4609	self.mask = bool(mask)
	4610
4470	4611
4471	4612	def _put(self,ndarray data,start,count,stride):
4472	4613	"""Private method to put data into a netCDF variable"""

4745	4886	else:
4746	4887	return data
4747	4888
	4889	def set_collective(self, value):
	4890	"""
	4891	`set_collective(self,True_or_False)`
	4892
	4893	turn on or off collective parallel IO access. Ignored if file is not
	4894	open for parallel access.
	4895	"""
	4896	IF HAS_NC_PAR:
	4897	# set collective MPI IO mode on or off
	4898	if value:
	4899	ierr = nc_var_par_access(self._grpid, self._varid,
	4900	NC_COLLECTIVE)
	4901	else:
	4902	ierr = nc_var_par_access(self._grpid, self._varid,
	4903	NC_INDEPENDENT)
	4904	_ensure_nc_success(ierr)
	4905	ELSE:
	4906	pass # does nothing
	4907
4748	4908	def __reduce__(self):
4749	4909	# raise error is user tries to pickle a Variable object.
4750	4910	raise NotImplementedError('Variable is not picklable')

6080	6240	class MFTime(_Variable):
6081	6241	"""
6082	6242	Class providing an interface to a MFDataset time Variable by imposing a unique common
6083		time unit to all files.
	6243	time unit and/or calendar to all files.
6084	6244
6085	6245	Example usage (See `netCDF4.MFTime.__init__` for more details):
6086	6246

6112	6272	32
6113	6273	"""
6114	6274
6115		def __init__(self, time, units=None):
6116		"""
6117		`__init__(self, time, units=None)`
	6275	def __init__(self, time, units=None, calendar=None):
	6276	"""
	6277	`__init__(self, time, units=None, calendar=None)`
6118	6278
6119	6279	Create a time Variable with units consistent across a multifile
6120	6280	dataset.
6121	6281
6122	6282	`time`: Time variable from a `netCDF4.MFDataset`.
6123	6283
6124		`units`: Time units, for example, `days since 1979-01-01`. If None, use
6125		the units from the master variable.
	6284	`units`: Time units, for example, `'days since 1979-01-01'`. If `None`,
	6285	use the units from the master variable.
	6286
	6287	`calendar`: Calendar overload to use across all files, for example,
	6288	`'standard'` or `'gregorian'`. If `None`, check that the calendar attribute
	6289	is present on each variable and values are unique across files raising a
	6290	`ValueError` otherwise.
6126	6291	"""
6127	6292	import datetime
6128	6293	self.__time = time

6131	6296	for name, value in time.__dict__.items():
6132	6297	self.__dict__[name] = value
6133	6298
6134		# make sure calendar attribute present in all files.
6135		for t in self._recVar:
6136		if not hasattr(t,'calendar'):
6137		raise ValueError('MFTime requires that the time variable in all files have a calendar attribute')
6138
6139		# Check that calendar is the same in all files.
6140		if len(set([t.calendar for t in self._recVar])) > 1:
6141		raise ValueError('MFTime requires that the same time calendar is used by all files.')
	6299	# Make sure calendar attribute present in all files if no default calendar
	6300	# is provided. Also assert this value is the same across files.
	6301	if calendar is None:
	6302	calendars = [None] * len(self._recVar)
	6303	for idx, t in enumerate(self._recVar):
	6304	if not hasattr(t, 'calendar'):
	6305	msg = 'MFTime requires that the time variable in all files ' \
	6306	'have a calendar attribute if no default calendar is provided.'
	6307	raise ValueError(msg)
	6308	else:
	6309	calendars[idx] = t.calendar
	6310	calendars = set(calendars)
	6311	if len(calendars) > 1:
	6312	msg = 'MFTime requires that the same time calendar is ' \
	6313	'used by all files if no default calendar is provided.'
	6314	raise ValueError(msg)
	6315	else:
	6316	calendar = list(calendars)[0]
	6317
	6318	# Set calendar using the default or the unique calendar value across all files.
	6319	self.calendar = calendar
6142	6320
6143	6321	# Override units if units is specified.
6144	6322	self.units = units or time.units

-1

setup.cfg less more

5	5	# Usually, nothing else is needed.
6	6	use_ncconfig=True
7	7	# path to nc-config script (use if not found in unix PATH).
8		#ncconfig=/usr/local/bin/nc-config
	8	#ncconfig=/usr/local/bin/nc-config
9	9	[directories]
10	10	#
11	11	# If nc-config doesn't do the trick, you can specify the locations

45	45	# If the libraries and include files are installed in separate locations,
46	46	# use curl_libdir and curl_incdir.
47	47	#curl_dir = /usr/local
	48	# location of mpi.h (needed for parallel support)
	49	#mpi_incdir=/opt/local/include/mpich-mp

+47

-10

setup.py less more

54	54	has_nc_inq_format_extended = False
55	55	has_cdf5_format = False
56	56	has_nc_open_mem = False
	57	has_nc_par = False
57	58
58	59	for d in inc_dirs:
59	60	try:

62	63	continue
63	64
64	65	has_nc_open_mem = os.path.exists(os.path.join(d, 'netcdf_mem.h'))
	66	has_nc_par = os.path.exists(os.path.join(d, 'netcdf_par.h'))
65	67
66	68	for line in f:
67	69	if line.startswith('nc_rename_grp'):

72	74	has_nc_inq_format_extended = True
73	75	if line.startswith('#define NC_FORMAT_64BIT_DATA'):
74	76	has_cdf5_format = True
	77
	78	ncmetapath = os.path.join(d,'netcdf_meta.h')
	79	if os.path.exists(ncmetapath):
	80	has_cdf5 = False
	81	for line in open(ncmetapath):
	82	if line.startswith('#define NC_HAS_CDF5'):
	83	has_cdf5 = True
75	84	break
76	85
77	86	return has_rename_grp, has_nc_inq_path, has_nc_inq_format_extended, \
78		has_cdf5_format, has_nc_open_mem
	87	has_cdf5_format, has_nc_open_mem, has_nc_par
79	88
80	89
81	90	def getnetcdfvers(libdirs):

138	147	curl_dir = os.environ.get('CURL_DIR')
139	148	curl_libdir = os.environ.get('CURL_LIBDIR')
140	149	curl_incdir = os.environ.get('CURL_INCDIR')
	150	mpi_incdir = os.environ.get('MPI_INCDIR')
141	151
142	152	USE_NCCONFIG = os.environ.get('USE_NCCONFIG')
143	153	if USE_NCCONFIG is not None:

229	239	pass
230	240	try:
231	241	curl_incdir = config.get("directories", "curl_incdir")
	242	except:
	243	pass
	244	try:
	245	mpi_incdir = config.get("directories","mpi_incdir")
232	246	except:
233	247	pass
234	248	try:

441	455	else:
442	456	# append numpy include dir.
443	457	import numpy
444
445	458	inc_dirs.append(numpy.get_include())
446	459
447	460	# get netcdf library version.

454	467	cmdclass = {}
455	468	netcdf4_src_root = osp.join('netCDF4', '_netCDF4')
456	469	netcdf4_src_c = netcdf4_src_root + '.c'
	470	netcdftime_src_root = osp.join('netcdftime', '_netcdftime')
	471	netcdftime_src_c = netcdftime_src_root + '.c'
457	472	if 'sdist' not in sys.argv[1:] and 'clean' not in sys.argv[1:]:
458	473	sys.stdout.write('using Cython to compile netCDF4.pyx...\n')
459		# remove netCDF4.c file if it exists, so cython will recompile netCDF4.pyx.
	474	# remove _netCDF4.c file if it exists, so cython will recompile _netCDF4.pyx.
460	475	# run for build and install (issue #263). Otherwise 'pip install' will
461		# not regenerate netCDF4.c, even if the C lib supports the new features.
462		if len(sys.argv) >= 2 and os.path.exists(netcdf4_src_c):
463		os.remove(netcdf4_src_c)
	476	# not regenerate _netCDF4.c, even if the C lib supports the new features.
	477	if len(sys.argv) >= 2:
	478	if os.path.exists(netcdf4_src_c):
	479	os.remove(netcdf4_src_c)
	480	# same for _netcdftime.c
	481	if os.path.exists(netcdftime_src_c):
	482	os.remove(netcdftime_src_c)
464	483	# this determines whether renameGroup and filepath methods will work.
465	484	has_rename_grp, has_nc_inq_path, has_nc_inq_format_extended, \
466		has_cdf5_format, has_nc_open_mem = check_api(inc_dirs)
	485	has_cdf5_format, has_nc_open_mem, has_nc_par = check_api(inc_dirs)
	486	try:
	487	import mpi4py
	488	except ImportError:
	489	has_nc_par = False
467	490
468	491	f = open(osp.join('include', 'constants.pyx'), 'w')
469	492	if has_rename_grp:

502	525	sys.stdout.write('netcdf lib does not have cdf-5 format capability\n')
503	526	f.write('DEF HAS_CDF5_FORMAT = 0\n')
504	527
	528	if has_nc_par:
	529	sys.stdout.write('netcdf lib has netcdf4 parallel functions\n')
	530	f.write('DEF HAS_NC_PAR = 1\n')
	531	else:
	532	sys.stdout.write('netcdf lib does not have netcdf4 parallel functions\n')
	533	f.write('DEF HAS_NC_PAR = 0\n')
	534
505	535	f.close()
	536
	537	if has_nc_par:
	538	inc_dirs.append(mpi4py.get_include())
	539	# mpi_incdir should not be needed if using nc-config
	540	# (should be included in nc-config --cflags)
	541	if mpi_incdir is not None: inc_dirs.append(mpi_incdir)
	542
506	543	ext_modules = [Extension("netCDF4._netCDF4",
507	544	[netcdf4_src_root + '.pyx'],
508	545	libraries=libs,

510	547	include_dirs=inc_dirs + ['include'],
511	548	runtime_library_dirs=runtime_lib_dirs),
512	549	Extension('netcdftime._netcdftime',
513		['netcdftime/_netcdftime.pyx'])]
	550	[netcdftime_src_root + '.pyx'])]
514	551	else:
515	552	ext_modules = None
516	553
517	554	setup(name="netCDF4",
518	555	cmdclass=cmdclass,
519		version="1.3.0",
520		long_description="netCDF version 4 has many features not found in earlier versions of the library, such as hierarchical groups, zlib compression, multiple unlimited dimensions, and new data types. It is implemented on top of HDF5. This module implements most of the new features, and can read and write netCDF files compatible with older versions of the library. The API is modelled after Scientific.IO.NetCDF, and should be familiar to users of that module.\n\nThis project has a `Subversion repository <http://code.google.com/p/netcdf4-python/source>`_ where you may access the most up-to-date source.",
	556	version="1.3.1",
	557	long_description="netCDF version 4 has many features not found in earlier versions of the library, such as hierarchical groups, zlib compression, multiple unlimited dimensions, and new data types. It is implemented on top of HDF5. This module implements most of the new features, and can read and write netCDF files compatible with older versions of the library. The API is modelled after Scientific.IO.NetCDF, and should be familiar to users of that module.\n\nThis project is hosted on a `GitHub repository <https://github.com/Unidata/netcdf4-python>`_ where you may access the most up-to-date source.",
521	558	author="Jeff Whitaker",
522	559	author_email="jeffrey.s.whitaker@noaa.gov",
523	560	url="http://github.com/Unidata/netcdf4-python",

-5

test/run_all.py less more

0		import glob, os, sys, unittest, netCDF4
	0	import glob, os, sys, unittest
1	1	from netCDF4 import getlibversion,__hdf5libversion__,__netcdf4libversion__,__version__
	2	from netCDF4 import __has_cdf5_format__, __has_nc_inq_path__, __has_nc_par__
2	3
3	4	# can also just run
4	5	# python -m unittest discover . 'tst*py'

13	14	else:
14	15	test_files.remove('tst_unicode3.py')
15	16	sys.stdout.write('not running tst_unicode3.py ...\n')
16		if __netcdf4libversion__ < '4.2.1':
	17	if __netcdf4libversion__ < '4.2.1' or __has_nc_par__:
17	18	test_files.remove('tst_diskless.py')
18	19	sys.stdout.write('not running tst_diskless.py ...\n')
19		if __netcdf4libversion__ < '4.1.2':
	20	if not __has_nc_inq_path__:
20	21	test_files.remove('tst_filepath.py')
21	22	sys.stdout.write('not running tst_filepath.py ...\n')
22		if __netcdf4libversion__ < '4.4.0' or sys.maxsize < 2**32:
	23	if not __has_cdf5_format__:
23	24	test_files.remove('tst_cdf5.py')
24	25	sys.stdout.write('not running tst_cdf5.py ...\n')
25	26

40	41	runner.run(testsuite)
41	42
42	43	if __name__ == '__main__':
43		import numpy
	44	import numpy, cython
44	45	sys.stdout.write('\n')
45	46	sys.stdout.write('netcdf4-python version: %s\n' % __version__)
46	47	sys.stdout.write('HDF5 lib version: %s\n' % __hdf5libversion__)
47	48	sys.stdout.write('netcdf lib version: %s\n' % __netcdf4libversion__)
48	49	sys.stdout.write('numpy version %s\n' % numpy.__version__)
	50	sys.stdout.write('cython version %s\n' % cython.__version__)
49	51	runner = unittest.TextTestRunner(verbosity=1)
50	52	result = runner.run(testsuite)
51	53	if not result.wasSuccessful():

-0

test/tst_filepath.py less more

1	1	import tempfile
2	2	import unittest
3	3	import netCDF4
	4
4	5
5	6	class test_filepath(unittest.TestCase):
6	7

22	23	nc.close()
23	24	shutil.rmtree(tmpdir)
24	25
	26	def test_no_such_file_raises(self):
	27	fname = 'not_a_nc_file.nc'
	28	with self.assertRaisesRegexp(IOError, fname):
	29	netCDF4.Dataset(fname, 'r')
	30
	31
25	32	if __name__ == '__main__':
26	33	unittest.main()

+13

-1

test/tst_masked5.py less more

4	4	import numpy as np
5	5	from numpy import ma
6	6	from numpy.testing import assert_array_equal
7		from netCDF4 import Dataset
	7	from netCDF4 import Dataset, __netcdf4libversion__
8	8
9	9	# Test use of vector of missing values.
10	10

22	22	f = Dataset(self.testfile, 'w')
23	23	d = f.createDimension('x',6)
24	24	v = f.createVariable('v', "i2", 'x')
	25	# issue 730: set fill_value for vlen str vars
	26	v2 = f.createVariable('v2',str,'x',fill_value=u'<missing>')
25	27
26	28	v.missing_value = self.missing_values
27	29	v[:] = self.v
	30	v2[0]='first'
28	31
29	32	f.close()
30	33

40	43
41	44	f = Dataset(self.testfile)
42	45	v = f.variables["v"]
	46	v2 = f.variables["v2"]
43	47	self.assertTrue(isinstance(v[:], ma.core.MaskedArray))
44	48	assert_array_equal(v[:], self.v_ma)
45	49	assert_array_equal(v[2],self.v[2]) # issue #624.

47	51	self.assertTrue(isinstance(v[:], np.ndarray))
48	52	assert_array_equal(v[:], self.v)
49	53
	54	# issue 730
	55	# this part fails with netcdf 4.1.3
	56	# a bug in vlen strings?
	57	if __netcdf4libversion__ >= '4.4.0':
	58	assert (v2[0]==u'first')
	59	assert (v2[1]==u'<missing>')
	60
	61
50	62	f.close()
51	63
52	64

+27

-4

test/tst_multifile.py less more

85	85	yr = 1979+nfile
86	86	time.units = 'days since %s-01-01' % yr
87	87
88		time.calendar = 'standard'
	88	# Do not set the calendar attribute on the created files to test calendar
	89	# overload.
	90	# time.calendar = 'standard'
89	91
90	92	x = f.createVariable('x','f',('time', 'y', 'z'))
91	93	x.units = 'potatoes per square mile'

105	107
106	108
107	109	def runTest(self):
	110	# The test files have no calendar attribute on the time variable.
	111	calendar = 'standard'
	112
108	113	# Get the real dates
109	114	dates = []
110	115	for file in self.files:
111	116	f = Dataset(file)
112	117	t = f.variables['time']
113		dates.extend(num2date(t[:], t.units, t.calendar))
	118	dates.extend(num2date(t[:], t.units, calendar))
114	119	f.close()
115	120
116	121	# Compare with the MF dates
117	122	f = MFDataset(self.files,check=True)
118	123	t = f.variables['time']
119		mfdates = num2date(t[:], t.units, t.calendar)
120	124
121		T = MFTime(t)
	125	T = MFTime(t, calendar=calendar)
	126	assert_equal(T.calendar, calendar)
122	127	assert_equal(len(T), len(t))
123	128	assert_equal(T.shape, t.shape)
124	129	assert_equal(T.dimensions, t.dimensions)

127	132	assert_equal(date2index(datetime.datetime(1980, 1, 2), T), 366)
128	133	f.close()
129	134
	135	# Test exception is raised when no calendar attribute is available on the
	136	# time variable.
	137	with MFDataset(self.files, check=True) as ds:
	138	with self.assertRaises(ValueError):
	139	MFTime(ds.variables['time'])
	140
	141	# Test exception is raised when the calendar attribute is different on the
	142	# variables. First, add calendar attributes to file. Note this will modify
	143	# the files inplace.
	144	calendars = ['standard', 'gregorian']
	145	for idx, f in enumerate(self.files):
	146	with Dataset(f, 'a') as ds:
	147	ds.variables['time'].calendar = calendars[idx]
	148	with MFDataset(self.files, check=True) as ds:
	149	with self.assertRaises(ValueError):
	150	MFTime(ds.variables['time'])
	151
	152
130	153	if __name__ == '__main__':
131	154	unittest.main()