Commit 277ded1cf2b9e321b5c09ea9ac335fc7b08a8bb1 - python-pyeclib

Merge tag '1.3.1' into debian/newton * New upstream release * Removed Pre-Depends on dpkg 1.3.1 release (to fix release tooling) Change-Id: Icf15c40fac98a60ff9d740d1ee7fabfb7b55b04c Ondřej Nový 7 years ago

14 changed file(s) with 531 addition(s) and 29 deletion(s). Raw diff Collapse all Expand all

-0

AUTHORS less more

7	7	Timur Alperovich (timuralp@swiftstack.com)
8	8	Thiago da Silva (thiago@redhat.com)
9	9	Eric Lambert (eric_lambert@xyratex.com)
	10	Ondřej Nový (ondrej.novy@firma.seznam.cz)
10	11	Davanum Srinivas (davanum@gmail.com)
11	12	Victor Stinner (vstinner@redhat.com)
12	13	Mark Storer (Mark.Storer@evault.com)

+21

-0

ChangeLog less more

	0	New in 1.3.1
	1	------------
	2
	3	* Updated name in setup.py to work with release tooling.
	4
	5
	6	New in 1.3.0
	7	------------
	8
	9	* Updated liberasurecode dependency to 1.2.0.
	10
	11	* Fixed memory leak in get_segment_info (Launchpad bug #1604335).
	12
	13	* Properly return an error code if liberasurecode returns an
	14	invalid fragment size.
	15
	16	* ECDriver() now requries "k" and "m" values to be passed in.
	17
	18	* Fix some requirements and installation instruction.
	19
	20
0	21	New in 1.2.1
1	22	------------
2	23

+14

-3

README.md less more

228	228	Install pre-requisites:
229	229
230	230	* Python 2.6, 2.7 or 3.x (including development packages), argparse, setuptools
231		* liberasurecode v1.1.0 or greater [3]
	231	* liberasurecode v1.2.0 or greater [3]
232	232	* Erasure code backend libraries, gf-complete and Jerasure [1],[2], ISA-L [4] etc
233	233
	234	An example for ubuntu to install dependency packages:
	235	$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
	236	$ sudo pip install -U bindep -r test-requirement.txt
	237
	238	If you want to confirm all dependency packages installed succuessfully, try:
	239	$ sudo bindep -f bindep.txt
	240
	241	That shows missing dependency packages for you, http://docs.openstack.org/infra/bindep/
	242
	243	Note: currently liberasurecode-dev/liberasurecode-devel in package repo is older
	244	than v1.2.0
	245
234	246	Install PyECLib::
235
236	247	$ sudo python setup.py install
237	248
238	249	Run test suite included::

258	269
259	270	[2] Greenan, Kevin M et al, "Flat XOR-based erasure codes in storage systems", http://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
260	271
261		[3] liberasurecode, C API abstraction layer for erasure coding backends, https://bitbucket.org/tsg-/liberasurecode
	272	[3] liberasurecode, C API abstraction layer for erasure coding backends, https://github.com/openstack/liberasurecode
262	273
263	274	[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
264	275

+10

-0

bindep.txt less more

	0	# This is a cross-platform list tracking distribution packages needed by tests;
	1	# see http://docs.openstack.org/infra/bindep/ for additional information.
	2
	3	build-essential [platform:dpkg]
	4	gcc [platform:rpm]
	5	make [platform:rpm]
	6	liberasurecode-dev [platform:dpkg]
	7	liberasurecode-devel [platform:rpm]
	8	python-dev [platform:dpkg]
	9	python-devel [platform:rpm]

-0

debian/changelog less more

	0	python-pyeclib (1.3.1-1) unstable; urgency=medium
	1
	2	* New upstream release
	3	* Removed Pre-Depends on dpkg
	4
	5	-- Ondřej Nový <onovy@debian.org> Sat, 08 Oct 2016 20:28:22 +0200
	6
0	7	python-pyeclib (1.2.1-2) unstable; urgency=medium
1	8
2	9	* d/{control,copyright}: Use my @debian.org email address

-2

debian/control less more

20	20
21	21	Package: python-pyeclib
22	22	Architecture: any
23		Pre-Depends: dpkg (>= 1.15.6~)
24	23	Depends: ${misc:Depends}, ${python:Depends}, ${shlibs:Depends}
25	24	Description: interface for implementing erasure codes - Python 2.x
26	25	This library provides a simple Python interface for implementing erasure

39	38
40	39	Package: python3-pyeclib
41	40	Architecture: any
42		Pre-Depends: dpkg (>= 1.15.6~)
43	41	Depends: ${misc:Depends}, ${python3:Depends}, ${shlibs:Depends}
44	42	Description: interface for implementing erasure codes - Python 3.x
45	43	This library provides a simple Python interface for implementing erasure

+242

-0

doc/source/conf.py less more

	0	# -- coding: utf-8 --
	1	#
	2	# PyECLib documentation build configuration file, created by
	3	# sphinx-quickstart on Thu Aug 11 19:59:50 2016.
	4	#
	5	# This file is execfile()d with the current directory set to its containing dir.
	6	#
	7	# Note that not all possible configuration values are present in this
	8	# autogenerated file.
	9	#
	10	# All configuration values have a default; values that are commented out
	11	# serve to show the default.
	12
	13	import sys, os
	14
	15	# If extensions (or modules to document with autodoc) are in another directory,
	16	# add these directories to sys.path here. If the directory is relative to the
	17	# documentation root, use os.path.abspath to make it absolute, like shown here.
	18	#sys.path.insert(0, os.path.abspath('.'))
	19
	20	# -- General configuration -----------------------------------------------------
	21
	22	# If your documentation needs a minimal Sphinx version, state it here.
	23	#needs_sphinx = '1.0'
	24
	25	# Add any Sphinx extension module names here, as strings. They can be extensions
	26	# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
	27	extensions = []
	28
	29	# Add any paths that contain templates here, relative to this directory.
	30	templates_path = ['_templates']
	31
	32	# The suffix of source filenames.
	33	source_suffix = '.rst'
	34
	35	# The encoding of source files.
	36	#source_encoding = 'utf-8-sig'
	37
	38	# The master toctree document.
	39	master_doc = 'index'
	40
	41	# General information about the project.
	42	project = u'PyECLib'
	43	copyright = u'2016, Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki'
	44
	45	# The version info for the project you're documenting, acts as replacement for
	46	# \|version\| and \|release\|, also used in various other places throughout the
	47	# built documents.
	48	#
	49	# The short X.Y version.
	50	version = '1.3.1'
	51	# The full version, including alpha/beta/rc tags.
	52	release = '1.3.1'
	53
	54	# The language for content autogenerated by Sphinx. Refer to documentation
	55	# for a list of supported languages.
	56	#language = None
	57
	58	# There are two options for replacing \|today\|: either, you set today to some
	59	# non-false value, then it is used:
	60	#today = ''
	61	# Else, today_fmt is used as the format for a strftime call.
	62	#today_fmt = '%B %d, %Y'
	63
	64	# List of patterns, relative to source directory, that match files and
	65	# directories to ignore when looking for source files.
	66	exclude_patterns = []
	67
	68	# The reST default role (used for this markup: `text`) to use for all documents.
	69	#default_role = None
	70
	71	# If true, '()' will be appended to :func: etc. cross-reference text.
	72	#add_function_parentheses = True
	73
	74	# If true, the current module name will be prepended to all description
	75	# unit titles (such as .. function::).
	76	#add_module_names = True
	77
	78	# If true, sectionauthor and moduleauthor directives will be shown in the
	79	# output. They are ignored by default.
	80	#show_authors = False
	81
	82	# The name of the Pygments (syntax highlighting) style to use.
	83	pygments_style = 'sphinx'
	84
	85	# A list of ignored prefixes for module index sorting.
	86	#modindex_common_prefix = []
	87
	88
	89	# -- Options for HTML output ---------------------------------------------------
	90
	91	# The theme to use for HTML and HTML Help pages. See the documentation for
	92	# a list of builtin themes.
	93	html_theme = 'default'
	94
	95	# Theme options are theme-specific and customize the look and feel of a theme
	96	# further. For a list of options available for each theme, see the
	97	# documentation.
	98	#html_theme_options = {}
	99
	100	# Add any paths that contain custom themes here, relative to this directory.
	101	#html_theme_path = []
	102
	103	# The name for this set of Sphinx documents. If None, it defaults to
	104	# "<project> v<release> documentation".
	105	#html_title = None
	106
	107	# A shorter title for the navigation bar. Default is the same as html_title.
	108	#html_short_title = None
	109
	110	# The name of an image file (relative to this directory) to place at the top
	111	# of the sidebar.
	112	#html_logo = None
	113
	114	# The name of an image file (within the static path) to use as favicon of the
	115	# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
	116	# pixels large.
	117	#html_favicon = None
	118
	119	# Add any paths that contain custom static files (such as style sheets) here,
	120	# relative to this directory. They are copied after the builtin static files,
	121	# so a file named "default.css" will overwrite the builtin "default.css".
	122	html_static_path = ['_static']
	123
	124	# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
	125	# using the given strftime format.
	126	#html_last_updated_fmt = '%b %d, %Y'
	127
	128	# If true, SmartyPants will be used to convert quotes and dashes to
	129	# typographically correct entities.
	130	#html_use_smartypants = True
	131
	132	# Custom sidebar templates, maps document names to template names.
	133	#html_sidebars = {}
	134
	135	# Additional templates that should be rendered to pages, maps page names to
	136	# template names.
	137	#html_additional_pages = {}
	138
	139	# If false, no module index is generated.
	140	#html_domain_indices = True
	141
	142	# If false, no index is generated.
	143	#html_use_index = True
	144
	145	# If true, the index is split into individual pages for each letter.
	146	#html_split_index = False
	147
	148	# If true, links to the reST sources are added to the pages.
	149	#html_show_sourcelink = True
	150
	151	# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
	152	#html_show_sphinx = True
	153
	154	# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
	155	#html_show_copyright = True
	156
	157	# If true, an OpenSearch description file will be output, and all pages will
	158	# contain a <link> tag referring to it. The value of this option must be the
	159	# base URL from which the finished HTML is served.
	160	#html_use_opensearch = ''
	161
	162	# This is the file name suffix for HTML files (e.g. ".xhtml").
	163	#html_file_suffix = None
	164
	165	# Output file base name for HTML help builder.
	166	htmlhelp_basename = 'PyECLibdoc'
	167
	168
	169	# -- Options for LaTeX output --------------------------------------------------
	170
	171	latex_elements = {
	172	# The paper size ('letterpaper' or 'a4paper').
	173	#'papersize': 'letterpaper',
	174
	175	# The font size ('10pt', '11pt' or '12pt').
	176	#'pointsize': '10pt',
	177
	178	# Additional stuff for the LaTeX preamble.
	179	#'preamble': '',
	180	}
	181
	182	# Grouping the document tree into LaTeX files. List of tuples
	183	# (source start file, target name, title, author, documentclass [howto/manual]).
	184	latex_documents = [
	185	('index', 'PyECLib.tex', u'PyECLib Documentation',
	186	u'Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki', 'manual'),
	187	]
	188
	189	# The name of an image file (relative to this directory) to place at the top of
	190	# the title page.
	191	#latex_logo = None
	192
	193	# For "manual" documents, if this is true, then toplevel headings are parts,
	194	# not chapters.
	195	#latex_use_parts = False
	196
	197	# If true, show page references after internal links.
	198	#latex_show_pagerefs = False
	199
	200	# If true, show URL addresses after external links.
	201	#latex_show_urls = False
	202
	203	# Documents to append as an appendix to all manuals.
	204	#latex_appendices = []
	205
	206	# If false, no module index is generated.
	207	#latex_domain_indices = True
	208
	209
	210	# -- Options for manual page output --------------------------------------------
	211
	212	# One entry per manual page. List of tuples
	213	# (source start file, name, description, authors, manual section).
	214	man_pages = [
	215	('index', 'pyeclib', u'PyECLib Documentation',
	216	[u'Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki'], 1)
	217	]
	218
	219	# If true, show URL addresses after external links.
	220	#man_show_urls = False
	221
	222
	223	# -- Options for Texinfo output ------------------------------------------------
	224
	225	# Grouping the document tree into Texinfo files. List of tuples
	226	# (source start file, target name, title, author,
	227	# dir menu entry, description, category)
	228	texinfo_documents = [
	229	('index', 'PyECLib', u'PyECLib Documentation',
	230	u'Kevin Greenan, Tushar Gohad, Kota Tsuyuzaki', 'PyECLib', 'One line description of project.',
	231	'Miscellaneous'),
	232	]
	233
	234	# Documents to append as an appendix to all manuals.
	235	#texinfo_appendices = []
	236
	237	# If false, no module index is generated.
	238	#texinfo_domain_indices = True
	239
	240	# How to display URL addresses: 'footnote', 'no', or 'inline'.
	241	#texinfo_show_urls = 'footnote'

+22

-0

doc/source/index.rst less more

	0	.. PyECLib documentation master file, created by
	1	sphinx-quickstart on Thu Aug 11 19:59:50 2016.
	2	You can adapt this file completely to your liking, but it should at least
	3	contain the root `toctree` directive.
	4
	5	Welcome to PyECLib's documentation!
	6	===================================
	7
	8	Contents:
	9
	10	.. toctree::
	11	:maxdepth: 2
	12
	13
	14
	15	Indices and tables
	16	==================
	17
	18	* :ref:`genindex`
	19	* :ref:`modindex`
	20	* :ref:`search`
	21

+49

-8

pyeclib/ec_iface.py less more

25	25	from .enum import unique
26	26	from .utils import create_instance
27	27	from .utils import positive_int_value
28		from pyeclib_c import check_backend_available
	28	from pyeclib_c import get_liberasurecode_version
	29
	30
	31	def check_backend_available(backend_name):
	32	try:
	33	from pyeclib_c import check_backend_available
	34
	35	if backend_name.startswith('flat_xor_hd'):
	36	int_type = PyECLib_EC_Types.get_by_name('flat_xor_hd')
	37	else:
	38	int_type = PyECLib_EC_Types.get_by_name(backend_name)
	39	if not int_type:
	40	return False
	41	return check_backend_available(int_type.value)
	42	except ImportError:
	43	# check_backend_available has been supported since
	44	# liberasurecode>=1.2.0 so we need to define the func for older
	45	# liberasurecode version
	46
	47	# select available k, m values
	48	if backend_name.startswith('flat_xor_hd'):
	49	k, m = (10, 5)
	50	else:
	51	k, m = (10, 4)
	52	try:
	53	driver = ECDriver(ec_type=backend_name, k=k, m=m)
	54	except ECDriverError:
	55	return False
	56	return True
29	57
30	58
31	59	def PyECLibVersion(z, y, x):

113	141	self.ec_type = None
114	142	self.chksum_type = None
115	143	self.validate = False
	144
	145	for required in ('k', 'm'):
	146	if required not in kwargs:
	147	raise ECDriverError(
	148	"Invalid Argument: %s is required" % required)
	149
116	150	for (key, value) in kwargs.items():
117	151	if key == "k":
118	152	try:

476	510	def _PyECLibValidECTypes():
477	511	available_ec_types = []
478	512	for _type in ALL_EC_TYPES:
479		if _type.startswith('flat_xor_hd'):
480		int_type = PyECLib_EC_Types.get_by_name('flat_xor_hd')
481		else:
482		int_type = PyECLib_EC_Types.get_by_name(_type)
483		if not int_type:
484		continue
485		if check_backend_available(int_type.value):
	513	if check_backend_available(_type):
486	514	available_ec_types.append(_type)
487	515	return available_ec_types
488	516
489	517
490	518	VALID_EC_TYPES = _PyECLibValidECTypes()
	519
	520
	521	def _liberasurecode_version():
	522	version_int = get_liberasurecode_version()
	523	version_hex_str = hex(version_int)
	524	version_hex_str = version_hex_str.lstrip('0x')
	525	major = str(int(version_hex_str[-6:-4]))
	526	minor = str(int(version_hex_str[-4:-2]))
	527	rev = str(int(version_hex_str[-2:]))
	528	version_str = '.'.join([major, minor, rev])
	529	return version_str
	530
	531	LIBERASURECODE_VERSION = _liberasurecode_version()

-5

setup.py less more

71	71	library_basename = "liberasurecode"
72	72	library_version = "1"
73	73	library = library_basename + "-" + library_version
74		library_url = "https://bitbucket.org/tsg-/liberasurecode.git"
	74	library_url = "https://github.com/openstack/liberasurecode"
75	75
76	76	found_path = _find_library("erasurecode")
77	77	if found_path:

158	158
159	159	module = Extension('pyeclib_c',
160	160	define_macros=[('MAJOR VERSION', '1'),
161		('MINOR VERSION', '2')],
	161	('MINOR VERSION', '3')],
162	162	include_dirs=[default_python_incdir,
163	163	'src/c/pyeclib_c',
164	164	'/usr/include',

170	170	# extra_compile_args=['-g', '-O0'],
171	171	sources=['src/c/pyeclib_c/pyeclib_c.c'])
172	172
173		setup(name='PyECLib',
174		version='1.2.1',
	173	setup(name='pyeclib',
	174	version='1.3.1',
175	175	author='Kevin Greenan',
176	176	author_email='kmgreen2@gmail.com',
177	177	maintainer='Kevin Greenan and Tushar Gohad',
178	178	maintainer_email='kmgreen2@gmail.com, tusharsg@gmail.com',
179		url='https://bitbucket.org/kmgreen2/pyeclib',
	179	url='http://git.openstack.org/cgit/openstack/pyeclib/',
180	180	description='This library provides a simple Python interface for \
181	181	implementing erasure codes. To obtain the best possible \
182	182	performance, the underlying erasure code algorithms are \

188	188	package_dir={'pyeclib': 'pyeclib'},
189	189	cmdclass={'build': build, 'install': install, 'clean': clean},
190	190	py_modules=['pyeclib.ec_iface', 'pyeclib.core'],
	191	command_options={
	192	'build_sphinx': {
	193	'build_dir': ('setup.py', 'doc/build')}},
191	194	test_suite='test')

+81

-9

src/c/pyeclib_c/pyeclib_c.c less more

84	84	static PyObject * pyeclib_c_decode(PyObject self, PyObject args);
85	85	static PyObject * pyeclib_c_get_metadata(PyObject self, PyObject args);
86	86	static PyObject * pyeclib_c_check_metadata(PyObject self, PyObject args);
	87	static PyObject * pyeclib_c_liberasurecode_version(PyObject self, PyObject args);
87	88
88	89	static PyObject import_class(const char module, const char *cls)
89	90	{

352	353	PyObject *pyeclib_obj_handle = NULL;
353	354	pyeclib_t *pyeclib_handle = NULL;
354	355	PyObject ret_dict = NULL; / python dictionary to return */
	356
	357	// Prepare variables for return dict to cleanup on exit
	358	PyObject *segment_size_key = NULL;
	359	PyObject *segment_size_value = NULL;
	360	PyObject *last_segment_size_key = NULL;
	361	PyObject *last_segment_size_value = NULL;
	362	PyObject *fragment_size_key = NULL;
	363	PyObject *fragment_size_value = NULL;
	364	PyObject *last_fragment_size_key = NULL;
	365	PyObject *last_fragment_size_value = NULL;
	366	PyObject *num_segments_key = NULL;
	367	PyObject *num_segments_value = NULL;
	368
355	369	int data_len; /* data length from user in bytes */
356	370	int segment_size, last_segment_size; /* segment sizes in bytes */
357	371	int num_segments; /* total number of segments */

371	385
372	386	/* The minimum segment size depends on the EC algorithm */
373	387	min_segment_size = liberasurecode_get_minimum_encode_size(pyeclib_handle->ec_desc);
	388	if (min_segment_size < 0) {
	389	pyeclib_c_seterr(-EINVALIDPARAMS, "pyeclib_c_get_segment_info ERROR: ");
	390	return NULL;
	391	}
374	392
375	393	/* Get the number of segments */
376	394	num_segments = (int)ceill((double)data_len / segment_size);

394	412	* This will retrieve fragment_size calculated by liberasurecode with
395	413	* specified backend.
396	414	*/
	415
397	416	fragment_size = liberasurecode_get_fragment_size(pyeclib_handle->ec_desc, data_len);
	417	if (fragment_size < 0) {
	418	pyeclib_c_seterr(-EINVALIDPARAMS, "pyeclib_c_get_segment_info ERROR: ");
	419	return NULL;
	420	}
398	421
399	422	/* Segment size is the user-provided segment size */
400	423	segment_size = data_len;

407	430	*/
408	431
409	432	fragment_size = liberasurecode_get_fragment_size(pyeclib_handle->ec_desc, segment_size);
	433	if (fragment_size < 0) {
	434	pyeclib_c_seterr(-EINVALIDPARAMS, "pyeclib_c_get_segment_info ERROR: ");
	435	return NULL;
	436	}
410	437
411	438	last_segment_size = data_len - (segment_size * (num_segments - 1));
412	439

423	450	}
424	451
425	452	last_fragment_size = liberasurecode_get_fragment_size(pyeclib_handle->ec_desc, last_segment_size);
	453	if (fragment_size < 0) {
	454	pyeclib_c_seterr(-EINVALIDPARAMS, "pyeclib_c_get_segment_info ERROR: ");
	455	return NULL;
	456	}
426	457	}
427	458
428	459	/* Add header to fragment sizes */

432	463	/* Create and return the python dictionary of segment info */
433	464	ret_dict = PyDict_New();
434	465	if (NULL == ret_dict) {
	466	goto error;
	467	} else {
	468	if((segment_size_key = PyString_FromString("segment_size\0")) == NULL \|\|
	469	(segment_size_value = PyInt_FromLong(segment_size)) == NULL \|\|
	470	PyDict_SetItem(ret_dict, segment_size_key, segment_size_value)) goto error;
	471
	472	if((last_segment_size_key = PyString_FromString("last_segment_size\0")) == NULL \|\|
	473	(last_segment_size_value = PyInt_FromLong(last_segment_size)) == NULL \|\|
	474	PyDict_SetItem(ret_dict, last_segment_size_key, last_segment_size_value)) goto error;
	475
	476	if((fragment_size_key = PyString_FromString("fragment_size\0")) == NULL \|\|
	477	(fragment_size_value = PyInt_FromLong(fragment_size)) == NULL \|\|
	478	PyDict_SetItem(ret_dict, fragment_size_key, fragment_size_value)) goto error;
	479
	480	if((last_fragment_size_key = PyString_FromString("last_fragment_size\0")) == NULL \|\|
	481	(last_fragment_size_value = PyInt_FromLong(last_fragment_size)) == NULL \|\|
	482	PyDict_SetItem(ret_dict, last_fragment_size_key, last_fragment_size_value)) goto error;
	483
	484	if((num_segments_key = PyString_FromString("num_segments\0")) == NULL \|\|
	485	(num_segments_value = PyInt_FromLong(num_segments)) == NULL \|\|
	486	PyDict_SetItem(ret_dict, num_segments_key, num_segments_value)) goto error;
	487	}
	488
	489	exit:
	490	Py_XDECREF(segment_size_key);
	491	Py_XDECREF(segment_size_value);
	492	Py_XDECREF(last_segment_size_key);
	493	Py_XDECREF(last_segment_size_value);
	494	Py_XDECREF(fragment_size_key);
	495	Py_XDECREF(fragment_size_value);
	496	Py_XDECREF(last_fragment_size_key);
	497	Py_XDECREF(last_fragment_size_value);
	498	Py_XDECREF(num_segments_key);
	499	Py_XDECREF(num_segments_value);
	500	return ret_dict;
	501
	502	error:
	503	// To prevent unexpected call, this is placed after return call
435	504	pyeclib_c_seterr(-ENOMEM, "pyeclib_c_get_segment_info ERROR: ");
436		} else {
437		PyDict_SetItem(ret_dict, PyString_FromString("segment_size\0"), PyInt_FromLong(segment_size));
438		PyDict_SetItem(ret_dict, PyString_FromString("last_segment_size\0"), PyInt_FromLong(last_segment_size));
439		PyDict_SetItem(ret_dict, PyString_FromString("fragment_size\0"), PyInt_FromLong(fragment_size));
440		PyDict_SetItem(ret_dict, PyString_FromString("last_fragment_size\0"), PyInt_FromLong(last_fragment_size));
441		PyDict_SetItem(ret_dict, PyString_FromString("num_segments\0"), PyInt_FromLong(num_segments));
442		}
443
444		return ret_dict;
	505	Py_XDECREF(ret_dict);
	506	ret_dict = NULL;
	507	goto exit;
	508
445	509	}
446	510
447	511

1188	1252	Py_RETURN_FALSE;
1189	1253	}
1190	1254
	1255	static PyObject*
	1256	pyeclib_c_liberasurecode_version(PyObject self, PyObject args) {
	1257	return PyInt_FromLong(LIBERASURECODE_VERSION);
	1258	}
	1259
1191	1260	static PyMethodDef PyECLibMethods[] = {
1192	1261	{"init", pyeclib_c_init, METH_VARARGS, "Initialize a new erasure encoder/decoder"},
1193	1262	{"encode", pyeclib_c_encode, METH_VARARGS, "Create parity using source data"},

1197	1266	{"get_segment_info", pyeclib_c_get_segment_info, METH_VARARGS, "Return segment and fragment size information needed when encoding a segmented stream"},
1198	1267	{"get_metadata", pyeclib_c_get_metadata, METH_VARARGS, "Get the integrity checking metadata for a fragment"},
1199	1268	{"check_metadata", pyeclib_c_check_metadata, METH_VARARGS, "Check the integrity checking metadata for a set of fragments"},
	1269	{"get_liberasurecode_version", pyeclib_c_liberasurecode_version, METH_NOARGS, "Get libersaurecode version in use"},
	1270	#if ( LIBERASURECODE_VERSION >= _VERSION(1,2,0) )
1200	1271	{"check_backend_available", pyeclib_c_check_backend_available, METH_VARARGS, "Check if a backend is available"},
	1272	#endif
1201	1273	{NULL, NULL, 0, NULL} /* Sentinel */
1202	1274	};
1203	1275

+61

-0

test/test_pyeclib_api.py less more

25	25	import sys
26	26	import tempfile
27	27	import unittest
	28
	29	from distutils.version import StrictVersion
	30
28	31	from pyeclib.ec_iface import ECBackendNotSupported
29	32	from pyeclib.ec_iface import ECDriver
30	33	from pyeclib.ec_iface import ECDriverError

33	36	from pyeclib.ec_iface import PyECLib_EC_Types
34	37	from pyeclib.ec_iface import ALL_EC_TYPES
35	38	from pyeclib.ec_iface import VALID_EC_TYPES
	39	from pyeclib.ec_iface import LIBERASURECODE_VERSION
	40	import resource
	41
36	42
37	43	if sys.version < '3':
38	44	def b2i(b):

107	113
108	114	def tearDown(self):
109	115	pass
	116
	117	def test_invalid_km_args(self):
	118	for ec_type in VALID_EC_TYPES:
	119	# missing k
	120	with self.assertRaises(ECDriverError) as err_context:
	121	ECDriver(ec_type=ec_type, m=1)
	122
	123	self.assertEqual(str(err_context.exception),
	124	"Invalid Argument: k is required")
	125
	126	# missing m
	127	with self.assertRaises(ECDriverError) as err_context:
	128	ECDriver(ec_type=ec_type, k=1)
	129
	130	self.assertEqual(str(err_context.exception),
	131	"Invalid Argument: m is required")
	132
	133	with self.assertRaises(ECDriverError) as err_context:
	134	# m is smaller than 1
	135	ECDriver(ec_type=ec_type, k=-100, m=1)
	136	self.assertEqual(str(err_context.exception),
	137	"Invalid number of data fragments (k)")
	138
	139	with self.assertRaises(ECDriverError) as err_context:
	140	# m is smaller than 1
	141	ECDriver(ec_type=ec_type, k=1, m=-100)
	142	self.assertEqual(str(err_context.exception),
	143	"Invalid number of data fragments (m)")
110	144
111	145	def test_valid_ec_types(self):
112	146	# Build list of available types and compare to VALID_EC_TYPES

530	564	(first_fragment_to_corrupt + i) % len(fragments) for i in range(num_to_corrupt)
531	565	]
532	566
	567	if StrictVersion(LIBERASURECODE_VERSION) < \
	568	StrictVersion('1.2.0'):
	569	# if liberasurecode is older than the version supports
	570	# fragment integrity check, skip following test
	571	continue
533	572	i = 0
534	573	for fragment in fragments:
535	574	if i in fragments_to_corrupt:

572	611	self.assertTrue(
573	612	pyeclib_drivers[0].min_parity_fragments_needed() == 1)
574	613
	614	def test_get_segment_info_memory_usage(self):
	615	for ec_driver in self.get_pyeclib_testspec():
	616	self._test_get_segment_info_memory_usage(ec_driver)
	617
	618	def _test_get_segment_info_memory_usage(self, ec_driver):
	619	# 1. Preapre the expected memory allocation
	620	info = ec_driver.get_segment_info(10241024, 10241024)
	621	info = None
	622	loop_range = range(1000)
	623
	624	# 2. Get current memory usage
	625	usage = resource.getrusage(resource.RUSAGE_SELF)[2]
	626
	627	# 3. Loop to call get_segment_info
	628	for x in loop_range:
	629	ec_driver.get_segment_info(10241024, 10241024)
	630
	631	# 4. memory usage shoudln't be increased
	632	self.assertEqual(usage, resource.getrusage(resource.RUSAGE_SELF)[2],
	633	'Memory usage is increased unexpectedly %s - %s' %
	634	(usage, resource.getrusage(resource.RUSAGE_SELF)[2]))
	635
575	636
576	637	if __name__ == '__main__':
577	638	unittest.main()

-0

test-requirements.txt less more

	0	# The order of packages is significant, because pip processes them in the order
	1	# of appearance. Changing the order has an impact on the overall integration
	2	# process, which may cause wedges in the gate later.
	3
	4	coverage
	5	nose
	6	sphinx>=1.1.2,!=1.2.0,!=1.3b1,<1.3 # BSD

-2

tox.ini less more

1	1	envlist = py27,py34,pep8
2	2
3	3	[testenv]
4		deps=
5		nose
	4	deps =
	5	-r{toxinidir}/test-requirements.txt
6	6	commands=
7	7	nosetests test/
8	8

11	11	pep8
12	12	commands=
13	13	pep8 pyeclib/ setup.py
	14
	15	[testenv:venv]
	16	commands = {posargs}
	17
	18	[testenv:docs]
	19	commands = python setup.py build_sphinx