Commit ceac1d17c9074fa46612a3ee2f64076cbd6a95d2 - ros-vcstools

New upstream version 0.1.42 Jochen Sprickerhof 4 years ago

18 changed file(s) with 548 addition(s) and 243 deletion(s). Raw diff Collapse all Expand all

+15

-5

.gitignore less more

	0	*#
	1	*.DS_Store
	2	*.coverage
	3	*.deb
	4	*.egg-info
	5	*.eggs
	6	*.log
	7	*.orig
0	8	*.pyc
1		.DS_Store
2		.coverage
3		build
	9	*.swp
	10	*.tgz
	11	*~
	12	.tox
	13	build/*
	14	description-pak
4	15	dist
5		*.egg-info
	16	doc-pak
6	17	nosetests.xml
7		.eggs

-1

.travis.yml less more

2	2	- "2.7"
3	3	- "3.4"
4	4	- "3.5"
	5	- "3.6"
5	6	before_install:
6	7	- export REPO=`pwd`
7	8	install:

9	10	- sudo apt-get install -qq python-yaml python-dateutil
10	11	- sudo apt-get install -qq python3-yaml python3-dateutil
11	12	- sudo apt-get install -qq git mercurial bzr subversion
	13	- pip install -U setuptools tox tox-travis
12	14	- echo $PYTHONPATH
13	15	- python -c 'import sys;print(sys.path)'
14	16	- python setup.py install

18	20	- svn --version
19	21	# command to run tests
20	22	script:
21		- python -m nose --with-coverage --cover-package vcstools test
	23	- tox
	24
22	25	notifications:
23	26	email: false
	27	after_success:
	28	- coveralls

+89

-0

CONTRIBUTING.rst less more

	0	Contributing guide
	1	==================
	2
	3	Thanks for your interest in contributing to vcstools.
	4
	5	Any kinds of contributions are welcome: Bug reports, Documentation, Patches.
	6
	7	Developer Environment
	8	---------------------
	9
	10	For many tasks, it is okay to just develop using a single installed python version. But if you need to test/debug the project in multiple python versions, you need to install those version::
	11
	12	1. (Optional) Install multiple python versions
	13
	14	1. (Optional) Install [pyenv](https://github.com/pyenv/pyenv-installer) to manage python versions
	15	2. (Optional) Using pyenv, install the python versions used in testing::
	16
	17	pyenv install 2.7.16
	18	pyenv install 3.6.8
	19
	20	It may be okay to run and test python against locally installed libraries, but if you need to have a consistent build, it is recommended to manage your environment using `virtualenv <https://virtualenv.readthedocs.org/en/latest/>`_::
	21
	22	$ virtualenv ~/vcstools_venv
	23	$ source ~/vcstools_venv/bin/activate
	24
	25	Editable library install
	26	-------------------
	27
	28	It is common to work on rosinstall or wstool while also needing to make changes to the vcstools library. For that purpose, use::
	29
	30	$ pip install --editable /path/to/vcstools_source
	31
	32	For convenience also consider [virtualenvwrapper](https://pypi.org/project/virtualenvwrapper/ ).
	33
	34	At this point in any shell where you run ``source ~/vcstools_venv/bin/activate``, you can use vcstools and evny edits to files in the vcstools source will take effect immediately.
	35	This is the effect of ``pip install --editable``, see ``pip install --help``.
	36
	37	To setup a virtualenv for Python3 simply do this (from a clean terminal)::
	38
	39	$ virtualenv --python=python3 ~/vcstools_venv_py3
	40	$ source ~/vcstools_venv_py3
	41
	42	When you're done developing, you can exit any shells where you did ``source .../bin/activate`` and delete the virtualenv folder, e.g. ``~/vcstools_venv``.
	43
	44	Testing
	45	-------
	46
	47	Prerequisites:
	48
	49	* The tests require git, mercurial, bazaar and subversion to be installed.
	50
	51	Using the python library nose to test::
	52
	53	# run all tests using nose
	54	$ nosetests
	55	# run one test using nose
	56	$ nosetests {testname}
	57	# run all tests with coverage check
	58	$ python setup.py test
	59	# run all tests using python3
	60	$ python3 setup.py test
	61	# run all tests against multiple python versions (same as in travis)
	62	$ tox
	63
	64	Releasing
	65	---------
	66
	67	* Update `src/vcstools/__version__.py`
	68	* Check `doc/changelog` is up to date
	69	* Check `stdeb.cfg` is up to date
	70	* prepare release dependencies::
	71
	72	pip install --upgrade setuptools wheel twine
	73
	74	* Upload to testpypi::
	75
	76	python3 setup.py sdist bdist_wheel
	77	twine upload --repository testpypi dist/*
	78
	79	* Check testpypi download files and documentation look ok
	80	* Actually release::
	81
	82	twine upload dist/*
	83
	84	* Create and push tag::
	85
	86	git tag x.y.z
	87	git push
	88	git push --tags

+11

-0

MANIFEST.in less more

	0	include *.py
	1	include LICENSE
	2
	3	global-exclude .tox
	4	global-exclude *~
	5	global-exclude __pycache__
	6	global-exclude .coverage
	7	global-exclude *.py[co]
	8	global-exclude *.db
	9	global-exclude .git*
	10	global-exclude *.orig

+32

-50

README.rst less more

0	0	vcstools
1	1	========
2	2
3		The vcstools module provides a Python API for interacting with different version control systems (VCS/SCMs).
	3	.. image:: https://travis-ci.org/vcstools/vcstools.svg?branch=master
	4	:target: https://travis-ci.org/vcstools/vcstools
	5
	6	.. image:: https://coveralls.io/repos/github/vcstools/vcstools/badge.svg?branch=master
	7	:target: https://coveralls.io/github/vcstools/vcstools?branch=master
	8
	9	.. image:: https://img.shields.io/pypi/v/vcstools.svg
	10	:target: https://pypi.python.org/pypi/vcstools
	11
	12	.. image:: https://img.shields.io/pypi/pyversions/vcstools.svg
	13	:target: https://pypi.python.org/pypi/vcstools
	14
	15	.. image:: https://img.shields.io/pypi/status/vcstools.svg
	16	:target: https://pypi.python.org/pypi/vcstools
	17
	18	.. image:: https://img.shields.io/pypi/l/vcstools.svg
	19	:target: https://pypi.python.org/pypi/vcstools
	20
	21	.. image:: https://img.shields.io/pypi/dd/vcstools.svg
	22	:target: https://pypi.python.org/pypi/vcstools
	23
	24	.. image:: https://img.shields.io/pypi/dw/vcstools.svg
	25	:target: https://pypi.python.org/pypi/vcstools
	26
	27	.. image:: https://img.shields.io/pypi/dm/vcstools.svg
	28	:target: https://pypi.python.org/pypi/vcstools
	29
	30	The vcstools module provides a Python API for interacting with different version control systems (VCS/SCMs)
	31	It is used in tools like `wstool <https://wiki.ros.org/wstool>`_ and `rosinstall <https://docs.ros.org/independent/api/rosinstall/html/>`_, which are frequently used in ROS.
	32
	33	This should not be confused with ``vcstool`` (no trailing ``s``), which provides the ``vcs`` command line tool and is otherwise unrelated to this repository, see: https://github.com/dirk-thomas/vcstool
4	34
5	35	See http://www.ros.org/doc/independent/api/vcstools/html/
6	36
7	37	Installing
8	38	----------
9	39
10		Install the latest release on Ubuntu using apt-get::
11
12		$ sudo apt-get install vcstools
13
14		On other Systems, use the pypi package::
15
16		$ pip install vcstools
17
18		Developer Environment
19		---------------------
20
21		When testing or doing development on vcstools, use a `virtualenv <https://virtualenv.readthedocs.org/en/latest/>`_::
22
23		$ virtualenv ~/vcstools_venv
24		$ source ~/vcstools_venv/bin/activate
25		$ pip install --editable /path/to/vcstools_source
26
27		At this point in any shell where you run ``source ~/vcstools_venv/bin/activate``, you can use vcstools and evny edits to files in the vcstools source will take effect immediately.
28		This is the effect of ``pip install --editable``, see ``pip install --help``.
29
30		To setup a virtualenv for Python3 simply do this (from a clean terminal)::
31
32		$ virtualenv --python=python3 ~/vcstools_venv_py3
33		$ source ~/vcstools_venv_py3
34
35		When you're done developing, you can exit any shells where you did ``source .../bin/activate`` and delete the virtualenv folder, e.g. ``~/vcstools_venv``.
36
37		Testing
38		-------
39
40		Use the python library nose to test::
41
42		$ python setup.py test
43
44		To test with coverage, make sure to have python-coverage installed and run::
45
46		$ python setup.py test -n # this installs test dependencies only
47		$ nosetests --with-coverage --cover-package vcstools
48
49		To run python3 compatibility tests, run::
50
51		$ python3 setup.py test
52
53		Test Status
54		-----------
55
56		.. image:: https://travis-ci.org/vcstools/vcstools.svg?branch=master
57		:target: https://travis-ci.org/vcstools/vcstools
58
	40	See `documentation <http://docs.ros.org/independent/api/vcstools/html/>`_

+14

-0

doc/changelog.rst less more

2	2
3	3	0.1
4	4	===
	5
	6	0.1.42
	7	------
	8
	9	- remove cosmic and disco until we have hosting for it
	10
	11	0.1.41
	12	------
	13
	14	- fix git submodule error due to lack of quoting
	15	- Fix git update failure by refreshing git index before fast-forward
	16	- Fix python3 incompatibility due to wrong use of urlopen
	17	- Updating get_affected_files function by removing single quotes covering format (#129)
	18	- Fix export_upstream for git submodules with relative urls. (#130)
5	19
6	20	0.1.40
7	21	------

-0

setup.cfg less more

	0	[aliases]
	1	test = nosetests --with-coverage --cover-package=vcstools --where=test --cover-min-percentage=80

+29

-8

setup.py less more

1	1
2	2	import imp
3	3
	4	with open('README.rst') as readme_file:
	5	README = readme_file.read()
4	6
5	7	def get_version():
6	8	ver_file = None

13	15	if ver_file is not None:
14	16	ver_file.close()
15	17
	18	test_required = [
	19	"nose",
	20	"coverage",
	21	"coveralls",
	22	"mock",
	23	"pep8",
	24	# run checks in multiple environments
	25	"tox",
	26	"tox-pyenv",
	27	# code metrics
	28	"radon~=1.4.0; python_version > '3'",
	29	# coala lint checks only in newest python
	30	"coala; python_version > '3'",
	31	"coala-bears; python_version > '3'",
	32	# mypy typing checks only in newest python
	33	"mypy; python_version > '3'"
	34	]
16	35
17	36	setup(name='vcstools',
18	37	version=get_version(),

20	39	package_dir={'': 'src'},
21	40	scripts=[],
22	41	install_requires=['pyyaml', 'python-dateutil'],
23		tests_require=['nose', 'mock'],
24		test_suite="nose.collector",
	42	# tests_require automatically installed when running python setup.py test
	43	tests_require=test_required,
	44	# extras_require allow pip install .[test]
	45	extras_require={
	46	'test': test_required
	47	},
25	48	author="Tully Foote, Thibault Kruse, Ken Conley",
26	49	author_email="tfoote@osrfoundation.org",
27	50	url="http://wiki.ros.org/vcstools",
28		download_url="http://download.ros.org/downloads/vcstools/",
29	51	keywords=["scm", "vcs", "git", "svn", "hg", "bzr"],
30	52	classifiers=[
31	53	"Programming Language :: Python",
32	54	"Programming Language :: Python :: 2",
33	55	"Programming Language :: Python :: 3",
34		"License :: OSI Approved :: BSD License"
	56	"License :: OSI Approved :: BSD License",
	57	"Development Status :: 7 - Inactive",
	58	"Topic :: Software Development :: Version Control"
35	59	],
36	60	description="VCS/SCM source control library for svn, git, hg, and bzr",
37		long_description="""\
38		Library for managing source code trees from multiple version control systems.
39		Current supports svn, git, hg, and bzr.
40		""",
	61	long_description=README,
41	62	license="BSD")

-1

src/vcstools/__version__.py less more

0		version = '0.1.40'
	0	version = '0.1.42'

-1

src/vcstools/common.py less more

119	119	fdesc, fname = tempfile.mkstemp()
120	120	fhand = os.fdopen(fdesc, "wb")
121	121	# Copy the http response to the temporary file.
122		shutil.copyfileobj(resp.fp, fhand)
	122	shutil.copyfileobj(resp, fhand)
123	123	finally:
124	124	if fhand:
125	125	fhand.close()

+30

-4

src/vcstools/git.py less more

455	455	return response
456	456
457	457	def get_affected_files(self, revision):
458		cmd = "git show {0} --pretty='format:' --name-only".format(
	458	# Making changes for windows support
	459	cmd = "git show {0} --pretty=format: --name-only".format(
459	460	revision)
460	461	code, output, _ = run_shell_command(cmd, shell=True, cwd=self._path)
461	462	affected = []

517	518	rel_path,
518	519	line[3:])
519	520	if LooseVersion(self.gitversion) > LooseVersion('1.7'):
520		command = "git submodule foreach --recursive git status {0}".format(status_flag)
521	521	if not untracked:
522		command += " -uno"
	522	status_flag += " -uno"
	523	command = "git submodule foreach --recursive 'git status {0}'".format(status_flag)
523	524	_, response2, _ = run_shell_command(command,
524	525	shell=True,
525	526	cwd=self._path)

739	740	if not self.detect_presence():
740	741	return False
741	742
	743	# Export should work regardless of the state of the local dir (might have modified files),
	744	# so we clone to a temporary folder and checkout the specified version there.
	745	# If the repo has submodules with relative URLs, cloning to a temp dir doesn't work.
	746	# To support this case, first check if current version is already checked out and there are no
	747	# modifications and try to export from current local dir.
	748	# see https://github.com/vcstools/vcstools/pull/130
	749	current_sha = self.get_version()
	750	export_sha = self.get_version(version)
	751	if current_sha == export_sha and self.get_diff() == '':
	752	archiver = GitArchiver(main_repo_abspath=self.get_path(), force_sub=True)
	753	filepath = '{0}.tar.gz'.format(basepath)
	754	archiver.create(filepath)
	755	return filepath
	756
	757	# export a different version than currently checked out or there are local changes
742	758	try:
743	759	# since version may relate to remote branch / tag we do not
744	760	# know about yet, do fetch if not already done
745	761	self._do_fetch()
	762	# export should work regardless of the state of the local dir (might have modified files)
	763	# so we clone to a temporary folder and checkout the specified version there
746	764	tmpd_path = tempfile.mkdtemp()
747	765	try:
748	766	tmpgit = GitClient(tmpd_path)

826	844	return True
827	845	print("Cannot fast-forward, local repository and remote '%s' have diverged." % branch_parent)
828	846	return False
	847	# 'git reset --keep' doesn't refresh the index. Do it manually to avoid
	848	# errors as reported in: https://github.com/vcstools/wstool/issues/77
	849	cmd = "git update-index -q --refresh"
	850	run_shell_command(cmd,
	851	shell=True,
	852	cwd=self._path,
	853	show_stdout=False,
	854	verbose=verbose)
829	855	if verbose:
830	856	print("Rebasing repository")
831	857	# Rebase, do not pull, because somebody could have
832	858	# commited in the meantime.
833	859	if LooseVersion(self.gitversion) >= LooseVersion('1.7.1'):
834		# --keep allows o rebase even with local changes, as long as
	860	# --keep allows to rebase even with local changes, as long as
835	861	# local changes are not in files that change between versions
836	862	cmd = "git reset --keep remotes/%s/%s" % (default_remote, branch_parent)
837	863	value, _, _ = run_shell_command(cmd,

+84

-127

src/vcstools/git_archive_all.py less more

0		#! /usr/bin/env python
1	0	# coding=utf-8
2	1
3	2	# The MIT License (MIT)

23	22	# THE SOFTWARE.
24	23
25	24	# from
26		# https://github.com/Kentzo/git-archive-all/blob/497049571f1cfe1c183cd3513b69914fa7379824/git_archive_all.py
	25	# https://github.com/Kentzo/git-archive-all/blob/613fa6525e4815c37fed4122fb4ba6ca992d8ff9/git_archive_all.py
27	26
28	27
29	28	from __future__ import print_function

35	34	import sys
36	35	import tarfile
37	36	from zipfile import ZipFile, ZipInfo, ZIP_DEFLATED
38
39		__version__ = "1.12"
	37	import re
	38
	39	__version__ = "1.16.4"
40	40
41	41
42	42	class GitArchiver(object):

61	61	baz
62	62	foo/
63	63	bar
64		@type prefix: string
	64	@type prefix: str
65	65
66	66	@param exclude: Determines whether archiver should follow rules specified in .gitattributes files.
67		@type exclude: bool
	67	@type exclude: bool
68	68
69	69	@param force_sub: Determines whether submodules are initialized and updated before archiving.
70	70	@type force_sub: bool

76	76	If given path is path to a subdirectory (but not a submodule directory!) it will be replaced
77	77	with abspath to top-level directory of the repository.
78	78	If None, current cwd is used.
79		@type main_repo_abspath: string
	79	@type main_repo_abspath: str
80	80	"""
81	81	if extra is None:
82	82	extra = []

84	84	if main_repo_abspath is None:
85	85	main_repo_abspath = path.abspath('')
86	86	elif not path.isabs(main_repo_abspath):
87		raise ValueError("You MUST pass absolute path to the main git repository.")
	87	raise ValueError("main_repo_abspath must be an absolute path")
88	88
89	89	try:
90		self.run_shell("[ -d .git ] \|\| git rev-parse --git-dir > /dev/null 2>&1", main_repo_abspath)
91		except Exception as e:
92		raise ValueError("{0} not a git repository (or any of the parent directories).".format(main_repo_abspath))
93
94		main_repo_abspath = path.abspath(
95		self.read_git_shell('git rev-parse --show-toplevel', main_repo_abspath)
96		.rstrip()
97		)
	90	main_repo_abspath = path.abspath(self.run_git_shell('git rev-parse --show-toplevel', main_repo_abspath).rstrip())
	91	except CalledProcessError:
	92	raise ValueError("{0} is not part of a git repository".format(main_repo_abspath))
98	93
99	94	self.prefix = prefix
100	95	self.exclude = exclude

110	105	Supported formats are: gz, zip, bz2, xz, tar, tgz, txz
111	106
112	107	@param output_path: Output file path.
113		@type output_path: string
	108	@type output_path: str
114	109
115	110	@param dry_run: Determines whether create should do nothing but print what it would archive.
116	111	@type dry_run: bool
117	112
118	113	@param output_format: Determines format of the output archive. If None, format is determined from extension
119	114	of output_file_path.
120		@type output_format: string
	115	@type output_format: str
121	116	"""
122	117	if output_format is None:
123	118	file_name, file_ext = path.splitext(output_path)

151	146	def add_file(file_path, arcname):
152	147	archive.add(file_path, arcname)
153	148	else:
154		raise RuntimeError("Unknown format: {0}".format(output_format))
	149	raise RuntimeError("unknown format: {0}".format(output_format))
155	150
156	151	def archiver(file_path, arcname):
157	152	self.LOG.debug("Compressing {0} => {1}...".format(file_path, arcname))

162	157	def archiver(file_path, arcname):
163	158	self.LOG.info("{0} => {1}".format(file_path, arcname))
164	159
165		self.archive_all_files(archiver) # this will take care of submodule init and update
	160	self.archive_all_files(archiver)
166	161
167	162	if archive is not None:
168	163	archive.close()

172	167	Returns exclude patterns for a given repo. It looks for .gitattributes files in repo_file_paths.
173	168
174	169	Resulting dictionary will contain exclude patterns per path (relative to the repo_abspath).
175		E.g. {('.', 'Catalyst', 'Editions', 'Base'), ['Foo', 'Bar']}
176
177		@type repo_abspath: string
178		@param repo_abspath: Absolute path to the git repository.
179
	170	E.g. {('.', 'Catalyst', 'Editions', 'Base'): ['Foo', 'Bar']}
	171
	172	@param repo_abspath: Absolute path to the git repository.
	173	@type repo_abspath: str
	174
	175	@param repo_file_paths: List of paths relative to the repo_abspath that are under git control.
180	176	@type repo_file_paths: list
181		@param repo_file_paths: List of paths relative to the repo_abspath that are under git control.
182
183		@rtype: dict
184		@return: Dictionary representing exclude patterns.
185		Keys are tuples of strings. Values are lists of strings.
186		Returns None if self.exclude is not set.
	177
	178	@return: Dictionary representing exclude patterns.
	179	Keys are tuples of strings. Values are lists of strings.
	180	Returns None if self.exclude is not set.
	181	@rtype: dict or None
187	182	"""
188	183	if not self.exclude:
189	184	return None

203	198
204	199	# There may be no gitattributes.
205	200	try:
206		global_attributes_abspath = self.read_shell("git config --get core.attributesfile", repo_abspath).rstrip()
	201	global_attributes_abspath = self.run_git_shell("git config --get core.attributesfile", repo_abspath).rstrip()
207	202	exclude_patterns[()] = read_attributes(global_attributes_abspath)
208	203	except:
209	204	# And it's valid to not have them.

228	223	"""
229	224	Checks whether file at a given path is excluded.
230	225
231		@type repo_abspath: string
232	226	@param repo_abspath: Absolute path to the git repository.
233
234		@type repo_file_path: string
235		@param repo_file_path: Path to a file within repo_abspath.
236
237		@type exclude_patterns: dict
238		@param exclude_patterns: Exclude patterns with format specified for get_exclude_patterns.
239
	227	@type repo_abspath: str
	228
	229	@param repo_file_path: Path to a file within repo_abspath.
	230	@type repo_file_path: str
	231
	232	@param exclude_patterns: Exclude patterns with format specified for get_exclude_patterns.
	233	@type exclude_patterns: dict
	234
	235	@return: True if file should be excluded. Otherwise False.
240	236	@rtype: bool
241		@return: True if file should be excluded. Otherwise False.
242	237	"""
243	238	if exclude_patterns is None or not len(exclude_patterns):
244	239	return False

273	268
274	269	@param archiver: Callable that accepts 2 arguments:
275	270	abspath to file on the system and relative path within archive.
	271	@type archiver: Callable
276	272	"""
277	273	for file_path in self.extra:
278	274	archiver(path.abspath(file_path), path.join(self.prefix, file_path))

289	285
290	286	Recurs into submodules as well.
291	287
292		@type repo_path: string
293		@param repo_path: Path to the git submodule repository relative to main_repo_abspath.
294
295		@rtype: iterator
296		@return: Iterator to traverse files under git control relative to main_repo_abspath.
	288	@param repo_path: Path to the git submodule repository relative to main_repo_abspath.
	289	@type repo_path: str
	290
	291	@return: Iterator to traverse files under git control relative to main_repo_abspath.
	292	@rtype: Iterable
297	293	"""
298	294	repo_abspath = path.join(self.main_repo_abspath, repo_path)
299		repo_file_paths = self.read_git_shell(
	295	repo_file_paths = self.run_git_shell(
300	296	"git ls-files --cached --full-name --no-empty-directory",
301	297	repo_abspath
302	298	).splitlines()

305	301	for repo_file_path in repo_file_paths:
306	302	# Git puts path in quotes if file path has unicode characters.
307	303	repo_file_path = repo_file_path.strip('"') # file path relative to current repo
308		file_name = path.basename(repo_file_path)
	304	repo_file_abspath = path.join(repo_abspath, repo_file_path) # absolute file path
309	305	main_repo_file_path = path.join(repo_path, repo_file_path) # file path relative to the main repo
310	306
311		# Only list symlinks and files that don't start with git.
312		if file_name.startswith(".git") or (
313		not path.islink(main_repo_file_path) and path.isdir(main_repo_file_path)
314		):
	307	# Only list symlinks and files.
	308	if not path.islink(repo_file_abspath) and path.isdir(repo_file_abspath):
315	309	continue
316	310
317	311	if self.is_file_excluded(repo_abspath, repo_file_path, exclude_patterns):

320	314	yield main_repo_file_path
321	315
322	316	if self.force_sub:
323		self.run_shell("git submodule init", repo_abspath)
324		self.run_shell("git submodule update", repo_abspath)
325
326		for submodule_path in self.read_shell("git submodule --quiet foreach 'pwd -P'", repo_abspath).splitlines():
327		# Shell command returns absolute paths to submodules.
328		submodule_path = path.relpath(submodule_path, self.main_repo_abspath)
329		for file_path in self.walk_git_files(submodule_path):
330		yield file_path
	317	self.run_git_shell("git submodule init", repo_abspath)
	318	self.run_git_shell("git submodule update", repo_abspath)
	319
	320	try:
	321	repo_gitmodules_abspath = path.join(repo_abspath, ".gitmodules")
	322
	323	with open(repo_gitmodules_abspath) as f:
	324	lines = f.readlines()
	325
	326	for l in lines:
	327	m = re.match("^\spath\s=\s(.)\s*$", l)
	328
	329	if m:
	330	submodule_path = m.group(1)
	331	submodule_path = path.join(repo_path, submodule_path)
	332
	333	if self.is_file_excluded(repo_abspath, submodule_path, exclude_patterns):
	334	continue
	335
	336	for submodule_file_path in self.walk_git_files(submodule_path):
	337	if self.is_file_excluded(repo_abspath, submodule_file_path, exclude_patterns):
	338	continue
	339
	340	yield submodule_file_path
	341	except IOError:
	342	pass
331	343
332	344	@staticmethod
333	345	def get_path_components(repo_abspath, abspath):

339	351	'/Documents/Hobby/ParaView/Catalyst/Editions/Base/', function will return:
340	352	['.', 'Catalyst', 'Editions', 'Base']
341	353
342		First element is always '.' (concrete symbol depends on OS).
	354	First element is always os.curdir (concrete symbol depends on OS).
343	355
344	356	@param repo_abspath: Absolute path to the git repository. Normalized via os.path.normpath.
345		@type repo_abspath: string
	357	@type repo_abspath: str
346	358
347	359	@param abspath: Absolute path to a file within repo_abspath. Normalized via os.path.normpath.
348		@type abspath: string
	360	@type abspath: str
349	361
350	362	@return: List of path components.
351	363	@rtype: list

377	389	return components
378	390
379	391	@staticmethod
380		def run_shell(cmd, cwd=None):
381		"""
382		Runs shell command.
383
384		@type cmd: string
	392	def run_git_shell(cmd, cwd=None):
	393	"""
	394	Runs git shell command, reads output and decodes it into unicode string.
	395
385	396	@param cmd: Command to be executed.
386
387		@type cwd: string
	397	@type cmd: str
	398
	399	@type cwd: str
388	400	@param cwd: Working directory.
389	401
390		@rtype: int
391		@return: Return code of the command.
392
393		@raise CalledProcessError: Raises exception if return code of the command is non-zero.
394		"""
395		p = Popen(cmd, shell=True, cwd=cwd)
396		p.wait()
397
398		if p.returncode:
399		raise CalledProcessError(returncode=p.returncode, cmd=cmd)
400
401		return p.returncode
402
403		@staticmethod
404		def read_shell(cmd, cwd=None, encoding='utf-8'):
405		"""
406		Runs shell command and reads output.
407
408		@type cmd: string
409		@param cmd: Command to be executed.
410
411		@type cwd: string
412		@param cwd: Working directory.
413
414		@type encoding: string
415		@param encoding: Encoding used to decode bytes returned by Popen into string.
416
417		@rtype: string
418		@return: Output of the command.
419
420		@raise CalledProcessError: Raises exception if return code of the command is non-zero.
421		"""
422		p = Popen(cmd, shell=True, stdout=PIPE, cwd=cwd)
423		output, _ = p.communicate()
424		output = output.decode(encoding)
425
426		if p.returncode:
427		if sys.version_info > (2, 6):
428		raise CalledProcessError(returncode=p.returncode, cmd=cmd, output=output)
429		else:
430		raise CalledProcessError(returncode=p.returncode, cmd=cmd)
431
432		return output
433
434		@staticmethod
435		def read_git_shell(cmd, cwd=None):
436		"""
437		Runs git shell command, reads output and decodes it into unicode string
438
439		@type cmd: string
440		@param cmd: Command to be executed.
441
442		@type cwd: string
443		@param cwd: Working directory.
444
445		@rtype: string
446		@return: Output of the command.
	402	@rtype: str
	403	@return: Output of the command.
447	404
448	405	@raise CalledProcessError: Raises exception if return code of the command is non-zero.
449	406	"""

-0

src/vcstools/svn.py less more

(No changes)

-1

stdeb.cfg less more

0	0	[DEFAULT]
1	1	Depends: subversion, mercurial, git-core, bzr, python-yaml, python-dateutil
2	2	Depends3: subversion, mercurial, git-core, bzr, python3-yaml
3		Suite: oneiric precise quantal raring saucy trusty utopic vivid wily xenial yakkety zesty artful bionic wheezy jessie stretch
	3	Suite: oneiric precise quantal raring saucy trusty utopic vivid wily xenial yakkety zesty artful bionic wheezy jessie stretch buster
4	4	X-Python3-Version: >= 3.2

+85

-0

test/mock_server.py less more

	0	import os
	1	import json
	2	import re
	3	import socket
	4	import shutil
	5	import tempfile
	6	from threading import Thread
	7
	8	from io import BytesIO
	9	try:
	10	# py3k
	11	from http.server import BaseHTTPRequestHandler, HTTPServer
	12	except ImportError:
	13	# py2.7
	14	from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
	15
	16
	17	def get_free_port():
	18	s = socket.socket(socket.AF_INET, type=socket.SOCK_STREAM)
	19	s.bind(('localhost', 0))
	20	address, port = s.getsockname()
	21	s.close()
	22	return port
	23
	24
	25	def start_mock_server(file_content):
	26	port = get_free_port()
	27
	28	class MockServerRequestHandler(BaseHTTPRequestHandler):
	29	'''adapted from https://realpython.com/testing-third-party-apis-with-mock-servers
	30	serves file in file_content
	31	returns in chunks if chunked=true is part of url
	32	requires any Basic auth if auth=true is part of url
	33	:returns: base URL
	34	'''
	35	CHUNK_SIZE = 1024
	36
	37	def do_GET(self):
	38	if 'auth=true' in self.path:
	39	if not 'Authorization' in self.headers:
	40	self.send_response(401)
	41	self.send_header('Www-Authenticate', 'Basic realm="foo"')
	42	self.end_headers()
	43	return
	44	if re.search(re.compile(r'/downloads/.*'), self.path):
	45	# Add response status code.
	46	self.send_response(200)
	47
	48	# Add response headers.
	49	self.send_header('Content-Type', 'application/application/x-gzip;')
	50
	51	# Add response content.
	52	if 'chunked=true' in self.path:
	53	self.send_header('Transfer-Encoding', 'chunked')
	54	self.send_header('Connection', 'close')
	55	self.end_headers()
	56
	57	stream = BytesIO(file_content)
	58	while True:
	59	data = stream.read(self.CHUNK_SIZE)
	60	# python3.[0-4] cannot easily format bytes (see PEP 461)
	61	self.wfile.write(("%X\r\n" % len(data)).encode('ascii'))
	62	self.wfile.write(data)
	63	self.wfile.write(b"\r\n")
	64	# If there's no more data to read, stop streaming
	65	if not data:
	66	break
	67
	68	else:
	69	self.end_headers()
	70	self.wfile.write(file_content) # nonchunked
	71
	72	# Ensure any buffered output has been transmitted and close the stream
	73	self.wfile.flush()
	74	return
	75	if result_status is None:
	76	self.send_response(404)
	77	self.end_headers()
	78	return
	79
	80	mock_server = HTTPServer(('localhost', port), MockServerRequestHandler)
	81	mock_server_thread = Thread(target=mock_server.serve_forever)
	82	mock_server_thread.setDaemon(True)
	83	mock_server_thread.start()
	84	return 'http://localhost:{port}'.format(port=port)

+86

-37

test/test_git_subm.py less more

52	52	self.root_directory = tempfile.mkdtemp()
53	53	# helpful when setting tearDown to pass
54	54	self.directories = dict(setUp=self.root_directory)
55		self.remote_path = os.path.join(self.root_directory, "remote")
56		self.submodule_path = os.path.join(self.root_directory, "submodule")
57		self.subsubmodule_path = os.path.join(self.root_directory, "subsubmodule")
	55	self.remote_dir = os.path.join(self.root_directory, "remote")
	56	self.repo_path = os.path.join(self.remote_dir, "repo")
	57	self.submodule_path = os.path.join(self.remote_dir, "submodule")
	58	self.subsubmodule_path = os.path.join(self.remote_dir, "subsubmodule")
58	59	self.local_path = os.path.join(self.root_directory, "local")
59	60	self.sublocal_path = os.path.join(self.local_path, "submodule")
60	61	self.sublocal2_path = os.path.join(self.local_path, "submodule2")

65	66	self.subexport2_path = os.path.join(self.export_path, "submodule2")
66	67	self.subsubexport_path = os.path.join(self.subexport_path, "subsubmodule")
67	68	self.subsubexport2_path = os.path.join(self.subexport2_path, "subsubmodule")
68		os.makedirs(self.remote_path)
	69	os.makedirs(self.repo_path)
69	70	os.makedirs(self.submodule_path)
70	71	os.makedirs(self.subsubmodule_path)
71	72
72	73	# create a "remote" repo
73		subprocess.check_call("git init", shell=True, cwd=self.remote_path)
74		subprocess.check_call("touch fixed.txt", shell=True, cwd=self.remote_path)
75		subprocess.check_call("git add fixed.txt", shell=True, cwd=self.remote_path)
76		subprocess.check_call("git commit -m initial", shell=True, cwd=self.remote_path)
77		subprocess.check_call("git tag test_tag", shell=True, cwd=self.remote_path)
78		subprocess.check_call("git branch initial_branch", shell=True, cwd=self.remote_path)
	74	subprocess.check_call("git init", shell=True, cwd=self.repo_path)
	75	subprocess.check_call("touch fixed.txt", shell=True, cwd=self.repo_path)
	76	subprocess.check_call("git add fixed.txt", shell=True, cwd=self.repo_path)
	77	subprocess.check_call("git commit -m initial", shell=True, cwd=self.repo_path)
	78	subprocess.check_call("git tag test_tag", shell=True, cwd=self.repo_path)
	79	subprocess.check_call("git branch initial_branch", shell=True, cwd=self.repo_path)
79	80	po = subprocess.Popen("git log -n 1 --pretty=format:\"%H\"", shell=True,
80		cwd=self.remote_path, stdout=subprocess.PIPE)
	81	cwd=self.repo_path, stdout=subprocess.PIPE)
81	82	self.version_init = po.stdout.read().decode('UTF-8').rstrip('"').lstrip('"')
82	83
83	84	# create a submodule repo

110	111	self.subversion_final = po.stdout.read().decode('UTF-8').rstrip('"').lstrip('"')
111	112
112	113	# attach submodule somewhere, only in test_branch first
113		subprocess.check_call("git checkout master -b test_branch", shell=True, cwd=self.remote_path)
	114	subprocess.check_call("git checkout master -b test_branch", shell=True, cwd=self.repo_path)
114	115	subprocess.check_call("git submodule add %s %s" % (self.submodule_path,
115		"submodule2"), shell=True, cwd=self.remote_path)
	116	"submodule2"), shell=True, cwd=self.repo_path)
116	117
117	118	# this is needed only if git <= 1.7, during the time when submodules were being introduced (from 1.5.3)
118		subprocess.check_call("git submodule init", shell=True, cwd=self.remote_path)
119		subprocess.check_call("git submodule update", shell=True, cwd=self.remote_path)
120
121		subprocess.check_call("git commit -m submodule", shell=True, cwd=self.remote_path)
	119	subprocess.check_call("git submodule init", shell=True, cwd=self.repo_path)
	120	subprocess.check_call("git submodule update", shell=True, cwd=self.repo_path)
	121
	122	subprocess.check_call("git commit -m submodule", shell=True, cwd=self.repo_path)
122	123
123	124	po = subprocess.Popen("git log -n 1 --pretty=format:\"%H\"", shell=True,
124		cwd=self.remote_path, stdout=subprocess.PIPE)
	125	cwd=self.repo_path, stdout=subprocess.PIPE)
125	126	self.version_test = po.stdout.read().decode('UTF-8').rstrip('"').lstrip('"')
126	127
	128	# attach submodule using relative url, only in test_sub_relative
	129	subprocess.check_call("git checkout master -b test_sub_relative", shell=True, cwd=self.repo_path)
	130	subprocess.check_call("git submodule add %s %s" % (os.path.join('..', os.path.basename(self.submodule_path)),
	131	"submodule"), shell=True, cwd=self.repo_path)
	132
	133	# this is needed only if git <= 1.7, during the time when submodules were being introduced (from 1.5.3)
	134	subprocess.check_call("git submodule init", shell=True, cwd=self.repo_path)
	135	subprocess.check_call("git submodule update", shell=True, cwd=self.repo_path)
	136
	137	subprocess.check_call("git commit -m submodule", shell=True, cwd=self.repo_path)
	138
	139	po = subprocess.Popen("git log -n 1 --pretty=format:\"%H\"", shell=True,
	140	cwd=self.repo_path, stdout=subprocess.PIPE)
	141	self.version_relative = po.stdout.read().decode('UTF-8').rstrip('"').lstrip('"')
	142
127	143	# attach submodule to remote on master. CAREFUL : submodule2 is still in working tree (git does not clean it)
128		subprocess.check_call("git checkout master", shell=True, cwd=self.remote_path)
	144	subprocess.check_call("git checkout master", shell=True, cwd=self.repo_path)
129	145	subprocess.check_call("git submodule add %s %s" % (self.submodule_path, "submodule"),
130		shell=True, cwd=self.remote_path)
	146	shell=True, cwd=self.repo_path)
131	147
132	148	# this is needed only if git <= 1.7, during the time when submodules were being introduced (from 1.5.3)
133		subprocess.check_call("git submodule init", shell=True, cwd=self.remote_path)
134		subprocess.check_call("git submodule update", shell=True, cwd=self.remote_path)
135
136		subprocess.check_call("git commit -m submodule", shell=True, cwd=self.remote_path)
	149	subprocess.check_call("git submodule init", shell=True, cwd=self.repo_path)
	150	subprocess.check_call("git submodule update", shell=True, cwd=self.repo_path)
	151
	152	subprocess.check_call("git commit -m submodule", shell=True, cwd=self.repo_path)
137	153
138	154	po = subprocess.Popen("git log -n 1 --pretty=format:\"%H\"", shell=True,
139		cwd=self.remote_path, stdout=subprocess.PIPE)
	155	cwd=self.repo_path, stdout=subprocess.PIPE)
140	156	self.version_final = po.stdout.read().decode('UTF-8').rstrip('"').lstrip('"')
141		subprocess.check_call("git tag last_tag", shell=True, cwd=self.remote_path)
	157	subprocess.check_call("git tag last_tag", shell=True, cwd=self.repo_path)
	158
	159	print("setup done\n\n")
142	160
143	161	@classmethod
144	162	def tearDownClass(self):

155	173	class GitClientTest(GitClientTestSetups):
156	174
157	175	def test_checkout_master_with_subs(self):
158		url = self.remote_path
	176	url = self.repo_path
159	177	client = GitClient(self.local_path)
160	178	subclient = GitClient(self.sublocal_path)
161	179	subsubclient = GitClient(self.subsublocal_path)

173	191	self.assertEqual(self.subsubversion_final, subsubclient.get_version())
174	192
175	193	def test_export_master(self):
176		url = self.remote_path
	194	url = self.repo_path
177	195	client = GitClient(self.local_path)
178	196	subclient = GitClient(self.sublocal_path)
179	197	subsubclient = GitClient(self.subsublocal_path)

202	220	self.assertEqual(dirdiff.right_only, [])
203	221	self.assertEqual(dirdiff.diff_files, [])
204	222
	223	def test_export_relative(self):
	224	url = self.repo_path
	225	client = GitClient(self.local_path)
	226	subclient = GitClient(self.sublocal_path)
	227	subsubclient = GitClient(self.subsublocal_path)
	228	self.assertFalse(client.path_exists())
	229	self.assertFalse(client.detect_presence())
	230	self.assertFalse(os.path.exists(self.export_path))
	231	self.assertTrue(client.checkout(url, "test_sub_relative"))
	232	self.assertTrue(client.path_exists())
	233	self.assertTrue(subclient.path_exists())
	234	self.assertTrue(subsubclient.path_exists())
	235	#subprocess.call(["tree", self.root_directory])
	236	tarpath = client.export_repository("test_sub_relative", self.export_path)
	237	self.assertEqual(tarpath, self.export_path + '.tar.gz')
	238	os.mkdir(self.export_path)
	239	with closing(tarfile.open(tarpath, "r:gz")) as tarf:
	240	tarf.extractall(self.export_path)
	241	subsubdirdiff = filecmp.dircmp(self.subsubexport_path, self.subsublocal_path, ignore=['.git', '.gitmodules'])
	242	self.assertEqual(subsubdirdiff.left_only, [])
	243	self.assertEqual(subsubdirdiff.right_only, [])
	244	self.assertEqual(subsubdirdiff.diff_files, [])
	245	subdirdiff = filecmp.dircmp(self.subexport_path, self.sublocal_path, ignore=['.git', '.gitmodules'])
	246	self.assertEqual(subdirdiff.left_only, [])
	247	self.assertEqual(subdirdiff.right_only, [])
	248	self.assertEqual(subdirdiff.diff_files, [])
	249	dirdiff = filecmp.dircmp(self.export_path, self.local_path, ignore=['.git', '.gitmodules'])
	250	self.assertEqual(dirdiff.left_only, [])
	251	self.assertEqual(dirdiff.right_only, [])
	252	self.assertEqual(dirdiff.diff_files, [])
	253
205	254	def test_export_branch(self):
206		url = self.remote_path
	255	url = self.repo_path
207	256	client = GitClient(self.local_path)
208	257	subclient = GitClient(self.sublocal_path)
209	258	subclient2 = GitClient(self.sublocal2_path)

256	305	self.assertEqual(dirdiff.diff_files, [])
257	306
258	307	def test_export_hash(self):
259		url = self.remote_path
	308	url = self.repo_path
260	309	client = GitClient(self.local_path)
261	310	subclient = GitClient(self.sublocal_path)
262	311	subclient2 = GitClient(self.sublocal2_path)

309	358	self.assertEqual(dirdiff.diff_files, [])
310	359
311	360	def test_checkout_branch_without_subs(self):
312		url = self.remote_path
	361	url = self.repo_path
313	362	client = GitClient(self.local_path)
314	363	subclient = GitClient(self.sublocal_path)
315	364	subsubclient = GitClient(self.subsublocal_path)

323	372	self.assertFalse(subsubclient.path_exists())
324	373
325	374	def test_checkout_test_branch_with_subs(self):
326		url = self.remote_path
	375	url = self.repo_path
327	376	client = GitClient(self.local_path)
328	377	subclient = GitClient(self.sublocal_path)
329	378	subsubclient = GitClient(self.subsublocal_path)

341	390	self.assertTrue(subsubclient2.path_exists())
342	391
343	392	def test_checkout_master_with_subs2(self):
344		url = self.remote_path
	393	url = self.repo_path
345	394	client = GitClient(self.local_path)
346	395	subclient = GitClient(self.sublocal_path)
347	396	subsubclient = GitClient(self.subsublocal_path)

359	408	self.assertFalse(subsubclient2.path_exists())
360	409
361	410	def test_switch_branches(self):
362		url = self.remote_path
	411	url = self.repo_path
363	412	client = GitClient(self.local_path)
364	413	subclient = GitClient(self.sublocal_path)
365	414	subclient2 = GitClient(self.sublocal2_path)

388	437	self.assertTrue(subsubclient.path_exists())
389	438
390	439	def test_switch_branches_retrieve_local_subcommit(self):
391		url = self.remote_path
	440	url = self.repo_path
392	441	client = GitClient(self.local_path)
393	442	subclient = GitClient(self.sublocal_path)
394	443	subclient2 = GitClient(self.sublocal2_path)

429	478	self.assertTrue(os.path.exists(os.path.join(self.sublocal2_path, "submodif.txt")))
430	479
431	480	def test_status(self):
432		url = self.remote_path
	481	url = self.repo_path
433	482	client = GitClient(self.local_path)
434	483	self.assertTrue(client.checkout(url))
435	484	output = client.get_status(porcelain=True) # porcelain=True ensures stable format

479	528	?? local/subsubnew.txt''', output.rstrip())
480	529
481	530	def test_diff(self):
482		url = self.remote_path
	531	url = self.repo_path
483	532	client = GitClient(self.local_path)
484	533	self.assertTrue(client.checkout(url))
485	534	output = client.get_diff()

+47

-8

test/test_tar.py less more

33	33	from __future__ import absolute_import, print_function, unicode_literals
34	34	import os
35	35	import unittest
	36	import tarfile
36	37	import tempfile
37	38	import shutil
38	39	import subprocess
	40	import mock
39	41
40	42	from vcstools.tar import TarClient
	43	from test.mock_server import start_mock_server
	44
	45	def tarfile_contents():
	46	'''
	47	:returns: binary string of ROS package-release-like tarfile to serve
	48	'''
	49	tar_directory = tempfile.mkdtemp()
	50	filename = os.path.join(tar_directory, 'sample.tar.gz')
	51	pkgname = "sample-1.0"
	52	pkg_directory = os.path.join(tar_directory, pkgname)
	53	os.mkdir(pkg_directory)
	54	packagexml = os.path.join(pkg_directory, "package.xml")
	55	with open(packagexml, 'w+') as f:
	56	f.write('<?xml version="1.0"?><package>' + ('<name>sample</name>' * 1000) + '</package>')
	57
	58	with tarfile.open(filename, "w:gz") as tar_handle:
	59	tar_handle.addfile(tarfile.TarInfo(os.path.join(pkgname, "package.xml")), packagexml)
	60	tar_handle.close()
	61
	62	with open(filename, mode='rb') as file: # b is important -> binary
	63	result = file.read()
	64	shutil.rmtree(tar_directory)
	65	return result
	66
41	67
42	68
43	69	class TarClientTest(unittest.TestCase):
	70	'''Test against mock http server'''
44	71
45	72	@classmethod
46	73	def setUpClass(self):
47		self.remote_url = "https://github.com/ros-gbp/ros_comm-release/archive/release/jade/roswtf/1.11.13-0.tar.gz"
48		self.package_version = "ros_comm-release-release-jade-roswtf-1.11.13-0"
	74	baseURL = start_mock_server(tarfile_contents())
	75	self.remote_url = baseURL + '/downloads/1.0.tar.gz'
	76	self.package_version = "sample-1.0"
49	77
50	78	def setUp(self):
51	79	self.directories = {}

80	108	client = TarClient(local_path)
81	109	self.assertEqual(client.get_vcs_type_name(), 'tar')
82	110
83		def test_checkout(self):
	111
	112	@mock.patch('netrc.netrc') # cannot rely on local ~/.netrc file
	113	def test_checkout_parametrized(self, patched_netrc):
	114	netrc_mock = mock.Mock()
	115	netrc_mock.authenticators.return_value = ('user', '' , 'password')
	116	patched_netrc.return_value = netrc_mock
	117	for query_params in ['', '?chunked=true', '?auth=true', '?chunked=true&auth=true']:
	118	self.check_checkout(query_params)
	119
	120	# parametrized
	121	def check_checkout(self, query_params):
84	122	# checks out all subdirs
85	123	directory = tempfile.mkdtemp()
86	124	self.directories["checkout_test"] = directory

89	127	self.assertFalse(client.path_exists())
90	128	self.assertFalse(client.detect_presence())
91	129	self.assertFalse(client.detect_presence())
92		self.assertTrue(client.checkout(self.remote_url))
93		self.assertTrue(client.path_exists())
94		self.assertTrue(client.detect_presence())
95		self.assertEqual(client.get_path(), local_path)
96		self.assertEqual(client.get_url(), self.remote_url)
	130	self.assertTrue(client.checkout(self.remote_url + query_params))
	131	self.assertTrue(client.path_exists())
	132	self.assertTrue(client.detect_presence())
	133	self.assertEqual(client.get_path(), local_path)
	134	self.assertEqual(client.get_url(), self.remote_url + query_params)
97	135	# make sure the tarball subdirectory was promoted correctly.
98	136	self.assertTrue(os.path.exists(os.path.join(local_path,
99	137	self.package_version,

137	175
138	176
139	177	class TarClientTestLocal(unittest.TestCase):
	178	'''Tests with URL being a local filepath'''
140	179
141	180	def setUp(self):
142	181	self.root_directory = tempfile.mkdtemp()

+15

-0

tox.ini less more

	0	# Tox is the QA gateway before releasing
	1	# it can run against multiple python versions
	2	# it runs commands in virtualenvs prepared with python version and dependencies from setup.[py,cfg]
	3	# While tox can be configured to do plenty of things, it is bothersome to work with quickly, and it is not a lightweight dependency, so prefer to write build logic in setup.py or other commands that run without tox.
	4
	5	[tox]
	6	envlist = py27, py34, py35, py36
	7
	8	[testenv]
	9	# flawed due to https://github.com/tox-dev/tox/issues/149
	10	# deps = -rrequirements.txt
	11
	12	commands =
	13	pip install .[test]
	14	{envpython} setup.py test