New Upstream Release - git-annex-remote-rclone

Ready changes

Summary

Merged new upstream version: 0.7 (was: 0.6).

Resulting package

Built on 2023-08-11T10:53 (took 10m2s)

The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:

apt install -t fresh-releases git-annex-remote-rclone

Lintian Result

Diff

diff --git a/.github/workflows/shellcheck.yml b/.github/workflows/shellcheck.yml
new file mode 100644
index 0000000..9db519c
--- /dev/null
+++ b/.github/workflows/shellcheck.yml
@@ -0,0 +1,23 @@
+name: Shellcheck
+
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: '0 6 * * *'
+
+jobs:
+  shellcheck:
+    runs-on: ubuntu-18.04
+
+    steps:
+    - name: Set up environment
+      uses: actions/checkout@v1
+   
+    - name: Install dependencies
+      run: |
+        sudo apt update && sudo apt install shellcheck
+
+    - name: shellcheck
+      run: |
+        shellcheck git-annex-remote-rclone
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
new file mode 100644
index 0000000..1521ccd
--- /dev/null
+++ b/.github/workflows/test.yml
@@ -0,0 +1,69 @@
+name: Tests
+
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: '0 6 * * *'
+
+jobs:
+  test:
+    runs-on: ${{ matrix.os }}
+
+    strategy:
+      fail-fast: false
+      matrix:
+        os:
+          - ubuntu-18.04
+          #- windows-2019
+          - macos-latest
+        rclone:
+          - current
+          - v1.58.1  # current "current" -- the baseline
+          - v1.53.3  # Debian bullseye (current stable)
+          - v1.45    # Debian buster (current oldstable)
+          # older versions for which there is internal logic
+          - v1.33
+          - v1.30
+          # - v1.29 TODO: seems to need rclone config run -- must be done in test script
+        exclude:
+          # 1.30 - --version exits non-0
+          - os: macos-latest
+            rclone: v1.30
+          # 1.45 - has some odd handling of HOME on OSX - does not use overloaded $HOME
+          - os: macos-latest
+            rclone: v1.45
+
+    steps:
+    - name: Set up environment
+      uses: actions/checkout@v1
+
+    - name: Install Python
+      uses: actions/setup-python@v4
+      with:
+        python-version: '>=3.7'
+
+    - name: Install datalad-installer
+      run: pip install datalad-installer
+
+    - name: Install git-annex (Ubuntu)
+      if: startsWith(matrix.os, 'ubuntu-')
+      run: datalad-installer --sudo ok git-annex -m datalad/git-annex:release
+
+    - name: Install git-annex (macOS)
+      if: startsWith(matrix.os, 'macos-')
+      run: datalad-installer git-annex -m brew
+
+    - name: Install latest rclone
+      if: matrix.rclone == 'current'
+      run: datalad-installer --sudo ok rclone -m downloads.rclone.org
+
+    - name: Install specific version of rclone
+      if: matrix.rclone != 'current'
+      run: datalad-installer --sudo ok rclone=${{ matrix.rclone }} -m downloads.rclone.org
+
+    - name: ${{ matrix.module }} tests
+      run: |
+        PATH=$PWD:$PATH tests/all-in-one.sh
+        # Test without encryption so we do get 0-sized files in rclone remote store
+        PATH=$PWD:$PATH GARR_TEST_ENCRYPTION=none tests/all-in-one.sh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 00adb54..c2a8df5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,14 +12,4 @@ test:
     - curl 'https://downloads.rclone.org/rclone-current-linux-amd64.zip' > rclone.zip
     - unzip rclone.zip
     - mv rclone-*-linux-amd64/rclone /usr/local/bin
-    - echo -e '[local]\ntype = local\nnounc =' > ~/.rclone.conf
-    - git-annex version
-    - mkdir testrepo
-    - cd testrepo
-    - git init .
-    - git-annex init
-    - git-annex initremote GA-rclone-CI type=external externaltype=rclone target=local prefix=/tmp/GA-rclone-CI chunk=100MiB encryption=shared mac=HMACSHA512
-    - touch test
-    - git-annex add test
-    - git-annex copy test --to GA-rclone-CI
-    - git-annex testremote GA-rclone-CI --fast
+    - tests/all-in-one.sh
diff --git a/README.md b/README.md
index e210f6a..3f35b5d 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,8 @@ __Users are urged to upgrade to version 0.4 or higher immediately.
 Some ways of using earlier versions could result in data loss. [more information here](https://github.com/DanielDent/git-annex-remote-rclone/issues/8)__
 
 
-[![build status](https://gitlab.com/DanielDent/git-annex-remote-rclone/badges/master/build.svg)](https://gitlab.com/DanielDent/git-annex-remote-rclone/commits/master)
+[![Tests](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/test.yml/badge.svg)](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/test.yml)
+[![Shellcheck](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/shellcheck.yml/badge.svg)](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/shellcheck.yml)
 
 This wrapper around [rclone](http://rclone.org/) makes any destination supported by rclone usable with git-annex.
 
@@ -123,4 +124,4 @@ Please submit a pull request or create a new issue for problems or potential imp
 
 ## License
 
-Copyright 2016-2017 [Daniel Dent](https://www.danieldent.com/). Licensed under the GPLv3.
+Copyright 2016-2022 [Daniel Dent](https://www.danieldent.com/), 2022 [git-annex-remote-rclone contributors](https://github.com/DanielDent/git-annex-remote-rclone/graphs/contributors). Licensed under the GPLv3.
diff --git a/debian/changelog b/debian/changelog
index cd1ef25..ff9ead6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+git-annex-remote-rclone (0.7-1) UNRELEASED; urgency=low
+
+  * New upstream release.
+
+ -- Debian Janitor <janitor@jelmer.uk>  Fri, 11 Aug 2023 10:43:37 -0000
+
 git-annex-remote-rclone (0.6-1) unstable; urgency=medium
 
   * New minor bugfix upstream release
diff --git a/git-annex-remote-rclone b/git-annex-remote-rclone
index ad788fa..bde8de2 100755
--- a/git-annex-remote-rclone
+++ b/git-annex-remote-rclone
@@ -1,9 +1,13 @@
 #!/bin/bash
+# emacs: -*- mode: python; tab-width: 4; indent-tabs-mode: t -*-
+# ex: set sts=4 ts=4 sw=4 noet:
+#
 # git-annex-remote-rclone - wrapper to enable use of rclone-supported cloud providers as git-annex special remotes.
 #
 # Install in PATH as git-annex-remote-rclone
 #
-# Copyright (C) 2016-2017  Daniel Dent
+# Copyright (C) 2016-2022  Daniel Dent
+#               2022       git-annex-remote-rclone contributors
 #
 # This program is free software: you can redistribute it and/or modify it under the terms of version 3 of the GNU
 # General Public License as published by the Free Software Foundation.
@@ -44,7 +48,7 @@ validate_layout() {
 			echo "INITREMOTE-FAILURE rclone_layout setting not recognized"
 			exit 1
 			;;
-	esac		
+	esac
 }
 
 # Sets LOC to the location to use to store a key.
@@ -60,14 +64,14 @@ calclocation () {
 			;;
 		nodir)
 			LOC="$REMOTE_TARGET:$REMOTE_PREFIX/"
-			;;	
+			;;
 		mixed)
 			ask DIRHASH "$1"
 			LOC="$REMOTE_TARGET:$REMOTE_PREFIX/$RET"
 			;;
 		frankencase)
 			ask DIRHASH "$1"
-			lret=$(echo $RET|tr A-Z a-z)
+			lret=$(echo "$RET" | tr '[:upper:]' '[:lower:]')
 			LOC="$REMOTE_TARGET:$REMOTE_PREFIX/$lret"
 			;;
 	esac
@@ -77,32 +81,46 @@ calclocation () {
 ask () {
 	echo "$1" "$2"
 	read -r resp
-    # Strip trailing carriage return, if present
-    resp="${resp%$'\r'}"
-	if echo $resp|grep '^VALUE '>/dev/null; then
-	    RET=$(echo "$resp" | cut -f2- -d' ')
-    	else
-	    RET=""
+	# Strip trailing carriage return, if present
+	resp="${resp%$'\r'}"
+	if echo "$resp" | grep '^VALUE '>/dev/null; then
+		RET=$(echo "$resp" | cut -f2- -d' ')
+		else
+		RET=""
 	fi
 }
 
+GREP () {
+	set +e
+	out=$(grep "$@")
+	rc=$?
+	set -e
+	# Replace explicit newline since we must provide 1 line DEBUG
+	# Fancy sed is from Example 5 of https://linuxhint.com/newline_replace_sed
+	# which worked on Linux and OSX.
+	# shellcheck disable=SC2016
+	out_safe=$(echo "$out" | sed -ne 'H;${x;s/\n/\\n/g;s/^,//;p;}')
+	echo "DEBUG 'grep \"$*\"' exited with rc=$rc and stdout=${out_safe}"
+	exit $rc
+}
 
 # This has to come first, to get the protocol started.
 echo VERSION 1
 
 while read -r line; do
-    # Strip trailing carriage return, if present
-    line="${line%$'\r'}"
+	# Strip trailing carriage return, if present
+	line="${line%$'\r'}"
+	# shellcheck disable=SC2086
 	set -- $line
 	case "$1" in
 		INITREMOTE)
 			# Do anything necessary to create resources
 			# used by the remote. Try to be idempotent.
-			# 
+			#
 			# Use GETCONFIG to get any needed configuration
 			# settings, and SETCONFIG to set any persistent
 			# configuration settings.
-			# 
+			#
 			# (Note that this is not run every time, only when
 			# git annex initremote or git annex enableremote is
 			# run.)
@@ -110,31 +128,31 @@ while read -r line; do
 			getconfig prefix
 			REMOTE_PREFIX=$RET
 			if [ -z "$REMOTE_PREFIX" ]; then
-                		REMOTE_PREFIX="git-annex"
+				REMOTE_PREFIX="git-annex"
 			fi
 			if [ "$REMOTE_PREFIX" == "/" ]; then
-		                echo INITREMOTE-FAILURE "storing objects directly in the root (/) is not supported"
+				echo INITREMOTE-FAILURE "storing objects directly in the root (/) is not supported"
 			fi
 			setconfig prefix $REMOTE_PREFIX
 
 			getconfig target
 			REMOTE_TARGET=$RET
-			setconfig target $REMOTE_TARGET
-			
+			setconfig target "$REMOTE_TARGET"
+
 			getconfig rclone_layout
 			RCLONE_LAYOUT=$RET
 			validate_layout
-			setconfig rclone_layout $RCLONE_LAYOUT
+			setconfig rclone_layout "$RCLONE_LAYOUT"
 
 			if [ -z "$REMOTE_TARGET" ]; then
-		                echo INITREMOTE-FAILURE "rclone remote target must be specified (use target= parameter)"
+				echo INITREMOTE-FAILURE "rclone remote target must be specified (use target= parameter)"
 			fi
 
-            if runcmd rclone mkdir $REMOTE_TARGET:$REMOTE_PREFIX; then
-                echo INITREMOTE-SUCCESS
-            else
-                echo INITREMOTE-FAILURE "Failed to create directory on remote. Ensure that 'rclone config' has been run."
-            fi
+			if runcmd rclone mkdir "$REMOTE_TARGET:$REMOTE_PREFIX"; then
+				echo INITREMOTE-SUCCESS
+			else
+				echo INITREMOTE-FAILURE "Failed to create directory on remote. Ensure that 'rclone config' has been run."
+			fi
 		;;
 		PREPARE)
 			# Use GETCONFIG to get configuration settings,
@@ -143,20 +161,22 @@ while read -r line; do
 
 			getconfig prefix
 			REMOTE_PREFIX="$RET"
-			
+
 			getconfig target
 			REMOTE_TARGET="$RET"
-			
+
 			getconfig rclone_layout
 			RCLONE_LAYOUT="$RET"
 			validate_layout
 
-            echo PREPARE-SUCCESS
+			echo PREPARE-SUCCESS
 		;;
 		TRANSFER)
+			op="$2"
 			key="$3"
-			file="$4"
-			case "$2" in
+			shift 3
+			file="$*"
+			case "$op" in
 				STORE)
 					# Store the file to a location
 					# based on the key.
@@ -164,13 +184,13 @@ while read -r line; do
 					calclocation "$key"
 					if [ ! -e "$file" ]; then
 						echo TRANSFER-FAILURE STORE "$key" "asked to store non-existent file $file"
-					else	
+					else
 						if runcmd rclone copy "$file" "$LOC"; then
 							echo TRANSFER-SUCCESS STORE "$key"
 						else
 							echo TRANSFER-FAILURE STORE "$key"
 						fi
-					fi					
+					fi
 				;;
 				RETRIEVE)
 					# Retrieve from a location based on
@@ -179,9 +199,9 @@ while read -r line; do
 					calclocation "$key"
 					# http://stackoverflow.com/questions/31396985/why-is-mktemp-on-os-x-broken-with-a-command-that-worked-on-linux
 					if GA_RC_TEMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/rclone-annex-tmp.XXXXXXXXX") &&
-					    runcmd rclone copy "$LOC$key" $GA_RC_TEMP_DIR &&
-					    mv $GA_RC_TEMP_DIR/$key $file &&
-					    rmdir $GA_RC_TEMP_DIR; then
+						runcmd rclone copy "$LOC$key" "$GA_RC_TEMP_DIR" &&
+						mv "$GA_RC_TEMP_DIR/$key" "$file" &&
+						rmdir "$GA_RC_TEMP_DIR"; then
 						echo TRANSFER-SUCCESS RETRIEVE "$key"
 					else
 						echo TRANSFER-FAILURE RETRIEVE "$key"
@@ -197,14 +217,13 @@ while read -r line; do
 			# Some rclone backends support multiple
 			# files under one name.
 			if check_result=$(rclone size "$LOC$key" 2>&1) &&
-			    echo $check_result|grep -E 'Total objects: [1-9][0-9]* Total size' >&2 &&
-			    ! echo $check_result|grep 'Total size: 0 (0 bytes)' >&2; then
+				echo "$check_result" | GREP -E 'Total objects: [1-9][0-9]*\>'; then
 				echo CHECKPRESENT-SUCCESS "$key"
 			else
 				# rclone 1.29 used 'Total objects: 0'
 				# rclone 1.30 uses 'directory not found'
-				if echo $check_result|grep 'Total objects: 0' >&2 || 
-				   echo $check_result|grep ' directory not found' >&2; then 
+				if echo "$check_result" | GREP 'Total objects: 0' ||
+				   echo "$check_result" | GREP ' directory not found'; then
 					echo CHECKPRESENT-FAILURE "$key"
 				else
 					# When the directory does not exist,
@@ -222,18 +241,18 @@ while read -r line; do
 			# Note that it's not a failure to remove a
 			# key that is not present.
 			if remove_result=$(rclone delete --retries 1 "$LOC$key" 2>&1); then
-		                echo REMOVE-SUCCESS "$key"
-	                else
-	    		   	# rclone 1.29 used Failed to purge: Couldn't find directory:
+				echo REMOVE-SUCCESS "$key"
+			else
+				# rclone 1.29 used Failed to purge: Couldn't find directory:
 				# rclone 1.30 used no such file or directory
 				# rclone 1.33 uses directory not found
-		                if echo $remove_result | grep " Failed to purge: Couldn't find directory: " >&2 ||
-				   echo $remove_result | grep ' no such file or directory' ||
-				   echo $remove_result | grep ' directory not found' >&2
-				then   
+				if echo "$remove_result" | GREP " Failed to purge: Couldn't find directory: " ||
+					echo "$remove_result" | GREP ' no such file or directory' ||
+					echo "$remove_result" | GREP ' directory not found'
+				then
 					echo REMOVE-SUCCESS "$key"
-		                else
-                		        echo REMOVE-FAILURE "$key"
+				else
+					echo REMOVE-FAILURE "$key"
 				fi
 			fi
 		;;
@@ -243,7 +262,7 @@ while read -r line; do
 			# to say that any other request is unsupported.
 			echo UNSUPPORTED-REQUEST
 		;;
-	esac	
+	esac
 done
 
 
diff --git a/tests/all-in-one.sh b/tests/all-in-one.sh
new file mode 100755
index 0000000..f4a5663
--- /dev/null
+++ b/tests/all-in-one.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+
+cd "$(mktemp -d "${TMPDIR:-/tmp}/garr-XXXXXXX")"
+
+# recipe from https://stackoverflow.com/a/4024263/1265472
+# sort -V seems to work on OSX yoh has access to.
+verlte() {
+    [  "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]
+}
+
+verlt() {
+    [ "$1" = "$2" ] && return 1 || verlte $1 $2
+}
+
+
+set -eux
+
+# provide versioning information to possibly ease troubleshooting
+git annex version
+rclone --version
+
+export HOME=$PWD
+echo -e '[local]\ntype = local\nnounc =' > ~/.rclone.conf
+# to pacify git/git-annex
+git config --global user.name Me
+git config --global user.email me@example.com
+git config --global init.defaultBranch master
+
+git_annex_version=$(git annex version | awk '/git-annex version:/{print $3;}')
+
+# Prepare rclone remote local store
+mkdir rclone-local
+export RCLONE_PREFIX=$PWD/rclone-local
+
+git-annex version
+mkdir testrepo
+cd testrepo
+git init .
+git-annex init
+git-annex initremote GA-rclone-CI type=external externaltype=rclone target=local prefix=$RCLONE_PREFIX chunk=100MiB encryption=${GARR_TEST_ENCRYPTION:-shared} mac=HMACSHA512
+
+# Rudimentary test, spaces in the filename must be ok, 0 length files should be ok
+if verlte "10.20220525+git73" "$git_annex_version"; then
+	# Was fixed in 10.20220525-73-g13fc6a9b6
+	echo "I: Fixed git-annex $git_annex_version, adding empty file"
+	touch "test 0"
+fi
+
+echo 1 > "test 1"
+git-annex add *
+git-annex copy * --to GA-rclone-CI
+git-annex drop *
+git-annex get *
+
+# Do a cycle with --debug to ensure that we are passing desired DEBUG output
+git-annex --debug drop test\ 1 2>&1 | grep -q 'grep.*exited with rc='
+git-annex --debug get test\ 1 2>/dev/null
+
+# test copy/drop/get cycle with parallel execution and good number of files and spaces in the names, and duplicated content/keys
+set +x
+for f in `seq 1 20`; do echo "load $f" | tee "test-$f.dat" >| "test $f.dat"; done
+set -x
+git annex add -J5 --quiet .
+git-annex copy -J5 --quiet . --to GA-rclone-CI
+git-annex drop -J5 --quiet .
+git-annex get -J5 --quiet .
+git-annex drop --from GA-rclone-CI -J5 --quiet .
+
+# annex testremote --fast
+git-annex testremote GA-rclone-CI --fast

Debdiff

File lists identical (after any substitutions)

No differences were encountered in the control files

More details

Full run details