New Upstream Release - git-annex-remote-rclone
Ready changes
Summary
Merged new upstream version: 0.7 (was: 0.6).
Resulting package
Built on 2023-08-11T10:53 (took 10m2s)
The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:
apt install -t fresh-releases git-annex-remote-rclone
Lintian Result
Diff
diff --git a/.github/workflows/shellcheck.yml b/.github/workflows/shellcheck.yml
new file mode 100644
index 0000000..9db519c
--- /dev/null
+++ b/.github/workflows/shellcheck.yml
@@ -0,0 +1,23 @@
+name: Shellcheck
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 6 * * *'
+
+jobs:
+ shellcheck:
+ runs-on: ubuntu-18.04
+
+ steps:
+ - name: Set up environment
+ uses: actions/checkout@v1
+
+ - name: Install dependencies
+ run: |
+ sudo apt update && sudo apt install shellcheck
+
+ - name: shellcheck
+ run: |
+ shellcheck git-annex-remote-rclone
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
new file mode 100644
index 0000000..1521ccd
--- /dev/null
+++ b/.github/workflows/test.yml
@@ -0,0 +1,69 @@
+name: Tests
+
+on:
+ push:
+ pull_request:
+ schedule:
+ - cron: '0 6 * * *'
+
+jobs:
+ test:
+ runs-on: ${{ matrix.os }}
+
+ strategy:
+ fail-fast: false
+ matrix:
+ os:
+ - ubuntu-18.04
+ #- windows-2019
+ - macos-latest
+ rclone:
+ - current
+ - v1.58.1 # current "current" -- the baseline
+ - v1.53.3 # Debian bullseye (current stable)
+ - v1.45 # Debian buster (current oldstable)
+ # older versions for which there is internal logic
+ - v1.33
+ - v1.30
+ # - v1.29 TODO: seems to need rclone config run -- must be done in test script
+ exclude:
+ # 1.30 - --version exits non-0
+ - os: macos-latest
+ rclone: v1.30
+ # 1.45 - has some odd handling of HOME on OSX - does not use overloaded $HOME
+ - os: macos-latest
+ rclone: v1.45
+
+ steps:
+ - name: Set up environment
+ uses: actions/checkout@v1
+
+ - name: Install Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '>=3.7'
+
+ - name: Install datalad-installer
+ run: pip install datalad-installer
+
+ - name: Install git-annex (Ubuntu)
+ if: startsWith(matrix.os, 'ubuntu-')
+ run: datalad-installer --sudo ok git-annex -m datalad/git-annex:release
+
+ - name: Install git-annex (macOS)
+ if: startsWith(matrix.os, 'macos-')
+ run: datalad-installer git-annex -m brew
+
+ - name: Install latest rclone
+ if: matrix.rclone == 'current'
+ run: datalad-installer --sudo ok rclone -m downloads.rclone.org
+
+ - name: Install specific version of rclone
+ if: matrix.rclone != 'current'
+ run: datalad-installer --sudo ok rclone=${{ matrix.rclone }} -m downloads.rclone.org
+
+ - name: ${{ matrix.module }} tests
+ run: |
+ PATH=$PWD:$PATH tests/all-in-one.sh
+ # Test without encryption so we do get 0-sized files in rclone remote store
+ PATH=$PWD:$PATH GARR_TEST_ENCRYPTION=none tests/all-in-one.sh
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 00adb54..c2a8df5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,14 +12,4 @@ test:
- curl 'https://downloads.rclone.org/rclone-current-linux-amd64.zip' > rclone.zip
- unzip rclone.zip
- mv rclone-*-linux-amd64/rclone /usr/local/bin
- - echo -e '[local]\ntype = local\nnounc =' > ~/.rclone.conf
- - git-annex version
- - mkdir testrepo
- - cd testrepo
- - git init .
- - git-annex init
- - git-annex initremote GA-rclone-CI type=external externaltype=rclone target=local prefix=/tmp/GA-rclone-CI chunk=100MiB encryption=shared mac=HMACSHA512
- - touch test
- - git-annex add test
- - git-annex copy test --to GA-rclone-CI
- - git-annex testremote GA-rclone-CI --fast
+ - tests/all-in-one.sh
diff --git a/README.md b/README.md
index e210f6a..3f35b5d 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,8 @@ __Users are urged to upgrade to version 0.4 or higher immediately.
Some ways of using earlier versions could result in data loss. [more information here](https://github.com/DanielDent/git-annex-remote-rclone/issues/8)__
-[![build status](https://gitlab.com/DanielDent/git-annex-remote-rclone/badges/master/build.svg)](https://gitlab.com/DanielDent/git-annex-remote-rclone/commits/master)
+[![Tests](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/test.yml/badge.svg)](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/test.yml)
+[![Shellcheck](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/shellcheck.yml/badge.svg)](https://github.com/DanielDent/git-annex-remote-rclone/actions/workflows/shellcheck.yml)
This wrapper around [rclone](http://rclone.org/) makes any destination supported by rclone usable with git-annex.
@@ -123,4 +124,4 @@ Please submit a pull request or create a new issue for problems or potential imp
## License
-Copyright 2016-2017 [Daniel Dent](https://www.danieldent.com/). Licensed under the GPLv3.
+Copyright 2016-2022 [Daniel Dent](https://www.danieldent.com/), 2022 [git-annex-remote-rclone contributors](https://github.com/DanielDent/git-annex-remote-rclone/graphs/contributors). Licensed under the GPLv3.
diff --git a/debian/changelog b/debian/changelog
index cd1ef25..ff9ead6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+git-annex-remote-rclone (0.7-1) UNRELEASED; urgency=low
+
+ * New upstream release.
+
+ -- Debian Janitor <janitor@jelmer.uk> Fri, 11 Aug 2023 10:43:37 -0000
+
git-annex-remote-rclone (0.6-1) unstable; urgency=medium
* New minor bugfix upstream release
diff --git a/git-annex-remote-rclone b/git-annex-remote-rclone
index ad788fa..bde8de2 100755
--- a/git-annex-remote-rclone
+++ b/git-annex-remote-rclone
@@ -1,9 +1,13 @@
#!/bin/bash
+# emacs: -*- mode: python; tab-width: 4; indent-tabs-mode: t -*-
+# ex: set sts=4 ts=4 sw=4 noet:
+#
# git-annex-remote-rclone - wrapper to enable use of rclone-supported cloud providers as git-annex special remotes.
#
# Install in PATH as git-annex-remote-rclone
#
-# Copyright (C) 2016-2017 Daniel Dent
+# Copyright (C) 2016-2022 Daniel Dent
+# 2022 git-annex-remote-rclone contributors
#
# This program is free software: you can redistribute it and/or modify it under the terms of version 3 of the GNU
# General Public License as published by the Free Software Foundation.
@@ -44,7 +48,7 @@ validate_layout() {
echo "INITREMOTE-FAILURE rclone_layout setting not recognized"
exit 1
;;
- esac
+ esac
}
# Sets LOC to the location to use to store a key.
@@ -60,14 +64,14 @@ calclocation () {
;;
nodir)
LOC="$REMOTE_TARGET:$REMOTE_PREFIX/"
- ;;
+ ;;
mixed)
ask DIRHASH "$1"
LOC="$REMOTE_TARGET:$REMOTE_PREFIX/$RET"
;;
frankencase)
ask DIRHASH "$1"
- lret=$(echo $RET|tr A-Z a-z)
+ lret=$(echo "$RET" | tr '[:upper:]' '[:lower:]')
LOC="$REMOTE_TARGET:$REMOTE_PREFIX/$lret"
;;
esac
@@ -77,32 +81,46 @@ calclocation () {
ask () {
echo "$1" "$2"
read -r resp
- # Strip trailing carriage return, if present
- resp="${resp%$'\r'}"
- if echo $resp|grep '^VALUE '>/dev/null; then
- RET=$(echo "$resp" | cut -f2- -d' ')
- else
- RET=""
+ # Strip trailing carriage return, if present
+ resp="${resp%$'\r'}"
+ if echo "$resp" | grep '^VALUE '>/dev/null; then
+ RET=$(echo "$resp" | cut -f2- -d' ')
+ else
+ RET=""
fi
}
+GREP () {
+ set +e
+ out=$(grep "$@")
+ rc=$?
+ set -e
+ # Replace explicit newline since we must provide 1 line DEBUG
+ # Fancy sed is from Example 5 of https://linuxhint.com/newline_replace_sed
+ # which worked on Linux and OSX.
+ # shellcheck disable=SC2016
+ out_safe=$(echo "$out" | sed -ne 'H;${x;s/\n/\\n/g;s/^,//;p;}')
+ echo "DEBUG 'grep \"$*\"' exited with rc=$rc and stdout=${out_safe}"
+ exit $rc
+}
# This has to come first, to get the protocol started.
echo VERSION 1
while read -r line; do
- # Strip trailing carriage return, if present
- line="${line%$'\r'}"
+ # Strip trailing carriage return, if present
+ line="${line%$'\r'}"
+ # shellcheck disable=SC2086
set -- $line
case "$1" in
INITREMOTE)
# Do anything necessary to create resources
# used by the remote. Try to be idempotent.
- #
+ #
# Use GETCONFIG to get any needed configuration
# settings, and SETCONFIG to set any persistent
# configuration settings.
- #
+ #
# (Note that this is not run every time, only when
# git annex initremote or git annex enableremote is
# run.)
@@ -110,31 +128,31 @@ while read -r line; do
getconfig prefix
REMOTE_PREFIX=$RET
if [ -z "$REMOTE_PREFIX" ]; then
- REMOTE_PREFIX="git-annex"
+ REMOTE_PREFIX="git-annex"
fi
if [ "$REMOTE_PREFIX" == "/" ]; then
- echo INITREMOTE-FAILURE "storing objects directly in the root (/) is not supported"
+ echo INITREMOTE-FAILURE "storing objects directly in the root (/) is not supported"
fi
setconfig prefix $REMOTE_PREFIX
getconfig target
REMOTE_TARGET=$RET
- setconfig target $REMOTE_TARGET
-
+ setconfig target "$REMOTE_TARGET"
+
getconfig rclone_layout
RCLONE_LAYOUT=$RET
validate_layout
- setconfig rclone_layout $RCLONE_LAYOUT
+ setconfig rclone_layout "$RCLONE_LAYOUT"
if [ -z "$REMOTE_TARGET" ]; then
- echo INITREMOTE-FAILURE "rclone remote target must be specified (use target= parameter)"
+ echo INITREMOTE-FAILURE "rclone remote target must be specified (use target= parameter)"
fi
- if runcmd rclone mkdir $REMOTE_TARGET:$REMOTE_PREFIX; then
- echo INITREMOTE-SUCCESS
- else
- echo INITREMOTE-FAILURE "Failed to create directory on remote. Ensure that 'rclone config' has been run."
- fi
+ if runcmd rclone mkdir "$REMOTE_TARGET:$REMOTE_PREFIX"; then
+ echo INITREMOTE-SUCCESS
+ else
+ echo INITREMOTE-FAILURE "Failed to create directory on remote. Ensure that 'rclone config' has been run."
+ fi
;;
PREPARE)
# Use GETCONFIG to get configuration settings,
@@ -143,20 +161,22 @@ while read -r line; do
getconfig prefix
REMOTE_PREFIX="$RET"
-
+
getconfig target
REMOTE_TARGET="$RET"
-
+
getconfig rclone_layout
RCLONE_LAYOUT="$RET"
validate_layout
- echo PREPARE-SUCCESS
+ echo PREPARE-SUCCESS
;;
TRANSFER)
+ op="$2"
key="$3"
- file="$4"
- case "$2" in
+ shift 3
+ file="$*"
+ case "$op" in
STORE)
# Store the file to a location
# based on the key.
@@ -164,13 +184,13 @@ while read -r line; do
calclocation "$key"
if [ ! -e "$file" ]; then
echo TRANSFER-FAILURE STORE "$key" "asked to store non-existent file $file"
- else
+ else
if runcmd rclone copy "$file" "$LOC"; then
echo TRANSFER-SUCCESS STORE "$key"
else
echo TRANSFER-FAILURE STORE "$key"
fi
- fi
+ fi
;;
RETRIEVE)
# Retrieve from a location based on
@@ -179,9 +199,9 @@ while read -r line; do
calclocation "$key"
# http://stackoverflow.com/questions/31396985/why-is-mktemp-on-os-x-broken-with-a-command-that-worked-on-linux
if GA_RC_TEMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/rclone-annex-tmp.XXXXXXXXX") &&
- runcmd rclone copy "$LOC$key" $GA_RC_TEMP_DIR &&
- mv $GA_RC_TEMP_DIR/$key $file &&
- rmdir $GA_RC_TEMP_DIR; then
+ runcmd rclone copy "$LOC$key" "$GA_RC_TEMP_DIR" &&
+ mv "$GA_RC_TEMP_DIR/$key" "$file" &&
+ rmdir "$GA_RC_TEMP_DIR"; then
echo TRANSFER-SUCCESS RETRIEVE "$key"
else
echo TRANSFER-FAILURE RETRIEVE "$key"
@@ -197,14 +217,13 @@ while read -r line; do
# Some rclone backends support multiple
# files under one name.
if check_result=$(rclone size "$LOC$key" 2>&1) &&
- echo $check_result|grep -E 'Total objects: [1-9][0-9]* Total size' >&2 &&
- ! echo $check_result|grep 'Total size: 0 (0 bytes)' >&2; then
+ echo "$check_result" | GREP -E 'Total objects: [1-9][0-9]*\>'; then
echo CHECKPRESENT-SUCCESS "$key"
else
# rclone 1.29 used 'Total objects: 0'
# rclone 1.30 uses 'directory not found'
- if echo $check_result|grep 'Total objects: 0' >&2 ||
- echo $check_result|grep ' directory not found' >&2; then
+ if echo "$check_result" | GREP 'Total objects: 0' ||
+ echo "$check_result" | GREP ' directory not found'; then
echo CHECKPRESENT-FAILURE "$key"
else
# When the directory does not exist,
@@ -222,18 +241,18 @@ while read -r line; do
# Note that it's not a failure to remove a
# key that is not present.
if remove_result=$(rclone delete --retries 1 "$LOC$key" 2>&1); then
- echo REMOVE-SUCCESS "$key"
- else
- # rclone 1.29 used Failed to purge: Couldn't find directory:
+ echo REMOVE-SUCCESS "$key"
+ else
+ # rclone 1.29 used Failed to purge: Couldn't find directory:
# rclone 1.30 used no such file or directory
# rclone 1.33 uses directory not found
- if echo $remove_result | grep " Failed to purge: Couldn't find directory: " >&2 ||
- echo $remove_result | grep ' no such file or directory' ||
- echo $remove_result | grep ' directory not found' >&2
- then
+ if echo "$remove_result" | GREP " Failed to purge: Couldn't find directory: " ||
+ echo "$remove_result" | GREP ' no such file or directory' ||
+ echo "$remove_result" | GREP ' directory not found'
+ then
echo REMOVE-SUCCESS "$key"
- else
- echo REMOVE-FAILURE "$key"
+ else
+ echo REMOVE-FAILURE "$key"
fi
fi
;;
@@ -243,7 +262,7 @@ while read -r line; do
# to say that any other request is unsupported.
echo UNSUPPORTED-REQUEST
;;
- esac
+ esac
done
diff --git a/tests/all-in-one.sh b/tests/all-in-one.sh
new file mode 100755
index 0000000..f4a5663
--- /dev/null
+++ b/tests/all-in-one.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+
+cd "$(mktemp -d "${TMPDIR:-/tmp}/garr-XXXXXXX")"
+
+# recipe from https://stackoverflow.com/a/4024263/1265472
+# sort -V seems to work on OSX yoh has access to.
+verlte() {
+ [ "$1" = "`echo -e "$1\n$2" | sort -V | head -n1`" ]
+}
+
+verlt() {
+ [ "$1" = "$2" ] && return 1 || verlte $1 $2
+}
+
+
+set -eux
+
+# provide versioning information to possibly ease troubleshooting
+git annex version
+rclone --version
+
+export HOME=$PWD
+echo -e '[local]\ntype = local\nnounc =' > ~/.rclone.conf
+# to pacify git/git-annex
+git config --global user.name Me
+git config --global user.email me@example.com
+git config --global init.defaultBranch master
+
+git_annex_version=$(git annex version | awk '/git-annex version:/{print $3;}')
+
+# Prepare rclone remote local store
+mkdir rclone-local
+export RCLONE_PREFIX=$PWD/rclone-local
+
+git-annex version
+mkdir testrepo
+cd testrepo
+git init .
+git-annex init
+git-annex initremote GA-rclone-CI type=external externaltype=rclone target=local prefix=$RCLONE_PREFIX chunk=100MiB encryption=${GARR_TEST_ENCRYPTION:-shared} mac=HMACSHA512
+
+# Rudimentary test, spaces in the filename must be ok, 0 length files should be ok
+if verlte "10.20220525+git73" "$git_annex_version"; then
+ # Was fixed in 10.20220525-73-g13fc6a9b6
+ echo "I: Fixed git-annex $git_annex_version, adding empty file"
+ touch "test 0"
+fi
+
+echo 1 > "test 1"
+git-annex add *
+git-annex copy * --to GA-rclone-CI
+git-annex drop *
+git-annex get *
+
+# Do a cycle with --debug to ensure that we are passing desired DEBUG output
+git-annex --debug drop test\ 1 2>&1 | grep -q 'grep.*exited with rc='
+git-annex --debug get test\ 1 2>/dev/null
+
+# test copy/drop/get cycle with parallel execution and good number of files and spaces in the names, and duplicated content/keys
+set +x
+for f in `seq 1 20`; do echo "load $f" | tee "test-$f.dat" >| "test $f.dat"; done
+set -x
+git annex add -J5 --quiet .
+git-annex copy -J5 --quiet . --to GA-rclone-CI
+git-annex drop -J5 --quiet .
+git-annex get -J5 --quiet .
+git-annex drop --from GA-rclone-CI -J5 --quiet .
+
+# annex testremote --fast
+git-annex testremote GA-rclone-CI --fast
Debdiff
File lists identical (after any substitutions)
No differences were encountered in the control files