Codebase list s3cmd / upstream/2.0.2+git20190522.ae6cdde
Import upstream version 2.0.2+git20190522.ae6cdde, md5 8b129962cdfe23de7db44d7b04887426 Debian Janitor 4 years ago
31 changed file(s) with 3912 addition(s) and 175 deletion(s). Raw diff Collapse all Expand all
+10
-0
.gitignore less more
0 *.pyc
1 *.swp
2 testsuite
3 testsuite-out
4 /MANIFEST
5 /dist
6 build/*
7 s3cmd.egg-info
8 s3cmd.spec
9 .idea
+8
-0
.svnignore less more
0 ## Run 'svn propset svn:ignore -F .svnignore .' after you change this list
1 *.pyc
2 tst.*
3 MANIFEST
4 dist
5 build
6 .*.swp
7 s3cmd.1.gz
+75
-0
.travis.s3cfg less more
0 [default]
1 access_key = Q3AM3UQ867SPQQA43P2F
2 access_token =
3 add_encoding_exts =
4 add_headers =
5 bucket_location = us-east-1
6 ca_certs_file =
7 cache_file =
8 check_ssl_certificate = True
9 check_ssl_hostname = True
10 cloudfront_host = cloudfront.amazonaws.com
11 default_mime_type = binary/octet-stream
12 delay_updates = False
13 delete_after = False
14 delete_after_fetch = False
15 delete_removed = False
16 dry_run = False
17 enable_multipart = True
18 encoding = UTF-8
19 encrypt = False
20 expiry_date =
21 expiry_days =
22 expiry_prefix =
23 follow_symlinks = False
24 force = False
25 get_continue = False
26 gpg_command = /usr/bin/gpg
27 gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
28 gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
29 gpg_passphrase =
30 guess_mime_type = True
31 host_base = localhost:9000
32 host_bucket = localhost:9000
33 human_readable_sizes = False
34 invalidate_default_index_on_cf = False
35 invalidate_default_index_root_on_cf = True
36 invalidate_on_cf = False
37 kms_key =
38 limit = -1
39 limitrate = 0
40 list_md5 = False
41 log_target_prefix =
42 long_listing = False
43 max_delete = -1
44 mime_type =
45 multipart_chunk_size_mb = 15
46 multipart_max_chunks = 10000
47 preserve_attrs = True
48 progress_meter = True
49 proxy_host =
50 proxy_port = 0
51 put_continue = False
52 recursive = False
53 recv_chunk = 65536
54 reduced_redundancy = False
55 requester_pays = False
56 restore_days = 1
57 secret_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
58 send_chunk = 65536
59 server_side_encryption = False
60 signature_v2 = False
61 simpledb_host = sdb.amazonaws.com
62 skip_existing = False
63 socket_timeout = 300
64 stats = False
65 stop_on_error = False
66 storage_class =
67 urlencoding_mode = normal
68 use_http_expect = False
69 use_https = False
70 use_mime_magic = True
71 verbosity = WARNING
72 website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
73 website_error =
74 website_index = index.html
+38
-0
.travis.yml less more
0 language: python
1 python:
2 - "2.6"
3 - "2.7"
4 - "3.4"
5 - "3.5"
6 - "3.6"
7 #matrix:
8 # allow_failures:
9 # - python: "3.5"
10 notifications:
11 email: false
12 irc: "chat.freenode.net#s3cmd"
13 # command to install dependencies
14 cache:
15 directories:
16 - $HOME/cache
17 install:
18 - pip install .
19 # command to prepare tests
20 before_install:
21 - mkdir -p $HOME/cache
22 - test ! -e $HOME/cache/minio && wget -O $HOME/cache/minio https://dl.minio.io/server/minio/release/linux-amd64/minio || echo "Minio already in cache"
23 - mkdir -p $HOME/minio_tmp
24 # Start a local instance of minio
25 before_script:
26 - "export AWS_ACCESS_KEY_ID=Q3AM3UQ867SPQQA43P2F"
27 - "export AWS_SECRET_ACCESS_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
28 - "export MINIO_ACCESS_KEY=Q3AM3UQ867SPQQA43P2F"
29 - "export MINIO_SECRET_KEY=zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
30 - chmod +x $HOME/cache/minio
31 - $HOME/cache/minio server $HOME/minio_tmp &
32 - sleep 4 # give minio some time to start
33 # command to run tests
34 ## Tests stopped at test 23 because minio doesn't support "quote_plus" used in signatures.
35 script: python ./run-tests-minio.py -c .travis.s3cfg -p baseauto
36 after_script:
37 - killall minio
+1459
-0
ChangeLog less more
0 2011-06-06 Michal Ludvig <mludvig@logix.net.nz>
1
2 ===== Migrated to GIT =====
3
4 No longer keeping ChangeLog up to date, use git log instead!
5
6 * git://github.com/s3tools/s3cmd.git
7
8 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
9
10 * S3/S3Uri.py: Fixed cf:// uri parsing.
11 * S3/CloudFront.py: Don't fail if there are no cfinval
12 requests.
13
14 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
15
16 * S3/PkgInfo.py: Updated to 1.1.0-beta1
17 * NEWS: Updated.
18 * s3cmd.1: Regenerated.
19
20 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
21
22 * S3/Config.py: Increase socket_timeout from 10 secs to 5 mins.
23
24 2011-04-10 Michal Ludvig <mludvig@logix.net.nz>
25
26 * s3cmd, S3/CloudFront.py, S3/S3Uri.py: Support for checking
27 status of CF Invalidation Requests [cfinvalinfo].
28 * s3cmd, S3/CloudFront.py, S3/Config.py: Support for CloudFront
29 invalidation using [sync --cf-invalidate] command.
30 * S3/Utils.py: getDictFromTree() now recurses into
31 sub-trees.
32
33 2011-03-30 Michal Ludvig <mludvig@logix.net.nz>
34
35 * S3/CloudFront.py: Fix warning with Python 2.7
36 * S3/CloudFront.py: Cmd._get_dist_name_for_bucket() moved to
37 CloudFront class.
38
39 2011-01-13 Michal Ludvig <mludvig@logix.net.nz>
40
41 * s3cmd, S3/FileLists.py: Move file/object listing functions
42 to S3/FileLists.py
43
44 2011-01-09 Michal Ludvig <mludvig@logix.net.nz>
45
46 * Released version 1.0.0
47 ----------------------
48
49 * S3/PkgInfo.py: Updated to 1.0.0
50 * NEWS: Updated.
51
52 2011-01-02 Michal Ludvig <mludvig@logix.net.nz>
53
54 * s3cmd: Improved r457 (Don't crash when file disappears
55 before checking MD5).
56 * s3cmd, s3cmd.1, format-manpage.pl: Improved --help text
57 and manpage.
58 * s3cmd: Removed explicit processing of --follow-symlinks
59 (is cought by the default / main loop).
60
61 2010-12-24 Michal Ludvig <mludvig@logix.net.nz>
62
63 * s3cmd: Set 10s socket timeout for read()/write().
64 * s3cmd: Added --(no-)check-md5 for [sync].
65 * run-tests.py, testsuite.tar.gz: Added testsuite for
66 the above.
67 * NEWS: Document the above.
68 * s3cmd: Don't crash when file disappears before
69 checking MD5.
70
71 2010-12-09 Michal Ludvig <mludvig@logix.net.nz>
72
73 * Released version 1.0.0-rc2
74 --------------------------
75
76 * S3/PkgInfo.py: Updated to 1.0.0-rc2
77 * NEWS, TODO, s3cmd.1: Updated.
78
79 2010-11-13 Michal Ludvig <mludvig@logix.net.nz>
80
81 * s3cmd: Added support for remote-to-remote sync.
82 (Based on patch from Sundar Raman - thanks!)
83 * run-tests.py: Testsuite for the above.
84
85 2010-11-12 Michal Ludvig <mludvig@logix.net.nz>
86
87 * s3cmd: Fixed typo in "s3cmd du" error path.
88
89 2010-11-12 Michal Ludvig <mludvig@logix.net.nz>
90
91 * format-manpage.pl: new manpage auto-formatter
92 * s3cmd.1: Updated using the above helper script
93 * setup.py: Warn if manpage is too old.
94
95 2010-10-27 Michal Ludvig <mludvig@logix.net.nz>
96
97 * run-tests.py, testsuite.tar.gz: Keep the testsuite in
98 SVN as a tarball. There's too many "strange" things
99 in the directory for it to be kept in SVN.
100
101 2010-10-27 Michal Ludvig <mludvig@logix.net.nz>
102
103 * TODO: Updated.
104 * upload-to-sf.sh: Updated for new SF.net system
105
106 2010-10-26 Michal Ludvig <mludvig@logix.net.nz>
107
108 * Released version 1.0.0-rc1
109 --------------------------
110
111 * S3/PkgInfo.py: Updated to 1.0.0-rc1
112 * NEWS, TODO: Updated.
113
114 2010-10-26 Michal Ludvig <mludvig@logix.net.nz>
115
116 * s3cmd, S3/CloudFront.py, S3/Config.py: Added support
117 for CloudFront DefaultRootObject. Thanks to Luke Andrew.
118
119 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
120
121 * s3cmd: Improved 'fixbucket' command. Thanks to Srinivasa
122 Moorthy.
123 * s3cmd: Read config file even if User Profile directory on
124 Windows contains non-ascii symbols. Thx Slava Vishnyakov
125
126 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
127
128 * s3cmd: Don't fail when a local node is a directory
129 and we expected a file. (as if for example /etc/passwd
130 was a dir)
131
132 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
133
134 * s3cmd, S3/S3.py: Ignore inaccessible (and missing) files
135 on upload.
136 * run-tests.py: Extended [sync] test to verify correct
137 handling of inaccessible files.
138 * testsuite/permission-tests: New testsuite files.
139
140 2010-10-24 Michal Ludvig <mludvig@logix.net.nz>
141
142 * S3/S3.py: "Stringify" all headers. Httplib should do
143 it but some Python 2.7 users reported problems that should
144 now be fixed.
145 * run-tests.py: Fixed test #6
146
147 2010-07-25 Aaron Maxwell <amax@resymbol.net>
148
149 * S3/Config.py, testsuite/etc/, run-tests.py, s3cmd.1, s3cmd:
150 Option to follow local symlinks for sync and
151 put (--follow-symlinks option), including tests and documentation
152 * run-tests.py: --bucket-prefix option, to allow different
153 developers to run tests in their own sandbox
154
155 2010-07-08 Michal Ludvig <mludvig@logix.net.nz>
156
157 * run-tests.py, testsuite/crappy-file-name.tar.gz:
158 Updated testsuite, work around a problem with [s3cmd cp]
159 when the source file contains '?' or '\x7f'
160 (where the inability to copy '?' is especially annoying).
161
162 2010-07-08 Michal Ludvig <mludvig@logix.net.nz>
163
164 * S3/Utils.py, S3/S3Uri.py: Fixed names after moving
165 functions between modules.
166
167 2010-06-29 Timothee Groleau <kde@timotheegroleau.com>
168
169 * S3/ACL.py: Fix isAnonRead method on Grantees
170 * ChangeLog: Update name of contributor for Timothee Groleau
171
172 2010-06-13 Michal Ludvig <mludvig@logix.net.nz>
173
174 * s3cmd, S3/CloudFront.py: Both [accesslog] and [cfmodify]
175 access logging can now be disabled with --no-access-logging
176
177 2010-06-13 Michal Ludvig <mludvig@logix.net.nz>
178
179 * S3/CloudFront.py: Allow s3:// URI as well as cf:// URI
180 for most CloudFront-related commands.
181
182 2010-06-12 Michal Ludvig <mludvig@logix.net.nz>
183
184 * s3cmd, S3/CloudFront.py, S3/Config.py: Support access
185 logging for CloudFront distributions.
186 * S3/S3.py, S3/Utils.py: Moved some functions to Utils.py
187 to make them available to CloudFront.py
188 * NEWS: Document the above.
189
190 2010-05-27 Michal Ludvig <mludvig@logix.net.nz>
191
192 * S3/S3.py: Fix bucket listing for buckets with
193 over 1000 prefixes. (contributed by Timothee Groleau)
194 * S3/S3.py: Fixed code formating.
195
196 2010-05-21 Michal Ludvig <mludvig@logix.net.nz>
197
198 * s3cmd, S3/S3.py: Added support for bucket locations
199 outside US/EU (i.e. us-west-1 and ap-southeast-1 as of now).
200
201 2010-05-21 Michal Ludvig <mludvig@logix.net.nz>
202
203 * s3cmd, S3/S3.py, S3/Config.py: Added --reduced-redundancy
204 switch for Reduced Redundancy Storage.
205
206 2010-05-20 Michal Ludvig <mludvig@logix.net.nz>
207
208 * s3cmd, S3/ACL.py, S3/Config.py: Support for --acl-grant
209 and --acl-revoke (contributed by Timothee Groleau)
210 * s3cmd: Couple of fixes on top of the above commit.
211 * s3cmd: Pre-parse ACL parameters in OptionS3ACL()
212
213 2010-05-20 Michal Ludvig <mludvig@logix.net.nz>
214
215 * S3/Exceptions.py, S3/S3.py: Some HTTP_400 exceptions
216 are retriable.
217
218 2010-03-19 Michal Ludvig <mludvig@logix.net.nz>
219
220 * s3cmd, S3/ACL.py: Print all ACLs for a Grantee
221 (one Grantee can have multiple different Grant entries)
222
223 2010-03-19 Michal Ludvig <mludvig@logix.net.nz>
224
225 * s3cmd: Enable bucket-level ACL setting
226 * s3cmd, S3/AccessLog.py, ...: Added [accesslog] command.
227 * s3cmd: Fix imports from S3.Utils
228
229 2009-12-10 Michal Ludvig <mludvig@logix.net.nz>
230
231 * s3cmd: Path separator conversion on Windows hosts.
232
233 2009-10-08 Michal Ludvig <mludvig@logix.net.nz>
234
235 * Released version 0.9.9.91
236 -------------------------
237
238 * S3/PkgInfo.py: Updated to 0.9.9.91
239 * NEWS: News for 0.9.9.91
240
241 2009-10-08 Michal Ludvig <mludvig@logix.net.nz>
242
243 * S3/S3.py: fixed reference to _max_retries.
244
245 2009-10-06 Michal Ludvig <mludvig@logix.net.nz>
246
247 * Released version 0.9.9.90
248 -------------------------
249
250 * S3/PkgInfo.py: Updated to 0.9.9.90
251 * NEWS: News for 0.9.9.90
252
253 2009-10-06 Michal Ludvig <mludvig@logix.net.nz>
254
255 * S3/S3.py: Introduce throttling on upload only after
256 second failure. I.e. first retry at full speed.
257 * TODO: Updated with new ideas.
258
259 2009-06-02 Michal Ludvig <michal@logix.cz>
260
261 * s3cmd: New [fixbucket] command for fixing invalid object
262 names in a given Bucket. For instance names with &#x08; in
263 them (not sure how people manage to upload them but they do).
264 * S3/S3.py, S3/Utils.py, S3/Config.py: Support methods for
265 the above, plus advise user to run 'fixbucket' when XML parsing
266 fails.
267 * NEWS: Updated.
268
269 2009-05-29 Michal Ludvig <michal@logix.cz>
270
271 * S3/Utils.py: New function replace_nonprintables()
272 * s3cmd: Filter local filenames through the above function
273 to avoid problems with uploaded filenames containing invalid
274 XML entities, eg &#08;
275 * S3/S3.py: Warn if a non-printables char is passed to
276 urlencode_string() - they should have been replaced earlier
277 in the processing.
278 * run-tests.py, TODO, NEWS: Updated.
279 * testsuite/crappy-file-name.tar.gz: Tarball with a crappy-named
280 file. Untar for the testsuite.
281
282 2009-05-29 Michal Ludvig <michal@logix.cz>
283
284 * testsuite/blahBlah/*: Added files needed for run-tests.py
285
286 2009-05-28 Michal Ludvig <michal@logix.cz>
287
288 * S3/Utils.py (dateS3toPython): Be more relaxed about
289 timestamps format.
290
291 2009-05-28 Michal Ludvig <michal@logix.cz>
292
293 * s3cmd, run-test.py, TODO, NEWS: Added --dry-run
294 and --exclude/--include for [setacl].
295 * s3cmd, run-test.py, TODO, NEWS: Added --dry-run
296 and --exclude/--include for [del].
297
298 2009-05-28 Michal Ludvig <michal@logix.cz>
299
300 * s3cmd: Support for recursive [cp] and [mv], including
301 multiple-source arguments, --include/--exclude,
302 --dry-run, etc.
303 * run-tests.py: Tests for the above.
304 * S3/S3.py: Preserve metadata (eg ACL or MIME type)
305 during [cp] and [mv].
306 * NEWS, TODO: Updated.
307
308 2009-05-28 Michal Ludvig <michal@logix.cz>
309
310 * run-tests.py: Added --verbose mode.
311
312 2009-05-27 Michal Ludvig <michal@logix.cz>
313
314 * NEWS: Added info about --verbatim.
315 * TODO: Added more tasks.
316
317 2009-05-27 Michal Ludvig <michal@logix.cz>
318
319 * S3/SortedDict.py: Add case-sensitive mode.
320 * s3cmd, S3/S3.py, S3/Config.py: Use SortedDict() in
321 case-sensitive mode to avoid dropping filenames
322 differing only in capitalisation
323 * run-tests.py: Testsuite for the above.
324 * NEWS: Updated.
325
326 2009-03-20 Michal Ludvig <michal@logix.cz>
327
328 * S3/S3.py: Re-sign requests before retrial to avoid
329 RequestTimeTooSkewed errors on failed long-running
330 uploads.
331 BTW 'request' now has its own class S3Request.
332
333 2009-03-04 Michal Ludvig <michal@logix.cz>
334
335 * s3cmd, S3/Config.py, S3/S3.py: Support for --verbatim.
336
337 2009-02-25 Michal Ludvig <michal@logix.cz>
338
339 * s3cmd: Fixed "put file.ext s3://bkt" (ie just the bucket name).
340 * s3cmd: Fixed reporting of ImportError of S3 modules.
341 * s3cmd: Fixed Error: global name 'real_filename' is not defined
342
343 2009-02-24 Michal Ludvig <michal@logix.cz>
344
345 * s3cmd: New command [sign]
346 * S3/Utils.py: New function sign_string()
347 * S3/S3.py, S3/CloudFront.py: Use sign_string().
348 * NEWS: Updated.
349
350 2009-02-17 Michal Ludvig <michal@logix.cz>
351
352 * Released version 0.9.9
353 ----------------------
354
355 * S3/PkgInfo.py: Updated to 0.9.9
356 * NEWS: Compile a big news list for 0.9.9
357
358 2009-02-17 Michal Ludvig <michal@logix.cz>
359
360 * s3cmd.1: Document all the new options and commands.
361 * s3cmd, S3/Config.py: Updated some help texts. Removed
362 option --debug-syncmatch along the way (because --dry-run
363 with --debug is good enough).
364 * TODO: Updated.
365
366 2009-02-16 Michal Ludvig <michal@logix.cz>
367
368 * s3cmd: Check Python version >= 2.4 as soon as possible.
369
370 2009-02-14 Michal Ludvig <michal@logix.cz>
371
372 * s3cmd, S3/Config.py, S3/S3.py: Added --add-header option.
373 * NEWS: Documented --add-header.
374 * run-tests.py: Fixed for new messages.
375
376 2009-02-14 Michal Ludvig <michal@logix.cz>
377
378 * README: Updated for 0.9.9
379 * s3cmd, S3/PkgInfo.py, s3cmd.1: Replaced project
380 URLs with http://s3tools.org
381 * NEWS: Improved message.
382
383 2009-02-12 Michal Ludvig <michal@logix.cz>
384
385 * s3cmd: Added --list-md5 for 'ls' command.
386 * S3/Config.py: New setting list_md5
387
388 2009-02-12 Michal Ludvig <michal@logix.cz>
389
390 * s3cmd: Set Content-Length header for requests with 'body'.
391 * s3cmd: And send it for requests with no body as well...
392
393 2009-02-02 Michal Ludvig <michal@logix.cz>
394
395 * Released version 0.9.9-rc3
396 --------------------------
397
398 * S3/PkgInfo.py, NEWS: Updated for 0.9.9-rc3
399
400 2009-02-01 Michal Ludvig <michal@logix.cz>
401
402 * S3/Exceptions.py: Correct S3Exception.__str__() to
403 avoid crash in S3Error() subclass. Reported by '~t2~'.
404 * NEWS: Updated.
405
406 2009-01-30 Michal Ludvig <michal@logix.cz>
407
408 * Released version 0.9.9-rc2
409 --------------------------
410
411 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-rc2
412
413 2009-01-30 Michal Ludvig <michal@logix.cz>
414
415 * s3cmd: Under some circumstance s3cmd crashed
416 when put/get/sync had 0 files to transmit. Fixed now.
417
418 2009-01-28 Michal Ludvig <michal@logix.cz>
419
420 * s3cmd: Output 'delete:' in --dry-run only when
421 used together with --delete-removed. Otherwise
422 the user will think that without --dry-run it
423 would really delete the files.
424
425 2009-01-27 Michal Ludvig <michal@logix.cz>
426
427 * Released version 0.9.9-rc1
428 --------------------------
429
430 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-rc1
431
432 2009-01-26 Michal Ludvig <michal@logix.cz>
433
434 * Merged CloudFront support from branches/s3cmd-airlock
435 See the ChangeLog in that branch for details.
436
437 2009-01-25 W. Tell <w_tell -at- sourceforge>
438
439 * s3cmd: Implemented --include and friends.
440
441 2009-01-25 Michal Ludvig <michal@logix.cz>
442
443 * s3cmd: Enabled --dry-run and --exclude for 'put' and 'get'.
444 * S3/Exceptions.py: Remove DeprecationWarning about
445 BaseException.message in Python 2.6
446 * s3cmd: Rewritten gpg_command() to use subprocess.Popen()
447 instead of os.popen4() deprecated in 2.6
448 * TODO: Note about failing GPG.
449
450 2009-01-22 Michal Ludvig <michal@logix.cz>
451
452 * S3/Config.py: guess_mime_type = True (will affect new
453 installations only).
454
455 2009-01-22 Michal Ludvig <michal@logix.cz>
456
457 * Released version 0.9.9-pre5
458 ---------------------------
459
460 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-pre5
461
462 2009-01-22 Michal Ludvig <michal@logix.cz>
463
464 * run-tests.py: Updated paths for the new sync
465 semantics.
466 * s3cmd, S3/S3.py: Small fixes to make testsuite happy.
467
468 2009-01-21 Michal Ludvig <michal@logix.cz>
469
470 * s3cmd: Migrated 'sync' local->remote to the new
471 scheme with fetch_{local,remote}_list().
472 Enabled --dry-run for 'sync'.
473
474 2009-01-20 Michal Ludvig <michal@logix.cz>
475
476 * s3cmd: Migrated 'sync' remote->local to the new
477 scheme with fetch_{local,remote}_list().
478 Changed fetch_remote_list() to return dict() compatible
479 with fetch_local_list().
480 Re-implemented --exclude / --include processing.
481 * S3/Utils.py: functions for parsing RFC822 dates (for HTTP
482 header responses).
483 * S3/Config.py: placeholders for --include.
484
485 2009-01-15 Michal Ludvig <michal@logix.cz>
486
487 * s3cmd, S3/S3Uri.py, NEWS: Support for recursive 'put'.
488
489 2009-01-13 Michal Ludvig <michal@logix.cz>
490
491 * TODO: Updated.
492 * s3cmd: renamed (fetch_)remote_keys to remote_list and
493 a few other renames for consistency.
494
495 2009-01-08 Michal Ludvig <michal@logix.cz>
496
497 * S3/S3.py: Some errors during file upload were incorrectly
498 interpreted as MD5 mismatch. (bug #2384990)
499 * S3/ACL.py: Move attributes from class to instance.
500 * run-tests.py: Tests for ACL.
501 * s3cmd: Minor messages changes.
502
503 2009-01-07 Michal Ludvig <michal@logix.cz>
504
505 * s3cmd: New command 'setacl'.
506 * S3/S3.py: Implemented set_acl().
507 * S3/ACL.py: Fill in <Owner/> tag in ACL XML.
508 * NEWS: Info about 'setacl'.
509
510 2009-01-07 Michal Ludvig <michal@logix.cz>
511
512 * s3cmd: Factored remote_keys generation from cmd_object_get()
513 to fetch_remote_keys().
514 * s3cmd: Display Public URL in 'info' for AnonRead objects.
515 * S3/ACL.py: Generate XML from a current list of Grantees
516
517 2009-01-07 Michal Ludvig <michal@logix.cz>
518
519 * S3/ACL.py: Keep ACL internally as a list of of 'Grantee' objects.
520 * S3/Utils.py: Fix crash in stripNameSpace() when the XML has no NS.
521
522 2009-01-07 Michal Ludvig <michal@logix.cz>
523
524 * S3/ACL.py: New object for handling ACL issues.
525 * S3/S3.py: Moved most of S3.get_acl() to ACL class.
526 * S3/Utils.py: Reworked XML helpers - remove XMLNS before
527 parsing the input XML to avoid having all Tags prefixed
528 with {XMLNS} by ElementTree.
529
530 2009-01-03 Michal Ludvig <michal@logix.cz>
531
532 * s3cmd: Don't fail when neither $HOME nor %USERPROFILE% is set.
533 (fixes #2483388)
534
535 2009-01-01 W. Tell <w_tell -at- sourceforge>
536
537 * S3/S3.py, S3/Utils.py: Use 'hashlib' instead of md5 and sha
538 modules to avoid Python 2.6 warnings.
539
540 2008-12-31 Michal Ludvig <michal@logix.cz>
541
542 * Released version 0.9.9-pre4
543 ---------------------------
544
545 2008-12-31 Michal Ludvig <michal@logix.cz>
546
547 * s3cmd: Reworked internal handling of unicode vs encoded filenames.
548 Should replace unknown characters with '?' instead of baling out.
549
550 2008-12-31 Michal Ludvig <michal@logix.cz>
551
552 * run-tests.py: Display system encoding in use.
553 * s3cmd: Print a nice error message when --exclude-from
554 file is not readable.
555 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre4
556 * S3/Exceptions.py: Added missing imports.
557 * NEWS: Updated.
558 * testsuite: reorganised UTF-8 files, added GBK encoding files,
559 moved encoding-specific files to 'tar.gz' archives, removed
560 unicode dir.
561 * run-tests.py: Adapted to the above change.
562 * run-tests.sh: removed.
563 * testsuite/exclude.encodings: Added.
564 * run-tests.py: Don't assume utf-8, use preferred encoding
565 instead.
566 * s3cmd, S3/Utils.py, S3/Exceptions.py, S3/Progress.py,
567 S3/Config.py, S3/S3.py: Added --encoding switch and
568 Config.encoding variable. Don't assume utf-8 for filesystem
569 and terminal output anymore.
570 * s3cmd: Avoid ZeroDivisionError on fast links.
571 * s3cmd: Unicodised all info() output.
572
573 2008-12-30 Michal Ludvig <michal@logix.cz>
574
575 * s3cmd: Replace unknown Unicode characters with '?'
576 to avoid UnicodeEncodeError's. Also make all output strings
577 unicode.
578 * run-tests.py: Exit on failed test. Fixed order of tests.
579
580 2008-12-29 Michal Ludvig <michal@logix.cz>
581
582 * TODO, NEWS: Updated
583 * s3cmd: Improved wildcard get.
584 * run-tests.py: Improved testsuite, added parameters support
585 to run only specified tests, cleaned up win/posix integration.
586 * S3/Exception.py: Python 2.4 doesn't automatically set
587 Exception.message.
588
589 2008-12-29 Michal Ludvig <michal@logix.cz>
590
591 * s3cmd, run-tests.py: Make it work on Windows.
592
593 2008-12-26 Michal Ludvig <michal@logix.cz>
594
595 * setup.cfg: Remove explicit install prefix. That should fix
596 Mac OS X and Windows "setup.py install" runs.
597
598 2008-12-22 Michal Ludvig <michal@logix.cz>
599
600 * s3cmd, S3/S3.py, S3/Progress.py: Display "[X of Y]"
601 in --progress mode.
602 * s3cmd, S3/Config.py: Implemented recursive [get].
603 Added --skip-existing option for [get] and [sync].
604
605 2008-12-17 Michal Ludvig <michal@logix.cz>
606
607 * TODO: Updated
608
609 2008-12-14 Michal Ludvig <michal@logix.cz>
610
611 * S3/Progress.py: Restructured import Utils to avoid import
612 conflicts.
613
614 2008-12-12 Michal Ludvig <michal@logix.cz>
615
616 * s3cmd: Better Exception output. Print sys.path on ImportError,
617 don't print backtrace on KeyboardInterrupt
618
619 2008-12-11 Michal Ludvig <michal@logix.cz>
620
621 * s3cmd: Support for multiple sources in 'get' command.
622
623 2008-12-10 Michal Ludvig <michal@logix.cz>
624
625 * TODO: Updated list.
626 * s3cmd: Don't display download/upload completed message
627 in --progress mode.
628 * S3/S3.py: Pass src/dst names down to Progress class.
629 * S3/Progress.py: added new class ProgressCR - apparently
630 ProgressANSI doesn't work on MacOS-X (and perhaps elsewhere).
631 * S3/Config.py: Default progress meter is now ProgressCR
632 * s3cmd: Updated email address for reporting bugs.
633
634 2008-12-02 Michal Ludvig <michal@logix.cz>
635
636 * s3cmd, S3/S3.py, NEWS: Support for (non-)recursive 'ls'
637
638 2008-12-01 Michal Ludvig <michal@logix.cz>
639
640 * Released version 0.9.9-pre3
641 ---------------------------
642
643 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre3
644
645 2008-12-01 Michal Ludvig <michal@logix.cz>
646
647 * run-tests.py: Added a lot of new tests.
648 * testsuite/etc/logo.png: New file.
649
650 2008-11-30 Michal Ludvig <michal@logix.cz>
651
652 * S3/S3.py: object_get() -- make start_position argument optional.
653
654 2008-11-29 Michal Ludvig <michal@logix.cz>
655
656 * s3cmd: Delete local files with "sync --delete-removed"
657
658 2008-11-25 Michal Ludvig <michal@logix.cz>
659
660 * s3cmd, S3/Progress.py: Fixed Unicode output in Progress meter.
661 * s3cmd: Fixed 'del --recursive' without prefix (i.e. all objects).
662 * TODO: Updated list.
663 * upload-to-sf.sh: Helper script.
664 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre2+svn
665
666 2008-11-24 Michal Ludvig <michal@logix.cz>
667
668 * Released version 0.9.9-pre2
669 ------------------------
670
671 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre2
672 * NEWS: Added 0.9.9-pre2
673
674 2008-11-24 Michal Ludvig <michal@logix.cz>
675
676 * s3cmd, s3cmd.1, S3/S3.py: Display or don't display progress meter
677 default depends on whether we're on TTY (console) or not.
678
679 2008-11-24 Michal Ludvig <michal@logix.cz>
680
681 * s3cmd: Fixed 'get' conflict.
682 * s3cmd.1, TODO: Document 'mv' command.
683
684 2008-11-24 Michal Ludvig <michal@logix.cz>
685
686 * S3/S3.py, s3cmd, S3/Config.py, s3cmd.1: Added --continue for
687 'get' command, improved 'get' failure resiliency.
688 * S3/Progress.py: Support for progress meter not starting in 0.
689 * S3/S3.py: improved retrying in send_request() and send_file()
690
691 2008-11-24 Michal Ludvig <michal@logix.cz>
692
693 * s3cmd, S3/S3.py, NEWS: "s3cmd mv" for moving objects
694
695 2008-11-24 Michal Ludvig <michal@logix.cz>
696
697 * S3/Utils.py: Common XML parser.
698 * s3cmd, S3/Exeptions.py: Print info message on Error.
699
700 2008-11-21 Michal Ludvig <michal@logix.cz>
701
702 * s3cmd: Support for 'cp' command.
703 * S3/S3.py: Added S3.object.copy() method.
704 * s3cmd.1: Document 'cp' command.
705 * NEWS: Let everyone know ;-)
706 Thanks Andrew Ryan for a patch proposal!
707 https://sourceforge.net/forum/forum.php?thread_id=2346987&forum_id=618865
708
709 2008-11-17 Michal Ludvig <michal@logix.cz>
710
711 * S3/Progress.py: Two progress meter implementations.
712 * S3/Config.py, s3cmd: New --progress / --no-progress parameters
713 and Config() members.
714 * S3/S3.py: Call Progress() in send_file()/recv_file()
715 * NEWS: Let everyone know ;-)
716
717 2008-11-16 Michal Ludvig <michal@logix.cz>
718
719 * NEWS: Fetch 0.9.8.4 release news from 0.9.8.x branch.
720
721 2008-11-16 Michal Ludvig <michal@logix.cz>
722
723 Merge from 0.9.8.x branch, rel 251:
724 * S3/S3.py: Adjusting previous commit (orig 249) - it's not a good idea
725 to retry ALL failures. Especially not those code=4xx where AmazonS3
726 servers are not happy with our requests.
727 Merge from 0.9.8.x branch, rel 249:
728 * S3/S3.py, S3/Exception.py: Re-issue failed requests in S3.send_request()
729 Merge from 0.9.8.x branch, rel 248:
730 * s3cmd: Don't leak open filehandles in sync. Thx Patrick Linskey for report.
731 Merge from 0.9.8.x branch, rel 247:
732 * s3cmd: Re-raise the right exception.
733 Merge from 0.9.8.x branch, rel 246:
734 * s3cmd, S3/S3.py, S3/Exceptions.py: Don't abort 'sync' or 'put' on files
735 that can't be open (e.g. Permision denied). Print a warning and skip over
736 instead.
737 Merge from 0.9.8.x branch, rel 245:
738 * S3/S3.py: Escape parameters in strings. Fixes sync to and
739 ls of directories with spaces. (Thx Lubomir Rintel from Fedora Project)
740 Merge from 0.9.8.x branch, rel 244:
741 * s3cmd: Unicode brainfuck again. This time force all output
742 in UTF-8, will see how many complaints we'll get...
743
744 2008-09-16 Michal Ludvig <michal@logix.cz>
745
746 * NEWS: s3cmd 0.9.8.4 released from branches/0.9.8.x SVN branch.
747
748 2008-09-16 Michal Ludvig <michal@logix.cz>
749
750 * S3/S3.py: Don't run into ZeroDivisionError when speed counter
751 returns 0s elapsed on upload/download file.
752
753 2008-09-15 Michal Ludvig <michal@logix.cz>
754
755 * s3cmd, S3/S3.py, S3/Utils.py, S3/S3Uri.py, S3/Exceptions.py:
756 Yet anoter Unicode round. Unicodised all command line arguments
757 before processing.
758
759 2008-09-15 Michal Ludvig <michal@logix.cz>
760
761 * S3/S3.py: "s3cmd mb" can create upper-case buckets again
762 in US. Non-US (e.g. EU) bucket names must conform to strict
763 DNS-rules.
764 * S3/S3Uri.py: Display public URLs correctly for non-DNS buckets.
765
766 2008-09-10 Michal Ludvig <michal@logix.cz>
767
768 * testsuite, run-tests.py: Added testsuite with first few tests.
769
770 2008-09-10 Michal Ludvig <michal@logix.cz>
771
772 * s3cmd, S3/S3Uri.py, S3/S3.py: All internal representations of
773 S3Uri()s are Unicode (i.e. not UTF-8 but type()==unicode). It
774 still doesn't work on non-UTF8 systems though.
775
776 2008-09-04 Michal Ludvig <michal@logix.cz>
777
778 * s3cmd: Rework UTF-8 output to keep sys.stdout untouched (or it'd
779 break 's3cmd get' to stdout for binary files).
780
781 2008-09-03 Michal Ludvig <michal@logix.cz>
782
783 * s3cmd, S3/S3.py, S3/Config.py: Removed --use-old-connect-method
784 again. Autodetect the need for old connect method instead.
785
786 2008-09-03 Michal Ludvig <michal@logix.cz>
787
788 * s3cmd, S3/S3.py: Make --verbose mode more useful and default
789 mode less verbose.
790
791 2008-09-03 Michal Ludvig <michal@logix.cz>
792
793 * s3cmd, S3/Config.py: [rb] Allow removal of non-empty buckets
794 with --force.
795 [mb, rb] Allow multiple arguments, i.e. create or remove
796 multiple buckets at once.
797 [del] Perform recursive removal with --recursive (or -r).
798
799 2008-09-01 Michal Ludvig <michal@logix.cz>
800
801 * s3cmd: Refuse 'sync' together with '--encrypt'.
802 * S3/S3.py: removed object_{get,put,delete}_uri() functions
803 and made object_{get,put,delete}() accept URI instead of
804 bucket/object parameters.
805
806 2008-09-01 Michal Ludvig <michal@logix.cz>
807
808 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre1
809
810 2008-09-01 Michal Ludvig <michal@logix.cz>
811
812 * s3cmd, S3/S3.py, S3/Config.py: Allow access to upper-case
813 named buckets again with --use-old-connect-method
814 (uses http://s3.amazonaws.com/bucket/object instead of
815 http://bucket.s3.amazonaws.com/object)
816
817 2008-08-19 Michal Ludvig <michal@logix.cz>
818
819 * s3cmd: Always output UTF-8, even on output redirects.
820
821 2008-08-01 Michal Ludvig <michal@logix.cz>
822
823 * TODO: Add some items
824
825 2008-07-29 Michal Ludvig <michal@logix.cz>
826
827 * Released version 0.9.8.3
828 ------------------------
829
830 2008-07-29 Michal Ludvig <michal@logix.cz>
831
832 * S3/PkgInfo.py: Bumped up version to 0.9.8.3
833 * NEWS: Added 0.9.8.3
834
835 2008-07-29 Michal Ludvig <michal@logix.cz>
836
837 * S3/Utils.py (hash_file_md5): Hash files in 32kB chunks
838 instead of reading it all up to a memory first to avoid
839 OOM on large files.
840
841 2008-07-07 Michal Ludvig <michal@logix.cz>
842
843 * s3cmd.1: couple of syntax fixes from Mikhail Gusarov
844
845 2008-07-03 Michal Ludvig <michal@logix.cz>
846
847 * Released version 0.9.8.2
848 ------------------------
849
850 2008-07-03 Michal Ludvig <michal@logix.cz>
851
852 * S3/PkgInfo.py: Bumped up version to 0.9.8.2
853 * NEWS: Added 0.9.8.2
854 * s3cmd: Print version info on 'unexpected error' output.
855
856 2008-06-30 Michal Ludvig <michal@logix.cz>
857
858 * S3/S3.py: Re-upload when Amazon doesn't send ETag
859 in PUT response. It happens from time to time for
860 unknown reasons. Thanks "Burtc" for report and
861 "hermzz" for fix.
862
863 2008-06-27 Michal Ludvig <michal@logix.cz>
864
865 * Released version 0.9.8.1
866 ------------------------
867
868 2008-06-27 Michal Ludvig <michal@logix.cz>
869
870 * S3/PkgInfo.py: Bumped up version to 0.9.8.1
871 * NEWS: Added 0.9.8.1
872 * s3cmd: make 'cfg' global
873 * run-tests.sh: Sort-of testsuite
874
875 2008-06-23 Michal Ludvig <michal@logix.cz>
876
877 * Released version 0.9.8
878 ----------------------
879
880 2008-06-23 Michal Ludvig <michal@logix.cz>
881
882 * S3/PkgInfo.py: Bumped up version to 0.9.8
883 * NEWS: Added 0.9.8
884 * TODO: Removed completed tasks
885
886 2008-06-23 Michal Ludvig <michal@logix.cz>
887
888 * s3cmd: Last-minute compatibility fixes for Python 2.4
889 * s3cmd, s3cmd.1: --debug-exclude is an alias for --debug-syncmatch
890 * s3cmd: Don't require $HOME env variable to be set.
891 Fixes #2000133
892 * s3cmd: Wrapped all execution in a try/except block
893 to catch all exceptions and ask for a report.
894
895 2008-06-18 Michal Ludvig <michal@logix.cz>
896
897 * S3/PkgInfo.py: Version 0.9.8-rc3
898
899 2008-06-18 Michal Ludvig <michal@logix.cz>
900
901 * S3/S3.py: Bucket name can't contain upper-case letters (S3/DNS limitation).
902
903 2008-06-12 Michal Ludvig <michal@logix.cz>
904
905 * S3/PkgInfo.py: Version 0.9.8-rc2
906
907 2008-06-12 Michal Ludvig <michal@logix.cz>
908
909 * s3cmd, s3cmd.1: Added GLOB (shell-style wildcard) exclude, renamed
910 orig regexp-style --exclude to --rexclude
911
912 2008-06-11 Michal Ludvig <michal@logix.cz>
913
914 * S3/PkgInfo.py: Version 0.9.8-rc1
915
916 2008-06-11 Michal Ludvig <michal@logix.cz>
917
918 * s3cmd: Remove python 2.5 specific code (try/except/finally
919 block) and make s3cmd compatible with python 2.4 again.
920 * s3cmd, S3/Config.py, s3cmd.1: Added --exclude-from and --debug-syncmatch
921 switches for sync.
922
923 2008-06-10 Michal Ludvig <michal@logix.cz>
924
925 * s3cmd: Added --exclude switch for sync.
926 * s3cmd.1, NEWS: Document --exclude
927
928 2008-06-05 Michal Ludvig <michal@logix.cz>
929
930 * Released version 0.9.7
931 ----------------------
932
933 2008-06-05 Michal Ludvig <michal@logix.cz>
934
935 * S3/PkgInfo.py: Bumped up version to 0.9.7
936 * NEWS: Added 0.9.7
937 * TODO: Removed completed tasks
938 * s3cmd, s3cmd.1: Updated help texts,
939 removed --dry-run option as it's not implemented.
940
941 2008-06-05 Michal Ludvig <michal@logix.cz>
942
943 * S3/Config.py: Store more file attributes in sync to S3.
944 * s3cmd: Make sync remote2local more error-resilient.
945
946 2008-06-04 Michal Ludvig <michal@logix.cz>
947
948 * s3cmd: Implemented cmd_sync_remote2local() for restoring
949 backup from S3 to a local filesystem
950 * S3/S3.py: S3.object_get_uri() now requires writable stream
951 and not a path name.
952 * S3/Utils.py: Added mkdir_with_parents()
953
954 2008-06-04 Michal Ludvig <michal@logix.cz>
955
956 * s3cmd: Refactored cmd_sync() in preparation
957 for remote->local sync.
958
959 2008-04-30 Michal Ludvig <michal@logix.cz>
960
961 * s3db, S3/SimpleDB.py: Implemented almost full SimpleDB API.
962
963 2008-04-29 Michal Ludvig <michal@logix.cz>
964
965 * s3db, S3/SimpleDB.py: Initial support for Amazon SimpleDB.
966 For now implements ListDomains() call and most of the
967 infrastructure required for request creation.
968
969 2008-04-29 Michal Ludvig <michal@logix.cz>
970
971 * S3/Exceptions.py: Exceptions moved out of S3.S3
972 * S3/SortedDict.py: rewritten from scratch to preserve
973 case of keys while still sorting in case-ignore mode.
974
975 2008-04-28 Michal Ludvig <michal@logix.cz>
976
977 * S3/S3.py: send_file() now computes MD5 sum of the file
978 being uploaded, compares with ETag returned by Amazon
979 and retries upload if they don't match.
980
981 2008-03-05 Michal Ludvig <michal@logix.cz>
982
983 * s3cmd, S3/S3.py, S3/Utils.py: Throttle upload speed and retry
984 when upload failed.
985 Report download/upload speed and time elapsed.
986
987 2008-02-28 Michal Ludvig <michal@logix.cz>
988
989 * Released version 0.9.6
990 ----------------------
991
992 2008-02-28 Michal Ludvig <michal@logix.cz>
993
994 * S3/PkgInfo.py: bumped up version to 0.9.6
995 * NEWS: What's new in 0.9.6
996
997 2008-02-27 Michal Ludvig <michal@logix.cz>
998
999 * s3cmd, s3cmd.1: Updated help and man page.
1000 * S3/S3.py, S3/Utils.py, s3cmd: Support for 's3cmd info' command.
1001 * s3cmd: Fix crash when 'sync'ing files with unresolvable owner uid/gid.
1002 * S3/S3.py, S3/Utils.py: open files in binary mode (otherwise windows
1003 users have problems).
1004 * S3/S3.py: modify 'x-amz-date' format (problems reported on MacOS X).
1005 Thanks Jon Larkowski for fix.
1006
1007 2008-02-27 Michal Ludvig <michal@logix.cz>
1008
1009 * TODO: Updated wishlist.
1010
1011 2008-02-11 Michal Ludvig <michal@logix.cz>
1012
1013 * S3/S3.py: Properly follow RedirectPermanent responses for EU buckets
1014 * S3/S3.py: Create public buckets with -P (#1837328)
1015 * S3/S3.py, s3cmd: Correctly display public URL on uploads.
1016 * S3/S3.py, S3/Config.py: Support for MIME types. Both
1017 default and guessing. Fixes bug #1872192 (Thanks Martin Herr)
1018
1019 2007-11-13 Michal Ludvig <michal@logix.cz>
1020
1021 * Released version 0.9.5
1022 ----------------------
1023
1024 2007-11-13 Michal Ludvig <michal@logix.cz>
1025
1026 * S3/S3.py: Support for buckets stored in Europe, access now
1027 goes via <bucket>.s3.amazonaws.com where possible.
1028
1029 2007-11-12 Michal Ludvig <michal@logix.cz>
1030
1031 * s3cmd: Support for storing file attributes (like ownership,
1032 mode, etc) in sync operation.
1033 * s3cmd, S3/S3.py: New command 'ib' to get information about
1034 bucket (only 'LocationConstraint' supported for now).
1035
1036 2007-10-01 Michal Ludvig <michal@logix.cz>
1037
1038 * s3cmd: Fix typo in argument name (patch
1039 from Kim-Minh KAPLAN, SF #1804808)
1040
1041 2007-09-25 Michal Ludvig <michal@logix.cz>
1042
1043 * s3cmd: Exit with error code on error (patch
1044 from Kim-Minh KAPLAN, SF #1800583)
1045
1046 2007-09-25 Michal Ludvig <michal@logix.cz>
1047
1048 * S3/S3.py: Don't fail if bucket listing doesn't have
1049 <IsTruncated> node.
1050 * s3cmd: Create ~/.s3cfg with 0600 permissions.
1051
1052 2007-09-13 Michal Ludvig <michal@logix.cz>
1053
1054 * s3cmd: Improved 'sync'
1055 * S3/S3.py: Support for buckets with over 1000 objects.
1056
1057 2007-09-03 Michal Ludvig <michal@logix.cz>
1058
1059 * s3cmd: Small tweaks to --configure workflow.
1060
1061 2007-09-02 Michal Ludvig <michal@logix.cz>
1062
1063 * s3cmd: Initial support for 'sync' operation. For
1064 now only local->s3 direction. In this version doesn't
1065 work well with non-ASCII filenames and doesn't support
1066 encryption.
1067
1068 2007-08-24 Michal Ludvig <michal@logix.cz>
1069
1070 * s3cmd, S3/Util.py: More ElementTree imports cleanup
1071
1072 2007-08-19 Michal Ludvig <michal@logix.cz>
1073
1074 * NEWS: Added news for 0.9.5
1075
1076 2007-08-19 Michal Ludvig <michal@logix.cz>
1077
1078 * s3cmd: Better handling of multiple arguments for put, get and del
1079
1080 2007-08-14 Michal Ludvig <michal@logix.cz>
1081
1082 * setup.py, S3/Utils.py: Try import xml.etree.ElementTree
1083 or elementtree.ElementTree module.
1084
1085 2007-08-14 Michal Ludvig <michal@logix.cz>
1086
1087 * s3cmd.1: Add info about --encrypt parameter.
1088
1089 2007-08-14 Michal Ludvig <michal@logix.cz>
1090
1091 * S3/PkgInfo.py: Bump up version to 0.9.5-pre
1092
1093 2007-08-13 Michal Ludvig <michal@logix.cz>
1094
1095 * Released version 0.9.4
1096 ----------------------
1097
1098 2007-08-13 Michal Ludvig <michal@logix.cz>
1099
1100 * S3/S3.py: Added function urlencode_string() that encodes
1101 non-ascii characters in object name before sending it to S3.
1102
1103 2007-08-13 Michal Ludvig <michal@logix.cz>
1104
1105 * README: Updated Amazon S3 pricing overview
1106
1107 2007-08-13 Michal Ludvig <michal@logix.cz>
1108
1109 * s3cmd, S3/Config.py, S3/S3.py: HTTPS support
1110
1111 2007-07-20 Michal Ludvig <michal@logix.cz>
1112
1113 * setup.py: Check correct Python version and ElementTree availability.
1114
1115 2007-07-05 Michal Ludvig <michal@logix.cz>
1116
1117 * s3cmd: --configure support for Proxy
1118 * S3/S3.py: HTTP proxy support from
1119 John D. Rowell <jdrowell@exerciseyourbrain.com>
1120
1121 2007-06-19 Michal Ludvig <michal@logix.cz>
1122
1123 * setup.py: Check for S3CMD_PACKAGING and don't install
1124 manpages and docs if defined.
1125 * INSTALL: Document the above change.
1126 * MANIFEST.in: Include uncompressed manpage
1127
1128 2007-06-17 Michal Ludvig <michal@logix.cz>
1129
1130 * s3cmd: Added encryption key support to --configure
1131 * S3/PkgInfo.py: Bump up version to 0.9.4-pre
1132 * setup.py: Cleaned up some rpm-specific stuff that
1133 caused problems to Debian packager Mikhail Gusarov
1134 * setup.cfg: Removed [bdist_rpm] section
1135 * MANIFEST.in: Include S3/*.py
1136
1137 2007-06-16 Michal Ludvig <michal@logix.cz>
1138
1139 * s3cmd.1: Syntax fixes from Mikhail Gusarov <dottedmag@dottedmag.net>
1140
1141 2007-05-27 Michal Ludvig <michal@logix.cz>
1142
1143 * Support for on-the-fly GPG encryption.
1144
1145 2007-05-26 Michal Ludvig <michal@logix.cz>
1146
1147 * s3cmd.1: Add info about "s3cmd du" command.
1148
1149 2007-05-26 Michal Ludvig <michal@logix.cz>
1150
1151 * Released version 0.9.3
1152 ----------------------
1153
1154 2007-05-26 Michal Ludvig <michal@logix.cz>
1155
1156 * s3cmd: Patch from Basil Shubin <basil.shubin@gmail.com>
1157 adding support for "s3cmd du" command.
1158 * s3cmd: Modified output format of "s3cmd du" to conform
1159 with unix "du".
1160 * setup.cfg: Require Python 2.5 in RPM. Otherwise it needs
1161 to require additional python modules (e.g. ElementTree)
1162 which may have different names in different distros. It's
1163 indeed still possible to manually install s3cmd with
1164 Python 2.4 and appropriate modules.
1165
1166 2007-04-09 Michal Ludvig <michal@logix.cz>
1167
1168 * Released version 0.9.2
1169 ----------------------
1170
1171 2007-04-09 Michal Ludvig <michal@logix.cz>
1172
1173 * s3cmd.1: Added manpage
1174 * Updated infrastructure files to create "better"
1175 distribution archives.
1176
1177 2007-03-26 Michal Ludvig <michal@logix.cz>
1178
1179 * setup.py, S3/PkgInfo.py: Move package info out of setup.py
1180 * s3cmd: new parameter --version
1181 * s3cmd, S3/S3Uri.py: Output public HTTP URL for objects
1182 stored with Public ACL.
1183
1184 2007-02-28 Michal Ludvig <michal@logix.cz>
1185
1186 * s3cmd: Verify supplied accesskey and secretkey
1187 in interactive configuration path.
1188 * S3/Config.py: Hide access key and secret key
1189 from debug output.
1190 * S3/S3.py: Modify S3Error exception to work
1191 in python 2.4 (=> don't expect Exception is
1192 a new-style class).
1193 * s3cmd: Updated for the above change.
1194
1195 2007-02-19 Michal Ludvig <michal@logix.cz>
1196
1197 * NEWS, INSTALL, README, setup.py: Added
1198 more documentation.
1199
1200 2007-02-19 Michal Ludvig <michal@logix.cz>
1201
1202 * S3/S3.py, s3cmd: New feature - allow "get" to stdout
1203
1204 2007-02-19 Michal Ludvig <michal@logix.cz>
1205
1206 * S3/S3fs.py: Removed (development moved to branch s3fs-devel).
1207
1208 2007-02-08 Michal Ludvig <michal@logix.cz>
1209
1210 * S3/S3fs.py:
1211 - Implemented mknod()
1212 - Can create directory structure
1213 - Rewritten to use SQLite3. Currently can create
1214 the filesystem, and a root inode.
1215
1216 2007-02-07 Michal Ludvig <michal@logix.cz>
1217
1218 * s3cmd (from /s3py:74): Renamed SVN top-level project
1219 s3py to s3cmd
1220
1221 2007-02-07 Michal Ludvig <michal@logix.cz>
1222
1223 * setup.cfg: Only require Python 2.4, not 2.5
1224 * S3/Config.py: Removed show_uri - no longer needed,
1225 it's now default
1226
1227 2007-02-07 Michal Ludvig <michal@logix.cz>
1228
1229 * setup.py
1230 - Version 0.9.1
1231
1232 2007-02-07 Michal Ludvig <michal@logix.cz>
1233
1234 * s3cmd: Change all "exit()" calls to "sys.exit()"
1235 and allow for python 2.4
1236 * S3/S3.py: Removed dependency on hashlib -> allow for python 2.4
1237
1238 2007-01-27 Michal Ludvig <michal@logix.cz>
1239
1240 * S3/S3.py, S3/S3Uri.py: Case insensitive regex in S3Uri.py
1241
1242 2007-01-26 Michal Ludvig <michal@logix.cz>
1243
1244 * S3/S3fs.py: Added support for stroing/loading inodes.
1245 No data yet however.
1246
1247 2007-01-26 Michal Ludvig <michal@logix.cz>
1248
1249 * S3/S3fs.py: Initial version of S3fs module.
1250 Can create filesystem via "S3fs.mkfs()"
1251
1252 2007-01-26 Michal Ludvig <michal@logix.cz>
1253
1254 * S3/BidirMap.py, S3/Config.py, S3/S3.py, S3/S3Uri.py,
1255 S3/SortedDict.py, S3/Utils.py, s3cmd: Added headers with
1256 copyright to all files
1257 * S3/S3.py, S3/S3Uri.py: Removed S3.compose_uri(), introduced
1258 S3UriS3.compose_uri() instead.
1259
1260 2007-01-26 Michal Ludvig <michal@logix.cz>
1261
1262 * S3/S3.py, S3/S3Uri.py, s3cmd:
1263 - Converted all users of parse_uri to S3Uri class API
1264 - Removed "cp" command again. Will have to use 'put'
1265 and 'get' for now.
1266
1267 2007-01-25 Michal Ludvig <michal@logix.cz>
1268
1269 * S3/S3Uri.py: New module S3/S3Uri.py
1270 * S3/S3.py, s3cmd: Converted "put" operation to use
1271 the new S3Uri class.
1272
1273 2007-01-24 Michal Ludvig <michal@logix.cz>
1274
1275 * S3/S3.py
1276 * s3cmd
1277 - Added 'cp' command
1278 - Renamed parse_s3_uri to parse_uri (this will go away anyway)
1279
1280 2007-01-19 Michal Ludvig <michal@logix.cz>
1281
1282 * setup.cfg
1283 * setup.py
1284 - Include README into tarballs
1285
1286 2007-01-19 Michal Ludvig <michal@logix.cz>
1287
1288 * README
1289 - Added comprehensive README file
1290
1291 2007-01-19 Michal Ludvig <michal@logix.cz>
1292
1293 * setup.cfg
1294 * setup.py
1295 - Added configuration for setup.py sdist
1296
1297 2007-01-19 Michal Ludvig <michal@logix.cz>
1298
1299 * S3/Config.py
1300 * s3cmd
1301 - Added interactive configurator (--configure)
1302 - Added config dumper (--dump-config)
1303 - Improved --help output
1304
1305 2007-01-19 Michal Ludvig <michal@logix.cz>
1306
1307 * setup.cfg
1308 * setup.py
1309 Added info for building RPM packages.
1310
1311 2007-01-18 Michal Ludvig <michal@logix.cz>
1312
1313 * S3/Config.py
1314 * S3/S3.py
1315 * s3cmd
1316 Moved class Config from S3/S3.py to S3/Config.py
1317
1318 2007-01-18 Michal Ludvig <michal@logix.cz>
1319
1320 * S3/Config.py (from /s3py/trunk/S3/ConfigParser.py:47)
1321 * S3/ConfigParser.py
1322 * S3/S3.py
1323 Renamed S3/ConfigParser.py to S3/Config.py
1324
1325 2007-01-18 Michal Ludvig <michal@logix.cz>
1326
1327 * s3cmd
1328 Added info about homepage
1329
1330 2007-01-17 Michal Ludvig <michal@logix.cz>
1331
1332 * S3/S3.py
1333 * s3cmd
1334 - Use prefix for listings if specified.
1335 - List all commands in --help
1336
1337 2007-01-16 Michal Ludvig <michal@logix.cz>
1338
1339 * S3/S3.py
1340 * s3cmd
1341 Major rework of Config class:
1342 - Renamed from AwsConfig to Config
1343 - Converted to Singleton (see Config.__new__() and an article on
1344 Wikipedia)
1345 - No more explicit listing of options - use introspection to get them
1346 (class variables that of type str, int or bool that don't start with
1347 underscore)
1348 - Check values read from config file and verify their type.
1349
1350 Added OptionMimeType and -m/-M options. Not yet implemented
1351 functionality in the rest of S3/S3.py
1352
1353 2007-01-15 Michal Ludvig <michal@logix.cz>
1354
1355 * S3/S3.py
1356 * s3cmd
1357 - Merged list-buckets and bucket-list-objects operations into
1358 a single 'ls' command.
1359 - New parameter -P for uploading publicly readable objects
1360
1361 2007-01-14 Michal Ludvig <michal@logix.cz>
1362
1363 * s3.py
1364 * setup.py
1365 Renamed s3.py to s3cmd (take 2)
1366
1367 2007-01-14 Michal Ludvig <michal@logix.cz>
1368
1369 * s3cmd (from /s3py/trunk/s3.py:45)
1370 Renamed s3.py to s3cmd
1371
1372 2007-01-14 Michal Ludvig <michal@logix.cz>
1373
1374 * S3
1375 * S3/S3.py
1376 * s3.py
1377 * setup.py
1378 All classes from s3.py go to S3/S3.py
1379 Added setup.py
1380
1381 2007-01-14 Michal Ludvig <michal@logix.cz>
1382
1383 * s3.py
1384 Minor fix S3.utils -> S3.Utils
1385
1386 2007-01-14 Michal Ludvig <michal@logix.cz>
1387
1388 * .svnignore
1389 * BidirMap.py
1390 * ConfigParser.py
1391 * S3
1392 * S3/BidirMap.py (from /s3py/trunk/BidirMap.py:35)
1393 * S3/ConfigParser.py (from /s3py/trunk/ConfigParser.py:38)
1394 * S3/SortedDict.py (from /s3py/trunk/SortedDict.py:35)
1395 * S3/Utils.py (from /s3py/trunk/utils.py:39)
1396 * S3/__init__.py
1397 * SortedDict.py
1398 * s3.py
1399 * utils.py
1400 Moved modules to their own package
1401
1402 2007-01-12 Michal Ludvig <michal@logix.cz>
1403
1404 * s3.py
1405 Added "del" command
1406 Converted all (?) commands to accept s3-uri
1407 Added -u/--show-uri parameter
1408
1409 2007-01-11 Michal Ludvig <michal@logix.cz>
1410
1411 * s3.py
1412 Verify MD5 on received files
1413 Improved upload of multiple files
1414 Initial S3-URI support (more tbd)
1415
1416 2007-01-11 Michal Ludvig <michal@logix.cz>
1417
1418 * s3.py
1419 Minor fixes:
1420 - store names of parsed files in AwsConfig
1421 - Print total size with upload/download
1422
1423 2007-01-11 Michal Ludvig <michal@logix.cz>
1424
1425 * s3.py
1426 * utils.py
1427 Added support for sending and receiving files.
1428
1429 2007-01-11 Michal Ludvig <michal@logix.cz>
1430
1431 * ConfigParser.py
1432 * s3.py
1433 List all Objects in all Buckets command
1434 Yet another logging improvement
1435 Version check for Python 2.5 or higher
1436
1437 2007-01-11 Michal Ludvig <michal@logix.cz>
1438
1439 * ConfigParser.py
1440 * s3.py
1441 * utils.py
1442 Added ConfigParser
1443 Improved setting logging levels
1444 It can now quite reliably list buckets and objects
1445
1446 2007-01-11 Michal Ludvig <michal@logix.cz>
1447
1448 * .svnignore
1449 Added ignore list
1450
1451 2007-01-11 Michal Ludvig <michal@logix.cz>
1452
1453 * .svnignore
1454 * BidirMap.py
1455 * SortedDict.py
1456 * s3.py
1457 * utils.py
1458 Initial import
+44
-0
Makefile less more
0 SHELL := /bin/bash
1 VERSION := $(shell /usr/bin/env python2 -c 'from S3 import PkgInfo;print PkgInfo.version')
2 SPEC := s3cmd.spec
3 COMMIT := $(shell git rev-parse HEAD)
4 SHORTCOMMIT := $(shell git rev-parse --short=8 HEAD)
5 TARBALL = s3cmd-$(VERSION)-$(SHORTCOMMIT).tar.gz
6
7 release:
8 python2 setup.py register sdist upload --sign
9
10 clean:
11 -rm -rf s3cmd-*.tar.gz *.rpm *~ $(SPEC)
12 -find . -name \*.pyc -exec rm \{\} \;
13 -find . -name \*.pyo -exec rm \{\} \;
14
15 $(SPEC): $(SPEC).in
16 sed -e 's/##VERSION##/$(VERSION)/' \
17 -e 's/##COMMIT##/$(COMMIT)/' \
18 -e 's/##SHORTCOMMIT##/$(SHORTCOMMIT)/' \
19 $(SPEC).in > $(SPEC)
20
21 # fixme: python setup.py sdist also generates a PKG-INFO file which we don't have using straight git archive
22 git-tarball:
23 git archive --format tar --prefix s3cmd-$(COMMIT)/ HEAD S3/ s3cmd NEWS README.md LICENSE INSTALL setup.cfg s3cmd.1 setup.py| gzip -c > $(TARBALL)
24
25 # Use older digest algorithms for local rpmbuilds, as EPEL5 and
26 # earlier releases need this. When building using mock for a
27 # particular target, it will use the proper (newer) digests if that
28 # target supports it.
29 git-rpm: clean git-tarball $(SPEC)
30 tmp_dir=`mktemp -d` ; \
31 mkdir -p $${tmp_dir}/{BUILD,RPMS,SRPMS,SPECS,SOURCES} ; \
32 cp $(TARBALL) $${tmp_dir}/SOURCES ; \
33 cp $(SPEC) $${tmp_dir}/SPECS ; \
34 cd $${tmp_dir} > /dev/null 2>&1; \
35 rpmbuild -ba --define "_topdir $${tmp_dir}" \
36 --define "_source_filedigest_algorithm 0" \
37 --define "_binary_filedigest_algorithm 0" \
38 --define "dist %{nil}" \
39 SPECS/$(SPEC) ; \
40 cd - > /dev/null 2>&1; \
41 cp $${tmp_dir}/RPMS/noarch/* $${tmp_dir}/SRPMS/* . ; \
42 rm -rf $${tmp_dir} ; \
43 rpmlint *.rpm *.spec
+0
-45
PKG-INFO less more
0 Metadata-Version: 1.1
1 Name: s3cmd
2 Version: 2.0.2
3 Summary: Command line tool for managing Amazon S3 and CloudFront services
4 Home-page: http://s3tools.org
5 Author: github.com/mdomsch, github.com/matteobar, github.com/fviard
6 Author-email: s3tools-bugs@lists.sourceforge.net
7 License: GNU GPL v2+
8 Description-Content-Type: UNKNOWN
9 Description:
10
11 S3cmd lets you copy files from/to Amazon S3
12 (Simple Storage Service) using a simple to use
13 command line client. Supports rsync-like backup,
14 GPG encryption, and more. Also supports management
15 of Amazon's CloudFront content delivery network.
16
17
18 Authors:
19 --------
20 Michal Ludvig <michal@logix.cz>
21
22 Platform: UNKNOWN
23 Classifier: Development Status :: 5 - Production/Stable
24 Classifier: Environment :: Console
25 Classifier: Environment :: MacOS X
26 Classifier: Environment :: Win32 (MS Windows)
27 Classifier: Intended Audience :: End Users/Desktop
28 Classifier: Intended Audience :: System Administrators
29 Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)
30 Classifier: Natural Language :: English
31 Classifier: Operating System :: MacOS :: MacOS X
32 Classifier: Operating System :: Microsoft :: Windows
33 Classifier: Operating System :: POSIX
34 Classifier: Operating System :: Unix
35 Classifier: Programming Language :: Python :: 2
36 Classifier: Programming Language :: Python :: 2.6
37 Classifier: Programming Language :: Python :: 2.7
38 Classifier: Programming Language :: Python :: 3
39 Classifier: Programming Language :: Python :: 3.3
40 Classifier: Programming Language :: Python :: 3.4
41 Classifier: Programming Language :: Python :: 3.5
42 Classifier: Programming Language :: Python :: 3.6
43 Classifier: Topic :: System :: Archiving
44 Classifier: Topic :: Utilities
+1
-1
README.md less more
334334
335335 ### License
336336
337 Copyright (C) 2007-2017 TGRMN Software - http://www.tgrmn.com - and contributors
337 Copyright (C) 2007-2019 TGRMN Software - http://www.tgrmn.com - and contributors
338338
339339 This program is free software; you can redistribute it and/or modify
340340 it under the terms of the GNU General Public License as published by
+89
-0
RELEASE_INSTRUCTIONS less more
0 Instructions for s3cmd maintainers for doing a tagged release and publishing on sourceforge.net.
1 In the below, 1.5.0-rc1 is the example version being released. Salt to taste.
2
3 1. Make a fresh clone of the repo:
4 git clone ssh+git://git@github.com/s3tools/s3cmd s3cmd-release
5
6 2. Run ./run-tests.py to verify it all works OK.
7
8 3. Update version to 1.5.0-rc1 in S3/PkgInfo.py
9
10 4. Update manpage with ./s3cmd --help | ./format-manpage.pl > s3cmd.1
11
12 5. Update NEWS with info about new features. Best to extract from git
13 with: git log --no-merges v1.5.0-beta1..
14 (list all tags with: "git tag")
15
16
17 6. Verify the above changes:
18 git diff --check && git diff
19 git status
20 (The only changed files should be NEWS, s3cmd.1, S3/PkgInfo.py)
21
22 7. Remove testsuite (intentionally inaccessible files break the next
23 step):
24 chmod -R +rwx testsuite/permission-tests/permission-denied-dir && rm -rf testsuite
25
26 8. If everything worked fine commit the above changes:
27 git commit -a -m "Update version to 1.5.0-rc1"
28
29 9. Tag it:
30 git tag --sign -a v1.5.0-rc1 -m "Tag v1.5.0-rc1"
31
32 10. Push back to github:
33 git push --tags
34
35 11. Build the "Source Distribution":
36 python setup.py sdist upload --sign
37 -> Creates dist/s3cmd-1.5.0-rc1.tar.gz* and dist/s3cmd-1.5.0-rc1.zip*
38
39 The upload step publishes it to PyPi, so 'pip install s3cmd' downloads
40 the new version. It also generated the GPG signatures, so you don't
41 have to do that manually, and have them available for the next steps.
42
43
44 GitHub releases
45
46 1. Login to github.com/s3tools/s3cmd
47
48 2. You will see your new tag in the Tags tab. Click "Draft a new
49 release".
50
51 3. In the 'Tag version' drop-down, select your new tag.
52
53 4. In the 'Release title' field, name it v1.5.0-rc1.
54
55 5. In the 'Describe this release' text box, add in this release's
56 notes from the NEWS file.
57
58 6. Upload all 4 files from dist/.
59
60 7. Click "Publish release"
61
62
63
64 SourceForge releases
65
66 1. Login to sf.net
67
68 2. Go to https://sourceforge.net/p/s3tools/admin/
69
70 3. Files -> s3cmd -> Add Folder -> Enter "1.5.0-rc1" -> Create
71
72 4. Go into 1.5.0-rc1 -> Add File -> upload dist/s3cmd-1.5.0-rc1.tar.gz
73
74 5. Once uploaded click the little "i" icon on the right and click
75 "Select all" under "Default Download For:" to update the default
76 download button to this new version.
77
78 6. Give it a few minutes and verify on the Summary page that the
79 download button has been updated to s3cmd-1.5.0-rc1.tar.gz
80
81 Now it's time to send out an announcement email to
82 s3tools-announce@lists.sourceforge.net and
83 s3tools-general@lists.sourceforge.net (check out the s3cmd-announce
84 archive for an inspiration :)
85
86 And the last step is to ask the respective distribution maintainers
87 (Fedora, Debian, Ubuntu, OpenSuse, ...?) to update the package in
88 their builds.
+1
-1
S3/ACL.py less more
1515 except ImportError:
1616 import elementtree.ElementTree as ET
1717
18 PY3 = (sys.version_info >= (3,0))
18 PY3 = (sys.version_info >= (3, 0))
1919
2020 class Grantee(object):
2121 ALL_USERS_URI = "http://acs.amazonaws.com/groups/global/AllUsers"
+34
-8
S3/CloudFront.py less more
2121 from .S3 import S3
2222 from .Config import Config
2323 from .Exceptions import *
24 from .Utils import getTreeFromXml, appendXmlTextNode, getDictFromTree, dateS3toPython, getBucketFromHostname, getHostnameFromBucket, deunicodise, urlencode_string, convertHeaderTupleListToDict
24 from .Utils import (getTreeFromXml, appendXmlTextNode, getDictFromTree,
25 dateS3toPython, getBucketFromHostname,
26 getHostnameFromBucket, deunicodise, urlencode_string,
27 convertHeaderTupleListToDict, encode_to_s3, decode_from_s3)
2528 from .Crypto import sign_string_v2
2629 from .S3Uri import S3Uri, S3UriS3
2730 from .ConnMan import ConnMan
2831 from .SortedDict import SortedDict
32
33 PY3 = (sys.version_info >= (3, 0))
2934
3035 cloudfront_api_version = "2010-11-01"
3136 cloudfront_resource = "/%(api_ver)s/distribution" % { 'api_ver' : cloudfront_api_version }
175180 else:
176181 self.info['Logging'] = None
177182
178 def __str__(self):
183 def get_printable_tree(self):
179184 tree = ET.Element("DistributionConfig")
180185 tree.attrib['xmlns'] = DistributionConfig.xmlns
181186
196201 appendXmlTextNode("Bucket", getHostnameFromBucket(self.info['Logging'].bucket()), logging_el)
197202 appendXmlTextNode("Prefix", self.info['Logging'].object(), logging_el)
198203 tree.append(logging_el)
199 return ET.tostring(tree)
204 return tree
205
206 def __unicode__(self):
207 return decode_from_s3(ET.tostring(self.get_printable_tree()))
208
209 def __str__(self):
210 if PY3:
211 # Return unicode
212 return ET.tostring(self.get_printable_tree(), encoding="unicode")
213 else:
214 # Return bytes
215 return ET.tostring(self.get_printable_tree())
200216
201217 class Invalidation(object):
202218 ## Example:
284300 def get_reference(self):
285301 return self.reference
286302
287 def __str__(self):
303 def get_printable_tree(self):
288304 tree = ET.Element("InvalidationBatch")
289
290305 for path in self.paths:
291306 if len(path) < 1 or path[0] != "/":
292307 path = "/" + path
293308 appendXmlTextNode("Path", urlencode_string(path), tree)
294309 appendXmlTextNode("CallerReference", self.reference, tree)
295 return ET.tostring(tree)
310 return tree
311
312 def __unicode__(self):
313 return decode_from_s3(ET.tostring(self.get_printable_tree()))
314
315 def __str__(self):
316 if PY3:
317 # Return unicode
318 return ET.tostring(self.get_printable_tree(), encoding="unicode")
319 else:
320 # Return bytes
321 return ET.tostring(self.get_printable_tree())
296322
297323 class CloudFront(object):
298324 operations = {
563589
564590 def sign_request(self, headers):
565591 string_to_sign = headers['x-amz-date']
566 signature = sign_string_v2(string_to_sign)
592 signature = decode_from_s3(sign_string_v2(encode_to_s3(string_to_sign)))
567593 debug(u"CloudFront.sign_request('%s') = %s" % (string_to_sign, signature))
568594 return signature
569595
602628 continue
603629
604630 if CloudFront.dist_list.get(distListIndex, None) is None:
605 CloudFront.dist_list[distListIndex] = set()
631 CloudFront.dist_list[distListIndex] = set()
606632
607633 CloudFront.dist_list[distListIndex].add(d.uri())
608634
+7
-5
S3/Config.py less more
204204 # Maximum sleep duration for throtte / limitrate.
205205 # s3 will timeout if a request/transfer is stuck for more than a short time
206206 throttle_max = 100
207 public_url_use_https = False
207208
208209 ## Creating a singleton
209210 def __new__(self, configfile = None, access_key=None, secret_key=None, access_token=None):
259260 resp = conn.getresponse()
260261 files = resp.read()
261262 if resp.status == 200 and len(files)>1:
262 conn.request('GET', "/latest/meta-data/iam/security-credentials/%s"%files.decode('UTF-8'))
263 conn.request('GET', "/latest/meta-data/iam/security-credentials/%s" % files.decode('utf-8'))
263264 resp=conn.getresponse()
264265 if resp.status == 200:
265 creds=json.load(resp)
266 Config().update_option('access_key', creds['AccessKeyId'].encode('ascii'))
267 Config().update_option('secret_key', creds['SecretAccessKey'].encode('ascii'))
268 Config().update_option('access_token', creds['Token'].encode('ascii'))
266 resp_content = config_unicodise(resp.read())
267 creds=json.loads(resp_content)
268 Config().update_option('access_key', config_unicodise(creds['AccessKeyId']))
269 Config().update_option('secret_key', config_unicodise(creds['SecretAccessKey']))
270 Config().update_option('access_token', config_unicodise(creds['Token']))
269271 else:
270272 raise IOError
271273 else:
+1
-0
S3/Crypto.py less more
6262
6363 Useful for REST authentication. See http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
6464 string_to_sign should be utf-8 "bytes".
65 and returned signature will be utf-8 encoded "bytes".
6566 """
6667 secret_key = Config.Config().secret_key
6768 signature = base64.encodestring(hmac.new(encode_to_s3(secret_key), string_to_sign, sha1).digest()).strip()
+2
-4
S3/S3.py less more
381381 bucket_location = bucket_location.strip()
382382 if bucket_location.upper() == "EU":
383383 bucket_location = bucket_location.upper()
384 else:
385 bucket_location = bucket_location.lower()
386384 body = "<CreateBucketConfiguration><LocationConstraint>"
387385 body += bucket_location
388386 body += "</LocationConstraint></CreateBucketConfiguration>"
967965 request = self.create_request("BUCKET_LIST", bucket = uri.bucket(),
968966 uri_params = {'policy': None})
969967 response = self.send_request(request)
970 return response['data']
968 return decode_from_s3(response['data'])
971969
972970 def set_policy(self, uri, policy):
973971 headers = SortedDict(ignore_case = True)
990988 request = self.create_request("BUCKET_LIST", bucket = uri.bucket(),
991989 uri_params = {'cors': None})
992990 response = self.send_request(request)
993 return response['data']
991 return decode_from_s3(response['data'])
994992
995993 def set_cors(self, uri, cors):
996994 headers = SortedDict(ignore_case = True)
+6
-3
S3/S3Uri.py less more
8989 return check_bucket_name_dns_support(Config.Config().host_bucket, self._bucket)
9090
9191 def public_url(self):
92 public_url_protocol = "http"
93 if Config.Config().public_url_use_https:
94 public_url_protocol = "https"
9295 if self.is_dns_compatible():
93 return "http://%s.%s/%s" % (self._bucket, Config.Config().host_base, self._object)
94 else:
95 return "http://%s/%s/%s" % (Config.Config().host_base, self._bucket, self._object)
96 return "%s://%s.%s/%s" % (public_url_protocol, self._bucket, Config.Config().host_base, self._object)
97 else:
98 return "%s://%s/%s/%s" % (public_url_protocol, Config.Config().host_base, self._bucket, self._object)
9699
97100 def host_name(self):
98101 if self.is_dns_compatible():
+3
-3
S3/Utils.py less more
100100 __all__.append("stripNameSpace")
101101
102102 def getTreeFromXml(xml):
103 xml, xmlns = stripNameSpace(xml)
103 xml, xmlns = stripNameSpace(encode_to_s3(xml))
104104 try:
105105 tree = ET.fromstring(xml)
106106 if xmlns:
193193 def formatSize(size, human_readable = False, floating_point = False):
194194 size = floating_point and float(size) or int(size)
195195 if human_readable:
196 coeffs = ['k', 'M', 'G', 'T']
196 coeffs = ['K', 'M', 'G', 'T']
197197 coeff = ""
198198 while size > 2048:
199199 size /= 1024
200200 coeff = coeffs.pop(0)
201 return (size, coeff)
201 return (floating_point and float(size) or int(size), coeff)
202202 else:
203203 return (size, "")
204204 __all__.append("formatSize")
+52
-0
TODO less more
0 TODO list for s3cmd project
1 ===========================
2
3 - Before 1.0.0 (or asap after 1.0.0)
4 - Make 'sync s3://bkt/some-filename local/other-filename' work
5 (at the moment it'll always download).
6 - Enable --exclude for [ls].
7 - Allow change /tmp to somewhere else
8 - With --guess-mime use 'magic' module if available.
9 - Support --preserve for [put] and [get]. Update manpage.
10 - Don't let --continue fail if the file is already fully downloaded.
11 - Option --mime-type should set mime type with 'cp' and 'mv'.
12 If possible --guess-mime-type should do as well.
13 - Make upload throttling configurable.
14 - Allow removing 'DefaultRootObject' from CloudFront distributions.
15 - Get s3://bucket/non-existent creates empty local file 'non-existent'
16 - Add 'geturl' command, both Unicode and urlencoded output.
17 - Add a command for generating "Query String Authentication" URLs.
18 - Support --acl-grant (together with --acl-public/private) for [put] and [sync]
19 - Filter 's3cmd ls' output by --bucket-location=
20
21 - After 1.0.0
22 - Sync must backup non-files as well. At least directories,
23 symlinks and device nodes.
24 - Speed up upload / download with multiple threads.
25 (see http://blog.50projects.com/p/s3cmd-modifications.html)
26 - Sync should be able to update metadata (UID, timstamps, etc)
27 if only these change (i.e. same content, different metainfo).
28 - If GPG fails error() and exit. If un-GPG fails save the
29 file with .gpg extension.
30 - Keep backup files remotely on put/sync-to if requested
31 (move the old 'object' to e.g. 'object~' and only then upload
32 the new one). Could be more advanced to keep, say, last 5
33 copies, etc.
34 - Memory consumption on very large upload sets is terribly high.
35 - Implement per-bucket (or per-regexp?) default settings. For
36 example regarding ACLs, encryption, etc.
37
38 - Implement GPG for sync
39 (it's not that easy since it won't be easy to compare
40 the encrypted-remote-object size with local file.
41 either we can store the metadata in a dedicated file
42 where we face a risk of inconsistencies, or we'll store
43 the metadata encrypted in each object header where we'll
44 have to do large number for object/HEAD requests. tough
45 call).
46 Or we can only compare local timestamps with remote object
47 timestamps. If the local one is older we'll *assume* it
48 hasn't been changed. But what to do about remote2local sync?
49
50 - Keep man page up to date and write some more documentation
51 - Yeah, right ;-)
+223
-0
format-manpage.pl less more
0 #!/usr/bin/perl
1
2 # Format s3cmd.1 manpage
3 # Usage:
4 # s3cmd --help | format-manpage.pl > s3cmd.1
5
6 use strict;
7
8 my $commands = "";
9 my $cfcommands = "";
10 my $wscommands = "";
11 my $options = "";
12
13 while (<>) {
14 if (/^Commands:/) {
15 while (<>) {
16 last if (/^\s*$/);
17 my ($desc, $cmd, $cmdline);
18 ($desc = $_) =~ s/^\s*(.*?)\s*$/$1/;
19 ($cmdline = <>) =~ s/^\s*s3cmd (.*?) (.*?)\s*$/s3cmd \\fB$1\\fR \\fI$2\\fR/;
20 $cmd = $1;
21 $cmdline =~ s/-/\\-/g;
22 if ($cmd =~ /^cf/) {
23 $cfcommands .= ".TP\n$cmdline\n$desc\n";
24 } elsif ($cmd =~ /^ws/) {
25 $wscommands .= ".TP\n$cmdline\n$desc\n";
26 } else {
27 $commands .= ".TP\n$cmdline\n$desc\n";
28 }
29 }
30 }
31 if (/^Options:/) {
32 my ($opt, $desc);
33 while (<>) {
34 last if (/^\s*$/);
35 $_ =~ s/(.*?)\s*$/$1/;
36 $desc = "";
37 $opt = "";
38 if (/^ (-.*)/) {
39 $opt = $1;
40 if ($opt =~ / /) {
41 ($opt, $desc) = split(/\s\s+/, $opt, 2);
42 }
43 $opt =~ s/(-[^ ,=\.]+)/\\fB$1\\fR/g;
44 # escape all single dashes
45 $opt =~ s/-/\\-/g;
46 $options .= ".TP\n$opt\n";
47 } else {
48 $_ =~ s/\s*(.*?)\s*$/$1/;
49 $_ =~ s/(--[^ ,=\.]+)/\\fB$1\\fR/g;
50 # escape word 'Cache-Control'
51 $_ =~ s/'(\S+-\S+)'/\\&'$1'/g;
52 # escape all single dashes
53 $_ =~ s/-/\\-/g;
54 $desc .= $_;
55 }
56 if ($desc) {
57 $options .= "$desc\n";
58 }
59 }
60 }
61 }
62 print "
63 .\\\" !!! IMPORTANT: This file is generated from s3cmd \\-\\-help output using format-manpage.pl
64 .\\\" !!! Do your changes either in s3cmd file or in 'format\\-manpage.pl' otherwise
65 .\\\" !!! they will be overwritten!
66
67 .TH s3cmd 1
68 .SH NAME
69 s3cmd \\- tool for managing Amazon S3 storage space and Amazon CloudFront content delivery network
70 .SH SYNOPSIS
71 .B s3cmd
72 [\\fIOPTIONS\\fR] \\fICOMMAND\\fR [\\fIPARAMETERS\\fR]
73 .SH DESCRIPTION
74 .PP
75 .B s3cmd
76 is a command line client for copying files to/from
77 Amazon S3 (Simple Storage Service) and performing other
78 related tasks, for instance creating and removing buckets,
79 listing objects, etc.
80
81 .SH COMMANDS
82 .PP
83 .B s3cmd
84 can do several \\fIactions\\fR specified by the following \\fIcommands\\fR.
85 $commands
86
87 .PP
88 Commands for static WebSites configuration
89 $wscommands
90
91 .PP
92 Commands for CloudFront management
93 $cfcommands
94
95 .SH OPTIONS
96 .PP
97 Some of the below specified options can have their default
98 values set in
99 .B s3cmd
100 config file (by default \$HOME/.s3cmd). As it's a simple text file
101 feel free to open it with your favorite text editor and do any
102 changes you like.
103 $options
104
105 .SH EXAMPLES
106 One of the most powerful commands of \\fIs3cmd\\fR is \\fBs3cmd sync\\fR used for
107 synchronising complete directory trees to or from remote S3 storage. To some extent
108 \\fBs3cmd put\\fR and \\fBs3cmd get\\fR share a similar behaviour with \\fBsync\\fR.
109 .PP
110 Basic usage common in backup scenarios is as simple as:
111 .nf
112 s3cmd sync /local/path/ s3://test\\-bucket/backup/
113 .fi
114 .PP
115 This command will find all files under /local/path directory and copy them
116 to corresponding paths under s3://test\\-bucket/backup on the remote side.
117 For example:
118 .nf
119 /local/path/\\fBfile1.ext\\fR \\-> s3://bucket/backup/\\fBfile1.ext\\fR
120 /local/path/\\fBdir123/file2.bin\\fR \\-> s3://bucket/backup/\\fBdir123/file2.bin\\fR
121 .fi
122 .PP
123 However if the local path doesn't end with a slash the last directory's name
124 is used on the remote side as well. Compare these with the previous example:
125 .nf
126 s3cmd sync /local/path s3://test\\-bucket/backup/
127 .fi
128 will sync:
129 .nf
130 /local/\\fBpath/file1.ext\\fR \\-> s3://bucket/backup/\\fBpath/file1.ext\\fR
131 /local/\\fBpath/dir123/file2.bin\\fR \\-> s3://bucket/backup/\\fBpath/dir123/file2.bin\\fR
132 .fi
133 .PP
134 To retrieve the files back from S3 use inverted syntax:
135 .nf
136 s3cmd sync s3://test\\-bucket/backup/ ~/restore/
137 .fi
138 that will download files:
139 .nf
140 s3://bucket/backup/\\fBfile1.ext\\fR \\-> ~/restore/\\fBfile1.ext\\fR
141 s3://bucket/backup/\\fBdir123/file2.bin\\fR \\-> ~/restore/\\fBdir123/file2.bin\\fR
142 .fi
143 .PP
144 Without the trailing slash on source the behaviour is similar to
145 what has been demonstrated with upload:
146 .nf
147 s3cmd sync s3://test\\-bucket/backup ~/restore/
148 .fi
149 will download the files as:
150 .nf
151 s3://bucket/\\fBbackup/file1.ext\\fR \\-> ~/restore/\\fBbackup/file1.ext\\fR
152 s3://bucket/\\fBbackup/dir123/file2.bin\\fR \\-> ~/restore/\\fBbackup/dir123/file2.bin\\fR
153 .fi
154 .PP
155 All source file names, the bold ones above, are matched against \\fBexclude\\fR
156 rules and those that match are then re\\-checked against \\fBinclude\\fR rules to see
157 whether they should be excluded or kept in the source list.
158 .PP
159 For the purpose of \\fB\\-\\-exclude\\fR and \\fB\\-\\-include\\fR matching only the
160 bold file names above are used. For instance only \\fBpath/file1.ext\\fR is tested
161 against the patterns, not \\fI/local/\\fBpath/file1.ext\\fR
162 .PP
163 Both \\fB\\-\\-exclude\\fR and \\fB\\-\\-include\\fR work with shell\\-style wildcards (a.k.a. GLOB).
164 For a greater flexibility s3cmd provides Regular\\-expression versions of the two exclude options
165 named \\fB\\-\\-rexclude\\fR and \\fB\\-\\-rinclude\\fR.
166 The options with ...\\fB\\-from\\fR suffix (eg \\-\\-rinclude\\-from) expect a filename as
167 an argument. Each line of such a file is treated as one pattern.
168 .PP
169 There is only one set of patterns built from all \\fB\\-\\-(r)exclude(\\-from)\\fR options
170 and similarly for include variant. Any file excluded with eg \\-\\-exclude can
171 be put back with a pattern found in \\-\\-rinclude\\-from list.
172 .PP
173 Run s3cmd with \\fB\\-\\-dry\\-run\\fR to verify that your rules work as expected.
174 Use together with \\fB\\-\\-debug\\fR get detailed information
175 about matching file names against exclude and include rules.
176 .PP
177 For example to exclude all files with \".jpg\" extension except those beginning with a number use:
178 .PP
179 \\-\\-exclude '*.jpg' \\-\\-rinclude '[0\\-9].*\\.jpg'
180 .PP
181 To exclude all files except \"*.jpg\" extension, use:
182 .PP
183 \\-\\-exclude '*' \\-\\-include '*.jpg'
184 .PP
185 To exclude local directory 'somedir', be sure to use a trailing forward slash, as such:
186 .PP
187 \\-\\-exclude 'somedir/'
188 .PP
189
190 .SH SEE ALSO
191 For the most up to date list of options run:
192 .B s3cmd \\-\\-help
193 .br
194 For more info about usage, examples and other related info visit project homepage at:
195 .B http://s3tools.org
196 .SH AUTHOR
197 Written by Michal Ludvig and contributors
198 .SH CONTACT, SUPPORT
199 Preferred way to get support is our mailing list:
200 .br
201 .I s3tools\\-general\@lists.sourceforge.net
202 .br
203 or visit the project homepage:
204 .br
205 .B http://s3tools.org
206 .SH REPORTING BUGS
207 Report bugs to
208 .I s3tools\\-bugs\@lists.sourceforge.net
209 .SH COPYRIGHT
210 Copyright \\(co 2007\\-2015 TGRMN Software \\- http://www.tgrmn.com \\- and contributors
211 .br
212 .SH LICENSE
213 This program is free software; you can redistribute it and/or modify
214 it under the terms of the GNU General Public License as published by
215 the Free Software Foundation; either version 2 of the License, or
216 (at your option) any later version.
217 This program is distributed in the hope that it will be useful,
218 but WITHOUT ANY WARRANTY; without even the implied warranty of
219 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
220 GNU General Public License for more details.
221 .br
222 ";
+63
-0
magic less more
0 # Additional magic for common web file types
1
2 0 string/b {\ " JSON data
3 !:mime application/json
4 0 string/b {\ } JSON data
5 !:mime application/json
6 0 string/b [ JSON data
7 !:mime application/json
8
9 0 search/4000 function
10 >&0 search/32/b )\ { JavaScript program
11 !:mime application/javascript
12
13 0 search/4000 @media CSS stylesheet
14 !:mime text/css
15 0 search/4000 @import CSS stylesheet
16 !:mime text/css
17 0 search/4000 @namespace CSS stylesheet
18 !:mime text/css
19 0 search/4000/b {\ background CSS stylesheet
20 !:mime text/css
21 0 search/4000/b {\ border CSS stylesheet
22 !:mime text/css
23 0 search/4000/b {\ bottom CSS stylesheet
24 !:mime text/css
25 0 search/4000/b {\ color CSS stylesheet
26 !:mime text/css
27 0 search/4000/b {\ cursor CSS stylesheet
28 !:mime text/css
29 0 search/4000/b {\ direction CSS stylesheet
30 !:mime text/css
31 0 search/4000/b {\ display CSS stylesheet
32 !:mime text/css
33 0 search/4000/b {\ float CSS stylesheet
34 !:mime text/css
35 0 search/4000/b {\ font CSS stylesheet
36 !:mime text/css
37 0 search/4000/b {\ height CSS stylesheet
38 !:mime text/css
39 0 search/4000/b {\ left CSS stylesheet
40 !:mime text/css
41 0 search/4000/b {\ line- CSS stylesheet
42 !:mime text/css
43 0 search/4000/b {\ margin CSS stylesheet
44 !:mime text/css
45 0 search/4000/b {\ padding CSS stylesheet
46 !:mime text/css
47 0 search/4000/b {\ position CSS stylesheet
48 !:mime text/css
49 0 search/4000/b {\ right CSS stylesheet
50 !:mime text/css
51 0 search/4000/b {\ text- CSS stylesheet
52 !:mime text/css
53 0 search/4000/b {\ top CSS stylesheet
54 !:mime text/css
55 0 search/4000/b {\ width CSS stylesheet
56 !:mime text/css
57 0 search/4000/b {\ visibility CSS stylesheet
58 !:mime text/css
59 0 search/4000/b {\ -moz- CSS stylesheet
60 !:mime text/css
61 0 search/4000/b {\ -webkit- CSS stylesheet
62 !:mime text/css
+805
-0
run-tests-minio.py less more
0 #!/usr/bin/env python2
1 # -*- coding=utf-8 -*-
2
3 ## Amazon S3cmd - testsuite
4 ## Author: Michal Ludvig <michal@logix.cz>
5 ## http://www.logix.cz/michal
6 ## License: GPL Version 2
7 ## Copyright: TGRMN Software and contributors
8
9 from __future__ import absolute_import, print_function
10
11 import sys
12 import os
13 import re
14 import time
15 from subprocess import Popen, PIPE, STDOUT
16 import locale
17 import getpass
18 import S3.Exceptions
19 import S3.Config
20 from S3.ExitCodes import *
21
22 PY3 = (sys.version_info >= (3,0))
23
24 try:
25 unicode
26 except NameError:
27 # python 3 support
28 # In python 3, unicode -> str, and str -> bytes
29 unicode = str
30
31 count_pass = 0
32 count_fail = 0
33 count_skip = 0
34
35 test_counter = 0
36 run_tests = []
37 exclude_tests = []
38
39 verbose = False
40
41 encoding = locale.getpreferredencoding()
42 if not encoding:
43 print("Guessing current system encoding failed. Consider setting $LANG variable.")
44 sys.exit(1)
45 else:
46 print("System encoding: " + encoding)
47
48 try:
49 unicode
50 except NameError:
51 # python 3 support
52 # In python 3, unicode -> str, and str -> bytes
53 unicode = str
54
55 def unicodise(string, encoding = "utf-8", errors = "replace"):
56 """
57 Convert 'string' to Unicode or raise an exception.
58 Config can't use toolbox from Utils that is itself using Config
59 """
60 if type(string) == unicode:
61 return string
62
63 try:
64 return unicode(string, encoding, errors)
65 except UnicodeDecodeError:
66 raise UnicodeDecodeError("Conversion to unicode failed: %r" % string)
67
68 # https://stackoverflow.com/questions/377017/test-if-executable-exists-in-python/377028#377028
69 def which(program):
70 def is_exe(fpath):
71 return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
72
73 fpath, fname = os.path.split(program)
74 if fpath:
75 if is_exe(program):
76 return program
77 else:
78 for path in os.environ["PATH"].split(os.pathsep):
79 path = path.strip('"')
80 exe_file = os.path.join(path, program)
81 if is_exe(exe_file):
82 return exe_file
83
84 return None
85
86 if which('curl') is not None:
87 have_curl = True
88 else:
89 have_curl = False
90
91 config_file = None
92 if os.getenv("HOME"):
93 config_file = os.path.join(unicodise(os.getenv("HOME"), encoding), ".s3cfg")
94 elif os.name == "nt" and os.getenv("USERPROFILE"):
95 config_file = os.path.join(unicodise(os.getenv("USERPROFILE"), encoding),
96 os.getenv("APPDATA") and unicodise(os.getenv("APPDATA"), encoding)
97 or 'Application Data',
98 "s3cmd.ini")
99
100
101 ## Unpack testsuite/ directory
102 if not os.path.isdir('testsuite') and os.path.isfile('testsuite.tar.gz'):
103 os.system("tar -xz -f testsuite.tar.gz")
104 if not os.path.isdir('testsuite'):
105 print("Something went wrong while unpacking testsuite.tar.gz")
106 sys.exit(1)
107
108 os.system("tar -xf testsuite/checksum.tar -C testsuite")
109 if not os.path.isfile('testsuite/checksum/cksum33.txt'):
110 print("Something went wrong while unpacking testsuite/checkum.tar")
111 sys.exit(1)
112
113 ## Fix up permissions for permission-denied tests
114 os.chmod("testsuite/permission-tests/permission-denied-dir", 0o444)
115 os.chmod("testsuite/permission-tests/permission-denied.txt", 0o000)
116
117 ## Patterns for Unicode tests
118 patterns = {}
119 patterns['UTF-8'] = u"ŪņЇЌœđЗ/☺ unicode € rocks ™"
120 patterns['GBK'] = u"12月31日/1-特色條目"
121
122 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
123 if not have_encoding and os.path.isfile('testsuite/encodings/%s.tar.gz' % encoding):
124 os.system("tar xvz -C testsuite/encodings -f testsuite/encodings/%s.tar.gz" % encoding)
125 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
126
127 if have_encoding:
128 #enc_base_remote = "%s/xyz/%s/" % (pbucket(1), encoding)
129 enc_pattern = patterns[encoding]
130 else:
131 print(encoding + " specific files not found.")
132
133 def unicodise(string):
134 if type(string) == unicode:
135 return string
136
137 return unicode(string, "UTF-8", "replace")
138
139 def deunicodise(string):
140 if type(string) != unicode:
141 return string
142
143 return string.encode("UTF-8", "replace")
144
145 if not os.path.isdir('testsuite/crappy-file-name'):
146 os.system("tar xvz -C testsuite -f testsuite/crappy-file-name.tar.gz")
147 # TODO: also unpack if the tarball is newer than the directory timestamp
148 # for instance when a new version was pulled from SVN.
149
150 def test(label, cmd_args = [], retcode = 0, must_find = [], must_not_find = [], must_find_re = [], must_not_find_re = [], stdin = None):
151 def command_output():
152 print("----")
153 print(" ".join([" " in arg and "'%s'" % arg or arg for arg in cmd_args]))
154 print("----")
155 print(stdout)
156 print("----")
157
158 def failure(message = ""):
159 global count_fail
160 if message:
161 message = u" (%r)" % message
162 print(u"\x1b[31;1mFAIL%s\x1b[0m" % (message))
163 count_fail += 1
164 command_output()
165 #return 1
166 sys.exit(1)
167 def success(message = ""):
168 global count_pass
169 if message:
170 message = " (%r)" % message
171 print("\x1b[32;1mOK\x1b[0m%s" % (message))
172 count_pass += 1
173 if verbose:
174 command_output()
175 return 0
176 def skip(message = ""):
177 global count_skip
178 if message:
179 message = " (%r)" % message
180 print("\x1b[33;1mSKIP\x1b[0m%s" % (message))
181 count_skip += 1
182 return 0
183 def compile_list(_list, regexps = False):
184 if regexps == False:
185 _list = [re.escape(item) for item in _list]
186
187 return [re.compile(item, re.MULTILINE) for item in _list]
188
189 global test_counter
190 test_counter += 1
191 print(("%3d %s " % (test_counter, label)).ljust(30, "."), end=' ')
192 sys.stdout.flush()
193
194 if run_tests.count(test_counter) == 0 or exclude_tests.count(test_counter) > 0:
195 return skip()
196
197 if not cmd_args:
198 return skip()
199
200 p = Popen(cmd_args, stdin = stdin, stdout = PIPE, stderr = STDOUT, universal_newlines = True, close_fds = True)
201 stdout, stderr = p.communicate()
202 if type(retcode) not in [list, tuple]: retcode = [retcode]
203 if p.returncode not in retcode:
204 return failure("retcode: %d, expected one of: %s" % (p.returncode, retcode))
205
206 if type(must_find) not in [ list, tuple ]: must_find = [must_find]
207 if type(must_find_re) not in [ list, tuple ]: must_find_re = [must_find_re]
208 if type(must_not_find) not in [ list, tuple ]: must_not_find = [must_not_find]
209 if type(must_not_find_re) not in [ list, tuple ]: must_not_find_re = [must_not_find_re]
210
211 find_list = []
212 find_list.extend(compile_list(must_find))
213 find_list.extend(compile_list(must_find_re, regexps = True))
214 find_list_patterns = []
215 find_list_patterns.extend(must_find)
216 find_list_patterns.extend(must_find_re)
217
218 not_find_list = []
219 not_find_list.extend(compile_list(must_not_find))
220 not_find_list.extend(compile_list(must_not_find_re, regexps = True))
221 not_find_list_patterns = []
222 not_find_list_patterns.extend(must_not_find)
223 not_find_list_patterns.extend(must_not_find_re)
224
225 for index in range(len(find_list)):
226 stdout = unicodise(stdout)
227 match = find_list[index].search(stdout)
228 if not match:
229 return failure("pattern not found: %s" % find_list_patterns[index])
230 for index in range(len(not_find_list)):
231 match = not_find_list[index].search(stdout)
232 if match:
233 return failure("pattern found: %s (match: %s)" % (not_find_list_patterns[index], match.group(0)))
234
235 return success()
236
237 def test_s3cmd(label, cmd_args = [], **kwargs):
238 if not cmd_args[0].endswith("s3cmd"):
239 cmd_args.insert(0, "python")
240 cmd_args.insert(1, "s3cmd")
241 if config_file:
242 cmd_args.insert(2, "-c")
243 cmd_args.insert(3, config_file)
244
245 return test(label, cmd_args, **kwargs)
246
247 def test_mkdir(label, dir_name):
248 if os.name in ("posix", "nt"):
249 cmd = ['mkdir', '-p']
250 else:
251 print("Unknown platform: %s" % os.name)
252 sys.exit(1)
253 cmd.append(dir_name)
254 return test(label, cmd)
255
256 def test_rmdir(label, dir_name):
257 if os.path.isdir(dir_name):
258 if os.name == "posix":
259 cmd = ['rm', '-rf']
260 elif os.name == "nt":
261 cmd = ['rmdir', '/s/q']
262 else:
263 print("Unknown platform: %s" % os.name)
264 sys.exit(1)
265 cmd.append(dir_name)
266 return test(label, cmd)
267 else:
268 return test(label, [])
269
270 def test_flushdir(label, dir_name):
271 test_rmdir(label + "(rm)", dir_name)
272 return test_mkdir(label + "(mk)", dir_name)
273
274 def test_copy(label, src_file, dst_file):
275 if os.name == "posix":
276 cmd = ['cp', '-f']
277 elif os.name == "nt":
278 cmd = ['copy']
279 else:
280 print("Unknown platform: %s" % os.name)
281 sys.exit(1)
282 cmd.append(src_file)
283 cmd.append(dst_file)
284 return test(label, cmd)
285
286 def test_curl_HEAD(label, src_file, **kwargs):
287 cmd = ['curl', '--silent', '--head', '-include', '--location']
288 cmd.append(src_file)
289 return test(label, cmd, **kwargs)
290
291 bucket_prefix = u"%s-" % getpass.getuser().lower()
292
293 argv = sys.argv[1:]
294 while argv:
295 arg = argv.pop(0)
296 if arg.startswith('--bucket-prefix='):
297 print("Usage: '--bucket-prefix PREFIX', not '--bucket-prefix=PREFIX'")
298 sys.exit(0)
299 if arg in ("-h", "--help"):
300 print("%s A B K..O -N" % sys.argv[0])
301 print("Run tests number A, B and K through to O, except for N")
302 sys.exit(0)
303
304 if arg in ("-c", "--config"):
305 config_file = argv.pop(0)
306 continue
307 if arg in ("-l", "--list"):
308 exclude_tests = range(0, 999)
309 break
310 if arg in ("-v", "--verbose"):
311 verbose = True
312 continue
313 if arg in ("-p", "--bucket-prefix"):
314 try:
315 bucket_prefix = argv.pop(0)
316 except IndexError:
317 print("Bucket prefix option must explicitly supply a bucket name prefix")
318 sys.exit(0)
319 continue
320 if ".." in arg:
321 range_idx = arg.find("..")
322 range_start = arg[:range_idx] or 0
323 range_end = arg[range_idx+2:] or 999
324 run_tests.extend(range(int(range_start), int(range_end) + 1))
325 elif arg.startswith("-"):
326 exclude_tests.append(int(arg[1:]))
327 else:
328 run_tests.append(int(arg))
329
330 print("Using bucket prefix: '%s'" % bucket_prefix)
331
332 cfg = S3.Config.Config(config_file)
333
334 if not run_tests:
335 run_tests = range(0, 999)
336
337 # helper functions for generating bucket names
338 def bucket(tail):
339 '''Test bucket name'''
340 label = 'autotest'
341 if str(tail) == '3':
342 label = 'autotest'
343 return '%ss3cmd-%s-%s' % (bucket_prefix, label, tail)
344
345 def pbucket(tail):
346 '''Like bucket(), but prepends "s3://" for you'''
347 return 's3://' + bucket(tail)
348
349 ## ====== Remove test buckets
350 test_s3cmd("Remove test buckets", ['rb', '-r', '--force', pbucket(1), pbucket(2), pbucket(3)])
351
352 ## ====== verify they were removed
353 test_s3cmd("Verify no test buckets", ['ls'],
354 must_not_find = [pbucket(1), pbucket(2), pbucket(3)])
355
356
357 ## ====== Create one bucket (EU)
358 # Disabled for minio
359 #test_s3cmd("Create one bucket (EU)", ['mb', '--bucket-location=EU', pbucket(1)],
360 # must_find = "Bucket '%s/' created" % pbucket(1))
361 test_s3cmd("Create one bucket", ['mb', pbucket(1)],
362 must_find = "Bucket '%s/' created" % pbucket(1))
363
364
365 ## ====== Create multiple buckets
366 test_s3cmd("Create multiple buckets", ['mb', pbucket(2), pbucket(3)],
367 must_find = [ "Bucket '%s/' created" % pbucket(2), "Bucket '%s/' created" % pbucket(3)])
368
369
370 ## ====== Invalid bucket name
371 test_s3cmd("Invalid bucket name", ["mb", "--bucket-location=EU", pbucket('EU')],
372 retcode = EX_USAGE,
373 must_find = "ERROR: Parameter problem: Bucket name '%s' contains disallowed character" % bucket('EU'),
374 must_not_find_re = "Bucket.*created")
375
376
377 ## ====== Buckets list
378 # Modified for Minio
379 test_s3cmd("Buckets list", ["ls"],
380 must_find = [ "autotest-1", "autotest-2", "autotest-3" ], must_not_find_re = "autotest-EU")
381
382
383 ## ====== Sync to S3
384 # Modified for Minio (exclude crappy dir)
385 test_s3cmd("Sync to S3", ['sync', 'testsuite/', pbucket(1) + '/xyz/', '--exclude', 'demo/*', '--exclude', '*.png', '--no-encrypt', '--exclude-from', 'testsuite/exclude.encodings', '--exclude', 'crappy-file-name/*' ],
386 must_find = [ "ERROR: Upload of 'testsuite/permission-tests/permission-denied.txt' is not possible (Reason: Permission denied)",
387 ],
388 must_not_find_re = [ "demo/", "^(?!WARNING: Skipping).*\.png$", "permission-denied-dir" ],
389 retcode = EX_PARTIAL)
390
391 if have_encoding:
392 ## ====== Sync UTF-8 / GBK / ... to S3
393 test_s3cmd(u"Sync %s to S3" % encoding, ['sync', 'testsuite/encodings/' + encoding, '%s/xyz/encodings/' % pbucket(1), '--exclude', 'demo/*', '--no-encrypt' ],
394 must_find = [ u"'testsuite/encodings/%(encoding)s/%(pattern)s' -> '%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s'" % { 'encoding' : encoding, 'pattern' : enc_pattern , 'pbucket' : pbucket(1)} ])
395
396
397 ## ====== List bucket content
398 test_s3cmd("List bucket content", ['ls', '%s/xyz/' % pbucket(1) ],
399 must_find_re = [ u"DIR +%s/xyz/binary/$" % pbucket(1) , u"DIR +%s/xyz/etc/$" % pbucket(1) ],
400 must_not_find = [ u"random-crap.md5", u"/demo" ])
401
402
403 ## ====== List bucket recursive
404 must_find = [ u"%s/xyz/binary/random-crap.md5" % pbucket(1) ]
405 if have_encoding:
406 must_find.append(u"%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s" % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
407
408 test_s3cmd("List bucket recursive", ['ls', '--recursive', pbucket(1)],
409 must_find = must_find,
410 must_not_find = [ "logo.png" ])
411
412 ## ====== FIXME
413 test_s3cmd("Recursive put", ['put', '--recursive', 'testsuite/etc', '%s/xyz/' % pbucket(1) ])
414
415
416 ## ====== Clean up local destination dir
417 test_flushdir("Clean testsuite-out/", "testsuite-out")
418
419 ## ====== Put from stdin
420 f = open('testsuite/single-file/single-file.txt', 'r')
421 test_s3cmd("Put from stdin", ['put', '-', '%s/single-file/single-file.txt' % pbucket(1)],
422 must_find = ["'<stdin>' -> '%s/single-file/single-file.txt'" % pbucket(1)],
423 stdin = f)
424 f.close()
425
426 ## ====== Multipart put
427 os.system('mkdir -p testsuite-out')
428 os.system('dd if=/dev/urandom of=testsuite-out/urandom.bin bs=1M count=16 > /dev/null 2>&1')
429 test_s3cmd("Put multipart", ['put', '--multipart-chunk-size-mb=5', 'testsuite-out/urandom.bin', '%s/urandom.bin' % pbucket(1)],
430 must_not_find = ['abortmp'])
431
432 ## ====== Multipart put from stdin
433 f = open('testsuite-out/urandom.bin', 'r')
434 test_s3cmd("Multipart large put from stdin", ['put', '--multipart-chunk-size-mb=5', '-', '%s/urandom2.bin' % pbucket(1)],
435 must_find = ['%s/urandom2.bin' % pbucket(1)],
436 must_not_find = ['abortmp'],
437 stdin = f)
438 f.close()
439
440 ## ====== Clean up local destination dir
441 test_flushdir("Clean testsuite-out/", "testsuite-out")
442
443 ## ====== Moving things without trailing '/'
444 os.system('dd if=/dev/urandom of=testsuite-out/urandom1.bin bs=1k count=1 > /dev/null 2>&1')
445 os.system('dd if=/dev/urandom of=testsuite-out/urandom2.bin bs=1k count=1 > /dev/null 2>&1')
446 test_s3cmd("Put multiple files", ['put', 'testsuite-out/urandom1.bin', 'testsuite-out/urandom2.bin', '%s/' % pbucket(1)],
447 must_find = ["%s/urandom1.bin" % pbucket(1), "%s/urandom2.bin" % pbucket(1)])
448
449 test_s3cmd("Move without '/'", ['mv', '%s/urandom1.bin' % pbucket(1), '%s/urandom2.bin' % pbucket(1), '%s/dir' % pbucket(1)],
450 retcode = 64,
451 must_find = ['Destination must be a directory'])
452
453 test_s3cmd("Move recursive w/a '/'",
454 ['-r', 'mv', '%s/dir1' % pbucket(1), '%s/dir2' % pbucket(1)],
455 retcode = 64,
456 must_find = ['Destination must be a directory'])
457
458 ## ====== Moving multiple files into directory with trailing '/'
459 must_find = ["'%s/urandom1.bin' -> '%s/dir/urandom1.bin'" % (pbucket(1),pbucket(1)), "'%s/urandom2.bin' -> '%s/dir/urandom2.bin'" % (pbucket(1),pbucket(1))]
460 must_not_find = ["'%s/urandom1.bin' -> '%s/dir'" % (pbucket(1),pbucket(1)), "'%s/urandom2.bin' -> '%s/dir'" % (pbucket(1),pbucket(1))]
461 test_s3cmd("Move multiple files",
462 ['mv', '%s/urandom1.bin' % pbucket(1), '%s/urandom2.bin' % pbucket(1), '%s/dir/' % pbucket(1)],
463 must_find = must_find,
464 must_not_find = must_not_find)
465
466 ## ====== Clean up local destination dir
467 test_flushdir("Clean testsuite-out/", "testsuite-out")
468
469 ## ====== Sync from S3
470 must_find = [ "'%s/xyz/binary/random-crap.md5' -> 'testsuite-out/xyz/binary/random-crap.md5'" % pbucket(1) ]
471 if have_encoding:
472 must_find.append(u"'%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s' -> 'testsuite-out/xyz/encodings/%(encoding)s/%(pattern)s' " % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
473 test_s3cmd("Sync from S3", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
474 must_find = must_find)
475
476
477 ## ====== Remove 'demo' directory
478 test_rmdir("Remove 'dir-test/'", "testsuite-out/xyz/dir-test/")
479
480
481 ## ====== Create dir with name of a file
482 test_mkdir("Create file-dir dir", "testsuite-out/xyz/dir-test/file-dir")
483
484
485 ## ====== Skip dst dirs
486 test_s3cmd("Skip over dir", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
487 must_find = "ERROR: Download of 'xyz/dir-test/file-dir' failed (Reason: testsuite-out/xyz/dir-test/file-dir is a directory)",
488 retcode = EX_PARTIAL)
489
490
491 ## ====== Clean up local destination dir
492 test_flushdir("Clean testsuite-out/", "testsuite-out")
493
494
495 ## ====== Put public, guess MIME
496 test_s3cmd("Put public, guess MIME", ['put', '--guess-mime-type', '--acl-public', 'testsuite/etc/logo.png', '%s/xyz/etc/logo.png' % pbucket(1)],
497 must_find = [ "-> '%s/xyz/etc/logo.png'" % pbucket(1) ])
498
499
500 ## ====== Retrieve from URL
501 # Minio: disabled
502 #if have_curl:
503 # test_curl_HEAD("Retrieve from URL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
504 # must_find_re = ['Content-Length: 22059'])
505
506 ## ====== Change ACL to Private
507 # Minio: disabled
508 #test_s3cmd("Change ACL to Private", ['setacl', '--acl-private', '%s/xyz/etc/l*.png' % pbucket(1)],
509 # must_find = [ "logo.png: ACL set to Private" ])
510
511
512 ## ====== Verify Private ACL
513 # Minio: disabled
514 #if have_curl:
515 # test_curl_HEAD("Verify Private ACL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
516 # must_find_re = [ '403 Forbidden' ])
517
518
519 ## ====== Change ACL to Public
520 # Minio: disabled
521 #test_s3cmd("Change ACL to Public", ['setacl', '--acl-public', '--recursive', '%s/xyz/etc/' % pbucket(1) , '-v'],
522 # must_find = [ "logo.png: ACL set to Public" ])
523
524
525 ## ====== Verify Public ACL
526 # Minio: disabled
527 #if have_curl:
528 # test_curl_HEAD("Verify Public ACL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
529 # must_find_re = [ '200 OK',
530 # 'Content-Length: 22059'])
531
532
533 ## ====== Sync more to S3
534 # Modified for Minio (exclude crappy dir)
535 test_s3cmd("Sync more to S3", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--exclude', 'crappy-file-name/*' ],
536 must_find = [ "'testsuite/demo/some-file.xml' -> '%s/xyz/demo/some-file.xml' " % pbucket(1) ],
537 must_not_find = [ "'testsuite/etc/linked.png' -> '%s/xyz/etc/linked.png'" % pbucket(1) ],
538 retcode = EX_PARTIAL)
539
540
541 ## ====== Don't check MD5 sum on Sync
542 test_copy("Change file cksum1.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum1.txt")
543 test_copy("Change file cksum33.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum33.txt")
544 # Modified for Minio (exclude crappy dir)
545 test_s3cmd("Don't check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--no-check-md5', '--exclude', 'crappy-file-name/*'],
546 must_find = [ "cksum33.txt" ],
547 must_not_find = [ "cksum1.txt" ],
548 retcode = EX_PARTIAL)
549
550
551 ## ====== Check MD5 sum on Sync
552 # Modified for Minio (exclude crappy dir)
553 test_s3cmd("Check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--check-md5', '--exclude', 'crappy-file-name/*'],
554 must_find = [ "cksum1.txt" ],
555 retcode = EX_PARTIAL)
556
557
558 ## ====== Rename within S3
559 test_s3cmd("Rename within S3", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
560 must_find = [ "move: '%s/xyz/etc/logo.png' -> '%s/xyz/etc2/Logo.PNG'" % (pbucket(1), pbucket(1))])
561
562
563 ## ====== Rename (NoSuchKey)
564 test_s3cmd("Rename (NoSuchKey)", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
565 retcode = EX_NOTFOUND,
566 must_find_re = [ 'Key not found' ],
567 must_not_find = [ "move: '%s/xyz/etc/logo.png' -> '%s/xyz/etc2/Logo.PNG'" % (pbucket(1), pbucket(1)) ])
568
569 ## ====== Sync more from S3 (invalid src)
570 test_s3cmd("Sync more from S3 (invalid src)", ['sync', '--delete-removed', '%s/xyz/DOESNOTEXIST' % pbucket(1), 'testsuite-out'],
571 must_not_find = [ "delete: 'testsuite-out/logo.png'" ])
572
573 ## ====== Sync more from S3
574 test_s3cmd("Sync more from S3", ['sync', '--delete-removed', '%s/xyz' % pbucket(1), 'testsuite-out'],
575 must_find = [ "'%s/xyz/etc2/Logo.PNG' -> 'testsuite-out/xyz/etc2/Logo.PNG'" % pbucket(1),
576 "'%s/xyz/demo/some-file.xml' -> 'testsuite-out/xyz/demo/some-file.xml'" % pbucket(1) ],
577 must_not_find_re = [ "not-deleted.*etc/logo.png", "delete: 'testsuite-out/logo.png'" ])
578
579
580 ## ====== Make dst dir for get
581 test_rmdir("Remove dst dir for get", "testsuite-out")
582
583
584 ## ====== Get multiple files
585 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
586 retcode = EX_USAGE,
587 must_find = [ 'Destination must be a directory or stdout when downloading multiple sources.' ])
588
589 ## ====== put/get non-ASCII filenames
590 test_s3cmd("Put unicode filenames", ['put', u'testsuite/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo', u'%s/xyz/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo' % pbucket(1)],
591 retcode = 0,
592 must_find = [ '->' ])
593
594
595 ## ====== Make dst dir for get
596 test_mkdir("Make dst dir for get", "testsuite-out")
597
598
599 ## ====== put/get non-ASCII filenames
600 test_s3cmd("Get unicode filenames", ['get', u'%s/xyz/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo' % pbucket(1), 'testsuite-out'],
601 retcode = 0,
602 must_find = [ '->' ])
603
604
605 ## ====== Get multiple files
606 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
607 must_find = [ u"-> 'testsuite-out/Logo.PNG'",
608 u"-> 'testsuite-out/AtomicClockRadio.ttf'" ])
609
610 ## ====== Upload files differing in capitalisation
611 test_s3cmd("blah.txt / Blah.txt", ['put', '-r', 'testsuite/blahBlah', pbucket(1)],
612 must_find = [ '%s/blahBlah/Blah.txt' % pbucket(1), '%s/blahBlah/blah.txt' % pbucket(1)])
613
614 ## ====== Copy between buckets
615 test_s3cmd("Copy between buckets", ['cp', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc2/logo.png' % pbucket(3)],
616 must_find = [ "remote copy: '%s/xyz/etc2/Logo.PNG' -> '%s/xyz/etc2/logo.png'" % (pbucket(1), pbucket(3)) ])
617
618 ## ====== Recursive copy
619 test_s3cmd("Recursive copy, set ACL", ['cp', '-r', '--acl-public', '%s/xyz/' % pbucket(1), '%s/copy/' % pbucket(2), '--exclude', 'demo/dir?/*.txt', '--exclude', 'non-printables*'],
620 must_find = [ "remote copy: '%s/xyz/etc2/Logo.PNG' -> '%s/copy/etc2/Logo.PNG'" % (pbucket(1), pbucket(2)),
621 "remote copy: '%s/xyz/blahBlah/Blah.txt' -> '%s/copy/blahBlah/Blah.txt'" % (pbucket(1), pbucket(2)),
622 "remote copy: '%s/xyz/blahBlah/blah.txt' -> '%s/copy/blahBlah/blah.txt'" % (pbucket(1), pbucket(2)) ],
623 must_not_find = [ "demo/dir1/file1-1.txt" ])
624
625 ## ====== Verify ACL and MIME type
626 # Minio: disable acl check, not supported by minio
627 test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
628 must_find_re = [ "MIME type:.*image/png" ])
629
630 ## ====== modify MIME type
631 # Minio: disable acl check, not supported by minio
632 # Minio: modifying mime type alone not allowed as copy of same file for them
633 #test_s3cmd("Modify MIME type", ['modify', '--mime-type=binary/octet-stream', '%s/copy/etc2/Logo.PNG' % pbucket(2) ])
634
635 #test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
636 # must_find_re = [ "MIME type:.*binary/octet-stream" ])
637
638 # Minio: disable acl check, not supported by minio
639 #test_s3cmd("Modify MIME type back", ['modify', '--mime-type=image/png', '%s/copy/etc2/Logo.PNG' % pbucket(2) ])
640
641 # Minio: disable acl check, not supported by minio
642 #test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
643 # must_find_re = [ "MIME type:.*image/png" ])
644
645 #test_s3cmd("Add cache-control header", ['modify', '--add-header=cache-control: max-age=3600, public', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
646 # must_find_re = [ "modify: .*" ])
647
648 #if have_curl:
649 # test_curl_HEAD("HEAD check Cache-Control present", 'http://%s.%s/copy/etc2/Logo.PNG' % (bucket(2), cfg.host_base),
650 # must_find_re = [ "Cache-Control: max-age=3600" ])
651
652 #test_s3cmd("Remove cache-control header", ['modify', '--remove-header=cache-control', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
653 # must_find_re = [ "modify: .*" ])
654
655 #if have_curl:
656 # test_curl_HEAD("HEAD check Cache-Control not present", 'http://%s.%s/copy/etc2/Logo.PNG' % (bucket(2), cfg.host_base),
657 # must_not_find_re = [ "Cache-Control: max-age=3600" ])
658
659 ## ====== sign
660 test_s3cmd("sign string", ['sign', 's3cmd'], must_find_re = ["Signature:"])
661 test_s3cmd("signurl time", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), str(int(time.time()) + 60)], must_find_re = ["http://"])
662 test_s3cmd("signurl time offset", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), '+60'], must_find_re = ["https?://"])
663 test_s3cmd("signurl content disposition and type", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), '+60', '--content-disposition=inline; filename=video.mp4', '--content-type=video/mp4'], must_find_re = [ 'response-content-disposition', 'response-content-type' ] )
664
665 ## ====== Rename within S3
666 test_s3cmd("Rename within S3", ['mv', '%s/copy/etc2/Logo.PNG' % pbucket(2), '%s/copy/etc/logo.png' % pbucket(2)],
667 must_find = [ "move: '%s/copy/etc2/Logo.PNG' -> '%s/copy/etc/logo.png'" % (pbucket(2), pbucket(2))])
668
669 ## ====== Sync between buckets
670 test_s3cmd("Sync remote2remote", ['sync', '%s/xyz/' % pbucket(1), '%s/copy/' % pbucket(2), '--delete-removed', '--exclude', 'non-printables*'],
671 must_find = [ "remote copy: '%s/xyz/demo/dir1/file1-1.txt' -> '%s/copy/demo/dir1/file1-1.txt'" % (pbucket(1), pbucket(2)),
672 "remote copy: 'etc/logo.png' -> 'etc2/Logo.PNG'",
673 "delete: '%s/copy/etc/logo.png'" % pbucket(2) ],
674 must_not_find = [ "blah.txt" ])
675
676 ## ====== Don't Put symbolic link
677 test_s3cmd("Don't put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1), '--exclude', 'crappy-file-name/*'],
678 retcode = EX_USAGE,
679 must_find = ["WARNING: Skipping over symbolic link: testsuite/etc/linked1.png"],
680 must_not_find_re = ["^(?!WARNING: Skipping).*linked1.png"])
681
682 ## ====== Put symbolic link
683 test_s3cmd("Put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1),'--follow-symlinks' , '--exclude', 'crappy-file-name/*'],
684 must_find = [ "'testsuite/etc/linked1.png' -> '%s/xyz/linked1.png'" % pbucket(1)])
685
686 ## ====== Sync symbolic links
687 test_s3cmd("Sync symbolic links", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--follow-symlinks', '--exclude', 'crappy-file-name/*' ],
688 must_find = ["remote copy: 'etc2/Logo.PNG' -> 'etc/linked.png'"],
689 # Don't want to recursively copy linked directories!
690 must_not_find_re = ["etc/more/linked-dir/more/give-me-more.txt",
691 "etc/brokenlink.png"],
692 retcode = EX_PARTIAL)
693
694 ## ====== Multi source move
695 test_s3cmd("Multi-source move", ['mv', '-r', '%s/copy/blahBlah/Blah.txt' % pbucket(2), '%s/copy/etc/' % pbucket(2), '%s/moved/' % pbucket(2)],
696 must_find = [ "move: '%s/copy/blahBlah/Blah.txt' -> '%s/moved/Blah.txt'" % (pbucket(2), pbucket(2)),
697 "move: '%s/copy/etc/AtomicClockRadio.ttf' -> '%s/moved/AtomicClockRadio.ttf'" % (pbucket(2), pbucket(2)),
698 "move: '%s/copy/etc/TypeRa.ttf' -> '%s/moved/TypeRa.ttf'" % (pbucket(2), pbucket(2)) ],
699 must_not_find = [ "blah.txt" ])
700
701 ## ====== Verify move
702 test_s3cmd("Verify move", ['ls', '-r', pbucket(2)],
703 must_find = [ "%s/moved/Blah.txt" % pbucket(2),
704 "%s/moved/AtomicClockRadio.ttf" % pbucket(2),
705 "%s/moved/TypeRa.ttf" % pbucket(2),
706 "%s/copy/blahBlah/blah.txt" % pbucket(2) ],
707 must_not_find = [ "%s/copy/blahBlah/Blah.txt" % pbucket(2),
708 "%s/copy/etc/AtomicClockRadio.ttf" % pbucket(2),
709 "%s/copy/etc/TypeRa.ttf" % pbucket(2) ])
710
711 ## ====== List all
712 test_s3cmd("List all", ['la'],
713 must_find = [ "%s/urandom.bin" % pbucket(1)])
714
715 ## ====== Simple delete
716 test_s3cmd("Simple delete", ['del', '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
717 must_find = [ "delete: '%s/xyz/etc2/Logo.PNG'" % pbucket(1) ])
718
719 ## ====== Simple delete with rm
720 test_s3cmd("Simple delete with rm", ['rm', '%s/xyz/test_rm/TypeRa.ttf' % pbucket(1)],
721 must_find = [ "delete: '%s/xyz/test_rm/TypeRa.ttf'" % pbucket(1) ])
722
723 ## ====== Create expiration rule with days and prefix
724 # Minio: disabled
725 #test_s3cmd("Create expiration rule with days and prefix", ['expire', pbucket(1), '--expiry-days=365', '--expiry-prefix=log/'],
726 # must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
727
728 ## ====== Create expiration rule with date and prefix
729 # Minio: disabled
730 #test_s3cmd("Create expiration rule with date and prefix", ['expire', pbucket(1), '--expiry-date=2012-12-31T00:00:00.000Z', '--expiry-prefix=log/'],
731 # must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
732
733 ## ====== Create expiration rule with days only
734 # Minio: disabled
735 #test_s3cmd("Create expiration rule with days only", ['expire', pbucket(1), '--expiry-days=365'],
736 # must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
737
738 ## ====== Create expiration rule with date only
739 # Minio: disabled
740 #test_s3cmd("Create expiration rule with date only", ['expire', pbucket(1), '--expiry-date=2012-12-31T00:00:00.000Z'],
741 # must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
742
743 ## ====== Get current expiration setting
744 # Minio: disabled
745 #test_s3cmd("Get current expiration setting", ['info', pbucket(1)],
746 # must_find = [ "Expiration Rule: all objects in this bucket will expire in '2012-12-31T00:00:00.000Z'"])
747
748 ## ====== Delete expiration rule
749 # Minio: disabled
750 #test_s3cmd("Delete expiration rule", ['expire', pbucket(1)],
751 # must_find = [ "Bucket '%s/': expiration configuration is deleted." % pbucket(1)])
752
753 ## ====== set Requester Pays flag
754 # Minio: disabled
755 #test_s3cmd("Set requester pays", ['payer', '--requester-pays', pbucket(2)])
756
757 ## ====== get Requester Pays flag
758 # Minio: disabled
759 #test_s3cmd("Get requester pays flag", ['info', pbucket(2)],
760 # must_find = [ "Payer: Requester"])
761
762 ## ====== ls using Requester Pays flag
763 # Minio: disabled
764 #test_s3cmd("ls using requester pays flag", ['ls', '--requester-pays', pbucket(2)])
765
766 ## ====== clear Requester Pays flag
767 # Minio: disabled
768 #test_s3cmd("Clear requester pays", ['payer', pbucket(2)])
769
770 ## ====== get Requester Pays flag
771 # Minio: disabled
772 #test_s3cmd("Get requester pays flag", ['info', pbucket(2)],
773 # must_find = [ "Payer: BucketOwner"])
774
775 ## ====== Recursive delete maximum exceeed
776 test_s3cmd("Recursive delete maximum exceeded", ['del', '--recursive', '--max-delete=1', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
777 must_not_find = [ "delete: '%s/xyz/etc/TypeRa.ttf'" % pbucket(1) ])
778
779 ## ====== Recursive delete
780 test_s3cmd("Recursive delete", ['del', '--recursive', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
781 must_find = [ "delete: '%s/xyz/etc/TypeRa.ttf'" % pbucket(1) ],
782 must_find_re = [ "delete: '.*/etc/logo.png'" ],
783 must_not_find = [ "AtomicClockRadio.ttf" ])
784
785 ## ====== Recursive delete with rm
786 test_s3cmd("Recursive delete with rm", ['rm', '--recursive', '--exclude', 'Atomic*', '%s/xyz/test_rm' % pbucket(1)],
787 must_find = [ "delete: '%s/xyz/test_rm/more/give-me-more.txt'" % pbucket(1) ],
788 must_find_re = [ "delete: '.*/test_rm/logo.png'" ],
789 must_not_find = [ "AtomicClockRadio.ttf" ])
790
791 ## ====== Recursive delete all
792 test_s3cmd("Recursive delete all", ['del', '--recursive', '--force', pbucket(1)],
793 must_find_re = [ "delete: '.*binary/random-crap'" ])
794
795 ## ====== Remove empty bucket
796 test_s3cmd("Remove empty bucket", ['rb', pbucket(1)],
797 must_find = [ "Bucket '%s/' removed" % pbucket(1) ])
798
799 ## ====== Remove remaining buckets
800 test_s3cmd("Remove remaining buckets", ['rb', '--recursive', pbucket(2), pbucket(3)],
801 must_find = [ "Bucket '%s/' removed" % pbucket(2),
802 "Bucket '%s/' removed" % pbucket(3) ])
803
804 # vim:et:ts=4:sts=4:ai
+783
-0
run-tests.py less more
0 #!/usr/bin/env python2
1 # -*- coding=utf-8 -*-
2
3 ## Amazon S3cmd - testsuite
4 ## Author: Michal Ludvig <michal@logix.cz>
5 ## http://www.logix.cz/michal
6 ## License: GPL Version 2
7 ## Copyright: TGRMN Software and contributors
8
9 from __future__ import absolute_import, print_function
10
11 import sys
12 import os
13 import re
14 import time
15 from subprocess import Popen, PIPE, STDOUT
16 import locale
17 import getpass
18 import S3.Exceptions
19 import S3.Config
20 from S3.ExitCodes import *
21
22 PY3 = (sys.version_info >= (3,0))
23
24 try:
25 unicode
26 except NameError:
27 # python 3 support
28 # In python 3, unicode -> str, and str -> bytes
29 unicode = str
30
31 count_pass = 0
32 count_fail = 0
33 count_skip = 0
34
35 test_counter = 0
36 run_tests = []
37 exclude_tests = []
38
39 verbose = False
40
41 encoding = locale.getpreferredencoding()
42 if not encoding:
43 print("Guessing current system encoding failed. Consider setting $LANG variable.")
44 sys.exit(1)
45 else:
46 print("System encoding: " + encoding)
47
48 try:
49 unicode
50 except NameError:
51 # python 3 support
52 # In python 3, unicode -> str, and str -> bytes
53 unicode = str
54
55 def unicodise(string, encoding = "utf-8", errors = "replace"):
56 """
57 Convert 'string' to Unicode or raise an exception.
58 Config can't use toolbox from Utils that is itself using Config
59 """
60 if type(string) == unicode:
61 return string
62
63 try:
64 return unicode(string, encoding, errors)
65 except UnicodeDecodeError:
66 raise UnicodeDecodeError("Conversion to unicode failed: %r" % string)
67
68 # https://stackoverflow.com/questions/377017/test-if-executable-exists-in-python/377028#377028
69 def which(program):
70 def is_exe(fpath):
71 return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
72
73 fpath, fname = os.path.split(program)
74 if fpath:
75 if is_exe(program):
76 return program
77 else:
78 for path in os.environ["PATH"].split(os.pathsep):
79 path = path.strip('"')
80 exe_file = os.path.join(path, program)
81 if is_exe(exe_file):
82 return exe_file
83
84 return None
85
86 if which('curl') is not None:
87 have_curl = True
88 else:
89 have_curl = False
90
91 config_file = None
92 if os.getenv("HOME"):
93 config_file = os.path.join(unicodise(os.getenv("HOME"), encoding), ".s3cfg")
94 elif os.name == "nt" and os.getenv("USERPROFILE"):
95 config_file = os.path.join(unicodise(os.getenv("USERPROFILE"), encoding),
96 os.getenv("APPDATA") and unicodise(os.getenv("APPDATA"), encoding)
97 or 'Application Data',
98 "s3cmd.ini")
99
100
101 ## Unpack testsuite/ directory
102 if not os.path.isdir('testsuite') and os.path.isfile('testsuite.tar.gz'):
103 os.system("tar -xz -f testsuite.tar.gz")
104 if not os.path.isdir('testsuite'):
105 print("Something went wrong while unpacking testsuite.tar.gz")
106 sys.exit(1)
107
108 os.system("tar -xf testsuite/checksum.tar -C testsuite")
109 if not os.path.isfile('testsuite/checksum/cksum33.txt'):
110 print("Something went wrong while unpacking testsuite/checkum.tar")
111 sys.exit(1)
112
113 ## Fix up permissions for permission-denied tests
114 os.chmod("testsuite/permission-tests/permission-denied-dir", 0o444)
115 os.chmod("testsuite/permission-tests/permission-denied.txt", 0o000)
116
117 ## Patterns for Unicode tests
118 patterns = {}
119 patterns['UTF-8'] = u"ŪņЇЌœđЗ/☺ unicode € rocks ™"
120 patterns['GBK'] = u"12月31日/1-特色條目"
121
122 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
123 if not have_encoding and os.path.isfile('testsuite/encodings/%s.tar.gz' % encoding):
124 os.system("tar xvz -C testsuite/encodings -f testsuite/encodings/%s.tar.gz" % encoding)
125 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
126
127 if have_encoding:
128 #enc_base_remote = "%s/xyz/%s/" % (pbucket(1), encoding)
129 enc_pattern = patterns[encoding]
130 else:
131 print(encoding + " specific files not found.")
132
133 def unicodise(string):
134 if type(string) == unicode:
135 return string
136
137 return unicode(string, "UTF-8", "replace")
138
139 def deunicodise(string):
140 if type(string) != unicode:
141 return string
142
143 return string.encode("UTF-8", "replace")
144
145 if not os.path.isdir('testsuite/crappy-file-name'):
146 os.system("tar xvz -C testsuite -f testsuite/crappy-file-name.tar.gz")
147 # TODO: also unpack if the tarball is newer than the directory timestamp
148 # for instance when a new version was pulled from SVN.
149
150 def test(label, cmd_args = [], retcode = 0, must_find = [], must_not_find = [], must_find_re = [], must_not_find_re = [], stdin = None):
151 def command_output():
152 print("----")
153 print(" ".join([" " in arg and "'%s'" % arg or arg for arg in cmd_args]))
154 print("----")
155 print(stdout)
156 print("----")
157
158 def failure(message = ""):
159 global count_fail
160 if message:
161 message = u" (%r)" % message
162 print(u"\x1b[31;1mFAIL%s\x1b[0m" % (message))
163 count_fail += 1
164 command_output()
165 #return 1
166 sys.exit(1)
167 def success(message = ""):
168 global count_pass
169 if message:
170 message = " (%r)" % message
171 print("\x1b[32;1mOK\x1b[0m%s" % (message))
172 count_pass += 1
173 if verbose:
174 command_output()
175 return 0
176 def skip(message = ""):
177 global count_skip
178 if message:
179 message = " (%r)" % message
180 print("\x1b[33;1mSKIP\x1b[0m%s" % (message))
181 count_skip += 1
182 return 0
183 def compile_list(_list, regexps = False):
184 if regexps == False:
185 _list = [re.escape(item) for item in _list]
186
187 return [re.compile(item, re.MULTILINE) for item in _list]
188
189 global test_counter
190 test_counter += 1
191 print(("%3d %s " % (test_counter, label)).ljust(30, "."), end=' ')
192 sys.stdout.flush()
193
194 if run_tests.count(test_counter) == 0 or exclude_tests.count(test_counter) > 0:
195 return skip()
196
197 if not cmd_args:
198 return skip()
199
200 p = Popen(cmd_args, stdin = stdin, stdout = PIPE, stderr = STDOUT, universal_newlines = True, close_fds = True)
201 stdout, stderr = p.communicate()
202 if type(retcode) not in [list, tuple]: retcode = [retcode]
203 if p.returncode not in retcode:
204 return failure("retcode: %d, expected one of: %s" % (p.returncode, retcode))
205
206 if type(must_find) not in [ list, tuple ]: must_find = [must_find]
207 if type(must_find_re) not in [ list, tuple ]: must_find_re = [must_find_re]
208 if type(must_not_find) not in [ list, tuple ]: must_not_find = [must_not_find]
209 if type(must_not_find_re) not in [ list, tuple ]: must_not_find_re = [must_not_find_re]
210
211 find_list = []
212 find_list.extend(compile_list(must_find))
213 find_list.extend(compile_list(must_find_re, regexps = True))
214 find_list_patterns = []
215 find_list_patterns.extend(must_find)
216 find_list_patterns.extend(must_find_re)
217
218 not_find_list = []
219 not_find_list.extend(compile_list(must_not_find))
220 not_find_list.extend(compile_list(must_not_find_re, regexps = True))
221 not_find_list_patterns = []
222 not_find_list_patterns.extend(must_not_find)
223 not_find_list_patterns.extend(must_not_find_re)
224
225 for index in range(len(find_list)):
226 stdout = unicodise(stdout)
227 match = find_list[index].search(stdout)
228 if not match:
229 return failure("pattern not found: %s" % find_list_patterns[index])
230 for index in range(len(not_find_list)):
231 match = not_find_list[index].search(stdout)
232 if match:
233 return failure("pattern found: %s (match: %s)" % (not_find_list_patterns[index], match.group(0)))
234
235 return success()
236
237 def test_s3cmd(label, cmd_args = [], **kwargs):
238 if not cmd_args[0].endswith("s3cmd"):
239 cmd_args.insert(0, "python")
240 cmd_args.insert(1, "s3cmd")
241 if config_file:
242 cmd_args.insert(2, "-c")
243 cmd_args.insert(3, config_file)
244
245 return test(label, cmd_args, **kwargs)
246
247 def test_mkdir(label, dir_name):
248 if os.name in ("posix", "nt"):
249 cmd = ['mkdir', '-p']
250 else:
251 print("Unknown platform: %s" % os.name)
252 sys.exit(1)
253 cmd.append(dir_name)
254 return test(label, cmd)
255
256 def test_rmdir(label, dir_name):
257 if os.path.isdir(dir_name):
258 if os.name == "posix":
259 cmd = ['rm', '-rf']
260 elif os.name == "nt":
261 cmd = ['rmdir', '/s/q']
262 else:
263 print("Unknown platform: %s" % os.name)
264 sys.exit(1)
265 cmd.append(dir_name)
266 return test(label, cmd)
267 else:
268 return test(label, [])
269
270 def test_flushdir(label, dir_name):
271 test_rmdir(label + "(rm)", dir_name)
272 return test_mkdir(label + "(mk)", dir_name)
273
274 def test_copy(label, src_file, dst_file):
275 if os.name == "posix":
276 cmd = ['cp', '-f']
277 elif os.name == "nt":
278 cmd = ['copy']
279 else:
280 print("Unknown platform: %s" % os.name)
281 sys.exit(1)
282 cmd.append(src_file)
283 cmd.append(dst_file)
284 return test(label, cmd)
285
286 def test_curl_HEAD(label, src_file, **kwargs):
287 cmd = ['curl', '--silent', '--head', '-include', '--location']
288 cmd.append(src_file)
289 return test(label, cmd, **kwargs)
290
291 bucket_prefix = u"%s-" % getpass.getuser().lower()
292
293 argv = sys.argv[1:]
294 while argv:
295 arg = argv.pop(0)
296 if arg.startswith('--bucket-prefix='):
297 print("Usage: '--bucket-prefix PREFIX', not '--bucket-prefix=PREFIX'")
298 sys.exit(0)
299 if arg in ("-h", "--help"):
300 print("%s A B K..O -N" % sys.argv[0])
301 print("Run tests number A, B and K through to O, except for N")
302 sys.exit(0)
303
304 if arg in ("-c", "--config"):
305 config_file = argv.pop(0)
306 continue
307 if arg in ("-l", "--list"):
308 exclude_tests = range(0, 999)
309 break
310 if arg in ("-v", "--verbose"):
311 verbose = True
312 continue
313 if arg in ("-p", "--bucket-prefix"):
314 try:
315 bucket_prefix = argv.pop(0)
316 except IndexError:
317 print("Bucket prefix option must explicitly supply a bucket name prefix")
318 sys.exit(0)
319 continue
320 if ".." in arg:
321 range_idx = arg.find("..")
322 range_start = arg[:range_idx] or 0
323 range_end = arg[range_idx+2:] or 999
324 run_tests.extend(range(int(range_start), int(range_end) + 1))
325 elif arg.startswith("-"):
326 exclude_tests.append(int(arg[1:]))
327 else:
328 run_tests.append(int(arg))
329
330 print("Using bucket prefix: '%s'" % bucket_prefix)
331
332 cfg = S3.Config.Config(config_file)
333
334 if not run_tests:
335 run_tests = range(0, 999)
336
337 # helper functions for generating bucket names
338 def bucket(tail):
339 '''Test bucket name'''
340 label = 'autotest'
341 if str(tail) == '3':
342 label = 'autotest'
343 return '%ss3cmd-%s-%s' % (bucket_prefix, label, tail)
344
345 def pbucket(tail):
346 '''Like bucket(), but prepends "s3://" for you'''
347 return 's3://' + bucket(tail)
348
349 ## ====== Remove test buckets
350 test_s3cmd("Remove test buckets", ['rb', '-r', '--force', pbucket(1), pbucket(2), pbucket(3)])
351
352 ## ====== verify they were removed
353 test_s3cmd("Verify no test buckets", ['ls'],
354 must_not_find = [pbucket(1), pbucket(2), pbucket(3)])
355
356
357 ## ====== Create one bucket (EU)
358 test_s3cmd("Create one bucket (EU)", ['mb', '--bucket-location=EU', pbucket(1)],
359 must_find = "Bucket '%s/' created" % pbucket(1))
360
361
362
363 ## ====== Create multiple buckets
364 test_s3cmd("Create multiple buckets", ['mb', pbucket(2), pbucket(3)],
365 must_find = [ "Bucket '%s/' created" % pbucket(2), "Bucket '%s/' created" % pbucket(3)])
366
367
368 ## ====== Invalid bucket name
369 test_s3cmd("Invalid bucket name", ["mb", "--bucket-location=EU", pbucket('EU')],
370 retcode = EX_USAGE,
371 must_find = "ERROR: Parameter problem: Bucket name '%s' contains disallowed character" % bucket('EU'),
372 must_not_find_re = "Bucket.*created")
373
374
375 ## ====== Buckets list
376 test_s3cmd("Buckets list", ["ls"],
377 must_find = [ "autotest-1", "autotest-2", "Autotest-3" ], must_not_find_re = "autotest-EU")
378
379
380 ## ====== Sync to S3
381 test_s3cmd("Sync to S3", ['sync', 'testsuite/', pbucket(1) + '/xyz/', '--exclude', 'demo/*', '--exclude', '*.png', '--no-encrypt', '--exclude-from', 'testsuite/exclude.encodings' ],
382 must_find = [ "ERROR: Upload of 'testsuite/permission-tests/permission-denied.txt' is not possible (Reason: Permission denied)",
383 "WARNING: 32 non-printable characters replaced in: crappy-file-name/non-printables",
384 ],
385 must_not_find_re = [ "demo/", "^(?!WARNING: Skipping).*\.png$", "permission-denied-dir" ],
386 retcode = EX_PARTIAL)
387
388 if have_encoding:
389 ## ====== Sync UTF-8 / GBK / ... to S3
390 test_s3cmd(u"Sync %s to S3" % encoding, ['sync', 'testsuite/encodings/' + encoding, '%s/xyz/encodings/' % pbucket(1), '--exclude', 'demo/*', '--no-encrypt' ],
391 must_find = [ u"'testsuite/encodings/%(encoding)s/%(pattern)s' -> '%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s'" % { 'encoding' : encoding, 'pattern' : enc_pattern , 'pbucket' : pbucket(1)} ])
392
393
394 ## ====== List bucket content
395 test_s3cmd("List bucket content", ['ls', '%s/xyz/' % pbucket(1) ],
396 must_find_re = [ u"DIR +%s/xyz/binary/$" % pbucket(1) , u"DIR +%s/xyz/etc/$" % pbucket(1) ],
397 must_not_find = [ u"random-crap.md5", u"/demo" ])
398
399
400 ## ====== List bucket recursive
401 must_find = [ u"%s/xyz/binary/random-crap.md5" % pbucket(1) ]
402 if have_encoding:
403 must_find.append(u"%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s" % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
404
405 test_s3cmd("List bucket recursive", ['ls', '--recursive', pbucket(1)],
406 must_find = must_find,
407 must_not_find = [ "logo.png" ])
408
409 ## ====== FIXME
410 test_s3cmd("Recursive put", ['put', '--recursive', 'testsuite/etc', '%s/xyz/' % pbucket(1) ])
411
412
413 ## ====== Clean up local destination dir
414 test_flushdir("Clean testsuite-out/", "testsuite-out")
415
416 ## ====== Put from stdin
417 f = open('testsuite/single-file/single-file.txt', 'r')
418 test_s3cmd("Put from stdin", ['put', '-', '%s/single-file/single-file.txt' % pbucket(1)],
419 must_find = ["'<stdin>' -> '%s/single-file/single-file.txt'" % pbucket(1)],
420 stdin = f)
421 f.close()
422
423 ## ====== Multipart put
424 os.system('mkdir -p testsuite-out')
425 os.system('dd if=/dev/urandom of=testsuite-out/urandom.bin bs=1M count=16 > /dev/null 2>&1')
426 test_s3cmd("Put multipart", ['put', '--multipart-chunk-size-mb=5', 'testsuite-out/urandom.bin', '%s/urandom.bin' % pbucket(1)],
427 must_not_find = ['abortmp'])
428
429 ## ====== Multipart put from stdin
430 f = open('testsuite-out/urandom.bin', 'r')
431 test_s3cmd("Multipart large put from stdin", ['put', '--multipart-chunk-size-mb=5', '-', '%s/urandom2.bin' % pbucket(1)],
432 must_find = ['%s/urandom2.bin' % pbucket(1)],
433 must_not_find = ['abortmp'],
434 stdin = f)
435 f.close()
436
437 ## ====== Clean up local destination dir
438 test_flushdir("Clean testsuite-out/", "testsuite-out")
439
440 ## ====== Moving things without trailing '/'
441 os.system('dd if=/dev/urandom of=testsuite-out/urandom1.bin bs=1k count=1 > /dev/null 2>&1')
442 os.system('dd if=/dev/urandom of=testsuite-out/urandom2.bin bs=1k count=1 > /dev/null 2>&1')
443 test_s3cmd("Put multiple files", ['put', 'testsuite-out/urandom1.bin', 'testsuite-out/urandom2.bin', '%s/' % pbucket(1)],
444 must_find = ["%s/urandom1.bin" % pbucket(1), "%s/urandom2.bin" % pbucket(1)])
445
446 test_s3cmd("Move without '/'", ['mv', '%s/urandom1.bin' % pbucket(1), '%s/urandom2.bin' % pbucket(1), '%s/dir' % pbucket(1)],
447 retcode = 64,
448 must_find = ['Destination must be a directory'])
449
450 test_s3cmd("Move recursive w/a '/'",
451 ['-r', 'mv', '%s/dir1' % pbucket(1), '%s/dir2' % pbucket(1)],
452 retcode = 64,
453 must_find = ['Destination must be a directory'])
454
455 ## ====== Moving multiple files into directory with trailing '/'
456 must_find = ["'%s/urandom1.bin' -> '%s/dir/urandom1.bin'" % (pbucket(1),pbucket(1)), "'%s/urandom2.bin' -> '%s/dir/urandom2.bin'" % (pbucket(1),pbucket(1))]
457 must_not_find = ["'%s/urandom1.bin' -> '%s/dir'" % (pbucket(1),pbucket(1)), "'%s/urandom2.bin' -> '%s/dir'" % (pbucket(1),pbucket(1))]
458 test_s3cmd("Move multiple files",
459 ['mv', '%s/urandom1.bin' % pbucket(1), '%s/urandom2.bin' % pbucket(1), '%s/dir/' % pbucket(1)],
460 must_find = must_find,
461 must_not_find = must_not_find)
462
463 ## ====== Clean up local destination dir
464 test_flushdir("Clean testsuite-out/", "testsuite-out")
465
466 ## ====== Sync from S3
467 must_find = [ "'%s/xyz/binary/random-crap.md5' -> 'testsuite-out/xyz/binary/random-crap.md5'" % pbucket(1) ]
468 if have_encoding:
469 must_find.append(u"'%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s' -> 'testsuite-out/xyz/encodings/%(encoding)s/%(pattern)s' " % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
470 test_s3cmd("Sync from S3", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
471 must_find = must_find)
472
473 ## ====== Remove 'demo' directory
474 test_rmdir("Remove 'dir-test/'", "testsuite-out/xyz/dir-test/")
475
476
477 ## ====== Create dir with name of a file
478 test_mkdir("Create file-dir dir", "testsuite-out/xyz/dir-test/file-dir")
479
480
481 ## ====== Skip dst dirs
482 test_s3cmd("Skip over dir", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
483 must_find = "ERROR: Download of 'xyz/dir-test/file-dir' failed (Reason: testsuite-out/xyz/dir-test/file-dir is a directory)",
484 retcode = EX_PARTIAL)
485
486
487 ## ====== Clean up local destination dir
488 test_flushdir("Clean testsuite-out/", "testsuite-out")
489
490
491 ## ====== Put public, guess MIME
492 test_s3cmd("Put public, guess MIME", ['put', '--guess-mime-type', '--acl-public', 'testsuite/etc/logo.png', '%s/xyz/etc/logo.png' % pbucket(1)],
493 must_find = [ "-> '%s/xyz/etc/logo.png'" % pbucket(1) ])
494
495
496 ## ====== Retrieve from URL
497 if have_curl:
498 test_curl_HEAD("Retrieve from URL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
499 must_find_re = ['Content-Length: 22059'])
500
501 ## ====== Change ACL to Private
502 test_s3cmd("Change ACL to Private", ['setacl', '--acl-private', '%s/xyz/etc/l*.png' % pbucket(1)],
503 must_find = [ "logo.png: ACL set to Private" ])
504
505
506 ## ====== Verify Private ACL
507 if have_curl:
508 test_curl_HEAD("Verify Private ACL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
509 must_find_re = [ '403 Forbidden' ])
510
511
512 ## ====== Change ACL to Public
513 test_s3cmd("Change ACL to Public", ['setacl', '--acl-public', '--recursive', '%s/xyz/etc/' % pbucket(1) , '-v'],
514 must_find = [ "logo.png: ACL set to Public" ])
515
516
517 ## ====== Verify Public ACL
518 if have_curl:
519 test_curl_HEAD("Verify Public ACL", 'http://%s.%s/xyz/etc/logo.png' % (bucket(1), cfg.host_base),
520 must_find_re = [ '200 OK',
521 'Content-Length: 22059'])
522
523
524 ## ====== Sync more to S3
525 test_s3cmd("Sync more to S3", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt' ],
526 must_find = [ "'testsuite/demo/some-file.xml' -> '%s/xyz/demo/some-file.xml' " % pbucket(1) ],
527 must_not_find = [ "'testsuite/etc/linked.png' -> '%s/xyz/etc/linked.png'" % pbucket(1) ],
528 retcode = EX_PARTIAL)
529
530
531 ## ====== Don't check MD5 sum on Sync
532 test_copy("Change file cksum1.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum1.txt")
533 test_copy("Change file cksum33.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum33.txt")
534 test_s3cmd("Don't check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--no-check-md5'],
535 must_find = [ "cksum33.txt" ],
536 must_not_find = [ "cksum1.txt" ],
537 retcode = EX_PARTIAL)
538
539
540 ## ====== Check MD5 sum on Sync
541 test_s3cmd("Check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--check-md5'],
542 must_find = [ "cksum1.txt" ],
543 retcode = EX_PARTIAL)
544
545
546 ## ====== Rename within S3
547 test_s3cmd("Rename within S3", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
548 must_find = [ "move: '%s/xyz/etc/logo.png' -> '%s/xyz/etc2/Logo.PNG'" % (pbucket(1), pbucket(1))])
549
550
551 ## ====== Rename (NoSuchKey)
552 test_s3cmd("Rename (NoSuchKey)", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
553 retcode = EX_NOTFOUND,
554 must_find_re = [ 'Key not found' ],
555 must_not_find = [ "move: '%s/xyz/etc/logo.png' -> '%s/xyz/etc2/Logo.PNG'" % (pbucket(1), pbucket(1)) ])
556
557 ## ====== Sync more from S3 (invalid src)
558 test_s3cmd("Sync more from S3 (invalid src)", ['sync', '--delete-removed', '%s/xyz/DOESNOTEXIST' % pbucket(1), 'testsuite-out'],
559 must_not_find = [ "delete: 'testsuite-out/logo.png'" ])
560
561 ## ====== Sync more from S3
562 test_s3cmd("Sync more from S3", ['sync', '--delete-removed', '%s/xyz' % pbucket(1), 'testsuite-out'],
563 must_find = [ "'%s/xyz/etc2/Logo.PNG' -> 'testsuite-out/xyz/etc2/Logo.PNG'" % pbucket(1),
564 "'%s/xyz/demo/some-file.xml' -> 'testsuite-out/xyz/demo/some-file.xml'" % pbucket(1) ],
565 must_not_find_re = [ "not-deleted.*etc/logo.png", "delete: 'testsuite-out/logo.png'" ])
566
567
568 ## ====== Make dst dir for get
569 test_rmdir("Remove dst dir for get", "testsuite-out")
570
571
572 ## ====== Get multiple files
573 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
574 retcode = EX_USAGE,
575 must_find = [ 'Destination must be a directory or stdout when downloading multiple sources.' ])
576
577 ## ====== put/get non-ASCII filenames
578 test_s3cmd("Put unicode filenames", ['put', u'testsuite/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo', u'%s/xyz/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo' % pbucket(1)],
579 retcode = 0,
580 must_find = [ '->' ])
581
582
583 ## ====== Make dst dir for get
584 test_mkdir("Make dst dir for get", "testsuite-out")
585
586
587 ## ====== put/get non-ASCII filenames
588 test_s3cmd("Get unicode filenames", ['get', u'%s/xyz/encodings/UTF-8/ŪņЇЌœđЗ/Žůžo' % pbucket(1), 'testsuite-out'],
589 retcode = 0,
590 must_find = [ '->' ])
591
592
593 ## ====== Get multiple files
594 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
595 must_find = [ u"-> 'testsuite-out/Logo.PNG'",
596 u"-> 'testsuite-out/AtomicClockRadio.ttf'" ])
597
598 ## ====== Upload files differing in capitalisation
599 test_s3cmd("blah.txt / Blah.txt", ['put', '-r', 'testsuite/blahBlah', pbucket(1)],
600 must_find = [ '%s/blahBlah/Blah.txt' % pbucket(1), '%s/blahBlah/blah.txt' % pbucket(1)])
601
602 ## ====== Copy between buckets
603 test_s3cmd("Copy between buckets", ['cp', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc2/logo.png' % pbucket(3)],
604 must_find = [ "remote copy: '%s/xyz/etc2/Logo.PNG' -> '%s/xyz/etc2/logo.png'" % (pbucket(1), pbucket(3)) ])
605
606 ## ====== Recursive copy
607 test_s3cmd("Recursive copy, set ACL", ['cp', '-r', '--acl-public', '%s/xyz/' % pbucket(1), '%s/copy/' % pbucket(2), '--exclude', 'demo/dir?/*.txt', '--exclude', 'non-printables*'],
608 must_find = [ "remote copy: '%s/xyz/etc2/Logo.PNG' -> '%s/copy/etc2/Logo.PNG'" % (pbucket(1), pbucket(2)),
609 "remote copy: '%s/xyz/blahBlah/Blah.txt' -> '%s/copy/blahBlah/Blah.txt'" % (pbucket(1), pbucket(2)),
610 "remote copy: '%s/xyz/blahBlah/blah.txt' -> '%s/copy/blahBlah/blah.txt'" % (pbucket(1), pbucket(2)) ],
611 must_not_find = [ "demo/dir1/file1-1.txt" ])
612
613 ## ====== Verify ACL and MIME type
614 test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
615 must_find_re = [ "MIME type:.*image/png",
616 "ACL:.*\*anon\*: READ",
617 "URL:.*http://%s.%s/copy/etc2/Logo.PNG" % (bucket(2), cfg.host_base) ])
618
619 ## ====== modify MIME type
620 test_s3cmd("Modify MIME type", ['modify', '--mime-type=binary/octet-stream', '%s/copy/etc2/Logo.PNG' % pbucket(2) ])
621
622 test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
623 must_find_re = [ "MIME type:.*binary/octet-stream",
624 "ACL:.*\*anon\*: READ",
625 "URL:.*http://%s.%s/copy/etc2/Logo.PNG" % (bucket(2), cfg.host_base) ])
626
627 test_s3cmd("Modify MIME type back", ['modify', '--mime-type=image/png', '%s/copy/etc2/Logo.PNG' % pbucket(2) ])
628
629 test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
630 must_find_re = [ "MIME type:.*image/png",
631 "ACL:.*\*anon\*: READ",
632 "URL:.*http://%s.%s/copy/etc2/Logo.PNG" % (bucket(2), cfg.host_base) ])
633
634 test_s3cmd("Add cache-control header", ['modify', '--add-header=cache-control: max-age=3600, public', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
635 must_find_re = [ "modify: .*" ])
636
637 if have_curl:
638 test_curl_HEAD("HEAD check Cache-Control present", 'http://%s.%s/copy/etc2/Logo.PNG' % (bucket(2), cfg.host_base),
639 must_find_re = [ "Cache-Control: max-age=3600" ])
640
641 test_s3cmd("Remove cache-control header", ['modify', '--remove-header=cache-control', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
642 must_find_re = [ "modify: .*" ])
643
644 if have_curl:
645 test_curl_HEAD("HEAD check Cache-Control not present", 'http://%s.%s/copy/etc2/Logo.PNG' % (bucket(2), cfg.host_base),
646 must_not_find_re = [ "Cache-Control: max-age=3600" ])
647
648 ## ====== sign
649 test_s3cmd("sign string", ['sign', 's3cmd'], must_find_re = ["Signature:"])
650 test_s3cmd("signurl time", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), str(int(time.time()) + 60)], must_find_re = ["http://"])
651 test_s3cmd("signurl time offset", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), '+60'], must_find_re = ["https?://"])
652 test_s3cmd("signurl content disposition and type", ['signurl', '%s/copy/etc2/Logo.PNG' % pbucket(2), '+60', '--content-disposition=inline; filename=video.mp4', '--content-type=video/mp4'], must_find_re = [ 'response-content-disposition', 'response-content-type' ] )
653
654 ## ====== Rename within S3
655 test_s3cmd("Rename within S3", ['mv', '%s/copy/etc2/Logo.PNG' % pbucket(2), '%s/copy/etc/logo.png' % pbucket(2)],
656 must_find = [ "move: '%s/copy/etc2/Logo.PNG' -> '%s/copy/etc/logo.png'" % (pbucket(2), pbucket(2))])
657
658 ## ====== Sync between buckets
659 test_s3cmd("Sync remote2remote", ['sync', '%s/xyz/' % pbucket(1), '%s/copy/' % pbucket(2), '--delete-removed', '--exclude', 'non-printables*'],
660 must_find = [ "remote copy: '%s/xyz/demo/dir1/file1-1.txt' -> '%s/copy/demo/dir1/file1-1.txt'" % (pbucket(1), pbucket(2)),
661 "remote copy: 'etc/logo.png' -> 'etc2/Logo.PNG'",
662 "delete: '%s/copy/etc/logo.png'" % pbucket(2) ],
663 must_not_find = [ "blah.txt" ])
664
665 ## ====== Don't Put symbolic link
666 test_s3cmd("Don't put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1),],
667 retcode = EX_USAGE,
668 must_find = ["WARNING: Skipping over symbolic link: testsuite/etc/linked1.png"],
669 must_not_find_re = ["^(?!WARNING: Skipping).*linked1.png"])
670
671 ## ====== Put symbolic link
672 test_s3cmd("Put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1),'--follow-symlinks' ],
673 must_find = [ "'testsuite/etc/linked1.png' -> '%s/xyz/linked1.png'" % pbucket(1)])
674
675 ## ====== Sync symbolic links
676 test_s3cmd("Sync symbolic links", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--follow-symlinks' ],
677 must_find = ["remote copy: 'etc2/Logo.PNG' -> 'etc/linked.png'"],
678 # Don't want to recursively copy linked directories!
679 must_not_find_re = ["etc/more/linked-dir/more/give-me-more.txt",
680 "etc/brokenlink.png"],
681 retcode = EX_PARTIAL)
682
683 ## ====== Multi source move
684 test_s3cmd("Multi-source move", ['mv', '-r', '%s/copy/blahBlah/Blah.txt' % pbucket(2), '%s/copy/etc/' % pbucket(2), '%s/moved/' % pbucket(2)],
685 must_find = [ "move: '%s/copy/blahBlah/Blah.txt' -> '%s/moved/Blah.txt'" % (pbucket(2), pbucket(2)),
686 "move: '%s/copy/etc/AtomicClockRadio.ttf' -> '%s/moved/AtomicClockRadio.ttf'" % (pbucket(2), pbucket(2)),
687 "move: '%s/copy/etc/TypeRa.ttf' -> '%s/moved/TypeRa.ttf'" % (pbucket(2), pbucket(2)) ],
688 must_not_find = [ "blah.txt" ])
689
690 ## ====== Verify move
691 test_s3cmd("Verify move", ['ls', '-r', pbucket(2)],
692 must_find = [ "%s/moved/Blah.txt" % pbucket(2),
693 "%s/moved/AtomicClockRadio.ttf" % pbucket(2),
694 "%s/moved/TypeRa.ttf" % pbucket(2),
695 "%s/copy/blahBlah/blah.txt" % pbucket(2) ],
696 must_not_find = [ "%s/copy/blahBlah/Blah.txt" % pbucket(2),
697 "%s/copy/etc/AtomicClockRadio.ttf" % pbucket(2),
698 "%s/copy/etc/TypeRa.ttf" % pbucket(2) ])
699
700 ## ====== List all
701 test_s3cmd("List all", ['la'],
702 must_find = [ "%s/urandom.bin" % pbucket(1)])
703
704 ## ====== Simple delete
705 test_s3cmd("Simple delete", ['del', '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
706 must_find = [ "delete: '%s/xyz/etc2/Logo.PNG'" % pbucket(1) ])
707
708 ## ====== Simple delete with rm
709 test_s3cmd("Simple delete with rm", ['rm', '%s/xyz/test_rm/TypeRa.ttf' % pbucket(1)],
710 must_find = [ "delete: '%s/xyz/test_rm/TypeRa.ttf'" % pbucket(1) ])
711
712 ## ====== Create expiration rule with days and prefix
713 test_s3cmd("Create expiration rule with days and prefix", ['expire', pbucket(1), '--expiry-days=365', '--expiry-prefix=log/'],
714 must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
715
716 ## ====== Create expiration rule with date and prefix
717 test_s3cmd("Create expiration rule with date and prefix", ['expire', pbucket(1), '--expiry-date=2020-12-31T00:00:00.000Z', '--expiry-prefix=log/'],
718 must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
719
720 ## ====== Create expiration rule with days only
721 test_s3cmd("Create expiration rule with days only", ['expire', pbucket(1), '--expiry-days=365'],
722 must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
723
724 ## ====== Create expiration rule with date only
725 test_s3cmd("Create expiration rule with date only", ['expire', pbucket(1), '--expiry-date=2020-12-31T00:00:00.000Z'],
726 must_find = [ "Bucket '%s/': expiration configuration is set." % pbucket(1)])
727
728 ## ====== Get current expiration setting
729 test_s3cmd("Get current expiration setting", ['info', pbucket(1)],
730 must_find = [ "Expiration Rule: all objects in this bucket will expire in '2020-12-31T00:00:00.000Z'"])
731
732 ## ====== Delete expiration rule
733 test_s3cmd("Delete expiration rule", ['expire', pbucket(1)],
734 must_find = [ "Bucket '%s/': expiration configuration is deleted." % pbucket(1)])
735
736 ## ====== set Requester Pays flag
737 test_s3cmd("Set requester pays", ['payer', '--requester-pays', pbucket(2)])
738
739 ## ====== get Requester Pays flag
740 test_s3cmd("Get requester pays flag", ['info', pbucket(2)],
741 must_find = [ "Payer: Requester"])
742
743 ## ====== ls using Requester Pays flag
744 test_s3cmd("ls using requester pays flag", ['ls', '--requester-pays', pbucket(2)])
745
746 ## ====== clear Requester Pays flag
747 test_s3cmd("Clear requester pays", ['payer', pbucket(2)])
748
749 ## ====== get Requester Pays flag
750 test_s3cmd("Get requester pays flag", ['info', pbucket(2)],
751 must_find = [ "Payer: BucketOwner"])
752
753 ## ====== Recursive delete maximum exceeed
754 test_s3cmd("Recursive delete maximum exceeded", ['del', '--recursive', '--max-delete=1', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
755 must_not_find = [ "delete: '%s/xyz/etc/TypeRa.ttf'" % pbucket(1) ])
756
757 ## ====== Recursive delete
758 test_s3cmd("Recursive delete", ['del', '--recursive', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
759 must_find = [ "delete: '%s/xyz/etc/TypeRa.ttf'" % pbucket(1) ],
760 must_find_re = [ "delete: '.*/etc/logo.png'" ],
761 must_not_find = [ "AtomicClockRadio.ttf" ])
762
763 ## ====== Recursive delete with rm
764 test_s3cmd("Recursive delete with rm", ['rm', '--recursive', '--exclude', 'Atomic*', '%s/xyz/test_rm' % pbucket(1)],
765 must_find = [ "delete: '%s/xyz/test_rm/more/give-me-more.txt'" % pbucket(1) ],
766 must_find_re = [ "delete: '.*/test_rm/logo.png'" ],
767 must_not_find = [ "AtomicClockRadio.ttf" ])
768
769 ## ====== Recursive delete all
770 test_s3cmd("Recursive delete all", ['del', '--recursive', '--force', pbucket(1)],
771 must_find_re = [ "delete: '.*binary/random-crap'" ])
772
773 ## ====== Remove empty bucket
774 test_s3cmd("Remove empty bucket", ['rb', pbucket(1)],
775 must_find = [ "Bucket '%s/' removed" % pbucket(1) ])
776
777 ## ====== Remove remaining buckets
778 test_s3cmd("Remove remaining buckets", ['rb', '--recursive', pbucket(2), pbucket(3)],
779 must_find = [ "Bucket '%s/' removed" % pbucket(2),
780 "Bucket '%s/' removed" % pbucket(3) ])
781
782 # vim:et:ts=4:sts=4:ai
+27
-15
s3cmd less more
101101 buckets_size += size
102102 total_size, size_coeff = formatSize(buckets_size, cfg.human_readable_sizes)
103103 total_size_str = str(total_size) + size_coeff
104 output(u"".rjust(8, "-"))
105 output(u"%s Total" % (total_size_str.ljust(8)))
104 output(u"".rjust(12, "-"))
105 output(u"%s Total" % (total_size_str.ljust(12)))
106106 return size
107107
108108 def subcmd_bucket_usage(s3, uri):
130130 except KeyboardInterrupt as e:
131131 extra_info = u' [interrupted]'
132132
133 total_size, size_coeff = formatSize(bucket_size, Config().human_readable_sizes)
134 total_size_str = str(total_size) + size_coeff
135 output(u"%s %s objects %s%s" % (total_size_str.ljust(8), object_count, uri, extra_info))
133 total_size_str = u"%d%s" % formatSize(bucket_size,
134 Config().human_readable_sizes)
135 if Config().human_readable_sizes:
136 total_size_str = total_size_str.rjust(5)
137 else:
138 total_size_str = total_size_str.rjust(12)
139 output(u"%s %7s objects %s%s" % (total_size_str, object_count, uri,
140 extra_info))
136141 return bucket_size
137142
138143 def cmd_ls(args):
183188 error(S3.codes[e.info["Code"]] % bucket)
184189 raise
185190
191 # md5 are 32 char long, but for multipart there could be a suffix
192 if Config().human_readable_sizes:
193 # %(size)5s%(coeff)1s
194 format_size = u"%5d%1s"
195 dir_str = u"DIR".rjust(6)
196 else:
197 format_size = u"%12d%s"
198 dir_str = u"DIR".rjust(12)
186199 if cfg.long_listing:
187 format_string = u"%(timestamp)16s %(size)9s%(coeff)1s %(md5)32s %(storageclass)s %(uri)s"
200 format_string = u"%(timestamp)16s %(size)s %(md5)-35s %(storageclass)-11s %(uri)s"
188201 elif cfg.list_md5:
189 format_string = u"%(timestamp)16s %(size)9s%(coeff)1s %(md5)32s %(uri)s"
202 format_string = u"%(timestamp)16s %(size)s %(md5)-35s %(uri)s"
190203 else:
191 format_string = u"%(timestamp)16s %(size)9s%(coeff)1s %(uri)s"
204 format_string = u"%(timestamp)16s %(size)s %(uri)s"
192205
193206 for prefix in response['common_prefixes']:
194207 output(format_string % {
195208 "timestamp": "",
196 "size": "DIR",
197 "coeff": "",
209 "size": dir_str,
198210 "md5": "",
199211 "storageclass": "",
200212 "uri": uri.compose_uri(bucket, prefix["Prefix"])})
212224 except KeyError:
213225 pass
214226
215 size, size_coeff = formatSize(object["Size"], Config().human_readable_sizes)
227 size_and_coeff = formatSize(object["Size"],
228 Config().human_readable_sizes)
216229 output(format_string % {
217230 "timestamp": formatDateTime(object["LastModified"]),
218 "size" : str(size),
219 "coeff": size_coeff,
231 "size" : format_size % size_and_coeff,
220232 "md5" : md5,
221233 "storageclass" : storageclass,
222234 "uri": uri.compose_uri(bucket, object["Key"]),
26492661 optparser.add_option( "--no-encrypt", dest="encrypt", action="store_false", help="Don't encrypt files.")
26502662 optparser.add_option("-f", "--force", dest="force", action="store_true", help="Force overwrite and other dangerous operations.")
26512663 optparser.add_option( "--continue", dest="get_continue", action="store_true", help="Continue getting a partially downloaded file (only for [get] command).")
2652 optparser.add_option( "--continue-put", dest="put_continue", action="store_true", help="Continue uploading partially uploaded files or multipart upload parts. Restarts/parts files that don't have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk.")
2664 optparser.add_option( "--continue-put", dest="put_continue", action="store_true", help="Continue uploading partially uploaded files or multipart upload parts. Restarts parts/files that don't have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk.")
26532665 optparser.add_option( "--upload-id", dest="upload_id", help="UploadId for Multipart Upload, in case you want continue an existing upload (equivalent to --continue-put) and there are multiple partial uploads. Use s3cmd multipart [URI] to see what UploadIds are associated with the given URI.")
26542666 optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).")
26552667 optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.")
26602672 optparser.add_option( "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")
26612673 optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")
26622674
2663 optparser.add_option("-D", "--restore-days", dest="restore_days", action="store", help="Number of days to keep restored file available (only for 'restore' command).", metavar="NUM")
2675 optparser.add_option("-D", "--restore-days", dest="restore_days", action="store", help="Number of days to keep restored file available (only for 'restore' command). Default is 1 day.", metavar="NUM")
26642676 optparser.add_option( "--restore-priority", dest="restore_priority", action="store", choices=['standard', 'expedited', 'bulk'], help="Priority for restoring files from S3 Glacier (only for 'restore' command). Choices available: bulk, standard, expedited")
26652677
26662678 optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete destination objects with no corresponding source file [sync]")
+0
-45
s3cmd.egg-info/PKG-INFO less more
0 Metadata-Version: 1.1
1 Name: s3cmd
2 Version: 2.0.2
3 Summary: Command line tool for managing Amazon S3 and CloudFront services
4 Home-page: http://s3tools.org
5 Author: github.com/mdomsch, github.com/matteobar, github.com/fviard
6 Author-email: s3tools-bugs@lists.sourceforge.net
7 License: GNU GPL v2+
8 Description-Content-Type: UNKNOWN
9 Description:
10
11 S3cmd lets you copy files from/to Amazon S3
12 (Simple Storage Service) using a simple to use
13 command line client. Supports rsync-like backup,
14 GPG encryption, and more. Also supports management
15 of Amazon's CloudFront content delivery network.
16
17
18 Authors:
19 --------
20 Michal Ludvig <michal@logix.cz>
21
22 Platform: UNKNOWN
23 Classifier: Development Status :: 5 - Production/Stable
24 Classifier: Environment :: Console
25 Classifier: Environment :: MacOS X
26 Classifier: Environment :: Win32 (MS Windows)
27 Classifier: Intended Audience :: End Users/Desktop
28 Classifier: Intended Audience :: System Administrators
29 Classifier: License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)
30 Classifier: Natural Language :: English
31 Classifier: Operating System :: MacOS :: MacOS X
32 Classifier: Operating System :: Microsoft :: Windows
33 Classifier: Operating System :: POSIX
34 Classifier: Operating System :: Unix
35 Classifier: Programming Language :: Python :: 2
36 Classifier: Programming Language :: Python :: 2.6
37 Classifier: Programming Language :: Python :: 2.7
38 Classifier: Programming Language :: Python :: 3
39 Classifier: Programming Language :: Python :: 3.3
40 Classifier: Programming Language :: Python :: 3.4
41 Classifier: Programming Language :: Python :: 3.5
42 Classifier: Programming Language :: Python :: 3.6
43 Classifier: Topic :: System :: Archiving
44 Classifier: Topic :: Utilities
+0
-36
s3cmd.egg-info/SOURCES.txt less more
0 INSTALL
1 LICENSE
2 MANIFEST.in
3 NEWS
4 README.md
5 s3cmd
6 s3cmd.1
7 setup.cfg
8 setup.py
9 S3/ACL.py
10 S3/AccessLog.py
11 S3/BidirMap.py
12 S3/CloudFront.py
13 S3/Config.py
14 S3/ConnMan.py
15 S3/Crypto.py
16 S3/Custom_httplib27.py
17 S3/Custom_httplib3x.py
18 S3/Exceptions.py
19 S3/ExitCodes.py
20 S3/FileDict.py
21 S3/FileLists.py
22 S3/HashCache.py
23 S3/MultiPart.py
24 S3/PkgInfo.py
25 S3/Progress.py
26 S3/S3.py
27 S3/S3Uri.py
28 S3/SortedDict.py
29 S3/Utils.py
30 S3/__init__.py
31 s3cmd.egg-info/PKG-INFO
32 s3cmd.egg-info/SOURCES.txt
33 s3cmd.egg-info/dependency_links.txt
34 s3cmd.egg-info/requires.txt
35 s3cmd.egg-info/top_level.txt⏎
+0
-1
s3cmd.egg-info/dependency_links.txt less more
0
+0
-2
s3cmd.egg-info/requires.txt less more
0 python-dateutil
1 python-magic
+0
-1
s3cmd.egg-info/top_level.txt less more
0 S3
+174
-0
s3cmd.spec.in less more
0 %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
1
2 %global commit ##COMMIT##
3 %global shortcommit ##SHORTCOMMIT##
4
5 Name: s3cmd
6 Version: ##VERSION##
7 Release: 1%{dist}
8 Summary: Tool for accessing Amazon Simple Storage Service
9
10 Group: Applications/Internet
11 License: GPLv2
12 URL: http://s3tools.com
13 # git clone https://github.com/s3tools/s3cmd
14 # python setup.py sdist
15 Source0: https://github.com/s3tools/s3cmd/archive/%{commit}/%{name}-%{version}-%{shortcommit}.tar.gz
16 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
17 BuildArch: noarch
18
19 %if %{!?fedora:16}%{?fedora} < 16 || %{!?rhel:7}%{?rhel} < 7
20 BuildRequires: python-devel
21 %else
22 BuildRequires: python2-devel
23 %endif
24 %if %{!?fedora:8}%{?fedora} < 8 || %{!?rhel:6}%{?rhel} < 6
25 # This is in standard library since 2.5
26 BuildRequires: python-elementtree
27 Requires: python-elementtree
28 %endif
29 BuildRequires: python-dateutil
30 BuildRequires: python-setuptools
31 Requires: python-dateutil
32 Requires: python-magic
33
34 %description
35 S3cmd lets you copy files from/to Amazon S3
36 (Simple Storage Service) using a simple to use
37 command line client.
38
39
40 %prep
41 %setup -q -n s3cmd-%{commit}
42
43 %build
44
45
46 %install
47 rm -rf $RPM_BUILD_ROOT
48 S3CMD_PACKAGING=Yes python setup.py install --prefix=%{_prefix} --root=$RPM_BUILD_ROOT
49 install -d $RPM_BUILD_ROOT%{_mandir}/man1
50 install -m 644 s3cmd.1 $RPM_BUILD_ROOT%{_mandir}/man1
51
52
53 %clean
54 rm -rf $RPM_BUILD_ROOT
55
56
57 %files
58 %defattr(-,root,root,-)
59 %{_bindir}/s3cmd
60 %{_mandir}/man1/s3cmd.1*
61 %{python_sitelib}/S3
62 %if 0%{?fedora} >= 9 || 0%{?rhel} >= 6
63 %{python_sitelib}/s3cmd*.egg-info
64 %endif
65 %doc NEWS README.md LICENSE
66
67
68 %changelog
69 * Thu Feb 5 2015 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.1.2-5
70 - add Requires: python-magic
71
72 * Wed Feb 4 2015 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.1.2-4
73 - upstream 1.5.1.2, mostly bug fixes
74 - add dependency on python-setuptools
75
76 Mon Jan 12 2015 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-1
77 - upstream 1.5.0 final
78
79 * Tue Jul 1 2014 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.6.rc1
80 - upstream 1.5.0-rc1
81
82 * Sun Mar 23 2014 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.4.git
83 - upstream 1.5.0-beta1 plus even newer upstream fixes
84
85 * Sun Feb 02 2014 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.3.git
86 - upstream 1.5.0-beta1 plus newer upstream fixes
87
88 * Wed May 29 2013 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.2.gita122d97
89 - more upstream bugfixes
90 - drop pyxattr dep, that codepath got dropped in this release
91
92 * Mon May 20 2013 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.1.gitb1ae0fbe
93 - upstream 1.5.0-alpha3 plus fixes
94 - add dep on pyxattr for the --xattr option
95
96 * Tue Jun 19 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.4.git11e5755e
97 - add local MD5 cache
98
99 * Mon Jun 18 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.3.git7de0789d
100 - parallelize local->remote syncs
101
102 * Mon Jun 18 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.2.gitf881b162
103 - add hardlink / duplicate file detection support
104
105 * Fri Mar 9 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.1.git2dfe4a65
106 - build from git for mdomsch patches to s3cmd sync
107
108 * Thu Feb 23 2012 Dennis Gilmore <dennis@ausil.us> - 1.0.1-1
109 - update to 1.0.1 release
110
111 * Sat Jan 14 2012 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 1.0.0-4
112 - Rebuilt for https://fedoraproject.org/wiki/Fedora_17_Mass_Rebuild
113
114 * Thu May 05 2011 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 1.0.0-3
115 - No hashlib hackery
116
117 * Wed Feb 09 2011 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 1.0.0-2
118 - Rebuilt for https://fedoraproject.org/wiki/Fedora_15_Mass_Rebuild
119
120 * Tue Jan 11 2011 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 1.0.0-1
121 - New upstream release
122
123 * Mon Nov 29 2010 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 0.9.9.91-3
124 - Patch for broken f14 httplib
125
126 * Thu Jul 22 2010 David Malcolm <dmalcolm@redhat.com> - 0.9.9.91-2.1
127 - Rebuilt for https://fedoraproject.org/wiki/Features/Python_2.7/MassRebuild
128
129 * Wed Apr 28 2010 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 0.9.9.91-1.1
130 - Do not use sha1 from hashlib
131
132 * Sun Feb 21 2010 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.9.91-1
133 - New upstream release
134
135 * Sun Jul 26 2009 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 0.9.9-2
136 - Rebuilt for https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild
137
138 * Tue Feb 24 2009 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.9-1
139 - New upstream release
140
141 * Sat Nov 29 2008 Ignacio Vazquez-Abrams <ivazqueznet+rpm@gmail.com> - 0.9.8.4-2
142 - Rebuild for Python 2.6
143
144 * Tue Nov 11 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.4-1
145 - New upstream release, URI encoding patch upstreamed
146
147 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-4
148 - Try 3/65536
149
150 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-3
151 - Whoops, forgot to actually apply the patch.
152
153 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-2
154 - Fix listing of directories with special characters in names
155
156 * Thu Jul 31 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-1
157 - New upstream release: Avoid running out-of-memory in MD5'ing large files.
158
159 * Fri Jul 25 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.2-1.1
160 - Fix a typo
161
162 * Tue Jul 15 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.2-1
163 - New upstream
164
165 * Fri Jul 04 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-3
166 - Be satisfied with ET provided by 2.5 python
167
168 * Fri Jul 04 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-2
169 - Added missing python-devel BR, thanks to Marek Mahut
170 - Packaged the Python egg file
171
172 * Wed Jul 02 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-1
173 - Initial packaging attempt
+0
-5
setup.cfg less more
00 [sdist]
11 formats = gztar,zip
2
3 [egg_info]
4 tag_build =
5 tag_date = 0
6
testsuite.tar.gz less more
Binary diff not shown
+7
-0
upload-to-sf.sh less more
0 #!/bin/sh
1
2 VERSION=$(./s3cmd --version | awk '{print $NF}')
3 echo -e "Uploading \033[32ms3cmd \033[31m${VERSION}\033[0m ..."
4 #rsync -avP dist/s3cmd-${VERSION}.* ludvigm@frs.sourceforge.net:uploads/
5 ln -f NEWS README.txt
6 rsync -avP dist/s3cmd-${VERSION}.* README.txt ludvigm,s3tools@frs.sourceforge.net:/home/frs/project/s/s3/s3tools/s3cmd/${VERSION}/