Codebase list s3cmd / 89c35b7
Imported Upstream version 1.5.0~20140213 Gianfranco Costamagna 8 years ago
44 changed file(s) with 5176 addition(s) and 836 deletion(s). Raw diff Collapse all Expand all
+7
-0
.gitignore less more
0 *.pyc
1 *.swp
2 testsuite
3 /MANIFEST
4 /dist
5 build/*
6 s3cmd.spec
+8
-0
.svnignore less more
0 ## Run 'svn propset svn:ignore -F .svnignore .' after you change this list
1 *.pyc
2 tst.*
3 MANIFEST
4 dist
5 build
6 .*.swp
7 s3cmd.1.gz
+1462
-0
ChangeLog less more
0 2011-06-06 Michal Ludvig <mludvig@logix.net.nz>
1
2 ===== Migrated to GIT =====
3
4 No longer keeping ChangeLog up to date, use git log instead!
5
6 Two "official" repositories (both the same content):
7
8 * git://github.com/s3tools/s3cmd.git (primary)
9 * git://s3tools.git.sourceforge.net/gitroot/s3tools/s3cmd.git
10
11 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
12
13 * S3/S3Uri.py: Fixed cf:// uri parsing.
14 * S3/CloudFront.py: Don't fail if there are no cfinval
15 requests.
16
17 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
18
19 * S3/PkgInfo.py: Updated to 1.1.0-beta1
20 * NEWS: Updated.
21 * s3cmd.1: Regenerated.
22
23 2011-04-11 Michal Ludvig <mludvig@logix.net.nz>
24
25 * S3/Config.py: Increase socket_timeout from 10 secs to 5 mins.
26
27 2011-04-10 Michal Ludvig <mludvig@logix.net.nz>
28
29 * s3cmd, S3/CloudFront.py, S3/S3Uri.py: Support for checking
30 status of CF Invalidation Requests [cfinvalinfo].
31 * s3cmd, S3/CloudFront.py, S3/Config.py: Support for CloudFront
32 invalidation using [sync --cf-invalidate] command.
33 * S3/Utils.py: getDictFromTree() now recurses into
34 sub-trees.
35
36 2011-03-30 Michal Ludvig <mludvig@logix.net.nz>
37
38 * S3/CloudFront.py: Fix warning with Python 2.7
39 * S3/CloudFront.py: Cmd._get_dist_name_for_bucket() moved to
40 CloudFront class.
41
42 2011-01-13 Michal Ludvig <mludvig@logix.net.nz>
43
44 * s3cmd, S3/FileLists.py: Move file/object listing functions
45 to S3/FileLists.py
46
47 2011-01-09 Michal Ludvig <mludvig@logix.net.nz>
48
49 * Released version 1.0.0
50 ----------------------
51
52 * S3/PkgInfo.py: Updated to 1.0.0
53 * NEWS: Updated.
54
55 2011-01-02 Michal Ludvig <mludvig@logix.net.nz>
56
57 * s3cmd: Improved r457 (Don't crash when file disappears
58 before checking MD5).
59 * s3cmd, s3cmd.1, format-manpage.pl: Improved --help text
60 and manpage.
61 * s3cmd: Removed explicit processing of --follow-symlinks
62 (is cought by the default / main loop).
63
64 2010-12-24 Michal Ludvig <mludvig@logix.net.nz>
65
66 * s3cmd: Set 10s socket timeout for read()/write().
67 * s3cmd: Added --(no-)check-md5 for [sync].
68 * run-tests.py, testsuite.tar.gz: Added testsuite for
69 the above.
70 * NEWS: Document the above.
71 * s3cmd: Don't crash when file disappears before
72 checking MD5.
73
74 2010-12-09 Michal Ludvig <mludvig@logix.net.nz>
75
76 * Released version 1.0.0-rc2
77 --------------------------
78
79 * S3/PkgInfo.py: Updated to 1.0.0-rc2
80 * NEWS, TODO, s3cmd.1: Updated.
81
82 2010-11-13 Michal Ludvig <mludvig@logix.net.nz>
83
84 * s3cmd: Added support for remote-to-remote sync.
85 (Based on patch from Sundar Raman - thanks!)
86 * run-tests.py: Testsuite for the above.
87
88 2010-11-12 Michal Ludvig <mludvig@logix.net.nz>
89
90 * s3cmd: Fixed typo in "s3cmd du" error path.
91
92 2010-11-12 Michal Ludvig <mludvig@logix.net.nz>
93
94 * format-manpage.pl: new manpage auto-formatter
95 * s3cmd.1: Updated using the above helper script
96 * setup.py: Warn if manpage is too old.
97
98 2010-10-27 Michal Ludvig <mludvig@logix.net.nz>
99
100 * run-tests.py, testsuite.tar.gz: Keep the testsuite in
101 SVN as a tarball. There's too many "strange" things
102 in the directory for it to be kept in SVN.
103
104 2010-10-27 Michal Ludvig <mludvig@logix.net.nz>
105
106 * TODO: Updated.
107 * upload-to-sf.sh: Updated for new SF.net system
108
109 2010-10-26 Michal Ludvig <mludvig@logix.net.nz>
110
111 * Released version 1.0.0-rc1
112 --------------------------
113
114 * S3/PkgInfo.py: Updated to 1.0.0-rc1
115 * NEWS, TODO: Updated.
116
117 2010-10-26 Michal Ludvig <mludvig@logix.net.nz>
118
119 * s3cmd, S3/CloudFront.py, S3/Config.py: Added support
120 for CloudFront DefaultRootObject. Thanks to Luke Andrew.
121
122 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
123
124 * s3cmd: Improved 'fixbucket' command. Thanks to Srinivasa
125 Moorthy.
126 * s3cmd: Read config file even if User Profile directory on
127 Windows contains non-ascii symbols. Thx Slava Vishnyakov
128
129 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
130
131 * s3cmd: Don't fail when a local node is a directory
132 and we expected a file. (as if for example /etc/passwd
133 was a dir)
134
135 2010-10-25 Michal Ludvig <mludvig@logix.net.nz>
136
137 * s3cmd, S3/S3.py: Ignore inaccessible (and missing) files
138 on upload.
139 * run-tests.py: Extended [sync] test to verify correct
140 handling of inaccessible files.
141 * testsuite/permission-tests: New testsuite files.
142
143 2010-10-24 Michal Ludvig <mludvig@logix.net.nz>
144
145 * S3/S3.py: "Stringify" all headers. Httplib should do
146 it but some Python 2.7 users reported problems that should
147 now be fixed.
148 * run-tests.py: Fixed test #6
149
150 2010-07-25 Aaron Maxwell <amax@resymbol.net>
151
152 * S3/Config.py, testsuite/etc/, run-tests.py, s3cmd.1, s3cmd:
153 Option to follow local symlinks for sync and
154 put (--follow-symlinks option), including tests and documentation
155 * run-tests.py: --bucket-prefix option, to allow different
156 developers to run tests in their own sandbox
157
158 2010-07-08 Michal Ludvig <mludvig@logix.net.nz>
159
160 * run-tests.py, testsuite/crappy-file-name.tar.gz:
161 Updated testsuite, work around a problem with [s3cmd cp]
162 when the source file contains '?' or '\x7f'
163 (where the inability to copy '?' is especially annoying).
164
165 2010-07-08 Michal Ludvig <mludvig@logix.net.nz>
166
167 * S3/Utils.py, S3/S3Uri.py: Fixed names after moving
168 functions between modules.
169
170 2010-06-29 Timothee Groleau <kde@timotheegroleau.com>
171
172 * S3/ACL.py: Fix isAnonRead method on Grantees
173 * ChangeLog: Update name of contributor for Timothee Groleau
174
175 2010-06-13 Michal Ludvig <mludvig@logix.net.nz>
176
177 * s3cmd, S3/CloudFront.py: Both [accesslog] and [cfmodify]
178 access logging can now be disabled with --no-access-logging
179
180 2010-06-13 Michal Ludvig <mludvig@logix.net.nz>
181
182 * S3/CloudFront.py: Allow s3:// URI as well as cf:// URI
183 for most CloudFront-related commands.
184
185 2010-06-12 Michal Ludvig <mludvig@logix.net.nz>
186
187 * s3cmd, S3/CloudFront.py, S3/Config.py: Support access
188 logging for CloudFront distributions.
189 * S3/S3.py, S3/Utils.py: Moved some functions to Utils.py
190 to make them available to CloudFront.py
191 * NEWS: Document the above.
192
193 2010-05-27 Michal Ludvig <mludvig@logix.net.nz>
194
195 * S3/S3.py: Fix bucket listing for buckets with
196 over 1000 prefixes. (contributed by Timothee Groleau)
197 * S3/S3.py: Fixed code formating.
198
199 2010-05-21 Michal Ludvig <mludvig@logix.net.nz>
200
201 * s3cmd, S3/S3.py: Added support for bucket locations
202 outside US/EU (i.e. us-west-1 and ap-southeast-1 as of now).
203
204 2010-05-21 Michal Ludvig <mludvig@logix.net.nz>
205
206 * s3cmd, S3/S3.py, S3/Config.py: Added --reduced-redundancy
207 switch for Reduced Redundancy Storage.
208
209 2010-05-20 Michal Ludvig <mludvig@logix.net.nz>
210
211 * s3cmd, S3/ACL.py, S3/Config.py: Support for --acl-grant
212 and --acl-revoke (contributed by Timothee Groleau)
213 * s3cmd: Couple of fixes on top of the above commit.
214 * s3cmd: Pre-parse ACL parameters in OptionS3ACL()
215
216 2010-05-20 Michal Ludvig <mludvig@logix.net.nz>
217
218 * S3/Exceptions.py, S3/S3.py: Some HTTP_400 exceptions
219 are retriable.
220
221 2010-03-19 Michal Ludvig <mludvig@logix.net.nz>
222
223 * s3cmd, S3/ACL.py: Print all ACLs for a Grantee
224 (one Grantee can have multiple different Grant entries)
225
226 2010-03-19 Michal Ludvig <mludvig@logix.net.nz>
227
228 * s3cmd: Enable bucket-level ACL setting
229 * s3cmd, S3/AccessLog.py, ...: Added [accesslog] command.
230 * s3cmd: Fix imports from S3.Utils
231
232 2009-12-10 Michal Ludvig <mludvig@logix.net.nz>
233
234 * s3cmd: Path separator conversion on Windows hosts.
235
236 2009-10-08 Michal Ludvig <mludvig@logix.net.nz>
237
238 * Released version 0.9.9.91
239 -------------------------
240
241 * S3/PkgInfo.py: Updated to 0.9.9.91
242 * NEWS: News for 0.9.9.91
243
244 2009-10-08 Michal Ludvig <mludvig@logix.net.nz>
245
246 * S3/S3.py: fixed reference to _max_retries.
247
248 2009-10-06 Michal Ludvig <mludvig@logix.net.nz>
249
250 * Released version 0.9.9.90
251 -------------------------
252
253 * S3/PkgInfo.py: Updated to 0.9.9.90
254 * NEWS: News for 0.9.9.90
255
256 2009-10-06 Michal Ludvig <mludvig@logix.net.nz>
257
258 * S3/S3.py: Introduce throttling on upload only after
259 second failure. I.e. first retry at full speed.
260 * TODO: Updated with new ideas.
261
262 2009-06-02 Michal Ludvig <michal@logix.cz>
263
264 * s3cmd: New [fixbucket] command for fixing invalid object
265 names in a given Bucket. For instance names with &#x08; in
266 them (not sure how people manage to upload them but they do).
267 * S3/S3.py, S3/Utils.py, S3/Config.py: Support methods for
268 the above, plus advise user to run 'fixbucket' when XML parsing
269 fails.
270 * NEWS: Updated.
271
272 2009-05-29 Michal Ludvig <michal@logix.cz>
273
274 * S3/Utils.py: New function replace_nonprintables()
275 * s3cmd: Filter local filenames through the above function
276 to avoid problems with uploaded filenames containing invalid
277 XML entities, eg &#08;
278 * S3/S3.py: Warn if a non-printables char is passed to
279 urlencode_string() - they should have been replaced earlier
280 in the processing.
281 * run-tests.py, TODO, NEWS: Updated.
282 * testsuite/crappy-file-name.tar.gz: Tarball with a crappy-named
283 file. Untar for the testsuite.
284
285 2009-05-29 Michal Ludvig <michal@logix.cz>
286
287 * testsuite/blahBlah/*: Added files needed for run-tests.py
288
289 2009-05-28 Michal Ludvig <michal@logix.cz>
290
291 * S3/Utils.py (dateS3toPython): Be more relaxed about
292 timestamps format.
293
294 2009-05-28 Michal Ludvig <michal@logix.cz>
295
296 * s3cmd, run-test.py, TODO, NEWS: Added --dry-run
297 and --exclude/--include for [setacl].
298 * s3cmd, run-test.py, TODO, NEWS: Added --dry-run
299 and --exclude/--include for [del].
300
301 2009-05-28 Michal Ludvig <michal@logix.cz>
302
303 * s3cmd: Support for recursive [cp] and [mv], including
304 multiple-source arguments, --include/--exclude,
305 --dry-run, etc.
306 * run-tests.py: Tests for the above.
307 * S3/S3.py: Preserve metadata (eg ACL or MIME type)
308 during [cp] and [mv].
309 * NEWS, TODO: Updated.
310
311 2009-05-28 Michal Ludvig <michal@logix.cz>
312
313 * run-tests.py: Added --verbose mode.
314
315 2009-05-27 Michal Ludvig <michal@logix.cz>
316
317 * NEWS: Added info about --verbatim.
318 * TODO: Added more tasks.
319
320 2009-05-27 Michal Ludvig <michal@logix.cz>
321
322 * S3/SortedDict.py: Add case-sensitive mode.
323 * s3cmd, S3/S3.py, S3/Config.py: Use SortedDict() in
324 case-sensitive mode to avoid dropping filenames
325 differing only in capitalisation
326 * run-tests.py: Testsuite for the above.
327 * NEWS: Updated.
328
329 2009-03-20 Michal Ludvig <michal@logix.cz>
330
331 * S3/S3.py: Re-sign requests before retrial to avoid
332 RequestTimeTooSkewed errors on failed long-running
333 uploads.
334 BTW 'request' now has its own class S3Request.
335
336 2009-03-04 Michal Ludvig <michal@logix.cz>
337
338 * s3cmd, S3/Config.py, S3/S3.py: Support for --verbatim.
339
340 2009-02-25 Michal Ludvig <michal@logix.cz>
341
342 * s3cmd: Fixed "put file.ext s3://bkt" (ie just the bucket name).
343 * s3cmd: Fixed reporting of ImportError of S3 modules.
344 * s3cmd: Fixed Error: global name 'real_filename' is not defined
345
346 2009-02-24 Michal Ludvig <michal@logix.cz>
347
348 * s3cmd: New command [sign]
349 * S3/Utils.py: New function sign_string()
350 * S3/S3.py, S3/CloudFront.py: Use sign_string().
351 * NEWS: Updated.
352
353 2009-02-17 Michal Ludvig <michal@logix.cz>
354
355 * Released version 0.9.9
356 ----------------------
357
358 * S3/PkgInfo.py: Updated to 0.9.9
359 * NEWS: Compile a big news list for 0.9.9
360
361 2009-02-17 Michal Ludvig <michal@logix.cz>
362
363 * s3cmd.1: Document all the new options and commands.
364 * s3cmd, S3/Config.py: Updated some help texts. Removed
365 option --debug-syncmatch along the way (because --dry-run
366 with --debug is good enough).
367 * TODO: Updated.
368
369 2009-02-16 Michal Ludvig <michal@logix.cz>
370
371 * s3cmd: Check Python version >= 2.4 as soon as possible.
372
373 2009-02-14 Michal Ludvig <michal@logix.cz>
374
375 * s3cmd, S3/Config.py, S3/S3.py: Added --add-header option.
376 * NEWS: Documented --add-header.
377 * run-tests.py: Fixed for new messages.
378
379 2009-02-14 Michal Ludvig <michal@logix.cz>
380
381 * README: Updated for 0.9.9
382 * s3cmd, S3/PkgInfo.py, s3cmd.1: Replaced project
383 URLs with http://s3tools.org
384 * NEWS: Improved message.
385
386 2009-02-12 Michal Ludvig <michal@logix.cz>
387
388 * s3cmd: Added --list-md5 for 'ls' command.
389 * S3/Config.py: New setting list_md5
390
391 2009-02-12 Michal Ludvig <michal@logix.cz>
392
393 * s3cmd: Set Content-Length header for requests with 'body'.
394 * s3cmd: And send it for requests with no body as well...
395
396 2009-02-02 Michal Ludvig <michal@logix.cz>
397
398 * Released version 0.9.9-rc3
399 --------------------------
400
401 * S3/PkgInfo.py, NEWS: Updated for 0.9.9-rc3
402
403 2009-02-01 Michal Ludvig <michal@logix.cz>
404
405 * S3/Exceptions.py: Correct S3Exception.__str__() to
406 avoid crash in S3Error() subclass. Reported by '~t2~'.
407 * NEWS: Updated.
408
409 2009-01-30 Michal Ludvig <michal@logix.cz>
410
411 * Released version 0.9.9-rc2
412 --------------------------
413
414 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-rc2
415
416 2009-01-30 Michal Ludvig <michal@logix.cz>
417
418 * s3cmd: Under some circumstance s3cmd crashed
419 when put/get/sync had 0 files to transmit. Fixed now.
420
421 2009-01-28 Michal Ludvig <michal@logix.cz>
422
423 * s3cmd: Output 'delete:' in --dry-run only when
424 used together with --delete-removed. Otherwise
425 the user will think that without --dry-run it
426 would really delete the files.
427
428 2009-01-27 Michal Ludvig <michal@logix.cz>
429
430 * Released version 0.9.9-rc1
431 --------------------------
432
433 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-rc1
434
435 2009-01-26 Michal Ludvig <michal@logix.cz>
436
437 * Merged CloudFront support from branches/s3cmd-airlock
438 See the ChangeLog in that branch for details.
439
440 2009-01-25 W. Tell <w_tell -at- sourceforge>
441
442 * s3cmd: Implemented --include and friends.
443
444 2009-01-25 Michal Ludvig <michal@logix.cz>
445
446 * s3cmd: Enabled --dry-run and --exclude for 'put' and 'get'.
447 * S3/Exceptions.py: Remove DeprecationWarning about
448 BaseException.message in Python 2.6
449 * s3cmd: Rewritten gpg_command() to use subprocess.Popen()
450 instead of os.popen4() deprecated in 2.6
451 * TODO: Note about failing GPG.
452
453 2009-01-22 Michal Ludvig <michal@logix.cz>
454
455 * S3/Config.py: guess_mime_type = True (will affect new
456 installations only).
457
458 2009-01-22 Michal Ludvig <michal@logix.cz>
459
460 * Released version 0.9.9-pre5
461 ---------------------------
462
463 * S3/PkgInfo.py, NEWS, TODO: Updated for 0.9.9-pre5
464
465 2009-01-22 Michal Ludvig <michal@logix.cz>
466
467 * run-tests.py: Updated paths for the new sync
468 semantics.
469 * s3cmd, S3/S3.py: Small fixes to make testsuite happy.
470
471 2009-01-21 Michal Ludvig <michal@logix.cz>
472
473 * s3cmd: Migrated 'sync' local->remote to the new
474 scheme with fetch_{local,remote}_list().
475 Enabled --dry-run for 'sync'.
476
477 2009-01-20 Michal Ludvig <michal@logix.cz>
478
479 * s3cmd: Migrated 'sync' remote->local to the new
480 scheme with fetch_{local,remote}_list().
481 Changed fetch_remote_list() to return dict() compatible
482 with fetch_local_list().
483 Re-implemented --exclude / --include processing.
484 * S3/Utils.py: functions for parsing RFC822 dates (for HTTP
485 header responses).
486 * S3/Config.py: placeholders for --include.
487
488 2009-01-15 Michal Ludvig <michal@logix.cz>
489
490 * s3cmd, S3/S3Uri.py, NEWS: Support for recursive 'put'.
491
492 2009-01-13 Michal Ludvig <michal@logix.cz>
493
494 * TODO: Updated.
495 * s3cmd: renamed (fetch_)remote_keys to remote_list and
496 a few other renames for consistency.
497
498 2009-01-08 Michal Ludvig <michal@logix.cz>
499
500 * S3/S3.py: Some errors during file upload were incorrectly
501 interpreted as MD5 mismatch. (bug #2384990)
502 * S3/ACL.py: Move attributes from class to instance.
503 * run-tests.py: Tests for ACL.
504 * s3cmd: Minor messages changes.
505
506 2009-01-07 Michal Ludvig <michal@logix.cz>
507
508 * s3cmd: New command 'setacl'.
509 * S3/S3.py: Implemented set_acl().
510 * S3/ACL.py: Fill in <Owner/> tag in ACL XML.
511 * NEWS: Info about 'setacl'.
512
513 2009-01-07 Michal Ludvig <michal@logix.cz>
514
515 * s3cmd: Factored remote_keys generation from cmd_object_get()
516 to fetch_remote_keys().
517 * s3cmd: Display Public URL in 'info' for AnonRead objects.
518 * S3/ACL.py: Generate XML from a current list of Grantees
519
520 2009-01-07 Michal Ludvig <michal@logix.cz>
521
522 * S3/ACL.py: Keep ACL internally as a list of of 'Grantee' objects.
523 * S3/Utils.py: Fix crash in stripNameSpace() when the XML has no NS.
524
525 2009-01-07 Michal Ludvig <michal@logix.cz>
526
527 * S3/ACL.py: New object for handling ACL issues.
528 * S3/S3.py: Moved most of S3.get_acl() to ACL class.
529 * S3/Utils.py: Reworked XML helpers - remove XMLNS before
530 parsing the input XML to avoid having all Tags prefixed
531 with {XMLNS} by ElementTree.
532
533 2009-01-03 Michal Ludvig <michal@logix.cz>
534
535 * s3cmd: Don't fail when neither $HOME nor %USERPROFILE% is set.
536 (fixes #2483388)
537
538 2009-01-01 W. Tell <w_tell -at- sourceforge>
539
540 * S3/S3.py, S3/Utils.py: Use 'hashlib' instead of md5 and sha
541 modules to avoid Python 2.6 warnings.
542
543 2008-12-31 Michal Ludvig <michal@logix.cz>
544
545 * Released version 0.9.9-pre4
546 ---------------------------
547
548 2008-12-31 Michal Ludvig <michal@logix.cz>
549
550 * s3cmd: Reworked internal handling of unicode vs encoded filenames.
551 Should replace unknown characters with '?' instead of baling out.
552
553 2008-12-31 Michal Ludvig <michal@logix.cz>
554
555 * run-tests.py: Display system encoding in use.
556 * s3cmd: Print a nice error message when --exclude-from
557 file is not readable.
558 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre4
559 * S3/Exceptions.py: Added missing imports.
560 * NEWS: Updated.
561 * testsuite: reorganised UTF-8 files, added GBK encoding files,
562 moved encoding-specific files to 'tar.gz' archives, removed
563 unicode dir.
564 * run-tests.py: Adapted to the above change.
565 * run-tests.sh: removed.
566 * testsuite/exclude.encodings: Added.
567 * run-tests.py: Don't assume utf-8, use preferred encoding
568 instead.
569 * s3cmd, S3/Utils.py, S3/Exceptions.py, S3/Progress.py,
570 S3/Config.py, S3/S3.py: Added --encoding switch and
571 Config.encoding variable. Don't assume utf-8 for filesystem
572 and terminal output anymore.
573 * s3cmd: Avoid ZeroDivisionError on fast links.
574 * s3cmd: Unicodised all info() output.
575
576 2008-12-30 Michal Ludvig <michal@logix.cz>
577
578 * s3cmd: Replace unknown Unicode characters with '?'
579 to avoid UnicodeEncodeError's. Also make all output strings
580 unicode.
581 * run-tests.py: Exit on failed test. Fixed order of tests.
582
583 2008-12-29 Michal Ludvig <michal@logix.cz>
584
585 * TODO, NEWS: Updated
586 * s3cmd: Improved wildcard get.
587 * run-tests.py: Improved testsuite, added parameters support
588 to run only specified tests, cleaned up win/posix integration.
589 * S3/Exception.py: Python 2.4 doesn't automatically set
590 Exception.message.
591
592 2008-12-29 Michal Ludvig <michal@logix.cz>
593
594 * s3cmd, run-tests.py: Make it work on Windows.
595
596 2008-12-26 Michal Ludvig <michal@logix.cz>
597
598 * setup.cfg: Remove explicit install prefix. That should fix
599 Mac OS X and Windows "setup.py install" runs.
600
601 2008-12-22 Michal Ludvig <michal@logix.cz>
602
603 * s3cmd, S3/S3.py, S3/Progress.py: Display "[X of Y]"
604 in --progress mode.
605 * s3cmd, S3/Config.py: Implemented recursive [get].
606 Added --skip-existing option for [get] and [sync].
607
608 2008-12-17 Michal Ludvig <michal@logix.cz>
609
610 * TODO: Updated
611
612 2008-12-14 Michal Ludvig <michal@logix.cz>
613
614 * S3/Progress.py: Restructured import Utils to avoid import
615 conflicts.
616
617 2008-12-12 Michal Ludvig <michal@logix.cz>
618
619 * s3cmd: Better Exception output. Print sys.path on ImportError,
620 don't print backtrace on KeyboardInterrupt
621
622 2008-12-11 Michal Ludvig <michal@logix.cz>
623
624 * s3cmd: Support for multiple sources in 'get' command.
625
626 2008-12-10 Michal Ludvig <michal@logix.cz>
627
628 * TODO: Updated list.
629 * s3cmd: Don't display download/upload completed message
630 in --progress mode.
631 * S3/S3.py: Pass src/dst names down to Progress class.
632 * S3/Progress.py: added new class ProgressCR - apparently
633 ProgressANSI doesn't work on MacOS-X (and perhaps elsewhere).
634 * S3/Config.py: Default progress meter is now ProgressCR
635 * s3cmd: Updated email address for reporting bugs.
636
637 2008-12-02 Michal Ludvig <michal@logix.cz>
638
639 * s3cmd, S3/S3.py, NEWS: Support for (non-)recursive 'ls'
640
641 2008-12-01 Michal Ludvig <michal@logix.cz>
642
643 * Released version 0.9.9-pre3
644 ---------------------------
645
646 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre3
647
648 2008-12-01 Michal Ludvig <michal@logix.cz>
649
650 * run-tests.py: Added a lot of new tests.
651 * testsuite/etc/logo.png: New file.
652
653 2008-11-30 Michal Ludvig <michal@logix.cz>
654
655 * S3/S3.py: object_get() -- make start_position argument optional.
656
657 2008-11-29 Michal Ludvig <michal@logix.cz>
658
659 * s3cmd: Delete local files with "sync --delete-removed"
660
661 2008-11-25 Michal Ludvig <michal@logix.cz>
662
663 * s3cmd, S3/Progress.py: Fixed Unicode output in Progress meter.
664 * s3cmd: Fixed 'del --recursive' without prefix (i.e. all objects).
665 * TODO: Updated list.
666 * upload-to-sf.sh: Helper script.
667 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre2+svn
668
669 2008-11-24 Michal Ludvig <michal@logix.cz>
670
671 * Released version 0.9.9-pre2
672 ------------------------
673
674 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre2
675 * NEWS: Added 0.9.9-pre2
676
677 2008-11-24 Michal Ludvig <michal@logix.cz>
678
679 * s3cmd, s3cmd.1, S3/S3.py: Display or don't display progress meter
680 default depends on whether we're on TTY (console) or not.
681
682 2008-11-24 Michal Ludvig <michal@logix.cz>
683
684 * s3cmd: Fixed 'get' conflict.
685 * s3cmd.1, TODO: Document 'mv' command.
686
687 2008-11-24 Michal Ludvig <michal@logix.cz>
688
689 * S3/S3.py, s3cmd, S3/Config.py, s3cmd.1: Added --continue for
690 'get' command, improved 'get' failure resiliency.
691 * S3/Progress.py: Support for progress meter not starting in 0.
692 * S3/S3.py: improved retrying in send_request() and send_file()
693
694 2008-11-24 Michal Ludvig <michal@logix.cz>
695
696 * s3cmd, S3/S3.py, NEWS: "s3cmd mv" for moving objects
697
698 2008-11-24 Michal Ludvig <michal@logix.cz>
699
700 * S3/Utils.py: Common XML parser.
701 * s3cmd, S3/Exeptions.py: Print info message on Error.
702
703 2008-11-21 Michal Ludvig <michal@logix.cz>
704
705 * s3cmd: Support for 'cp' command.
706 * S3/S3.py: Added S3.object.copy() method.
707 * s3cmd.1: Document 'cp' command.
708 * NEWS: Let everyone know ;-)
709 Thanks Andrew Ryan for a patch proposal!
710 https://sourceforge.net/forum/forum.php?thread_id=2346987&forum_id=618865
711
712 2008-11-17 Michal Ludvig <michal@logix.cz>
713
714 * S3/Progress.py: Two progress meter implementations.
715 * S3/Config.py, s3cmd: New --progress / --no-progress parameters
716 and Config() members.
717 * S3/S3.py: Call Progress() in send_file()/recv_file()
718 * NEWS: Let everyone know ;-)
719
720 2008-11-16 Michal Ludvig <michal@logix.cz>
721
722 * NEWS: Fetch 0.9.8.4 release news from 0.9.8.x branch.
723
724 2008-11-16 Michal Ludvig <michal@logix.cz>
725
726 Merge from 0.9.8.x branch, rel 251:
727 * S3/S3.py: Adjusting previous commit (orig 249) - it's not a good idea
728 to retry ALL failures. Especially not those code=4xx where AmazonS3
729 servers are not happy with our requests.
730 Merge from 0.9.8.x branch, rel 249:
731 * S3/S3.py, S3/Exception.py: Re-issue failed requests in S3.send_request()
732 Merge from 0.9.8.x branch, rel 248:
733 * s3cmd: Don't leak open filehandles in sync. Thx Patrick Linskey for report.
734 Merge from 0.9.8.x branch, rel 247:
735 * s3cmd: Re-raise the right exception.
736 Merge from 0.9.8.x branch, rel 246:
737 * s3cmd, S3/S3.py, S3/Exceptions.py: Don't abort 'sync' or 'put' on files
738 that can't be open (e.g. Permision denied). Print a warning and skip over
739 instead.
740 Merge from 0.9.8.x branch, rel 245:
741 * S3/S3.py: Escape parameters in strings. Fixes sync to and
742 ls of directories with spaces. (Thx Lubomir Rintel from Fedora Project)
743 Merge from 0.9.8.x branch, rel 244:
744 * s3cmd: Unicode brainfuck again. This time force all output
745 in UTF-8, will see how many complaints we'll get...
746
747 2008-09-16 Michal Ludvig <michal@logix.cz>
748
749 * NEWS: s3cmd 0.9.8.4 released from branches/0.9.8.x SVN branch.
750
751 2008-09-16 Michal Ludvig <michal@logix.cz>
752
753 * S3/S3.py: Don't run into ZeroDivisionError when speed counter
754 returns 0s elapsed on upload/download file.
755
756 2008-09-15 Michal Ludvig <michal@logix.cz>
757
758 * s3cmd, S3/S3.py, S3/Utils.py, S3/S3Uri.py, S3/Exceptions.py:
759 Yet anoter Unicode round. Unicodised all command line arguments
760 before processing.
761
762 2008-09-15 Michal Ludvig <michal@logix.cz>
763
764 * S3/S3.py: "s3cmd mb" can create upper-case buckets again
765 in US. Non-US (e.g. EU) bucket names must conform to strict
766 DNS-rules.
767 * S3/S3Uri.py: Display public URLs correctly for non-DNS buckets.
768
769 2008-09-10 Michal Ludvig <michal@logix.cz>
770
771 * testsuite, run-tests.py: Added testsuite with first few tests.
772
773 2008-09-10 Michal Ludvig <michal@logix.cz>
774
775 * s3cmd, S3/S3Uri.py, S3/S3.py: All internal representations of
776 S3Uri()s are Unicode (i.e. not UTF-8 but type()==unicode). It
777 still doesn't work on non-UTF8 systems though.
778
779 2008-09-04 Michal Ludvig <michal@logix.cz>
780
781 * s3cmd: Rework UTF-8 output to keep sys.stdout untouched (or it'd
782 break 's3cmd get' to stdout for binary files).
783
784 2008-09-03 Michal Ludvig <michal@logix.cz>
785
786 * s3cmd, S3/S3.py, S3/Config.py: Removed --use-old-connect-method
787 again. Autodetect the need for old connect method instead.
788
789 2008-09-03 Michal Ludvig <michal@logix.cz>
790
791 * s3cmd, S3/S3.py: Make --verbose mode more useful and default
792 mode less verbose.
793
794 2008-09-03 Michal Ludvig <michal@logix.cz>
795
796 * s3cmd, S3/Config.py: [rb] Allow removal of non-empty buckets
797 with --force.
798 [mb, rb] Allow multiple arguments, i.e. create or remove
799 multiple buckets at once.
800 [del] Perform recursive removal with --recursive (or -r).
801
802 2008-09-01 Michal Ludvig <michal@logix.cz>
803
804 * s3cmd: Refuse 'sync' together with '--encrypt'.
805 * S3/S3.py: removed object_{get,put,delete}_uri() functions
806 and made object_{get,put,delete}() accept URI instead of
807 bucket/object parameters.
808
809 2008-09-01 Michal Ludvig <michal@logix.cz>
810
811 * S3/PkgInfo.py: Bumped up version to 0.9.9-pre1
812
813 2008-09-01 Michal Ludvig <michal@logix.cz>
814
815 * s3cmd, S3/S3.py, S3/Config.py: Allow access to upper-case
816 named buckets again with --use-old-connect-method
817 (uses http://s3.amazonaws.com/bucket/object instead of
818 http://bucket.s3.amazonaws.com/object)
819
820 2008-08-19 Michal Ludvig <michal@logix.cz>
821
822 * s3cmd: Always output UTF-8, even on output redirects.
823
824 2008-08-01 Michal Ludvig <michal@logix.cz>
825
826 * TODO: Add some items
827
828 2008-07-29 Michal Ludvig <michal@logix.cz>
829
830 * Released version 0.9.8.3
831 ------------------------
832
833 2008-07-29 Michal Ludvig <michal@logix.cz>
834
835 * S3/PkgInfo.py: Bumped up version to 0.9.8.3
836 * NEWS: Added 0.9.8.3
837
838 2008-07-29 Michal Ludvig <michal@logix.cz>
839
840 * S3/Utils.py (hash_file_md5): Hash files in 32kB chunks
841 instead of reading it all up to a memory first to avoid
842 OOM on large files.
843
844 2008-07-07 Michal Ludvig <michal@logix.cz>
845
846 * s3cmd.1: couple of syntax fixes from Mikhail Gusarov
847
848 2008-07-03 Michal Ludvig <michal@logix.cz>
849
850 * Released version 0.9.8.2
851 ------------------------
852
853 2008-07-03 Michal Ludvig <michal@logix.cz>
854
855 * S3/PkgInfo.py: Bumped up version to 0.9.8.2
856 * NEWS: Added 0.9.8.2
857 * s3cmd: Print version info on 'unexpected error' output.
858
859 2008-06-30 Michal Ludvig <michal@logix.cz>
860
861 * S3/S3.py: Re-upload when Amazon doesn't send ETag
862 in PUT response. It happens from time to time for
863 unknown reasons. Thanks "Burtc" for report and
864 "hermzz" for fix.
865
866 2008-06-27 Michal Ludvig <michal@logix.cz>
867
868 * Released version 0.9.8.1
869 ------------------------
870
871 2008-06-27 Michal Ludvig <michal@logix.cz>
872
873 * S3/PkgInfo.py: Bumped up version to 0.9.8.1
874 * NEWS: Added 0.9.8.1
875 * s3cmd: make 'cfg' global
876 * run-tests.sh: Sort-of testsuite
877
878 2008-06-23 Michal Ludvig <michal@logix.cz>
879
880 * Released version 0.9.8
881 ----------------------
882
883 2008-06-23 Michal Ludvig <michal@logix.cz>
884
885 * S3/PkgInfo.py: Bumped up version to 0.9.8
886 * NEWS: Added 0.9.8
887 * TODO: Removed completed tasks
888
889 2008-06-23 Michal Ludvig <michal@logix.cz>
890
891 * s3cmd: Last-minute compatibility fixes for Python 2.4
892 * s3cmd, s3cmd.1: --debug-exclude is an alias for --debug-syncmatch
893 * s3cmd: Don't require $HOME env variable to be set.
894 Fixes #2000133
895 * s3cmd: Wrapped all execution in a try/except block
896 to catch all exceptions and ask for a report.
897
898 2008-06-18 Michal Ludvig <michal@logix.cz>
899
900 * S3/PkgInfo.py: Version 0.9.8-rc3
901
902 2008-06-18 Michal Ludvig <michal@logix.cz>
903
904 * S3/S3.py: Bucket name can't contain upper-case letters (S3/DNS limitation).
905
906 2008-06-12 Michal Ludvig <michal@logix.cz>
907
908 * S3/PkgInfo.py: Version 0.9.8-rc2
909
910 2008-06-12 Michal Ludvig <michal@logix.cz>
911
912 * s3cmd, s3cmd.1: Added GLOB (shell-style wildcard) exclude, renamed
913 orig regexp-style --exclude to --rexclude
914
915 2008-06-11 Michal Ludvig <michal@logix.cz>
916
917 * S3/PkgInfo.py: Version 0.9.8-rc1
918
919 2008-06-11 Michal Ludvig <michal@logix.cz>
920
921 * s3cmd: Remove python 2.5 specific code (try/except/finally
922 block) and make s3cmd compatible with python 2.4 again.
923 * s3cmd, S3/Config.py, s3cmd.1: Added --exclude-from and --debug-syncmatch
924 switches for sync.
925
926 2008-06-10 Michal Ludvig <michal@logix.cz>
927
928 * s3cmd: Added --exclude switch for sync.
929 * s3cmd.1, NEWS: Document --exclude
930
931 2008-06-05 Michal Ludvig <michal@logix.cz>
932
933 * Released version 0.9.7
934 ----------------------
935
936 2008-06-05 Michal Ludvig <michal@logix.cz>
937
938 * S3/PkgInfo.py: Bumped up version to 0.9.7
939 * NEWS: Added 0.9.7
940 * TODO: Removed completed tasks
941 * s3cmd, s3cmd.1: Updated help texts,
942 removed --dry-run option as it's not implemented.
943
944 2008-06-05 Michal Ludvig <michal@logix.cz>
945
946 * S3/Config.py: Store more file attributes in sync to S3.
947 * s3cmd: Make sync remote2local more error-resilient.
948
949 2008-06-04 Michal Ludvig <michal@logix.cz>
950
951 * s3cmd: Implemented cmd_sync_remote2local() for restoring
952 backup from S3 to a local filesystem
953 * S3/S3.py: S3.object_get_uri() now requires writable stream
954 and not a path name.
955 * S3/Utils.py: Added mkdir_with_parents()
956
957 2008-06-04 Michal Ludvig <michal@logix.cz>
958
959 * s3cmd: Refactored cmd_sync() in preparation
960 for remote->local sync.
961
962 2008-04-30 Michal Ludvig <michal@logix.cz>
963
964 * s3db, S3/SimpleDB.py: Implemented almost full SimpleDB API.
965
966 2008-04-29 Michal Ludvig <michal@logix.cz>
967
968 * s3db, S3/SimpleDB.py: Initial support for Amazon SimpleDB.
969 For now implements ListDomains() call and most of the
970 infrastructure required for request creation.
971
972 2008-04-29 Michal Ludvig <michal@logix.cz>
973
974 * S3/Exceptions.py: Exceptions moved out of S3.S3
975 * S3/SortedDict.py: rewritten from scratch to preserve
976 case of keys while still sorting in case-ignore mode.
977
978 2008-04-28 Michal Ludvig <michal@logix.cz>
979
980 * S3/S3.py: send_file() now computes MD5 sum of the file
981 being uploaded, compares with ETag returned by Amazon
982 and retries upload if they don't match.
983
984 2008-03-05 Michal Ludvig <michal@logix.cz>
985
986 * s3cmd, S3/S3.py, S3/Utils.py: Throttle upload speed and retry
987 when upload failed.
988 Report download/upload speed and time elapsed.
989
990 2008-02-28 Michal Ludvig <michal@logix.cz>
991
992 * Released version 0.9.6
993 ----------------------
994
995 2008-02-28 Michal Ludvig <michal@logix.cz>
996
997 * S3/PkgInfo.py: bumped up version to 0.9.6
998 * NEWS: What's new in 0.9.6
999
1000 2008-02-27 Michal Ludvig <michal@logix.cz>
1001
1002 * s3cmd, s3cmd.1: Updated help and man page.
1003 * S3/S3.py, S3/Utils.py, s3cmd: Support for 's3cmd info' command.
1004 * s3cmd: Fix crash when 'sync'ing files with unresolvable owner uid/gid.
1005 * S3/S3.py, S3/Utils.py: open files in binary mode (otherwise windows
1006 users have problems).
1007 * S3/S3.py: modify 'x-amz-date' format (problems reported on MacOS X).
1008 Thanks Jon Larkowski for fix.
1009
1010 2008-02-27 Michal Ludvig <michal@logix.cz>
1011
1012 * TODO: Updated wishlist.
1013
1014 2008-02-11 Michal Ludvig <michal@logix.cz>
1015
1016 * S3/S3.py: Properly follow RedirectPermanent responses for EU buckets
1017 * S3/S3.py: Create public buckets with -P (#1837328)
1018 * S3/S3.py, s3cmd: Correctly display public URL on uploads.
1019 * S3/S3.py, S3/Config.py: Support for MIME types. Both
1020 default and guessing. Fixes bug #1872192 (Thanks Martin Herr)
1021
1022 2007-11-13 Michal Ludvig <michal@logix.cz>
1023
1024 * Released version 0.9.5
1025 ----------------------
1026
1027 2007-11-13 Michal Ludvig <michal@logix.cz>
1028
1029 * S3/S3.py: Support for buckets stored in Europe, access now
1030 goes via <bucket>.s3.amazonaws.com where possible.
1031
1032 2007-11-12 Michal Ludvig <michal@logix.cz>
1033
1034 * s3cmd: Support for storing file attributes (like ownership,
1035 mode, etc) in sync operation.
1036 * s3cmd, S3/S3.py: New command 'ib' to get information about
1037 bucket (only 'LocationConstraint' supported for now).
1038
1039 2007-10-01 Michal Ludvig <michal@logix.cz>
1040
1041 * s3cmd: Fix typo in argument name (patch
1042 from Kim-Minh KAPLAN, SF #1804808)
1043
1044 2007-09-25 Michal Ludvig <michal@logix.cz>
1045
1046 * s3cmd: Exit with error code on error (patch
1047 from Kim-Minh KAPLAN, SF #1800583)
1048
1049 2007-09-25 Michal Ludvig <michal@logix.cz>
1050
1051 * S3/S3.py: Don't fail if bucket listing doesn't have
1052 <IsTruncated> node.
1053 * s3cmd: Create ~/.s3cfg with 0600 permissions.
1054
1055 2007-09-13 Michal Ludvig <michal@logix.cz>
1056
1057 * s3cmd: Improved 'sync'
1058 * S3/S3.py: Support for buckets with over 1000 objects.
1059
1060 2007-09-03 Michal Ludvig <michal@logix.cz>
1061
1062 * s3cmd: Small tweaks to --configure workflow.
1063
1064 2007-09-02 Michal Ludvig <michal@logix.cz>
1065
1066 * s3cmd: Initial support for 'sync' operation. For
1067 now only local->s3 direction. In this version doesn't
1068 work well with non-ASCII filenames and doesn't support
1069 encryption.
1070
1071 2007-08-24 Michal Ludvig <michal@logix.cz>
1072
1073 * s3cmd, S3/Util.py: More ElementTree imports cleanup
1074
1075 2007-08-19 Michal Ludvig <michal@logix.cz>
1076
1077 * NEWS: Added news for 0.9.5
1078
1079 2007-08-19 Michal Ludvig <michal@logix.cz>
1080
1081 * s3cmd: Better handling of multiple arguments for put, get and del
1082
1083 2007-08-14 Michal Ludvig <michal@logix.cz>
1084
1085 * setup.py, S3/Utils.py: Try import xml.etree.ElementTree
1086 or elementtree.ElementTree module.
1087
1088 2007-08-14 Michal Ludvig <michal@logix.cz>
1089
1090 * s3cmd.1: Add info about --encrypt parameter.
1091
1092 2007-08-14 Michal Ludvig <michal@logix.cz>
1093
1094 * S3/PkgInfo.py: Bump up version to 0.9.5-pre
1095
1096 2007-08-13 Michal Ludvig <michal@logix.cz>
1097
1098 * Released version 0.9.4
1099 ----------------------
1100
1101 2007-08-13 Michal Ludvig <michal@logix.cz>
1102
1103 * S3/S3.py: Added function urlencode_string() that encodes
1104 non-ascii characters in object name before sending it to S3.
1105
1106 2007-08-13 Michal Ludvig <michal@logix.cz>
1107
1108 * README: Updated Amazon S3 pricing overview
1109
1110 2007-08-13 Michal Ludvig <michal@logix.cz>
1111
1112 * s3cmd, S3/Config.py, S3/S3.py: HTTPS support
1113
1114 2007-07-20 Michal Ludvig <michal@logix.cz>
1115
1116 * setup.py: Check correct Python version and ElementTree availability.
1117
1118 2007-07-05 Michal Ludvig <michal@logix.cz>
1119
1120 * s3cmd: --configure support for Proxy
1121 * S3/S3.py: HTTP proxy support from
1122 John D. Rowell <jdrowell@exerciseyourbrain.com>
1123
1124 2007-06-19 Michal Ludvig <michal@logix.cz>
1125
1126 * setup.py: Check for S3CMD_PACKAGING and don't install
1127 manpages and docs if defined.
1128 * INSTALL: Document the above change.
1129 * MANIFEST.in: Include uncompressed manpage
1130
1131 2007-06-17 Michal Ludvig <michal@logix.cz>
1132
1133 * s3cmd: Added encryption key support to --configure
1134 * S3/PkgInfo.py: Bump up version to 0.9.4-pre
1135 * setup.py: Cleaned up some rpm-specific stuff that
1136 caused problems to Debian packager Mikhail Gusarov
1137 * setup.cfg: Removed [bdist_rpm] section
1138 * MANIFEST.in: Include S3/*.py
1139
1140 2007-06-16 Michal Ludvig <michal@logix.cz>
1141
1142 * s3cmd.1: Syntax fixes from Mikhail Gusarov <dottedmag@dottedmag.net>
1143
1144 2007-05-27 Michal Ludvig <michal@logix.cz>
1145
1146 * Support for on-the-fly GPG encryption.
1147
1148 2007-05-26 Michal Ludvig <michal@logix.cz>
1149
1150 * s3cmd.1: Add info about "s3cmd du" command.
1151
1152 2007-05-26 Michal Ludvig <michal@logix.cz>
1153
1154 * Released version 0.9.3
1155 ----------------------
1156
1157 2007-05-26 Michal Ludvig <michal@logix.cz>
1158
1159 * s3cmd: Patch from Basil Shubin <basil.shubin@gmail.com>
1160 adding support for "s3cmd du" command.
1161 * s3cmd: Modified output format of "s3cmd du" to conform
1162 with unix "du".
1163 * setup.cfg: Require Python 2.5 in RPM. Otherwise it needs
1164 to require additional python modules (e.g. ElementTree)
1165 which may have different names in different distros. It's
1166 indeed still possible to manually install s3cmd with
1167 Python 2.4 and appropriate modules.
1168
1169 2007-04-09 Michal Ludvig <michal@logix.cz>
1170
1171 * Released version 0.9.2
1172 ----------------------
1173
1174 2007-04-09 Michal Ludvig <michal@logix.cz>
1175
1176 * s3cmd.1: Added manpage
1177 * Updated infrastructure files to create "better"
1178 distribution archives.
1179
1180 2007-03-26 Michal Ludvig <michal@logix.cz>
1181
1182 * setup.py, S3/PkgInfo.py: Move package info out of setup.py
1183 * s3cmd: new parameter --version
1184 * s3cmd, S3/S3Uri.py: Output public HTTP URL for objects
1185 stored with Public ACL.
1186
1187 2007-02-28 Michal Ludvig <michal@logix.cz>
1188
1189 * s3cmd: Verify supplied accesskey and secretkey
1190 in interactive configuration path.
1191 * S3/Config.py: Hide access key and secret key
1192 from debug output.
1193 * S3/S3.py: Modify S3Error exception to work
1194 in python 2.4 (=> don't expect Exception is
1195 a new-style class).
1196 * s3cmd: Updated for the above change.
1197
1198 2007-02-19 Michal Ludvig <michal@logix.cz>
1199
1200 * NEWS, INSTALL, README, setup.py: Added
1201 more documentation.
1202
1203 2007-02-19 Michal Ludvig <michal@logix.cz>
1204
1205 * S3/S3.py, s3cmd: New feature - allow "get" to stdout
1206
1207 2007-02-19 Michal Ludvig <michal@logix.cz>
1208
1209 * S3/S3fs.py: Removed (development moved to branch s3fs-devel).
1210
1211 2007-02-08 Michal Ludvig <michal@logix.cz>
1212
1213 * S3/S3fs.py:
1214 - Implemented mknod()
1215 - Can create directory structure
1216 - Rewritten to use SQLite3. Currently can create
1217 the filesystem, and a root inode.
1218
1219 2007-02-07 Michal Ludvig <michal@logix.cz>
1220
1221 * s3cmd (from /s3py:74): Renamed SVN top-level project
1222 s3py to s3cmd
1223
1224 2007-02-07 Michal Ludvig <michal@logix.cz>
1225
1226 * setup.cfg: Only require Python 2.4, not 2.5
1227 * S3/Config.py: Removed show_uri - no longer needed,
1228 it's now default
1229
1230 2007-02-07 Michal Ludvig <michal@logix.cz>
1231
1232 * setup.py
1233 - Version 0.9.1
1234
1235 2007-02-07 Michal Ludvig <michal@logix.cz>
1236
1237 * s3cmd: Change all "exit()" calls to "sys.exit()"
1238 and allow for python 2.4
1239 * S3/S3.py: Removed dependency on hashlib -> allow for python 2.4
1240
1241 2007-01-27 Michal Ludvig <michal@logix.cz>
1242
1243 * S3/S3.py, S3/S3Uri.py: Case insensitive regex in S3Uri.py
1244
1245 2007-01-26 Michal Ludvig <michal@logix.cz>
1246
1247 * S3/S3fs.py: Added support for stroing/loading inodes.
1248 No data yet however.
1249
1250 2007-01-26 Michal Ludvig <michal@logix.cz>
1251
1252 * S3/S3fs.py: Initial version of S3fs module.
1253 Can create filesystem via "S3fs.mkfs()"
1254
1255 2007-01-26 Michal Ludvig <michal@logix.cz>
1256
1257 * S3/BidirMap.py, S3/Config.py, S3/S3.py, S3/S3Uri.py,
1258 S3/SortedDict.py, S3/Utils.py, s3cmd: Added headers with
1259 copyright to all files
1260 * S3/S3.py, S3/S3Uri.py: Removed S3.compose_uri(), introduced
1261 S3UriS3.compose_uri() instead.
1262
1263 2007-01-26 Michal Ludvig <michal@logix.cz>
1264
1265 * S3/S3.py, S3/S3Uri.py, s3cmd:
1266 - Converted all users of parse_uri to S3Uri class API
1267 - Removed "cp" command again. Will have to use 'put'
1268 and 'get' for now.
1269
1270 2007-01-25 Michal Ludvig <michal@logix.cz>
1271
1272 * S3/S3Uri.py: New module S3/S3Uri.py
1273 * S3/S3.py, s3cmd: Converted "put" operation to use
1274 the new S3Uri class.
1275
1276 2007-01-24 Michal Ludvig <michal@logix.cz>
1277
1278 * S3/S3.py
1279 * s3cmd
1280 - Added 'cp' command
1281 - Renamed parse_s3_uri to parse_uri (this will go away anyway)
1282
1283 2007-01-19 Michal Ludvig <michal@logix.cz>
1284
1285 * setup.cfg
1286 * setup.py
1287 - Include README into tarballs
1288
1289 2007-01-19 Michal Ludvig <michal@logix.cz>
1290
1291 * README
1292 - Added comprehensive README file
1293
1294 2007-01-19 Michal Ludvig <michal@logix.cz>
1295
1296 * setup.cfg
1297 * setup.py
1298 - Added configuration for setup.py sdist
1299
1300 2007-01-19 Michal Ludvig <michal@logix.cz>
1301
1302 * S3/Config.py
1303 * s3cmd
1304 - Added interactive configurator (--configure)
1305 - Added config dumper (--dump-config)
1306 - Improved --help output
1307
1308 2007-01-19 Michal Ludvig <michal@logix.cz>
1309
1310 * setup.cfg
1311 * setup.py
1312 Added info for building RPM packages.
1313
1314 2007-01-18 Michal Ludvig <michal@logix.cz>
1315
1316 * S3/Config.py
1317 * S3/S3.py
1318 * s3cmd
1319 Moved class Config from S3/S3.py to S3/Config.py
1320
1321 2007-01-18 Michal Ludvig <michal@logix.cz>
1322
1323 * S3/Config.py (from /s3py/trunk/S3/ConfigParser.py:47)
1324 * S3/ConfigParser.py
1325 * S3/S3.py
1326 Renamed S3/ConfigParser.py to S3/Config.py
1327
1328 2007-01-18 Michal Ludvig <michal@logix.cz>
1329
1330 * s3cmd
1331 Added info about homepage
1332
1333 2007-01-17 Michal Ludvig <michal@logix.cz>
1334
1335 * S3/S3.py
1336 * s3cmd
1337 - Use prefix for listings if specified.
1338 - List all commands in --help
1339
1340 2007-01-16 Michal Ludvig <michal@logix.cz>
1341
1342 * S3/S3.py
1343 * s3cmd
1344 Major rework of Config class:
1345 - Renamed from AwsConfig to Config
1346 - Converted to Singleton (see Config.__new__() and an article on
1347 Wikipedia)
1348 - No more explicit listing of options - use introspection to get them
1349 (class variables that of type str, int or bool that don't start with
1350 underscore)
1351 - Check values read from config file and verify their type.
1352
1353 Added OptionMimeType and -m/-M options. Not yet implemented
1354 functionality in the rest of S3/S3.py
1355
1356 2007-01-15 Michal Ludvig <michal@logix.cz>
1357
1358 * S3/S3.py
1359 * s3cmd
1360 - Merged list-buckets and bucket-list-objects operations into
1361 a single 'ls' command.
1362 - New parameter -P for uploading publicly readable objects
1363
1364 2007-01-14 Michal Ludvig <michal@logix.cz>
1365
1366 * s3.py
1367 * setup.py
1368 Renamed s3.py to s3cmd (take 2)
1369
1370 2007-01-14 Michal Ludvig <michal@logix.cz>
1371
1372 * s3cmd (from /s3py/trunk/s3.py:45)
1373 Renamed s3.py to s3cmd
1374
1375 2007-01-14 Michal Ludvig <michal@logix.cz>
1376
1377 * S3
1378 * S3/S3.py
1379 * s3.py
1380 * setup.py
1381 All classes from s3.py go to S3/S3.py
1382 Added setup.py
1383
1384 2007-01-14 Michal Ludvig <michal@logix.cz>
1385
1386 * s3.py
1387 Minor fix S3.utils -> S3.Utils
1388
1389 2007-01-14 Michal Ludvig <michal@logix.cz>
1390
1391 * .svnignore
1392 * BidirMap.py
1393 * ConfigParser.py
1394 * S3
1395 * S3/BidirMap.py (from /s3py/trunk/BidirMap.py:35)
1396 * S3/ConfigParser.py (from /s3py/trunk/ConfigParser.py:38)
1397 * S3/SortedDict.py (from /s3py/trunk/SortedDict.py:35)
1398 * S3/Utils.py (from /s3py/trunk/utils.py:39)
1399 * S3/__init__.py
1400 * SortedDict.py
1401 * s3.py
1402 * utils.py
1403 Moved modules to their own package
1404
1405 2007-01-12 Michal Ludvig <michal@logix.cz>
1406
1407 * s3.py
1408 Added "del" command
1409 Converted all (?) commands to accept s3-uri
1410 Added -u/--show-uri parameter
1411
1412 2007-01-11 Michal Ludvig <michal@logix.cz>
1413
1414 * s3.py
1415 Verify MD5 on received files
1416 Improved upload of multiple files
1417 Initial S3-URI support (more tbd)
1418
1419 2007-01-11 Michal Ludvig <michal@logix.cz>
1420
1421 * s3.py
1422 Minor fixes:
1423 - store names of parsed files in AwsConfig
1424 - Print total size with upload/download
1425
1426 2007-01-11 Michal Ludvig <michal@logix.cz>
1427
1428 * s3.py
1429 * utils.py
1430 Added support for sending and receiving files.
1431
1432 2007-01-11 Michal Ludvig <michal@logix.cz>
1433
1434 * ConfigParser.py
1435 * s3.py
1436 List all Objects in all Buckets command
1437 Yet another logging improvement
1438 Version check for Python 2.5 or higher
1439
1440 2007-01-11 Michal Ludvig <michal@logix.cz>
1441
1442 * ConfigParser.py
1443 * s3.py
1444 * utils.py
1445 Added ConfigParser
1446 Improved setting logging levels
1447 It can now quite reliably list buckets and objects
1448
1449 2007-01-11 Michal Ludvig <michal@logix.cz>
1450
1451 * .svnignore
1452 Added ignore list
1453
1454 2007-01-11 Michal Ludvig <michal@logix.cz>
1455
1456 * .svnignore
1457 * BidirMap.py
1458 * SortedDict.py
1459 * s3.py
1460 * utils.py
1461 Initial import
+339
-0
LICENSE less more
0 GNU GENERAL PUBLIC LICENSE
1 Version 2, June 1991
2
3 Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
4 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
5 Everyone is permitted to copy and distribute verbatim copies
6 of this license document, but changing it is not allowed.
7
8 Preamble
9
10 The licenses for most software are designed to take away your
11 freedom to share and change it. By contrast, the GNU General Public
12 License is intended to guarantee your freedom to share and change free
13 software--to make sure the software is free for all its users. This
14 General Public License applies to most of the Free Software
15 Foundation's software and to any other program whose authors commit to
16 using it. (Some other Free Software Foundation software is covered by
17 the GNU Lesser General Public License instead.) You can apply it to
18 your programs, too.
19
20 When we speak of free software, we are referring to freedom, not
21 price. Our General Public Licenses are designed to make sure that you
22 have the freedom to distribute copies of free software (and charge for
23 this service if you wish), that you receive source code or can get it
24 if you want it, that you can change the software or use pieces of it
25 in new free programs; and that you know you can do these things.
26
27 To protect your rights, we need to make restrictions that forbid
28 anyone to deny you these rights or to ask you to surrender the rights.
29 These restrictions translate to certain responsibilities for you if you
30 distribute copies of the software, or if you modify it.
31
32 For example, if you distribute copies of such a program, whether
33 gratis or for a fee, you must give the recipients all the rights that
34 you have. You must make sure that they, too, receive or can get the
35 source code. And you must show them these terms so they know their
36 rights.
37
38 We protect your rights with two steps: (1) copyright the software, and
39 (2) offer you this license which gives you legal permission to copy,
40 distribute and/or modify the software.
41
42 Also, for each author's protection and ours, we want to make certain
43 that everyone understands that there is no warranty for this free
44 software. If the software is modified by someone else and passed on, we
45 want its recipients to know that what they have is not the original, so
46 that any problems introduced by others will not reflect on the original
47 authors' reputations.
48
49 Finally, any free program is threatened constantly by software
50 patents. We wish to avoid the danger that redistributors of a free
51 program will individually obtain patent licenses, in effect making the
52 program proprietary. To prevent this, we have made it clear that any
53 patent must be licensed for everyone's free use or not licensed at all.
54
55 The precise terms and conditions for copying, distribution and
56 modification follow.
57
58 GNU GENERAL PUBLIC LICENSE
59 TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
60
61 0. This License applies to any program or other work which contains
62 a notice placed by the copyright holder saying it may be distributed
63 under the terms of this General Public License. The "Program", below,
64 refers to any such program or work, and a "work based on the Program"
65 means either the Program or any derivative work under copyright law:
66 that is to say, a work containing the Program or a portion of it,
67 either verbatim or with modifications and/or translated into another
68 language. (Hereinafter, translation is included without limitation in
69 the term "modification".) Each licensee is addressed as "you".
70
71 Activities other than copying, distribution and modification are not
72 covered by this License; they are outside its scope. The act of
73 running the Program is not restricted, and the output from the Program
74 is covered only if its contents constitute a work based on the
75 Program (independent of having been made by running the Program).
76 Whether that is true depends on what the Program does.
77
78 1. You may copy and distribute verbatim copies of the Program's
79 source code as you receive it, in any medium, provided that you
80 conspicuously and appropriately publish on each copy an appropriate
81 copyright notice and disclaimer of warranty; keep intact all the
82 notices that refer to this License and to the absence of any warranty;
83 and give any other recipients of the Program a copy of this License
84 along with the Program.
85
86 You may charge a fee for the physical act of transferring a copy, and
87 you may at your option offer warranty protection in exchange for a fee.
88
89 2. You may modify your copy or copies of the Program or any portion
90 of it, thus forming a work based on the Program, and copy and
91 distribute such modifications or work under the terms of Section 1
92 above, provided that you also meet all of these conditions:
93
94 a) You must cause the modified files to carry prominent notices
95 stating that you changed the files and the date of any change.
96
97 b) You must cause any work that you distribute or publish, that in
98 whole or in part contains or is derived from the Program or any
99 part thereof, to be licensed as a whole at no charge to all third
100 parties under the terms of this License.
101
102 c) If the modified program normally reads commands interactively
103 when run, you must cause it, when started running for such
104 interactive use in the most ordinary way, to print or display an
105 announcement including an appropriate copyright notice and a
106 notice that there is no warranty (or else, saying that you provide
107 a warranty) and that users may redistribute the program under
108 these conditions, and telling the user how to view a copy of this
109 License. (Exception: if the Program itself is interactive but
110 does not normally print such an announcement, your work based on
111 the Program is not required to print an announcement.)
112
113 These requirements apply to the modified work as a whole. If
114 identifiable sections of that work are not derived from the Program,
115 and can be reasonably considered independent and separate works in
116 themselves, then this License, and its terms, do not apply to those
117 sections when you distribute them as separate works. But when you
118 distribute the same sections as part of a whole which is a work based
119 on the Program, the distribution of the whole must be on the terms of
120 this License, whose permissions for other licensees extend to the
121 entire whole, and thus to each and every part regardless of who wrote it.
122
123 Thus, it is not the intent of this section to claim rights or contest
124 your rights to work written entirely by you; rather, the intent is to
125 exercise the right to control the distribution of derivative or
126 collective works based on the Program.
127
128 In addition, mere aggregation of another work not based on the Program
129 with the Program (or with a work based on the Program) on a volume of
130 a storage or distribution medium does not bring the other work under
131 the scope of this License.
132
133 3. You may copy and distribute the Program (or a work based on it,
134 under Section 2) in object code or executable form under the terms of
135 Sections 1 and 2 above provided that you also do one of the following:
136
137 a) Accompany it with the complete corresponding machine-readable
138 source code, which must be distributed under the terms of Sections
139 1 and 2 above on a medium customarily used for software interchange; or,
140
141 b) Accompany it with a written offer, valid for at least three
142 years, to give any third party, for a charge no more than your
143 cost of physically performing source distribution, a complete
144 machine-readable copy of the corresponding source code, to be
145 distributed under the terms of Sections 1 and 2 above on a medium
146 customarily used for software interchange; or,
147
148 c) Accompany it with the information you received as to the offer
149 to distribute corresponding source code. (This alternative is
150 allowed only for noncommercial distribution and only if you
151 received the program in object code or executable form with such
152 an offer, in accord with Subsection b above.)
153
154 The source code for a work means the preferred form of the work for
155 making modifications to it. For an executable work, complete source
156 code means all the source code for all modules it contains, plus any
157 associated interface definition files, plus the scripts used to
158 control compilation and installation of the executable. However, as a
159 special exception, the source code distributed need not include
160 anything that is normally distributed (in either source or binary
161 form) with the major components (compiler, kernel, and so on) of the
162 operating system on which the executable runs, unless that component
163 itself accompanies the executable.
164
165 If distribution of executable or object code is made by offering
166 access to copy from a designated place, then offering equivalent
167 access to copy the source code from the same place counts as
168 distribution of the source code, even though third parties are not
169 compelled to copy the source along with the object code.
170
171 4. You may not copy, modify, sublicense, or distribute the Program
172 except as expressly provided under this License. Any attempt
173 otherwise to copy, modify, sublicense or distribute the Program is
174 void, and will automatically terminate your rights under this License.
175 However, parties who have received copies, or rights, from you under
176 this License will not have their licenses terminated so long as such
177 parties remain in full compliance.
178
179 5. You are not required to accept this License, since you have not
180 signed it. However, nothing else grants you permission to modify or
181 distribute the Program or its derivative works. These actions are
182 prohibited by law if you do not accept this License. Therefore, by
183 modifying or distributing the Program (or any work based on the
184 Program), you indicate your acceptance of this License to do so, and
185 all its terms and conditions for copying, distributing or modifying
186 the Program or works based on it.
187
188 6. Each time you redistribute the Program (or any work based on the
189 Program), the recipient automatically receives a license from the
190 original licensor to copy, distribute or modify the Program subject to
191 these terms and conditions. You may not impose any further
192 restrictions on the recipients' exercise of the rights granted herein.
193 You are not responsible for enforcing compliance by third parties to
194 this License.
195
196 7. If, as a consequence of a court judgment or allegation of patent
197 infringement or for any other reason (not limited to patent issues),
198 conditions are imposed on you (whether by court order, agreement or
199 otherwise) that contradict the conditions of this License, they do not
200 excuse you from the conditions of this License. If you cannot
201 distribute so as to satisfy simultaneously your obligations under this
202 License and any other pertinent obligations, then as a consequence you
203 may not distribute the Program at all. For example, if a patent
204 license would not permit royalty-free redistribution of the Program by
205 all those who receive copies directly or indirectly through you, then
206 the only way you could satisfy both it and this License would be to
207 refrain entirely from distribution of the Program.
208
209 If any portion of this section is held invalid or unenforceable under
210 any particular circumstance, the balance of the section is intended to
211 apply and the section as a whole is intended to apply in other
212 circumstances.
213
214 It is not the purpose of this section to induce you to infringe any
215 patents or other property right claims or to contest validity of any
216 such claims; this section has the sole purpose of protecting the
217 integrity of the free software distribution system, which is
218 implemented by public license practices. Many people have made
219 generous contributions to the wide range of software distributed
220 through that system in reliance on consistent application of that
221 system; it is up to the author/donor to decide if he or she is willing
222 to distribute software through any other system and a licensee cannot
223 impose that choice.
224
225 This section is intended to make thoroughly clear what is believed to
226 be a consequence of the rest of this License.
227
228 8. If the distribution and/or use of the Program is restricted in
229 certain countries either by patents or by copyrighted interfaces, the
230 original copyright holder who places the Program under this License
231 may add an explicit geographical distribution limitation excluding
232 those countries, so that distribution is permitted only in or among
233 countries not thus excluded. In such case, this License incorporates
234 the limitation as if written in the body of this License.
235
236 9. The Free Software Foundation may publish revised and/or new versions
237 of the General Public License from time to time. Such new versions will
238 be similar in spirit to the present version, but may differ in detail to
239 address new problems or concerns.
240
241 Each version is given a distinguishing version number. If the Program
242 specifies a version number of this License which applies to it and "any
243 later version", you have the option of following the terms and conditions
244 either of that version or of any later version published by the Free
245 Software Foundation. If the Program does not specify a version number of
246 this License, you may choose any version ever published by the Free Software
247 Foundation.
248
249 10. If you wish to incorporate parts of the Program into other free
250 programs whose distribution conditions are different, write to the author
251 to ask for permission. For software which is copyrighted by the Free
252 Software Foundation, write to the Free Software Foundation; we sometimes
253 make exceptions for this. Our decision will be guided by the two goals
254 of preserving the free status of all derivatives of our free software and
255 of promoting the sharing and reuse of software generally.
256
257 NO WARRANTY
258
259 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
260 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
261 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
262 PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
263 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
264 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
265 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
266 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
267 REPAIR OR CORRECTION.
268
269 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
270 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
271 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
272 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
273 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
274 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
275 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
276 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
277 POSSIBILITY OF SUCH DAMAGES.
278
279 END OF TERMS AND CONDITIONS
280
281 How to Apply These Terms to Your New Programs
282
283 If you develop a new program, and you want it to be of the greatest
284 possible use to the public, the best way to achieve this is to make it
285 free software which everyone can redistribute and change under these terms.
286
287 To do so, attach the following notices to the program. It is safest
288 to attach them to the start of each source file to most effectively
289 convey the exclusion of warranty; and each file should have at least
290 the "copyright" line and a pointer to where the full notice is found.
291
292 <one line to give the program's name and a brief idea of what it does.>
293 Copyright (C) <year> <name of author>
294
295 This program is free software; you can redistribute it and/or modify
296 it under the terms of the GNU General Public License as published by
297 the Free Software Foundation; either version 2 of the License, or
298 (at your option) any later version.
299
300 This program is distributed in the hope that it will be useful,
301 but WITHOUT ANY WARRANTY; without even the implied warranty of
302 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
303 GNU General Public License for more details.
304
305 You should have received a copy of the GNU General Public License along
306 with this program; if not, write to the Free Software Foundation, Inc.,
307 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
308
309 Also add information on how to contact you by electronic and paper mail.
310
311 If the program is interactive, make it output a short notice like this
312 when it starts in an interactive mode:
313
314 Gnomovision version 69, Copyright (C) year name of author
315 Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
316 This is free software, and you are welcome to redistribute it
317 under certain conditions; type `show c' for details.
318
319 The hypothetical commands `show w' and `show c' should show the appropriate
320 parts of the General Public License. Of course, the commands you use may
321 be called something other than `show w' and `show c'; they could even be
322 mouse-clicks or menu items--whatever suits your program.
323
324 You should also get your employer (if you work as a programmer) or your
325 school, if any, to sign a "copyright disclaimer" for the program, if
326 necessary. Here is a sample; alter the names:
327
328 Yoyodyne, Inc., hereby disclaims all copyright interest in the program
329 `Gnomovision' (which makes passes at compilers) written by James Hacker.
330
331 <signature of Ty Coon>, 1 April 1989
332 Ty Coon, President of Vice
333
334 This General Public License does not permit incorporating your program into
335 proprietary programs. If your program is a subroutine library, you may
336 consider it more useful to permit linking proprietary applications with the
337 library. If this is what you want to do, use the GNU Lesser General
338 Public License instead of this License.
+2
-0
MANIFEST.in less more
0 include INSTALL README NEWS
1 include s3cmd.1
+43
-0
Makefile less more
0 VERSION := 1.5.0
1 SHELL := /bin/bash
2 SPEC := s3cmd.spec
3 COMMIT := $(shell git rev-parse HEAD)
4 SHORTCOMMIT := $(shell git rev-parse --short=8 HEAD)
5 TARBALL = s3cmd-$(VERSION)-$(SHORTCOMMIT).tar.gz
6
7 release:
8 python setup.py register sdist upload
9
10 clean:
11 -rm -rf s3cmd-*.tar.gz *.rpm *~ $(SPEC)
12 -find . -name \*.pyc -exec rm \{\} \;
13 -find . -name \*.pyo -exec rm \{\} \;
14
15 $(SPEC): $(SPEC).in
16 sed -e 's/##VERSION##/$(VERSION)/' \
17 -e 's/##COMMIT##/$(COMMIT)/' \
18 -e 's/##SHORTCOMMIT##/$(SHORTCOMMIT)/' \
19 $(SPEC).in > $(SPEC)
20
21 tarball:
22 git archive --format tar --prefix s3cmd-$(COMMIT)/ HEAD | gzip -c > $(TARBALL)
23
24 # Use older digest algorithms for local rpmbuilds, as EPEL5 and
25 # earlier releases need this. When building using mock for a
26 # particular target, it will use the proper (newer) digests if that
27 # target supports it.
28 rpm: clean tarball $(SPEC)
29 tmp_dir=`mktemp -d` ; \
30 mkdir -p $${tmp_dir}/{BUILD,RPMS,SRPMS,SPECS,SOURCES} ; \
31 cp $(TARBALL) $${tmp_dir}/SOURCES ; \
32 cp $(SPEC) $${tmp_dir}/SPECS ; \
33 cd $${tmp_dir} > /dev/null 2>&1; \
34 rpmbuild -ba --define "_topdir $${tmp_dir}" \
35 --define "_source_filedigest_algorithm 0" \
36 --define "_binary_filedigest_algorithm 0" \
37 --define "dist %{nil}" \
38 SPECS/$(SPEC) ; \
39 cd - > /dev/null 2>&1; \
40 cp $${tmp_dir}/RPMS/noarch/* $${tmp_dir}/SRPMS/* . ; \
41 rm -rf $${tmp_dir} ; \
42 rpmlint *.rpm *.spec
+50
-1
NEWS less more
0 s3cmd 1.1.0 - ???
0 s3cmd 1.5.0-beta1 - 2013-12-02
1 =================
2 * Brougt to you by Matt Domsch and contributors, thanks guys! :)
3 * Multipart upload improvements (Eugene Brevdo, UENISHI Kota)
4 * Allow --acl-grant on AWS groups (Dale Lovelace)
5 * Added Server-Side Encryption support (Kevin Daub)
6 * Improved MIME types detections and content encoding (radomir,
7 Eric Drechsel, George Melika)
8 * Various smaller changes and bugfixes from many contributors
9
10 s3cmd 1.5.0-alpha3 - 2013-03-11
11 ==================
12 * Persistent HTTP/HTTPS connections for massive speedup (Michal Ludvig)
13 * New switch --quiet for suppressing all output (Siddarth Prakash)
14 * Honour "umask" on file downloads (Jason Dalton)
15 * Various bugfixes from many contributors
16
17 s3cmd 1.5.0-alpha2 - 2013-03-04
18 ==================
19 * IAM roles support (David Kohen, Eric Dowd)
20 * Manage bucket policies (Kota Uenishi)
21 * Various bugfixes from many contributors
22
23 s3cmd 1.5.0-alpha1 - 2013-02-19
24 ==================
25 * Server-side copy for hardlinks/softlinks to improve performance
26 (Matt Domsch)
27 * New [signurl] command (Craig Ringer)
28 * Improved symlink-loop detection (Michal Ludvig)
29 * Add --delete-after option for sync (Matt Domsch)
30 * Handle empty return bodies when processing S3 errors.
31 (Kelly McLaughlin)
32 * Upload from STDIN (Eric Connell)
33 * Updated bucket locations (Stefhen Hovland)
34 * Support custom HTTP headers (Brendan O'Connor, Karl Matthias)
35 * Improved MIME support (Karsten Sperling, Christopher Noyes)
36 * Added support for --acl-grant/--acl-revoke to 'sync' command
37 (Michael Tyson)
38 * CloudFront: Support default index and default root invalidation
39 (Josep del Rio)
40 * Command line options for access/secret keys (Matt Sweeney)
41 * Support [setpolicy] for setting bucket policies (Joe Fiorini)
42 * Respect the $TZ environment variable (James Brown)
43 * Reduce memory consumption for [s3cmd du] (Charlie Schluting)
44 * Rate limit progress updates (Steven Noonan)
45 * Download from S3 to a temp file first (Sumit Kumar)
46 * Reuse a single connection when doing a bucket list (Kelly McLaughlin)
47 * Delete empty files if object_get() failed (Oren Held)
48
49 s3cmd 1.1.0 - (never released)
150 ===========
251 * MultiPart upload enabled for both [put] and [sync]. Default chunk
352 size is 15MB.
+0
-22
PKG-INFO less more
0 Metadata-Version: 1.0
1 Name: s3cmd
2 Version: 1.1.0-beta3
3 Summary: Command line tool for managing Amazon S3 and CloudFront services
4 Home-page: http://s3tools.org
5 Author: Michal Ludvig
6 Author-email: michal@logix.cz
7 License: GPL version 2
8 Description:
9
10 S3cmd lets you copy files from/to Amazon S3
11 (Simple Storage Service) using a simple to use
12 command line client. Supports rsync-like backup,
13 GPG encryption, and more. Also supports management
14 of Amazon's CloudFront content delivery network.
15
16
17 Authors:
18 --------
19 Michal Ludvig <michal@logix.cz>
20
21 Platform: UNKNOWN
+4
-1
README less more
179179 case sensitive and must be entered accurately or you'll
180180 keep getting errors about invalid signatures or similar.
181181
182 Remember to add ListAllMyBuckets permissions to the keys
183 or you will get an AccessDenied error while testing access.
184
182185 3) Run "s3cmd ls" to list all your buckets.
183186 As you just started using S3 there are no buckets owned by
184187 you as of now. So the output will be empty.
234237
235238 Use --recursive (or -r) to list all the remote files:
236239
237 ~$ s3cmd ls s3://public.s3tools.org
240 ~$ s3cmd ls --recursive s3://public.s3tools.org
238241 2009-02-10 05:10 123456 s3://public.s3tools.org/somefile.xml
239242 2009-02-10 05:13 18 s3://public.s3tools.org/somewhere/dir1/file1-1.txt
240243 2009-02-10 05:13 8 s3://public.s3tools.org/somewhere/dir1/file1-2.txt
+9
-5
S3/ACL.py less more
145145 if self.hasGrant(name, permission):
146146 return
147147
148 name = name.lower()
149148 permission = permission.upper()
150149
151150 if "ALL" == permission:
158157 grantee.name = name
159158 grantee.permission = permission
160159
161 if name.find('@') <= -1: # ultra lame attempt to differenciate emails id from canonical ids
160 if name.find('@') > -1:
161 grantee.name = grantee.name.lower
162 grantee.xsi_type = "AmazonCustomerByEmail"
163 grantee.tag = "EmailAddress"
164 elif name.find('http://acs.amazonaws.com/groups/') > -1:
165 grantee.xsi_type = "Group"
166 grantee.tag = "URI"
167 else:
168 grantee.name = grantee.name.lower
162169 grantee.xsi_type = "CanonicalUser"
163170 grantee.tag = "ID"
164 else:
165 grantee.xsi_type = "AmazonCustomerByEmail"
166 grantee.tag = "EmailAddress"
167171
168172 self.appendGrantee(grantee)
169173
+41
-12
S3/CloudFront.py less more
132132 ## </Logging>
133133 ## </DistributionConfig>
134134
135 EMPTY_CONFIG = "<DistributionConfig><Origin/><CallerReference/><Enabled>true</Enabled></DistributionConfig>"
135 EMPTY_CONFIG = "<DistributionConfig><S3Origin><DNSName/></S3Origin><CallerReference/><Enabled>true</Enabled></DistributionConfig>"
136136 xmlns = "http://cloudfront.amazonaws.com/doc/%(api_ver)s/" % { 'api_ver' : cloudfront_api_version }
137137 def __init__(self, xml = None, tree = None):
138138 if xml is None:
173173 tree.attrib['xmlns'] = DistributionConfig.xmlns
174174
175175 ## Retain the order of the following calls!
176 appendXmlTextNode("Origin", self.info['Origin'], tree)
176 s3org = appendXmlTextNode("S3Origin", '', tree)
177 appendXmlTextNode("DNSName", self.info['S3Origin']['DNSName'], s3org)
177178 appendXmlTextNode("CallerReference", self.info['CallerReference'], tree)
178179 for cname in self.info['CNAME']:
179180 appendXmlTextNode("CNAME", cname.lower(), tree)
280281 tree = ET.Element("InvalidationBatch")
281282
282283 for path in self.paths:
283 if path[0] != "/":
284 if len(path) < 1 or path[0] != "/":
284285 path = "/" + path
285286 appendXmlTextNode("Path", path, tree)
286287 appendXmlTextNode("CallerReference", self.reference, tree)
321322 def CreateDistribution(self, uri, cnames_add = [], comment = None, logging = None, default_root_object = None):
322323 dist_config = DistributionConfig()
323324 dist_config.info['Enabled'] = True
324 dist_config.info['Origin'] = uri.host_name()
325 dist_config.info['S3Origin']['DNSName'] = uri.host_name()
325326 dist_config.info['CallerReference'] = str(uri)
326327 dist_config.info['DefaultRootObject'] = default_root_object
327328 if comment == None:
422423 body = request_body, headers = headers)
423424 return response
424425
425 def InvalidateObjects(self, uri, paths):
426 def InvalidateObjects(self, uri, paths, default_index_file, invalidate_default_index_on_cf, invalidate_default_index_root_on_cf):
427 # joseprio: if the user doesn't want to invalidate the default index
428 # path, or if the user wants to invalidate the root of the default
429 # index, we need to process those paths
430 if default_index_file is not None and (not invalidate_default_index_on_cf or invalidate_default_index_root_on_cf):
431 new_paths = []
432 default_index_suffix = '/' + default_index_file
433 for path in paths:
434 if path.endswith(default_index_suffix) or path == default_index_file:
435 if invalidate_default_index_on_cf:
436 new_paths.append(path)
437 if invalidate_default_index_root_on_cf:
438 new_paths.append(path[:-len(default_index_file)])
439 else:
440 new_paths.append(path)
441 paths = new_paths
442
426443 # uri could be either cf:// or s3:// uri
427444 cfuri = self.get_dist_name_for_bucket(uri)
428445 if len(paths) > 999:
492509 warning(unicode(e))
493510 warning("Waiting %d sec..." % self._fail_wait(retries))
494511 time.sleep(self._fail_wait(retries))
495 return self.send_request(op_name, dist_id, body, retries - 1)
512 return self.send_request(op_name, dist_id, body, retries = retries - 1)
496513 else:
497514 raise e
498515
515532
516533 if not headers.has_key("x-amz-date"):
517534 headers["x-amz-date"] = time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())
535
536 if len(self.config.access_token)>0:
537 self.config.role_refresh()
538 headers['x-amz-security-token']=self.config.access_token
518539
519540 signature = self.sign_request(headers)
520541 headers["Authorization"] = "AWS "+self.config.access_key+":"+signature
545566 if (uri.type == "cf"):
546567 return uri
547568 if (uri.type != "s3"):
548 raise ParameterError("CloudFront or S3 URI required instead of: %s" % arg)
569 raise ParameterError("CloudFront or S3 URI required instead of: %s" % uri)
549570
550571 debug("_get_dist_name_for_bucket(%r)" % uri)
551572 if CloudFront.dist_list is None:
554575 for d in response['dist_list'].dist_summs:
555576 if d.info.has_key("S3Origin"):
556577 CloudFront.dist_list[getBucketFromHostname(d.info['S3Origin']['DNSName'])[0]] = d.uri()
578 elif d.info.has_key("CustomOrigin"):
579 # Aral: This used to skip over distributions with CustomOrigin, however, we mustn't
580 # do this since S3 buckets that are set up as websites use custom origins.
581 # Thankfully, the custom origin URLs they use start with the URL of the
582 # S3 bucket. Here, we make use this naming convention to support this use case.
583 distListIndex = getBucketFromHostname(d.info['CustomOrigin']['DNSName'])[0];
584 distListIndex = distListIndex[:len(uri.bucket())]
585 CloudFront.dist_list[distListIndex] = d.uri()
557586 else:
558 # Skip over distributions with CustomOrigin
587 # Aral: I'm not sure when this condition will be reached, but keeping it in there.
559588 continue
560589 debug("dist_list: %s" % CloudFront.dist_list)
561590 try:
562591 return CloudFront.dist_list[uri.bucket()]
563592 except Exception, e:
564593 debug(e)
565 raise ParameterError("Unable to translate S3 URI to CloudFront distribution name: %s" % arg)
594 raise ParameterError("Unable to translate S3 URI to CloudFront distribution name: %s" % uri)
566595
567596 class Cmd(object):
568597 """
645674 for arg in args:
646675 uri = S3Uri(arg)
647676 if uri.type != "s3":
648 raise ParameterError("Bucket can only be created from a s3:// URI instead of: %s" % arg)
677 raise ParameterError("Distribution can only be created from a s3:// URI instead of: %s" % arg)
649678 if uri.object():
650679 raise ParameterError("Use s3:// URI with a bucket name only instead of: %s" % arg)
651680 if not uri.is_dns_compatible():
662691 d = response['distribution']
663692 dc = d.info['DistributionConfig']
664693 output("Distribution created:")
665 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['Origin']))
694 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['S3Origin']['DNSName']))
666695 pretty_output("DistId", d.uri())
667696 pretty_output("DomainName", d.info['DomainName'])
668697 pretty_output("CNAMEs", ", ".join(dc.info['CNAME']))
704733 response = cf.GetDistInfo(cfuri)
705734 d = response['distribution']
706735 dc = d.info['DistributionConfig']
707 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['Origin']))
736 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['S3Origin']['DNSName']))
708737 pretty_output("DistId", d.uri())
709738 pretty_output("DomainName", d.info['DomainName'])
710739 pretty_output("Status", d.info['Status'])
+105
-2
S3/Config.py less more
66 from logging import debug, info, warning, error
77 import re
88 import os
9 import sys
910 import Progress
1011 from SortedDict import SortedDict
12 import httplib
13 try:
14 import json
15 except ImportError, e:
16 pass
1117
1218 class Config(object):
1319 _instance = None
1521 _doc = {}
1622 access_key = ""
1723 secret_key = ""
24 access_token = ""
1825 host_base = "s3.amazonaws.com"
1926 host_bucket = "%(bucket)s.s3.amazonaws.com"
2027 simpledb_host = "sdb.amazonaws.com"
2835 human_readable_sizes = False
2936 extra_headers = SortedDict(ignore_case = True)
3037 force = False
38 server_side_encryption = False
3139 enable = None
3240 get_continue = False
41 put_continue = False
42 upload_id = None
3343 skip_existing = False
3444 recursive = False
45 restore_days = 1
3546 acl_public = None
3647 acl_grants = []
3748 acl_revokes = []
3950 proxy_port = 3128
4051 encrypt = False
4152 dry_run = False
53 add_encoding_exts = ""
4254 preserve_attrs = True
4355 preserve_attrs_list = [
4456 'uname', # Verbose owner Name (e.g. 'root')
4961 'mtime', # Modification timestamp
5062 'ctime', # Creation timestamp
5163 'mode', # File mode (e.g. rwxr-xr-x = 755)
64 'md5', # File MD5 (if known)
5265 #'acl', # Full ACL (not yet supported)
5366 ]
5467 delete_removed = False
68 delete_after = False
69 delete_after_fetch = False
70 max_delete = -1
5571 _doc['delete_removed'] = "[sync] Remove remote S3 objects when local file has been deleted"
72 delay_updates = False
5673 gpg_passphrase = ""
5774 gpg_command = ""
5875 gpg_encrypt = "%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s"
6178 bucket_location = "US"
6279 default_mime_type = "binary/octet-stream"
6380 guess_mime_type = True
81 use_mime_magic = True
6482 mime_type = ""
6583 enable_multipart = True
6684 multipart_chunk_size_mb = 15 # MB
7391 debug_exclude = {}
7492 debug_include = {}
7593 encoding = "utf-8"
94 add_content_encoding = True
7695 urlencoding_mode = "normal"
7796 log_target_prefix = ""
7897 reduced_redundancy = False
7998 follow_symlinks = False
8099 socket_timeout = 300
81100 invalidate_on_cf = False
101 # joseprio: new flags for default index invalidation
102 invalidate_default_index_on_cf = False
103 invalidate_default_index_root_on_cf = True
82104 website_index = "index.html"
83105 website_error = ""
84106 website_endpoint = "http://%(bucket)s.s3-website-%(location)s.amazonaws.com/"
107 additional_destinations = []
108 files_from = []
109 cache_file = ""
110 add_headers = ""
111 ignore_failed_copy = False
85112
86113 ## Creating a singleton
87114 def __new__(self, configfile = None):
91118
92119 def __init__(self, configfile = None):
93120 if configfile:
94 self.read_config_file(configfile)
121 try:
122 self.read_config_file(configfile)
123 except IOError, e:
124 if 'AWS_CREDENTIAL_FILE' in os.environ:
125 self.env_config()
126 if len(self.access_key)==0:
127 self.role_config()
128
129 def role_config(self):
130 if sys.version_info[0] * 10 + sys.version_info[1] < 26:
131 error("IAM authentication requires Python 2.6 or newer")
132 raise
133 if not 'json' in sys.modules:
134 error("IAM authentication not available -- missing module json")
135 raise
136 try:
137 conn = httplib.HTTPConnection(host='169.254.169.254', timeout = 2)
138 conn.request('GET', "/latest/meta-data/iam/security-credentials/")
139 resp = conn.getresponse()
140 files = resp.read()
141 if resp.status == 200 and len(files)>1:
142 conn.request('GET', "/latest/meta-data/iam/security-credentials/%s"%files)
143 resp=conn.getresponse()
144 if resp.status == 200:
145 creds=json.load(resp)
146 Config().update_option('access_key', creds['AccessKeyId'].encode('ascii'))
147 Config().update_option('secret_key', creds['SecretAccessKey'].encode('ascii'))
148 Config().update_option('access_token', creds['Token'].encode('ascii'))
149 else:
150 raise IOError
151 else:
152 raise IOError
153 except:
154 raise
155
156 def role_refresh(self):
157 try:
158 self.role_config()
159 except:
160 warning("Could not refresh role")
161
162 def env_config(self):
163 cred_content = ""
164 try:
165 cred_file = open(os.environ['AWS_CREDENTIAL_FILE'],'r')
166 cred_content = cred_file.read()
167 except IOError, e:
168 debug("Error %d accessing credentials file %s" % (e.errno,os.environ['AWS_CREDENTIAL_FILE']))
169 r_data = re.compile("^\s*(?P<orig_key>\w+)\s*=\s*(?P<value>.*)")
170 r_quotes = re.compile("^\"(.*)\"\s*$")
171 if len(cred_content)>0:
172 for line in cred_content.splitlines():
173 is_data = r_data.match(line)
174 is_data = r_data.match(line)
175 if is_data:
176 data = is_data.groupdict()
177 if r_quotes.match(data["value"]):
178 data["value"] = data["value"][1:-1]
179 if data["orig_key"]=="AWSAccessKeyId":
180 data["key"] = "access_key"
181 elif data["orig_key"]=="AWSSecretKey":
182 data["key"] = "secret_key"
183 else:
184 del data["key"]
185 if "key" in data:
186 Config().update_option(data["key"], data["value"])
187 if data["key"] in ("access_key", "secret_key", "gpg_passphrase"):
188 print_value = ("%s...%d_chars...%s") % (data["value"][:2], len(data["value"]) - 3, data["value"][-1:])
189 else:
190 print_value = data["value"]
191 debug("env_Config: %s->%s" % (data["key"], print_value))
95192
96193 def option_list(self):
97194 retval = []
111208 cp = ConfigParser(configfile)
112209 for option in self.option_list():
113210 self.update_option(option, cp.get(option))
211
212 if cp.get('add_headers'):
213 for option in cp.get('add_headers').split(","):
214 (key, value) = option.split(':')
215 self.extra_headers[key.replace('_', '-').strip()] = value.strip()
216
114217 self._parsed_files.append(configfile)
115218
116219 def dump_config(self, stream):
176279 data["value"] = data["value"][1:-1]
177280 self.__setitem__(data["key"], data["value"])
178281 if data["key"] in ("access_key", "secret_key", "gpg_passphrase"):
179 print_value = (data["value"][:2]+"...%d_chars..."+data["value"][-1:]) % (len(data["value"]) - 3)
282 print_value = ("%s...%d_chars...%s") % (data["value"][:2], len(data["value"]) - 3, data["value"][-1:])
180283 else:
181284 print_value = data["value"]
182285 debug("ConfigParser: %s->%s" % (data["key"], print_value))
+71
-0
S3/ConnMan.py less more
0 import httplib
1 from urlparse import urlparse
2 from threading import Semaphore
3 from logging import debug, info, warning, error
4
5 from Config import Config
6 from Exceptions import ParameterError
7
8 __all__ = [ "ConnMan" ]
9
10 class http_connection(object):
11 def __init__(self, id, hostname, ssl, cfg):
12 self.hostname = hostname
13 self.ssl = ssl
14 self.id = id
15 self.counter = 0
16 if cfg.proxy_host != "":
17 self.c = httplib.HTTPConnection(cfg.proxy_host, cfg.proxy_port)
18 elif not ssl:
19 self.c = httplib.HTTPConnection(hostname)
20 else:
21 self.c = httplib.HTTPSConnection(hostname)
22
23 class ConnMan(object):
24 conn_pool_sem = Semaphore()
25 conn_pool = {}
26 conn_max_counter = 800 ## AWS closes connection after some ~90 requests
27
28 @staticmethod
29 def get(hostname, ssl = None):
30 cfg = Config()
31 if ssl == None:
32 ssl = cfg.use_https
33 conn = None
34 if cfg.proxy_host != "":
35 if ssl:
36 raise ParameterError("use_ssl=True can't be used with proxy")
37 conn_id = "proxy://%s:%s" % (cfg.proxy_host, cfg.proxy_port)
38 else:
39 conn_id = "http%s://%s" % (ssl and "s" or "", hostname)
40 ConnMan.conn_pool_sem.acquire()
41 if not ConnMan.conn_pool.has_key(conn_id):
42 ConnMan.conn_pool[conn_id] = []
43 if len(ConnMan.conn_pool[conn_id]):
44 conn = ConnMan.conn_pool[conn_id].pop()
45 debug("ConnMan.get(): re-using connection: %s#%d" % (conn.id, conn.counter))
46 ConnMan.conn_pool_sem.release()
47 if not conn:
48 debug("ConnMan.get(): creating new connection: %s" % conn_id)
49 conn = http_connection(conn_id, hostname, ssl, cfg)
50 conn.c.connect()
51 conn.counter += 1
52 return conn
53
54 @staticmethod
55 def put(conn):
56 if conn.id.startswith("proxy://"):
57 conn.c.close()
58 debug("ConnMan.put(): closing proxy connection (keep-alive not yet supported)")
59 return
60
61 if conn.counter >= ConnMan.conn_max_counter:
62 conn.c.close()
63 debug("ConnMan.put(): closing over-used connection")
64 return
65
66 ConnMan.conn_pool_sem.acquire()
67 ConnMan.conn_pool[conn.id].append(conn)
68 ConnMan.conn_pool_sem.release()
69 debug("ConnMan.put(): connection put back to pool (%s#%d)" % (conn.id, conn.counter))
70
+1
-1
S3/Exceptions.py less more
4343 if response.has_key("headers"):
4444 for header in response["headers"]:
4545 debug("HttpHeader: %s: %s" % (header, response["headers"][header]))
46 if response.has_key("data"):
46 if response.has_key("data") and response["data"]:
4747 tree = getTreeFromXml(response["data"])
4848 error_node = tree
4949 if not error_node.tag == "Error":
+54
-0
S3/FileDict.py less more
0 ## Amazon S3 manager
1 ## Author: Michal Ludvig <michal@logix.cz>
2 ## http://www.logix.cz/michal
3 ## License: GPL Version 2
4
5 from SortedDict import SortedDict
6 import Utils
7
8 class FileDict(SortedDict):
9 def __init__(self, mapping = {}, ignore_case = True, **kwargs):
10 SortedDict.__init__(self, mapping = mapping, ignore_case = ignore_case, **kwargs)
11 self.hardlinks = dict() # { dev: { inode : {'md5':, 'relative_files':}}}
12 self.by_md5 = dict() # {md5: set(relative_files)}
13
14 def record_md5(self, relative_file, md5):
15 if md5 not in self.by_md5:
16 self.by_md5[md5] = set()
17 self.by_md5[md5].add(relative_file)
18
19 def find_md5_one(self, md5):
20 try:
21 return list(self.by_md5.get(md5, set()))[0]
22 except:
23 return None
24
25 def get_md5(self, relative_file):
26 """returns md5 if it can, or raises IOError if file is unreadable"""
27 md5 = None
28 if 'md5' in self[relative_file]:
29 return self[relative_file]['md5']
30 md5 = self.get_hardlink_md5(relative_file)
31 if md5 is None:
32 md5 = Utils.hash_file_md5(self[relative_file]['full_name'])
33 self.record_md5(relative_file, md5)
34 self[relative_file]['md5'] = md5
35 return md5
36
37 def record_hardlink(self, relative_file, dev, inode, md5):
38 if dev == 0 or inode == 0: return # Windows
39 if dev not in self.hardlinks:
40 self.hardlinks[dev] = dict()
41 if inode not in self.hardlinks[dev]:
42 self.hardlinks[dev][inode] = dict(md5=md5, relative_files=set())
43 self.hardlinks[dev][inode]['relative_files'].add(relative_file)
44
45 def get_hardlink_md5(self, relative_file):
46 md5 = None
47 dev = self[relative_file]['dev']
48 inode = self[relative_file]['inode']
49 try:
50 md5 = self.hardlinks[dev][inode]['md5']
51 except:
52 pass
53 return md5
+358
-122
S3/FileLists.py less more
55 from S3 import S3
66 from Config import Config
77 from S3Uri import S3Uri
8 from SortedDict import SortedDict
8 from FileDict import FileDict
99 from Utils import *
1010 from Exceptions import ParameterError
11 from HashCache import HashCache
1112
1213 from logging import debug, info, warning, error
1314
1415 import os
16 import sys
1517 import glob
18 import copy
19 import re
20 import errno
1621
1722 __all__ = ["fetch_local_list", "fetch_remote_list", "compare_filelists", "filter_exclude_include"]
1823
1924 def _fswalk_follow_symlinks(path):
20 '''
21 Walk filesystem, following symbolic links (but without recursion), on python2.4 and later
22
23 If a recursive directory link is detected, emit a warning and skip.
24 '''
25 assert os.path.isdir(path) # only designed for directory argument
26 walkdirs = set([path])
27 targets = set()
28 for dirpath, dirnames, filenames in os.walk(path):
29 for dirname in dirnames:
30 current = os.path.join(dirpath, dirname)
31 target = os.path.realpath(current)
32 if os.path.islink(current):
33 if target in targets:
34 warning("Skipping recursively symlinked directory %s" % dirname)
35 else:
36 walkdirs.add(current)
37 targets.add(target)
38 for walkdir in walkdirs:
39 for value in os.walk(walkdir):
40 yield value
41
42 def _fswalk(path, follow_symlinks):
43 '''
44 Directory tree generator
45
46 path (str) is the root of the directory tree to walk
47
48 follow_symlinks (bool) indicates whether to descend into symbolically linked directories
49 '''
50 if follow_symlinks:
51 return _fswalk_follow_symlinks(path)
52 return os.walk(path)
25 '''
26 Walk filesystem, following symbolic links (but without recursion), on python2.4 and later
27
28 If a symlink directory loop is detected, emit a warning and skip.
29 E.g.: dir1/dir2/sym-dir -> ../dir2
30 '''
31 assert os.path.isdir(path) # only designed for directory argument
32 walkdirs = set([path])
33 for dirpath, dirnames, filenames in os.walk(path):
34 handle_exclude_include_walk(dirpath, dirnames, [])
35 real_dirpath = os.path.realpath(dirpath)
36 for dirname in dirnames:
37 current = os.path.join(dirpath, dirname)
38 real_current = os.path.realpath(current)
39 if os.path.islink(current):
40 if (real_dirpath == real_current or
41 real_dirpath.startswith(real_current + os.path.sep)):
42 warning("Skipping recursively symlinked directory %s" % dirname)
43 else:
44 walkdirs.add(current)
45 for walkdir in walkdirs:
46 for dirpath, dirnames, filenames in os.walk(walkdir):
47 handle_exclude_include_walk(dirpath, dirnames, [])
48 yield (dirpath, dirnames, filenames)
49
50 def _fswalk_no_symlinks(path):
51 '''
52 Directory tree generator
53
54 path (str) is the root of the directory tree to walk
55 '''
56 for dirpath, dirnames, filenames in os.walk(path):
57 handle_exclude_include_walk(dirpath, dirnames, filenames)
58 yield (dirpath, dirnames, filenames)
5359
5460 def filter_exclude_include(src_list):
5561 info(u"Applying --exclude/--include")
5662 cfg = Config()
57 exclude_list = SortedDict(ignore_case = False)
63 exclude_list = FileDict(ignore_case = False)
5864 for file in src_list.keys():
5965 debug(u"CHECK: %s" % file)
6066 excluded = False
7783 del(src_list[file])
7884 continue
7985 else:
80 debug(u"PASS: %s" % (file))
86 debug(u"PASS: %r" % (file))
8187 return src_list, exclude_list
8288
83 def fetch_local_list(args, recursive = None):
84 def _get_filelist_local(local_uri):
89 def handle_exclude_include_walk(root, dirs, files):
90 cfg = Config()
91 copydirs = copy.copy(dirs)
92 copyfiles = copy.copy(files)
93
94 # exclude dir matches in the current directory
95 # this prevents us from recursing down trees we know we want to ignore
96 for x in copydirs:
97 d = os.path.join(root, x, '')
98 debug(u"CHECK: %r" % d)
99 excluded = False
100 for r in cfg.exclude:
101 if r.search(d):
102 excluded = True
103 debug(u"EXCL-MATCH: '%s'" % (cfg.debug_exclude[r]))
104 break
105 if excluded:
106 ## No need to check for --include if not excluded
107 for r in cfg.include:
108 if r.search(d):
109 excluded = False
110 debug(u"INCL-MATCH: '%s'" % (cfg.debug_include[r]))
111 break
112 if excluded:
113 ## Still excluded - ok, action it
114 debug(u"EXCLUDE: %r" % d)
115 dirs.remove(x)
116 continue
117 else:
118 debug(u"PASS: %r" % (d))
119
120 # exclude file matches in the current directory
121 for x in copyfiles:
122 file = os.path.join(root, x)
123 debug(u"CHECK: %r" % file)
124 excluded = False
125 for r in cfg.exclude:
126 if r.search(file):
127 excluded = True
128 debug(u"EXCL-MATCH: '%s'" % (cfg.debug_exclude[r]))
129 break
130 if excluded:
131 ## No need to check for --include if not excluded
132 for r in cfg.include:
133 if r.search(file):
134 excluded = False
135 debug(u"INCL-MATCH: '%s'" % (cfg.debug_include[r]))
136 break
137 if excluded:
138 ## Still excluded - ok, action it
139 debug(u"EXCLUDE: %s" % file)
140 files.remove(x)
141 continue
142 else:
143 debug(u"PASS: %r" % (file))
144
145
146 def _get_filelist_from_file(cfg, local_path):
147 def _append(d, key, value):
148 if key not in d:
149 d[key] = [value]
150 else:
151 d[key].append(value)
152
153 filelist = {}
154 for fname in cfg.files_from:
155 if fname == u'-':
156 f = sys.stdin
157 else:
158 try:
159 f = open(fname, 'r')
160 except IOError, e:
161 warning(u"--files-from input file %s could not be opened for reading (%s), skipping." % (fname, e.strerror))
162 continue
163
164 for line in f:
165 line = line.strip()
166 line = os.path.normpath(os.path.join(local_path, line))
167 dirname = os.path.dirname(line)
168 basename = os.path.basename(line)
169 _append(filelist, dirname, basename)
170 if f != sys.stdin:
171 f.close()
172
173 # reformat to match os.walk()
174 result = []
175 keys = filelist.keys()
176 keys.sort()
177 for key in keys:
178 values = filelist[key]
179 values.sort()
180 result.append((key, [], values))
181 return result
182
183 def fetch_local_list(args, is_src = False, recursive = None):
184 def _get_filelist_local(loc_list, local_uri, cache):
85185 info(u"Compiling list of local files...")
186
187 if deunicodise(local_uri.basename()) == "-":
188 try:
189 uid = os.geteuid()
190 gid = os.getegid()
191 except:
192 uid = 0
193 gid = 0
194 loc_list["-"] = {
195 'full_name_unicode' : '-',
196 'full_name' : '-',
197 'size' : -1,
198 'mtime' : -1,
199 'uid' : uid,
200 'gid' : gid,
201 'dev' : 0,
202 'inode': 0,
203 }
204 return loc_list, True
86205 if local_uri.isdir():
87206 local_base = deunicodise(local_uri.basename())
88207 local_path = deunicodise(local_uri.path())
89 filelist = _fswalk(local_path, cfg.follow_symlinks)
90 single_file = False
208 if is_src and len(cfg.files_from):
209 filelist = _get_filelist_from_file(cfg, local_path)
210 single_file = False
211 else:
212 if cfg.follow_symlinks:
213 filelist = _fswalk_follow_symlinks(local_path)
214 else:
215 filelist = _fswalk_no_symlinks(local_path)
216 single_file = False
91217 else:
92218 local_base = ""
93219 local_path = deunicodise(local_uri.dirname())
94220 filelist = [( local_path, [], [deunicodise(local_uri.basename())] )]
95221 single_file = True
96 loc_list = SortedDict(ignore_case = False)
97222 for root, dirs, files in filelist:
98223 rel_root = root.replace(local_path, local_base, 1)
99224 for f in files:
111236 relative_file = replace_nonprintables(relative_file)
112237 if relative_file.startswith('./'):
113238 relative_file = relative_file[2:]
114 sr = os.stat_result(os.lstat(full_name))
239 try:
240 sr = os.stat_result(os.stat(full_name))
241 except OSError, e:
242 if e.errno == errno.ENOENT:
243 # file was removed async to us getting the list
244 continue
245 else:
246 raise
115247 loc_list[relative_file] = {
116248 'full_name_unicode' : unicodise(full_name),
117249 'full_name' : full_name,
118250 'size' : sr.st_size,
119251 'mtime' : sr.st_mtime,
252 'dev' : sr.st_dev,
253 'inode' : sr.st_ino,
254 'uid' : sr.st_uid,
255 'gid' : sr.st_gid,
256 'sr': sr # save it all, may need it in preserve_attrs_list
120257 ## TODO: Possibly more to save here...
121258 }
259 if 'md5' in cfg.sync_checks:
260 md5 = cache.md5(sr.st_dev, sr.st_ino, sr.st_mtime, sr.st_size)
261 if md5 is None:
262 try:
263 md5 = loc_list.get_md5(relative_file) # this does the file I/O
264 except IOError:
265 continue
266 cache.add(sr.st_dev, sr.st_ino, sr.st_mtime, sr.st_size, md5)
267 loc_list.record_hardlink(relative_file, sr.st_dev, sr.st_ino, md5)
122268 return loc_list, single_file
123269
270 def _maintain_cache(cache, local_list):
271 # if getting the file list from files_from, it is going to be
272 # a subset of the actual tree. We should not purge content
273 # outside of that subset as we don't know if it's valid or
274 # not. Leave it to a non-files_from run to purge.
275 if cfg.cache_file and len(cfg.files_from) == 0:
276 cache.mark_all_for_purge()
277 for i in local_list.keys():
278 cache.unmark_for_purge(local_list[i]['dev'], local_list[i]['inode'], local_list[i]['mtime'], local_list[i]['size'])
279 cache.purge()
280 cache.save(cfg.cache_file)
281
124282 cfg = Config()
283
284 cache = HashCache()
285 if cfg.cache_file:
286 try:
287 cache.load(cfg.cache_file)
288 except IOError:
289 info(u"No cache file found, creating it.")
290
125291 local_uris = []
126 local_list = SortedDict(ignore_case = False)
292 local_list = FileDict(ignore_case = False)
127293 single_file = False
128294
129295 if type(args) not in (list, tuple):
141307 local_uris.append(uri)
142308
143309 for uri in local_uris:
144 list_for_uri, single_file = _get_filelist_local(uri)
145 local_list.update(list_for_uri)
310 list_for_uri, single_file = _get_filelist_local(local_list, uri, cache)
146311
147312 ## Single file is True if and only if the user
148313 ## specified one local URI and that URI represents
152317 if len(local_list) > 1:
153318 single_file = False
154319
320 _maintain_cache(cache, local_list)
321
155322 return local_list, single_file
156323
157324 def fetch_remote_list(args, require_attribs = False, recursive = None):
325 def _get_remote_attribs(uri, remote_item):
326 response = S3(cfg).object_info(uri)
327 remote_item.update({
328 'size': int(response['headers']['content-length']),
329 'md5': response['headers']['etag'].strip('"\''),
330 'timestamp' : dateRFC822toUnix(response['headers']['date'])
331 })
332 try:
333 md5 = response['s3cmd-attrs']['md5']
334 remote_item.update({'md5': md5})
335 debug(u"retreived md5=%s from headers" % md5)
336 except KeyError:
337 pass
338
158339 def _get_filelist_remote(remote_uri, recursive = True):
159340 ## If remote_uri ends with '/' then all remote files will have
160341 ## the remote_uri prefix removed in the relative path.
182363 rem_base = rem_base[:rem_base.rfind('/')+1]
183364 remote_uri = S3Uri("s3://%s/%s" % (remote_uri.bucket(), rem_base))
184365 rem_base_len = len(rem_base)
185 rem_list = SortedDict(ignore_case = False)
366 rem_list = FileDict(ignore_case = False)
186367 break_now = False
187368 for object in response['list']:
188 if object['Key'] == rem_base_original and object['Key'][-1] != os.path.sep:
369 if object['Key'] == rem_base_original and object['Key'][-1] != "/":
189370 ## We asked for one file and we got that file :-)
190371 key = os.path.basename(object['Key'])
191372 object_uri_str = remote_uri_original.uri()
192373 break_now = True
193 rem_list = {} ## Remove whatever has already been put to rem_list
374 rem_list = FileDict(ignore_case = False) ## Remove whatever has already been put to rem_list
194375 else:
195376 key = object['Key'][rem_base_len:] ## Beware - this may be '' if object['Key']==rem_base !!
196377 object_uri_str = remote_uri.uri() + key
201382 'object_key' : object['Key'],
202383 'object_uri_str' : object_uri_str,
203384 'base_uri' : remote_uri,
385 'dev' : None,
386 'inode' : None,
204387 }
388 if rem_list[key]['md5'].find("-") > 0: # always get it for multipart uploads
389 _get_remote_attribs(S3Uri(object_uri_str), rem_list[key])
390 md5 = rem_list[key]['md5']
391 rem_list.record_md5(key, md5)
205392 if break_now:
206393 break
207394 return rem_list
208395
209396 cfg = Config()
210397 remote_uris = []
211 remote_list = SortedDict(ignore_case = False)
398 remote_list = FileDict(ignore_case = False)
212399
213400 if type(args) not in (list, tuple):
214401 args = [args]
227414 objectlist = _get_filelist_remote(uri)
228415 for key in objectlist:
229416 remote_list[key] = objectlist[key]
417 remote_list.record_md5(key, objectlist.get_md5(key))
230418 else:
231419 for uri in remote_uris:
232420 uri_str = str(uri)
233421 ## Wildcards used in remote URI?
234422 ## If yes we'll need a bucket listing...
235 if uri_str.find('*') > -1 or uri_str.find('?') > -1:
236 first_wildcard = uri_str.find('*')
237 first_questionmark = uri_str.find('?')
238 if first_questionmark > -1 and first_questionmark < first_wildcard:
239 first_wildcard = first_questionmark
240 prefix = uri_str[:first_wildcard]
241 rest = uri_str[first_wildcard+1:]
423 wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
424 if len(wildcard_split_result) == 2: # wildcards found
425 prefix, rest = wildcard_split_result
242426 ## Only request recursive listing if the 'rest' of the URI,
243427 ## i.e. the part after first wildcard, contains '/'
244 need_recursion = rest.find('/') > -1
428 need_recursion = '/' in rest
245429 objectlist = _get_filelist_remote(S3Uri(prefix), recursive = need_recursion)
246430 for key in objectlist:
247431 ## Check whether the 'key' matches the requested wildcards
258442 'object_key': uri.object()
259443 }
260444 if require_attribs:
261 response = S3(cfg).object_info(uri)
262 remote_item.update({
263 'size': int(response['headers']['content-length']),
264 'md5': response['headers']['etag'].strip('"\''),
265 'timestamp' : dateRFC822toUnix(response['headers']['date'])
266 })
445 _get_remote_attribs(uri, remote_item)
446
267447 remote_list[key] = remote_item
448 md5 = remote_item.get('md5')
449 if md5:
450 remote_list.record_md5(key, md5)
268451 return remote_list
269452
270 def compare_filelists(src_list, dst_list, src_remote, dst_remote):
453
454 def compare_filelists(src_list, dst_list, src_remote, dst_remote, delay_updates = False):
271455 def __direction_str(is_remote):
272456 return is_remote and "remote" or "local"
273457
274 # We don't support local->local sync, use 'rsync' or something like that instead ;-)
458 def _compare(src_list, dst_lst, src_remote, dst_remote, file):
459 """Return True if src_list[file] matches dst_list[file], else False"""
460 attribs_match = True
461 if not (src_list.has_key(file) and dst_list.has_key(file)):
462 info(u"%s: does not exist in one side or the other: src_list=%s, dst_list=%s" % (file, src_list.has_key(file), dst_list.has_key(file)))
463 return False
464
465 ## check size first
466 if 'size' in cfg.sync_checks and dst_list[file]['size'] != src_list[file]['size']:
467 debug(u"xfer: %s (size mismatch: src=%s dst=%s)" % (file, src_list[file]['size'], dst_list[file]['size']))
468 attribs_match = False
469
470 ## check md5
471 compare_md5 = 'md5' in cfg.sync_checks
472 # Multipart-uploaded files don't have a valid md5 sum - it ends with "...-nn"
473 if compare_md5:
474 if (src_remote == True and src_list[file]['md5'].find("-") >= 0) or (dst_remote == True and dst_list[file]['md5'].find("-") >= 0):
475 compare_md5 = False
476 info(u"disabled md5 check for %s" % file)
477 if attribs_match and compare_md5:
478 try:
479 src_md5 = src_list.get_md5(file)
480 dst_md5 = dst_list.get_md5(file)
481 except (IOError,OSError), e:
482 # md5 sum verification failed - ignore that file altogether
483 debug(u"IGNR: %s (disappeared)" % (file))
484 warning(u"%s: file disappeared, ignoring." % (file))
485 raise
486
487 if src_md5 != dst_md5:
488 ## checksums are different.
489 attribs_match = False
490 debug(u"XFER: %s (md5 mismatch: src=%s dst=%s)" % (file, src_md5, dst_md5))
491
492 return attribs_match
493
494 # we don't support local->local sync, use 'rsync' or something like that instead ;-)
275495 assert(not(src_remote == False and dst_remote == False))
276496
277497 info(u"Verifying attributes...")
278498 cfg = Config()
279 exists_list = SortedDict(ignore_case = False)
499 ## Items left on src_list will be transferred
500 ## Items left on update_list will be transferred after src_list
501 ## Items left on copy_pairs will be copied from dst1 to dst2
502 update_list = FileDict(ignore_case = False)
503 ## Items left on dst_list will be deleted
504 copy_pairs = []
280505
281506 debug("Comparing filelists (direction: %s -> %s)" % (__direction_str(src_remote), __direction_str(dst_remote)))
282 debug("src_list.keys: %s" % src_list.keys())
283 debug("dst_list.keys: %s" % dst_list.keys())
284
285 for file in src_list.keys():
286 debug(u"CHECK: %s" % file)
287 if dst_list.has_key(file):
507
508 for relative_file in src_list.keys():
509 debug(u"CHECK: %s" % (relative_file))
510
511 if dst_list.has_key(relative_file):
288512 ## Was --skip-existing requested?
289513 if cfg.skip_existing:
290 debug(u"IGNR: %s (used --skip-existing)" % (file))
291 exists_list[file] = src_list[file]
292 del(src_list[file])
293 ## Remove from destination-list, all that is left there will be deleted
294 del(dst_list[file])
514 debug(u"IGNR: %s (used --skip-existing)" % (relative_file))
515 del(src_list[relative_file])
516 del(dst_list[relative_file])
295517 continue
296518
297 attribs_match = True
298 ## Check size first
299 if 'size' in cfg.sync_checks and dst_list[file]['size'] != src_list[file]['size']:
300 debug(u"XFER: %s (size mismatch: src=%s dst=%s)" % (file, src_list[file]['size'], dst_list[file]['size']))
301 attribs_match = False
302
303 ## Check MD5
304 compare_md5 = 'md5' in cfg.sync_checks
305 # Multipart-uploaded files don't have a valid MD5 sum - it ends with "...-NN"
306 if compare_md5 and (src_remote == True and src_list[file]['md5'].find("-") >= 0) or (dst_remote == True and dst_list[file]['md5'].find("-") >= 0):
307 compare_md5 = False
308 info(u"Disabled MD5 check for %s" % file)
309 if attribs_match and compare_md5:
519 try:
520 same_file = _compare(src_list, dst_list, src_remote, dst_remote, relative_file)
521 except (IOError,OSError), e:
522 debug(u"IGNR: %s (disappeared)" % (relative_file))
523 warning(u"%s: file disappeared, ignoring." % (relative_file))
524 del(src_list[relative_file])
525 del(dst_list[relative_file])
526 continue
527
528 if same_file:
529 debug(u"IGNR: %s (transfer not needed)" % relative_file)
530 del(src_list[relative_file])
531 del(dst_list[relative_file])
532
533 else:
534 # look for matching file in src
310535 try:
311 if src_remote == False and dst_remote == True:
312 src_md5 = hash_file_md5(src_list[file]['full_name'])
313 dst_md5 = dst_list[file]['md5']
314 elif src_remote == True and dst_remote == False:
315 src_md5 = src_list[file]['md5']
316 dst_md5 = hash_file_md5(dst_list[file]['full_name'])
317 elif src_remote == True and dst_remote == True:
318 src_md5 = src_list[file]['md5']
319 dst_md5 = dst_list[file]['md5']
320 except (IOError,OSError), e:
321 # MD5 sum verification failed - ignore that file altogether
322 debug(u"IGNR: %s (disappeared)" % (file))
323 warning(u"%s: file disappeared, ignoring." % (file))
324 del(src_list[file])
325 del(dst_list[file])
326 continue
327
328 if src_md5 != dst_md5:
329 ## Checksums are different.
330 attribs_match = False
331 debug(u"XFER: %s (md5 mismatch: src=%s dst=%s)" % (file, src_md5, dst_md5))
332
333 if attribs_match:
334 ## Remove from source-list, all that is left there will be transferred
335 debug(u"IGNR: %s (transfer not needed)" % file)
336 exists_list[file] = src_list[file]
337 del(src_list[file])
338
339 ## Remove from destination-list, all that is left there will be deleted
340 del(dst_list[file])
341
342 return src_list, dst_list, exists_list
536 md5 = src_list.get_md5(relative_file)
537 except IOError:
538 md5 = None
539 if md5 is not None and dst_list.by_md5.has_key(md5):
540 # Found one, we want to copy
541 dst1 = list(dst_list.by_md5[md5])[0]
542 debug(u"DST COPY src: %s -> %s" % (dst1, relative_file))
543 copy_pairs.append((src_list[relative_file], dst1, relative_file))
544 del(src_list[relative_file])
545 del(dst_list[relative_file])
546 else:
547 # record that we will get this file transferred to us (before all the copies), so if we come across it later again,
548 # we can copy from _this_ copy (e.g. we only upload it once, and copy thereafter).
549 dst_list.record_md5(relative_file, md5)
550 update_list[relative_file] = src_list[relative_file]
551 del src_list[relative_file]
552 del dst_list[relative_file]
553
554 else:
555 # dst doesn't have this file
556 # look for matching file elsewhere in dst
557 try:
558 md5 = src_list.get_md5(relative_file)
559 except IOError:
560 md5 = None
561 dst1 = dst_list.find_md5_one(md5)
562 if dst1 is not None:
563 # Found one, we want to copy
564 debug(u"DST COPY dst: %s -> %s" % (dst1, relative_file))
565 copy_pairs.append((src_list[relative_file], dst1, relative_file))
566 del(src_list[relative_file])
567 else:
568 # we don't have this file, and we don't have a copy of this file elsewhere. Get it.
569 # record that we will get this file transferred to us (before all the copies), so if we come across it later again,
570 # we can copy from _this_ copy (e.g. we only upload it once, and copy thereafter).
571 dst_list.record_md5(relative_file, md5)
572
573 for f in dst_list.keys():
574 if src_list.has_key(f) or update_list.has_key(f):
575 # leave only those not on src_list + update_list
576 del dst_list[f]
577
578 return src_list, dst_list, update_list, copy_pairs
343579
344580 # vim:et:ts=4:sts=4:ai
+57
-0
S3/HashCache.py less more
0 import cPickle as pickle
1
2 class HashCache(object):
3 def __init__(self):
4 self.inodes = dict()
5
6 def add(self, dev, inode, mtime, size, md5):
7 if dev == 0 or inode == 0: return # Windows
8 if dev not in self.inodes:
9 self.inodes[dev] = dict()
10 if inode not in self.inodes[dev]:
11 self.inodes[dev][inode] = dict()
12 self.inodes[dev][inode][mtime] = dict(md5=md5, size=size)
13
14 def md5(self, dev, inode, mtime, size):
15 try:
16 d = self.inodes[dev][inode][mtime]
17 if d['size'] != size:
18 return None
19 except:
20 return None
21 return d['md5']
22
23 def mark_all_for_purge(self):
24 for d in self.inodes.keys():
25 for i in self.inodes[d].keys():
26 for c in self.inodes[d][i].keys():
27 self.inodes[d][i][c]['purge'] = True
28
29 def unmark_for_purge(self, dev, inode, mtime, size):
30 try:
31 d = self.inodes[dev][inode][mtime]
32 except KeyError:
33 return
34 if d['size'] == size and 'purge' in d:
35 del self.inodes[dev][inode][mtime]['purge']
36
37 def purge(self):
38 for d in self.inodes.keys():
39 for i in self.inodes[d].keys():
40 for m in self.inodes[d][i].keys():
41 if 'purge' in self.inodes[d][i][m]:
42 del self.inodes[d][i]
43 break
44
45 def save(self, f):
46 d = dict(inodes=self.inodes, version=1)
47 f = open(f, 'w')
48 p = pickle.dump(d, f)
49 f.close()
50
51 def load(self, f):
52 f = open(f, 'r')
53 d = pickle.load(f)
54 f.close()
55 if d.get('version') == 1 and 'inodes' in d:
56 self.inodes = d['inodes']
+115
-28
S3/MultiPart.py less more
22 ## License: GPL Version 2
33
44 import os
5 import sys
56 from stat import ST_SIZE
67 from logging import debug, info, warning, error
7 from Utils import getTextFromXml, formatSize, unicodise
8 from Utils import getTextFromXml, getTreeFromXml, formatSize, unicodise, calculateChecksum, parseNodes
89 from Exceptions import S3UploadError
10 from collections import defaultdict
911
1012 class MultiPartUpload(object):
1113
2123 self.headers_baseline = headers_baseline
2224 self.upload_id = self.initiate_multipart_upload()
2325
26 def get_parts_information(self, uri, upload_id):
27 multipart_response = self.s3.list_multipart(uri, upload_id)
28 tree = getTreeFromXml(multipart_response['data'])
29
30 parts = defaultdict(lambda: None)
31 for elem in parseNodes(tree):
32 try:
33 parts[int(elem['PartNumber'])] = {'checksum': elem['ETag'], 'size': elem['Size']}
34 except KeyError:
35 pass
36
37 return parts
38
39 def get_unique_upload_id(self, uri):
40 upload_id = None
41 multipart_response = self.s3.get_multipart(uri)
42 tree = getTreeFromXml(multipart_response['data'])
43 for mpupload in parseNodes(tree):
44 try:
45 mp_upload_id = mpupload['UploadId']
46 mp_path = mpupload['Key']
47 info("mp_path: %s, object: %s" % (mp_path, uri.object()))
48 if mp_path == uri.object():
49 if upload_id is not None:
50 raise ValueError("More than one UploadId for URI %s. Disable multipart upload, or use\n %s multipart %s\nto list the Ids, then pass a unique --upload-id into the put command." % (uri, sys.argv[0], uri))
51 upload_id = mp_upload_id
52 except KeyError:
53 pass
54
55 return upload_id
56
2457 def initiate_multipart_upload(self):
2558 """
2659 Begin a multipart upload
2760 http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadInitiate.html
2861 """
29 request = self.s3.create_request("OBJECT_POST", uri = self.uri, headers = self.headers_baseline, extra = "?uploads")
30 response = self.s3.send_request(request)
31 data = response["data"]
32 self.upload_id = getTextFromXml(data, "UploadId")
62 if self.s3.config.upload_id is not None:
63 self.upload_id = self.s3.config.upload_id
64 elif self.s3.config.put_continue:
65 self.upload_id = self.get_unique_upload_id(self.uri)
66 else:
67 self.upload_id = None
68
69 if self.upload_id is None:
70 request = self.s3.create_request("OBJECT_POST", uri = self.uri, headers = self.headers_baseline, extra = "?uploads")
71 response = self.s3.send_request(request)
72 data = response["data"]
73 self.upload_id = getTextFromXml(data, "UploadId")
74
3375 return self.upload_id
3476
3577 def upload_all_parts(self):
4183 if not self.upload_id:
4284 raise RuntimeError("Attempting to use a multipart upload that has not been initiated.")
4385
44 size_left = file_size = os.stat(self.file.name)[ST_SIZE]
4586 self.chunk_size = self.s3.config.multipart_chunk_size_mb * 1024 * 1024
46 nr_parts = file_size / self.chunk_size + (file_size % self.chunk_size and 1)
47 debug("MultiPart: Uploading %s in %d parts" % (self.file.name, nr_parts))
87
88 if self.file.name != "<stdin>":
89 size_left = file_size = os.stat(self.file.name)[ST_SIZE]
90 nr_parts = file_size / self.chunk_size + (file_size % self.chunk_size and 1)
91 debug("MultiPart: Uploading %s in %d parts" % (self.file.name, nr_parts))
92 else:
93 debug("MultiPart: Uploading from %s" % (self.file.name))
94
95 remote_statuses = defaultdict(lambda: None)
96 if self.s3.config.put_continue:
97 remote_statuses = self.get_parts_information(self.uri, self.upload_id)
4898
4999 seq = 1
50 while size_left > 0:
51 offset = self.chunk_size * (seq - 1)
52 current_chunk_size = min(file_size - offset, self.chunk_size)
53 size_left -= current_chunk_size
54 labels = {
55 'source' : unicodise(self.file.name),
56 'destination' : unicodise(self.uri.uri()),
57 'extra' : "[part %d of %d, %s]" % (seq, nr_parts, "%d%sB" % formatSize(current_chunk_size, human_readable = True))
58 }
59 try:
60 self.upload_part(seq, offset, current_chunk_size, labels)
61 except:
62 error(u"Upload of '%s' part %d failed. Aborting multipart upload." % (self.file.name, seq))
63 self.abort_upload()
64 raise
65 seq += 1
100 if self.file.name != "<stdin>":
101 while size_left > 0:
102 offset = self.chunk_size * (seq - 1)
103 current_chunk_size = min(file_size - offset, self.chunk_size)
104 size_left -= current_chunk_size
105 labels = {
106 'source' : unicodise(self.file.name),
107 'destination' : unicodise(self.uri.uri()),
108 'extra' : "[part %d of %d, %s]" % (seq, nr_parts, "%d%sB" % formatSize(current_chunk_size, human_readable = True))
109 }
110 try:
111 self.upload_part(seq, offset, current_chunk_size, labels, remote_status = remote_statuses[seq])
112 except:
113 error(u"\nUpload of '%s' part %d failed. Use\n %s abortmp %s %s\nto abort the upload, or\n %s --upload-id %s put ...\nto continue the upload."
114 % (self.file.name, seq, sys.argv[0], self.uri, self.upload_id, sys.argv[0], self.upload_id))
115 raise
116 seq += 1
117 else:
118 while True:
119 buffer = self.file.read(self.chunk_size)
120 offset = self.chunk_size * (seq - 1)
121 current_chunk_size = len(buffer)
122 labels = {
123 'source' : unicodise(self.file.name),
124 'destination' : unicodise(self.uri.uri()),
125 'extra' : "[part %d, %s]" % (seq, "%d%sB" % formatSize(current_chunk_size, human_readable = True))
126 }
127 if len(buffer) == 0: # EOF
128 break
129 try:
130 self.upload_part(seq, offset, current_chunk_size, labels, buffer, remote_status = remote_statuses[seq])
131 except:
132 error(u"\nUpload of '%s' part %d failed. Use\n %s abortmp %s %s\nto abort, or\n %s --upload-id %s put ...\nto continue the upload."
133 % (self.file.name, seq, self.uri, sys.argv[0], self.upload_id, sys.argv[0], self.upload_id))
134 raise
135 seq += 1
66136
67137 debug("MultiPart: Upload finished: %d parts", seq - 1)
68138
69 def upload_part(self, seq, offset, chunk_size, labels):
139 def upload_part(self, seq, offset, chunk_size, labels, buffer = '', remote_status = None):
70140 """
71141 Upload a file chunk
72142 http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadUploadPart.html
73143 """
74144 # TODO implement Content-MD5
75145 debug("Uploading part %i of %r (%s bytes)" % (seq, self.upload_id, chunk_size))
146
147 if remote_status is not None:
148 if int(remote_status['size']) == chunk_size:
149 checksum = calculateChecksum(buffer, self.file, offset, chunk_size, self.s3.config.send_chunk)
150 remote_checksum = remote_status['checksum'].strip('"')
151 if remote_checksum == checksum:
152 warning("MultiPart: size and md5sum match for %s part %d, skipping." % (self.uri, seq))
153 self.parts[seq] = remote_status['checksum']
154 return
155 else:
156 warning("MultiPart: checksum (%s vs %s) does not match for %s part %d, reuploading."
157 % (remote_checksum, checksum, self.uri, seq))
158 else:
159 warning("MultiPart: size (%d vs %d) does not match for %s part %d, reuploading."
160 % (int(remote_status['size']), chunk_size, self.uri, seq))
161
76162 headers = { "content-length": chunk_size }
77163 query_string = "?partNumber=%i&uploadId=%s" % (seq, self.upload_id)
78164 request = self.s3.create_request("OBJECT_PUT", uri = self.uri, headers = headers, extra = query_string)
79 response = self.s3.send_file(request, self.file, labels, offset = offset, chunk_size = chunk_size)
165 response = self.s3.send_file(request, self.file, labels, buffer, offset = offset, chunk_size = chunk_size)
80166 self.parts[seq] = response["headers"]["etag"]
81167 return response
82168
105191 http://docs.amazonwebservices.com/AmazonS3/latest/API/index.html?mpUploadAbort.html
106192 """
107193 debug("MultiPart: Aborting upload: %s" % self.upload_id)
108 request = self.s3.create_request("OBJECT_DELETE", uri = self.uri, extra = "?uploadId=%s" % (self.upload_id))
109 response = self.s3.send_request(request)
194 #request = self.s3.create_request("OBJECT_DELETE", uri = self.uri, extra = "?uploadId=%s" % (self.upload_id))
195 #response = self.s3.send_request(request)
196 response = None
110197 return response
111198
112199 # vim:et:ts=4:sts=4:ai
+1
-1
S3/PkgInfo.py less more
00 package = "s3cmd"
1 version = "1.1.0-beta3"
1 version = "1.5.0-beta1"
22 url = "http://s3tools.org"
33 license = "GPL version 2"
44 short_description = "Command line tool for managing Amazon S3 and CloudFront services"
+18
-1
S3/Progress.py less more
44
55 import sys
66 import datetime
7 import time
78 import Utils
89
910 class Progress(object):
1011 _stdout = sys.stdout
12 _last_display = 0
1113
1214 def __init__(self, labels, total_size):
1315 self._stdout = sys.stdout
4749 self._stdout.write(u"%(source)s -> %(destination)s %(extra)s\n" % self.labels)
4850 self._stdout.flush()
4951
52 def _display_needed(self):
53 # We only need to update the display every so often.
54 if time.time() - self._last_display > 1:
55 self._last_display = time.time()
56 return True
57 return False
58
5059 def display(self, new_file = False, done_message = None):
5160 """
5261 display(new_file = False[/True], done = False[/True])
6978 self._stdout.flush()
7079 return
7180
72 rel_position = selfself.current_position * 100 / self.total_size
81 rel_position = self.current_position * 100 / self.total_size
7382 if rel_position >= self.last_milestone:
7483 self.last_milestone = (int(rel_position) / 5) * 5
7584 self._stdout.write("%d%% ", self.last_milestone)
95104 self.output_labels()
96105 self._stdout.write(self.ANSI_save_cursor_pos)
97106 self._stdout.flush()
107 return
108
109 # Only display progress every so often
110 if not (new_file or done_message) and not self._display_needed():
98111 return
99112
100113 timedelta = self.time_current - self.time_start
131144 self.output_labels()
132145 return
133146
147 # Only display progress every so often
148 if not (new_file or done_message) and not self._display_needed():
149 return
150
134151 timedelta = self.time_current - self.time_start
135152 sec_elapsed = timedelta.days * 86400 + timedelta.seconds + float(timedelta.microseconds)/1000000.0
136153 if (sec_elapsed > 0):
+236
-54
S3/S3.py less more
55 import sys
66 import os, os.path
77 import time
8 import errno
89 import httplib
910 import logging
1011 import mimetypes
2627 from Exceptions import *
2728 from MultiPart import MultiPartUpload
2829 from S3Uri import S3Uri
30 from ConnMan import ConnMan
2931
3032 try:
31 import magic
33 import magic, gzip
3234 try:
3335 ## https://github.com/ahupp/python-magic
3436 magic_ = magic.Magic(mime=True)
35 def mime_magic(file):
37 def mime_magic_file(file):
3638 return magic_.from_file(file)
37 except (TypeError, AttributeError):
39 def mime_magic_buffer(buffer):
40 return magic_.from_buffer(buffer)
41 except TypeError:
42 ## http://pypi.python.org/pypi/filemagic
43 try:
44 magic_ = magic.Magic(flags=magic.MAGIC_MIME)
45 def mime_magic_file(file):
46 return magic_.id_filename(file)
47 def mime_magic_buffer(buffer):
48 return magic_.id_buffer(buffer)
49 except TypeError:
50 ## file-5.11 built-in python bindings
51 magic_ = magic.open(magic.MAGIC_MIME)
52 magic_.load()
53 def mime_magic_file(file):
54 return magic_.file(file)
55 def mime_magic_buffer(buffer):
56 return magic_.buffer(buffer)
57
58 except AttributeError:
3859 ## Older python-magic versions
3960 magic_ = magic.open(magic.MAGIC_MIME)
4061 magic_.load()
41 def mime_magic(file):
62 def mime_magic_file(file):
4263 return magic_.file(file)
64 def mime_magic_buffer(buffer):
65 return magic_.buffer(buffer)
66
67 def mime_magic(file):
68 type = mime_magic_file(file)
69 if type != "application/x-gzip; charset=binary":
70 return (type, None)
71 else:
72 return (mime_magic_buffer(gzip.open(file).read(8192)), 'gzip')
73
4374 except ImportError, e:
4475 if str(e).find("magic") >= 0:
4576 magic_message = "Module python-magic is not available."
5283 if (not magic_warned):
5384 warning(magic_message)
5485 magic_warned = True
55 return mimetypes.guess_type(file)[0]
86 return mimetypes.guess_type(file)
5687
5788 __all__ = []
5889 class S3Request(object):
5990 def __init__(self, s3, method_string, resource, headers, params = {}):
6091 self.s3 = s3
6192 self.headers = SortedDict(headers or {}, ignore_case = True)
93 # Add in any extra headers from s3 config object
94 if self.s3.config.extra_headers:
95 self.headers.update(self.s3.config.extra_headers)
96 if len(self.s3.config.access_token)>0:
97 self.s3.config.role_refresh()
98 self.headers['x-amz-security-token']=self.s3.config.access_token
6299 self.resource = resource
63100 self.method_string = method_string
64101 self.params = params
153190
154191 def __init__(self, config):
155192 self.config = config
156
157 def get_connection(self, bucket):
158 if self.config.proxy_host != "":
159 return httplib.HTTPConnection(self.config.proxy_host, self.config.proxy_port)
160 else:
161 if self.config.use_https:
162 return httplib.HTTPSConnection(self.get_hostname(bucket))
163 else:
164 return httplib.HTTPConnection(self.get_hostname(bucket))
165193
166194 def get_hostname(self, bucket):
167195 if bucket and check_bucket_name_dns_conformity(bucket):
338366
339367 return response
340368
369 def add_encoding(self, filename, content_type):
370 if content_type.find("charset=") != -1:
371 return False
372 exts = self.config.add_encoding_exts.split(',')
373 if exts[0]=='':
374 return False
375 parts = filename.rsplit('.',2)
376 if len(parts) < 2:
377 return False
378 ext = parts[1]
379 if ext in exts:
380 return True
381 else:
382 return False
383
341384 def object_put(self, filename, uri, extra_headers = None, extra_label = ""):
342385 # TODO TODO
343386 # Make it consistent with stream-oriented object_get()
344387 if uri.type != "s3":
345388 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
346389
347 if not os.path.isfile(filename):
390 if filename != "-" and not os.path.isfile(filename):
348391 raise InvalidFileError(u"%s is not a regular file" % unicodise(filename))
349392 try:
350 file = open(filename, "rb")
351 size = os.stat(filename)[ST_SIZE]
393 if filename == "-":
394 file = sys.stdin
395 size = 0
396 else:
397 file = open(filename, "rb")
398 size = os.stat(filename)[ST_SIZE]
352399 except (IOError, OSError), e:
353400 raise InvalidFileError(u"%s: %s" % (unicodise(filename), e.strerror))
354401
356403 if extra_headers:
357404 headers.update(extra_headers)
358405
406 ## Set server side encryption
407 if self.config.server_side_encryption:
408 headers["x-amz-server-side-encryption"] = "AES256"
409
359410 ## MIME-type handling
360411 content_type = self.config.mime_type
361 if not content_type and self.config.guess_mime_type:
362 content_type = mime_magic(filename)
412 content_encoding = None
413 if filename != "-" and not content_type and self.config.guess_mime_type:
414 if self.config.use_mime_magic:
415 (content_type, content_encoding) = mime_magic(filename)
416 else:
417 (content_type, content_encoding) = mimetypes.guess_type(filename)
363418 if not content_type:
364419 content_type = self.config.default_mime_type
365 debug("Content-Type set to '%s'" % content_type)
420
421 ## add charset to content type
422 if self.add_encoding(filename, content_type):
423 content_type = content_type + "; charset=" + self.config.encoding.upper()
424
366425 headers["content-type"] = content_type
426 if content_encoding is not None and self.config.add_content_encoding:
427 headers["content-encoding"] = content_encoding
367428
368429 ## Other Amazon S3 attributes
369430 if self.config.acl_public:
373434
374435 ## Multipart decision
375436 multipart = False
437 if not self.config.enable_multipart and filename == "-":
438 raise ParameterError("Multi-part upload is required to upload from stdin")
376439 if self.config.enable_multipart:
377 if size > self.config.multipart_chunk_size_mb * 1024 * 1024:
440 if size > self.config.multipart_chunk_size_mb * 1024 * 1024 or filename == "-":
378441 multipart = True
379442 if multipart:
380443 # Multipart requests are quite different... drop here
381444 return self.send_file_multipart(file, headers, uri, size)
382445
383446 ## Not multipart...
447 if self.config.put_continue:
448 # Note, if input was stdin, we would be performing multipart upload.
449 # So this will always work as long as the file already uploaded was
450 # not uploaded via MultiUpload, in which case its ETag will not be
451 # an md5.
452 try:
453 info = self.object_info(uri)
454 except:
455 info = None
456
457 if info is not None:
458 remote_size = int(info['headers']['content-length'])
459 remote_checksum = info['headers']['etag'].strip('"')
460 if size == remote_size:
461 checksum = calculateChecksum('', file, 0, size, self.config.send_chunk)
462 if remote_checksum == checksum:
463 warning("Put: size and md5sum match for %s, skipping." % uri)
464 return
465 else:
466 warning("MultiPart: checksum (%s vs %s) does not match for %s, reuploading."
467 % (remote_checksum, checksum, uri))
468 else:
469 warning("MultiPart: size (%d vs %d) does not match for %s, reuploading."
470 % (remote_size, size, uri))
471
384472 headers["content-length"] = size
385473 request = self.create_request("OBJECT_PUT", uri = uri, headers = headers)
386474 labels = { 'source' : unicodise(filename), 'destination' : unicodise(uri.uri()), 'extra' : extra_label }
400488 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
401489 request = self.create_request("OBJECT_DELETE", uri = uri)
402490 response = self.send_request(request)
491 return response
492
493 def object_restore(self, uri):
494 if uri.type != "s3":
495 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
496 body = '<RestoreRequest xmlns="http://s3.amazonaws.com/doc/2006-3-01">'
497 body += (' <Days>%s</Days>' % self.config.restore_days)
498 body += '</RestoreRequest>'
499 request = self.create_request("OBJECT_POST", uri = uri, extra = "?restore")
500 debug("About to send request '%s' with body '%s'" % (request, body))
501 response = self.send_request(request, body)
502 debug("Received response '%s'" % (response))
403503 return response
404504
405505 def object_copy(self, src_uri, dst_uri, extra_headers = None):
417517 headers["x-amz-storage-class"] = "REDUCED_REDUNDANCY"
418518 # if extra_headers:
419519 # headers.update(extra_headers)
520
521 ## Set server side encryption
522 if self.config.server_side_encryption:
523 headers["x-amz-server-side-encryption"] = "AES256"
524
420525 request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers)
421526 response = self.send_request(request)
422527 return response
453558 body = str(acl)
454559 debug(u"set_acl(%s): acl-xml: %s" % (uri, body))
455560 response = self.send_request(request, body)
561 return response
562
563 def get_policy(self, uri):
564 request = self.create_request("BUCKET_LIST", bucket = uri.bucket(), extra = "?policy")
565 response = self.send_request(request)
566 return response['data']
567
568 def set_policy(self, uri, policy):
569 headers = {}
570 # TODO check policy is proper json string
571 headers['content-type'] = 'application/json'
572 request = self.create_request("BUCKET_CREATE", uri = uri,
573 extra = "?policy", headers=headers)
574 body = policy
575 debug(u"set_policy(%s): policy-json: %s" % (uri, body))
576 request.sign()
577 response = self.send_request(request, body=body)
578 return response
579
580 def delete_policy(self, uri):
581 request = self.create_request("BUCKET_DELETE", uri = uri, extra = "?policy")
582 debug(u"delete_policy(%s)" % uri)
583 response = self.send_request(request)
584 return response
585
586 def get_multipart(self, uri):
587 request = self.create_request("BUCKET_LIST", bucket = uri.bucket(), extra = "?uploads")
588 response = self.send_request(request)
589 return response
590
591 def abort_multipart(self, uri, id):
592 request = self.create_request("OBJECT_DELETE", uri=uri,
593 extra = ("?uploadId=%s" % id))
594 response = self.send_request(request)
595 return response
596
597 def list_multipart(self, uri, id):
598 request = self.create_request("OBJECT_GET", uri=uri,
599 extra = ("?uploadId=%s" % id))
600 response = self.send_request(request)
456601 return response
457602
458603 def get_accesslog(self, uri):
579724 # "Stringify" all headers
580725 for header in headers.keys():
581726 headers[header] = str(headers[header])
582 conn = self.get_connection(resource['bucket'])
727 conn = ConnMan.get(self.get_hostname(resource['bucket']))
583728 uri = self.format_uri(resource)
584729 debug("Sending request method_string=%r, uri=%r, headers=%r, body=(%i bytes)" % (method_string, uri, headers, len(body or "")))
585 conn.request(method_string, uri, body, headers)
730 conn.c.request(method_string, uri, body, headers)
586731 response = {}
587 http_response = conn.getresponse()
732 http_response = conn.c.getresponse()
588733 response["status"] = http_response.status
589734 response["reason"] = http_response.reason
590735 response["headers"] = convertTupleListToDict(http_response.getheaders())
591736 response["data"] = http_response.read()
737 if response["headers"].has_key("x-amz-meta-s3cmd-attrs"):
738 attrs = parse_attrs_header(response["headers"]["x-amz-meta-s3cmd-attrs"])
739 response["s3cmd-attrs"] = attrs
592740 debug("Response: " + str(response))
593 conn.close()
741 ConnMan.put(conn)
742 except ParameterError, e:
743 raise
594744 except Exception, e:
595745 if retries:
596746 warning("Retrying failed request: %s (%s)" % (resource['uri'], e))
624774
625775 return response
626776
627 def send_file(self, request, file, labels, throttle = 0, retries = _max_retries, offset = 0, chunk_size = -1):
777 def send_file(self, request, file, labels, buffer = '', throttle = 0, retries = _max_retries, offset = 0, chunk_size = -1):
628778 method_string, resource, headers = request.get_triplet()
629779 size_left = size_total = headers.get("content-length")
630780 if self.config.progress_meter:
633783 info("Sending file '%s', please wait..." % file.name)
634784 timestamp_start = time.time()
635785 try:
636 conn = self.get_connection(resource['bucket'])
637 conn.connect()
638 conn.putrequest(method_string, self.format_uri(resource))
786 conn = ConnMan.get(self.get_hostname(resource['bucket']))
787 conn.c.putrequest(method_string, self.format_uri(resource))
639788 for header in headers.keys():
640 conn.putheader(header, str(headers[header]))
641 conn.endheaders()
789 conn.c.putheader(header, str(headers[header]))
790 conn.c.endheaders()
791 except ParameterError, e:
792 raise
642793 except Exception, e:
643794 if self.config.progress_meter:
644795 progress.done("failed")
647798 warning("Waiting %d sec..." % self._fail_wait(retries))
648799 time.sleep(self._fail_wait(retries))
649800 # Connection error -> same throttle value
650 return self.send_file(request, file, labels, throttle, retries - 1, offset, chunk_size)
801 return self.send_file(request, file, labels, buffer, throttle, retries - 1, offset, chunk_size)
651802 else:
652803 raise S3UploadError("Upload failed for: %s" % resource['uri'])
653 file.seek(offset)
804 if buffer == '':
805 file.seek(offset)
654806 md5_hash = md5()
807
655808 try:
656809 while (size_left > 0):
657 #debug("SendFile: Reading up to %d bytes from '%s'" % (self.config.send_chunk, file.name))
658 data = file.read(min(self.config.send_chunk, size_left))
810 #debug("SendFile: Reading up to %d bytes from '%s' - remaining bytes: %s" % (self.config.send_chunk, file.name, size_left))
811 if buffer == '':
812 data = file.read(min(self.config.send_chunk, size_left))
813 else:
814 data = buffer
815
659816 md5_hash.update(data)
660 conn.send(data)
817 conn.c.send(data)
661818 if self.config.progress_meter:
662819 progress.update(delta_position = len(data))
663820 size_left -= len(data)
664821 if throttle:
665822 time.sleep(throttle)
666823 md5_computed = md5_hash.hexdigest()
824
667825 response = {}
668 http_response = conn.getresponse()
826 http_response = conn.c.getresponse()
669827 response["status"] = http_response.status
670828 response["reason"] = http_response.reason
671829 response["headers"] = convertTupleListToDict(http_response.getheaders())
672830 response["data"] = http_response.read()
673831 response["size"] = size_total
674 conn.close()
832 ConnMan.put(conn)
675833 debug(u"Response: %s" % response)
834 except ParameterError, e:
835 raise
676836 except Exception, e:
677837 if self.config.progress_meter:
678838 progress.done("failed")
684844 warning("Waiting %d sec..." % self._fail_wait(retries))
685845 time.sleep(self._fail_wait(retries))
686846 # Connection error -> same throttle value
687 return self.send_file(request, file, labels, throttle, retries - 1, offset, chunk_size)
847 return self.send_file(request, file, labels, buffer, throttle, retries - 1, offset, chunk_size)
688848 else:
689849 debug("Giving up on '%s' %s" % (file.name, e))
690850 raise S3UploadError("Upload failed for: %s" % resource['uri'])
694854 response["speed"] = response["elapsed"] and float(response["size"]) / response["elapsed"] or float(-1)
695855
696856 if self.config.progress_meter:
697 ## The above conn.close() takes some time -> update() progress meter
857 ## Finalising the upload takes some time -> update() progress meter
698858 ## to correct the average speed. Otherwise people will complain that
699859 ## 'progress' and response["speed"] are inconsistent ;-)
700860 progress.update()
706866 redir_hostname = getTextFromXml(response['data'], ".//Endpoint")
707867 self.set_hostname(redir_bucket, redir_hostname)
708868 warning("Redirected to: %s" % (redir_hostname))
709 return self.send_file(request, file, labels, offset = offset, chunk_size = chunk_size)
869 return self.send_file(request, file, labels, buffer, offset = offset, chunk_size = chunk_size)
710870
711871 # S3 from time to time doesn't send ETag back in a response :-(
712872 # Force re-upload here.
729889 warning("Upload failed: %s (%s)" % (resource['uri'], S3Error(response)))
730890 warning("Waiting %d sec..." % self._fail_wait(retries))
731891 time.sleep(self._fail_wait(retries))
732 return self.send_file(request, file, labels, throttle, retries - 1, offset, chunk_size)
892 return self.send_file(request, file, labels, buffer, throttle, retries - 1, offset, chunk_size)
733893 else:
734894 warning("Too many failures. Giving up on '%s'" % (file.name))
735895 raise S3UploadError
742902 warning("MD5 Sums don't match!")
743903 if retries:
744904 warning("Retrying upload of %s" % (file.name))
745 return self.send_file(request, file, labels, throttle, retries - 1, offset, chunk_size)
905 return self.send_file(request, file, labels, buffer, throttle, retries - 1, offset, chunk_size)
746906 else:
747907 warning("Too many failures. Giving up on '%s'" % (file.name))
748908 raise S3UploadError
751911
752912 def send_file_multipart(self, file, headers, uri, size):
753913 chunk_size = self.config.multipart_chunk_size_mb * 1024 * 1024
914 timestamp_start = time.time()
754915 upload = MultiPartUpload(self, file, uri, headers)
755916 upload.upload_all_parts()
756917 response = upload.complete_multipart_upload()
757 response["speed"] = 0 # XXX
918 timestamp_end = time.time()
919 response["elapsed"] = timestamp_end - timestamp_start
758920 response["size"] = size
921 response["speed"] = response["elapsed"] and float(response["size"]) / response["elapsed"] or float(-1)
759922 return response
760923
761924 def recv_file(self, request, stream, labels, start_position = 0, retries = _max_retries):
766929 info("Receiving file '%s', please wait..." % stream.name)
767930 timestamp_start = time.time()
768931 try:
769 conn = self.get_connection(resource['bucket'])
770 conn.connect()
771 conn.putrequest(method_string, self.format_uri(resource))
932 conn = ConnMan.get(self.get_hostname(resource['bucket']))
933 conn.c.putrequest(method_string, self.format_uri(resource))
772934 for header in headers.keys():
773 conn.putheader(header, str(headers[header]))
935 conn.c.putheader(header, str(headers[header]))
774936 if start_position > 0:
775937 debug("Requesting Range: %d .. end" % start_position)
776 conn.putheader("Range", "bytes=%d-" % start_position)
777 conn.endheaders()
938 conn.c.putheader("Range", "bytes=%d-" % start_position)
939 conn.c.endheaders()
778940 response = {}
779 http_response = conn.getresponse()
941 http_response = conn.c.getresponse()
780942 response["status"] = http_response.status
781943 response["reason"] = http_response.reason
782944 response["headers"] = convertTupleListToDict(http_response.getheaders())
945 if response["headers"].has_key("x-amz-meta-s3cmd-attrs"):
946 attrs = parse_attrs_header(response["headers"]["x-amz-meta-s3cmd-attrs"])
947 response["s3cmd-attrs"] = attrs
783948 debug("Response: %s" % response)
949 except ParameterError, e:
950 raise
784951 except Exception, e:
785952 if self.config.progress_meter:
786953 progress.done("failed")
822989 while (current_position < size_total):
823990 this_chunk = size_left > self.config.recv_chunk and self.config.recv_chunk or size_left
824991 data = http_response.read(this_chunk)
992 if len(data) == 0:
993 raise S3Error("EOF from S3!")
994
825995 stream.write(data)
826996 if start_position == 0:
827997 md5_hash.update(data)
829999 ## Call progress meter from here...
8301000 if self.config.progress_meter:
8311001 progress.update(delta_position = len(data))
832 conn.close()
1002 ConnMan.put(conn)
8331003 except Exception, e:
8341004 if self.config.progress_meter:
8351005 progress.done("failed")
8651035 warning("Unable to verify MD5. Assume it matches.")
8661036 response["md5"] = response["headers"]["etag"]
8671037
868 response["md5match"] = response["headers"]["etag"].find(response["md5"]) >= 0
1038 md5_hash = response["headers"]["etag"]
1039 try:
1040 md5_hash = response["s3cmd-attrs"]["md5"]
1041 except KeyError:
1042 pass
1043
1044 response["md5match"] = md5_hash.find(response["md5"]) >= 0
8691045 response["elapsed"] = timestamp_end - timestamp_start
8701046 response["size"] = current_position
8711047 response["speed"] = response["elapsed"] and float(response["size"]) / response["elapsed"] or float(-1)
8751051 debug("ReceiveFile: Computed MD5 = %s" % response["md5"])
8761052 if not response["md5match"]:
8771053 warning("MD5 signatures do not match: computed=%s, received=%s" % (
878 response["md5"], response["headers"]["etag"]))
1054 response["md5"], md5_hash))
8791055 return response
8801056 __all__.append("S3")
8811057
1058 def parse_attrs_header(attrs_header):
1059 attrs = {}
1060 for attr in attrs_header.split("/"):
1061 key, val = attr.split(":")
1062 attrs[key] = val
1063 return attrs
8821064 # vim:et:ts=4:sts=4:ai
+3
-2
S3/S3Uri.py less more
99 from logging import debug
1010 import S3
1111 from Utils import unicodise, check_bucket_name_dns_conformity
12 import Config
1213
1314 class S3Uri(object):
1415 type = None
7980
8081 def public_url(self):
8182 if self.is_dns_compatible():
82 return "http://%s.s3.amazonaws.com/%s" % (self._bucket, self._object)
83 return "http://%s.%s/%s" % (self._bucket, Config.Config().host_base, self._object)
8384 else:
84 return "http://s3.amazonaws.com/%s/%s" % (self._bucket, self._object)
85 return "http://%s/%s/%s" % (Config.Config().host_base, self._bucket, self._object)
8586
8687 def host_name(self):
8788 if self.is_dns_compatible():
+0
-175
S3/SimpleDB.py less more
0 ## Amazon SimpleDB library
1 ## Author: Michal Ludvig <michal@logix.cz>
2 ## http://www.logix.cz/michal
3 ## License: GPL Version 2
4
5 """
6 Low-level class for working with Amazon SimpleDB
7 """
8
9 import time
10 import urllib
11 import base64
12 import hmac
13 import sha
14 import httplib
15 from logging import debug, info, warning, error
16
17 from Utils import convertTupleListToDict
18 from SortedDict import SortedDict
19 from Exceptions import *
20
21 class SimpleDB(object):
22 # API Version
23 # See http://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/DeveloperGuide/
24 Version = "2007-11-07"
25 SignatureVersion = 1
26
27 def __init__(self, config):
28 self.config = config
29
30 ## ------------------------------------------------
31 ## Methods implementing SimpleDB API
32 ## ------------------------------------------------
33
34 def ListDomains(self, MaxNumberOfDomains = 100):
35 '''
36 Lists all domains associated with our Access Key. Returns
37 domain names up to the limit set by MaxNumberOfDomains.
38 '''
39 parameters = SortedDict()
40 parameters['MaxNumberOfDomains'] = MaxNumberOfDomains
41 return self.send_request("ListDomains", DomainName = None, parameters = parameters)
42
43 def CreateDomain(self, DomainName):
44 return self.send_request("CreateDomain", DomainName = DomainName)
45
46 def DeleteDomain(self, DomainName):
47 return self.send_request("DeleteDomain", DomainName = DomainName)
48
49 def PutAttributes(self, DomainName, ItemName, Attributes):
50 parameters = SortedDict()
51 parameters['ItemName'] = ItemName
52 seq = 0
53 for attrib in Attributes:
54 if type(Attributes[attrib]) == type(list()):
55 for value in Attributes[attrib]:
56 parameters['Attribute.%d.Name' % seq] = attrib
57 parameters['Attribute.%d.Value' % seq] = unicode(value)
58 seq += 1
59 else:
60 parameters['Attribute.%d.Name' % seq] = attrib
61 parameters['Attribute.%d.Value' % seq] = unicode(Attributes[attrib])
62 seq += 1
63 ## TODO:
64 ## - support for Attribute.N.Replace
65 ## - support for multiple values for one attribute
66 return self.send_request("PutAttributes", DomainName = DomainName, parameters = parameters)
67
68 def GetAttributes(self, DomainName, ItemName, Attributes = []):
69 parameters = SortedDict()
70 parameters['ItemName'] = ItemName
71 seq = 0
72 for attrib in Attributes:
73 parameters['AttributeName.%d' % seq] = attrib
74 seq += 1
75 return self.send_request("GetAttributes", DomainName = DomainName, parameters = parameters)
76
77 def DeleteAttributes(self, DomainName, ItemName, Attributes = {}):
78 """
79 Remove specified Attributes from ItemName.
80 Attributes parameter can be either:
81 - not specified, in which case the whole Item is removed
82 - list, e.g. ['Attr1', 'Attr2'] in which case these parameters are removed
83 - dict, e.g. {'Attr' : 'One', 'Attr' : 'Two'} in which case the
84 specified values are removed from multi-value attributes.
85 """
86 parameters = SortedDict()
87 parameters['ItemName'] = ItemName
88 seq = 0
89 for attrib in Attributes:
90 parameters['Attribute.%d.Name' % seq] = attrib
91 if type(Attributes) == type(dict()):
92 parameters['Attribute.%d.Value' % seq] = unicode(Attributes[attrib])
93 seq += 1
94 return self.send_request("DeleteAttributes", DomainName = DomainName, parameters = parameters)
95
96 def Query(self, DomainName, QueryExpression = None, MaxNumberOfItems = None, NextToken = None):
97 parameters = SortedDict()
98 if QueryExpression:
99 parameters['QueryExpression'] = QueryExpression
100 if MaxNumberOfItems:
101 parameters['MaxNumberOfItems'] = MaxNumberOfItems
102 if NextToken:
103 parameters['NextToken'] = NextToken
104 return self.send_request("Query", DomainName = DomainName, parameters = parameters)
105 ## Handle NextToken? Or maybe not - let the upper level do it
106
107 ## ------------------------------------------------
108 ## Low-level methods for handling SimpleDB requests
109 ## ------------------------------------------------
110
111 def send_request(self, *args, **kwargs):
112 request = self.create_request(*args, **kwargs)
113 #debug("Request: %s" % repr(request))
114 conn = self.get_connection()
115 conn.request("GET", self.format_uri(request['uri_params']))
116 http_response = conn.getresponse()
117 response = {}
118 response["status"] = http_response.status
119 response["reason"] = http_response.reason
120 response["headers"] = convertTupleListToDict(http_response.getheaders())
121 response["data"] = http_response.read()
122 conn.close()
123
124 if response["status"] < 200 or response["status"] > 299:
125 debug("Response: " + str(response))
126 raise S3Error(response)
127
128 return response
129
130 def create_request(self, Action, DomainName, parameters = None):
131 if not parameters:
132 parameters = SortedDict()
133 parameters['AWSAccessKeyId'] = self.config.access_key
134 parameters['Version'] = self.Version
135 parameters['SignatureVersion'] = self.SignatureVersion
136 parameters['Action'] = Action
137 parameters['Timestamp'] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
138 if DomainName:
139 parameters['DomainName'] = DomainName
140 parameters['Signature'] = self.sign_request(parameters)
141 parameters.keys_return_lowercase = False
142 uri_params = urllib.urlencode(parameters)
143 request = {}
144 request['uri_params'] = uri_params
145 request['parameters'] = parameters
146 return request
147
148 def sign_request(self, parameters):
149 h = ""
150 parameters.keys_sort_lowercase = True
151 parameters.keys_return_lowercase = False
152 for key in parameters:
153 h += "%s%s" % (key, parameters[key])
154 #debug("SignRequest: %s" % h)
155 return base64.encodestring(hmac.new(self.config.secret_key, h, sha).digest()).strip()
156
157 def get_connection(self):
158 if self.config.proxy_host != "":
159 return httplib.HTTPConnection(self.config.proxy_host, self.config.proxy_port)
160 else:
161 if self.config.use_https:
162 return httplib.HTTPSConnection(self.config.simpledb_host)
163 else:
164 return httplib.HTTPConnection(self.config.simpledb_host)
165
166 def format_uri(self, uri_params):
167 if self.config.proxy_host != "":
168 uri = "http://%s/?%s" % (self.config.simpledb_host, uri_params)
169 else:
170 uri = "/?%s" % uri_params
171 #debug('format_uri(): ' + uri)
172 return uri
173
174 # vim:et:ts=4:sts=4:ai
+3
-0
S3/SortedDict.py less more
33 ## License: GPL Version 2
44
55 from BidirMap import BidirMap
6 import Utils
67
78 class SortedDictIterator(object):
89 def __init__(self, sorted_dict, keys):
4445 def __iter__(self):
4546 return SortedDictIterator(self, self.keys())
4647
48
49
4750 if __name__ == "__main__":
4851 d = { 'AWS' : 1, 'Action' : 2, 'america' : 3, 'Auckland' : 4, 'America' : 5 }
4952 sd = SortedDict(d)
+126
-5
S3/Utils.py less more
22 ## http://www.logix.cz/michal
33 ## License: GPL Version 2
44
5 import datetime
56 import os
67 import sys
78 import time
1213 import hmac
1314 import base64
1415 import errno
16 import urllib
1517
1618 from logging import debug, info, warning, error
19
1720
1821 import Config
1922 import Exceptions
162165 __all__.append("formatSize")
163166
164167 def formatDateTime(s3timestamp):
165 return time.strftime("%Y-%m-%d %H:%M", dateS3toPython(s3timestamp))
168 try:
169 import pytz
170 timezone = pytz.timezone(os.environ.get('TZ', 'UTC'))
171 tz = pytz.timezone('UTC')
172 ## Can't unpack args and follow that with kwargs in python 2.5
173 ## So we pass them all as kwargs
174 params = zip(('year', 'month', 'day', 'hour', 'minute', 'second', 'tzinfo'),
175 dateS3toPython(s3timestamp)[0:6] + (tz,))
176 params = dict(params)
177 utc_dt = datetime.datetime(**params)
178 dt_object = utc_dt.astimezone(timezone)
179 except ImportError:
180 dt_object = datetime.datetime(*dateS3toPython(s3timestamp)[0:6])
181 return dt_object.strftime("%Y-%m-%d %H:%M")
166182 __all__.append("formatDateTime")
167183
168184 def convertTupleListToDict(list):
200216 return dirname
201217 __all__.append("mktmpsomething")
202218
203 def mktmpdir(prefix = "/tmp/tmpdir-", randchars = 10):
219 def mktmpdir(prefix = os.getenv('TMP','/tmp') + "/tmpdir-", randchars = 10):
204220 return mktmpsomething(prefix, randchars, os.mkdir)
205221 __all__.append("mktmpdir")
206222
207 def mktmpfile(prefix = "/tmp/tmpfile-", randchars = 20):
223 def mktmpfile(prefix = os.getenv('TMP','/tmp') + "/tmpfile-", randchars = 20):
208224 createfunc = lambda filename : os.close(os.open(filename, os.O_CREAT | os.O_EXCL))
209225 return mktmpsomething(prefix, randchars, createfunc)
210226 __all__.append("mktmpfile")
318334 __all__.append("replace_nonprintables")
319335
320336 def sign_string(string_to_sign):
321 #debug("string_to_sign: %s" % string_to_sign)
337 """Sign a string with the secret key, returning base64 encoded results.
338 By default the configured secret key is used, but may be overridden as
339 an argument.
340
341 Useful for REST authentication. See http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
342 """
322343 signature = base64.encodestring(hmac.new(Config.Config().secret_key, string_to_sign, sha1).digest()).strip()
323 #debug("signature: %s" % signature)
324344 return signature
325345 __all__.append("sign_string")
346
347 def sign_url(url_to_sign, expiry):
348 """Sign a URL in s3://bucket/object form with the given expiry
349 time. The object will be accessible via the signed URL until the
350 AWS key and secret are revoked or the expiry time is reached, even
351 if the object is otherwise private.
352
353 See: http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
354 """
355 return sign_url_base(
356 bucket = url_to_sign.bucket(),
357 object = url_to_sign.object(),
358 expiry = expiry
359 )
360 __all__.append("sign_url")
361
362 def sign_url_base(**parms):
363 """Shared implementation of sign_url methods. Takes a hash of 'bucket', 'object' and 'expiry' as args."""
364 parms['expiry']=time_to_epoch(parms['expiry'])
365 parms['access_key']=Config.Config().access_key
366 debug("Expiry interpreted as epoch time %s", parms['expiry'])
367 signtext = 'GET\n\n\n%(expiry)d\n/%(bucket)s/%(object)s' % parms
368 debug("Signing plaintext: %r", signtext)
369 parms['sig'] = urllib.quote_plus(sign_string(signtext))
370 debug("Urlencoded signature: %s", parms['sig'])
371 return "http://%(bucket)s.s3.amazonaws.com/%(object)s?AWSAccessKeyId=%(access_key)s&Expires=%(expiry)d&Signature=%(sig)s" % parms
372
373 def time_to_epoch(t):
374 """Convert time specified in a variety of forms into UNIX epoch time.
375 Accepts datetime.datetime, int, anything that has a strftime() method, and standard time 9-tuples
376 """
377 if isinstance(t, int):
378 # Already an int
379 return t
380 elif isinstance(t, tuple) or isinstance(t, time.struct_time):
381 # Assume it's a time 9-tuple
382 return int(time.mktime(t))
383 elif hasattr(t, 'timetuple'):
384 # Looks like a datetime object or compatible
385 return int(time.mktime(t.timetuple()))
386 elif hasattr(t, 'strftime'):
387 # Looks like the object supports standard srftime()
388 return int(t.strftime('%s'))
389 elif isinstance(t, str) or isinstance(t, unicode):
390 # See if it's a string representation of an epoch
391 try:
392 return int(t)
393 except ValueError:
394 # Try to parse it as a timestamp string
395 try:
396 return time.strptime(t)
397 except ValueError, ex:
398 # Will fall through
399 debug("Failed to parse date with strptime: %s", ex)
400 pass
401 raise Exceptions.ParameterError('Unable to convert %r to an epoch time. Pass an epoch time. Try `date -d \'now + 1 year\' +%%s` (shell) or time.mktime (Python).' % t)
402
326403
327404 def check_bucket_name(bucket, dns_strict = True):
328405 if dns_strict:
381458 return Config.Config().host_bucket % { 'bucket' : bucket }
382459 __all__.append("getHostnameFromBucket")
383460
461
462 def calculateChecksum(buffer, mfile, offset, chunk_size, send_chunk):
463 md5_hash = md5()
464 size_left = chunk_size
465 if buffer == '':
466 mfile.seek(offset)
467 while size_left > 0:
468 data = mfile.read(min(send_chunk, size_left))
469 md5_hash.update(data)
470 size_left -= len(data)
471 else:
472 md5_hash.update(buffer)
473
474 return md5_hash.hexdigest()
475
476
477 __all__.append("calculateChecksum")
478
479
480 # Deal with the fact that pwd and grp modules don't exist for Windos
481 try:
482 import pwd
483 def getpwuid_username(uid):
484 """returns a username from the password databse for the given uid"""
485 return pwd.getpwuid(uid).pw_name
486 except ImportError:
487 def getpwuid_username(uid):
488 return getpass.getuser()
489 __all__.append("getpwuid_username")
490
491 try:
492 import grp
493 def getgrgid_grpname(gid):
494 """returns a groupname from the group databse for the given gid"""
495 return grp.getgrgid(gid).gr_name
496 except ImportError:
497 def getgrgid_grpname(gid):
498 return "nobody"
499
500 __all__.append("getgrgid_grpname")
501
502
503
384504 # vim:et:ts=4:sts=4:ai
505
+52
-0
TODO less more
0 TODO list for s3cmd project
1 ===========================
2
3 - Before 1.0.0 (or asap after 1.0.0)
4 - Make 'sync s3://bkt/some-filename local/other-filename' work
5 (at the moment it'll always download).
6 - Enable --exclude for [ls].
7 - Allow change /tmp to somewhere else
8 - With --guess-mime use 'magic' module if available.
9 - Support --preserve for [put] and [get]. Update manpage.
10 - Don't let --continue fail if the file is already fully downloaded.
11 - Option --mime-type should set mime type with 'cp' and 'mv'.
12 If possible --guess-mime-type should do as well.
13 - Make upload throttling configurable.
14 - Allow removing 'DefaultRootObject' from CloudFront distributions.
15 - Get s3://bucket/non-existent creates empty local file 'non-existent'
16 - Add 'geturl' command, both Unicode and urlencoded output.
17 - Add a command for generating "Query String Authentication" URLs.
18 - Support --acl-grant (together with --acl-public/private) for [put] and [sync]
19 - Filter 's3cmd ls' output by --bucket-location=
20
21 - After 1.0.0
22 - Sync must backup non-files as well. At least directories,
23 symlinks and device nodes.
24 - Speed up upload / download with multiple threads.
25 (see http://blog.50projects.com/p/s3cmd-modifications.html)
26 - Sync should be able to update metadata (UID, timstamps, etc)
27 if only these change (i.e. same content, different metainfo).
28 - If GPG fails error() and exit. If un-GPG fails save the
29 file with .gpg extension.
30 - Keep backup files remotely on put/sync-to if requested
31 (move the old 'object' to e.g. 'object~' and only then upload
32 the new one). Could be more advanced to keep, say, last 5
33 copies, etc.
34 - Memory consumption on very large upload sets is terribly high.
35 - Implement per-bucket (or per-regexp?) default settings. For
36 example regarding ACLs, encryption, etc.
37
38 - Implement GPG for sync
39 (it's not that easy since it won't be easy to compare
40 the encrypted-remote-object size with local file.
41 either we can store the metadata in a dedicated file
42 where we face a risk of inconsistencies, or we'll store
43 the metadata encrypted in each object header where we'll
44 have to do large number for object/HEAD requests. tough
45 call).
46 Or we can only compare local timestamps with remote object
47 timestamps. If the local one is older we'll *assume* it
48 hasn't been changed. But what to do about remote2local sync?
49
50 - Keep man page up to date and write some more documentation
51 - Yeah, right ;-)
artwork/AtomicClockRadio.ttf less more
Binary diff not shown
artwork/TypeRa.ttf less more
Binary diff not shown
artwork/site-top-full-size.xcf less more
Binary diff not shown
artwork/site-top-label-download.png less more
Binary diff not shown
artwork/site-top-label-s3cmd.png less more
Binary diff not shown
artwork/site-top-label-s3sync.png less more
Binary diff not shown
artwork/site-top-s3tools-logo.png less more
Binary diff not shown
artwork/site-top.jpg less more
Binary diff not shown
artwork/site-top.png less more
Binary diff not shown
artwork/site-top.xcf less more
Binary diff not shown
+200
-0
format-manpage.pl less more
0 #!/usr/bin/perl
1
2 # Format s3cmd.1 manpage
3 # Usage:
4 # s3cmd --help | format-manpage.pl > s3cmd.1
5
6 use strict;
7
8 my $commands = "";
9 my $cfcommands = "";
10 my $wscommands = "";
11 my $options = "";
12
13 while (<>) {
14 if (/^Commands:/) {
15 while (<>) {
16 last if (/^\s*$/);
17 my ($desc, $cmd, $cmdline);
18 ($desc = $_) =~ s/^\s*(.*?)\s*$/$1/;
19 ($cmdline = <>) =~ s/^\s*s3cmd (.*?) (.*?)\s*$/s3cmd \\fB$1\\fR \\fI$2\\fR/;
20 $cmd = $1;
21 if ($cmd =~ /^cf/) {
22 $cfcommands .= ".TP\n$cmdline\n$desc\n";
23 } elsif ($cmd =~ /^ws/) {
24 $wscommands .= ".TP\n$cmdline\n$desc\n";
25 } else {
26 $commands .= ".TP\n$cmdline\n$desc\n";
27 }
28 }
29 }
30 if (/^Options:/) {
31 my ($opt, $desc);
32 while (<>) {
33 last if (/^\s*$/);
34 $_ =~ s/(.*?)\s*$/$1/;
35 $desc = "";
36 $opt = "";
37 if (/^ (-.*)/) {
38 $opt = $1;
39 if ($opt =~ / /) {
40 ($opt, $desc) = split(/\s\s+/, $opt, 2);
41 }
42 $opt =~ s/(-[^ ,=\.]+)/\\fB$1\\fR/g;
43 $opt =~ s/-/\\-/g;
44 $options .= ".TP\n$opt\n";
45 } else {
46 $_ =~ s/\s*(.*?)\s*$/$1/;
47 $_ =~ s/(--[^ ,=\.]+)/\\fB$1\\fR/g;
48 $desc .= $_;
49 }
50 if ($desc) {
51 $options .= "$desc\n";
52 }
53 }
54 }
55 }
56 print "
57 .\\\" !!! IMPORTANT: This file is generated from s3cmd --help output using format-manpage.pl
58 .\\\" !!! Do your changes either in s3cmd file or in 'format-manpage.pl' otherwise
59 .\\\" !!! they will be overwritten!
60
61 .TH s3cmd 1
62 .SH NAME
63 s3cmd \\- tool for managing Amazon S3 storage space and Amazon CloudFront content delivery network
64 .SH SYNOPSIS
65 .B s3cmd
66 [\\fIOPTIONS\\fR] \\fICOMMAND\\fR [\\fIPARAMETERS\\fR]
67 .SH DESCRIPTION
68 .PP
69 .B s3cmd
70 is a command line client for copying files to/from
71 Amazon S3 (Simple Storage Service) and performing other
72 related tasks, for instance creating and removing buckets,
73 listing objects, etc.
74
75 .SH COMMANDS
76 .PP
77 .B s3cmd
78 can do several \\fIactions\\fR specified by the following \\fIcommands\\fR.
79 $commands
80
81 .PP
82 Commands for static WebSites configuration
83 $wscommands
84
85 .PP
86 Commands for CloudFront management
87 $cfcommands
88
89 .SH OPTIONS
90 .PP
91 Some of the below specified options can have their default
92 values set in
93 .B s3cmd
94 config file (by default \$HOME/.s3cmd). As it's a simple text file
95 feel free to open it with your favorite text editor and do any
96 changes you like.
97 $options
98
99 .SH EXAMPLES
100 One of the most powerful commands of \\fIs3cmd\\fR is \\fBs3cmd sync\\fR used for
101 synchronising complete directory trees to or from remote S3 storage. To some extent
102 \\fBs3cmd put\\fR and \\fBs3cmd get\\fR share a similar behaviour with \\fBsync\\fR.
103 .PP
104 Basic usage common in backup scenarios is as simple as:
105 .nf
106 s3cmd sync /local/path/ s3://test-bucket/backup/
107 .fi
108 .PP
109 This command will find all files under /local/path directory and copy them
110 to corresponding paths under s3://test-bucket/backup on the remote side.
111 For example:
112 .nf
113 /local/path/\\fBfile1.ext\\fR \\-> s3://bucket/backup/\\fBfile1.ext\\fR
114 /local/path/\\fBdir123/file2.bin\\fR \\-> s3://bucket/backup/\\fBdir123/file2.bin\\fR
115 .fi
116 .PP
117 However if the local path doesn't end with a slash the last directory's name
118 is used on the remote side as well. Compare these with the previous example:
119 .nf
120 s3cmd sync /local/path s3://test-bucket/backup/
121 .fi
122 will sync:
123 .nf
124 /local/\\fBpath/file1.ext\\fR \\-> s3://bucket/backup/\\fBpath/file1.ext\\fR
125 /local/\\fBpath/dir123/file2.bin\\fR \\-> s3://bucket/backup/\\fBpath/dir123/file2.bin\\fR
126 .fi
127 .PP
128 To retrieve the files back from S3 use inverted syntax:
129 .nf
130 s3cmd sync s3://test-bucket/backup/ /tmp/restore/
131 .fi
132 that will download files:
133 .nf
134 s3://bucket/backup/\\fBfile1.ext\\fR \\-> /tmp/restore/\\fBfile1.ext\\fR
135 s3://bucket/backup/\\fBdir123/file2.bin\\fR \\-> /tmp/restore/\\fBdir123/file2.bin\\fR
136 .fi
137 .PP
138 Without the trailing slash on source the behaviour is similar to
139 what has been demonstrated with upload:
140 .nf
141 s3cmd sync s3://test-bucket/backup /tmp/restore/
142 .fi
143 will download the files as:
144 .nf
145 s3://bucket/\\fBbackup/file1.ext\\fR \\-> /tmp/restore/\\fBbackup/file1.ext\\fR
146 s3://bucket/\\fBbackup/dir123/file2.bin\\fR \\-> /tmp/restore/\\fBbackup/dir123/file2.bin\\fR
147 .fi
148 .PP
149 All source file names, the bold ones above, are matched against \\fBexclude\\fR
150 rules and those that match are then re\\-checked against \\fBinclude\\fR rules to see
151 whether they should be excluded or kept in the source list.
152 .PP
153 For the purpose of \\fB\\-\\-exclude\\fR and \\fB\\-\\-include\\fR matching only the
154 bold file names above are used. For instance only \\fBpath/file1.ext\\fR is tested
155 against the patterns, not \\fI/local/\\fBpath/file1.ext\\fR
156 .PP
157 Both \\fB\\-\\-exclude\\fR and \\fB\\-\\-include\\fR work with shell-style wildcards (a.k.a. GLOB).
158 For a greater flexibility s3cmd provides Regular-expression versions of the two exclude options
159 named \\fB\\-\\-rexclude\\fR and \\fB\\-\\-rinclude\\fR.
160 The options with ...\\fB\\-from\\fR suffix (eg \\-\\-rinclude\\-from) expect a filename as
161 an argument. Each line of such a file is treated as one pattern.
162 .PP
163 There is only one set of patterns built from all \\fB\\-\\-(r)exclude(\\-from)\\fR options
164 and similarly for include variant. Any file excluded with eg \\-\\-exclude can
165 be put back with a pattern found in \\-\\-rinclude\\-from list.
166 .PP
167 Run s3cmd with \\fB\\-\\-dry\\-run\\fR to verify that your rules work as expected.
168 Use together with \\fB\\-\\-debug\\fR get detailed information
169 about matching file names against exclude and include rules.
170 .PP
171 For example to exclude all files with \".jpg\" extension except those beginning with a number use:
172 .PP
173 \\-\\-exclude '*.jpg' \\-\\-rinclude '[0-9].*\\.jpg'
174 .SH SEE ALSO
175 For the most up to date list of options run
176 .B s3cmd \\-\\-help
177 .br
178 For more info about usage, examples and other related info visit project homepage at
179 .br
180 .B http://s3tools.org
181 .SH DONATIONS
182 Please consider a donation if you have found s3cmd useful:
183 .br
184 .B http://s3tools.org/donate
185 .SH AUTHOR
186 Written by Michal Ludvig <mludvig\@logix.net.nz> and 15+ contributors
187 .SH CONTACT, SUPPORT
188 Preferred way to get support is our mailing list:
189 .I s3tools\\-general\@lists.sourceforge.net
190 .SH REPORTING BUGS
191 Report bugs to
192 .I s3tools\\-bugs\@lists.sourceforge.net
193 .SH COPYRIGHT
194 Copyright \\(co 2007,2008,2009,2010,2011,2012 Michal Ludvig <http://www.logix.cz/michal>
195 .br
196 This is free software. You may redistribute copies of it under the terms of
197 the GNU General Public License version 2 <http://www.gnu.org/licenses/gpl.html>.
198 There is NO WARRANTY, to the extent permitted by law.
199 ";
+63
-0
magic less more
0 # Additional magic for common web file types
1
2 0 string/b {\ " JSON data
3 !:mime application/json
4 0 string/b {\ } JSON data
5 !:mime application/json
6 0 string/b [ JSON data
7 !:mime application/json
8
9 0 search/4000 function
10 >&0 search/32/b )\ { JavaScript program
11 !:mime application/javascript
12
13 0 search/4000 @media CSS stylesheet
14 !:mime text/css
15 0 search/4000 @import CSS stylesheet
16 !:mime text/css
17 0 search/4000 @namespace CSS stylesheet
18 !:mime text/css
19 0 search/4000/b {\ background CSS stylesheet
20 !:mime text/css
21 0 search/4000/b {\ border CSS stylesheet
22 !:mime text/css
23 0 search/4000/b {\ bottom CSS stylesheet
24 !:mime text/css
25 0 search/4000/b {\ color CSS stylesheet
26 !:mime text/css
27 0 search/4000/b {\ cursor CSS stylesheet
28 !:mime text/css
29 0 search/4000/b {\ direction CSS stylesheet
30 !:mime text/css
31 0 search/4000/b {\ display CSS stylesheet
32 !:mime text/css
33 0 search/4000/b {\ float CSS stylesheet
34 !:mime text/css
35 0 search/4000/b {\ font CSS stylesheet
36 !:mime text/css
37 0 search/4000/b {\ height CSS stylesheet
38 !:mime text/css
39 0 search/4000/b {\ left CSS stylesheet
40 !:mime text/css
41 0 search/4000/b {\ line- CSS stylesheet
42 !:mime text/css
43 0 search/4000/b {\ margin CSS stylesheet
44 !:mime text/css
45 0 search/4000/b {\ padding CSS stylesheet
46 !:mime text/css
47 0 search/4000/b {\ position CSS stylesheet
48 !:mime text/css
49 0 search/4000/b {\ right CSS stylesheet
50 !:mime text/css
51 0 search/4000/b {\ text- CSS stylesheet
52 !:mime text/css
53 0 search/4000/b {\ top CSS stylesheet
54 !:mime text/css
55 0 search/4000/b {\ width CSS stylesheet
56 !:mime text/css
57 0 search/4000/b {\ visibility CSS stylesheet
58 !:mime text/css
59 0 search/4000/b {\ -moz- CSS stylesheet
60 !:mime text/css
61 0 search/4000/b {\ -webkit- CSS stylesheet
62 !:mime text/css
+540
-0
run-tests.py less more
0 #!/usr/bin/env python
1 # -*- coding=utf-8 -*-
2
3 ## Amazon S3cmd - testsuite
4 ## Author: Michal Ludvig <michal@logix.cz>
5 ## http://www.logix.cz/michal
6 ## License: GPL Version 2
7
8 import sys
9 import os
10 import re
11 from subprocess import Popen, PIPE, STDOUT
12 import locale
13 import getpass
14
15 count_pass = 0
16 count_fail = 0
17 count_skip = 0
18
19 test_counter = 0
20 run_tests = []
21 exclude_tests = []
22
23 verbose = False
24
25 if os.name == "posix":
26 have_wget = True
27 elif os.name == "nt":
28 have_wget = False
29 else:
30 print "Unknown platform: %s" % os.name
31 sys.exit(1)
32
33 ## Unpack testsuite/ directory
34 if not os.path.isdir('testsuite') and os.path.isfile('testsuite.tar.gz'):
35 os.system("tar -xz -f testsuite.tar.gz")
36 if not os.path.isdir('testsuite'):
37 print "Something went wrong while unpacking testsuite.tar.gz"
38 sys.exit(1)
39
40 os.system("tar -xf testsuite/checksum.tar -C testsuite")
41 if not os.path.isfile('testsuite/checksum/cksum33.txt'):
42 print "Something went wrong while unpacking testsuite/checkum.tar"
43 sys.exit(1)
44
45 ## Fix up permissions for permission-denied tests
46 os.chmod("testsuite/permission-tests/permission-denied-dir", 0444)
47 os.chmod("testsuite/permission-tests/permission-denied.txt", 0000)
48
49 ## Patterns for Unicode tests
50 patterns = {}
51 patterns['UTF-8'] = u"ŪņЇЌœđЗ/☺ unicode € rocks ™"
52 patterns['GBK'] = u"12月31日/1-特色條目"
53
54 encoding = locale.getpreferredencoding()
55 if not encoding:
56 print "Guessing current system encoding failed. Consider setting $LANG variable."
57 sys.exit(1)
58 else:
59 print "System encoding: " + encoding
60
61 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
62 if not have_encoding and os.path.isfile('testsuite/encodings/%s.tar.gz' % encoding):
63 os.system("tar xvz -C testsuite/encodings -f testsuite/encodings/%s.tar.gz" % encoding)
64 have_encoding = os.path.isdir('testsuite/encodings/' + encoding)
65
66 if have_encoding:
67 #enc_base_remote = "%s/xyz/%s/" % (pbucket(1), encoding)
68 enc_pattern = patterns[encoding]
69 else:
70 print encoding + " specific files not found."
71
72 if not os.path.isdir('testsuite/crappy-file-name'):
73 os.system("tar xvz -C testsuite -f testsuite/crappy-file-name.tar.gz")
74 # TODO: also unpack if the tarball is newer than the directory timestamp
75 # for instance when a new version was pulled from SVN.
76
77 def test(label, cmd_args = [], retcode = 0, must_find = [], must_not_find = [], must_find_re = [], must_not_find_re = []):
78 def command_output():
79 print "----"
80 print " ".join([arg.find(" ")>=0 and "'%s'" % arg or arg for arg in cmd_args])
81 print "----"
82 print stdout
83 print "----"
84
85 def failure(message = ""):
86 global count_fail
87 if message:
88 message = " (%r)" % message
89 print "\x1b[31;1mFAIL%s\x1b[0m" % (message)
90 count_fail += 1
91 command_output()
92 #return 1
93 sys.exit(1)
94 def success(message = ""):
95 global count_pass
96 if message:
97 message = " (%r)" % message
98 print "\x1b[32;1mOK\x1b[0m%s" % (message)
99 count_pass += 1
100 if verbose:
101 command_output()
102 return 0
103 def skip(message = ""):
104 global count_skip
105 if message:
106 message = " (%r)" % message
107 print "\x1b[33;1mSKIP\x1b[0m%s" % (message)
108 count_skip += 1
109 return 0
110 def compile_list(_list, regexps = False):
111 if regexps == False:
112 _list = [re.escape(item.encode(encoding, "replace")) for item in _list]
113
114 return [re.compile(item, re.MULTILINE) for item in _list]
115
116 global test_counter
117 test_counter += 1
118 print ("%3d %s " % (test_counter, label)).ljust(30, "."),
119 sys.stdout.flush()
120
121 if run_tests.count(test_counter) == 0 or exclude_tests.count(test_counter) > 0:
122 return skip()
123
124 if not cmd_args:
125 return skip()
126
127 p = Popen(cmd_args, stdout = PIPE, stderr = STDOUT, universal_newlines = True)
128 stdout, stderr = p.communicate()
129 if retcode != p.returncode:
130 return failure("retcode: %d, expected: %d" % (p.returncode, retcode))
131
132 if type(must_find) not in [ list, tuple ]: must_find = [must_find]
133 if type(must_find_re) not in [ list, tuple ]: must_find_re = [must_find_re]
134 if type(must_not_find) not in [ list, tuple ]: must_not_find = [must_not_find]
135 if type(must_not_find_re) not in [ list, tuple ]: must_not_find_re = [must_not_find_re]
136
137 find_list = []
138 find_list.extend(compile_list(must_find))
139 find_list.extend(compile_list(must_find_re, regexps = True))
140 find_list_patterns = []
141 find_list_patterns.extend(must_find)
142 find_list_patterns.extend(must_find_re)
143
144 not_find_list = []
145 not_find_list.extend(compile_list(must_not_find))
146 not_find_list.extend(compile_list(must_not_find_re, regexps = True))
147 not_find_list_patterns = []
148 not_find_list_patterns.extend(must_not_find)
149 not_find_list_patterns.extend(must_not_find_re)
150
151 for index in range(len(find_list)):
152 match = find_list[index].search(stdout)
153 if not match:
154 return failure("pattern not found: %s" % find_list_patterns[index])
155 for index in range(len(not_find_list)):
156 match = not_find_list[index].search(stdout)
157 if match:
158 return failure("pattern found: %s (match: %s)" % (not_find_list_patterns[index], match.group(0)))
159
160 return success()
161
162 def test_s3cmd(label, cmd_args = [], **kwargs):
163 if not cmd_args[0].endswith("s3cmd"):
164 cmd_args.insert(0, "python")
165 cmd_args.insert(1, "s3cmd")
166
167 return test(label, cmd_args, **kwargs)
168
169 def test_mkdir(label, dir_name):
170 if os.name in ("posix", "nt"):
171 cmd = ['mkdir', '-p']
172 else:
173 print "Unknown platform: %s" % os.name
174 sys.exit(1)
175 cmd.append(dir_name)
176 return test(label, cmd)
177
178 def test_rmdir(label, dir_name):
179 if os.path.isdir(dir_name):
180 if os.name == "posix":
181 cmd = ['rm', '-rf']
182 elif os.name == "nt":
183 cmd = ['rmdir', '/s/q']
184 else:
185 print "Unknown platform: %s" % os.name
186 sys.exit(1)
187 cmd.append(dir_name)
188 return test(label, cmd)
189 else:
190 return test(label, [])
191
192 def test_flushdir(label, dir_name):
193 test_rmdir(label + "(rm)", dir_name)
194 return test_mkdir(label + "(mk)", dir_name)
195
196 def test_copy(label, src_file, dst_file):
197 if os.name == "posix":
198 cmd = ['cp', '-f']
199 elif os.name == "nt":
200 cmd = ['copy']
201 else:
202 print "Unknown platform: %s" % os.name
203 sys.exit(1)
204 cmd.append(src_file)
205 cmd.append(dst_file)
206 return test(label, cmd)
207
208 bucket_prefix = u"%s-" % getpass.getuser()
209 print "Using bucket prefix: '%s'" % bucket_prefix
210
211 argv = sys.argv[1:]
212 while argv:
213 arg = argv.pop(0)
214 if arg.startswith('--bucket-prefix='):
215 print "Usage: '--bucket-prefix PREFIX', not '--bucket-prefix=PREFIX'"
216 sys.exit(0)
217 if arg in ("-h", "--help"):
218 print "%s A B K..O -N" % sys.argv[0]
219 print "Run tests number A, B and K through to O, except for N"
220 sys.exit(0)
221 if arg in ("-l", "--list"):
222 exclude_tests = range(0, 999)
223 break
224 if arg in ("-v", "--verbose"):
225 verbose = True
226 continue
227 if arg in ("-p", "--bucket-prefix"):
228 try:
229 bucket_prefix = argv.pop(0)
230 except IndexError:
231 print "Bucket prefix option must explicitly supply a bucket name prefix"
232 sys.exit(0)
233 continue
234 if arg.find("..") >= 0:
235 range_idx = arg.find("..")
236 range_start = arg[:range_idx] or 0
237 range_end = arg[range_idx+2:] or 999
238 run_tests.extend(range(int(range_start), int(range_end) + 1))
239 elif arg.startswith("-"):
240 exclude_tests.append(int(arg[1:]))
241 else:
242 run_tests.append(int(arg))
243
244 if not run_tests:
245 run_tests = range(0, 999)
246
247 # helper functions for generating bucket names
248 def bucket(tail):
249 '''Test bucket name'''
250 label = 'autotest'
251 if str(tail) == '3':
252 label = 'Autotest'
253 return '%ss3cmd-%s-%s' % (bucket_prefix, label, tail)
254
255 def pbucket(tail):
256 '''Like bucket(), but prepends "s3://" for you'''
257 return 's3://' + bucket(tail)
258
259 ## ====== Remove test buckets
260 test_s3cmd("Remove test buckets", ['rb', '-r', pbucket(1), pbucket(2), pbucket(3)],
261 must_find = [ "Bucket '%s/' removed" % pbucket(1),
262 "Bucket '%s/' removed" % pbucket(2),
263 "Bucket '%s/' removed" % pbucket(3) ])
264
265
266 ## ====== Create one bucket (EU)
267 test_s3cmd("Create one bucket (EU)", ['mb', '--bucket-location=EU', pbucket(1)],
268 must_find = "Bucket '%s/' created" % pbucket(1))
269
270
271
272 ## ====== Create multiple buckets
273 test_s3cmd("Create multiple buckets", ['mb', pbucket(2), pbucket(3)],
274 must_find = [ "Bucket '%s/' created" % pbucket(2), "Bucket '%s/' created" % pbucket(3)])
275
276
277 ## ====== Invalid bucket name
278 test_s3cmd("Invalid bucket name", ["mb", "--bucket-location=EU", pbucket('EU')],
279 retcode = 1,
280 must_find = "ERROR: Parameter problem: Bucket name '%s' contains disallowed character" % bucket('EU'),
281 must_not_find_re = "Bucket.*created")
282
283
284 ## ====== Buckets list
285 test_s3cmd("Buckets list", ["ls"],
286 must_find = [ "autotest-1", "autotest-2", "Autotest-3" ], must_not_find_re = "autotest-EU")
287
288
289 ## ====== Sync to S3
290 test_s3cmd("Sync to S3", ['sync', 'testsuite/', pbucket(1) + '/xyz/', '--exclude', 'demo/*', '--exclude', '*.png', '--no-encrypt', '--exclude-from', 'testsuite/exclude.encodings' ],
291 must_find = [ "WARNING: 32 non-printable characters replaced in: crappy-file-name/non-printables ^A^B^C^D^E^F^G^H^I^J^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_^? +-[\]^<>%%\"'#{}`&?.end",
292 "WARNING: File can not be uploaded: testsuite/permission-tests/permission-denied.txt: Permission denied",
293 "stored as '%s/xyz/crappy-file-name/non-printables ^A^B^C^D^E^F^G^H^I^J^K^L^M^N^O^P^Q^R^S^T^U^V^W^X^Y^Z^[^\^]^^^_^? +-[\\]^<>%%%%\"'#{}`&?.end'" % pbucket(1) ],
294 must_not_find_re = [ "demo/", "\.png$", "permission-denied-dir" ])
295
296 if have_encoding:
297 ## ====== Sync UTF-8 / GBK / ... to S3
298 test_s3cmd("Sync %s to S3" % encoding, ['sync', 'testsuite/encodings/' + encoding, '%s/xyz/encodings/' % pbucket(1), '--exclude', 'demo/*', '--no-encrypt' ],
299 must_find = [ u"File 'testsuite/encodings/%(encoding)s/%(pattern)s' stored as '%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s'" % { 'encoding' : encoding, 'pattern' : enc_pattern , 'pbucket' : pbucket(1)} ])
300
301
302 ## ====== List bucket content
303 test_s3cmd("List bucket content", ['ls', '%s/xyz/' % pbucket(1) ],
304 must_find_re = [ u"DIR %s/xyz/binary/$" % pbucket(1) , u"DIR %s/xyz/etc/$" % pbucket(1) ],
305 must_not_find = [ u"random-crap.md5", u"/demo" ])
306
307
308 ## ====== List bucket recursive
309 must_find = [ u"%s/xyz/binary/random-crap.md5" % pbucket(1) ]
310 if have_encoding:
311 must_find.append(u"%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s" % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
312
313 test_s3cmd("List bucket recursive", ['ls', '--recursive', pbucket(1)],
314 must_find = must_find,
315 must_not_find = [ "logo.png" ])
316
317 ## ====== FIXME
318 # test_s3cmd("Recursive put", ['put', '--recursive', 'testsuite/etc', '%s/xyz/' % pbucket(1) ])
319
320
321 ## ====== Clean up local destination dir
322 test_flushdir("Clean testsuite-out/", "testsuite-out")
323
324
325 ## ====== Sync from S3
326 must_find = [ "File '%s/xyz/binary/random-crap.md5' stored as 'testsuite-out/xyz/binary/random-crap.md5'" % pbucket(1) ]
327 if have_encoding:
328 must_find.append(u"File '%(pbucket)s/xyz/encodings/%(encoding)s/%(pattern)s' stored as 'testsuite-out/xyz/encodings/%(encoding)s/%(pattern)s' " % { 'encoding' : encoding, 'pattern' : enc_pattern, 'pbucket' : pbucket(1) })
329 test_s3cmd("Sync from S3", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
330 must_find = must_find)
331
332
333 ## ====== Remove 'demo' directory
334 test_rmdir("Remove 'dir-test/'", "testsuite-out/xyz/dir-test/")
335
336
337 ## ====== Create dir with name of a file
338 test_mkdir("Create file-dir dir", "testsuite-out/xyz/dir-test/file-dir")
339
340
341 ## ====== Skip dst dirs
342 test_s3cmd("Skip over dir", ['sync', '%s/xyz' % pbucket(1), 'testsuite-out'],
343 must_find = "WARNING: testsuite-out/xyz/dir-test/file-dir is a directory - skipping over")
344
345
346 ## ====== Clean up local destination dir
347 test_flushdir("Clean testsuite-out/", "testsuite-out")
348
349
350 ## ====== Put public, guess MIME
351 test_s3cmd("Put public, guess MIME", ['put', '--guess-mime-type', '--acl-public', 'testsuite/etc/logo.png', '%s/xyz/etc/logo.png' % pbucket(1)],
352 must_find = [ "stored as '%s/xyz/etc/logo.png'" % pbucket(1) ])
353
354
355 ## ====== Retrieve from URL
356 if have_wget:
357 test("Retrieve from URL", ['wget', '-O', 'testsuite-out/logo.png', 'http://%s.s3.amazonaws.com/xyz/etc/logo.png' % bucket(1)],
358 must_find_re = [ 'logo.png.*saved \[22059/22059\]' ])
359
360
361 ## ====== Change ACL to Private
362 test_s3cmd("Change ACL to Private", ['setacl', '--acl-private', '%s/xyz/etc/l*.png' % pbucket(1)],
363 must_find = [ "logo.png: ACL set to Private" ])
364
365
366 ## ====== Verify Private ACL
367 if have_wget:
368 test("Verify Private ACL", ['wget', '-O', 'testsuite-out/logo.png', 'http://%s.s3.amazonaws.com/xyz/etc/logo.png' % bucket(1)],
369 retcode = 8,
370 must_find_re = [ 'ERROR 403: Forbidden' ])
371
372
373 ## ====== Change ACL to Public
374 test_s3cmd("Change ACL to Public", ['setacl', '--acl-public', '--recursive', '%s/xyz/etc/' % pbucket(1) , '-v'],
375 must_find = [ "logo.png: ACL set to Public" ])
376
377
378 ## ====== Verify Public ACL
379 if have_wget:
380 test("Verify Public ACL", ['wget', '-O', 'testsuite-out/logo.png', 'http://%s.s3.amazonaws.com/xyz/etc/logo.png' % bucket(1)],
381 must_find_re = [ 'logo.png.*saved \[22059/22059\]' ])
382
383
384 ## ====== Sync more to S3
385 test_s3cmd("Sync more to S3", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt' ],
386 must_find = [ "File 'testsuite/demo/some-file.xml' stored as '%s/xyz/demo/some-file.xml' " % pbucket(1) ],
387 must_not_find = [ "File 'testsuite/etc/linked.png' stored as '%s/xyz/etc/linked.png" % pbucket(1) ])
388
389
390 ## ====== Don't check MD5 sum on Sync
391 test_copy("Change file cksum1.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum1.txt")
392 test_copy("Change file cksum33.txt", "testsuite/checksum/cksum2.txt", "testsuite/checksum/cksum33.txt")
393 test_s3cmd("Don't check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--no-check-md5'],
394 must_find = [ "cksum33.txt" ],
395 must_not_find = [ "cksum1.txt" ])
396
397
398 ## ====== Check MD5 sum on Sync
399 test_s3cmd("Check MD5", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--check-md5'],
400 must_find = [ "cksum1.txt" ])
401
402
403 ## ====== Rename within S3
404 test_s3cmd("Rename within S3", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
405 must_find = [ 'File %s/xyz/etc/logo.png moved to %s/xyz/etc2/Logo.PNG' % (pbucket(1), pbucket(1))])
406
407
408 ## ====== Rename (NoSuchKey)
409 test_s3cmd("Rename (NoSuchKey)", ['mv', '%s/xyz/etc/logo.png' % pbucket(1), '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
410 retcode = 1,
411 must_find_re = [ 'ERROR:.*NoSuchKey' ],
412 must_not_find = [ 'File %s/xyz/etc/logo.png moved to %s/xyz/etc2/Logo.PNG' % (pbucket(1), pbucket(1)) ])
413
414 ## ====== Sync more from S3 (invalid src)
415 test_s3cmd("Sync more from S3 (invalid src)", ['sync', '--delete-removed', '%s/xyz/DOESNOTEXIST' % pbucket(1), 'testsuite-out'],
416 must_not_find = [ "deleted: testsuite-out/logo.png" ])
417
418 ## ====== Sync more from S3
419 test_s3cmd("Sync more from S3", ['sync', '--delete-removed', '%s/xyz' % pbucket(1), 'testsuite-out'],
420 must_find = [ "deleted: testsuite-out/logo.png",
421 "File '%s/xyz/etc2/Logo.PNG' stored as 'testsuite-out/xyz/etc2/Logo.PNG' (22059 bytes" % pbucket(1),
422 "File '%s/xyz/demo/some-file.xml' stored as 'testsuite-out/xyz/demo/some-file.xml' " % pbucket(1) ],
423 must_not_find_re = [ "not-deleted.*etc/logo.png" ])
424
425
426 ## ====== Make dst dir for get
427 test_rmdir("Remove dst dir for get", "testsuite-out")
428
429
430 ## ====== Get multiple files
431 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
432 retcode = 1,
433 must_find = [ 'Destination must be a directory or stdout when downloading multiple sources.' ])
434
435
436 ## ====== Make dst dir for get
437 test_mkdir("Make dst dir for get", "testsuite-out")
438
439
440 ## ====== Get multiple files
441 test_s3cmd("Get multiple files", ['get', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc/AtomicClockRadio.ttf' % pbucket(1), 'testsuite-out'],
442 must_find = [ u"saved as 'testsuite-out/Logo.PNG'", u"saved as 'testsuite-out/AtomicClockRadio.ttf'" ])
443
444 ## ====== Upload files differing in capitalisation
445 test_s3cmd("blah.txt / Blah.txt", ['put', '-r', 'testsuite/blahBlah', pbucket(1)],
446 must_find = [ '%s/blahBlah/Blah.txt' % pbucket(1), '%s/blahBlah/blah.txt' % pbucket(1)])
447
448 ## ====== Copy between buckets
449 test_s3cmd("Copy between buckets", ['cp', '%s/xyz/etc2/Logo.PNG' % pbucket(1), '%s/xyz/etc2/logo.png' % pbucket(3)],
450 must_find = [ "File %s/xyz/etc2/Logo.PNG copied to %s/xyz/etc2/logo.png" % (pbucket(1), pbucket(3)) ])
451
452 ## ====== Recursive copy
453 test_s3cmd("Recursive copy, set ACL", ['cp', '-r', '--acl-public', '%s/xyz/' % pbucket(1), '%s/copy' % pbucket(2), '--exclude', 'demo/dir?/*.txt', '--exclude', 'non-printables*'],
454 must_find = [ "File %s/xyz/etc2/Logo.PNG copied to %s/copy/etc2/Logo.PNG" % (pbucket(1), pbucket(2)),
455 "File %s/xyz/blahBlah/Blah.txt copied to %s/copy/blahBlah/Blah.txt" % (pbucket(1), pbucket(2)),
456 "File %s/xyz/blahBlah/blah.txt copied to %s/copy/blahBlah/blah.txt" % (pbucket(1), pbucket(2)) ],
457 must_not_find = [ "demo/dir1/file1-1.txt" ])
458
459 ## ====== Verify ACL and MIME type
460 test_s3cmd("Verify ACL and MIME type", ['info', '%s/copy/etc2/Logo.PNG' % pbucket(2) ],
461 must_find_re = [ "MIME type:.*image/png",
462 "ACL:.*\*anon\*: READ",
463 "URL:.*http://%s.s3.amazonaws.com/copy/etc2/Logo.PNG" % bucket(2) ])
464
465 ## ====== Rename within S3
466 test_s3cmd("Rename within S3", ['mv', '%s/copy/etc2/Logo.PNG' % pbucket(2), '%s/copy/etc/logo.png' % pbucket(2)],
467 must_find = [ 'File %s/copy/etc2/Logo.PNG moved to %s/copy/etc/logo.png' % (pbucket(2), pbucket(2))])
468
469 ## ====== Sync between buckets
470 test_s3cmd("Sync remote2remote", ['sync', '%s/xyz/' % pbucket(1), '%s/copy/' % pbucket(2), '--delete-removed', '--exclude', 'non-printables*'],
471 must_find = [ "File %s/xyz/demo/dir1/file1-1.txt copied to %s/copy/demo/dir1/file1-1.txt" % (pbucket(1), pbucket(2)),
472 "remote copy: etc/logo.png -> etc2/Logo.PNG",
473 "deleted: '%s/copy/etc/logo.png'" % pbucket(2) ],
474 must_not_find = [ "blah.txt" ])
475
476 ## ====== Don't Put symbolic link
477 test_s3cmd("Don't put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1),],
478 must_not_find_re = [ "linked1.png"])
479
480 ## ====== Put symbolic link
481 test_s3cmd("Put symbolic links", ['put', 'testsuite/etc/linked1.png', 's3://%s/xyz/' % bucket(1),'--follow-symlinks' ],
482 must_find = [ "File 'testsuite/etc/linked1.png' stored as '%s/xyz/linked1.png'" % pbucket(1)])
483
484 ## ====== Sync symbolic links
485 test_s3cmd("Sync symbolic links", ['sync', 'testsuite/', 's3://%s/xyz/' % bucket(1), '--no-encrypt', '--follow-symlinks' ],
486 must_find = ["remote copy: etc2/Logo.PNG -> etc/linked.png"],
487 # Don't want to recursively copy linked directories!
488 must_not_find_re = ["etc/more/linked-dir/more/give-me-more.txt",
489 "etc/brokenlink.png"],
490 )
491
492 ## ====== Multi source move
493 test_s3cmd("Multi-source move", ['mv', '-r', '%s/copy/blahBlah/Blah.txt' % pbucket(2), '%s/copy/etc/' % pbucket(2), '%s/moved/' % pbucket(2)],
494 must_find = [ "File %s/copy/blahBlah/Blah.txt moved to %s/moved/Blah.txt" % (pbucket(2), pbucket(2)),
495 "File %s/copy/etc/AtomicClockRadio.ttf moved to %s/moved/AtomicClockRadio.ttf" % (pbucket(2), pbucket(2)),
496 "File %s/copy/etc/TypeRa.ttf moved to %s/moved/TypeRa.ttf" % (pbucket(2), pbucket(2)) ],
497 must_not_find = [ "blah.txt" ])
498
499 ## ====== Verify move
500 test_s3cmd("Verify move", ['ls', '-r', pbucket(2)],
501 must_find = [ "%s/moved/Blah.txt" % pbucket(2),
502 "%s/moved/AtomicClockRadio.ttf" % pbucket(2),
503 "%s/moved/TypeRa.ttf" % pbucket(2),
504 "%s/copy/blahBlah/blah.txt" % pbucket(2) ],
505 must_not_find = [ "%s/copy/blahBlah/Blah.txt" % pbucket(2),
506 "%s/copy/etc/AtomicClockRadio.ttf" % pbucket(2),
507 "%s/copy/etc/TypeRa.ttf" % pbucket(2) ])
508
509 ## ====== Simple delete
510 test_s3cmd("Simple delete", ['del', '%s/xyz/etc2/Logo.PNG' % pbucket(1)],
511 must_find = [ "File %s/xyz/etc2/Logo.PNG deleted" % pbucket(1) ])
512
513
514 ## ====== Recursive delete maximum exceeed
515 test_s3cmd("Recursive delete maximum exceeded", ['del', '--recursive', '--max-delete=1', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
516 must_not_find = [ "File %s/xyz/etc/TypeRa.ttf deleted" % pbucket(1) ])
517
518 ## ====== Recursive delete
519 test_s3cmd("Recursive delete", ['del', '--recursive', '--exclude', 'Atomic*', '%s/xyz/etc' % pbucket(1)],
520 must_find = [ "File %s/xyz/etc/TypeRa.ttf deleted" % pbucket(1) ],
521 must_find_re = [ "File .*/etc/logo.png deleted" ],
522 must_not_find = [ "AtomicClockRadio.ttf" ])
523
524 ## ====== Recursive delete all
525 test_s3cmd("Recursive delete all", ['del', '--recursive', '--force', pbucket(1)],
526 must_find_re = [ "File .*binary/random-crap deleted" ])
527
528
529 ## ====== Remove empty bucket
530 test_s3cmd("Remove empty bucket", ['rb', pbucket(1)],
531 must_find = [ "Bucket '%s/' removed" % pbucket(1) ])
532
533
534 ## ====== Remove remaining buckets
535 test_s3cmd("Remove remaining buckets", ['rb', '--recursive', pbucket(2), pbucket(3)],
536 must_find = [ "Bucket '%s/' removed" % pbucket(2),
537 "Bucket '%s/' removed" % pbucket(3) ])
538
539 # vim:et:ts=4:sts=4:ai
+940
-394
s3cmd less more
0 #!/usr/bin/python
0 #!/usr/bin/env python
11
22 ## Amazon S3 manager
33 ## Author: Michal Ludvig <michal@logix.cz>
2222 import subprocess
2323 import htmlentitydefs
2424 import socket
25 import shutil
26 import tempfile
2527
2628 from copy import copy
2729 from optparse import OptionParser, Option, OptionValueError, IndentedHelpFormatter
3032
3133 def output(message):
3234 sys.stdout.write(message + "\n")
35 sys.stdout.flush()
3336
3437 def check_args_type(args, type, verbose_type):
3538 for arg in args:
6467
6568 if object.endswith('*'):
6669 object = object[:-1]
67 try:
68 response = s3.bucket_list(bucket, prefix = object, recursive = True)
69 except S3Error, e:
70 if S3.codes.has_key(e.info["Code"]):
71 error(S3.codes[e.info["Code"]] % bucket)
72 return
73 else:
74 raise
70
7571 bucket_size = 0
76 for object in response["list"]:
77 size, size_coeff = formatSize(object["Size"], False)
78 bucket_size += size
72 # iterate and store directories to traverse, while summing objects:
73 dirs = [object]
74 while dirs:
75 try:
76 response = s3.bucket_list(bucket, prefix=dirs.pop())
77 except S3Error, e:
78 if S3.codes.has_key(e.info["Code"]):
79 error(S3.codes[e.info["Code"]] % bucket)
80 return
81 else:
82 raise
83
84 # objects in the current scope:
85 for obj in response["list"]:
86 bucket_size += int(obj["Size"])
87
88 # directories found in current scope:
89 for obj in response["common_prefixes"]:
90 dirs.append(obj["Prefix"])
91
7992 total_size, size_coeff = formatSize(bucket_size, Config().human_readable_sizes)
8093 total_size_str = str(total_size) + size_coeff
8194 output(u"%s %s" % (total_size_str.ljust(8), uri))
138151 "uri": uri.compose_uri(bucket, prefix["Prefix"])})
139152
140153 for object in response["list"]:
154 md5 = object['ETag'].strip('"')
155 if cfg.list_md5:
156 if md5.find('-') >= 0: # need to get md5 from the object
157 object_uri = uri.compose_uri(bucket, object["Key"])
158 info_response = s3.object_info(S3Uri(object_uri))
159 try:
160 md5 = info_response['s3cmd-attrs']['md5']
161 except KeyError:
162 pass
163
141164 size, size_coeff = formatSize(object["Size"], Config().human_readable_sizes)
142165 output(format_string % {
143166 "timestamp": formatDateTime(object["LastModified"]),
144167 "size" : str(size),
145168 "coeff": size_coeff,
146 "md5" : object['ETag'].strip('"'),
169 "md5" : md5,
147170 "uri": uri.compose_uri(bucket, object["Key"]),
148171 })
149172
256279 if len(args) == 0:
257280 raise ParameterError("Nothing to upload. Expecting a local file or directory.")
258281
259 local_list, single_file_local = fetch_local_list(args)
282 local_list, single_file_local = fetch_local_list(args, is_src = True)
260283
261284 local_list, exclude_list = filter_exclude_include(local_list)
262285
265288 info(u"Summary: %d local files to upload" % local_count)
266289
267290 if local_count > 0:
268 if not destination_base.endswith("/"):
291 if not single_file_local and '-' in local_list.keys():
292 raise ParameterError("Cannot specify multiple local files if uploading from '-' (ie stdin)")
293 elif single_file_local and local_list.keys()[0] == "-" and destination_base.endswith("/"):
294 raise ParameterError("Destination S3 URI must not end with '/' when uploading from stdin.")
295 elif not destination_base.endswith("/"):
269296 if not single_file_local:
270297 raise ParameterError("Destination S3 URI must end with '/' (ie must refer to a directory on the remote side).")
271298 local_list[local_list.keys()[0]]['remote_uri'] = unicodise(destination_base)
277304 for key in exclude_list:
278305 output(u"exclude: %s" % unicodise(key))
279306 for key in local_list:
280 output(u"upload: %s -> %s" % (local_list[key]['full_name_unicode'], local_list[key]['remote_uri']))
281
282 warning(u"Exitting now because of --dry-run")
307 if key != "-":
308 nicekey = local_list[key]['full_name_unicode']
309 else:
310 nicekey = "<stdin>"
311 output(u"upload: %s -> %s" % (nicekey, local_list[key]['remote_uri']))
312
313 warning(u"Exiting now because of --dry-run")
283314 return
284315
285316 seq = 0
294325 seq_label = "[%d of %d]" % (seq, local_count)
295326 if Config().encrypt:
296327 exitcode, full_name, extra_headers["x-amz-meta-s3tools-gpgenc"] = gpg_encrypt(full_name_orig)
328 if cfg.preserve_attrs or local_list[key]['size'] > (cfg.multipart_chunk_size_mb * 1024 * 1024):
329 attr_header = _build_attr_header(local_list, key)
330 debug(u"attr_header: %s" % attr_header)
331 extra_headers.update(attr_header)
297332 try:
298333 response = s3.object_put(full_name, uri_final, extra_headers, extra_label = seq_label)
299334 except S3UploadError, e:
302337 except InvalidFileError, e:
303338 warning(u"File can not be uploaded: %s" % e)
304339 continue
305 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
306 if not Config().progress_meter:
307 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
308 (unicodise(full_name_orig), uri_final, response["size"], response["elapsed"],
309 speed_fmt[0], speed_fmt[1], seq_label))
340 if response is not None:
341 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
342 if not Config().progress_meter:
343 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
344 (unicodise(full_name_orig), uri_final, response["size"], response["elapsed"],
345 speed_fmt[0], speed_fmt[1], seq_label))
310346 if Config().acl_public:
311347 output(u"Public URL of the object is: %s" %
312 (uri_final.public_url()))
348 (uri_final.public_url()))
313349 if Config().encrypt and full_name != full_name_orig:
314350 debug(u"Removing temporary encrypted file: %s" % unicodise(full_name))
315351 os.remove(full_name)
392428 for key in remote_list:
393429 output(u"download: %s -> %s" % (remote_list[key]['object_uri_str'], remote_list[key]['local_filename']))
394430
395 warning(u"Exitting now because of --dry-run")
431 warning(u"Exiting now because of --dry-run")
396432 return
397433
398434 seq = 0
409445 if destination == "-":
410446 ## stdout
411447 dst_stream = sys.__stdout__
448 file_exists = True
412449 else:
413450 ## File
414451 try:
439476 except IOError, e:
440477 error(u"Skipping %s: %s" % (destination, e.strerror))
441478 continue
442 response = s3.object_get(uri, dst_stream, start_position = start_position, extra_label = seq_label)
479 try:
480 response = s3.object_get(uri, dst_stream, start_position = start_position, extra_label = seq_label)
481 except S3Error, e:
482 if not file_exists: # Delete, only if file didn't exist before!
483 debug(u"object_get failed for '%s', deleting..." % (destination,))
484 os.unlink(destination)
485 raise
486
443487 if response["headers"].has_key("x-amz-meta-s3tools-gpgenc"):
444488 gpg_decrypt(destination, response["headers"]["x-amz-meta-s3tools-gpgenc"])
445489 response["size"] = os.stat(destination)[6]
447491 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
448492 output(u"File %s saved as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s)" %
449493 (uri, destination, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1]))
494 if Config().delete_after_fetch:
495 s3.object_delete(uri)
496 output(u"File %s removed after fetch" % (uri))
450497
451498 def cmd_object_del(args):
452499 for uri_str in args:
472519 remote_count = len(remote_list)
473520
474521 info(u"Summary: %d remote files to delete" % remote_count)
522 if cfg.max_delete > 0 and remote_count > cfg.max_delete:
523 warning(u"delete: maximum requested number of deletes would be exceeded, none performed.")
524 return
475525
476526 if cfg.dry_run:
477527 for key in exclude_list:
479529 for key in remote_list:
480530 output(u"delete: %s" % remote_list[key]['object_uri_str'])
481531
482 warning(u"Exitting now because of --dry-run")
532 warning(u"Exiting now because of --dry-run")
483533 return
484534
485535 for key in remote_list:
486536 item = remote_list[key]
487537 response = s3.object_delete(S3Uri(item['object_uri_str']))
488538 output(u"File %s deleted" % item['object_uri_str'])
539
540 def cmd_object_restore(args):
541 s3 = S3(cfg)
542
543 if cfg.restore_days < 1:
544 raise ParameterError("You must restore a file for 1 or more days")
545
546 remote_list = fetch_remote_list(args, require_attribs = False, recursive = cfg.recursive)
547 remote_list, exclude_list = filter_exclude_include(remote_list)
548
549 remote_count = len(remote_list)
550
551 info(u"Summary: Restoring %d remote files for %d days" % (remote_count, cfg.restore_days))
552
553 if cfg.dry_run:
554 for key in exclude_list:
555 output(u"exclude: %s" % unicodise(key))
556 for key in remote_list:
557 output(u"restore: %s" % remote_list[key]['object_uri_str'])
558
559 warning(u"Exiting now because of --dry-run")
560 return
561
562 for key in remote_list:
563 item = remote_list[key]
564
565 uri = S3Uri(item['object_uri_str'])
566 if not item['object_uri_str'].endswith("/"):
567 response = s3.object_restore(S3Uri(item['object_uri_str']))
568 output(u"File %s restoration started" % item['object_uri_str'])
569 else:
570 debug(u"Skipping directory since only files may be restored")
571
489572
490573 def subcmd_cp_mv(args, process_fce, action_str, message):
491574 if len(args) < 2:
520603 for key in remote_list:
521604 output(u"%s: %s -> %s" % (action_str, remote_list[key]['object_uri_str'], remote_list[key]['dest_name']))
522605
523 warning(u"Exitting now because of --dry-run")
606 warning(u"Exiting now because of --dry-run")
524607 return
525608
526609 seq = 0
533616 dst_uri = S3Uri(item['dest_name'])
534617
535618 extra_headers = copy(cfg.extra_headers)
536 response = process_fce(src_uri, dst_uri, extra_headers)
537 output(message % { "src" : src_uri, "dst" : dst_uri })
538 if Config().acl_public:
539 info(u"Public URL is: %s" % dst_uri.public_url())
619 try:
620 response = process_fce(src_uri, dst_uri, extra_headers)
621 output(message % { "src" : src_uri, "dst" : dst_uri })
622 if Config().acl_public:
623 info(u"Public URL is: %s" % dst_uri.public_url())
624 except S3Error, e:
625 if cfg.ignore_failed_copy and e.code == "NoSuchKey":
626 warning(u"Key not found %s" % item['object_uri_str'])
627 else:
628 raise
540629
541630 def cmd_cp(args):
542631 s3 = S3(Config())
562651 output(u" File size: %s" % info['headers']['content-length'])
563652 output(u" Last mod: %s" % info['headers']['last-modified'])
564653 output(u" MIME type: %s" % info['headers']['content-type'])
565 output(u" MD5 sum: %s" % info['headers']['etag'].strip('"'))
654 md5 = info['headers']['etag'].strip('"')
655 try:
656 md5 = info['s3cmd-attrs']['md5']
657 except KeyError:
658 pass
659 output(u" MD5 sum: %s" % md5)
660 if 'x-amz-server-side-encryption' in info['headers']:
661 output(u" SSE: %s" % info['headers']['x-amz-server-side-encryption'])
662 else:
663 output(u" SSE: NONE")
664
566665 else:
567666 info = s3.bucket_info(uri)
568667 output(u"%s (bucket):" % uri.uri())
569668 output(u" Location: %s" % info['bucket-location'])
570669 acl = s3.get_acl(uri)
571670 acl_grant_list = acl.getGrantList()
671
672 try:
673 policy = s3.get_policy(uri)
674 output(u" policy: %s" % policy)
675 except:
676 output(u" policy: none")
677
572678 for grant in acl_grant_list:
573679 output(u" ACL: %s: %s" % (grant['grantee'], grant['permission']))
574680 if acl.isAnonRead():
575681 output(u" URL: %s" % uri.public_url())
682
576683 except S3Error, e:
577684 if S3.codes.has_key(e.info["Code"]):
578685 error(S3.codes[e.info["Code"]] % uri.bucket())
580687 else:
581688 raise
582689
690 def filedicts_to_keys(*args):
691 keys = set()
692 for a in args:
693 keys.update(a.keys())
694 keys = list(keys)
695 keys.sort()
696 return keys
697
583698 def cmd_sync_remote2remote(args):
584 s3 = S3(Config())
585
586 # Normalise s3://uri (e.g. assert trailing slash)
587 destination_base = unicode(S3Uri(args[-1]))
588
589 src_list = fetch_remote_list(args[:-1], recursive = True, require_attribs = True)
590 dst_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)
591
592 src_count = len(src_list)
593 dst_count = len(dst_list)
594
595 info(u"Found %d source files, %d destination files" % (src_count, dst_count))
596
597 src_list, exclude_list = filter_exclude_include(src_list)
598
599 src_list, dst_list, existing_list = compare_filelists(src_list, dst_list, src_remote = True, dst_remote = True)
600
601 src_count = len(src_list)
602 dst_count = len(dst_list)
603
604 print(u"Summary: %d source files to copy, %d files at destination to delete" % (src_count, dst_count))
605
606 if src_count > 0:
607 ### Populate 'remote_uri' only if we've got something to sync from src to dst
608 for key in src_list:
609 src_list[key]['target_uri'] = destination_base + key
610
611 if cfg.dry_run:
612 for key in exclude_list:
613 output(u"exclude: %s" % unicodise(key))
614 if cfg.delete_removed:
615 for key in dst_list:
616 output(u"delete: %s" % dst_list[key]['object_uri_str'])
617 for key in src_list:
618 output(u"Sync: %s -> %s" % (src_list[key]['object_uri_str'], src_list[key]['target_uri']))
619 warning(u"Exitting now because of --dry-run")
620 return
621
622 # Delete items in destination that are not in source
623 if cfg.delete_removed:
699 def _do_deletes(s3, dst_list):
700 if cfg.max_delete > 0 and len(dst_list) > cfg.max_delete:
701 warning(u"delete: maximum requested number of deletes would be exceeded, none performed.")
702 return
703 # Delete items in destination that are not in source
624704 if cfg.dry_run:
625705 for key in dst_list:
626706 output(u"delete: %s" % dst_list[key]['object_uri_str'])
630710 s3.object_delete(uri)
631711 output(u"deleted: '%s'" % uri)
632712
713 s3 = S3(Config())
714
715 # Normalise s3://uri (e.g. assert trailing slash)
716 destination_base = unicode(S3Uri(args[-1]))
717
718 src_list = fetch_remote_list(args[:-1], recursive = True, require_attribs = True)
719 dst_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)
720
721 src_count = len(src_list)
722 orig_src_count = src_count
723 dst_count = len(dst_list)
724
725 info(u"Found %d source files, %d destination files" % (src_count, dst_count))
726
727 src_list, src_exclude_list = filter_exclude_include(src_list)
728 dst_list, dst_exclude_list = filter_exclude_include(dst_list)
729
730 src_list, dst_list, update_list, copy_pairs = compare_filelists(src_list, dst_list, src_remote = True, dst_remote = True, delay_updates = cfg.delay_updates)
731
732 src_count = len(src_list)
733 update_count = len(update_list)
734 dst_count = len(dst_list)
735
736 print(u"Summary: %d source files to copy, %d files at destination to delete" % (src_count, dst_count))
737
738 ### Populate 'target_uri' only if we've got something to sync from src to dst
739 for key in src_list:
740 src_list[key]['target_uri'] = destination_base + key
741 for key in update_list:
742 update_list[key]['target_uri'] = destination_base + key
743
744 if cfg.dry_run:
745 keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
746 for key in keys:
747 output(u"exclude: %s" % unicodise(key))
748 if cfg.delete_removed:
749 for key in dst_list:
750 output(u"delete: %s" % dst_list[key]['object_uri_str'])
751 for key in src_list:
752 output(u"Sync: %s -> %s" % (src_list[key]['object_uri_str'], src_list[key]['target_uri']))
753 warning(u"Exiting now because of --dry-run")
754 return
755
756 # if there are copy pairs, we can't do delete_before, on the chance
757 # we need one of the to-be-deleted files as a copy source.
758 if len(copy_pairs) > 0:
759 cfg.delete_after = True
760
761 if cfg.delete_removed and orig_src_count == 0 and len(dst_list) and not cfg.force:
762 warning(u"delete: cowardly refusing to delete because no source files were found. Use --force to override.")
763 cfg.delete_removed = False
764
765 # Delete items in destination that are not in source
766 if cfg.delete_removed and not cfg.delete_after:
767 _do_deletes(s3, dst_list)
768
769 def _upload(src_list, seq, src_count):
770 file_list = src_list.keys()
771 file_list.sort()
772 for file in file_list:
773 seq += 1
774 item = src_list[file]
775 src_uri = S3Uri(item['object_uri_str'])
776 dst_uri = S3Uri(item['target_uri'])
777 seq_label = "[%d of %d]" % (seq, src_count)
778 extra_headers = copy(cfg.extra_headers)
779 try:
780 response = s3.object_copy(src_uri, dst_uri, extra_headers)
781 output("File %(src)s copied to %(dst)s" % { "src" : src_uri, "dst" : dst_uri })
782 except S3Error, e:
783 error("File %(src)s could not be copied: %(e)s" % { "src" : src_uri, "e" : e })
784 return seq
785
633786 # Perform the synchronization of files
634787 timestamp_start = time.time()
635788 seq = 0
636 file_list = src_list.keys()
637 file_list.sort()
638 for file in file_list:
639 seq += 1
640 item = src_list[file]
641 src_uri = S3Uri(item['object_uri_str'])
642 dst_uri = S3Uri(item['target_uri'])
643 seq_label = "[%d of %d]" % (seq, src_count)
644 extra_headers = copy(cfg.extra_headers)
645 try:
646 response = s3.object_copy(src_uri, dst_uri, extra_headers)
647 output("File %(src)s copied to %(dst)s" % { "src" : src_uri, "dst" : dst_uri })
648 except S3Error, e:
649 error("File %(src)s could not be copied: %(e)s" % { "src" : src_uri, "e" : e })
789 seq = _upload(src_list, seq, src_count + update_count)
790 seq = _upload(update_list, seq, src_count + update_count)
791 n_copied, bytes_saved, failed_copy_files = remote_copy(s3, copy_pairs, destination_base)
792
793 #process files not copied
794 debug("Process files that was not remote copied")
795 failed_copy_count = len (failed_copy_files)
796 for key in failed_copy_files:
797 failed_copy_files[key]['target_uri'] = destination_base + key
798 seq = _upload(failed_copy_files, seq, failed_copy_count)
799
650800 total_elapsed = time.time() - timestamp_start
801 if total_elapsed == 0.0:
802 total_elapsed = 1.0
651803 outstr = "Done. Copied %d files in %0.1f seconds, %0.2f files/s" % (seq, total_elapsed, seq/total_elapsed)
652804 if seq > 0:
653805 output(outstr)
654806 else:
655807 info(outstr)
656808
809 # Delete items in destination that are not in source
810 if cfg.delete_removed and cfg.delete_after:
811 _do_deletes(s3, dst_list)
812
657813 def cmd_sync_remote2local(args):
658 def _parse_attrs_header(attrs_header):
659 attrs = {}
660 for attr in attrs_header.split("/"):
661 key, val = attr.split(":")
662 attrs[key] = val
663 return attrs
814 def _do_deletes(local_list):
815 if cfg.max_delete > 0 and len(local_list) > cfg.max_delete:
816 warning(u"delete: maximum requested number of deletes would be exceeded, none performed.")
817 return
818 for key in local_list:
819 os.unlink(local_list[key]['full_name'])
820 output(u"deleted: %s" % local_list[key]['full_name_unicode'])
664821
665822 s3 = S3(Config())
666823
667824 destination_base = args[-1]
668 local_list, single_file_local = fetch_local_list(destination_base, recursive = True)
825 local_list, single_file_local = fetch_local_list(destination_base, is_src = False, recursive = True)
669826 remote_list = fetch_remote_list(args[:-1], recursive = True, require_attribs = True)
670827
671828 local_count = len(local_list)
672829 remote_count = len(remote_list)
830 orig_remote_count = remote_count
673831
674832 info(u"Found %d remote files, %d local files" % (remote_count, local_count))
675833
676 remote_list, exclude_list = filter_exclude_include(remote_list)
677
678 remote_list, local_list, existing_list = compare_filelists(remote_list, local_list, src_remote = True, dst_remote = False)
834 remote_list, src_exclude_list = filter_exclude_include(remote_list)
835 local_list, dst_exclude_list = filter_exclude_include(local_list)
836
837 remote_list, local_list, update_list, copy_pairs = compare_filelists(remote_list, local_list, src_remote = True, dst_remote = False, delay_updates = cfg.delay_updates)
679838
680839 local_count = len(local_list)
681840 remote_count = len(remote_list)
682
683 info(u"Summary: %d remote files to download, %d local files to delete" % (remote_count, local_count))
684
685 if not os.path.isdir(destination_base):
686 ## We were either given a file name (existing or not) or want STDOUT
687 if remote_count > 1:
688 raise ParameterError("Destination must be a directory when downloading multiple sources.")
689 remote_list[remote_list.keys()[0]]['local_filename'] = deunicodise(destination_base)
690 else:
691 if destination_base[-1] != os.path.sep:
692 destination_base += os.path.sep
693 for key in remote_list:
694 local_filename = destination_base + key
695 if os.path.sep != "/":
696 local_filename = os.path.sep.join(local_filename.split("/"))
697 remote_list[key]['local_filename'] = deunicodise(local_filename)
841 update_count = len(update_list)
842 copy_pairs_count = len(copy_pairs)
843
844 info(u"Summary: %d remote files to download, %d local files to delete, %d local files to hardlink" % (remote_count + update_count, local_count, copy_pairs_count))
845
846 empty_fname_re = re.compile(r'\A\s*\Z')
847 def _set_local_filename(remote_list, destination_base):
848 if len(remote_list) == 0:
849 return
850 if not os.path.isdir(destination_base):
851 ## We were either given a file name (existing or not) or want STDOUT
852 if len(remote_list) > 1:
853 raise ParameterError("Destination must be a directory when downloading multiple sources.")
854 remote_list[remote_list.keys()[0]]['local_filename'] = deunicodise(destination_base)
855 else:
856 if destination_base[-1] != os.path.sep:
857 destination_base += os.path.sep
858 for key in remote_list:
859 local_basename = key
860 if empty_fname_re.match(key):
861 # Objects may exist on S3 with empty names (''), which don't map so well to common filesystems.
862 local_basename = '__AWS-EMPTY-OBJECT-NAME__'
863 warning(u"Empty object name on S3 found, saving locally as %s" % (local_basename))
864 local_filename = destination_base + local_basename
865 if os.path.sep != "/":
866 local_filename = os.path.sep.join(local_filename.split("/"))
867 remote_list[key]['local_filename'] = deunicodise(local_filename)
868
869 _set_local_filename(remote_list, destination_base)
870 _set_local_filename(update_list, destination_base)
698871
699872 if cfg.dry_run:
700 for key in exclude_list:
873 keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
874 for key in keys:
701875 output(u"exclude: %s" % unicodise(key))
702876 if cfg.delete_removed:
703877 for key in local_list:
704878 output(u"delete: %s" % local_list[key]['full_name_unicode'])
705879 for key in remote_list:
706 output(u"download: %s -> %s" % (remote_list[key]['object_uri_str'], remote_list[key]['local_filename']))
707
708 warning(u"Exitting now because of --dry-run")
880 output(u"download: %s -> %s" % (unicodise(remote_list[key]['object_uri_str']), unicodise(remote_list[key]['local_filename'])))
881 for key in update_list:
882 output(u"download: %s -> %s" % (update_list[key]['object_uri_str'], update_list[key]['local_filename']))
883
884 warning(u"Exiting now because of --dry-run")
709885 return
710886
711 if cfg.delete_removed:
712 for key in local_list:
713 os.unlink(local_list[key]['full_name'])
714 output(u"deleted: %s" % local_list[key]['full_name_unicode'])
887 # if there are copy pairs, we can't do delete_before, on the chance
888 # we need one of the to-be-deleted files as a copy source.
889 if len(copy_pairs) > 0:
890 cfg.delete_after = True
891
892 if cfg.delete_removed and orig_remote_count == 0 and len(local_list) and not cfg.force:
893 warning(u"delete: cowardly refusing to delete because no source files were found. Use --force to override.")
894 cfg.delete_removed = False
895
896 if cfg.delete_removed and not cfg.delete_after:
897 _do_deletes(local_list)
898
899 def _download(remote_list, seq, total, total_size, dir_cache):
900 file_list = remote_list.keys()
901 file_list.sort()
902 for file in file_list:
903 seq += 1
904 item = remote_list[file]
905 uri = S3Uri(item['object_uri_str'])
906 dst_file = item['local_filename']
907 seq_label = "[%d of %d]" % (seq, total)
908 try:
909 dst_dir = os.path.dirname(dst_file)
910 if not dir_cache.has_key(dst_dir):
911 dir_cache[dst_dir] = Utils.mkdir_with_parents(dst_dir)
912 if dir_cache[dst_dir] == False:
913 warning(u"%s: destination directory not writable: %s" % (file, dst_dir))
914 continue
915 try:
916 debug(u"dst_file=%s" % unicodise(dst_file))
917 # create temporary files (of type .s3cmd.XXXX.tmp) in the same directory
918 # for downloading and then rename once downloaded
919 chkptfd, chkptfname = tempfile.mkstemp(".tmp",".s3cmd.",os.path.dirname(dst_file))
920 debug(u"created chkptfname=%s" % unicodise(chkptfname))
921 dst_stream = os.fdopen(chkptfd, "wb")
922 response = s3.object_get(uri, dst_stream, extra_label = seq_label)
923 dst_stream.close()
924 # download completed, rename the file to destination
925 os.rename(chkptfname, dst_file)
926
927 # set permissions on destination file
928 original_umask = os.umask(0);
929 os.umask(original_umask);
930 mode = 0777 - original_umask;
931 debug(u"mode=%s" % oct(mode))
932 os.chmod(dst_file, mode);
933 debug(u"renamed chkptfname=%s to dst_file=%s" % (unicodise(chkptfname), unicodise(dst_file)))
934 if response.has_key('s3cmd-attrs') and cfg.preserve_attrs:
935 attrs = response['s3cmd-attrs']
936 if attrs.has_key('mode'):
937 os.chmod(dst_file, int(attrs['mode']))
938 if attrs.has_key('mtime') or attrs.has_key('atime'):
939 mtime = attrs.has_key('mtime') and int(attrs['mtime']) or int(time.time())
940 atime = attrs.has_key('atime') and int(attrs['atime']) or int(time.time())
941 os.utime(dst_file, (atime, mtime))
942 if attrs.has_key('uid') and attrs.has_key('gid'):
943 uid = int(attrs['uid'])
944 gid = int(attrs['gid'])
945 os.lchown(dst_file,uid,gid)
946 except OSError, e:
947 try:
948 dst_stream.close()
949 os.remove(chkptfname)
950 except: pass
951 if e.errno == errno.EEXIST:
952 warning(u"%s exists - not overwriting" % (dst_file))
953 continue
954 if e.errno in (errno.EPERM, errno.EACCES):
955 warning(u"%s not writable: %s" % (dst_file, e.strerror))
956 continue
957 if e.errno == errno.EISDIR:
958 warning(u"%s is a directory - skipping over" % dst_file)
959 continue
960 raise e
961 except KeyboardInterrupt:
962 try:
963 dst_stream.close()
964 os.remove(chkptfname)
965 except: pass
966 warning(u"Exiting after keyboard interrupt")
967 return
968 except Exception, e:
969 try:
970 dst_stream.close()
971 os.remove(chkptfname)
972 except: pass
973 error(u"%s: %s" % (file, e))
974 continue
975 # We have to keep repeating this call because
976 # Python 2.4 doesn't support try/except/finally
977 # construction :-(
978 try:
979 dst_stream.close()
980 os.remove(chkptfname)
981 except: pass
982 except S3DownloadError, e:
983 error(u"%s: download failed too many times. Skipping that file." % file)
984 continue
985 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
986 if not Config().progress_meter:
987 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
988 (uri, unicodise(dst_file), response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
989 seq_label))
990 total_size += response["size"]
991 if Config().delete_after_fetch:
992 s3.object_delete(uri)
993 output(u"File '%s' removed after syncing" % (uri))
994 return seq, total_size
715995
716996 total_size = 0
717997 total_elapsed = 0.0
718998 timestamp_start = time.time()
999 dir_cache = {}
7191000 seq = 0
720 dir_cache = {}
721 file_list = remote_list.keys()
722 file_list.sort()
723 for file in file_list:
724 seq += 1
725 item = remote_list[file]
726 uri = S3Uri(item['object_uri_str'])
727 dst_file = item['local_filename']
728 seq_label = "[%d of %d]" % (seq, remote_count)
729 try:
730 dst_dir = os.path.dirname(dst_file)
731 if not dir_cache.has_key(dst_dir):
732 dir_cache[dst_dir] = Utils.mkdir_with_parents(dst_dir)
733 if dir_cache[dst_dir] == False:
734 warning(u"%s: destination directory not writable: %s" % (file, dst_dir))
735 continue
736 try:
737 open_flags = os.O_CREAT
738 open_flags |= os.O_TRUNC
739 # open_flags |= os.O_EXCL
740
741 debug(u"dst_file=%s" % unicodise(dst_file))
742 # This will have failed should the file exist
743 os.close(os.open(dst_file, open_flags))
744 # Yeah I know there is a race condition here. Sadly I don't know how to open() in exclusive mode.
745 dst_stream = open(dst_file, "wb")
746 response = s3.object_get(uri, dst_stream, extra_label = seq_label)
747 dst_stream.close()
748 if response['headers'].has_key('x-amz-meta-s3cmd-attrs') and cfg.preserve_attrs:
749 attrs = _parse_attrs_header(response['headers']['x-amz-meta-s3cmd-attrs'])
750 if attrs.has_key('mode'):
751 os.chmod(dst_file, int(attrs['mode']))
752 if attrs.has_key('mtime') or attrs.has_key('atime'):
753 mtime = attrs.has_key('mtime') and int(attrs['mtime']) or int(time.time())
754 atime = attrs.has_key('atime') and int(attrs['atime']) or int(time.time())
755 os.utime(dst_file, (atime, mtime))
756 ## FIXME: uid/gid / uname/gname handling comes here! TODO
757 except OSError, e:
758 try: dst_stream.close()
759 except: pass
760 if e.errno == errno.EEXIST:
761 warning(u"%s exists - not overwriting" % (dst_file))
762 continue
763 if e.errno in (errno.EPERM, errno.EACCES):
764 warning(u"%s not writable: %s" % (dst_file, e.strerror))
765 continue
766 if e.errno == errno.EISDIR:
767 warning(u"%s is a directory - skipping over" % dst_file)
768 continue
769 raise e
770 except KeyboardInterrupt:
771 try: dst_stream.close()
772 except: pass
773 warning(u"Exiting after keyboard interrupt")
774 return
775 except Exception, e:
776 try: dst_stream.close()
777 except: pass
778 error(u"%s: %s" % (file, e))
779 continue
780 # We have to keep repeating this call because
781 # Python 2.4 doesn't support try/except/finally
782 # construction :-(
783 try: dst_stream.close()
784 except: pass
785 except S3DownloadError, e:
786 error(u"%s: download failed too many times. Skipping that file." % file)
787 continue
788 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
789 if not Config().progress_meter:
790 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
791 (uri, unicodise(dst_file), response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
792 seq_label))
793 total_size += response["size"]
1001 seq, total_size = _download(remote_list, seq, remote_count + update_count, total_size, dir_cache)
1002 seq, total_size = _download(update_list, seq, remote_count + update_count, total_size, dir_cache)
1003
1004 failed_copy_list = local_copy(copy_pairs, destination_base)
1005 _set_local_filename(failed_copy_list, destination_base)
1006 seq, total_size = _download(failed_copy_list, seq, len(failed_copy_list) + remote_count + update_count, total_size, dir_cache)
7941007
7951008 total_elapsed = time.time() - timestamp_start
7961009 speed_fmt = formatSize(total_size/total_elapsed, human_readable = True, floating_point = True)
8031016 else:
8041017 info(outstr)
8051018
1019 if cfg.delete_removed and cfg.delete_after:
1020 _do_deletes(local_list)
1021
1022 def local_copy(copy_pairs, destination_base):
1023 # Do NOT hardlink local files by default, that'd be silly
1024 # For instance all empty files would become hardlinked together!
1025 encoding = sys.getfilesystemencoding()
1026 failed_copy_list = FileDict()
1027 for (src_obj, dst1, relative_file) in copy_pairs:
1028 src_file = os.path.join(destination_base, dst1)
1029 dst_file = os.path.join(destination_base, relative_file)
1030 dst_dir = os.path.dirname(dst_file)
1031 try:
1032 if not os.path.isdir(dst_dir):
1033 debug("MKDIR %s" % dst_dir)
1034 os.makedirs(dst_dir)
1035 debug(u"Copying %s to %s" % (src_file, dst_file))
1036 shutil.copy2(src_file.encode(encoding), dst_file.encode(encoding))
1037 except (IOError, OSError), e:
1038 warning(u'Unable to hardlink or copy files %s -> %s: %s' % (src_file, dst_file, e))
1039 failed_copy_list[relative_file] = src_obj
1040 return failed_copy_list
1041
1042 def remote_copy(s3, copy_pairs, destination_base):
1043 saved_bytes = 0
1044 failed_copy_list = FileDict()
1045 for (src_obj, dst1, dst2) in copy_pairs:
1046 debug(u"Remote Copying from %s to %s" % (dst1, dst2))
1047 dst1_uri = S3Uri(destination_base + dst1)
1048 dst2_uri = S3Uri(destination_base + dst2)
1049 extra_headers = copy(cfg.extra_headers)
1050 try:
1051 s3.object_copy(dst1_uri, dst2_uri, extra_headers)
1052 info = s3.object_info(dst2_uri)
1053 saved_bytes = saved_bytes + int(info['headers']['content-length'])
1054 output(u"remote copy: %s -> %s" % (dst1, dst2))
1055 except:
1056 warning(u'Unable to remote copy files %s -> %s' % (dst1_uri, dst2_uri))
1057 failed_copy_list[dst2] = src_obj
1058 return (len(copy_pairs), saved_bytes, failed_copy_list)
1059
1060
1061 def _build_attr_header(local_list, src):
1062 attrs = {}
1063 for attr in cfg.preserve_attrs_list:
1064 if attr == 'uname':
1065 try:
1066 val = Utils.getpwuid_username(local_list[src]['uid'])
1067 except KeyError:
1068 attr = "uid"
1069 val = local_list[src].get('uid')
1070 warning(u"%s: Owner username not known. Storing UID=%d instead." % (src, val))
1071 elif attr == 'gname':
1072 try:
1073 val = Utils.getgrgid_grpname(local_list[src].get('gid'))
1074 except KeyError:
1075 attr = "gid"
1076 val = local_list[src].get('gid')
1077 warning(u"%s: Owner groupname not known. Storing GID=%d instead." % (src, val))
1078 elif attr == 'md5':
1079 try:
1080 val = local_list.get_md5(src)
1081 except IOError:
1082 val = None
1083 else:
1084 try:
1085 val = getattr(local_list[src]['sr'], 'st_' + attr)
1086 except:
1087 val = None
1088 if val is not None:
1089 attrs[attr] = val
1090
1091 if 'md5' in attrs and attrs['md5'] is None:
1092 del attrs['md5']
1093
1094 result = ""
1095 for k in attrs: result += "%s:%s/" % (k, attrs[k])
1096 return { 'x-amz-meta-s3cmd-attrs' : result[:-1] }
1097
1098
8061099 def cmd_sync_local2remote(args):
807 def _build_attr_header(src):
808 import pwd, grp
809 attrs = {}
810 src = deunicodise(src)
811 try:
812 st = os.stat_result(os.stat(src))
813 except OSError, e:
814 raise InvalidFileError(u"%s: %s" % (unicodise(src), e.strerror))
815 for attr in cfg.preserve_attrs_list:
816 if attr == 'uname':
1100
1101 def _do_deletes(s3, remote_list):
1102 if cfg.max_delete > 0 and len(remote_list) > cfg.max_delete:
1103 warning(u"delete: maximum requested number of deletes would be exceeded, none performed.")
1104 return
1105 for key in remote_list:
1106 uri = S3Uri(remote_list[key]['object_uri_str'])
1107 s3.object_delete(uri)
1108 output(u"deleted: '%s'" % uri)
1109
1110 def _single_process(local_list):
1111 for dest in destinations:
1112 ## Normalize URI to convert s3://bkt to s3://bkt/ (trailing slash)
1113 destination_base_uri = S3Uri(dest)
1114 if destination_base_uri.type != 's3':
1115 raise ParameterError("Destination must be S3Uri. Got: %s" % destination_base_uri)
1116 destination_base = str(destination_base_uri)
1117 _child(destination_base, local_list)
1118 return destination_base_uri
1119
1120 def _parent():
1121 # Now that we've done all the disk I/O to look at the local file system and
1122 # calculate the md5 for each file, fork for each destination to upload to them separately
1123 # and in parallel
1124 child_pids = []
1125
1126 for dest in destinations:
1127 ## Normalize URI to convert s3://bkt to s3://bkt/ (trailing slash)
1128 destination_base_uri = S3Uri(dest)
1129 if destination_base_uri.type != 's3':
1130 raise ParameterError("Destination must be S3Uri. Got: %s" % destination_base_uri)
1131 destination_base = str(destination_base_uri)
1132 child_pid = os.fork()
1133 if child_pid == 0:
1134 _child(destination_base, local_list)
1135 os._exit(0)
1136 else:
1137 child_pids.append(child_pid)
1138
1139 while len(child_pids):
1140 (pid, status) = os.wait()
1141 child_pids.remove(pid)
1142
1143 return
1144
1145 def _child(destination_base, local_list):
1146 def _set_remote_uri(local_list, destination_base, single_file_local):
1147 if len(local_list) > 0:
1148 ## Populate 'remote_uri' only if we've got something to upload
1149 if not destination_base.endswith("/"):
1150 if not single_file_local:
1151 raise ParameterError("Destination S3 URI must end with '/' (ie must refer to a directory on the remote side).")
1152 local_list[local_list.keys()[0]]['remote_uri'] = unicodise(destination_base)
1153 else:
1154 for key in local_list:
1155 local_list[key]['remote_uri'] = unicodise(destination_base + key)
1156
1157 def _upload(local_list, seq, total, total_size):
1158 file_list = local_list.keys()
1159 file_list.sort()
1160 for file in file_list:
1161 seq += 1
1162 item = local_list[file]
1163 src = item['full_name']
1164 uri = S3Uri(item['remote_uri'])
1165 seq_label = "[%d of %d]" % (seq, total)
1166 extra_headers = copy(cfg.extra_headers)
8171167 try:
818 val = pwd.getpwuid(st.st_uid).pw_name
819 except KeyError:
820 attr = "uid"
821 val = st.st_uid
822 warning(u"%s: Owner username not known. Storing UID=%d instead." % (unicodise(src), val))
823 elif attr == 'gname':
824 try:
825 val = grp.getgrgid(st.st_gid).gr_name
826 except KeyError:
827 attr = "gid"
828 val = st.st_gid
829 warning(u"%s: Owner groupname not known. Storing GID=%d instead." % (unicodise(src), val))
830 else:
831 val = getattr(st, 'st_' + attr)
832 attrs[attr] = val
833 result = ""
834 for k in attrs: result += "%s:%s/" % (k, attrs[k])
835 return { 'x-amz-meta-s3cmd-attrs' : result[:-1] }
836
1168 if cfg.preserve_attrs:
1169 attr_header = _build_attr_header(local_list, file)
1170 debug(u"attr_header: %s" % attr_header)
1171 extra_headers.update(attr_header)
1172 response = s3.object_put(src, uri, extra_headers, extra_label = seq_label)
1173 except InvalidFileError, e:
1174 warning(u"File can not be uploaded: %s" % e)
1175 continue
1176 except S3UploadError, e:
1177 error(u"%s: upload failed too many times. Skipping that file." % item['full_name_unicode'])
1178 continue
1179 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
1180 if not cfg.progress_meter:
1181 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
1182 (item['full_name_unicode'], uri, response["size"], response["elapsed"],
1183 speed_fmt[0], speed_fmt[1], seq_label))
1184 total_size += response["size"]
1185 uploaded_objects_list.append(uri.object())
1186 return seq, total_size
1187
1188 remote_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)
1189
1190 local_count = len(local_list)
1191 orig_local_count = local_count
1192 remote_count = len(remote_list)
1193
1194 info(u"Found %d local files, %d remote files" % (local_count, remote_count))
1195
1196 local_list, src_exclude_list = filter_exclude_include(local_list)
1197 remote_list, dst_exclude_list = filter_exclude_include(remote_list)
1198
1199 if single_file_local and len(local_list) == 1 and len(remote_list) == 1:
1200 ## Make remote_key same as local_key for comparison if we're dealing with only one file
1201 remote_list_entry = remote_list[remote_list.keys()[0]]
1202 # Flush remote_list, by the way
1203 remote_list = FileDict()
1204 remote_list[local_list.keys()[0]] = remote_list_entry
1205
1206 local_list, remote_list, update_list, copy_pairs = compare_filelists(local_list, remote_list, src_remote = False, dst_remote = True, delay_updates = cfg.delay_updates)
1207
1208 local_count = len(local_list)
1209 update_count = len(update_list)
1210 copy_count = len(copy_pairs)
1211 remote_count = len(remote_list)
1212
1213 info(u"Summary: %d local files to upload, %d files to remote copy, %d remote files to delete" % (local_count + update_count, copy_count, remote_count))
1214
1215 _set_remote_uri(local_list, destination_base, single_file_local)
1216 _set_remote_uri(update_list, destination_base, single_file_local)
1217
1218 if cfg.dry_run:
1219 keys = filedicts_to_keys(src_exclude_list, dst_exclude_list)
1220 for key in keys:
1221 output(u"exclude: %s" % unicodise(key))
1222 for key in local_list:
1223 output(u"upload: %s -> %s" % (local_list[key]['full_name_unicode'], local_list[key]['remote_uri']))
1224 for key in update_list:
1225 output(u"upload: %s -> %s" % (update_list[key]['full_name_unicode'], update_list[key]['remote_uri']))
1226 for (src_obj, dst1, dst2) in copy_pairs:
1227 output(u"remote copy: %s -> %s" % (dst1, dst2))
1228 if cfg.delete_removed:
1229 for key in remote_list:
1230 output(u"delete: %s" % remote_list[key]['object_uri_str'])
1231
1232 warning(u"Exiting now because of --dry-run")
1233 return
1234
1235 # if there are copy pairs, we can't do delete_before, on the chance
1236 # we need one of the to-be-deleted files as a copy source.
1237 if len(copy_pairs) > 0:
1238 cfg.delete_after = True
1239
1240 if cfg.delete_removed and orig_local_count == 0 and len(remote_list) and not cfg.force:
1241 warning(u"delete: cowardly refusing to delete because no source files were found. Use --force to override.")
1242 cfg.delete_removed = False
1243
1244 if cfg.delete_removed and not cfg.delete_after:
1245 _do_deletes(s3, remote_list)
1246
1247 total_size = 0
1248 total_elapsed = 0.0
1249 timestamp_start = time.time()
1250 n, total_size = _upload(local_list, 0, local_count, total_size)
1251 n, total_size = _upload(update_list, n, local_count, total_size)
1252 n_copies, saved_bytes, failed_copy_files = remote_copy(s3, copy_pairs, destination_base)
1253
1254 #upload file that could not be copied
1255 debug("Process files that was not remote copied")
1256 failed_copy_count = len(failed_copy_files)
1257 _set_remote_uri(failed_copy_files, destination_base, single_file_local)
1258 n, total_size = _upload(failed_copy_files, n, failed_copy_count, total_size)
1259
1260 if cfg.delete_removed and cfg.delete_after:
1261 _do_deletes(s3, remote_list)
1262 total_elapsed = time.time() - timestamp_start
1263 total_speed = total_elapsed and total_size/total_elapsed or 0.0
1264 speed_fmt = formatSize(total_speed, human_readable = True, floating_point = True)
1265
1266 # Only print out the result if any work has been done or
1267 # if the user asked for verbose output
1268 outstr = "Done. Uploaded %d bytes in %0.1f seconds, %0.2f %sB/s. Copied %d files saving %d bytes transfer." % (total_size, total_elapsed, speed_fmt[0], speed_fmt[1], n_copies, saved_bytes)
1269 if total_size + saved_bytes > 0:
1270 output(outstr)
1271 else:
1272 info(outstr)
1273
1274 return
1275
1276 def _invalidate_on_cf(destination_base_uri):
1277 cf = CloudFront(cfg)
1278 default_index_file = None
1279 if cfg.invalidate_default_index_on_cf or cfg.invalidate_default_index_root_on_cf:
1280 info_response = s3.website_info(destination_base_uri, cfg.bucket_location)
1281 if info_response:
1282 default_index_file = info_response['index_document']
1283 if len(default_index_file) < 1:
1284 default_index_file = None
1285
1286 result = cf.InvalidateObjects(destination_base_uri, uploaded_objects_list, default_index_file, cfg.invalidate_default_index_on_cf, cfg.invalidate_default_index_root_on_cf)
1287 if result['status'] == 201:
1288 output("Created invalidation request for %d paths" % len(uploaded_objects_list))
1289 output("Check progress with: s3cmd cfinvalinfo cf://%s/%s" % (result['dist_id'], result['request_id']))
1290
1291
1292 # main execution
8371293 s3 = S3(cfg)
1294 uploaded_objects_list = []
8381295
8391296 if cfg.encrypt:
8401297 error(u"S3cmd 'sync' doesn't yet support GPG encryption, sorry.")
8421299 error(u"or disable encryption with --no-encrypt parameter.")
8431300 sys.exit(1)
8441301
845 ## Normalize URI to convert s3://bkt to s3://bkt/ (trailing slash)
846 destination_base_uri = S3Uri(args[-1])
847 if destination_base_uri.type != 's3':
848 raise ParameterError("Destination must be S3Uri. Got: %s" % destination_base_uri)
849 destination_base = str(destination_base_uri)
850
851 local_list, single_file_local = fetch_local_list(args[:-1], recursive = True)
852 remote_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)
853
854 local_count = len(local_list)
855 remote_count = len(remote_list)
856
857 info(u"Found %d local files, %d remote files" % (local_count, remote_count))
858
859 local_list, exclude_list = filter_exclude_include(local_list)
860
861 if single_file_local and len(local_list) == 1 and len(remote_list) == 1:
862 ## Make remote_key same as local_key for comparison if we're dealing with only one file
863 remote_list_entry = remote_list[remote_list.keys()[0]]
864 # Flush remote_list, by the way
865 remote_list = { local_list.keys()[0] : remote_list_entry }
866
867 local_list, remote_list, existing_list = compare_filelists(local_list, remote_list, src_remote = False, dst_remote = True)
868
869 local_count = len(local_list)
870 remote_count = len(remote_list)
871
872 info(u"Summary: %d local files to upload, %d remote files to delete" % (local_count, remote_count))
873
874 if local_count > 0:
875 ## Populate 'remote_uri' only if we've got something to upload
876 if not destination_base.endswith("/"):
877 if not single_file_local:
878 raise ParameterError("Destination S3 URI must end with '/' (ie must refer to a directory on the remote side).")
879 local_list[local_list.keys()[0]]['remote_uri'] = unicodise(destination_base)
880 else:
881 for key in local_list:
882 local_list[key]['remote_uri'] = unicodise(destination_base + key)
883
884 if cfg.dry_run:
885 for key in exclude_list:
886 output(u"exclude: %s" % unicodise(key))
887 if cfg.delete_removed:
888 for key in remote_list:
889 output(u"delete: %s" % remote_list[key]['object_uri_str'])
890 for key in local_list:
891 output(u"upload: %s -> %s" % (local_list[key]['full_name_unicode'], local_list[key]['remote_uri']))
892
893 warning(u"Exitting now because of --dry-run")
894 return
895
896 if cfg.delete_removed:
897 for key in remote_list:
898 uri = S3Uri(remote_list[key]['object_uri_str'])
899 s3.object_delete(uri)
900 output(u"deleted: '%s'" % uri)
901
902 uploaded_objects_list = []
903 total_size = 0
904 total_elapsed = 0.0
905 timestamp_start = time.time()
906 seq = 0
907 file_list = local_list.keys()
908 file_list.sort()
909 for file in file_list:
910 seq += 1
911 item = local_list[file]
912 src = item['full_name']
913 uri = S3Uri(item['remote_uri'])
914 seq_label = "[%d of %d]" % (seq, local_count)
915 extra_headers = copy(cfg.extra_headers)
916 try:
917 if cfg.preserve_attrs:
918 attr_header = _build_attr_header(src)
919 debug(u"attr_header: %s" % attr_header)
920 extra_headers.update(attr_header)
921 response = s3.object_put(src, uri, extra_headers, extra_label = seq_label)
922 except InvalidFileError, e:
923 warning(u"File can not be uploaded: %s" % e)
924 continue
925 except S3UploadError, e:
926 error(u"%s: upload failed too many times. Skipping that file." % item['full_name_unicode'])
927 continue
928 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
929 if not cfg.progress_meter:
930 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
931 (item['full_name_unicode'], uri, response["size"], response["elapsed"],
932 speed_fmt[0], speed_fmt[1], seq_label))
933 total_size += response["size"]
934 uploaded_objects_list.append(uri.object())
935
936 total_elapsed = time.time() - timestamp_start
937 total_speed = total_elapsed and total_size/total_elapsed or 0.0
938 speed_fmt = formatSize(total_speed, human_readable = True, floating_point = True)
939
940 # Only print out the result if any work has been done or
941 # if the user asked for verbose output
942 outstr = "Done. Uploaded %d bytes in %0.1f seconds, %0.2f %sB/s" % (total_size, total_elapsed, speed_fmt[0], speed_fmt[1])
943 if total_size > 0:
944 output(outstr)
1302 local_list, single_file_local = fetch_local_list(args[:-1], is_src = True, recursive = True)
1303
1304 destinations = [args[-1]]
1305 if cfg.additional_destinations:
1306 destinations = destinations + cfg.additional_destinations
1307
1308 if 'fork' not in os.__all__ or len(destinations) < 2:
1309 destination_base_uri = _single_process(local_list)
1310 if cfg.invalidate_on_cf:
1311 if len(uploaded_objects_list) == 0:
1312 info("Nothing to invalidate in CloudFront")
1313 else:
1314 _invalidate_on_cf(destination_base_uri)
9451315 else:
946 info(outstr)
947
948 if cfg.invalidate_on_cf:
949 if len(uploaded_objects_list) == 0:
950 info("Nothing to invalidate in CloudFront")
951 else:
952 # 'uri' from the last iteration is still valid at this point
953 cf = CloudFront(cfg)
954 result = cf.InvalidateObjects(uri, uploaded_objects_list)
955 if result['status'] == 201:
956 output("Created invalidation request for %d paths" % len(uploaded_objects_list))
957 output("Check progress with: s3cmd cfinvalinfo cf://%s/%s" % (result['dist_id'], result['request_id']))
1316 _parent()
1317 if cfg.invalidate_on_cf:
1318 error(u"You cannot use both --cf-invalidate and --add-destination.")
9581319
9591320 def cmd_sync(args):
9601321 if (len(args) < 2):
9691330 raise ParameterError("Invalid source/destination: '%s'" % "' '".join(args))
9701331
9711332 def cmd_setacl(args):
972 def _update_acl(uri, seq_label = ""):
973 something_changed = False
974 acl = s3.get_acl(uri)
975 debug(u"acl: %s - %r" % (uri, acl.grantees))
976 if cfg.acl_public == True:
977 if acl.isAnonRead():
978 info(u"%s: already Public, skipping %s" % (uri, seq_label))
979 else:
980 acl.grantAnonRead()
981 something_changed = True
982 elif cfg.acl_public == False: # we explicitely check for False, because it could be None
983 if not acl.isAnonRead():
984 info(u"%s: already Private, skipping %s" % (uri, seq_label))
985 else:
986 acl.revokeAnonRead()
987 something_changed = True
988
989 # update acl with arguments
990 # grant first and revoke later, because revoke has priority
991 if cfg.acl_grants:
992 something_changed = True
993 for grant in cfg.acl_grants:
994 acl.grant(**grant);
995
996 if cfg.acl_revokes:
997 something_changed = True
998 for revoke in cfg.acl_revokes:
999 acl.revoke(**revoke);
1000
1001 if not something_changed:
1002 return
1003
1004 retsponse = s3.set_acl(uri, acl)
1005 if retsponse['status'] == 200:
1006 if cfg.acl_public in (True, False):
1007 output(u"%s: ACL set to %s %s" % (uri, set_to_acl, seq_label))
1008 else:
1009 output(u"%s: ACL updated" % uri)
1010
10111333 s3 = S3(cfg)
10121334
10131335 set_to_acl = cfg.acl_public and "Public" or "Private"
10231345 else:
10241346 info("Setting bucket-level ACL for %s" % (uri.uri()))
10251347 if not cfg.dry_run:
1026 _update_acl(uri)
1348 update_acl(s3, uri)
10271349 else:
10281350 args.append(arg)
10291351
10401362 for key in remote_list:
10411363 output(u"setacl: %s" % remote_list[key]['object_uri_str'])
10421364
1043 warning(u"Exitting now because of --dry-run")
1365 warning(u"Exiting now because of --dry-run")
10441366 return
10451367
10461368 seq = 0
10481370 seq += 1
10491371 seq_label = "[%d of %d]" % (seq, remote_count)
10501372 uri = S3Uri(remote_list[key]['object_uri_str'])
1051 _update_acl(uri, seq_label)
1373 update_acl(s3, uri, seq_label)
1374
1375 def cmd_setpolicy(args):
1376 s3 = S3(cfg)
1377 uri = S3Uri(args[1])
1378 policy_file = args[0]
1379 policy = open(policy_file, 'r').read()
1380
1381 if cfg.dry_run: return
1382
1383 response = s3.set_policy(uri, policy)
1384
1385 #if retsponse['status'] == 200:
1386 debug(u"response - %s" % response['status'])
1387 if response['status'] == 204:
1388 output(u"%s: Policy updated" % uri)
1389
1390 def cmd_delpolicy(args):
1391 s3 = S3(cfg)
1392 uri = S3Uri(args[0])
1393 if cfg.dry_run: return
1394
1395 response = s3.delete_policy(uri)
1396
1397 #if retsponse['status'] == 200:
1398 debug(u"response - %s" % response['status'])
1399 output(u"%s: Policy deleted" % uri)
1400
1401
1402 def cmd_multipart(args):
1403 s3 = S3(cfg)
1404 uri = S3Uri(args[0])
1405
1406 #id = ''
1407 #if(len(args) > 1): id = args[1]
1408
1409 response = s3.get_multipart(uri)
1410 debug(u"response - %s" % response['status'])
1411 output(u"%s" % uri)
1412 tree = getTreeFromXml(response['data'])
1413 debug(parseNodes(tree))
1414 output(u"Initiated\tPath\tId")
1415 for mpupload in parseNodes(tree):
1416 try:
1417 output("%s\t%s\t%s" % (mpupload['Initiated'], "s3://" + uri.bucket() + "/" + mpupload['Key'], mpupload['UploadId']))
1418 except KeyError:
1419 pass
1420
1421 def cmd_abort_multipart(args):
1422 '''{"cmd":"abortmp", "label":"abort a multipart upload", "param":"s3://BUCKET Id", "func":cmd_abort_multipart, "argc":2},'''
1423 s3 = S3(cfg)
1424 uri = S3Uri(args[0])
1425 id = args[1]
1426 response = s3.abort_multipart(uri, id)
1427 debug(u"response - %s" % response['status'])
1428 output(u"%s" % uri)
1429
1430 def cmd_list_multipart(args):
1431 '''{"cmd":"abortmp", "label":"list a multipart upload", "param":"s3://BUCKET Id", "func":cmd_list_multipart, "argc":2},'''
1432 s3 = S3(cfg)
1433 uri = S3Uri(args[0])
1434 id = args[1]
1435
1436 response = s3.list_multipart(uri, id)
1437 debug(u"response - %s" % response['status'])
1438 tree = getTreeFromXml(response['data'])
1439 output(u"LastModified\t\t\tPartNumber\tETag\tSize")
1440 for mpupload in parseNodes(tree):
1441 try:
1442 output("%s\t%s\t%s\t%s" % (mpupload['LastModified'], mpupload['PartNumber'], mpupload['ETag'], mpupload['Size']))
1443 except:
1444 pass
10521445
10531446 def cmd_accesslog(args):
10541447 s3 = S3(cfg)
10761469 debug("string-to-sign: %r" % string_to_sign)
10771470 signature = Utils.sign_string(string_to_sign)
10781471 output("Signature: %s" % signature)
1472
1473 def cmd_signurl(args):
1474 expiry = args.pop()
1475 url_to_sign = S3Uri(args.pop())
1476 if url_to_sign.type != 's3':
1477 raise ParameterError("Must be S3Uri. Got: %s" % url_to_sign)
1478 debug("url to sign: %r" % url_to_sign)
1479 signed_url = Utils.sign_url(url_to_sign, expiry)
1480 output(signed_url)
10791481
10801482 def cmd_fixbucket(args):
10811483 def _unescape(text):
11991601 ("gpg_passphrase", "Encryption password", "Encryption password is used to protect your files from reading\nby unauthorized persons while in transfer to S3"),
12001602 ("gpg_command", "Path to GPG program"),
12011603 ("use_https", "Use HTTPS protocol", "When using secure HTTPS protocol all communication with Amazon S3\nservers is protected from 3rd party eavesdropping. This method is\nslower than plain HTTP and can't be used if you're behind a proxy"),
1202 ("proxy_host", "HTTP Proxy server name", "On some networks all internet access must go through a HTTP proxy.\nTry setting it here if you can't conect to S3 directly"),
1604 ("proxy_host", "HTTP Proxy server name", "On some networks all internet access must go through a HTTP proxy.\nTry setting it here if you can't connect to S3 directly"),
12031605 ("proxy_port", "HTTP Proxy server port"),
12041606 ]
12051607 ## Option-specfic defaults
12971699
12981700 except Exception, e:
12991701 error(u"Test failed: %s" % (e))
1702 if e.find('403') != -1:
1703 error(u"Are you sure your keys have ListAllMyBuckets permissions?")
13001704 val = raw_input("\nRetry configuration? [Y/n] ")
13011705 if val.lower().startswith("y") or val == "":
13021706 continue
13851789 {"cmd":"get", "label":"Get file from bucket", "param":"s3://BUCKET/OBJECT LOCAL_FILE", "func":cmd_object_get, "argc":1},
13861790 {"cmd":"del", "label":"Delete file from bucket", "param":"s3://BUCKET/OBJECT", "func":cmd_object_del, "argc":1},
13871791 #{"cmd":"mkdir", "label":"Make a virtual S3 directory", "param":"s3://BUCKET/path/to/dir", "func":cmd_mkdir, "argc":1},
1792 {"cmd":"restore", "label":"Restore file from Glacier storage", "param":"s3://BUCKET/OBJECT", "func":cmd_object_restore, "argc":1},
13881793 {"cmd":"sync", "label":"Synchronize a directory tree to S3", "param":"LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR", "func":cmd_sync, "argc":2},
13891794 {"cmd":"du", "label":"Disk usage by buckets", "param":"[s3://BUCKET[/PREFIX]]", "func":cmd_du, "argc":0},
13901795 {"cmd":"info", "label":"Get various information about Buckets or Files", "param":"s3://BUCKET[/OBJECT]", "func":cmd_info, "argc":1},
13911796 {"cmd":"cp", "label":"Copy object", "param":"s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]", "func":cmd_cp, "argc":2},
13921797 {"cmd":"mv", "label":"Move object", "param":"s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]", "func":cmd_mv, "argc":2},
13931798 {"cmd":"setacl", "label":"Modify Access control list for Bucket or Files", "param":"s3://BUCKET[/OBJECT]", "func":cmd_setacl, "argc":1},
1799
1800 {"cmd":"setpolicy", "label":"Modify Bucket Policy", "param":"FILE s3://BUCKET", "func":cmd_setpolicy, "argc":2},
1801 {"cmd":"delpolicy", "label":"Delete Bucket Policy", "param":"s3://BUCKET", "func":cmd_delpolicy, "argc":1},
1802
1803 {"cmd":"multipart", "label":"show multipart uploads", "param":"s3://BUCKET [Id]", "func":cmd_multipart, "argc":1},
1804 {"cmd":"abortmp", "label":"abort a multipart upload", "param":"s3://BUCKET/OBJECT Id", "func":cmd_abort_multipart, "argc":2},
1805
1806 {"cmd":"listmp", "label":"list parts of a multipart upload", "param":"s3://BUCKET/OBJECT Id", "func":cmd_list_multipart, "argc":2},
1807
13941808 {"cmd":"accesslog", "label":"Enable/disable bucket access logging", "param":"s3://BUCKET", "func":cmd_accesslog, "argc":1},
13951809 {"cmd":"sign", "label":"Sign arbitrary string using the secret key", "param":"STRING-TO-SIGN", "func":cmd_sign, "argc":1},
1810 {"cmd":"signurl", "label":"Sign an S3 URL to provide limited public access with expiry", "param":"s3://BUCKET/OBJECT expiry_epoch", "func":cmd_signurl, "argc":2},
13961811 {"cmd":"fixbucket", "label":"Fix invalid file names in a bucket", "param":"s3://BUCKET[/PREFIX]", "func":cmd_fixbucket, "argc":1},
13971812
13981813 ## Website commands
14151830 for cmd in commands_list:
14161831 help += " %s\n %s %s %s\n" % (cmd["label"], progname, cmd["cmd"], cmd["param"])
14171832 return help
1833
1834
1835 def update_acl(s3, uri, seq_label=""):
1836 something_changed = False
1837 acl = s3.get_acl(uri)
1838 debug(u"acl: %s - %r" % (uri, acl.grantees))
1839 if cfg.acl_public == True:
1840 if acl.isAnonRead():
1841 info(u"%s: already Public, skipping %s" % (uri, seq_label))
1842 else:
1843 acl.grantAnonRead()
1844 something_changed = True
1845 elif cfg.acl_public == False: # we explicitely check for False, because it could be None
1846 if not acl.isAnonRead():
1847 info(u"%s: already Private, skipping %s" % (uri, seq_label))
1848 else:
1849 acl.revokeAnonRead()
1850 something_changed = True
1851
1852 # update acl with arguments
1853 # grant first and revoke later, because revoke has priority
1854 if cfg.acl_grants:
1855 something_changed = True
1856 for grant in cfg.acl_grants:
1857 acl.grant(**grant)
1858
1859 if cfg.acl_revokes:
1860 something_changed = True
1861 for revoke in cfg.acl_revokes:
1862 acl.revoke(**revoke)
1863
1864 if not something_changed:
1865 return
1866
1867 retsponse = s3.set_acl(uri, acl)
1868 if retsponse['status'] == 200:
1869 if cfg.acl_public in (True, False):
1870 set_to_acl = cfg.acl_public and "Public" or "Private"
1871 output(u"%s: ACL set to %s %s" % (uri, set_to_acl, seq_label))
1872 else:
1873 output(u"%s: ACL updated" % uri)
14181874
14191875 class OptionMimeType(Option):
14201876 def check_mimetype(option, opt, value):
14771933 optparser.set_defaults(config = config_file)
14781934 optparser.set_defaults(verbosity = default_verbosity)
14791935
1480 optparser.add_option( "--configure", dest="run_configure", action="store_true", help="Invoke interactive (re)configuration tool. Optionally use as '--configure s3://come-bucket' to test access to a specific bucket instead of attempting to list them all.")
1936 optparser.add_option( "--configure", dest="run_configure", action="store_true", help="Invoke interactive (re)configuration tool. Optionally use as '--configure s3://some-bucket' to test access to a specific bucket instead of attempting to list them all.")
14811937 optparser.add_option("-c", "--config", dest="config", metavar="FILE", help="Config file name. Defaults to %default")
14821938 optparser.add_option( "--dump-config", dest="dump_config", action="store_true", help="Dump current configuration after parsing config files and command line options and exit.")
1939 optparser.add_option( "--access_key", dest="access_key", help="AWS Access Key")
1940 optparser.add_option( "--secret_key", dest="secret_key", help="AWS Secret Key")
14831941
14841942 optparser.add_option("-n", "--dry-run", dest="dry_run", action="store_true", help="Only show what should be uploaded or downloaded but don't actually do it. May still perform S3 requests to get bucket listings and other information though (only for file transfer commands)")
14851943
14871945 optparser.add_option( "--no-encrypt", dest="encrypt", action="store_false", help="Don't encrypt files.")
14881946 optparser.add_option("-f", "--force", dest="force", action="store_true", help="Force overwrite and other dangerous operations.")
14891947 optparser.add_option( "--continue", dest="get_continue", action="store_true", help="Continue getting a partially downloaded file (only for [get] command).")
1948 optparser.add_option( "--continue-put", dest="put_continue", action="store_true", help="Continue uploading partially uploaded files or multipart upload parts. Restarts/parts files that don't have matching size and md5. Skips files/parts that do. Note: md5sum checks are not always sufficient to check (part) file equality. Enable this at your own risk.")
1949 optparser.add_option( "--upload-id", dest="upload_id", help="UploadId for Multipart Upload, in case you want continue an existing upload (equivalent to --continue-put) and there are multiple partial uploads. Use s3cmd multipart [URI] to see what UploadIds are associated with the given URI.")
14901950 optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).")
14911951 optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.")
14921952 optparser.add_option( "--check-md5", dest="check_md5", action="store_true", help="Check MD5 sums when comparing files for [sync]. (default)")
14961956 optparser.add_option( "--acl-grant", dest="acl_grants", type="s3acl", action="append", metavar="PERMISSION:EMAIL or USER_CANONICAL_ID", help="Grant stated permission to a given amazon user. Permission is one of: read, write, read_acp, write_acp, full_control, all")
14971957 optparser.add_option( "--acl-revoke", dest="acl_revokes", type="s3acl", action="append", metavar="PERMISSION:USER_CANONICAL_ID", help="Revoke stated permission for a given amazon user. Permission is one of: read, write, read_acp, wr ite_acp, full_control, all")
14981958
1959 optparser.add_option("-D", "--restore-days", dest="restore_days", action="store", help="Number of days to keep restored file available (only for 'restore' command).", metavar="NUM")
1960
14991961 optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]")
15001962 optparser.add_option( "--no-delete-removed", dest="delete_removed", action="store_false", help="Don't delete remote objects.")
1963 optparser.add_option( "--delete-after", dest="delete_after", action="store_true", help="Perform deletes after new uploads [sync]")
1964 optparser.add_option( "--delay-updates", dest="delay_updates", action="store_true", help="Put all updated files into place at end [sync]")
1965 optparser.add_option( "--max-delete", dest="max_delete", action="store", help="Do not delete more than NUM files. [del] and [sync]", metavar="NUM")
1966 optparser.add_option( "--add-destination", dest="additional_destinations", action="append", help="Additional destination for parallel uploads, in addition to last arg. May be repeated.")
1967 optparser.add_option( "--delete-after-fetch", dest="delete_after_fetch", action="store_true", help="Delete remote objects after fetching to local file (only for [get] and [sync] commands).")
15011968 optparser.add_option("-p", "--preserve", dest="preserve_attrs", action="store_true", help="Preserve filesystem attributes (mode, ownership, timestamps). Default for [sync] command.")
15021969 optparser.add_option( "--no-preserve", dest="preserve_attrs", action="store_false", help="Don't store FS attributes")
15031970 optparser.add_option( "--exclude", dest="exclude", action="append", metavar="GLOB", help="Filenames and paths matching GLOB will be excluded from sync")
15081975 optparser.add_option( "--include-from", dest="include_from", action="append", metavar="FILE", help="Read --include GLOBs from FILE")
15091976 optparser.add_option( "--rinclude", dest="rinclude", action="append", metavar="REGEXP", help="Same as --include but uses REGEXP (regular expression) instead of GLOB")
15101977 optparser.add_option( "--rinclude-from", dest="rinclude_from", action="append", metavar="FILE", help="Read --rinclude REGEXPs from FILE")
1511
1512 optparser.add_option( "--bucket-location", dest="bucket_location", help="Datacentre to create bucket in. As of now the datacenters are: US (default), EU, us-west-1, and ap-southeast-1")
1978 optparser.add_option( "--ignore-failed-copy", dest="ignore_failed_copy", action="store_true", help="Don't exit unsuccessfully because of missing keys")
1979
1980 optparser.add_option( "--files-from", dest="files_from", action="append", metavar="FILE", help="Read list of source-file names from FILE. Use - to read from stdin.")
1981 optparser.add_option( "--bucket-location", dest="bucket_location", help="Datacentre to create bucket in. As of now the datacenters are: US (default), EU, ap-northeast-1, ap-southeast-1, sa-east-1, us-west-1 and us-west-2")
15131982 optparser.add_option( "--reduced-redundancy", "--rr", dest="reduced_redundancy", action="store_true", help="Store object with 'Reduced redundancy'. Lower per-GB price. [put, cp, mv]")
15141983
15151984 optparser.add_option( "--access-logging-target-prefix", dest="log_target_prefix", help="Target prefix for access logs (S3 URI) (for [cfmodify] and [accesslog] commands)")
15161985 optparser.add_option( "--no-access-logging", dest="log_target_prefix", action="store_false", help="Disable access logging (for [cfmodify] and [accesslog] commands)")
15171986
1518 optparser.add_option( "--default-mime-type", dest="default_mime_type", action="store_true", help="Default MIME-type for stored objects. Application default is binary/octet-stream.")
1519 optparser.add_option( "--guess-mime-type", dest="guess_mime_type", action="store_true", help="Guess MIME-type of files by their extension or mime magic. Fall back to default MIME-Type as specified by --default-mime-type option")
1987 optparser.add_option( "--default-mime-type", dest="default_mime_type", type="mimetype", action="store", help="Default MIME-type for stored objects. Application default is binary/octet-stream.")
1988 optparser.add_option("-M", "--guess-mime-type", dest="guess_mime_type", action="store_true", help="Guess MIME-type of files by their extension or mime magic. Fall back to default MIME-Type as specified by --default-mime-type option")
15201989 optparser.add_option( "--no-guess-mime-type", dest="guess_mime_type", action="store_false", help="Don't guess MIME-type and use the default type instead.")
1990 optparser.add_option( "--no-mime-magic", dest="use_mime_magic", action="store_false", help="Don't use mime magic when guessing MIME-type.")
15211991 optparser.add_option("-m", "--mime-type", dest="mime_type", type="mimetype", metavar="MIME/TYPE", help="Force MIME-type. Override both --default-mime-type and --guess-mime-type.")
15221992
15231993 optparser.add_option( "--add-header", dest="add_header", action="append", metavar="NAME:VALUE", help="Add a given HTTP header to the upload request. Can be used multiple times. For instance set 'Expires' or 'Cache-Control' headers (or both) using this options if you like.")
15241994
1995 optparser.add_option( "--server-side-encryption", dest="server_side_encryption", action="store_true", help="Specifies that server-side encryption will be used when putting objects.")
1996
15251997 optparser.add_option( "--encoding", dest="encoding", metavar="ENCODING", help="Override autodetected terminal and filesystem encoding (character set). Autodetected: %s" % preferred_encoding)
1998 optparser.add_option( "--disable-content-encoding", dest="add_content_encoding", action="store_false", help="Don't include a Content-encoding header to the the uploaded objects")
1999 optparser.add_option( "--add-encoding-exts", dest="add_encoding_exts", metavar="EXTENSIONs", help="Add encoding to these comma delimited extensions i.e. (css,js,html) when uploading to S3 )")
15262000 optparser.add_option( "--verbatim", dest="urlencoding_mode", action="store_const", const="verbatim", help="Use the S3 name as given on the command line. No pre-processing, encoding, etc. Use with caution!")
15272001
15282002 optparser.add_option( "--disable-multipart", dest="enable_multipart", action="store_false", help="Disable multipart upload on files bigger than --multipart-chunk-size-mb")
15312005 optparser.add_option( "--list-md5", dest="list_md5", action="store_true", help="Include MD5 sums in bucket listings (only for 'ls' command).")
15322006 optparser.add_option("-H", "--human-readable-sizes", dest="human_readable_sizes", action="store_true", help="Print sizes in human readable form (eg 1kB instead of 1234).")
15332007
1534 optparser.add_option( "--ws-index", dest="website_index", action="store", help="Name of error-document (only for [ws-create] command)")
1535 optparser.add_option( "--ws-error", dest="website_error", action="store", help="Name of index-document (only for [ws-create] command)")
2008 optparser.add_option( "--ws-index", dest="website_index", action="store", help="Name of index-document (only for [ws-create] command)")
2009 optparser.add_option( "--ws-error", dest="website_error", action="store", help="Name of error-document (only for [ws-create] command)")
15362010
15372011 optparser.add_option( "--progress", dest="progress_meter", action="store_true", help="Display progress meter (default on TTY).")
15382012 optparser.add_option( "--no-progress", dest="progress_meter", action="store_false", help="Don't display progress meter (default on non-TTY).")
15392013 optparser.add_option( "--enable", dest="enable", action="store_true", help="Enable given CloudFront distribution (only for [cfmodify] command)")
15402014 optparser.add_option( "--disable", dest="enable", action="store_false", help="Enable given CloudFront distribution (only for [cfmodify] command)")
15412015 optparser.add_option( "--cf-invalidate", dest="invalidate_on_cf", action="store_true", help="Invalidate the uploaded filed in CloudFront. Also see [cfinval] command.")
2016 # joseprio: adding options to invalidate the default index and the default
2017 # index root
2018 optparser.add_option( "--cf-invalidate-default-index", dest="invalidate_default_index_on_cf", action="store_true", help="When using Custom Origin and S3 static website, invalidate the default index file.")
2019 optparser.add_option( "--cf-no-invalidate-default-index-root", dest="invalidate_default_index_root_on_cf", action="store_false", help="When using Custom Origin and S3 static website, don't invalidate the path to the default index file.")
15422020 optparser.add_option( "--cf-add-cname", dest="cf_cnames_add", action="append", metavar="CNAME", help="Add given CNAME to a CloudFront distribution (only for [cfcreate] and [cfmodify] commands)")
15432021 optparser.add_option( "--cf-remove-cname", dest="cf_cnames_remove", action="append", metavar="CNAME", help="Remove given CNAME from a CloudFront distribution (only for [cfmodify] command)")
15442022 optparser.add_option( "--cf-comment", dest="cf_comment", action="store", metavar="COMMENT", help="Set COMMENT for a given CloudFront distribution (only for [cfcreate] and [cfmodify] commands)")
15472025 optparser.add_option("-d", "--debug", dest="verbosity", action="store_const", const=logging.DEBUG, help="Enable debug output.")
15482026 optparser.add_option( "--version", dest="show_version", action="store_true", help="Show s3cmd version (%s) and exit." % (PkgInfo.version))
15492027 optparser.add_option("-F", "--follow-symlinks", dest="follow_symlinks", action="store_true", default=False, help="Follow symbolic links as if they are regular files")
2028 optparser.add_option( "--cache-file", dest="cache_file", action="store", default="", metavar="FILE", help="Cache FILE containing local source MD5 values")
2029 optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False, help="Silence output on stdout")
15502030
15512031 optparser.set_usage(optparser.usage + " COMMAND [parameters]")
15522032 optparser.set_description('S3cmd is a tool for managing objects in '+
15542034 '"buckets" and uploading, downloading and removing '+
15552035 '"objects" from these buckets.')
15562036 optparser.epilog = format_commands(optparser.get_prog_name(), commands_list)
1557 optparser.epilog += ("\nFor more informations see the progect homepage:\n%s\n" % PkgInfo.url)
2037 optparser.epilog += ("\nFor more information see the project homepage:\n%s\n" % PkgInfo.url)
15582038 optparser.epilog += ("\nConsider a donation if you have found s3cmd useful:\n%s/donate\n" % PkgInfo.url)
15592039
15602040 (options, args) = optparser.parse_args()
15682048 if options.show_version:
15692049 output(u"s3cmd version %s" % PkgInfo.version)
15702050 sys.exit(0)
2051
2052 if options.quiet:
2053 try:
2054 f = open("/dev/null", "w")
2055 sys.stdout.close()
2056 sys.stdout = f
2057 except IOError:
2058 warning(u"Unable to open /dev/null: --quiet disabled.")
15712059
15722060 ## Now finally parse the config file
15732061 if not options.config:
16322120 if options.check_md5 == False:
16332121 try:
16342122 cfg.sync_checks.remove("md5")
2123 cfg.preserve_attrs_list.remove("md5")
16352124 except Exception:
16362125 pass
1637 if options.check_md5 == True and cfg.sync_checks.count("md5") == 0:
1638 cfg.sync_checks.append("md5")
2126 if options.check_md5 == True:
2127 if cfg.sync_checks.count("md5") == 0:
2128 cfg.sync_checks.append("md5")
2129 if cfg.preserve_attrs_list.count("md5") == 0:
2130 cfg.preserve_attrs_list.append("md5")
16392131
16402132 ## Update Config with other parameters
16412133 for option in cfg.option_list():
16572149 if cfg.multipart_chunk_size_mb > MultiPartUpload.MAX_CHUNK_SIZE_MB:
16582150 raise ParameterError("Chunk size %d MB is too large, must be <= %d MB. Please adjust --multipart-chunk-size-mb" % (cfg.multipart_chunk_size_mb, MultiPartUpload.MAX_CHUNK_SIZE_MB))
16592151
2152 ## If an UploadId was provided, set put_continue True
2153 if options.upload_id is not None:
2154 cfg.upload_id = options.upload_id
2155 cfg.put_continue = True
2156
2157 if cfg.upload_id and not cfg.multipart_chunk_size_mb:
2158 raise ParameterError("Must have --multipart-chunk-size-mb if using --put-continue or --upload-id")
2159
16602160 ## CloudFront's cf_enable and Config's enable share the same --enable switch
16612161 options.cf_enable = options.enable
16622162
16722172 except AttributeError:
16732173 ## Some CloudFront.Cmd.Options() options are not settable from command line
16742174 pass
2175
2176 if options.additional_destinations:
2177 cfg.additional_destinations = options.additional_destinations
2178 if options.files_from:
2179 cfg.files_from = options.files_from
16752180
16762181 ## Set output and filesystem encoding for printing out filenames.
16772182 sys.stdout = codecs.getwriter(cfg.encoding)(sys.stdout, "replace")
17332238 sys.exit(1)
17342239
17352240 if len(args) < commands[command]["argc"]:
1736 error(u"Not enough paramters for command '%s'" % command)
2241 error(u"Not enough parameters for command '%s'" % command)
17372242 sys.exit(1)
17382243
17392244 try:
17422247 error(u"S3 error: %s" % e)
17432248 sys.exit(1)
17442249
1745 def report_exception(e):
2250 def report_exception(e, msg=''):
17462251 sys.stderr.write("""
17472252 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
17482253 An unexpected error has occurred.
1749 Please report the following lines to:
2254 Please try reproducing the error using
2255 the latest s3cmd code from the git master
2256 branch found at:
2257 https://github.com/s3tools/s3cmd
2258 If the error persists, please report the
2259 following lines (removing any private
2260 info as necessary) to:
17502261 s3tools-bugs@lists.sourceforge.net
2262 %s
17512263 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
17522264
1753 """)
2265 """ % msg)
2266 s = ' '.join(sys.argv)
2267 sys.stderr.write("""Invoked as: %s""" % s)
2268
17542269 tb = traceback.format_exc(sys.exc_info())
17552270 e_class = str(e.__class__)
17562271 e_class = e_class[e_class.rfind(".")+1 : -2]
17592274 sys.stderr.write("S3cmd: %s\n" % PkgInfo.version)
17602275 except NameError:
17612276 sys.stderr.write("S3cmd: unknown version. Module import problem?\n")
2277 sys.stderr.write("python: %s\n" % sys.version)
2278 sys.stderr.write("environment LANG=%s\n" % os.getenv("LANG"))
17622279 sys.stderr.write("\n")
17632280 sys.stderr.write(unicode(tb, errors="replace"))
17642281
17722289 sys.stderr.write("""
17732290 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
17742291 An unexpected error has occurred.
1775 Please report the above lines to:
2292 Please try reproducing the error using
2293 the latest s3cmd code from the git master
2294 branch found at:
2295 https://github.com/s3tools/s3cmd
2296 If the error persists, please report the
2297 above lines (removing any private
2298 info as necessary) to:
17762299 s3tools-bugs@lists.sourceforge.net
17772300 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
17782301 """)
17872310 from S3.S3 import S3
17882311 from S3.Config import Config
17892312 from S3.SortedDict import SortedDict
2313 from S3.FileDict import FileDict
17902314 from S3.S3Uri import S3Uri
17912315 from S3 import Utils
17922316 from S3.Utils import *
18142338 sys.stderr.write("See ya!\n")
18152339 sys.exit(1)
18162340
2341 except MemoryError:
2342 msg = """
2343 MemoryError! You have exceeded the amount of memory available for this process.
2344 This usually occurs when syncing >750,000 files on a 32-bit python instance.
2345 The solutions to this are:
2346 1) sync several smaller subtrees; or
2347 2) use a 64-bit python on a 64-bit OS with >8GB RAM
2348 """
2349 sys.stderr.write(msg)
2350 sys.exit(1)
2351
2352 except UnicodeEncodeError, e:
2353 lang = os.getenv("LANG")
2354 msg = """
2355 You have encountered a UnicodeEncodeError. Your environment
2356 variable LANG=%s may not specify a Unicode encoding (e.g. UTF-8).
2357 Please set LANG=en_US.UTF-8 or similar in your environment before
2358 invoking s3cmd.
2359 """ % lang
2360 report_exception(e, msg)
2361 sys.exit(1)
2362
18172363 except Exception, e:
18182364 report_exception(e)
18192365 sys.exit(1)
+106
-10
s3cmd.1 less more
0
1 .\" !!! IMPORTANT: This file is generated from s3cmd --help output using format-manpage.pl
2 .\" !!! Do your changes either in s3cmd file or in 'format-manpage.pl' otherwise
3 .\" !!! they will be overwritten!
04
15 .TH s3cmd 1
26 .SH NAME
5660 s3cmd \fBsetacl\fR \fIs3://BUCKET[/OBJECT]\fR
5761 Modify Access control list for Bucket or Files
5862 .TP
63 s3cmd \fBsetpolicy\fR \fIFILE s3://BUCKET\fR
64 Modify Bucket Policy
65 .TP
66 s3cmd \fBdelpolicy\fR \fIs3://BUCKET\fR
67 Delete Bucket Policy
68 .TP
69 s3cmd \fBmultipart\fR \fIs3://BUCKET [Id]\fR
70 show multipart uploads
71 .TP
72 s3cmd \fBabortmp\fR \fIs3://BUCKET/OBJECT Id\fR
73 abort a multipart upload
74 .TP
75 s3cmd \fBlistmp\fR \fIs3://BUCKET/OBJECT Id\fR
76 list parts of a multipart upload
77 .TP
5978 s3cmd \fBaccesslog\fR \fIs3://BUCKET\fR
6079 Enable/disable bucket access logging
6180 .TP
6281 s3cmd \fBsign\fR \fISTRING-TO-SIGN\fR
6382 Sign arbitrary string using the secret key
83 .TP
84 s3cmd \fBsignurl\fR \fIs3://BUCKET/OBJECT expiry_epoch\fR
85 Sign an S3 URL to provide limited public access with expiry
6486 .TP
6587 s3cmd \fBfixbucket\fR \fIs3://BUCKET[/PREFIX]\fR
6688 Fix invalid file names in a bucket
115137 .TP
116138 \fB\-\-configure\fR
117139 Invoke interactive (re)configuration tool. Optionally
118 use as '\fB--configure\fR s3://come-bucket' to test access
140 use as '\fB--configure\fR s3://some-bucket' to test access
119141 to a specific bucket instead of attempting to list
120142 them all.
121143 .TP
125147 \fB\-\-dump\-config\fR
126148 Dump current configuration after parsing config files
127149 and command line options and exit.
150 .TP
151 \fB\-\-access_key\fR=ACCESS_KEY
152 AWS Access Key
153 .TP
154 \fB\-\-secret_key\fR=SECRET_KEY
155 AWS Secret Key
128156 .TP
129157 \fB\-n\fR, \fB\-\-dry\-run\fR
130158 Only show what should be uploaded or downloaded but
145173 Continue getting a partially downloaded file (only for
146174 [get] command).
147175 .TP
176 \fB\-\-continue\-put\fR
177 Continue uploading partially uploaded files or
178 multipart upload parts. Restarts/parts files that
179 don't have matching size and md5. Skips files/parts
180 that do. Note: md5sum checks are not always
181 sufficient to check (part) file equality. Enable this
182 at your own risk.
183 .TP
184 \fB\-\-upload\-id\fR=UPLOAD_ID
185 UploadId for Multipart Upload, in case you want
186 continue an existing upload (equivalent to \fB--continue-\fR
187 put) and there are multiple partial uploads. Use
188 s3cmd multipart [URI] to see what UploadIds are
189 associated with the given URI.
190 .TP
148191 \fB\-\-skip\-existing\fR
149192 Skip over files that exist at the destination (only
150193 for [get] and [sync] commands).
185228 \fB\-\-no\-delete\-removed\fR
186229 Don't delete remote objects.
187230 .TP
231 \fB\-\-delete\-after\fR
232 Perform deletes after new uploads [sync]
233 .TP
234 \fB\-\-delay\-updates\fR
235 Put all updated files into place at end [sync]
236 .TP
237 \fB\-\-max\-delete\fR=NUM
238 Do not delete more than NUM files. [del] and [sync]
239 .TP
240 \fB\-\-add\-destination\fR=ADDITIONAL_DESTINATIONS
241 Additional destination for parallel uploads, in
242 addition to last arg. May be repeated.
243 .TP
244 \fB\-\-delete\-after\-fetch\fR
245 Delete remote objects after fetching to local file
246 (only for [get] and [sync] commands).
247 .TP
188248 \fB\-p\fR, \fB\-\-preserve\fR
189249 Preserve filesystem attributes (mode, ownership,
190250 timestamps). Default for [sync] command.
221281 \fB\-\-rinclude\-from\fR=FILE
222282 Read --rinclude REGEXPs from FILE
223283 .TP
284 \fB\-\-ignore\-failed\-copy\fR
285 Don't exit unsuccessfully because of missing keys
286 .TP
287 \fB\-\-files\-from\fR=FILE
288 Read list of source-file names from FILE. Use - to
289 read from stdin.
290 .TP
224291 \fB\-\-bucket\-location\fR=BUCKET_LOCATION
225292 Datacentre to create bucket in. As of now the
226 datacenters are: US (default), EU, us-west-1, and ap-
227 southeast-1
293 datacenters are: US (default), EU, ap-northeast-1, ap-
294 southeast-1, sa-east-1, us-west-1 and us-west-2
228295 .TP
229296 \fB\-\-reduced\-redundancy\fR, \fB\-\-rr\fR
230297 Store object with 'Reduced redundancy'. Lower per-GB
238305 Disable access logging (for [cfmodify] and [accesslog]
239306 commands)
240307 .TP
241 \fB\-\-default\-mime\-type\fR
308 \fB\-\-default\-mime\-type\fR=DEFAULT_MIME_TYPE
242309 Default MIME-type for stored objects. Application
243310 default is binary/octet-stream.
244311 .TP
245 \fB\-\-guess\-mime\-type\fR
312 \fB\-M\fR, \fB\-\-guess\-mime\-type\fR
246313 Guess MIME-type of files by their extension or mime
247314 magic. Fall back to default MIME-Type as specified by
248315 \fB--default-mime-type\fR option
250317 \fB\-\-no\-guess\-mime\-type\fR
251318 Don't guess MIME-type and use the default type
252319 instead.
320 .TP
321 \fB\-\-no\-mime\-magic\fR
322 Don't use mime magic when guessing MIME-type.
253323 .TP
254324 \fB\-m\fR MIME/TYPE, \fB\-\-mime\-type\fR=MIME/TYPE
255325 Force MIME-type. Override both \fB--default-mime-type\fR and
261331 'Cache-Control' headers (or both) using this options
262332 if you like.
263333 .TP
334 \fB\-\-server\-side\-encryption\fR
335 Specifies that server-side encryption will be used
336 when putting objects.
337 .TP
264338 \fB\-\-encoding\fR=ENCODING
265339 Override autodetected terminal and filesystem encoding
266340 (character set). Autodetected: UTF-8
341 .TP
342 \fB\-\-disable\-content\-encoding\fR
343 Don't include a Content-encoding header to the the
344 uploaded objects
345 .TP
346 \fB\-\-add\-encoding\-exts\fR=EXTENSIONs
347 Add encoding to these comma delimited extensions i.e.
348 (css,js,html) when uploading to S3 )
267349 .TP
268350 \fB\-\-verbatim\fR
269351 Use the S3 name as given on the command line. No pre-
290372 1234).
291373 .TP
292374 \fB\-\-ws\-index\fR=WEBSITE_INDEX
375 Name of index-document (only for [ws-create] command)
376 .TP
377 \fB\-\-ws\-error\fR=WEBSITE_ERROR
293378 Name of error-document (only for [ws-create] command)
294 .TP
295 \fB\-\-ws\-error\fR=WEBSITE_ERROR
296 Name of index-document (only for [ws-create] command)
297379 .TP
298380 \fB\-\-progress\fR
299381 Display progress meter (default on TTY).
312394 \fB\-\-cf\-invalidate\fR
313395 Invalidate the uploaded filed in CloudFront. Also see
314396 [cfinval] command.
397 .TP
398 \fB\-\-cf\-invalidate\-default\-index\fR
399 When using Custom Origin and S3 static website,
400 invalidate the default index file.
401 .TP
402 \fB\-\-cf\-no\-invalidate\-default\-index\-root\fR
403 When using Custom Origin and S3 static website, don't
404 invalidate the path to the default index file.
315405 .TP
316406 \fB\-\-cf\-add\-cname\fR=CNAME
317407 Add given CNAME to a CloudFront distribution (only for
339429 Enable debug output.
340430 .TP
341431 \fB\-\-version\fR
342 Show s3cmd version (1.1.0-beta3) and exit.
432 Show s3cmd version (1.5.0-beta1) and exit.
343433 .TP
344434 \fB\-F\fR, \fB\-\-follow\-symlinks\fR
345435 Follow symbolic links as if they are regular files
436 .TP
437 \fB\-\-cache\-file\fR=FILE
438 Cache FILE containing local source MD5 values
439 .TP
440 \fB\-q\fR, \fB\-\-quiet\fR
441 Silence output on stdout
346442
347443
348444 .SH EXAMPLES
434530 .SH AUTHOR
435531 Written by Michal Ludvig <mludvig@logix.net.nz> and 15+ contributors
436532 .SH CONTACT, SUPPORT
437 Prefered way to get support is our mailing list:
533 Preferred way to get support is our mailing list:
438534 .I s3tools\-general@lists.sourceforge.net
439535 .SH REPORTING BUGS
440536 Report bugs to
+155
-0
s3cmd.spec.in less more
0 %{!?python_sitelib: %define python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")}
1
2 %global commit ##COMMIT##
3 %global shortcommit ##SHORTCOMMIT##
4
5 Name: s3cmd
6 Version: ##VERSION##
7 Release: 0.3.git%{shortcommit}%{?dist}
8 Summary: Tool for accessing Amazon Simple Storage Service
9
10 Group: Applications/Internet
11 License: GPLv2
12 URL: http://s3tools.logix.cz/s3cmd
13 # git clone git@github.com:mdomsch/s3cmd.git
14 # git checkout -b origin/merge
15 #git archive --format tar --prefix s3cmd-1.1.0-beta3-2dfe4a65/ HEAD | gzip -c > s3cmd-1.1.0-beta1-2dfe4a65.tar.gz
16
17 Source0: https://github.com/s3tools/s3cmd/archive/%{commit}/%{name}-%{version}-%{shortcommit}.tar.gz
18 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
19 BuildArch: noarch
20
21 %if %{!?fedora:16}%{?fedora} < 16 || %{!?rhel:7}%{?rhel} < 7
22 BuildRequires: python-devel
23 %else
24 BuildRequires: python2-devel
25 %endif
26 %if %{!?fedora:8}%{?fedora} < 8 || %{!?rhel:6}%{?rhel} < 6
27 # This is in standard library since 2.5
28 Requires: python-elementtree
29 %endif
30
31 %description
32 S3cmd lets you copy files from/to Amazon S3
33 (Simple Storage Service) using a simple to use
34 command line client.
35
36
37 %prep
38 %setup -q -n s3cmd-%{commit}
39
40 %build
41
42
43 %install
44 rm -rf $RPM_BUILD_ROOT
45 S3CMD_PACKAGING=Yes python setup.py install --prefix=%{_prefix} --root=$RPM_BUILD_ROOT
46 install -d $RPM_BUILD_ROOT%{_mandir}/man1
47 install -m 644 s3cmd.1 $RPM_BUILD_ROOT%{_mandir}/man1
48
49
50 %clean
51 rm -rf $RPM_BUILD_ROOT
52
53
54 %files
55 %defattr(-,root,root,-)
56 %{_bindir}/s3cmd
57 %{_mandir}/man1/s3cmd.1*
58 %{python_sitelib}/S3
59 %if 0%{?fedora} >= 9 || 0%{?rhel} >= 6
60 %{python_sitelib}/s3cmd*.egg-info
61 %endif
62 %doc NEWS README
63
64
65 %changelog
66 * Sun Feb 02 2014 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.3.git
67 - upstream 1.5.0-beta1 plus newer upstream fixes
68
69 * Wed May 29 2013 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.2.gita122d97
70 - more upstream bugfixes
71 - drop pyxattr dep, that codepath got dropped in this release
72
73 * Mon May 20 2013 Matt Domsch <mdomsch@fedoraproject.org> - 1.5.0-0.1.gitb1ae0fbe
74 - upstream 1.5.0-alpha3 plus fixes
75 - add dep on pyxattr for the --xattr option
76
77 * Tue Jun 19 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.4.git11e5755e
78 - add local MD5 cache
79
80 * Mon Jun 18 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.3.git7de0789d
81 - parallelize local->remote syncs
82
83 * Mon Jun 18 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.2.gitf881b162
84 - add hardlink / duplicate file detection support
85
86 * Fri Mar 9 2012 Matt Domsch <mdomsch@fedoraproject.org> - 1.1.0-0.1.git2dfe4a65
87 - build from git for mdomsch patches to s3cmd sync
88
89 * Thu Feb 23 2012 Dennis Gilmore <dennis@ausil.us> - 1.0.1-1
90 - update to 1.0.1 release
91
92 * Sat Jan 14 2012 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 1.0.0-4
93 - Rebuilt for https://fedoraproject.org/wiki/Fedora_17_Mass_Rebuild
94
95 * Thu May 05 2011 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 1.0.0-3
96 - No hashlib hackery
97
98 * Wed Feb 09 2011 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 1.0.0-2
99 - Rebuilt for https://fedoraproject.org/wiki/Fedora_15_Mass_Rebuild
100
101 * Tue Jan 11 2011 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 1.0.0-1
102 - New upstream release
103
104 * Mon Nov 29 2010 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 0.9.9.91-3
105 - Patch for broken f14 httplib
106
107 * Thu Jul 22 2010 David Malcolm <dmalcolm@redhat.com> - 0.9.9.91-2.1
108 - Rebuilt for https://fedoraproject.org/wiki/Features/Python_2.7/MassRebuild
109
110 * Wed Apr 28 2010 Lubomir Rintel (GoodData) <lubo.rintel@gooddata.com> - 0.9.9.91-1.1
111 - Do not use sha1 from hashlib
112
113 * Sun Feb 21 2010 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.9.91-1
114 - New upstream release
115
116 * Sun Jul 26 2009 Fedora Release Engineering <rel-eng@lists.fedoraproject.org> - 0.9.9-2
117 - Rebuilt for https://fedoraproject.org/wiki/Fedora_12_Mass_Rebuild
118
119 * Tue Feb 24 2009 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.9-1
120 - New upstream release
121
122 * Sat Nov 29 2008 Ignacio Vazquez-Abrams <ivazqueznet+rpm@gmail.com> - 0.9.8.4-2
123 - Rebuild for Python 2.6
124
125 * Tue Nov 11 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.4-1
126 - New upstream release, URI encoding patch upstreamed
127
128 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-4
129 - Try 3/65536
130
131 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-3
132 - Whoops, forgot to actually apply the patch.
133
134 * Fri Sep 26 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-2
135 - Fix listing of directories with special characters in names
136
137 * Thu Jul 31 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.3-1
138 - New upstream release: Avoid running out-of-memory in MD5'ing large files.
139
140 * Fri Jul 25 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.2-1.1
141 - Fix a typo
142
143 * Tue Jul 15 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.2-1
144 - New upstream
145
146 * Fri Jul 04 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-3
147 - Be satisfied with ET provided by 2.5 python
148
149 * Fri Jul 04 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-2
150 - Added missing python-devel BR, thanks to Marek Mahut
151 - Packaged the Python egg file
152
153 * Wed Jul 02 2008 Lubomir Rintel (Good Data) <lubo.rintel@gooddata.com> - 0.9.8.1-1
154 - Initial packaging attempt
testsuite.tar.gz less more
Binary diff not shown
+7
-0
upload-to-sf.sh less more
0 #!/bin/sh
1
2 VERSION=$(./s3cmd --version | awk '{print $NF}')
3 echo -e "Uploading \033[32ms3cmd \033[31m${VERSION}\033[0m ..."
4 #rsync -avP dist/s3cmd-${VERSION}.* ludvigm@frs.sourceforge.net:uploads/
5 ln -f NEWS README.txt
6 rsync -avP dist/s3cmd-${VERSION}.* README.txt ludvigm,s3tools@frs.sourceforge.net:/home/frs/project/s/s3/s3tools/s3cmd/${VERSION}/