Codebase list s3cmd / 95869e0
Imported Upstream version 0.9.9~rc3 Gianfranco Costamagna 8 years ago
14 changed file(s) with 2340 addition(s) and 670 deletion(s). Raw diff Collapse all Expand all
+59
-0
NEWS less more
0 s3cmd 0.9.9-rc3 - 2009-02-02
1 ===============
2 * Fixed crash in S3Error().__str__() (typically Amazon's Internal
3 errors, etc).
4
5 s3cmd 0.9.9-rc2 - 2009-01-30
6 ===============
7 * Fixed s3cmd crash when put / get / sync found
8 zero files to transfer.
9 * In --dry-run output files to be deleted only
10 with --delete-removed, otherwise users get scared.
11
12 s3cmd 0.9.9-rc1 - 2009-01-27
13 ===============
14 * CloudFront support through cfcreate, cfdelete,
15 cfmodify and cfinfo commands.
16 * Added --include / --rinclude / --(r)include-from
17 options to override --exclude exclusions.
18 * Enabled --dry-run for [put] and [get].
19 * Fixed GPG (--encrypt) compatibility with Python 2.6.
20
21 s3cmd 0.9.9-pre5 - 2009-01-22
22 ================
23 * New command 'setacl' for setting ACL on existing objects.
24 * Recursive [put] with a slightly different semantic.
25 * Multiple sources for [sync] and slightly different semantics.
26 * Support for --dry-run with [sync]
27
28 s3cmd 0.9.9-pre4 - 2008-12-30
29 ================
30 * Support for non-recursive [ls]
31 * Support for multiple sources and recursive [get].
32 * Improved wildcard [get].
33 * New option --skip-existing for [get] and [sync].
34 * Improved Progress class (fixes Mac OS X)
35 * Fixed installation on Windows and Mac OS X.
36 * Don't print nasty backtrace on KeyboardInterrupt.
37 * Should work fine on non-UTF8 systems, provided all
38 the files are in current system encoding.
39 * System encoding can be overriden using --encoding.
40
41 s3cmd 0.9.9-pre3 - 2008-12-01
42 ================
43 * Bugfixes only
44 - Fixed sync from S3 to local
45 - Fixed progress meter with Unicode chars
46
47 s3cmd 0.9.9-pre2 - 2008-11-24
48 ================
49 * Implemented progress meter (--progress / --no-progress)
50 * Removing of non-empty buckets with --force
51 * Recursively remove objects from buckets with a given
52 prefix with --recursive (-r)
53 * Copying and moving objects, within or between buckets.
54 (Andrew Ryan)
55 * Continue getting partially downloaded files with --continue
56 * Improved resistance to communication errors (Connection
57 reset by peer, etc.)
58
059 s3cmd 0.9.8.4 - 2008-11-07
160 =============
261 * Stabilisation / bugfix release:
+5
-3
PKG-INFO less more
00 Metadata-Version: 1.0
11 Name: s3cmd
2 Version: 0.9.8.4
3 Summary: S3cmd is a tool for managing Amazon S3 storage space.
2 Version: 0.9.9-rc3
3 Summary: Command line tool for managing Amazon S3 and CloudFront services
44 Home-page: http://s3tools.logix.cz
55 Author: Michal Ludvig
66 Author-email: michal@logix.cz
99
1010 S3cmd lets you copy files from/to Amazon S3
1111 (Simple Storage Service) using a simple to use
12 command line client.
12 command line client. Supports rsync-like backup,
13 GPG encryption, and more. Also supports management
14 of Amazon's CloudFront content delivery network.
1315
1416
1517 Authors:
+157
-0
S3/ACL.py less more
0 ## Amazon S3 - Access Control List representation
1 ## Author: Michal Ludvig <michal@logix.cz>
2 ## http://www.logix.cz/michal
3 ## License: GPL Version 2
4
5 from Utils import *
6
7 try:
8 import xml.etree.ElementTree as ET
9 except ImportError:
10 import elementtree.ElementTree as ET
11
12 class Grantee(object):
13 ALL_USERS_URI = "http://acs.amazonaws.com/groups/global/AllUsers"
14
15 def __init__(self):
16 self.xsi_type = None
17 self.tag = None
18 self.name = None
19 self.display_name = None
20 self.permission = None
21
22 def __repr__(self):
23 return 'Grantee("%(tag)s", "%(name)s", "%(permission)s")' % {
24 "tag" : self.tag,
25 "name" : self.name,
26 "permission" : self.permission
27 }
28
29 def isAllUsers(self):
30 return self.tag == "URI" and self.name == Grantee.ALL_USERS_URI
31
32 def isAnonRead(self):
33 return self.isAllUsers and self.permission == "READ"
34
35 def getElement(self):
36 el = ET.Element("Grant")
37 grantee = ET.SubElement(el, "Grantee", {
38 'xmlns:xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
39 'xsi:type' : self.xsi_type
40 })
41 name = ET.SubElement(grantee, self.tag)
42 name.text = self.name
43 permission = ET.SubElement(el, "Permission")
44 permission.text = self.permission
45 return el
46
47 class GranteeAnonRead(Grantee):
48 def __init__(self):
49 Grantee.__init__(self)
50 self.xsi_type = "Group"
51 self.tag = "URI"
52 self.name = Grantee.ALL_USERS_URI
53 self.permission = "READ"
54
55 class ACL(object):
56 EMPTY_ACL = "<AccessControlPolicy><Owner><ID></ID></Owner><AccessControlList></AccessControlList></AccessControlPolicy>"
57
58 def __init__(self, xml = None):
59 if not xml:
60 xml = ACL.EMPTY_ACL
61
62 self.grantees = []
63 self.owner_id = ""
64 self.owner_nick = ""
65
66 tree = getTreeFromXml(xml)
67 self.parseOwner(tree)
68 self.parseGrants(tree)
69
70 def parseOwner(self, tree):
71 self.owner_id = tree.findtext(".//Owner//ID")
72 self.owner_nick = tree.findtext(".//Owner//DisplayName")
73
74 def parseGrants(self, tree):
75 for grant in tree.findall(".//Grant"):
76 grantee = Grantee()
77 g = grant.find(".//Grantee")
78 grantee.xsi_type = g.attrib['{http://www.w3.org/2001/XMLSchema-instance}type']
79 grantee.permission = grant.find('Permission').text
80 for el in g:
81 if el.tag == "DisplayName":
82 grantee.display_name = el.text
83 else:
84 grantee.tag = el.tag
85 grantee.name = el.text
86 self.grantees.append(grantee)
87
88 def getGrantList(self):
89 acl = {}
90 for grantee in self.grantees:
91 if grantee.display_name:
92 user = grantee.display_name
93 elif grantee.isAllUsers():
94 user = "*anon*"
95 else:
96 user = grantee.name
97 acl[user] = grantee.permission
98 return acl
99
100 def getOwner(self):
101 return { 'id' : self.owner_id, 'nick' : self.owner_nick }
102
103 def isAnonRead(self):
104 for grantee in self.grantees:
105 if grantee.isAnonRead():
106 return True
107 return False
108
109 def grantAnonRead(self):
110 if not self.isAnonRead():
111 self.grantees.append(GranteeAnonRead())
112
113 def revokeAnonRead(self):
114 self.grantees = [g for g in self.grantees if not g.isAnonRead()]
115
116 def __str__(self):
117 tree = getTreeFromXml(ACL.EMPTY_ACL)
118 tree.attrib['xmlns'] = "http://s3.amazonaws.com/doc/2006-03-01/"
119 owner = tree.find(".//Owner//ID")
120 owner.text = self.owner_id
121 acl = tree.find(".//AccessControlList")
122 for grantee in self.grantees:
123 acl.append(grantee.getElement())
124 return ET.tostring(tree)
125
126 if __name__ == "__main__":
127 xml = """<?xml version="1.0" encoding="UTF-8"?>
128 <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
129 <Owner>
130 <ID>12345678901234567890</ID>
131 <DisplayName>owner-nickname</DisplayName>
132 </Owner>
133 <AccessControlList>
134 <Grant>
135 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
136 <ID>12345678901234567890</ID>
137 <DisplayName>owner-nickname</DisplayName>
138 </Grantee>
139 <Permission>FULL_CONTROL</Permission>
140 </Grant>
141 <Grant>
142 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group">
143 <URI>http://acs.amazonaws.com/groups/global/AllUsers</URI>
144 </Grantee>
145 <Permission>READ</Permission>
146 </Grant>
147 </AccessControlList>
148 </AccessControlPolicy>
149 """
150 acl = ACL(xml)
151 print "Grants:", acl.getGrantList()
152 acl.revokeAnonRead()
153 print "Grants:", acl.getGrantList()
154 acl.grantAnonRead()
155 print "Grants:", acl.getGrantList()
156 print acl
+487
-0
S3/CloudFront.py less more
0 ## Amazon CloudFront support
1 ## Author: Michal Ludvig <michal@logix.cz>
2 ## http://www.logix.cz/michal
3 ## License: GPL Version 2
4
5 import sys
6 import base64
7 import time
8 import httplib
9 from logging import debug, info, warning, error
10
11 try:
12 from hashlib import md5, sha1
13 except ImportError:
14 from md5 import md5
15 import sha as sha1
16 import hmac
17
18 try:
19 import xml.etree.ElementTree as ET
20 except ImportError:
21 import elementtree.ElementTree as ET
22
23 from Config import Config
24 from Exceptions import *
25 from Utils import getTreeFromXml, appendXmlTextNode, getDictFromTree, dateS3toPython
26 from S3Uri import S3Uri, S3UriS3
27
28 def output(message):
29 sys.stdout.write(message + "\n")
30
31 def pretty_output(label, message):
32 #label = ("%s " % label).ljust(20, ".")
33 label = ("%s:" % label).ljust(15)
34 output("%s %s" % (label, message))
35
36 class DistributionSummary(object):
37 ## Example:
38 ##
39 ## <DistributionSummary>
40 ## <Id>1234567890ABC</Id>
41 ## <Status>Deployed</Status>
42 ## <LastModifiedTime>2009-01-16T11:49:02.189Z</LastModifiedTime>
43 ## <DomainName>blahblahblah.cloudfront.net</DomainName>
44 ## <Origin>example.bucket.s3.amazonaws.com</Origin>
45 ## <Enabled>true</Enabled>
46 ## </DistributionSummary>
47
48 def __init__(self, tree):
49 if tree.tag != "DistributionSummary":
50 raise ValueError("Expected <DistributionSummary /> xml, got: <%s />" % tree.tag)
51 self.parse(tree)
52
53 def parse(self, tree):
54 self.info = getDictFromTree(tree)
55 self.info['Enabled'] = (self.info['Enabled'].lower() == "true")
56
57 def uri(self):
58 return S3Uri("cf://%s" % self.info['Id'])
59
60 class DistributionList(object):
61 ## Example:
62 ##
63 ## <DistributionList xmlns="http://cloudfront.amazonaws.com/doc/2008-06-30/">
64 ## <Marker />
65 ## <MaxItems>100</MaxItems>
66 ## <IsTruncated>false</IsTruncated>
67 ## <DistributionSummary>
68 ## ... handled by DistributionSummary() class ...
69 ## </DistributionSummary>
70 ## </DistributionList>
71
72 def __init__(self, xml):
73 tree = getTreeFromXml(xml)
74 if tree.tag != "DistributionList":
75 raise ValueError("Expected <DistributionList /> xml, got: <%s />" % tree.tag)
76 self.parse(tree)
77
78 def parse(self, tree):
79 self.info = getDictFromTree(tree)
80 ## Normalise some items
81 self.info['IsTruncated'] = (self.info['IsTruncated'].lower() == "true")
82
83 self.dist_summs = []
84 for dist_summ in tree.findall(".//DistributionSummary"):
85 self.dist_summs.append(DistributionSummary(dist_summ))
86
87 class Distribution(object):
88 ## Example:
89 ##
90 ## <Distribution xmlns="http://cloudfront.amazonaws.com/doc/2008-06-30/">
91 ## <Id>1234567890ABC</Id>
92 ## <Status>InProgress</Status>
93 ## <LastModifiedTime>2009-01-16T13:07:11.319Z</LastModifiedTime>
94 ## <DomainName>blahblahblah.cloudfront.net</DomainName>
95 ## <DistributionConfig>
96 ## ... handled by DistributionConfig() class ...
97 ## </DistributionConfig>
98 ## </Distribution>
99
100 def __init__(self, xml):
101 tree = getTreeFromXml(xml)
102 if tree.tag != "Distribution":
103 raise ValueError("Expected <Distribution /> xml, got: <%s />" % tree.tag)
104 self.parse(tree)
105
106 def parse(self, tree):
107 self.info = getDictFromTree(tree)
108 ## Normalise some items
109 self.info['LastModifiedTime'] = dateS3toPython(self.info['LastModifiedTime'])
110
111 self.info['DistributionConfig'] = DistributionConfig(tree = tree.find(".//DistributionConfig"))
112
113 def uri(self):
114 return S3Uri("cf://%s" % self.info['Id'])
115
116 class DistributionConfig(object):
117 ## Example:
118 ##
119 ## <DistributionConfig>
120 ## <Origin>somebucket.s3.amazonaws.com</Origin>
121 ## <CallerReference>s3://somebucket/</CallerReference>
122 ## <Comment>http://somebucket.s3.amazonaws.com/</Comment>
123 ## <Enabled>true</Enabled>
124 ## </DistributionConfig>
125
126 EMPTY_CONFIG = "<DistributionConfig><Origin/><CallerReference/><Enabled>true</Enabled></DistributionConfig>"
127 xmlns = "http://cloudfront.amazonaws.com/doc/2008-06-30/"
128 def __init__(self, xml = None, tree = None):
129 if not xml:
130 xml = DistributionConfig.EMPTY_CONFIG
131
132 if not tree:
133 tree = getTreeFromXml(xml)
134
135 if tree.tag != "DistributionConfig":
136 raise ValueError("Expected <DistributionConfig /> xml, got: <%s />" % tree.tag)
137 self.parse(tree)
138
139 def parse(self, tree):
140 self.info = getDictFromTree(tree)
141 self.info['Enabled'] = (self.info['Enabled'].lower() == "true")
142 if not self.info.has_key("CNAME"):
143 self.info['CNAME'] = []
144 if type(self.info['CNAME']) != list:
145 self.info['CNAME'] = [self.info['CNAME']]
146 self.info['CNAME'] = [cname.lower() for cname in self.info['CNAME']]
147 if not self.info.has_key("Comment"):
148 self.info['Comment'] = ""
149
150 def __str__(self):
151 tree = ET.Element("DistributionConfig")
152 tree.attrib['xmlns'] = DistributionConfig.xmlns
153
154 ## Retain the order of the following calls!
155 appendXmlTextNode("Origin", self.info['Origin'], tree)
156 appendXmlTextNode("CallerReference", self.info['CallerReference'], tree)
157 for cname in self.info['CNAME']:
158 appendXmlTextNode("CNAME", cname.lower(), tree)
159 if self.info['Comment']:
160 appendXmlTextNode("Comment", self.info['Comment'], tree)
161 appendXmlTextNode("Enabled", str(self.info['Enabled']).lower(), tree)
162
163 return ET.tostring(tree)
164
165 class CloudFront(object):
166 operations = {
167 "CreateDist" : { 'method' : "POST", 'resource' : "" },
168 "DeleteDist" : { 'method' : "DELETE", 'resource' : "/%(dist_id)s" },
169 "GetList" : { 'method' : "GET", 'resource' : "" },
170 "GetDistInfo" : { 'method' : "GET", 'resource' : "/%(dist_id)s" },
171 "GetDistConfig" : { 'method' : "GET", 'resource' : "/%(dist_id)s/config" },
172 "SetDistConfig" : { 'method' : "PUT", 'resource' : "/%(dist_id)s/config" },
173 }
174
175 ## Maximum attempts of re-issuing failed requests
176 _max_retries = 5
177
178 def __init__(self, config):
179 self.config = config
180
181 ## --------------------------------------------------
182 ## Methods implementing CloudFront API
183 ## --------------------------------------------------
184
185 def GetList(self):
186 response = self.send_request("GetList")
187 response['dist_list'] = DistributionList(response['data'])
188 if response['dist_list'].info['IsTruncated']:
189 raise NotImplementedError("List is truncated. Ask s3cmd author to add support.")
190 ## TODO: handle Truncated
191 return response
192
193 def CreateDistribution(self, uri, cnames_add = [], comment = None):
194 dist_config = DistributionConfig()
195 dist_config.info['Enabled'] = True
196 dist_config.info['Origin'] = uri.host_name()
197 dist_config.info['CallerReference'] = str(uri)
198 if comment == None:
199 dist_config.info['Comment'] = uri.public_url()
200 else:
201 dist_config.info['Comment'] = comment
202 for cname in cnames_add:
203 if dist_config.info['CNAME'].count(cname) == 0:
204 dist_config.info['CNAME'].append(cname)
205 request_body = str(dist_config)
206 debug("CreateDistribution(): request_body: %s" % request_body)
207 response = self.send_request("CreateDist", body = request_body)
208 response['distribution'] = Distribution(response['data'])
209 return response
210
211 def ModifyDistribution(self, cfuri, cnames_add = [], cnames_remove = [],
212 comment = None, enabled = None):
213 if cfuri.type != "cf":
214 raise ValueError("Expected CFUri instead of: %s" % cfuri)
215 # Get current dist status (enabled/disabled) and Etag
216 info("Checking current status of %s" % cfuri)
217 response = self.GetDistConfig(cfuri)
218 dc = response['dist_config']
219 if enabled != None:
220 dc.info['Enabled'] = enabled
221 if comment != None:
222 dc.info['Comment'] = comment
223 for cname in cnames_add:
224 if dc.info['CNAME'].count(cname) == 0:
225 dc.info['CNAME'].append(cname)
226 for cname in cnames_remove:
227 while dc.info['CNAME'].count(cname) > 0:
228 dc.info['CNAME'].remove(cname)
229 response = self.SetDistConfig(cfuri, dc, response['headers']['etag'])
230 return response
231
232 def DeleteDistribution(self, cfuri):
233 if cfuri.type != "cf":
234 raise ValueError("Expected CFUri instead of: %s" % cfuri)
235 # Get current dist status (enabled/disabled) and Etag
236 info("Checking current status of %s" % cfuri)
237 response = self.GetDistConfig(cfuri)
238 if response['dist_config'].info['Enabled']:
239 info("Distribution is ENABLED. Disabling first.")
240 response['dist_config'].info['Enabled'] = False
241 response = self.SetDistConfig(cfuri, response['dist_config'],
242 response['headers']['etag'])
243 warning("Waiting for Distribution to become disabled.")
244 warning("This may take several minutes, please wait.")
245 while True:
246 response = self.GetDistInfo(cfuri)
247 d = response['distribution']
248 if d.info['Status'] == "Deployed" and d.info['Enabled'] == False:
249 info("Distribution is now disabled")
250 break
251 warning("Still waiting...")
252 time.sleep(10)
253 headers = {}
254 headers['if-match'] = response['headers']['etag']
255 response = self.send_request("DeleteDist", dist_id = cfuri.dist_id(),
256 headers = headers)
257 return response
258
259 def GetDistInfo(self, cfuri):
260 if cfuri.type != "cf":
261 raise ValueError("Expected CFUri instead of: %s" % cfuri)
262 response = self.send_request("GetDistInfo", dist_id = cfuri.dist_id())
263 response['distribution'] = Distribution(response['data'])
264 return response
265
266 def GetDistConfig(self, cfuri):
267 if cfuri.type != "cf":
268 raise ValueError("Expected CFUri instead of: %s" % cfuri)
269 response = self.send_request("GetDistConfig", dist_id = cfuri.dist_id())
270 response['dist_config'] = DistributionConfig(response['data'])
271 return response
272
273 def SetDistConfig(self, cfuri, dist_config, etag = None):
274 if etag == None:
275 debug("SetDistConfig(): Etag not set. Fetching it first.")
276 etag = self.GetDistConfig(cfuri)['headers']['etag']
277 debug("SetDistConfig(): Etag = %s" % etag)
278 request_body = str(dist_config)
279 debug("SetDistConfig(): request_body: %s" % request_body)
280 headers = {}
281 headers['if-match'] = etag
282 response = self.send_request("SetDistConfig", dist_id = cfuri.dist_id(),
283 body = request_body, headers = headers)
284 return response
285
286 ## --------------------------------------------------
287 ## Low-level methods for handling CloudFront requests
288 ## --------------------------------------------------
289
290 def send_request(self, op_name, dist_id = None, body = None, headers = {}, retries = _max_retries):
291 operation = self.operations[op_name]
292 if body:
293 headers['content-type'] = 'text/plain'
294 request = self.create_request(operation, dist_id, headers)
295 conn = self.get_connection()
296 debug("send_request(): %s %s" % (request['method'], request['resource']))
297 conn.request(request['method'], request['resource'], body, request['headers'])
298 http_response = conn.getresponse()
299 response = {}
300 response["status"] = http_response.status
301 response["reason"] = http_response.reason
302 response["headers"] = dict(http_response.getheaders())
303 response["data"] = http_response.read()
304 conn.close()
305
306 debug("CloudFront: response: %r" % response)
307
308 if response["status"] >= 500:
309 e = CloudFrontError(response)
310 if retries:
311 warning(u"Retrying failed request: %s" % op_name)
312 warning(unicode(e))
313 warning("Waiting %d sec..." % self._fail_wait(retries))
314 time.sleep(self._fail_wait(retries))
315 return self.send_request(op_name, dist_id, body, retries - 1)
316 else:
317 raise e
318
319 if response["status"] < 200 or response["status"] > 299:
320 raise CloudFrontError(response)
321
322 return response
323
324 def create_request(self, operation, dist_id = None, headers = None):
325 resource = self.config.cloudfront_resource + (
326 operation['resource'] % { 'dist_id' : dist_id })
327
328 if not headers:
329 headers = {}
330
331 if headers.has_key("date"):
332 if not headers.has_key("x-amz-date"):
333 headers["x-amz-date"] = headers["date"]
334 del(headers["date"])
335
336 if not headers.has_key("x-amz-date"):
337 headers["x-amz-date"] = time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime())
338
339 signature = self.sign_request(headers)
340 headers["Authorization"] = "AWS "+self.config.access_key+":"+signature
341
342 request = {}
343 request['resource'] = resource
344 request['headers'] = headers
345 request['method'] = operation['method']
346
347 return request
348
349 def sign_request(self, headers):
350 string_to_sign = headers['x-amz-date']
351 signature = base64.encodestring(hmac.new(self.config.secret_key, string_to_sign, sha1).digest()).strip()
352 debug(u"CloudFront.sign_request('%s') = %s" % (string_to_sign, signature))
353 return signature
354
355 def get_connection(self):
356 if self.config.proxy_host != "":
357 raise ParameterError("CloudFront commands don't work from behind a HTTP proxy")
358 return httplib.HTTPSConnection(self.config.cloudfront_host)
359
360 def _fail_wait(self, retries):
361 # Wait a few seconds. The more it fails the more we wait.
362 return (self._max_retries - retries + 1) * 3
363
364 class Cmd(object):
365 """
366 Class that implements CloudFront commands
367 """
368
369 class Options(object):
370 cf_cnames_add = []
371 cf_cnames_remove = []
372 cf_comment = None
373 cf_enable = None
374
375 def option_list(self):
376 return [opt for opt in dir(self) if opt.startswith("cf_")]
377
378 def update_option(self, option, value):
379 setattr(Cmd.options, option, value)
380
381 options = Options()
382
383 @staticmethod
384 def info(args):
385 cf = CloudFront(Config())
386 if not args:
387 response = cf.GetList()
388 for d in response['dist_list'].dist_summs:
389 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(d.info['Origin']))
390 pretty_output("DistId", d.uri())
391 pretty_output("DomainName", d.info['DomainName'])
392 pretty_output("Status", d.info['Status'])
393 pretty_output("Enabled", d.info['Enabled'])
394 output("")
395 else:
396 cfuris = []
397 for arg in args:
398 cfuris.append(S3Uri(arg))
399 if cfuris[-1].type != 'cf':
400 raise ParameterError("CloudFront URI required instead of: %s" % arg)
401 for cfuri in cfuris:
402 response = cf.GetDistInfo(cfuri)
403 d = response['distribution']
404 dc = d.info['DistributionConfig']
405 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['Origin']))
406 pretty_output("DistId", d.uri())
407 pretty_output("DomainName", d.info['DomainName'])
408 pretty_output("Status", d.info['Status'])
409 pretty_output("CNAMEs", ", ".join(dc.info['CNAME']))
410 pretty_output("Comment", dc.info['Comment'])
411 pretty_output("Enabled", dc.info['Enabled'])
412 pretty_output("Etag", response['headers']['etag'])
413
414 @staticmethod
415 def create(args):
416 cf = CloudFront(Config())
417 buckets = []
418 for arg in args:
419 uri = S3Uri(arg)
420 if uri.type != "s3":
421 raise ParameterError("Bucket can only be created from a s3:// URI instead of: %s" % arg)
422 if uri.object():
423 raise ParameterError("Use s3:// URI with a bucket name only instead of: %s" % arg)
424 if not uri.is_dns_compatible():
425 raise ParameterError("CloudFront can only handle lowercase-named buckets.")
426 buckets.append(uri)
427 if not buckets:
428 raise ParameterError("No valid bucket names found")
429 for uri in buckets:
430 info("Creating distribution from: %s" % uri)
431 response = cf.CreateDistribution(uri, cnames_add = Cmd.options.cf_cnames_add,
432 comment = Cmd.options.cf_comment)
433 d = response['distribution']
434 dc = d.info['DistributionConfig']
435 output("Distribution created:")
436 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['Origin']))
437 pretty_output("DistId", d.uri())
438 pretty_output("DomainName", d.info['DomainName'])
439 pretty_output("CNAMEs", ", ".join(dc.info['CNAME']))
440 pretty_output("Comment", dc.info['Comment'])
441 pretty_output("Status", d.info['Status'])
442 pretty_output("Enabled", dc.info['Enabled'])
443 pretty_output("Etag", response['headers']['etag'])
444
445 @staticmethod
446 def delete(args):
447 cf = CloudFront(Config())
448 cfuris = []
449 for arg in args:
450 cfuris.append(S3Uri(arg))
451 if cfuris[-1].type != 'cf':
452 raise ParameterError("CloudFront URI required instead of: %s" % arg)
453 for cfuri in cfuris:
454 response = cf.DeleteDistribution(cfuri)
455 if response['status'] >= 400:
456 error("Distribution %s could not be deleted: %s" % (cfuri, response['reason']))
457 output("Distribution %s deleted" % cfuri)
458
459 @staticmethod
460 def modify(args):
461 cf = CloudFront(Config())
462 cfuri = S3Uri(args.pop(0))
463 if cfuri.type != 'cf':
464 raise ParameterError("CloudFront URI required instead of: %s" % arg)
465 if len(args):
466 raise ParameterError("Too many parameters. Modify one Distribution at a time.")
467
468 response = cf.ModifyDistribution(cfuri,
469 cnames_add = Cmd.options.cf_cnames_add,
470 cnames_remove = Cmd.options.cf_cnames_remove,
471 comment = Cmd.options.cf_comment,
472 enabled = Cmd.options.cf_enable)
473 if response['status'] >= 400:
474 error("Distribution %s could not be modified: %s" % (cfuri, response['reason']))
475 output("Distribution modified: %s" % cfuri)
476 response = cf.GetDistInfo(cfuri)
477 d = response['distribution']
478 dc = d.info['DistributionConfig']
479 pretty_output("Origin", S3UriS3.httpurl_to_s3uri(dc.info['Origin']))
480 pretty_output("DistId", d.uri())
481 pretty_output("DomainName", d.info['DomainName'])
482 pretty_output("Status", d.info['Status'])
483 pretty_output("CNAMEs", ", ".join(dc.info['CNAME']))
484 pretty_output("Comment", dc.info['Comment'])
485 pretty_output("Enabled", dc.info['Enabled'])
486 pretty_output("Etag", response['headers']['etag'])
+15
-2
S3/Config.py less more
55 import logging
66 from logging import debug, info, warning, error
77 import re
8 import Progress
89
910 class Config(object):
1011 _instance = None
1516 host_base = "s3.amazonaws.com"
1617 host_bucket = "%(bucket)s.s3.amazonaws.com"
1718 simpledb_host = "sdb.amazonaws.com"
19 cloudfront_host = "cloudfront.amazonaws.com"
20 cloudfront_resource = "/2008-06-30/distribution"
1821 verbosity = logging.WARNING
22 progress_meter = True
23 progress_class = Progress.ProgressCR
1924 send_chunk = 4096
2025 recv_chunk = 4096
2126 human_readable_sizes = False
2227 force = False
28 get_continue = False
29 skip_existing = False
30 recursive = False
2331 acl_public = False
2432 proxy_host = ""
2533 proxy_port = 3128
4654 use_https = False
4755 bucket_location = "US"
4856 default_mime_type = "binary/octet-stream"
49 guess_mime_type = False
57 guess_mime_type = True
5058 debug_syncmatch = False
59 # List of checks to be performed for 'sync'
60 sync_checks = ['size', 'md5'] # 'weak-timestamp'
5161 # List of compiled REGEXPs
5262 exclude = []
63 include = []
5364 # Dict mapping compiled REGEXPs back to their textual form
5465 debug_exclude = {}
66 debug_include = {}
67 encoding = "utf-8"
5568
5669 ## Creating a singleton
5770 def __new__(self, configfile = None):
118131 self.parse_file(file, sections)
119132
120133 def parse_file(self, file, sections = []):
121 info("ConfigParser: Reading file '%s'" % file)
134 debug("ConfigParser: Reading file '%s'" % file)
122135 if type(sections) != type([]):
123136 sections = [sections]
124137 in_our_section = True
+27
-9
S3/Exceptions.py less more
22 ## http://www.logix.cz/michal
33 ## License: GPL Version 2
44
5 from Utils import getTreeFromXml, unicodise, deunicodise
56 from logging import debug, info, warning, error
67
78 try:
1011 import elementtree.ElementTree as ET
1112
1213 class S3Exception(Exception):
14 def __init__(self, message = ""):
15 self.message = unicodise(message)
16
1317 def __str__(self):
14 ## Is this legal?
15 return unicode(self)
18 ## Call unicode(self) instead of self.message because
19 ## __unicode__() method could be overriden in subclasses!
20 return deunicodise(unicode(self))
1621
1722 def __unicode__(self):
1823 return self.message
24
25 ## (Base)Exception.message has been deprecated in Python 2.6
26 def _get_message(self):
27 return self._message
28 def _set_message(self, message):
29 self._message = message
30 message = property(_get_message, _set_message)
31
1932
2033 class S3Error (S3Exception):
2134 def __init__(self, response):
2740 for header in response["headers"]:
2841 debug("HttpHeader: %s: %s" % (header, response["headers"][header]))
2942 if response.has_key("data"):
30 tree = ET.fromstring(response["data"])
31 for child in tree.getchildren():
43 tree = getTreeFromXml(response["data"])
44 error_node = tree
45 if not error_node.tag == "Error":
46 error_node = tree.find(".//Error")
47 for child in error_node.getchildren():
3248 if child.text != "":
3349 debug("ErrorXML: " + child.tag + ": " + repr(child.text))
3450 self.info[child.tag] = child.text
3551
3652 def __unicode__(self):
37 retval = "%d (%s)" % (self.status, self.reason)
38 try:
39 retval += (": %s" % self.info["Code"])
40 except (AttributeError, KeyError):
41 pass
53 retval = u"%d " % (self.status)
54 retval += (u"(%s)" % (self.info.has_key("Code") and self.info["Code"] or self.reason))
55 if self.info.has_key("Message"):
56 retval += (u": %s" % self.info["Message"])
4257 return retval
4358
59 class CloudFrontError(S3Error):
60 pass
61
4462 class S3UploadError(S3Exception):
4563 pass
4664
+5
-3
S3/PkgInfo.py less more
00 package = "s3cmd"
1 version = "0.9.8.4"
1 version = "0.9.9-rc3"
22 url = "http://s3tools.logix.cz"
33 license = "GPL version 2"
4 short_description = "S3cmd is a tool for managing Amazon S3 storage space."
4 short_description = "Command line tool for managing Amazon S3 and CloudFront services"
55 long_description = """
66 S3cmd lets you copy files from/to Amazon S3
77 (Simple Storage Service) using a simple to use
8 command line client.
8 command line client. Supports rsync-like backup,
9 GPG encryption, and more. Also supports management
10 of Amazon's CloudFront content delivery network.
911 """
1012
+154
-0
S3/Progress.py less more
0 ## Amazon S3 manager
1 ## Author: Michal Ludvig <michal@logix.cz>
2 ## http://www.logix.cz/michal
3 ## License: GPL Version 2
4
5 import sys
6 import datetime
7 import Utils
8
9 class Progress(object):
10 _stdout = sys.stdout
11
12 def __init__(self, labels, total_size):
13 self._stdout = sys.stdout
14 self.new_file(labels, total_size)
15
16 def new_file(self, labels, total_size):
17 self.labels = labels
18 self.total_size = total_size
19 # Set initial_position to something in the
20 # case we're not counting from 0. For instance
21 # when appending to a partially downloaded file.
22 # Setting initial_position will let the speed
23 # be computed right.
24 self.initial_position = 0
25 self.current_position = self.initial_position
26 self.time_start = datetime.datetime.now()
27 self.time_last = self.time_start
28 self.time_current = self.time_start
29
30 self.display(new_file = True)
31
32 def update(self, current_position = -1, delta_position = -1):
33 self.time_last = self.time_current
34 self.time_current = datetime.datetime.now()
35 if current_position > -1:
36 self.current_position = current_position
37 elif delta_position > -1:
38 self.current_position += delta_position
39 #else:
40 # no update, just call display()
41 self.display()
42
43 def done(self, message):
44 self.display(done_message = message)
45
46 def output_labels(self):
47 self._stdout.write(u"%(source)s -> %(destination)s %(extra)s\n" % self.labels)
48 self._stdout.flush()
49
50 def display(self, new_file = False, done_message = None):
51 """
52 display(new_file = False[/True], done = False[/True])
53
54 Override this method to provide a nicer output.
55 """
56 if new_file:
57 self.output_labels()
58 self.last_milestone = 0
59 return
60
61 if self.current_position == self.total_size:
62 print_size = Utils.formatSize(self.current_position, True)
63 if print_size[1] != "": print_size[1] += "B"
64 timedelta = self.time_current - self.time_start
65 sec_elapsed = timedelta.days * 86400 + timedelta.seconds + float(timedelta.microseconds)/1000000.0
66 print_speed = Utils.formatSize((self.current_position - self.initial_position) / sec_elapsed, True, True)
67 self._stdout.write("100%% %s%s in %.2fs (%.2f %sB/s)\n" %
68 (print_size[0], print_size[1], sec_elapsed, print_speed[0], print_speed[1]))
69 self._stdout.flush()
70 return
71
72 rel_position = selfself.current_position * 100 / self.total_size
73 if rel_position >= self.last_milestone:
74 self.last_milestone = (int(rel_position) / 5) * 5
75 self._stdout.write("%d%% ", self.last_milestone)
76 self._stdout.flush()
77 return
78
79 class ProgressANSI(Progress):
80 ## http://en.wikipedia.org/wiki/ANSI_escape_code
81 SCI = '\x1b['
82 ANSI_hide_cursor = SCI + "?25l"
83 ANSI_show_cursor = SCI + "?25h"
84 ANSI_save_cursor_pos = SCI + "s"
85 ANSI_restore_cursor_pos = SCI + "u"
86 ANSI_move_cursor_to_column = SCI + "%uG"
87 ANSI_erase_to_eol = SCI + "0K"
88 ANSI_erase_current_line = SCI + "2K"
89
90 def display(self, new_file = False, done_message = None):
91 """
92 display(new_file = False[/True], done_message = None)
93 """
94 if new_file:
95 self.output_labels()
96 self._stdout.write(self.ANSI_save_cursor_pos)
97 self._stdout.flush()
98 return
99
100 timedelta = self.time_current - self.time_start
101 sec_elapsed = timedelta.days * 86400 + timedelta.seconds + float(timedelta.microseconds)/1000000.0
102 if (sec_elapsed > 0):
103 print_speed = Utils.formatSize((self.current_position - self.initial_position) / sec_elapsed, True, True)
104 else:
105 print_speed = (0, "")
106 self._stdout.write(self.ANSI_restore_cursor_pos)
107 self._stdout.write(self.ANSI_erase_to_eol)
108 self._stdout.write("%(current)s of %(total)s %(percent)3d%% in %(elapsed)ds %(speed).2f %(speed_coeff)sB/s" % {
109 "current" : str(self.current_position).rjust(len(str(self.total_size))),
110 "total" : self.total_size,
111 "percent" : self.total_size and (self.current_position * 100 / self.total_size) or 0,
112 "elapsed" : sec_elapsed,
113 "speed" : print_speed[0],
114 "speed_coeff" : print_speed[1]
115 })
116
117 if done_message:
118 self._stdout.write(" %s\n" % done_message)
119
120 self._stdout.flush()
121
122 class ProgressCR(Progress):
123 ## Uses CR char (Carriage Return) just like other progress bars do.
124 CR_char = chr(13)
125
126 def display(self, new_file = False, done_message = None):
127 """
128 display(new_file = False[/True], done_message = None)
129 """
130 if new_file:
131 self.output_labels()
132 return
133
134 timedelta = self.time_current - self.time_start
135 sec_elapsed = timedelta.days * 86400 + timedelta.seconds + float(timedelta.microseconds)/1000000.0
136 if (sec_elapsed > 0):
137 print_speed = Utils.formatSize((self.current_position - self.initial_position) / sec_elapsed, True, True)
138 else:
139 print_speed = (0, "")
140 self._stdout.write(self.CR_char)
141 output = " %(current)s of %(total)s %(percent)3d%% in %(elapsed)4ds %(speed)7.2f %(speed_coeff)sB/s" % {
142 "current" : str(self.current_position).rjust(len(str(self.total_size))),
143 "total" : self.total_size,
144 "percent" : self.total_size and (self.current_position * 100 / self.total_size) or 0,
145 "elapsed" : sec_elapsed,
146 "speed" : print_speed[0],
147 "speed_coeff" : print_speed[1]
148 }
149 self._stdout.write(output)
150 if done_message:
151 self._stdout.write(" %s\n" % done_message)
152
153 self._stdout.flush()
+298
-147
S3/S3.py less more
55 import sys
66 import os, os.path
77 import base64
8 import md5
9 import sha
10 import hmac
8 import time
119 import httplib
1210 import logging
1311 import mimetypes
1412 from logging import debug, info, warning, error
1513 from stat import ST_SIZE
1614
15 try:
16 from hashlib import md5, sha1
17 except ImportError:
18 from md5 import md5
19 import sha as sha1
20 import hmac
21
1722 from Utils import *
1823 from SortedDict import SortedDict
1924 from BidirMap import BidirMap
2025 from Config import Config
2126 from Exceptions import *
27 from ACL import ACL
2228
2329 class S3(object):
2430 http_methods = BidirMap(
5763 ## S3 sometimes sends HTTP-307 response
5864 redir_map = {}
5965
66 ## Maximum attempts of re-issuing failed requests
67 _max_retries = 5
68
6069 def __init__(self, config):
6170 self.config = config
6271
100109 response["list"] = getListFromXml(response["data"], "Bucket")
101110 return response
102111
103 def bucket_list(self, bucket, prefix = None):
112 def bucket_list(self, bucket, prefix = None, recursive = None):
104113 def _list_truncated(data):
105114 ## <IsTruncated> can either be "true" or "false" or be missing completely
106115 is_truncated = getTextFromXml(data, ".//IsTruncated") or "false"
109118 def _get_contents(data):
110119 return getListFromXml(data, "Contents")
111120
112 prefix = self.urlencode_string(prefix)
113 request = self.create_request("BUCKET_LIST", bucket = bucket, prefix = prefix)
121 def _get_common_prefixes(data):
122 return getListFromXml(data, "CommonPrefixes")
123
124 uri_params = {}
125 if prefix:
126 uri_params['prefix'] = self.urlencode_string(prefix)
127 if not self.config.recursive and not recursive:
128 uri_params['delimiter'] = "/"
129 request = self.create_request("BUCKET_LIST", bucket = bucket, **uri_params)
114130 response = self.send_request(request)
115131 #debug(response)
116132 list = _get_contents(response["data"])
133 prefixes = _get_common_prefixes(response["data"])
117134 while _list_truncated(response["data"]):
118 marker = list[-1]["Key"]
119 info("Listing continues after '%s'" % marker)
120 request = self.create_request("BUCKET_LIST", bucket = bucket,
121 prefix = prefix,
122 marker = self.urlencode_string(marker))
135 uri_params['marker'] = self.urlencode_string(list[-1]["Key"])
136 debug("Listing continues after '%s'" % uri_params['marker'])
137 request = self.create_request("BUCKET_LIST", bucket = bucket, **uri_params)
123138 response = self.send_request(request)
124139 list += _get_contents(response["data"])
140 prefixes += _get_common_prefixes(response["data"])
125141 response['list'] = list
142 response['common_prefixes'] = prefixes
126143 return response
127144
128145 def bucket_create(self, bucket, bucket_location = None):
154171 response['bucket-location'] = getTextFromXml(response['data'], "LocationConstraint") or "any"
155172 return response
156173
157 def object_put(self, filename, bucket, object, extra_headers = None):
174 def object_put(self, filename, uri, extra_headers = None, extra_label = ""):
175 # TODO TODO
176 # Make it consistent with stream-oriented object_get()
177 if uri.type != "s3":
178 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
179
158180 if not os.path.isfile(filename):
159 raise InvalidFileError("%s is not a regular file" % filename)
181 raise InvalidFileError(u"%s is not a regular file" % unicodise(filename))
160182 try:
161183 file = open(filename, "rb")
162184 size = os.stat(filename)[ST_SIZE]
163185 except IOError, e:
164 raise InvalidFileError("%s: %s" % (filename, e.strerror))
186 raise InvalidFileError(u"%s: %s" % (unicodise(filename), e.strerror))
165187 headers = SortedDict()
166188 if extra_headers:
167189 headers.update(extra_headers)
175197 headers["content-type"] = content_type
176198 if self.config.acl_public:
177199 headers["x-amz-acl"] = "public-read"
178 request = self.create_request("OBJECT_PUT", bucket = bucket, object = object, headers = headers)
179 response = self.send_file(request, file)
180 return response
181
182 def object_get_uri(self, uri, stream):
200 request = self.create_request("OBJECT_PUT", uri = uri, headers = headers)
201 labels = { 'source' : unicodise(filename), 'destination' : unicodise(uri.uri()), 'extra' : extra_label }
202 response = self.send_file(request, file, labels)
203 return response
204
205 def object_get(self, uri, stream, start_position = 0, extra_label = ""):
183206 if uri.type != "s3":
184207 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
185 request = self.create_request("OBJECT_GET", bucket = uri.bucket(), object = uri.object())
186 response = self.recv_file(request, stream)
187 return response
188
189 def object_delete(self, bucket, object):
190 request = self.create_request("OBJECT_DELETE", bucket = bucket, object = object)
191 response = self.send_request(request)
192 return response
193
194 def object_put_uri(self, filename, uri, extra_headers = None):
195 # TODO TODO
196 # Make it consistent with stream-oriented object_get_uri()
208 request = self.create_request("OBJECT_GET", uri = uri)
209 labels = { 'source' : unicodise(uri.uri()), 'destination' : unicodise(stream.name), 'extra' : extra_label }
210 response = self.recv_file(request, stream, labels, start_position)
211 return response
212
213 def object_delete(self, uri):
197214 if uri.type != "s3":
198215 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
199 return self.object_put(filename, uri.bucket(), uri.object(), extra_headers)
200
201 def object_delete_uri(self, uri):
202 if uri.type != "s3":
203 raise ValueError("Expected URI type 's3', got '%s'" % uri.type)
204 return self.object_delete(uri.bucket(), uri.object())
216 request = self.create_request("OBJECT_DELETE", uri = uri)
217 response = self.send_request(request)
218 return response
219
220 def object_copy(self, src_uri, dst_uri, extra_headers = None):
221 if src_uri.type != "s3":
222 raise ValueError("Expected URI type 's3', got '%s'" % src_uri.type)
223 if dst_uri.type != "s3":
224 raise ValueError("Expected URI type 's3', got '%s'" % dst_uri.type)
225 headers = SortedDict()
226 headers['x-amz-copy-source'] = "/%s/%s" % (src_uri.bucket(), self.urlencode_string(src_uri.object()))
227 if self.config.acl_public:
228 headers["x-amz-acl"] = "public-read"
229 if extra_headers:
230 headers.update(extra_headers)
231 request = self.create_request("OBJECT_PUT", uri = dst_uri, headers = headers)
232 response = self.send_request(request)
233 return response
234
235 def object_move(self, src_uri, dst_uri, extra_headers = None):
236 response_copy = self.object_copy(src_uri, dst_uri, extra_headers)
237 debug("Object %s copied to %s" % (src_uri, dst_uri))
238 if getRootTagName(response_copy["data"]) == "CopyObjectResult":
239 response_delete = self.object_delete(src_uri)
240 debug("Object %s deleted" % src_uri)
241 return response_copy
205242
206243 def object_info(self, uri):
207 request = self.create_request("OBJECT_HEAD", bucket = uri.bucket(), object = uri.object())
244 request = self.create_request("OBJECT_HEAD", uri = uri)
208245 response = self.send_request(request)
209246 return response
210247
211248 def get_acl(self, uri):
212249 if uri.has_object():
213 request = self.create_request("OBJECT_GET", bucket = uri.bucket(), object = uri.object(), extra = "?acl")
250 request = self.create_request("OBJECT_GET", uri = uri, extra = "?acl")
214251 else:
215252 request = self.create_request("BUCKET_LIST", bucket = uri.bucket(), extra = "?acl")
216 acl = {}
217 response = self.send_request(request)
218 grants = getListFromXml(response['data'], "Grant")
219 for grant in grants:
220 if grant['Grantee'][0].has_key('DisplayName'):
221 user = grant['Grantee'][0]['DisplayName']
222 if grant['Grantee'][0].has_key('URI'):
223 user = grant['Grantee'][0]['URI']
224 if user == 'http://acs.amazonaws.com/groups/global/AllUsers':
225 user = "*anon*"
226 perm = grant['Permission']
227 acl[user] = perm
253
254 response = self.send_request(request)
255 acl = ACL(response['data'])
228256 return acl
257
258 def set_acl(self, uri, acl):
259 if uri.has_object():
260 request = self.create_request("OBJECT_PUT", uri = uri, extra = "?acl")
261 else:
262 request = self.create_request("BUCKET_CREATE", bucket = uri.bucket(), extra = "?acl")
263
264 body = str(acl)
265 debug(u"set_acl(%s): acl-xml: %s" % (uri, body))
266 response = self.send_request(request, body)
267 return response
229268
230269 ## Low level methods
231270 def urlencode_string(self, string):
267306 debug("String '%s' encoded to '%s'" % (string, encoded))
268307 return encoded
269308
270 def create_request(self, operation, bucket = None, object = None, headers = None, extra = None, **params):
309 def create_request(self, operation, uri = None, bucket = None, object = None, headers = None, extra = None, **params):
271310 resource = { 'bucket' : None, 'uri' : "/" }
311
312 if uri and (bucket or object):
313 raise ValueError("Both 'uri' and either 'bucket' or 'object' parameters supplied")
314 ## If URI is given use that instead of bucket/object parameters
315 if uri:
316 bucket = uri.bucket()
317 object = uri.has_object() and uri.object() or None
318
272319 if bucket:
273320 resource['bucket'] = str(bucket)
274321 if object:
301348 debug("CreateRequest: resource[uri]=" + resource['uri'])
302349 return (method_string, resource, headers)
303350
304 def send_request(self, request, body = None, retries = 5):
351 def _fail_wait(self, retries):
352 # Wait a few seconds. The more it fails the more we wait.
353 return (self._max_retries - retries + 1) * 3
354
355 def send_request(self, request, body = None, retries = _max_retries):
305356 method_string, resource, headers = request
306 info("Processing request, please wait...")
357 debug("Processing request, please wait...")
307358 try:
308359 conn = self.get_connection(resource['bucket'])
309360 conn.request(method_string, self.format_uri(resource), body, headers)
318369 except Exception, e:
319370 if retries:
320371 warning("Retrying failed request: %s (%s)" % (resource['uri'], e))
372 warning("Waiting %d sec..." % self._fail_wait(retries))
373 time.sleep(self._fail_wait(retries))
321374 return self.send_request(request, body, retries - 1)
322375 else:
323376 raise S3RequestError("Request failed for: %s" % resource['uri'])
327380 redir_bucket = getTextFromXml(response['data'], ".//Bucket")
328381 redir_hostname = getTextFromXml(response['data'], ".//Endpoint")
329382 self.set_hostname(redir_bucket, redir_hostname)
330 info("Redirected to: %s" % (redir_hostname))
383 warning("Redirected to: %s" % (redir_hostname))
331384 return self.send_request(request, body)
332385
333386 if response["status"] >= 500:
335388 if retries:
336389 warning(u"Retrying failed request: %s" % resource['uri'])
337390 warning(unicode(e))
391 warning("Waiting %d sec..." % self._fail_wait(retries))
392 time.sleep(self._fail_wait(retries))
338393 return self.send_request(request, body, retries - 1)
339394 else:
340395 raise e
344399
345400 return response
346401
347 def send_file(self, request, file, throttle = 0, retries = 3):
402 def send_file(self, request, file, labels, throttle = 0, retries = _max_retries):
348403 method_string, resource, headers = request
349 info("Sending file '%s', please wait..." % file.name)
350 conn = self.get_connection(resource['bucket'])
351 conn.connect()
352 conn.putrequest(method_string, self.format_uri(resource))
353 for header in headers.keys():
354 conn.putheader(header, str(headers[header]))
355 conn.endheaders()
404 size_left = size_total = headers.get("content-length")
405 if self.config.progress_meter:
406 progress = self.config.progress_class(labels, size_total)
407 else:
408 info("Sending file '%s', please wait..." % file.name)
409 timestamp_start = time.time()
410 try:
411 conn = self.get_connection(resource['bucket'])
412 conn.connect()
413 conn.putrequest(method_string, self.format_uri(resource))
414 for header in headers.keys():
415 conn.putheader(header, str(headers[header]))
416 conn.endheaders()
417 except Exception, e:
418 if self.config.progress_meter:
419 progress.done("failed")
420 if retries:
421 warning("Retrying failed request: %s (%s)" % (resource['uri'], e))
422 warning("Waiting %d sec..." % self._fail_wait(retries))
423 time.sleep(self._fail_wait(retries))
424 # Connection error -> same throttle value
425 return self.send_file(request, file, labels, throttle, retries - 1)
426 else:
427 raise S3UploadError("Upload failed for: %s" % resource['uri'])
356428 file.seek(0)
357 timestamp_start = time.time()
358 md5_hash = md5.new()
359 size_left = size_total = headers.get("content-length")
360 while (size_left > 0):
361 debug("SendFile: Reading up to %d bytes from '%s'" % (self.config.send_chunk, file.name))
362 data = file.read(self.config.send_chunk)
363 md5_hash.update(data)
364 debug("SendFile: Sending %d bytes to the server" % len(data))
365 try:
429 md5_hash = md5()
430 try:
431 while (size_left > 0):
432 #debug("SendFile: Reading up to %d bytes from '%s'" % (self.config.send_chunk, file.name))
433 data = file.read(self.config.send_chunk)
434 md5_hash.update(data)
366435 conn.send(data)
367 except Exception, e:
368 ## When an exception occurs insert a
369 if retries:
370 conn.close()
371 warning("Upload of '%s' failed %s " % (file.name, e))
372 throttle = throttle and throttle * 5 or 0.01
373 warning("Retrying on lower speed (throttle=%0.2f)" % throttle)
374 return self.send_file(request, file, throttle, retries - 1)
375 else:
376 debug("Giving up on '%s' %s" % (file.name, e))
377 raise S3UploadError
378
379 size_left -= len(data)
380 if throttle:
381 time.sleep(throttle)
382 debug("Sent %d bytes (%d %% of %d)" % (
383 (size_total - size_left),
384 (size_total - size_left) * 100 / size_total,
385 size_total))
436 if self.config.progress_meter:
437 progress.update(delta_position = len(data))
438 size_left -= len(data)
439 if throttle:
440 time.sleep(throttle)
441 md5_computed = md5_hash.hexdigest()
442 response = {}
443 http_response = conn.getresponse()
444 response["status"] = http_response.status
445 response["reason"] = http_response.reason
446 response["headers"] = convertTupleListToDict(http_response.getheaders())
447 response["data"] = http_response.read()
448 response["size"] = size_total
449 conn.close()
450 debug(u"Response: %s" % response)
451 except Exception, e:
452 if self.config.progress_meter:
453 progress.done("failed")
454 if retries:
455 throttle = throttle and throttle * 5 or 0.01
456 warning("Upload failed: %s (%s)" % (resource['uri'], e))
457 warning("Retrying on lower speed (throttle=%0.2f)" % throttle)
458 warning("Waiting %d sec..." % self._fail_wait(retries))
459 time.sleep(self._fail_wait(retries))
460 # Connection error -> same throttle value
461 return self.send_file(request, file, labels, throttle, retries - 1)
462 else:
463 debug("Giving up on '%s' %s" % (file.name, e))
464 raise S3UploadError("Upload failed for: %s" % resource['uri'])
465
386466 timestamp_end = time.time()
387 md5_computed = md5_hash.hexdigest()
388 response = {}
389 http_response = conn.getresponse()
390 response["status"] = http_response.status
391 response["reason"] = http_response.reason
392 response["headers"] = convertTupleListToDict(http_response.getheaders())
393 response["data"] = http_response.read()
394467 response["elapsed"] = timestamp_end - timestamp_start
395 response["size"] = size_total
396468 response["speed"] = response["elapsed"] and float(response["size"]) / response["elapsed"] or float(-1)
397 conn.close()
469
470 if self.config.progress_meter:
471 ## The above conn.close() takes some time -> update() progress meter
472 ## to correct the average speed. Otherwise people will complain that
473 ## 'progress' and response["speed"] are inconsistent ;-)
474 progress.update()
475 progress.done("done")
398476
399477 if response["status"] == 307:
400478 ## RedirectPermanent
401479 redir_bucket = getTextFromXml(response['data'], ".//Bucket")
402480 redir_hostname = getTextFromXml(response['data'], ".//Endpoint")
403481 self.set_hostname(redir_bucket, redir_hostname)
404 info("Redirected to: %s" % (redir_hostname))
405 return self.send_file(request, file)
482 warning("Redirected to: %s" % (redir_hostname))
483 return self.send_file(request, file, labels)
406484
407485 # S3 from time to time doesn't send ETag back in a response :-(
408486 # Force re-upload here.
409487 if not response['headers'].has_key('etag'):
410488 response['headers']['etag'] = ''
411489
490 if response["status"] < 200 or response["status"] > 299:
491 if response["status"] >= 500:
492 ## AWS internal error - retry
493 if retries:
494 warning("Upload failed: %s (%s)" % (resource['uri'], S3Error(response)))
495 warning("Waiting %d sec..." % self._fail_wait(retries))
496 time.sleep(self._fail_wait(retries))
497 return self.send_file(request, file, labels, throttle, retries - 1)
498 else:
499 warning("Too many failures. Giving up on '%s'" % (file.name))
500 raise S3UploadError
501 ## Non-recoverable error
502 raise S3Error(response)
503
412504 debug("MD5 sums: computed=%s, received=%s" % (md5_computed, response["headers"]["etag"]))
413505 if response["headers"]["etag"].strip('"\'') != md5_hash.hexdigest():
414506 warning("MD5 Sums don't match!")
415507 if retries:
416 info("Retrying upload.")
417 return self.send_file(request, file, throttle, retries - 1)
418 else:
419 debug("Too many failures. Giving up on '%s'" % (file.name))
508 warning("Retrying upload of %s" % (file.name))
509 return self.send_file(request, file, labels, throttle, retries - 1)
510 else:
511 warning("Too many failures. Giving up on '%s'" % (file.name))
420512 raise S3UploadError
421513
422 if response["status"] < 200 or response["status"] > 299:
423 raise S3Error(response)
424 return response
425
426 def recv_file(self, request, stream):
514 return response
515
516 def recv_file(self, request, stream, labels, start_position = 0, retries = _max_retries):
427517 method_string, resource, headers = request
428 info("Receiving file '%s', please wait..." % stream.name)
429 conn = self.get_connection(resource['bucket'])
430 conn.connect()
431 conn.putrequest(method_string, self.format_uri(resource))
432 for header in headers.keys():
433 conn.putheader(header, str(headers[header]))
434 conn.endheaders()
435 response = {}
436 http_response = conn.getresponse()
437 response["status"] = http_response.status
438 response["reason"] = http_response.reason
439 response["headers"] = convertTupleListToDict(http_response.getheaders())
518 if self.config.progress_meter:
519 progress = self.config.progress_class(labels, 0)
520 else:
521 info("Receiving file '%s', please wait..." % stream.name)
522 timestamp_start = time.time()
523 try:
524 conn = self.get_connection(resource['bucket'])
525 conn.connect()
526 conn.putrequest(method_string, self.format_uri(resource))
527 for header in headers.keys():
528 conn.putheader(header, str(headers[header]))
529 if start_position > 0:
530 debug("Requesting Range: %d .. end" % start_position)
531 conn.putheader("Range", "bytes=%d-" % start_position)
532 conn.endheaders()
533 response = {}
534 http_response = conn.getresponse()
535 response["status"] = http_response.status
536 response["reason"] = http_response.reason
537 response["headers"] = convertTupleListToDict(http_response.getheaders())
538 debug("Response: %s" % response)
539 except Exception, e:
540 if self.config.progress_meter:
541 progress.done("failed")
542 if retries:
543 warning("Retrying failed request: %s (%s)" % (resource['uri'], e))
544 warning("Waiting %d sec..." % self._fail_wait(retries))
545 time.sleep(self._fail_wait(retries))
546 # Connection error -> same throttle value
547 return self.recv_file(request, stream, labels, start_position, retries - 1)
548 else:
549 raise S3DownloadError("Download failed for: %s" % resource['uri'])
440550
441551 if response["status"] == 307:
442552 ## RedirectPermanent
444554 redir_bucket = getTextFromXml(response['data'], ".//Bucket")
445555 redir_hostname = getTextFromXml(response['data'], ".//Endpoint")
446556 self.set_hostname(redir_bucket, redir_hostname)
447 info("Redirected to: %s" % (redir_hostname))
448 return self.recv_file(request, stream)
557 warning("Redirected to: %s" % (redir_hostname))
558 return self.recv_file(request, stream, labels)
449559
450560 if response["status"] < 200 or response["status"] > 299:
451561 raise S3Error(response)
452562
453 md5_hash = md5.new()
454 size_left = size_total = int(response["headers"]["content-length"])
455 size_recvd = 0
456 timestamp_start = time.time()
457 while (size_recvd < size_total):
458 this_chunk = size_left > self.config.recv_chunk and self.config.recv_chunk or size_left
459 debug("ReceiveFile: Receiving up to %d bytes from the server" % this_chunk)
460 data = http_response.read(this_chunk)
461 debug("ReceiveFile: Writing %d bytes to file '%s'" % (len(data), stream.name))
462 stream.write(data)
463 md5_hash.update(data)
464 size_recvd += len(data)
465 debug("Received %d bytes (%d %% of %d)" % (
466 size_recvd,
467 size_recvd * 100 / size_total,
468 size_total))
469 conn.close()
563 if start_position == 0:
564 # Only compute MD5 on the fly if we're downloading from beginning
565 # Otherwise we'd get a nonsense.
566 md5_hash = md5()
567 size_left = int(response["headers"]["content-length"])
568 size_total = start_position + size_left
569 current_position = start_position
570
571 if self.config.progress_meter:
572 progress.total_size = size_total
573 progress.initial_position = current_position
574 progress.current_position = current_position
575
576 try:
577 while (current_position < size_total):
578 this_chunk = size_left > self.config.recv_chunk and self.config.recv_chunk or size_left
579 data = http_response.read(this_chunk)
580 stream.write(data)
581 if start_position == 0:
582 md5_hash.update(data)
583 current_position += len(data)
584 ## Call progress meter from here...
585 if self.config.progress_meter:
586 progress.update(delta_position = len(data))
587 conn.close()
588 except Exception, e:
589 if self.config.progress_meter:
590 progress.done("failed")
591 if retries:
592 warning("Retrying failed request: %s (%s)" % (resource['uri'], e))
593 warning("Waiting %d sec..." % self._fail_wait(retries))
594 time.sleep(self._fail_wait(retries))
595 # Connection error -> same throttle value
596 return self.recv_file(request, stream, labels, current_position, retries - 1)
597 else:
598 raise S3DownloadError("Download failed for: %s" % resource['uri'])
599
600 stream.flush()
470601 timestamp_end = time.time()
471 response["md5"] = md5_hash.hexdigest()
602
603 if self.config.progress_meter:
604 ## The above stream.flush() may take some time -> update() progress meter
605 ## to correct the average speed. Otherwise people will complain that
606 ## 'progress' and response["speed"] are inconsistent ;-)
607 progress.update()
608 progress.done("done")
609
610 if start_position == 0:
611 # Only compute MD5 on the fly if we were downloading from the beginning
612 response["md5"] = md5_hash.hexdigest()
613 else:
614 # Otherwise try to compute MD5 of the output file
615 try:
616 response["md5"] = hash_file_md5(stream.name)
617 except IOError, e:
618 if e.errno != errno.ENOENT:
619 warning("Unable to open file: %s: %s" % (stream.name, e))
620 warning("Unable to verify MD5. Assume it matches.")
621 response["md5"] = response["headers"]["etag"]
622
472623 response["md5match"] = response["headers"]["etag"].find(response["md5"]) >= 0
473624 response["elapsed"] = timestamp_end - timestamp_start
474 response["size"] = size_recvd
625 response["size"] = current_position
475626 response["speed"] = response["elapsed"] and float(response["size"]) / response["elapsed"] or float(-1)
476 if response["size"] != long(response["headers"]["content-length"]):
627 if response["size"] != start_position + long(response["headers"]["content-length"]):
477628 warning("Reported size (%s) does not match received size (%s)" % (
478 response["headers"]["content-length"], response["size"]))
629 start_position + response["headers"]["content-length"], response["size"]))
479630 debug("ReceiveFile: Computed MD5 = %s" % response["md5"])
480631 if not response["md5match"]:
481632 warning("MD5 signatures do not match: computed=%s, received=%s" % (
494645 h += "/" + resource['bucket']
495646 h += resource['uri']
496647 debug("SignHeaders: " + repr(h))
497 return base64.encodestring(hmac.new(self.config.secret_key, h, sha).digest()).strip()
648 return base64.encodestring(hmac.new(self.config.secret_key, h, sha1).digest()).strip()
498649
499650 @staticmethod
500651 def check_bucket_name(bucket, dns_strict = True):
+67
-1
S3/S3Uri.py less more
22 ## http://www.logix.cz/michal
33 ## License: GPL Version 2
44
5 import os
56 import re
67 import sys
78 from BidirMap import BidirMap
4142 def public_url(self):
4243 raise ValueError("This S3 URI does not have Anonymous URL representation")
4344
45 def basename(self):
46 return self.__unicode__().split("/")[-1]
47
4448 class S3UriS3(S3Uri):
4549 type = "s3"
4650 _re = re.compile("^s3://([^/]+)/?(.*)", re.IGNORECASE)
6771 def uri(self):
6872 return "/".join(["s3:/", self._bucket, self._object])
6973
74 def is_dns_compatible(self):
75 return S3.check_bucket_name_dns_conformity(self._bucket)
76
7077 def public_url(self):
71 if S3.check_bucket_name_dns_conformity(self._bucket):
78 if self.is_dns_compatible():
7279 return "http://%s.s3.amazonaws.com/%s" % (self._bucket, self._object)
7380 else:
7481 return "http://s3.amazonaws.com/%s/%s" % (self._bucket, self._object)
7582
83 def host_name(self):
84 if self.is_dns_compatible():
85 return "%s.s3.amazonaws.com" % (self._bucket)
86 else:
87 return "s3.amazonaws.com"
88
7689 @staticmethod
7790 def compose_uri(bucket, object = ""):
7891 return "s3://%s/%s" % (bucket, object)
7992
93 @staticmethod
94 def httpurl_to_s3uri(http_url):
95 m=re.match("(https?://)?([^/]+)/?(.*)", http_url, re.IGNORECASE)
96 hostname, object = m.groups()[1:]
97 hostname = hostname.lower()
98 if hostname == "s3.amazonaws.com":
99 ## old-style url: http://s3.amazonaws.com/bucket/object
100 if object.count("/") == 0:
101 ## no object given
102 bucket = object
103 object = ""
104 else:
105 ## bucket/object
106 bucket, object = object.split("/", 1)
107 elif hostname.endswith(".s3.amazonaws.com"):
108 ## new-style url: http://bucket.s3.amazonaws.com/object
109 bucket = hostname[:-(len(".s3.amazonaws.com"))]
110 else:
111 raise ValueError("Unable to parse URL: %s" % http_url)
112 return S3Uri("s3://%(bucket)s/%(object)s" % {
113 'bucket' : bucket,
114 'object' : object })
115
80116 class S3UriS3FS(S3Uri):
81117 type = "s3fs"
82118 _re = re.compile("^s3fs://([^/]*)/?(.*)", re.IGNORECASE)
113149 def uri(self):
114150 return "/".join(["file:/", self.path()])
115151
152 def isdir(self):
153 return os.path.isdir(self.path())
154
155 def dirname(self):
156 return os.path.dirname(self.path())
157
158 class S3UriCloudFront(S3Uri):
159 type = "cf"
160 _re = re.compile("^cf://([^/]*)/?", re.IGNORECASE)
161 def __init__(self, string):
162 match = self._re.match(string)
163 if not match:
164 raise ValueError("%s: not a CloudFront URI" % string)
165 groups = match.groups()
166 self._dist_id = groups[0]
167
168 def dist_id(self):
169 return self._dist_id
170
171 def uri(self):
172 return "/".join(["cf:/", self.dist_id()])
173
116174 if __name__ == "__main__":
117175 uri = S3Uri("s3://bucket/object")
118176 print "type() =", type(uri)
142200 print "uri.type=", uri.type
143201 print "path =", uri.path()
144202 print
203
204 uri = S3Uri("cf://1234567890ABCD/")
205 print "type() =", type(uri)
206 print "uri =", uri
207 print "uri.type=", uri.type
208 print "dist_id =", uri.dist_id()
209 print
210
+105
-44
S3/Utils.py less more
77 import re
88 import string
99 import random
10 import md5
10 import rfc822
11 try:
12 from hashlib import md5
13 except ImportError:
14 from md5 import md5
1115 import errno
1216
1317 from logging import debug, info, warning, error
18
19 import Config
1420
1521 try:
1622 import xml.etree.ElementTree as ET
1723 except ImportError:
1824 import elementtree.ElementTree as ET
1925
20 def stripTagXmlns(xmlns, tag):
21 """
22 Returns a function that, given a tag name argument, removes
23 eventual ElementTree xmlns from it.
24
25 Example:
26 stripTagXmlns("{myXmlNS}tag") -> "tag"
27 """
28 if not xmlns:
29 return tag
30 return re.sub(xmlns, "", tag)
31
32 def fixupXPath(xmlns, xpath, max = 0):
33 if not xmlns:
34 return xpath
35 retval = re.subn("//", "//%s" % xmlns, xpath, max)[0]
36 return retval
37
38 def parseNodes(nodes, xmlns = ""):
26 def parseNodes(nodes):
3927 ## WARNING: Ignores text nodes from mixed xml/text.
4028 ## For instance <tag1>some text<tag2>other text</tag2></tag1>
4129 ## will be ignore "some text" node
4331 for node in nodes:
4432 retval_item = {}
4533 for child in node.getchildren():
46 name = stripTagXmlns(xmlns, child.tag)
34 name = child.tag
4735 if child.getchildren():
48 retval_item[name] = parseNodes([child], xmlns)
36 retval_item[name] = parseNodes([child])
4937 else:
5038 retval_item[name] = node.findtext(".//%s" % child.tag)
5139 retval.append(retval_item)
5240 return retval
5341
54 def getNameSpace(element):
55 if not element.tag.startswith("{"):
56 return ""
57 return re.compile("^(\{[^}]+\})").match(element.tag).groups()[0]
58
42 def stripNameSpace(xml):
43 """
44 removeNameSpace(xml) -- remove top-level AWS namespace
45 """
46 r = re.compile('^(<?[^>]+?>\s?)(<\w+) xmlns=[\'"](http://[^\'"]+)[\'"](.*)', re.MULTILINE)
47 if r.match(xml):
48 xmlns = r.match(xml).groups()[2]
49 xml = r.sub("\\1\\2\\4", xml)
50 else:
51 xmlns = None
52 return xml, xmlns
53
54 def getTreeFromXml(xml):
55 xml, xmlns = stripNameSpace(xml)
56 tree = ET.fromstring(xml)
57 if xmlns:
58 tree.attrib['xmlns'] = xmlns
59 return tree
60
5961 def getListFromXml(xml, node):
60 tree = ET.fromstring(xml)
61 xmlns = getNameSpace(tree)
62 nodes = tree.findall('.//%s%s' % (xmlns, node))
63 return parseNodes(nodes, xmlns)
64
62 tree = getTreeFromXml(xml)
63 nodes = tree.findall('.//%s' % (node))
64 return parseNodes(nodes)
65
66 def getDictFromTree(tree):
67 ret_dict = {}
68 for child in tree.getchildren():
69 if child.getchildren():
70 ## Complex-type child. We're not interested
71 continue
72 if ret_dict.has_key(child.tag):
73 if not type(ret_dict[child.tag]) == list:
74 ret_dict[child.tag] = [ret_dict[child.tag]]
75 ret_dict[child.tag].append(child.text or "")
76 else:
77 ret_dict[child.tag] = child.text or ""
78 return ret_dict
79
6580 def getTextFromXml(xml, xpath):
66 tree = ET.fromstring(xml)
67 xmlns = getNameSpace(tree)
81 tree = getTreeFromXml(xml)
6882 if tree.tag.endswith(xpath):
6983 return tree.text
7084 else:
71 return tree.findtext(fixupXPath(xmlns, xpath))
85 return tree.findtext(xpath)
86
87 def getRootTagName(xml):
88 tree = getTreeFromXml(xml)
89 return tree.tag
90
91 def xmlTextNode(tag_name, text):
92 el = ET.Element(tag_name)
93 el.text = unicode(text)
94 return el
95
96 def appendXmlTextNode(tag_name, text, parent):
97 """
98 Creates a new <tag_name> Node and sets
99 its content to 'text'. Then appends the
100 created Node to 'parent' element if given.
101 Returns the newly created Node.
102 """
103 parent.append(xmlTextNode(tag_name, text))
72104
73105 def dateS3toPython(date):
74106 date = re.compile("\.\d\d\dZ").sub(".000Z", date)
79111 ## Currently the argument to strptime() is GMT but mktime()
80112 ## treats it as "localtime". Anyway...
81113 return time.mktime(dateS3toPython(date))
114
115 def dateRFC822toPython(date):
116 return rfc822.parsedate(date)
117
118 def dateRFC822toUnix(date):
119 return time.mktime(dateRFC822toPython(date))
82120
83121 def formatSize(size, human_readable = False, floating_point = False):
84122 size = floating_point and float(size) or int(size)
136174 return mktmpsomething(prefix, randchars, createfunc)
137175
138176 def hash_file_md5(filename):
139 h = md5.new()
177 h = md5()
140178 f = open(filename, "rb")
141179 while True:
142180 # Hash 32kB chunks
173211 return False
174212 return True
175213
176 def unicodise(string):
214 def unicodise(string, encoding = None, errors = "replace"):
177215 """
178216 Convert 'string' to Unicode or raise an exception.
179217 """
180 debug("Unicodising %r" % string)
218
219 if not encoding:
220 encoding = Config.Config().encoding
221
181222 if type(string) == unicode:
182223 return string
224 debug("Unicodising %r using %s" % (string, encoding))
183225 try:
184 return string.decode("utf-8")
226 return string.decode(encoding, errors)
185227 except UnicodeDecodeError:
186228 raise UnicodeDecodeError("Conversion to unicode failed: %r" % string)
187229
188 def try_unicodise(string):
230 def deunicodise(string, encoding = None, errors = "replace"):
231 """
232 Convert unicode 'string' to <type str>, by default replacing
233 all invalid characters with '?' or raise an exception.
234 """
235
236 if not encoding:
237 encoding = Config.Config().encoding
238
239 if type(string) != unicode:
240 return str(string)
241 debug("DeUnicodising %r using %s" % (string, encoding))
189242 try:
190 return unicodise(string)
191 except UnicodeDecodeError:
192 return string
193
243 return string.encode(encoding, errors)
244 except UnicodeEncodeError:
245 raise UnicodeEncodeError("Conversion from unicode failed: %r" % string)
246
247 def unicodise_safe(string, encoding = None):
248 """
249 Convert 'string' to Unicode according to current encoding
250 and replace all invalid characters with '?'
251 """
252
253 return unicodise(deunicodise(string, encoding), encoding).replace(u'\ufffd', '?')
254
+947
-458
s3cmd less more
1010 import os
1111 import re
1212 import errno
13 import pwd, grp
1413 import glob
1514 import traceback
1615 import codecs
16 import locale
17 import subprocess
1718
1819 from copy import copy
1920 from optparse import OptionParser, Option, OptionValueError, IndentedHelpFormatter
2021 from logging import debug, info, warning, error
2122 from distutils.spawn import find_executable
2223
23 ## Output native on TTY, UTF-8 otherwise (redirects)
24 #_stdout = sys.stdout.isatty() and sys.stdout or codecs.getwriter("utf-8")(sys.stdout)
25 #_stderr = sys.stderr.isatty() and sys.stderr or codecs.getwriter("utf-8")(sys.stderr)
26 ## Output UTF-8 in all cases
27 _stdout = codecs.getwriter("utf-8")(sys.stdout)
28 _stderr = codecs.getwriter("utf-8")(sys.stderr)
29 ## Leave it to the terminal
30 #_stdout = sys.stdout
31 #_stderr = sys.stderr
32
3324 def output(message):
34 _stdout.write(message + "\n")
25 sys.stdout.write(message + "\n")
3526
3627 def check_args_type(args, type, verbose_type):
3728 for arg in args:
5748 buckets_size += size
5849 total_size, size_coeff = formatSize(buckets_size, Config().human_readable_sizes)
5950 total_size_str = str(total_size) + size_coeff
60 output("".rjust(8, "-"))
61 output("%s Total" % (total_size_str.ljust(8)))
51 output(u"".rjust(8, "-"))
52 output(u"%s Total" % (total_size_str.ljust(8)))
6253
6354 def subcmd_bucket_usage(s3, uri):
6455 bucket = uri.bucket()
6758 if object.endswith('*'):
6859 object = object[:-1]
6960 try:
70 response = s3.bucket_list(bucket, prefix = object)
61 response = s3.bucket_list(bucket, prefix = object, recursive = True)
7162 except S3Error, e:
7263 if S3.codes.has_key(e.Code):
7364 error(S3.codes[e.Code] % bucket)
8071 bucket_size += size
8172 total_size, size_coeff = formatSize(bucket_size, Config().human_readable_sizes)
8273 total_size_str = str(total_size) + size_coeff
83 output("%s %s" % (total_size_str.ljust(8), uri))
74 output(u"%s %s" % (total_size_str.ljust(8), uri))
8475 return bucket_size
8576
8677 def cmd_ls(args):
9990
10091 for bucket in response["list"]:
10192 subcmd_bucket_list(s3, S3Uri("s3://" + bucket["Name"]))
102 output("")
93 output(u"")
10394
10495
10596 def subcmd_buckets_list_all(s3):
10697 response = s3.list_all_buckets()
10798 for bucket in response["list"]:
108 output("%s s3://%s" % (
99 output(u"%s s3://%s" % (
109100 formatDateTime(bucket["CreationDate"]),
110101 bucket["Name"],
111102 ))
112103
113104 def subcmd_bucket_list(s3, uri):
114105 bucket = uri.bucket()
115 object = uri.object()
116
117 output("Bucket 's3://%s':" % bucket)
118 if object.endswith('*'):
119 object = object[:-1]
106 prefix = uri.object()
107
108 debug(u"Bucket 's3://%s':" % bucket)
109 if prefix.endswith('*'):
110 prefix = prefix[:-1]
120111 try:
121 response = s3.bucket_list(bucket, prefix = object)
112 response = s3.bucket_list(bucket, prefix = prefix)
122113 except S3Error, e:
123114 if S3.codes.has_key(e.info["Code"]):
124115 error(S3.codes[e.info["Code"]] % bucket)
125116 return
126117 else:
127118 raise
119
120 for prefix in response['common_prefixes']:
121 output(u"%s %s" % (
122 "DIR".rjust(26),
123 uri.compose_uri(bucket, prefix["Prefix"])))
124
128125 for object in response["list"]:
129126 size, size_coeff = formatSize(object["Size"], Config().human_readable_sizes)
130 output("%s %s%s %s" % (
127 output(u"%s %s%s %s" % (
131128 formatDateTime(object["LastModified"]),
132129 str(size).rjust(8), size_coeff.ljust(1),
133130 uri.compose_uri(bucket, object["Key"]),
134131 ))
135132
136133 def cmd_bucket_create(args):
137 uri = S3Uri(args[0])
138 if not uri.type == "s3" or not uri.has_bucket() or uri.has_object():
139 raise ParameterError("Expecting S3 URI with just the bucket name set instead of '%s'" % args[0])
140 try:
141 s3 = S3(Config())
142 response = s3.bucket_create(uri.bucket(), cfg.bucket_location)
143 except S3Error, e:
144 if S3.codes.has_key(e.info["Code"]):
145 error(S3.codes[e.info["Code"]] % uri.bucket())
146 return
147 else:
148 raise
149 output("Bucket '%s' created" % uri.bucket())
150
151 def cmd_bucket_delete(args):
152 uri = S3Uri(args[0])
153 if not uri.type == "s3" or not uri.has_bucket() or uri.has_object():
154 raise ParameterError("Expecting S3 URI with just the bucket name set instead of '%s'" % args[0])
155 try:
156 s3 = S3(Config())
157 response = s3.bucket_delete(uri.bucket())
158 except S3Error, e:
159 if S3.codes.has_key(e.info["Code"]):
160 error(S3.codes[e.info["Code"]] % uri.bucket())
161 return
162 else:
163 raise
164 output("Bucket '%s' removed" % uri.bucket())
165
166 def cmd_object_put(args):
167134 s3 = S3(Config())
168
169 uri_arg = args.pop()
170 check_args_type(args, 'file', 'filename')
171
172 uri = S3Uri(uri_arg)
173 if uri.type != "s3":
174 raise ParameterError("Expecting S3 URI instead of '%s'" % uri_arg)
175
176 if len(args) > 1 and uri.object() != "" and not Config().force:
177 error("When uploading multiple files the last argument must")
178 error("be a S3 URI specifying just the bucket name")
179 error("WITHOUT object name!")
180 error("Alternatively use --force argument and the specified")
181 error("object name will be prefixed to all stored filenames.")
182 sys.exit(1)
183
184 seq = 0
185 total = len(args)
186 for file in args:
187 seq += 1
188 uri_arg_final = str(uri)
189 if len(args) > 1 or uri.object() == "":
190 uri_arg_final += os.path.basename(file)
191
192 uri_final = S3Uri(uri_arg_final)
193 extra_headers = {}
194 real_filename = file
195 if Config().encrypt:
196 exitcode, real_filename, extra_headers["x-amz-meta-s3tools-gpgenc"] = gpg_encrypt(file)
135 for arg in args:
136 uri = S3Uri(arg)
137 if not uri.type == "s3" or not uri.has_bucket() or uri.has_object():
138 raise ParameterError("Expecting S3 URI with just the bucket name set instead of '%s'" % arg)
197139 try:
198 response = s3.object_put_uri(real_filename, uri_final, extra_headers)
199 except S3UploadError, e:
200 error("Upload of '%s' failed too many times. Skipping that file." % real_filename)
201 continue
202 except InvalidFileError, e:
203 warning("File can not be uploaded: %s" % e)
204 continue
205 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
206 output("File '%s' stored as %s (%d bytes in %0.1f seconds, %0.2f %sB/s) [%d of %d]" %
207 (file, uri_final, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
208 seq, total))
209 if Config().acl_public:
210 output("Public URL of the object is: %s" %
211 (uri_final.public_url()))
212 if Config().encrypt and real_filename != file:
213 debug("Removing temporary encrypted file: %s" % real_filename)
214 os.remove(real_filename)
215
216 def cmd_object_get(args):
217 s3 = S3(Config())
218
219 if not S3Uri(args[0]).type == 's3':
220 raise ParameterError("Expecting S3 URI instead of '%s'" % args[0])
221
222 destination_dir = None
223 destination_file = None
224 if len(args) > 1:
225 if S3Uri(args[-1]).type == 's3':
226 # all S3, use object names to local dir
227 check_args_type(args, type="s3", verbose_type="S3 URI") # May raise ParameterError
228 else:
229 if (len(args) > 2):
230 # last must be dir, all preceding S3
231 if not os.path.isdir(args[-1]):
232 raise ParameterError("Last parameter must be a directory")
233 destination_dir = args.pop()
234 check_args_type(args, type="s3", verbose_type="S3 URI") # May raise ParameterError
235 else:
236 # last must be a dir or a filename
237 if os.path.isdir(args[-1]):
238 destination_dir = args.pop()
239 else:
240 destination_file = args.pop()
241
242 while (len(args)):
243 uri_arg = args.pop(0)
244 uri = S3Uri(uri_arg)
245
246 if destination_file:
247 destination = destination_file
248 elif destination_dir:
249 destination = destination_dir + "/" + uri.object()
250 else:
251 # By default the destination filename is the object name
252 destination = uri.object()
253 if destination == "-":
254 ## stdout
255 dst_stream = sys.stdout
256 else:
257 ## File
258 if not Config().force and os.path.exists(destination):
259 raise ParameterError("File %s already exists. Use --force to overwrite it" % destination)
260 try:
261 dst_stream = open(destination, "wb")
262 except IOError, e:
263 error("Skipping %s: %s" % (destination, e.strerror))
264 continue
265 response = s3.object_get_uri(uri, dst_stream)
266 if response["headers"].has_key("x-amz-meta-s3tools-gpgenc"):
267 gpg_decrypt(destination, response["headers"]["x-amz-meta-s3tools-gpgenc"])
268 response["size"] = os.stat(destination)[6]
269 if destination != "-":
270 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
271 output("Object %s saved as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s)" %
272 (uri, destination, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1]))
273
274 def cmd_object_del(args):
275 s3 = S3(Config())
276
277 while (len(args)):
278 uri_arg = args.pop(0)
279 uri = S3Uri(uri_arg)
280 if uri.type != "s3" or not uri.has_object():
281 raise ParameterError("Expecting S3 URI instead of '%s'" % uri_arg)
282
283 response = s3.object_delete_uri(uri)
284 output("Object %s deleted" % uri)
285
286 def cmd_info(args):
287 s3 = S3(Config())
288
289 while (len(args)):
290 uri_arg = args.pop(0)
291 uri = S3Uri(uri_arg)
292 if uri.type != "s3" or not uri.has_bucket():
293 raise ParameterError("Expecting S3 URI instead of '%s'" % uri_arg)
294
295 try:
296 if uri.has_object():
297 info = s3.object_info(uri)
298 output("%s (object):" % uri.uri())
299 output(" File size: %s" % info['headers']['content-length'])
300 output(" Last mod: %s" % info['headers']['last-modified'])
301 output(" MIME type: %s" % info['headers']['content-type'])
302 output(" MD5 sum: %s" % info['headers']['etag'].strip('"'))
303 else:
304 info = s3.bucket_info(uri)
305 output("%s (bucket):" % uri.uri())
306 output(" Location: %s" % info['bucket-location'])
307 acl = s3.get_acl(uri)
308 for user in acl.keys():
309 output(" ACL: %s: %s" % (user, acl[user]))
140 response = s3.bucket_create(uri.bucket(), cfg.bucket_location)
141 output(u"Bucket '%s' created" % uri.uri())
310142 except S3Error, e:
311143 if S3.codes.has_key(e.info["Code"]):
312144 error(S3.codes[e.info["Code"]] % uri.bucket())
314146 else:
315147 raise
316148
149 def cmd_bucket_delete(args):
150 def _bucket_delete_one(uri):
151 try:
152 response = s3.bucket_delete(uri.bucket())
153 except S3Error, e:
154 if e.info['Code'] == 'BucketNotEmpty' and (cfg.force or cfg.recursive):
155 warning(u"Bucket is not empty. Removing all the objects from it first. This may take some time...")
156 subcmd_object_del_uri(uri, recursive = True)
157 return _bucket_delete_one(uri)
158 elif S3.codes.has_key(e.info["Code"]):
159 error(S3.codes[e.info["Code"]] % uri.bucket())
160 return
161 else:
162 raise
163
164 s3 = S3(Config())
165 for arg in args:
166 uri = S3Uri(arg)
167 if not uri.type == "s3" or not uri.has_bucket() or uri.has_object():
168 raise ParameterError("Expecting S3 URI with just the bucket name set instead of '%s'" % arg)
169 _bucket_delete_one(uri)
170 output(u"Bucket '%s' removed" % uri.uri())
171
172 def fetch_local_list(args, recursive = None):
173 local_uris = []
174 local_list = SortedDict()
175 single_file = False
176
177 if type(args) not in (list, tuple):
178 args = [args]
179
180 if recursive == None:
181 recursive = cfg.recursive
182
183 for arg in args:
184 uri = S3Uri(arg)
185 if not uri.type == 'file':
186 raise ParameterError("Expecting filename or directory instead of: %s" % arg)
187 if uri.isdir() and not recursive:
188 raise ParameterError("Use --recursive to upload a directory: %s" % arg)
189 local_uris.append(uri)
190
191 for uri in local_uris:
192 list_for_uri, single_file = _get_filelist_local(uri)
193 local_list.update(list_for_uri)
194
195 ## Single file is True if and only if the user
196 ## specified one local URI and that URI represents
197 ## a FILE. Ie it is False if the URI was of a DIR
198 ## and that dir contained only one FILE. That's not
199 ## a case of single_file==True.
200 if len(local_list) > 1:
201 single_file = False
202
203 return local_list, single_file
204
205 def fetch_remote_list(args, require_attribs = False, recursive = None):
206 remote_uris = []
207 remote_list = SortedDict()
208
209 if type(args) not in (list, tuple):
210 args = [args]
211
212 if recursive == None:
213 recursive = cfg.recursive
214
215 for arg in args:
216 uri = S3Uri(arg)
217 if not uri.type == 's3':
218 raise ParameterError("Expecting S3 URI instead of '%s'" % arg)
219 remote_uris.append(uri)
220
221 if recursive:
222 for uri in remote_uris:
223 objectlist = _get_filelist_remote(uri)
224 for key in objectlist:
225 remote_list[key] = objectlist[key]
226 else:
227 for uri in remote_uris:
228 uri_str = str(uri)
229 ## Wildcards used in remote URI?
230 ## If yes we'll need a bucket listing...
231 if uri_str.find('*') > -1 or uri_str.find('?') > -1:
232 first_wildcard = uri_str.find('*')
233 first_questionmark = uri_str.find('?')
234 if first_questionmark > -1 and first_questionmark < first_wildcard:
235 first_wildcard = first_questionmark
236 prefix = uri_str[:first_wildcard]
237 rest = uri_str[first_wildcard+1:]
238 ## Only request recursive listing if the 'rest' of the URI,
239 ## i.e. the part after first wildcard, contains '/'
240 need_recursion = rest.find('/') > -1
241 objectlist = _get_filelist_remote(S3Uri(prefix), recursive = need_recursion)
242 for key in objectlist:
243 ## Check whether the 'key' matches the requested wildcards
244 if glob.fnmatch.fnmatch(objectlist[key]['object_uri_str'], uri_str):
245 remote_list[key] = objectlist[key]
246 else:
247 ## No wildcards - simply append the given URI to the list
248 key = os.path.basename(uri.object())
249 if not key:
250 raise ParameterError(u"Expecting S3 URI with a filename or --recursive: %s" % uri.uri())
251 remote_item = {
252 'base_uri': uri,
253 'object_uri_str': unicode(uri),
254 'object_key': uri.object()
255 }
256 if require_attribs:
257 response = S3(cfg).object_info(uri)
258 remote_item.update({
259 'size': int(response['headers']['content-length']),
260 'md5': response['headers']['etag'].strip('"\''),
261 'timestamp' : Utils.dateRFC822toUnix(response['headers']['date'])
262 })
263 remote_list[key] = remote_item
264 return remote_list
265
266 def cmd_object_put(args):
267 cfg = Config()
268 s3 = S3(cfg)
269
270 if len(args) == 0:
271 raise ParameterError("Nothing to upload. Expecting a local file or directory and a S3 URI destination.")
272
273 destination_base = args.pop()
274 if S3Uri(destination_base).type != 's3':
275 raise ParameterError("Destination must be S3Uri. Got: %s" % destination_base)
276
277 if len(args) == 0:
278 raise ParameterError("Nothing to upload. Expecting a local file or directory.")
279
280 local_list, single_file_local = fetch_local_list(args)
281
282 local_list, exclude_list = _filelist_filter_exclude_include(local_list)
283
284 local_count = len(local_list)
285
286 info(u"Summary: %d local files to upload" % local_count)
287
288 if local_count > 0:
289 if not destination_base.endswith("/"):
290 if not single_file_local:
291 raise ParameterError("Destination S3 URI must end with '/' (ie must refer to a directory on the remote side).")
292 local_list[local_list.keys()[0]]['remote_uri'] = unicodise(destination_base)
293 else:
294 for key in local_list:
295 local_list[key]['remote_uri'] = unicodise(destination_base + key)
296
297 if cfg.dry_run:
298 for key in exclude_list:
299 output(u"exclude: %s" % unicodise(key))
300 for key in local_list:
301 output(u"upload: %s -> %s" % (local_list[key]['full_name_unicode'], local_list[key]['remote_uri']))
302
303 warning(u"Exitting now because of --dry-run")
304 return
305
306 seq = 0
307 for key in local_list:
308 seq += 1
309
310 uri_final = S3Uri(local_list[key]['remote_uri'])
311
312 extra_headers = {}
313 full_name_orig = local_list[key]['full_name']
314 full_name = full_name_orig
315 seq_label = "[%d of %d]" % (seq, local_count)
316 if Config().encrypt:
317 exitcode, full_name, extra_headers["x-amz-meta-s3tools-gpgenc"] = gpg_encrypt(full_name_orig)
318 try:
319 response = s3.object_put(full_name, uri_final, extra_headers, extra_label = seq_label)
320 except S3UploadError, e:
321 error(u"Upload of '%s' failed too many times. Skipping that file." % real_filename)
322 continue
323 except InvalidFileError, e:
324 warning(u"File can not be uploaded: %s" % e)
325 continue
326 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
327 if not Config().progress_meter:
328 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
329 (unicodise(full_name_orig), uri_final, response["size"], response["elapsed"],
330 speed_fmt[0], speed_fmt[1], seq_label))
331 if Config().acl_public:
332 output(u"Public URL of the object is: %s" %
333 (uri_final.public_url()))
334 if Config().encrypt and full_name != full_name_orig:
335 debug(u"Removing temporary encrypted file: %s" % unicodise(full_name))
336 os.remove(full_name)
337
338 def cmd_object_get(args):
339 cfg = Config()
340 s3 = S3(cfg)
341
342 ## Check arguments:
343 ## if not --recursive:
344 ## - first N arguments must be S3Uri
345 ## - if the last one is S3 make current dir the destination_base
346 ## - if the last one is a directory:
347 ## - take all 'basenames' of the remote objects and
348 ## make the destination name be 'destination_base'+'basename'
349 ## - if the last one is a file or not existing:
350 ## - if the number of sources (N, above) == 1 treat it
351 ## as a filename and save the object there.
352 ## - if there's more sources -> Error
353 ## if --recursive:
354 ## - first N arguments must be S3Uri
355 ## - for each Uri get a list of remote objects with that Uri as a prefix
356 ## - apply exclude/include rules
357 ## - each list item will have MD5sum, Timestamp and pointer to S3Uri
358 ## used as a prefix.
359 ## - the last arg may be a local directory - destination_base
360 ## - if the last one is S3 make current dir the destination_base
361 ## - if the last one doesn't exist check remote list:
362 ## - if there is only one item and its_prefix==its_name
363 ## download that item to the name given in last arg.
364 ## - if there are more remote items use the last arg as a destination_base
365 ## and try to create the directory (incl. all parents).
366 ##
367 ## In both cases we end up with a list mapping remote object names (keys) to local file names.
368
369 ## Each item will be a dict with the following attributes
370 # {'remote_uri', 'local_filename'}
371 download_list = []
372
373 if len(args) == 0:
374 raise ParameterError("Nothing to download. Expecting S3 URI.")
375
376 if S3Uri(args[-1]).type == 'file':
377 destination_base = args.pop()
378 else:
379 destination_base = "."
380
381 if len(args) == 0:
382 raise ParameterError("Nothing to download. Expecting S3 URI.")
383
384 remote_list = fetch_remote_list(args, require_attribs = False)
385 remote_list, exclude_list = _filelist_filter_exclude_include(remote_list)
386
387 remote_count = len(remote_list)
388
389 info(u"Summary: %d remote files to download" % remote_count)
390
391 if remote_count > 0:
392 if not os.path.isdir(destination_base) or destination_base == '-':
393 ## We were either given a file name (existing or not) or want STDOUT
394 if remote_count > 1:
395 raise ParameterError("Destination must be a directory when downloading multiple sources.")
396 remote_list[remote_list.keys()[0]]['local_filename'] = deunicodise(destination_base)
397 elif os.path.isdir(destination_base):
398 if destination_base[-1] != os.path.sep:
399 destination_base += os.path.sep
400 for key in remote_list:
401 remote_list[key]['local_filename'] = destination_base + key
402 else:
403 raise InternalError("WTF? Is it a dir or not? -- %s" % destination_base)
404
405 if cfg.dry_run:
406 for key in exclude_list:
407 output(u"exclude: %s" % unicodise(key))
408 for key in remote_list:
409 output(u"download: %s -> %s" % (remote_list[key]['object_uri_str'], remote_list[key]['local_filename']))
410
411 warning(u"Exitting now because of --dry-run")
412 return
413
414 seq = 0
415 for key in remote_list:
416 seq += 1
417 item = remote_list[key]
418 uri = S3Uri(item['object_uri_str'])
419 ## Encode / Decode destination with "replace" to make sure it's compatible with current encoding
420 destination = unicodise_safe(item['local_filename'])
421 seq_label = "[%d of %d]" % (seq, remote_count)
422
423 start_position = 0
424
425 if destination == "-":
426 ## stdout
427 dst_stream = sys.__stdout__
428 else:
429 ## File
430 try:
431 file_exists = os.path.exists(destination)
432 try:
433 dst_stream = open(destination, "ab")
434 except IOError, e:
435 if e.errno == errno.ENOENT:
436 basename = destination[:destination.rindex(os.path.sep)]
437 info(u"Creating directory: %s" % basename)
438 os.makedirs(basename)
439 dst_stream = open(destination, "ab")
440 else:
441 raise
442 if file_exists:
443 if Config().get_continue:
444 start_position = dst_stream.tell()
445 elif Config().force:
446 start_position = 0L
447 dst_stream.seek(0L)
448 dst_stream.truncate()
449 elif Config().skip_existing:
450 info(u"Skipping over existing file: %s" % (destination))
451 continue
452 else:
453 dst_stream.close()
454 raise ParameterError(u"File %s already exists. Use either of --force / --continue / --skip-existing or give it a new name." % destination)
455 except IOError, e:
456 error(u"Skipping %s: %s" % (destination, e.strerror))
457 continue
458 response = s3.object_get(uri, dst_stream, start_position = start_position, extra_label = seq_label)
459 if response["headers"].has_key("x-amz-meta-s3tools-gpgenc"):
460 gpg_decrypt(destination, response["headers"]["x-amz-meta-s3tools-gpgenc"])
461 response["size"] = os.stat(destination)[6]
462 if not Config().progress_meter and destination != "-":
463 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
464 output(u"File %s saved as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s)" %
465 (uri, destination, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1]))
466
467 def cmd_object_del(args):
468 while (len(args)):
469 uri_arg = args.pop(0)
470 uri = S3Uri(uri_arg)
471 if uri.type != "s3":
472 raise ParameterError("Expecting S3 URI instead of '%s'" % uri_arg)
473 if not uri.has_object():
474 if Config().recursive and not Config().force:
475 raise ParameterError("Please use --force to delete ALL contents of %s" % uri)
476 elif not Config().recursive:
477 raise ParameterError("Object name required, not only the bucket name")
478 subcmd_object_del_uri(uri)
479
480 def subcmd_object_del_uri(uri, recursive = None):
481 s3 = S3(Config())
482 if recursive is None:
483 recursive = cfg.recursive
484 uri_list = []
485 if recursive:
486 filelist = _get_filelist_remote(uri)
487 uri_base = 's3://' + uri.bucket() + "/"
488 for idx in filelist:
489 object = filelist[idx]
490 debug(u"Adding URI " + uri_base + object['object_key'])
491 uri_list.append(S3Uri(uri_base + object['object_key']))
492 else:
493 uri_list.append(uri)
494 for _uri in uri_list:
495 response = s3.object_delete(_uri)
496 output(u"Object %s deleted" % _uri)
497
498 def subcmd_cp_mv(args, process_fce, message):
499 src_uri = S3Uri(args.pop(0))
500 dst_uri = S3Uri(args.pop(0))
501
502 if len(args):
503 raise ParameterError("Too many parameters! Expected: %s" % commands['cp']['param'])
504
505 if src_uri.type != "s3" or dst_uri.type != "s3":
506 raise ParameterError("Parameters are not URIs! Expected: %s" % commands['cp']['param'])
507
508 if dst_uri.object() == "":
509 dst_uri = S3Uri(dst_uri.uri() + src_uri.object())
510
511 response = process_fce(src_uri, dst_uri)
512 output(message % { "src" : src_uri, "dst" : dst_uri})
513 if Config().acl_public:
514 output(u"Public URL is: %s" % dst_uri.public_url())
515
516 def cmd_cp(args):
517 s3 = S3(Config())
518 subcmd_cp_mv(args, s3.object_copy, "Object %(src)s copied to %(dst)s")
519
520 def cmd_mv(args):
521 s3 = S3(Config())
522 subcmd_cp_mv(args, s3.object_move, "Object %(src)s moved to %(dst)s")
523
524 def cmd_info(args):
525 s3 = S3(Config())
526
527 while (len(args)):
528 uri_arg = args.pop(0)
529 uri = S3Uri(uri_arg)
530 if uri.type != "s3" or not uri.has_bucket():
531 raise ParameterError("Expecting S3 URI instead of '%s'" % uri_arg)
532
533 try:
534 if uri.has_object():
535 info = s3.object_info(uri)
536 output(u"%s (object):" % uri.uri())
537 output(u" File size: %s" % info['headers']['content-length'])
538 output(u" Last mod: %s" % info['headers']['last-modified'])
539 output(u" MIME type: %s" % info['headers']['content-type'])
540 output(u" MD5 sum: %s" % info['headers']['etag'].strip('"'))
541 else:
542 info = s3.bucket_info(uri)
543 output(u"%s (bucket):" % uri.uri())
544 output(u" Location: %s" % info['bucket-location'])
545 acl = s3.get_acl(uri)
546 acl_list = acl.getGrantList()
547 for user in acl_list:
548 output(u" ACL: %s: %s" % (user, acl_list[user]))
549 if acl.isAnonRead():
550 output(u" URL: %s" % uri.public_url())
551 except S3Error, e:
552 if S3.codes.has_key(e.info["Code"]):
553 error(S3.codes[e.info["Code"]] % uri.bucket())
554 return
555 else:
556 raise
557
317558 def _get_filelist_local(local_uri):
318 output("Compiling list of local files...")
319 local_path = local_uri.path()
320 if os.path.isdir(local_path):
321 loc_base = os.path.join(local_path, "")
559 info(u"Compiling list of local files...")
560 if local_uri.isdir():
561 local_base = deunicodise(local_uri.basename())
562 local_path = deunicodise(local_uri.path())
322563 filelist = os.walk(local_path)
564 single_file = False
323565 else:
324 loc_base = "./"
325 filelist = [( '.', [], [local_path] )]
326 loc_base_len = len(loc_base)
327 loc_list = {}
566 local_base = ""
567 local_path = deunicodise(local_uri.dirname())
568 filelist = [( local_path, [], [deunicodise(local_uri.basename())] )]
569 single_file = True
570 loc_list = SortedDict()
328571 for root, dirs, files in filelist:
329 ## TODO: implement explicit exclude
572 rel_root = root.replace(local_path, local_base, 1)
330573 for f in files:
331574 full_name = os.path.join(root, f)
332575 if not os.path.isfile(full_name):
335578 ## Synchronize symlinks... one day
336579 ## for now skip over
337580 continue
338 file = full_name[loc_base_len:]
581 relative_file = unicodise(os.path.join(rel_root, f))
582 if relative_file.startswith('./'):
583 relative_file = relative_file[2:]
339584 sr = os.stat_result(os.lstat(full_name))
340 loc_list[file] = {
585 loc_list[relative_file] = {
586 'full_name_unicode' : unicodise(full_name),
341587 'full_name' : full_name,
342588 'size' : sr.st_size,
343589 'mtime' : sr.st_mtime,
344590 ## TODO: Possibly more to save here...
345591 }
346 return loc_list
347
348 def _get_filelist_remote(remote_uri):
349 output("Retrieving list of remote files...")
592 return loc_list, single_file
593
594 def _get_filelist_remote(remote_uri, recursive = True):
595 ## If remote_uri ends with '/' then all remote files will have
596 ## the remote_uri prefix removed in the relative path.
597 ## If, on the other hand, the remote_uri ends with something else
598 ## (probably alphanumeric symbol) we'll use the last path part
599 ## in the relative path.
600 ##
601 ## Complicated, eh? See an example:
602 ## _get_filelist_remote("s3://bckt/abc/def") may yield:
603 ## { 'def/file1.jpg' : {}, 'def/xyz/blah.txt' : {} }
604 ## _get_filelist_remote("s3://bckt/abc/def/") will yield:
605 ## { 'file1.jpg' : {}, 'xyz/blah.txt' : {} }
606 ## Furthermore a prefix-magic can restrict the return list:
607 ## _get_filelist_remote("s3://bckt/abc/def/x") yields:
608 ## { 'xyz/blah.txt' : {} }
609
610 info(u"Retrieving list of remote files for %s ..." % remote_uri)
350611
351612 s3 = S3(Config())
352 response = s3.bucket_list(remote_uri.bucket(), prefix = remote_uri.object())
353
354 rem_base = remote_uri.object()
613 response = s3.bucket_list(remote_uri.bucket(), prefix = remote_uri.object(), recursive = recursive)
614
615 rem_base_original = rem_base = remote_uri.object()
616 remote_uri_original = remote_uri
617 if rem_base != '' and rem_base[-1] != '/':
618 rem_base = rem_base[:rem_base.rfind('/')+1]
619 remote_uri = S3Uri("s3://%s/%s" % (remote_uri.bucket(), rem_base))
355620 rem_base_len = len(rem_base)
356 rem_list = {}
621 rem_list = SortedDict()
622 break_now = False
357623 for object in response['list']:
358 key = object['Key'][rem_base_len:]
624 if object['Key'] == rem_base_original and object['Key'][-1] != os.path.sep:
625 ## We asked for one file and we got that file :-)
626 key = os.path.basename(object['Key'])
627 object_uri_str = remote_uri_original.uri()
628 break_now = True
629 rem_list = {} ## Remove whatever has already been put to rem_list
630 else:
631 key = object['Key'][rem_base_len:] ## Beware - this may be '' if object['Key']==rem_base !!
632 object_uri_str = remote_uri.uri() + key
359633 rem_list[key] = {
360634 'size' : int(object['Size']),
361 # 'mtime' : dateS3toUnix(object['LastModified']), ## That's upload time, not our lastmod time :-(
635 'timestamp' : dateS3toUnix(object['LastModified']), ## Sadly it's upload time, not our lastmod time :-(
362636 'md5' : object['ETag'][1:-1],
363 'object_key' : object['Key']
637 'object_key' : object['Key'],
638 'object_uri_str' : object_uri_str,
639 'base_uri' : remote_uri,
364640 }
641 if break_now:
642 break
365643 return rem_list
366
367 def _compare_filelists(src_list, dst_list, src_is_local_and_dst_is_remote):
368 output("Verifying checksums...")
644
645 def _filelist_filter_exclude_include(src_list):
646 info(u"Applying --exclude/--include")
369647 cfg = Config()
370 exists_list = {}
371 exclude_list = {}
372 if cfg.debug_syncmatch:
373 logging.root.setLevel(logging.DEBUG)
648 exclude_list = SortedDict()
374649 for file in src_list.keys():
375 if not cfg.debug_syncmatch:
376 debug("CHECK: %s" % (os.sep + file))
650 debug(u"CHECK: %s" % file)
377651 excluded = False
378652 for r in cfg.exclude:
379 ## all paths start with '/' from the base dir
380 if r.search(os.sep + file):
381 ## Can't directly 'continue' to the outer loop
382 ## therefore this awkward excluded switch :-(
653 if r.search(file):
383654 excluded = True
384 if cfg.debug_syncmatch:
385 debug("EXCL: %s" % (os.sep + file))
386 debug("RULE: '%s'" % (cfg.debug_exclude[r]))
387 else:
388 info("%s: excluded" % file)
655 debug(u"EXCL-MATCH: '%s'" % (cfg.debug_exclude[r]))
389656 break
390657 if excluded:
391 exclude_list = src_list[file]
658 ## No need to check for --include if not excluded
659 for r in cfg.include:
660 if r.search(file):
661 excluded = False
662 debug(u"INCL-MATCH: '%s'" % (cfg.debug_include[r]))
663 break
664 if excluded:
665 ## Still excluded - ok, action it
666 debug(u"EXCLUDE: %s" % file)
667 exclude_list[file] = src_list[file]
392668 del(src_list[file])
393669 continue
394670 else:
395 debug("PASS: %s" % (os.sep + file))
671 debug(u"PASS: %s" % (file))
672 return src_list, exclude_list
673
674 def _compare_filelists(src_list, dst_list, src_is_local_and_dst_is_remote):
675 info(u"Verifying attributes...")
676 cfg = Config()
677 exists_list = SortedDict()
678 if cfg.debug_syncmatch:
679 logging.root.setLevel(logging.DEBUG)
680
681 for file in src_list.keys():
682 if not cfg.debug_syncmatch:
683 debug(u"CHECK: %s" % file)
396684 if dst_list.has_key(file):
685 ## Was --skip-existing requested?
686 if cfg.skip_existing:
687 debug(u"IGNR: %s (used --skip-existing)" % (file))
688 exists_list[file] = src_list[file]
689 del(src_list[file])
690 ## Remove from destination-list, all that is left there will be deleted
691 del(dst_list[file])
692 continue
693
694 attribs_match = True
397695 ## Check size first
398 if dst_list[file]['size'] == src_list[file]['size']:
399 #debug("%s same size: %s" % (file, dst_list[file]['size']))
696 if 'size' in cfg.sync_checks and dst_list[file]['size'] != src_list[file]['size']:
697 debug(u"XFER: %s (size mismatch: src=%s dst=%s)" % (file, src_list[file]['size'], dst_list[file]['size']))
698 attribs_match = False
699
700 if attribs_match and 'md5' in cfg.sync_checks:
400701 ## ... same size, check MD5
401702 if src_is_local_and_dst_is_remote:
402703 src_md5 = Utils.hash_file_md5(src_list[file]['full_name'])
404705 else:
405706 src_md5 = src_list[file]['md5']
406707 dst_md5 = Utils.hash_file_md5(dst_list[file]['full_name'])
407 if src_md5 == dst_md5:
408 #debug("%s md5 matches: %s" % (file, dst_md5))
409 ## Checksums are the same.
410 ## Remove from source-list, all that is left there will be transferred
411 debug("IGNR: %s (transfer not needed: MD5 OK, Size OK)" % file)
412 exists_list[file] = src_list[file]
413 del(src_list[file])
414 else:
415 debug("XFER: %s (md5 mismatch: src=%s dst=%s)" % (file, src_md5, dst_md5))
416 else:
417 debug("XFER: %s (size mismatch: src=%s dst=%s)" % (file, src_list[file]['size'], dst_list[file]['size']))
418
708 if src_md5 != dst_md5:
709 ## Checksums are different.
710 attribs_match = False
711 debug(u"XFER: %s (md5 mismatch: src=%s dst=%s)" % (file, src_md5, dst_md5))
712
713 if attribs_match:
714 ## Remove from source-list, all that is left there will be transferred
715 debug(u"IGNR: %s (transfer not needed)" % file)
716 exists_list[file] = src_list[file]
717 del(src_list[file])
718
419719 ## Remove from destination-list, all that is left there will be deleted
420 #debug("%s removed from destination list" % file)
421720 del(dst_list[file])
721
422722 if cfg.debug_syncmatch:
423 warning("Exiting because of --debug-syncmatch")
424 sys.exit(0)
425
426 return src_list, dst_list, exists_list, exclude_list
427
428 def cmd_sync_remote2local(src, dst):
723 warning(u"Exiting because of --debug-syncmatch")
724 sys.exit(1)
725
726 return src_list, dst_list, exists_list
727
728 def cmd_sync_remote2local(args):
429729 def _parse_attrs_header(attrs_header):
430730 attrs = {}
431731 for attr in attrs_header.split("/"):
435735
436736 s3 = S3(Config())
437737
438 src_uri = S3Uri(src)
439 dst_uri = S3Uri(dst)
440
441 rem_list = _get_filelist_remote(src_uri)
442 rem_count = len(rem_list)
443
444 loc_list = _get_filelist_local(dst_uri)
445 loc_count = len(loc_list)
446
447 output("Found %d remote files, %d local files" % (rem_count, loc_count))
448
449 _compare_filelists(rem_list, loc_list, False)
450
451 output("Summary: %d remote files to download, %d local files to delete" % (len(rem_list), len(loc_list)))
452
453 for file in loc_list:
738 destination_base = args[-1]
739 local_list, single_file_local = fetch_local_list(destination_base, recursive = True)
740 remote_list = fetch_remote_list(args[:-1], recursive = True, require_attribs = True)
741
742 local_count = len(local_list)
743 remote_count = len(remote_list)
744
745 info(u"Found %d remote files, %d local files" % (remote_count, local_count))
746
747 remote_list, exclude_list = _filelist_filter_exclude_include(remote_list)
748
749 remote_list, local_list, existing_list = _compare_filelists(remote_list, local_list, False)
750
751 local_count = len(local_list)
752 remote_count = len(remote_list)
753
754 info(u"Summary: %d remote files to download, %d local files to delete" % (remote_count, local_count))
755
756 if not os.path.isdir(destination_base):
757 ## We were either given a file name (existing or not) or want STDOUT
758 if remote_count > 1:
759 raise ParameterError("Destination must be a directory when downloading multiple sources.")
760 remote_list[remote_list.keys()[0]]['local_filename'] = deunicodise(destination_base)
761 else:
762 if destination_base[-1] != os.path.sep:
763 destination_base += os.path.sep
764 for key in remote_list:
765 remote_list[key]['local_filename'] = deunicodise(destination_base + key)
766
767 if cfg.dry_run:
768 for key in exclude_list:
769 output(u"exclude: %s" % unicodise(key))
454770 if cfg.delete_removed:
455 # os.unlink(file)
456 output("deleted '%s'" % file)
457 else:
458 output("not-deleted '%s'" % file)
771 for key in local_list:
772 output(u"delete: %s" % local_list[key]['full_name_unicode'])
773 for key in remote_list:
774 output(u"download: %s -> %s" % (remote_list[key]['object_uri_str'], remote_list[key]['local_filename']))
775
776 warning(u"Exitting now because of --dry-run")
777 return
778
779 if cfg.delete_removed:
780 for key in local_list:
781 os.unlink(local_list[key]['full_name'])
782 output(u"deleted: %s" % local_list[key]['full_name_unicode'])
459783
460784 total_size = 0
461 total_count = len(rem_list)
462785 total_elapsed = 0.0
463786 timestamp_start = time.time()
464787 seq = 0
465788 dir_cache = {}
466 src_base = src_uri.uri()
467 dst_base = dst_uri.path()
468 if not src_base[-1] == "/": src_base += "/"
469 file_list = rem_list.keys()
789 file_list = remote_list.keys()
470790 file_list.sort()
471791 for file in file_list:
472792 seq += 1
473 uri = S3Uri(src_base + file)
474 dst_file = dst_base + file
793 item = remote_list[file]
794 uri = S3Uri(item['object_uri_str'])
795 dst_file = item['local_filename']
796 seq_label = "[%d of %d]" % (seq, remote_count)
475797 try:
476798 dst_dir = os.path.dirname(dst_file)
477799 if not dir_cache.has_key(dst_dir):
478800 dir_cache[dst_dir] = Utils.mkdir_with_parents(dst_dir)
479801 if dir_cache[dst_dir] == False:
480 warning("%s: destination directory not writable: %s" % (file, dst_dir))
802 warning(u"%s: destination directory not writable: %s" % (file, dst_dir))
481803 continue
482804 try:
483805 open_flags = os.O_CREAT
484 if cfg.force:
485 open_flags |= os.O_TRUNC
486 else:
487 open_flags |= os.O_EXCL
488
489 debug("dst_file=%s" % dst_file)
806 open_flags |= os.O_TRUNC
807 # open_flags |= os.O_EXCL
808
809 debug(u"dst_file=%s" % unicodise(dst_file))
490810 # This will have failed should the file exist
491811 os.close(os.open(dst_file, open_flags))
492812 # Yeah I know there is a race condition here. Sadly I don't know how to open() in exclusive mode.
493813 dst_stream = open(dst_file, "wb")
494 response = s3.object_get_uri(uri, dst_stream)
814 response = s3.object_get(uri, dst_stream, extra_label = seq_label)
495815 dst_stream.close()
496816 if response['headers'].has_key('x-amz-meta-s3cmd-attrs') and cfg.preserve_attrs:
497817 attrs = _parse_attrs_header(response['headers']['x-amz-meta-s3cmd-attrs'])
506826 try: dst_stream.close()
507827 except: pass
508828 if e.errno == errno.EEXIST:
509 warning("%s exists - not overwriting" % (dst_file))
829 warning(u"%s exists - not overwriting" % (dst_file))
510830 continue
511831 if e.errno in (errno.EPERM, errno.EACCES):
512 warning("%s not writable: %s" % (dst_file, e.strerror))
832 warning(u"%s not writable: %s" % (dst_file, e.strerror))
513833 continue
514834 raise e
515835 except KeyboardInterrupt:
516836 try: dst_stream.close()
517837 except: pass
518 warning("Exiting after keyboard interrupt")
838 warning(u"Exiting after keyboard interrupt")
519839 return
520840 except Exception, e:
521841 try: dst_stream.close()
522842 except: pass
523 error("%s: %s" % (file, e))
843 error(u"%s: %s" % (file, e))
524844 continue
525845 # We have to keep repeating this call because
526846 # Python 2.4 doesn't support try/except/finally
528848 try: dst_stream.close()
529849 except: pass
530850 except S3DownloadError, e:
531 error("%s: download failed too many times. Skipping that file." % file)
851 error(u"%s: download failed too many times. Skipping that file." % file)
532852 continue
533853 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
534 output("File '%s' stored as %s (%d bytes in %0.1f seconds, %0.2f %sB/s) [%d of %d]" %
535 (uri, dst_file, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
536 seq, total_count))
854 if not Config().progress_meter:
855 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
856 (uri, unicodise(dst_file), response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
857 seq_label))
537858 total_size += response["size"]
538859
539860 total_elapsed = time.time() - timestamp_start
540861 speed_fmt = formatSize(total_size/total_elapsed, human_readable = True, floating_point = True)
541 output("Done. Downloaded %d bytes in %0.1f seconds, %0.2f %sB/s" %
542 (total_size, total_elapsed, speed_fmt[0], speed_fmt[1]))
543
544 def cmd_sync_local2remote(src, dst):
862
863 # Only print out the result if any work has been done or
864 # if the user asked for verbose output
865 outstr = "Done. Downloaded %d bytes in %0.1f seconds, %0.2f %sB/s" % (total_size, total_elapsed, speed_fmt[0], speed_fmt[1])
866 if total_size > 0:
867 output(outstr)
868 else:
869 info(outstr)
870
871 def cmd_sync_local2remote(args):
545872 def _build_attr_header(src):
873 import pwd, grp
546874 attrs = {}
875 src = deunicodise(src)
547876 st = os.stat_result(os.stat(src))
548877 for attr in cfg.preserve_attrs_list:
549878 if attr == 'uname':
552881 except KeyError:
553882 attr = "uid"
554883 val = st.st_uid
555 warning("%s: Owner username not known. Storing UID=%d instead." % (src, val))
884 warning(u"%s: Owner username not known. Storing UID=%d instead." % (unicodise(src), val))
556885 elif attr == 'gname':
557886 try:
558887 val = grp.getgrgid(st.st_gid).gr_name
559888 except KeyError:
560889 attr = "gid"
561890 val = st.st_gid
562 warning("%s: Owner groupname not known. Storing GID=%d instead." % (src, val))
891 warning(u"%s: Owner groupname not known. Storing GID=%d instead." % (unicodise(src), val))
563892 else:
564893 val = getattr(st, 'st_' + attr)
565894 attrs[attr] = val
570899 s3 = S3(cfg)
571900
572901 if cfg.encrypt:
573 error("S3cmd 'sync' doesn't support GPG encryption, sorry.")
574 error("Either use unconditional 's3cmd put --recursive'")
575 error("or disable encryption with --no-encryption parameter.")
902 error(u"S3cmd 'sync' doesn't yet support GPG encryption, sorry.")
903 error(u"Either use unconditional 's3cmd put --recursive'")
904 error(u"or disable encryption with --no-encrypt parameter.")
576905 sys.exit(1)
577906
578
579 src_uri = S3Uri(src)
580 dst_uri = S3Uri(dst)
581
582 loc_list = _get_filelist_local(src_uri)
583 loc_count = len(loc_list)
584
585 rem_list = _get_filelist_remote(dst_uri)
586 rem_count = len(rem_list)
587
588 output("Found %d local files, %d remote files" % (loc_count, rem_count))
589
590 _compare_filelists(loc_list, rem_list, True)
591
592 output("Summary: %d local files to upload, %d remote files to delete" % (len(loc_list), len(rem_list)))
593
594 for file in rem_list:
595 uri = S3Uri("s3://" + dst_uri.bucket()+"/"+rem_list[file]['object_key'])
907 destination_base = args[-1]
908 local_list, single_file_local = fetch_local_list(args[:-1], recursive = True)
909 remote_list = fetch_remote_list(destination_base, recursive = True, require_attribs = True)
910
911 local_count = len(local_list)
912 remote_count = len(remote_list)
913
914 info(u"Found %d local files, %d remote files" % (local_count, remote_count))
915
916 local_list, exclude_list = _filelist_filter_exclude_include(local_list)
917
918 if single_file_local and len(local_list) == 1 and len(remote_list) == 1:
919 ## Make remote_key same as local_key for comparison if we're dealing with only one file
920 remote_list_entry = remote_list[remote_list.keys()[0]]
921 # Flush remote_list, by the way
922 remote_list = { local_list.keys()[0] : remote_list_entry }
923
924 local_list, remote_list, existing_list = _compare_filelists(local_list, remote_list, True)
925
926 local_count = len(local_list)
927 remote_count = len(remote_list)
928
929 info(u"Summary: %d local files to upload, %d remote files to delete" % (local_count, remote_count))
930
931 if local_count > 0:
932 ## Populate 'remote_uri' only if we've got something to upload
933 if not destination_base.endswith("/"):
934 if not single_file_local:
935 raise ParameterError("Destination S3 URI must end with '/' (ie must refer to a directory on the remote side).")
936 local_list[local_list.keys()[0]]['remote_uri'] = unicodise(destination_base)
937 else:
938 for key in local_list:
939 local_list[key]['remote_uri'] = unicodise(destination_base + key)
940
941 if cfg.dry_run:
942 for key in exclude_list:
943 output(u"exclude: %s" % unicodise(key))
596944 if cfg.delete_removed:
597 response = s3.object_delete_uri(uri)
598 output("deleted '%s'" % uri)
599 else:
600 output("not-deleted '%s'" % uri)
945 for key in remote_list:
946 output(u"delete: %s" % remote_list[key]['object_uri_str'])
947 for key in local_list:
948 output(u"upload: %s -> %s" % (local_list[key]['full_name_unicode'], local_list[key]['remote_uri']))
949
950 warning(u"Exitting now because of --dry-run")
951 return
952
953 if cfg.delete_removed:
954 for key in remote_list:
955 uri = S3Uri(remote_list[key]['object_uri_str'])
956 s3.object_delete(uri)
957 output(u"deleted: '%s'" % uri)
601958
602959 total_size = 0
603 total_count = len(loc_list)
604960 total_elapsed = 0.0
605961 timestamp_start = time.time()
606962 seq = 0
607 dst_base = dst_uri.uri()
608 if not dst_base[-1] == "/": dst_base += "/"
609 file_list = loc_list.keys()
963 file_list = local_list.keys()
610964 file_list.sort()
611965 for file in file_list:
612966 seq += 1
613 src = loc_list[file]['full_name']
614 uri = S3Uri(dst_base + file)
967 item = local_list[file]
968 src = item['full_name']
969 uri = S3Uri(item['remote_uri'])
970 seq_label = "[%d of %d]" % (seq, local_count)
971 attr_header = None
615972 if cfg.preserve_attrs:
616973 attr_header = _build_attr_header(src)
617974 debug(attr_header)
618975 try:
619 response = s3.object_put_uri(src, uri, attr_header)
976 response = s3.object_put(src, uri, attr_header, extra_label = seq_label)
620977 except S3UploadError, e:
621 error("%s: upload failed too many times. Skipping that file." % src)
978 error(u"%s: upload failed too many times. Skipping that file." % item['full_name_unicode'])
622979 continue
623980 except InvalidFileError, e:
624 warning("File can not be uploaded: %s" % e)
981 warning(u"File can not be uploaded: %s" % e)
625982 continue
626983 speed_fmt = formatSize(response["speed"], human_readable = True, floating_point = True)
627 output("File '%s' stored as %s (%d bytes in %0.1f seconds, %0.2f %sB/s) [%d of %d]" %
628 (src, uri, response["size"], response["elapsed"], speed_fmt[0], speed_fmt[1],
629 seq, total_count))
984 if not cfg.progress_meter:
985 output(u"File '%s' stored as '%s' (%d bytes in %0.1f seconds, %0.2f %sB/s) %s" %
986 (item['full_name_unicode'], uri, response["size"], response["elapsed"],
987 speed_fmt[0], speed_fmt[1], seq_label))
630988 total_size += response["size"]
631989
632990 total_elapsed = time.time() - timestamp_start
633 speed_fmt = formatSize(total_size/total_elapsed, human_readable = True, floating_point = True)
634 output("Done. Uploaded %d bytes in %0.1f seconds, %0.2f %sB/s" %
635 (total_size, total_elapsed, speed_fmt[0], speed_fmt[1]))
991 total_speed = total_elapsed and total_size/total_elapsed or 0.0
992 speed_fmt = formatSize(total_speed, human_readable = True, floating_point = True)
993
994 # Only print out the result if any work has been done or
995 # if the user asked for verbose output
996 outstr = "Done. Uploaded %d bytes in %0.1f seconds, %0.2f %sB/s" % (total_size, total_elapsed, speed_fmt[0], speed_fmt[1])
997 if total_size > 0:
998 output(outstr)
999 else:
1000 info(outstr)
6361001
6371002 def cmd_sync(args):
638 src = args.pop(0)
639 dst = args.pop(0)
640 if (len(args)):
641 raise ParameterError("Too many parameters! Expected: %s" % commands['sync']['param'])
642
643 if S3Uri(src).type == "s3" and not src.endswith('/'):
644 src += "/"
645
646 if not dst.endswith('/'):
647 dst += "/"
648
649 if S3Uri(src).type == "file" and S3Uri(dst).type == "s3":
650 return cmd_sync_local2remote(src, dst)
651 if S3Uri(src).type == "s3" and S3Uri(dst).type == "file":
652 return cmd_sync_remote2local(src, dst)
653
1003 if (len(args) < 2):
1004 raise ParameterError("Too few parameters! Expected: %s" % commands['sync']['param'])
1005
1006 if S3Uri(args[0]).type == "file" and S3Uri(args[-1]).type == "s3":
1007 return cmd_sync_local2remote(args)
1008 if S3Uri(args[0]).type == "s3" and S3Uri(args[-1]).type == "file":
1009 return cmd_sync_remote2local(args)
1010 raise ParameterError("Invalid source/destination: '%s'" % "' '".join(args))
1011
1012 def cmd_setacl(args):
1013 s3 = S3(cfg)
1014
1015 set_to_acl = cfg.acl_public and "Public" or "Private"
1016
1017 remote_list = fetch_remote_list(args)
1018 remote_count = len(remote_list)
1019 seq = 0
1020 for key in remote_list:
1021 seq += 1
1022 seq_label = "[%d of %d]" % (seq, remote_count)
1023 uri = S3Uri(remote_list[key]['object_uri_str'])
1024 acl = s3.get_acl(uri)
1025 debug(u"acl: %s - %r" % (uri, acl.grantees))
1026 if cfg.acl_public:
1027 if acl.isAnonRead():
1028 info(u"%s: already Public, skipping %s" % (uri, seq_label))
1029 continue
1030 acl.grantAnonRead()
1031 else:
1032 if not acl.isAnonRead():
1033 info(u"%s: already Private, skipping %s" % (uri, seq_label))
1034 continue
1035 acl.revokeAnonRead()
1036 retsponse = s3.set_acl(uri, acl)
1037 if retsponse['status'] == 200:
1038 output(u"%s: ACL set to %s %s" % (uri, set_to_acl, seq_label))
1039
6541040 def resolve_list(lst, args):
6551041 retval = []
6561042 for item in lst:
6581044 return retval
6591045
6601046 def gpg_command(command, passphrase = ""):
661 p_in, p_out = os.popen4(command)
662 if command.count("--passphrase-fd"):
663 p_in.write(passphrase+"\n")
664 p_in.flush()
665 for line in p_out:
666 info(line.strip())
667 p_pid, p_exitcode = os.wait()
1047 debug("GPG command: " + " ".join(command))
1048 p = subprocess.Popen(command, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
1049 p_stdout, p_stderr = p.communicate(passphrase + "\n")
1050 debug("GPG output:")
1051 for line in p_stdout.split("\n"):
1052 debug("GPG: " + line)
1053 p_exitcode = p.wait()
6681054 return p_exitcode
6691055
6701056 def gpg_encrypt(filename):
6751061 "input_file" : filename,
6761062 "output_file" : tmp_filename,
6771063 }
678 info("Encrypting file %(input_file)s to %(output_file)s..." % args)
1064 info(u"Encrypting file %(input_file)s to %(output_file)s..." % args)
6791065 command = resolve_list(cfg.gpg_encrypt.split(" "), args)
6801066 code = gpg_command(command, cfg.gpg_passphrase)
6811067 return (code, tmp_filename, "gpg")
6881074 "input_file" : filename,
6891075 "output_file" : tmp_filename,
6901076 }
691 info("Decrypting file %(input_file)s to %(output_file)s..." % args)
1077 info(u"Decrypting file %(input_file)s to %(output_file)s..." % args)
6921078 command = resolve_list(cfg.gpg_decrypt.split(" "), args)
6931079 code = gpg_command(command, cfg.gpg_passphrase)
6941080 if code == 0 and in_place:
695 debug("Renaming %s to %s" % (tmp_filename, filename))
1081 debug(u"Renaming %s to %s" % (tmp_filename, filename))
6961082 os.unlink(filename)
6971083 os.rename(tmp_filename, filename)
6981084 tmp_filename = filename
7211107
7221108 try:
7231109 while 1:
724 output("\nEnter new values or accept defaults in brackets with Enter.")
725 output("Refer to user manual for detailed description of all options.")
1110 output(u"\nEnter new values or accept defaults in brackets with Enter.")
1111 output(u"Refer to user manual for detailed description of all options.")
7261112 for option in options:
7271113 prompt = option[1]
7281114 ## Option-specific handling
7431129 pass
7441130
7451131 if len(option) >= 3:
746 output("\n%s" % option[2])
1132 output(u"\n%s" % option[2])
7471133
7481134 val = raw_input(prompt + ": ")
7491135 if val != "":
7511137 # Turn 'Yes' into True, everything else into False
7521138 val = val.lower().startswith('y')
7531139 setattr(cfg, option[0], val)
754 output("\nNew settings:")
1140 output(u"\nNew settings:")
7551141 for option in options:
756 output(" %s: %s" % (option[1], getattr(cfg, option[0])))
1142 output(u" %s: %s" % (option[1], getattr(cfg, option[0])))
7571143 val = raw_input("\nTest access with supplied credentials? [Y/n] ")
7581144 if val.lower().startswith("y") or val == "":
7591145 try:
760 output("Please wait...")
1146 output(u"Please wait...")
7611147 S3(Config()).bucket_list("", "")
762 output("Success. Your access key and secret key worked fine :-)")
763
764 output("\nNow verifying that encryption works...")
1148 output(u"Success. Your access key and secret key worked fine :-)")
1149
1150 output(u"\nNow verifying that encryption works...")
7651151 if not getattr(cfg, "gpg_command") or not getattr(cfg, "gpg_passphrase"):
766 output("Not configured. Never mind.")
1152 output(u"Not configured. Never mind.")
7671153 else:
7681154 if not getattr(cfg, "gpg_command"):
7691155 raise Exception("Path to GPG program not set")
7891175 raise Exception("Encryption verification error.")
7901176
7911177 except Exception, e:
792 error("Test failed: %s" % (e))
1178 error(u"Test failed: %s" % (e))
7931179 val = raw_input("\nRetry configuration? [Y/n] ")
7941180 if val.lower().startswith("y") or val == "":
7951181 continue
8131199 os.umask(old_mask)
8141200 cfg.dump_config(f)
8151201 f.close()
816 output("Configuration saved to '%s'" % config_file)
1202 output(u"Configuration saved to '%s'" % config_file)
8171203
8181204 except (EOFError, KeyboardInterrupt):
819 output("\nConfiguration aborted. Changes were NOT saved.")
1205 output(u"\nConfiguration aborted. Changes were NOT saved.")
8201206 return
8211207
8221208 except IOError, e:
823 error("Writing config file failed: %s: %s" % (config_file, e.strerror))
1209 error(u"Writing config file failed: %s: %s" % (config_file, e.strerror))
8241210 sys.exit(1)
8251211
826 def process_exclude_from_file(exf, exclude_array):
827 exfi = open(exf, "rt")
828 for ex in exfi:
829 ex = ex.strip()
830 if re.match("^#", ex) or re.match("^\s*$", ex):
1212 def process_patterns_from_file(fname, patterns_list):
1213 try:
1214 fn = open(fname, "rt")
1215 except IOError, e:
1216 error(e)
1217 sys.exit(1)
1218 for pattern in fn:
1219 pattern = pattern.strip()
1220 if re.match("^#", pattern) or re.match("^\s*$", pattern):
8311221 continue
832 debug("adding rule: %s" % ex)
833 exclude_array.append(ex)
834
835 commands = {}
836 commands_list = [
1222 debug(u"%s: adding rule: %s" % (fname, pattern))
1223 patterns_list.append(pattern)
1224
1225 return patterns_list
1226
1227 def process_patterns(patterns_list, patterns_from, is_glob, option_txt = ""):
1228 """
1229 process_patterns(patterns, patterns_from, is_glob, option_txt = "")
1230 Process --exclude / --include GLOB and REGEXP patterns.
1231 'option_txt' is 'exclude' / 'include' / 'rexclude' / 'rinclude'
1232 Returns: patterns_compiled, patterns_text
1233 """
1234
1235 patterns_compiled = []
1236 patterns_textual = {}
1237
1238 if patterns_list is None:
1239 patterns_list = []
1240
1241 if patterns_from:
1242 ## Append patterns from glob_from
1243 for fname in patterns_from:
1244 debug(u"processing --%s-from %s" % (option_txt, fname))
1245 patterns_list = process_patterns_from_file(fname, patterns_list)
1246
1247 for pattern in patterns_list:
1248 debug(u"processing %s rule: %s" % (option_txt, patterns_list))
1249 if is_glob:
1250 pattern = glob.fnmatch.translate(pattern)
1251 r = re.compile(pattern)
1252 patterns_compiled.append(r)
1253 patterns_textual[r] = pattern
1254
1255 return patterns_compiled, patterns_textual
1256
1257 def get_commands_list():
1258 return [
8371259 {"cmd":"mb", "label":"Make bucket", "param":"s3://BUCKET", "func":cmd_bucket_create, "argc":1},
8381260 {"cmd":"rb", "label":"Remove bucket", "param":"s3://BUCKET", "func":cmd_bucket_delete, "argc":1},
8391261 {"cmd":"ls", "label":"List objects or buckets", "param":"[s3://BUCKET[/PREFIX]]", "func":cmd_ls, "argc":0},
8451267 {"cmd":"sync", "label":"Synchronize a directory tree to S3", "param":"LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR", "func":cmd_sync, "argc":2},
8461268 {"cmd":"du", "label":"Disk usage by buckets", "param":"[s3://BUCKET[/PREFIX]]", "func":cmd_du, "argc":0},
8471269 {"cmd":"info", "label":"Get various information about Buckets or Objects", "param":"s3://BUCKET[/OBJECT]", "func":cmd_info, "argc":1},
848 #{"cmd":"setacl", "label":"Modify Access control list for Bucket or Object", "param":"s3://BUCKET[/OBJECT]", "func":cmd_setacl, "argc":1},
1270 {"cmd":"cp", "label":"Copy object", "param":"s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]", "func":cmd_cp, "argc":2},
1271 {"cmd":"mv", "label":"Move object", "param":"s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]", "func":cmd_mv, "argc":2},
1272 {"cmd":"setacl", "label":"Modify Access control list for Bucket or Object", "param":"s3://BUCKET[/OBJECT]", "func":cmd_setacl, "argc":1},
1273 ## CloudFront commands
1274 {"cmd":"cflist", "label":"List CloudFront distribution points", "param":"", "func":CfCmd.info, "argc":0},
1275 {"cmd":"cfinfo", "label":"Display CloudFront distribution point parameters", "param":"[cf://DIST_ID]", "func":CfCmd.info, "argc":0},
1276 {"cmd":"cfcreate", "label":"Create CloudFront distribution point", "param":"s3://BUCKET", "func":CfCmd.create, "argc":1},
1277 {"cmd":"cfdelete", "label":"Delete CloudFront distribution point", "param":"cf://DIST_ID", "func":CfCmd.delete, "argc":1},
1278 {"cmd":"cfmodify", "label":"Change CloudFront distribution point parameters", "param":"cf://DIST_ID", "func":CfCmd.modify, "argc":1},
8491279 ]
8501280
851 def format_commands(progname):
1281 def format_commands(progname, commands_list):
8521282 help = "Commands:\n"
8531283 for cmd in commands_list:
8541284 help += " %s\n %s %s %s\n" % (cmd["label"], progname, cmd["cmd"], cmd["param"])
8771307 sys.stderr.write("ERROR: Python 2.4 or higher required, sorry.\n")
8781308 sys.exit(1)
8791309
1310 commands_list = get_commands_list()
1311 commands = {}
1312
8801313 ## Populate "commands" from "commands_list"
8811314 for cmd in commands_list:
8821315 if cmd.has_key("cmd"):
8861319 optparser = OptionParser(option_class=OptionMimeType, formatter=MyHelpFormatter())
8871320 #optparser.disable_interspersed_args()
8881321
1322 config_file = None
8891323 if os.getenv("HOME"):
890 optparser.set_defaults(config=os.getenv("HOME")+"/.s3cfg")
891
1324 config_file = os.path.join(os.getenv("HOME"), ".s3cfg")
1325 elif os.name == "nt" and os.getenv("USERPROFILE"):
1326 config_file = os.path.join(os.getenv("USERPROFILE"), "Application Data", "s3cmd.ini")
1327
1328 preferred_encoding = locale.getpreferredencoding() or "UTF-8"
1329
1330 optparser.set_defaults(encoding = preferred_encoding)
1331 optparser.set_defaults(config = config_file)
8921332 optparser.set_defaults(verbosity = default_verbosity)
8931333
8941334 optparser.add_option( "--configure", dest="run_configure", action="store_true", help="Invoke interactive (re)configuration tool.")
8951335 optparser.add_option("-c", "--config", dest="config", metavar="FILE", help="Config file name. Defaults to %default")
8961336 optparser.add_option( "--dump-config", dest="dump_config", action="store_true", help="Dump current configuration after parsing config files and command line options and exit.")
8971337
898 #optparser.add_option("-n", "--dry-run", dest="dry_run", action="store_true", help="Only show what should be uploaded or downloaded but don't actually do it. May still perform S3 requests to get bucket listings and other information though.")
1338 optparser.add_option("-n", "--dry-run", dest="dry_run", action="store_true", help="Only show what should be uploaded or downloaded but don't actually do it. May still perform S3 requests to get bucket listings and other information though (only for [sync] command)")
8991339
9001340 optparser.add_option("-e", "--encrypt", dest="encrypt", action="store_true", help="Encrypt files before uploading to S3.")
9011341 optparser.add_option( "--no-encrypt", dest="encrypt", action="store_false", help="Don't encrypt files.")
9021342 optparser.add_option("-f", "--force", dest="force", action="store_true", help="Force overwrite and other dangerous operations.")
1343 optparser.add_option( "--continue", dest="get_continue", action="store_true", help="Continue getting a partially downloaded file (only for [get] command).")
1344 optparser.add_option( "--skip-existing", dest="skip_existing", action="store_true", help="Skip over files that exist at the destination (only for [get] and [sync] commands).")
1345 optparser.add_option("-r", "--recursive", dest="recursive", action="store_true", help="Recursive upload, download or removal.")
9031346 optparser.add_option("-P", "--acl-public", dest="acl_public", action="store_true", help="Store objects with ACL allowing read for anyone.")
9041347 optparser.add_option( "--acl-private", dest="acl_public", action="store_false", help="Store objects with default ACL allowing access for you only.")
9051348 optparser.add_option( "--delete-removed", dest="delete_removed", action="store_true", help="Delete remote objects with no corresponding local file [sync]")
9101353 optparser.add_option( "--exclude-from", dest="exclude_from", action="append", metavar="FILE", help="Read --exclude GLOBs from FILE")
9111354 optparser.add_option( "--rexclude", dest="rexclude", action="append", metavar="REGEXP", help="Filenames and paths matching REGEXP (regular expression) will be excluded from sync")
9121355 optparser.add_option( "--rexclude-from", dest="rexclude_from", action="append", metavar="FILE", help="Read --rexclude REGEXPs from FILE")
1356 optparser.add_option( "--include", dest="include", action="append", metavar="GLOB", help="Filenames and paths matching GLOB will be included even if previously excluded by one of --(r)exclude(-from) patterns")
1357 optparser.add_option( "--include-from", dest="include_from", action="append", metavar="FILE", help="Read --include GLOBs from FILE")
1358 optparser.add_option( "--rinclude", dest="rinclude", action="append", metavar="REGEXP", help="Same as --include but uses REGEXP (regular expression) instead of GLOB")
1359 optparser.add_option( "--rinclude-from", dest="rinclude_from", action="append", metavar="FILE", help="Read --rinclude REGEXPs from FILE")
9131360 optparser.add_option( "--debug-syncmatch", "--debug-exclude", dest="debug_syncmatch", action="store_true", help="Output detailed information about remote vs. local filelist matching and --exclude processing and then exit")
9141361
9151362 optparser.add_option( "--bucket-location", dest="bucket_location", help="Datacentre to create bucket in. Either EU or US (default)")
9171364 optparser.add_option("-m", "--mime-type", dest="default_mime_type", type="mimetype", metavar="MIME/TYPE", help="Default MIME-type to be set for objects stored.")
9181365 optparser.add_option("-M", "--guess-mime-type", dest="guess_mime_type", action="store_true", help="Guess MIME-type of files by their extension. Falls back to default MIME-Type as specified by --mime-type option")
9191366
1367 optparser.add_option( "--encoding", dest="encoding", metavar="ENCODING", help="Override autodetected terminal and filesystem encoding (character set). Autodetected: %s" % preferred_encoding)
1368
9201369 optparser.add_option("-H", "--human-readable-sizes", dest="human_readable_sizes", action="store_true", help="Print sizes in human readable form.")
9211370
1371 optparser.add_option( "--progress", dest="progress_meter", action="store_true", help="Display progress meter (default on TTY).")
1372 optparser.add_option( "--no-progress", dest="progress_meter", action="store_false", help="Don't display progress meter (default on non-TTY).")
1373 optparser.add_option( "--enable", dest="cf_enable", action="store_true", help="Enable given CloudFront distribution (only for [cfmodify] command)")
1374 optparser.add_option( "--disable", dest="cf_enable", action="store_false", help="Enable given CloudFront distribution (only for [cfmodify] command)")
1375 optparser.add_option( "--cf-add-cname", dest="cf_cnames_add", action="append", metavar="CNAME", help="Add given CNAME to a CloudFront distribution (only for [cfcreate] and [cfmodify] commands)")
1376 optparser.add_option( "--cf-remove-cname", dest="cf_cnames_remove", action="append", metavar="CNAME", help="Remove given CNAME from a CloudFront distribution (only for [cfmodify] command)")
1377 optparser.add_option( "--cf-comment", dest="cf_comment", action="store", metavar="COMMENT", help="Set COMMENT for a given CloudFront distribution (only for [cfcreate] and [cfmodify] commands)")
9221378 optparser.add_option("-v", "--verbose", dest="verbosity", action="store_const", const=logging.INFO, help="Enable verbose output.")
9231379 optparser.add_option("-d", "--debug", dest="verbosity", action="store_const", const=logging.DEBUG, help="Enable debug output.")
9241380 optparser.add_option( "--version", dest="show_version", action="store_true", help="Show s3cmd version (%s) and exit." % (PkgInfo.version))
9281384 'Amazon S3 storage. It allows for making and removing '+
9291385 '"buckets" and uploading, downloading and removing '+
9301386 '"objects" from these buckets.')
931 optparser.epilog = format_commands(optparser.get_prog_name())
1387 optparser.epilog = format_commands(optparser.get_prog_name(), commands_list)
9321388 optparser.epilog += ("\nSee program homepage for more information at\n%s\n" % PkgInfo.url)
9331389
9341390 (options, args) = optparser.parse_args()
9371393 ## debugging/verbose output for config file parser on request
9381394 logging.basicConfig(level=options.verbosity,
9391395 format='%(levelname)s: %(message)s',
940 stream = _stderr)
1396 stream = sys.stderr)
9411397
9421398 if options.show_version:
943 output("s3cmd version %s" % PkgInfo.version)
1399 output(u"s3cmd version %s" % PkgInfo.version)
9441400 sys.exit(0)
9451401
9461402 ## Now finally parse the config file
9471403 if not options.config:
948 error("Can't find a config file. Please use --config option.")
1404 error(u"Can't find a config file. Please use --config option.")
9491405 sys.exit(1)
9501406
9511407 try:
9541410 if options.run_configure:
9551411 cfg = Config()
9561412 else:
957 error("%s: %s" % (options.config, e.strerror))
958 error("Configuration file not available.")
959 error("Consider using --configure parameter to create one.")
1413 error(u"%s: %s" % (options.config, e.strerror))
1414 error(u"Configuration file not available.")
1415 error(u"Consider using --configure parameter to create one.")
9601416 sys.exit(1)
9611417
9621418 ## And again some logging level adjustments
9651421 cfg.verbosity = options.verbosity
9661422 logging.root.setLevel(cfg.verbosity)
9671423
1424 ## Default to --progress on TTY devices, --no-progress elsewhere
1425 ## Can be overriden by actual --(no-)progress parameter
1426 cfg.update_option('progress_meter', sys.stdout.isatty())
1427
1428 ## Unsupported features on Win32 platform
1429 if os.name == "nt":
1430 if cfg.preserve_attrs:
1431 error(u"Option --preserve is not yet supported on MS Windows platform. Assuming --no-preserve.")
1432 cfg.preserve_attrs = False
1433 if cfg.progress_meter:
1434 error(u"Option --progress is not yet supported on MS Windows platform. Assuming --no-progress.")
1435 cfg.progress_meter = False
1436
9681437 ## Update Config with other parameters
9691438 for option in cfg.option_list():
9701439 try:
9711440 if getattr(options, option) != None:
972 debug("Updating %s -> %s" % (option, getattr(options, option)))
1441 debug(u"Updating Config.Config %s -> %s" % (option, getattr(options, option)))
9731442 cfg.update_option(option, getattr(options, option))
9741443 except AttributeError:
9751444 ## Some Config() options are not settable from command line
9761445 pass
977
978 ## Process GLOB (shell wildcard style) excludes
979 if options.exclude is None:
980 options.exclude = []
981
982 if options.exclude_from:
983 for exf in options.exclude_from:
984 debug("processing --exclude-from %s" % exf)
985 process_exclude_from_file(exf, options.exclude)
986
987 if options.exclude:
988 for ex in options.exclude:
989 debug("processing rule: %s" % ex)
990 exc = re.compile(glob.fnmatch.translate(ex))
991 cfg.exclude.append(exc)
992 if options.debug_syncmatch:
993 cfg.debug_exclude[exc] = ex
994
995 ## Process REGEXP style excludes
996 if options.rexclude is None:
997 options.rexclude = []
998
999 if options.rexclude_from:
1000 for exf in options.rexclude_from:
1001 debug("processing --rexclude-from %s" % exf)
1002 process_exclude_from_file(exf, options.rexclude)
1003
1004 if options.rexclude:
1005 for ex in options.rexclude:
1006 debug("processing rule: %s" % ex)
1007 exc = re.compile(ex)
1008 cfg.exclude.append(exc)
1009 if options.debug_syncmatch:
1010 cfg.debug_exclude[exc] = ex
1446
1447 ## Update CloudFront options if some were set
1448 for option in CfCmd.options.option_list():
1449 try:
1450 if getattr(options, option) != None:
1451 debug(u"Updating CloudFront.Cmd %s -> %s" % (option, getattr(options, option)))
1452 CfCmd.options.update_option(option, getattr(options, option))
1453 except AttributeError:
1454 ## Some CloudFront.Cmd.Options() options are not settable from command line
1455 pass
1456
1457 ## Set output and filesystem encoding for printing out filenames.
1458 sys.stdout = codecs.getwriter(cfg.encoding)(sys.stdout, "replace")
1459 sys.stderr = codecs.getwriter(cfg.encoding)(sys.stderr, "replace")
1460
1461 ## Process --exclude and --exclude-from
1462 patterns_list, patterns_textual = process_patterns(options.exclude, options.exclude_from, is_glob = True, option_txt = "exclude")
1463 cfg.exclude.extend(patterns_list)
1464 cfg.debug_exclude.update(patterns_textual)
1465
1466 ## Process --rexclude and --rexclude-from
1467 patterns_list, patterns_textual = process_patterns(options.rexclude, options.rexclude_from, is_glob = False, option_txt = "rexclude")
1468 cfg.exclude.extend(patterns_list)
1469 cfg.debug_exclude.update(patterns_textual)
1470
1471 ## Process --include and --include-from
1472 patterns_list, patterns_textual = process_patterns(options.include, options.include_from, is_glob = True, option_txt = "include")
1473 cfg.include.extend(patterns_list)
1474 cfg.debug_include.update(patterns_textual)
1475
1476 ## Process --rinclude and --rinclude-from
1477 patterns_list, patterns_textual = process_patterns(options.rinclude, options.rinclude_from, is_glob = False, option_txt = "rinclude")
1478 cfg.include.extend(patterns_list)
1479 cfg.debug_include.update(patterns_textual)
10111480
10121481 if cfg.encrypt and cfg.gpg_passphrase == "":
1013 error("Encryption requested but no passphrase set in config file.")
1014 error("Please re-run 's3cmd --configure' and supply it.")
1482 error(u"Encryption requested but no passphrase set in config file.")
1483 error(u"Please re-run 's3cmd --configure' and supply it.")
10151484 sys.exit(1)
10161485
10171486 if options.dump_config:
10231492 sys.exit(0)
10241493
10251494 if len(args) < 1:
1026 error("Missing command. Please run with --help for more information.")
1495 error(u"Missing command. Please run with --help for more information.")
10271496 sys.exit(1)
10281497
10291498 ## Unicodise all remaining arguments:
10311500
10321501 command = args.pop(0)
10331502 try:
1034 debug("Command: %s" % commands[command]["cmd"])
1503 debug(u"Command: %s" % commands[command]["cmd"])
10351504 ## We must do this lookup in extra step to
10361505 ## avoid catching all KeyError exceptions
10371506 ## from inner functions.
10381507 cmd_func = commands[command]["func"]
10391508 except KeyError, e:
1040 error("Invalid command: %s" % e)
1509 error(u"Invalid command: %s" % e)
10411510 sys.exit(1)
10421511
10431512 if len(args) < commands[command]["argc"]:
1044 error("Not enough paramters for command '%s'" % command)
1513 error(u"Not enough paramters for command '%s'" % command)
10451514 sys.exit(1)
10461515
10471516 try:
10481517 cmd_func(args)
10491518 except S3Error, e:
1050 error("S3 error: %s" % e)
1519 error(u"S3 error: %s" % e)
10511520 sys.exit(1)
10521521 except ParameterError, e:
1053 error("Parameter problem: %s" % e)
1522 error(u"Parameter problem: %s" % e)
10541523 sys.exit(1)
10551524
10561525 if __name__ == '__main__':
10651534 from S3 import Utils
10661535 from S3.Exceptions import *
10671536 from S3.Utils import unicodise
1537 from S3.Progress import Progress
1538 from S3.CloudFront import Cmd as CfCmd
10681539
10691540 main()
10701541 sys.exit(0)
1542
10711543 except SystemExit, e:
10721544 sys.exit(e.code)
1545
1546 except KeyboardInterrupt:
1547 sys.stderr.write("See ya!\n")
1548 sys.exit(1)
10731549
10741550 except Exception, e:
10751551 sys.stderr.write("""
10761552 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10771553 An unexpected error has occurred.
10781554 Please report the following lines to:
1079 s3tools-general@lists.sourceforge.net
1555 s3tools-bugs@lists.sourceforge.net
10801556 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10811557
10821558 """)
1083 sys.stderr.write("S3cmd: %s\n" % PkgInfo.version)
1084 sys.stderr.write("Python: %s\n" % sys.version.replace('\n', ' '))
1559 tb = traceback.format_exc(sys.exc_info())
1560 e_class = str(e.__class__)
1561 e_class = e_class[e_class.rfind(".")+1 : -2]
1562 sys.stderr.write(u"Problem: %s: %s\n" % (e_class, e))
1563 try:
1564 sys.stderr.write("S3cmd: %s\n" % PkgInfo.version)
1565 except NameError:
1566 sys.stderr.write("S3cmd: unknown version. Module import problem?\n")
1567 sys.stderr.write("Python: %s\n" % sys.version.replace('\n', ' '))
10851568 sys.stderr.write("\n")
1086 sys.stderr.write(traceback.format_exc(sys.exc_info())+"\n")
1569 sys.stderr.write(unicode(tb, errors="replace"))
1570 if type(e) == ImportError:
1571 sys.stderr.write("\n")
1572 sys.stderr.write("Your sys.path contains these entries:\n")
1573 for path in sys.path:
1574 sys.stderr.write(u"\t%s\n" % path)
1575 sys.stderr.write("Now the question is where has S3/S3.py been installed?\n")
10871576 sys.stderr.write("""
10881577 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10891578 An unexpected error has occurred.
10901579 Please report the above lines to:
1091 s3tools-general@lists.sourceforge.net
1580 s3tools-bugs@lists.sourceforge.net
10921581 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
10931582 """)
10941583 sys.exit(1)
+14
-0
s3cmd.1 less more
4141 \fBsync\fR \fIs3://BUCKET[/PREFIX] LOCAL_DIR\fR
4242 Restore a tree from S3 to local directory
4343 .TP
44 \fBcp\fR \fIs3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]\fR
45 \fBmv\fR \fIs3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]\fR
46 Make a copy of a file (\fIcp\fR) or move a file (\fImv\fR).
47 Destination can be in the same bucket with a different name
48 or in another bucket with the same or different name.
49 Adding \fI\-\-acl\-public\fR will make the destination object
50 publicly accessible (see below).
51 .TP
4452 \fBinfo\fR \fIs3://BUCKET[/OBJECT]\fR
4553 Get various information about a Bucket or Object
4654 .TP
108116 \fB\-f\fR, \fB\-\-force\fR
109117 Force overwrite and other dangerous operations.
110118 .TP
119 \fB\-\-continue\fR
120 Continue getting a partially downloaded file (only for \fIget\fR command). This comes handy once download of a large file, say an ISO image, from a S3 bucket fails and a partially downloaded file is left on the disk. Unfortunately \fIput\fR command doesn't support restarting of failed upload due to Amazon S3 limitation.
121 .TP
111122 \fB\-P\fR, \fB\-\-acl\-public\fR
112123 Store objects with permissions allowing read for anyone.
113124 .TP
133144 .\".TP
134145 .\"\fB\-u\fR, \fB\-\-show\-uri\fR
135146 .\"Show complete S3 URI in listings.
147 .TP
148 \fB\-\-progress\fR, \fB\-\-no\-progress\fR
149 Display or don't display progress meter. When running on TTY (e.g. console or xterm) the default is to display progress meter. If not on TTY (e.g. output is redirected somewhere or running from cron) the default is to not display progress meter.
136150 .TP
137151 \fB\-v\fR, \fB\-\-verbose\fR
138152 Enable verbose output.
+0
-3
setup.cfg less more
00 [sdist]
11 formats = gztar,zip
2
3 [install]
4 prefix = /usr