Package list feed2imap / 37fde67
Imported Upstream version 1.0 Lucas Nussbaum 10 years ago
32 changed file(s) with 8195 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
0 GNU GENERAL PUBLIC LICENSE
1 Version 2, June 1991
2
3 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
4 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
5 Everyone is permitted to copy and distribute verbatim copies
6 of this license document, but changing it is not allowed.
7
8 Preamble
9
10 The licenses for most software are designed to take away your
11 freedom to share and change it. By contrast, the GNU General Public
12 License is intended to guarantee your freedom to share and change free
13 software--to make sure the software is free for all its users. This
14 General Public License applies to most of the Free Software
15 Foundation's software and to any other program whose authors commit to
16 using it. (Some other Free Software Foundation software is covered by
17 the GNU Library General Public License instead.) You can apply it to
18 your programs, too.
19
20 When we speak of free software, we are referring to freedom, not
21 price. Our General Public Licenses are designed to make sure that you
22 have the freedom to distribute copies of free software (and charge for
23 this service if you wish), that you receive source code or can get it
24 if you want it, that you can change the software or use pieces of it
25 in new free programs; and that you know you can do these things.
26
27 To protect your rights, we need to make restrictions that forbid
28 anyone to deny you these rights or to ask you to surrender the rights.
29 These restrictions translate to certain responsibilities for you if you
30 distribute copies of the software, or if you modify it.
31
32 For example, if you distribute copies of such a program, whether
33 gratis or for a fee, you must give the recipients all the rights that
34 you have. You must make sure that they, too, receive or can get the
35 source code. And you must show them these terms so they know their
36 rights.
37
38 We protect your rights with two steps: (1) copyright the software, and
39 (2) offer you this license which gives you legal permission to copy,
40 distribute and/or modify the software.
41
42 Also, for each author's protection and ours, we want to make certain
43 that everyone understands that there is no warranty for this free
44 software. If the software is modified by someone else and passed on, we
45 want its recipients to know that what they have is not the original, so
46 that any problems introduced by others will not reflect on the original
47 authors' reputations.
48
49 Finally, any free program is threatened constantly by software
50 patents. We wish to avoid the danger that redistributors of a free
51 program will individually obtain patent licenses, in effect making the
52 program proprietary. To prevent this, we have made it clear that any
53 patent must be licensed for everyone's free use or not licensed at all.
54
55 The precise terms and conditions for copying, distribution and
56 modification follow.
57
58 GNU GENERAL PUBLIC LICENSE
59 TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
60
61 0. This License applies to any program or other work which contains
62 a notice placed by the copyright holder saying it may be distributed
63 under the terms of this General Public License. The "Program", below,
64 refers to any such program or work, and a "work based on the Program"
65 means either the Program or any derivative work under copyright law:
66 that is to say, a work containing the Program or a portion of it,
67 either verbatim or with modifications and/or translated into another
68 language. (Hereinafter, translation is included without limitation in
69 the term "modification".) Each licensee is addressed as "you".
70
71 Activities other than copying, distribution and modification are not
72 covered by this License; they are outside its scope. The act of
73 running the Program is not restricted, and the output from the Program
74 is covered only if its contents constitute a work based on the
75 Program (independent of having been made by running the Program).
76 Whether that is true depends on what the Program does.
77
78 1. You may copy and distribute verbatim copies of the Program's
79 source code as you receive it, in any medium, provided that you
80 conspicuously and appropriately publish on each copy an appropriate
81 copyright notice and disclaimer of warranty; keep intact all the
82 notices that refer to this License and to the absence of any warranty;
83 and give any other recipients of the Program a copy of this License
84 along with the Program.
85
86 You may charge a fee for the physical act of transferring a copy, and
87 you may at your option offer warranty protection in exchange for a fee.
88
89 2. You may modify your copy or copies of the Program or any portion
90 of it, thus forming a work based on the Program, and copy and
91 distribute such modifications or work under the terms of Section 1
92 above, provided that you also meet all of these conditions:
93
94 a) You must cause the modified files to carry prominent notices
95 stating that you changed the files and the date of any change.
96
97 b) You must cause any work that you distribute or publish, that in
98 whole or in part contains or is derived from the Program or any
99 part thereof, to be licensed as a whole at no charge to all third
100 parties under the terms of this License.
101
102 c) If the modified program normally reads commands interactively
103 when run, you must cause it, when started running for such
104 interactive use in the most ordinary way, to print or display an
105 announcement including an appropriate copyright notice and a
106 notice that there is no warranty (or else, saying that you provide
107 a warranty) and that users may redistribute the program under
108 these conditions, and telling the user how to view a copy of this
109 License. (Exception: if the Program itself is interactive but
110 does not normally print such an announcement, your work based on
111 the Program is not required to print an announcement.)
112
113 These requirements apply to the modified work as a whole. If
114 identifiable sections of that work are not derived from the Program,
115 and can be reasonably considered independent and separate works in
116 themselves, then this License, and its terms, do not apply to those
117 sections when you distribute them as separate works. But when you
118 distribute the same sections as part of a whole which is a work based
119 on the Program, the distribution of the whole must be on the terms of
120 this License, whose permissions for other licensees extend to the
121 entire whole, and thus to each and every part regardless of who wrote it.
122
123 Thus, it is not the intent of this section to claim rights or contest
124 your rights to work written entirely by you; rather, the intent is to
125 exercise the right to control the distribution of derivative or
126 collective works based on the Program.
127
128 In addition, mere aggregation of another work not based on the Program
129 with the Program (or with a work based on the Program) on a volume of
130 a storage or distribution medium does not bring the other work under
131 the scope of this License.
132
133 3. You may copy and distribute the Program (or a work based on it,
134 under Section 2) in object code or executable form under the terms of
135 Sections 1 and 2 above provided that you also do one of the following:
136
137 a) Accompany it with the complete corresponding machine-readable
138 source code, which must be distributed under the terms of Sections
139 1 and 2 above on a medium customarily used for software interchange; or,
140
141 b) Accompany it with a written offer, valid for at least three
142 years, to give any third party, for a charge no more than your
143 cost of physically performing source distribution, a complete
144 machine-readable copy of the corresponding source code, to be
145 distributed under the terms of Sections 1 and 2 above on a medium
146 customarily used for software interchange; or,
147
148 c) Accompany it with the information you received as to the offer
149 to distribute corresponding source code. (This alternative is
150 allowed only for noncommercial distribution and only if you
151 received the program in object code or executable form with such
152 an offer, in accord with Subsection b above.)
153
154 The source code for a work means the preferred form of the work for
155 making modifications to it. For an executable work, complete source
156 code means all the source code for all modules it contains, plus any
157 associated interface definition files, plus the scripts used to
158 control compilation and installation of the executable. However, as a
159 special exception, the source code distributed need not include
160 anything that is normally distributed (in either source or binary
161 form) with the major components (compiler, kernel, and so on) of the
162 operating system on which the executable runs, unless that component
163 itself accompanies the executable.
164
165 If distribution of executable or object code is made by offering
166 access to copy from a designated place, then offering equivalent
167 access to copy the source code from the same place counts as
168 distribution of the source code, even though third parties are not
169 compelled to copy the source along with the object code.
170
171 4. You may not copy, modify, sublicense, or distribute the Program
172 except as expressly provided under this License. Any attempt
173 otherwise to copy, modify, sublicense or distribute the Program is
174 void, and will automatically terminate your rights under this License.
175 However, parties who have received copies, or rights, from you under
176 this License will not have their licenses terminated so long as such
177 parties remain in full compliance.
178
179 5. You are not required to accept this License, since you have not
180 signed it. However, nothing else grants you permission to modify or
181 distribute the Program or its derivative works. These actions are
182 prohibited by law if you do not accept this License. Therefore, by
183 modifying or distributing the Program (or any work based on the
184 Program), you indicate your acceptance of this License to do so, and
185 all its terms and conditions for copying, distributing or modifying
186 the Program or works based on it.
187
188 6. Each time you redistribute the Program (or any work based on the
189 Program), the recipient automatically receives a license from the
190 original licensor to copy, distribute or modify the Program subject to
191 these terms and conditions. You may not impose any further
192 restrictions on the recipients' exercise of the rights granted herein.
193 You are not responsible for enforcing compliance by third parties to
194 this License.
195
196 7. If, as a consequence of a court judgment or allegation of patent
197 infringement or for any other reason (not limited to patent issues),
198 conditions are imposed on you (whether by court order, agreement or
199 otherwise) that contradict the conditions of this License, they do not
200 excuse you from the conditions of this License. If you cannot
201 distribute so as to satisfy simultaneously your obligations under this
202 License and any other pertinent obligations, then as a consequence you
203 may not distribute the Program at all. For example, if a patent
204 license would not permit royalty-free redistribution of the Program by
205 all those who receive copies directly or indirectly through you, then
206 the only way you could satisfy both it and this License would be to
207 refrain entirely from distribution of the Program.
208
209 If any portion of this section is held invalid or unenforceable under
210 any particular circumstance, the balance of the section is intended to
211 apply and the section as a whole is intended to apply in other
212 circumstances.
213
214 It is not the purpose of this section to induce you to infringe any
215 patents or other property right claims or to contest validity of any
216 such claims; this section has the sole purpose of protecting the
217 integrity of the free software distribution system, which is
218 implemented by public license practices. Many people have made
219 generous contributions to the wide range of software distributed
220 through that system in reliance on consistent application of that
221 system; it is up to the author/donor to decide if he or she is willing
222 to distribute software through any other system and a licensee cannot
223 impose that choice.
224
225 This section is intended to make thoroughly clear what is believed to
226 be a consequence of the rest of this License.
227
228 8. If the distribution and/or use of the Program is restricted in
229 certain countries either by patents or by copyrighted interfaces, the
230 original copyright holder who places the Program under this License
231 may add an explicit geographical distribution limitation excluding
232 those countries, so that distribution is permitted only in or among
233 countries not thus excluded. In such case, this License incorporates
234 the limitation as if written in the body of this License.
235
236 9. The Free Software Foundation may publish revised and/or new versions
237 of the General Public License from time to time. Such new versions will
238 be similar in spirit to the present version, but may differ in detail to
239 address new problems or concerns.
240
241 Each version is given a distinguishing version number. If the Program
242 specifies a version number of this License which applies to it and "any
243 later version", you have the option of following the terms and conditions
244 either of that version or of any later version published by the Free
245 Software Foundation. If the Program does not specify a version number of
246 this License, you may choose any version ever published by the Free Software
247 Foundation.
248
249 10. If you wish to incorporate parts of the Program into other free
250 programs whose distribution conditions are different, write to the author
251 to ask for permission. For software which is copyrighted by the Free
252 Software Foundation, write to the Free Software Foundation; we sometimes
253 make exceptions for this. Our decision will be guided by the two goals
254 of preserving the free status of all derivatives of our free software and
255 of promoting the sharing and reuse of software generally.
256
257 NO WARRANTY
258
259 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
260 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
261 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
262 PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
263 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
264 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
265 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
266 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
267 REPAIR OR CORRECTION.
268
269 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
270 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
271 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
272 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
273 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
274 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
275 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
276 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
277 POSSIBILITY OF SUCH DAMAGES.
278
279 END OF TERMS AND CONDITIONS
280
281 How to Apply These Terms to Your New Programs
282
283 If you develop a new program, and you want it to be of the greatest
284 possible use to the public, the best way to achieve this is to make it
285 free software which everyone can redistribute and change under these terms.
286
287 To do so, attach the following notices to the program. It is safest
288 to attach them to the start of each source file to most effectively
289 convey the exclusion of warranty; and each file should have at least
290 the "copyright" line and a pointer to where the full notice is found.
291
292 <one line to give the program's name and a brief idea of what it does.>
293 Copyright (C) <year> <name of author>
294
295 This program is free software; you can redistribute it and/or modify
296 it under the terms of the GNU General Public License as published by
297 the Free Software Foundation; either version 2 of the License, or
298 (at your option) any later version.
299
300 This program is distributed in the hope that it will be useful,
301 but WITHOUT ANY WARRANTY; without even the implied warranty of
302 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
303 GNU General Public License for more details.
304
305 You should have received a copy of the GNU General Public License
306 along with this program; if not, write to the Free Software
307 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
308
309
310 Also add information on how to contact you by electronic and paper mail.
311
312 If the program is interactive, make it output a short notice like this
313 when it starts in an interactive mode:
314
315 Gnomovision version 69, Copyright (C) year name of author
316 Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
317 This is free software, and you are welcome to redistribute it
318 under certain conditions; type `show c' for details.
319
320 The hypothetical commands `show w' and `show c' should show the appropriate
321 parts of the General Public License. Of course, the commands you use may
322 be called something other than `show w' and `show c'; they could even be
323 mouse-clicks or menu items--whatever suits your program.
324
325 You should also get your employer (if you work as a programmer) or your
326 school, if any, to sign a "copyright disclaimer" for the program, if
327 necessary. Here is a sample; alter the names:
328
329 Yoyodyne, Inc., hereby disclaims all copyright interest in the program
330 `Gnomovision' (which makes passes at compilers) written by James Hacker.
331
332 <signature of Ty Coon>, 1 April 1989
333 Ty Coon, President of Vice
334
335 This General Public License does not permit incorporating your program into
336 proprietary programs. If your program is a subroutine library, you may
337 consider it more useful to permit linking proprietary applications with the
338 library. If this is what you want to do, use the GNU Library General
339 Public License instead of this License.
0 Feed2Imap 1.0 (18/04/2010)
1 ==========================
2 * Removed patch from Haegar as it's no longer needed with the new
3 rubyimap.rb (see discussion in https://gna.org/bugs/?13977)
4 * Support writing to maildirs instead of through IMAP
5 * Use Message-ID instead of X-CacheIndex
6 * Do not use acme.com
7 * Update rubyimap.rb
8 * Provide a way to disable SSL certification verification when
9 connecting to IMAPS accounts
10
11 Feed2Imap 0.9.4 (27/07/2009)
12 ============================
13 * Warn (INFO level, so only displayed with feed2imap -v) if fetching a
14 feed takes more than 10s, as this might cause massive delays
15 in feed2imap run times.
16 * Fix encoding of email headers
17 * Only include images once when used several times in the same item
18 * New version of Net::Imap
19 * Use Message-Id instead of X-CacheIndex
20 * Fix MIME formatting when including images
21 * Require ruby-feedparser 0.7, better email formatting
22 * Made it possible to re-use substrings in targets
23 * Fix buffering problem with filters
24 * Added a patch from Haegar to fix problem with dovecot 1.2.1
25
26 Feed2Imap 0.9.3 (23/07/2008)
27 ============================
28 * Check the return code from external commands, and warn if != 0. Fixes
29 Gna bug #10516.
30 * Support for including images in the mails (see include-images config
31 option). Based on a patch by Pavel Avgustinov, and with help from
32 Arnt Gulbrandsen.
33 * Dropped rubymail_patch.rb
34 * Added option to wrap text output. Thanks to Maxime Petazzoni for
35 the patch.
36 * When updating an email, remove the Recent flag.
37 * You need to use ruby-feedparser 0.6 or greater with that release.
38
39 Feed2Imap 0.9.2 (28/10/2007)
40 ============================
41 * resynchronized rubyimap.rb with stdlib, and fixed send! -> send.
42 * upload items in reverse order, to upload the older first
43 Closes Gna bug #8986. Thanks go do Rial Juan for the patch.
44 * Don't allow more than 16 fetchers to run at the same time.
45 16 should be a reasonable default for everybody.
46 Closes Gna #9032.
47 * Reduce the default HTTP timeout to 30s.
48 * Don't update the cache if uploading items failed (should avoid
49 missing some items).
50 * Safer cache writing. Should avoid some cache corruption problems.
51 * Now exits when we receive an IMAP error, instead of trying to recover. We
52 shouldn't receive IMAP errors anyway. Closes Debian #405070.
53 * Fixed content-type-encoding in HTML emails. Reported by Arnt Gulbrandse.
54
55 Feed2Imap 0.9.1 (15/05/2007)
56 ============================
57 * Fixed bug with folder creation.
58
59 Feed2Imap 0.9 (15/05/2007)
60 ============================
61 * Folder creation moved to upload. This should make feed2imap run
62 slightly faster.
63 * Per-feed dumpdir option (helps debugging)
64 * Now uses Content-Transfer-Encoding: 8bit (thanks Arnt Gulbrandsen
65 <arnt@gulbrandsen.priv.no>)
66 * Now supports Snowscripts, using the 'execurl' and 'filter' config
67 keywords. For more information, see the example configuration file.
68 * Slightly better option parsing. Thanks to Paul van Tilburg for the
69 patch.
70 * A debug mode was added, and the normal mode was improved, so it is
71 no longer necessary to redirect feed2imap output to /dev/null:
72 transient errors are only reported after they have happened a
73 certain number of times (default 10).
74 * An ignore-hash option was added for feeds whose content change all
75 the time.
76
77 Feed2Imap 0.8 (28/06/2006)
78 ============================
79 * Uses the http_proxy environment variable to determine the proxy server
80 if available. (fixes gna bug #5820, all credits go to Boyd Adamson
81 <boyd-adamson@usa.net>)
82 * Fixes flocking on Solaris (fixes gna bug #5819). Again, all credits go to
83 Boyd Adamson <boyd-adamson@usa.net>.
84 * Rewrite of the "find updated and new items" code. It should work much better
85 now. Also, a debug-updated configuration variable was added to make it
86 easier to debug those issues.
87 * New always-new flag in the config file to consider all items as new (for
88 feeds where items are wrongly marked as updated, e.g mediawiki feeds).
89 See example configuration file for more information (fixes Debian bug
90 #366878).
91 * When disconnecting from the IMAP server, don't display an exception in
92 non-verbose mode if the "connection is reset by peer" (fixes Debian bug
93 #367282).
94 * Handling of exceptions in needMIME (fixes gna bug #5872).
95
96 Feed2Imap 0.7 (17/02/2006)
97 ============================
98 * Fixes the IMAPS disconnection problem (patch provided by Gael Utard
99 <gael.utard@laposte.net>) (fixes gna bug #2178).
100 * Fixes some issues regarding parallel fetching of feeds.
101 * Now displays the feed creator as sender of emails. (fixes gna bug #5043).
102 * Don't display the password in error messages (fixes debian bug #350370).
103 * Upload mail with the Item's time, not the upload's time (fixes debian
104 bug #350371).
105
106 Feed2Imap 0.6 (11/01/2006)
107 ============================
108 * Moved the RSS/Atom parser to a separate library (ruby-feedparser).
109 * Locks the Cache file to avoid concurrent instances of feed2imap.
110 * Issue a warning if the config file is world readable.
111 * Fixed a small bug in Atom feeds parsing.
112 * Fix another bug related to escaped HTML.
113 * Minor fixes.
114
115 Feed2Imap 0.5 (19/09/2005)
116 ============================
117 * Fixed a parser problem with items with multiple children in the description.
118 * Mails were upload with \n only, but \r\n are needed.
119 * Feed2Imap can now work without libopenssl.
120 * Fixed a bug in the HTML2Text converter with <a> tags without href.
121 * Reserved characters (eg @) can now be included in the login, password or
122 folder. You just need to escape them.
123 * Feed2Imap is now included in Debian (package name: feed2imap).
124 * Much better handling of feeds with escaped HTML (LinuxFR for example).
125
126 Feed2Imap 0.4 (25/07/2005)
127 ============================
128 * now available as a Debian package.
129 * added manpages for everything.
130 * added min-frequency and disable config options. Added doc in example config.
131 * You can now use WordPress's feed:http://something urls in feed2imaprc.
132 * Switched to a real SGML parser for the text version.
133 * Much better output for the text version of emails.
134 * New feed2imap-cleaner to remove old mails seen but not flagged
135 * Feed2Imap version number wasn't displayed in the User-Agent
136 * Better exception handling when parsing errors occur
137 * added feed2imap's RSS feed to the default feeds in the config file
138
139 Feed2Imap 0.3 (04/06/2005)
140 ============================
141 * New releases are now advertised using a RSS feed
142 * Cleaner way to manage duplicate IDs (#1773)
143 * Fixed a problem with pseudo-duplicate entries from Mediawiki
144 * Fixed a problem with updated items being seen as updated at each update.
145 * Fixed a problem when the disconnection from the IMAP server failed.
146 reported by Ludovic Gomez <ludogomez@chez.com>
147
148 Feed2Imap 0.2 (30/04/2005)
149 ============================
150 * Fixed a problem with feeds with strange caching bugs (old items going away
151 and coming back)
152 * The text version is now encoded in iso-8859-1 instead of utf-8.
153 * The subject is now MIME-encoded in utf-8. It works with mutt & evo.
154 * No longer overwrite mail flags (Read, Important,..) when updating an item.
155 * HTTP fetching is now multithreaded and is much faster (about 300%).
156 * Fetching over HTTPS works.
157 * HTTP fetching is fully unit-tested.
158
159 Feed2Imap 0.1 (02/04/2005)
160 ==========================
161 * first public release.
0 Feed2Imap
1 -------------
2 by Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 Currently, all the information is provided on
5
6 http://home.gna.org/feed2imap
7
8 Copyright (c) 2005-2010 Lucas Nussbaum <lucas@lucas-nussbaum.net>
9
10 This program is free software; you can redistribute it and/or modify
11 it under the terms of the GNU General Public License as published by
12 the Free Software Foundation; either version 2 of the License, or
13 (at your option) any later version.
14
15 This program is distributed in the hope that it will be useful,
16 but WITHOUT ANY WARRANTY; without even the implied warranty of
17 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
18 GNU General Public License for more details.
19
20 You should have received a copy of the GNU General Public License
21 along with this program; if not, write to the Free Software
22 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
0 require 'rake/testtask'
1 require 'rake/rdoctask'
2 require 'rake/packagetask'
3 require 'rake'
4 require 'find'
5
6 task :default => [:package]
7
8 PKG_NAME = 'feed2imap'
9 PKG_VERSION = '1.0'
10 PKG_FILES = [ 'ChangeLog', 'README', 'COPYING', 'setup.rb', 'Rakefile']
11 Find.find('bin/', 'lib/', 'test/', 'data/') do |f|
12 if FileTest.directory?(f) and f =~ /\.svn/
13 Find.prune
14 else
15 PKG_FILES << f
16 end
17 end
18 Rake::TestTask.new do |t|
19 t.libs << "libs/feed2imap"
20 t.libs << "test"
21 t.test_files = FileList['test/tc_*.rb']
22 end
23
24 Rake::RDocTask.new do |rd|
25 rd.main = 'README'
26 rd.rdoc_files.include('lib/*.rb', 'lib/feed2imap/*.rb')
27 rd.options << '--all'
28 rd.options << '--diagram'
29 rd.options << '--fileboxes'
30 rd.options << '--inline-source'
31 rd.options << '--line-numbers'
32 rd.rdoc_dir = 'rdoc'
33 end
34
35 Rake::PackageTask.new(PKG_NAME, PKG_VERSION) do |p|
36 p.need_tar = true
37 p.need_zip = true
38 p.package_files = PKG_FILES
39 end
40
41 # "Gem" part of the Rakefile
42 begin
43 require 'rake/gempackagetask'
44
45 spec = Gem::Specification.new do |s|
46 s.platform = Gem::Platform::RUBY
47 s.summary = "RSS/Atom feed aggregator"
48 s.name = PKG_NAME
49 s.version = PKG_VERSION
50 s.requirements << 'feedparser'
51 s.require_path = 'lib'
52 s.files = PKG_FILES
53 s.description = "RSS/Atom feed aggregator"
54 end
55
56 Rake::GemPackageTask.new(spec) do |pkg|
57 pkg.need_zip = true
58 pkg.need_tar = true
59 end
60 rescue LoadError
61 puts "Will not generate gem."
62 end
0 #!/usr/bin/ruby
1
2 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
3
4 require 'feed2imap/feed2imap'
5 require 'optparse'
6
7 verbose = false
8 version = false
9 cacherebuild = false
10 configf = ENV['HOME'] + '/.feed2imaprc'
11 progname = File::basename($PROGRAM_NAME)
12 opts = OptionParser::new do |opts|
13 opts.program_name = progname
14 opts.banner = "Usage: #{progname} [options]"
15 opts.separator ""
16 opts.separator "Options:"
17
18 opts.on("-v", "--verbose", "Verbose mode") do |v|
19 verbose = true
20 end
21
22 opts.on("-d", "--debug", "Debug mode") do |v|
23 verbose = :debug
24 end
25
26 opts.on("-V", "--version", "Display Feed2Imap version") do |v|
27 version = true
28 end
29 opts.on("-c", "--rebuild-cache", "Cache rebuilding run : will fetch everything and add to cache, without uploading to the IMAP server. Useful if your cache file was lost, and you don't want to re-read all the items.") do |c|
30 cacherebuild = true
31 end
32 opts.on("-f", "--config <file>", "Select alternate config file") do |f|
33 configf = f
34 end
35 end
36 begin
37 opts.parse!(ARGV)
38 rescue OptionParser::ParseError => pe
39 opts.warn pe
40 puts opts
41 exit 1
42 end
43
44 if version
45 puts "Feed2Imap v.#{F2I_VERSION}"
46 else
47 Feed2Imap::new(verbose, cacherebuild, configf)
48 end
0 #!/usr/bin/ruby
1
2 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
3
4 require 'feed2imap/feed2imap'
5 require 'optparse'
6
7 configf = ENV['HOME'] + '/.feed2imaprc'
8 dryrun = false
9
10 opts = OptionParser::new do |opts|
11 opts.banner = "Usage: feed2imap-cleaner [options]"
12 opts.separator ""
13 opts.separator "Options:"
14 opts.on("-d", "--dry-run", "Dont really remove messages") do |v|
15 dryrun = true
16 end
17 opts.on("-f", "--config <file>", "Select alternate config file") do |f|
18 configf = f
19 end
20 end
21 opts.parse!(ARGV)
22
23 config = nil
24 File::open(configf) { |f| config = F2IConfig::new(f) }
25 config.imap_accounts.each_value do |ac|
26 ac.connect
27 end
28 config.feeds.each do |f|
29 f.imapaccount.cleanup(f.folder, dryrun)
30 end
31
0 #!/usr/bin/ruby
1
2 =begin
3 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
4 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
5
6 This program is free software; you can redistribute it and/or modify
7 it under the terms of the GNU General Public License as published by
8 the Free Software Foundation; either version 2 of the License, or
9 (at your option) any later version.
10
11 This program is distributed in the hope that it will be useful,
12 but WITHOUT ANY WARRANTY; without even the implied warranty of
13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 GNU General Public License for more details.
15
16 You should have received a copy of the GNU General Public License
17 along with this program; if not, write to the Free Software
18 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
19 =end
20
21 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
22
23 require 'feed2imap/config'
24 require 'optparse'
25
26 configf = ENV['HOME'] + '/.feed2imaprc'
27 opts = OptionParser::new do |opts|
28 opts.banner = "Usage: ./dumpconfig.rb [options]"
29 opts.separator ""
30 opts.separator "Options:"
31 opts.on("-f", "--config <file>", "Select alternate config file") do |f|
32 configf = f
33 end
34 end
35 opts.parse!(ARGV)
36
37 if not File::exist?(configf)
38 puts "Configuration file #{configfile} not found."
39 exit(1)
40 end
41 File::open(configf) { |f| puts F2IConfig::new(f).to_s }
0 #!/usr/bin/ruby
1
2 =begin
3 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
4 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
5
6 This program is free software; you can redistribute it and/or modify
7 it under the terms of the GNU General Public License as published by
8 the Free Software Foundation; either version 2 of the License, or
9 (at your option) any later version.
10
11 This program is distributed in the hope that it will be useful,
12 but WITHOUT ANY WARRANTY; without even the implied warranty of
13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 GNU General Public License for more details.
15
16 You should have received a copy of the GNU General Public License
17 along with this program; if not, write to the Free Software
18 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
19 =end
20
21 $:.unshift File.join(File.dirname(__FILE__), '..', 'lib')
22
23 require 'rexml/document'
24 require 'yaml'
25
26 DEFAULTIMAPFOLDER = 'imap://login:password@imapserver/folder.folder2'
27
28 opml = ARGV[0]
29 doc = nil
30 doc = REXML::Document::new(IO.read(opml))
31 feeds = []
32 doc.root.each_element('//outline') do |e|
33 if u = e.attribute('xmlUrl') || e.attribute('htmlUrl')
34 # dirty liferea hack
35 next if u.value == 'vfolder'
36 # get title
37 t = e.attribute('text') || e.attribute('Title') || nil
38 if t.nil?
39 title = '*** FEED TITLE (must be unique) ***'
40 else
41 title = t.value
42 end
43 url = u.value
44 feeds.push({'name' => title, 'url' => url, 'target' => DEFAULTIMAPFOLDER})
45 end
46 end
47 YAML::dump({'feeds' => feeds}, $stdout)
0 # Global options:
1 # max-failures: maximum number of failures allowed before they are reported in
2 # normal mode (default 10). By default, failures are only visible in verbose
3 # mode. Most feeds tend to suffer from temporary failures.
4 # dumpdir: (for debugging purposes) directory where all fetched feeds will be
5 # dumped.
6 # debug-updated: (for debugging purposes) if true, display a lot of information
7 # about the "updated-items" algorithm.
8 # include-images: download images and include them in the mail? (true/false)
9 # default-email: default email address in the format foo@example.com
10 # disable-ssl-verification: disable SSL certification when connecting
11 # to IMAPS accounts (true/false)
12 #
13 # Per-feed options:
14 # name: name of the feed (must be unique)
15 # url: HTTP[S] address where the feed has to be fetched
16 # target: the IMAP URI where to put emails. Should start with imap:// for IMAP,
17 # imaps:// for IMAPS and maildir:// for a path to a local maildir.
18 # min-frequency: (in HOURS) is the minimum frequency with which this particular
19 # feed will be fetched
20 # disable: if set to something, the feed will be ignored
21 # include-images: download images and include them in the mail? (true/false)
22 # always-new: feed2imap tries to use a clever algorithm to determine whether
23 # an item is new or has been updated. It doesn't work well with some web apps
24 # like mediawiki. When this flag is enabled, all items which don't match
25 # exactly a previously downloaded item are considered as new items.
26 # ignore-hash: Some feeds change the content of their items all the time, so
27 # feed2imap detects that they have been updated at each run. When this flag
28 # is enabled, feed2imap ignores the content of an item when determining
29 # whether the item is already known.
30 # dumpdir: (for debugging purposes) directory where all fetched feeds will be
31 # dumped.
32 # Snownews/Liferea scripts support :
33 # execurl: Command to execute that will display the RSS/Atom feed on stdout
34 # filter: Command to execute which will receive the RSS/Atom feed on stdin,
35 # modify it, and output it on stdout.
36 # For more information: http://kiza.kcore.de/software/snownews/snowscripts/
37 #
38 #
39 # If your login contains an @ character, replace it with %40. Other reserved
40 # characters can be escaped in the same way (see man ascii to get their code)
41 feeds:
42 - name: feed2imap
43 url: http://home.gna.org/feed2imap/feed2imap.rss
44 target: imap://luser:password@imap.apinc.org/INBOX.Feeds.Feed2Imap
45 - name: lucas
46 url: http://www.lucas-nussbaum.net/blog/?feed=rss2
47 target: imap://luser:password@imap.apinc.org/INBOX.Feeds.Lucas
48 - name: JabberFrWiki
49 url: http://wiki.jabberfr.org/index.php?title=Special:Recentchanges&feed=rss
50 target: imaps://luser:password@imap.apinc.org/INBOX.Feeds.JabberFR
51 always-new: true
52 - name: LeMonde
53 execurl: "wget -q -O /dev/stdout http://www.lemonde.fr/rss/sequence/0,2-3208,1-0,0.xml"
54 filter: "/home/lucas/lemonde_getbody"
55 target: imap://luser:password@imap.apinc.org/INBOX.Feeds.LeMonde
56 # It is also possible to reuse the same string in the target parameter:
57 # target-refix: &target "imap://user:pass@host/rss."
58 # feeds:
59 # - name: test1
60 # target: [ *target, 'test1' ]
61 # ...
62 # - name: test2
63 # target: [ *target, 'test2' ]
64 # ...
0 .TH feed2imap\-cleaner 1 "Jul 25, 2005"
1 .SH NAME
2 feed2imap\-cleaner \- Removes old items from IMAP folders
3 .SH SYNOPSIS
4 \fBfeed2imap\-cleaner\fR [OPTIONS]
5 .SH DESCRIPTION
6 feed2imap\-cleaner deletes old items from IMAP folders specified in the configuration file. The actual query string used to determine whether an item is old is :
7 "SEEN NOT FLAGGED BEFORE (3 days ago)". Which means that an item WON'T be deleted if it satisfies one of the following conditions :
8 .TP 0.2i
9 \(bu
10 It isn't 3 days old ;
11 .TP 0.2i
12 \(bu
13 It hasn't been read yet ;
14 .TP 0.2i
15 \(bu
16 It is flagged (marked as Important, for example).
17 .TP
18 \fB\-d\fR, \fB\-\-dry\-run\fR
19 Don't remove anything, but show what would be removed if run without this option.
20 .TP
21 \fB\-f\fR, \fB\-\-config \fIfile\fB\fR
22 Use another config file (~/.feed2imaprc is the default).
23 .SH BUGS
24 Deletion criterias should probably be more configurable.
25 .SH "SEE ALSO"
26 Homepage :
27 http://home.gna.org/feed2imap/
28 .PP
29 \fBfeed2imaprc\fR(5),
30 \fBfeed2imap\fR(1)
31 .SH AUTHOR
32 Copyright (C) 2005 Lucas Nussbaum lucas@lucas\-nussbaum.net
33 .PP
34 This program is free software; you can redistribute it and/or modify
35 it under the terms of the GNU General Public License as published by the
36 Free Software Foundation; either version 2 of the License, or (at your
37 option) any later version.
38 .PP
39 This program is distributed in the hope that it will be useful, but
40 WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
41 or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
42 more details.
0 .TH feed2imap\-dumpconfig 1 "Jul 25, 2005"
1 .SH NAME
2 feed2imap\-dumpconfig \- Dump feed2imap config
3 .SH SYNOPSIS
4 \fBfeed2imap\-dumpconfig\fR [OPTIONS]
5 .SH DESCRIPTION
6 feed2imap\-dumpconfig dumps the content of your feed2imaprc to screen.
7 .TP
8 \fB\-f\fR, \fB\-\-config \fIfile\fB\fR
9 Use another config file (~/.feed2imaprc is the default).
10 .SH "SEE ALSO"
11 Homepage :
12 http://home.gna.org/feed2imap/
13 .PP
14 \fBfeed2imaprc\fR(5),
15 \fBfeed2imap\fR(1)
16 .SH AUTHOR
17 Copyright (C) 2005 Lucas Nussbaum lucas@lucas\-nussbaum.net
18 .PP
19 This program is free software; you can redistribute it and/or modify
20 it under the terms of the GNU General Public License as published by the
21 Free Software Foundation; either version 2 of the License, or (at your
22 option) any later version.
23 .PP
24 This program is distributed in the hope that it will be useful, but
25 WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
26 or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
27 more details.
0 .TH feed2imap\-opmlimport 1 "Jul 25, 2005"
1 .SH NAME
2 feed2imap\-opmlimport \- Convert an OPML subscription list to a feed2imap config file
3 .SH SYNOPSIS
4 \fBfeed2imap\-opmlimport\fR
5 .SH DESCRIPTION
6 feed2imap\-opmlimport reads an OPML subscription list on standard input and outputs a feed2imap configuration file on standard output. The resulting configuration file will require some tweaking.
7 .SH BUGS
8 Should probably accept parameters to be able to change default values.
9 .SH "SEE ALSO"
10 Homepage :
11 http://home.gna.org/feed2imap/
12 .PP
13 \fBfeed2imaprc\fR(5),
14 \fBfeed2imap\fR(1)
15 .SH AUTHOR
16 Copyright (C) 2005 Lucas Nussbaum lucas@lucas\-nussbaum.net
17 .PP
18 This program is free software; you can redistribute it and/or modify
19 it under the terms of the GNU General Public License as published by the
20 Free Software Foundation; either version 2 of the License, or (at your
21 option) any later version.
22 .PP
23 This program is distributed in the hope that it will be useful, but
24 WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
25 or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
26 more details.
0 .TH feed2imap 1 "Jul 25, 2005"
1 .SH NAME
2 feed2imap \- clever RSS/ATOM feed aggregator
3 .SH SYNOPSIS
4 \fBfeed2imap\fR [OPTIONS]
5 .SH DESCRIPTION
6 feed2imap is an RSS/Atom feed aggregator. After
7 Downloading feeds (over HTTP or HTTPS), it uploads them to a specified
8 folder of an IMAP mail server. The user can then access the feeds using
9 Mutt, Evolution, Mozilla Thunderbird or even a webmail.
10 .TP
11 \fB\-V\fR, \fB\-\-version\fR
12 Show version information.
13 .TP
14 \fB\-v\fR, \fB\-\-verbose\fR
15 Run in verbose mode.
16 .TP
17 \fB\-c\fR, \fB\-\-rebuild\-cache\fR
18 Rebuilds the cache. Fetches all items and mark them as already seen. Useful if you lose your .feed2imap.cache file.
19 .TP
20 \fB\-f\fR, \fB\-\-config \fIfile\fB\fR
21 Use another config file (~/.feed2imaprc is the default).
22 .SH "SEE ALSO"
23 Homepage :
24 http://home.gna.org/feed2imap/
25 .PP
26 \fBfeed2imaprc\fR(5),
27 \fBfeed2imap\-cleaner\fR(1),
28 \fBfeed2imap\-dumpconfig\fR(1),
29 \fBfeed2imap\-opmlimport\fR(1)
30 .SH AUTHOR
31 Copyright (C) 2005 Lucas Nussbaum lucas@lucas\-nussbaum.net
32 .PP
33 This program is free software; you can redistribute it and/or modify
34 it under the terms of the GNU General Public License as published by the
35 Free Software Foundation; either version 2 of the License, or (at your
36 option) any later version.
37 .PP
38 This program is distributed in the hope that it will be useful, but
39 WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
40 or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
41 more details.
0 .TH feed2imaprc 5 "Jul 25, 2005"
1 .SH NAME
2 feed2imaprc \- feed2imap configuration file
3 .SH SYNOPSIS
4 \fBfeed2imaprc\fR is feed2imap's configuration file. It is usually located in \fB~/.feed2imaprc\fR.
5 .SH EXAMPLE
6 See \fB/usr/share/doc/feed2imap/examples/feed2imaprc\fR.
7 .SH "RESERVED CHARACTERS"
8 Some characters are reserved in RFC2396 (URI). If you need to include a reserved character in the login/password part of your target URI, replace it with its hex code. For example, @ can be replaced by %40.
9 .SH BUGS
10 This manpage should probably give more details. However, the example configuration file is
11 very well documented.
12 .SH "SEE ALSO"
13 Homepage :
14 http://home.gna.org/feed2imap/
15 .PP
16 \fBfeed2imap\fR(1)
17 .SH AUTHOR
18 Copyright (C) 2005 Lucas Nussbaum lucas@lucas\-nussbaum.net
19 .PP
20 This program is free software; you can redistribute it and/or modify
21 it under the terms of the GNU General Public License as published by the
22 Free Software Foundation; either version 2 of the License, or (at your
23 option) any later version.
24 .PP
25 This program is distributed in the hope that it will be useful, but
26 WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
27 or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
28 more details.
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 # debug mode
20 $updateddebug = false
21
22 # This class manages a cache of items
23 # (items which have already been seen)
24
25 require 'digest/md5'
26
27 class ItemCache
28 def initialize(debug = false)
29 @channels = {}
30 @@cacheidx = 0
31 $updateddebug = debug
32 self
33 end
34
35 # Returns the really new items amongst items
36 def get_new_items(id, items, always_new = false, ignore_hash = false)
37 if $updateddebug
38 puts "======================================================="
39 puts "GET_NEW_ITEMS FOR #{id}... (#{Time::now})"
40 end
41 @channels[id] ||= CachedChannel::new
42 @channels[id].parsefailures = 0
43 return @channels[id].get_new_items(items, always_new, ignore_hash)
44 end
45
46 # Commit changes to the cache
47 def commit_cache(id)
48 @channels[id] ||= CachedChannel::new
49 @channels[id].commit
50 end
51
52 # Get the last time the cache was updated
53 def get_last_check(id)
54 @channels[id] ||= CachedChannel::new
55 @channels[id].lastcheck
56 end
57
58 # Get the last time the cache was updated
59 def set_last_check(id, time)
60 @channels[id] ||= CachedChannel::new
61 @channels[id].lastcheck = time
62 @channels[id].failures = 0
63 self
64 end
65
66 # Fetching failure.
67 # returns number of failures
68 def fetch_failed(id)
69 @channels[id].fetch_failed
70 end
71
72 # Parsing failure.
73 # returns number of failures
74 def parse_failed(id)
75 @channels[id].parse_failed
76 end
77
78 # Load the cache from an IO stream
79 def load(io)
80 begin
81 @@cacheidx, @channels = Marshal.load(io)
82 rescue
83 @channels = Marshal.load(io)
84 @@cacheidx = 0
85 end
86 end
87
88 # Save the cache to an IO stream
89 def save(io)
90 Marshal.dump([@@cacheidx, @channels], io)
91 end
92
93 # Return the number of channels in the cache
94 def nbchannels
95 @channels.length
96 end
97
98 # Return the number of items in the cache
99 def nbitems
100 nb = 0
101 @channels.each_value { |c|
102 nb += c.nbitems
103 }
104 nb
105 end
106
107 def ItemCache.getindex
108 i = @@cacheidx
109 @@cacheidx += 1
110 i
111 end
112 end
113
114 class CachedChannel
115 # Size of the cache for each feed
116 # 100 items should be enough for everybody, even quite busy feeds
117 CACHESIZE = 100
118
119 attr_accessor :lastcheck, :items, :failures, :parsefailures
120
121 def initialize
122 @lastcheck = Time::at(0)
123 @items = []
124 @itemstemp = [] # see below
125 @nbnewitems = 0
126 @failures = 0
127 @parsefailures = 0
128 end
129
130 # Let's explain @items and @itemstemp.
131 # @items contains the CachedItems serialized to the disk cache.
132 # The - quite complicated - get_new_items method fills in @itemstemp
133 # but leaves @items unchanged.
134 # Later, the commit() method replaces @items with @itemstemp and
135 # empties @itemstemp. This way, if something wrong happens during the
136 # upload to the IMAP server, items aren't lost.
137 # @nbnewitems is set by get_new_items, and is used to limit the number
138 # of (old) items serialized.
139
140 # Returns the really new items amongst items
141 def get_new_items(items, always_new = false, ignore_hash = false)
142 # save number of new items
143 @nbnewitems = items.length
144 # set items' cached version if not set yet
145 newitems = []
146 updateditems = []
147 @itemstemp = @items
148 items.each { |i| i.cacheditem ||= CachedItem::new(i) }
149 if $updateddebug
150 puts "-------Items downloaded before dups removal (#{items.length}) :----------"
151 items.each { |i| puts "#{i.cacheditem.to_s}" }
152 end
153 # remove dups
154 dups = true
155 while dups
156 dups = false
157 for i in 0...items.length do
158 for j in i+1...items.length do
159 if items[i].cacheditem == items[j].cacheditem
160 if $updateddebug
161 puts "## Removed duplicate #{items[j].cacheditem.to_s}"
162 end
163 items.delete_at(j)
164 dups = true
165 break
166 end
167 end
168 break if dups
169 end
170 end
171 # debug : dump interesting info to stdout.
172 if $updateddebug
173 puts "-------Items downloaded after dups removal (#{items.length}) :----------"
174 items.each { |i| puts "#{i.cacheditem.to_s}" }
175 puts "-------Items already there (#{@items.length}) :----------"
176 @items.each { |i| puts "#{i.to_s}" }
177 puts "Items always considered as new: #{always_new.to_s}"
178 puts "Items compared ignoring the hash: #{ignore_hash.to_s}"
179 end
180 items.each do |i|
181 found = false
182 # Try to find a perfect match
183 @items.each do |j|
184 # note that simple_compare only CachedItem, not RSSItem, so we have to use
185 # j.simple_compare(i) and not i.simple_compare(j)
186 if (i.cacheditem == j and not ignore_hash) or
187 (j.simple_compare(i) and ignore_hash)
188 i.cacheditem.index = j.index
189 found = true
190 # let's put j in front of itemstemp
191 @itemstemp.delete(j)
192 @itemstemp.unshift(j)
193 break
194 end
195 end
196 next if found
197 if not always_new
198 # Try to find an updated item
199 @items.each do |j|
200 # Do we need a better heuristic ?
201 if j.is_ancestor_of(i)
202 i.cacheditem.index = j.index
203 i.cacheditem.updated = true
204 updateditems.push(i)
205 found = true
206 # let's put j in front of itemstemp
207 @itemstemp.delete(j)
208 @itemstemp.unshift(i.cacheditem)
209 break
210 end
211 end
212 end
213 next if found
214 # add as new
215 i.cacheditem.create_index
216 newitems.push(i)
217 # add i.cacheditem to @itemstemp
218 @itemstemp.unshift(i.cacheditem)
219 end
220 if $updateddebug
221 puts "-------New items :----------"
222 newitems.each { |i| puts "#{i.cacheditem.to_s}" }
223 puts "-------Updated items :----------"
224 updateditems.each { |i| puts "#{i.cacheditem.to_s}" }
225 end
226 return [newitems, updateditems]
227 end
228
229 def commit
230 # too old items must be dropped
231 n = @nbnewitems > CACHESIZE ? @nbnewitems : CACHESIZE
232 @items = @itemstemp[0..n]
233 if $updateddebug
234 puts "Committing: new items: #{@nbnewitems} / items kept: #{@items.length}"
235 end
236 @itemstemp = []
237 self
238 end
239
240 # returns the number of items
241 def nbitems
242 @items.length
243 end
244
245 def parse_failed
246 @parsefailures = 0 if @parsefailures.nil?
247 @parsefailures += 1
248 return @parsefailures
249 end
250
251 def fetch_failed
252 @failures = 0 if @failures.nil?
253 @failures += 1
254 return @failures
255 end
256 end
257
258 # This class is the only thing kept in the cache
259 class CachedItem
260 attr_reader :title, :link, :creator, :date, :hash
261 attr_accessor :index
262 attr_accessor :updated
263
264 def initialize(item)
265 @title = item.title
266 @link = item.link
267 @date = item.date
268 @creator = item.creator
269 if item.content.nil?
270 @hash = nil
271 else
272 @hash = Digest::MD5.hexdigest(item.content.to_s)
273 end
274 end
275
276 def ==(other)
277 if $updateddebug
278 puts "Comparing #{self.to_s} and #{other.to_s}:"
279 puts "Title: #{@title == other.title}"
280 puts "Link: #{@link == other.link}"
281 puts "Creator: #{@creator == other.creator}"
282 puts "Date: #{@date == other.date}"
283 puts "Hash: #{@hash == other.hash}"
284 end
285 @title == other.title and @link == other.link and
286 (@creator.nil? or other.creator.nil? or @creator == other.creator) and
287 (@date.nil? or other.date.nil? or @date == other.date) and @hash == other.hash
288 end
289
290 def simple_compare(other)
291 @title == other.title and @link == other.link and
292 (@creator.nil? or other.creator.nil? or @creator == other.creator)
293 end
294
295 def create_index
296 @index = ItemCache.getindex
297 end
298
299 def is_ancestor_of(other)
300 (@link and other.link and @link == other.link) and
301 ((@creator and other.creator and @creator == other.creator) or (@creator.nil?))
302 end
303
304 def to_s
305 "\"#{@title}\" #{@creator}/#{@date} #{@link} #{@hash}"
306 end
307 end
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 require 'yaml'
20 require 'uri'
21 require 'feed2imap/imap'
22 require 'feed2imap/maildir'
23 require 'etc'
24 require 'socket'
25
26 # Default cache file
27 DEFCACHE = ENV['HOME'] + '/.feed2imap.cache'
28
29 # Hostname and login name of the current user
30 HOSTNAME = Socket.gethostname
31 LOGNAME = Etc.getlogin
32
33 # Feed2imap configuration
34 class F2IConfig
35 attr_reader :imap_accounts, :cache, :feeds, :dumpdir, :updateddebug, :max_failures, :include_images, :default_email, :hostname
36
37 # Load the configuration from the IO stream
38 # TODO should do some sanity check on the data read.
39 def initialize(io)
40 @conf = YAML::load(io)
41 @cache = @conf['cache'] || DEFCACHE
42 @dumpdir = @conf['dumpdir'] || nil
43 @conf['feeds'] ||= []
44 @feeds = []
45 @max_failures = (@conf['max-failures'] || 10).to_i
46 @updateddebug = (@conf['debug-updated'] and @conf['debug-updated'] != 'false')
47 @include_images = (@conf['include-images'] and @conf['include-images'] != 'false')
48 @default_email = (@conf['default-email'] || "#{LOGNAME}@#{HOSTNAME}")
49 ImapAccount.no_ssl_verify = (@conf['disable-ssl-verification'] and @conf['disable-ssl-verification'] != 'false')
50 @hostname = HOSTNAME # FIXME: should this be configurable as well?
51 @imap_accounts = ImapAccounts::new
52 maildir_account = MaildirAccount::new
53 @conf['feeds'].each do |f|
54 if f['disable'].nil?
55 uri = URI::parse(f['target'].to_s)
56 path = URI::unescape(uri.path)
57 path = path[1..-1] if path[0,1] == '/'
58 if uri.scheme == 'maildir'
59 @feeds.push(ConfigFeed::new(f, maildir_account, path, self))
60 else
61 @feeds.push(ConfigFeed::new(f, @imap_accounts.add_account(uri), path, self))
62 end
63 end
64 end
65 end
66
67 def to_s
68 s = "Your Feed2Imap config :\n"
69 s += "=======================\n"
70 s += "Cache file: #{@cache}\n\n"
71 s += "Imap accounts I'll have to connect to :\n"
72 s += "---------------------------------------\n"
73 @imap_accounts.each_value { |i| s += i.to_s + "\n" }
74 s += "\nFeeds :\n"
75 s += "-------\n"
76 i = 1
77 @feeds.each do |f|
78 s += "#{i}. #{f.name}\n"
79 s += " URL: #{f.url}\n"
80 s += " IMAP Account: #{f.imapaccount}\n"
81 s += " Folder: #{f.folder}\n"
82
83 if not f.wrapto
84 s += " Not wrapped.\n"
85 end
86
87 s += "\n"
88 i += 1
89 end
90 s
91 end
92 end
93
94 # A configured feed. simple data container.
95 class ConfigFeed
96 attr_reader :name, :url, :imapaccount, :folder, :always_new, :execurl, :filter, :ignore_hash, :dumpdir, :wrapto, :include_images
97 attr_accessor :body
98
99 def initialize(f, imapaccount, folder, f2iconfig)
100 @name = f['name']
101 @url = f['url']
102 @url.sub!(/^feed:/, '') if @url =~ /^feed:/
103 @imapaccount, @folder = imapaccount, folder
104 @freq = f['min-frequency']
105 @always_new = (f['always-new'] and f['always-new'] != 'false')
106 @execurl = f['execurl']
107 @filter = f['filter']
108 @ignore_hash = f['ignore-hash'] || false
109 @freq = @freq.to_i if @freq
110 @dumpdir = f['dumpdir'] || nil
111 @wrapto = if f['wrapto'] == nil then 72 else f['wrapto'].to_i end
112 @include_images = f2iconfig.include_images
113 if f['include-images']
114 @include_images = (f['include-images'] != 'false')
115 end
116 end
117
118 def needfetch(lastcheck)
119 return true if @freq.nil?
120 return (lastcheck + @freq * 3600) < Time::now
121 end
122 end
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 # Feed2Imap version
20 F2I_VERSION = '1.0'
21 F2I_WARNFETCHTIME = 10
22
23 require 'feed2imap/config'
24 require 'feed2imap/cache'
25 require 'feed2imap/httpfetcher'
26 require 'logger'
27 require 'thread'
28 require 'feedparser'
29 require 'feed2imap/itemtomail'
30 require 'open3'
31
32 class Feed2Imap
33 def Feed2Imap.version
34 return F2I_VERSION
35 end
36
37 def initialize(verbose, cacherebuild, configfile)
38 @logger = Logger::new(STDOUT)
39 if verbose == :debug
40 @logger.level = Logger::DEBUG
41 require 'pp'
42 elsif verbose == true
43 @logger.level = Logger::INFO
44 else
45 @logger.level = Logger::WARN
46 end
47 @logger.info("Feed2Imap V.#{F2I_VERSION} started")
48 # reading config
49 @logger.info('Reading configuration file ...')
50 if not File::exist?(configfile)
51 @logger.fatal("Configuration file #{configfile} not found.")
52 exit(1)
53 end
54 if (File::stat(configfile).mode & 044) != 0
55 @logger.warn("Configuration file is readable by other users. It " +
56 "probably contains your password.")
57 end
58 begin
59 File::open(configfile) {
60 |f| @config = F2IConfig::new(f)
61 }
62 rescue
63 @logger.fatal("Error while reading configuration file, exiting: #{$!}")
64 exit(1)
65 end
66 if @logger.level == Logger::DEBUG
67 @logger.debug("Configuration read:")
68 pp(@config)
69 end
70
71 # init cache
72 @logger.info('Initializing cache ...')
73 @cache = ItemCache::new(@config.updateddebug)
74 if not File::exist?(@config.cache + '.lock')
75 f = File::new(@config.cache + '.lock', 'w')
76 f.close
77 end
78 if File::new(@config.cache + '.lock', 'w').flock(File::LOCK_EX | File::LOCK_NB) == false
79 @logger.fatal("Another instance of feed2imap is already locking the cache file")
80 exit(1)
81 end
82 if not File::exist?(@config.cache)
83 @logger.warn("Cache file #{@config.cache} not found, using a new one")
84 else
85 File::open(@config.cache) do |f|
86 @cache.load(f)
87 end
88 end
89
90 # connecting all IMAP accounts
91 @logger.info('Connecting to IMAP accounts ...')
92 @config.imap_accounts.each_value do |ac|
93 begin
94 ac.connect
95 rescue
96 @logger.fatal("Error while connecting to #{ac}, exiting: #{$!}")
97 exit(1)
98 end
99 end
100
101 # for each feed, fetch, upload to IMAP and cache
102 @logger.info("Fetching and filtering feeds ...")
103 ths = []
104 mutex = Mutex::new
105 sparefetchers = 16 # max number of fetchers running at the same time.
106 sparefetchers_mutex = Mutex::new
107 sparefetchers_cond = ConditionVariable::new
108 @config.feeds.each do |f|
109 ths << Thread::new(f) do |feed|
110 begin
111 mutex.lock
112 lastcheck = @cache.get_last_check(feed.name)
113 if feed.needfetch(lastcheck)
114 mutex.unlock
115 sparefetchers_mutex.synchronize do
116 while sparefetchers <= 0
117 sparefetchers_cond.wait(sparefetchers_mutex)
118 end
119 sparefetchers -= 1
120 end
121 fetch_start = Time::now
122 if feed.url
123 s = HTTPFetcher::fetch(feed.url, @cache.get_last_check(feed.name))
124 elsif feed.execurl
125 # avoid running more than one command at the same time.
126 # We need it because the called command might not be
127 # thread-safe, and we need to get the right exitcode
128 mutex.lock
129 s = %x{#{feed.execurl}}
130 if $?.exitstatus != 0
131 @logger.warn("Command for #{feed.name} exited with status #{$?.exitstatus} !")
132 end
133 mutex.unlock
134 else
135 @logger.warn("No way to fetch feed #{feed.name} !")
136 end
137 if feed.filter and s != nil
138 # avoid running more than one command at the same time.
139 # We need it because the called command might not be
140 # thread-safe, and we need to get the right exitcode.
141 mutex.lock
142 # hack hack hack, avoid buffering problems
143 stdin, stdout, stderr = Open3::popen3(feed.filter)
144 inth = Thread::new do
145 stdin.puts s
146 stdin.close
147 end
148 output = nil
149 outh = Thread::new do
150 output = stdout.read
151 end
152 inth.join
153 outh.join
154 s = output
155 if $?.exitstatus != 0
156 @logger.warn("Filter command for #{feed.name} exited with status #{$?.exitstatus}. Output might be corrupted !")
157 end
158 mutex.unlock
159 end
160 if Time::now - fetch_start > F2I_WARNFETCHTIME
161 @logger.info("Fetching feed #{feed.name} took #{(Time::now - fetch_start).to_i}s")
162 end
163 sparefetchers_mutex.synchronize do
164 sparefetchers += 1
165 sparefetchers_cond.signal
166 end
167 mutex.lock
168 feed.body = s
169 @cache.set_last_check(feed.name, Time::now)
170 else
171 @logger.debug("Feed #{feed.name} doesn't need to be checked again for now.")
172 end
173 mutex.unlock
174 # dump if requested
175 if @config.dumpdir
176 mutex.synchronize do
177 if feed.body
178 fname = @config.dumpdir + '/' + feed.name + '-' + Time::now.xmlschema
179 File::open(fname, 'w') { |file| file.puts feed.body }
180 end
181 end
182 end
183 # dump this feed if requested
184 if feed.dumpdir
185 mutex.synchronize do
186 if feed.body
187 fname = feed.dumpdir + '/' + feed.name + '-' + Time::now.xmlschema
188 File::open(fname, 'w') { |file| file.puts feed.body }
189 end
190 end
191 end
192 rescue Timeout::Error
193 mutex.synchronize do
194 n = @cache.fetch_failed(feed.name)
195 m = "Timeout::Error while fetching #{feed.url}: #{$!} (failed #{n} times)"
196 if n > @config.max_failures
197 @logger.fatal(m)
198 else
199 @logger.info(m)
200 end
201 end
202 rescue
203 mutex.synchronize do
204 n = @cache.fetch_failed(feed.name)
205 m = "Error while fetching #{feed.url}: #{$!} (failed #{n} times)"
206 if n > @config.max_failures
207 @logger.fatal(m)
208 else
209 @logger.info(m)
210 end
211 end
212 end
213 end
214 end
215 ths.each { |t| t.join }
216 @logger.info("Parsing and uploading ...")
217 @config.feeds.each do |f|
218 if f.body.nil? # means 304
219 @logger.debug("Feed #{f.name} did not change.")
220 next
221 end
222 begin
223 feed = FeedParser::Feed::new(f.body)
224 rescue Exception
225 n = @cache.parse_failed(f.name)
226 m = "Error while parsing #{f.name}: #{$!} (failed #{n} times)"
227 if n > @config.max_failures
228 @logger.fatal(m)
229 else
230 @logger.info(m)
231 end
232 next
233 end
234 begin
235 newitems, updateditems = @cache.get_new_items(f.name, feed.items, f.always_new, f.ignore_hash)
236 rescue
237 @logger.fatal("Exception caught when selecting new items for #{f.name}: #{$!}")
238 puts $!.backtrace
239 next
240 end
241 @logger.info("#{f.name}: #{newitems.length} new items, #{updateditems.length} updated items.") if newitems.length > 0 or updateditems.length > 0 or @logger.level == Logger::DEBUG
242 begin
243 if !cacherebuild
244 fn = f.name.gsub(/[^0-9A-Za-z]/,'')
245 updateditems.each do |i|
246 id = "<#{fn}-#{i.cacheditem.index}@#{@config.hostname}>"
247 email = item_to_mail(@config, i, id, true, f.name, f.include_images, f.wrapto)
248 f.imapaccount.updatemail(f.folder, email,
249 id, i.date || Time::new)
250 end
251 # reverse is needed to upload older items first (fixes gna#8986)
252 newitems.reverse.each do |i|
253 id = "<#{fn}-#{i.cacheditem.index}@#{@config.hostname}>"
254 email = item_to_mail(@config, i, id, false, f.name, f.include_images, f.wrapto)
255 f.imapaccount.putmail(f.folder, email, i.date || Time::new)
256 end
257 end
258 rescue
259 @logger.fatal("Exception caught while uploading mail to #{f.folder}: #{$!}")
260 puts $!.backtrace
261 @logger.fatal("We can't recover from IMAP errors, so we are exiting.")
262 exit(1)
263 end
264 begin
265 @cache.commit_cache(f.name)
266 rescue
267 @logger.fatal("Exception caught while updating cache for #{f.name}: #{$!}")
268 next
269 end
270 end
271 @logger.info("Finished. Saving cache ...")
272 begin
273 File::open("#{@config.cache}.new", 'w') { |f| @cache.save(f) }
274 rescue
275 @logger.fatal("Exception caught while writing new cache to #{@config.cache}.new: #{$!}")
276 end
277 begin
278 File::rename("#{@config.cache}.new", @config.cache)
279 rescue
280 @logger.fatal("Exception caught while renaming #{@config.cache}.new to #{@config.cache}: #{$!}")
281 end
282 @logger.info("Closing IMAP connections ...")
283 @config.imap_accounts.each_value do |ac|
284 begin
285 ac.disconnect
286 rescue
287 # servers tend to cause an exception to be raised here, hence the INFO level.
288 @logger.info("Exception caught while closing connection to #{ac.to_s}: #{$!}")
289 end
290 end
291 end
292 end
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 require 'feed2imap/sgml-parser'
20
21 # this class provides a simple SGML parser that removes HTML tags
22 class HTML2TextParser < SGMLParser
23
24 attr_reader :savedata
25
26 def initialize(verbose = false)
27 @savedata = ''
28 @pre = false
29 @href = nil
30 @links = []
31 super(verbose)
32 end
33
34 def handle_data(data)
35 # let's remove all CR
36 data.gsub!(/\n/, '') if not @pre
37
38 @savedata << data
39 end
40
41 def unknown_starttag(tag, attrs)
42 case tag
43 when 'p'
44 @savedata << "\n\n"
45 when 'br'
46 @savedata << "\n"
47 when 'b'
48 @savedata << '*'
49 when 'u'
50 @savedata << '_'
51 when 'i'
52 @savedata << '/'
53 when 'pre'
54 @savedata << "\n\n"
55 @pre = true
56 when 'a'
57 # find href in args
58 @href = nil
59 attrs.each do |a|
60 if a[0] == 'href'
61 @href = a[1]
62 end
63 end
64 if @href
65 @links << @href.gsub(/^("|'|)(.*)("|')$/,'\2')
66 end
67 end
68 end
69
70 def close
71 super
72 if @links.length > 0
73 @savedata << "\n\n"
74 @links.each_index do |i|
75 @savedata << "[#{i+1}] #{@links[i]}\n"
76 end
77 end
78 end
79
80 def unknown_endtag(tag)
81 case tag
82 when 'b'
83 @savedata << '*'
84 when 'u'
85 @savedata << '_'
86 when 'i'
87 @savedata << '/'
88 when 'pre'
89 @savedata << "\n\n"
90 @pre = false
91 when 'a'
92 if @href
93 @savedata << "[#{@links.length}]"
94 @href = nil
95 end
96 end
97 end
98 end
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 require 'net/http'
20 # get openssl if available
21 begin
22 require 'net/https'
23 rescue LoadError
24 end
25 require 'uri'
26
27
28 # max number of redirections
29 MAXREDIR = 5
30
31 HTTPDEBUG = false
32
33 # Class used to retrieve the feed over HTTP
34 class HTTPFetcher
35 def HTTPFetcher::fetcher(baseuri, uri, lastcheck, recursion)
36 proxy_host = nil
37 proxy_port = nil
38 proxy_user = nil
39 proxy_pass = nil
40 if ENV['http_proxy']
41 proxy_uri = URI.parse(ENV['http_proxy'])
42 proxy_host = proxy_uri.host
43 proxy_port = proxy_uri.port
44 proxy_user, proxy_pass = proxy_uri.userinfo.split(/:/) if proxy_uri.userinfo
45 end
46
47 http = Net::HTTP::Proxy(proxy_host,
48 proxy_port,
49 proxy_user,
50 proxy_pass ).new(uri.host, uri.port)
51 http.read_timeout = 30 # should be enough for everybody...
52 http.open_timeout = 30
53 if uri.scheme == 'https'
54 http.use_ssl = true
55 http.verify_mode = OpenSSL::SSL::VERIFY_NONE
56 end
57 if defined?(Feed2Imap)
58 useragent = "Feed2Imap v#{Feed2Imap.version} http://home.gna.org/feed2imap/"
59 else
60 useragent = 'Feed2Imap http://home.gna.org/feed2imap/'
61 end
62
63 if lastcheck == Time::at(0)
64 req = Net::HTTP::Get::new(uri.request_uri, {'User-Agent' => useragent })
65 else
66 req = Net::HTTP::Get::new(uri.request_uri, {'User-Agent' => useragent, 'If-Modified-Since' => lastcheck.httpdate})
67 end
68 if uri.userinfo
69 login, pw = uri.userinfo.split(':')
70 req.basic_auth(login, pw)
71 # workaround. eg. wikini redirects and loses auth info.
72 elsif uri.host == baseuri.host and baseuri.userinfo
73 login, pw = baseuri.userinfo.split(':')
74 req.basic_auth(login, pw)
75 end
76 begin
77 response = http.request(req)
78 rescue Timeout::Error
79 raise "Timeout while fetching #{baseuri.to_s}"
80 end
81 case response
82 when Net::HTTPSuccess
83 return response.body
84 when Net::HTTPRedirection
85 # if not modified
86 if Net::HTTPNotModified === response
87 puts "HTTPNotModified on #{uri}" if HTTPDEBUG
88 return nil
89 end
90 if recursion > 0
91 redir = URI::join(uri.to_s, response['location'])
92 return fetcher(baseuri, redir, lastcheck, recursion - 1)
93 else
94 raise "Too many redirections while fetching #{baseuri.to_s}"
95 end
96 else
97 raise "#{response.code}: #{response.message} while fetching #{baseuri.to_s}"
98 end
99 end
100
101 def HTTPFetcher::fetch(url, lastcheck)
102 uri = URI::parse(url)
103 return HTTPFetcher::fetcher(uri, uri, lastcheck, MAXREDIR)
104 end
105 end
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 # Imap connection handling
20 require 'feed2imap/rubyimap'
21 begin
22 require 'openssl'
23 rescue LoadError
24 end
25 require 'uri'
26
27 # This class is a container of IMAP accounts.
28 # Thanks to it, accounts are re-used : several feeds
29 # using the same IMAP account will create only one
30 # IMAP connection.
31 class ImapAccounts < Hash
32
33 def add_account(uri)
34 u = URI::Generic::build({ :scheme => uri.scheme,
35 :userinfo => uri.userinfo,
36 :host => uri.host,
37 :port => uri.port })
38 if not include?(u)
39 ac = ImapAccount::new(u)
40 self[u] = ac
41 end
42 return self[u]
43 end
44 end
45
46 # This class is an IMAP account, with the given fd
47 # once the connection has been established
48 class ImapAccount
49 attr_reader :uri
50
51 @@no_ssl_verify = false
52 def ImapAccount::no_ssl_verify=(v)
53 @@no_ssl_verify = v
54 end
55
56 def initialize(uri)
57 @uri = uri
58 @existing_folders = []
59 self
60 end
61
62 # connects to the IMAP server
63 # raises an exception if it fails
64 def connect
65 port = 143
66 usessl = false
67 if uri.scheme == 'imap'
68 port = 143
69 usessl = false
70 elsif uri.scheme == 'imaps'
71 port = 993
72 usessl = true
73 else
74 raise "Unknown scheme: #{uri.scheme}"
75 end
76 # use given port if port given
77 port = uri.port if uri.port
78 @connection = Net::IMAP::new(uri.host, port, usessl, nil, !@@no_ssl_verify)
79 user, password = URI::unescape(uri.userinfo).split(':',2)
80 @connection.login(user, password)
81 self
82 end
83
84 # disconnect from the IMAP server
85 def disconnect
86 if @connection
87 @connection.logout
88 @connection.disconnect
89 end
90 end
91
92 # tests if the folder exists and create it if not
93 def create_folder_if_not_exists(folder)
94 return if @existing_folders.include?(folder)
95 if @connection.list('', folder).nil?
96 @connection.create(folder)
97 @connection.subscribe(folder)
98 end
99 @existing_folders << folder
100 end
101
102 # Put the mail in the given folder
103 # You should check whether the folder exist first.
104 def putmail(folder, mail, date = Time::now)
105 create_folder_if_not_exists(folder)
106 @connection.append(folder, mail.gsub(/\n/, "\r\n"), nil, date)
107 end
108
109 # update a mail
110 def updatemail(folder, mail, id, date = Time::now)
111 create_folder_if_not_exists(folder)
112 @connection.select(folder)
113 searchres = @connection.search(['HEADER', 'Message-Id', id])
114 flags = nil
115 if searchres.length > 0
116 # we get the flags from the first result and delete everything
117 flags = @connection.fetch(searchres[0], 'FLAGS')[0].attr['FLAGS']
118 searchres.each { |m| @connection.store(m, "+FLAGS", [:Deleted]) }
119 @connection.expunge
120 flags -= [ :Recent ] # avoids errors with dovecot
121 end
122 @connection.append(folder, mail.gsub(/\n/, "\r\n"), flags, date)
123 end
124
125 # convert to string
126 def to_s
127 u2 = uri.clone
128 u2.password = 'PASSWORD'
129 u2.to_s
130 end
131
132 # remove mails in a folder according to a criteria
133 def cleanup(folder, dryrun = false)
134 puts "-- Considering #{folder}:"
135 @connection.select(folder)
136 a = ['SEEN', 'NOT', 'FLAGGED', 'BEFORE', (Date::today - 3).strftime('%d-%b-%Y')]
137 todel = @connection.search(a)
138 todel.each do |m|
139 f = @connection.fetch(m, "FULL")
140 d = f[0].attr['INTERNALDATE']
141 s = f[0].attr['ENVELOPE'].subject
142 if s =~ /^=\?utf-8\?b\?/
143 s = Base64::decode64(s.gsub(/^=\?utf-8\?b\?(.*)\?=$/, '\1')).toISO_8859_1('utf-8')
144 end
145 if dryrun
146 puts "To remove: #{s} (#{d})"
147 else
148 puts "Removing: #{s} (#{d})"
149 @connection.store(m, "+FLAGS", [:Deleted])
150 end
151 end
152 puts "-- Deleted #{todel.length} messages."
153 if not dryrun
154 @connection.expunge
155 end
156 return todel.length
157 end
158 end
159
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This file contains classes to parse a feed and store it as a Channel object.
5
6 This program is free software; you can redistribute it and/or modify
7 it under the terms of the GNU General Public License as published by
8 the Free Software Foundation; either version 2 of the License, or
9 (at your option) any later version.
10
11 This program is distributed in the hope that it will be useful,
12 but WITHOUT ANY WARRANTY; without even the implied warranty of
13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 GNU General Public License for more details.
15
16 You should have received a copy of the GNU General Public License
17 along with this program; if not, write to the Free Software
18 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
19 =end
20
21 require 'rexml/document'
22 require 'time'
23 require 'rmail'
24 require 'feedparser'
25 require 'feedparser/text-output'
26 require 'feedparser/html-output'
27 require 'base64'
28 require 'rmail'
29 require 'digest/md5'
30
31 class String
32 def needMIME
33 utf8 = false
34 begin
35 self.unpack('U*').each do |c|
36 if c > 127
37 utf8 = true
38 break
39 end
40 end
41 rescue
42 # safe fallback in case of problems
43 utf8 = true
44 end
45 utf8
46 end
47 end
48
49 def item_to_mail(config, item, id, updated, from = 'Feed2Imap', inline_images = false, wrapto = false)
50 message = RMail::Message::new
51 if item.creator and item.creator != ''
52 if item.creator.include?('@')
53 message.header['From'] = item.creator.chomp
54 else
55 message.header['From'] = "=?utf-8?b?#{Base64::encode64(item.creator.chomp).gsub("\n",'')}?= <#{config.default_email}>"
56 end
57 else
58 message.header['From'] = "=?utf-8?b?#{Base64::encode64(from).gsub("\n",'')}?= <#{config.default_email}>"
59 end
60 message.header['To'] = "=?utf-8?b?#{Base64::encode64(from).gsub("\n",'')}?= <#{config.default_email}>"
61
62 if item.date.nil?
63 message.header['Date'] = Time::new.rfc2822
64 else
65 message.header['Date'] = item.date.rfc2822
66 end
67 message.header['X-Feed2Imap-Version'] = F2I_VERSION if defined?(F2I_VERSION)
68 message.header['Message-Id'] = id
69 message.header['X-F2IStatus'] = "Updated" if updated
70 # treat subject. Might need MIME encoding.
71 subj = item.title or (item.date and item.date.to_s) or item.link
72 if subj
73 if subj.needMIME
74 message.header['Subject'] = "=?utf-8?b?#{Base64::encode64(subj).gsub("\n",'')}?="
75 else
76 message.header['Subject'] = subj
77 end
78 end
79 textpart = RMail::Message::new
80 textpart.header['Content-Type'] = 'text/plain; charset=utf-8; format=flowed'
81 textpart.header['Content-Transfer-Encoding'] = '8bit'
82 textpart.body = item.to_text(true, wrapto, false)
83 htmlpart = RMail::Message::new
84 htmlpart.header['Content-Type'] = 'text/html; charset=utf-8'
85 htmlpart.header['Content-Transfer-Encoding'] = '8bit'
86 htmlpart.body = item.to_html
87
88 # inline images as attachments
89 imgs = []
90 if inline_images
91 cids = []
92 htmlpart.body.gsub!(/(<img[^>]+)src="(\S+?\/([^\/]+?\.(png|gif|jpe?g)))"([^>]*>)/i) do |match|
93 # $2 contains url, $3 the image name, $4 the image extension
94 begin
95 image = Base64.encode64(HTTPFetcher::fetch($2, Time.at(0)).chomp) + "\n"
96 cid = "#{Digest::MD5.hexdigest($2)}@#{config.hostname}"
97 if not cids.include?(cid)
98 cids << cid
99 imgpart = RMail::Message.new
100 imgpart.header.set('Content-ID', "<#{cid}>")
101 type = $4
102 type = 'jpeg' if type.downcase == 'jpg' # hack hack hack
103 imgpart.header.set('Content-Type', "image/#{type}", 'name' => $3)
104 imgpart.header.set('Content-Disposition', 'attachment', 'filename' => $3)
105 imgpart.header.set('Content-Transfer-Encoding', 'base64')
106 imgpart.body = image
107 imgs << imgpart
108 end
109 # now to specify what to replace with
110 newtag = "#{$1}src=\"cid:#{cid}\"#{$5}"
111 #print "#{cid}: Replacing '#{$&}' with '#{newtag}'...\n"
112 newtag
113 rescue
114 #print "Error while fetching image #{$2}: #{$!}...\n"
115 $& # don't modify on exception
116 end
117 end
118 end
119 if imgs.length > 0
120 message.header.set('Content-Type', 'multipart/related', 'type'=> 'multipart/alternative')
121 texthtml = RMail::Message::new
122 texthtml.header.set('Content-Type', 'multipart/alternative')
123 texthtml.add_part(textpart)
124 texthtml.add_part(htmlpart)
125 message.add_part(texthtml)
126 imgs.each do |i|
127 message.add_part(i)
128 end
129 else
130 message.header['Content-Type'] = 'multipart/alternative'
131 message.add_part(textpart)
132 message.add_part(htmlpart)
133 end
134 return message.to_s
135 end
136
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server, or local Maildir
2 Copyright (c) 2009 Andreas Rottmann
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program. If not, see <http://www.gnu.org/licenses/>.
16 =end
17
18 require 'uri'
19 require 'fileutils'
20 require 'fcntl'
21
22 class MaildirAccount
23 MYHOSTNAME = Socket.gethostname
24
25 attr_reader :uri
26
27 def putmail(folder, mail, date = Time::now)
28 store_message(folder_dir(folder), date, nil) do |f|
29 f.puts(mail)
30 end
31 end
32
33 def updatemail(folder, mail, idx, date = Time::now)
34 dir = folder_dir(folder)
35 guarantee_maildir(dir)
36 mail_files = find_mails(dir, idx)
37 flags = nil
38 if mail_files.length > 0
39 # get the info from the first result and delete everything
40 info = maildir_file_info(mail_files[0])
41 mail_files.each { |f| File.delete(File.join(dir, f)) }
42 end
43 store_message(dir, date, info) { |f| f.puts(mail) }
44 end
45
46 def to_s
47 uri.to_s
48 end
49
50 def cleanup(folder, dryrun = false)
51 dir = folder_dir(folder)
52 puts "-- Considering #{dir}:"
53 guarantee_maildir(dir)
54
55 del_count = 0
56 recent_time = Time.now() -- (3 * 24 * 60 * 60) # 3 days
57 Dir[File.join(dir, 'cur', '*')].each do |fn|
58 flags = maildir_file_info_flags(fn)
59 # don't consider not-seen, flagged, or recent messages
60 mtime = File.mtime(fn)
61 next if (not flags.index('S') or
62 flags.index('F') or
63 mtime > recent_time)
64 File.open(fn) do |f|
65 mail = RMail::Parser.read(f)
66 end
67 if dryrun
68 puts "To remove: #{subject} #{mtime}"
69 else
70 puts "Removing: #{subject} #{mtime}"
71 File.delete(fn)
72 end
73 del_count += 1
74 end
75 puts "-- Deleted #{del_count} messages"
76 return del_count
77 end
78
79 private
80
81 def folder_dir(folder)
82 return File.join('/', folder)
83 end
84
85 def store_message(dir, date, info, &block)
86 # TODO: handle `date'
87
88 guarantee_maildir(dir)
89
90 stored = false
91 Dir.chdir(dir) do |d|
92 timer = 30
93 fd = nil
94 while timer >= 0
95 new_fn = new_maildir_basefn
96 tmp_path = File.join(dir, 'tmp', new_fn)
97 new_path = File.join(dir, 'new', new_fn)
98 begin
99 fd = IO::sysopen(tmp_path,
100 Fcntl::O_WRONLY | Fcntl::O_EXCL | Fcntl::O_CREAT)
101 break
102 rescue Errno::EEXIST
103 sleep 2
104 timer -= 2
105 next
106 end
107 end
108
109 if fd
110 begin
111 f = IO.open(fd)
112 # provide a writable interface for the caller
113 yield f
114 f.fsync
115 File.link tmp_path, new_path
116 stored = true
117 ensure
118 File.unlink tmp_path if File.exists? tmp_path
119 end
120 end
121
122 if stored and info
123 cur_path = File.join(dir, 'cur', new_fn + ':' + info)
124 File.rename(new_path, cur_path)
125 end
126 end # Dir.chdir
127
128 return stored
129 end
130
131 def find_mails(dir, idx)
132 dir_paths = []
133 ['cur', 'new'].each do |d|
134 subdir = File.join(dir, d)
135 raise "#{subdir} not a directory" unless File.directory? subdir
136 Dir[File.join(subdir, '*')].each do |fn|
137 File.open(fn) do |f|
138 mail = RMail::Parser.read(f)
139 cache_index = mail.header['Message-Id']
140 next if not (cache_index and cache_index == idx)
141 dir_paths.push(File.join(d, File.basename(fn)))
142 end
143 end
144 end
145 return dir_paths
146 end
147
148 def guarantee_maildir(dir)
149 # Ensure maildir-folderness
150 ['new', 'cur', 'tmp'].each do |d|
151 FileUtils.mkdir_p(File.join(dir, d))
152 end
153 end
154
155 def maildir_file_info(file)
156 basename = File.basename(file)
157 colon = basename.rindex(':')
158
159 return (colon and basename.slice(colon + 1, -1))
160 end
161
162 # Shamelessly taken from
163 # http://gitorious.org/sup/mainline/blobs/master/lib/sup/maildir.rb
164 def new_maildir_basefn
165 Kernel::srand()
166 "#{Time.now.to_i.to_s}.#{$$}#{Kernel.rand(1000000)}.#{MYHOSTNAME}"
167 end
168 end
169
0 =begin
1 Feed2Imap - RSS/Atom Aggregator uploading to an IMAP Server
2 Copyright (c) 2005 Lucas Nussbaum <lucas@lucas-nussbaum.net>
3
4 This program is free software; you can redistribute it and/or modify
5 it under the terms of the GNU General Public License as published by
6 the Free Software Foundation; either version 2 of the License, or
7 (at your option) any later version.
8
9 This program is distributed in the hope that it will be useful,
10 but WITHOUT ANY WARRANTY; without even the implied warranty of
11 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 GNU General Public License for more details.
13
14 You should have received a copy of the GNU General Public License
15 along with this program; if not, write to the Free Software
16 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17 =end
18
19 require 'feedparser'
20
21 # Patch for REXML
22 # Very ugly patch to make REXML error-proof.
23 # The problem is REXML uses IConv, which isn't error-proof at all.
24 # With those changes, it uses unpack/pack with some error handling
25 module REXML
26 module Encoding
27 def decode(str)
28 return str.toUTF8(@encoding)
29 end
30
31 def encode(str)
32 return str
33 end
34
35 def encoding=(enc)
36 return if defined? @encoding and enc == @encoding
37 @encoding = enc || 'utf-8'
38 end
39 end
40
41 class Element
42 def children
43 @children
44 end
45 end
46 end
0 # File fetched from
1 # http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/trunk/lib/net/imap.rb?view=log
2 # Current rev: 27336
3 ############################################################################
4 #
5 # = net/imap.rb
6 #
7 # Copyright (C) 2000 Shugo Maeda <shugo@ruby-lang.org>
8 #
9 # This library is distributed under the terms of the Ruby license.
10 # You can freely distribute/modify this library.
11 #
12 # Documentation: Shugo Maeda, with RDoc conversion and overview by William
13 # Webber.
14 #
15 # See Net::IMAP for documentation.
16 #
17
18
19 require "socket"
20 require "monitor"
21 require "digest/md5"
22 require "strscan"
23 begin
24 require "openssl"
25 rescue LoadError
26 end
27
28 module Net
29
30 #
31 # Net::IMAP implements Internet Message Access Protocol (IMAP) client
32 # functionality. The protocol is described in [IMAP].
33 #
34 # == IMAP Overview
35 #
36 # An IMAP client connects to a server, and then authenticates
37 # itself using either #authenticate() or #login(). Having
38 # authenticated itself, there is a range of commands
39 # available to it. Most work with mailboxes, which may be
40 # arranged in an hierarchical namespace, and each of which
41 # contains zero or more messages. How this is implemented on
42 # the server is implementation-dependent; on a UNIX server, it
43 # will frequently be implemented as a files in mailbox format
44 # within a hierarchy of directories.
45 #
46 # To work on the messages within a mailbox, the client must
47 # first select that mailbox, using either #select() or (for
48 # read-only access) #examine(). Once the client has successfully
49 # selected a mailbox, they enter _selected_ state, and that
50 # mailbox becomes the _current_ mailbox, on which mail-item
51 # related commands implicitly operate.
52 #
53 # Messages have two sorts of identifiers: message sequence
54 # numbers, and UIDs.
55 #
56 # Message sequence numbers number messages within a mail box
57 # from 1 up to the number of items in the mail box. If new
58 # message arrives during a session, it receives a sequence
59 # number equal to the new size of the mail box. If messages
60 # are expunged from the mailbox, remaining messages have their
61 # sequence numbers "shuffled down" to fill the gaps.
62 #
63 # UIDs, on the other hand, are permanently guaranteed not to
64 # identify another message within the same mailbox, even if
65 # the existing message is deleted. UIDs are required to
66 # be assigned in ascending (but not necessarily sequential)
67 # order within a mailbox; this means that if a non-IMAP client
68 # rearranges the order of mailitems within a mailbox, the
69 # UIDs have to be reassigned. An IMAP client cannot thus
70 # rearrange message orders.
71 #
72 # == Examples of Usage
73 #
74 # === List sender and subject of all recent messages in the default mailbox
75 #
76 # imap = Net::IMAP.new('mail.example.com')
77 # imap.authenticate('LOGIN', 'joe_user', 'joes_password')
78 # imap.examine('INBOX')
79 # imap.search(["RECENT"]).each do |message_id|
80 # envelope = imap.fetch(message_id, "ENVELOPE")[0].attr["ENVELOPE"]
81 # puts "#{envelope.from[0].name}: \t#{envelope.subject}"
82 # end
83 #
84 # === Move all messages from April 2003 from "Mail/sent-mail" to "Mail/sent-apr03"
85 #
86 # imap = Net::IMAP.new('mail.example.com')
87 # imap.authenticate('LOGIN', 'joe_user', 'joes_password')
88 # imap.select('Mail/sent-mail')
89 # if not imap.list('Mail/', 'sent-apr03')
90 # imap.create('Mail/sent-apr03')
91 # end
92 # imap.search(["BEFORE", "30-Apr-2003", "SINCE", "1-Apr-2003"]).each do |message_id|
93 # imap.copy(message_id, "Mail/sent-apr03")
94 # imap.store(message_id, "+FLAGS", [:Deleted])
95 # end
96 # imap.expunge
97 #
98 # == Thread Safety
99 #
100 # Net::IMAP supports concurrent threads. For example,
101 #
102 # imap = Net::IMAP.new("imap.foo.net", "imap2")
103 # imap.authenticate("cram-md5", "bar", "password")
104 # imap.select("inbox")
105 # fetch_thread = Thread.start { imap.fetch(1..-1, "UID") }
106 # search_result = imap.search(["BODY", "hello"])
107 # fetch_result = fetch_thread.value
108 # imap.disconnect
109 #
110 # This script invokes the FETCH command and the SEARCH command concurrently.
111 #
112 # == Errors
113 #
114 # An IMAP server can send three different types of responses to indicate
115 # failure:
116 #
117 # NO:: the attempted command could not be successfully completed. For
118 # instance, the username/password used for logging in are incorrect;
119 # the selected mailbox does not exists; etc.
120 #
121 # BAD:: the request from the client does not follow the server's
122 # understanding of the IMAP protocol. This includes attempting
123 # commands from the wrong client state; for instance, attempting
124 # to perform a SEARCH command without having SELECTed a current
125 # mailbox. It can also signal an internal server
126 # failure (such as a disk crash) has occurred.
127 #
128 # BYE:: the server is saying goodbye. This can be part of a normal
129 # logout sequence, and can be used as part of a login sequence
130 # to indicate that the server is (for some reason) unwilling
131 # to accept our connection. As a response to any other command,
132 # it indicates either that the server is shutting down, or that
133 # the server is timing out the client connection due to inactivity.
134 #
135 # These three error response are represented by the errors
136 # Net::IMAP::NoResponseError, Net::IMAP::BadResponseError, and
137 # Net::IMAP::ByeResponseError, all of which are subclasses of
138 # Net::IMAP::ResponseError. Essentially, all methods that involve
139 # sending a request to the server can generate one of these errors.
140 # Only the most pertinent instances have been documented below.
141 #
142 # Because the IMAP class uses Sockets for communication, its methods
143 # are also susceptible to the various errors that can occur when
144 # working with sockets. These are generally represented as
145 # Errno errors. For instance, any method that involves sending a
146 # request to the server and/or receiving a response from it could
147 # raise an Errno::EPIPE error if the network connection unexpectedly
148 # goes down. See the socket(7), ip(7), tcp(7), socket(2), connect(2),
149 # and associated man pages.
150 #
151 # Finally, a Net::IMAP::DataFormatError is thrown if low-level data
152 # is found to be in an incorrect format (for instance, when converting
153 # between UTF-8 and UTF-16), and Net::IMAP::ResponseParseError is
154 # thrown if a server response is non-parseable.
155 #
156 #
157 # == References
158 #
159 # [[IMAP]]
160 # M. Crispin, "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1",
161 # RFC 2060, December 1996. (Note: since obsoleted by RFC 3501)
162 #
163 # [[LANGUAGE-TAGS]]
164 # Alvestrand, H., "Tags for the Identification of
165 # Languages", RFC 1766, March 1995.
166 #
167 # [[MD5]]
168 # Myers, J., and M. Rose, "The Content-MD5 Header Field", RFC
169 # 1864, October 1995.
170 #
171 # [[MIME-IMB]]
172 # Freed, N., and N. Borenstein, "MIME (Multipurpose Internet
173 # Mail Extensions) Part One: Format of Internet Message Bodies", RFC
174 # 2045, November 1996.
175 #
176 # [[RFC-822]]
177 # Crocker, D., "Standard for the Format of ARPA Internet Text
178 # Messages", STD 11, RFC 822, University of Delaware, August 1982.
179 #
180 # [[RFC-2087]]
181 # Myers, J., "IMAP4 QUOTA extension", RFC 2087, January 1997.
182 #
183 # [[RFC-2086]]
184 # Myers, J., "IMAP4 ACL extension", RFC 2086, January 1997.
185 #
186 # [[RFC-2195]]
187 # Klensin, J., Catoe, R., and Krumviede, P., "IMAP/POP AUTHorize Extension
188 # for Simple Challenge/Response", RFC 2195, September 1997.
189 #
190 # [[SORT-THREAD-EXT]]
191 # Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - SORT and THREAD
192 # Extensions", draft-ietf-imapext-sort, May 2003.
193 #
194 # [[OSSL]]
195 # http://www.openssl.org
196 #
197 # [[RSSL]]
198 # http://savannah.gnu.org/projects/rubypki
199 #
200 # [[UTF7]]
201 # Goldsmith, D. and Davis, M., "UTF-7: A Mail-Safe Transformation Format of
202 # Unicode", RFC 2152, May 1997.
203 #
204 class IMAP
205 include MonitorMixin
206 if defined?(OpenSSL)
207 include OpenSSL
208 include SSL
209 end
210
211 # Returns an initial greeting response from the server.
212 attr_reader :greeting
213
214 # Returns recorded untagged responses. For example:
215 #
216 # imap.select("inbox")
217 # p imap.responses["EXISTS"][-1]
218 # #=> 2
219 # p imap.responses["UIDVALIDITY"][-1]
220 # #=> 968263756
221 attr_reader :responses
222
223 # Returns all response handlers.
224 attr_reader :response_handlers
225
226 # The thread to receive exceptions.
227 attr_accessor :client_thread
228
229 # Flag indicating a message has been seen
230 SEEN = :Seen
231
232 # Flag indicating a message has been answered
233 ANSWERED = :Answered
234
235 # Flag indicating a message has been flagged for special or urgent
236 # attention
237 FLAGGED = :Flagged
238
239 # Flag indicating a message has been marked for deletion. This
240 # will occur when the mailbox is closed or expunged.
241 DELETED = :Deleted
242
243 # Flag indicating a message is only a draft or work-in-progress version.
244 DRAFT = :Draft
245
246 # Flag indicating that the message is "recent", meaning that this
247 # session is the first session in which the client has been notified
248 # of this message.
249 RECENT = :Recent
250
251 # Flag indicating that a mailbox context name cannot contain
252 # children.
253 NOINFERIORS = :Noinferiors
254
255 # Flag indicating that a mailbox is not selected.
256 NOSELECT = :Noselect
257
258 # Flag indicating that a mailbox has been marked "interesting" by
259 # the server; this commonly indicates that the mailbox contains
260 # new messages.
261 MARKED = :Marked
262
263 # Flag indicating that the mailbox does not contains new messages.
264 UNMARKED = :Unmarked
265
266 # Returns the debug mode.
267 def self.debug
268 return @@debug
269 end
270
271 # Sets the debug mode.
272 def self.debug=(val)
273 return @@debug = val
274 end
275
276 # Returns the max number of flags interned to symbols.
277 def self.max_flag_count
278 return @@max_flag_count
279 end
280
281 # Sets the max number of flags interned to symbols.
282 def self.max_flag_count=(count)
283 @@max_flag_count = count
284 end
285
286 # Adds an authenticator for Net::IMAP#authenticate. +auth_type+
287 # is the type of authentication this authenticator supports
288 # (for instance, "LOGIN"). The +authenticator+ is an object
289 # which defines a process() method to handle authentication with
290 # the server. See Net::IMAP::LoginAuthenticator,
291 # Net::IMAP::CramMD5Authenticator, and Net::IMAP::DigestMD5Authenticator
292 # for examples.
293 #
294 #
295 # If +auth_type+ refers to an existing authenticator, it will be
296 # replaced by the new one.
297 def self.add_authenticator(auth_type, authenticator)
298 @@authenticators[auth_type] = authenticator
299 end
300
301 # Disconnects from the server.
302 def disconnect
303 begin
304 begin
305 # try to call SSL::SSLSocket#io.
306 @sock.io.shutdown
307 rescue NoMethodError
308 # @sock is not an SSL::SSLSocket.
309 @sock.shutdown
310 end
311 rescue Errno::ENOTCONN
312 # ignore `Errno::ENOTCONN: Socket is not connected' on some platforms.
313 end
314 @receiver_thread.join
315 @sock.close
316 end
317
318 # Returns true if disconnected from the server.
319 def disconnected?
320 return @sock.closed?
321 end
322
323 # Sends a CAPABILITY command, and returns an array of
324 # capabilities that the server supports. Each capability
325 # is a string. See [IMAP] for a list of possible
326 # capabilities.
327 #
328 # Note that the Net::IMAP class does not modify its
329 # behaviour according to the capabilities of the server;
330 # it is up to the user of the class to ensure that
331 # a certain capability is supported by a server before
332 # using it.
333 def capability
334 synchronize do
335 send_command("CAPABILITY")
336 return @responses.delete("CAPABILITY")[-1]
337 end
338 end
339
340 # Sends a NOOP command to the server. It does nothing.
341 def noop
342 send_command("NOOP")
343 end
344
345 # Sends a LOGOUT command to inform the server that the client is
346 # done with the connection.
347 def logout
348 send_command("LOGOUT")
349 end
350
351 # Sends a STARTTLS command to start TLS session.
352 def starttls(options = {}, verify = true)
353 send_command("STARTTLS") do |resp|
354 if resp.kind_of?(TaggedResponse) && resp.name == "OK"
355 begin
356 # for backward compatibility
357 certs = options.to_str
358 options = create_ssl_params(certs, verify)
359 rescue NoMethodError
360 end
361 start_tls_session(options)
362 end
363 end
364 end
365
366 # Sends an AUTHENTICATE command to authenticate the client.
367 # The +auth_type+ parameter is a string that represents
368 # the authentication mechanism to be used. Currently Net::IMAP
369 # supports authentication mechanisms:
370 #
371 # LOGIN:: login using cleartext user and password.
372 # CRAM-MD5:: login with cleartext user and encrypted password
373 # (see [RFC-2195] for a full description). This
374 # mechanism requires that the server have the user's
375 # password stored in clear-text password.
376 #
377 # For both these mechanisms, there should be two +args+: username
378 # and (cleartext) password. A server may not support one or other
379 # of these mechanisms; check #capability() for a capability of
380 # the form "AUTH=LOGIN" or "AUTH=CRAM-MD5".
381 #
382 # Authentication is done using the appropriate authenticator object:
383 # see @@authenticators for more information on plugging in your own
384 # authenticator.
385 #
386 # For example:
387 #
388 # imap.authenticate('LOGIN', user, password)
389 #
390 # A Net::IMAP::NoResponseError is raised if authentication fails.
391 def authenticate(auth_type, *args)
392 auth_type = auth_type.upcase
393 unless @@authenticators.has_key?(auth_type)
394 raise ArgumentError,
395 format('unknown auth type - "%s"', auth_type)
396 end
397 authenticator = @@authenticators[auth_type].new(*args)
398 send_command("AUTHENTICATE", auth_type) do |resp|
399 if resp.instance_of?(ContinuationRequest)
400 data = authenticator.process(resp.data.text.unpack("m")[0])
401 s = [data].pack("m").gsub(/\n/, "")
402 send_string_data(s)
403 put_string(CRLF)
404 end
405 end
406 end
407
408 # Sends a LOGIN command to identify the client and carries
409 # the plaintext +password+ authenticating this +user+. Note
410 # that, unlike calling #authenticate() with an +auth_type+
411 # of "LOGIN", #login() does *not* use the login authenticator.
412 #
413 # A Net::IMAP::NoResponseError is raised if authentication fails.
414 def login(user, password)
415 send_command("LOGIN", user, password)
416 end
417
418 # Sends a SELECT command to select a +mailbox+ so that messages
419 # in the +mailbox+ can be accessed.
420 #
421 # After you have selected a mailbox, you may retrieve the
422 # number of items in that mailbox from @responses["EXISTS"][-1],
423 # and the number of recent messages from @responses["RECENT"][-1].
424 # Note that these values can change if new messages arrive
425 # during a session; see #add_response_handler() for a way of
426 # detecting this event.
427 #
428 # A Net::IMAP::NoResponseError is raised if the mailbox does not
429 # exist or is for some reason non-selectable.
430 def select(mailbox)
431 synchronize do
432 @responses.clear
433 send_command("SELECT", mailbox)
434 end
435 end
436
437 # Sends a EXAMINE command to select a +mailbox+ so that messages
438 # in the +mailbox+ can be accessed. Behaves the same as #select(),
439 # except that the selected +mailbox+ is identified as read-only.
440 #
441 # A Net::IMAP::NoResponseError is raised if the mailbox does not
442 # exist or is for some reason non-examinable.
443 def examine(mailbox)
444 synchronize do
445 @responses.clear
446 send_command("EXAMINE", mailbox)
447 end
448 end
449
450 # Sends a CREATE command to create a new +mailbox+.
451 #
452 # A Net::IMAP::NoResponseError is raised if a mailbox with that name
453 # cannot be created.
454 def create(mailbox)
455 send_command("CREATE", mailbox)
456 end
457
458 # Sends a DELETE command to remove the +mailbox+.
459 #
460 # A Net::IMAP::NoResponseError is raised if a mailbox with that name
461 # cannot be deleted, either because it does not exist or because the
462 # client does not have permission to delete it.
463 def delete(mailbox)
464 send_command("DELETE", mailbox)
465 end
466
467 # Sends a RENAME command to change the name of the +mailbox+ to
468 # +newname+.
469 #
470 # A Net::IMAP::NoResponseError is raised if a mailbox with the
471 # name +mailbox+ cannot be renamed to +newname+ for whatever
472 # reason; for instance, because +mailbox+ does not exist, or
473 # because there is already a mailbox with the name +newname+.
474 def rename(mailbox, newname)
475 send_command("RENAME", mailbox, newname)
476 end
477
478 # Sends a SUBSCRIBE command to add the specified +mailbox+ name to
479 # the server's set of "active" or "subscribed" mailboxes as returned
480 # by #lsub().
481 #
482 # A Net::IMAP::NoResponseError is raised if +mailbox+ cannot be
483 # subscribed to, for instance because it does not exist.
484 def subscribe(mailbox)
485 send_command("SUBSCRIBE", mailbox)
486 end
487
488 # Sends a UNSUBSCRIBE command to remove the specified +mailbox+ name
489 # from the server's set of "active" or "subscribed" mailboxes.
490 #
491 # A Net::IMAP::NoResponseError is raised if +mailbox+ cannot be
492 # unsubscribed from, for instance because the client is not currently
493 # subscribed to it.
494 def unsubscribe(mailbox)
495 send_command("UNSUBSCRIBE", mailbox)
496 end
497
498 # Sends a LIST command, and returns a subset of names from
499 # the complete set of all names available to the client.
500 # +refname+ provides a context (for instance, a base directory
501 # in a directory-based mailbox hierarchy). +mailbox+ specifies
502 # a mailbox or (via wildcards) mailboxes under that context.
503 # Two wildcards may be used in +mailbox+: '*', which matches
504 # all characters *including* the hierarchy delimiter (for instance,
505 # '/' on a UNIX-hosted directory-based mailbox hierarchy); and '%',
506 # which matches all characters *except* the hierarchy delimiter.
507 #
508 # If +refname+ is empty, +mailbox+ is used directly to determine
509 # which mailboxes to match. If +mailbox+ is empty, the root
510 # name of +refname+ and the hierarchy delimiter are returned.
511 #
512 # The return value is an array of +Net::IMAP::MailboxList+. For example:
513 #
514 # imap.create("foo/bar")
515 # imap.create("foo/baz")
516 # p imap.list("", "foo/%")
517 # #=> [#<Net::IMAP::MailboxList attr=[:Noselect], delim="/", name="foo/">, \\
518 # #<Net::IMAP::MailboxList attr=[:Noinferiors, :Marked], delim="/", name="foo/bar">, \\
519 # #<Net::IMAP::MailboxList attr=[:Noinferiors], delim="/", name="foo/baz">]
520 def list(refname, mailbox)
521 synchronize do
522 send_command("LIST", refname, mailbox)
523 return @responses.delete("LIST")
524 end
525 end
526
527 # Sends the GETQUOTAROOT command along with specified +mailbox+.
528 # This command is generally available to both admin and user.
529 # If mailbox exists, returns an array containing objects of
530 # Net::IMAP::MailboxQuotaRoot and Net::IMAP::MailboxQuota.
531 def getquotaroot(mailbox)
532 synchronize do
533 send_command("GETQUOTAROOT", mailbox)
534 result = []
535 result.concat(@responses.delete("QUOTAROOT"))
536 result.concat(@responses.delete("QUOTA"))
537 return result
538 end
539 end
540
541 # Sends the GETQUOTA command along with specified +mailbox+.
542 # If this mailbox exists, then an array containing a
543 # Net::IMAP::MailboxQuota object is returned. This
544 # command generally is only available to server admin.
545 def getquota(mailbox)
546 synchronize do
547 send_command("GETQUOTA", mailbox)
548 return @responses.delete("QUOTA")
549 end
550 end
551
552 # Sends a SETQUOTA command along with the specified +mailbox+ and
553 # +quota+. If +quota+ is nil, then quota will be unset for that
554 # mailbox. Typically one needs to be logged in as server admin
555 # for this to work. The IMAP quota commands are described in
556 # [RFC-2087].
557 def setquota(mailbox, quota)
558 if quota.nil?
559 data = '()'
560 else
561 data = '(STORAGE ' + quota.to_s + ')'
562 end
563 send_command("SETQUOTA", mailbox, RawData.new(data))
564 end
565
566 # Sends the SETACL command along with +mailbox+, +user+ and the
567 # +rights+ that user is to have on that mailbox. If +rights+ is nil,
568 # then that user will be stripped of any rights to that mailbox.
569 # The IMAP ACL commands are described in [RFC-2086].
570 def setacl(mailbox, user, rights)
571 if rights.nil?
572 send_command("SETACL", mailbox, user, "")
573 else
574 send_command("SETACL", mailbox, user, rights)
575 end
576 end
577
578 # Send the GETACL command along with specified +mailbox+.
579 # If this mailbox exists, an array containing objects of
580 # Net::IMAP::MailboxACLItem will be returned.
581 def getacl(mailbox)
582 synchronize do
583 send_command("GETACL", mailbox)
584 return @responses.delete("ACL")[-1]
585 end
586 end
587
588 # Sends a LSUB command, and returns a subset of names from the set
589 # of names that the user has declared as being "active" or
590 # "subscribed". +refname+ and +mailbox+ are interpreted as
591 # for #list().
592 # The return value is an array of +Net::IMAP::MailboxList+.
593 def lsub(refname, mailbox)
594 synchronize do
595 send_command("LSUB", refname, mailbox)
596 return @responses.delete("LSUB")
597 end
598 end
599
600 # Sends a STATUS command, and returns the status of the indicated
601 # +mailbox+. +attr+ is a list of one or more attributes that
602 # we are request the status of. Supported attributes include:
603 #
604 # MESSAGES:: the number of messages in the mailbox.
605 # RECENT:: the number of recent messages in the mailbox.
606 # UNSEEN:: the number of unseen messages in the mailbox.
607 #
608 # The return value is a hash of attributes. For example:
609 #
610 # p imap.status("inbox", ["MESSAGES", "RECENT"])
611