Codebase list ctdconverter / d212575
New upstream version 2.0 Michael R. Crusoe 5 years ago
16 changed file(s) with 2794 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
+38
-0
.gitignore less more
0 *.py[cod]
1
2 # C extensions
3 *.so
4
5 # Packages
6 *.egg
7 *.egg-info
8 build
9 eggs
10 parts
11 bin
12 var
13 sdist
14 develop-eggs
15 .installed.cfg
16 lib
17 lib64
18 __pycache__
19
20 # Installer logs
21 pip-log.txt
22
23 # Unit test / coverage reports
24 .coverage
25 .tox
26 nosetests.xml
27
28 # Translations
29 *.mo
30
31 # Mr Developer
32 .mr.developer.cfg
33 .project
34 .pydevproject
35
36 # ignore PyCharm stuff
37 .idea
+2
-0
.settings/org.eclipse.core.resources.prefs less more
0 eclipse.preferences.version=1
1 encoding//galaxyconfiggenerator/generator.py=utf-8
+674
-0
LICENSE less more
0 GNU GENERAL PUBLIC LICENSE
1 Version 3, 29 June 2007
2
3 Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
4 Everyone is permitted to copy and distribute verbatim copies
5 of this license document, but changing it is not allowed.
6
7 Preamble
8
9 The GNU General Public License is a free, copyleft license for
10 software and other kinds of works.
11
12 The licenses for most software and other practical works are designed
13 to take away your freedom to share and change the works. By contrast,
14 the GNU General Public License is intended to guarantee your freedom to
15 share and change all versions of a program--to make sure it remains free
16 software for all its users. We, the Free Software Foundation, use the
17 GNU General Public License for most of our software; it applies also to
18 any other work released this way by its authors. You can apply it to
19 your programs, too.
20
21 When we speak of free software, we are referring to freedom, not
22 price. Our General Public Licenses are designed to make sure that you
23 have the freedom to distribute copies of free software (and charge for
24 them if you wish), that you receive source code or can get it if you
25 want it, that you can change the software or use pieces of it in new
26 free programs, and that you know you can do these things.
27
28 To protect your rights, we need to prevent others from denying you
29 these rights or asking you to surrender the rights. Therefore, you have
30 certain responsibilities if you distribute copies of the software, or if
31 you modify it: responsibilities to respect the freedom of others.
32
33 For example, if you distribute copies of such a program, whether
34 gratis or for a fee, you must pass on to the recipients the same
35 freedoms that you received. You must make sure that they, too, receive
36 or can get the source code. And you must show them these terms so they
37 know their rights.
38
39 Developers that use the GNU GPL protect your rights with two steps:
40 (1) assert copyright on the software, and (2) offer you this License
41 giving you legal permission to copy, distribute and/or modify it.
42
43 For the developers' and authors' protection, the GPL clearly explains
44 that there is no warranty for this free software. For both users' and
45 authors' sake, the GPL requires that modified versions be marked as
46 changed, so that their problems will not be attributed erroneously to
47 authors of previous versions.
48
49 Some devices are designed to deny users access to install or run
50 modified versions of the software inside them, although the manufacturer
51 can do so. This is fundamentally incompatible with the aim of
52 protecting users' freedom to change the software. The systematic
53 pattern of such abuse occurs in the area of products for individuals to
54 use, which is precisely where it is most unacceptable. Therefore, we
55 have designed this version of the GPL to prohibit the practice for those
56 products. If such problems arise substantially in other domains, we
57 stand ready to extend this provision to those domains in future versions
58 of the GPL, as needed to protect the freedom of users.
59
60 Finally, every program is threatened constantly by software patents.
61 States should not allow patents to restrict development and use of
62 software on general-purpose computers, but in those that do, we wish to
63 avoid the special danger that patents applied to a free program could
64 make it effectively proprietary. To prevent this, the GPL assures that
65 patents cannot be used to render the program non-free.
66
67 The precise terms and conditions for copying, distribution and
68 modification follow.
69
70 TERMS AND CONDITIONS
71
72 0. Definitions.
73
74 "This License" refers to version 3 of the GNU General Public License.
75
76 "Copyright" also means copyright-like laws that apply to other kinds of
77 works, such as semiconductor masks.
78
79 "The Program" refers to any copyrightable work licensed under this
80 License. Each licensee is addressed as "you". "Licensees" and
81 "recipients" may be individuals or organizations.
82
83 To "modify" a work means to copy from or adapt all or part of the work
84 in a fashion requiring copyright permission, other than the making of an
85 exact copy. The resulting work is called a "modified version" of the
86 earlier work or a work "based on" the earlier work.
87
88 A "covered work" means either the unmodified Program or a work based
89 on the Program.
90
91 To "propagate" a work means to do anything with it that, without
92 permission, would make you directly or secondarily liable for
93 infringement under applicable copyright law, except executing it on a
94 computer or modifying a private copy. Propagation includes copying,
95 distribution (with or without modification), making available to the
96 public, and in some countries other activities as well.
97
98 To "convey" a work means any kind of propagation that enables other
99 parties to make or receive copies. Mere interaction with a user through
100 a computer network, with no transfer of a copy, is not conveying.
101
102 An interactive user interface displays "Appropriate Legal Notices"
103 to the extent that it includes a convenient and prominently visible
104 feature that (1) displays an appropriate copyright notice, and (2)
105 tells the user that there is no warranty for the work (except to the
106 extent that warranties are provided), that licensees may convey the
107 work under this License, and how to view a copy of this License. If
108 the interface presents a list of user commands or options, such as a
109 menu, a prominent item in the list meets this criterion.
110
111 1. Source Code.
112
113 The "source code" for a work means the preferred form of the work
114 for making modifications to it. "Object code" means any non-source
115 form of a work.
116
117 A "Standard Interface" means an interface that either is an official
118 standard defined by a recognized standards body, or, in the case of
119 interfaces specified for a particular programming language, one that
120 is widely used among developers working in that language.
121
122 The "System Libraries" of an executable work include anything, other
123 than the work as a whole, that (a) is included in the normal form of
124 packaging a Major Component, but which is not part of that Major
125 Component, and (b) serves only to enable use of the work with that
126 Major Component, or to implement a Standard Interface for which an
127 implementation is available to the public in source code form. A
128 "Major Component", in this context, means a major essential component
129 (kernel, window system, and so on) of the specific operating system
130 (if any) on which the executable work runs, or a compiler used to
131 produce the work, or an object code interpreter used to run it.
132
133 The "Corresponding Source" for a work in object code form means all
134 the source code needed to generate, install, and (for an executable
135 work) run the object code and to modify the work, including scripts to
136 control those activities. However, it does not include the work's
137 System Libraries, or general-purpose tools or generally available free
138 programs which are used unmodified in performing those activities but
139 which are not part of the work. For example, Corresponding Source
140 includes interface definition files associated with source files for
141 the work, and the source code for shared libraries and dynamically
142 linked subprograms that the work is specifically designed to require,
143 such as by intimate data communication or control flow between those
144 subprograms and other parts of the work.
145
146 The Corresponding Source need not include anything that users
147 can regenerate automatically from other parts of the Corresponding
148 Source.
149
150 The Corresponding Source for a work in source code form is that
151 same work.
152
153 2. Basic Permissions.
154
155 All rights granted under this License are granted for the term of
156 copyright on the Program, and are irrevocable provided the stated
157 conditions are met. This License explicitly affirms your unlimited
158 permission to run the unmodified Program. The output from running a
159 covered work is covered by this License only if the output, given its
160 content, constitutes a covered work. This License acknowledges your
161 rights of fair use or other equivalent, as provided by copyright law.
162
163 You may make, run and propagate covered works that you do not
164 convey, without conditions so long as your license otherwise remains
165 in force. You may convey covered works to others for the sole purpose
166 of having them make modifications exclusively for you, or provide you
167 with facilities for running those works, provided that you comply with
168 the terms of this License in conveying all material for which you do
169 not control copyright. Those thus making or running the covered works
170 for you must do so exclusively on your behalf, under your direction
171 and control, on terms that prohibit them from making any copies of
172 your copyrighted material outside their relationship with you.
173
174 Conveying under any other circumstances is permitted solely under
175 the conditions stated below. Sublicensing is not allowed; section 10
176 makes it unnecessary.
177
178 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
179
180 No covered work shall be deemed part of an effective technological
181 measure under any applicable law fulfilling obligations under article
182 11 of the WIPO copyright treaty adopted on 20 December 1996, or
183 similar laws prohibiting or restricting circumvention of such
184 measures.
185
186 When you convey a covered work, you waive any legal power to forbid
187 circumvention of technological measures to the extent such circumvention
188 is effected by exercising rights under this License with respect to
189 the covered work, and you disclaim any intention to limit operation or
190 modification of the work as a means of enforcing, against the work's
191 users, your or third parties' legal rights to forbid circumvention of
192 technological measures.
193
194 4. Conveying Verbatim Copies.
195
196 You may convey verbatim copies of the Program's source code as you
197 receive it, in any medium, provided that you conspicuously and
198 appropriately publish on each copy an appropriate copyright notice;
199 keep intact all notices stating that this License and any
200 non-permissive terms added in accord with section 7 apply to the code;
201 keep intact all notices of the absence of any warranty; and give all
202 recipients a copy of this License along with the Program.
203
204 You may charge any price or no price for each copy that you convey,
205 and you may offer support or warranty protection for a fee.
206
207 5. Conveying Modified Source Versions.
208
209 You may convey a work based on the Program, or the modifications to
210 produce it from the Program, in the form of source code under the
211 terms of section 4, provided that you also meet all of these conditions:
212
213 a) The work must carry prominent notices stating that you modified
214 it, and giving a relevant date.
215
216 b) The work must carry prominent notices stating that it is
217 released under this License and any conditions added under section
218 7. This requirement modifies the requirement in section 4 to
219 "keep intact all notices".
220
221 c) You must license the entire work, as a whole, under this
222 License to anyone who comes into possession of a copy. This
223 License will therefore apply, along with any applicable section 7
224 additional terms, to the whole of the work, and all its parts,
225 regardless of how they are packaged. This License gives no
226 permission to license the work in any other way, but it does not
227 invalidate such permission if you have separately received it.
228
229 d) If the work has interactive user interfaces, each must display
230 Appropriate Legal Notices; however, if the Program has interactive
231 interfaces that do not display Appropriate Legal Notices, your
232 work need not make them do so.
233
234 A compilation of a covered work with other separate and independent
235 works, which are not by their nature extensions of the covered work,
236 and which are not combined with it such as to form a larger program,
237 in or on a volume of a storage or distribution medium, is called an
238 "aggregate" if the compilation and its resulting copyright are not
239 used to limit the access or legal rights of the compilation's users
240 beyond what the individual works permit. Inclusion of a covered work
241 in an aggregate does not cause this License to apply to the other
242 parts of the aggregate.
243
244 6. Conveying Non-Source Forms.
245
246 You may convey a covered work in object code form under the terms
247 of sections 4 and 5, provided that you also convey the
248 machine-readable Corresponding Source under the terms of this License,
249 in one of these ways:
250
251 a) Convey the object code in, or embodied in, a physical product
252 (including a physical distribution medium), accompanied by the
253 Corresponding Source fixed on a durable physical medium
254 customarily used for software interchange.
255
256 b) Convey the object code in, or embodied in, a physical product
257 (including a physical distribution medium), accompanied by a
258 written offer, valid for at least three years and valid for as
259 long as you offer spare parts or customer support for that product
260 model, to give anyone who possesses the object code either (1) a
261 copy of the Corresponding Source for all the software in the
262 product that is covered by this License, on a durable physical
263 medium customarily used for software interchange, for a price no
264 more than your reasonable cost of physically performing this
265 conveying of source, or (2) access to copy the
266 Corresponding Source from a network server at no charge.
267
268 c) Convey individual copies of the object code with a copy of the
269 written offer to provide the Corresponding Source. This
270 alternative is allowed only occasionally and noncommercially, and
271 only if you received the object code with such an offer, in accord
272 with subsection 6b.
273
274 d) Convey the object code by offering access from a designated
275 place (gratis or for a charge), and offer equivalent access to the
276 Corresponding Source in the same way through the same place at no
277 further charge. You need not require recipients to copy the
278 Corresponding Source along with the object code. If the place to
279 copy the object code is a network server, the Corresponding Source
280 may be on a different server (operated by you or a third party)
281 that supports equivalent copying facilities, provided you maintain
282 clear directions next to the object code saying where to find the
283 Corresponding Source. Regardless of what server hosts the
284 Corresponding Source, you remain obligated to ensure that it is
285 available for as long as needed to satisfy these requirements.
286
287 e) Convey the object code using peer-to-peer transmission, provided
288 you inform other peers where the object code and Corresponding
289 Source of the work are being offered to the general public at no
290 charge under subsection 6d.
291
292 A separable portion of the object code, whose source code is excluded
293 from the Corresponding Source as a System Library, need not be
294 included in conveying the object code work.
295
296 A "User Product" is either (1) a "consumer product", which means any
297 tangible personal property which is normally used for personal, family,
298 or household purposes, or (2) anything designed or sold for incorporation
299 into a dwelling. In determining whether a product is a consumer product,
300 doubtful cases shall be resolved in favor of coverage. For a particular
301 product received by a particular user, "normally used" refers to a
302 typical or common use of that class of product, regardless of the status
303 of the particular user or of the way in which the particular user
304 actually uses, or expects or is expected to use, the product. A product
305 is a consumer product regardless of whether the product has substantial
306 commercial, industrial or non-consumer uses, unless such uses represent
307 the only significant mode of use of the product.
308
309 "Installation Information" for a User Product means any methods,
310 procedures, authorization keys, or other information required to install
311 and execute modified versions of a covered work in that User Product from
312 a modified version of its Corresponding Source. The information must
313 suffice to ensure that the continued functioning of the modified object
314 code is in no case prevented or interfered with solely because
315 modification has been made.
316
317 If you convey an object code work under this section in, or with, or
318 specifically for use in, a User Product, and the conveying occurs as
319 part of a transaction in which the right of possession and use of the
320 User Product is transferred to the recipient in perpetuity or for a
321 fixed term (regardless of how the transaction is characterized), the
322 Corresponding Source conveyed under this section must be accompanied
323 by the Installation Information. But this requirement does not apply
324 if neither you nor any third party retains the ability to install
325 modified object code on the User Product (for example, the work has
326 been installed in ROM).
327
328 The requirement to provide Installation Information does not include a
329 requirement to continue to provide support service, warranty, or updates
330 for a work that has been modified or installed by the recipient, or for
331 the User Product in which it has been modified or installed. Access to a
332 network may be denied when the modification itself materially and
333 adversely affects the operation of the network or violates the rules and
334 protocols for communication across the network.
335
336 Corresponding Source conveyed, and Installation Information provided,
337 in accord with this section must be in a format that is publicly
338 documented (and with an implementation available to the public in
339 source code form), and must require no special password or key for
340 unpacking, reading or copying.
341
342 7. Additional Terms.
343
344 "Additional permissions" are terms that supplement the terms of this
345 License by making exceptions from one or more of its conditions.
346 Additional permissions that are applicable to the entire Program shall
347 be treated as though they were included in this License, to the extent
348 that they are valid under applicable law. If additional permissions
349 apply only to part of the Program, that part may be used separately
350 under those permissions, but the entire Program remains governed by
351 this License without regard to the additional permissions.
352
353 When you convey a copy of a covered work, you may at your option
354 remove any additional permissions from that copy, or from any part of
355 it. (Additional permissions may be written to require their own
356 removal in certain cases when you modify the work.) You may place
357 additional permissions on material, added by you to a covered work,
358 for which you have or can give appropriate copyright permission.
359
360 Notwithstanding any other provision of this License, for material you
361 add to a covered work, you may (if authorized by the copyright holders of
362 that material) supplement the terms of this License with terms:
363
364 a) Disclaiming warranty or limiting liability differently from the
365 terms of sections 15 and 16 of this License; or
366
367 b) Requiring preservation of specified reasonable legal notices or
368 author attributions in that material or in the Appropriate Legal
369 Notices displayed by works containing it; or
370
371 c) Prohibiting misrepresentation of the origin of that material, or
372 requiring that modified versions of such material be marked in
373 reasonable ways as different from the original version; or
374
375 d) Limiting the use for publicity purposes of names of licensors or
376 authors of the material; or
377
378 e) Declining to grant rights under trademark law for use of some
379 trade names, trademarks, or service marks; or
380
381 f) Requiring indemnification of licensors and authors of that
382 material by anyone who conveys the material (or modified versions of
383 it) with contractual assumptions of liability to the recipient, for
384 any liability that these contractual assumptions directly impose on
385 those licensors and authors.
386
387 All other non-permissive additional terms are considered "further
388 restrictions" within the meaning of section 10. If the Program as you
389 received it, or any part of it, contains a notice stating that it is
390 governed by this License along with a term that is a further
391 restriction, you may remove that term. If a license document contains
392 a further restriction but permits relicensing or conveying under this
393 License, you may add to a covered work material governed by the terms
394 of that license document, provided that the further restriction does
395 not survive such relicensing or conveying.
396
397 If you add terms to a covered work in accord with this section, you
398 must place, in the relevant source files, a statement of the
399 additional terms that apply to those files, or a notice indicating
400 where to find the applicable terms.
401
402 Additional terms, permissive or non-permissive, may be stated in the
403 form of a separately written license, or stated as exceptions;
404 the above requirements apply either way.
405
406 8. Termination.
407
408 You may not propagate or modify a covered work except as expressly
409 provided under this License. Any attempt otherwise to propagate or
410 modify it is void, and will automatically terminate your rights under
411 this License (including any patent licenses granted under the third
412 paragraph of section 11).
413
414 However, if you cease all violation of this License, then your
415 license from a particular copyright holder is reinstated (a)
416 provisionally, unless and until the copyright holder explicitly and
417 finally terminates your license, and (b) permanently, if the copyright
418 holder fails to notify you of the violation by some reasonable means
419 prior to 60 days after the cessation.
420
421 Moreover, your license from a particular copyright holder is
422 reinstated permanently if the copyright holder notifies you of the
423 violation by some reasonable means, this is the first time you have
424 received notice of violation of this License (for any work) from that
425 copyright holder, and you cure the violation prior to 30 days after
426 your receipt of the notice.
427
428 Termination of your rights under this section does not terminate the
429 licenses of parties who have received copies or rights from you under
430 this License. If your rights have been terminated and not permanently
431 reinstated, you do not qualify to receive new licenses for the same
432 material under section 10.
433
434 9. Acceptance Not Required for Having Copies.
435
436 You are not required to accept this License in order to receive or
437 run a copy of the Program. Ancillary propagation of a covered work
438 occurring solely as a consequence of using peer-to-peer transmission
439 to receive a copy likewise does not require acceptance. However,
440 nothing other than this License grants you permission to propagate or
441 modify any covered work. These actions infringe copyright if you do
442 not accept this License. Therefore, by modifying or propagating a
443 covered work, you indicate your acceptance of this License to do so.
444
445 10. Automatic Licensing of Downstream Recipients.
446
447 Each time you convey a covered work, the recipient automatically
448 receives a license from the original licensors, to run, modify and
449 propagate that work, subject to this License. You are not responsible
450 for enforcing compliance by third parties with this License.
451
452 An "entity transaction" is a transaction transferring control of an
453 organization, or substantially all assets of one, or subdividing an
454 organization, or merging organizations. If propagation of a covered
455 work results from an entity transaction, each party to that
456 transaction who receives a copy of the work also receives whatever
457 licenses to the work the party's predecessor in interest had or could
458 give under the previous paragraph, plus a right to possession of the
459 Corresponding Source of the work from the predecessor in interest, if
460 the predecessor has it or can get it with reasonable efforts.
461
462 You may not impose any further restrictions on the exercise of the
463 rights granted or affirmed under this License. For example, you may
464 not impose a license fee, royalty, or other charge for exercise of
465 rights granted under this License, and you may not initiate litigation
466 (including a cross-claim or counterclaim in a lawsuit) alleging that
467 any patent claim is infringed by making, using, selling, offering for
468 sale, or importing the Program or any portion of it.
469
470 11. Patents.
471
472 A "contributor" is a copyright holder who authorizes use under this
473 License of the Program or a work on which the Program is based. The
474 work thus licensed is called the contributor's "contributor version".
475
476 A contributor's "essential patent claims" are all patent claims
477 owned or controlled by the contributor, whether already acquired or
478 hereafter acquired, that would be infringed by some manner, permitted
479 by this License, of making, using, or selling its contributor version,
480 but do not include claims that would be infringed only as a
481 consequence of further modification of the contributor version. For
482 purposes of this definition, "control" includes the right to grant
483 patent sublicenses in a manner consistent with the requirements of
484 this License.
485
486 Each contributor grants you a non-exclusive, worldwide, royalty-free
487 patent license under the contributor's essential patent claims, to
488 make, use, sell, offer for sale, import and otherwise run, modify and
489 propagate the contents of its contributor version.
490
491 In the following three paragraphs, a "patent license" is any express
492 agreement or commitment, however denominated, not to enforce a patent
493 (such as an express permission to practice a patent or covenant not to
494 sue for patent infringement). To "grant" such a patent license to a
495 party means to make such an agreement or commitment not to enforce a
496 patent against the party.
497
498 If you convey a covered work, knowingly relying on a patent license,
499 and the Corresponding Source of the work is not available for anyone
500 to copy, free of charge and under the terms of this License, through a
501 publicly available network server or other readily accessible means,
502 then you must either (1) cause the Corresponding Source to be so
503 available, or (2) arrange to deprive yourself of the benefit of the
504 patent license for this particular work, or (3) arrange, in a manner
505 consistent with the requirements of this License, to extend the patent
506 license to downstream recipients. "Knowingly relying" means you have
507 actual knowledge that, but for the patent license, your conveying the
508 covered work in a country, or your recipient's use of the covered work
509 in a country, would infringe one or more identifiable patents in that
510 country that you have reason to believe are valid.
511
512 If, pursuant to or in connection with a single transaction or
513 arrangement, you convey, or propagate by procuring conveyance of, a
514 covered work, and grant a patent license to some of the parties
515 receiving the covered work authorizing them to use, propagate, modify
516 or convey a specific copy of the covered work, then the patent license
517 you grant is automatically extended to all recipients of the covered
518 work and works based on it.
519
520 A patent license is "discriminatory" if it does not include within
521 the scope of its coverage, prohibits the exercise of, or is
522 conditioned on the non-exercise of one or more of the rights that are
523 specifically granted under this License. You may not convey a covered
524 work if you are a party to an arrangement with a third party that is
525 in the business of distributing software, under which you make payment
526 to the third party based on the extent of your activity of conveying
527 the work, and under which the third party grants, to any of the
528 parties who would receive the covered work from you, a discriminatory
529 patent license (a) in connection with copies of the covered work
530 conveyed by you (or copies made from those copies), or (b) primarily
531 for and in connection with specific products or compilations that
532 contain the covered work, unless you entered into that arrangement,
533 or that patent license was granted, prior to 28 March 2007.
534
535 Nothing in this License shall be construed as excluding or limiting
536 any implied license or other defenses to infringement that may
537 otherwise be available to you under applicable patent law.
538
539 12. No Surrender of Others' Freedom.
540
541 If conditions are imposed on you (whether by court order, agreement or
542 otherwise) that contradict the conditions of this License, they do not
543 excuse you from the conditions of this License. If you cannot convey a
544 covered work so as to satisfy simultaneously your obligations under this
545 License and any other pertinent obligations, then as a consequence you may
546 not convey it at all. For example, if you agree to terms that obligate you
547 to collect a royalty for further conveying from those to whom you convey
548 the Program, the only way you could satisfy both those terms and this
549 License would be to refrain entirely from conveying the Program.
550
551 13. Use with the GNU Affero General Public License.
552
553 Notwithstanding any other provision of this License, you have
554 permission to link or combine any covered work with a work licensed
555 under version 3 of the GNU Affero General Public License into a single
556 combined work, and to convey the resulting work. The terms of this
557 License will continue to apply to the part which is the covered work,
558 but the special requirements of the GNU Affero General Public License,
559 section 13, concerning interaction through a network will apply to the
560 combination as such.
561
562 14. Revised Versions of this License.
563
564 The Free Software Foundation may publish revised and/or new versions of
565 the GNU General Public License from time to time. Such new versions will
566 be similar in spirit to the present version, but may differ in detail to
567 address new problems or concerns.
568
569 Each version is given a distinguishing version number. If the
570 Program specifies that a certain numbered version of the GNU General
571 Public License "or any later version" applies to it, you have the
572 option of following the terms and conditions either of that numbered
573 version or of any later version published by the Free Software
574 Foundation. If the Program does not specify a version number of the
575 GNU General Public License, you may choose any version ever published
576 by the Free Software Foundation.
577
578 If the Program specifies that a proxy can decide which future
579 versions of the GNU General Public License can be used, that proxy's
580 public statement of acceptance of a version permanently authorizes you
581 to choose that version for the Program.
582
583 Later license versions may give you additional or different
584 permissions. However, no additional obligations are imposed on any
585 author or copyright holder as a result of your choosing to follow a
586 later version.
587
588 15. Disclaimer of Warranty.
589
590 THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
591 APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
592 HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
593 OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
594 THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
595 PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
596 IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
597 ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
598
599 16. Limitation of Liability.
600
601 IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
602 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
603 THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
604 GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
605 USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
606 DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
607 PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
608 EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
609 SUCH DAMAGES.
610
611 17. Interpretation of Sections 15 and 16.
612
613 If the disclaimer of warranty and limitation of liability provided
614 above cannot be given local legal effect according to their terms,
615 reviewing courts shall apply local law that most closely approximates
616 an absolute waiver of all civil liability in connection with the
617 Program, unless a warranty or assumption of liability accompanies a
618 copy of the Program in return for a fee.
619
620 END OF TERMS AND CONDITIONS
621
622 How to Apply These Terms to Your New Programs
623
624 If you develop a new program, and you want it to be of the greatest
625 possible use to the public, the best way to achieve this is to make it
626 free software which everyone can redistribute and change under these terms.
627
628 To do so, attach the following notices to the program. It is safest
629 to attach them to the start of each source file to most effectively
630 state the exclusion of warranty; and each file should have at least
631 the "copyright" line and a pointer to where the full notice is found.
632
633 {one line to give the program's name and a brief idea of what it does.}
634 Copyright (C) {year} {name of author}
635
636 This program is free software: you can redistribute it and/or modify
637 it under the terms of the GNU General Public License as published by
638 the Free Software Foundation, either version 3 of the License, or
639 (at your option) any later version.
640
641 This program is distributed in the hope that it will be useful,
642 but WITHOUT ANY WARRANTY; without even the implied warranty of
643 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
644 GNU General Public License for more details.
645
646 You should have received a copy of the GNU General Public License
647 along with this program. If not, see <http://www.gnu.org/licenses/>.
648
649 Also add information on how to contact you by electronic and paper mail.
650
651 If the program does terminal interaction, make it output a short
652 notice like this when it starts in an interactive mode:
653
654 {project} Copyright (C) {year} {fullname}
655 This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
656 This is free software, and you are welcome to redistribute it
657 under certain conditions; type `show c' for details.
658
659 The hypothetical commands `show w' and `show c' should show the appropriate
660 parts of the General Public License. Of course, your program's commands
661 might be different; for a GUI interface, you would use an "about box".
662
663 You should also get your employer (if you work as a programmer) or school,
664 if any, to sign a "copyright disclaimer" for the program, if necessary.
665 For more information on this, and how to apply and follow the GNU GPL, see
666 <http://www.gnu.org/licenses/>.
667
668 The GNU General Public License does not permit incorporating your program
669 into proprietary programs. If your program is a subroutine library, you
670 may consider it more useful to permit linking proprietary applications with
671 the library. If this is what you want to do, use the GNU Lesser General
672 Public License instead of this License. But first, please read
673 <http://www.gnu.org/philosophy/why-not-lgpl.html>.
+184
-0
README.md less more
0 # CTDConverter
1 Given one or more CTD files, `CTD2Converter` generates the needed wrappers to include them in workflow engines, such as Galaxy and CWL.
2
3 ## Dependencies
4 `CTDConverter` has the following python dependencies:
5
6 - [CTDopts]
7 - `lxml`
8 - `ruamel.yaml`
9
10 ### Installing Dependencies
11 We recommend the use of `conda` to manage all dependencies. If you're not sure what `conda` is, make sure to read the [using-conda](conda documentation).
12
13 The easiest way to get you started with CTD conversion is to create a `conda` environment on which you'll install all dependencies. Using environments in `conda` allows you to have parallel, independent python environments, thus avoiding conflicts between libraries. If you haven't installed `conda`, check [conda-install](conda's installation guide).
14
15 Once you've installed `conda`, create an environment named `ctd-converter`, like so:
16
17 ```sh
18 $ conda create --name ctd-converter
19 ```
20
21 You will now need to *activate* the environment by executing the following command:
22
23 ```sh
24 $ source activate ctd-converter
25 ```
26
27 Install the required dependencies as follows (the order of execution **is actually important**, due to transitive dependencies):
28
29 ```sh
30 $ conda install --channel workflowconversion ctdopts
31 $ conda install lxml
32 $ conda install --channel conda-forge ruamel.yaml
33 $ conda install libxml2=2.9.2
34 ```
35
36 `lxml` depends on `libxml2`. When you install `lxml` you'll get the latest version of `libxml2` (2.9.4) by default. You would usually want the latest version, but there is, however, a bug in validating XML files against a schema in this version of `libxml2`.
37
38 If you require validation of input CTDs against a schema (which we recommend), you will need to downgrade to the latest known version of `libxml2` that works, namely, 2.9.2.
39
40 You could just download dependencies manually and make them available through your `PYTHONPATH` environment variable, if you're into that. To get more information about how to install python modules without using `conda`, visit: https://docs.python.org/2/install/.
41
42 ## How to install `CTDConverter`
43 `CTDConverter` is not a python module, rather, a series of scripts, so installing it is as easy as downloading the source code from https://github.com/genericworkflownodes/CTDConverter. Once you've installed all dependencies, downloaded `CTDConverter` and activated your `conda` environment, you're good to go.
44
45 ## Usage
46 The first thing that you need to tell `CTDConverter` is the output format of the converted wrappers. `CTDConverter` supports conversion of CTDs into Galaxy and CWL. Invoking it is as simple as follows:
47
48 $ python convert.py [FORMAT] [ADDITIONAL_PARAMETERS ...]
49
50 Here `[FORMAT]` can be any of the supported formats (i.e., `cwl`, `galaxy`). `CTDConverter` offers a series of format-specific scripts and we've designed these scripts to behave *somewhat* similarly. All converter scripts have the same core functionality, that is, read CTD files, parse them using [CTDopts], validate against a schema, etc. Of course, each converter script might add extra functionality that is not present in other engines. Only the Galaxy converter script supports generation of a `tool_conf.xml` file, for instance.
51
52 The following sections in this file describe the parameters that all converter scripts share.
53
54 Please refer to the detailed documentation for each of the converters for more information:
55
56 - [Generation of Galaxy ToolConfig files](galaxy/README.md)
57 - [Generation of CWL task files](cwl/README.md)
58
59 ## Fail Policy while processing several Files
60 `CTDConverter` can parse several CTDs and convert them. However, the process will be interrupted and an error code will be returned at the first encountered error (e.g., a CTD is not valid, there are missing support files, etc.).
61
62 ## Converting a single CTD
63 In its simplest form, the converter takes an input CTD file and generates an output file. The following usage of `CTDConverter`:
64
65 $ python convert.py [FORMAT] -i /data/sample_input.ctd -o /data/sample_output.xml
66
67 will parse `/data/sample_input.ctd` and generate an appropriate converted file under `/data/sample_output.xml`. The generated file can be added to your workflow engine as usual.
68
69 ## Converting several CTDs
70 When converting several CTDs, the expected value for the `-o`/`--output` parameter is a folder. For example:
71
72 $ python convert.py [FORMAT] -i /data/ctds/one.ctd /data/ctds/two.ctd -o /data/converted-files
73
74 Will convert `/data/ctds/one.ctd` into `/data/converted-files/one.[EXT]` and `/data/ctds/two.ctd` into `/data/converted-files/two.[EXT]`. Each converter has a preferred extension, here shown as a variable (`[EXT]`). Galaxy prefers `xml`, while CWL prefers `cwl`.
75
76 You can use wildcard expansion, as supported by most modern operating systems:
77
78 $ python convert.py [FORMAT] -i /data/ctds/*.ctd -o /data/converted-files
79
80 ## Common Parameters
81 ### Input File(s)
82 * Purpose: Provide input CTD file(s) to convert.
83 * Short/long version: `-i` / `--input`
84 * Required: yes.
85 * Taken values: a list of input CTD files.
86
87 Examples:
88
89 Any of the following invocations will convert `/data/input_one.ctd` and `/data/input_two.ctd`:
90
91 $ python convert.py [FORMAT] -i /data/input_one.ctd -i /data/input_two.ctd -o /data/generated
92 $ python convert.py [FORMAT] -i /data/input_one.ctd /data/input_two.ctd -o /data/generated
93 $ python convert.py [FORMAT] --input /data/input_one.ctd /data/input_two.ctd -o /data/generated
94 $ python convert.py [FORMAT] --input /data/input_one.ctd --input /data/input_two.ctd -o /data/generated
95
96 The following invocation will convert `/data/input.ctd` into `/data/output.xml`:
97
98 $ python convert.py [FORMAT] -i /data/input.ctd -o /data/output.xml
99
100 Of course, you can also use wildcards, which will be automatically expanded by any modern operating system. This is extremely useful if you want to convert several files at a time. Let's assume that the folder `/data/ctds` contains three files: `input_one.ctd`, `input_two.ctd` and `input_three.ctd`. The following two invocations will produce the same output in the `/data/wrappers` folder:
101
102 $ python convert.py [FORMAT] -i /data/input_one.ctd /data/input_two.ctd /data/input_three.ctd -o /data/wrappers
103 $ python convert.py [FORMAT] -i /data/*.ctd -o /data/wrappers
104
105 ### Output Destination
106 * Purpose: Provide output destination for the converted wrapper files.
107 * Short/long version: `-o` / `--output-destination`
108 * Required: yes.
109 * Taken values: if a single input file is given, then a single output file is expected. If multiple input files are given, then an existent folder in which all converted CTDs will be written is expected.
110
111 Examples:
112
113 A single input is given, and the output will be generated into `/data/output.xml`:
114
115 $ python convert.py [FORMAT] -i /data/input.ctd -o /data/output.xml
116
117 Several inputs are given. The output is the already existent folder, `/data/wrappers`, and at the end of the operation, the files `/data/wrappers/input_one.[EXT]` and `/data/wrappers/input_two.[EXT]` will be generated:
118
119 $ python convert.py [FORMAT] -i /data/ctds/input_one.ctd /data/ctds/input_two.ctd -o /data/stubs
120
121 Please note that the output file name is **not** taken from the name of the input file, rather from the name of the tool, that is, from the `name` attribute in the `<tool>` element in its corresponding CTD. By convention, the name of the CTD file and the name of the tool match.
122
123 ### Blacklisting Parameters
124 * Purpose: Some parameters present in the CTD are not to be exposed on the output files. Think of parameters such as `--help`, `--debug` that might won't make much sense to be exposed to final users in a workflow management system.
125 * Short/long version: `-b` / `--blacklist-parameters`
126 * Required: no.
127 * Taken values: A list of parameters to be blacklisted.
128
129 Example:
130
131 $ pythonconvert.py [FORMAT] ... -b h help quiet
132
133 In this case, `CTDConverter` will not process any of the parameters named `h`, `help`, or `quiet`, that is, they will not appear in the generated output files.
134
135 ### Schema Validation
136 * Purpose: Provide validation of input CTDs against a schema file (i.e, a XSD file).
137 * Short/long version: `-V` / `--validation-schema`
138 * Required: no.
139 * Taken values: location of the schema file (e.g., CTD.xsd).
140
141 CTDs can be validated against a schema. The master version of the schema can be found on [CTDSchema].
142
143 If a schema is provided, all input CTDs will be validated against it.
144
145 **NOTE:** Please make sure to read the [section on issues with schema validation](#issues-with-libxml2-and-schema-validation) if you require validation of CTDs against a schema.
146
147 ### Hardcoding Parameters
148 * Purpose: Fixing the value of a parameter and hide it from the end user.
149 * Short/long version: `-p` / `--hardcoded-parameters`
150 * Required: no.
151 * Taken values: The path of a file containing the mapping between parameter names and hardcoded values to use.
152
153 It is sometimes required that parameters are hidden from the end user in workflow systems and that they take a predetermined, fixed value. Allowing end users to control parameters similar to `--verbosity`, `--threads`, etc., might create more problems than solving them. For this purpose, the parameter `-p`/`--hardcoded-parameters` takes the path of a file that contains up to three columns separated by whitespace that map parameter names to the hardcoded value. The first column contains the name of the parameter and the second one the hardcoded value. Only the first two columns are mandatory.
154
155 If the parameter is to be hardcoded only for certain tools, a third column containing a comma separated list of tool names for which the hardcoding will apply can be added.
156
157 Lines starting with `#` will be ignored. The following is an example of a valid file:
158
159 # Parameter name # Value # Tool(s)
160 threads 8
161 mode quiet
162 xtandem_executable xtandem XTandemAdapter
163 verbosity high Foo, Bar
164
165 The parameters `threads` and `mode` will be set to `8` and `quiet`, respectively, for all parsed CTDs. However, the `xtandem_executable` parameter will be set to `xtandem` only for the `XTandemAdapter` tool. Similarly, the parameter `verbosity` will be set to `high` for the `Foo` and `Bar` tools only.
166
167 ### Providing a default executable Path
168 * Purpose: Help workflow engines locate tools by providing a path.
169 * Short/long version: `-x` / `--default-executable-path`
170 * Required: no.
171 * Taken values: The default executable path of the tools in the target workflow engine.
172
173 CTDs can contain an `<executablePath>` element that will be used when executing the tool binary. If this element is missing, the value provided by this parameter will be used as a prefix when building the appropriate sections in the output files.
174
175 The following invocation of the converter will use `/opt/suite/bin` as a prefix when providing the executable path in the output files for any input CTD that lacks the `<executablePath>` section:
176
177 $ python convert.py [FORMAT] -x /opt/suite/bin ...
178
179
180 [CTDopts]: https://github.com/genericworkflownodes/CTDopts
181 [CTDSchema]: https://github.com/WorkflowConversion/CTDSchema
182 [conda-install]: https://conda.io/docs/install/quick.html
183 [using-conda]: https://conda.io/docs/using/envs.html⏎
+0
-0
common/__init__.py less more
(New empty file)
+45
-0
common/exceptions.py less more
0 #!/usr/bin/env python
1 # encoding: utf-8
2
3 """
4 @author: delagarza
5 """
6
7 from CTDopts.CTDopts import ModelError
8
9
10 class CLIError(Exception):
11 # Generic exception to raise and log different fatal errors.
12 def __init__(self, msg):
13 super(CLIError).__init__(type(self))
14 self.msg = "E: %s" % msg
15
16 def __str__(self):
17 return self.msg
18
19 def __unicode__(self):
20 return self.msg
21
22
23 class InvalidModelException(ModelError):
24 def __init__(self, message):
25 super(InvalidModelException, self).__init__()
26 self.message = message
27
28 def __str__(self):
29 return self.message
30
31 def __repr__(self):
32 return self.message
33
34
35 class ApplicationException(Exception):
36 def __init__(self, msg):
37 super(ApplicationException).__init__(type(self))
38 self.msg = msg
39
40 def __str__(self):
41 return self.msg
42
43 def __unicode__(self):
44 return self.msg⏎
+23
-0
common/logger.py less more
0 #!/usr/bin/env python
1 # encoding: utf-8
2 import sys
3
4 MESSAGE_INDENTATION_INCREMENT = 2
5
6
7 def _get_indented_text(text, indentation_level):
8 return ("%(indentation)s%(text)s" %
9 {"indentation": " " * (MESSAGE_INDENTATION_INCREMENT * indentation_level),
10 "text": text})
11
12
13 def warning(warning_text, indentation_level=0):
14 sys.stdout.write(_get_indented_text("WARNING: %s\n" % warning_text, indentation_level))
15
16
17 def error(error_text, indentation_level=0):
18 sys.stderr.write(_get_indented_text("ERROR: %s\n" % error_text, indentation_level))
19
20
21 def info(info_text, indentation_level=0):
22 sys.stdout.write(_get_indented_text("INFO: %s\n" % info_text, indentation_level))
+318
-0
common/utils.py less more
0 #!/usr/bin/env python
1 # encoding: utf-8
2 import ntpath
3 import os
4
5 from lxml import etree
6 from string import strip
7 from logger import info, error, warning
8
9 from common.exceptions import ApplicationException
10 from CTDopts.CTDopts import CTDModel, ParameterGroup
11
12
13 MESSAGE_INDENTATION_INCREMENT = 2
14
15
16 # simple struct-class containing a tuple with input/output location and the in-memory CTDModel
17 class ParsedCTD:
18 def __init__(self, ctd_model=None, input_file=None, suggested_output_file=None):
19 self.ctd_model = ctd_model
20 self.input_file = input_file
21 self.suggested_output_file = suggested_output_file
22
23
24 class ParameterHardcoder:
25 def __init__(self):
26 # map whose keys are the composite names of tools and parameters in the following pattern:
27 # [ToolName][separator][ParameterName] -> HardcodedValue
28 # if the parameter applies to all tools, then the following pattern is used:
29 # [ParameterName] -> HardcodedValue
30
31 # examples (assuming separator is '#'):
32 # threads -> 24
33 # XtandemAdapter#adapter -> xtandem.exe
34 # adapter -> adapter.exe
35 self.separator = "!"
36 self.parameter_map = {}
37
38 # the most specific value will be returned in case of overlap
39 def get_hardcoded_value(self, parameter_name, tool_name):
40 # look for the value that would apply for all tools
41 generic_value = self.parameter_map.get(parameter_name, None)
42 specific_value = self.parameter_map.get(self.build_key(parameter_name, tool_name), None)
43 if specific_value is not None:
44 return specific_value
45
46 return generic_value
47
48 def register_parameter(self, parameter_name, parameter_value, tool_name=None):
49 self.parameter_map[self.build_key(parameter_name, tool_name)] = parameter_value
50
51 def build_key(self, parameter_name, tool_name):
52 if tool_name is None:
53 return parameter_name
54 return "%s%s%s" % (parameter_name, self.separator, tool_name)
55
56
57 def validate_path_exists(path):
58 if not os.path.isfile(path) or not os.path.exists(path):
59 raise ApplicationException("The provided path (%s) does not exist or is not a valid file path." % path)
60
61
62 def validate_argument_is_directory(args, argument_name):
63 file_name = getattr(args, argument_name)
64 if file_name is not None and os.path.isdir(file_name):
65 raise ApplicationException("The provided output file name (%s) points to a directory." % file_name)
66
67
68 def validate_argument_is_valid_path(args, argument_name):
69 paths_to_check = []
70 # check if we are handling a single file or a list of files
71 member_value = getattr(args, argument_name)
72 if member_value is not None:
73 if isinstance(member_value, list):
74 for file_name in member_value:
75 paths_to_check.append(strip(str(file_name)))
76 else:
77 paths_to_check.append(strip(str(member_value)))
78
79 for path_to_check in paths_to_check:
80 validate_path_exists(path_to_check)
81
82
83 # taken from
84 # http://stackoverflow.com/questions/8384737/python-extract-file-name-from-path-no-matter-what-the-os-path-format
85 def get_filename(path):
86 head, tail = ntpath.split(path)
87 return tail or ntpath.basename(head)
88
89
90 def get_filename_without_suffix(path):
91 root, ext = os.path.splitext(os.path.basename(path))
92 return root
93
94
95 def parse_input_ctds(xsd_location, input_ctds, output_destination, output_file_extension):
96 is_converting_multiple_ctds = len(input_ctds) > 1
97 parsed_ctds = []
98 schema = None
99 if xsd_location is not None:
100 try:
101 info("Loading validation schema from %s" % xsd_location, 0)
102 schema = etree.XMLSchema(etree.parse(xsd_location))
103 except Exception, e:
104 error("Could not load validation schema %s. Reason: %s" % (xsd_location, str(e)), 0)
105 else:
106 warning("Validation against a schema has not been enabled.", 0)
107
108 for input_ctd in input_ctds:
109 if schema is not None:
110 validate_against_schema(input_ctd, schema)
111
112 output_file = output_destination
113 # if multiple inputs are being converted, we need to generate a different output_file for each input
114 if is_converting_multiple_ctds:
115 output_file = os.path.join(output_file, get_filename_without_suffix(input_ctd) + "." + output_file_extension)
116 info("Parsing %s" % input_ctd)
117 parsed_ctds.append(ParsedCTD(CTDModel(from_file=input_ctd), input_ctd, output_file))
118
119 return parsed_ctds
120
121
122 def flatten_list_of_lists(args, list_name):
123 setattr(args, list_name, [item for sub_list in getattr(args, list_name) for item in sub_list])
124
125
126 def validate_against_schema(ctd_file, schema):
127 try:
128 parser = etree.XMLParser(schema=schema)
129 etree.parse(ctd_file, parser=parser)
130 except etree.XMLSyntaxError, e:
131 raise ApplicationException("Invalid CTD file %s. Reason: %s" % (ctd_file, str(e)))
132
133
134 def add_common_parameters(parser, version, last_updated):
135 parser.add_argument("FORMAT", default=None, help="Output format (mandatory). Can be one of: cwl, galaxy.")
136 parser.add_argument("-i", "--input", dest="input_files", default=[], required=True, nargs="+", action="append",
137 help="List of CTD files to convert.")
138 parser.add_argument("-o", "--output-destination", dest="output_destination", required=True,
139 help="If multiple input files are given, then a folder in which all converted "
140 "files will be generated is expected; "
141 "if a single input file is given, then a destination file is expected.")
142 parser.add_argument("-x", "--default-executable-path", dest="default_executable_path",
143 help="Use this executable path when <executablePath> is not present in the CTD",
144 default=None, required=False)
145 parser.add_argument("-b", "--blacklist-parameters", dest="blacklisted_parameters", default=[], nargs="+",
146 action="append",
147 help="List of parameters that will be ignored and won't appear on the galaxy stub",
148 required=False)
149 parser.add_argument("-p", "--hardcoded-parameters", dest="hardcoded_parameters", default=None, required=False,
150 help="File containing hardcoded values for the given parameters. Run with '-h' or '--help' "
151 "to see a brief example on the format of this file.")
152 parser.add_argument("-V", "--validation-schema", dest="xsd_location", default=None, required=False,
153 help="Location of the schema to use to validate CTDs. If not provided, no schema validation "
154 "will take place.")
155
156 # TODO: add verbosity, maybe?
157 program_version = "v%s" % version
158 program_build_date = str(last_updated)
159 program_version_message = "%%(prog)s %s (%s)" % (program_version, program_build_date)
160 parser.add_argument("-v", "--version", action="version", version=program_version_message)
161
162
163 def parse_hardcoded_parameters(hardcoded_parameters_file):
164 parameter_hardcoder = ParameterHardcoder()
165 if hardcoded_parameters_file is not None:
166 line_number = 0
167 with open(hardcoded_parameters_file) as f:
168 for line in f:
169 line_number += 1
170 if line is None or not line.strip() or line.strip().startswith("#"):
171 pass
172 else:
173 # the third column must not be obtained as a whole, and not split
174 parsed_hardcoded_parameter = line.strip().split(None, 2)
175 # valid lines contain two or three columns
176 if len(parsed_hardcoded_parameter) != 2 and len(parsed_hardcoded_parameter) != 3:
177 warning("Invalid line at line number %d of the given hardcoded parameters file. Line will be"
178 "ignored:\n%s" % (line_number, line), 0)
179 continue
180
181 parameter_name = parsed_hardcoded_parameter[0]
182 hardcoded_value = parsed_hardcoded_parameter[1]
183 tool_names = None
184 if len(parsed_hardcoded_parameter) == 3:
185 tool_names = parsed_hardcoded_parameter[2].split(',')
186 if tool_names:
187 for tool_name in tool_names:
188 parameter_hardcoder.register_parameter(parameter_name, hardcoded_value, tool_name.strip())
189 else:
190 parameter_hardcoder.register_parameter(parameter_name, hardcoded_value)
191
192 return parameter_hardcoder
193
194
195 def extract_tool_help_text(ctd_model):
196 manual = ""
197 doc_url = None
198 if "manual" in ctd_model.opt_attribs.keys():
199 manual += "%s\n\n" % ctd_model.opt_attribs["manual"]
200 if "docurl" in ctd_model.opt_attribs.keys():
201 doc_url = ctd_model.opt_attribs["docurl"]
202
203 help_text = "No help available"
204 if manual is not None:
205 help_text = manual
206 if doc_url is not None:
207 help_text = ("" if manual is None else manual) + "\nFor more information, visit %s" % doc_url
208
209 return help_text
210
211
212 def extract_tool_executable_path(model, default_executable_path):
213 # rules to build the executable path:
214 # if executablePath is null, then use default_executable_path
215 # if executablePath is null and executableName is null, then the name of the tool will be used
216 # if executablePath is null and executableName is not null, then executableName will be used
217 # if executablePath is not null and executableName is null,
218 # then executablePath and the name of the tool will be used
219 # if executablePath is not null and executableName is not null, then both will be used
220
221 # first, check if the model has executablePath / executableName defined
222 executable_path = model.opt_attribs.get("executablePath", None)
223 executable_name = model.opt_attribs.get("executableName", None)
224
225 # check if we need to use the default_executable_path
226 if executable_path is None:
227 executable_path = default_executable_path
228
229 # fix the executablePath to make sure that there is a '/' in the end
230 if executable_path is not None:
231 executable_path = executable_path.strip()
232 if not executable_path.endswith("/"):
233 executable_path += "/"
234
235 # assume that we have all information present
236 command = str(executable_path) + str(executable_name)
237 if executable_path is None:
238 if executable_name is None:
239 command = model.name
240 else:
241 command = executable_name
242 else:
243 if executable_name is None:
244 command = executable_path + model.name
245 return command
246
247
248 def extract_and_flatten_parameters(ctd_model):
249 parameters = []
250 if len(ctd_model.parameters.parameters) > 0:
251 # use this to put parameters that are to be processed
252 # we know that CTDModel has one parent ParameterGroup
253 pending = [ctd_model.parameters]
254 while len(pending) > 0:
255 # take one element from 'pending'
256 parameter = pending.pop()
257 if type(parameter) is not ParameterGroup:
258 parameters.append(parameter)
259 else:
260 # append the first-level children of this ParameterGroup
261 pending.extend(parameter.parameters.values())
262 # returned the reversed list of parameters (as it is now,
263 # we have the last parameter in the CTD as first in the list)
264 return reversed(parameters)
265
266
267 # some parameters are mapped to command line options, this method helps resolve those mappings, if any
268 def resolve_param_mapping(param, ctd_model):
269 # go through all mappings and find if the given param appears as a reference name in a mapping element
270 param_mapping = None
271 for cli_element in ctd_model.cli:
272 for mapping_element in cli_element.mappings:
273 if mapping_element.reference_name == param.name:
274 if param_mapping is not None:
275 warning("The parameter %s has more than one mapping in the <cli> section. "
276 "The first found mapping, %s, will be used." % (param.name, param_mapping), 1)
277 else:
278 param_mapping = cli_element.option_identifier
279
280 return param_mapping if param_mapping is not None else param.name
281
282
283 def _extract_param_cli_name(param, ctd_model):
284 # we generate parameters with colons for subgroups, but not for the two topmost parents (OpenMS legacy)
285 if type(param.parent) == ParameterGroup:
286 if not hasattr(param.parent.parent, 'parent'):
287 return resolve_param_mapping(param, ctd_model)
288 elif not hasattr(param.parent.parent.parent, 'parent'):
289 return resolve_param_mapping(param, ctd_model)
290 else:
291 if ctd_model.cli:
292 warning("Using nested parameter sections (NODE elements) is not compatible with <cli>", 1)
293 return extract_param_name(param.parent) + ":" + resolve_param_mapping(param, ctd_model)
294 else:
295 return resolve_param_mapping(param, ctd_model)
296
297
298 def extract_param_name(param):
299 # we generate parameters with colons for subgroups, but not for the two topmost parents (OpenMS legacy)
300 if type(param.parent) == ParameterGroup:
301 if not hasattr(param.parent.parent, "parent"):
302 return param.name
303 elif not hasattr(param.parent.parent.parent, "parent"):
304 return param.name
305 else:
306 return extract_param_name(param.parent) + ":" + param.name
307 else:
308 return param.name
309
310
311 def extract_command_line_prefix(param, ctd_model):
312 param_name = extract_param_name(param)
313 param_cli_name = _extract_param_cli_name(param, ctd_model)
314 if param_name == param_cli_name:
315 # there was no mapping, so for the cli name we will use a '-' in the prefix
316 param_cli_name = "-" + param_name
317 return param_cli_name
+272
-0
convert.py less more
0 import os
1 import sys
2 import traceback
3 import common.utils as utils
4
5 from argparse import ArgumentParser
6 from argparse import RawDescriptionHelpFormatter
7 from common.exceptions import ApplicationException, ModelError
8
9 __all__ = []
10 __version__ = 2.0
11 __date__ = '2014-09-17'
12 __updated__ = '2017-08-09'
13
14 program_version = "v%s" % __version__
15 program_build_date = str(__updated__)
16 program_version_message = '%%(prog)s %s (%s)' % (program_version, program_build_date)
17 program_short_description = "CTDConverter - A project from the WorkflowConversion family " \
18 "(https://github.com/WorkflowConversion/CTDConverter)"
19 program_usage = '''
20 USAGE:
21
22 $ python convert.py [FORMAT] [ARGUMENTS ...]
23
24 FORMAT can be either one of the supported output formats: cwl, galaxy.
25
26 There is one converter for each supported FORMAT, each taking a different set of arguments. Please consult the detailed
27 documentation for each of the converters. Nevertheless, all converters have the following common parameters/options:
28
29
30 I - Parsing a single CTD file and convert it:
31
32 $ python convert.py [FORMAT] -i [INPUT_FILE] -o [OUTPUT_FILE]
33
34
35 II - Parsing several CTD files, output converted wrappers in a given folder:
36
37 $ python converter.py [FORMAT] -i [INPUT_FILES] -o [OUTPUT_DIRECTORY]
38
39
40 III - Hardcoding parameters
41
42 It is possible to hardcode parameters. This makes sense if you want to set a tool in 'quiet' mode or if your tools
43 support multi-threading and accept the number of threads via a parameter, without giving end users the chance to
44 change the values for these parameters.
45
46 In order to generate hardcoded parameters, you need to provide a simple file. Each line of this file contains
47 two or three columns separated by whitespace. Any line starting with a '#' will be ignored. The first column contains
48 the name of the parameter, the second column contains the value that will always be set for this parameter. Only the
49 first two columns are mandatory.
50
51 If the parameter is to be hardcoded only for a set of tools, then a third column can be added. This column contains
52 a comma-separated list of tool names for which the parameter will be hardcoded. If a third column is not present,
53 then all processed tools containing the given parameter will get a hardcoded value for it.
54
55 The following is an example of a valid file:
56
57 ##################################### HARDCODED PARAMETERS example #####################################
58 # Every line starting with a # will be handled as a comment and will not be parsed.
59 # The first column is the name of the parameter and the second column is the value that will be used.
60
61 # Parameter name # Value # Tool(s)
62 threads 8
63 mode quiet
64 xtandem_executable xtandem XTandemAdapter
65 verbosity high Foo, Bar
66
67 #########################################################################################################
68
69 Using the above file will produce a command-line similar to:
70
71 [TOOL] ... -threads 8 -mode quiet ...
72
73 for all tools. For XTandemAdapter, however, the command-line will look like:
74
75 XtandemAdapter ... -threads 8 -mode quiet -xtandem_executable xtandem ...
76
77 And for tools Foo and Bar, the command-line will be similar to:
78
79 Foo -threads 8 -mode quiet -verbosity high ...
80
81
82 IV - Engine-specific parameters
83
84 i - Galaxy
85
86 a. Providing file formats, mimetypes
87
88 Galaxy supports the concept of file format in order to connect compatible ports, that is, input ports of a
89 certain data format will be able to receive data from a port from the same format. This converter allows you
90 to provide a personalized file in which you can relate the CTD data formats with supported Galaxy data formats.
91 The layout of this file consists of lines, each of either one or four columns separated by any amount of
92 whitespace. The content of each column is as follows:
93
94 * 1st column: file extension
95 * 2nd column: data type, as listed in Galaxy
96 * 3rd column: full-named Galaxy data type, as it will appear on datatypes_conf.xml
97 * 4th column: mimetype (optional)
98
99 The following is an example of a valid "file formats" file:
100
101 ########################################## FILE FORMATS example ##########################################
102 # Every line starting with a # will be handled as a comment and will not be parsed.
103 # The first column is the file format as given in the CTD and second column is the Galaxy data format. The
104 # second, third, fourth and fifth columns can be left empty if the data type has already been registered
105 # in Galaxy, otherwise, all but the mimetype must be provided.
106
107 # CTD type # Galaxy type # Long Galaxy data type # Mimetype
108 csv tabular galaxy.datatypes.data:Text
109 fasta
110 ini txt galaxy.datatypes.data:Text
111 txt
112 idxml txt galaxy.datatypes.xml:GenericXml application/xml
113 options txt galaxy.datatypes.data:Text
114 grid grid galaxy.datatypes.data:Grid
115 ##########################################################################################################
116
117 Note that each line consists precisely of either one, three or four columns. In the case of data types already
118 registered in Galaxy (such as fasta and txt in the above example), only the first column is needed. In the
119 case of data types that haven't been yet registered in Galaxy, the first three columns are needed
120 (mimetype is optional).
121
122 For information about Galaxy data types and subclasses, see the following page:
123 https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes
124
125
126 b. Finer control over which tools will be converted
127
128 Sometimes only a subset of CTDs needs to be converted. It is possible to either explicitly specify which tools
129 will be converted or which tools will not be converted.
130
131 The value of the -s/--skip-tools parameter is a file in which each line will be interpreted as the name of a
132 tool that will not be converted. Conversely, the value of the -r/--required-tools is a file in which each line
133 will be interpreted as a tool that is required. Only one of these parameters can be specified at a given time.
134
135 The format of both files is exactly the same. As stated before, each line will be interpreted as the name of a
136 tool. Any line starting with a '#' will be ignored.
137
138
139 ii - CWL
140
141 There are, for now, no CWL-specific parameters or options.
142
143 '''
144
145 program_license = '''%(short_description)s
146
147 Copyright 2017, WorklfowConversion
148
149 Licensed under the Apache License, Version 2.0 (the "License");
150 you may not use this file except in compliance with the License.
151 You may obtain a copy of the License at
152
153 http://www.apache.org/licenses/LICENSE-2.0
154
155 Unless required by applicable law or agreed to in writing, software
156 distributed under the License is distributed on an "AS IS" BASIS,
157 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
158 See the License for the specific language governing permissions and
159 limitations under the License.
160
161 %(usage)s
162 ''' % {'short_description': program_short_description, 'usage': program_usage}
163
164
165 def main(argv=None):
166 if argv is None:
167 argv = sys.argv
168 else:
169 sys.argv.extend(argv)
170
171 # check that we have, at least, one argument provided
172 # at this point we cannot parse the arguments, because each converter takes different arguments, meaning each
173 # converter will register its own parameters after we've registered the basic ones... we have to do it old school
174 if len(argv) < 2:
175 utils.error("Not enough arguments provided")
176 print("\nUsage: $ python convert.py [TARGET] [ARGUMENTS]\n\n" +
177 "Where:\n" +
178 " target: one of 'cwl' or 'galaxy'\n\n" +
179 "Run again using the -h/--help option to print more detailed help.\n")
180 return 1
181
182 # TODO: at some point this should look like real software engineering and use a map containing converter instances
183 # whose keys would be the name of the converter (e.g., cwl, galaxy), but for the time being, only two formats
184 # are supported
185 target = str.lower(argv[1])
186 if target == 'cwl':
187 from cwl import converter
188 elif target == 'galaxy':
189 from galaxy import converter
190 elif target == '-h' or target == '--help' or target == '--h' or target == 'help':
191 print(program_license)
192 return 0
193 else:
194 utils.error("Unrecognized target engine. Supported targets are 'cwl' and 'galaxy'.")
195 return 1
196
197 utils.info("Using %s converter" % target)
198
199 try:
200 # Setup argument parser
201 parser = ArgumentParser(prog="CTDConverter", description=program_license,
202 formatter_class=RawDescriptionHelpFormatter, add_help=True)
203 utils.add_common_parameters(parser, program_version_message, program_build_date)
204
205 # add tool-specific arguments
206 converter.add_specific_args(parser)
207
208 # parse arguments and perform some basic, common validation
209 args = parser.parse_args()
210 validate_and_prepare_common_arguments(args)
211
212 # parse the input CTD files into CTDModels
213 parsed_ctds = utils.parse_input_ctds(args.xsd_location, args.input_files, args.output_destination,
214 converter.get_preferred_file_extension())
215
216 # let the converter do its own thing
217 converter.convert_models(args, parsed_ctds)
218 return 0
219
220 except KeyboardInterrupt:
221 print("Interrupted...")
222 return 0
223
224 except ApplicationException, e:
225 traceback.print_exc()
226 utils.error("CTDConverter could not complete the requested operation.", 0)
227 utils.error("Reason: " + e.msg, 0)
228 return 1
229
230 except ModelError, e:
231 traceback.print_exc()
232 utils.error("There seems to be a problem with one of your input CTDs.", 0)
233 utils.error("Reason: " + e.msg, 0)
234 return 1
235
236 except Exception, e:
237 traceback.print_exc()
238 utils.error("CTDConverter could not complete the requested operation.", 0)
239 utils.error("Reason: " + e.msg, 0)
240 return 2
241
242
243 def validate_and_prepare_common_arguments(args):
244 # flatten lists of lists to a list containing elements
245 lists_to_flatten = ["input_files", "blacklisted_parameters"]
246 for list_to_flatten in lists_to_flatten:
247 utils.flatten_list_of_lists(args, list_to_flatten)
248
249 # if input is a single file, we expect output to be a file (and not a dir that already exists)
250 if len(args.input_files) == 1:
251 if os.path.isdir(args.output_destination):
252 raise ApplicationException("If a single input file is provided, output (%s) is expected to be a file "
253 "and not a folder.\n" % args.output_destination)
254
255 # if input is a list of files, we expect output to be a folder
256 if len(args.input_files) > 1:
257 if not os.path.isdir(args.output_destination):
258 raise ApplicationException("If several input files are provided, output (%s) is expected to be an "
259 "existing directory.\n" % args.output_destination)
260
261 # check that the provided input files, if provided, contain a valid file path
262 input_arguments_to_check = ["xsd_location", "input_files", "hardcoded_parameters"]
263 for argument_name in input_arguments_to_check:
264 utils.validate_argument_is_valid_path(args, argument_name)
265
266 # add the parameter hardcoder
267 args.parameter_hardcoder = utils.parse_hardcoded_parameters(args.hardcoded_parameters)
268
269
270 if __name__ == "__main__":
271 sys.exit(main())⏎
+5
-0
cwl/README.md less more
0 # Conversion of CTD Files to CWL
1
2 ## How to use: Parameters in Detail
3 The CWL converter has, for now, only the basic parameters described in the [top README file](../README.md).
4
+0
-0
cwl/__init__.py less more
(New empty file)
+196
-0
cwl/converter.py less more
0 #!/usr/bin/env python
1 # encoding: utf-8
2
3 # instead of using cwlgen, we decided to use PyYAML directly
4 # we promptly found a problem with cwlgen, namely, it is not possible to construct something like:
5 # some_paramter:
6 # type: ['null', string]
7 # which kind of sucks, because this seems to be the way to state that a parameter is truly optional and has no default
8 # since cwlgen is just "fancy classes" around the yaml.dump() method, we implemented our own generation of yaml
9
10
11 import ruamel.yaml as yaml
12
13 from CTDopts.CTDopts import _InFile, _OutFile, ParameterGroup, _Choices, _NumericRange, _FileFormat, ModelError, _Null
14 from common import utils, logger
15
16 # all cwl-related properties are defined here
17
18 CWL_SHEBANG = "#!/usr/bin/env cwl-runner"
19 CURRENT_CWL_VERSION = 'v1.0'
20 CWL_VERSION = 'cwlVersion'
21 CLASS = 'class'
22 BASE_COMMAND = 'baseCommand'
23 INPUTS = 'inputs'
24 ID = 'id'
25 TYPE = 'type'
26 INPUT_BINDING = 'inputBinding'
27 OUTPUT_BINDING = 'outputBinding'
28 PREFIX = 'prefix'
29 OUTPUTS = 'outputs'
30 POSITION = 'position'
31 VALUE_FROM = 'valueFrom'
32 GLOB = 'glob'
33 LABEL = 'label'
34 DOC = 'doc'
35 DEFAULT = 'default'
36
37 # types
38 TYPE_NULL = 'null'
39 TYPE_BOOLEAN = 'boolean'
40 TYPE_INT = 'int'
41 TYPE_LONG = 'long'
42 TYPE_FLOAT = 'float'
43 TYPE_DOUBLE = 'double'
44 TYPE_STRING = 'string'
45 TYPE_FILE = 'File'
46 TYPE_DIRECTORY = 'Directory'
47
48 TYPE_TO_CWL_TYPE = {int: TYPE_INT, float: TYPE_DOUBLE, str: TYPE_STRING, bool: TYPE_BOOLEAN, _InFile: TYPE_FILE,
49 _OutFile: TYPE_FILE, _Choices: TYPE_STRING}
50
51
52 def add_specific_args(parser):
53 # no specific arguments for CWL conversion, for now
54 # however, this method has to be defined, otherwise ../convert.py won't work for CWL
55 pass
56
57
58 def get_preferred_file_extension():
59 return "cwl"
60
61
62 def convert_models(args, parsed_ctds):
63 # go through each ctd model and perform the conversion, easy as pie!
64 for parsed_ctd in parsed_ctds:
65 model = parsed_ctd.ctd_model
66 origin_file = parsed_ctd.input_file
67 output_file = parsed_ctd.suggested_output_file
68
69 logger.info("Converting %s (source %s)" % (model.name, utils.get_filename(origin_file)))
70 cwl_tool = convert_to_cwl(model, args)
71
72 logger.info("Writing to %s" % utils.get_filename(output_file), 1)
73
74 stream = file(output_file, 'w')
75 stream.write(CWL_SHEBANG + '\n\n')
76 stream.write("# This CWL file was automatically generated using CTDConverter.\n")
77 stream.write("# Visit https://github.com/WorkflowConversion/CTDConverter for more information.\n\n")
78 yaml.dump(cwl_tool, stream, default_flow_style=False)
79 stream.close()
80
81
82 # returns a dictionary
83 def convert_to_cwl(ctd_model, args):
84 # create cwl_tool object with the basic information
85 base_command = utils.extract_tool_executable_path(ctd_model, args.default_executable_path)
86
87 # add basic properties
88 cwl_tool = {}
89 cwl_tool[CWL_VERSION] = CURRENT_CWL_VERSION
90 cwl_tool[CLASS] = 'CommandLineTool'
91 cwl_tool[LABEL] = ctd_model.opt_attribs["description"]
92 cwl_tool[DOC] = utils.extract_tool_help_text(ctd_model)
93 cwl_tool[BASE_COMMAND] = base_command
94
95 # TODO: test with optional output files
96
97 # add inputs/outputs
98 for param in utils.extract_and_flatten_parameters(ctd_model):
99 if param.name in args.blacklisted_parameters:
100 continue
101
102 param_name = utils.extract_param_name(param)
103 cwl_fixed_param_name = fix_param_name(param_name)
104 hardcoded_value = args.parameter_hardcoder.get_hardcoded_value(param_name, ctd_model.name)
105 param_default = str(param.default) if param.default is not _Null and param.default is not None else None
106
107 if param.type is _OutFile:
108 create_lists_if_missing(cwl_tool, [INPUTS, OUTPUTS])
109 # we know the only outputs are of type _OutFile
110 # we need an input of type string that will contain the name of the output file
111 label = "Filename for %s output file" % param_name
112 input_name_for_output_filename = get_input_name_for_output_filename(param)
113 input_param = {}
114 input_param[ID] = input_name_for_output_filename
115 input_param[DOC] = label
116 input_param[LABEL] = label
117 if param_default is not None:
118 input_param[DEFAULT] = param_default
119 input_param[TYPE] = generate_cwl_param_type(param, TYPE_STRING)
120 insert_input_binding(ctd_model, param, hardcoded_value, input_param)
121
122 output_binding = {}
123 output_binding[GLOB] = "$(inputs.%s)" % input_name_for_output_filename
124
125 output_param = {}
126 output_param[ID] = cwl_fixed_param_name
127 output_param[OUTPUT_BINDING] = output_binding
128 output_param[DOC] = param.description
129 output_param[LABEL] = param.description
130 output_param[TYPE] = generate_cwl_param_type(param)
131
132 cwl_tool[INPUTS].append(input_param)
133 cwl_tool[OUTPUTS].append(output_param)
134
135 else:
136 create_lists_if_missing(cwl_tool, [INPUTS])
137 # we know that anything that is not an _OutFile is an input
138 input_param = {}
139 input_param[ID] = cwl_fixed_param_name
140 input_param[DOC] = param.description
141 input_param[LABEL] = param.description
142 if param_default is not None:
143 input_param[DEFAULT] = param_default
144 input_param[TYPE] = generate_cwl_param_type(param)
145 insert_input_binding(ctd_model, param, hardcoded_value, input_param)
146
147 cwl_tool[INPUTS].append(input_param)
148
149 return cwl_tool
150
151
152 def create_lists_if_missing(cwl_tool, keys):
153 for key in keys:
154 if key not in cwl_tool:
155 cwl_tool[key] = []
156
157
158 def get_input_name_for_output_filename(param):
159 assert param.type is _OutFile, "Only output files can get a generated filename input parameter."
160 return fix_param_name(utils.extract_param_name(param)) + "_filename"
161
162
163 def fix_param_name(param_name):
164 # IMPORTANT: there seems to be a problem in CWL if the prefix and the parameter name are the same, so we need to
165 # prepend something to the parameter name that will be registered in CWL, also, using colons in parameter
166 # names seems to bring all sorts of problems for cwl-runner
167 return 'param_' + param_name.replace(":", "_")
168
169
170 # in order to provide "true" optional params, the parameter type should be something like ['null', <CWLType>],
171 # for instance ['null', int]
172 def generate_cwl_param_type(param, forced_type=None):
173 cwl_type = TYPE_TO_CWL_TYPE[param.type] if forced_type is None else forced_type
174 return cwl_type if param.required else ['null', cwl_type]
175
176
177 # generate, and insert, the inputBinding
178 def insert_input_binding(ctd_model, param, hardcoded_value, cwl_input_param):
179 prefix = utils.extract_command_line_prefix(param, ctd_model)
180 prefix = None if prefix is None or not prefix.strip() else prefix
181
182 input_binding = {}
183
184 if prefix is not None:
185 input_binding[PREFIX] = prefix
186
187 if hardcoded_value is not None:
188 input_binding[VALUE_FROM] = hardcoded_value
189
190 if param.is_positional():
191 input_binding[POSITION] = param.position
192
193 # insert input binding if there's something in it
194 if input_binding:
195 cwl_input_param[INPUT_BINDING] = input_binding
+139
-0
galaxy/README.md less more
0 # Conversion of CTD Files to Galaxy ToolConfigs
1 ## Generating a `tool_conf.xml` File
2 * Purpose: Galaxy uses a file `tool_conf.xml` in which other tools can be included. `CTDConverter` can also generate this file. Categories will be extracted from the provided input CTDs and for each category, a different `<section>` will be generated. Any input CTD lacking a category will be sorted under the provided default category.
3 * Short/long version: `-t` / `--tool-conf-destination`
4 * Required: no.
5 * Taken values: The destination of the file.
6
7 $ python convert.py galaxy -i /data/ctds/*.ctd -o /data/generated-galaxy-stubs -t /data/generated-galaxy-stubs/tool_conf.xml
8
9
10 ## Adding Parameters to the Command-line
11 * Purpose: Galaxy *ToolConfig* files include a `<command>` element in which the command line to invoke the tool can be given. Sometimes it is needed to invoke your tools in a certain way (i.e., passing certain parameters). For instance, some tools offer the possibility to be invoked in a verbose or quiet way or even to be invoked in a headless way (i.e., without GUI).
12 * Short/long version: `-a` / `--add-to-command-line`
13 * Required: no.
14 * Taken values: The command(s) to be added to the command line.
15
16 Example:
17
18 $ python convert.py galaxy ... -a "--quiet --no-gui"
19
20 Will generate the following `<command>` element in the generated Galaxy *ToolConfig*:
21
22 <command>TOOL_NAME --quiet --no-gui ...</command>
23
24 ## Providing a default Category
25 * Purpose: Input CTDs that lack a category will be sorted under the value given to this parameter. If this parameter is not provided, then the category `DEFAULT` will be used.
26 * Short/long version: `-c` / `--default-category`
27 * Required: no.
28 * Taken values: The value for the default category to use for input CTDs lacking a category.
29
30 Example:
31
32 Suppose there is a folder containing several CTD files. Some of those CTDs don't have the optional attribute `category` and the rest belong to the `Data Processing` category. The following invocation:
33
34 $ python convert.py galaxy ... -c Other
35
36 will generate, for each of the categories, a different section. Additionally, CTDs lacking a category will be sorted under the given category, `Other`, as shown:
37
38 <section id="category-id-dataprocessing" name="Data Processing">
39 <tool file="some_path/tool_one.xml" />
40 <tool file="some_path/tool_two.xml" />
41 ...
42 </section>
43
44 <section id="category-id-other" name="Other">
45 <tool file="some_path/tool_three.xml" />
46 <tool file="some_path/tool_four.xml" />
47 ...
48 </section>
49
50 ## Providing a Path for the Location of the *ToolConfig* Files
51 * Purpose: The `tool_conf.xml` file contains references to files which in turn contain Galaxy *ToolConfig* files. Using this parameter, you can provide information about the location of your wrappers on your Galaxy instance.
52 * Short/long version: `-g` / `--galaxy-tool-path`
53 * Required: no.
54 * Taken values: The path relative to your `$GALAXY_ROOT/tools` folder on which your tools are located.
55
56 Example:
57
58 $ python convert.py galaxy ... -g my_tools_folder
59
60 Will generate `<tool>` elements in the generated `tool_conf.xml` as follows:
61
62 <tool file="my_tools_folder/some_tool.xml" />
63
64 In this example, `tool_conf.xml` refers to a file located on `$GALAXY_ROOT/tools/my_tools_folder/some_tool.xml`.
65
66 ## Including additional Macros Files
67 * Purpose: Include external macros files.
68 * Short/long version: `-m` / `--macros`
69 * Required: no.
70 * Default: `macros.xml`
71 * Taken values: List of paths of macros files to include.
72
73 *ToolConfig* supports elaborate sections such as `<stdio>`, `<requirements>`, etc., that are identical across tools of the same suite. Macros files assist in the task of including external xml sections into *ToolConfig* files. For more information about the syntax of macros files, see: https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#Reusing_Repeated_Configuration_Elements
74
75 There are some macros that are required, namely `stdio`, `requirements` and `advanced_options`. A template macro file is included in [macros.xml]. It can be edited to suit your needs and you could add extra macros or leave it as it is and include additional files. Every macro found in the provided files will be expanded.
76
77 Please note that the used macros files **must** be copied to your Galaxy installation on the same location in which you place the generated *ToolConfig* files, otherwise Galaxy will not be able to parse the generated *ToolConfig* files!
78
79 ## Generating a `datatypes_conf.xml` File
80 * Purpose: Specify the destination of a generated `datatypes_conf.xml` file.
81 * Short/long version: `-d` / `--datatypes-destination`
82 * Required: no.
83 * Taken values: The path in which `datatypes_conf.xml` will be generated.
84
85 It is likely that your tools use file formats or mimetypes that have not been registered in Galaxy. The generator allows you to specify a path in which an automatically generated `datatypes_conf.xml` file will be created. Consult the next section to get information about how to register file formats and mimetypes.
86
87 ## Providing Galaxy File Formats
88 * Purpose: Register new file formats and mimetypes.
89 * Short/long version: `-f` / `--formats-file`
90 * Required: no.
91 * Taken values: The path of a file describing formats.
92
93 Galaxy supports the concept of file format in order to connect compatible ports, that is, input ports of a certain data format will be able to receive data from a port from the same format. This converter allows you to provide a personalized file in which you can relate the CTD data formats with supported Galaxy data formats. The format file is a simple text file, each line containing several columns separated by whitespace. The content of each column is as follows:
94
95 * 1st column: file extension, this column is required.
96 * 2nd column: data type, as listed in Galaxy, this column is optional.
97 * 3rd column: full-named Galaxy data type, as it will appear on datatypes_conf.xml; this column is required if the second column is included.
98 * 4th column: mimetype, this column is optional.
99
100 The following is an example of a valid "file formats" file:
101
102 # CTD type # Galaxy type # Long Galaxy data type # Mimetype
103 csv tabular galaxy.datatypes.data:Text
104 fasta
105 ini txt galaxy.datatypes.data:Text
106 txt
107 idxml txt galaxy.datatypes.xml:GenericXml application/xml
108 options txt galaxy.datatypes.data:Text
109 grid grid galaxy.datatypes.data:Grid
110
111 Note that each line consists of either one, three or four columns. In the case of data types already registered in Galaxy (such as `fasta` and `txt` in the above example), only the first column is needed. In the case of data types that haven't been yet registered in Galaxy, the first three columns are needed (mimetype is optional).
112
113 For information about Galaxy data types and subclasses, consult the following page: https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes
114
115 ## Remarks about some of the *OpenMS* Tools
116 * Most of the tools can be generated automatically. However, some of the tools need some extra work (for now).
117 * The following adapters need to be changed, such that you provide the path to the executable:
118 * FidoAdapter (add `-exe fido` in the command tag, delete the `$param_exe` in the command tag, delete the parameter from the input list).
119 * MSGFPlusAdapter (add `-executable msgfplus.jar` in the command tag, delete the `$param_executable` in the command tag, delete the parameter from the input list).
120 * MyriMatchAdapter (add `-myrimatch_executable myrimatch` in the command tag, delete the `$param_myrimatch_executable` in the command tag, delete the parameter from the input list).
121 * OMSSAAdapter (add `-omssa_executable omssa` in the command tag, delete the `$param_omssa_executable` in the command tag, delete the parameter from the input list).
122 * PepNovoAdapter (add `-pepnovo_executable pepnovo` in the command tag, delete the `$param_pepnovo_executable` in the command tag, delete the parameter from the input list).
123 * XTandemAdapter (add `-xtandem_executable xtandem` in the command tag, delete the $param_xtandem_executable in the command tag, delete the parameter from the input list).
124 * To avoid the deletion in the inputs you can also add these parameters to the blacklist
125
126 $ python convert.py galaxy -b exe executable myrimatch_excutable omssa_executable pepnovo_executable xtandem_executable
127
128 * The following tools have multiple outputs (number of inputs = number of outputs) which is not yet supported in Galaxy-stable:
129 * SeedListGenerator
130 * SpecLibSearcher
131 * MapAlignerIdentification
132 * MapAlignerPoseClustering
133 * MapAlignerSpectrum
134 * MapAlignerRTTransformer
135
136 [CTDopts]: https://github.com/genericworkflownodes/CTDopts
137 [macros.xml]: https://github.com/WorkflowConversion/CTDConverter/blob/master/galaxy/macros.xml
138 [CTDSchema]: https://github.com/genericworkflownodes/CTDSchema⏎
+0
-0
galaxy/__init__.py less more
(New empty file)
+867
-0
galaxy/converter.py less more
0 #!/usr/bin/env python
1 # encoding: utf-8
2 import os
3 import string
4
5 from collections import OrderedDict
6 from string import strip
7 from lxml import etree
8 from lxml.etree import SubElement, Element, ElementTree, ParseError, parse
9
10 from common import utils, logger
11 from common.exceptions import ApplicationException, InvalidModelException
12
13 from CTDopts.CTDopts import _InFile, _OutFile, ParameterGroup, _Choices, _NumericRange, _FileFormat, ModelError, _Null
14
15
16 TYPE_TO_GALAXY_TYPE = {int: 'integer', float: 'float', str: 'text', bool: 'boolean', _InFile: 'data',
17 _OutFile: 'data', _Choices: 'select'}
18 STDIO_MACRO_NAME = "stdio"
19 REQUIREMENTS_MACRO_NAME = "requirements"
20 ADVANCED_OPTIONS_MACRO_NAME = "advanced_options"
21
22 REQUIRED_MACROS = [STDIO_MACRO_NAME, REQUIREMENTS_MACRO_NAME, ADVANCED_OPTIONS_MACRO_NAME]
23
24
25 class ExitCode:
26 def __init__(self, code_range="", level="", description=None):
27 self.range = code_range
28 self.level = level
29 self.description = description
30
31
32 class DataType:
33 def __init__(self, extension, galaxy_extension=None, galaxy_type=None, mimetype=None):
34 self.extension = extension
35 self.galaxy_extension = galaxy_extension
36 self.galaxy_type = galaxy_type
37 self.mimetype = mimetype
38
39
40 def add_specific_args(parser):
41 parser.add_argument("-f", "--formats-file", dest="formats_file",
42 help="File containing the supported file formats. Run with '-h' or '--help' to see a "
43 "brief example on the layout of this file.", default=None, required=False)
44 parser.add_argument("-a", "--add-to-command-line", dest="add_to_command_line",
45 help="Adds content to the command line", default="", required=False)
46 parser.add_argument("-d", "--datatypes-destination", dest="data_types_destination",
47 help="Specify the location of a datatypes_conf.xml to modify and add the registered "
48 "data types. If the provided destination does not exist, a new file will be created.",
49 default=None, required=False)
50 parser.add_argument("-c", "--default-category", dest="default_category", default="DEFAULT", required=False,
51 help="Default category to use for tools lacking a category when generating tool_conf.xml")
52 parser.add_argument("-t", "--tool-conf-destination", dest="tool_conf_destination", default=None, required=False,
53 help="Specify the location of an existing tool_conf.xml that will be modified to include "
54 "the converted tools. If the provided destination does not exist, a new file will"
55 "be created.")
56 parser.add_argument("-g", "--galaxy-tool-path", dest="galaxy_tool_path", default=None, required=False,
57 help="The path that will be prepended to the file names when generating tool_conf.xml")
58 parser.add_argument("-r", "--required-tools", dest="required_tools_file", default=None, required=False,
59 help="Each line of the file will be interpreted as a tool name that needs translation. "
60 "Run with '-h' or '--help' to see a brief example on the format of this file.")
61 parser.add_argument("-s", "--skip-tools", dest="skip_tools_file", default=None, required=False,
62 help="File containing a list of tools for which a Galaxy stub will not be generated. "
63 "Run with '-h' or '--help' to see a brief example on the format of this file.")
64 parser.add_argument("-m", "--macros", dest="macros_files", default=[], nargs="*",
65 action="append", required=None, help="Import the additional given file(s) as macros. "
66 "The macros stdio, requirements and advanced_options are "
67 "required. Please see galaxy/macros.xml for an example of a "
68 "valid macros file. All defined macros will be imported.")
69
70
71 def convert_models(args, parsed_ctds):
72 # validate and prepare the passed arguments
73 validate_and_prepare_args(args)
74
75 # extract the names of the macros and check that we have found the ones we need
76 macros_to_expand = parse_macros_files(args.macros_files)
77
78 # parse the given supported file-formats file
79 supported_file_formats = parse_file_formats(args.formats_file)
80
81 # parse the skip/required tools files
82 skip_tools = parse_tools_list_file(args.skip_tools_file)
83 required_tools = parse_tools_list_file(args.required_tools_file)
84
85 _convert_internal(parsed_ctds,
86 supported_file_formats=supported_file_formats,
87 default_executable_path=args.default_executable_path,
88 add_to_command_line=args.add_to_command_line,
89 blacklisted_parameters=args.blacklisted_parameters,
90 required_tools=required_tools,
91 skip_tools=skip_tools,
92 macros_file_names=args.macros_files,
93 macros_to_expand=macros_to_expand,
94 parameter_hardcoder=args.parameter_hardcoder)
95
96 # generation of galaxy stubs is ready... now, let's see if we need to generate a tool_conf.xml
97 if args.tool_conf_destination is not None:
98 generate_tool_conf(parsed_ctds, args.tool_conf_destination,
99 args.galaxy_tool_path, args.default_category)
100
101 # generate datatypes_conf.xml
102 if args.data_types_destination is not None:
103 generate_data_type_conf(supported_file_formats, args.data_types_destination)
104
105
106 def parse_tools_list_file(tools_list_file):
107 tools_list = None
108 if tools_list_file is not None:
109 tools_list = []
110 with open(tools_list_file) as f:
111 for line in f:
112 if line is None or not line.strip() or line.strip().startswith("#"):
113 continue
114 else:
115 tools_list.append(line.strip())
116
117 return tools_list
118
119
120 def parse_macros_files(macros_file_names):
121 macros_to_expand = set()
122
123 for macros_file_name in macros_file_names:
124 try:
125 macros_file = open(macros_file_name)
126 logger.info("Loading macros from %s" % macros_file_name, 0)
127 root = parse(macros_file).getroot()
128 for xml_element in root.findall("xml"):
129 name = xml_element.attrib["name"]
130 if name in macros_to_expand:
131 logger.warning("Macro %s has already been found. Duplicate found in file %s." %
132 (name, macros_file_name), 0)
133 else:
134 logger.info("Macro %s found" % name, 1)
135 macros_to_expand.add(name)
136 except ParseError, e:
137 raise ApplicationException("The macros file " + macros_file_name + " could not be parsed. Cause: " +
138 str(e))
139 except IOError, e:
140 raise ApplicationException("The macros file " + macros_file_name + " could not be opened. Cause: " +
141 str(e))
142
143 # we depend on "stdio", "requirements" and "advanced_options" to exist on all the given macros files
144 missing_needed_macros = []
145 for required_macro in REQUIRED_MACROS:
146 if required_macro not in macros_to_expand:
147 missing_needed_macros.append(required_macro)
148
149 if missing_needed_macros:
150 raise ApplicationException(
151 "The following required macro(s) were not found in any of the given macros files: %s, "
152 "see galaxy/macros.xml for an example of a valid macros file."
153 % ", ".join(missing_needed_macros))
154
155 # we do not need to "expand" the advanced_options macro
156 macros_to_expand.remove(ADVANCED_OPTIONS_MACRO_NAME)
157 return macros_to_expand
158
159
160 def parse_file_formats(formats_file):
161 supported_formats = {}
162 if formats_file is not None:
163 line_number = 0
164 with open(formats_file) as f:
165 for line in f:
166 line_number += 1
167 if line is None or not line.strip() or line.strip().startswith("#"):
168 # ignore (it'd be weird to have something like:
169 # if line is not None and not (not line.strip()) ...
170 pass
171 else:
172 # not an empty line, no comment
173 # strip the line and split by whitespace
174 parsed_formats = line.strip().split()
175 # valid lines contain either one or four columns
176 if not (len(parsed_formats) == 1 or len(parsed_formats) == 3 or len(parsed_formats) == 4):
177 logger.warning(
178 "Invalid line at line number %d of the given formats file. Line will be ignored:\n%s" %
179 (line_number, line), 0)
180 # ignore the line
181 continue
182 elif len(parsed_formats) == 1:
183 supported_formats[parsed_formats[0]] = DataType(parsed_formats[0], parsed_formats[0])
184 else:
185 mimetype = None
186 # check if mimetype was provided
187 if len(parsed_formats) == 4:
188 mimetype = parsed_formats[3]
189 supported_formats[parsed_formats[0]] = DataType(parsed_formats[0], parsed_formats[1],
190 parsed_formats[2], mimetype)
191 return supported_formats
192
193
194 def validate_and_prepare_args(args):
195 # check that only one of skip_tools_file and required_tools_file has been provided
196 if args.skip_tools_file is not None and args.required_tools_file is not None:
197 raise ApplicationException(
198 "You have provided both a file with tools to ignore and a file with required tools.\n"
199 "Only one of -s/--skip-tools, -r/--required-tools can be provided.")
200
201 # flatten macros_files to make sure that we have a list containing file names and not a list of lists
202 utils.flatten_list_of_lists(args, "macros_files")
203
204 # check that the arguments point to a valid, existing path
205 input_variables_to_check = ["skip_tools_file", "required_tools_file", "macros_files", "formats_file"]
206 for variable_name in input_variables_to_check:
207 utils.validate_argument_is_valid_path(args, variable_name)
208
209 # check that the provided output files, if provided, contain a valid file path (i.e., not a folder)
210 output_variables_to_check = ["data_types_destination", "tool_conf_destination"]
211 for variable_name in output_variables_to_check:
212 file_name = getattr(args, variable_name)
213 if file_name is not None and os.path.isdir(file_name):
214 raise ApplicationException("The provided output file name (%s) points to a directory." % file_name)
215
216 if not args.macros_files:
217 # list is empty, provide the default value
218 logger.warning("Using default macros from galaxy/macros.xml", 0)
219 args.macros_files = ["galaxy/macros.xml"]
220
221
222 def get_preferred_file_extension():
223 return "xml"
224
225
226 def _convert_internal(parsed_ctds, **kwargs):
227 # parse all input files into models using CTDopts (via utils)
228 # the output is a tuple containing the model, output destination, origin file
229 for parsed_ctd in parsed_ctds:
230 model = parsed_ctd.ctd_model
231 origin_file = parsed_ctd.input_file
232 output_file = parsed_ctd.suggested_output_file
233
234 if kwargs["skip_tools"] is not None and model.name in kwargs["skip_tools"]:
235 logger.info("Skipping tool %s" % model.name, 0)
236 continue
237 elif kwargs["required_tools"] is not None and model.name not in kwargs["required_tools"]:
238 logger.info("Tool %s is not required, skipping it" % model.name, 0)
239 continue
240 else:
241 logger.info("Converting %s (source %s)" % (model.name, utils.get_filename(origin_file)), 0)
242 tool = create_tool(model)
243 write_header(tool, model)
244 create_description(tool, model)
245 expand_macros(tool, model, **kwargs)
246 create_command(tool, model, **kwargs)
247 create_inputs(tool, model, **kwargs)
248 create_outputs(tool, model, **kwargs)
249 create_help(tool, model)
250
251 # wrap our tool element into a tree to be able to serialize it
252 tree = ElementTree(tool)
253 logger.info("Writing to %s" % utils.get_filename(output_file), 1)
254 tree.write(open(output_file, 'w'), encoding="UTF-8", xml_declaration=True, pretty_print=True)
255
256
257 def write_header(tool, model):
258 tool.addprevious(etree.Comment(
259 "This is a configuration file for the integration of a tools into Galaxy (https://galaxyproject.org/). "
260 "This file was automatically generated using CTDConverter."))
261 tool.addprevious(etree.Comment('Proposed Tool Section: [%s]' % model.opt_attribs.get("category", "")))
262
263
264 def generate_tool_conf(parsed_ctds, tool_conf_destination, galaxy_tool_path, default_category):
265 # for each category, we keep a list of models corresponding to it
266 categories_to_tools = dict()
267 for parsed_ctd in parsed_ctds:
268 category = strip(parsed_ctd.ctd_model.opt_attribs.get("category", ""))
269 if not category.strip():
270 category = default_category
271 if category not in categories_to_tools:
272 categories_to_tools[category] = []
273 categories_to_tools[category].append(utils.get_filename(parsed_ctd.suggested_output_file))
274
275 # at this point, we should have a map for all categories->tools
276 toolbox_node = Element("toolbox")
277
278 if galaxy_tool_path is not None and not galaxy_tool_path.strip().endswith("/"):
279 galaxy_tool_path = galaxy_tool_path.strip() + "/"
280 if galaxy_tool_path is None:
281 galaxy_tool_path = ""
282
283 for category, file_names in categories_to_tools.iteritems():
284 section_node = add_child_node(toolbox_node, "section")
285 section_node.attrib["id"] = "section-id-" + "".join(category.split())
286 section_node.attrib["name"] = category
287
288 for filename in file_names:
289 tool_node = add_child_node(section_node, "tool")
290 tool_node.attrib["file"] = galaxy_tool_path + filename
291
292 toolconf_tree = ElementTree(toolbox_node)
293 toolconf_tree.write(open(tool_conf_destination,'w'), encoding="UTF-8", xml_declaration=True, pretty_print=True)
294 logger.info("Generated Galaxy tool_conf.xml in %s" % tool_conf_destination, 0)
295
296
297 def generate_data_type_conf(supported_file_formats, data_types_destination):
298 data_types_node = Element("datatypes")
299 registration_node = add_child_node(data_types_node, "registration")
300 registration_node.attrib["converters_path"] = "lib/galaxy/datatypes/converters"
301 registration_node.attrib["display_path"] = "display_applications"
302
303 for format_name in supported_file_formats:
304 data_type = supported_file_formats[format_name]
305 # add only if it's a data type that does not exist in Galaxy
306 if data_type.galaxy_type is not None:
307 data_type_node = add_child_node(registration_node, "datatype")
308 # we know galaxy_extension is not None
309 data_type_node.attrib["extension"] = data_type.galaxy_extension
310 data_type_node.attrib["type"] = data_type.galaxy_type
311 if data_type.mimetype is not None:
312 data_type_node.attrib["mimetype"] = data_type.mimetype
313
314 data_types_tree = ElementTree(data_types_node)
315 data_types_tree.write(open(data_types_destination,'w'), encoding="UTF-8", xml_declaration=True, pretty_print=True)
316 logger.info("Generated Galaxy datatypes_conf.xml in %s" % data_types_destination, 0)
317
318
319 def create_tool(model):
320 return Element("tool", OrderedDict([("id", model.name), ("name", model.name), ("version", model.version)]))
321
322
323 def create_description(tool, model):
324 if "description" in model.opt_attribs.keys() and model.opt_attribs["description"] is not None:
325 description = SubElement(tool,"description")
326 description.text = model.opt_attribs["description"]
327
328
329 def create_command(tool, model, **kwargs):
330 final_command = utils.extract_tool_executable_path(model, kwargs["default_executable_path"]) + '\n'
331 final_command += kwargs["add_to_command_line"] + '\n'
332 advanced_command_start = "#if $adv_opts.adv_opts_selector=='advanced':\n"
333 advanced_command_end = "#end if"
334 advanced_command = ""
335 parameter_hardcoder = kwargs["parameter_hardcoder"]
336
337 found_output_parameter = False
338 for param in utils.extract_and_flatten_parameters(model):
339 if param.type is _OutFile:
340 found_output_parameter = True
341 command = ""
342 param_name = utils.extract_param_name(param)
343 command_line_prefix = utils.extract_command_line_prefix(param, model)
344
345 if param.name in kwargs["blacklisted_parameters"]:
346 continue
347
348 hardcoded_value = parameter_hardcoder.get_hardcoded_value(param_name, model.name)
349 if hardcoded_value:
350 command += "%s %s\n" % (command_line_prefix, hardcoded_value)
351 else:
352 # parameter is neither blacklisted nor hardcoded...
353 galaxy_parameter_name = get_galaxy_parameter_name(param)
354 repeat_galaxy_parameter_name = get_repeat_galaxy_parameter_name(param)
355
356 # logic for ITEMLISTs
357 if param.is_list:
358 if param.type is _InFile:
359 command += command_line_prefix + "\n"
360 command += " #for token in $" + galaxy_parameter_name + ":\n"
361 command += " $token\n"
362 command += " #end for\n"
363 else:
364 command += "\n#if $" + repeat_galaxy_parameter_name + ":\n"
365 command += command_line_prefix + "\n"
366 command += " #for token in $" + repeat_galaxy_parameter_name + ":\n"
367 command += " #if \" \" in str(token):\n"
368 command += " \"$token." + galaxy_parameter_name + "\"\n"
369 command += " #else\n"
370 command += " $token." + galaxy_parameter_name + "\n"
371 command += " #end if\n"
372 command += " #end for\n"
373 command += "#end if\n"
374 # logic for other ITEMs
375 else:
376 if param.advanced and param.type is not _OutFile:
377 actual_parameter = "$adv_opts.%s" % galaxy_parameter_name
378 else:
379 actual_parameter = "$%s" % galaxy_parameter_name
380 # TODO only useful for text fields, integers or floats
381 # not useful for choices, input fields ...
382
383 if not is_boolean_parameter(param) and type(param.restrictions) is _Choices :
384 command += "#if " + actual_parameter + ":\n"
385 command += " %s\n" % command_line_prefix
386 command += " #if \" \" in str(" + actual_parameter + "):\n"
387 command += " \"" + actual_parameter + "\"\n"
388 command += " #else\n"
389 command += " " + actual_parameter + "\n"
390 command += " #end if\n"
391 command += "#end if\n"
392 elif is_boolean_parameter(param):
393 command += "#if " + actual_parameter + ":\n"
394 command += " %s\n" % command_line_prefix
395 command += "#end if\n"
396 elif TYPE_TO_GALAXY_TYPE[param.type] is 'text':
397 command += "#if " + actual_parameter + ":\n"
398 command += " %s " % command_line_prefix
399 command += " \"" + actual_parameter + "\"\n"
400 command += "#end if\n"
401 else:
402 command += "#if " + actual_parameter + ":\n"
403 command += " %s " % command_line_prefix
404 command += actual_parameter + "\n"
405 command += "#end if\n"
406
407 if param.advanced and param.type is not _OutFile:
408 advanced_command += " %s" % command
409 else:
410 final_command += command
411
412 if advanced_command:
413 final_command += "%s%s%s\n" % (advanced_command_start, advanced_command, advanced_command_end)
414
415 if not found_output_parameter:
416 final_command += "> $param_stdout\n"
417
418 command_node = add_child_node(tool, "command")
419 command_node.text = final_command
420
421
422 # creates the xml elements needed to import the needed macros files
423 # and to "expand" the macros
424 def expand_macros(tool, model, **kwargs):
425 macros_node = add_child_node(tool, "macros")
426 token_node = add_child_node(macros_node, "token")
427 token_node.attrib["name"] = "@EXECUTABLE@"
428 token_node.text = utils.extract_tool_executable_path(model, kwargs["default_executable_path"])
429
430 # add <import> nodes
431 for macro_file_name in kwargs["macros_file_names"]:
432 macro_file = open(macro_file_name)
433 import_node = add_child_node(macros_node, "import")
434 # do not add the path of the file, rather, just its basename
435 import_node.text = os.path.basename(macro_file.name)
436
437 # add <expand> nodes
438 for expand_macro in kwargs["macros_to_expand"]:
439 expand_node = add_child_node(tool, "expand")
440 expand_node.attrib["macro"] = expand_macro
441
442
443 def get_galaxy_parameter_name(param):
444 return "param_%s" % utils.extract_param_name(param).replace(":", "_").replace("-", "_")
445
446
447 def get_input_with_same_restrictions(out_param, model, supported_file_formats):
448 for param in utils.extract_and_flatten_parameters(model):
449 if param.type is _InFile:
450 if param.restrictions is not None:
451 in_param_formats = get_supported_file_types(param.restrictions.formats, supported_file_formats)
452 out_param_formats = get_supported_file_types(out_param.restrictions.formats, supported_file_formats)
453 if in_param_formats == out_param_formats:
454 return param
455
456
457 def create_inputs(tool, model, **kwargs):
458 inputs_node = SubElement(tool, "inputs")
459
460 # some suites (such as OpenMS) need some advanced options when handling inputs
461 expand_advanced_node = add_child_node(tool, "expand", OrderedDict([("macro", ADVANCED_OPTIONS_MACRO_NAME)]))
462 parameter_hardcoder = kwargs["parameter_hardcoder"]
463
464 # treat all non output-file parameters as inputs
465 for param in utils.extract_and_flatten_parameters(model):
466 # no need to show hardcoded parameters
467 hardcoded_value = parameter_hardcoder.get_hardcoded_value(param.name, model.name)
468 if param.name in kwargs["blacklisted_parameters"] or hardcoded_value:
469 # let's not use an extra level of indentation and use NOP
470 continue
471 if param.type is not _OutFile:
472 if param.advanced:
473 if expand_advanced_node is not None:
474 parent_node = expand_advanced_node
475 else:
476 # something went wrong... we are handling an advanced parameter and the
477 # advanced input macro was not set... inform the user about it
478 logger.info("The parameter %s has been set as advanced, but advanced_input_macro has "
479 "not been set." % param.name, 1)
480 # there is not much we can do, other than use the inputs_node as a parent node!
481 parent_node = inputs_node
482 else:
483 parent_node = inputs_node
484
485 # for lists we need a repeat tag
486 if param.is_list and param.type is not _InFile:
487 rep_node = add_child_node(parent_node, "repeat")
488 create_repeat_attribute_list(rep_node, param)
489 parent_node = rep_node
490
491 param_node = add_child_node(parent_node, "param")
492 create_param_attribute_list(param_node, param, kwargs["supported_file_formats"])
493
494 # advanced parameter selection should be at the end
495 # and only available if an advanced parameter exists
496 if expand_advanced_node is not None and len(expand_advanced_node) > 0:
497 inputs_node.append(expand_advanced_node)
498
499
500 def get_repeat_galaxy_parameter_name(param):
501 return "rep_" + get_galaxy_parameter_name(param)
502
503
504 def create_repeat_attribute_list(rep_node, param):
505 rep_node.attrib["name"] = get_repeat_galaxy_parameter_name(param)
506 if param.required:
507 rep_node.attrib["min"] = "1"
508 else:
509 rep_node.attrib["min"] = "0"
510 # for the ITEMLISTs which have LISTITEM children we only
511 # need one parameter as it is given as a string
512 if param.default is not None:
513 rep_node.attrib["max"] = "1"
514 rep_node.attrib["title"] = get_galaxy_parameter_name(param)
515
516
517 def create_param_attribute_list(param_node, param, supported_file_formats):
518 param_node.attrib["name"] = get_galaxy_parameter_name(param)
519
520 param_type = TYPE_TO_GALAXY_TYPE[param.type]
521 if param_type is None:
522 raise ModelError("Unrecognized parameter type %(type)s for parameter %(name)s"
523 % {"type": param.type, "name": param.name})
524
525 if param.is_list:
526 param_type = "text"
527
528 if is_selection_parameter(param):
529 param_type = "select"
530 if len(param.restrictions.choices) < 5:
531 param_node.attrib["display"] = "radio"
532
533 if is_boolean_parameter(param):
534 param_type = "boolean"
535
536 if param.type is _InFile:
537 # assume it's just text unless restrictions are provided
538 param_format = "txt"
539 if param.restrictions is not None:
540 # join all formats of the file, take mapping from supported_file if available for an entry
541 if type(param.restrictions) is _FileFormat:
542 param_format = ",".join([get_supported_file_type(i, supported_file_formats) if
543 get_supported_file_type(i, supported_file_formats)
544 else i for i in param.restrictions.formats])
545 else:
546 raise InvalidModelException("Expected 'file type' restrictions for input file [%(name)s], "
547 "but instead got [%(type)s]"
548 % {"name": param.name, "type": type(param.restrictions)})
549
550 param_node.attrib["type"] = "data"
551 param_node.attrib["format"] = param_format
552 # in the case of multiple input set multiple flag
553 if param.is_list:
554 param_node.attrib["multiple"] = "true"
555
556 else:
557 param_node.attrib["type"] = param_type
558
559 # check for parameters with restricted values (which will correspond to a "select" in galaxy)
560 if param.restrictions is not None:
561 # it could be either _Choices or _NumericRange, with special case for boolean types
562 if param_type == "boolean":
563 create_boolean_parameter(param_node, param)
564 elif type(param.restrictions) is _Choices:
565 # create as many <option> elements as restriction values
566 for choice in param.restrictions.choices:
567 option_node = add_child_node(param_node, "option", OrderedDict([("value", str(choice))]))
568 option_node.text = str(choice)
569
570 # preselect the default value
571 if param.default == choice:
572 option_node.attrib["selected"] = "true"
573
574 elif type(param.restrictions) is _NumericRange:
575 if param.type is not int and param.type is not float:
576 raise InvalidModelException("Expected either 'int' or 'float' in the numeric range restriction for "
577 "parameter [%(name)s], but instead got [%(type)s]" %
578 {"name": param.name, "type": type(param.restrictions)})
579 # extract the min and max values and add them as attributes
580 # validate the provided min and max values
581 if param.restrictions.n_min is not None:
582 param_node.attrib["min"] = str(param.restrictions.n_min)
583 if param.restrictions.n_max is not None:
584 param_node.attrib["max"] = str(param.restrictions.n_max)
585 elif type(param.restrictions) is _FileFormat:
586 param_node.attrib["format"] = ','.join([get_supported_file_type(i, supported_file_formats) if
587 get_supported_file_type(i, supported_file_formats)
588 else i for i in param.restrictions.formats])
589 else:
590 raise InvalidModelException("Unrecognized restriction type [%(type)s] for parameter [%(name)s]"
591 % {"type": type(param.restrictions), "name": param.name})
592
593 if param_type == "select" and param.default in param.restrictions.choices:
594 param_node.attrib["optional"] = "False"
595 else:
596 param_node.attrib["optional"] = str(not param.required)
597
598 if param_type == "text":
599 # add size attribute... this is the length of a textbox field in Galaxy (it could also be 15x2, for instance)
600 param_node.attrib["size"] = "30"
601 # add sanitizer nodes, this is needed for special character like "["
602 # which are used for example by FeatureFinderMultiplex
603 sanitizer_node = SubElement(param_node, "sanitizer")
604
605 valid_node = SubElement(sanitizer_node, "valid", OrderedDict([("initial", "string.printable")]))
606 add_child_node(valid_node, "remove", OrderedDict([("value", '\'')]))
607 add_child_node(valid_node, "remove", OrderedDict([("value", '"')]))
608
609 # check for default value
610 if param.default is not None and param.default is not _Null:
611 if type(param.default) is list:
612 # we ASSUME that a list of parameters looks like:
613 # $ tool -ignore He Ar Xe
614 # meaning, that, for example, Helium, Argon and Xenon will be ignored
615 param_node.attrib["value"] = ' '.join(map(str, param.default))
616
617 elif param_type != "boolean":
618 param_node.attrib["value"] = str(param.default)
619
620 else:
621 # simple boolean with a default
622 if param.default is True:
623 param_node.attrib["checked"] = "true"
624 else:
625 if param.type is int or param.type is float:
626 # galaxy requires "value" to be included for int/float
627 # since no default was included, we need to figure out one in a clever way... but let the user know
628 # that we are "thinking" for him/her
629 logger.warning("Generating default value for parameter [%s]. "
630 "Galaxy requires the attribute 'value' to be set for integer/floats. "
631 "Edit the CTD file and provide a suitable default value." % param.name, 1)
632 # check if there's a min/max and try to use them
633 default_value = None
634 if param.restrictions is not None:
635 if type(param.restrictions) is _NumericRange:
636 default_value = param.restrictions.n_min
637 if default_value is None:
638 default_value = param.restrictions.n_max
639 if default_value is None:
640 # no min/max provided... just use 0 and see what happens
641 default_value = 0
642 else:
643 # should never be here, since we have validated this anyway...
644 # this code is here just for documentation purposes
645 # however, better safe than sorry!
646 # (it could be that the code changes and then we have an ugly scenario)
647 raise InvalidModelException("Expected either a numeric range for parameter [%(name)s], "
648 "but instead got [%(type)s]"
649 % {"name": param.name, "type": type(param.restrictions)})
650 else:
651 # no restrictions and no default value provided...
652 # make up something
653 default_value = 0
654 param_node.attrib["value"] = str(default_value)
655
656 label = "%s parameter" % param.name
657 help_text = ""
658
659 if param.description is not None:
660 label, help_text = generate_label_and_help(param.description)
661
662 param_node.attrib["label"] = label
663 param_node.attrib["help"] = "(-%s)" % param.name + " " + help_text
664
665
666 def generate_label_and_help(desc):
667 help_text = ""
668 # This tag is found in some descriptions
669 if not isinstance(desc, basestring):
670 desc = str(desc)
671 desc = desc.encode("utf8").replace("#br#", " <br>")
672 # Get rid of dots in the end
673 if desc.endswith("."):
674 desc = desc.rstrip(".")
675 # Check if first word is a normal word and make it uppercase
676 if str(desc).find(" ") > -1:
677 first_word, rest = str(desc).split(" ", 1)
678 if str(first_word).islower():
679 # check if label has a quotient of the form a/b
680 if first_word.find("/") != 1 :
681 first_word.capitalize()
682 desc = first_word + " " + rest
683 label = desc.decode("utf8")
684
685 # Try to split the label if it is too long
686 if len(desc) > 50:
687 # find an example and put everything before in the label and the e.g. in the help
688 if desc.find("e.g.") > 1 :
689 label, help_text = desc.split("e.g.",1)
690 help_text = "e.g." + help_text
691 else:
692 # find the end of the first sentence
693 # look for ". " because some labels contain .file or something similar
694 delimiter = ""
695 if desc.find(". ") > 1 and desc.find("? ") > 1:
696 if desc.find(". ") < desc.find("? "):
697 delimiter = ". "
698 else:
699 delimiter = "? "
700 elif desc.find(". ") > 1:
701 delimiter = ". "
702 elif desc.find("? ") > 1:
703 delimiter = "? "
704 if delimiter != "":
705 label, help_text = desc.split(delimiter, 1)
706
707 # add the question mark back
708 if delimiter == "? ":
709 label += "? "
710
711 # remove all linebreaks
712 label = label.rstrip().rstrip('<br>').rstrip()
713 return label, help_text
714
715
716 # determines if the given choices are boolean (basically, if the possible values are yes/no, true/false)
717 def is_boolean_parameter(param):
718 # detect boolean selects of OpenMS
719 if is_selection_parameter(param):
720 if len(param.restrictions.choices) == 2:
721 # check that default value is false to make sure it is an actual flag
722 if "false" in param.restrictions.choices and \
723 "true" in param.restrictions.choices and \
724 param.default == "false":
725 return True
726 else:
727 return param.type is bool
728
729
730 # determines if there are choices for the parameter
731 def is_selection_parameter(param):
732 return type(param.restrictions) is _Choices
733
734
735 def get_lowercase_list(some_list):
736 lowercase_list = map(str, some_list)
737 lowercase_list = map(string.lower, lowercase_list)
738 lowercase_list = map(strip, lowercase_list)
739 return lowercase_list
740
741
742 # creates a galaxy boolean parameter type
743 # this method assumes that param has restrictions, and that only two restictions are present
744 # (either yes/no or true/false)
745 def create_boolean_parameter(param_node, param):
746 # first, determine the 'truevalue' and the 'falsevalue'
747 """TODO: true and false values can be way more than 'true' and 'false'
748 but for that we need CTD support
749 """
750 # by default, 'true' and 'false' are handled as flags, like the verbose flag (i.e., -v)
751 true_value = "-%s" % utils.extract_param_name(param)
752 false_value = ""
753 choices = get_lowercase_list(param.restrictions.choices)
754 if "yes" in choices:
755 true_value = "yes"
756 false_value = "no"
757 param_node.attrib["truevalue"] = true_value
758 param_node.attrib["falsevalue"] = false_value
759
760 # set the checked attribute
761 if param.default is not None:
762 checked_value = "false"
763 default = strip(string.lower(param.default))
764 if default == "yes" or default == "true":
765 checked_value = "true"
766 param_node.attrib["checked"] = checked_value
767
768
769 def create_outputs(parent, model, **kwargs):
770 outputs_node = add_child_node(parent, "outputs")
771 parameter_hardcoder = kwargs["parameter_hardcoder"]
772
773 for param in utils.extract_and_flatten_parameters(model):
774
775 # no need to show hardcoded parameters
776 hardcoded_value = parameter_hardcoder.get_hardcoded_value(param.name, model.name)
777 if param.name in kwargs["blacklisted_parameters"] or hardcoded_value:
778 # let's not use an extra level of indentation and use NOP
779 continue
780 if param.type is _OutFile:
781 create_output_node(outputs_node, param, model, kwargs["supported_file_formats"])
782
783 # If there are no outputs defined in the ctd the node will have no children
784 # and the stdout will be used as output
785 if len(outputs_node) == 0:
786 add_child_node(outputs_node, "data",
787 OrderedDict([("name", "param_stdout"), ("format", "txt"), ("label", "Output from stdout")]))
788
789
790 def create_output_node(parent, param, model, supported_file_formats):
791 data_node = add_child_node(parent, "data")
792 data_node.attrib["name"] = get_galaxy_parameter_name(param)
793
794 data_format = "data"
795 if param.restrictions is not None:
796 if type(param.restrictions) is _FileFormat:
797 # set the first data output node to the first file format
798
799 # check if there are formats that have not been registered yet...
800 output = list()
801 for format_name in param.restrictions.formats:
802 if not format_name in supported_file_formats.keys():
803 output.append(str(format_name))
804
805 # warn only if there's about to complain
806 if output:
807 logger.warning("Parameter " + param.name + " has the following unsupported format(s):"
808 + ','.join(output), 1)
809 data_format = ','.join(output)
810
811 formats = get_supported_file_types(param.restrictions.formats, supported_file_formats)
812 try:
813 data_format = formats.pop()
814 except KeyError:
815 # there is not much we can do, other than catching the exception
816 pass
817 # if there are more than one output file formats try to take the format from the input parameter
818 if formats:
819 corresponding_input = get_input_with_same_restrictions(param, model, supported_file_formats)
820 if corresponding_input is not None:
821 data_format = "input"
822 data_node.attrib["metadata_source"] = get_galaxy_parameter_name(corresponding_input)
823 else:
824 raise InvalidModelException("Unrecognized restriction type [%(type)s] "
825 "for output [%(name)s]" % {"type": type(param.restrictions),
826 "name": param.name})
827 data_node.attrib["format"] = data_format
828
829 # TODO: find a smarter label ?
830 return data_node
831
832
833 # Get the supported file format for one given format
834 def get_supported_file_type(format_name, supported_file_formats):
835 if format_name in supported_file_formats.keys():
836 return supported_file_formats.get(format_name, DataType(format_name, format_name)).galaxy_extension
837 else:
838 return None
839
840
841 def get_supported_file_types(formats, supported_file_formats):
842 return set([supported_file_formats.get(format_name, DataType(format_name, format_name)).galaxy_extension
843 for format_name in formats if format_name in supported_file_formats.keys()])
844
845
846 def create_change_format_node(parent, data_formats, input_ref):
847 # <change_format>
848 # <when input="secondary_structure" value="true" format="txt"/>
849 # </change_format>
850 change_format_node = add_child_node(parent, "change_format")
851 for data_format in data_formats:
852 add_child_node(change_format_node, "when",
853 OrderedDict([("input", input_ref), ("value", data_format), ("format", data_format)]))
854
855
856 # Shows basic information about the file, such as data ranges and file type.
857 def create_help(tool, model):
858 help_node = add_child_node(tool, "help")
859 # TODO: do we need CDATA Section here?
860 help_node.text = utils.extract_tool_help_text(model)
861
862
863 # adds and returns a child node using the given name to the given parent node
864 def add_child_node(parent_node, child_node_name, attributes=OrderedDict([])):
865 child_node = SubElement(parent_node, child_node_name, attributes)
866 return child_node
+31
-0
galaxy/macros.xml less more
0 <?xml version='1.0' encoding='UTF-8'?>
1 <!-- CTD2Galaxy depends on this file and on the stdio, advanced_options macros!
2 You can edit this file to add your own macros, if you so desire, or you can
3 add additional macro files using the m/macros parameter -->
4 <macros>
5 <xml name="requirements">
6 <requirements>
7 <requirement type="binary">@EXECUTABLE@</requirement>
8 </requirements>
9 </xml>
10 <xml name="stdio">
11 <stdio>
12 <exit_code range="1:"/>
13 <exit_code range=":-1"/>
14 <regex match="Error:"/>
15 <regex match="Exception:"/>
16 </stdio>
17 </xml>
18 <xml name="advanced_options">
19 <conditional name="adv_opts">
20 <param name="adv_opts_selector" type="select" label="Advanced Options">
21 <option value="basic" selected="True">Hide Advanced Options</option>
22 <option value="advanced">Show Advanced Options</option>
23 </param>
24 <when value="basic"/>
25 <when value="advanced">
26 <yield/>
27 </when>
28 </conditional>
29 </xml>
30 </macros>