Update upstream source from tag 'upstream/3.5'
Update to upstream version '3.5'
with Debian dir 54a1b3621fa1bc4e810d02773d2bef8c87d9c3ad
Andreas Tille
1 year, 6 months ago
| 0 | 0 | ------------------------------------------------------------*- mode: text -*- |
| 1 | 1 | |
| 2 | #12 ------------------------------------------------------------ | |
| 2 | Please report bugs here: https://github.com/MLopez-Ibanez/irace/issues | |
| 3 | 3 | |
| 4 | irace --check or checkIraceScenario do not work with targetRunnerParallel if | |
| 5 | the user does not use target.runner at all. | |
| 6 | ||
| 7 | #11 ------------------------------------------------------------ | |
| 8 | ||
| 9 | The semantics of recovery are not clear. Should it recover the exact parameters | |
| 10 | given to the original run? If so, then moving to a new execdir will mess up | |
| 11 | things. Should it allow changing parameters after recovery (that is, what to do | |
| 12 | if it finds a new scenario file with different values or it is passed | |
| 13 | additional command-line parameters?). | |
| 14 | ||
| 15 | One possible behavior could be: Recover everything as given originally; if new | |
| 16 | settings are given, either by a configuration file or command-line, override | |
| 17 | them; for settings that are still set to default values in the original, | |
| 18 | recompute the default (because overriden settings may influence the default of | |
| 19 | other settings). Report everything that is restored and overriden. | |
| 20 | ||
| 21 | This means recovery should happen much earlier, before reading | |
| 22 | command-line/scenario file. | |
| 23 | ||
| 24 | ||
| 25 | #9 ------------------------------------------------------------ | |
| 26 | ||
| 27 | By default | |
| 28 | # trainInstancesDir = "./Instances/" | |
| 29 | # testInstancesDir = "" | |
| 30 | ||
| 31 | So not specifying any directory, like: | |
| 32 | ||
| 33 | ## File with a list of instances and (optionally) parameters. | |
| 34 | ## If empty or NULL, do not use a file. | |
| 35 | trainInstancesFile = "instances-train.txt" | |
| 36 | ||
| 37 | ## File containing a list of test instances and optionally additional | |
| 38 | ## parameters for them. If empty or NULL, do not use a file. | |
| 39 | testInstancesFile = "instances-test.txt" | |
| 40 | ||
| 41 | defaults to for train instances to Instances/train-instance-1 ... | |
| 42 | but to ./test-instance-1 ... for test instances, which is unexpected. | |
| 43 | ||
| 44 | # FIXME: Use NA as default value for testInstancesFile/testInstancesDir. | |
| 45 | If testInstancesFile != NA and testInstancesDir == NA, | |
| 46 | then make testInstancesDir = trainInstancesDir. Maybe give a warning when doing this. | |
| 47 | ||
| 48 | ||
| 49 | #8 ------------------------------------------------------------ | |
| 50 | ||
| 51 | We do not re-evaluate the elitist (using which.exe), but we do call | |
| 52 | target.evaluator on them. See this code in race.R: | |
| 53 | ||
| 54 | # Execute commands | |
| 55 | if (length(which.exe) > 0) { | |
| 56 | which.exps <- which(which.alive %in% which.exe) | |
| 57 | target.output[which.exps] <- execute.experiments (experiments[which.exps], scenario) | |
| 58 | } | |
| 59 | irace.assert(!any(sapply(target.output, is.null))) | |
| 60 | ||
| 61 | # target.evaluator may be NULL. If so, target.output must | |
| 62 | # contain the right output already. | |
| 63 | if (!is.null(.irace$target.evaluator)) | |
| 64 | target.output <- execute.evaluator (experiments, scenario, target.output, | |
| 65 | configurations[which.alive, ".ID."]) | |
| 66 | ||
| 67 | The reason for that is, in some scenarios, we want to normalize the output of | |
| 68 | the algorithm, so when we evaluate new configurations, there is a need to | |
| 69 | re-normalize the output of the elites as well. So we need to call | |
| 70 | target-evaluator again. | |
| 71 | ||
| 72 | However, when recovering, the files needed by target.evaluator are likely | |
| 73 | lost. We would need to call again target-runner. | |
| 74 | ||
| 75 | This is perhaps a design mistake: We could simply do the normalization internally in irace or create a separate target script (target-re-evaluator?) that only people needing something like re-normalization will use. | |
| 76 | ||
| 77 | As a short-term fix, we could implement something like: | |
| 78 | ||
| 79 | If recovering and target-evaluator is in use, keep elites as elites, but re-run target-runner on them. (Perhaps we need a new .irace$ flag saying: first race from recovery). | |
| 80 | ||
| 81 | ||
| 82 | #6 ------------------------------------------------------------ | |
| 83 | ||
| 84 | Sometimes system2 does not say what exactly failed or what command was | |
| 85 | run when something failed, it just prints: "error in running command". | |
| 86 | Can we provide more details? | |
| 87 | ||
| 88 | ||
| 89 | #2 ------------------------------------------------------------ | |
| 90 | ||
| 91 | A parameter like: | |
| 92 | ||
| 93 | a "" c (0, 5, 10, 20) | |
| 94 | ||
| 95 | and a condition like a > 10 will return TRUE when a is 5, because a | |
| 96 | is actually "5" so the comparison is lexicographic. How to solve this? | |
| 97 | ||
| 98 | - Fix 1. Check if a can be converted to numeric, then force it before | |
| 99 | evaluating conditions. However, we cannot know for sure if the user | |
| 100 | actually wants to convert a to numeric or not. Imagine a = "+3", then | |
| 101 | a == "+3" would fail! | |
| 102 | ||
| 103 | - Fix 2. Force users to use quotes when defining categorical and | |
| 104 | ordinal parameters. Hopefully this will make obvious that these | |
| 105 | values are strings and cannot be compared with other numbers. | |
| 106 | ||
| 107 | This is already documented in the user-guide. | |
| 108 | ||
| 4 | This file is only kept for historical purposes. | |
| 109 | 5 | |
| 110 | 6 | ############################################################### |
| 111 | 7 | # FIXED bugs |
| 6 | 6 | settings given a set of instances of an optimization problem. |
| 7 | 7 | M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, |
| 8 | 8 | and M. Birattari (2016) <doi:10.1016/j.orp.2016.09.002>. |
| 9 | Version: 3.4.1 | |
| 9 | Version: 3.5 | |
| 10 | 10 | Authors@R: c(person("Manuel", "López-Ibáñez", role = c("aut", "cre"), |
| 11 | 11 | email = "manuel.lopez-ibanez@manchester.ac.uk", |
| 12 | 12 | comment = c(ORCID = "0000-0001-9974-1295")), |
| 20 | 20 | Depends: R (>= 3.2.0) |
| 21 | 21 | Imports: stats, utils, compiler, R6 |
| 22 | 22 | Suggests: Rmpi (>= 0.6.0), parallel, knitr, testthat, withr, mlr (>= |
| 23 | 2.15.0), ParamHelpers, devtools | |
| 23 | 2.15.0), ParamHelpers, devtools, covr | |
| 24 | 24 | VignetteBuilder: knitr |
| 25 | 25 | License: GPL (>= 2) |
| 26 | URL: http://iridia.ulb.ac.be/irace, | |
| 26 | URL: https://mlopez-ibanez.github.io/irace/, | |
| 27 | 27 | https://github.com/MLopez-Ibanez/irace |
| 28 | 28 | BugReports: https://github.com/MLopez-Ibanez/irace/issues |
| 29 | 29 | ByteCompile: yes |
| 30 | LazyData: yes | |
| 31 | 30 | Encoding: UTF-8 |
| 32 | RoxygenNote: 6.1.1 | |
| 31 | RoxygenNote: 7.1.1 | |
| 33 | 32 | SystemRequirements: GNU make |
| 34 | 33 | NeedsCompilation: yes |
| 35 | Packaged: 2020-03-31 08:07:18 UTC; manu | |
| 34 | Packaged: 2022-10-23 13:44:18 UTC; manu | |
| 36 | 35 | Author: Manuel López-Ibáñez [aut, cre] |
| 37 | 36 | (<https://orcid.org/0000-0001-9974-1295>), |
| 38 | 37 | Jérémie Dubois-Lacoste [aut], |
| 44 | 43 | Nguyen Dang [ctb] |
| 45 | 44 | Maintainer: Manuel López-Ibáñez <manuel.lopez-ibanez@manchester.ac.uk> |
| 46 | 45 | Repository: CRAN |
| 47 | Date/Publication: 2020-03-31 13:40:03 UTC | |
| 46 | Date/Publication: 2022-10-23 14:55:06 UTC | |
| 0 | 8f9f201288b16a339b6f57273a707d1f *BUGS | |
| 1 | 90442f63f91cc4c296eef4b18123d897 *DESCRIPTION | |
| 2 | 929993335cf3a237b7d27e9c1c1a0486 *NAMESPACE | |
| 3 | da2fd1953cd33a19dfc3f9d78a435863 *NEWS.md | |
| 4 | f728d8206d29e524728b98b5e5c98bc4 *R/ablation.R | |
| 5 | 47187f0811097d94d2f3abcf83e97dca *R/argparser.R | |
| 6 | 05a84aa85867928116ca1da97c6bc56f *R/cluster.R | |
| 7 | 77a6154db7d6d3202a998bffd562cf7b *R/generation.R | |
| 8 | 1570a3c84a2eabf122b979f12e47a11f *R/irace-options.R | |
| 9 | 019a192de8c07048f5b05b6294a42fa1 *R/irace-package.R | |
| 10 | 21751b754f699b991754b2d5310ba37a *R/irace.R | |
| 11 | 8eed0fa02814e45224e5d3b37e91b038 *R/irace2pyimp.R | |
| 12 | f55a403c2ea36bad4fb84f79096acfa1 *R/main.R | |
| 13 | 477a890824f2689de002705efa5d5e7b *R/model.R | |
| 14 | 607c6e1c1944510033038d794664c271 *R/parameterAnalysis.R | |
| 15 | e403df46a7a45f16e35b104948f42b02 *R/parameterExploration.R | |
| 16 | b5f0dad93e467221992a310ab8ab80d3 *R/race-wrapper.R | |
| 17 | d585220e5670cc23ce454f67a2353a7a *R/race.R | |
| 18 | 9745a0ec7a762830726686a84ef8d5f2 *R/readConfiguration.R | |
| 19 | fc1fc0c4096fabbfbd1af7f46860ba76 *R/readParameters.R | |
| 20 | 4c0cd07ce0136e79a6dc457ddb15f57f *R/testing.R | |
| 21 | 58e5c86ea54797458928b7171e1beaed *R/tnorm.R | |
| 22 | 2f7cc13b106b43c8c01d1531a12a5c4d *R/utils.R | |
| 23 | 49ef85c2dd41fbca000ecccd19ea0807 *R/version.R | |
| 24 | 87efc0b972f0c2596be88c2b8bf5238a *README.md | |
| 25 | 8bba10219a86dfabf2c028f10e24c361 *build/partial.rdb | |
| 26 | 6d005f12fcd5b771eb3775e7e6bb9ae3 *build/vignette.rds | |
| 27 | 5c3dba114664cbebb390e2912437dd22 *cleanup | |
| 0 | 69a04f8f4c0abbd39178a3d0c7f3ce10 *BUGS | |
| 1 | 5045e033d6ef0ac2cadce585596446bd *DESCRIPTION | |
| 2 | a63f58132973b19bf43020d7a5f15a8d *NAMESPACE | |
| 3 | 038a593734b0b4fcccde699be7b7ce20 *NEWS.md | |
| 4 | b1f319dfbac17e70234c634a2490b632 *R/ablation.R | |
| 5 | 9c80cd3c08b5cb818c94fd65f81907a2 *R/argparser.R | |
| 6 | 5e50e779e5017e05cd2fa56b684f6c06 *R/cluster.R | |
| 7 | a0a9019a8fff086a6b04aba345855c11 *R/generation.R | |
| 8 | 92e15c99068d5698c8b5f396abf50212 *R/irace-options.R | |
| 9 | 76968ff9efe28f18642b4d4a3163ae66 *R/irace-package.R | |
| 10 | b56a08ad22985062d39a85653a4f4e4f *R/irace.R | |
| 11 | 915e1242e234fc7b6ec17a0793781a3c *R/main.R | |
| 12 | fef3c5475713f5f09a5580f3225ab771 *R/model.R | |
| 13 | d1301df655fad850e615a3f62d536179 *R/parameterAnalysis.R | |
| 14 | 86ead6121525dc45cf04a3f54ead0c19 *R/parameterExploration.R | |
| 15 | 068f4659da9911b4e24e68c81ba0614f *R/path_rel2abs.R | |
| 16 | 213f69a714e687be269920999074c87c *R/race-wrapper.R | |
| 17 | 816656412e80a1f170fc8938d968c25d *R/race.R | |
| 18 | ee9dff2757fd0537a3f43759518bc37b *R/readConfiguration.R | |
| 19 | bc45aa88d1d4ff5d05a22778123b18a4 *R/readParameters.R | |
| 20 | 519ff55d3b1b32a21363ad4e4bbfa133 *R/testing.R | |
| 21 | bc04ea89773012f517adc374a6cc6cd9 *R/timer.R | |
| 22 | 55b4b74d608899de8109a8a136490e96 *R/tnorm.R | |
| 23 | 4fd568645b95fe328c7d2843442f1c89 *R/utils.R | |
| 24 | 4904572120e0067c272b3b2335d5a8d1 *R/version.R | |
| 25 | 2a118b7354fdedff5c74246d8423e5ad *R/zzz.R | |
| 26 | 559d78ef5ba4df1b8666192ac468df59 *README.md | |
| 27 | 92d3ace7e0265aaf74b473cadbcf470a *build/partial.rdb | |
| 28 | a852616f9564563cd4aadf7640345486 *build/vignette.rds | |
| 29 | 76b4951f93d078c5a1598e781fd9e670 *cleanup | |
| 28 | 30 | 4fb959d1be3d68345289530627355ca9 *inst/CITATION |
| 29 | 31 | f694f4bed8633608f1914f539c2dfee4 *inst/bin/parallel-irace |
| 30 | 2b6592cf4f79610bdbcc504a4e70bc28 *inst/bin/parallel-irace-mpi | |
| 31 | 89f5edb040e5d72e0b93ea460b11d42d *inst/bin/parallel-irace-qsub | |
| 32 | 6033d97deebd3a6c8e4b1bd4b7433b8e *inst/doc/irace-package.R | |
| 33 | c6f6dde37bededb32d96fa6889891b2a *inst/doc/irace-package.Rnw | |
| 34 | 25df86126dd2756b08d7f257caafbc1d *inst/doc/irace-package.pdf | |
| 32 | 36995f96e183cdd8e7e25207c4d95394 *inst/bin/parallel-irace-mpi | |
| 33 | 404f816e7b4fdf2a142c65c931f300ce *inst/bin/parallel-irace-qsub | |
| 34 | 8eda33b154a7c9cff1a90c0e168da70e *inst/bin/parallel-irace-slurm | |
| 35 | 1263d7dbd890bf549235501fbaf60c05 *inst/doc/irace-package.R | |
| 36 | c2e0a6fcffab3c32cdfc05ff2d5aa126 *inst/doc/irace-package.Rnw | |
| 37 | afa8bf9cb475e5f5f0cde602756d274b *inst/doc/irace-package.pdf | |
| 38 | 6f5adad387e00e68a7802e8d543b18ce *inst/examples/README.md | |
| 35 | 39 | 067fe2821c16cf8262dece7a4c36144e *inst/examples/Spear/README |
| 36 | 40 | 4a9044d0341a321c352ec901cb97a8f0 *inst/examples/Spear/parameters-cat.txt |
| 37 | 41 | cb2da2a31c96ffc3a4274e9a8ef3832a *inst/examples/Spear/parameters-mixed.txt |
| 38 | 42 | ed518f04134d2ca0f90b7393467b4e15 *inst/examples/Spear/scenario.txt |
| 39 | 43 | 039051117ef71079e1d2ff106d6c4e7a *inst/examples/Spear/target-runner |
| 40 | f2c52ac9107d5cfcd53901d5c7ef94e9 *inst/examples/acotsp/README | |
| 44 | d24870bcca1038abb7a62d97df77cbd2 *inst/examples/acotsp/README | |
| 41 | 45 | d14e5bb16ddb374e204cabfc0be4545d *inst/examples/acotsp/default.txt |
| 42 | 46 | 0f16709b42a05455f08906576ba3ee18 *inst/examples/acotsp/forbidden.txt |
| 43 | 47 | 45771f3ceccfe39d9568abda93d7df24 *inst/examples/acotsp/parameters-acotsp.txt |
| 44 | 48 | cc17eef505606543c0d5536f2c34b079 *inst/examples/acotsp/scenario.txt |
| 45 | 60be927405606a9e2f2668039ec1fde3 *inst/examples/acotsp/target-runner | |
| 49 | addd8e7ae0e2ca926476aa9b9e13cb08 *inst/examples/acotsp/target-runner | |
| 46 | 50 | b7d067aba94e5b7fc1e92237a6f0a1b4 *inst/examples/batchmode-cluster/README |
| 47 | 51 | d960f69eeadff7405e4452f8f50c710a *inst/examples/batchmode-cluster/irace-sge-cluster |
| 48 | 1f0d72ccb5c598003e563fee4e06dcb4 *inst/examples/batchmode-cluster/target-evaluator | |
| 52 | 0b3da55eb761846dcf247a23060ffb0a *inst/examples/batchmode-cluster/target-evaluator | |
| 53 | cb7e8b282835fd66d0745aa16dfdecbf *inst/examples/batchmode-cluster/target-runner-htcondor | |
| 49 | 54 | 682e5b6a084e3e94e0ae045b60de8f9f *inst/examples/batchmode-cluster/target-runner-pbs |
| 50 | 55 | 3cd7828292c3927406d34868ad9266b0 *inst/examples/batchmode-cluster/target-runner-sge |
| 51 | 56 | 40fbfe617cda99828251f4b2128a4076 *inst/examples/batchmode-cluster/target-runner-slurm |
| 52 | 57 | 3523d53e5d616ae1343cbb787b703b5c *inst/examples/hypervolume/README |
| 53 | 58 | 5d16974368bcb97b8dcf9594b19152f2 *inst/examples/hypervolume/target-evaluator |
| 54 | 59 | 74406c35bc1a89a6e202f9aa27f48817 *inst/examples/hypervolume/target-runner |
| 55 | 70ca30740880fa79fc905341f90dfc09 *inst/examples/irace2pyimp/002-TemplateDesign/irace.Rdata | |
| 56 | 6a0d04e2c0edf0c32c442bb98678a4e4 *inst/examples/irace2pyimp/002-TemplateDesign/run.sh | |
| 57 | 902005160f28d6d22be7c517b6ffb04d *inst/examples/irace2pyimp/acotsp-forbidden/features.csv | |
| 58 | 25bdee597b648c0b2bdf94f8e3634d33 *inst/examples/irace2pyimp/acotsp-forbidden/irace.Rdata | |
| 59 | 397ac9afae6495e8602cc0af0c0037ee *inst/examples/irace2pyimp/acotsp-forbidden/run.sh | |
| 60 | 902005160f28d6d22be7c517b6ffb04d *inst/examples/irace2pyimp/acotsp/features.csv | |
| 61 | e76df5487875f5c1900196d164809e6d *inst/examples/irace2pyimp/acotsp/irace.Rdata | |
| 62 | 25f67c1549f30889eb348d6a6e6a3ff1 *inst/examples/irace2pyimp/acotsp/run.sh | |
| 63 | 3e4174afe9c5d85bcecb1ba765e94b40 *inst/examples/matlab/RUN.m | |
| 64 | ef092e94145f01eea192cef2f1cdf3d8 *inst/examples/matlab/parameters.txt | |
| 65 | 6200a5e84d27116627d145d0ecbd0ece *inst/examples/matlab/scenario2.txt | |
| 66 | 4401349ba53ea145723c8393cb6ababb *inst/examples/matlab/target-runner | |
| 67 | a7eb13541113d2c2b51f6c71e57dbe7c *inst/examples/moaco/README | |
| 60 | 7a908b35eff27e6ed9fe12d5936f8018 *inst/examples/matlab/Main.m | |
| 61 | 3b0332e02daabf31651a5a0d81ba830a *inst/examples/matlab/instances.txt | |
| 62 | 570fbc605de77856756be963e4fb8c91 *inst/examples/matlab/parameters.txt | |
| 63 | a7c2b75263639028f078e5096a8c2be0 *inst/examples/matlab/scenario.txt | |
| 64 | 117ed65e989d5b1d901431e35c392b00 *inst/examples/moaco/README | |
| 68 | 65 | 6bd5d5787a551af4e396b6681960681f *inst/examples/moaco/parameters.txt |
| 69 | 66 | 047f8de89d4747cd3526a4c7e11075c3 *inst/examples/moaco/scenario.txt |
| 70 | 67 | 5157ef3fc2efb9230f76aef1cfe73291 *inst/examples/moaco/target-evaluator |
| 71 | 68 | e250b38c64ed8030bc71ee66ec4f8941 *inst/examples/moaco/target-runner |
| 72 | 69 | de151ff3fce62d302ea77b93c8a5c92b *inst/examples/target-runner-python/target-runner-acotsp.py |
| 73 | 70 | 5ab7fbf168799bc1c7a54c9b8549a159 *inst/examples/target-runner-python/target-runner-advanced.py |
| 71 | fc5b988b87d11f4e54a72ced66b72fb2 *inst/examples/target-runner-python/target-runner-python-win.bat | |
| 74 | 72 | 2286cd7e0e526fa1526ea38a612d476b *inst/examples/target-runner-python/trivial/instances.txt |
| 75 | 73 | e3f5b6aee118bd70096440b0d648e812 *inst/examples/target-runner-python/trivial/parameters.txt |
| 76 | 74 | e35dda63b5e6a98c1c15b392693ac633 *inst/examples/target-runner-python/trivial/scenario.txt |
| 77 | 75 | 3dcae77aa87c659dff7dd85639ed2e27 *inst/examples/target-runner-python/trivial/target-runner.py |
| 78 | 4dd3073b98f2b4723f68ebcb4eeace03 *inst/exdata/irace-acotsp.Rdata | |
| 79 | a86b3efa1a616252fbefff57796ea8b5 *inst/exdata/sann.rda | |
| 76 | fdf194d8f19dada2867eff7a0695139d *inst/exdata/irace-acotsp.Rdata | |
| 77 | 15d6164b0d369f64e77ed992d0ec3385 *inst/exdata/log-ablation.Rdata | |
| 78 | 38aec57159de03f0ff4d710331413041 *inst/exdata/sann.rda | |
| 79 | a523dd89b6ec7ba2adc26e2ed3a3a179 *inst/irace.sindef | |
| 80 | 80 | 5718f32d8359ec715f2d8bc1a0ec02b3 *inst/templates/configurations.txt.tmpl |
| 81 | 81 | 635cce18d757a309cf38afafd9ebda9d *inst/templates/forbidden.txt.tmpl |
| 82 | 2733cea5852f923b71bc3086b5d04592 *inst/templates/instances-list.txt.tmpl | |
| 82 | af9376428b2c2d911710fd6e51ecf784 *inst/templates/instances-list.txt.tmpl | |
| 83 | 83 | 261e74e936e2217cb612278c5d231887 *inst/templates/parameters.txt.tmpl |
| 84 | 9c73861922818cb0a328735cfa033332 *inst/templates/scenario.txt.tmpl | |
| 84 | 65bde26c9a3d3b154c68a90381fe889e *inst/templates/scenario.txt.tmpl | |
| 85 | 85 | 0185c79ae7b6f5994f89da433acd7b94 *inst/templates/target-evaluator.tmpl |
| 86 | 94feb17231e156778d4e28ea3f45f1ca *inst/templates/target-runner.tmpl | |
| 86 | ce1e0647e76140eac038f0a1e0f30d2a *inst/templates/target-runner.tmpl | |
| 87 | 87 | 4850d7fbd68fbc7c06af53d3c510734f *inst/templates/windows/target-runner.bat |
| 88 | 79e960ad8a32ad97019252d27cef0f85 *man/ablation.Rd | |
| 89 | f7c1bfa57a8d880f6c0fe00d2c84c0d4 *man/buildCommandLine.Rd | |
| 90 | 1fedf949064f213cb77f96163204ae1e *man/checkIraceScenario.Rd | |
| 91 | b3b549648e43a6e29ba0a9b9ec9bb80d *man/checkScenario.Rd | |
| 92 | 852a7b0994e6e5000f5760213f5c3b9e *man/configurations.print.Rd | |
| 93 | b0a694214053c371c5a8d0847fbdad53 *man/configurations.print.command.Rd | |
| 94 | ffa323a082c23e3383246e6fd1a0bf83 *man/configurationsBoxplot.Rd | |
| 95 | 625fcb458cc61915be09c476fcfde5ba *man/defaultScenario.Rd | |
| 96 | 9adffac2d7742ba4fc2747034f2692f2 *man/getConfigurationById.Rd | |
| 97 | e5e93fa55f64b5b5b5edf401319939f5 *man/getConfigurationByIteration.Rd | |
| 98 | e0253c1596549c13a10a29b5c86f4215 *man/getFinalElites.Rd | |
| 99 | 449aa819717c8ae99831b0861d36d92f *man/irace-package.Rd | |
| 100 | 9ed0683b875e228e95ba34ccc4ff8db5 *man/irace.Rd | |
| 101 | 0b166a31d7d1f224695714e5526d9d9c *man/irace.cmdline.Rd | |
| 102 | 72e86d95156bc51a47f00b8f9d477265 *man/irace.license.Rd | |
| 103 | 2ffe037626ebe04abc9cb2038b61d751 *man/irace.main.Rd | |
| 104 | 2ade5885c5cf1fb824d8df9cdd67160b *man/irace.usage.Rd | |
| 105 | a740f833d3612fedf1e33d4bf64ef75a *man/irace.version.Rd | |
| 106 | b8dade7a46d233dacfda4398f2f7bb5f *man/irace2pyimp.Rd | |
| 107 | 1988aeacc8c81ba9734e4cd7715feb66 *man/irace2pyimp_cmdline.Rd | |
| 108 | fec6572fb0e35b89e77627333b0a4f71 *man/parallelCoordinatesPlot.Rd | |
| 109 | c9f68cd60c69d04e3a29804e32fe5cec *man/parameterFrequency.Rd | |
| 110 | 44b2288280813116a2c1bb598fbb34d6 *man/plotAblation.Rd | |
| 111 | 31acaee9cb001ea0df185df8b60a21cb *man/printScenario.Rd | |
| 112 | 430e7a0ed7ae11d1b8b67376a2642559 *man/psRace.Rd | |
| 113 | c3299a7e0324778ba81bad825e192501 *man/readConfigurationsFile.Rd | |
| 114 | b5032e1d497cedeac151f24e68039460 *man/readParameters.Rd | |
| 115 | 1de19eb53f010a4d76089623502ed92d *man/readScenario.Rd | |
| 116 | 17f8b6b6965e8888dd556d2aabf0f221 *man/read_pcs_file.Rd | |
| 117 | f8c4ca3085c228adc5c1387f6661c27c *man/removeConfigurationsMetaData.Rd | |
| 118 | ba808bcb9c03f9e8872669bbc7c4452c *man/scenario.update.paths.Rd | |
| 119 | 05a86b94b4bd798628b916ae1bc60d39 *man/target.evaluator.default.Rd | |
| 120 | 1946a8c451d554ea18de6acb1e53e406 *man/target.runner.default.Rd | |
| 121 | 26530cb06e42ce4114cfd085af70b723 *man/testConfigurations.Rd | |
| 122 | 6900bec9835be9cfcc765d20ad500b27 *man/testing.main.Rd | |
| 123 | 96d2e468dfc0ea6d9ff783c9245ffc9e *src/Makevars | |
| 124 | b225187001f851aea8395d09173a6583 *src/Makevars.win | |
| 125 | 30d0a08b1017171d15bdc9ebc332d159 *src/install.libs.R | |
| 88 | 994adc0a852364fdcea6b274bbc4ba47 *man/CommandArgsParser.Rd | |
| 89 | 6b0dde5d934c2c16a9555cc6305bc8c1 *man/ablation.Rd | |
| 90 | 078aae4ce581e120cd576971577f5e12 *man/ablation_cmdline.Rd | |
| 91 | c1211469bba0c10e00fa80c47a384d7a *man/buildCommandLine.Rd | |
| 92 | d412ab580b7ba52bfc02370a93fe99eb *man/checkIraceScenario.Rd | |
| 93 | 7a1d06d2da2f300a7eed9fe508838824 *man/checkParameters.Rd | |
| 94 | 04e6a8b6f89522fe45d545285d6c3f4f *man/checkScenario.Rd | |
| 95 | 7e5aa8f3c75eaedc7d8e19a902864403 *man/configurations.print.Rd | |
| 96 | 14e033558771c9106b429f53bfb72320 *man/configurations.print.command.Rd | |
| 97 | c10db4cc76effd892d2d71f48d67c292 *man/defaultScenario.Rd | |
| 98 | cd7f7baaa8734863c8f7a62ae97a045b *man/getConfigurationById.Rd | |
| 99 | 9a72690609110ba97808c4284a4121ee *man/getConfigurationByIteration.Rd | |
| 100 | 147fc1fd2f1fe41f7d19efa46d981955 *man/getFinalElites.Rd | |
| 101 | ab24f5f4834c020a9e13d5d0b27456d9 *man/irace-package.Rd | |
| 102 | 9fda11a0c9063c4d668b58fbd06970ed *man/irace.Rd | |
| 103 | fc7d521a068ac95854f7aa0cd07fa8eb *man/irace.cmdline.Rd | |
| 104 | 8fb6cd730c0c3f68f950035aab00f116 *man/irace.license.Rd | |
| 105 | bfee6562847f9202097c048aea8fc38e *man/irace.main.Rd | |
| 106 | c61fcfc9965c72d1792982c4ac05487b *man/irace.version.Rd | |
| 107 | f851af63937121b12ba3818fc89d9cc4 *man/path_rel2abs.Rd | |
| 108 | 88e40f9047a1e78663eb5bb714d5862c *man/plotAblation.Rd | |
| 109 | 1e954c71985f87b3d84d20832e64ba8e *man/printParameters.Rd | |
| 110 | 1fa672e70f48ba69c3c48a9c634c199c *man/printScenario.Rd | |
| 111 | 0866f93c1c8b41b8ddf20cda0d2b37f8 *man/psRace.Rd | |
| 112 | 3933746918dd3372aef6825c963c2f6f *man/readConfigurationsFile.Rd | |
| 113 | 6416a3ab3b2c54c915f5297beba44166 *man/readParameters.Rd | |
| 114 | fd4014da95182daa5f4e036d42716aff *man/readScenario.Rd | |
| 115 | f38a57a9b8da01e0cb6e227853a7bbd2 *man/read_logfile.Rd | |
| 116 | 0013e857af07f83b4c3657a1a88525cb *man/read_pcs_file.Rd | |
| 117 | fe958fdb0536802ef73a17e55b3c2ddb *man/removeConfigurationsMetaData.Rd | |
| 118 | 8d2c92c883e214f8ac76f4a4d63a750f *man/scenario_update_paths.Rd | |
| 119 | f4388a116d42c37a4bc6d8f64cb281bd *man/target.evaluator.default.Rd | |
| 120 | 7c150396daf48a28b3dae46ecbeca84a *man/target.runner.default.Rd | |
| 121 | 8499f76c5044d2aa52df9bcf03cacbb5 *man/testConfigurations.Rd | |
| 122 | 269490247d3ceb4ea52130c545f5ee00 *man/testing_fromfile.Rd | |
| 123 | a5e75a420330b3074709671da1e3d0e7 *man/testing_fromlog.Rd | |
| 124 | a886f8cb75354603c3204d2cebc07888 *src/Makevars | |
| 125 | 7906feceff998d2b2309454c9cbb2f10 *src/Makevars-common | |
| 126 | ca5fd252fe6221d98e26e1eab7b8d9c6 *src/Makevars.win | |
| 127 | b9f27579825988f77dd3bd9cc80449c1 *src/install.libs.R | |
| 128 | 4fb82b74a5fcc75be149ef7b2ee9b2a7 *src/iracebin/ablation.c | |
| 126 | 129 | 58ea452025942c848ac07b11f57427ea *src/iracebin/irace.c |
| 127 | 130 | f6939d43ee41ad7076812ca9404cf4e8 *src/iracebin/irace.h |
| 128 | 156cf581cb11fb3f1827f81530593ae0 *src/iracebin/irace2pyimp.c | |
| 129 | 0b87b4635dc04ba67212afa13fa019b1 *tests/testthat.R | |
| 131 | 4c3b685dcd16cc3d2091da956863d966 *tests/testthat.R | |
| 132 | b735d8cdd247647949863dc0db5242c5 *tests/testthat/bad_scenario.txt | |
| 133 | d2a04968b98d1dc030087ed5a3973a24 *tests/testthat/bug-13-lookup.rds | |
| 130 | 134 | 392472d97647beac3650ff564bd64455 *tests/testthat/bug_large_new_instances-confs.txt |
| 131 | 3e458de2e4dd97fc94a3c3831d72deca *tests/testthat/bug_large_new_instances.Rdata | |
| 135 | c37af8399644c4c02083ce46b2a72631 *tests/testthat/bug_large_new_instances.Rdata | |
| 132 | 136 | 0c70d9f2df1bdaae7a5427383deae61f *tests/testthat/configurations.txt |
| 137 | 5392765ea169c97660c1a8f5e38962d5 *tests/testthat/dependencies.txt | |
| 138 | 72b9700b9daeb38f2b3d56b07e761337 *tests/testthat/dependencies2.txt | |
| 133 | 139 | a96f6d23df9bca94fd33c47b8fe54835 *tests/testthat/dummy_wrapper.py |
| 134 | 140 | db236f7a1008af30779f40caf07060f7 *tests/testthat/forbidden.txt |
| 135 | 5500ec952b6f8dd4b6472c20eff359a0 *tests/testthat/helper-common.R | |
| 136 | edde2fb3f78e939587d0648fd74d83e5 *tests/testthat/irace2pyimp/002-TemplateDesign/features.csv | |
| 137 | 9b48de17b419b215a12be667a0ea0c72 *tests/testthat/irace2pyimp/002-TemplateDesign/instances.txt | |
| 138 | 70e839a3df66606579446e283067da69 *tests/testthat/irace2pyimp/002-TemplateDesign/params.pcs | |
| 139 | 365683596b87b0c2b409c548f0a426a0 *tests/testthat/irace2pyimp/002-TemplateDesign/runhistory.json.gz | |
| 140 | a7f86b05a53f425857cfc03186aed9c9 *tests/testthat/irace2pyimp/002-TemplateDesign/scenario.txt | |
| 141 | 9f77d33d5c3f24a30ea673dc093dfd3a *tests/testthat/irace2pyimp/002-TemplateDesign/traj_aclib2.json | |
| 142 | cece95303d20fd3bd1580fdd7740e2f6 *tests/testthat/irace2pyimp/acotsp_1/features.csv | |
| 143 | 4a996468edd9f178280824b6ef814b83 *tests/testthat/irace2pyimp/acotsp_1/instances.txt | |
| 144 | 94c9b9d0a582ae85c22367dac120d440 *tests/testthat/irace2pyimp/acotsp_1/params.pcs | |
| 145 | da20c33a03f92e4dfb2cb55b58253f0d *tests/testthat/irace2pyimp/acotsp_1/runhistory.json.gz | |
| 146 | e409ace6c8879c98144e09f6cf1ed82a *tests/testthat/irace2pyimp/acotsp_1/scenario.txt | |
| 147 | f3bbc537d647447eefdb23ca1c03a87c *tests/testthat/irace2pyimp/acotsp_1/traj_aclib2.json | |
| 148 | cece95303d20fd3bd1580fdd7740e2f6 *tests/testthat/irace2pyimp/acotsp_4/features.csv | |
| 149 | 4a996468edd9f178280824b6ef814b83 *tests/testthat/irace2pyimp/acotsp_4/instances.txt | |
| 150 | 94c9b9d0a582ae85c22367dac120d440 *tests/testthat/irace2pyimp/acotsp_4/params.pcs | |
| 151 | f8b5cb41296b3abeeb4da342f174f5a1 *tests/testthat/irace2pyimp/acotsp_4/runhistory.json.gz | |
| 152 | e409ace6c8879c98144e09f6cf1ed82a *tests/testthat/irace2pyimp/acotsp_4/scenario.txt | |
| 153 | 283ffe2d398e7e3615983559f29cba69 *tests/testthat/irace2pyimp/acotsp_4/traj_aclib2.json | |
| 141 | e53bf0109c13ffc1734dc5d66c77caac *tests/testthat/helper-common.R | |
| 154 | 142 | 5a08d4053b2e1733ac189074f84a62cc *tests/testthat/logparameters.txt |
| 155 | 143 | 1e4c15b1cb77a4738b5431e83f59cf33 *tests/testthat/parameters.txt |
| 156 | 144 | 34ecd3091ca49fabe295aaa76720e449 *tests/testthat/saved_maxim_bug.rds |
| 157 | 145 | bfcde32a37798cf3fa4a0cd3a47933c9 *tests/testthat/saved_maxim_bug2.rds |
| 158 | 4fd4da106e4a97ca4ac34a1e94f99394 *tests/testthat/test-GenericWrapper4AC.R | |
| 159 | 396ac6d04ed9c0f755adbf11e7259544 *tests/testthat/test-bugs.R | |
| 160 | e1df5cacd43a5f253eed310cbdb09b8c *tests/testthat/test-capping.R | |
| 161 | d603700d9179ad177a49d1020362637b *tests/testthat/test-forbidden.R | |
| 162 | 59f1c6fa5b4ca51785df8648cfbcbc6a *tests/testthat/test-irace2pyimp.R | |
| 163 | d9b37c6098677a850f7d148454056426 *tests/testthat/test-maxTime.R | |
| 146 | 802b1377a53b451f13bc190dd692eb9e *tests/testthat/scenario.txt | |
| 147 | fa5811cf38687ea0ba22857d07e43708 *tests/testthat/setup.R | |
| 148 | 390e7a8b3437a82b11f614fd1c1976f1 *tests/testthat/teardown.R | |
| 149 | f738821c6a6716067263dcb035a8f499 *tests/testthat/test-GenericWrapper4AC.R | |
| 150 | c76a2bb7c81016dd0769ea1463db40f9 *tests/testthat/test-bad_scenario.R | |
| 151 | 9001476440addf62d0bddf2d83e66391 *tests/testthat/test-bug-10.R | |
| 152 | e72a8ebe9175644123b6939ac21d9e79 *tests/testthat/test-bug-13.R | |
| 153 | 53a1fc7b10969c0a91f94585abe9a7ea *tests/testthat/test-bugs.R | |
| 154 | 968331f372e347ce39e0d3da72f58c70 *tests/testthat/test-capping.R | |
| 155 | 74f00ef9d7219ebcf467f6e63f1de3d1 *tests/testthat/test-dependencies.R | |
| 156 | 158ae05d47f940a435ee6770ec56ae86 *tests/testthat/test-forbidden.R | |
| 157 | a86211242111165f2e32a8a9105bdc54 *tests/testthat/test-maxTime.R | |
| 164 | 158 | 56754b543d36c2c63459a4befbd50c03 *tests/testthat/test-mlr.R |
| 165 | 7f1baf58cc9b20a1a2a4156a3b1b4395 *tests/testthat/test-path.R | |
| 166 | 5053ba8defc4c86962e3b8b84d4103ff *tests/testthat/test-sann-irace.R | |
| 159 | 1dbfbc6930ca68aac00a2ecf119b0b0a *tests/testthat/test-path.R | |
| 160 | 827ba08b388c13ad94d7c47ec643b48b *tests/testthat/test-raceconfs.R | |
| 161 | 386a21c6965ec4ee93c55c6b2d4dff5d *tests/testthat/test-readconfs.R | |
| 162 | c6dbfe891bc46013458c5858ce22e7bf *tests/testthat/test-sann-irace.R | |
| 167 | 163 | 54aba5c7c5e6cacd5d809cef03a4e4ac *tests/testthat/test-similar.R |
| 168 | 164 | 8ebd654ea8317ed80746c261c4e90b5d *tests/testthat/test-targetRunnerParallel.R |
| 169 | 165 | 30e04b3da8a10ca41b7457a0f553d059 *tests/testthat/test-targeteval.R |
| 170 | da2fd1953cd33a19dfc3f9d78a435863 *vignettes/NEWS.txt | |
| 166 | 038a593734b0b4fcccde699be7b7ce20 *vignettes/NEWS.txt | |
| 171 | 167 | 289ab2a15fa9dca53bb4a94dff1817e3 *vignettes/Warning-icon.png |
| 172 | c8940c257353249fb90b2435afe5f1af *vignettes/examples.Rdata | |
| 173 | 60f556ce3296211bb986740daba10d91 *vignettes/irace-acotsp-stdout.txt | |
| 174 | 4dd3073b98f2b4723f68ebcb4eeace03 *vignettes/irace-acotsp.Rdata | |
| 175 | c6f6dde37bededb32d96fa6889891b2a *vignettes/irace-package.Rnw | |
| 176 | 02f58d0f325d4de0c5961854092e5cb1 *vignettes/irace-package.bib | |
| 168 | 29ce28ad39dd3182608f02197ec1ed45 *vignettes/examples.Rdata | |
| 169 | 4784f9bce400ae4624ca874739cd7c4b *vignettes/fig1u-acotsp-instances.pdf | |
| 170 | df916e74a88ae7e49d09735b04c82cd8 *vignettes/irace-acotsp-stdout.txt | |
| 171 | fdf194d8f19dada2867eff7a0695139d *vignettes/irace-acotsp.Rdata | |
| 172 | c2e0a6fcffab3c32cdfc05ff2d5aa126 *vignettes/irace-package.Rnw | |
| 173 | 003c544a7fe10a9552d74ac6bdc0a1ac *vignettes/irace-package.bib | |
| 177 | 174 | f44025a8b672c55f7c68873d309336ad *vignettes/irace-scheme.pdf |
| 178 | 175 | 0b5a399b2478c44749ab63304194f6a7 *vignettes/light-bulb-icon.png |
| 179 | eb7d9a9a7d61a14e2377e09ee0b9450d *vignettes/log-ablation.Rdata | |
| 180 | 90c0dfc0fa6e1da79e2a602ea0085fc4 *vignettes/section/irace-options.tex | |
| 176 | 9d6ff066c6a773ca2b8c41999f684e0b *vignettes/section/irace-options.tex |
| 0 | 0 | # Generated by roxygen2: do not edit by hand |
| 1 | 1 | |
| 2 | export(CommandArgsParser) | |
| 2 | 3 | export(ablation) |
| 4 | export(ablation_cmdline) | |
| 3 | 5 | export(buildCommandLine) |
| 4 | 6 | export(checkIraceScenario) |
| 7 | export(checkParameters) | |
| 5 | 8 | export(checkScenario) |
| 9 | export(cmdline_usage) | |
| 6 | 10 | export(configurations.print) |
| 7 | 11 | export(configurations.print.command) |
| 8 | export(configurationsBoxplot) | |
| 9 | 12 | export(defaultScenario) |
| 10 | 13 | export(getConfigurationById) |
| 11 | 14 | export(getConfigurationByIteration) |
| 14 | 17 | export(irace.cmdline) |
| 15 | 18 | export(irace.license) |
| 16 | 19 | export(irace.main) |
| 17 | export(irace.usage) | |
| 18 | 20 | export(irace.version) |
| 19 | export(irace2pyimp) | |
| 20 | export(irace2pyimp_cmdline) | |
| 21 | export(parallelCoordinatesPlot) | |
| 22 | export(parameterFrequency) | |
| 21 | export(path_rel2abs) | |
| 23 | 22 | export(plotAblation) |
| 23 | export(printParameters) | |
| 24 | 24 | export(printScenario) |
| 25 | 25 | export(psRace) |
| 26 | 26 | export(readConfigurationsFile) |
| 27 | 27 | export(readParameters) |
| 28 | 28 | export(readScenario) |
| 29 | export(read_logfile) | |
| 29 | 30 | export(read_pcs_file) |
| 30 | 31 | export(removeConfigurationsMetaData) |
| 31 | 32 | export(scenario.update.paths) |
| 33 | export(scenario_update_paths) | |
| 32 | 34 | export(target.evaluator.default) |
| 33 | 35 | export(target.runner.default) |
| 34 | 36 | export(testConfigurations) |
| 35 | export(testing.main) | |
| 37 | export(testing_fromfile) | |
| 38 | export(testing_fromlog) | |
| 36 | 39 | import(compiler) |
| 37 | 40 | import(stats) |
| 38 | 41 | import(utils) |
| 39 | 42 | importFrom(R6,R6Class) |
| 40 | importFrom(grDevices,cairo_pdf) | |
| 41 | 43 | importFrom(grDevices,dev.new) |
| 42 | 44 | importFrom(grDevices,dev.off) |
| 43 | 45 | importFrom(grDevices,pdf) |
| 44 | importFrom(grDevices,rgb) | |
| 45 | 46 | importFrom(graphics,abline) |
| 46 | 47 | importFrom(graphics,axis) |
| 47 | 48 | importFrom(graphics,barplot) |
| 48 | 49 | importFrom(graphics,boxplot) |
| 49 | 50 | importFrom(graphics,bxp) |
| 50 | 51 | importFrom(graphics,grid) |
| 51 | importFrom(graphics,hist) | |
| 52 | 52 | importFrom(graphics,lines) |
| 53 | 53 | importFrom(graphics,matplot) |
| 54 | 54 | importFrom(graphics,mtext) |
| 0 | # irace 3.5 | |
| 1 | ||
| 2 | ## New features and improvements | |
| 3 | ||
| 4 | * Handling of dependent parameter domains: These should be specified in the | |
| 5 | parameter domain definition and, for now, only numerical parameter can | |
| 6 | define dependent domains. A numerical domain can be dependent on one bound, | |
| 7 | e.g. `(1, "param1*2")`, where the dependent bound can include basic | |
| 8 | arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 9 | ||
| 10 | * The package now provides an `ablation` executable (`ablation.exe` in | |
| 11 | Windows) that makes easier to perform ablation analysis without having any R | |
| 12 | knowledge. | |
| 13 | ||
| 14 | * The interface to functions `ablation()` and `plotAblation()` has been | |
| 15 | simplified. The `ablation()` function now allows overriding scenario | |
| 16 | settings. The `plotAblation()` function will not create the plot if the | |
| 17 | ablation log does not contain a complete ablation. | |
| 18 | (Manuel López-Ibáñez) | |
| 19 | ||
| 20 | * The argument `n.instances` of `ablation()` has been renamed to `n_instances` and it is now a factor that multiplies `scenario$firstTest`. | |
| 21 | (Manuel López-Ibáñez) | |
| 22 | ||
| 23 | * New command-line option `--quiet` to run without producing any output | |
| 24 | except errors (also available as a scenario option). | |
| 25 | (Manuel López-Ibáñez) | |
| 26 | ||
| 27 | * New command-line option `--init` to initialize a scenario. (Deyao Chen) | |
| 28 | ||
| 29 | * Added support for HTCondor cluster framework to `--batchmode`. | |
| 30 | (Filippo Bistaffa) | |
| 31 | ||
| 32 | * `--check` now also check the contents of `configurationsFile` and runs | |
| 33 | configurations provided via `initConfigurations`. | |
| 34 | (Manuel López-Ibáñez, reported by Andreea Avramescu) | |
| 35 | ||
| 36 | * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs` | |
| 37 | to help in cases where the target-runner must be invoked via another | |
| 38 | software with particular options (such as `python.exe` in Windows). | |
| 39 | (Manuel López-Ibáñez) | |
| 40 | ||
| 41 | * New scenario option `minMeasurableTime`. | |
| 42 | (Manuel López-Ibáñez) | |
| 43 | ||
| 44 | * An error is produced if a variable set in the scenario file is not known to | |
| 45 | irace. If your scenario file contains R code, then use variable names | |
| 46 | beginning with a dot `'.'`, which will be ignored by irace. | |
| 47 | (Manuel López-Ibáñez) | |
| 48 | ||
| 49 | * Plotting functions have been moved to the new package | |
| 50 | [iraceplot](https://auto-optimization.github.io/iraceplot/). In particular, | |
| 51 | `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and | |
| 52 | `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by | |
| 53 | `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and | |
| 54 | `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`. | |
| 55 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 56 | ||
| 57 | * The user-guide now contains a detailed section on "Hyper-parameter | |
| 58 | optimization of machine learning methods". | |
| 59 | (Manuel López-Ibáñez) | |
| 60 | ||
| 61 | * When `testType="F-test"` and only two configurations remain, the elimination | |
| 62 | test now uses the pseudo-median estimated by the Wilcoxon signed-rank test | |
| 63 | to decide which configuration is the best one instead of comparing the | |
| 64 | median difference. | |
| 65 | (Manuel López-Ibáñez) | |
| 66 | ||
| 67 | * New functions `testing_fromlog()` and `testing_fromfile()` for independently | |
| 68 | executing the testing phase. The function `testing.main()` was removed as it | |
| 69 | is superseded by the new ones. | |
| 70 | (Manuel López-Ibáñez) | |
| 71 | ||
| 72 | * New function `read_logfile()` to easily read the log file produced by irace. | |
| 73 | (Manuel López-Ibáñez) | |
| 74 | ||
| 75 | * New function `printParameters()` that prints a parameters R object as a valid input text. | |
| 76 | (Manuel López-Ibáñez) | |
| 77 | ||
| 78 | * `irace2pyimp` moved to its own R package. | |
| 79 | (Manuel López-Ibáñez) | |
| 80 | ||
| 81 | * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`. | |
| 82 | (Manuel López-Ibáñez, reported by Johann Dreo) | |
| 83 | ||
| 84 | * `path_rel2abs()` and `checkParameters()` are now exported so that other | |
| 85 | packages may use them. | |
| 86 | (Manuel López-Ibáñez) | |
| 87 | ||
| 88 | * `path_rel2abs()` also searches in system paths. (Manuel López-Ibáñez) | |
| 89 | ||
| 90 | * `readConfigurationsFile()` will now detect duplicated configurations and | |
| 91 | error. (Manuel López-Ibáñez) | |
| 92 | ||
| 93 | * The interface to functions `getFinalElites()`, `getConfigurationById()` and | |
| 94 | `getConfigurationByIteration()` has been simplified. | |
| 95 | ||
| 96 | * The package provides a `irace.sindef` file that may be used for building a | |
| 97 | standalone container of irace using Singularity. See the `README.md` file | |
| 98 | for instructions. (Contributed by Johann Dreo) | |
| 99 | ||
| 100 | * New example `examples/target-runner-python/target-runner-python-win.bat` | |
| 101 | contributed by Levi Ribeiro. | |
| 102 | ||
| 103 | * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters. | |
| 104 | (Manuel López-Ibáñez) | |
| 105 | ||
| 106 | * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez) | |
| 107 | ||
| 108 | ## Fixes | |
| 109 | ||
| 110 | * Correctly handle clear out-performance cases despite strong bi-modality. | |
| 111 | (Reported by Nguyen Dang, | |
| 112 | fixed by Manuel López-Ibáñez) | |
| 113 | ||
| 114 | * Fix error when recovering from a parallel run on Windows. | |
| 115 | (Manuel López-Ibáñez, reported by Tarek Gamal) | |
| 116 | ||
| 117 | * `testNbElites` now controls how many iteration elites are tested when | |
| 118 | `testIterationElites=1`. This is the documented behavior in the user guide. | |
| 119 | (Manuel López-Ibáñez, reported by Marcelo de Souza) | |
| 120 | ||
| 121 | * Fixes to the Matlab example. (Manuel López-Ibáñez) | |
| 122 | ||
| 123 | * The default of `testType` is now set to `t-test` when capping is enabled. | |
| 124 | (Manuel López-Ibáñez, reported by Jovana Radjenovic) | |
| 125 | ||
| 126 | * Fix various issues in the user guide. | |
| 127 | (Manuel López-Ibáñez, reported by Jovana Radjenovic) | |
| 128 | ||
| 129 | * Remove duplicated elites. | |
| 130 | (Manuel López-Ibáñez, reported by Federico Naldini) | |
| 131 | ||
| 132 | * Fix (#7): warnings with partial matched parameters. | |
| 133 | (Manuel López-Ibáñez, reported by Marc Becker) | |
| 134 | ||
| 135 | * Fix (#10): wrong assert with `elitist=0`. (Manuel López-Ibáñez) | |
| 136 | ||
| 137 | * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r). | |
| 138 | ||
| 139 | * Fix (#13): Maximum number configurations immediately rejected reached. | |
| 140 | (Manuel López-Ibáñez) | |
| 141 | ||
| 142 | * Fix: when setting the scenario file in the command-line, `scenarioFile` was | |
| 143 | not set correctly. The correct scenario was used, however, the debug output | |
| 144 | and the value stored in the log / recovery file was wrong. | |
| 145 | (Manuel López-Ibáñez, reported by Richard Schoonhoven) | |
| 146 | ||
| 147 | * With `sampleInstances = FALSE`, elitist irace does not change the order of | |
| 148 | instances already seen. However, if you want to make sure that the order of | |
| 149 | the instances is enforced, you also need to set `elitistNewInstances=0`. | |
| 150 | ||
| 151 | * The function `irace.usage()` was removed. It was not really useful for R | |
| 152 | users as the same result can be obtained by calling | |
| 153 | `irace.cmdline("--help")`. | |
| 154 | (Manuel López-Ibáñez) | |
| 155 | ||
| 156 | ||
| 0 | 157 | # irace 3.4.1 (31/03/2020) |
| 1 | 158 | |
| 2 | 159 | * `NEWS` converted to markdown. |
| 20 | 177 | versions. |
| 21 | 178 | (Manuel López-Ibáñez) |
| 22 | 179 | * Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres) |
| 23 | ||
| 180 | ||
| 24 | 181 | ``` |
| 25 | 182 | value >= 1L && value <= length(possibleValues) is not TRUE |
| 26 | 183 | ``` |
| 29 | 186 | of a script. On Windows, `irace.exe` replaces `irace.bat` |
| 30 | 187 | (Manuel López-Ibáñez) |
| 31 | 188 | |
| 32 | * inst/examples/Spear contains the Spear (SAT solver) configuration scenario. | |
| 189 | * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario. | |
| 33 | 190 | (Manuel López-Ibáñez) |
| 34 | 191 | |
| 35 | * Fixed bug when reporting minimum maxTime required. | |
| 192 | * Fixed bug when reporting minimum `maxTime` required. | |
| 36 | 193 | (Reported by Luciana Salete Buriol, |
| 37 | 194 | fixed by Manuel López-Ibáñez) |
| 38 | 195 | |
| 41 | 198 | ```R |
| 42 | 199 | all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE |
| 43 | 200 | ``` |
| 44 | ||
| 201 | ||
| 45 | 202 | (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez) |
| 46 | 203 | |
| 47 | 204 | |
| 60 | 217 | satisfy `digits` (up to `digits=15`). |
| 61 | 218 | (Manuel López-Ibáñez) |
| 62 | 219 | |
| 63 | * It is possible to specify boundMax without capping. | |
| 220 | * It is possible to specify `boundMax` without capping. | |
| 64 | 221 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) |
| 65 | 222 | |
| 66 | 223 | * `irace --check` will exit with code 1 if the check is unsuccessful |
| 68 | 225 | |
| 69 | 226 | * Print where irace is installed with `--help`. (Manuel López-Ibáñez) |
| 70 | 227 | |
| 71 | * irace will now complain if the output of target-runner or target-evaluator | |
| 228 | * irace will now complain if the output of `target-runner` or `target-evaluator` | |
| 72 | 229 | contains extra lines even if the first line of output is correct. This is to |
| 73 | 230 | avoid parsing the wrong output. Unfortunately, this may break setups that |
| 74 | 231 | relied on this behavior. The solution is to only print the output that irace |
| 81 | 238 | |
| 82 | 239 | * New option `aclib=` (`--aclib 1`) enables compatibility with the |
| 83 | 240 | GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by |
| 84 | AClib (http://aclib.net/). This is EXPERIMENTAL. | |
| 85 | `--aclib 1` also sets digits to 15 for compatibility with AClib defaults. | |
| 241 | AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets | |
| 242 | digits to 15 for compatibility with AClib defaults. | |
| 86 | 243 | (Manuel López-Ibáñez) |
| 87 | 244 | |
| 88 | 245 | * Fix printing of output when capping is enabled. |
| 107 | 264 | * Fix bug in `checkTargetFiles()` (`--check`) with capping. |
| 108 | 265 | (Leslie Pérez Cáceres) |
| 109 | 266 | |
| 110 | * Clarify a few errors/warnings when `maxTime` > 0. | |
| 267 | * Clarify a few errors/warnings when `maxTime > 0`. | |
| 111 | 268 | (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos) |
| 112 | 269 | |
| 113 | 270 | |
| 190 | 347 | evaluation noticeably faster. |
| 191 | 348 | (Manuel López-Ibáñez) |
| 192 | 349 | |
| 193 | * The argument 'experiment' passed to the R function targetRunner does not | |
| 194 | contain anymore an element 'extra.params'. Similarly, the 'scenario' | |
| 195 | structure does not contain anymore the elements 'instances.extra.params' and | |
| 196 | 'testInstances.extra.params'. Any instance-specific parameters values now | |
| 350 | * The argument `'experiment'` passed to the R function `targetRunner` does not | |
| 351 | contain anymore an element `'extra.params'`. Similarly, the `'scenario'` | |
| 352 | structure does not contain anymore the elements `'instances.extra.params'` and | |
| 353 | `'testInstances.extra.params'`. Any instance-specific parameters values now | |
| 197 | 354 | form part of the character string that defines an instance and it is up to |
| 198 | the user-defined targetRunner to parse them appropriately. These changes | |
| 355 | the user-defined `targetRunner` to parse them appropriately. These changes | |
| 199 | 356 | make no difference when targetRunner is an external script, or when |
| 200 | 357 | instances and instance-specific parameter values are read from a file. |
| 201 | 358 | (Manuel López-Ibáñez) |
| 202 | 359 | |
| 203 | 360 | # irace 2.3 |
| 204 | 361 | |
| 205 | * Fix bug that will cause iraceResults$experimentLog to count calls to | |
| 206 | targetEvaluator as experiments, even if no call to targetRunner was | |
| 362 | * Fix bug that will cause `iraceResults$experimentLog` to count calls to | |
| 363 | `targetEvaluator` as experiments, even if no call to `targetRunner` was | |
| 207 | 364 | performed. This does not affect the computation of the budget consumed and, |
| 208 | 365 | thus, it does not affect the termination criteria of irace. The bug triggers |
| 209 | 366 | an assertion that terminates irace, thus no run that was successful with |
| 221 | 378 | |
| 222 | 379 | * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by |
| 223 | 380 | `--batchmode` (`batchmode`). It is now the responsibility of the target-runner |
| 224 | to parse the output of the batch job submission command (e.g., qsub or | |
| 225 | squeue), and return just the job ID. Values supported are: "sge", "torque", | |
| 381 | to parse the output of the batch job submission command (e.g., `qsub` or | |
| 382 | `squeue`), and return just the job ID. Values supported are: "sge", "torque", | |
| 226 | 383 | "pbs" and "slurm". (Manuel López-Ibáñez) |
| 227 | 384 | |
| 228 | 385 | * The option `--parallel` can now be combined with `--batchmode` to limit the |
| 239 | 396 | ```R |
| 240 | 397 | eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE)) |
| 241 | 398 | ``` |
| 242 | ||
| 399 | ||
| 243 | 400 | This feature is VERY experimental and the syntax is likely to change in the |
| 244 | 401 | future. (Manuel López-Ibáñez) |
| 245 | 402 | |
| 256 | 413 | (Manuel López-Ibáñez, Leslie Pérez Cáceres) |
| 257 | 414 | |
| 258 | 415 | * Update manual and vignette with details about the expected arguments and |
| 259 | return value of targetRunner and targetEvaluator. (Manuel López-Ibáñez) | |
| 416 | return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez) | |
| 260 | 417 | |
| 261 | 418 | * Many updates to the User Guide vignette. (Manuel López-Ibáñez) |
| 262 | 419 | |
| 263 | * Fix \dontrun example in irace-package.Rd (Manuel López-Ibáñez) | |
| 420 | * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez) | |
| 264 | 421 | |
| 265 | 422 | * Fix bug: If testInstances contains duplicates, results of testing are not |
| 266 | correctly saved in iraceResults$testing$experiments nor reported correctly | |
| 267 | at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for | |
| 423 | correctly saved in `iraceResults$testing$experiments` nor reported correctly | |
| 424 | at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for | |
| 268 | 425 | each testing instance. These IDs are used for the rownames of |
| 269 | iraceResults$testing$experiments and the names of the scenario$testInstances | |
| 270 | and iraceResults$testing$seeds vectors. (Manuel López-Ibáñez) | |
| 271 | ||
| 272 | * Fix bug where irace keeps retrying the target-runner call even if it | |
| 426 | `iraceResults$testing$experiments` and the names of the | |
| 427 | `scenario$testInstances` | |
| 428 | and `iraceResults$testing$seeds` vectors. (Manuel López-Ibáñez) | |
| 429 | ||
| 430 | * Fix bug where irace keeps retrying the `target-runner` call even if it | |
| 273 | 431 | succeeds. (Manuel López-Ibáñez) |
| 274 | 432 | |
| 275 | 433 | * New command-line parameter |
| 280 | 438 | instances defined by the scenario. Useful if you decide on the testing |
| 281 | 439 | instances only after running irace. (Manuel López-Ibáñez) |
| 282 | 440 | |
| 283 | * Bugfix: When using maxTime != 0, the number of experiments performed may be | |
| 441 | * Bugfix: When using `maxTime != 0`, the number of experiments performed may be | |
| 284 | 442 | miscounted in some cases. (Manuel López-Ibáñez) |
| 285 | 443 | |
| 286 | 444 | |
| 304 | 462 | comparisons), |
| 305 | 463 | - `t-test-holm` (t-test with Holm's correction for multiple comparisons) |
| 306 | 464 | |
| 307 | * MPI does not create log files with --debug-level 0. | |
| 465 | * MPI does not create log files with `--debug-level 0`. | |
| 308 | 466 | (Manuel López-Ibáñez) |
| 309 | 467 | |
| 310 | * For simplicity, the parallel-irace-* scripts do not use an auxiliary | |
| 468 | * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary | |
| 311 | 469 | `tune-main` script. For customizing them, make a copy and edit them |
| 312 | 470 | directly. |
| 313 | 471 | (Manuel López-Ibáñez) |
| 316 | 474 | ``` |
| 317 | 475 | --target-runner-retries : Retry target-runner this many times in case of error. |
| 318 | 476 | ``` |
| 319 | ||
| 477 | ||
| 320 | 478 | |
| 321 | 479 | * We print diversity measures after evaluating on each instance: |
| 322 | 480 | (Leslie Pérez Cáceres) |
| 379 | 537 | |
| 380 | 538 | * The best configurations found, either at the end or at each iteration of an |
| 381 | 539 | irace run, can now be applied to a set of test instances different from the |
| 382 | training instances. See options testInstanceDir, testInstanceFile, | |
| 383 | testNbElites, and testIterationElites. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 384 | ||
| 385 | * The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed. | |
| 540 | training instances. See options `testInstanceDir`, `testInstanceFile`, | |
| 541 | `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 542 | ||
| 543 | * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed. | |
| 386 | 544 | See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of |
| 387 | 545 | the new interfaces. |
| 388 | 546 | |
| 390 | 548 | IDs, and numbers are printed in a more human-readable format. |
| 391 | 549 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) |
| 392 | 550 | |
| 393 | * Reduce memory use for very large values of maxExperiments. | |
| 551 | * Reduce memory use for very large values of `maxExperiments`. | |
| 394 | 552 | (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) |
| 395 | 553 | |
| 396 | 554 | * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing |
| 417 | 575 | |
| 418 | 576 | * New configuration options, mainly for R users: |
| 419 | 577 | |
| 420 | - hookRunParallel: Optional R function to provide custom | |
| 421 | parallelization of hook.run. | |
| 422 | ||
| 423 | - hookRunData: Optional data passed to hookRun. This is ignored by | |
| 424 | the default hookRun function, but it may be used by custom hookRun R | |
| 425 | functions to pass persistent data around. | |
| 426 | (Manuel López-Ibáñez) | |
| 578 | - `hookRunParallel`: Optional R function to provide custom | |
| 579 | parallelization of `hook.run`. | |
| 580 | ||
| 581 | - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the | |
| 582 | default `hookRun` function, but it may be used by custom `hookRun` R | |
| 583 | functions to pass persistent data around. (Manuel López-Ibáñez) | |
| 427 | 584 | |
| 428 | 585 | # irace 1.05 |
| 429 | 586 | |
| 448 | 605 | See `--forbidden-file` and `inst/templates/forbidden.tmpl`. |
| 449 | 606 | (Manuel López-Ibáñez) |
| 450 | 607 | |
| 451 | * New option `--recovery-file` (recoveryFile) allows resuming a | |
| 608 | * New option `--recovery-file` (`recoveryFile`) allows resuming a | |
| 452 | 609 | previous irace run. (Leslie Pérez Cáceres) |
| 453 | 610 | |
| 454 | 611 | * The confidence level for the elimination test is now |
| 469 | 626 | * Print elapsed time for calls to hook-run if `debugLevel >=1`. |
| 470 | 627 | (Manuel López-Ibáñez) |
| 471 | 628 | |
| 472 | * `examples/hook-run-python/hook-run`: A multi-purpose hook-run written | |
| 629 | * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written | |
| 473 | 630 | in Python. (Franco Mascia) |
| 474 | 631 | |
| 475 | 632 | * Parallel mode in an SGE cluster (`--sge-cluster`) is more |
| 499 | 656 | |
| 500 | 657 | * More concise output. |
| 501 | 658 | |
| 502 | * The parameters expName and expDescription are now useless and they | |
| 659 | * The parameters `expName` and `expDescription` are now useless and they | |
| 503 | 660 | were removed. |
| 504 | 661 | |
| 505 | 662 | * Faster computation of similar candidates (Jeremie Dubois-Lacoste |
| 506 | 663 | and Leslie Pérez Cáceres). |
| 507 | 664 | |
| 508 | * Fix bug when saving instances in tunerResults$experiments. | |
| 665 | * Fix bug when saving instances in `tunerResults$experiments`. | |
| 509 | 666 | |
| 510 | 667 | * `irace.cmdline ("--help")` does not try to quit R anymore. |
| 511 | 668 | |
| 0 | .ablation.params.def <- read.table(header=TRUE, stringsAsFactors = FALSE, text=" | |
| 1 | name type short long default description | |
| 2 | iraceResults p -l --log-file NA 'Path to the (.Rdata) file created by irace from which the \"iraceResults\" object will be loaded.' | |
| 3 | src i -S --src 1 'Source configuration ID.' | |
| 4 | target i -T --target NA 'Target configuration ID. By default the best configuration found by irace.' | |
| 5 | ab.params s -P --params '' 'Specific parameter names to be used for the ablation (separated with commas). By default use all' | |
| 6 | type s -t --type 'full' 'Type of ablation to perform: \"full\" will execute each configuration on all \"--n-instances\" to determine the best-performing one; \"racing\" will apply racing to find the best configurations.' | |
| 7 | n_instances i -n --n-instances 1 'Number of instances used in \"full\" ablation will be n_instances * scenario$firstTest.' | |
| 8 | seed i '' --seed 1234567 'Integer value to use as seed for the random number generation.' | |
| 9 | ablationLogFile p -o --output-file 'log-ablation.Rdata' 'Log file to save the ablation log. If \"\", the results are not saved to a file.' | |
| 10 | plot s -p --plot '' 'Output filename (.pdf) for the plot. If not given, no plot is created.' | |
| 11 | plot_type s -O --plot-type 'mean' 'Type of plot. Supported values are \"mean\" and \"boxplot\".' | |
| 12 | old_path p '' --old-path NA 'Old path found in the log-file (.Rdata) given as input to be replaced by --new-path.' | |
| 13 | new_path p '' --new-path NA 'New path to replace the path found in the log-file (.Rdata) given as input.' | |
| 14 | execDir p -e --exec-dir NA 'Directory where the target runner will be run.' | |
| 15 | scenarioFile p -s --scenario NA 'Scenario file to override the scenario given in the log-file (.Rdata)' | |
| 16 | parallel i '' --parallel NA 'Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.' | |
| 17 | ") | |
| 18 | ||
| 19 | cat_ablation_license <- function() | |
| 20 | { | |
| 21 | ablation_license <- | |
| 22 | '#------------------------------------------------------------------------------ | |
| 23 | # ablation: An implementation in R of Ablation Analysis | |
| 24 | # Version: __VERSION__ | |
| 25 | # Copyright (C) 2020--2022 | |
| 26 | # Manuel Lopez-Ibanez <manuel.lopez-ibanez@manchester.ac.uk> | |
| 27 | # Leslie Perez Caceres <leslie.perez.caceres@ulb.ac.be> | |
| 28 | # | |
| 29 | # This is free software, and you are welcome to redistribute it under certain | |
| 30 | # conditions. See the GNU General Public License for details. There is NO | |
| 31 | # WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | |
| 32 | #------------------------------------------------------------------------------ | |
| 33 | ' | |
| 34 | cat(sub("__VERSION__", irace.version, ablation_license, fixed=TRUE)) | |
| 35 | } | |
| 36 | ||
| 37 | #' Launch ablation with command-line options. | |
| 38 | #' | |
| 39 | #' Launch [ablation()] with the same command-line options as the command-line | |
| 40 | #' executable (`ablation.exe` in Windows). | |
| 41 | #' | |
| 42 | #' @param argv (`character()`) \cr The arguments | |
| 43 | #' provided on the R command line as a character vector, e.g., | |
| 44 | #' `c("-i", "irace.Rdata", "--src", 1)`. | |
| 45 | #' | |
| 46 | #' @details The function reads the parameters given on the command line | |
| 47 | #' used to invoke R, launches [ablation()] and possibly [plotAblation()]. | |
| 48 | #' | |
| 49 | #' List of command-line options: | |
| 50 | #' ```{r echo=FALSE,comment=NA} | |
| 51 | #' cmdline_usage(.ablation.params.def) | |
| 52 | #' ``` | |
| 53 | #' @template ret_ablog | |
| 54 | #' @examples | |
| 55 | #' # ablation_cmdline("--help") | |
| 56 | #' | |
| 57 | #' @author Manuel López-Ibáñez | |
| 58 | #' @concept running | |
| 59 | #' @export | |
| 60 | ablation_cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) | |
| 61 | { | |
| 62 | op <- options(width = 9999L) # Do not wrap the output. | |
| 63 | on.exit(options(op), add = TRUE) | |
| 64 | ||
| 65 | cat_ablation_license() | |
| 66 | cat ("# installed at: ", system.file(package="irace"), "\n", | |
| 67 | "# called with: ", paste(argv, collapse = " "), "\n", sep = "") | |
| 68 | parser <- CommandArgsParser$new(argv = argv, argsdef = .ablation.params.def) | |
| 69 | if (!is.null(parser$readArg (short = "-h", long = "--help"))) { | |
| 70 | parser$cmdline_usage() | |
| 71 | return(invisible(NULL)) | |
| 72 | } | |
| 73 | ||
| 74 | if (!is.null(parser$readArg (short = "-v", long = "--version"))) { | |
| 75 | print(citation(package="irace")) | |
| 76 | return(invisible(NULL)) | |
| 77 | } | |
| 78 | ||
| 79 | params <- parser$readAll() | |
| 80 | # TODO: Send the other options to the irace command-line parser so the user | |
| 81 | # can override options in scenario via the command-line. | |
| 82 | if (length(parser$argv) > 0) | |
| 83 | stop("Unknown command-line options: ", paste(parser$argv, collapse = " ")) | |
| 84 | ||
| 85 | if (is.null(params$iraceResults)) { | |
| 86 | irace.error("You must provide the path to the '.Rdata' file that contains the 'iraceResults' object generated by irace.") | |
| 87 | return(invisible(NULL)) | |
| 88 | } | |
| 89 | iraceResults <- read_logfile(params$iraceResults) | |
| 90 | if (is.null(params$old_path) != is.null(params$new_path)) { | |
| 91 | irace.error("To update paths you must provide both --old-path and --new-path.") | |
| 92 | return(invisible(NULL)) | |
| 93 | } else if (!is.null(params$old_path)) { | |
| 94 | iraceResults$scenario <- scenario_update_paths(iraceResults$scenario, params$old_path, params$new_path) | |
| 95 | } | |
| 96 | if (!is.null(params$scenarioFile)) { | |
| 97 | scenario <- readScenario(params$scenarioFile) | |
| 98 | } | |
| 99 | if (is_null_or_empty_or_na(trim(params$ablationLogFile))) { | |
| 100 | params$ablationLogFile <- NULL | |
| 101 | } | |
| 102 | for (p in c("execDir", "parallel")) { | |
| 103 | if (!is.null(params[[p]])) scenario[[p]] <- params[[p]] | |
| 104 | } | |
| 105 | ||
| 106 | if (!is.null(params$ablationLogFile)) | |
| 107 | params$ablationLogFile <- path_rel2abs(params$ablationLogFile) | |
| 108 | ||
| 109 | if (!is.null(params$ab.params)) | |
| 110 | params$ab.params <- sapply(strsplit(params$ab.params, ",")[[1]], trimws, USE.NAMES=FALSE) | |
| 111 | ||
| 112 | ablog <- do.call(ablation, | |
| 113 | args = c(list(iraceResults = iraceResults, | |
| 114 | src = params$src, target = params$target, | |
| 115 | ab.params = params$ab.params, type = params$type, | |
| 116 | n_instances = params$n_instances, seed = params$seed, | |
| 117 | ablationLogFile = params$ablationLogFile), | |
| 118 | scenario)) | |
| 119 | if (!is.null(params[["plot"]]) || base::interactive()) { | |
| 120 | plotAblation(ablog, pdf.file = params[["plot"]], type = params[["plot_type"]]) | |
| 121 | } | |
| 122 | invisible(ablog) | |
| 123 | } | |
| 124 | ||
| 0 | 125 | ## This function fixes dependent parameters when a parameter value has been |
| 1 | 126 | ## changed. |
| 2 | 127 | fixDependenciesWithReference <- function(configuration, ref.configuration, parameters) |
| 50 | 175 | configurations <- rbind.data.frame(configurations, new.configuration) |
| 51 | 176 | } |
| 52 | 177 | rownames(configurations) <- NULL |
| 53 | return (list(configurations=configurations, changed.params=changed.params)) | |
| 54 | } | |
| 55 | ||
| 56 | #' Performs ablation between two configurations. | |
| 178 | list(configurations=configurations, changed.params=changed.params) | |
| 179 | } | |
| 180 | ||
| 181 | report_duplicated_results <- function(experiments, configurations) | |
| 182 | { | |
| 183 | x <- t(experiments) | |
| 184 | x <- x[duplicated(x) | duplicated(x, fromLast = TRUE), , drop=FALSE] | |
| 185 | if (nrow(x) == 0L) return(NULL) # No duplicates | |
| 186 | dups <- split(rownames(x), apply(x, 1, paste0, collapse="")) | |
| 187 | names(dups) <- NULL | |
| 188 | for (g in dups) { | |
| 189 | cat("Warning: The following configuration produced the same results (the different parameters had not effect):\n") | |
| 190 | print(configurations[configurations$.ID. %in% g, , drop=FALSE]) | |
| 191 | } | |
| 192 | dups | |
| 193 | } | |
| 194 | ||
| 195 | #' Performs ablation between two configurations (from source to target). | |
| 57 | 196 | #' |
| 58 | 197 | #' @description Ablation is a method for analyzing the differences between two configurations. |
| 59 | #' | |
| 60 | #' @param iraceLogFile Log file created by \pkg{irace}, this file must contain | |
| 61 | #' the \code{iraceResults} object. | |
| 62 | #' @param iraceResults Object created by \pkg{irace} and saved in | |
| 63 | #' \code{scenario$logFile}. | |
| 64 | #' @param src,target Source and target configuration IDs. If \code{NULL}, then | |
| 65 | #' the first configuration ever evaluated is used as source and the best | |
| 66 | #' configuration found is used as target. | |
| 67 | #' @param ab.params Parameter names to be used for the ablation. They must be | |
| 68 | #' in parameters$names. | |
| 69 | #' @param n.instances Number of instances to be used for the "full" ablation, | |
| 70 | #' if not provided firstTest instances are used. | |
| 71 | #' @param type Type of ablation to perform, "full" will execute all instances | |
| 72 | #' in the configurations to determine the best performing, "racing" will | |
| 73 | #' apply racing to find the best configurations. | |
| 74 | #' @param seed Numerical value to use as seed for the random number generation. | |
| 75 | #' @param ablationLogFile Log file to save the ablation log. | |
| 76 | #' @param pdf.file Prefix that will be used to save the plot file of the | |
| 77 | #' ablation results. | |
| 78 | #' @param pdf.width Width provided to create the pdf file. | |
| 79 | #' @param mar Vector with the margins for the ablation plot. | |
| 80 | #' @template arg_debuglevel | |
| 198 | #' | |
| 199 | #' @template arg_iraceresults | |
| 200 | #' @param src,target Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as `src` and the best configuration found by irace is used as target. | |
| 201 | #' @param ab.params Specific parameter names to be used for the ablation. They must be in `parameters$names`. By default, use all parameters. | |
| 202 | #' @param type Type of ablation to perform: `"full"` will execute each configuration on all `n_instances` to determine the best-performing one; `"racing"` will apply racing to find the best configurations. | |
| 203 | #' @param n_instances (`integer(1)`) Number of instances used in `"full"` ablation will be `n_instances * scenario$firstTest`. | |
| 204 | #' @param seed (`integer(1)`) Integer value to use as seed for the random number generation. | |
| 205 | #' @param ablationLogFile (`character(1)`) Log file to save the ablation log. If `NULL`, the results are not saved to a file. | |
| 206 | #' @param ... Further arguments to override scenario settings, e.g., `debugLevel`, `parallel`, etc. | |
| 81 | 207 | #' |
| 82 | 208 | #' @references |
| 83 | 209 | #' C. Fawcett and H. H. Hoos. Analysing differences between algorithm |
| 84 | 210 | #' configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016. |
| 85 | 211 | #' |
| 86 | #' @return A list containing the following elements: | |
| 87 | #' \describe{ | |
| 88 | #' \item{configurations}{Configurations tested in the ablation.} | |
| 89 | #' \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 90 | #' instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} | |
| 91 | #' \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 92 | #' \item{scenario}{Scenario object with the settings used for the experiments.} | |
| 93 | #' \item{trajectory}{IDs of the best configurations at each step of the ablation.} | |
| 94 | #' \item{best}{Best configuration found in the experiments.} | |
| 95 | #' } | |
| 96 | #' | |
| 212 | #' @template ret_ablog | |
| 213 | #' @seealso [plotAblation()] | |
| 97 | 214 | #' @examples |
| 98 | 215 | #' \donttest{ |
| 99 | #' irace.logfile <- file.path(system.file(package="irace"), "exdata", "sann.rda") | |
| 100 | #' load(irace.logfile) | |
| 216 | #' logfile <- system.file(package="irace", "exdata", "sann.rda") | |
| 101 | 217 | #' # Execute ablation between the first and the best configuration found by irace. |
| 102 | #' ablation(iraceResults = iraceResults, ablationLogFile = NULL) | |
| 218 | #' ablog <- ablation(logfile, ablationLogFile = NULL) | |
| 219 | #' plotAblation(ablog) | |
| 103 | 220 | #' # Execute ablation between two selected configurations, and selecting only a |
| 104 | 221 | #' # subset of parameters, directly reading the setup from the irace log file. |
| 105 | #' ablation(iraceLogFile = irace.logfile, src = 1, target = 10, | |
| 106 | #' ab.params = c("temp"), ablationLogFile = NULL) | |
| 222 | #' ablog <- ablation(logfile, src = 1, target = 10, | |
| 223 | #' ab.params = c("temp"), ablationLogFile = NULL) | |
| 224 | #' plotAblation(ablog) | |
| 107 | 225 | #' } |
| 108 | 226 | #' |
| 109 | 227 | #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez |
| 110 | 228 | #' @export |
| 111 | #FIXME: Add check for the inputs! | |
| 112 | ablation <- function(iraceLogFile = NULL, iraceResults = NULL, | |
| 113 | src = NULL, target = NULL, | |
| 114 | ab.params = NULL, n.instances = NULL, | |
| 115 | type = "full", seed = 1234567, | |
| 116 | ablationLogFile = "log-ablation.Rdata", | |
| 117 | pdf.file = NULL, pdf.width = 20, mar = c(12,5,4,1), | |
| 118 | debugLevel = NULL) | |
| 119 | { | |
| 229 | ablation <- function(iraceResults, src = 1L, target = NULL, | |
| 230 | ab.params = NULL, type = c("full", "racing"), | |
| 231 | n_instances = 1L, seed = 1234567, | |
| 232 | ablationLogFile = "log-ablation.Rdata", ...) | |
| 233 | { | |
| 234 | # Input check | |
| 235 | if (missing(iraceResults) || is.null(iraceResults)) | |
| 236 | stop("You must provide an 'iraceResults' object generated by irace or the path to the '.Rdata' file that contains this object.") | |
| 237 | ||
| 238 | type <- match.arg(type) | |
| 239 | if (n_instances != 1L && type == "racing") | |
| 240 | stop("'n_instances' has no effect when type == 'racing'") | |
| 241 | ||
| 242 | if (!is.null(ablationLogFile)) | |
| 243 | file.check(ablationLogFile, writeable = TRUE, text = 'logFile') | |
| 244 | ||
| 245 | save_ablog <- function(complete) { | |
| 246 | ablog <- list(changes = changes, | |
| 247 | configurations = all_configurations, | |
| 248 | experiments = results, | |
| 249 | instances = instances, | |
| 250 | scenario = scenario, | |
| 251 | trajectory = trajectory, | |
| 252 | best = best.configuration, | |
| 253 | complete = complete) | |
| 254 | if (!is.null(ablationLogFile)) save(ablog, file = ablationLogFile, version = 2) | |
| 255 | ablog | |
| 256 | } | |
| 257 | ||
| 120 | 258 | # FIXME: The previous seed needs to be saved and restored at the end. |
| 121 | 259 | set.seed(seed) |
| 122 | ||
| 123 | # Input check | |
| 124 | if (is.null(iraceLogFile) && is.null(iraceResults)) | |
| 125 | irace.error("You must provide a Rdata file or an iraceResults object.") | |
| 126 | if (!(type %in% c("full", "racing"))) | |
| 127 | irace.error("Type of ablation", type, "not recognized.") | |
| 128 | ||
| 129 | 260 | # Load the data of the log file |
| 130 | if (!is.null(iraceLogFile)) load(iraceLogFile) | |
| 131 | ||
| 132 | if (is.null(src)) src <- 1 | |
| 261 | iraceResults <- read_logfile(iraceResults) | |
| 133 | 262 | if (is.null(target)) target <- iraceResults$iterationElites[length(iraceResults$iterationElites)] |
| 134 | 263 | |
| 135 | irace.note ("Starting ablation from ", src, " to ", target, " :\n# Seed:", seed, "\n") | |
| 264 | irace.note ("Starting ablation from ", src, " to ", target, "\n# Seed: ", seed, "\n") | |
| 136 | 265 | |
| 137 | 266 | if (src %!in% iraceResults$allConfigurations$.ID.) |
| 138 | irace.error("Source configuration ID (", src, ") cannot be found") | |
| 267 | stop("Source configuration ID (", src, ") cannot be found") | |
| 139 | 268 | |
| 140 | 269 | if (target %!in% iraceResults$allConfigurations$.ID.) |
| 141 | irace.error("Target configuration ID (", target, ") cannot be found") | |
| 270 | stop("Target configuration ID (", target, ") cannot be found") | |
| 142 | 271 | |
| 143 | 272 | src.configuration <- iraceResults$allConfigurations[src, , drop = FALSE] |
| 144 | 273 | target.configuration <- iraceResults$allConfigurations[target, , drop = FALSE] |
| 145 | 274 | |
| 146 | 275 | parameters <- iraceResults$parameters |
| 147 | 276 | scenario <- iraceResults$scenario |
| 148 | ||
| 149 | if (!is.null(ablationLogFile)) scenario$logFile <- ablationLogFile | |
| 150 | if (!is.null(debugLevel)) scenario$debugLevel <- debugLevel | |
| 151 | ||
| 277 | scenario_args <- list(...) | |
| 278 | if (length(scenario_args) > 0L) { | |
| 279 | unknown_scenario_args <- setdiff(names(scenario_args), names(scenario)) | |
| 280 | if (length(unknown_scenario_args) > 0L) | |
| 281 | irace.error("Unknown scenario settings given: ", paste0(unknown_scenario_args, collapse=", ")) | |
| 282 | scenario <- modifyList(scenario, scenario_args) | |
| 283 | } | |
| 284 | scenario$logFile <- "" | |
| 152 | 285 | scenario <- checkScenario (scenario) |
| 153 | ||
| 154 | 286 | startParallel(scenario) |
| 155 | 287 | on.exit(stopParallel(), add = TRUE) |
| 156 | 288 | |
| 157 | # LESLIE: we should decide how to select the instances to perform the ablation | |
| 158 | # for now we generate an instace list with one instance. | |
| 159 | ## MANUEL: I think this should be obtained from parameters like | |
| 160 | ## instanceFile/instanceDir/instance and re-use the code from irace. And by | |
| 161 | ## default, it should use something like firstTest random training instances. | |
| 162 | ## LESLIE: Ok by default we use firstTest and I added a parameter to change it in case needed. | |
| 163 | if (type == "racing") | |
| 164 | n.instances <- length(scenario$instances) | |
| 165 | else if (is.null(n.instances)) | |
| 166 | n.instances <- scenario$firstTest | |
| 167 | instances <- generateInstances(scenario, n.instances) | |
| 289 | n_instances <- if (type == "racing") length(scenario$instances) else n_instances * scenario$firstTest | |
| 290 | instances <- generateInstances(scenario, n_instances) | |
| 168 | 291 | .irace$instancesList <- instances |
| 169 | 292 | |
| 170 | 293 | # Select the parameters used for ablation |
| 171 | 294 | if (is.null(ab.params)) { |
| 172 | ab.params <- parameters$names | |
| 295 | ab.params <- parameters$names | |
| 173 | 296 | } else if (!all(ab.params %in% parameters$names)) { |
| 174 | irace.error("Some of the parameters provided are not defined in the parameters object.") | |
| 175 | } | |
| 297 | irace.error("Some of the parameters provided (", paste0(setdiff(ab.params, parameters$names), collapse=", "), | |
| 298 | ") are not defined in the parameter space.") | |
| 299 | } | |
| 300 | ||
| 301 | cat("# Source configuration (row number is ID):\n") | |
| 302 | configurations.print(src.configuration) | |
| 303 | cat("# Target configuration (row number is ID):\n") | |
| 304 | configurations.print(target.configuration) | |
| 305 | ||
| 176 | 306 | # Select parameters that are different in both configurations |
| 177 | 307 | neq.params <- which(src.configuration[,ab.params] != target.configuration[,ab.params]) |
| 178 | 308 | |
| 180 | 310 | irace.error("Candidates are equal considering the parameters selected\n") |
| 181 | 311 | param.names <- colnames(src.configuration[,ab.params])[neq.params] |
| 182 | 312 | |
| 183 | cat("# Configurations (row number is ID):\n") | |
| 184 | configurations.print(rbind(src.configuration, target.configuration)) | |
| 185 | ||
| 186 | 313 | # FIXME: Do we really need to override the ID? |
| 187 | 314 | src.configuration$.ID. <- best.id <- 1 |
| 188 | best.configuration <- all.configurations <- src.configuration | |
| 315 | best.configuration <- all_configurations <- src.configuration | |
| 189 | 316 | |
| 190 | 317 | # Execute source and target configurations. |
| 191 | 318 | ## FIXME: We may already have these experiments in the logFile! |
| 196 | 323 | seeds = instances[, "seed"], |
| 197 | 324 | scenario = scenario, |
| 198 | 325 | bounds = scenario$boundMax) |
| 199 | irace.note("Executing source and target configurations on the given instances...\n") | |
| 326 | # Define variables needed | |
| 327 | trajectory <- 1 | |
| 328 | names(trajectory) <- "source" | |
| 329 | # FIXME: changes should only store the changed parameters. | |
| 330 | changes <- list() | |
| 331 | irace.note("Executing source and target configurations on the given instances (", nrow(instances), ")...\n") | |
| 200 | 332 | target.output <- execute.experiments(experiments, scenario) |
| 201 | 333 | if (!is.null(scenario$targetEvaluator)) |
| 202 | 334 | target.output <- execute.evaluator (experiments, scenario, target.output, |
| 206 | 338 | results <- matrix(NA, ncol = 1, nrow = nrow(instances), |
| 207 | 339 | dimnames = list(seq(1,nrow(instances)), 1)) |
| 208 | 340 | results[,1] <- output[1:nrow(instances)] |
| 209 | lastres <- output[(nrow(instances)+1):(2*nrow(instances))] | |
| 210 | ||
| 211 | # Define variables needed | |
| 212 | trajectory <- 1 | |
| 213 | names(trajectory) <- "source" | |
| 214 | changes <- list() | |
| 215 | ||
| 216 | # Start ablation | |
| 341 | lastres <- output[(nrow(instances)+1):(2 * nrow(instances))] | |
| 217 | 342 | step <- 1 |
| 343 | ablog <- save_ablog(complete = FALSE) | |
| 218 | 344 | while (length(param.names) > 1) { |
| 219 | 345 | # Generate ablation configurations |
| 220 | 346 | cat("# Generating configurations (row number is ID):", param.names,"\n") |
| 229 | 355 | # New configurations ids |
| 230 | 356 | ## FIXME: These should be generated with respect to the logFile to make |
| 231 | 357 | ## sure we don't have duplicate IDs. |
| 232 | new.ids <- seq(max(all.configurations$.ID.) + 1, | |
| 233 | max(all.configurations$.ID.) + nrow(aconfigurations)) | |
| 234 | aconfigurations[,".ID."] <- new.ids | |
| 358 | aconfigurations[,".ID."] <- seq(max(all_configurations$.ID.) + 1, | |
| 359 | max(all_configurations$.ID.) + nrow(aconfigurations)) | |
| 235 | 360 | configurations.print(aconfigurations, metadata = FALSE) |
| 236 | all.configurations <- rbind(all.configurations, aconfigurations) | |
| 361 | all_configurations <- rbind(all_configurations, aconfigurations) | |
| 237 | 362 | |
| 238 | 363 | # Set variables for the racing procedure |
| 239 | 364 | if (scenario$capping) { |
| 249 | 374 | scenario$elitist <- FALSE |
| 250 | 375 | .irace$next.instance <- 1 |
| 251 | 376 | } |
| 252 | ||
| 253 | # All instances for the firstTest for the full | |
| 254 | if (type == "full") scenario$firstTest <- nrow(instances) | |
| 255 | ||
| 377 | ||
| 256 | 378 | irace.note("Ablation (", type, ") of ", nrow(aconfigurations), |
| 257 | 379 | " configurations on ", nrow(instances), " instances.\n") |
| 258 | # MANUEL: Is this racing or just running the configurations? | |
| 380 | # Force the race to see all instances in "full" mode | |
| 381 | if (type == "full") scenario$firstTest <- nrow(instances) | |
| 259 | 382 | race.output <- race(maxExp = nrow(aconfigurations) * nrow(instances), |
| 260 | 383 | minSurvival = 1, |
| 261 | 384 | elite.data = elite.data, |
| 265 | 388 | elitistNewInstances = 0) |
| 266 | 389 | results <- merge.matrix (results, race.output$experiments) |
| 267 | 390 | |
| 268 | # Save temp log | |
| 269 | ab.log <- list(configurations = all.configurations, | |
| 270 | instances = instances, | |
| 271 | experiments = results, | |
| 272 | scenario = scenario, | |
| 273 | trajectory = trajectory, | |
| 274 | changes = changes) | |
| 275 | if (!is.null(ablationLogFile)) | |
| 276 | save(ab.log, file = ablationLogFile, version = 2) | |
| 391 | # Save log | |
| 392 | ablog <- save_ablog(complete = FALSE) | |
| 277 | 393 | |
| 278 | 394 | # Get the best configuration based on the criterion of irace |
| 279 | 395 | # MANUEL: Doesn't race.output already give you all this info??? |
| 280 | 396 | cranks <- overall.ranks(results[,aconfigurations$.ID.,drop=FALSE], scenario$testType) |
| 281 | best.ids <- which.min(cranks) | |
| 282 | cand.mean <- colMeans(results[,aconfigurations$.ID.,drop=FALSE], na.rm=TRUE) | |
| 397 | best_id <- which.min(cranks)[1] | |
| 398 | # cand.mean <- colMeans(results[,aconfigurations$.ID.,drop=FALSE], na.rm=TRUE) | |
| 283 | 399 | changes[[step]] <- ab.aux$changed.params |
| 284 | best.change <- changes[[step]][[best.ids[1]]] | |
| 285 | trajectory <- c(trajectory, aconfigurations[best.ids[1], ".ID."]) | |
| 400 | best.change <- changes[[step]][[best_id]] | |
| 401 | trajectory <- c(trajectory, aconfigurations[best_id, ".ID."]) | |
| 286 | 402 | |
| 287 | 403 | # Report best |
| 288 | 404 | # FIXME: This ID does not actually match the configuration ID |
| 289 | 405 | # The race already reports the best. |
| 290 | #cat("# Best configuration ID:", best.ids[1], "mean:", cand.mean[best.ids[1]],"\n") | |
| 291 | for(i in seq_along(best.change)) { | |
| 292 | cat("# ", best.change[i], best.configuration[,best.change[i]], "->", | |
| 293 | aconfigurations[best.ids[1], best.change[i]], "\n") | |
| 406 | cat("# Best changed parameters:\n") | |
| 407 | for (i in seq_along(best.change)) { | |
| 408 | cat("#", best.change[i], ":", best.configuration[,best.change[i]], "->", | |
| 409 | aconfigurations[best_id, best.change[i]], "\n") | |
| 294 | 410 | } |
| 295 | 411 | |
| 296 | best.configuration <- aconfigurations[best.ids[1],,drop=FALSE] | |
| 412 | best.configuration <- aconfigurations[best_id,,drop=FALSE] | |
| 297 | 413 | best.id <- best.configuration$.ID. |
| 298 | 414 | param.names <- param.names[!(param.names %in% best.change)] |
| 299 | 415 | step <- step + 1 |
| 301 | 417 | |
| 302 | 418 | # Add last configuration and its results |
| 303 | 419 | # FIXME: This may be overriding the ID of an existing configuration!!! |
| 304 | target.configuration$.ID. <- max(all.configurations$.ID.) + 1 | |
| 305 | all.configurations <- rbind(all.configurations, target.configuration) | |
| 420 | target.configuration$.ID. <- max(all_configurations$.ID.) + 1 | |
| 421 | all_configurations <- rbind(all_configurations, target.configuration) | |
| 306 | 422 | results <- cbind(results, matrix(lastres, ncol = 1, |
| 307 | 423 | dimnames=list(seq(1, nrow(instances)), |
| 308 | 424 | target.configuration$.ID.))) |
| 310 | 426 | |
| 311 | 427 | # Get the overall best |
| 312 | 428 | cranks <- overall.ranks(results[,trajectory, drop=FALSE], scenario$testType) |
| 313 | best.ids <- which.min(cranks) | |
| 314 | best.configuration <- all.configurations[trajectory[best.ids[1]],,drop=FALSE] | |
| 429 | best_id <- which.min(cranks)[1] | |
| 430 | ## FIXME: At this point, the rownames of all_configurations does not match | |
| 431 | ## all_configurations$.ID. That is confusing and a potential source of | |
| 432 | ## bugs. Instead of fixing it here, we should not generate the discrepancy | |
| 433 | ## ever. | |
| 434 | best.configuration <- all_configurations[trajectory[best_id],,drop=FALSE] | |
| 315 | 435 | irace.note("Final best configuration:\n") |
| 316 | 436 | configurations.print(best.configuration) |
| 317 | ||
| 437 | ||
| 438 | # Check for duplicated results: | |
| 439 | report_duplicated_results(results, all_configurations) | |
| 440 | ||
| 318 | 441 | # LESLIE: If we use racing we can have a matrix of results that is not |
| 319 | 442 | # complete, how should we do the plots? |
| 320 | 443 | # MANUEL: Do not plot anything that was discarded |
| 321 | 444 | |
| 322 | # Save final log. | |
| 323 | ab.log <- list(configurations = all.configurations, | |
| 324 | instances = instances, | |
| 325 | experiments = results, | |
| 326 | scenario = scenario, | |
| 327 | trajectory = trajectory, | |
| 328 | best = best.configuration) | |
| 329 | ||
| 330 | if (!is.null(ablationLogFile)) | |
| 331 | save(ab.log, file = ablationLogFile, version = 2) | |
| 332 | ||
| 333 | plotAblation(ab.log = ab.log, pdf.file = pdf.file, | |
| 334 | pdf.width = pdf.width, mar = mar, main = "Ablation") | |
| 335 | return(ab.log) | |
| 445 | save_ablog(complete = TRUE) | |
| 336 | 446 | } |
| 337 | 447 | |
| 338 | 448 | ablation.labels <- function(trajectory, configurations) |
| 354 | 464 | |
| 355 | 465 | #' Create plot from an ablation log |
| 356 | 466 | #' |
| 357 | #' @param ab.log Ablation log returned by \code{\link{ablation}}. | |
| 358 | #' @param abLogFile Rdata file containing the ablation log. | |
| 467 | #' @param ablog (`list()`|`character(1)`) Ablation log object returned by [ablation()]. Alternatively, the path to an `.Rdata` file, e.g., `"log-ablation.Rdata"`, from which the object will be loaded. | |
| 468 | ||
| 359 | 469 | #' @param pdf.file Output filename. |
| 360 | 470 | #' @param pdf.width Width provided to create the pdf file. |
| 361 | #' @param type Type of plots. Supported values are \code{"mean"} and | |
| 362 | #' \code{"boxplot"}. | |
| 471 | #' @param type Type of plot. Supported values are `"mean"` and `"boxplot"`. | |
| 363 | 472 | #' @param mar Vector with the margins for the ablation plot. |
| 364 | 473 | #' @param ylab Label of y-axis. |
| 474 | #' @param ylim Numeric vector of length 2, giving the y coordinates ranges. | |
| 365 | 475 | #' @param ... Further graphical parameters may also be supplied as |
| 366 | #' arguments. See \code{plot.default}. | |
| 476 | #' arguments. See [graphics::plot.default()]. | |
| 367 | 477 | #' |
| 368 | 478 | #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez |
| 369 | #' @seealso \code{\link{ablation}} | |
| 479 | #' @seealso [ablation()] | |
| 480 | #' @examples | |
| 481 | #' logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 482 | #' plotAblation(ablog = logfile) | |
| 370 | 483 | #' @export |
| 371 | plotAblation <- function (ab.log = NULL, abLogFile = NULL, | |
| 372 | pdf.file = NULL, pdf.width = 20, | |
| 484 | plotAblation <- function (ablog, pdf.file = NULL, pdf.width = 20, | |
| 373 | 485 | type = c("mean", "boxplot"), |
| 374 | 486 | mar = par("mar"), |
| 375 | ylab = "Mean configuration cost", ...) | |
| 487 | ylab = "Mean configuration cost", ylim = NULL, | |
| 488 | ...) | |
| 376 | 489 | { |
| 377 | 490 | type <- match.arg(type) |
| 378 | if (is.null(ab.log)) { | |
| 379 | if (is.null(abLogFile)) | |
| 380 | irace.error("You must provide a log file or an ablation log object") | |
| 381 | else { | |
| 382 | load(abLogFile) | |
| 383 | if (is.null(ab.log)) | |
| 384 | irace.error("abLogFile '", abLogFile, "' does not contain ab.log") | |
| 385 | } | |
| 386 | } | |
| 387 | ||
| 491 | if (missing(ablog) || is.null(ablog)) { | |
| 492 | irace.error("You must provide an 'ablog' object generated by ablation() or the path to the '.Rdata' file that contains this object.") | |
| 493 | } | |
| 494 | ||
| 495 | ablog <- read_logfile(ablog, name = "ablog") | |
| 496 | if (!ablog$complete) | |
| 497 | stop("The ablog shows that the ablation procedure did not complete cleanly and only contains partial information") | |
| 498 | ||
| 499 | ||
| 388 | 500 | if (!is.null(pdf.file)) { |
| 389 | 501 | if (!is.file.extension(pdf.file, ".pdf")) |
| 390 | 502 | pdf.file <- paste0(pdf.file, ".pdf") |
| 394 | 506 | on.exit(dev.off(), add = TRUE) |
| 395 | 507 | } |
| 396 | 508 | |
| 397 | trajectory <- ab.log$trajectory | |
| 398 | configurations <- ab.log$configurations | |
| 509 | trajectory <- ablog$trajectory | |
| 510 | configurations <- ablog$configurations | |
| 399 | 511 | # Generate labels |
| 400 | 512 | # FIXME: allow overriding these labels. |
| 401 | 513 | labels <- ablation.labels(trajectory, configurations) |
| 402 | 514 | |
| 403 | 515 | inches_to_lines <- (par("mar") / par("mai"))[1] |
| 404 | 516 | lab.width <- max(strwidth(labels, units = "inches")) * inches_to_lines |
| 405 | old.par <- par(mar = mar + c(lab.width - 2, 0, 0, 0), cex.axis = 1) | |
| 406 | on.exit(par(old.par), add = TRUE) | |
| 407 | ||
| 408 | experiments <- ab.log$experiments | |
| 409 | ||
| 410 | # FIXME: We should also show the other alternatives at each step not just the | |
| 517 | old.par <- par(mar = c(lab.width + 2.1, 4.1, 0.1, 0.1), cex.axis = 1) | |
| 518 | if (!is.null(pdf.file)) | |
| 519 | on.exit(par(old.par), add = TRUE) | |
| 520 | ||
| 521 | experiments <- ablog$experiments | |
| 522 | ||
| 523 | # FIXME: We could also show the other alternatives at each step not just the | |
| 411 | 524 | # one selected. See Leonardo's thesis. |
| 412 | ylim <- NULL | |
| 413 | 525 | if (type == "boxplot") { |
| 414 | 526 | bx <- boxplot(experiments[, trajectory], plot=FALSE) |
| 415 | ylim <- range(ylim, bx$stats[is.finite(bx$stats)], | |
| 416 | bx$out[is.finite(bx$out)], | |
| 417 | bx$conf[is.finite(bx$conf)]) | |
| 527 | if (is.null(ylim)) { | |
| 528 | ylim <- range(bx$stats[is.finite(bx$stats)], | |
| 529 | bx$out[is.finite(bx$out)], | |
| 530 | bx$conf[is.finite(bx$conf)]) | |
| 531 | } | |
| 418 | 532 | } |
| 419 | 533 | costs.avg <- colMeans(experiments[, trajectory]) |
| 420 | 534 | |
| 0 | #' R6 Class for parsing command-line arguments | |
| 1 | #' | |
| 2 | #' @export | |
| 0 | 3 | CommandArgsParser <- R6::R6Class("CommandArgsParser", cloneable = FALSE, list( |
| 1 | 4 | argv = NULL, |
| 2 | 5 | argsdef = NULL, |
| 7 | 10 | argv <- strsplit(trim(argv), " +")[[1]] |
| 8 | 11 | } |
| 9 | 12 | self$argv <- argv |
| 13 | required_colnames <- c("name", "short", "long", "type", "default") | |
| 14 | if (any(required_colnames %!in% colnames(argsdef))) { | |
| 15 | stop("argsdef must contain the column names: ", paste0(required_colnames, collapse=", ")) | |
| 16 | } | |
| 10 | 17 | self$argsdef <- argsdef |
| 18 | rownames(self$argsdef) <- argsdef$name | |
| 19 | self | |
| 11 | 20 | }, |
| 12 | 21 | readCmdLineParameter = function (paramName, default = NULL) { |
| 13 | value <- self$readArg (short = self$argsdef[paramName, "short"], | |
| 14 | long = self$argsdef[paramName,"long"]) | |
| 22 | value <- self$readArg(short = self$argsdef[paramName, "short"], | |
| 23 | long = self$argsdef[paramName,"long"]) | |
| 15 | 24 | if (is.null(value)) { |
| 16 | 25 | value <- if (is.null(default)) self$argsdef[paramName, "default"] else default |
| 17 | } else if (is.na(value) && self$argsdef[paramName,"type"] != 'x' ) { | |
| 18 | irace.error ("option '", self$argsdef[paramName,"long"], | |
| 19 | "' requires an argument\n") | |
| 26 | } else if (is.na(value) && self$argsdef[paramName,"type"] != 'x') { | |
| 27 | stop("option '", self$argsdef[paramName,"long"],"' requires an argument", call. = FALSE) | |
| 20 | 28 | } |
| 21 | 29 | return(value) |
| 22 | 30 | }, |
| 26 | 34 | readArg = function(short = "", long = "") { |
| 27 | 35 | argv <- self$argv |
| 28 | 36 | pos <- c() |
| 29 | if (length (short) > 0) { | |
| 30 | pos <- grep (paste0("^", short, "$"), argv) | |
| 31 | if (length (pos) == 0) { | |
| 32 | pos <- grep (paste0("^", short, "="), argv) | |
| 37 | if (length(short) > 0) { | |
| 38 | # FIXME: use match() | |
| 39 | pos <- grep(paste0("^", short, "$"), argv) | |
| 40 | if (length(pos) == 0) { | |
| 41 | # FIXME: use pmatch() | |
| 42 | pos <- grep(paste0("^", short, "="), argv) | |
| 33 | 43 | } |
| 34 | 44 | } |
| 35 | if (length (long) > 0 && length (pos) == 0) { | |
| 36 | pos <- grep (paste0("^", long, "$"), argv) | |
| 37 | if (length (pos) == 0) { | |
| 38 | pos <- grep (paste0("^", long, "="), argv) | |
| 45 | if (length(long) > 0 && length(pos) == 0) { | |
| 46 | pos <- grep(paste0("^", long, "$"), argv) | |
| 47 | if (length(pos) == 0) { | |
| 48 | pos <- grep(paste0("^", long, "="), argv) | |
| 39 | 49 | } |
| 40 | 50 | } |
| 41 | if (length (pos) == 0) { | |
| 42 | return (NULL) | |
| 43 | } else if (length(pos) > 0) { | |
| 51 | if (length(pos) == 0) { | |
| 52 | return(NULL) | |
| 53 | } else if(length(pos) > 0) { | |
| 44 | 54 | # Allow repeated parameters |
| 45 | 55 | pos <- max(pos) |
| 46 | 56 | } |
| 47 | 57 | |
| 48 | value <- unlist(strsplit (argv[pos], '=', fixed = TRUE))[2] | |
| 58 | value <- unlist(strsplit(argv[pos], '=', fixed = TRUE))[2] | |
| 49 | 59 | if (is.null (value) || is.na(value)) { |
| 50 | 60 | value <- argv[pos + 1] |
| 51 | 61 | self$argv <- argv[-(pos + 1)] |
| 52 | 62 | } |
| 53 | 63 | self$argv <- self$argv[-pos] |
| 54 | 64 | return (value) |
| 65 | }, | |
| 66 | readAll = function() { | |
| 67 | params <- list() | |
| 68 | for (param in self$argsdef$name[self$argsdef$type != 'x']) { | |
| 69 | value <- self$readCmdLineParameter(paramName = param) | |
| 70 | if (is.na(value) || (length(value) > 0 && value == "")) value <- NULL | |
| 71 | params[[param]] <- value | |
| 72 | } | |
| 73 | params | |
| 74 | }, | |
| 75 | cmdline_usage = function(){ | |
| 76 | irace::cmdline_usage(self$argsdef) | |
| 55 | 77 | }) |
| 56 | ) | |
| 78 | ) | |
| 79 | ||
| 80 | #' `cmdline_usage()` prints the output of `--help` | |
| 81 | #' | |
| 82 | #' @param cmdline_args Definition of the command-line arguments. | |
| 83 | #' | |
| 84 | #' @rdname CommandArgsParser | |
| 85 | #' @export | |
| 86 | cmdline_usage <- function(cmdline_args) | |
| 87 | { | |
| 88 | for (i in seq_len(nrow(cmdline_args))) { | |
| 89 | short <- cmdline_args[i,"short"] | |
| 90 | long <- cmdline_args[i,"long"] | |
| 91 | desc <- cmdline_args[i,"description"] | |
| 92 | if (desc == "" || (short == "" && long == "")) next | |
| 93 | if (short != "") short <- paste0(short,",") | |
| 94 | default <- cmdline_args[i,"default"] | |
| 95 | if (!is_null_or_empty_or_na(default)) { | |
| 96 | desc <- paste0(desc, " Default: ", default, ".") | |
| 97 | } | |
| 98 | cat(sep = "\n", strwrap(desc, width = 80, | |
| 99 | initial = sprintf("%3s%-20s ", short, long), | |
| 100 | exdent = 25)) | |
| 101 | } | |
| 102 | } | |
| 0 | 0 | ### Submit/wait for jobs in batch clusters. |
| 1 | 1 | sge.job.finished <- function(jobid) |
| 2 | { | |
| 3 | return(system (paste0("qstat -j ", jobid), | |
| 4 | ignore.stdout = TRUE, ignore.stderr = TRUE, | |
| 5 | intern = FALSE, wait = TRUE)) | |
| 6 | } | |
| 2 | system (paste0("qstat -j ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, | |
| 3 | intern = FALSE, wait = TRUE) | |
| 7 | 4 | |
| 8 | 5 | pbs.job.finished <- function(jobid) |
| 9 | { | |
| 10 | return(system (paste0("qstat ", jobid), | |
| 11 | ignore.stdout = TRUE, ignore.stderr = TRUE, | |
| 12 | intern = FALSE, wait = TRUE)) | |
| 13 | } | |
| 6 | system (paste0("qstat ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, | |
| 7 | intern = FALSE, wait = TRUE) | |
| 14 | 8 | |
| 15 | 9 | torque.job.finished <- function(jobid) |
| 16 | 10 | { |
| 31 | 25 | # output. If the 5th token in the last line is a 'C', then the job has |
| 32 | 26 | # terminated and its output files can be processed. Otherwise the job is not |
| 33 | 27 | # completed (queued, running, exiting...) |
| 34 | return(any(grepl(paste0(jobid, ".*\\sC\\s"), output))) | |
| 28 | any(grepl(paste0(jobid, ".*\\sC\\s"), output)) | |
| 35 | 29 | } |
| 36 | 30 | |
| 37 | 31 | slurm.job.finished <- function(jobid) |
| 43 | 37 | if (!is.null(attr(output, "status"))) return(TRUE) |
| 44 | 38 | # If may return zero, but the job is not in the system anymore because it |
| 45 | 39 | # completed. This is different from the Torque case. |
| 46 | return (!any(grepl(paste0("\\s", jobid, "\\s"), output))) | |
| 40 | !any(grepl(paste0("\\s", jobid, "\\s"), output)) | |
| 41 | } | |
| 42 | ||
| 43 | htcondor.job.finished <- function(jobid) | |
| 44 | { | |
| 45 | output <- suppressWarnings(system2("condor_q", jobid, stdout = TRUE, stderr = TRUE)) | |
| 46 | # Check if job is still in the queue, otherwise it is considered finished | |
| 47 | !any(grepl(paste0("ID:\\s", jobid), output)) | |
| 47 | 48 | } |
| 48 | 49 | |
| 49 | 50 | ## Launch a job with qsub and return its jobID. This function does not |
| 51 | 52 | ## invokes qsub and returns a jobID. |
| 52 | 53 | target.runner.qsub <- function(experiment, scenario) |
| 53 | 54 | { |
| 54 | debugLevel <- scenario$debugLevel | |
| 55 | configuration.id <- experiment$id.configuration | |
| 56 | instance.id <- experiment$id.instance | |
| 57 | seed <- experiment$seed | |
| 58 | configuration <- experiment$configuration | |
| 59 | instance <- experiment$instance | |
| 60 | switches <- experiment$switches | |
| 61 | ||
| 62 | targetRunner <- scenario$targetRunner | |
| 63 | if (as.logical(file.access(targetRunner, mode = 1))) { | |
| 64 | irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") | |
| 65 | } | |
| 66 | ||
| 67 | args <- paste(configuration.id, instance.id, seed, instance, | |
| 68 | buildCommandLine(configuration, switches)) | |
| 69 | output <- runcommand(targetRunner, args, configuration.id, debugLevel) | |
| 55 | debugLevel <- scenario$debugLevel | |
| 56 | res <- run_target_runner(experiment, scenario) | |
| 57 | cmd <- res$cmd | |
| 58 | output <- res$output | |
| 59 | args <- res$args | |
| 70 | 60 | |
| 71 | 61 | jobID <- NULL |
| 72 | 62 | outputRaw <- output$output |
| 73 | 63 | err.msg <- output$error |
| 74 | 64 | if (is.null(err.msg)) { |
| 75 | 65 | # We cannot use parse.output because that tries to convert to numeric. |
| 76 | if (scenario$debugLevel >= 2) { cat (outputRaw, sep = "\n") } | |
| 66 | if (debugLevel >= 2) { cat (outputRaw, sep = "\n") } | |
| 77 | 67 | # Initialize output as raw. If it is empty stays like this. |
| 78 | 68 | # strsplit crashes if outputRaw == character(0) |
| 79 | 69 | if (length(outputRaw) > 0) { |
| 84 | 74 | jobID <- NULL |
| 85 | 75 | } |
| 86 | 76 | } |
| 87 | return(list(jobID = jobID, error = err.msg, outputRaw = outputRaw, | |
| 88 | call = paste(targetRunner, args))) | |
| 77 | list(jobID = jobID, error = err.msg, outputRaw = outputRaw, | |
| 78 | call = paste(cmd, args)) | |
| 89 | 79 | } |
| 90 | 80 | |
| 91 | 81 | cluster.lapply <- function(X, scenario, poll.time = 2) |
| 98 | 88 | pbs = pbs.job.finished, |
| 99 | 89 | torque = torque.job.finished, |
| 100 | 90 | slurm = slurm.job.finished, |
| 91 | htcondor = htcondor.job.finished, | |
| 101 | 92 | irace.error ("Invalid value of scenario$batchmode = ", scenario$batchmode)) |
| 102 | 93 | |
| 103 | 94 | # Parallel controls how many jobs we send at once. Some clusters have low |
| 130 | 121 | } |
| 131 | 122 | } |
| 132 | 123 | } |
| 133 | return(output) | |
| 124 | output | |
| 134 | 125 | } |
| 17 | 17 | return(v) |
| 18 | 18 | } |
| 19 | 19 | |
| 20 | new.empty.configuration <- function(parameters) | |
| 20 | new_empty_configuration <- function(parameters) | |
| 21 | 21 | { |
| 22 | 22 | newConfigurationsColnames <- c(names(parameters$conditions), ".PARENT.") |
| 23 | 23 | return(setNames(as.list(rep(NA, length(newConfigurationsColnames))), |
| 38 | 38 | } |
| 39 | 39 | } |
| 40 | 40 | |
| 41 | ## Calculates the parameter bounds when parameters domain is dependent | |
| 42 | getDependentBound <- function(parameters, param, configuration) | |
| 43 | { | |
| 44 | values <- parameters$domain[[param]] | |
| 45 | if (is.expression(values)) { | |
| 46 | # Depends contains parameters that enable param and parameters that define | |
| 47 | # its domain. If this is a partial configuration, we need only the latter. | |
| 48 | # Use names() here in case the configuration is simply a list. | |
| 49 | deps <- intersect(names(configuration), parameters$depends[[param]]) | |
| 50 | # If it depends on a parameter that is disabled, then this is disabled. | |
| 51 | if (anyNA(configuration[deps])) return(NA) | |
| 52 | ||
| 53 | values <- sapply(values, eval, configuration) | |
| 54 | irace.assert(all(is.finite(values))) | |
| 55 | # Value gets truncated (defined from robotics initial requirements) | |
| 56 | if (parameters$types[param] == "i") values <- as.integer(values) | |
| 57 | if (values[1] > values[2]) { | |
| 58 | irace.error ("Invalid domain (", paste0(values, collapse=", "), | |
| 59 | ") generated for parameter '", param, | |
| 60 | "' that depends on parameters (", | |
| 61 | paste0(parameters$depends[[param]], collapse=", "), | |
| 62 | "). This is NOT a bug in irace. Check the definition of these parameters.") | |
| 63 | } | |
| 64 | } | |
| 65 | ||
| 66 | return(values) | |
| 67 | } | |
| 68 | ||
| 41 | 69 | ### Uniform sampling for the initial generation |
| 42 | 70 | sampleUniform <- function (parameters, nbConfigurations, digits, |
| 43 | 71 | forbidden = NULL, repair = NULL) |
| 44 | 72 | { |
| 73 | if (is.null(repair)) repair <- function(c, p, d) c | |
| 74 | ||
| 45 | 75 | namesParameters <- names(parameters$conditions) |
| 46 | 76 | newConfigurations <- |
| 47 | 77 | as.data.frame(matrix(nrow = nbConfigurations, |
| 48 | 78 | ncol = length(namesParameters) + 1, |
| 49 | 79 | dimnames = list(NULL, c(namesParameters, ".PARENT.")) |
| 50 | )) | |
| 51 | empty.configuration <- new.empty.configuration(parameters) | |
| 80 | ), stringsAsFactors=FALSE) | |
| 81 | empty_configuration <- new_empty_configuration(parameters) | |
| 52 | 82 | |
| 53 | 83 | for (idxConfiguration in seq_len(nbConfigurations)) { |
| 54 | 84 | forbidden.retries <- 0 |
| 55 | 85 | while (forbidden.retries < 100) { |
| 56 | configuration <- empty.configuration | |
| 86 | configuration <- empty_configuration | |
| 57 | 87 | for (p in seq_along(namesParameters)) { |
| 58 | 88 | currentParameter <- namesParameters[p] |
| 59 | 89 | if (!conditionsSatisfied(parameters, configuration, currentParameter)) { |
| 68 | 98 | # We don't even need to sample, there is only one possible value ! |
| 69 | 99 | newVal <- get.fixed.value (currentParameter, parameters) |
| 70 | 100 | # The parameter is not a fixed and should be sampled |
| 71 | } else if (currentType %in% c("i", "r")) { | |
| 72 | newVal <- sample.unif(currentParameter, parameters, currentType, digits) | |
| 101 | } else if (currentType %in% c("i","r")) { | |
| 102 | domain <- getDependentBound(parameters, currentParameter, configuration) | |
| 103 | newVal <- sample_unif(currentType, domain, | |
| 104 | transf = parameters$transform[[currentParameter]], | |
| 105 | digits) | |
| 73 | 106 | } else { |
| 74 | 107 | irace.assert(currentType %in% c("c","o")) |
| 75 | 108 | possibleValues <- parameters$domain[[currentParameter]] |
| 78 | 111 | configuration[[p]] <- newVal |
| 79 | 112 | } |
| 80 | 113 | configuration <- as.data.frame(configuration, stringsAsFactors=FALSE) |
| 81 | if (!is.null(repair)) { | |
| 82 | configuration <- repair(configuration, parameters, digits) | |
| 83 | } | |
| 114 | configuration <- repair(configuration, parameters, digits) | |
| 84 | 115 | |
| 85 | 116 | if (is.null(forbidden) |
| 86 | 117 | || nrow(checkForbidden(configuration, forbidden)) == 1) { |
| 103 | 134 | nbNewConfigurations, digits, forbidden = NULL, |
| 104 | 135 | repair = NULL) |
| 105 | 136 | { |
| 137 | if (is.null(repair)) repair <- function(c, p, d) c | |
| 138 | ||
| 106 | 139 | if (nbNewConfigurations <= 0) { |
| 107 | 140 | irace.error ("The number of configurations to generate appears to be negative or zero.") |
| 108 | 141 | } |
| 111 | 144 | as.data.frame(matrix(nrow = nbNewConfigurations, |
| 112 | 145 | ncol = length(namesParameters) + 1, |
| 113 | 146 | dimnames = list(NULL, c(namesParameters, ".PARENT.")) |
| 114 | )) | |
| 115 | empty.configuration <- new.empty.configuration(parameters) | |
| 147 | ), stringsAsFactors=FALSE) | |
| 148 | empty_configuration <- new_empty_configuration(parameters) | |
| 116 | 149 | |
| 117 | 150 | for (idxConfiguration in seq_len(nbNewConfigurations)) { |
| 118 | 151 | forbidden.retries <- 0 |
| 122 | 155 | prob = eliteConfigurations[[".WEIGHT."]]) |
| 123 | 156 | eliteParent <- eliteConfigurations[indexEliteParent, ] |
| 124 | 157 | idEliteParent <- eliteParent[[".ID."]] |
| 125 | configuration <- empty.configuration | |
| 158 | configuration <- empty_configuration | |
| 126 | 159 | configuration[[".PARENT."]] <- idEliteParent |
| 127 | 160 | |
| 128 | 161 | # Sample a value for every parameter of the new configuration. |
| 134 | 167 | currentType <- parameters$types[[currentParameter]] |
| 135 | 168 | if (!conditionsSatisfied(parameters, configuration, currentParameter)) { |
| 136 | 169 | # Some conditions are unsatisfied. |
| 137 | # Should be useless, NA is ?always? assigned when matrix created | |
| 170 | # Should be useless, NA is (always?) assigned when matrix created | |
| 138 | 171 | newVal <- NA |
| 139 | 172 | |
| 140 | 173 | } else if (isFixed(currentParameter, parameters)) { |
| 142 | 175 | newVal <- get.fixed.value (currentParameter, parameters) |
| 143 | 176 | # The parameter is not a fixed and should be sampled |
| 144 | 177 | } else if (currentType %in% c("i", "r")) { |
| 178 | domain <- getDependentBound(parameters, currentParameter, configuration) | |
| 145 | 179 | mean <- as.numeric(eliteParent[currentParameter]) |
| 146 | # If there is not value we obtain it from the model | |
| 180 | # If there is not value we obtain it from the model or the mean obtained is | |
| 181 | # not in the current domain, this can happen when using dependent domains | |
| 147 | 182 | if (is.na(mean)) mean <- model[[currentParameter]][[as.character(idEliteParent)]][2] |
| 148 | if (is.na(mean)) { | |
| 183 | if (is.na(mean) || !inNumericDomain(mean, domain)) { | |
| 149 | 184 | # The elite parent does not have any value for this parameter, |
| 150 | 185 | # let's sample uniformly. |
| 151 | newVal <- sample.unif(currentParameter, parameters, currentType, digits) | |
| 152 | ||
| 186 | newVal <- sample_unif(currentType, domain, | |
| 187 | transf = parameters$transform[[currentParameter]], digits) | |
| 153 | 188 | } else { |
| 154 | 189 | stdDev <- model[[currentParameter]][[as.character(idEliteParent)]][1] |
| 155 | newVal <- sample.norm(mean, stdDev, currentParameter, parameters, currentType, digits) | |
| 190 | # If parameters are dependent standard deviation must be computed | |
| 191 | # based on the current domain | |
| 192 | if (parameters$isDependent[currentParameter]) { | |
| 193 | # Conditions should be satisfied for the parameter, thus domain cannot be NA | |
| 194 | stdDev <- (domain[2] - domain[1]) * stdDev | |
| 195 | } | |
| 196 | newVal <- sample_norm(mean, stdDev, currentType, domain, | |
| 197 | transf = parameters$transform[[currentParameter]], | |
| 198 | digits) | |
| 156 | 199 | } |
| 157 | 200 | } else if (currentType == "o") { |
| 158 | 201 | possibleValues <- paramDomain(currentParameter, parameters) |
| 170 | 213 | stdDev <- model[[currentParameter]][[as.character(idEliteParent)]] |
| 171 | 214 | |
| 172 | 215 | # Sample with truncated normal distribution as an integer. |
| 173 | # See sample.norm() for an explanation. | |
| 216 | # See sample_norm() for an explanation. | |
| 174 | 217 | newValAsInt <- floor(rtnorm(1, mean + 0.5, stdDev, lower = 1, |
| 175 | 218 | upper = length(possibleValues) + 1L)) |
| 176 | 219 | |
| 196 | 239 | } |
| 197 | 240 | |
| 198 | 241 | configuration <- as.data.frame(configuration, stringsAsFactors = FALSE) |
| 199 | if (!is.null(repair)) { | |
| 200 | configuration <- repair(configuration, parameters, digits) | |
| 201 | } | |
| 242 | configuration <- repair(configuration, parameters, digits) | |
| 243 | ||
| 202 | 244 | if (is.null(forbidden) |
| 203 | 245 | || nrow(checkForbidden(configuration, forbidden)) == 1) { |
| 204 | 246 | newConfigurations[idxConfiguration,] <- configuration |
| 270 | 312 | # except for the case of log-transformed negative domains, where we have to |
| 271 | 313 | # translate by -0.5. |
| 272 | 314 | # |
| 273 | numeric.value.round <- function(type, value, lowerBound, upperBound, digits) | |
| 315 | numeric_value_round <- function(type, value, lowerBound, upperBound, digits) | |
| 274 | 316 | { |
| 275 | 317 | irace.assert(is.finite(value)) |
| 276 | 318 | if (type == "i") { |
| 277 | 319 | value <- floor(value) |
| 278 | upperBound <- upperBound - 1L # undo the above for the assert | |
| 279 | 320 | # The probability of this happening is very small, but it could happen. |
| 280 | 321 | if (value == upperBound + 1L) |
| 281 | 322 | value <- upperBound |
| 287 | 328 | } |
| 288 | 329 | |
| 289 | 330 | # Sample value for a numerical parameter. |
| 290 | sample.unif <- function(param, parameters, type, digits = NULL) | |
| 291 | { | |
| 292 | lowerBound <- paramLowerBound(param, parameters) | |
| 293 | upperBound <- paramUpperBound(param, parameters) | |
| 294 | transf <- parameters$transform[[param]] | |
| 331 | sample_unif <- function(type, domain, transf, digits) | |
| 332 | { | |
| 333 | # Dependent domains could be not available because of inactivity of parameters | |
| 334 | # on which they are depedent. In this case, the dependent parameter becomes | |
| 335 | # not active and we return NA. | |
| 336 | if (anyNA(domain)) return(NA) | |
| 337 | ||
| 338 | lowerBound <- domain[1] | |
| 339 | upperBound <- domain[2] | |
| 340 | ||
| 295 | 341 | if (type == "i") { |
| 296 | 342 | # +1 for correct rounding before floor() |
| 297 | 343 | upperBound <- 1L + upperBound |
| 302 | 348 | } else { |
| 303 | 349 | value <- runif(1, min = lowerBound, max = upperBound) |
| 304 | 350 | } |
| 305 | value <- numeric.value.round(type, value, lowerBound, upperBound, digits) | |
| 351 | # We use original upperBound, not the +1L for 'i'. | |
| 352 | value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) | |
| 306 | 353 | return(value) |
| 307 | 354 | } |
| 308 | 355 | |
| 309 | sample.norm <- function(mean, sd, param, parameters, type, digits = NULL) | |
| 310 | { | |
| 311 | lowerBound <- paramLowerBound(param, parameters) | |
| 312 | upperBound <- paramUpperBound(param, parameters) | |
| 313 | transf <- parameters$transform[[param]] | |
| 356 | sample_norm <- function(mean, sd, type, domain, transf, digits) | |
| 357 | { | |
| 358 | # Dependent domains could be not available because of inactivity of parameters | |
| 359 | # on which they are depedent. In this case, the dependent parameter becomes | |
| 360 | # not active and we return NA. | |
| 361 | if (anyNA(domain)) return(NA) | |
| 362 | ||
| 363 | lowerBound <- domain[1] | |
| 364 | upperBound <- domain[2] | |
| 365 | ||
| 314 | 366 | if (type == "i") { |
| 315 | 367 | upperBound <- 1L + upperBound |
| 316 | 368 | # Because negative domains are log-transformed to positive domains. |
| 324 | 376 | } else { |
| 325 | 377 | value <- rtnorm(1, mean, sd, lowerBound, upperBound) |
| 326 | 378 | } |
| 327 | ||
| 328 | value <- numeric.value.round(type, value, lowerBound, upperBound, digits) | |
| 379 | # We use original upperBound, not the +1L for 'i'. | |
| 380 | value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) | |
| 329 | 381 | return(value) |
| 330 | 382 | } |
| 2 | 2 | # Variables that do not have a command-line option have description == "" |
| 3 | 3 | # Types are b(oolean), i(nteger), s(tring), r(eal), p(ath), x (R object or no value) |
| 4 | 4 | # FIXME: Add special type for R functions. |
| 5 | # FIXME: For i and r add their range. | |
| 6 | .irace.params.def <- structure(list(name = c(".help", ".version", ".check", ".onlytest", | |
| 7 | "scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", | |
| 8 | "instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", | |
| 9 | "configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", | |
| 10 | "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", | |
| 11 | "targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", | |
| 12 | "digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", | |
| 13 | "sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", | |
| 14 | "nbConfigurations", "mu", "confidence", "deterministic", "seed", | |
| 15 | "parallel", "loadBalancing", "mpi", "batchmode", "softRestart", | |
| 16 | "softRestartThreshold", "testInstancesDir", "testInstancesFile", | |
| 17 | "testInstances", "testNbElites", "testIterationElites", "elitist", | |
| 18 | "elitistNewInstances", "elitistLimit", "repairConfiguration", | |
| 19 | "capping", "cappingType", "boundType", "boundMax", "boundDigits", | |
| 20 | "boundPar", "boundAsTimeout", "postselection", "aclib"), type = c("x", | |
| 21 | "x", "x", "p", "p", "p", "p", "p", "p", "s", "x", "p", "p", "p", | |
| 22 | "x", "p", "p", "i", "x", "x", "p", "i", "i", "r", "i", "i", "i", | |
| 23 | "i", "b", "s", "i", "i", "i", "i", "i", "r", "b", "i", "i", "b", | |
| 24 | "b", "s", "b", "r", "p", "p", "x", "i", "b", "b", "i", "i", "x", | |
| 25 | "b", "s", "s", "i", "i", "i", "b", "r", "b"), short = c("-h", | |
| 26 | "-v", "-c", "", "-s", "-p", "", "-l", "", "", "", "", "", "", | |
| 5 | # FIXME: For i and r add their domain. | |
| 6 | .irace.params.def <- structure(list(name = c(".help", ".version", ".check", ".init", | |
| 7 | ".onlytest", "scenarioFile", "execDir", "parameterFile", "forbiddenExps", | |
| 8 | "forbiddenFile", "initConfigurations", "configurationsFile", | |
| 9 | "logFile", "recoveryFile", "instances", "trainInstancesDir", | |
| 10 | "trainInstancesFile", "sampleInstances", "testInstancesDir", | |
| 11 | "testInstancesFile", "testInstances", "testNbElites", "testIterationElites", | |
| 12 | "testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", | |
| 13 | "targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", | |
| 14 | "targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", | |
| 15 | "maxTime", "budgetEstimation", "minMeasurableTime", "parallel", | |
| 16 | "loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", | |
| 17 | "seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", | |
| 18 | "elitistLimit", "repairConfiguration", "capping", "cappingType", | |
| 19 | "boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", | |
| 20 | "postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", | |
| 21 | "minNbSurvival", "nbConfigurations", "mu", "confidence"), type = c("x", | |
| 22 | "x", "x", "x", "p", "p", "p", "p", "x", "p", "x", "p", "p", "p", | |
| 23 | "s", "p", "p", "b", "p", "p", "x", "i", "b", "s", "i", "i", "p", | |
| 24 | "p", "s", "i", "x", "x", "p", "b", "i", "i", "r", "r", "i", "b", | |
| 25 | "b", "s", "i", "b", "i", "i", "b", "r", "b", "i", "i", "x", "b", | |
| 26 | "s", "s", "i", "i", "i", "b", "r", "b", "i", "i", "i", "i", "i", | |
| 27 | "r"), short = c("-h", "-v", "-c", "-i", "", "-s", "", "-p", "", | |
| 28 | "", "", "", "-l", "", "", "", "", "", "", "", "", "", "", "", | |
| 27 | 29 | "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", |
| 28 | "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", | |
| 29 | "", "", "", "-e", "", "", "", "", "", "", "", "", "", "", "", | |
| 30 | ""), long = c("--help", "--version", "--check", "--only-test", | |
| 31 | "--scenario", "--parameter-file", "--exec-dir", "--log-file", | |
| 32 | "--recovery-file", "", "", "--train-instances-dir", "--train-instances-file", | |
| 33 | "--configurations-file", "", "--forbidden-file", "--target-runner", | |
| 34 | "--target-runner-retries", "", "", "--target-evaluator", "--max-experiments", | |
| 35 | "--max-time", "--budget-estimation", "--digits", "--debug-level", | |
| 36 | "--iterations", "--experiments-per-iteration", "--sample-instances", | |
| 37 | "--test-type", "--first-test", "--each-test", "--min-survival", | |
| 38 | "--num-configurations", "--mu", "--confidence", "--deterministic", | |
| 39 | "--seed", "--parallel", "--load-balancing", "--mpi", "--batchmode", | |
| 40 | "--soft-restart", "--soft-restart-threshold", "--test-instances-dir", | |
| 41 | "--test-instances-file", "", "--test-num-elites", "--test-iteration-elites", | |
| 30 | "", "", "", "-q", "", "", "", "", "-e", "", "", "", "", "", "", | |
| 31 | "", "", "", "", "", "", "", "", "", "", "", ""), long = c("--help", | |
| 32 | "--version", "--check", "--init", "--only-test", "--scenario", | |
| 33 | "--exec-dir", "--parameter-file", "", "--forbidden-file", "", | |
| 34 | "--configurations-file", "--log-file", "--recovery-file", "", | |
| 35 | "--train-instances-dir", "--train-instances-file", "--sample-instances", | |
| 36 | "--test-instances-dir", "--test-instances-file", "", "--test-num-elites", | |
| 37 | "--test-iteration-elites", "--test-type", "--first-test", "--each-test", | |
| 38 | "--target-runner", "--target-runner-launcher", "--target-runner-args", | |
| 39 | "--target-runner-retries", "", "", "--target-evaluator", "--deterministic", | |
| 40 | "--max-experiments", "--max-time", "--budget-estimation", "--min-measurable-time", | |
| 41 | "--parallel", "--load-balancing", "--mpi", "--batchmode", "--digits", | |
| 42 | "--quiet", "--debug-level", "--seed", "--soft-restart", "--soft-restart-threshold", | |
| 42 | 43 | "--elitist", "--elitist-new-instances", "--elitist-limit", "", |
| 43 | 44 | "--capping", "--capping-type", "--bound-type", "--bound-max", |
| 44 | 45 | "--bound-digits", "--bound-par", "--bound-as-timeout", "--postselection", |
| 45 | "--aclib"), default = c(NA, NA, NA, "", "./scenario.txt", "./parameters.txt", | |
| 46 | "./", "./irace.Rdata", "", "", "", "./Instances", "", "", "", | |
| 47 | "", "./target-runner", "0", "", "", "", "0", "0", "0.02", "4", | |
| 48 | "0", "0", "0", "1", "F-test", "5", "1", "0", "0", "5", "0.95", | |
| 49 | "0", NA, "0", "1", "0", "0", "1", "", "", "", "", "1", "0", "1", | |
| 50 | "1", "2", "", "0", "median", "candidate", "0", "0", "1", "1", | |
| 51 | "0", "0"), description = c("Show this help.", "Show irace package version.", | |
| 52 | "Check scenario.", "Only test the configurations given in the file passed as argument.", | |
| 46 | "--aclib", "--iterations", "--experiments-per-iteration", "--min-survival", | |
| 47 | "--num-configurations", "--mu", "--confidence"), default = c(NA, | |
| 48 | NA, NA, "", "", "./scenario.txt", "./", "./parameters.txt", "", | |
| 49 | "", "", "", "./irace.Rdata", "", "", "./Instances", "", "1", | |
| 50 | "", "", "", "1", "0", "", "5", "1", "./target-runner", "", "{targetRunner} {targetRunnerArgs}", | |
| 51 | "0", "", "", "", "0", "0", "0", "0.02", "0.01", "0", "1", "0", | |
| 52 | "0", "4", "0", "0", NA, "1", "", "1", "1", "2", "", "0", "median", | |
| 53 | "candidate", "0", "0", "1", "1", "0", "0", "0", "0", "0", "0", | |
| 54 | "5", "0.95"), domain = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, | |
| 55 | NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "F-test,t-test,t-test-holm,t-test-bonferroni", | |
| 56 | NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, | |
| 57 | NA, "sge,pbs,torque,slurm,htcondor", NA, NA, NA, NA, NA, NA, | |
| 58 | NA, NA, NA, NA, NA, "median,mean,worst,best", "instance,candidate", | |
| 59 | NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA), description = c("Show this help.", | |
| 60 | "Show irace package version.", "Check scenario.", "Initialize the working directory with template config files.", | |
| 61 | "Only test the configurations given in the file passed as argument.", | |
| 53 | 62 | "File that describes the configuration scenario setup and other irace settings.", |
| 54 | "File that contains the description of the parameters of the target algorithm.", | |
| 55 | "Directory where the programs will be run.", "File to save tuning results as an R dataset, either absolute path or relative to execDir.", | |
| 63 | "Directory where the programs will be run.", "File that contains the description of the parameters of the target algorithm.", | |
| 64 | "", "File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions.", | |
| 65 | "", "File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated.", | |
| 66 | "File to save tuning results as an R dataset, either absolute path or relative to execDir.", | |
| 56 | 67 | "Previously saved log file to recover the execution of irace, either absolute path or relative to the current directory. If empty or NULL, recovery is not performed.", |
| 57 | "", "", "Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances.", | |
| 68 | "", "Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances.", | |
| 58 | 69 | "File that contains a list of training instances and optionally additional parameters for them. If trainInstancesDir is provided, irace will search for the files in this folder.", |
| 59 | "File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated.", | |
| 60 | "", "File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions.", | |
| 61 | "Script called for each configuration that executes the target algorithm to be tuned. See templates.", | |
| 70 | "Randomly sample the training instances or use them in the order given.", | |
| 71 | "Directory where testing instances are located, either absolute or relative to current directory.", | |
| 72 | "File containing a list of test instances and optionally additional parameters for them.", | |
| 73 | "", "Number of elite configurations returned by irace that will be tested if test instances are provided.", | |
| 74 | "Enable/disable testing the elite configurations found at each iteration.", | |
| 75 | "Statistical test used for elimination. The default value selects t-test if capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons).", | |
| 76 | "Number of instances evaluated before the first elimination test. It must be a multiple of eachTest.", | |
| 77 | "Number of instances evaluated between elimination tests.", "Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided.", | |
| 78 | "Executable that will be used to launch the target runner, when targetRunner cannot be executed directly (.e.g, a Python script in Windows).", | |
| 79 | "Command-line arguments provided to targetRunnerLauncher. The substrings \\{targetRunner\\} and \\{targetRunnerArgs\\} will be replaced by the value of the option targetRunner and by the arguments usually passed when calling targetRunner, respectively. Example: \"-m {targetRunner --args {targetRunnerArgs}\"}.", | |
| 62 | 80 | "Number of times to retry a call to targetRunner if the call failed.", |
| 63 | 81 | "Optional data passed to targetRunner. This is ignored by the default targetRunner function, but it may be used by custom targetRunner functions to pass persistent data around.", |
| 64 | 82 | "Optional R function to provide custom parallelization of targetRunner.", |
| 65 | 83 | "Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl", |
| 84 | "If the target algorithm is deterministic, configurations will be evaluated only once per instance.", | |
| 66 | 85 | "Maximum number of runs (invocations of targetRunner) that will be performed. It determines the maximum budget of experiments for the tuning.", |
| 67 | 86 | "Maximum total execution time in seconds for the executions of targetRunner. targetRunner must return two values: cost and time.", |
| 68 | 87 | "Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when maxTime > 0", |
| 69 | "Maximum number of decimal places that are significant for numerical (real) parameters.", | |
| 70 | "Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages.", | |
| 71 | "Number of iterations.", "Number of runs of the target algorithm per iteration.", | |
| 72 | "Randomly sample the training instances or use them in the order given.", | |
| 73 | "Statistical test used for elimination. Default test is always F-test unless capping is enabled, in which case the default test is t-test. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons).", | |
| 74 | "Number of instances evaluated before the first elimination test. It must be a multiple of eachTest.", | |
| 75 | "Number of instances evaluated between elimination tests.", "Minimum number of configurations needed to continue the execution of each race (iteration).", | |
| 76 | "Number of configurations to be sampled and evaluated at each iteration.", | |
| 77 | "Parameter used to define the number of configurations sampled and evaluated at each iteration.", | |
| 78 | "Confidence level for the elimination test.", "If the target algorithm is deterministic, configurations will be evaluated only once per instance.", | |
| 79 | "Seed of the random number generator (by default, generate a random seed).", | |
| 88 | "Minimum time unit that is still (significantly) measureable.", | |
| 80 | 89 | "Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.", |
| 81 | 90 | "Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster.", |
| 82 | 91 | "Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter parallel is the number of slaves).", |
| 83 | "Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque or slurm. targetRunner must submit jobs to the cluster using, for example, qsub.", | |
| 92 | "Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must submit jobs to the cluster using, for example, qsub.", | |
| 93 | "Maximum number of decimal places that are significant for numerical (real) parameters.", | |
| 94 | "Reduce the output generated by irace to a minimum.", "Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages.", | |
| 95 | "Seed of the random number generator (by default, generate a random seed).", | |
| 84 | 96 | "Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model.", |
| 85 | 97 | "Soft restart threshold value for numerical parameters. If NA, NULL or \"\", it is computed as 10^-digits.", |
| 86 | "Directory where testing instances are located, either absolute or relative to current directory.", | |
| 87 | "File containing a list of test instances and optionally additional parameters for them.", | |
| 88 | "", "Number of elite configurations returned by irace that will be tested if test instances are provided.", | |
| 89 | "Enable/disable testing the elite configurations found at each iteration.", | |
| 90 | 98 | "Enable/disable elitist irace.", "Number of instances added to the execution list before previous instances in elitist irace.", |
| 91 | 99 | "In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit.", |
| 92 | 100 | "User-defined R function that takes a configuration generated by irace and repairs it.", |
| 98 | 106 | "Penalization constant for timed out executions (executions that reach boundMax execution time).", |
| 99 | 107 | "Replace the configuration cost of bounded executions with boundMax.", |
| 100 | 108 | "Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace.", |
| 101 | "Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script." | |
| 102 | )), .Names = c("name", "type", "short", "long", "default", "description" | |
| 103 | ), row.names = c(".help", ".version", ".check", ".onlytest", | |
| 104 | "scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", | |
| 105 | "instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", | |
| 106 | "configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", | |
| 107 | "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", | |
| 108 | "targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", | |
| 109 | "digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", | |
| 110 | "sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", | |
| 111 | "nbConfigurations", "mu", "confidence", "deterministic", "seed", | |
| 112 | "parallel", "loadBalancing", "mpi", "batchmode", "softRestart", | |
| 113 | "softRestartThreshold", "testInstancesDir", "testInstancesFile", | |
| 114 | "testInstances", "testNbElites", "testIterationElites", "elitist", | |
| 115 | "elitistNewInstances", "elitistLimit", "repairConfiguration", | |
| 109 | "Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script.", | |
| 110 | "Maximum number of iterations.", "Number of runs of the target algorithm per iteration.", | |
| 111 | "Minimum number of configurations needed to continue the execution of each race (iteration).", | |
| 112 | "Number of configurations to be sampled and evaluated at each iteration.", | |
| 113 | "Parameter used to define the number of configurations sampled and evaluated at each iteration.", | |
| 114 | "Confidence level for the elimination test.")), row.names = c(".help", | |
| 115 | ".version", ".check", ".init", ".onlytest", "scenarioFile", "execDir", | |
| 116 | "parameterFile", "forbiddenExps", "forbiddenFile", "initConfigurations", | |
| 117 | "configurationsFile", "logFile", "recoveryFile", "instances", | |
| 118 | "trainInstancesDir", "trainInstancesFile", "sampleInstances", | |
| 119 | "testInstancesDir", "testInstancesFile", "testInstances", "testNbElites", | |
| 120 | "testIterationElites", "testType", "firstTest", "eachTest", "targetRunner", | |
| 121 | "targetRunnerLauncher", "targetRunnerLauncherArgs", "targetRunnerRetries", | |
| 122 | "targetRunnerData", "targetRunnerParallel", "targetEvaluator", | |
| 123 | "deterministic", "maxExperiments", "maxTime", "budgetEstimation", | |
| 124 | "minMeasurableTime", "parallel", "loadBalancing", "mpi", "batchmode", | |
| 125 | "digits", "quiet", "debugLevel", "seed", "softRestart", "softRestartThreshold", | |
| 126 | "elitist", "elitistNewInstances", "elitistLimit", "repairConfiguration", | |
| 116 | 127 | "capping", "cappingType", "boundType", "boundMax", "boundDigits", |
| 117 | "boundPar", "boundAsTimeout", "postselection", "aclib"), class = "data.frame") | |
| 118 | .irace.params.names <- c("scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", | |
| 119 | "instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", | |
| 120 | "configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", | |
| 121 | "targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", | |
| 122 | "targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", | |
| 123 | "digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", | |
| 124 | "sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", | |
| 125 | "nbConfigurations", "mu", "confidence", "deterministic", "seed", | |
| 126 | "parallel", "loadBalancing", "mpi", "batchmode", "softRestart", | |
| 127 | "softRestartThreshold", "testInstancesDir", "testInstancesFile", | |
| 128 | "testInstances", "testNbElites", "testIterationElites", "elitist", | |
| 129 | "elitistNewInstances", "elitistLimit", "repairConfiguration", | |
| 130 | "capping", "cappingType", "boundType", "boundMax", "boundDigits", | |
| 131 | "boundPar", "boundAsTimeout", "postselection", "aclib") | |
| 128 | "boundPar", "boundAsTimeout", "postselection", "aclib", "nbIterations", | |
| 129 | "nbExperimentsPerIteration", "minNbSurvival", "nbConfigurations", | |
| 130 | "mu", "confidence"), class = "data.frame") | |
| 131 | .irace.params.names <- c("scenarioFile", "execDir", "parameterFile", "forbiddenExps", | |
| 132 | "forbiddenFile", "initConfigurations", "configurationsFile", | |
| 133 | "logFile", "recoveryFile", "instances", "trainInstancesDir", | |
| 134 | "trainInstancesFile", "sampleInstances", "testInstancesDir", | |
| 135 | "testInstancesFile", "testInstances", "testNbElites", "testIterationElites", | |
| 136 | "testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", | |
| 137 | "targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", | |
| 138 | "targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", | |
| 139 | "maxTime", "budgetEstimation", "minMeasurableTime", "parallel", | |
| 140 | "loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", | |
| 141 | "seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", | |
| 142 | "elitistLimit", "repairConfiguration", "capping", "cappingType", | |
| 143 | "boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", | |
| 144 | "postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", | |
| 145 | "minNbSurvival", "nbConfigurations", "mu", "confidence") | |
| 132 | 146 | ## FIXME: If these values are special perhaps they should be saved in $state ? |
| 133 | 147 | .irace.params.recover <- c("instances", "seed", "testInstances", |
| 134 | 148 | # We need this because this data may mutate |
| 5 | 5 | #' @docType package |
| 6 | 6 | #' @import stats utils compiler |
| 7 | 7 | #' @importFrom R6 R6Class |
| 8 | #' @importFrom grDevices dev.new dev.off pdf cairo_pdf rgb | |
| 9 | #' @importFrom graphics abline axis barplot boxplot hist lines matplot mtext par plot points strwidth text bxp grid | |
| 8 | #' @importFrom grDevices dev.new dev.off pdf | |
| 9 | #' @importFrom graphics abline axis barplot boxplot lines matplot mtext par plot points strwidth text bxp grid | |
| 10 | 10 | #' |
| 11 | 11 | #' |
| 12 | 12 | #' @details License: GPL (>= 2) |
| 58 | 58 | #' sum(x * x - 10 * cos(2 * pi * x) + 10) |
| 59 | 59 | #' } |
| 60 | 60 | #' |
| 61 | #' ## We generate 200 instances (in this case, weights): | |
| 62 | #' weights <- rnorm(200, mean = 0.9, sd = 0.02) | |
| 61 | #' ## We generate 20 instances (in this case, weights): | |
| 62 | #' weights <- rnorm(20, mean = 0.9, sd = 0.02) | |
| 63 | 63 | #' |
| 64 | 64 | #' ## On this set of instances, we are interested in optimizing two |
| 65 | 65 | #' ## parameters of the SANN algorithm: tmax and temp. We setup the |
| 66 | 66 | #' ## parameter space as follows: |
| 67 | #' parameters.table <- ' | |
| 68 | #' tmax "" i (1, 5000) | |
| 67 | #' parameters_table <- ' | |
| 68 | #' tmax "" i,log (1, 5000) | |
| 69 | 69 | #' temp "" r (0, 100) |
| 70 | 70 | #' ' |
| 71 | 71 | #' |
| 72 | 72 | #' ## We use the irace function readParameters to read this table: |
| 73 | #' parameters <- readParameters(text = parameters.table) | |
| 73 | #' parameters <- readParameters(text = parameters_table) | |
| 74 | 74 | #' |
| 75 | 75 | #' ## Next, we define the function that will evaluate each candidate |
| 76 | 76 | #' ## configuration on a single instance. For simplicity, we restrict to |
| 77 | 77 | #' ## three-dimensional functions and we set the maximum number of |
| 78 | #' ## iterations of SANN to 5000. | |
| 79 | #' target.runner <- function(experiment, scenario) | |
| 78 | #' ## iterations of SANN to 1000. | |
| 79 | #' target_runner <- function(experiment, scenario) | |
| 80 | 80 | #' { |
| 81 | 81 | #' instance <- experiment$instance |
| 82 | 82 | #' configuration <- experiment$configuration |
| 88 | 88 | #' return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) |
| 89 | 89 | #' } |
| 90 | 90 | #' res <- stats::optim(par,fn, method="SANN", |
| 91 | #' control=list(maxit=5000 | |
| 91 | #' control=list(maxit=1000 | |
| 92 | 92 | #' , tmax = as.numeric(configuration[["tmax"]]) |
| 93 | 93 | #' , temp = as.numeric(configuration[["temp"]]) |
| 94 | 94 | #' )) |
| 97 | 97 | #' ## - 'error' is a string used to report an error |
| 98 | 98 | #' ## - 'outputRaw' is a string used to report the raw output of calls to |
| 99 | 99 | #' ## an external program or function. |
| 100 | #' ## - 'call' is a string used to report how target.runner called the | |
| 100 | #' ## - 'call' is a string used to report how target_runner called the | |
| 101 | 101 | #' ## external program or function. |
| 102 | 102 | #' return(list(cost = res$value)) |
| 103 | 103 | #' } |
| 104 | 104 | #' |
| 105 | 105 | #' ## We define a configuration scenario by setting targetRunner to the |
| 106 | #' ## function define above, instances to the first 100 random weights, and | |
| 107 | #' ## a maximum budget of 1000 calls to targetRunner. | |
| 108 | #' scenario <- list(targetRunner = target.runner, | |
| 109 | #' instances = weights[1:100], | |
| 110 | #' maxExperiments = 1000, | |
| 106 | #' ## function define above, instances to the first 10 random weights, and | |
| 107 | #' ## a maximum budget of 'maxExperiments' calls to targetRunner. | |
| 108 | #' scenario <- list(targetRunner = target_runner, | |
| 109 | #' instances = weights[1:10], | |
| 110 | #' maxExperiments = 500, | |
| 111 | 111 | #' # Do not create a logFile |
| 112 | 112 | #' logFile = "") |
| 113 | 113 | #' |
| 114 | 114 | #' ## We check that the scenario is valid. This will also try to execute |
| 115 | #' ## target.runner. | |
| 115 | #' ## target_runner. | |
| 116 | 116 | #' checkIraceScenario(scenario, parameters = parameters) |
| 117 | 117 | #' |
| 118 | 118 | #' \donttest{ |
| 119 | 119 | #' ## We are now ready to launch irace. We do it by means of the irace |
| 120 | 120 | #' ## function. The function will print information about its |
| 121 | 121 | #' ## progress. This may require a few minutes, so it is not run by default. |
| 122 | #' tuned.confs <- irace(scenario = scenario, parameters = parameters) | |
| 122 | #' tuned_confs <- irace(scenario = scenario, parameters = parameters) | |
| 123 | 123 | #' |
| 124 | 124 | #' ## We can print the best configurations found by irace as follows: |
| 125 | #' configurations.print(tuned.confs) | |
| 125 | #' configurations.print(tuned_confs) | |
| 126 | 126 | #' |
| 127 | 127 | #' ## We can evaluate the quality of the best configuration found by |
| 128 | 128 | #' ## irace versus the default configuration of the SANN algorithm on |
| 129 | #' ## the other 100 instances previously generated. | |
| 129 | #' ## the other 10 instances previously generated. | |
| 130 | 130 | #' ## To do so, first we apply the default configuration of the SANN |
| 131 | 131 | #' ## algorithm to these instances: |
| 132 | 132 | #' test <- function(configuration) |
| 133 | 133 | #' { |
| 134 | #' res <- lapply(weights[101:200], | |
| 135 | #' function(x) target.runner( | |
| 134 | #' res <- lapply(weights[11:20], | |
| 135 | #' function(x) target_runner( | |
| 136 | 136 | #' experiment = list(instance = x, |
| 137 | 137 | #' configuration = configuration), |
| 138 | 138 | #' scenario = scenario)) |
| 142 | 142 | |
| 143 | 143 | #' ## We extract and apply the winning configuration found by irace |
| 144 | 144 | #' ## to these instances: |
| 145 | #' tuned <- test (removeConfigurationsMetaData(tuned.confs[1,])) | |
| 145 | #' tuned <- test(removeConfigurationsMetaData(tuned_confs[1,])) | |
| 146 | 146 | #' |
| 147 | 147 | #' ## Finally, we can compare using a boxplot the quality obtained with the |
| 148 | 148 | #' ## default parametrization of SANN and the quality obtained with the |
| 155 | 155 | #' |
| 156 | 156 | NULL |
| 157 | 157 | |
| 158 | # Prefix for printing messages to the user. | |
| 159 | .msg.prefix <- "== irace == " | |
| 24 | 24 | # .Random.seed and .irace are special |
| 25 | 25 | for (name in setdiff(names(iraceResults$state), c(".Random.seed", ".irace"))) |
| 26 | 26 | assign(name, iraceResults$state[[name]]) |
| 27 | # FIXME: Check that irace.version matches and warn if not. | |
| 27 | 28 | assign(".Random.seed", iraceResults$state$.Random.seed, .GlobalEnv) |
| 28 | 29 | for (name in ls(iraceResults$state$.irace)) |
| 29 | 30 | assign(name, get(name, envir = iraceResults$state$.irace), envir = .irace) |
| 49 | 50 | selected <- 1:nrow(configurations) |
| 50 | 51 | for (i in seq_along(param.names)) { |
| 51 | 52 | param <- param.names[i] |
| 52 | lower <- parameters$domain[[param]][1] | |
| 53 | upper <- parameters$domain[[param]][2] | |
| 53 | x.domain <- getDependentBound(parameters, param, x) | |
| 54 | x.range <- diff(x.domain) | |
| 54 | 55 | |
| 55 | 56 | X <- x[[param]] |
| 56 | 57 | # FIXME: Since at the end we select a subset of configurations, we could use selected here. |
| 68 | 69 | # FIXME: Why is this updating d[j]? It seems that if the difference is |
| 69 | 70 | # large for one configuration, then it will be assumed to be large for |
| 70 | 71 | # the rest. |
| 71 | d[j] <- max(d[j], abs((as.numeric(X) - as.numeric(Y)) / (upper - lower))) | |
| 72 | if (parameters$isDependent[param]) { | |
| 73 | # Compare depedent domains by normalising their values to their own ranges first | |
| 74 | # and calculating the difference. (When possible) | |
| 75 | y.domain <- getDependentBound(parameters, param, configurations[selected[j],]) | |
| 76 | y.range <- diff(x.domain) | |
| 77 | dx <- ifelse (x.range == 0, 0, (as.numeric(X) - x.domain[1]) / x.range) | |
| 78 | dy <- ifelse (y.range == 0, 0, (as.numeric(Y) - y.domain[1]) / y.range) | |
| 79 | ||
| 80 | d[j] <- max(d[j], abs(dx - dy)) | |
| 81 | } else { | |
| 82 | # FIXME: We should calculate (X - x.domain[1]) / x.range once for all configurations | |
| 83 | # and all parameters, then calculate the differences using vectorization. | |
| 84 | d[j] <- max(d[j], abs((as.numeric(X) - as.numeric(Y)) / x.range)) | |
| 85 | } | |
| 72 | 86 | if (d[j] > threshold) isSimilar.mat[j,i] <- FALSE |
| 73 | 87 | } |
| 74 | 88 | } |
| 126 | 140 | ## filtering them out: |
| 127 | 141 | configurations <- configurations [keepIdx, , drop=FALSE] |
| 128 | 142 | ## filtering their strings out (to use them to define blocks): |
| 129 | strings <- strings [keepIdx] | |
| 143 | strings <- strings[keepIdx] | |
| 130 | 144 | |
| 131 | 145 | ## if everything is already filtered out, return |
| 132 | 146 | if (nrow(configurations) == 0) { |
| 190 | 204 | |
| 191 | 205 | |
| 192 | 206 | ## Number of iterations. |
| 193 | computeNbIterations <- function(nbParameters) | |
| 194 | { | |
| 195 | return (2 + log2(nbParameters)) | |
| 196 | } | |
| 207 | computeNbIterations <- function(nbParameters) (2 + log2(nbParameters)) | |
| 197 | 208 | |
| 198 | 209 | ## Computational budget at each iteration. |
| 199 | 210 | computeComputationalBudget <- function(remainingBudget, indexIteration, |
| 218 | 229 | |
| 219 | 230 | ## Termination of a race at each iteration. The race will stop if the |
| 220 | 231 | ## number of surviving configurations is equal or less than this number. |
| 221 | computeTerminationOfRace <- function(nbParameters) | |
| 222 | { | |
| 223 | return (2 + log2(nbParameters)) | |
| 224 | } | |
| 232 | computeTerminationOfRace <- function(nbParameters) (2 + log2(nbParameters)) | |
| 225 | 233 | |
| 226 | 234 | ## Compute the minimum budget required, and exit early in case the |
| 227 | 235 | ## budget given by the user is insufficient. |
| 329 | 337 | } else { |
| 330 | 338 | requireNamespace("parallel", quietly = TRUE) |
| 331 | 339 | if (.Platform$OS.type == 'windows' && is.null(.irace$cluster)) { |
| 340 | # FIXME: makeCluster does not print the output generated by the workers | |
| 341 | # on Windows. We need to use the future package for that: | |
| 342 | # https://stackoverflow.com/questions/56501937/how-to-print-from-clusterapply | |
| 332 | 343 | .irace$cluster <- parallel::makeCluster(parallel) |
| 344 | if (scenario$debugLevel >= 1) irace.note("makeCluster initialized for ", parallel, " jobs.") | |
| 333 | 345 | # In Windows, this needs to be exported, or we get: |
| 334 | 346 | ## Error in checkForRemoteErrors(val) : |
| 335 | 347 | ## 2 nodes produced errors; first error: could not find function "target.runner" |
| 348 | parallel::clusterExport(.irace$cluster, ls(environment(startParallel)), envir=environment(startParallel)) | |
| 336 | 349 | parallel::clusterExport(.irace$cluster, list("target.runner"), envir=.irace) |
| 337 | 350 | # In addition, we export the global environment because the user may |
| 338 | 351 | # have defined stuff there. There must be a better way to do this, but |
| 346 | 359 | stopParallel <- function() |
| 347 | 360 | { |
| 348 | 361 | if (!is.null(.irace$cluster)) { |
| 349 | parallel::stopCluster(.irace$cluster) | |
| 362 | try(parallel::stopCluster(.irace$cluster), silent=TRUE) | |
| 350 | 363 | .irace$cluster <- NULL |
| 351 | 364 | } |
| 352 | 365 | } |
| 355 | 368 | { |
| 356 | 369 | # We need to do this here to use/recover .Random.seed later. |
| 357 | 370 | if (is.na(scenario$seed)) { |
| 371 | # FIXME: We should store this seed in state not in scenario. We should not modify scenario. | |
| 358 | 372 | scenario$seed <- trunc(runif(1, 1, .Machine$integer.max)) |
| 359 | 373 | } |
| 360 | 374 | set.seed(scenario$seed) |
| 362 | 376 | irace.note("RNGkind: ", paste0(RNGkind(), collapse = " "), "\n") |
| 363 | 377 | irace.note(".Random.seed: ", paste0(.Random.seed, collapse = ", "), "\n") |
| 364 | 378 | } |
| 365 | return(scenario) | |
| 379 | scenario | |
| 366 | 380 | } |
| 367 | 381 | |
| 368 | 382 | ## Generate instances + seed. |
| 379 | 393 | # Sample instances index in groups (ntimes) |
| 380 | 394 | sindex <- as.vector(sapply(rep(length(instances), ntimes), sample.int, replace = FALSE)) |
| 381 | 395 | } else { |
| 382 | sindex <- rep(1:length(instances), ntimes) | |
| 396 | sindex <- rep(1L:length(instances), ntimes) | |
| 383 | 397 | } |
| 384 | 398 | # Sample seeds. |
| 385 | 399 | # 2147483647 is the maximum value for a 32-bit signed integer. |
| 386 | 400 | # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. |
| 387 | tmp <- data.frame (instance = sindex, | |
| 388 | seed = sample.int(2147483647, size = ntimes * length(instances), replace = TRUE)) | |
| 389 | return(tmp) | |
| 401 | data.frame(instance = sindex, | |
| 402 | seed = sample.int(2147483647L, size = length(sindex), replace = TRUE), stringsAsFactors=FALSE) | |
| 390 | 403 | } |
| 391 | 404 | |
| 392 | 405 | addInstances <- function(scenario, instancesList, n.instances) |
| 395 | 408 | if (is.null.or.empty(instancesList)) |
| 396 | 409 | instancesList <- generateInstances(scenario, n.instances) |
| 397 | 410 | # If deterministic, we have already added all instances. |
| 398 | else if (! scenario$deterministic) | |
| 411 | else if (!scenario$deterministic) | |
| 399 | 412 | instancesList <- rbind(instancesList, generateInstances(scenario, n.instances)) |
| 400 | 413 | |
| 401 | 414 | # FIXME: Something is adding rownames. Clear them to avoid future problems. |
| 468 | 481 | scenario$configurationsFile, "'.") |
| 469 | 482 | cat("# Adding", nrow(initConfigurations), "initial configuration(s)\n") |
| 470 | 483 | if (scenario$debugLevel >= 2) |
| 471 | print(as.data.frame(scenario$initConfigurations, stringAsFactor = FALSE), digits=15) | |
| 484 | print(as.data.frame(scenario$initConfigurations, stringsAsFactors = FALSE), digits=15) | |
| 472 | 485 | } else { |
| 473 | 486 | initConfigurations <- confs_from_file |
| 474 | 487 | } |
| 490 | 503 | allConfigurations <- |
| 491 | 504 | as.data.frame(matrix(ncol = length(configurations.colnames), |
| 492 | 505 | nrow = 0, |
| 493 | dimnames = list(NULL, configurations.colnames))) | |
| 494 | } | |
| 495 | return(allConfigurations) | |
| 506 | dimnames = list(NULL, configurations.colnames)), | |
| 507 | stringsAsFactors=FALSE) | |
| 508 | } | |
| 509 | allConfigurations | |
| 496 | 510 | } |
| 497 | 511 | |
| 498 | 512 | #' irace |
| 499 | 513 | #' |
| 500 | #' \code{irace} implements iterated Race. It receives some parameters to be tuned | |
| 514 | #' `irace` implements iterated Race. It receives some parameters to be tuned | |
| 501 | 515 | #' and returns the best configurations found, namely, the elite configurations |
| 502 | 516 | #' obtained from the last iterations (and sorted by rank). |
| 503 | 517 | #' |
| 504 | 518 | #' @template arg_scenario |
| 505 | 519 | #' @template arg_parameters |
| 506 | 520 | #' |
| 507 | #' @details The function \code{irace} executes the tuning procedure using | |
| 508 | #' the information provided in \code{scenario} and \code{parameters}. Initially it checks | |
| 509 | #' the correctness of \code{scenario} and recovers a previous execution if | |
| 510 | #' \code{scenario$recoveryFile} is set. A R data file log of the execution is created | |
| 511 | #' in \code{scenario$logFile}. | |
| 521 | #' @details The function `irace` executes the tuning procedure using | |
| 522 | #' the information provided in `scenario` and `parameters`. Initially it checks | |
| 523 | #' the correctness of `scenario` and recovers a previous execution if | |
| 524 | #' `scenario$recoveryFile` is set. A R data file log of the execution is created | |
| 525 | #' in `scenario$logFile`. | |
| 512 | 526 | #' |
| 513 | 527 | #' @template return_irace |
| 514 | 528 | #' @examples |
| 515 | 529 | #' \dontrun{ |
| 516 | 530 | #' parameters <- readParameters("parameters.txt") |
| 517 | #' scenario <- readScenario(filename = "scenario.txt", | |
| 518 | #' scenario = defaultScenario()) | |
| 531 | #' scenario <- readScenario(filename = "scenario.txt") | |
| 519 | 532 | #' irace(scenario = scenario, parameters = parameters) |
| 520 | 533 | #' } |
| 521 | 534 | #' |
| 522 | 535 | #' @seealso |
| 523 | 536 | #' \describe{ |
| 524 | #' \item{\code{\link{irace.main}}}{a higher-level command-line interface to \code{irace}.} | |
| 525 | #' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 526 | #' \item{\code{\link{readParameters}}}{read the target algorithm parameters from a file.} | |
| 527 | #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 528 | #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 537 | #' \item{[irace.main()]}{a higher-level interface to `irace`.} | |
| 538 | #' \item{[irace.cmdline()]}{a command-line interface to `irace`.} | |
| 539 | #' \item{[readScenario()]}{for reading a configuration scenario from a file.} | |
| 540 | #' \item{[readParameters()]}{read the target algorithm parameters from a file.} | |
| 541 | #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} | |
| 542 | #' \item{[checkScenario()]}{to check that the scenario is valid.} | |
| 529 | 543 | #' } |
| 530 | 544 | #' |
| 531 | 545 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 546 | #' @concept running | |
| 532 | 547 | #' @export |
| 533 | 548 | irace <- function(scenario, parameters) |
| 534 | { | |
| 535 | catInfo <- function(..., verbose = TRUE) { | |
| 549 | irace_common(scenario, parameters, simple = TRUE) | |
| 550 | ||
| 551 | irace_common <- function(scenario, parameters, simple, output.width = 9999L) | |
| 552 | { | |
| 553 | if (!simple) { | |
| 554 | op <- options(width = output.width) # Do not wrap the output. | |
| 555 | on.exit(options(op), add = TRUE) | |
| 556 | } | |
| 557 | scenario <- checkScenario(scenario) | |
| 558 | debugLevel <- scenario$debugLevel | |
| 559 | ||
| 560 | if (debugLevel >= 1) { | |
| 561 | op.debug <- options(warning.length = 8170, | |
| 562 | error = if (interactive()) utils::recover | |
| 563 | else irace.dump.frames) | |
| 564 | on.exit(options(op.debug), add = TRUE) | |
| 565 | printScenario (scenario) | |
| 566 | } | |
| 567 | ||
| 568 | if (missing(parameters)) { | |
| 569 | # Read parameters definition | |
| 570 | parameters <- readParameters (file = scenario$parameterFile, | |
| 571 | digits = scenario$digits, | |
| 572 | debugLevel = debugLevel) | |
| 573 | } else { | |
| 574 | parameters <- checkParameters(parameters) | |
| 575 | } | |
| 576 | ||
| 577 | eliteConfigurations <- irace_run(scenario = scenario, parameters = parameters) | |
| 578 | if (simple) return(eliteConfigurations) | |
| 579 | ||
| 580 | if (!scenario$quiet) { | |
| 581 | cat("# Best configurations (first number is the configuration ID;", | |
| 582 | " listed from best to worst according to the ", | |
| 583 | test.type.order.str(scenario$testType), "):\n", sep = "") | |
| 584 | configurations.print(eliteConfigurations) | |
| 585 | ||
| 586 | cat("# Best configurations as commandlines (first number is the configuration ID; same order as above):\n") | |
| 587 | configurations.print.command (eliteConfigurations, parameters) | |
| 588 | } | |
| 589 | ||
| 590 | if (scenario$postselection > 0) | |
| 591 | psRace(iraceLogFile=scenario$logFile, postselection=scenario$postselection, elites=TRUE) | |
| 592 | ||
| 593 | testing_fromlog(logFile = scenario$logFile) | |
| 594 | ||
| 595 | invisible(eliteConfigurations) | |
| 596 | } | |
| 597 | ||
| 598 | irace_run <- function(scenario, parameters) | |
| 599 | { | |
| 600 | quiet <- scenario$quiet | |
| 601 | catInfo <- if (quiet) do_nothing else function(..., verbose = TRUE) { | |
| 536 | 602 | irace.note (..., "\n") |
| 537 | 603 | if (verbose) { |
| 538 | 604 | cat ("# Iteration: ", indexIteration, "\n", |
| 540 | 606 | "# experimentsUsedSoFar: ", experimentsUsedSoFar, "\n", |
| 541 | 607 | "# timeUsed: ", timeUsed, "\n", |
| 542 | 608 | "# remainingBudget: ", remainingBudget, "\n", |
| 543 | "# currentBudget: ", currentBudget, "\n", | |
| 609 | "# currentBudget: ", currentBudget, "\n", | |
| 544 | 610 | "# number of elites: ", nrow(eliteConfigurations), "\n", |
| 545 | 611 | "# nbConfigurations: ", nbConfigurations, "\n", |
| 546 | 612 | sep = "") |
| 547 | 613 | } |
| 548 | 614 | } |
| 549 | ||
| 550 | scenario <- checkScenario(defaultScenario(scenario)) | |
| 615 | ||
| 616 | irace_finish <- function(iraceResults, scenario, reason) { | |
| 617 | elapsed <- timer$elapsed() | |
| 618 | if (!quiet) | |
| 619 | cat("# Total CPU user time: ", elapsed["user"], ", CPU sys time: ", elapsed["system"], | |
| 620 | ", Wall-clock time: ", elapsed["wallclock"], "\n", sep="") | |
| 621 | iraceResults$state$elapsed = elapsed | |
| 622 | iraceResults$state$completed = reason | |
| 623 | irace_save_logfile(iraceResults, scenario) | |
| 624 | iraceResults$state$eliteConfigurations | |
| 625 | } | |
| 626 | ||
| 627 | timer <- Timer$new() | |
| 628 | debugLevel <- scenario$debugLevel | |
| 551 | 629 | |
| 552 | 630 | # Recover state from file? |
| 553 | if (!is.null(scenario$recoveryFile)) { | |
| 631 | if (!is.null.or.empty(scenario$recoveryFile)) { | |
| 554 | 632 | irace.note ("Resuming from file: '", scenario$recoveryFile,"'\n") |
| 555 | 633 | recoverFromFile(scenario$recoveryFile) |
| 556 | 634 | # We call checkScenario again to fix any inconsistencies in the recovered |
| 557 | 635 | # data. |
| 636 | # FIXME: Do not call checkScenario earlier and instead do the minimum to check recoveryFile. | |
| 637 | scenario <- checkScenario(scenario) | |
| 558 | 638 | firstRace <- FALSE |
| 559 | scenario <- checkScenario(scenario) | |
| 639 | stopParallel() | |
| 560 | 640 | startParallel(scenario) |
| 561 | 641 | on.exit(stopParallel(), add = TRUE) |
| 562 | ||
| 563 | 642 | } else { # Do not recover |
| 564 | 643 | firstRace <- TRUE |
| 565 | scenario <- irace.init (scenario) | |
| 644 | scenario <- irace.init(scenario) | |
| 566 | 645 | forbiddenExps <- scenario$forbiddenExps |
| 567 | debugLevel <- scenario$debugLevel | |
| 568 | 646 | # Set options controlling debug level. |
| 569 | 647 | # FIXME: This should be the other way around, the options set the debugLevel. |
| 570 | 648 | options(.race.debug.level = debugLevel) |
| 571 | 649 | options(.irace.debug.level = debugLevel) |
| 572 | ||
| 573 | 650 | # Create a data frame of all configurations ever generated. |
| 574 | 651 | allConfigurations <- allConfigurationsInit(scenario, parameters) |
| 575 | 652 | nbUserConfigurations <- nrow(allConfigurations) |
| 587 | 664 | ) |
| 588 | 665 | model <- NULL |
| 589 | 666 | nbConfigurations <- 0 |
| 590 | eliteConfigurations <- data.frame() | |
| 667 | eliteConfigurations <- data.frame(stringsAsFactors=FALSE) | |
| 591 | 668 | |
| 592 | 669 | nbIterations <- ifelse (scenario$nbIterations == 0, |
| 593 | 670 | computeNbIterations(parameters$nbVariable), |
| 651 | 728 | output <- do.experiments(configurations = allConfigurations[next.configuration:nconfigurations, ], |
| 652 | 729 | ninstances = ninstances, scenario = scenario, parameters = parameters) |
| 653 | 730 | iraceResults$experimentLog <- rbind(iraceResults$experimentLog, |
| 654 | cbind(rep(0, nrow(output$experimentLog)), | |
| 655 | output$experimentLog)) | |
| 731 | # These experiments are assigned iteration 0 | |
| 732 | cbind(iteration=0L, output$experimentLog)) | |
| 656 | 733 | |
| 657 | 734 | iraceResults$experiments <- merge.matrix (iraceResults$experiments, |
| 658 | 735 | output$experiments) |
| 686 | 763 | } else { |
| 687 | 764 | nconfigurations <- min(1024, nconfigurations + new.conf) |
| 688 | 765 | } |
| 689 | } | |
| 766 | } # end of while(TRUE) | |
| 690 | 767 | |
| 691 | 768 | if (length(rejectedIDs) > 0) { |
| 692 | 769 | irace.note ("Immediately rejected configurations: ", |
| 763 | 840 | if (scenario$capping) |
| 764 | 841 | paste0("# capping: ", scenario$cappingType, "\n", |
| 765 | 842 | "# type bound: ", scenario$boundType, "\n", |
| 766 | "# maxBound: ", scenario$boundMax, "\n", | |
| 843 | "# boundMax: ", scenario$boundMax, "\n", | |
| 767 | 844 | "# par bound: ", scenario$boundPar, "\n", |
| 768 | 845 | "# bound digits: ", scenario$boundDigits, "\n") |
| 769 | 846 | else if (!is.null(scenario$boundMax)) |
| 770 | paste0("# maxBound: ", scenario$boundMax, "\n"), | |
| 847 | paste0("# boundMax: ", scenario$boundMax, "\n"), | |
| 771 | 848 | verbose = FALSE) |
| 772 | 849 | |
| 773 | 850 | |
| 789 | 866 | boundEstimate = boundEstimate, |
| 790 | 867 | rejectedIDs = rejectedIDs, |
| 791 | 868 | forbiddenExps = forbiddenExps, |
| 792 | completed = list(flag=FALSE, msg="")) | |
| 793 | # Consistency checks | |
| 794 | irace.assert(sum(!is.na(iraceResults$experiments)) == experimentsUsedSoFar) | |
| 795 | irace.assert(nrow(iraceResults$experimentLog) == experimentsUsedSoFar) | |
| 796 | ||
| 869 | completed = "Incomplete") | |
| 797 | 870 | ## Save to the log file |
| 798 | 871 | iraceResults$allConfigurations <- allConfigurations |
| 799 | 872 | irace_save_logfile(iraceResults, scenario) |
| 800 | 873 | |
| 874 | # Consistency checks | |
| 875 | irace.assert(nrow(iraceResults$experimentLog) == experimentsUsedSoFar) | |
| 876 | # With elitist=0 we may re-run the same configuration on the same (instance,seed) pair | |
| 877 | # FIXME: This assert is failing when sampleInstances=FALSE because we are re-executing again an instance that was executed at the start of the race. | |
| 878 | if (FALSE && scenario$elitist) | |
| 879 | irace.assert(sum(!is.na(iraceResults$experiments)) == experimentsUsedSoFar) | |
| 880 | ||
| 801 | 881 | if (remainingBudget <= 0) { |
| 802 | 882 | catInfo("Stopped because budget is exhausted") |
| 803 | iraceResults$state$completed$flag = TRUE | |
| 804 | iraceResults$state$completed$msg = "Budget exhausted" | |
| 805 | irace_save_logfile(iraceResults, scenario) | |
| 806 | return (eliteConfigurations) | |
| 883 | return(irace_finish(iraceResults, scenario, reason = "Budget exhausted")) | |
| 807 | 884 | } |
| 808 | 885 | if (scenario$maxTime > 0 && timeUsed >= scenario$maxTime) { |
| 809 | 886 | catInfo("Stopped because time budget is exhausted") |
| 810 | iraceResults$state$completed$flag = TRUE | |
| 811 | iraceResults$state$completed$msg = "Time budget exhausted" | |
| 812 | irace_save_logfile(iraceResults, scenario) | |
| 813 | return (eliteConfigurations) | |
| 887 | return(irace_finish(iraceResults, scenario, reason = "Time budget exhausted")) | |
| 814 | 888 | } |
| 815 | 889 | |
| 816 | 890 | if (indexIteration > nbIterations) { |
| 820 | 894 | if (debugLevel >= 1) { |
| 821 | 895 | catInfo("Limit of iterations reached", verbose = FALSE) |
| 822 | 896 | } |
| 823 | iraceResults$state$completed$flag = TRUE | |
| 824 | iraceResults$state$completed$msg = "Limit of iterations reached" | |
| 825 | irace_save_logfile(iraceResults, scenario) | |
| 826 | return (eliteConfigurations) | |
| 897 | return(irace_finish(iraceResults, scenario, reason = "Limit of iterations reached")) | |
| 827 | 898 | } |
| 828 | 899 | } |
| 829 | 900 | # Compute the current budget (nb of experiments for this iteration), |
| 867 | 938 | } else { |
| 868 | 939 | catInfo("Stopped because ", |
| 869 | 940 | "there is not enough budget to enforce the value of nbConfigurations.") |
| 870 | iraceResults$state$completed$flag = TRUE | |
| 871 | iraceResults$state$completed$msg = "Not enough budget to enforce the value of nbConfigurations" | |
| 872 | irace_save_logfile(iraceResults, scenario) | |
| 873 | return (eliteConfigurations) | |
| 941 | return(irace_finish(iraceResults, scenario, reason = "Not enough budget to enforce the value of nbConfigurations")) | |
| 874 | 942 | } |
| 875 | 943 | } |
| 876 | 944 | |
| 879 | 947 | catInfo("Stopped because there is not enough budget left to race more than ", |
| 880 | 948 | "the minimum (", minSurvival,")\n", |
| 881 | 949 | "# You may either increase the budget or set 'minNbSurvival' to a lower value") |
| 882 | iraceResults$state$completed$flag = TRUE | |
| 883 | iraceResults$state$completed$msg = "Not enough budget to race more than the minimum configurations" | |
| 884 | irace_save_logfile(iraceResults, scenario) | |
| 885 | return (eliteConfigurations) | |
| 950 | return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race more than the minimum configurations")) | |
| 886 | 951 | } |
| 887 | 952 | |
| 888 | 953 | |
| 900 | 965 | catInfo("Stopped because ", |
| 901 | 966 | "there is not enough budget left to race newly sampled configurations") |
| 902 | 967 | #(number of elites + 1) * (mu + min(5, indexIteration)) > remainingBudget" |
| 903 | iraceResults$state$completed$flag = TRUE | |
| 904 | iraceResults$state$completed$msg = "Not enough budget left to race newly sampled configurations" | |
| 905 | irace_save_logfile(iraceResults, scenario) | |
| 906 | return (eliteConfigurations) | |
| 968 | return(irace_finish(iraceResults, scenario, reason = "Not enough budget left to race newly sampled configurations")) | |
| 907 | 969 | } |
| 908 | 970 | |
| 909 | 971 | if (scenario$elitist) { |
| 913 | 975 | + nrow(eliteConfigurations) * min(scenario$elitistNewInstances, max(scenario$mu, scenario$firstTest)) |
| 914 | 976 | > currentBudget) { |
| 915 | 977 | catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") |
| 916 | iraceResults$state$completed$flag = TRUE | |
| 917 | iraceResults$state$completed$msg = "Not enough budget to race all configurations up to the first test (or mu)" | |
| 918 | irace_save_logfile(iraceResults, scenario) | |
| 919 | return (eliteConfigurations) | |
| 978 | return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) | |
| 920 | 979 | } |
| 921 | 980 | } else if (nbConfigurations * max(scenario$mu, scenario$firstTest) |
| 922 | 981 | > currentBudget) { |
| 923 | 982 | catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") |
| 924 | iraceResults$state$completed$flag = TRUE | |
| 925 | iraceResults$state$completed$msg = "Not enough budget to race all configurations up to the first test (or mu)" | |
| 926 | irace_save_logfile(iraceResults, scenario) | |
| 927 | return (eliteConfigurations) | |
| 983 | return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) | |
| 928 | 984 | } |
| 929 | 985 | |
| 930 | 986 | catInfo("Iteration ", indexIteration, " of ", nbIterations, "\n", |
| 958 | 1014 | allConfigurations <- rbind(allConfigurations, newConfigurations) |
| 959 | 1015 | rownames(allConfigurations) <- allConfigurations$.ID. |
| 960 | 1016 | raceConfigurations <- allConfigurations[allConfigurations$.ID. %!in% rejectedIDs, , drop = FALSE] |
| 961 | } else if (nbNewConfigurations < 0) { | |
| 1017 | } else if (nbNewConfigurations <= 0) { | |
| 962 | 1018 | # We let the user know that not all configurations will be used. |
| 963 | 1019 | if (nbUserConfigurations > nbConfigurations) { |
| 964 | 1020 | catInfo("Only ", nbConfigurations, |
| 981 | 1037 | nbNewConfigurations <- nbConfigurations - nrow(eliteConfigurations) |
| 982 | 1038 | |
| 983 | 1039 | # Update the model based on elites configurations |
| 984 | if (debugLevel >= 1) { irace.note("Update model\n") } | |
| 1040 | if (debugLevel >= 1) irace.note("Update model\n") | |
| 985 | 1041 | model <- updateModel(parameters, eliteConfigurations, model, indexIteration, |
| 986 | 1042 | nbIterations, nbNewConfigurations, scenario) |
| 987 | if (debugLevel >= 2) { printModel (model) } | |
| 1043 | if (debugLevel >= 2) printModel (model) | |
| 1044 | if (debugLevel >= 1) | |
| 1045 | irace.note("Sample ", nbNewConfigurations, " configurations from model\n") | |
| 988 | 1046 | |
| 989 | if (debugLevel >= 1) { | |
| 990 | irace.note("Sample ", nbNewConfigurations, " configurations from model\n") | |
| 991 | } | |
| 992 | 1047 | newConfigurations <- sampleModel(parameters, eliteConfigurations, |
| 993 | 1048 | model, nbNewConfigurations, |
| 994 | 1049 | digits = scenario$digits, |
| 1115 | 1170 | # FIXME: Since we only actually keep the alive ones, we don't need |
| 1116 | 1171 | # to carry around rejected ones in raceResults$configurations. This |
| 1117 | 1172 | # would reduce overhead. |
| 1118 | eliteConfigurations <- extractElites(raceResults$configurations, | |
| 1173 | eliteConfigurations <- extractElites(scenario, parameters, | |
| 1174 | raceResults$configurations, | |
| 1119 | 1175 | min(raceResults$nbAlive, minSurvival)) |
| 1120 | 1176 | irace.note("Elite configurations (first number is the configuration ID;", |
| 1121 | 1177 | " listed from best to worst according to the ", |
| 1122 | 1178 | test.type.order.str(scenario$testType), "):\n") |
| 1123 | configurations.print(eliteConfigurations, metadata = debugLevel >= 1) | |
| 1179 | if (!quiet) configurations.print(eliteConfigurations, metadata = debugLevel >= 1) | |
| 1124 | 1180 | iraceResults$iterationElites <- c(iraceResults$iterationElites, eliteConfigurations$.ID.[1]) |
| 1125 | 1181 | iraceResults$allElites[[indexIteration]] <- eliteConfigurations$.ID. |
| 1126 | 1182 | |
| 1145 | 1201 | irace.print.memUsed() |
| 1146 | 1202 | } |
| 1147 | 1203 | } |
| 1148 | # This code is actually never executed because we return above. | |
| 1149 | # Leslie: adding this just in case | |
| 1150 | iraceResults$state$completed$flag = TRUE | |
| 1151 | irace_save_logfile(iraceResults, scenario) | |
| 1152 | return (eliteConfigurations) | |
| 1153 | } | |
| 1204 | irace.internal.error("This code is actually never executed because we return above") | |
| 1205 | } | |
| 0 | # Convert an irace.Rdata file into input format supported by the parameter | |
| 1 | # importance analysis tool PyImp | |
| 2 | # (https://github.com/automl/ParameterImportance). | |
| 3 | # | |
| 4 | # PyImp is developed by the AutoML-Freiburgh group (https://github.com/automl) | |
| 5 | ||
| 6 | single_quote <- function(s) paste0("'",s,"'") | |
| 7 | double_quote <- function(s) paste0('"',s,'"') | |
| 8 | ||
| 9 | .irace2pyimp_header <- | |
| 10 | '#------------------------------------------------------------------------------ | |
| 11 | # irace2pyimp converts an irace.Rdata file into input format supported by the | |
| 12 | # parameter importance analysis tool PyImp (https://github.com/automl/ParameterImportance) | |
| 13 | # | |
| 14 | # irace2pyimp was developed by Nguyen Dang <nttd@st-andrews.ac.uk> | |
| 15 | # | |
| 16 | # PyImp is developed by the AutoML-Freiburg-Hannover group (https://www.automl.org/) | |
| 17 | # | |
| 18 | # To see usage: irace2pyimp --help | |
| 19 | # See examples in `$IRACE_HOME/inst/examples/irace2pyimp/acotsp/run.sh` | |
| 20 | # and in `$IRACE_HOME/inst/examples/irace2pyimp/002-TemplateDesign/run.sh` | |
| 21 | # | |
| 22 | # irace2pyimp can also be called as an R function instead. To see usage of the | |
| 23 | # function, run in R console: | |
| 24 | # > library(irace) | |
| 25 | # > ?irace2pyimp | |
| 26 | # | |
| 27 | # The generated files include: | |
| 28 | # * `params.pcs` : a text file containing the parameter space definition. | |
| 29 | # * `runhistory.json` : a JSON file containing the list of algorithm configurations | |
| 30 | # evaluated during the tuning and the performance data obtained. | |
| 31 | # * `traj_aclib2.json` : a JSON file containing the best configurations after | |
| 32 | # each iteration of irace. | |
| 33 | # * `scenario.txt` : a text file containing the definition of the tuning scenario. | |
| 34 | # * `instances.txt` : a text file containing the list of instances. | |
| 35 | # * `features.csv` : a text file containing instance features. | |
| 36 | #------------------------------------------------------------------------------ | |
| 37 | \n' | |
| 38 | ||
| 39 | # list of command line arguments | |
| 40 | .irace2pyimp_args <- read.table(header=TRUE, stringsAsFactors = FALSE, row.names = "name", text=' | |
| 41 | name type short long default description | |
| 42 | .help x "-h" "--help" NA "Show this help." | |
| 43 | normalise s "-n" "--normalise" "none" "Normalise the cost metric values into the range of [0,1] before converting to PyImp format. Values: {none(default), instance, feature}, which correspond to no-normalisation, normalisation based on instance, and normalisation based on features, respectively." | |
| 44 | outDir s "-o" "--out-dir" "./pyimp-input" "Directory where all generated data are stored. Default: ./pyimp-input" | |
| 45 | instanceFeatureFile s "-f" "--instance-feature-file" NA " A .csv file containing instance features (one line per instance, sorted in the same order as the list of instances input to irace). The first line contains feature names. By default, instance index is used as a feature." | |
| 46 | filterConditions s "-c" "--filter-conditions" "NA" "Only extract configurations that satisfies the given conditions. The conditions are in R expression format. Default: no filter" | |
| 47 | defaultConfigurationID i "-i" "--default-configuration-index" 1 "Index of default configuration (starting from 1), used by ablation analysis. Default value: 1" | |
| 48 | iraceRdataFile s "-r" "--irace-data-file" NA "The log .Rdata file generated by irace" | |
| 49 | ignoreUnsupported b "-ignore" "--ignore-unsupported" 0 "Forbidden configurations and repairConfiguration are not supported by the script. Set this flag to 1 to ignore them and proceed with your own risk. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis\'s path. Values: {0,1}" | |
| 50 | ') | |
| 51 | ||
| 52 | load_irace_rdata <- function(filename) | |
| 53 | { | |
| 54 | load(filename) | |
| 55 | ||
| 56 | if ('iraceResults' %!in% ls()) { | |
| 57 | irace.error(filename, " is not a valid .Rdata file generated by irace") | |
| 58 | } | |
| 59 | ||
| 60 | # Some fields are not present in the old version of irace, so we assign them as new version's default values | |
| 61 | if ('capping' %!in% names(iraceResults$scenario)) | |
| 62 | iraceResults$scenario$capping <- FALSE | |
| 63 | ||
| 64 | return(iraceResults) | |
| 65 | } | |
| 66 | ||
| 67 | filter_data <- function(rdata, conditions, defaultConfigurationID) | |
| 68 | { | |
| 69 | # - remove all configurations not satistfying conditions in: | |
| 70 | # + rdata$allConfigurations | |
| 71 | # + rdata$iterationElites | |
| 72 | # + rdata$experiments & rdata$experimentLog | |
| 73 | # - also re-assign default configuration if necessary | |
| 74 | ||
| 75 | conditions <- parse(text=conditions) | |
| 76 | ||
| 77 | # update allConfigurations | |
| 78 | allConfigurations <- rdata$allConfigurations | |
| 79 | allConfigurations <- allConfigurations[eval(conditions, allConfigurations), , drop = FALSE] | |
| 80 | allConfigurations <- allConfigurations[order(allConfigurations$.ID.), , drop = FALSE] | |
| 81 | ||
| 82 | # update defaultConfigurationID | |
| 83 | if (defaultConfigurationID %!in% allConfigurations$.ID.) { | |
| 84 | irace.warning("default configuration '", defaultConfigurationID, | |
| 85 | "' does not satisfy condition '", as.character(conditions), | |
| 86 | "', thus setting configuration ", allConfigurations$.ID.[1], | |
| 87 | " as default configuration instead.") | |
| 88 | defaultConfigurationID <- allConfigurations$.ID.[1] | |
| 89 | } | |
| 90 | ||
| 91 | # update iterationElites | |
| 92 | removed <- setdiff(rdata$iterationElites, allConfigurations$.ID.) | |
| 93 | if (length(removed) > 0) | |
| 94 | irace.warning("elite configuration(s) ", paste0(removed, collapse=", "), | |
| 95 | " did not satisfy condition ", as.character(conditions), | |
| 96 | ' and were eliminated') | |
| 97 | ||
| 98 | iterationElites <- setdiff(rdata$iterationElites, removed) | |
| 99 | ||
| 100 | # update experiments | |
| 101 | experiments <- rdata$experiments[, allConfigurations$.ID., drop = FALSE] | |
| 102 | ||
| 103 | # update experimentLog | |
| 104 | experimentLog <- rdata$experimentLog[rdata$experimentLog[,'configuration'] %in% allConfigurations$.ID., ,drop = FALSE] | |
| 105 | ||
| 106 | rdata$allConfigurations <- allConfigurations | |
| 107 | rdata$experiments <- experiments | |
| 108 | rdata$experimentLog <- experimentLog | |
| 109 | rdata$iterationElites <- iterationElites | |
| 110 | rdata$defaultConfigurationID <- defaultConfigurationID | |
| 111 | return(rdata) | |
| 112 | } | |
| 113 | ||
| 114 | generate_feature_file <- function(file = "features.csv", instances, instanceFeatureFile) | |
| 115 | { | |
| 116 | # Generate features.csv | |
| 117 | cat(sep="", "Generating feature file '", file, "' ...\n") | |
| 118 | ||
| 119 | #--- features.csv ----- | |
| 120 | tFeatures <- data.frame(instance=instances) | |
| 121 | ||
| 122 | # if no instance features are provided, instance index will be the only | |
| 123 | # feature. | |
| 124 | if (is.null.or.na(instanceFeatureFile)) { | |
| 125 | tFeatures$id <- 1:length(instances) | |
| 126 | ||
| 127 | } else {# otherwise, add features | |
| 128 | if (!file.exists(instanceFeatureFile)) { | |
| 129 | irace.error("Instance feature file '", instanceFeatureFile,"' does not exists.") | |
| 130 | } | |
| 131 | features <- read.table(instanceFeatureFile, header=TRUE) | |
| 132 | if (nrow(features) != length(instances)) { | |
| 133 | irace.error("The number of instances (", nrow(features), ") in '", instanceFeatureFile, | |
| 134 | "' does not match the instance list given to irace (", length(instances),")") | |
| 135 | } | |
| 136 | tFeatures <- cbind(tFeatures, features) | |
| 137 | } | |
| 138 | ||
| 139 | # write to features.csv | |
| 140 | write.csv(tFeatures,file = file, row.names=FALSE, quote=FALSE) | |
| 141 | return (tFeatures) | |
| 142 | } | |
| 143 | ||
| 144 | remove_fixed_parameters <- function(rdata) | |
| 145 | { | |
| 146 | # remove fixed parameters, as we don't need them in the analyses | |
| 147 | parameters <- rdata$parameters | |
| 148 | ||
| 149 | # update parameters | |
| 150 | fixedParams <- names(which(parameters$isFixed)) | |
| 151 | if (length(fixedParams) > 0) { | |
| 152 | cat(sep="", "Removing fixed parameters: ", paste0(fixedParams, collapse=", "), " ...\n") | |
| 153 | varParams <- names(which(!parameters$isFixed)) | |
| 154 | # Go over 'names','types','switches','hierarchy', etc and keep only | |
| 155 | # varParams (except for elements that contain a single value as | |
| 156 | # $nbParameters) | |
| 157 | parameters <- lapply(parameters, function(x) { | |
| 158 | if (length(x) == 1) return(x) | |
| 159 | else return(x[varParams]) | |
| 160 | }) | |
| 161 | parameters$nbParameters <- parameters$nbParameters - length(fixedParams) | |
| 162 | parameters <- rdata$parameters <- parameters | |
| 163 | rdata$allConfigurations <- rdata$allConfigurations[, varParams] | |
| 164 | } | |
| 165 | return (rdata) | |
| 166 | } | |
| 167 | ||
| 168 | generate_pcs_file <- function(file, parameters, defaultConfiguration) | |
| 169 | { | |
| 170 | #---- generate param file in SMAC's format ------ | |
| 171 | cat(sep="", "Generating parameter definition file '", file, "' ...\n") | |
| 172 | ||
| 173 | # param types, ranges, and default value | |
| 174 | paramlines <- c() | |
| 175 | for (param in parameters$names) { | |
| 176 | type <- parameters$types[[param]] | |
| 177 | domain <- parameters$domain[[param]] | |
| 178 | isLogTransform <- FALSE | |
| 179 | if ('transform' %in% names(parameters) && parameters$transform[[param]]=='log') | |
| 180 | isLogTransform <- TRUE | |
| 181 | ||
| 182 | val <- defaultConfiguration[[param]] | |
| 183 | if (is.na(val)) val <- domain[1] | |
| 184 | ||
| 185 | if (type %in% c('r','i')) { | |
| 186 | line <- paste0(param, ' [', domain[1], ',',domain[2],'] [', val, ']', | |
| 187 | if (type == 'i') 'i' else '', | |
| 188 | if (isLogTransform) 'l' else '') | |
| 189 | } else { | |
| 190 | line <- paste0(param, ' {', paste(domain, collapse = ','), '} [', val, ']') | |
| 191 | } | |
| 192 | paramlines <- c(paramlines, line) | |
| 193 | } | |
| 194 | ||
| 195 | # param conditions | |
| 196 | conditionlines <- c() | |
| 197 | for (param in parameters$names) { | |
| 198 | # FIXME: This will not work if we byte-compile the conditions | |
| 199 | irace.assert(!is.bytecode(parameters$conditions[[param]])) | |
| 200 | conditions <- as.character(parameters$conditions[[param]]) | |
| 201 | # Convert to SMAC's param format. | |
| 202 | if (conditions != "TRUE" && conditions != "FALSE") { | |
| 203 | conditions <- gsub("%in%", "in", conditions) # remove % | |
| 204 | conditions <- gsub("\"","", conditions) # remove double quote | |
| 205 | conditions <- gsub("\'","", conditions) # remove single quote | |
| 206 | ||
| 207 | # replace c() by {} | |
| 208 | conditions <- gsub("c[[:space:]]*\\(([^()]+)\\)", "{\\1}", conditions) | |
| 209 | ||
| 210 | # remove "(" and ")" at the beginning and end of the condition | |
| 211 | conditions <- trimws(conditions) | |
| 212 | conditionlines <- c(conditionlines, paste0(param," | ", conditions)) | |
| 213 | } | |
| 214 | } | |
| 215 | lines <- c(paramlines, '\n', '#Conditions:', conditionlines) | |
| 216 | # FIXME, TODO: forbidden conditions? | |
| 217 | # nguyen: SMAC's parameter space only allows particular forbidden constraints in the format of {parameter_name_1=value_1, ..., parameter_name_N=value_N}, while irace allows more general forbidden constraints. I'm not sure how to deal with it at the moment... | |
| 218 | ||
| 219 | writeLines(lines, file) | |
| 220 | } | |
| 221 | ||
| 222 | process_experiments <- function(rdata, normalise, tFeatures) | |
| 223 | { | |
| 224 | #---- reformat rdata$experiments and add extra information into it ---# | |
| 225 | cat(sep="", "Preprocessing experiment data...\n") | |
| 226 | experiments <- rdata$experiments | |
| 227 | ||
| 228 | # instance_id and seed | |
| 229 | tInstanceSeeds <- rdata$state$.irace$instancesList | |
| 230 | colnames(tInstanceSeeds)[colnames(tInstanceSeeds)=='instance'] <- 'instance_id' | |
| 231 | tInstanceSeeds <- cbind(instance_seed_id = 1:nrow(tInstanceSeeds), tInstanceSeeds) | |
| 232 | ||
| 233 | # add instance names into tInstanceSeeds | |
| 234 | instances <- rdata$scenario$instances | |
| 235 | tInstances <- data.frame(instance_id = 1:length(instances), instance=instances) | |
| 236 | tInstanceSeeds <- merge(tInstanceSeeds, tInstances, by = c('instance_id')) | |
| 237 | ||
| 238 | # melt experiments | |
| 239 | experiments <- cbind(instance_seed_id = 1:nrow(experiments), experiments) | |
| 240 | experiments <- na.omit(reshape(as.data.frame(experiments), direction='long', | |
| 241 | idvar='instance_seed_id', | |
| 242 | times=colnames(experiments)[2:ncol(experiments)], | |
| 243 | timevar='candidate_id', | |
| 244 | varying=list(colnames(experiments)[2:ncol(experiments)]), | |
| 245 | v.names='cost')) | |
| 246 | rownames(experiments) <- NULL | |
| 247 | ||
| 248 | # add instance names and instance_seed_id into experiments | |
| 249 | experiments <- merge(experiments, tInstanceSeeds, by='instance_seed_id') | |
| 250 | ||
| 251 | # normalise cost values if necessary | |
| 252 | if (normalise %!in% c('none','instance','feature')) | |
| 253 | irace.error("Invalid normalisation method '", normalise,"'") | |
| 254 | ||
| 255 | if (normalise != 'none' && rdata$scenario$capping) { | |
| 256 | irace.warning("normalisation for capped data is not tested and can be buggy.") | |
| 257 | } | |
| 258 | if (normalise != 'none') { | |
| 259 | t1 <- experiments # experiments table with minCost and maxCost for normalisation | |
| 260 | ||
| 261 | # get minCost and maxCost over each instance | |
| 262 | if (normalise == 'instance'){ | |
| 263 | tMinCost <- aggregate(cost~instance_id, t1, FUN=min) | |
| 264 | colnames(tMinCost)[2] <- 'minCost' | |
| 265 | tMaxCost <- aggregate(cost~instance_id, t1, FUN=max) | |
| 266 | colnames(tMaxCost)[2] <- 'maxCost' | |
| 267 | t1 <- merge(t1, tMinCost, by='instance_id') | |
| 268 | t1 <- merge(t1, tMaxCost, by='instance_id') | |
| 269 | } else { # get minCost and maxCost over each feature vector (i.e., over many instances with the same feature values) | |
| 270 | t2 <- merge(t1, tFeatures, by='instance') | |
| 271 | featureNames <- colnames(tFeatures)[-c(1)] | |
| 272 | featureVals <- as.list(t2[, featureNames, drop=FALSE]) | |
| 273 | tMinCost <- aggregate(t2$cost, by=featureVals, min) | |
| 274 | colnames(tMinCost)[colnames(tMinCost)=='x'] <- 'minCost' | |
| 275 | tMaxCost <- aggregate(t2$cost, by=featureVals, max) | |
| 276 | colnames(tMaxCost)[colnames(tMaxCost)=='x'] <- 'maxCost' | |
| 277 | t2 <- merge(t2, tMinCost, by=featureNames) | |
| 278 | t2 <- merge(t2, tMaxCost, by=featureNames) | |
| 279 | t1 <- t2[, names(t1) %!in% featureNames] | |
| 280 | } | |
| 281 | ||
| 282 | # calculate normalised cost | |
| 283 | # TODO: allow user-defined normalisation method | |
| 284 | t1$cost <- (t1$cost - t1$minCost) / (t1$maxCost - t1$minCost) | |
| 285 | ||
| 286 | # remove rows with normalisedCost == Inf | |
| 287 | t1 <- t1[is.finite(t1$cost), ] | |
| 288 | experiments <- t1[, names(t1) %!in% c('minCost','maxCost')] | |
| 289 | } | |
| 290 | ||
| 291 | # add extra information on iteration index & bound (when irace's capping is enabled) | |
| 292 | tLog <- data.frame(rdata$experimentLog) | |
| 293 | colnames(tLog)[colnames(tLog)=='instance'] <- 'instance_seed_id' | |
| 294 | colnames(tLog)[colnames(tLog)=='configuration'] <- 'candidate_id' | |
| 295 | tLog$candidate_id <- as.integer(tLog$candidate_id) | |
| 296 | log_columns <- c('instance_seed_id','candidate_id','iteration') | |
| 297 | if (('capping' %in% names(rdata$scenario)) && rdata$scenario$capping) { | |
| 298 | log_columns <- c(log_columns, 'time', 'bound') | |
| 299 | } | |
| 300 | experiments <- merge(experiments, tLog[, log_columns], | |
| 301 | by=c('instance_seed_id','candidate_id')) | |
| 302 | return(experiments) | |
| 303 | } | |
| 304 | ||
| 305 | generate_runhistory <- function(file = "runhistory.json", rdata, experiments) | |
| 306 | { | |
| 307 | #------- generate runhistory.json ------# | |
| 308 | cat(sep="", "Generating runhistory JSON file '", file, "' ...\n") | |
| 309 | ||
| 310 | # TODO: If we ever use the json R package for something else, we could use it | |
| 311 | # here as well, however, in runhistory.json, values are written with/without | |
| 312 | # quotes depending on data types. Therefore, we need to make sure they are | |
| 313 | # written correctly. | |
| 314 | ||
| 315 | # Configurations: The first columns is .ID., and the last column is | |
| 316 | # .PARENT. We are extracting parameter names only. | |
| 317 | ls_param_names <- setdiff(colnames(rdata$allConfigurations), c(".ID.", ".PARENT.")) | |
| 318 | output <- paste0('{', double_quote('configs'), ': {\n') | |
| 319 | for (row_id in seq_len(nrow(rdata$allConfigurations))) { | |
| 320 | cand <- rdata$allConfigurations[row_id, ] | |
| 321 | cand_id <- cand$.ID. | |
| 322 | ls_params_vals <- sapply(ls_param_names, function(pname) { | |
| 323 | val <- cand[[pname]] | |
| 324 | if (is.na(val)) return("") | |
| 325 | if (rdata$parameters$types[[pname]] == 'c') | |
| 326 | val <- double_quote(val) | |
| 327 | return(paste0(double_quote(pname), ': ', val)) | |
| 328 | }) | |
| 329 | if (row_id > 1) | |
| 330 | output <- paste0(output, ',\n') | |
| 331 | ls_params_vals <- ls_params_vals[ls_params_vals != ""] | |
| 332 | output <- paste0(output, double_quote(cand_id),': {', paste0(ls_params_vals, collapse = ', '),'}') | |
| 333 | } | |
| 334 | output <- paste0(output, '}') | |
| 335 | ||
| 336 | # data | |
| 337 | output <- paste0(output, ',\n', double_quote('data'), ': [\n') | |
| 338 | experiments <- experiments[order(experiments$candidate_id,experiments$instance_seed_id),] # order rows in experiments, so that runhistory.json is always the same across different runs | |
| 339 | for (row_id in seq_len(nrow(experiments))) { | |
| 340 | row <- experiments[row_id,] | |
| 341 | cost <- row$cost | |
| 342 | ||
| 343 | if (('capping' %in% rdata$scenario) && rdata$scenario$capping) { | |
| 344 | time <- row$time | |
| 345 | if (row$time >= row$bound){ | |
| 346 | if (row$bound < (rdata$scenario$boundMax - 0.01)) | |
| 347 | status <- 'StatusType.CAPPED' | |
| 348 | else | |
| 349 | status <- 'StatusType.TIMEOUT' | |
| 350 | } else | |
| 351 | status <- 'StatusType.SUCCESS' | |
| 352 | ||
| 353 | } else { | |
| 354 | time <- 0.9 | |
| 355 | status <- 'StatusType.SUCCESS' | |
| 356 | } | |
| 357 | ||
| 358 | if (row_id > 1) | |
| 359 | output <- paste0(output, ',\n') | |
| 360 | #rs = [[conf_id, inst, seed], [cost, time, status, {}]] | |
| 361 | s1 <- paste(row$candidate_id, double_quote(row$instance), row$seed, sep=', ') | |
| 362 | s2 <- paste(as.character(cost), as.character(time), paste('{',double_quote('__enum__'),': ', double_quote(status),'}'), '{}', sep=', ') | |
| 363 | output <- paste0(output,'[[', s1, '], [', s2, ']]') | |
| 364 | } | |
| 365 | output <- paste0(output,'\n]}\n') | |
| 366 | cat(output, file = file) | |
| 367 | } | |
| 368 | ||
| 369 | generate_trajectory <- function(file = 'traj_aclib2.json', rdata, experiments) | |
| 370 | { | |
| 371 | #-------------------- traj_aclib2.json ------------------------------# | |
| 372 | cat(sep="", "Generating '", file, "' ...\n") | |
| 373 | ||
| 374 | # Configurations: The first columns is .ID., and the last column is | |
| 375 | # .PARENT. We are extracting parameter names only. | |
| 376 | ls_param_names <- setdiff(colnames(rdata$allConfigurations), c(".ID.", ".PARENT.")) | |
| 377 | output <- "" | |
| 378 | for (iterationId in seq_along(rdata$iterationElites)) { | |
| 379 | confId <- rdata$iterationElites[iterationId] | |
| 380 | # if this elite was eliminated due to filterConditions, ignore it | |
| 381 | t1 <- experiments[(experiments$candidate_id == confId) & (experiments$iteration <= iterationId), ] | |
| 382 | ||
| 383 | # cpu time | |
| 384 | cpu_time <- paste0(double_quote('cputime'),': ', confId) | |
| 385 | total_cpu_time <- paste0(double_quote('total_cpu_time'),': null') | |
| 386 | wallclock_time <- paste0(double_quote('wallclock_time'),': ', confId) | |
| 387 | # evaluations | |
| 388 | evaluations <- paste0(double_quote('evaluations'),': ', nrow(t1)) | |
| 389 | # cost | |
| 390 | cost <- paste0(double_quote('cost'),': ', mean(t1$cost)) | |
| 391 | # configuration string | |
| 392 | cand <- rdata$allConfigurations[confId, ] | |
| 393 | ls_params_vals <- sapply(ls_param_names, function(pname) { | |
| 394 | val <- cand[[pname]] | |
| 395 | if (is.na(val)) return("") | |
| 396 | if (rdata$parameters$types[[pname]] == 'c') | |
| 397 | val <- single_quote(val) | |
| 398 | # FIMXE: if type is 'c', we do single_quote(val) two times? | |
| 399 | # nguyen: One single_quote is indeed sufficient! I've updated the next line accordingly :) | |
| 400 | return(double_quote(paste0(pname, '=', val))) | |
| 401 | }) | |
| 402 | ls_params_vals <- ls_params_vals[ls_params_vals != ""] | |
| 403 | configuration_string <- paste0(double_quote("incumbent"),': [', | |
| 404 | paste0(ls_params_vals, collapse = ', '),']') | |
| 405 | # combine everything | |
| 406 | output <- paste0(output, '{', | |
| 407 | paste(cpu_time, evaluations, cost, configuration_string, total_cpu_time, wallclock_time, sep=', '),'}\n') | |
| 408 | } | |
| 409 | cat(output, file = file) | |
| 410 | } | |
| 411 | ||
| 412 | generate_scenario_file <- function(file = "scenario.txt", rdata) | |
| 413 | { | |
| 414 | #------ create scenario file ------- | |
| 415 | cat(sep="", "Generating scenario file '", file, "' ...\n") | |
| 416 | lss <- list() | |
| 417 | lss[['algo']] <- rdata$scenario$targetRunner | |
| 418 | lss[['execDir']] <- './' | |
| 419 | if (rdata$scenario$deterministic) | |
| 420 | lss[['deterministic']] <- 'true' | |
| 421 | else | |
| 422 | lss[['deterministic']] <- 'false' | |
| 423 | if (rdata$scenario$capping) { | |
| 424 | lss[['run_obj']] <- 'runtime' | |
| 425 | lss[['cutoff_time']] <- rdata$scenario$boundMax | |
| 426 | lss[['overall_obj']] <- paste0('par',rdata$scenario$boundPar) | |
| 427 | } else { | |
| 428 | lss[['run_obj']] <- 'quality' | |
| 429 | lss[['cutoff_time']] <- 1 | |
| 430 | } | |
| 431 | lss[['tunerTimeout']] <- 999999 | |
| 432 | lss[['overall_obj']] <- 'mean' | |
| 433 | lss[['paramfile']] <- 'params.pcs' | |
| 434 | lss[['instance_file']] <- 'instances.txt' | |
| 435 | lss[['feature_file']] <- 'features.csv' | |
| 436 | writeLines(paste0(names(lss), "=", lss), file) | |
| 437 | } | |
| 438 | ||
| 439 | ||
| 440 | #' Command-line interface to irace2pyimp | |
| 441 | #' | |
| 442 | #' This is a command-line interface for calling the [`irace2pyimp`] function, | |
| 443 | #' which converts an `irace.Rdata` file into the input format supported by the | |
| 444 | #' parameter importance analysis tool `PyImp` | |
| 445 | #' (https://github.com/automl/ParameterImportance). \cr The best way to use | |
| 446 | #' this command line interface is to run the script `irace-to-pyimp`. \cr To | |
| 447 | #' see usage of the script, run: irace-to-pyimp --help | |
| 448 | #' | |
| 449 | #' @param argv (`character()`) \cr Command-line arguments. | |
| 450 | #' | |
| 451 | #' @return None. | |
| 452 | #' | |
| 453 | #' @examples | |
| 454 | #' | |
| 455 | #' irace2pyimp_cmdline("--help") | |
| 456 | #' | |
| 457 | #' @seealso | |
| 458 | #' [`irace2pyimp`] | |
| 459 | #' | |
| 460 | #' @author Nguyen Dang and Manuel López-Ibáñez | |
| 461 | #' @export | |
| 462 | #' @md | |
| 463 | irace2pyimp_cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) | |
| 464 | { | |
| 465 | parser <- CommandArgsParser$new(argv = argv, argsdef = .irace2pyimp_args) | |
| 466 | ||
| 467 | if (!is.null(parser$readArg(short = "-h", long = "--help"))) { | |
| 468 | cat(.irace2pyimp_header) | |
| 469 | cmdline_usage(.irace2pyimp_args) | |
| 470 | return(invisible(NULL)) | |
| 471 | } | |
| 472 | argvalues <- lapply(rownames(.irace2pyimp_args), parser$readCmdLineParameter) | |
| 473 | names(argv) <- rownames(.irace2pyimp_args) | |
| 474 | ||
| 475 | irace2pyimp(file = argvalues$iraceRdataFile, normalise = argvalues$normalise, | |
| 476 | outdir = argvalues$outDir, | |
| 477 | instanceFeatureFile = argvalues$instanceFeatureFile, | |
| 478 | filterConditions = argvalues$filterConditions, | |
| 479 | defaultConfigurationID = argvalues$defaultConfigurationID, | |
| 480 | ignoreUnsupported=argvalues$ignoreUnsupported) | |
| 481 | } | |
| 482 | ||
| 483 | #' Convert an `irace.Rdata` file into the format supported by PyImp | |
| 484 | #' | |
| 485 | #' This function converts an `irace.Rdata` file generated by irace into the | |
| 486 | #' input format supported by the parameter importance analysis tool `PyImp` | |
| 487 | #' (https://github.com/automl/ParameterImportance). | |
| 488 | ||
| 489 | #' | |
| 490 | #' The generated files include: | |
| 491 | #' | |
| 492 | #' * `params.pcs` : a text file containing the parameter space definition. | |
| 493 | #' * `runhistory.json` : a JSON file containing the list of algorithm configurations evaluated during the tuning and the performance data obtained. | |
| 494 | #' * `traj_aclib2.json` : a JSON file containing the best configurations after each iteration of irace. The last configuration will be used as the target configuration in ablation analysis. | |
| 495 | #' * `scenario.txt` : a text file containing the definition of the tuning scenario. | |
| 496 | #' * `instances.txt` : a text file containing the list of instances. | |
| 497 | #' * `features.csv` : a text file containing instance features. If no instance features are provided, the index of each instance will be used as a feature. | |
| 498 | #' | |
| 499 | #' @param file (`character(1)`) \cr Filename of the `.Rdata` file generated by | |
| 500 | #' irace after a tuning run is finished. | |
| 501 | #' | |
| 502 | #' @param normalise (`none` | `instance` | `feature`) \cr Normalise the cost metric values into `[0, 1]` range before converting to PyImp format. Possible values are:\cr | |
| 503 | #' * `none` (default): no normalisation. \cr | |
| 504 | #' * `instance` : normalisation is done per instance. \cr | |
| 505 | #' * `feature` : normalisation is based on features, i.e., instances with the same feature-vector values are grouped together and the normalised cost is calculated per group. | |
| 506 | #' | |
| 507 | #' @param outdir (`character(1)`) \cr Directory where all generated files are stored. | |
| 508 | #' @param instanceFeatureFile (`character(1)`) \cr A `.csv` file containing instance features (one line per instance, | |
| 509 | #' sorted in the same order as the list of instances input to irace). The first line contains feature names. | |
| 510 | #' @param filterConditions Only extract data that satisfies the given conditions. The conditions are in R expression format. | |
| 511 | #' @param defaultConfigurationID Index of default configuration (starting from 1), used by ablation analysis. | |
| 512 | #' @param ignoreUnsupported Forbidden configurations and repairConfiguration are not supported by the script. Set ignoreUnsupported=1 to ignore them and proceed with your own risk. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis's path. | |
| 513 | #' | |
| 514 | #' @examples | |
| 515 | #' \dontrun{ | |
| 516 | #' irace2pyimp(file='irace.Rdata', outdir='pyimp-run') | |
| 517 | #' irace2pyimp(file='irace.Rdata', normalise='feature', | |
| 518 | #' instanceFeatureFile='feature.csv', filterConditions="algorithm!='mas'") | |
| 519 | #' } | |
| 520 | #' cat("See more examples in '", | |
| 521 | #' file.path(system.file(package="irace"), "examples/irace2pyimp/acotsp/run.sh"), | |
| 522 | #' "' and in '", | |
| 523 | #' file.path(system.file(package="irace"), "examples/irace2pyimp/002-TemplateDesign/run.sh"), | |
| 524 | #' "'\n") | |
| 525 | #' | |
| 526 | #' @author Nguyen Dang and Manuel López-Ibáñez | |
| 527 | #' @export | |
| 528 | #' @md | |
| 529 | irace2pyimp <- function(file='./irace.Rdata', normalise='none', outdir='./pyimp-input/', | |
| 530 | instanceFeatureFile=NA, filterConditions=NA, defaultConfigurationID=1, ignoreUnsupported=0) | |
| 531 | { | |
| 532 | if (str_sub(outdir, start=-1) != "/") outdir <- paste0(outdir, "/") | |
| 533 | # create output dir if it doesn't exist | |
| 534 | if (!file.exists(outdir)) dir.create(outdir, recursive = TRUE) | |
| 535 | ||
| 536 | rdata <- load_irace_rdata(file) | |
| 537 | ||
| 538 | # check forbidden constraint and repairConfiguration | |
| 539 | if ((ignoreUnsupported==0) && (!is.null.or.empty(rdata$scenario$forbiddenExps) || !is.null.or.empty(rdata$scenario$repairConfiguration))){ | |
| 540 | irace.error("Forbidden constraints and/or repairConfiguration are not yet supported. To ignore them and proceed with your own risk, please set '--ignore-unsupported 1' (command line), or 'ignoreUnsupported=1' (R console) while calling irace2pyimp. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis's path.") | |
| 541 | } | |
| 542 | # print warning about forbidden constraint and repairConfiguration before proceeding | |
| 543 | if (!is.null.or.empty(rdata$scenario$forbiddenExps) || !is.null.or.empty(rdata$scenario$repairConfiguration)){ | |
| 544 | irace.warning("Forbidden constraints and repairConfiguration are ignored. Proceeding with your own risk...") | |
| 545 | } | |
| 546 | ||
| 547 | ops <- options(scipen=999) # Disable scientific notation | |
| 548 | on.exit(options(ops)) | |
| 549 | ||
| 550 | # filter data | |
| 551 | if (!is.na(filterConditions) && trimws(filterConditions) != '') { | |
| 552 | rdata <- filter_data(rdata = rdata, conditions = filterConditions, | |
| 553 | defaultConfigurationID = defaultConfigurationID) | |
| 554 | defaultConfigurationID <- rdata$defaultConfigurationID | |
| 555 | } | |
| 556 | ||
| 557 | # Generate instances.txt and features.csv | |
| 558 | #--- instances.txt ---- | |
| 559 | cat(sep="", "Generating instance list file '", paste0(outdir, 'instances.txt'), "' ...\n") | |
| 560 | writeLines(rdata$scenario$instances, paste0(outdir, 'instances.txt')) | |
| 561 | ||
| 562 | tFeatures <- generate_feature_file(file = paste0(outdir, 'features.csv'), | |
| 563 | instances = rdata$scenario$instances, | |
| 564 | instanceFeatureFile = instanceFeatureFile) | |
| 565 | ||
| 566 | # remove fixed parameters | |
| 567 | rdata <- remove_fixed_parameters(rdata) | |
| 568 | ||
| 569 | # generate params.pcs | |
| 570 | generate_pcs_file(file = paste0(outdir, 'params.pcs'), | |
| 571 | parameters = rdata$parameters, | |
| 572 | defaultConfiguration = rdata$allConfigurations[defaultConfigurationID,]) | |
| 573 | ||
| 574 | # generate runhistory.json and traj_aclib2.json | |
| 575 | experiments <- process_experiments(rdata, normalise, tFeatures) | |
| 576 | ||
| 577 | generate_runhistory(file = paste0(outdir, 'runhistory.json'), rdata = rdata, experiments = experiments) | |
| 578 | generate_trajectory(file = paste0(outdir, 'traj_aclib2.json'), rdata = rdata, experiments = experiments) | |
| 579 | ||
| 580 | # generate scenario.txt | |
| 581 | generate_scenario_file(file = paste0(outdir, "scenario.txt"), rdata = rdata) | |
| 582 | } | |
| 583 |
| 52 | 52 | # Copyright (C) 2003 Mauro Birattari |
| 53 | 53 | #------------------------------------------------------------------------------ |
| 54 | 54 | ' |
| 55 | cat.irace.license <- function() | |
| 55 | cat_irace_license <- function() | |
| 56 | 56 | { |
| 57 | 57 | cat(sub("__VERSION__", irace.version, irace.license, fixed=TRUE)) |
| 58 | 58 | } |
| 59 | 59 | |
| 60 | cmdline_usage <- function(cmdline_args) | |
| 61 | { | |
| 62 | for (i in seq_len(nrow(cmdline_args))) { | |
| 63 | short <- cmdline_args[i,"short"] | |
| 64 | long <- cmdline_args[i,"long"] | |
| 65 | desc <- cmdline_args[i,"description"] | |
| 66 | if (desc == "" || (short == "" && long == "")) next | |
| 67 | cat(sep = "\n", strwrap(desc, width = 80, | |
| 68 | initial = sprintf("%2s %-20s ", short, long), | |
| 69 | exdent = 25)) | |
| 70 | } | |
| 71 | } | |
| 72 | ||
| 73 | #' irace.usage | |
| 74 | #' | |
| 75 | #' \code{irace.usage} This function prints all command-line options of \pkg{irace}, | |
| 76 | #' with the corresponding switches and a short description. | |
| 77 | #' | |
| 78 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste | |
| 79 | #' @export | |
| 80 | irace.usage <- function () | |
| 81 | { | |
| 82 | cat.irace.license() | |
| 83 | cat ("# installed at: ", system.file(package="irace"), "\n", sep = "") | |
| 84 | cmdline_usage(.irace.params.def) | |
| 85 | } | |
| 86 | ||
| 87 | #' irace.main | |
| 88 | #' | |
| 89 | #' \code{irace.main} is a higher-level interface to invoke \code{\link{irace}}. | |
| 90 | #' | |
| 60 | #' Higher-level interface to launch irace. | |
| 61 | #' | |
| 91 | 62 | #' @template arg_scenario |
| 92 | 63 | #' |
| 93 | #' @param output.width (\code{integer(1)}) The width that must be used for the screen | |
| 94 | #' output. | |
| 95 | #' | |
| 96 | #' @details The function \code{irace.main} checks the correctness of the | |
| 97 | #' scenario, prints it, reads the parameter space from | |
| 98 | #' \code{scenario$parameterFile}, invokes \code{\link{irace}} and | |
| 99 | #' prints its results in various formatted ways. If you want a | |
| 100 | #' lower-level interface, please see function \code{\link{irace}}. | |
| 64 | #' @param output.width (\code{integer(1)}) The width used for the screen | |
| 65 | #' output. | |
| 66 | #' | |
| 67 | #' @details This function checks the correctness of the scenario, reads the | |
| 68 | #' parameter space from \code{scenario$parameterFile}, invokes [irace()], | |
| 69 | #' prints its results in various formatted ways, (optionally) calls | |
| 70 | #' [psRace()] and, finally, evaluates the best configurations on the test | |
| 71 | #' instances (if provided). If you want a lower-level interface that just | |
| 72 | #' runs irace, please see function [irace()]. | |
| 101 | 73 | #' |
| 102 | 74 | #' @templateVar return_invisible TRUE |
| 103 | 75 | #' @template return_irace |
| 104 | 76 | #' @seealso |
| 105 | #' \code{\link{irace.cmdline}} a higher-level command-line interface to | |
| 106 | #' \code{irace.main}. | |
| 107 | #' \code{\link{readScenario}} to read the scenario setup from a file. | |
| 108 | #' \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. | |
| 77 | #' [irace.cmdline()] a higher-level command-line interface to | |
| 78 | #' [irace()] | |
| 79 | #' [readScenario()] to read the scenario setup from a file. | |
| 80 | #' [defaultScenario()] to provide a default scenario for \pkg{irace}. | |
| 109 | 81 | #' |
| 110 | 82 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 111 | #' @export | |
| 112 | irace.main <- function(scenario = defaultScenario(), output.width = 9999L) | |
| 113 | { | |
| 114 | op <- options(width = output.width) # Do not wrap the output. | |
| 115 | on.exit(options(op), add = TRUE) | |
| 116 | ||
| 117 | scenario <- checkScenario (scenario) | |
| 118 | debug.level <- scenario$debugLevel | |
| 119 | ||
| 120 | if (debug.level >= 1) { | |
| 121 | op.debug <- options(warning.length = 8170, | |
| 122 | error = if (interactive()) utils::recover | |
| 123 | else irace.dump.frames) | |
| 124 | on.exit(options(op.debug), add = TRUE) | |
| 125 | printScenario (scenario) | |
| 126 | } | |
| 127 | ||
| 128 | # Read parameters definition | |
| 129 | parameters <- readParameters (file = scenario$parameterFile, | |
| 130 | digits = scenario$digits, | |
| 131 | debugLevel = debug.level) | |
| 132 | ||
| 133 | if (debug.level >= 2) { irace.note("Parameters have been read\n") } | |
| 134 | ||
| 135 | eliteConfigurations <- irace (scenario = scenario, parameters = parameters) | |
| 136 | ||
| 137 | cat("# Best configurations (first number is the configuration ID;", | |
| 138 | " listed from best to worst according to the ", | |
| 139 | test.type.order.str(scenario$testType), "):\n", sep = "") | |
| 140 | configurations.print(eliteConfigurations) | |
| 141 | ||
| 142 | cat("# Best configurations as commandlines (first number is the configuration ID; same order as above):\n") | |
| 143 | configurations.print.command (eliteConfigurations, parameters) | |
| 144 | ||
| 145 | if (scenario$postselection > 0) | |
| 146 | psRace(iraceLogFile=scenario$logFile, postselection=scenario$postselection, elites=TRUE) | |
| 147 | ||
| 148 | if (length(eliteConfigurations) > 0 && | |
| 149 | (scenario$testIterationElites != 0 || scenario$testNbElites != 0)) | |
| 150 | testing.main(logFile = scenario$logFile) | |
| 151 | ||
| 152 | invisible(eliteConfigurations) | |
| 153 | } | |
| 154 | ||
| 155 | #' testing.main | |
| 156 | #' | |
| 157 | #' \code{testing.main} executes the testing of the target | |
| 158 | #' algorithm configurations found on an \pkg{irace} execution. | |
| 159 | #' | |
| 160 | #' @param logFile Path to the \code{.Rdata} file produced by \pkg{irace}. | |
| 161 | #' | |
| 162 | #' @return Boolean. TRUE if the testing ended successfully otherwise, returns | |
| 163 | #' FALSE. | |
| 164 | #' | |
| 165 | #' @details The function \code{testing.main} loads the \code{logFile} and | |
| 166 | #' obtains the needed configurations according to the specified test. Use the | |
| 167 | #' \code{scenario$testNbElites} to test N final elite configurations or use | |
| 168 | #' \code{scenario$testIterationElites} to test the best configuration of each | |
| 169 | #' iteration. A test instance set must be provided through | |
| 170 | #' \code{scenario$testInstancesDir} and \code{testInstancesFile}. | |
| 171 | #' | |
| 172 | #' @seealso | |
| 173 | #' \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. | |
| 83 | #' @concept running | |
| 84 | #' @export | |
| 85 | irace.main <- function(scenario, output.width = 9999L) | |
| 86 | irace_common(scenario = scenario, simple=FALSE, output.width = output.width) | |
| 87 | ||
| 88 | #' Test configurations given in `.Rdata` file | |
| 89 | #' | |
| 90 | #' `testing_fromlog` executes the testing of the target algorithm configurations | |
| 91 | #' found by an \pkg{irace} execution. | |
| 92 | #' | |
| 93 | #' @param logFile Path to the `.Rdata` file produced by \pkg{irace}. | |
| 94 | #' | |
| 95 | #' @param testNbElites Number of (final) elite configurations to test. Overrides | |
| 96 | #' the value found in `logFile`. | |
| 97 | #' | |
| 98 | #' @param testIterationElites (`logical(1)`) If `FALSE`, only the final | |
| 99 | #' `testNbElites` configurations are tested; otherwise, also test the best | |
| 100 | #' configurations of each iteration. Overrides the value found in `logFile`. | |
| 101 | #' | |
| 102 | #' @param testInstancesDir Directory where testing instances are located, either absolute or relative to current directory. | |
| 103 | #' | |
| 104 | #' @param testInstancesFile File containing a list of test instances and optionally additional parameters for them. | |
| 105 | #' | |
| 106 | #' @param testInstances Character vector of the instances to be used in the `targetRunner` when executing the testing. | |
| 107 | #' | |
| 108 | #' @return Boolean. `TRUE` if the testing ended successfully otherwise, `FALSE`. | |
| 109 | #' | |
| 110 | #' @details The function `testing_fromlog` loads the `logFile` and obtains the | |
| 111 | #' testing setup and configurations to be tested. Within the `logFile`, the | |
| 112 | #' variable `scenario$testNbElites` specifies how many final elite | |
| 113 | #' configurations to test and `scenario$testIterationElites` indicates | |
| 114 | #' whether test the best configuration of each iteration. The values may be | |
| 115 | #' overridden by setting the corresponding arguments in this function. The | |
| 116 | #' set of testing instances must appear in `scenario[["testInstances"]]`. | |
| 117 | #' | |
| 118 | #' @seealso [defaultScenario()] to provide a default scenario for \pkg{irace}. | |
| 119 | #' [testing_fromfile()] provides a different interface for testing. | |
| 174 | 120 | #' |
| 175 | 121 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 176 | #' @export | |
| 177 | testing.main <- function(logFile) | |
| 122 | #' @concept running | |
| 123 | #' @export | |
| 124 | testing_fromlog <- function(logFile, testNbElites, testIterationElites, | |
| 125 | testInstancesDir, testInstancesFile, testInstances) | |
| 178 | 126 | { |
| 179 | 127 | if (is.null.or.empty(logFile)) { |
| 180 | 128 | irace.note("No logFile provided to perform the testing of configurations. Skipping testing.\n") |
| 181 | 129 | return(FALSE) |
| 182 | 130 | } |
| 131 | iraceResults <- read_logfile(logFile) | |
| 132 | scenario <- iraceResults[["scenario"]] | |
| 133 | parameters <- iraceResults[["parameters"]] | |
| 134 | instances_changed <- FALSE | |
| 183 | 135 | |
| 184 | file.check(logFile, readable = TRUE, text = "irace log file") | |
| 185 | ||
| 186 | load (logFile) | |
| 187 | scenario <- iraceResults$scenario | |
| 188 | parameters <- iraceResults$parameters | |
| 189 | ||
| 190 | if (is.null.or.empty(scenario$testInstances)) { | |
| 136 | if (!missing(testNbElites)) | |
| 137 | scenario[["testNbElites"]] <- testNbElites | |
| 138 | if (!missing(testIterationElites)) | |
| 139 | scenario$testIterationElites <- testIterationElites | |
| 140 | ||
| 141 | if (!missing(testInstances)) | |
| 142 | scenario[["testInstances"]] <- testInstances | |
| 143 | ||
| 144 | if (!missing(testInstancesDir)) { | |
| 145 | scenario$testInstancesDir <- testInstancesDir | |
| 146 | instances_changed <- TRUE | |
| 147 | } | |
| 148 | if (!missing(testInstancesFile)) { | |
| 149 | scenario$testInstancesFile <- testInstancesFile | |
| 150 | instances_changed <- TRUE | |
| 151 | } | |
| 152 | ||
| 153 | cat("\n\n# Testing of elite configurations:", scenario$testNbElites, | |
| 154 | "\n# Testing iteration configurations:", scenario$testIterationElites,"\n") | |
| 155 | if (scenario$testNbElites <= 0) | |
| 191 | 156 | return (FALSE) |
| 157 | ||
| 158 | # If they are already setup, don't change them. | |
| 159 | if (instances_changed || is.null.or.empty(scenario[["testInstances"]])) { | |
| 160 | scenario <- setup_test_instances(scenario) | |
| 161 | if (is.null.or.empty(scenario[["testInstances"]])) { | |
| 162 | irace.note("No test instances, skip testing\n") | |
| 163 | return(FALSE) | |
| 164 | } | |
| 192 | 165 | } |
| 193 | 166 | |
| 194 | # Get configurations | |
| 195 | testing.id <- c() | |
| 167 | # Get configurations that will be tested | |
| 196 | 168 | if (scenario$testIterationElites) |
| 197 | testing.id <- c(testing.id, iraceResults$iterationElites) | |
| 198 | if (scenario$testNbElites > 0) { | |
| 169 | testing_id <- sapply(iraceResults$allElites, function(x) | |
| 170 | x[1:min(length(x), scenario$testNbElites)]) | |
| 171 | else { | |
| 199 | 172 | tmp <- iraceResults$allElites[[length(iraceResults$allElites)]] |
| 200 | testing.id <- c(testing.id, tmp[1:min(length(tmp), scenario$testNbElites)]) | |
| 201 | } | |
| 202 | testing.id <- unique(testing.id) | |
| 203 | configurations <- iraceResults$allConfigurations[testing.id, , drop=FALSE] | |
| 204 | ||
| 205 | cat(" \n\n") | |
| 206 | irace.note ("Testing configurations (in no particular order): ", paste(testing.id, collapse=" "), "\n") | |
| 207 | configurations.print(configurations) | |
| 208 | cat("# Testing of elite configurations:", scenario$testNbElites, | |
| 209 | "\n# Testing iteration configurations:", scenario$testIterationElites,"\n") | |
| 210 | ||
| 211 | iraceResults$testing <- testConfigurations(configurations, scenario, parameters) | |
| 212 | ||
| 213 | # FIXME : We should print the seeds also. As an additional column? | |
| 214 | irace.note ("Testing results (column number is configuration ID in no particular order):\n") | |
| 215 | print(iraceResults$testing$experiments) | |
| 216 | irace_save_logfile (iraceResults, scenario) | |
| 217 | ||
| 218 | irace.note ("Finished testing\n") | |
| 173 | testing_id <- tmp[1:min(length(tmp), scenario$testNbElites)] | |
| 174 | } | |
| 175 | testing_id <- unique.default(unlist(testing_id)) | |
| 176 | configurations <- iraceResults$allConfigurations[testing_id, , drop=FALSE] | |
| 177 | ||
| 178 | irace.note ("Testing configurations (in no particular order): ", paste(testing_id, collapse=" "), "\n") | |
| 179 | testing_common(configurations, scenario, parameters, iraceResults) | |
| 219 | 180 | return(TRUE) |
| 220 | 181 | } |
| 221 | 182 | |
| 222 | testing.cmdline <- function(filename, scenario) | |
| 183 | #' Test configurations given an explicit table of configurations and a scenario file | |
| 184 | #' | |
| 185 | #' Executes the testing of an explicit list of configurations given in | |
| 186 | #' `filename` (same format as in [readConfigurationsFile()]). A `logFile` is | |
| 187 | #' created unless disabled in `scenario`. This may overwrite an existing one! | |
| 188 | #' | |
| 189 | #' @param filename Path to a file containing configurations: one configuration | |
| 190 | #' per line, one parameter per column, parameter names in header. | |
| 191 | #' | |
| 192 | #' @template arg_scenario | |
| 193 | #' | |
| 194 | #' @return iraceResults | |
| 195 | #' | |
| 196 | #' @seealso [testing_fromlog()] provides a different interface for testing. | |
| 197 | #' | |
| 198 | #' @author Manuel López-Ibáñez | |
| 199 | #' @concept running | |
| 200 | #' @export | |
| 201 | testing_fromfile <- function(filename, scenario) | |
| 223 | 202 | { |
| 224 | 203 | irace.note ("Checking scenario\n") |
| 225 | 204 | scenario <- checkScenario(scenario) |
| 226 | printScenario(scenario) | |
| 205 | if (!scenario$quiet) printScenario(scenario) | |
| 227 | 206 | |
| 228 | 207 | irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") |
| 229 | 208 | parameters <- readParameters (file = scenario$parameterFile, |
| 230 | 209 | digits = scenario$digits) |
| 231 | allConfigurations <- readConfigurationsFile (filename, parameters) | |
| 232 | allConfigurations <- cbind(.ID. = 1:nrow(allConfigurations), | |
| 233 | allConfigurations, | |
| 210 | configurations <- readConfigurationsFile (filename, parameters) | |
| 211 | configurations <- cbind(.ID. = 1:nrow(configurations), | |
| 212 | configurations, | |
| 234 | 213 | .PARENT. = NA) |
| 235 | rownames(allConfigurations) <- allConfigurations$.ID. | |
| 236 | num <- nrow(allConfigurations) | |
| 237 | allConfigurations <- checkForbidden(allConfigurations, scenario$forbiddenExps) | |
| 238 | if (nrow(allConfigurations) < num) { | |
| 239 | cat("# Warning: some of the configurations in the configurations file were forbidden", | |
| 240 | "and, thus, discarded\n") | |
| 241 | } | |
| 242 | ||
| 214 | rownames(configurations) <- configurations$.ID. | |
| 215 | num <- nrow(configurations) | |
| 216 | configurations <- checkForbidden(configurations, scenario$forbiddenExps) | |
| 217 | if (nrow(configurations) < num) { | |
| 218 | irace.warning("Some of the configurations in the configurations file were forbidden", | |
| 219 | "and, thus, discarded") | |
| 220 | } | |
| 243 | 221 | # To save the logs |
| 244 | 222 | iraceResults <- list(scenario = scenario, |
| 245 | 223 | irace.version = irace.version, |
| 246 | 224 | parameters = parameters, |
| 247 | allConfigurations = allConfigurations) | |
| 225 | allConfigurations = configurations) | |
| 248 | 226 | |
| 249 | 227 | irace.note ("Testing configurations (in the order given as input): \n") |
| 250 | configurations.print(allConfigurations) | |
| 251 | iraceResults$testing <- testConfigurations(allConfigurations, scenario, parameters) | |
| 252 | ||
| 228 | iraceResults <- testing_common(configurations, scenario, parameters, iraceResults) | |
| 229 | return(iraceResults) | |
| 230 | } | |
| 231 | ||
| 232 | testing_common <- function(configurations, scenario, parameters, iraceResults) | |
| 233 | { | |
| 234 | verbose <- !scenario$quiet | |
| 235 | if (verbose) configurations.print(configurations) | |
| 236 | iraceResults$testing <- testConfigurations(configurations, scenario, parameters) | |
| 237 | irace_save_logfile (iraceResults, scenario) | |
| 253 | 238 | # FIXME : We should print the seeds also. As an additional column? |
| 254 | 239 | irace.note ("Testing results (column number is configuration ID in no particular order):\n") |
| 255 | print(iraceResults$testing$experiments) | |
| 256 | irace_save_logfile(iraceResults, scenario) | |
| 240 | if (verbose) print(iraceResults$testing$experiments) | |
| 257 | 241 | irace.note ("Finished testing\n") |
| 258 | 242 | return(iraceResults) |
| 259 | 243 | } |
| 260 | 244 | |
| 261 | 245 | #' Test that the given irace scenario can be run. |
| 262 | 246 | #' |
| 263 | #' @description \code{checkIraceScenario} tests that the given irace scenario | |
| 264 | #' can be run by checking the scenario settings provided and trying to run | |
| 265 | #' the target-algorithm. | |
| 247 | #' Test that the given irace scenario can be run by checking the scenario | |
| 248 | #' settings provided and trying to run the target-algorithm. | |
| 266 | 249 | #' |
| 267 | 250 | #' @template arg_scenario |
| 268 | 251 | #' @template arg_parameters |
| 269 | 252 | #' |
| 270 | #' @return returns \code{TRUE} if succesful and gives an error and returns | |
| 253 | #' @return returns \code{TRUE} if successful and gives an error and returns | |
| 271 | 254 | #' \code{FALSE} otherwise. |
| 272 | 255 | #' |
| 273 | #' @details Provide the \code{parameters} argument only if the parameter list | |
| 274 | #' should not be obtained from the parameter file given by the scenario. If | |
| 275 | #' the parameter list is provided it will not be checked. This function will | |
| 256 | #' @details If the `parameters` argument is missing, then the parameters | |
| 257 | #' will be read from the file `parameterFile` given by `scenario`. If | |
| 258 | #' `parameters` is provided, then `parameterFile` will not be read. This function will | |
| 276 | 259 | #' try to execute the target-algorithm. |
| 277 | 260 | #' |
| 278 | 261 | #' @seealso |
| 285 | 268 | #' |
| 286 | 269 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 287 | 270 | #' @export |
| 288 | checkIraceScenario <- function(scenario, parameters = NULL) | |
| 271 | checkIraceScenario <- function(scenario, parameters) | |
| 289 | 272 | { |
| 290 | 273 | irace.note ("Checking scenario\n") |
| 291 | 274 | scenario$debugLevel <- 2 |
| 292 | 275 | scenario <- checkScenario(scenario) |
| 293 | printScenario(scenario) | |
| 276 | if (!scenario$quiet) printScenario(scenario) | |
| 294 | 277 | |
| 295 | if (is.null(parameters)) { | |
| 278 | if (missing(parameters)) { | |
| 296 | 279 | irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") |
| 297 | 280 | parameters <- readParameters (file = scenario$parameterFile, |
| 298 | 281 | digits = scenario$digits, |
| 299 | 282 | debugLevel = 2) |
| 300 | 283 | } else if (!is.null.or.empty(scenario$parameterFile)) { |
| 301 | cat("# Parameters provided by user.\n", | |
| 302 | "# Parameter file '", scenario$parameterFile, "' will be ignored\n", sep = "") | |
| 303 | } | |
| 304 | ||
| 305 | irace.note("Checking target execution.\n") | |
| 284 | if (!scenario$quiet) | |
| 285 | cat("# checkIraceScenario(): 'parameters' provided by user. ", | |
| 286 | "Parameter file '", scenario$parameterFile, "' will be ignored\n", sep = "") | |
| 287 | } | |
| 288 | checkParameters(parameters) | |
| 289 | irace.note("Checking target runner.\n") | |
| 306 | 290 | if (checkTargetFiles(scenario = scenario, parameters = parameters)) { |
| 307 | irace.note("Check succesful.\n") | |
| 291 | irace.note("Check successful.\n") | |
| 308 | 292 | return(TRUE) |
| 309 | 293 | } else { |
| 310 | 294 | irace.error("Check unsuccessful.\n") |
| 312 | 296 | } |
| 313 | 297 | } |
| 314 | 298 | |
| 315 | ||
| 316 | #' irace.cmdline | |
| 317 | #' | |
| 318 | #' \code{irace.cmdline} starts \pkg{irace} using the parameters | |
| 319 | #' of the command line used to invoke R. | |
| 299 | init <- function() | |
| 300 | { | |
| 301 | irace.note("Initializing working directory...\n") | |
| 302 | libPath <- system.file(package = "irace") | |
| 303 | tmplFiles <- list.files(file.path(libPath, "templates")) | |
| 304 | for (file in tmplFiles) { | |
| 305 | if (grepl(".tmpl", file) && (file != "target-evaluator.tmpl")) { | |
| 306 | newFile <- gsub(".tmpl", "", file) | |
| 307 | if ((file == "target-runner.tmpl") && .Platform$OS.type == 'windows') { | |
| 308 | file.copy(file.path(libPath, "templates", "windows", "target-runner.bat"), file.path(getwd(), "target-runner.bat"), overwrite = FALSE) | |
| 309 | } else { | |
| 310 | file.copy(file.path(libPath, "templates", file), file.path(getwd(), newFile), overwrite = FALSE) | |
| 311 | } | |
| 312 | } | |
| 313 | } | |
| 314 | } | |
| 315 | ||
| 316 | ||
| 317 | #' Launch `irace` with command-line options. | |
| 318 | #' | |
| 319 | #' Calls [irace.main()] using command-line options, maybe parsed from the | |
| 320 | #' command line used to invoke R. | |
| 320 | 321 | #' |
| 321 | 322 | #' @param argv (\code{character()}) \cr The arguments |
| 322 | 323 | #' provided on the R command line as a character vector, e.g., |
| 327 | 328 | #' @details The function reads the parameters given on the command line |
| 328 | 329 | #' used to invoke R, finds the name of the scenario file, |
| 329 | 330 | #' initializes the scenario from the file (with the function |
| 330 | #' \code{\link{readScenario}}) and possibly from parameters passed on | |
| 331 | #' \code{\link{readScenario}}) and possibly from parameters passed in | |
| 331 | 332 | #' the command line. It finally starts \pkg{irace} by calling |
| 332 | 333 | #' \code{\link{irace.main}}. |
| 333 | 334 | #' |
| 335 | #' List of command-line options: | |
| 336 | #' ```{r echo=FALSE,comment=NA} | |
| 337 | #' cmdline_usage(.irace.params.def) | |
| 338 | #' ``` | |
| 339 | #' | |
| 334 | 340 | #' @templateVar return_invisible TRUE |
| 335 | 341 | #' @template return_irace |
| 336 | 342 | #' |
| 337 | 343 | #' @seealso |
| 338 | #' \code{\link{irace.main}} to start \pkg{irace} with a given scenario. | |
| 339 | #' | |
| 344 | #' [irace.main()] to start \pkg{irace} with a given scenario. | |
| 345 | #' @examples | |
| 346 | #' irace.cmdline("--version") | |
| 340 | 347 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 341 | #' @export | |
| 342 | irace.cmdline <- function(argv = commandArgs (trailingOnly = TRUE)) | |
| 348 | #' @concept running | |
| 349 | #' @export | |
| 350 | irace.cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) | |
| 343 | 351 | { |
| 344 | 352 | parser <- CommandArgsParser$new(argv = argv, argsdef = .irace.params.def) |
| 353 | quiet <- !is.null(parser$readArg (short = "-q", long = "--quiet")) | |
| 354 | if (quiet) { | |
| 355 | op <- options(.irace.quiet = TRUE) | |
| 356 | on.exit(options(op)) | |
| 357 | } else { | |
| 358 | cat_irace_license() | |
| 359 | cat("# installed at: ", system.file(package="irace"), "\n", | |
| 360 | "# called with: ", paste(argv, collapse = " "), "\n", sep = "") | |
| 361 | } | |
| 345 | 362 | if (!is.null(parser$readArg (short = "-h", long = "--help"))) { |
| 346 | irace.usage() | |
| 363 | parser$cmdline_usage() | |
| 347 | 364 | return(invisible(NULL)) |
| 348 | 365 | } |
| 349 | ||
| 350 | if (!is.null(parser$readArg (short = "-v", long = "--version"))) { | |
| 351 | cat.irace.license() | |
| 352 | cat ("# installed at: ", system.file(package="irace"), "\n", sep = "") | |
| 366 | if (!is.null(parser$readArg(short = "-v", long = "--version"))) { | |
| 353 | 367 | print(citation(package="irace")) |
| 354 | 368 | return(invisible(NULL)) |
| 355 | 369 | } |
| 356 | cat.irace.license() | |
| 357 | cat ("# installed at: ", system.file(package="irace"), "\n", | |
| 358 | "# called with: ", paste(argv, collapse = " "), "\n", sep = "") | |
| 370 | ||
| 371 | if (!is.null(parser$readArg(short = "-i", long = "--init"))) { | |
| 372 | init() | |
| 373 | return(invisible(NULL)) | |
| 374 | } | |
| 359 | 375 | |
| 360 | 376 | # Read the scenario file and the command line |
| 361 | 377 | scenarioFile <- parser$readCmdLineParameter ("scenarioFile", default = "") |
| 365 | 381 | parser$readCmdLineParameter(paramName = param, |
| 366 | 382 | default = scenario[[param]]) |
| 367 | 383 | } |
| 384 | if (quiet) scenario$quiet <- TRUE | |
| 368 | 385 | |
| 369 | 386 | # Check scenario |
| 370 | 387 | if (!is.null(parser$readArg (short = "-c", long = "--check"))) { |
| 375 | 392 | # Only do testing |
| 376 | 393 | testFile <- parser$readArg (long = "--only-test") |
| 377 | 394 | if (!is.null(testFile)) { |
| 378 | return(invisible(testing.cmdline(testFile, scenario))) | |
| 395 | return(invisible(testing_fromfile(testFile, scenario))) | |
| 379 | 396 | } |
| 380 | 397 | |
| 381 | 398 | if (length(parser$argv) > 0) { |
| 382 | 399 | irace.error ("Unknown command-line options: ", paste(parser$argv, collapse = " ")) |
| 383 | 400 | } |
| 384 | ||
| 385 | irace.main(scenario) | |
| 386 | } | |
| 401 | ||
| 402 | irace_common(scenario = scenario, simple=FALSE) | |
| 403 | } | |
| 25 | 25 | irace.assert(type == "o") |
| 26 | 26 | value <- (nbValues - 1) / 2 |
| 27 | 27 | } |
| 28 | ||
| 28 | 29 | param <- list() |
| 29 | 30 | for (indexConfig in seq_len(nbConfigurations)) { |
| 30 | 31 | idCurrentConfig <- as.character(configurations[indexConfig, ".ID."]) |
| 152 | 153 | model[[param]][[id]] <- probVector / sum(probVector) |
| 153 | 154 | } else { |
| 154 | 155 | if (type == "i" || type == "r") { |
| 155 | value <- init.model.numeric(param, parameters) | |
| 156 | # We keep the value of the configuration as last known | |
| 157 | value[2] <- configurations[id, param] | |
| 156 | value <- c(init.model.numeric(param, parameters), | |
| 157 | # We keep the value of the configuration as last known | |
| 158 | configurations[id, param]) | |
| 158 | 159 | } else { |
| 159 | 160 | irace.assert(type == "o") |
| 160 | 161 | value <- (length(parameters$domain[[param]]) - 1) / 2 |
| 170 | 171 | } |
| 171 | 172 | |
| 172 | 173 | # Initialise model in case of numerical variables. |
| 173 | # it retuns an array size 2, first number indicates the | |
| 174 | # it retuns an array size 2, first number indicates the | |
| 174 | 175 | # standard deviation and second the last known value (initially NA) |
| 175 | 176 | init.model.numeric <- function(param, parameters) |
| 176 | 177 | { |
| 177 | lower <- paramLowerBound(param, parameters) | |
| 178 | upper <- paramUpperBound(param, parameters) | |
| 178 | # Dependent parameters define the standard deviation as | |
| 179 | # a portion of the size of the domain interval. In this case, | |
| 180 | # 0.5 indicates half of the interval, equivalent to | |
| 181 | # (domain[2] - domain[1]) * 0.5 | |
| 182 | if (parameters$isDependent[[param]]) { | |
| 183 | return(0.5) | |
| 184 | } | |
| 185 | ||
| 179 | 186 | transf <- parameters$transform[[param]] |
| 180 | 187 | if (transf == "log") { |
| 181 | lower <- 0 | |
| 182 | upper <- 1 | |
| 188 | domain <- c(0,1) | |
| 189 | } else { | |
| 190 | domain <- parameters$domain[[param]] | |
| 183 | 191 | } |
| 184 | value <- (upper - lower) / 2.0 | |
| 192 | value <- (domain[2] - domain[1]) / 2.0 | |
| 185 | 193 | irace.assert(is.finite(value)) |
| 186 | return(c(value, NA)) | |
| 194 | return(value) | |
| 187 | 195 | } |
| 0 | #' Plot of histogram of parameter values | |
| 1 | #' | |
| 2 | #' \code{parameterFrequency} plots the frequency of the parameters values in a | |
| 3 | #' set of target algorithm configurations. It generates plots showing the | |
| 4 | #' frequency of parameter values for each parameter, with \code{rows} * | |
| 5 | #' \code{cols} parameters being shown per plot. If a filename is provided the | |
| 6 | #' plots are saved in one or more files. | |
| 7 | #' | |
| 8 | #' @template arg_configurations | |
| 9 | #' @template arg_parameters | |
| 10 | #' @param rows Number of plots per column. | |
| 11 | #' @param cols Number of plots per row. | |
| 12 | #' @param filename Filename prefix to generate the plots. If \code{NULL} the plot | |
| 13 | #' displayed but not saved. | |
| 14 | #' @param pdf.width Width for the pdf file generated. | |
| 15 | #' @param col Color of the bar plot. | |
| 16 | #' | |
| 17 | #' @examples | |
| 18 | #' \donttest{ | |
| 19 | #' ## To use data obtained by irace | |
| 20 | #' | |
| 21 | #' # First, load the data produced by irace. | |
| 22 | #' irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") | |
| 23 | #' load(irace.logfile) | |
| 24 | #' attach(iraceResults) | |
| 25 | #' parameterFrequency(allConfigurations, parameters) | |
| 26 | #' } | |
| 27 | #' | |
| 28 | #' @seealso | |
| 29 | #' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. | |
| 30 | #' \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from | |
| 31 | #' a configurations file. | |
| 32 | #' | |
| 33 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 34 | #' @export | |
| 35 | # TODO: | |
| 36 | # * change slightly background of conditional parameters | |
| 37 | # | |
| 38 | # * accept a configuration (e.g, best one) and mark its values of the best | |
| 39 | # configuration somehow (bold?) | |
| 40 | # | |
| 41 | # * print number of NAs in numerical parameters within the plot "# NA = Total | |
| 42 | # (Percentage%)" | |
| 43 | # | |
| 44 | # * export this function and add an R manual page, perhaps with example. | |
| 45 | # | |
| 46 | # * how to resize the Window when filename == NULL to have a rows/cols aspect ratio. | |
| 47 | # | |
| 48 | # * use ggplot2 ? | |
| 49 | # | |
| 50 | # * add tests! | |
| 51 | #parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 52 | #best <- colnames(experiments)[which(apply(experiments, 2, function(x) { sum(!is.na(x))}) > 5)] | |
| 53 | #parameterFrequency(allConfigurations[best,], parameters) | |
| 54 | parameterFrequency <- function(configurations, parameters, | |
| 55 | rows = 4, cols = 3, | |
| 56 | filename = NULL, | |
| 57 | pdf.width = 12, | |
| 58 | col = "gray") | |
| 59 | { | |
| 60 | xlab <- "values" | |
| 61 | ylab.cat <- "Frequency" | |
| 62 | ylab.num <- "Probability density" | |
| 63 | ||
| 64 | def.par <- par(no.readonly = TRUE) # save default, for resetting... | |
| 65 | on.exit(par(def.par), add = TRUE) #- reset to default | |
| 66 | ||
| 67 | configurations <- removeConfigurationsMetaData(configurations) | |
| 68 | ||
| 69 | param.names <- as.character(parameters$names) | |
| 70 | nparams <- parameters$nbVariable | |
| 71 | nlines <- ceiling(nparams / cols) | |
| 72 | if (nlines < rows) rows <- nlines | |
| 73 | ||
| 74 | nplot <- 1 | |
| 75 | cplot <- 1 | |
| 76 | if (!is.null(filename)) { | |
| 77 | # Remove possible ".pdf" extension. | |
| 78 | filename <- sub(".pdf", "", filename, fixed = TRUE) | |
| 79 | pdf(file = paste0(filename, "-", cplot, ".pdf"), onefile = TRUE, width = pdf.width) | |
| 80 | on.exit(dev.off(), add = TRUE) | |
| 81 | } | |
| 82 | par(mfrow=c(rows,cols), mar=0.1 + c(4,3,3,1)) | |
| 83 | ||
| 84 | for (param.name in param.names) { | |
| 85 | ||
| 86 | if (parameters$isFixed[[param.name]]){ | |
| 87 | cat("Skipping fixed parameter:", param.name, "\n") | |
| 88 | next | |
| 89 | } else { | |
| 90 | cat("Plotting:", param.name, "\n") | |
| 91 | } | |
| 92 | ||
| 93 | if (nplot > rows * cols) { | |
| 94 | cat("Make new plot\n") | |
| 95 | cplot <- cplot + 1 | |
| 96 | if (!is.null(filename)) { | |
| 97 | dev.off() | |
| 98 | pdf(file = paste0(filename, "-", cplot, ".pdf"), onefile = TRUE, width = pdf.width) | |
| 99 | } else { | |
| 100 | dev.new() | |
| 101 | } | |
| 102 | par(mfrow=c(rows, cols)) | |
| 103 | nplot <- 1 | |
| 104 | } | |
| 105 | ||
| 106 | data <- configurations[, param.name] | |
| 107 | type <- parameters$types[[param.name]] | |
| 108 | domain <- parameters$domain[[param.name]] | |
| 109 | if (type %in% c("c", "o")) { | |
| 110 | data <- factor(data, domain) | |
| 111 | data <- addNA(data, ifany = TRUE) | |
| 112 | levels(data)[is.na(levels(data))] <- "<NA>" | |
| 113 | data <- table(data) | |
| 114 | barplot(data, main = param.name, xlab = xlab, ylab = ylab.cat, col = col) | |
| 115 | } else if (type %in% c("i", "r")) { | |
| 116 | data <- data[!is.na(data)] | |
| 117 | if (length(data) == 0) { | |
| 118 | cat("All values are NA for: ", param.name, ", skipping plot\n") | |
| 119 | next | |
| 120 | } else { | |
| 121 | hist(data, xlim = domain, prob = TRUE, | |
| 122 | main = param.name, xlab = xlab, ylab = ylab.num, col = col) | |
| 123 | if (length(data) > 1) { | |
| 124 | lines(density(data), col = "blue", lwd = 2) | |
| 125 | } else { | |
| 126 | abline(v = data[1], col = "blue", lwd = 2) | |
| 127 | } | |
| 128 | } | |
| 129 | } | |
| 130 | nplot <- nplot + 1 | |
| 131 | } | |
| 132 | } | |
| 133 | ||
| 134 | # Function parcoordlabel plots parallel coordinates for categorical and | |
| 135 | # numerical configurations. Idea from | |
| 136 | # http://stackoverflow.com/questions/23553210/add-ticks-to-parcoord-parallel-coordinates-plot | |
| 137 | parcoordlabel <- function (configurations, parameters, col = "green", lty = 1, | |
| 138 | lblcol="blue", title="Parameters parallel coordinates", ...) | |
| 139 | { | |
| 140 | replace.cat <- function(y, vals){ | |
| 141 | x <- rep(NA, length(y)) | |
| 142 | for(i in 1:length(vals)) | |
| 143 | x[y %in% vals[i]] <- i | |
| 144 | return(x) | |
| 145 | } | |
| 146 | ||
| 147 | replace.na <- function(x,r) { | |
| 148 | x <- unlist(x) | |
| 149 | x <- (x-r[1])/(r[2]-r[1]) | |
| 150 | x[is.na(x)] <- 1 | |
| 151 | return(x) | |
| 152 | } | |
| 153 | ||
| 154 | add.level <- function(x, bound, type){ | |
| 155 | if (type == "i" || type == "r"){ | |
| 156 | x <- c(x, x[length(x)] + (x[2]-x[1])) | |
| 157 | } else { | |
| 158 | x <- c(x, x[length(x)] + x[length(x)]/length(bound)) | |
| 159 | } | |
| 160 | return(x) | |
| 161 | } | |
| 162 | ||
| 163 | param.names <- colnames(configurations) | |
| 164 | bound.num <- parameters$domain | |
| 165 | # Categorical -> Numerical | |
| 166 | for (i in 1:ncol(configurations)) { | |
| 167 | pname <- param.names[i] | |
| 168 | if (parameters$types[[pname]] %in% c("c","o")) { | |
| 169 | configurations[,i]<- as.numeric(replace.cat(configurations[,i], bound.num[[pname]])) | |
| 170 | bound.num[[pname]] <- seq(1,length(bound.num[[pname]])) | |
| 171 | } | |
| 172 | } | |
| 173 | ||
| 174 | # Intervals | |
| 175 | pr <- lapply(bound.num[param.names], pretty) | |
| 176 | # add extra level for NA values | |
| 177 | for(param in param.names) | |
| 178 | pr[[param]] <- add.level( pr[[param]], bound.num[[param]], parameters$types[[param]]) | |
| 179 | # Total range | |
| 180 | rx <- lapply(pr, range, na.rm = TRUE) | |
| 181 | ||
| 182 | # Values x plot | |
| 183 | configurations <- mapply(replace.na, as.data.frame(configurations), rx) | |
| 184 | ||
| 185 | matplot(1:ncol(configurations), t(configurations), type = "l", col = col, | |
| 186 | lty = lty, xlab = "", ylab = "", ylim=c(0,1), axes = FALSE, main=title, ...) | |
| 187 | ||
| 188 | axis(1, at = 1:ncol(configurations), labels = colnames(configurations), las=2) | |
| 189 | ||
| 190 | for (i in 1:ncol(configurations)) { | |
| 191 | pnames <- param.names[i] | |
| 192 | lines(c(i, i), c(0, 1), col = "grey70") | |
| 193 | if(parameters$types[[param.names[i]]]=="c"){ | |
| 194 | labels <- c(parameters$domain[[param.names[i]]], "NA") | |
| 195 | }else{ | |
| 196 | labels <- pr[[pnames]] | |
| 197 | labels[length(labels)] <- "<NA>" | |
| 198 | } | |
| 199 | text(c(i, i), seq(0,1,length.out=length(labels)), labels = labels, xpd = NA, col=lblcol) | |
| 200 | } | |
| 201 | invisible() | |
| 202 | } | |
| 203 | ||
| 204 | #' parallelCoordinatesPlot | |
| 205 | #' | |
| 206 | #' \code{parallelCoordinatesPlot} plots a set of parameter configurations in | |
| 207 | #' parallel coordinates. | |
| 208 | #' | |
| 209 | #' @template arg_configurations | |
| 210 | #' @template arg_parameters | |
| 211 | #' @param param_names Parameters names that should be included. Default: parameters$names. | |
| 212 | #' @param hierarchy If \code{TRUE} conditional parameters will be displayed in a different | |
| 213 | #' plot. Default \code{TRUE}. | |
| 214 | #' @param filename Filename prefix to generate the plots. If \code{NULL} the plot | |
| 215 | #' displayed but not saved. | |
| 216 | #' @param pdf.width Width for the pdf file generated. | |
| 217 | #' @param mar Margin to use for the plot. See \code{\link{par}}. | |
| 218 | #' | |
| 219 | #' @return A set of parallel coordinates plots showing the parameters values. | |
| 220 | #' If a filename is provided this plots are saved in one or more files. | |
| 221 | #' | |
| 222 | #' @examples | |
| 223 | #' \donttest{ | |
| 224 | #' ## To use data obtained by irace | |
| 225 | #' # First, load the data produced by irace. | |
| 226 | #' irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") | |
| 227 | #' load(irace.logfile) | |
| 228 | #' attach(iraceResults) | |
| 229 | #' parallelCoordinatesPlot(allConfigurations, parameters, hierarchy = FALSE) | |
| 230 | #' } | |
| 231 | #' | |
| 232 | #' @seealso | |
| 233 | #' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. | |
| 234 | #' \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from | |
| 235 | #' a configurations file. | |
| 236 | #' | |
| 237 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 238 | #' @export | |
| 239 | # TODO: | |
| 240 | # * add color scheme | |
| 241 | # | |
| 242 | ## Filters and prepare data to plots parallel coordinate of configurations | |
| 243 | ## configurations: structure as in iraceResults | |
| 244 | ## parameters: structure as in iraceResults | |
| 245 | ## param_names: selection of parameters names that must be plotted | |
| 246 | ## hierarchy: separate plots according to dependencies of parameters (one level) | |
| 247 | ## mar: margin for the plots | |
| 248 | ## filename: prefix to save pdf files | |
| 249 | parallelCoordinatesPlot <- | |
| 250 | function(configurations, parameters, param_names=parameters$names, | |
| 251 | hierarchy = TRUE, filename = NULL, pdf.width = 14 , mar = c(8,1,4,1)) | |
| 252 | { | |
| 253 | getDependency <- function() { | |
| 254 | sdep <- list() | |
| 255 | independent <- c() | |
| 256 | for (param in param_names) { | |
| 257 | ## FIXME: If we ever byte-compile conditions, we need to use | |
| 258 | ## all.vars(.Internal(disassemble(condition))[[3]][[1]] | |
| 259 | constraint <- all.vars(parameters$conditions[[param]]) | |
| 260 | if (length(constraint) < 1) { | |
| 261 | independent <- unique(c(independent, param)) | |
| 262 | next | |
| 263 | } | |
| 264 | ||
| 265 | for(cc in constraint) { | |
| 266 | if(is.null(sdep[[cc]]) || is.na(sdep[[cc]])) | |
| 267 | sdep[[cc]] <- param | |
| 268 | else | |
| 269 | sdep[[cc]] <- c(sdep[[cc]], param) | |
| 270 | } | |
| 271 | } | |
| 272 | return(list(independent=independent, dependencies=sdep)) | |
| 273 | } | |
| 274 | ||
| 275 | ||
| 276 | configurations <- configurations[, grep("^\\.", colnames(configurations), invert=TRUE), drop = FALSE] | |
| 277 | configurations <- configurations[, param_names] | |
| 278 | ||
| 279 | if(hierarchy){ | |
| 280 | aux <- getDependency() | |
| 281 | if(length(aux$independent) >= 2){ | |
| 282 | indconfigurations <- configurations[, aux$independent] | |
| 283 | if(!is.null(filename)) | |
| 284 | pdf(file = paste0(filename,"_independent.pdf"), width=pdf.width) | |
| 285 | else dev.new() | |
| 286 | par(mar=mar) | |
| 287 | parcoordlabel(indconfigurations, parameters, title="Independent parameter parallel coordinates") | |
| 288 | if(!is.null(filename)) dev.off() | |
| 289 | }else{ | |
| 290 | cat("Skipping independent parameters",aux$independent,"\n") | |
| 291 | } | |
| 292 | ||
| 293 | dep <- aux$dependencies | |
| 294 | dnames <- names(dep) | |
| 295 | for(i in seq_along(dep)){ | |
| 296 | if(length(dep[[i]]) < 2){ | |
| 297 | cat("Skipping parameters",dnames[i],"\n") | |
| 298 | next | |
| 299 | } | |
| 300 | depconfigurations <- configurations[, dep[[i]]] | |
| 301 | depconfigurations <- depconfigurations[!apply(apply(depconfigurations, 1, is.na), 2, all),] | |
| 302 | ||
| 303 | if(nrow(depconfigurations) < 2 ){ | |
| 304 | cat("Skipping parameter",dnames[i],"\n") | |
| 305 | next | |
| 306 | } | |
| 307 | if(!is.null(filename)) | |
| 308 | pdf(file = paste0(filename,"_",dnames[i],".pdf"), width=pdf.width) | |
| 309 | else dev.new() | |
| 310 | par(mar=mar) | |
| 311 | parcoordlabel(depconfigurations, parameters, | |
| 312 | title = paste0("Parameters of dependent of ", dnames[i], " parallel coordinates")) | |
| 313 | if (!is.null(filename)) dev.off() | |
| 314 | } | |
| 315 | ||
| 316 | }else{ | |
| 317 | if(!is.null(filename)) | |
| 318 | pdf(file = paste0(filename,".pdf"), width=pdf.width) | |
| 319 | #else dev.new() | |
| 320 | par(mar=mar) | |
| 321 | parcoordlabel(configurations, parameters) | |
| 322 | if(!is.null(filename)) dev.off() | |
| 323 | } | |
| 324 | ||
| 325 | } | |
| 326 | ||
| 327 | 0 | #' Return the elite configurations of the final iteration. |
| 328 | 1 | #' |
| 329 | #' @param iraceResults Object created by \pkg{irace} and saved in \code{scenario$logFile}. | |
| 330 | #' @param logFile Log file created by \pkg{irace}, this file must contain the | |
| 331 | #' \code{iraceResults} object. | |
| 2 | #' @template arg_iraceresults | |
| 332 | 3 | #' @param n Number of elite configurations to return, if \code{n} is larger than the |
| 333 | #' number of configurations, then only the existing ones are returned. | |
| 4 | #' number of configurations, then only the existing ones are returned. The default (\code{n=0}) returns all of them. | |
| 334 | 5 | #' @param drop.metadata Remove metadata, such the configuration ID and |
| 335 | 6 | #' the ID of the parent, from the returned configurations. See |
| 336 | 7 | #' \code{\link{removeConfigurationsMetaData}}. |
| 337 | 8 | #' |
| 338 | 9 | #' @return A data frame containing the elite configurations required. |
| 339 | 10 | #' |
| 11 | #' @examples | |
| 12 | #' log_file <- system.file("exdata/irace-acotsp.Rdata", package="irace", mustWork=TRUE) | |
| 13 | #' print(removeConfigurationsMetaData(getFinalElites(log_file, n=1))) | |
| 340 | 14 | #' |
| 341 | 15 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 16 | #' @concept analysis | |
| 342 | 17 | #' @export |
| 343 | getFinalElites <- function(iraceResults = NULL, logFile = NULL, n = 0, | |
| 344 | drop.metadata = FALSE) | |
| 18 | getFinalElites <- function(iraceResults, n = 0L, drop.metadata = FALSE) | |
| 345 | 19 | { |
| 346 | if (is.null(iraceResults)) { | |
| 347 | if (is.null(logFile)) | |
| 348 | stop("You must supply either 'iraceResults' or 'logFile' argument.\n") | |
| 349 | else | |
| 350 | load(logFile) | |
| 351 | } | |
| 352 | ||
| 20 | if (missing(iraceResults)) stop("argument 'iraceResults' is missing") | |
| 21 | iraceResults <- read_logfile(iraceResults) | |
| 22 | ||
| 353 | 23 | last.elites <- iraceResults$allElites[[length(iraceResults$allElites)]] |
| 354 | 24 | |
| 355 | 25 | if (n == 0) |
| 356 | 26 | n <- length(last.elites) |
| 357 | 27 | |
| 358 | 28 | if (length(last.elites) < n) { |
| 359 | cat("Only", length(last.elites), "configurations available, reducing n,\n") | |
| 29 | cat("Only", length(last.elites), "configurations available, reducing n,") | |
| 360 | 30 | n <- length(last.elites) |
| 361 | 31 | } |
| 362 | 32 | last.elites <- last.elites[1:n] |
| 371 | 41 | |
| 372 | 42 | #' Returns the configurations selected by ID. |
| 373 | 43 | #' |
| 374 | #' @param iraceResults Object created by \pkg{irace} and saved in \code{scenario$logFile}. | |
| 375 | #' @param logFile Log file created by \pkg{irace}, this file must contain the | |
| 376 | #' \code{iraceResults} object. | |
| 44 | #' @template arg_iraceresults | |
| 377 | 45 | #' @param ids The id or a vector of ids of the candidates configurations to obtain. |
| 378 | 46 | #' @param drop.metadata Remove metadata, such the configuration ID and |
| 379 | 47 | #' the ID of the parent, from the returned configurations. See |
| 380 | #' \code{\link{removeConfigurationsMetaData}}. | |
| 48 | #' [removeConfigurationsMetaData()]. | |
| 381 | 49 | #' |
| 382 | 50 | #' @return A data frame containing the elite configurations required. |
| 383 | 51 | #' |
| 384 | 52 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 53 | #' @concept analysis | |
| 385 | 54 | #' @export |
| 386 | ## Get configuration(s) by the id(s). | |
| 387 | ## iraceResults: object created by irace and saved in scenario$logFile. | |
| 388 | ## iraceLog: log file created by irace, this file must contain the iraceResults object. | |
| 389 | ## ids: the id or a vector of ids of the candidates configurations to obtain. | |
| 390 | ## drop.metadata: Remove the internal identifier and parent identifier from the returned | |
| 391 | ## configurations data frame. | |
| 392 | ## * iraceResults or iraceLog must be provided, in case both are give iraceResults will be used. | |
| 393 | ## This function returns a data frame containing the selected candidate configurations | |
| 394 | getConfigurationById <- function(iraceResults = NULL, logFile = NULL, | |
| 395 | ids, drop.metadata = FALSE) | |
| 55 | getConfigurationById <- function(iraceResults, ids, drop.metadata = FALSE) | |
| 396 | 56 | { |
| 397 | if (is.null(iraceResults)) { | |
| 398 | if (is.null(logFile)) | |
| 399 | stop("You must supply either iraceResults or iraceLog argument.\n") | |
| 400 | else | |
| 401 | load(logFile) | |
| 402 | } | |
| 403 | ||
| 404 | if (length(ids) < 1) stop("You must provide at least one configuration id.\n") | |
| 57 | if (missing(iraceResults)) stop("argument 'iraceResults' is missing") | |
| 58 | iraceResults <- read_logfile(iraceResults) | |
| 59 | ||
| 60 | if (length(ids) < 1) stop("You must provide at least one configuration id.") | |
| 405 | 61 | |
| 406 | 62 | selection <- iraceResults$allConfigurations[,".ID."] %in% ids |
| 407 | 63 | |
| 408 | if (length(selection) < 1) stop("No configuration found with id", ids,".\n") | |
| 64 | if (length(selection) < 1) stop("No configuration found with id", ids,".") | |
| 409 | 65 | |
| 410 | 66 | configurations <-iraceResults$allConfigurations[selection, , drop = FALSE] |
| 411 | 67 | |
| 416 | 72 | |
| 417 | 73 | #' Returns the configurations by the iteration in which they were executed. |
| 418 | 74 | #' |
| 419 | #' @param iraceResults (\code{NULL}) Object created by \pkg{irace} and saved in \code{scenario$logFile}. | |
| 420 | #' @param logFile (\code{NULL}) Log file created by \pkg{irace}, this file must contain the | |
| 421 | #' \code{iraceResults} object. | |
| 75 | #' @template arg_iraceresults | |
| 422 | 76 | #' @param iterations The iteration number or a vector of iteration numbers from where |
| 423 | 77 | #' the configurations should be obtained. |
| 424 | 78 | #' @param drop.metadata (\code{FALSE}) Remove metadata, such the configuration ID and |
| 428 | 82 | #' @return A data frame containing the elite configurations required. |
| 429 | 83 | #' |
| 430 | 84 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 85 | #' @concept analysis | |
| 431 | 86 | #' @export |
| 432 | ## Get configuration(s) by the iteration in which they were executed. | |
| 433 | ## iraceResults: object created by irace and saved in scenario$logFile. | |
| 434 | ## iraceLog: log file created by irace, this file must contain the iraceResults object. | |
| 435 | ## iterations: the iteration or a vector of iterations from where the configurations should be obtained. | |
| 436 | ## drop.metadata: Remove the internal identifier and parent identifier from the returned | |
| 437 | ## configurations data frame. | |
| 438 | ## * iraceResults or iraceLog must be provided, in case both are give iraceResults will be used. | |
| 439 | ## This function returns a data frame containing the selected candidate configurations | |
| 440 | getConfigurationByIteration <- function(iraceResults = NULL, logFile = NULL, | |
| 87 | getConfigurationByIteration <- function(iraceResults, | |
| 441 | 88 | iterations, drop.metadata = FALSE) |
| 442 | 89 | { |
| 443 | if (is.null(iraceResults)) { | |
| 444 | if (is.null(logFile)) | |
| 445 | stop("You must supply either iraceResults or iraceLog argument.\n") | |
| 446 | else | |
| 447 | load(logFile) | |
| 448 | } | |
| 449 | ||
| 90 | if (missing(iraceResults)) stop("argument 'iraceResults' is missing") | |
| 91 | iraceResults <- read_logfile(iraceResults) | |
| 92 | ||
| 450 | 93 | if (length(iterations) < 1) |
| 451 | stop("You must provide at least one configuration id.\n") | |
| 94 | stop("You must provide at least one configuration id.") | |
| 452 | 95 | |
| 453 | 96 | # To silence warning. |
| 454 | 97 | iteration <- NULL |
| 458 | 101 | |
| 459 | 102 | selection <- iraceResults$allConfigurations[,".ID."] %in% ids |
| 460 | 103 | |
| 461 | if (length(selection) < 1) stop("No configuration found with id", ids,".\n") | |
| 104 | if (length(selection) < 1) stop("No configuration found with id", ids,".") | |
| 462 | 105 | |
| 463 | 106 | configurations <- iraceResults$allConfigurations[selection, , drop=FALSE] |
| 464 | 107 | |
| 467 | 110 | return(configurations) |
| 468 | 111 | } |
| 469 | 112 | |
| 470 | #' Creates box plots of the quality of configurations. | |
| 471 | #' | |
| 472 | #' @param experiments Matrix of performance of configurations (columns) over a set of instances (rows). | |
| 473 | #' @param title (\code{NULL}) Title for the plot. | |
| 474 | #' @param xlabel Label for the x axis. | |
| 475 | #' @param ylabel Label for the y axis. | |
| 476 | #' @param filename (\code{NULL}) Filename prefix to create a pdf file with the plot. | |
| 477 | #' | |
| 478 | #' @return Box plot of the performance of the configurations. | |
| 479 | #' | |
| 480 | #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 481 | #' @export | |
| 482 | configurationsBoxplot <- function(experiments, title = NULL, | |
| 483 | xlabel = "Configuration ID", | |
| 484 | ylabel = "Configuration cost", | |
| 485 | filename = NULL) | |
| 486 | { | |
| 487 | plot.jitter.points <- function(x, y, factor = 10 / x, pch = 20, ...) | |
| 488 | points(jitter(rep(x, length(y)), factor = factor), | |
| 489 | y, pch = pch, col = rgb(0,0,0,.2), ...) | |
| 490 | ||
| 491 | if (any(colSums(is.na(experiments)) > 0)) | |
| 492 | cat("Warning: There are NA values in the experiment results provided.\n") | |
| 493 | ||
| 494 | data.labels <- colnames(experiments) | |
| 495 | if (is.null(data.labels)) data.labels <- 1:ncol(experiments) | |
| 496 | ||
| 497 | # These parameters could be exposed to configurate the plot | |
| 498 | ||
| 499 | if (!is.null(filename)) { | |
| 500 | filename <- paste0(filename, ".pdf.") | |
| 501 | cat("Creating file", filename,"\n") | |
| 502 | # MANUEL: Why cairo_pdf = | |
| 503 | cairo_pdf(filename = filename, width=20, height=8) | |
| 504 | on.exit(dev.off(), add = TRUE) | |
| 505 | plot.mar <- c(7,11,4,1) | |
| 506 | plot.lwd <- 5 | |
| 507 | cex.axis <- 3 | |
| 508 | cex.main <- 3 | |
| 509 | x.add <- 2 | |
| 510 | } else { | |
| 511 | plot.mar <- c(2.5,9,4,1) | |
| 512 | plot.lwd <- 2 | |
| 513 | cex.axis <- 1 | |
| 514 | cex.main <- 1 | |
| 515 | x.add <- 0 | |
| 516 | } | |
| 517 | ||
| 518 | if (is.null(title)) plot.mar[3] <- 1 | |
| 519 | ||
| 520 | old.par <- par(las=1, mar=plot.mar, cex.axis=cex.axis, cex.main=cex.main, lwd=plot.lwd) | |
| 521 | on.exit(old.par, add = TRUE) | |
| 522 | ||
| 523 | boxplot(experiments, main = title, xaxt = "n", outline = TRUE) | |
| 524 | for (i in 1:ncol(experiments)) { | |
| 525 | plot.jitter.points (i, experiments[,i], cex = 1.5 * cex.axis) | |
| 526 | } | |
| 527 | ||
| 528 | # X axis | |
| 529 | axis(1, at=c(1:length(data.labels)), labels=data.labels, line = -0.5 + x.add, | |
| 530 | tick=FALSE, las=1, cex.axis=cex.axis) | |
| 531 | mtext(xlabel, side=1, line=1.5 + 2*x.add, cex=cex.axis, las=0) | |
| 532 | # Y axis | |
| 533 | mtext(ylabel, side=2, line=5+1.8*x.add, cex=cex.axis, las=0) | |
| 534 | } | |
| 535 | ||
| 29 | 29 | #' @examples |
| 30 | 30 | #' \dontrun{ |
| 31 | 31 | #' # Execute the postselection automatically after irace |
| 32 | #' scenario <- readScenario(filename="scenario.txt") | |
| 32 | 33 | #' parameters <- readParameters("parameters.txt") |
| 33 | #' scenario <- readScenario(filename="scenario.txt", | |
| 34 | #' scenario=defaultScenario()) | |
| 35 | 34 | #' # Use 10% of the total budget |
| 36 | 35 | #' scenario$postselection <- 0.1 |
| 37 | 36 | #' irace(scenario=scenario, parameters=parameters) |
| 41 | 40 | #' |
| 42 | 41 | #' @author Leslie Pérez Cáceres |
| 43 | 42 | #' @export |
| 44 | # This function executes a post selection race | |
| 45 | # elites: test all elites | |
| 46 | 43 | psRace <- function(iraceLogFile=NULL, iraceResults=NULL, conf.ids=NULL, |
| 47 | 44 | postselection=NULL, max.experiments=NULL, elites=FALSE, seed=1234567) |
| 48 | 45 | { |
| 50 | 47 | if (is.null(iraceLogFile) && is.null(iraceResults)) |
| 51 | 48 | irace.error("You must provide a Rdata file or an iraceResults object.") |
| 52 | 49 | |
| 53 | irace.note ("Stating post-selection:\n# Seed:", seed, "\n") | |
| 50 | irace.note ("Starting post-selection:\n# Seed:", seed, "\n") | |
| 54 | 51 | if (!is.null(iraceLogFile)) |
| 55 | 52 | cat("# Log file:",iraceLogFile,"\n") |
| 56 | 53 | |
| 57 | 54 | # Load the data of the log file |
| 58 | 55 | if (!is.null(iraceLogFile)) |
| 59 | load(iraceLogFile) | |
| 56 | iraceResults <- read_logfile(iraceLogFile) | |
| 60 | 57 | |
| 61 | 58 | parameters <- iraceResults$parameters |
| 62 | 59 | scenario <- iraceResults$scenario |
| 65 | 62 | if (!is.null(conf.ids)) { |
| 66 | 63 | if (!all(conf.ids %in% iraceResults$allConfigurations$.ID.)) |
| 67 | 64 | irace.error("Configuration ids provided", conf.ids,"cannot be found in the configurations.") |
| 68 | configurations <- iraceResults$allConfigurations[iraceResults$allConfigurations$.ID.%in% conf.ids,,drop=FALSE] | |
| 65 | configurations <- iraceResults$allConfigurations[iraceResults$allConfigurations$.ID.%in% conf.ids,,drop=FALSE] | |
| 69 | 66 | } else { |
| 70 | 67 | which.elites <- if (elites) unlist(iraceResults$allElites) else iraceResults$iterationElites |
| 71 | 68 | which.elites <- unique(which.elites) |
| 112 | 109 | elitistNewInstances = 0) |
| 113 | 110 | experiments <- race.output$experiments |
| 114 | 111 | |
| 115 | elite.configurations <- extractElites(race.output$configurations, | |
| 112 | elite.configurations <- extractElites(scenario, parameters, | |
| 113 | race.output$configurations, | |
| 116 | 114 | min(race.output$nbAlive, 1)) |
| 117 | 115 | irace.note("Elite configurations (first number is the configuration ID;", |
| 118 | 116 | " listed from best to worst according to the ", |
| 0 | #' Converts a relative path to an absolute path. It tries really hard to create | |
| 1 | #' canonical paths. | |
| 2 | #' | |
| 3 | #' @param path (`character(1)`) Character string representing a relative path. | |
| 4 | #' @param cwd (`character(1)`) Current working directory. | |
| 5 | #' | |
| 6 | #' @return (`character(1)`) Character string representing the absolute path | |
| 7 | #' | |
| 8 | #' @examples | |
| 9 | #' path_rel2abs("..") | |
| 10 | #' @export | |
| 11 | path_rel2abs <- function (path, cwd = getwd()) | |
| 12 | { | |
| 13 | # Keep doing gsub as long as x keeps changing. | |
| 14 | gsub.all <- function(pattern, repl, x, ...) { | |
| 15 | repeat { | |
| 16 | newx <- gsub(pattern, repl, x, ...) | |
| 17 | if (newx == x) return(newx) | |
| 18 | x <- newx | |
| 19 | } | |
| 20 | } | |
| 21 | # FIXME: Why NA and not FALSE? | |
| 22 | irace_normalize_path <- function(path) | |
| 23 | suppressWarnings(normalizePath(path, winslash = "/", mustWork = NA)) | |
| 24 | ||
| 25 | if (is.null.or.na(path)) { | |
| 26 | return (NULL) | |
| 27 | } else if (path == "") { | |
| 28 | return ("") | |
| 29 | } | |
| 30 | ||
| 31 | # Using .Platform$file.sep is too fragile. Better just use "/" everywhere. | |
| 32 | s <- "/" | |
| 33 | ||
| 34 | # Possibly expand ~/path to /home/user/path. | |
| 35 | path <- path.expand(path) | |
| 36 | # Remove winslashes if given. | |
| 37 | path <- gsub("\\", s, path, fixed = TRUE) | |
| 38 | ||
| 39 | # Detect a Windows drive | |
| 40 | windrive.regex <- "^[A-Za-z]:" | |
| 41 | windrive <- "" | |
| 42 | if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { | |
| 43 | m <- regexpr(windrive.regex, path) | |
| 44 | windrive <- regmatches(path, m) | |
| 45 | path <- sub(windrive.regex, "", path) | |
| 46 | } | |
| 47 | ||
| 48 | # Change "/./" to "/" to get a canonical form | |
| 49 | path <- gsub.all(paste0(s, ".", s), s, path, fixed = TRUE) | |
| 50 | # Change "//" to "/" to get a canonical form | |
| 51 | path <- gsub(paste0(s, s, "+"), s, path) | |
| 52 | # Change "/.$" to "/" to get a canonical form | |
| 53 | path <- sub(paste0(s, "\\.$"), s, path) | |
| 54 | # Drop final "/" | |
| 55 | path <- sub(paste0(s, "$"), "", path) | |
| 56 | if (path == "") path <- s | |
| 57 | ||
| 58 | # Prefix the current cwd to the path if it doesn't start with | |
| 59 | # / \\ or whatever separator. | |
| 60 | if (path == "." || !startsWith(path, s)) { | |
| 61 | # There is no need to normalize cwd if it was returned by getwd() | |
| 62 | if (!missing(cwd)) { | |
| 63 | # Recurse to get absolute cwd | |
| 64 | cwd <- path_rel2abs(cwd) | |
| 65 | } | |
| 66 | ||
| 67 | # Speed-up the most common cases. | |
| 68 | # If it is just "." | |
| 69 | if (path == ".") return (irace_normalize_path(cwd)) | |
| 70 | ||
| 71 | # If it does not contain separators at all and does not start with ".." | |
| 72 | if (!startsWith(path, "..") && !grepl(s, path)) { | |
| 73 | # it may be a command in the path. | |
| 74 | sys_path <- suppressWarnings(Sys.which(path)) | |
| 75 | if (nchar(sys_path) > 0 | |
| 76 | && file.access(sys_path, mode = 1) == 0 | |
| 77 | && !file.info(sys_path)$isdir) { | |
| 78 | return(irace_normalize_path(as.vector(sys_path))) | |
| 79 | } | |
| 80 | } | |
| 81 | ||
| 82 | # Remove "./" from the start of path. | |
| 83 | path <- sub(paste0("^\\.", s), "", path) | |
| 84 | # Make it absolute but avoid doubling s | |
| 85 | if (substring(cwd, nchar(cwd)) == s) path <- paste0(cwd, path) | |
| 86 | else path <- paste0(cwd, s, path) | |
| 87 | # If it is just a path without ".." inside | |
| 88 | if (!grepl(paste0(s,"\\.\\."), path)) { | |
| 89 | return (irace_normalize_path(path)) | |
| 90 | } | |
| 91 | # Detect a Windows drive | |
| 92 | if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { | |
| 93 | m <- regexpr(windrive.regex, path) | |
| 94 | windrive <- regmatches(path, m) | |
| 95 | path <- sub(windrive.regex, "", path) | |
| 96 | } | |
| 97 | } | |
| 98 | # else | |
| 99 | ||
| 100 | # Change "/x/.." to "/" to get a canonical form | |
| 101 | prevdir.regex <- paste0(s, "[^", s,"]+", s, "\\.\\.") | |
| 102 | repeat { | |
| 103 | # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" | |
| 104 | tmp <- sub(paste0(prevdir.regex, s), s, path) | |
| 105 | if (tmp == path) break | |
| 106 | path <- tmp | |
| 107 | } | |
| 108 | # Handle "/something/..$" to "/" that is, when ".." is the last thing in the path. | |
| 109 | path <- sub(paste0(prevdir.regex, "$"), s, path) | |
| 110 | ||
| 111 | # Handle "^/../../.." to "/" that is, going up at the root just returns the root. | |
| 112 | repeat { | |
| 113 | # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" | |
| 114 | tmp <- sub(paste0("^", s, "\\.\\.", s), s, path) | |
| 115 | if (tmp == path) break | |
| 116 | path <- tmp | |
| 117 | } | |
| 118 | # Handle "^/..$" to "/" that is, when ".." is the last thing in the path. | |
| 119 | path <- sub(paste0("^", s, "\\.\\.$"), s, path) | |
| 120 | # Add back Windows drive, if any. | |
| 121 | path <- paste0(windrive, path) | |
| 122 | ||
| 123 | # We use normalizePath, which will further simplify the path if | |
| 124 | # the path exists. | |
| 125 | irace_normalize_path(path) | |
| 126 | } |
| 33 | 33 | #' irace.logfile <- file.path(system.file(package="irace"), |
| 34 | 34 | #' "exdata", "irace-acotsp.Rdata") |
| 35 | 35 | #' load(irace.logfile) |
| 36 | #' attach(iraceResults) | |
| 36 | #' allConfigurations <- iraceResults$allConfigurations | |
| 37 | #' parameters <- iraceResults$parameters | |
| 37 | 38 | #' apply(allConfigurations[1:10, unlist(parameters$names)], 1, buildCommandLine, |
| 38 | 39 | #' unlist(parameters$switches)) |
| 39 | 40 | #' |
| 40 | 41 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 41 | #' @md | |
| 42 | 42 | #' @export |
| 43 | 43 | buildCommandLine <- function(values, switches) |
| 44 | 44 | { |
| 77 | 77 | target.evaluator.call = NULL) |
| 78 | 78 | { |
| 79 | 79 | if (!is.null(target.evaluator.call)) { |
| 80 | err.msg <- paste0(err.msg, "\n", .irace.prefix, | |
| 80 | err.msg <- paste0(err.msg, "\n", .msg.prefix, | |
| 81 | 81 | "The call to targetEvaluator was:\n", target.evaluator.call) |
| 82 | 82 | } |
| 83 | 83 | if (!is.null(target.runner.call)) { |
| 84 | err.msg <- paste0(err.msg, "\n", .irace.prefix, | |
| 84 | err.msg <- paste0(err.msg, "\n", .msg.prefix, | |
| 85 | 85 | "The call to targetRunner was:\n", target.runner.call) |
| 86 | 86 | } |
| 87 | 87 | if (is.null(output$outputRaw)) { |
| 100 | 100 | " Try to run the command(s) above from the execution directory '", |
| 101 | 101 | scenario$execDir, "' to investigate the issue. See also Appendix B (targetRunner troubleshooting checklist) of the User Guide (https://cran.r-project.org/package=irace/vignettes/irace-package.pdf).") |
| 102 | 102 | } |
| 103 | irace.error(err.msg, "\n", .irace.prefix, | |
| 103 | irace.error(err.msg, "\n", .msg.prefix, | |
| 104 | 104 | "The output was:\n", paste(output$outputRaw, collapse = "\n"), |
| 105 | "\n", .irace.prefix, advice.txt) | |
| 105 | "\n", .msg.prefix, advice.txt) | |
| 106 | 106 | } |
| 107 | 107 | |
| 108 | 108 | check.output.target.evaluator <- function (output, scenario, target.runner.call = NULL) |
| 145 | 145 | scenario, target.runner.call) |
| 146 | 146 | check.output.target.evaluator (output, scenario, target.runner.call = target.runner.call) |
| 147 | 147 | # Fix too small time. |
| 148 | output$time <- if (is.null(output$time)) NA else max(output$time, 0.01) | |
| 148 | output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) | |
| 149 | 149 | return (output) |
| 150 | 150 | } |
| 151 | 151 | |
| 193 | 193 | #' |
| 194 | 194 | #' |
| 195 | 195 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 196 | #' @md | |
| 197 | 196 | #' @export |
| 198 | 197 | target.evaluator.default <- function(experiment, num.configurations, all.conf.id, |
| 199 | 198 | scenario, target.runner.call) |
| 211 | 210 | } |
| 212 | 211 | |
| 213 | 212 | cwd <- setwd (scenario$execDir) |
| 214 | # FIXME: I think we don't even need to paste the args, since system2 handles this by itself. | |
| 215 | args <- paste(configuration.id, instance.id, seed, instance, num.configurations, all.conf.id) | |
| 213 | args <- c(configuration.id, instance.id, seed, instance, num.configurations, all.conf.id) | |
| 216 | 214 | output <- runcommand(targetEvaluator, args, configuration.id, debugLevel) |
| 217 | 215 | setwd (cwd) |
| 218 | 216 | |
| 233 | 231 | } |
| 234 | 232 | return(list(cost = cost, time = time, |
| 235 | 233 | error = err.msg, outputRaw = output$output, |
| 236 | call = paste(targetEvaluator, args))) | |
| 234 | call = paste(targetEvaluator, args, collapse=" "))) | |
| 237 | 235 | } |
| 238 | 236 | |
| 239 | 237 | check.output.target.runner <- function (output, scenario) |
| 289 | 287 | target.error (err.msg, output, scenario, target.runner.call = output$call) |
| 290 | 288 | } |
| 291 | 289 | # Fix too small time. |
| 292 | output$time <- if (is.null(output$time)) NA else max(output$time, 0.01) | |
| 290 | output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) | |
| 293 | 291 | return (output) |
| 294 | 292 | } |
| 295 | 293 | |
| 301 | 299 | doit <- function(experiment, scenario) |
| 302 | 300 | { |
| 303 | 301 | x <- target.runner(experiment, scenario) |
| 304 | return (check.output.target.runner (x, scenario)) | |
| 302 | return (check.output.target.runner(x, scenario)) | |
| 305 | 303 | } |
| 306 | 304 | |
| 307 | 305 | retries <- scenario$targetRunnerRetries |
| 353 | 351 | |
| 354 | 352 | target.runner.aclib <- function(experiment, scenario) |
| 355 | 353 | { |
| 354 | debugLevel <- scenario$debugLevel | |
| 355 | res <- run_target_runner(experiment, scenario) | |
| 356 | cmd <- res$cmd | |
| 357 | output <- res$output | |
| 358 | args <- res$args | |
| 359 | ||
| 360 | err.msg <- output$error | |
| 361 | if (is.null(err.msg)) { | |
| 362 | return(c(parse.aclib.output(output$output), | |
| 363 | list(outputRaw = output$output, call = paste(cmd, args)))) | |
| 364 | } | |
| 365 | ||
| 366 | list(cost = NULL, time = NULL, error = err.msg, | |
| 367 | outputRaw = output$output, call = paste(cmd, args)) | |
| 368 | } | |
| 369 | ||
| 370 | check_launcher_args <- function(targetRunnerLauncherArgs) | |
| 371 | { | |
| 372 | if (!grepl("{targetRunner}", targetRunnerLauncherArgs, fixed=TRUE)) | |
| 373 | irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunner}'") | |
| 374 | if (!grepl("{targetRunnerArgs}", targetRunnerLauncherArgs, fixed=TRUE)) | |
| 375 | irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunnerArgs}'") | |
| 376 | } | |
| 377 | ||
| 378 | process_launcher_args <- function(targetRunnerLauncherArgs, targetRunner, args) | |
| 379 | { | |
| 380 | check_launcher_args(targetRunnerLauncherArgs) | |
| 381 | targetRunnerLauncherArgs <- gsub("{targetRunner}", targetRunner, targetRunnerLauncherArgs, fixed=TRUE) | |
| 382 | targetRunnerLauncherArgs <- gsub("{targetRunnerArgs}", args, targetRunnerLauncherArgs, fixed=TRUE) | |
| 383 | targetRunnerLauncherArgs | |
| 384 | } | |
| 385 | ||
| 386 | run_target_runner <- function(experiment, scenario) | |
| 387 | { | |
| 356 | 388 | configuration.id <- experiment$id.configuration |
| 357 | 389 | instance.id <- experiment$id.instance |
| 358 | 390 | seed <- experiment$seed |
| 361 | 393 | switches <- experiment$switches |
| 362 | 394 | bound <- experiment$bound |
| 363 | 395 | |
| 364 | debugLevel <- scenario$debugLevel | |
| 365 | 396 | targetRunner <- scenario$targetRunner |
| 366 | if (as.logical(file.access(targetRunner, mode = 1))) { | |
| 367 | irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") | |
| 368 | } | |
| 369 | ||
| 370 | has_value <- !is.na(configuration) | |
| 371 | # <executable> [<arg>] [<arg>] ... [--cutoff <cutoff time>] [--instance <instance name>] | |
| 372 | # [--seed <seed>] --config [-param_name_1 value_1] [-param_name_2 value_2] ... | |
| 373 | args <- paste("--instance", instance, "--seed", seed, "--config", | |
| 374 | paste0("-", switches[has_value], " ", configuration[has_value], | |
| 375 | collapse = " ")) | |
| 376 | if (!is.null.or.na(bound)) | |
| 377 | args <- paste("--cutoff", bound, args) | |
| 378 | ||
| 379 | output <- runcommand(targetRunner, args, configuration.id, debugLevel) | |
| 380 | ||
| 381 | err.msg <- output$error | |
| 382 | if (is.null(err.msg)) { | |
| 383 | return(c(parse.aclib.output (output$output), | |
| 384 | list(outputRaw = output$output, call = paste(targetRunner, args)))) | |
| 385 | } | |
| 386 | ||
| 387 | return(list(cost = NULL, time = NULL, error = err.msg, | |
| 388 | outputRaw = output$output, call = paste(targetRunner, args))) | |
| 389 | } | |
| 390 | ||
| 391 | ||
| 392 | #' target.runner.default | |
| 397 | debugLevel <- scenario$debugLevel | |
| 398 | ||
| 399 | if (scenario$aclib) { | |
| 400 | has_value <- !is.na(configuration) | |
| 401 | # <executable> [<arg>] [<arg>] ... [--cutoff <cutoff time>] [--instance <instance name>] | |
| 402 | # [--seed <seed>] --config [-param_name_1 value_1] [-param_name_2 value_2] ... | |
| 403 | args <- paste("--instance", instance, "--seed", seed, "--config", | |
| 404 | paste0("-", switches[has_value], " ", configuration[has_value], | |
| 405 | collapse = " ")) | |
| 406 | if (!is.null.or.na(bound)) | |
| 407 | args <- paste("--cutoff", bound, args) | |
| 408 | } else { | |
| 409 | args <- paste(configuration.id, instance.id, seed, instance, bound, | |
| 410 | buildCommandLine(configuration, switches)) | |
| 411 | } | |
| 412 | ||
| 413 | targetRunnerLauncher <- scenario$targetRunnerLauncher | |
| 414 | if (is.null.or.empty(scenario$targetRunnerLauncher)) { | |
| 415 | if (as.logical(file.access(targetRunner, mode = 1))) { | |
| 416 | irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") | |
| 417 | } | |
| 418 | output <- runcommand(targetRunner, args, configuration.id, debugLevel) | |
| 419 | return(list(cmd=targetRunner, output=output, args=args)) | |
| 420 | } | |
| 421 | if (as.logical(file.access(targetRunnerLauncher, mode = 1))) { | |
| 422 | irace.error ("targetRunnerLauncher ", shQuote(targetRunnerLauncher), " cannot be found or is not executable!\n") | |
| 423 | } | |
| 424 | ||
| 425 | args <- process_launcher_args(scenario$targetRunnerLauncherArgs, targetRunner, args) | |
| 426 | output <- runcommand(targetRunnerLauncher, args, configuration.id, debugLevel) | |
| 427 | return(list(cmd=targetRunnerLauncher, output=output, args=args)) | |
| 428 | } | |
| 429 | ||
| 430 | #' Default `targetRunner` function. | |
| 393 | 431 | #' |
| 394 | #' `target.runner.default` is the default targetRunner function. | |
| 395 | #' You can use it as an advanced example of how to create your own targetRunner | |
| 396 | #' function. | |
| 432 | #' Use it as an advanced example of how to create your own `targetRunner` function. | |
| 397 | 433 | #' |
| 398 | 434 | #' @param experiment A list describing the experiment. It contains at least: |
| 399 | 435 | #' \describe{ |
| 430 | 466 | #' |
| 431 | 467 | #' |
| 432 | 468 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 433 | #' @md | |
| 434 | 469 | #' @export |
| 435 | 470 | target.runner.default <- function(experiment, scenario) |
| 436 | 471 | { |
| 437 | configuration.id <- experiment$id.configuration | |
| 438 | instance.id <- experiment$id.instance | |
| 439 | seed <- experiment$seed | |
| 440 | configuration <- experiment$configuration | |
| 441 | instance <- experiment$instance | |
| 442 | switches <- experiment$switches | |
| 443 | bound <- experiment$bound | |
| 444 | ||
| 445 | debugLevel <- scenario$debugLevel | |
| 446 | targetRunner <- scenario$targetRunner | |
| 447 | if (as.logical(file.access(targetRunner, mode = 1))) { | |
| 448 | irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") | |
| 449 | } | |
| 450 | ||
| 451 | args <- paste(configuration.id, instance.id, seed, instance, bound, | |
| 452 | buildCommandLine(configuration, switches)) | |
| 453 | output <- runcommand(targetRunner, args, configuration.id, debugLevel) | |
| 454 | ||
| 472 | res <- run_target_runner(experiment, scenario) | |
| 473 | cmd <- res$cmd | |
| 474 | output <- res$output | |
| 475 | args <- res$args | |
| 476 | ||
| 477 | debugLevel <- scenario$debugLevel | |
| 455 | 478 | cost <- time <- NULL |
| 456 | 479 | err.msg <- output$error |
| 457 | 480 | if (is.null(err.msg)) { |
| 458 | v.output <- parse.output(output$output, verbose = (scenario$debugLevel >= 2)) | |
| 481 | v.output <- parse.output(output$output, verbose = (debugLevel >= 2)) | |
| 459 | 482 | if (length(v.output) > 2) { |
| 460 | 483 | err.msg <- "The output of targetRunner should not be more than two numbers!" |
| 461 | 484 | } else if (length(v.output) == 1) { |
| 469 | 492 | time <- v.output[2] |
| 470 | 493 | } |
| 471 | 494 | } |
| 472 | return(list(cost = cost, time = time, | |
| 473 | error = err.msg, outputRaw = output$output, | |
| 474 | call = paste(targetRunner, args))) | |
| 495 | list(cost = cost, time = time, | |
| 496 | error = err.msg, outputRaw = output$output, | |
| 497 | call = paste(cmd, args, collapse = " ")) | |
| 475 | 498 | } |
| 476 | 499 | |
| 477 | 500 | execute.experiments <- function(experiments, scenario) |
| 569 | 592 | scenario = scenario, |
| 570 | 593 | target.runner = target.runner) |
| 571 | 594 | } |
| 572 | ||
| 573 | return(target.output) | |
| 595 | target.output | |
| 574 | 596 | } |
| 575 | 597 | |
| 576 | 598 | execute.evaluator <- function(experiments, scenario, target.output, configurations.id) |
| 577 | 599 | { |
| 578 | 600 | ## FIXME: We do not need the configurations.id argument: |
| 579 | # configurations.id <- sapply(experiments, function(x) x[["id.configuration"]]) | |
| 601 | irace.assert(isTRUE(all.equal(configurations.id, | |
| 602 | sapply(experiments, getElement, "id.configuration")))) | |
| 580 | 603 | all.conf.id <- paste(configurations.id, collapse = " ") |
| 581 | 604 | |
| 582 | 605 | ## Evaluate configurations sequentially |
| 77 | 77 | # which.alive nor which.exe |
| 78 | 78 | which.alive, which.exe, parameters, scenario) |
| 79 | 79 | { |
| 80 | debugLevel <- scenario$debugLevel | |
| 81 | ||
| 82 | irace.assert (isTRUE(parameters$nbParameters > 0)) | |
| 80 | irace.assert (parameters$nbVariable > 0) | |
| 83 | 81 | irace.assert (length(parameters$names) == parameters$nbParameters) |
| 84 | 82 | |
| 85 | 83 | # FIXME: Accessing 'seed' and 'instance' should be moved to createExperimentList. |
| 108 | 106 | if (!is.null(scenario$targetEvaluator)) |
| 109 | 107 | target.output <- execute.evaluator (experiments, scenario, target.output, |
| 110 | 108 | configurations[, ".ID."]) |
| 111 | return(target.output) | |
| 109 | target.output | |
| 112 | 110 | } |
| 113 | 111 | |
| 114 | 112 | aux2.friedman <- function(y, I, alive, conf.level = 0.95) |
| 159 | 157 | if (no.alive == 2) { |
| 160 | 158 | best <- NULL |
| 161 | 159 | ranks <- NULL |
| 160 | dropped.any <- TRUE | |
| 161 | PVAL <- 0 | |
| 162 | 162 | # If only 2 configurations are left, switch to Wilcoxon |
| 163 | V1 <- results[, which.alive[1]] | |
| 164 | V2 <- results[, which.alive[2]] | |
| 165 | PVAL <- wilcox.test(V1, V2, paired = TRUE, exact = FALSE)$p.value | |
| 166 | if (!is.nan(PVAL) && !is.na(PVAL) && PVAL < 1 - conf.level) { | |
| 167 | dropped.any <- TRUE | |
| 168 | if (median(V1 - V2) < 0) { | |
| 169 | best <- which.alive[1] | |
| 170 | ranks <- c(1,2) | |
| 171 | alive[which.alive[2]] <- FALSE | |
| 172 | } else { | |
| 173 | best <- which.alive[2] | |
| 174 | ranks <- c(2,1) | |
| 175 | alive[which.alive[1]] <- FALSE | |
| 176 | } | |
| 163 | V1 <- results[, which.alive[1]] | |
| 164 | V2 <- results[, which.alive[2]] | |
| 165 | ||
| 166 | # Avoid the test if the answer is obvious | |
| 167 | if (all(V1 <= V2)) { | |
| 168 | ranks <- c(1,2) | |
| 169 | } else if (all(V2 <= V1)) { | |
| 170 | ranks <- c(2,1) | |
| 177 | 171 | } else { |
| 178 | dropped.any <- FALSE | |
| 179 | if (median(V1 - V2) < 0) { | |
| 180 | best <- which.alive[1] | |
| 181 | ranks <- c(1,2) | |
| 182 | } else { | |
| 183 | best <- which.alive[2] | |
| 184 | ranks <- c(2,1) | |
| 185 | } | |
| 186 | } | |
| 172 | res <- wilcox.test(V1, V2, paired = TRUE, conf.int = TRUE) | |
| 173 | PVAL <- res$p.value | |
| 174 | irace.assert(!is.nan(PVAL) & !is.na(PVAL)) | |
| 175 | if (PVAL >= 1 - conf.level) dropped.any <- FALSE | |
| 176 | # We use the pseudo median estimated by the test. | |
| 177 | ranks <- if (res$estimate <= 0) c(1,2) else c(2,1) | |
| 178 | } | |
| 179 | best <- which.alive[ranks[1]] | |
| 180 | if (dropped.any) | |
| 181 | alive[which.alive[ranks[2]]] <- FALSE | |
| 182 | ||
| 187 | 183 | irace.assert(which.alive[which.min(ranks)] == best) |
| 188 | 184 | return(list(best = best, ranks = ranks, alive = alive, dropped.any = dropped.any, p.value = PVAL)) |
| 189 | 185 | } else { |
| 192 | 188 | } |
| 193 | 189 | } |
| 194 | 190 | |
| 195 | aux.ttest <- function(results, alive, which.alive, conf.level, | |
| 196 | adjust = c("none","bonferroni","holm")) | |
| 191 | aux.one_ttest <- function(results, alive, which.alive, conf.level, | |
| 192 | adjust = c("none","bonferroni","holm")) | |
| 197 | 193 | { |
| 198 | 194 | adjust <- match.arg(adjust) |
| 199 | 195 | irace.assert(sum(alive) == length(which.alive)) |
| 200 | ||
| 201 | 196 | results <- results[, which.alive] |
| 202 | 197 | means <- colMeans(results) |
| 203 | 198 | best <- which.min(means) |
| 210 | 205 | for (j in which_test) { |
| 211 | 206 | PVAL <- pvals[j] |
| 212 | 207 | if (PVAL == 1.0) next |
| 213 | results_j <- results[ , j] | |
| 208 | results_j <- results[, j] | |
| 214 | 209 | # t.test may fail if the data in each group is almost constant. Hence, we |
| 215 | 210 | # surround the call in a try() and we initialize p with 1 if the means are |
| 216 | 211 | # equal or zero if they are different |
| 217 | 212 | if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next |
| 218 | try(PVAL <- t.test(results_best, results_j, paired = TRUE)$p.value) | |
| 213 | # The t.test may fail if the data are not normal despite one configuration | |
| 214 | # clearly dominating the other. | |
| 215 | if (all(results_best <= results_j)) next | |
| 216 | try(PVAL <- t.test(results_best, results_j, alternative = "less", | |
| 217 | paired = TRUE)$p.value) | |
| 219 | 218 | irace.assert(!is.nan(PVAL) & !is.na(PVAL)) |
| 220 | 219 | pvals[j] <- PVAL |
| 221 | 220 | } |
| 228 | 227 | dropped.any = dropped_any, p.value = min(pvals))) |
| 229 | 228 | } |
| 230 | 229 | |
| 231 | race.init.instances <- function(deterministic, max.instances) | |
| 232 | { | |
| 233 | if (.irace$next.instance <= max.instances) { | |
| 234 | race.instances <- .irace$next.instance : max.instances | |
| 235 | } else { | |
| 236 | # This may happen if the scenario is deterministic and we would need | |
| 237 | # more instances than what we have. | |
| 238 | irace.assert(deterministic) | |
| 239 | race.instances <- 1 : max.instances | |
| 240 | } | |
| 241 | return(race.instances) | |
| 242 | } | |
| 243 | ||
| 244 | elitrace.init.instances <- function(race.env, deterministic, max.instances) | |
| 230 | aux.ttest <- function(results, alive, which.alive, conf.level, | |
| 231 | adjust = c("none","bonferroni","holm")) | |
| 232 | { | |
| 233 | adjust <- match.arg(adjust) | |
| 234 | irace.assert(sum(alive) == length(which.alive)) | |
| 235 | ||
| 236 | results <- results[, which.alive] | |
| 237 | means <- colMeans(results) | |
| 238 | # FIXME: break ties using median or ranks? | |
| 239 | best <- which.min(means) | |
| 240 | mean_best <- means[best] | |
| 241 | pvals <- sapply(means, function(x) as.numeric(isTRUE( | |
| 242 | all.equal.numeric(mean_best[[1]], x[[1]], check.attributes = FALSE)))) | |
| 243 | results_best <- results[, best] | |
| 244 | var_best <- var(results_best) | |
| 245 | which_test <- which(pvals < 1.0) | |
| 246 | for (j in which_test) { | |
| 247 | PVAL <- pvals[j] | |
| 248 | if (PVAL == 1.0) next | |
| 249 | results_j <- results[, j] | |
| 250 | # t.test may fail if the data in each group is almost constant. Hence, we | |
| 251 | # surround the call in a try() and we initialize p with 1 if the means are | |
| 252 | # equal or zero if they are different | |
| 253 | if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next | |
| 254 | # The t.test may fail if the data are not normal despite one configuration | |
| 255 | # clearly dominating the other. | |
| 256 | if (all(results_best <= results_j)) next | |
| 257 | try(PVAL <- t.test(results_best, results_j, paired = TRUE)$p.value) | |
| 258 | irace.assert(!is.nan(PVAL) & !is.na(PVAL)) | |
| 259 | pvals[j] <- PVAL | |
| 260 | } | |
| 261 | pvals <- p.adjust(pvals, method = adjust) | |
| 262 | dropj <- which.alive[pvals < 1.0 - conf.level] | |
| 263 | dropped_any <- length(dropj) > 0 | |
| 264 | irace.assert(all(alive[dropj])) | |
| 265 | alive[dropj] <- FALSE | |
| 266 | return(list(best = which.alive[best], ranks = means, alive = alive, | |
| 267 | dropped.any = dropped_any, p.value = min(pvals))) | |
| 268 | } | |
| 269 | ||
| 270 | no_elitrace.init.instances <- function(deterministic, max_instances) | |
| 245 | 271 | { |
| 246 | 272 | # if next.instance == 1 then this is the first iteration. |
| 247 | if (.irace$next.instance != 1) { | |
| 248 | new.instances <- NULL | |
| 249 | last.new <- .irace$next.instance + race.env$elitistNewInstances - 1 | |
| 250 | # Do we need to add new instances? | |
| 251 | if (race.env$elitistNewInstances > 0) { | |
| 252 | if (last.new > max.instances) { | |
| 253 | # This may happen if the scenario is deterministic and we would need | |
| 254 | # more instances than what we have. | |
| 255 | irace.assert(deterministic) | |
| 256 | if (.irace$next.instance <= max.instances) { | |
| 257 | # Add all instances that we have not seen yet as new ones. | |
| 258 | last.new <- max.instances | |
| 259 | new.instances <- .irace$next.instance : last.new | |
| 260 | } # else new.instances remains NULL and last.new remains > number of instances. | |
| 261 | # We need to update this because the value is used below and now there | |
| 262 | # may be fewer than expected, even zero. | |
| 263 | race.env$elitistNewInstances <- length(new.instances) | |
| 264 | } else { | |
| 273 | # If deterministic consider all (do not resample). | |
| 274 | if (.irace$next.instance == 1 || deterministic) return(1:max_instances) | |
| 275 | irace.assert(.irace$next.instance < max_instances) | |
| 276 | return(.irace$next.instance : max_instances) | |
| 277 | } | |
| 278 | ||
| 279 | elitrace.init.instances <- function(race.env, deterministic, max_instances, sampleInstances) | |
| 280 | { | |
| 281 | # if next.instance == 1 then this is the first iteration. | |
| 282 | if (.irace$next.instance == 1) return(1:max_instances) # Consider all | |
| 283 | ||
| 284 | new.instances <- NULL | |
| 285 | last.new <- .irace$next.instance + race.env$elitistNewInstances - 1 | |
| 286 | # Do we need to add new instances? | |
| 287 | if (race.env$elitistNewInstances > 0) { | |
| 288 | if (last.new > max_instances) { | |
| 289 | # This may happen if the scenario is deterministic and we would need | |
| 290 | # more instances than what we have. | |
| 291 | irace.assert(deterministic) | |
| 292 | if (.irace$next.instance <= max_instances) { | |
| 293 | # Add all instances that we have not seen yet as new ones. | |
| 294 | last.new <- max_instances | |
| 265 | 295 | new.instances <- .irace$next.instance : last.new |
| 266 | } | |
| 267 | } | |
| 268 | future.instances <- NULL | |
| 269 | if ((last.new + 1) <= max.instances) { | |
| 270 | future.instances <- (last.new + 1) : max.instances | |
| 271 | } | |
| 272 | # new.instances + past.instances + future.instances | |
| 273 | race.instances <- c(new.instances, sample.int(.irace$next.instance - 1), | |
| 274 | future.instances) | |
| 275 | } else { | |
| 276 | race.instances <- race.init.instances(deterministic, max.instances) | |
| 277 | } | |
| 278 | return(race.instances) | |
| 279 | } | |
| 280 | ||
| 296 | } # else new.instances remains NULL and last.new remains > number of instances. | |
| 297 | # We need to update this because the value is used below and now there | |
| 298 | # may be fewer than expected, even zero. | |
| 299 | race.env$elitistNewInstances <- length(new.instances) | |
| 300 | } else { | |
| 301 | new.instances <- .irace$next.instance : last.new | |
| 302 | } | |
| 303 | } | |
| 304 | future.instances <- NULL | |
| 305 | if ((last.new + 1) <= max_instances) { | |
| 306 | future.instances <- (last.new + 1) : max_instances | |
| 307 | } | |
| 308 | # new.instances + past.instances + future.instances | |
| 309 | # FIXME: we should sample taking into account the block-size, so we sample blocks, not instances. | |
| 310 | past_instances <- if (sampleInstances) | |
| 311 | sample.int(.irace$next.instance - 1) else | |
| 312 | 1:.irace$next.instance | |
| 313 | c(new.instances, past_instances, future.instances) | |
| 314 | } | |
| 315 | ||
| 316 | table_hline <- function(widths) { | |
| 317 | s <- "+" | |
| 318 | for(w in widths) { | |
| 319 | s <- paste0(s, strrep("-", w), "+") | |
| 320 | } | |
| 321 | return(paste0(s, "\n")) | |
| 322 | } | |
| 323 | table_sprint <- function(text, widths) { | |
| 324 | s <- "|" | |
| 325 | for (i in seq_along(text)) { | |
| 326 | s <- paste0(s, sprintf("%*s", widths[i], text[i]), "|") | |
| 327 | } | |
| 328 | return(paste0(s, "\n")) | |
| 329 | } | |
| 330 | .nocap_table_fields_width <- c(1, 11, 11, 11, 16, 11, 8, 5, 4, 6) | |
| 331 | .nocap_colum_names <- c(" ", "Instance", "Alive", "Best", "Mean best", "Exp so far", | |
| 332 | "W time", "rho", "KenW", "Qvar") | |
| 333 | .capping_table_fields_width <- c(1, 11, 8, 11, 11, 16, 11, 8, 5, 4, 6) | |
| 334 | .capping_colum_names <- c(" ", "Instance", "Bound", "Alive", "Best", "Mean best", "Exp so far", | |
| 335 | "W time", "rho", "KenW", "Qvar") | |
| 336 | capping_hline <- table_hline(.capping_table_fields_width) | |
| 337 | capping_header <- table_sprint(.capping_colum_names, .capping_table_fields_width) | |
| 338 | nocap_hline <- table_hline(.nocap_table_fields_width) | |
| 339 | nocap_header <- table_sprint(.nocap_colum_names, .nocap_table_fields_width) | |
| 340 | ||
| 341 | # FIXME: Depending on capping here is ugly. We should simply set-up the correct printing functions at the start of race(). | |
| 281 | 342 | race.print.header <- function(capping) |
| 282 | 343 | { |
| 283 | 344 | cat(sep = "", "# Markers: |
| 286 | 347 | - The test is performed and some configurations are discarded. |
| 287 | 348 | = The test is performed but no configuration is discarded. |
| 288 | 349 | ! The test is performed and configurations could be discarded but elite configurations are preserved. |
| 289 | . All alive configurations are elite and nothing is discarded\n") | |
| 290 | cat(sep = "", if (!capping) " | |
| 291 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 292 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 293 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 294 | " else " | |
| 295 | +-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 296 | | | Instance| Bound| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 297 | +-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 298 | ") | |
| 350 | . All alive configurations are elite and nothing is discarded\n\n") | |
| 351 | if (capping) | |
| 352 | cat(sep = "", capping_hline, capping_header, capping_hline) | |
| 353 | else | |
| 354 | cat(sep = "", nocap_hline, nocap_header, nocap_hline) | |
| 299 | 355 | } |
| 300 | 356 | |
| 301 | 357 | race.print.task <- function(res.symb, Results, |
| 312 | 368 | if (now <= start) return("00:00:00") |
| 313 | 369 | elapsed <- difftime(now, start, units = "secs") |
| 314 | 370 | # FIXME: Maybe better and faster if we only print seconds? |
| 315 | return(format(.POSIXct(elapsed, tz="GMT"), "%H:%M:%S")) | |
| 371 | format(.POSIXct(elapsed, tz="GMT"), "%H:%M:%S") | |
| 316 | 372 | } |
| 317 | 373 | # FIXME: This is the mean of the best, but perhaps it should |
| 318 | 374 | # be the sum of ranks in the case of test == friedman? |
| 319 | 375 | mean_best <- mean(Results[, best]) |
| 320 | 376 | time_str <- elapsed_wctime_str(Sys.time(), start.time) |
| 321 | 377 | cat(sprintf("|%s|%11d|", res.symb, instance)) |
| 322 | if (capping) cat(sprintf("%8.2f|", if(is.null(bound)) NA else bound)) | |
| 323 | cat(sprintf("%11d|%11d|%#15.10g|%11d|%s", | |
| 378 | if (capping) { | |
| 379 | if (is.null(bound)) cat(" NA|") else cat(sprintf("%8.2f|", bound)) | |
| 380 | } | |
| 381 | cat(sprintf(paste0("%11d|%11d|", .irace.format.perf, "|%11d|%s"), | |
| 324 | 382 | sum(alive), id.best, mean_best, experimentsUsed, time_str)) |
| 325 | 383 | |
| 326 | 384 | if (current.task > 1 && sum(alive) > 1) { |
| 327 | 385 | conc <- concordance(Results[1:current.task, alive, drop = FALSE]) |
| 328 | 386 | qvar <- dataVariance(Results[1:current.task, alive, drop = FALSE]) |
| 329 | cat(sprintf("|%+#4.2f|%.2f|%.4f|\n", conc$spearman.rho, conc$kendall.w, qvar)) | |
| 387 | # FIXME: We would like to use %+#4.2f but this causes problems with | |
| 388 | # https://github.com/oracle/fastr/issues/191 | |
| 389 | cat(sprintf("|%+4.2f|%.2f|%.4f|\n", conc$spearman.rho, conc$kendall.w, qvar)) | |
| 330 | 390 | } else { |
| 331 | 391 | cat("| NA| NA| NA|\n") |
| 332 | 392 | } |
| 335 | 395 | race.print.footer <- function(bestconf, mean.best, break.msg, debug.level, capping = FALSE) |
| 336 | 396 | { |
| 337 | 397 | cat(sep = "", |
| 338 | if (!capping) | |
| 339 | "+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+\n" | |
| 340 | else | |
| 341 | "+-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+\n", | |
| 398 | if (capping) capping_hline else nocap_hline, | |
| 342 | 399 | if (debug.level >= 1) paste0("# Stopped because ", break.msg, "\n"), |
| 343 | 400 | sprintf("Best-so-far configuration: %11d", bestconf[1, ".ID."]), |
| 344 | sprintf(" mean value: %#15.10g", mean.best), "\n", | |
| 401 | " mean value: ", | |
| 402 | sprintf(.irace.format.perf, mean.best), "\n", | |
| 345 | 403 | "Description of the best-so-far configuration:\n") |
| 346 | 404 | configurations.print(bestconf, metadata = TRUE) |
| 347 | 405 | cat("\n") |
| 428 | 486 | ## that exceed the maximum execution time (boundMax) |
| 429 | 487 | applyPAR <- function(results, boundMax, boundPar) |
| 430 | 488 | { |
| 489 | # We do not want to change Inf or -Inf because those represent rejection. | |
| 431 | 490 | if (boundPar != 1) |
| 432 | results[results >= boundMax] <- boundMax * boundPar | |
| 491 | results[is.finite(results) & results >= boundMax] <- boundMax * boundPar | |
| 433 | 492 | return(results) |
| 434 | 493 | } |
| 435 | 494 | |
| 442 | 501 | final.execution.bound <- function(experimentsTime, elites, no.configurations, |
| 443 | 502 | current.task, which.exe, scenario) |
| 444 | 503 | { |
| 445 | # FIXME: Make this a scenario parameter. The same parameter used in | |
| 446 | # race-wrapper when parsing target.runner output. | |
| 447 | minMeasurableTime <- 0.01 | |
| 448 | ||
| 449 | final.bounds <- rep(scenario$boundMax, no.configurations) | |
| 450 | total.time <- current.task * scenario$boundMax | |
| 451 | elite.bound <- scenario$boundMax | |
| 504 | minMeasurableTime <- scenario$minMeasurableTime | |
| 505 | boundMax <- scenario$boundMax | |
| 506 | # FIXME: should we use an adjusted boundMax | |
| 507 | final.bounds <- rep(boundMax, no.configurations) | |
| 508 | total.time <- current.task * boundMax | |
| 509 | elite.bound <- boundMax | |
| 452 | 510 | # Elite candidates can have NA values due to the rejection |
| 453 | 511 | if (length(elites) > 0) |
| 454 | 512 | elites <- elites[!is.na(experimentsTime[current.task,elites])] |
| 462 | 520 | if (scenario$boundType == "instance") { |
| 463 | 521 | elite.bound <- instanceBound(experimentsTime[current.task, elites], |
| 464 | 522 | type = scenario$cappingType) |
| 465 | final.bounds[which.exe] <- min(elite.bound + minMeasurableTime, scenario$boundMax) | |
| 523 | final.bounds[which.exe] <- min(elite.bound + minMeasurableTime, boundMax) | |
| 466 | 524 | final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) |
| 467 | 525 | } else { |
| 468 | 526 | elite.bound <- executionBound(experimentsTime[1:current.task, elites, drop = FALSE], |
| 469 | 527 | type = scenario$cappingType) |
| 470 | elite.bound <- min(elite.bound, scenario$boundMax) | |
| 528 | elite.bound <- min(elite.bound, boundMax) | |
| 471 | 529 | # FIXME: This minMeasurableTime should be a scenario setting and it |
| 472 | 530 | # should be the same value that we use in check.output.target.runner |
| 473 | 531 | total.time <- (current.task * elite.bound) + minMeasurableTime |
| 474 | 532 | time.left <- total.time - colSums(experimentsTime[1:current.task, which.exe, drop = FALSE], na.rm = TRUE) |
| 475 | final.bounds[which.exe] <- sapply(time.left, min, scenario$boundMax) | |
| 533 | final.bounds[which.exe] <- sapply(time.left, min, boundMax) | |
| 476 | 534 | # We round up the bounds up to the specified number of digits. This may |
| 477 | 535 | # be necessary if the target-algorithm does not support higher precision. |
| 478 | 536 | final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) |
| 562 | 620 | first.test <- scenario$firstTest |
| 563 | 621 | each.test <- scenario$eachTest |
| 564 | 622 | elitist <- scenario$elitist |
| 623 | capping <- scenario$capping | |
| 624 | quiet <- scenario$quiet | |
| 565 | 625 | no.configurations <- nrow(configurations) |
| 566 | 626 | experimentLog <- matrix(nrow = 0, ncol = 4, |
| 567 | 627 | dimnames = list(NULL, c("instance", "configuration", "time", "bound"))) |
| 595 | 655 | if (elitist) |
| 596 | 656 | race.instances <- elitrace.init.instances(race.env, |
| 597 | 657 | scenario$deterministic, |
| 598 | max.instances = nrow(.irace$instancesList)) | |
| 658 | max_instances = nrow(.irace$instancesList), | |
| 659 | sampleInstances = scenario$sampleInstances) | |
| 599 | 660 | else |
| 600 | race.instances <- race.init.instances(scenario$deterministic, | |
| 601 | max.instances = nrow(.irace$instancesList)) | |
| 661 | race.instances <- no_elitrace.init.instances(scenario$deterministic, | |
| 662 | max_instances = nrow(.irace$instancesList)) | |
| 602 | 663 | no.tasks <- length(race.instances) |
| 603 | 664 | |
| 604 | 665 | # Initialize some variables... |
| 630 | 691 | nrow = elite.safe, |
| 631 | 692 | ncol = no.configurations, |
| 632 | 693 | dimnames = list(elite.instances.ID, configurations.ID)) |
| 633 | if (scenario$capping) | |
| 694 | if (capping) | |
| 634 | 695 | experimentsTime <- matrix(NA, |
| 635 | 696 | nrow = elite.safe, |
| 636 | 697 | ncol = no.configurations, |
| 640 | 701 | Results[rownames(elite.data[["experiments"]]), |
| 641 | 702 | colnames(elite.data[["experiments"]])] <- elite.data[["experiments"]] |
| 642 | 703 | |
| 643 | if (scenario$capping) { | |
| 704 | if (capping) { | |
| 644 | 705 | experimentsTime[rownames(elite.data[["time"]]), |
| 645 | 706 | colnames(elite.data[["time"]])] <- elite.data[["time"]] |
| 646 | 707 | } |
| 647 | 708 | |
| 648 | 709 | # Preliminary execution of elite configurations to calculate |
| 649 | 710 | # the execution bound of initial configurations (capping only). |
| 650 | if (scenario$capping && elitistNewInstances != 0) { | |
| 711 | if (capping && elitistNewInstances != 0) { | |
| 651 | 712 | # FIXME: This should go into its own function. |
| 652 | 713 | n.elite <- ncol(elite.data[["experiments"]]) |
| 653 | 714 | which.elites <- which(rep(TRUE, n.elite)) |
| 716 | 777 | best <- 0 |
| 717 | 778 | race.ranks <- c() |
| 718 | 779 | no.elimination <- 0 # number of tasks without elimination. |
| 719 | ||
| 720 | race.print.header(scenario$capping) | |
| 780 | if (!quiet) | |
| 781 | race.print.header(capping) | |
| 721 | 782 | |
| 722 | 783 | # Test that all instances that have been previously seen have been evaluated |
| 723 | 784 | # by at least one configuration. |
| 724 | 785 | all_elite_instances_evaluated <- function() { |
| 786 | if (!elitist) | |
| 787 | return(TRUE) | |
| 725 | 788 | return(all(apply(!is.na(Results[ |
| 726 | 789 | as.character(seq_len(.irace$next.instance - 1)), |
| 727 | 790 | alive, drop=FALSE]), 1, any))) |
| 729 | 792 | |
| 730 | 793 | # Start main loop |
| 731 | 794 | break.msg <- NULL |
| 732 | for (current.task in seq_len (no.tasks)) { | |
| 795 | for (current.task in seq_len(no.tasks)) { | |
| 733 | 796 | which.alive <- which(alive) |
| 734 | 797 | nbAlive <- length(which.alive) |
| 735 | 798 | which.exe <- which.alive |
| 748 | 811 | # criterion is disabled) |
| 749 | 812 | ## MANUEL: So what is the reason to not immediately terminate here? Is |
| 750 | 813 | ## there a reason to continue? |
| 751 | race.print.task(".", Results[1:current.task, , drop = FALSE], | |
| 752 | race.instances[current.task], | |
| 753 | current.task, alive = alive, | |
| 754 | configurations[best, ".ID."], | |
| 755 | best = best, experimentsUsed, Sys.time(), | |
| 756 | bound = NA, scenario$capping) | |
| 814 | if (!quiet) | |
| 815 | race.print.task(".", Results[1:current.task, , drop = FALSE], | |
| 816 | race.instances[current.task], | |
| 817 | current.task, alive = alive, | |
| 818 | configurations[best, ".ID."], | |
| 819 | best = best, experimentsUsed, Sys.time(), | |
| 820 | bound = NA, capping) | |
| 757 | 821 | next |
| 758 | 822 | } |
| 759 | 823 | } |
| 767 | 831 | ## there are instances previously seen that have not been evaluated on any |
| 768 | 832 | ## alive configuration. |
| 769 | 833 | if (current.task > first.test) { |
| 770 | #if ((current.task > first.test && !scenario$capping) | |
| 834 | #if ((current.task > first.test && !capping) | |
| 771 | 835 | # MANUEL: This is new and I'm not sure what it does. |
| 772 | 836 | # LESLIE: When using capping, we dont finish any race until all |
| 773 | 837 | # previous instances have been executed (this makes sure that all non-elite |
| 774 | 838 | # configurations execute all the previous instances) |
| 775 | # || (scenario$capping && (current.task > elite.safe))) { | |
| 839 | # || (capping && (current.task > elite.safe))) { | |
| 776 | 840 | # MANUEL: How is this even possible? |
| 777 | 841 | # LESLIE: It can be that the capping eliminate all but one configuration |
| 778 | 842 | # (which should be an elite one) after we finish the new instances to be evaluated, |
| 825 | 889 | if (nrow(Results) < current.task) { |
| 826 | 890 | Results <- rbind(Results, rep(NA, ncol(Results))) |
| 827 | 891 | rownames(Results) <- race.instances[1:nrow(Results)] |
| 828 | if (scenario$capping) { | |
| 892 | if (capping) { | |
| 829 | 893 | experimentsTime <- rbind(experimentsTime, rep(NA, ncol(experimentsTime))) |
| 830 | 894 | rownames(experimentsTime) <- race.instances[1:nrow(experimentsTime)] |
| 831 | 895 | } |
| 838 | 902 | # Calculate bounds for executing if needed. |
| 839 | 903 | which.elite.exe <- intersect(which.exe, which(is.elite > 0)) |
| 840 | 904 | irace.assert(setequal(which.elite.exe, which(is.elite & is.na(Results[current.task,])))) |
| 841 | if (scenario$capping) { | |
| 905 | if (capping) { | |
| 842 | 906 | # Pre-execute elite configurations that are not yet executed in the current instance. |
| 843 | 907 | if (length(which.elite.exe)) { |
| 844 | 908 | # FIXME: This should go into its own function |
| 883 | 947 | elite.safe <- update.elite.safe(Results, is.elite) |
| 884 | 948 | } |
| 885 | 949 | which.exe <- setdiff(which.exe, which.elite.exe) |
| 950 | # FIXME: There is similar code above. | |
| 886 | 951 | if (length(which.exe) == 0L) { |
| 887 | 952 | is.elite <- update.is.elite(is.elite, which.elite.exe) |
| 888 | race.print.task(".", Results[1:current.task, , drop = FALSE], | |
| 889 | race.instances[current.task], | |
| 890 | current.task, alive = alive, | |
| 891 | configurations[best, ".ID."], | |
| 892 | best = best, experimentsUsed, start.time, | |
| 893 | bound = NA, scenario$capping) | |
| 953 | if (!quiet) | |
| 954 | race.print.task(".", Results[1:current.task, , drop = FALSE], | |
| 955 | race.instances[current.task], | |
| 956 | current.task, alive = alive, | |
| 957 | configurations[best, ".ID."], | |
| 958 | best = best, experimentsUsed, start.time, | |
| 959 | bound = NA, capping) | |
| 894 | 960 | next |
| 895 | 961 | } |
| 896 | 962 | } |
| 924 | 990 | irace.assert(length(vcost) == length(which.exps)) |
| 925 | 991 | # Set max execution bound to timed out executions which have execution |
| 926 | 992 | # times smaller than boundMax and implement parX if required |
| 927 | if (scenario$capping) { | |
| 993 | if (capping) { | |
| 928 | 994 | vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) |
| 929 | 995 | if (scenario$boundAsTimeout) |
| 930 | 996 | vcost[(vcost >= final.bounds[which.exps]) & (vcost < scenario$boundMax)] <- scenario$boundMax |
| 936 | 1002 | irace.assert(length(which.exps) == length(which.exe)) |
| 937 | 1003 | vtimes <- unlist(lapply(output[which.exps], "[[", "time")) |
| 938 | 1004 | irace.assert(length(vtimes) == length(which.exe)) |
| 939 | if (scenario$capping) { | |
| 1005 | if (capping) { | |
| 940 | 1006 | # Correct higher execution times. |
| 941 | 1007 | experimentsTime[current.task, which.exps] <- pmin(vtimes, final.bounds[which.exps]) |
| 942 | 1008 | } |
| 980 | 1046 | ## Dominance elimination (Capping only). |
| 981 | 1047 | # The second condition can be false if we eliminated via immediate |
| 982 | 1048 | # rejection |
| 983 | if (scenario$capping && sum(alive) > minSurvival) { | |
| 1049 | if (capping && sum(alive) > minSurvival) { | |
| 984 | 1050 | irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) |
| 985 | 1051 | cap.alive <- dom.elim(Results[1:current.task, , drop = FALSE], |
| 986 | 1052 | # Get current elite configurations |
| 1024 | 1090 | |
| 1025 | 1091 | # It may happen that the capping and the test eliminate together all |
| 1026 | 1092 | # configurations. In that case, we only trust the capping elimination. |
| 1027 | if (scenario$capping && !any(alive)) { | |
| 1093 | if (capping && !any(alive)) { | |
| 1028 | 1094 | if (scenario$debugLevel >= 2) { |
| 1029 | irace.note("Warning: Elimination tests have eliminated all configurations, keeping the capping results.\n") | |
| 1095 | irace.warning("Elimination tests have eliminated all configurations, keeping the capping results.\n") | |
| 1030 | 1096 | irace.note("Alive according to capping:", which(cap.alive), "\n") |
| 1031 | 1097 | irace.note("Alive according to test:", which(test.alive), "\n") |
| 1032 | 1098 | } |
| 1068 | 1134 | # Remove the ranks of those that are not alive anymore |
| 1069 | 1135 | race.ranks <- race.ranks[which.alive] |
| 1070 | 1136 | irace.assert(length(race.ranks) == sum(alive)) |
| 1071 | ||
| 1072 | race.print.task(res.symb, Results[1:current.task, , drop = FALSE], | |
| 1073 | race.instances[current.task], | |
| 1074 | current.task, alive = alive, | |
| 1075 | configurations[best, ".ID."], | |
| 1076 | best = best, experimentsUsed, start.time, | |
| 1077 | bound = elite.bound, scenario$capping) | |
| 1078 | ||
| 1137 | if (!quiet) | |
| 1138 | race.print.task(res.symb, Results[1:current.task, , drop = FALSE], | |
| 1139 | race.instances[current.task], | |
| 1140 | current.task, alive = alive, | |
| 1141 | configurations[best, ".ID."], | |
| 1142 | best = best, experimentsUsed, start.time, | |
| 1143 | bound = elite.bound, capping) | |
| 1144 | ||
| 1079 | 1145 | if (elitist) { |
| 1080 | 1146 | # Compute number of statistical tests without eliminations. |
| 1081 | 1147 | irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) |
| 1090 | 1156 | } |
| 1091 | 1157 | } |
| 1092 | 1158 | |
| 1093 | # Adding this given that when ncandidates = minsuvival+1 | |
| 1159 | if (is.null(break.msg)) | |
| 1160 | break.msg <- paste0("all instances (", no.tasks, ") evaluated") | |
| 1161 | ||
| 1162 | # Adding this given that when ncandidates = minsurvival+1 | |
| 1094 | 1163 | # and there one elite configuration that gets discarded in the new instances |
| 1095 | 1164 | # execution the race is finished with no executions. |
| 1096 | 1165 | # FIXME: we should handle this better, maybe allowing irace to handle no elite |
| 1097 | 1166 | # in irace() |
| 1098 | # MANUEL: I still don't understand how we reach this error if there are non-rejected elites. | |
| 1099 | if (current.task == 1 && !any(is.elite > 0)) | |
| 1167 | # MANUEL: Leslie, how can we reach this error in normal circumstances? | |
| 1168 | # Can we handle this better? | |
| 1169 | if (current.task == 1 && !any(is.elite > 0)) | |
| 1100 | 1170 | irace.error ("Maximum number configurations immediately rejected reached!") |
| 1101 | ||
| 1102 | if (is.null(break.msg)) | |
| 1103 | break.msg <- paste0("all instances (", no.tasks, ") evaluated") | |
| 1104 | ||
| 1171 | ||
| 1105 | 1172 | # All instances that are not new in this race must have been evaluated by at |
| 1106 | 1173 | # least one configuration. |
| 1107 | 1174 | irace.assert(all_elite_instances_evaluated(), |
| 1113 | 1180 | race.ranks <- overall.ranks(Results[, alive, drop = FALSE], stat.test = stat.test) |
| 1114 | 1181 | best <- which.alive[which.min(race.ranks)] |
| 1115 | 1182 | |
| 1116 | race.print.footer(bestconf = configurations[best, , drop = FALSE], | |
| 1117 | # FIXME: This is the mean of the best, but perhaps it | |
| 1118 | # should be the sum of ranks in the case of test == | |
| 1119 | # friedman? | |
| 1120 | mean.best = mean(Results[, best]), | |
| 1121 | break.msg = break.msg, debug.level = scenario$debugLevel, | |
| 1122 | capping = scenario$capping) | |
| 1123 | ||
| 1183 | if (!quiet) | |
| 1184 | race.print.footer(bestconf = configurations[best, , drop = FALSE], | |
| 1185 | # FIXME: This is the mean of the best, but perhaps it | |
| 1186 | # should be the sum of ranks in the case of test == | |
| 1187 | # friedman? | |
| 1188 | mean.best = mean(Results[, best]), | |
| 1189 | break.msg = break.msg, debug.level = scenario$debugLevel, | |
| 1190 | capping = capping) | |
| 1191 | ||
| 1124 | 1192 | nbAlive <- sum(alive) |
| 1125 | 1193 | configurations$.ALIVE. <- as.logical(alive) |
| 1126 | 1194 | # Assign the proper ranks in the configurations data.frame. |
| 0 | #' readConfigurationsFile | |
| 1 | #' | |
| 2 | #' `readConfigurationsFile` reads a set of target algorithms configurations | |
| 3 | #' from a file and puts them in \pkg{irace} format. The configurations are checked | |
| 4 | #' to match the parameters description provided. | |
| 5 | #' | |
| 6 | #' @param filename (`character(1)`) \cr Filename from which the configurations should be read. | |
| 0 | #' Read parameter configurations from a file | |
| 1 | #' | |
| 2 | #' Reads a set of target-algorithm configurations from a file and puts them in | |
| 3 | #' \pkg{irace} format. The configurations are checked to match the parameters | |
| 4 | #' description provided. | |
| 5 | #' | |
| 6 | #' @param filename (`character(1)`) \cr Filename from which the configurations should be read. The contents should be readable by `read.table( , header=TRUE)`. | |
| 7 | 7 | #' @template arg_parameters |
| 8 | 8 | #' @template arg_debuglevel |
| 9 | 9 | #' @template arg_text |
| 12 | 12 | #' Each row of the data frame is a candidate configuration, |
| 13 | 13 | #' the columns correspond to the parameter names in `parameters`. |
| 14 | 14 | #' |
| 15 | #' @details | |
| 16 | #' Example of an input file: | |
| 17 | #' ``` | |
| 18 | #' # This is a comment line | |
| 19 | #' <param_name_1> <param_name_2> ... | |
| 20 | #' 0.5 "value_1" ... | |
| 21 | #' 1.0 "value_2" ... | |
| 22 | #' 1.2 "value_3" ... | |
| 23 | #' ... ... | |
| 24 | #' ``` | |
| 25 | #' The order of the columns does not necessarily have to be the same | |
| 26 | #' as in the file containing the definition of the parameters. | |
| 27 | #' | |
| 15 | 28 | #' @seealso |
| 16 | #' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters list. | |
| 29 | #' [readParameters()] to obtain a valid parameter structure from a parameters file. | |
| 17 | 30 | #' |
| 18 | 31 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 19 | #' @md | |
| 20 | 32 | #' @export |
| 21 | ## Read some configurations from a file. | |
| 22 | ## Example of an input file, | |
| 23 | ## it should be readable with read.table( , header=TRUE). | |
| 24 | ## ------------------------------- | |
| 25 | ## <param_name_1> <param_name_2> | |
| 26 | ## 1 "value_1" | |
| 27 | ## 2 "value_2" | |
| 28 | ## # This is a comment line | |
| 29 | ## 3 "value_3" | |
| 30 | ## ------------------------------- | |
| 31 | ## The order of the columns does not necessarily have to be the same | |
| 32 | ## as in the file containing the definition of the parameters. | |
| 33 | ## | |
| 34 | 33 | ## FIXME: What about digits? |
| 35 | 34 | readConfigurationsFile <- function(filename, parameters, debugLevel = 0, text) |
| 36 | 35 | { |
| 50 | 49 | # Print the table that has been read. |
| 51 | 50 | cat("# Read ", nbConfigurations, " configuration(s) from file '", filename, "'\n", sep="") |
| 52 | 51 | if (debugLevel >= 2) { |
| 53 | print(as.data.frame(configurationTable, stringAsFactor = FALSE), digits=15) | |
| 52 | print(as.data.frame(configurationTable, stringsAsFactor = FALSE), digits=15) | |
| 54 | 53 | } |
| 55 | 54 | |
| 56 | 55 | namesParameters <- names(parameters$conditions) |
| 156 | 155 | } |
| 157 | 156 | } |
| 158 | 157 | } |
| 159 | return (configurationTable) | |
| 158 | if (anyDuplicated(configurationTable)) { | |
| 159 | irace.error("Duplicated configurations in file ", filename, " :\n", | |
| 160 | paste0(capture.output( | |
| 161 | configurationTable[duplicated(configurationTable), , drop=FALSE]), "\n")) | |
| 162 | } | |
| 163 | configurationTable | |
| 160 | 164 | } |
| 161 | 165 | # FIXME: It may be faster to create a single expression that concatenates all |
| 162 | 166 | # the elements of forbidden using '|' |
| 173 | 177 | ## would be faster to break as soon as nrow(configurations) < 1 |
| 174 | 178 | } |
| 175 | 179 | #print(nrow(configurations)) |
| 176 | return(configurations) | |
| 180 | configurations | |
| 177 | 181 | } |
| 178 | 182 | |
| 179 | 183 | compile.forbidden <- function(x) |
| 188 | 192 | |
| 189 | 193 | # We expect that there will be undefined variables, since the expressions |
| 190 | 194 | # will be evaluated within a data.frame later. |
| 191 | exp <- compiler::compile(substitute(is.na(x) | !(x), list(x = x)), | |
| 195 | expr <- compiler::compile(substitute(is.na(x) | !(x), list(x = x)), | |
| 192 | 196 | options = list(suppressUndefined=TRUE)) |
| 193 | attr(exp, "source") <- as.character(as.expression(x)) | |
| 194 | return(exp) | |
| 197 | attr(expr, "source") <- as.character(as.expression(x)) | |
| 198 | expr | |
| 195 | 199 | } |
| 196 | 200 | |
| 197 | 201 | readForbiddenFile <- function(filename) |
| 205 | 209 | |
| 206 | 210 | # FIXME: Instead of a list, we should generate a single expression that is |
| 207 | 211 | # the logical-OR of all elements of the list. |
| 208 | ||
| 212 | # First we would need to handle the "is.na(x) | !(x)" case here. | |
| 213 | # Maybe: sapply(forbiddenExps, function(x) substitute(is.na(x) | !(x), list(x=x))) | |
| 214 | # x <- parse(text=paste0("(", paste0(forbiddenExps,collapse=")||("), ")")) | |
| 209 | 215 | # Byte-compile them. |
| 210 | return(sapply(forbiddenExps, compile.forbidden)) | |
| 216 | sapply(forbiddenExps, compile.forbidden) | |
| 211 | 217 | } |
| 212 | 218 | |
| 213 | 219 | buildForbiddenExp <- function(configurations, parameters) |
| 226 | 232 | } |
| 227 | 233 | exps <- parse(text = lines) |
| 228 | 234 | # print(exps) |
| 229 | return(sapply(exps, compile.forbidden)) | |
| 235 | sapply(exps, compile.forbidden) | |
| 230 | 236 | } |
| 231 | 237 | |
| 232 | #' readScenario | |
| 233 | #' | |
| 234 | #' `readScenario` reads from a file the scenario settings to be used by | |
| 235 | #' \pkg{irace}.. | |
| 238 | #' Reads from a file the scenario settings to be used by \pkg{irace}. | |
| 236 | 239 | #' |
| 237 | 240 | #' @param filename (`character(1)`) \cr Filename from which the scenario will |
| 238 | 241 | #' be read. If empty, the default `scenarioFile` is used. An example |
| 241 | 244 | #' @templateVar arg_appendix This is an initial scenario that is overwritten |
| 242 | 245 | #' for every setting specified in the file to be read. |
| 243 | 246 | #' @template arg_scenario |
| 244 | #' | |
| 247 | #' @template arg_params_def | |
| 248 | #' | |
| 245 | 249 | #' @return The scenario list read from the file. The scenario settings not |
| 246 | 250 | #' present in the file are not present in the list, i.e., they are `NULL`. |
| 247 | 251 | #' |
| 248 | 252 | #' @seealso |
| 249 | 253 | #' \describe{ |
| 250 | #' \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 251 | #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 252 | #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 254 | #' \item{[printScenario()]}{prints the given scenario.} | |
| 255 | #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} | |
| 256 | #' \item{[checkScenario()]}{to check that the scenario is valid.} | |
| 253 | 257 | #' } |
| 254 | 258 | #' |
| 255 | 259 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 256 | #' @md | |
| 257 | 260 | #' @export |
| 258 | readScenario <- function(filename = "", scenario = list()) | |
| 261 | readScenario <- function(filename = "", scenario = list(), | |
| 262 | params_def = .irace.params.def) | |
| 259 | 263 | { |
| 260 | 264 | # This function allows recursively including scenario files. |
| 261 | envir <- environment() | |
| 262 | include.scenario <- function(rfilename, topfile = filename, envir. = envir) | |
| 265 | scenario_env <- new.env() | |
| 266 | include.scenario <- function(rfilename, topfile = filename, envir. = scenario_env) | |
| 263 | 267 | { |
| 264 | 268 | if (!file.exists (rfilename)) { |
| 265 | 269 | irace.error ("The scenario file ", shQuote(rfilename), " included from ", |
| 279 | 283 | |
| 280 | 284 | # First find out which file... |
| 281 | 285 | if (filename == "") { |
| 282 | filename <- .irace.params.def["scenarioFile","default"] | |
| 286 | filename <- path_rel2abs(params_def["scenarioFile","default"]) | |
| 283 | 287 | if (file.exists(filename)) { |
| 284 | cat("Warning: A default scenario file", shQuote(filename), | |
| 285 | "has been found and will be read\n") | |
| 288 | irace.warning("A default scenario file ", shQuote(filename), | |
| 289 | " has been found and will be read.") | |
| 286 | 290 | } else { |
| 287 | 291 | irace.error ("Not scenario file given (use ", |
| 288 | .irace.params.def["scenarioFile", "short"], " or ", | |
| 289 | .irace.params.def["scenarioFile", "long"], | |
| 292 | params_def["scenarioFile", "short"], " or ", | |
| 293 | params_def["scenarioFile", "long"], | |
| 290 | 294 | ") and no default scenario file ", shQuote(filename), |
| 291 | 295 | " has been found.") |
| 292 | 296 | } |
| 293 | } | |
| 297 | } else { | |
| 298 | filename <- path_rel2abs(filename) | |
| 299 | } | |
| 300 | ||
| 294 | 301 | if (file.exists (filename)) { |
| 295 | 302 | debug.level <- getOption(".irace.debug.level", default = 0) |
| 296 | 303 | if (debug.level >= 1) |
| 297 | cat ("# Reading scenario file", shQuote(filename), ".......") | |
| 304 | cat("# Reading scenario file", shQuote(filename), ".......") | |
| 298 | 305 | # chdir = TRUE to allow recursive sourcing. |
| 299 | 306 | handle.source.error <- function(e) { |
| 300 | 307 | irace.error("Reading scenario file ", shQuote(filename), |
| 303 | 310 | return(NULL) |
| 304 | 311 | } |
| 305 | 312 | withCallingHandlers( |
| 306 | tryCatch(source(filename, local = TRUE, chdir = TRUE), | |
| 313 | tryCatch(source(filename, local = scenario_env, chdir = TRUE), | |
| 307 | 314 | error = handle.source.error, warning = handle.source.error)) |
| 308 | 315 | if (debug.level >= 1) cat (" done!\n") |
| 309 | 316 | } else { |
| 310 | 317 | irace.error ("The scenario file ", shQuote(filename), " does not exist.") |
| 311 | 318 | } |
| 319 | ||
| 312 | 320 | ## Read scenario file variables. |
| 321 | scenario[["scenarioFile"]] <- filename | |
| 313 | 322 | # If these are given and relative, they should be relative to the |
| 314 | 323 | # scenario file (except logFile, which is relative to execDir). |
| 315 | pathParams <- setdiff(.irace.params.def[.irace.params.def[, "type"] == "p", | |
| 324 | pathParams <- setdiff(params_def[params_def[, "type"] == "p", | |
| 316 | 325 | "name"], "logFile") |
| 317 | for (param in .irace.params.names) { | |
| 318 | if (exists (param, inherits = FALSE)) { | |
| 319 | value <- get(param, inherits = FALSE) | |
| 326 | params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] | |
| 327 | ||
| 328 | for (param in params_names) { | |
| 329 | if (exists (param, envir = scenario_env, inherits = FALSE)) { | |
| 330 | value <- get(param, envir = scenario_env, inherits = FALSE) | |
| 320 | 331 | if (!is.null.or.empty(value) && is.character(value) |
| 321 | 332 | && (param %in% pathParams)) { |
| 322 | value <- path.rel2abs(value, cwd = dirname(filename)) | |
| 333 | value <- path_rel2abs(value, cwd = dirname(filename)) | |
| 323 | 334 | } |
| 324 | 335 | scenario[[param]] <- value |
| 325 | 336 | } |
| 326 | 337 | } |
| 327 | return (scenario) | |
| 338 | unknown_scenario_vars <- setdiff(ls(scenario_env), params_names) | |
| 339 | if (length(unknown_scenario_vars) > 0) { | |
| 340 | # We only accept variables that match irace.params.names and if the user | |
| 341 | # wants to define their own, they should use names starting with ".", which | |
| 342 | # are ignored by ls() | |
| 343 | irace.error("Scenario file ", shQuote(filename), " contains unknown variables: ", | |
| 344 | paste0(unknown_scenario_vars, collapse=", "), | |
| 345 | "\nMAKE SURE NO VARIABLE NAME IS MISSPELL (for example, 'parameterFile' is correct, while 'parametersFile' is not)", | |
| 346 | "\nIf you wish to use your own variables in the scenario file, use names beginning with a dot `.'") | |
| 347 | } | |
| 348 | scenario | |
| 328 | 349 | } |
| 350 | ||
| 351 | setup_test_instances <- function(scenario) | |
| 352 | { | |
| 353 | if (is.null.or.empty(scenario[["testInstances"]])) { | |
| 354 | if (!is.null.or.empty(scenario$testInstancesDir) || | |
| 355 | !is.null.or.empty(scenario$testInstancesFile)) { | |
| 356 | scenario$testInstancesDir <- path_rel2abs(scenario$testInstancesDir) | |
| 357 | if (!is.null.or.empty(scenario$testInstancesFile)) { | |
| 358 | scenario$testInstancesFile <- path_rel2abs(scenario$testInstancesFile) | |
| 359 | } | |
| 360 | scenario[["testInstances"]] <- | |
| 361 | readInstances(instancesDir = scenario$testInstancesDir, | |
| 362 | instancesFile = scenario$testInstancesFile) | |
| 363 | } else { | |
| 364 | scenario[["testInstances"]] <- NULL | |
| 365 | } | |
| 366 | } | |
| 367 | if (!is.null(scenario[["testInstances"]]) | |
| 368 | && is.null(names(scenario[["testInstances"]]))) { | |
| 369 | # Create unique IDs for testInstances | |
| 370 | names(scenario[["testInstances"]]) <- paste0(1:length(scenario[["testInstances"]]), "t") | |
| 371 | } | |
| 372 | scenario | |
| 373 | } | |
| 329 | 374 | |
| 330 | 375 | #' Check and correct the given scenario |
| 331 | 376 | #' |
| 332 | #' `checkScenario` takes a (possibly incomplete) scenario setup of | |
| 333 | #' \pkg{irace}, checks for errors and transforms it into a valid scenario. | |
| 377 | #' Checks for errors a (possibly incomplete) scenario setup of | |
| 378 | #' \pkg{irace} and transforms it into a valid scenario. | |
| 334 | 379 | #' |
| 335 | 380 | #' @template arg_scenario |
| 336 | 381 | #' |
| 346 | 391 | #' |
| 347 | 392 | #' @seealso |
| 348 | 393 | #' \describe{ |
| 349 | #' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 350 | #' \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 351 | #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 352 | #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 394 | #' \item{[readScenario()]}{for reading a configuration scenario from a file.} | |
| 395 | #' \item{[printScenario()]}{prints the given scenario.} | |
| 396 | #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} | |
| 397 | #' \item{[checkScenario()]}{to check that the scenario is valid.} | |
| 353 | 398 | #' } |
| 354 | 399 | #' |
| 355 | 400 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 356 | #' @md | |
| 357 | 401 | #' @export |
| 358 | 402 | ## FIXME: This function should only do checks and return TRUE/FALSE. There |
| 359 | 403 | ## should be other function that does the various transformations. |
| 361 | 405 | { |
| 362 | 406 | quote.param <- function(name) |
| 363 | 407 | { |
| 364 | return(paste0("'", name, "' (", .irace.params.def[name, "long"], ")")) | |
| 408 | if (.irace.params.def[name, "long"] != "") { | |
| 409 | return(paste0("'", name, "' (", .irace.params.def[name, "long"], ")")) | |
| 410 | } | |
| 411 | paste0("'", name, "'") | |
| 365 | 412 | } |
| 366 | 413 | |
| 367 | 414 | as.boolean.param <- function(x, name) |
| 370 | 417 | if (is.na (x) || (x != 0 && x != 1)) { |
| 371 | 418 | irace.error (quote.param(name), " must be either 0 or 1.") |
| 372 | 419 | } |
| 373 | return(as.logical(x)) | |
| 374 | } | |
| 375 | ||
| 376 | check.valid.param <- function(x, valid) | |
| 420 | as.logical(x) | |
| 421 | } | |
| 422 | ||
| 423 | check.valid.param <- function(x) | |
| 377 | 424 | { |
| 425 | cat(x, "\n") | |
| 426 | cat(.irace.params.def[x, "domain"], "\n") | |
| 427 | valid <- trimws(strsplit(.irace.params.def[x, "domain"],",",fixed=TRUE)[[1]]) | |
| 378 | 428 | if (scenario[[x]] %!in% valid) { |
| 379 | 429 | irace.error ("Invalid value '", scenario[[x]], "' of ", |
| 380 | 430 | quote.param(x), ", valid values are: ", |
| 381 | 431 | paste0(valid, collapse = ", ")) |
| 382 | 432 | } |
| 383 | 433 | } |
| 384 | ||
| 385 | ||
| 434 | ||
| 386 | 435 | # Fill possible unset (NULL) with default settings. |
| 387 | scenario <- defaultScenario (scenario) | |
| 436 | scenario <- defaultScenario(scenario) | |
| 388 | 437 | |
| 389 | 438 | # Duplicated entries will cause confusion. |
| 390 | 439 | dups <- anyDuplicated(names(scenario)) |
| 396 | 445 | for (p in boolParams) { |
| 397 | 446 | scenario[[p]] <- as.boolean.param (scenario[[p]], p) |
| 398 | 447 | } |
| 399 | ||
| 448 | options(.irace.quiet = scenario$quiet) | |
| 400 | 449 | ## Check that everything is fine with external parameters |
| 401 | 450 | # Check that the files exist and are readable. |
| 402 | scenario$parameterFile <- path.rel2abs(scenario$parameterFile) | |
| 451 | scenario$parameterFile <- path_rel2abs(scenario$parameterFile) | |
| 403 | 452 | # We don't read parameterFile here because the user may give the parameters |
| 404 | 453 | # explicitly. And it is validated in readParameters anyway. |
| 405 | scenario$execDir <- path.rel2abs(scenario$execDir) | |
| 454 | scenario$execDir <- path_rel2abs(scenario$execDir) | |
| 406 | 455 | file.check (scenario$execDir, isdir = TRUE, |
| 407 | 456 | text = paste0("execution directory ", quote.param("execDir"))) |
| 408 | 457 | options(.irace.execdir = scenario$execDir) |
| 409 | 458 | if (!is.null.or.empty(scenario$logFile)) { |
| 410 | scenario$logFile <- path.rel2abs(scenario$logFile, cwd = scenario$execDir) | |
| 411 | file.check(scenario$logFile, writeable = TRUE, | |
| 412 | text = quote.param('logFile')) | |
| 459 | scenario$logFile <- path_rel2abs(scenario$logFile, cwd = scenario$execDir) | |
| 460 | file.check(scenario$logFile, writeable = TRUE, text = quote.param('logFile')) | |
| 413 | 461 | } else { |
| 414 | scenario$logFile <- NULL | |
| 462 | # We cannot use NULL because defaultScenario() would override it. | |
| 463 | scenario$logFile <- "" | |
| 415 | 464 | } |
| 416 | 465 | |
| 417 | 466 | if (!is.null.or.empty(scenario$recoveryFile)) { |
| 418 | scenario$recoveryFile <- path.rel2abs(scenario$recoveryFile) | |
| 467 | scenario$recoveryFile <- path_rel2abs(scenario$recoveryFile) | |
| 419 | 468 | file.check(scenario$recoveryFile, readable = TRUE, |
| 420 | 469 | text = paste0("recovery file ", quote.param("recoveryFile"))) |
| 421 | 470 | |
| 422 | if (!is.null.or.empty(scenario$logFile) | |
| 471 | if (!is.null.or.empty(scenario$logFile) # Must have been set "" above. | |
| 423 | 472 | && scenario$recoveryFile == scenario$logFile) { |
| 424 | irace.error("log file and recovery file should be different ('", | |
| 473 | irace.error("log file and recovery file should be different '", | |
| 425 | 474 | scenario$logFile, "'") |
| 426 | 475 | } |
| 427 | 476 | } else { |
| 428 | scenario$recoveryFile <- NULL | |
| 477 | # We cannot use NULL because defaultScenario() would override it. | |
| 478 | scenario$recoveryFile <- "" | |
| 429 | 479 | } |
| 430 | 480 | |
| 431 | 481 | if (is.null.or.empty(scenario$targetRunnerParallel)) { |
| 450 | 500 | .irace$target.runner <- bytecompile(scenario$targetRunner) |
| 451 | 501 | } else if (is.null(scenario$targetRunnerParallel)) { |
| 452 | 502 | if (is.character(scenario$targetRunner)) { |
| 453 | scenario$targetRunner <- path.rel2abs(scenario$targetRunner) | |
| 454 | file.check (scenario$targetRunner, executable = TRUE, | |
| 455 | text = paste0("target runner ", quote.param("targetRunner"))) | |
| 503 | scenario$targetRunner <- path_rel2abs(scenario$targetRunner) | |
| 456 | 504 | .irace$target.runner <- if (scenario$aclib) |
| 457 | 505 | target.runner.aclib else target.runner.default |
| 506 | if (is.null.or.empty(scenario$targetRunnerLauncher)) { | |
| 507 | file.check (scenario$targetRunner, executable = TRUE, | |
| 508 | text = paste0("target runner ", quote.param("targetRunner"))) | |
| 509 | } else { | |
| 510 | scenario$targetRunnerLauncher <- path_rel2abs(scenario$targetRunnerLauncher) | |
| 511 | file.check (scenario$targetRunnerLauncher, executable = TRUE, | |
| 512 | text = paste0("target runner launcher ", quote.param("targetRunnerLauncher"))) | |
| 513 | check_launcher_args(scenario$targetRunnerLauncherArgs) | |
| 514 | } | |
| 458 | 515 | } else { |
| 459 | 516 | irace.error(quote.param ('targetRunner'), " must be a function or an executable program") |
| 460 | 517 | } |
| 467 | 524 | scenario$targetEvaluator <- get.function(scenario$targetEvaluator) |
| 468 | 525 | .irace$target.evaluator <- bytecompile(scenario$targetEvaluator) |
| 469 | 526 | } else if (is.character(scenario$targetEvaluator)) { |
| 470 | scenario$targetEvaluator <- path.rel2abs(scenario$targetEvaluator) | |
| 527 | scenario$targetEvaluator <- path_rel2abs(scenario$targetEvaluator) | |
| 471 | 528 | file.check (scenario$targetEvaluator, executable = TRUE, |
| 472 | 529 | text = "target evaluator") |
| 473 | 530 | .irace$target.evaluator <- target.evaluator.default |
| 474 | 531 | } else { |
| 475 | irace.error("'targetEvaluator' must be a function or an executable program") | |
| 532 | irace.error(quote.param('targetEvaluator'), " must be a function or an executable program") | |
| 476 | 533 | } |
| 477 | 534 | |
| 478 | 535 | irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) |
| 479 | 536 | |
| 480 | 537 | # Training instances |
| 481 | 538 | if (is.null.or.empty(scenario$instances)) { |
| 482 | scenario$trainInstancesDir <- path.rel2abs(scenario$trainInstancesDir) | |
| 539 | scenario$trainInstancesDir <- path_rel2abs(scenario$trainInstancesDir) | |
| 483 | 540 | if (!is.null.or.empty(scenario$trainInstancesFile)) { |
| 484 | scenario$trainInstancesFile <- path.rel2abs(scenario$trainInstancesFile) | |
| 541 | scenario$trainInstancesFile <- path_rel2abs(scenario$trainInstancesFile) | |
| 485 | 542 | } |
| 486 | 543 | if (is.null.or.empty(scenario$trainInstancesDir) |
| 487 | 544 | && is.null.or.empty(scenario$trainInstancesFile)) |
| 495 | 552 | } |
| 496 | 553 | |
| 497 | 554 | # Testing instances |
| 498 | if (is.null.or.empty(scenario$testInstances)) { | |
| 499 | if (!is.null.or.empty(scenario$testInstancesDir) || | |
| 500 | !is.null.or.empty(scenario$testInstancesFile)) { | |
| 501 | scenario$testInstancesDir <- path.rel2abs(scenario$testInstancesDir) | |
| 502 | if (!is.null.or.empty(scenario$testInstancesFile)) { | |
| 503 | scenario$testInstancesFile <- path.rel2abs(scenario$testInstancesFile) | |
| 504 | } | |
| 505 | scenario$testInstances <- | |
| 506 | readInstances(instancesDir = scenario$testInstancesDir, | |
| 507 | instancesFile = scenario$testInstancesFile) | |
| 508 | } else { | |
| 509 | scenario$testInstances <- NULL | |
| 510 | } | |
| 511 | } | |
| 512 | if (!is.null(scenario$testInstances) | |
| 513 | && is.null(names(scenario$testInstances))) { | |
| 514 | # Create unique IDs for testInstances | |
| 515 | names(scenario$testInstances) <- paste0(1:length(scenario$testInstances), "t") | |
| 516 | } | |
| 517 | ||
| 555 | scenario <- setup_test_instances(scenario) | |
| 556 | ||
| 518 | 557 | # Configurations file |
| 519 | 558 | if (!is.null.or.empty(scenario$configurationsFile)) { |
| 520 | scenario$configurationsFile <- path.rel2abs(scenario$configurationsFile) | |
| 559 | scenario$configurationsFile <- path_rel2abs(scenario$configurationsFile) | |
| 521 | 560 | file.check (scenario$configurationsFile, readable = TRUE, |
| 522 | 561 | text = "configurations file") |
| 523 | 562 | # We cannot read the configurations here because we need the parameters. |
| 534 | 573 | # the user specified them explicitly. |
| 535 | 574 | if (is.null.or.empty(scenario$forbiddenExps) |
| 536 | 575 | && !is.null.or.empty(scenario$forbiddenFile)) { |
| 537 | scenario$forbiddenFile <- path.rel2abs(scenario$forbiddenFile) | |
| 576 | scenario$forbiddenFile <- path_rel2abs(scenario$forbiddenFile) | |
| 538 | 577 | file.check (scenario$forbiddenFile, readable = TRUE, |
| 539 | 578 | text = "forbidden configurations file") |
| 540 | 579 | scenario$forbiddenExps <- readForbiddenFile(scenario$forbiddenFile) |
| 541 | cat("# ", length(scenario$forbiddenExps), | |
| 542 | " expression(s) specifying forbidden configurations read from '", | |
| 543 | scenario$forbiddenFile, "'\n", sep = "") | |
| 580 | irace.note(length(scenario$forbiddenExps), | |
| 581 | " expression(s) specifying forbidden configurations read from '", | |
| 582 | scenario$forbiddenFile, "'\n") | |
| 544 | 583 | } |
| 545 | 584 | |
| 546 | 585 | # Make it NULL if it is "" or NA |
| 547 | 586 | # FIXME: If it is a non-empty vector of strings, parse them as above. |
| 548 | if (is.null.or.empty(scenario$forbiddenExps) || is.null.or.na(scenario$forbiddenExps)) | |
| 587 | if (is_null_or_empty_or_na(scenario$forbiddenExps)) | |
| 549 | 588 | scenario$forbiddenExps <- NULL |
| 550 | 589 | |
| 551 | 590 | # We have characters everywhere, set to the right types to avoid |
| 576 | 615 | |
| 577 | 616 | if (scenario$mu < scenario$firstTest) { |
| 578 | 617 | if (scenario$debugLevel >= 1) { |
| 579 | cat("Warning: Assuming 'mu = firstTest' because 'mu' cannot be lower than 'firstTest'\n") | |
| 618 | irace.warning("Assuming 'mu = firstTest' because 'mu' cannot be lower than 'firstTest'\n") | |
| 580 | 619 | } |
| 581 | 620 | scenario$mu <- scenario$firstTest |
| 582 | 621 | } |
| 638 | 677 | " must be larger than 1 when mpi is enabled.") |
| 639 | 678 | } |
| 640 | 679 | |
| 680 | ||
| 641 | 681 | if (is.null.or.empty(scenario$batchmode)) |
| 642 | 682 | scenario$batchmode <- 0 |
| 643 | 683 | if (scenario$batchmode != 0) { |
| 644 | 684 | scenario$batchmode <- tolower(scenario$batchmode) |
| 645 | # FIXME: We should encode options in the large table in main.R | |
| 646 | check.valid.param("batchmode", valid = c("sge", "pbs", "torque", "slurm")) | |
| 685 | check.valid.param("batchmode") | |
| 647 | 686 | } |
| 648 | 687 | # Currently batchmode requires a targetEvaluator |
| 649 | 688 | if (scenario$batchmode != 0 && is.null(scenario$targetEvaluator)) { |
| 662 | 701 | if (scenario$boundMax <= 0) |
| 663 | 702 | irace.error("When capping == TRUE, boundMax (", scenario$boundMax, |
| 664 | 703 | ") must be > 0") |
| 665 | check.valid.param("cappingType", | |
| 666 | valid = c("median", "mean", "worst", "best")) | |
| 667 | check.valid.param("boundType", valid = c("instance", "candidate")) | |
| 668 | ||
| 704 | check.valid.param("cappingType") | |
| 705 | check.valid.param("boundType") | |
| 669 | 706 | if (scenario$boundPar < 1) |
| 670 | irace.error("Invalid value boundPar (", scenario$boundPar, | |
| 671 | ") must be >= 1") | |
| 672 | } else { # no capping | |
| 673 | if (scenario$boundMax <= 0 || is.na(scenario$boundMax)) | |
| 674 | scenario$boundMax <- NULL | |
| 675 | } | |
| 676 | ||
| 677 | ||
| 678 | ||
| 679 | if (is.null.or.empty(scenario$testType) || | |
| 680 | is.null.or.na(scenario$testType) || | |
| 681 | scenario$testType =="") { | |
| 682 | if (scenario$capping) scenario$testType <- "t-test" | |
| 683 | else scenario$testType <- "f-test" | |
| 707 | irace.error("Invalid value (", scenario$boundPar, ") ", | |
| 708 | quote.param("boundPar"), " must be >= 1") | |
| 709 | } else if (scenario$boundMax <= 0 || is.na(scenario$boundMax)) { # no capping | |
| 710 | scenario$boundMax <- NULL | |
| 711 | } | |
| 712 | ||
| 713 | if (is_null_or_empty_or_na(scenario$testType)) { | |
| 714 | if (scenario$capping) scenario$testType <- "t-test" | |
| 715 | else scenario$testType <- "f-test" | |
| 684 | 716 | } |
| 685 | 717 | |
| 686 | 718 | scenario$testType <- |
| 693 | 725 | "t.holm" = "t.holm", |
| 694 | 726 | "t-test-bonferroni" =, # Fall-through, |
| 695 | 727 | "t.bonferroni" = "t.bonferroni", |
| 696 | check.valid.param("testType", | |
| 697 | c("F-test, t-test, t-test-holm, t-test-bonferroni"))) | |
| 698 | ||
| 699 | return (scenario) | |
| 728 | check.valid.param("testType")) | |
| 729 | scenario | |
| 700 | 730 | } |
| 701 | 731 | |
| 702 | 732 | #' Prints the given scenario |
| 705 | 735 | #' |
| 706 | 736 | #' @seealso |
| 707 | 737 | #' \describe{ |
| 708 | #' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 709 | #' \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 710 | #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 711 | #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 738 | #' \item{[readScenario()]}{for reading a configuration scenario from a file.} | |
| 739 | #' \item{[printScenario()]}{prints the given scenario.} | |
| 740 | #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} | |
| 741 | #' \item{[checkScenario()]}{to check that the scenario is valid.} | |
| 712 | 742 | #' } |
| 713 | 743 | #' |
| 714 | 744 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 715 | #' @md | |
| 716 | 745 | #' @export |
| 717 | 746 | printScenario <- function(scenario) |
| 718 | 747 | { |
| 748 | params_names <- .irace.params.names | |
| 719 | 749 | cat("## irace scenario:\n") |
| 720 | for (param in .irace.params.names) { | |
| 750 | for (param in params_names) { | |
| 721 | 751 | if (param == "forbiddenExps") |
| 722 | 752 | extra <- paste0(" = expression(", paste0(collapse=", ", |
| 723 | 753 | sapply(scenario[[param]], attr, "source")), ")") |
| 729 | 759 | |
| 730 | 760 | #' Default scenario settings |
| 731 | 761 | #' |
| 732 | #' Return scenario with default values. | |
| 762 | #' Return scenario object with default values. | |
| 733 | 763 | #' |
| 734 | 764 | #' @template arg_scenario |
| 765 | #' | |
| 766 | #' @template arg_params_def | |
| 735 | 767 | #' |
| 736 | 768 | #' @return A list indexed by the \pkg{irace} parameter names, |
| 737 | 769 | #' containing the default values for each parameter, except for those |
| 741 | 773 | #' \itemize{ |
| 742 | 774 | #' \item General options: |
| 743 | 775 | #' \describe{ |
| 744 | #' \item{\code{scenarioFile}}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: \code{"./scenario.txt"})} | |
| 745 | #' \item{\code{execDir}}{Directory where the programs will be run. (Default: \code{"./"})} | |
| 746 | #' \item{\code{logFile}}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: \code{"./irace.Rdata"})} | |
| 747 | #' \item{\code{debugLevel}}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: \code{0})} | |
| 748 | #' \item{\code{seed}}{Seed of the random number generator (by default, generate a random seed). (Default: \code{NA})} | |
| 749 | #' \item{\code{repairConfiguration}}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: \code{""})} | |
| 750 | #' \item{\code{postselection}}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: \code{0})} | |
| 751 | #' \item{\code{aclib}}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: \code{0})} | |
| 752 | #' } | |
| 753 | #' \item Elitist \code{irace}: | |
| 754 | #' \describe{ | |
| 755 | #' \item{\code{elitist}}{Enable/disable elitist irace. (Default: \code{1})} | |
| 756 | #' \item{\code{elitistNewInstances}}{Number of instances added to the execution list before previous instances in elitist irace. (Default: \code{1})} | |
| 757 | #' \item{\code{elitistLimit}}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: \code{2})} | |
| 758 | #' } | |
| 759 | #' \item Internal \code{irace} options: | |
| 760 | #' \describe{ | |
| 761 | #' \item{\code{nbIterations}}{Number of iterations. (Default: \code{0})} | |
| 762 | #' \item{\code{nbExperimentsPerIteration}}{Number of runs of the target algorithm per iteration. (Default: \code{0})} | |
| 763 | #' \item{\code{sampleInstances}}{Randomly sample the training instances or use them in the order given. (Default: \code{1})} | |
| 764 | #' \item{\code{minNbSurvival}}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: \code{0})} | |
| 765 | #' \item{\code{nbConfigurations}}{Number of configurations to be sampled and evaluated at each iteration. (Default: \code{0})} | |
| 766 | #' \item{\code{mu}}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: \code{5})} | |
| 767 | #' \item{\code{softRestart}}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: \code{1})} | |
| 768 | #' \item{\code{softRestartThreshold}}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: \code{""})} | |
| 776 | #' \item{`scenarioFile`}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: `"./scenario.txt"`)} | |
| 777 | #' \item{`execDir`}{Directory where the programs will be run. (Default: `"./"`)} | |
| 778 | #' \item{`logFile`}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: `"./irace.Rdata"`)} | |
| 779 | #' \item{`quiet`}{Reduce the output generated by irace to a minimum. (Default: `0`)} | |
| 780 | #' \item{`debugLevel`}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: `0`)} | |
| 781 | #' \item{`seed`}{Seed of the random number generator (by default, generate a random seed). (Default: `NA`)} | |
| 782 | #' \item{`repairConfiguration`}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: `""`)} | |
| 783 | #' \item{`postselection`}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: `0`)} | |
| 784 | #' \item{`aclib`}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: `0`)} | |
| 785 | #' } | |
| 786 | #' \item Elitist `irace`: | |
| 787 | #' \describe{ | |
| 788 | #' \item{`elitist`}{Enable/disable elitist irace. (Default: `1`)} | |
| 789 | #' \item{`elitistNewInstances`}{Number of instances added to the execution list before previous instances in elitist irace. (Default: `1`)} | |
| 790 | #' \item{`elitistLimit`}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: `2`)} | |
| 791 | #' } | |
| 792 | #' \item Internal `irace` options: | |
| 793 | #' \describe{ | |
| 794 | #' \item{`sampleInstances`}{Randomly sample the training instances or use them in the order given. (Default: `1`)} | |
| 795 | #' \item{`softRestart`}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: `1`)} | |
| 796 | #' \item{`softRestartThreshold`}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: `""`)} | |
| 797 | #' \item{`nbIterations`}{Maximum number of iterations. (Default: `0`)} | |
| 798 | #' \item{`nbExperimentsPerIteration`}{Number of runs of the target algorithm per iteration. (Default: `0`)} | |
| 799 | #' \item{`minNbSurvival`}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: `0`)} | |
| 800 | #' \item{`nbConfigurations`}{Number of configurations to be sampled and evaluated at each iteration. (Default: `0`)} | |
| 801 | #' \item{`mu`}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: `5`)} | |
| 769 | 802 | #' } |
| 770 | 803 | #' \item Target algorithm parameters: |
| 771 | 804 | #' \describe{ |
| 772 | #' \item{\code{parameterFile}}{File that contains the description of the parameters of the target algorithm. (Default: \code{"./parameters.txt"})} | |
| 773 | #' \item{\code{forbiddenExps}}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: \code{""})} | |
| 774 | #' \item{\code{forbiddenFile}}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: \code{""})} | |
| 775 | #' \item{\code{digits}}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: \code{4})} | |
| 805 | #' \item{`parameterFile`}{File that contains the description of the parameters of the target algorithm. (Default: `"./parameters.txt"`)} | |
| 806 | #' \item{`forbiddenExps`}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: `""`)} | |
| 807 | #' \item{`forbiddenFile`}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: `""`)} | |
| 808 | #' \item{`digits`}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: `4`)} | |
| 776 | 809 | #' } |
| 777 | 810 | #' \item Target algorithm execution: |
| 778 | 811 | #' \describe{ |
| 779 | #' \item{\code{targetRunner}}{Script called for each configuration that executes the target algorithm to be tuned. See templates. (Default: \code{"./target-runner"})} | |
| 780 | #' \item{\code{targetRunnerRetries}}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: \code{0})} | |
| 781 | #' \item{\code{targetRunnerData}}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: \code{""})} | |
| 782 | #' \item{\code{targetRunnerParallel}}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: \code{""})} | |
| 783 | #' \item{\code{targetEvaluator}}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: \code{""})} | |
| 784 | #' \item{\code{deterministic}}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: \code{0})} | |
| 785 | #' \item{\code{parallel}}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: \code{0})} | |
| 786 | #' \item{\code{loadBalancing}}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: \code{1})} | |
| 787 | #' \item{\code{mpi}}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: \code{0})} | |
| 788 | #' \item{\code{batchmode}}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque or slurm. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: \code{0})} | |
| 812 | #' \item{`targetRunner`}{Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. (Default: `"./target-runner"`)} | |
| 813 | #' \item{`targetRunnerLauncher`}{Executable that will be used to launch the target runner, when \code{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). (Default: `""`)} | |
| 814 | #' \item{`targetRunnerLauncherArgs`}{Command-line arguments provided to \code{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \code{targetRunner} and by the arguments usually passed when calling \code{targetRunner}, respectively. Example: \code{"-m {targetRunner} --args {targetRunnerArgs}"}. (Default: `"{targetRunner} {targetRunnerArgs}"`)} | |
| 815 | #' \item{`targetRunnerRetries`}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: `0`)} | |
| 816 | #' \item{`targetRunnerData`}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: `""`)} | |
| 817 | #' \item{`targetRunnerParallel`}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: `""`)} | |
| 818 | #' \item{`targetEvaluator`}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: `""`)} | |
| 819 | #' \item{`deterministic`}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: `0`)} | |
| 820 | #' \item{`parallel`}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: `0`)} | |
| 821 | #' \item{`loadBalancing`}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: `1`)} | |
| 822 | #' \item{`mpi`}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: `0`)} | |
| 823 | #' \item{`batchmode`}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: `0`)} | |
| 789 | 824 | #' } |
| 790 | 825 | #' \item Initial configurations: |
| 791 | 826 | #' \describe{ |
| 792 | #' \item{\code{initConfigurations}}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: \code{""})} | |
| 793 | #' \item{\code{configurationsFile}}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: \code{""})} | |
| 827 | #' \item{`initConfigurations`}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: `""`)} | |
| 828 | #' \item{`configurationsFile`}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: `""`)} | |
| 794 | 829 | #' } |
| 795 | 830 | #' \item Training instances: |
| 796 | 831 | #' \describe{ |
| 797 | #' \item{\code{instances}}{Character vector of the instances to be used in the \code{targetRunner}. (Default: \code{""})} | |
| 798 | #' \item{\code{trainInstancesDir}}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: \code{"./Instances"})} | |
| 799 | #' \item{\code{trainInstancesFile}}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: \code{""})} | |
| 832 | #' \item{`instances`}{Character vector of the instances to be used in the \code{targetRunner}. (Default: `""`)} | |
| 833 | #' \item{`trainInstancesDir`}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: `"./Instances"`)} | |
| 834 | #' \item{`trainInstancesFile`}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: `""`)} | |
| 800 | 835 | #' } |
| 801 | 836 | #' \item Tuning budget: |
| 802 | 837 | #' \describe{ |
| 803 | #' \item{\code{maxExperiments}}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: \code{0})} | |
| 804 | #' \item{\code{maxTime}}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: \code{0})} | |
| 805 | #' \item{\code{budgetEstimation}}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: \code{0.02})} | |
| 838 | #' \item{`maxExperiments`}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: `0`)} | |
| 839 | #' \item{`maxTime`}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: `0`)} | |
| 840 | #' \item{`budgetEstimation`}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: `0.02`)} | |
| 841 | #' \item{`minMeasurableTime`}{Minimum time unit that is still (significantly) measureable. (Default: `0.01`)} | |
| 806 | 842 | #' } |
| 807 | 843 | #' \item Statistical test: |
| 808 | 844 | #' \describe{ |
| 809 | #' \item{\code{testType}}{Statistical test used for elimination. Default test is always \code{F-test} unless \code{capping} is enabled, in which case the default test is \code{t-test}. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: \code{"F-test"})} | |
| 810 | #' \item{\code{firstTest}}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: \code{5})} | |
| 811 | #' \item{\code{eachTest}}{Number of instances evaluated between elimination tests. (Default: \code{1})} | |
| 812 | #' \item{\code{confidence}}{Confidence level for the elimination test. (Default: \code{0.95})} | |
| 845 | #' \item{`testType`}{Statistical test used for elimination. The default value selects \code{t-test} if \code{capping} is enabled or \code{F-test}, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: `""`)} | |
| 846 | #' \item{`firstTest`}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: `5`)} | |
| 847 | #' \item{`eachTest`}{Number of instances evaluated between elimination tests. (Default: `1`)} | |
| 848 | #' \item{`confidence`}{Confidence level for the elimination test. (Default: `0.95`)} | |
| 813 | 849 | #' } |
| 814 | 850 | #' \item Adaptive capping: |
| 815 | 851 | #' \describe{ |
| 816 | #' \item{\code{capping}}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: \code{0})} | |
| 817 | #' \item{\code{cappingType}}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: \code{"median"})} | |
| 818 | #' \item{\code{boundType}}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: \code{"candidate"})} | |
| 819 | #' \item{\code{boundMax}}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: \code{0})} | |
| 820 | #' \item{\code{boundDigits}}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: \code{0})} | |
| 821 | #' \item{\code{boundPar}}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: \code{1})} | |
| 822 | #' \item{\code{boundAsTimeout}}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: \code{1})} | |
| 852 | #' \item{`capping`}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: `0`)} | |
| 853 | #' \item{`cappingType`}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: `"median"`)} | |
| 854 | #' \item{`boundType`}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: `"candidate"`)} | |
| 855 | #' \item{`boundMax`}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: `0`)} | |
| 856 | #' \item{`boundDigits`}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: `0`)} | |
| 857 | #' \item{`boundPar`}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: `1`)} | |
| 858 | #' \item{`boundAsTimeout`}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: `1`)} | |
| 823 | 859 | #' } |
| 824 | 860 | #' \item Recovery: |
| 825 | 861 | #' \describe{ |
| 826 | #' \item{\code{recoveryFile}}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: \code{""})} | |
| 862 | #' \item{`recoveryFile`}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: `""`)} | |
| 827 | 863 | #' } |
| 828 | 864 | #' \item Testing: |
| 829 | 865 | #' \describe{ |
| 830 | #' \item{\code{testInstancesDir}}{Directory where testing instances are located, either absolute or relative to current directory. (Default: \code{""})} | |
| 831 | #' \item{\code{testInstancesFile}}{File containing a list of test instances and optionally additional parameters for them. (Default: \code{""})} | |
| 832 | #' \item{\code{testInstances}}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: \code{""})} | |
| 833 | #' \item{\code{testNbElites}}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: \code{1})} | |
| 834 | #' \item{\code{testIterationElites}}{Enable/disable testing the elite configurations found at each iteration. (Default: \code{0})} | |
| 866 | #' \item{`testInstancesDir`}{Directory where testing instances are located, either absolute or relative to current directory. (Default: `""`)} | |
| 867 | #' \item{`testInstancesFile`}{File containing a list of test instances and optionally additional parameters for them. (Default: `""`)} | |
| 868 | #' \item{`testInstances`}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: `""`)} | |
| 869 | #' \item{`testNbElites`}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: `1`)} | |
| 870 | #' \item{`testIterationElites`}{Enable/disable testing the elite configurations found at each iteration. (Default: `0`)} | |
| 835 | 871 | #' } |
| 836 | 872 | #' } |
| 837 | 873 | # __IRACE_OPTIONS__END__ |
| 838 | 874 | #' |
| 839 | 875 | #' @seealso |
| 840 | 876 | #' \describe{ |
| 841 | #' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 842 | #' \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 843 | #' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 844 | #' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 877 | #' \item{[readScenario()]}{for reading a configuration scenario from a file.} | |
| 878 | #' \item{[printScenario()]}{prints the given scenario.} | |
| 879 | #' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} | |
| 880 | #' \item{[checkScenario()]}{to check that the scenario is valid.} | |
| 845 | 881 | #' } |
| 846 | 882 | #' |
| 847 | 883 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 848 | 884 | #' @export |
| 849 | defaultScenario <- function(scenario = list()) | |
| 850 | { | |
| 851 | if (!is.null(names(scenario)) | |
| 852 | && !all(names(scenario) %in% .irace.params.names)) { | |
| 885 | defaultScenario <- function(scenario = list(), | |
| 886 | params_def = .irace.params.def) | |
| 887 | { | |
| 888 | params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] | |
| 889 | if (is.null(names(scenario))) { | |
| 890 | scenario <- setNames(as.list(params_def[params_names,"default"]), params_names) | |
| 891 | } else if (!all(names(scenario) %in% params_names)) { | |
| 853 | 892 | irace.error("Unknown scenario parameters: ", |
| 854 | paste(names(scenario)[which(!names(scenario) | |
| 855 | %in% .irace.params.names)], | |
| 893 | paste(names(scenario)[!(names(scenario) %in% params_names)], | |
| 856 | 894 | collapse = ", ")) |
| 857 | } | |
| 858 | ||
| 859 | for (k in .irace.params.names) { | |
| 860 | if (is.null.or.na(scenario[[k]])) { | |
| 861 | scenario[[k]] <- .irace.params.def[k, "default"] | |
| 862 | } | |
| 863 | } | |
| 864 | return (scenario) | |
| 895 | } else { | |
| 896 | for (k in params_names) { | |
| 897 | if (is.null.or.na(scenario[[k]])) { | |
| 898 | scenario[[k]] <- params_def[k, "default"] | |
| 899 | } | |
| 900 | } | |
| 901 | } | |
| 902 | scenario | |
| 865 | 903 | } |
| 866 | 904 | |
| 867 | 905 | readInstances <- function(instancesDir = NULL, instancesFile = NULL) |
| 878 | 916 | instances <- sub("#.*$", "", instances) # Remove comments |
| 879 | 917 | instances <- sub("^[[:space:]]+", "", instances) # Remove leading whitespace |
| 880 | 918 | instances <- instances[instances != ""] # Delete empty lines |
| 881 | # FIXME: is.null.or.empty should handle length() == 0. | |
| 882 | if (is.null.or.empty(instances) || length(instances) == 0) | |
| 919 | if (is.null.or.empty(instances)) | |
| 883 | 920 | irace.error("No instances found in '", instancesFile, |
| 884 | 921 | "' (whitespace and comments starting with '#' are ignored)") |
| 885 | 922 | if (!is.null.or.empty(instancesDir)) |
| 894 | 931 | if (length (instances) == 0) |
| 895 | 932 | irace.error("No instances found in `", instancesDir, "'") |
| 896 | 933 | } |
| 897 | ||
| 898 | return(instances) | |
| 934 | instances | |
| 899 | 935 | } |
| 900 | 936 | |
| 901 | 937 | ## Check targetRunner execution |
| 902 | 938 | checkTargetFiles <- function(scenario, parameters) |
| 903 | 939 | { |
| 904 | result <- TRUE | |
| 905 | 940 | ## Create two random configurations |
| 906 | 941 | conf.id <- c("testConfig1", "testConfig2") |
| 907 | 942 | configurations <- sampleUniform(parameters, length(conf.id), |
| 908 | 943 | digits = scenario$digits, |
| 909 | 944 | forbidden = scenario$forbiddenExps, |
| 910 | 945 | repair = scenario$repairConfiguration) |
| 911 | configurations <- cbind (.ID. = conf.id, configurations) | |
| 912 | ||
| 946 | configurations <- cbind(.ID. = conf.id, configurations, stringsAsFactors=FALSE) | |
| 947 | ||
| 948 | # Read initial configurations provided by the user. | |
| 949 | initConfigurations <- allConfigurationsInit(scenario, parameters) | |
| 950 | if (nrow(initConfigurations) > 0L) { | |
| 951 | irace.assert(all(colnames(configurations) == colnames(initConfigurations))) | |
| 952 | configurations <- rbind(configurations, initConfigurations) | |
| 953 | } | |
| 913 | 954 | bounds <- rep(scenario$boundMax, nrow(configurations)) |
| 914 | 955 | |
| 915 | 956 | instances.ID <- if (scenario$sampleInstances) |
| 916 | sample.int(length(scenario$instances), 1) else 1 | |
| 957 | sample.int(length(scenario$instances), 1L) else 1L | |
| 917 | 958 | experiments <- createExperimentList( |
| 918 | 959 | configurations, parameters, instances = scenario$instances, |
| 919 | 960 | instances.ID = instances.ID, seeds = 1234567, scenario, bounds = bounds) |
| 924 | 965 | # FIXME: Create a function try.call(err.msg,warn.msg, fun, ...) |
| 925 | 966 | # Executing targetRunner |
| 926 | 967 | cat("# Executing targetRunner (", nrow(configurations), "times)...\n") |
| 968 | result <- TRUE | |
| 927 | 969 | output <- withCallingHandlers( |
| 928 | 970 | tryCatch(execute.experiments(experiments, scenario), |
| 929 | 971 | error = function(e) { |
| 944 | 986 | } |
| 945 | 987 | |
| 946 | 988 | irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) |
| 947 | ||
| 989 | if (!result) return(FALSE) | |
| 990 | ||
| 948 | 991 | if (!is.null(scenario$targetEvaluator)) { |
| 949 | 992 | cat("# Executing targetEvaluator...\n") |
| 950 | 993 | output <- withCallingHandlers( |
| 965 | 1008 | print(output, digits = 15) |
| 966 | 1009 | } |
| 967 | 1010 | } |
| 968 | return(result) | |
| 1011 | result | |
| 969 | 1012 | } |
| 970 | 1013 | |
| 971 | 1014 | |
| 0 | #' readParameters | |
| 1 | #' | |
| 2 | #' `readParameters` reads the parameters to be tuned by | |
| 3 | #' \pkg{irace} from a file or directly from a character string. | |
| 0 | #' Reads the parameters to be tuned by \pkg{irace} from a file or from a | |
| 1 | #' character string. | |
| 4 | 2 | #' |
| 5 | 3 | #' @param file (`character(1)`) \cr Filename containing the definitions of |
| 6 | 4 | #' the parameters to be tuned. |
| 8 | 6 | #' parameters. |
| 9 | 7 | #' @template arg_debuglevel |
| 10 | 8 | #' @template arg_text |
| 11 | #' | |
| 9 | #' | |
| 12 | 10 | #' @return A list containing the definitions of the parameters read. The list is |
| 13 | 11 | #' structured as follows: |
| 14 | 12 | #' \describe{ |
| 28 | 26 | #' \item{`nbParameters`}{An integer, the total number of parameters.} |
| 29 | 27 | #' \item{`nbFixed`}{An integer, the number of parameters with a fixed value.} |
| 30 | 28 | #' \item{`nbVariable`}{Number of variable (to be tuned) parameters.} |
| 29 | #' \item{`depends`}{List of character vectors, each vector specifies | |
| 30 | #' which parameters depend on this one.} | |
| 31 | #' \item{`isDependent`}{Logical vector that specifies which parameter has | |
| 32 | #' a dependent domain.} | |
| 31 | 33 | #' } |
| 32 | 34 | #' |
| 33 | 35 | #' @details Either `file` or `text` must be given. If `file` is given, the |
| 50 | 52 | #' @examples |
| 51 | 53 | #' ## Read the parameters directly from text |
| 52 | 54 | #' parameters.table <- ' |
| 53 | #' # name switch type values [conditions (using R syntax)] | |
| 54 | #' algorithm "--" c (as,mmas,eas,ras,acs) | |
| 55 | #' localsearch "--localsearch " c (0, 1, 2, 3) | |
| 56 | #' alpha "--alpha " r (0.00, 5.00) | |
| 57 | #' beta "--beta " r (0.00, 10.00) | |
| 58 | #' rho "--rho " r (0.01, 1.00) | |
| 59 | #' ants "--ants " i (5, 100) | |
| 60 | #' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 61 | #' rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 62 | #' elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 63 | #' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) | |
| 64 | #' dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 55 | #' # name switch type values [conditions (using R syntax)] | |
| 56 | #' algorithm "--" c (as,mmas,eas,ras,acs) | |
| 57 | #' localsearch "--localsearch " c (0, 1, 2, 3) | |
| 58 | #' alpha "--alpha " r (0.00, 5.00) | |
| 59 | #' beta "--beta " r (0.00, 10.00) | |
| 60 | #' rho "--rho " r (0.01, 1.00) | |
| 61 | #' ants "--ants " i,log (5, 100) | |
| 62 | #' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 63 | #' rasrank "--rasranks " i (1, "min(ants, 10)") | algorithm == "ras" | |
| 64 | #' elitistants "--elitistants " i (1, ants) | algorithm == "eas" | |
| 65 | #' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) | |
| 66 | #' dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 65 | 67 | #' ' |
| 66 | 68 | #' parameters <- readParameters(text=parameters.table) |
| 67 | 69 | #' str(parameters) |
| 68 | 70 | #' |
| 69 | 71 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 70 | #' @md | |
| 71 | 72 | #' @export |
| 72 | readParameters <- function(file, digits = 4, debugLevel = 0, text) | |
| 73 | readParameters <- function (file, digits = 4, debugLevel = 0, text) | |
| 73 | 74 | { |
| 74 | 75 | if (missing(file) && !missing(text)) { |
| 75 | 76 | filename <- strcat("text=", deparse(substitute(text))) |
| 166 | 167 | # The following line detects cycles |
| 167 | 168 | if (child == rootParam) |
| 168 | 169 | irace.error("A cycle detected in subordinate parameters! ", |
| 169 | "Check definition of conditions.\n", | |
| 170 | "Check definition of conditions and/or dependent domains.\n", | |
| 170 | 171 | "One parameter of this cycle is '", rootParam, "'") |
| 171 | 172 | |
| 172 | 173 | # The following line detects a missing definition |
| 196 | 197 | transform.domain <- function(transf, domain, type) |
| 197 | 198 | { |
| 198 | 199 | if (transf == "") return(transf) |
| 200 | ||
| 201 | # We do not support transformation of dependent parameters, yet | |
| 202 | # TODO: think about dependent domain transfomation | |
| 203 | if (is.expression(domain)) | |
| 204 | irace.error("Parameter domain transformations are not yet available for", | |
| 205 | " dependent parameter domains.") | |
| 199 | 206 | |
| 200 | 207 | lower <- domain[1] |
| 201 | 208 | upper <- domain[2] |
| 216 | 223 | } |
| 217 | 224 | irace.internal.error("unrecognized transformation type '", transf, "'") |
| 218 | 225 | } |
| 219 | ||
| 226 | ||
| 227 | # Checks that variables in the expressions are within | |
| 228 | # the parameters names. | |
| 229 | check_parameter_dependencies <- function (parameters) { | |
| 230 | for (p in names(Filter(length, parameters$depends))) { | |
| 231 | vars <- parameters$depends[[p]] | |
| 232 | flag <- vars %in% parameters$names | |
| 233 | if (!all(flag)) { | |
| 234 | irace.error ("Domain (", paste0(parameters$domain[[p]], collapse=", "), | |
| 235 | ") of parameter '", p, "' is not valid: '", | |
| 236 | paste0(vars[!flag], collapse=", "), | |
| 237 | "' cannot be found in the scenario parameters: ", | |
| 238 | paste0(parameters$names, collapse=" , ")," .") | |
| 239 | } | |
| 240 | flag <- parameters$types[vars] %in% c("i", "r") | |
| 241 | if (!all(flag)) { | |
| 242 | irace.error ("Domain of parameter '", p, "' depends on non-numerical", | |
| 243 | " parameters: ", paste0(vars[!flag], collapse=", "), " .") | |
| 244 | } | |
| 245 | ||
| 246 | # Supported operations for dependent domains | |
| 247 | allowed.fx <- c("+", "-", "*", "/", "%%", "min", "max", "round", "floor", "ceiling", "trunc") | |
| 248 | fx <- setdiff(all.names(parameters$domain[[p]], unique=TRUE), | |
| 249 | all.vars(parameters$domain[[p]], unique=TRUE)) | |
| 250 | flag <- fx %in% allowed.fx | |
| 251 | if (!all(flag)) { | |
| 252 | irace.error ("Domain of parameter '", p, "' uses function(s) ", | |
| 253 | "not yet supported by irace: ", | |
| 254 | paste0(fx[!flag], collapse=", "), " .") | |
| 255 | } | |
| 256 | } | |
| 257 | return(TRUE) | |
| 258 | } | |
| 259 | ||
| 220 | 260 | parameters <- list(names = character(0), |
| 221 | 261 | types = character(0), |
| 222 | 262 | switches = character(0), |
| 223 | 263 | domain = list(), |
| 224 | 264 | conditions = list(), |
| 225 | 265 | isFixed = logical(0), |
| 226 | transform = list()) | |
| 266 | # FIXME: This has to be a list because we assign | |
| 267 | # attributes to elements. | |
| 268 | transform = list(), | |
| 269 | isDependent = logical(0)) | |
| 227 | 270 | |
| 228 | 271 | conditions <- list() |
| 229 | 272 | lines <- readLines(con = file) |
| 281 | 324 | } |
| 282 | 325 | |
| 283 | 326 | ## Match param.value (delimited by parenthesis) |
| 284 | result <- field.match (line, "\\([^)]+\\)", delimited = TRUE, sep = "") | |
| 327 | # Regexp to detect dependent domains of the type ("min(p1)", 100) | |
| 328 | result <- field.match (line, "\\([^|]+\\)", delimited = TRUE, sep = "") | |
| 285 | 329 | param.value <- result$match |
| 286 | 330 | line <- result$line |
| 287 | if (is.null (result$match)) { | |
| 331 | if (is.null (param.value)) { | |
| 288 | 332 | errReadParameters (filename, nbLines, line, |
| 289 | 333 | "Allowed values must be a list within parenthesis") |
| 290 | 334 | } |
| 291 | 335 | |
| 292 | param.value <- string2vector(param.value) | |
| 336 | # For numerical parameters domains could be dependent | |
| 337 | # thus, we keep the string values in a variable | |
| 338 | # for example (10, param1+2) | |
| 339 | param.value.str <- string2vector(param.value) | |
| 293 | 340 | if (param.type %in% c("r","i")) { |
| 294 | param.value <- suppressWarnings(as.numeric(param.value)) | |
| 295 | if (any(is.na(param.value)) || length(param.value) != 2) { | |
| 341 | # For dependent domains param.value will be NA (we will parse | |
| 342 | # it later) | |
| 343 | param.value <- suppressWarnings(as.numeric(param.value.str)) | |
| 344 | if (length(param.value) != 2) { | |
| 296 | 345 | errReadParameters (filename, nbLines, NULL, |
| 297 | 346 | "incorrect numeric range (", result$match, |
| 298 | 347 | ") for parameter '", param.name, "'") |
| 299 | } else if (param.value[1] > param.value[2]) { | |
| 300 | errReadParameters (filename, nbLines, NULL, | |
| 301 | "lower bound must be smaller than upper bound in numeric range (", | |
| 302 | result$match, ") for parameter '", param.name, "'") | |
| 303 | 348 | } |
| 304 | 349 | |
| 305 | 350 | if (param.type == "r") { |
| 306 | 351 | # FIXME: Given (0.01,0.99) and digits=1, this produces (0, 1), which is |
| 307 | 352 | # probably not what the user wants. |
| 308 | 353 | param.value <- round(param.value, digits = digits) |
| 309 | } else if (param.type == "i" && !all(is.wholenumber(param.value))) { | |
| 354 | } else if (param.type == "i" && any(!is.wholenumber(param.value[!is.na(param.value)]))) { | |
| 355 | errReadParameters (filename, nbLines, NULL, | |
| 356 | "for parameter type 'i' values must be integers (", | |
| 357 | result$match, ") for parameter '", param.name, "'") | |
| 358 | } | |
| 359 | ||
| 360 | # Time to parse dependent domains or check values | |
| 361 | if (any(is.na(param.value))) { | |
| 362 | try(param.value[is.na(param.value)] <- parse(text=param.value.str[is.na(param.value)])) | |
| 363 | } else if (param.value[1] >= param.value[2]) { | |
| 310 | 364 | errReadParameters (filename, nbLines, NULL, |
| 311 | "for parameter type 'i' values must be integers (", | |
| 365 | "lower bound must be smaller than upper bound in numeric range (", | |
| 312 | 366 | result$match, ") for parameter '", param.name, "'") |
| 313 | 367 | } |
| 314 | ||
| 368 | ||
| 315 | 369 | param.transform <- transform.domain(param.transform, param.value, param.type) |
| 316 | 370 | if (is.null(param.transform)) { |
| 317 | 371 | errReadParameters (filename, nbLines, NULL, "The domain of parameter '", |
| 318 | 372 | param.name, "' of type 'log' cannot contain zero") |
| 319 | 373 | } |
| 320 | 374 | } else { |
| 375 | param.value <- param.value.str | |
| 321 | 376 | if (anyDuplicated(param.value)) { |
| 322 | 377 | dups <- duplicated(param.value) |
| 323 | 378 | errReadParameters (filename, nbLines, NULL, |
| 384 | 439 | "check that the parameter file is not empty") |
| 385 | 440 | } |
| 386 | 441 | |
| 442 | # Generate dependency flag | |
| 443 | # FIXME: check if we really need this vector | |
| 444 | parameters$isDependent <- sapply(parameters$domain, is.expression) | |
| 445 | ||
| 446 | names(parameters$types) <- | |
| 447 | names(parameters$switches) <- | |
| 448 | names(parameters$domain) <- | |
| 449 | names(parameters$isFixed) <- | |
| 450 | names(parameters$transform) <- | |
| 451 | names(parameters$isDependent) <- parameters$names | |
| 452 | ||
| 387 | 453 | # Obtain the variables in each condition |
| 388 | 454 | ## FIXME: In R 3.2, all.vars does not work with byte-compiled expressions, |
| 389 | 455 | ## thus we do not byte-compile them; but we could use |
| 390 | 456 | ## all.vars(.Internal(disassemble(condition))[[3]][[1]]) |
| 391 | parameters$depends <- lapply(conditions, all.vars) | |
| 457 | ## LESLIE: should we make then an all.vars in utils.R so we can | |
| 458 | ## use it without problems? | |
| 459 | parameters$depends <- lapply(parameters$domain, all.vars) | |
| 460 | # Check that dependencies are ok | |
| 461 | check_parameter_dependencies(parameters) | |
| 462 | # Merge dependencies and conditions | |
| 463 | parameters$depends <- Map(c, parameters$depends, lapply(conditions, all.vars)) | |
| 464 | parameters$depends <- lapply(parameters$depends, unique) | |
| 465 | ||
| 392 | 466 | # Sort parameters in 'conditions' in the proper order according to |
| 393 | 467 | # conditions |
| 394 | 468 | hierarchyLevel <- sapply(parameters$names, treeLevel, |
| 396 | 470 | parameters$hierarchy <- hierarchyLevel |
| 397 | 471 | parameters$conditions <- conditions[order(hierarchyLevel)] |
| 398 | 472 | |
| 399 | names(parameters$types) <- | |
| 400 | names(parameters$switches) <- | |
| 401 | names(parameters$domain) <- | |
| 402 | names(parameters$isFixed) <- | |
| 403 | names(parameters$hierarchy) <- | |
| 404 | names(parameters$transform) <- parameters$names | |
| 473 | names(parameters$hierarchy) <- parameters$names | |
| 405 | 474 | |
| 406 | 475 | # Print the hierarchy vector: |
| 407 | 476 | if (debugLevel >= 1) { |
| 408 | 477 | cat ("# --- Parameters Hierarchy ---\n") |
| 409 | print(paste0(names(parameters$hierarchy))) | |
| 410 | print(parameters$hierarchy) | |
| 411 | print(sapply(parameters$depends, paste0, collapse=", ")) | |
| 412 | 478 | print(data.frame(Parameter = paste0(names(parameters$hierarchy)), |
| 413 | 479 | Level = parameters$hierarchy, |
| 414 | 480 | "Depends on" = sapply(parameters$depends, paste0, collapse=", "), |
| 421 | 487 | parameters$nbParameters <- length(parameters$names) |
| 422 | 488 | parameters$nbFixed <- sum(parameters$isFixed == TRUE) |
| 423 | 489 | parameters$nbVariable <- sum(parameters$isFixed == FALSE) |
| 424 | if (debugLevel >= 2) print(parameters, digits = 15) | |
| 425 | return (parameters) | |
| 490 | if (debugLevel >= 2) { | |
| 491 | print(parameters, digits = 15) | |
| 492 | irace.note("Parameters have been read\n") | |
| 493 | } | |
| 494 | parameters | |
| 426 | 495 | } |
| 427 | 496 | |
| 428 | #' read_pcs_file | |
| 429 | #' | |
| 430 | 497 | #' Read parameters in PCS (AClib) format and write them in irace format. |
| 431 | 498 | #' |
| 432 | 499 | #' @param file (`character(1)`) \cr Filename containing the definitions of |
| 446 | 513 | #' for details. If none of these parameters is given, \pkg{irace} |
| 447 | 514 | #' will stop with an error. |
| 448 | 515 | #' |
| 516 | #' **FIXME:** Forbidden configurations, default configuration and transformations ("log") are currently ignored. See <https://github.com/MLopez-Ibanez/irace/issues/31> | |
| 517 | #' | |
| 518 | #' @references | |
| 519 | #' Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. **AClib: A Benchmark Library for Algorithm Configuration**. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, _Learning and Intelligent Optimization, 8th International Conference, LION 8_, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, Heidelberg, 2014. | |
| 520 | #' | |
| 449 | 521 | #' @examples |
| 450 | 522 | #' ## Read the parameters directly from text |
| 451 | #' pcs.table <- ' | |
| 452 | #' # name values [conditions (using R syntax)] | |
| 523 | #' pcs_table <- ' | |
| 524 | #' # name domain | |
| 453 | 525 | #' algorithm {as,mmas,eas,ras,acs}[as] |
| 454 | 526 | #' localsearch {0, 1, 2, 3}[0] |
| 455 | 527 | #' alpha [0.00, 5.00][1] |
| 468 | 540 | #' nnls | localsearch in {1,2,3} |
| 469 | 541 | #' dlb | localsearch in {1,2,3} |
| 470 | 542 | #' ' |
| 471 | #' parameters.table <- read_pcs_file(text=pcs.table) | |
| 472 | #' parameters <- readParameters(text=parameters.table) | |
| 543 | #' parameters_table <- read_pcs_file(text=pcs_table) | |
| 544 | #' cat(parameters_table) | |
| 545 | #' parameters <- readParameters(text=parameters_table) | |
| 473 | 546 | #' str(parameters) |
| 474 | 547 | #' |
| 475 | 548 | #' @author Manuel López-Ibáñez |
| 476 | #' @md | |
| 477 | 549 | #' @export |
| 478 | 550 | read_pcs_file <- function(file, digits = 4, debugLevel = 0, text) |
| 479 | 551 | { |
| 563 | 635 | } |
| 564 | 636 | irace.error("unrecognized line: ", line) |
| 565 | 637 | } |
| 566 | return(output) | |
| 638 | output | |
| 567 | 639 | } |
| 640 | ||
| 641 | #' checkParameters | |
| 642 | #' | |
| 643 | #' FIXME: This is incomplete, for now we only repair inputs from previous irace | |
| 644 | #' versions. | |
| 645 | #' | |
| 646 | #' @template arg_parameters | |
| 647 | #' @export | |
| 648 | checkParameters <- function(parameters) | |
| 649 | { | |
| 650 | if (is.null(parameters$isDependent)) { | |
| 651 | parameters$isDependent <- sapply(parameters$domain, is.expression) | |
| 652 | names(parameters$isDependent) <- parameters$names | |
| 653 | } | |
| 654 | parameters | |
| 655 | } | |
| 656 | ||
| 657 | #' Print parameter space in the textual format accepted by irace. | |
| 658 | #' | |
| 659 | #' FIXME: Dependent parameter bounds are not supported yet. | |
| 660 | #' | |
| 661 | #' @param params (`list()`) Parameter object stored in `irace.Rdata` or read with `irace::readParameters()`. | |
| 662 | #' | |
| 663 | #' @param digits (`integer()`) The desired number of digits after the decimal point for real-valued parameters. Default is 15, but it should be the value in `scenario$digits`. | |
| 664 | #' | |
| 665 | #' @examples | |
| 666 | #' parameters.table <- ' | |
| 667 | #' # name switch type values [conditions (using R syntax)] | |
| 668 | #' algorithm "--" c (as,mmas,eas,ras,acs) | |
| 669 | #' localsearch "--localsearch " c (0, 1, 2, 3) | |
| 670 | #' ants "--ants " i,log (5, 100) | |
| 671 | #' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 672 | #' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) | |
| 673 | #' ' | |
| 674 | #' parameters <- readParameters(text=parameters.table) | |
| 675 | #' printParameters(parameters) | |
| 676 | #' @export | |
| 677 | printParameters <- function(params, digits = 15L) | |
| 678 | { | |
| 679 | names_len <- max(nchar(params$names)) | |
| 680 | switches_len <- max(nchar(params$switches)) + 2 | |
| 681 | for (name in params$names) { | |
| 682 | switch <- paste0('"', params$switches[[name]], '"') | |
| 683 | type <- params$types[[name]] | |
| 684 | transf <- params$transform[[name]] | |
| 685 | domain <- params$domain[[name]] | |
| 686 | if (type == "r") domain <- formatC(domain, digits=digits, format="f", drop0trailing=TRUE) | |
| 687 | domain <- paste0('(', paste0(domain, collapse=","), ')') | |
| 688 | condition <- params$conditions[[name]] | |
| 689 | condition <- if (isTRUE(condition)) "" else paste0(" | ", condition) | |
| 690 | if (!is.null(transf) && transf != "") type <- paste0(type, ",", transf) | |
| 691 | cat(sprintf('%*s %*s %s %-15s%s\n', -names_len, name, -switches_len, switch, type, domain, condition)) | |
| 692 | } | |
| 693 | } | |
| 0 | #' testConfigurations | |
| 0 | #' Execute the given configurations on the testing instances specified in the | |
| 1 | #' scenario | |
| 1 | 2 | #' |
| 2 | #' \code{testConfigurations} executes the given configurations on the | |
| 3 | #' testing instances specified in the scenario. | |
| 4 | #' | |
| 5 | 3 | #' @template arg_configurations |
| 6 | 4 | #' @template arg_scenario |
| 7 | 5 | #' @template arg_parameters |
| 12 | 10 | #' \item{\code{seeds}}{Array of the instance seeds used in the experiments.} |
| 13 | 11 | #' } |
| 14 | 12 | #' |
| 15 | #' @details A test instance set must be provided through \code{scenario$testInstances}. | |
| 13 | #' @details A test instance set must be provided through `scenario[["testInstances"]]`. | |
| 16 | 14 | #' |
| 17 | 15 | #' @seealso |
| 18 | #' \code{\link{testing.main}} | |
| 16 | #' [testing_fromlog()] | |
| 19 | 17 | #' |
| 20 | 18 | #' @author Manuel López-Ibáñez |
| 21 | 19 | #' @export |
| 23 | 21 | { |
| 24 | 22 | # We need to set up a default scenario (and repeat all checks) in case |
| 25 | 23 | # we are called directly instead of being called after executing irace. |
| 26 | scenario <- checkScenario(defaultScenario(scenario)) | |
| 24 | scenario <- checkScenario(scenario) | |
| 27 | 25 | |
| 28 | testInstances <- scenario$testInstances | |
| 26 | testInstances <- scenario[["testInstances"]] | |
| 29 | 27 | instances.ID <- names(testInstances) |
| 30 | 28 | if (length(testInstances) == 0) irace.error("No test instances given") |
| 31 | 29 | if (is.null(instances.ID)) irace.error("testInstances must have names") |
| 32 | 30 | |
| 33 | 31 | # 2147483647 is the maximum value for a 32-bit signed integer. |
| 34 | 32 | # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. |
| 35 | ## FIXME: scenario$testInstances and scenario$instances behave differently, | |
| 33 | ## FIXME: scenario[["testInstances"]] and scenario$instances behave differently, | |
| 36 | 34 | ## we should unify them so that the seeds are also saved in scenario. |
| 37 | 35 | instanceSeed <- sample.int(2147483647, size = length(testInstances), replace = TRUE) |
| 38 | 36 | names(instanceSeed) <- instances.ID |
| 0 | Timer <- R6::R6Class("Timer", cloneable = FALSE, list( | |
| 1 | start = NULL, | |
| 2 | initialize = function() { | |
| 3 | self$start <- proc.time() | |
| 4 | self | |
| 5 | }, | |
| 6 | elapsed = function() { | |
| 7 | x <- proc.time() - self$start | |
| 8 | if (!is.na(x[4L])) | |
| 9 | x[1L] <- x[1L] + x[4L] | |
| 10 | if (!is.na(x[5L])) | |
| 11 | x[2L] <- x[2L] + x[5L] | |
| 12 | x <- x[1L:3L] | |
| 13 | names(x) <- c("user", "system", "wallclock") | |
| 14 | x | |
| 15 | }) | |
| 16 | ) | |
| 17 |
| 42 | 42 | rtnorm <- function (n, mean = 0, sd = 1, lower = -Inf, upper = Inf) { |
| 43 | 43 | if (length(n) > 1) |
| 44 | 44 | n <- length(n) |
| 45 | mean <- rep(mean, length=n) | |
| 46 | sd <- rep(sd, length=n) | |
| 47 | lower <- rep(lower, length=n) | |
| 48 | upper <- rep(upper, length=n) | |
| 45 | mean <- rep_len(mean, n) | |
| 46 | sd <- rep_len(sd, n) | |
| 47 | lower <- rep_len(lower, n) | |
| 48 | upper <- rep_len(upper, n) | |
| 49 | 49 | lower <- (lower - mean) / sd ## Algorithm works on mean 0, sd 1 scale |
| 50 | 50 | upper <- (upper - mean) / sd |
| 51 | ind <- seq(length=n) | |
| 51 | ind <- seq_len(n) | |
| 52 | 52 | ret <- numeric(n) |
| 53 | 53 | ## Different algorithms depending on where upper/lower limits lie. |
| 54 | 54 | alg <- ifelse( |
| 8 | 8 | options(error = if (interactive()) utils::recover else |
| 9 | 9 | quote(utils::dump.frames("iracedump", TRUE))) |
| 10 | 10 | } |
| 11 | ||
| 12 | .irace.prefix <- "== irace == " | |
| 13 | 11 | |
| 14 | 12 | irace.print.memUsed <- function(objects) |
| 15 | 13 | { |
| 41 | 39 | # cannot help the user to understand why the program failed. |
| 42 | 40 | irace.warning <- function(...) |
| 43 | 41 | { |
| 44 | cat(sep="", "WARNING: ", ..., "\n") | |
| 42 | if (getOption(".irace.quiet", default=FALSE)) return() | |
| 43 | cat(sep="", .msg.prefix, "WARNING: ", ..., "\n") | |
| 45 | 44 | } |
| 46 | 45 | |
| 47 | 46 | # Print a user-level fatal error message, when the calling context |
| 52 | 51 | # value allowed up to R 3.0.2 |
| 53 | 52 | op <- options(warning.length = 8170) |
| 54 | 53 | on.exit(options(op)) |
| 55 | stop (.irace.prefix, ..., call. = FALSE) | |
| 56 | } | |
| 57 | ||
| 58 | ## utils::dump.frames is broken and cannot be used with bquote, so we need a wrapper. | |
| 59 | ## See help(dump.frames) | |
| 54 | stop (.msg.prefix, ..., call. = FALSE) | |
| 55 | } | |
| 56 | ||
| 57 | ## utils::dump.frames is broken and cannot be used with bquote, so we need a wrapper. When irace crashes, it generates a file "iracedump.rda". To debug the crash use: | |
| 58 | ## R> load("iracedump.rda") | |
| 59 | ## R> debugger(iracedump) | |
| 60 | ## | |
| 61 | ## See help(dump.frames) for more details. | |
| 60 | 62 | irace.dump.frames <- function() |
| 61 | 63 | { |
| 62 | 64 | execDir <- getOption(".irace.execdir") |
| 79 | 81 | { |
| 80 | 82 | .irace.bug.report <- |
| 81 | 83 | paste0("An unexpected condition occurred. ", |
| 82 | "Please report this bug to the authors of the irace package <http://iridia.ulb.ac.be/irace>") | |
| 84 | "Please report this bug to the authors of the irace package <https://github.com/MLopez-Ibanez/irace/issues>") | |
| 83 | 85 | |
| 84 | 86 | op <- options(warning.length = 8170, |
| 85 | 87 | error = if (interactive()) utils::recover |
| 88 | 90 | # 6 to not show anything below irace.assert() |
| 89 | 91 | bt <- capture.output(traceback(6)) |
| 90 | 92 | warnings() |
| 91 | stop (.irace.prefix, paste0(..., collapse = "\n"), | |
| 93 | stop (.msg.prefix, paste0(..., collapse = "\n"), | |
| 92 | 94 | paste0(bt, collapse= "\n"), "\n", |
| 93 | .irace.prefix, "\n", .irace.bug.report, call. = FALSE) | |
| 95 | .msg.prefix, "\n", .irace.bug.report, call. = FALSE) | |
| 94 | 96 | invisible() |
| 95 | 97 | } |
| 96 | 98 | |
| 103 | 105 | msg <- paste0(deparse(mc), " is not TRUE\n") |
| 104 | 106 | if (!is.null(eval.after)) { |
| 105 | 107 | msg.after <- eval.parent(capture.output(eval.after)) |
| 106 | msg <- paste0(msg, "\n", msg.after) | |
| 108 | msg <- paste0(msg, "\n", paste0(msg.after, collapse="\n")) | |
| 107 | 109 | } |
| 108 | 110 | irace.internal.error(msg) |
| 109 | 111 | invisible() |
| 111 | 113 | |
| 112 | 114 | irace.note <- function(...) |
| 113 | 115 | { |
| 116 | # FIXME: If this was a function within an irace object, we could replace it | |
| 117 | # when using quiet. | |
| 118 | if (getOption(".irace.quiet", default=FALSE)) return() | |
| 114 | 119 | cat ("# ", format(Sys.time(), usetz=TRUE), ": ", |
| 115 | 120 | paste0(..., collapse = ""), sep = "") |
| 116 | 121 | } |
| 126 | 131 | if (!is.character(file) || is.null.or.empty(file)) { |
| 127 | 132 | irace.error (text, " ", shQuote(file), " is not a vaild filename") |
| 128 | 133 | } |
| 129 | file <- path.rel2abs(file) | |
| 134 | file <- path_rel2abs(file) | |
| 130 | 135 | ## The above should remove the trailing separator if present for windows OS |
| 131 | 136 | ## compatibility, except when we have just C:/, where the trailing separator |
| 132 | 137 | ## must remain. |
| 203 | 208 | (length(x) == 0) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") |
| 204 | 209 | } |
| 205 | 210 | |
| 211 | is_null_or_empty_or_na <- function(x) | |
| 212 | { | |
| 213 | (length(x) == 0) || is.na.nowarn(x) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") | |
| 214 | } | |
| 215 | ||
| 206 | 216 | is.function.name <- function(FUN) |
| 207 | 217 | { |
| 208 | 218 | # FIXME: Is there a simpler way to do this check? |
| 241 | 251 | strcat <- function(...) |
| 242 | 252 | { |
| 243 | 253 | do.call(paste0, args = list(..., collapse = NULL)) |
| 244 | } | |
| 245 | ||
| 246 | # Function to convert a relative to an absolute path. CWD is the | |
| 247 | # working directory to complete relative paths. It tries really hard | |
| 248 | # to create canonical paths. | |
| 249 | path.rel2abs <- function (path, cwd = getwd()) | |
| 250 | { | |
| 251 | # Keep doing gsub as long as x keeps changing. | |
| 252 | gsub.all <- function(pattern, repl, x, ...) { | |
| 253 | repeat { | |
| 254 | newx <- gsub(pattern, repl, x, ...) | |
| 255 | if (newx == x) return(newx) | |
| 256 | x <- newx | |
| 257 | } | |
| 258 | } | |
| 259 | irace.normalize.path <- function(path) { | |
| 260 | return(suppressWarnings(normalizePath(path, winslash = "/", mustWork = NA))) | |
| 261 | } | |
| 262 | ||
| 263 | if (is.null.or.na(path)) { | |
| 264 | return (NULL) | |
| 265 | } else if (path == "") { | |
| 266 | return ("") | |
| 267 | } | |
| 268 | # Using .Platform$file.sep is too fragile. Better just use "/" everywhere. | |
| 269 | s <- "/" | |
| 270 | ||
| 271 | # Possibly expand ~/path to /home/user/path. | |
| 272 | path <- path.expand(path) | |
| 273 | # Remove winslashes if given. | |
| 274 | path <- gsub("\\", s, path, fixed = TRUE) | |
| 275 | ||
| 276 | # Detect a Windows drive | |
| 277 | windrive.regex <- "^[A-Za-z]:" | |
| 278 | windrive <- "" | |
| 279 | if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { | |
| 280 | m <- regexpr(windrive.regex, path) | |
| 281 | windrive <- regmatches(path, m) | |
| 282 | path <- sub(windrive.regex, "", path) | |
| 283 | } | |
| 284 | ||
| 285 | ||
| 286 | # Change "/./" to "/" to get a canonical form | |
| 287 | path <- gsub.all(paste0(s, ".", s), s, path, fixed = TRUE) | |
| 288 | # Change "//" to "/" to get a canonical form | |
| 289 | path <- gsub(paste0(s, s, "+"), s, path) | |
| 290 | # Change "/.$" to "/" to get a canonical form | |
| 291 | path <- sub(paste0(s, "\\.$"), s, path) | |
| 292 | # Drop final "/" | |
| 293 | path <- sub(paste0(s, "$"), "", path) | |
| 294 | if (path == "") path <- s | |
| 295 | ||
| 296 | # Prefix the current cwd to the path if it doesn't start with | |
| 297 | # / \\ or whatever separator. | |
| 298 | if (path == "." || !grepl(paste0("^",s), path)) { | |
| 299 | # There is no need to normalize cwd if it was returned by getwd() | |
| 300 | if (!missing(cwd)) { | |
| 301 | # Recurse to get absolute cwd | |
| 302 | cwd <- path.rel2abs(cwd) | |
| 303 | } | |
| 304 | ||
| 305 | # Speed-up the most common cases. | |
| 306 | # If it is just "." | |
| 307 | if (path == ".") { | |
| 308 | return (irace.normalize.path(cwd)) | |
| 309 | } | |
| 310 | # Remove "./" from the start of path. | |
| 311 | path <- sub(paste0("^\\.", s), "", path) | |
| 312 | # Make it absolute but avoid doubling s | |
| 313 | if (substring(cwd, nchar(cwd)) == s) path <- paste0(cwd, path) | |
| 314 | else path <- paste0(cwd, s, path) | |
| 315 | # If it is just a path without ".." inside | |
| 316 | if (!grepl(paste0(s,"\\.\\."), path)) { | |
| 317 | return (irace.normalize.path(path)) | |
| 318 | } | |
| 319 | # Detect a Windows drive | |
| 320 | if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { | |
| 321 | m <- regexpr(windrive.regex, path) | |
| 322 | windrive <- regmatches(path, m) | |
| 323 | path <- sub(windrive.regex, "", path) | |
| 324 | } | |
| 325 | } | |
| 326 | # else | |
| 327 | ||
| 328 | # Change "/x/.." to "/" to get a canonical form | |
| 329 | prevdir.regex <- paste0(s, "[^", s,"]+", s, "\\.\\.") | |
| 330 | repeat { | |
| 331 | # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" | |
| 332 | tmp <- sub(paste0(prevdir.regex, s), s, path) | |
| 333 | if (tmp == path) break | |
| 334 | path <- tmp | |
| 335 | } | |
| 336 | # Handle "/something/..$" to "/" that is, when ".." is the last thing in the path. | |
| 337 | path <- sub(paste0(prevdir.regex, "$"), s, path) | |
| 338 | ||
| 339 | # Handle "^/../../.." to "/" that is, going up at the root just returns the root. | |
| 340 | repeat { | |
| 341 | # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" | |
| 342 | tmp <- sub(paste0("^", s, "\\.\\.", s), s, path) | |
| 343 | if (tmp == path) break | |
| 344 | path <- tmp | |
| 345 | } | |
| 346 | # Handle "^/..$" to "/" that is, when ".." is the last thing in the path. | |
| 347 | path <- sub(paste0("^", s, "\\.\\.$"), s, path) | |
| 348 | ||
| 349 | # Add back Windows drive, if any. | |
| 350 | path <- paste0(windrive, path) | |
| 351 | ||
| 352 | # We use normalizePath, which will further simplify the path if | |
| 353 | # the path exists. | |
| 354 | return (irace.normalize.path(path)) | |
| 355 | 254 | } |
| 356 | 255 | |
| 357 | 256 | #' Update filesystem paths of a scenario consistently. |
| 361 | 260 | #' |
| 362 | 261 | #' @template arg_scenario |
| 363 | 262 | #' @param from character string containing a regular expression (or character |
| 364 | #' string for \code{fixed = TRUE}) to be matched. | |
| 365 | #' @param to the replacement string.character string. For \code{fixed = FALSE} | |
| 366 | #' this can include backreferences \code{"\1"} to \code{"\9"} to | |
| 367 | #' parenthesized subexpressions of \code{from}. | |
| 368 | #' @param fixed logical. If \code{TRUE}, \code{from} is a string to be matched | |
| 263 | #' string for `fixed = TRUE`) to be matched. | |
| 264 | #' @param to the replacement string.character string. For `fixed = FALSE` | |
| 265 | #' this can include backreferences `"\1"` to `"\9"` to | |
| 266 | #' parenthesized subexpressions of `from`. | |
| 267 | #' @param fixed logical. If `TRUE`, `from` is a string to be matched | |
| 369 | 268 | #' as is. |
| 370 | 269 | #' @return The updated scenario |
| 371 | 270 | #' @examples |
| 372 | 271 | #' \dontrun{ |
| 373 | 272 | #' scenario <- readScenario(filename = "scenario.txt") |
| 374 | #' scenario <- scenario.update.paths(scenario, from = "/home/manuel/", to = "/home/leslie") | |
| 273 | #' scenario <- scenario_update_paths(scenario, from = "/home/manuel/", to = "/home/leslie") | |
| 375 | 274 | #' } |
| 376 | #' @seealso \code{\link[base]{grep}} | |
| 275 | #' @seealso [base::grep()] | |
| 377 | 276 | #' @export |
| 378 | scenario.update.paths <- function(scenario, from, to, fixed = TRUE) | |
| 277 | scenario_update_paths <- function(scenario, from, to, fixed = TRUE) | |
| 379 | 278 | { |
| 380 | 279 | pathParams <- .irace.params.def[.irace.params.def[, "type"] == "p", "name"] |
| 381 | 280 | # Only consider the ones that actually appear in scenario. |
| 382 | 281 | pathParams <- intersect(pathParams, names(scenario)) |
| 383 | 282 | scenario[pathParams] <- lapply(scenario[pathParams], sub, pattern = from, replacement = to, fixed = fixed) |
| 384 | return(scenario) | |
| 283 | scenario | |
| 284 | } | |
| 285 | ||
| 286 | #' @rdname scenario_update_paths | |
| 287 | #' @export | |
| 288 | scenario.update.paths <- function(scenario, from, to, fixed = TRUE) | |
| 289 | { | |
| 290 | .Deprecated("scenario_update_paths") | |
| 291 | scenario_update_paths(scenario=scenario, from=from, to=to, fixed=fixed) | |
| 385 | 292 | } |
| 386 | 293 | |
| 387 | 294 | # This function is used to trim potentially large strings for printing, since |
| 388 | 295 | # the maximum error/warning length is 8170 characters (R 3.0.2) |
| 389 | 296 | strlimit <- function(str, limit = 5000) |
| 390 | 297 | { |
| 391 | if (nchar(str) > limit) { | |
| 392 | return(paste0(substr(str, 1, limit - 3), "...")) | |
| 393 | } | |
| 298 | if (nchar(str) > limit) return(paste0(substr(str, 1, limit - 3), "...")) | |
| 394 | 299 | return(str) |
| 395 | 300 | } |
| 396 | 301 | |
| 397 | 302 | test.type.order.str <- function(test.type) |
| 398 | 303 | { |
| 399 | return (switch(test.type, | |
| 400 | friedman = "sum of ranks", | |
| 401 | t.none =, # Fall-throught | |
| 402 | t.holm =, # Fall-throught | |
| 403 | t.bonferroni = "mean value", | |
| 404 | irace.internal.error ("test.type.order.str() Invalid value '", | |
| 405 | test.type, "' of test.type"))) | |
| 304 | switch(test.type, | |
| 305 | friedman = "sum of ranks", | |
| 306 | t.none =, # Fall-throught | |
| 307 | t.holm =, # Fall-throught | |
| 308 | t.bonferroni = "mean value", | |
| 309 | irace.internal.error ("test.type.order.str() Invalid value '", | |
| 310 | test.type, "' of test.type")) | |
| 406 | 311 | } |
| 407 | 312 | |
| 408 | 313 | trim.leading <- function(str) |
| 409 | { | |
| 410 | return (sub('^[[:space:]]+', '', str)) ## white space, POSIX-style | |
| 411 | } | |
| 314 | sub('^[[:space:]]+', '', str) ## white space, POSIX-style | |
| 315 | ||
| 412 | 316 | trim.trailing <- function(str) |
| 413 | { | |
| 414 | return (sub('[[:space:]]+$', '', str)) ## white space, POSIX-style | |
| 415 | } | |
| 317 | sub('[[:space:]]+$', '', str) ## white space, POSIX-style | |
| 318 | ||
| 416 | 319 | # remove leading and trailing white space characters |
| 417 | trim <- function(str) | |
| 418 | { | |
| 419 | return (trim.trailing(trim.leading(str))) | |
| 420 | } | |
| 320 | trim <- function(str) trim.trailing(trim.leading(str)) | |
| 421 | 321 | |
| 422 | 322 | isFixed <- function (paramName, parameters) |
| 423 | { | |
| 424 | return (as.logical(parameters$isFixed[paramName])) | |
| 425 | } | |
| 323 | as.logical(parameters$isFixed[paramName]) | |
| 426 | 324 | |
| 427 | 325 | paramDomain <- function (paramName, parameters) |
| 428 | { | |
| 429 | return (parameters$domain[[paramName]]) | |
| 430 | } | |
| 326 | parameters$domain[[paramName]] | |
| 431 | 327 | |
| 432 | 328 | paramLowerBound <- function (paramName, parameters) |
| 433 | { | |
| 434 | return (as.numeric(parameters$domain[[paramName]][1])) | |
| 435 | } | |
| 329 | as.numeric(parameters$domain[[paramName]][1]) | |
| 436 | 330 | |
| 437 | 331 | paramUpperBound <- function (paramName, parameters) |
| 438 | { | |
| 439 | return (as.numeric(parameters$domain[[paramName]][2])) | |
| 440 | } | |
| 441 | ||
| 442 | nbParam <- function (parameters) | |
| 443 | { | |
| 444 | return (length(parameters$names)) | |
| 445 | } | |
| 332 | as.numeric(parameters$domain[[paramName]][2]) | |
| 333 | ||
| 334 | ||
| 335 | inNumericDomain <- function(value, domain) (value >= domain[1] && value <= domain[2]) | |
| 336 | ||
| 337 | nbParam <- function (parameters) length(parameters$names) | |
| 446 | 338 | |
| 447 | 339 | ## This function takes two matrices x and y and merges them such that the |
| 448 | 340 | ## resulting matrix z has: |
| 480 | 372 | # Input: the configurations with the .RANK. field filled. |
| 481 | 373 | # the number of elites wished |
| 482 | 374 | # Output: nbElites elites, sorted by ranks, with the weights assigned. |
| 483 | extractElites <- function(configurations, nbElites) | |
| 484 | { | |
| 375 | extractElites <- function(scenario, parameters, configurations, nbElites) | |
| 376 | { | |
| 377 | # Keep only alive configurations. | |
| 378 | ## FIXME: Shouldn't this be done by the caller? | |
| 379 | configurations <- configurations[configurations$.ALIVE., , drop = FALSE] | |
| 485 | 380 | if (nbElites < 1) { |
| 486 | ## FIXME: Should this be an error or should we handle it in some other way? | |
| 487 | irace.error("nbElites is lower or equal to zero.") | |
| 488 | } | |
| 381 | irace.internal.error("nbElites is lower or equal to zero.") | |
| 382 | } | |
| 383 | # Remove duplicated. Duplicated configurations may be generated, however, it | |
| 384 | # is too slow to check at generation time. Nevertheless, we can check now | |
| 385 | # since we typically have very few elites. | |
| 386 | ## FIXME: Use a variant of similarConfigurations. | |
| 387 | configurations <- configurations[order(configurations$.ID.), , drop = FALSE] | |
| 388 | before <- nrow(configurations) | |
| 389 | configurations <- configurations[!duplicated(removeConfigurationsMetaData(configurations)), | |
| 390 | , drop = FALSE] | |
| 391 | after <- nrow(configurations) | |
| 392 | if (after < before && scenario$debugLevel >= 1) { | |
| 393 | irace.note("Dropped ", before - after, " duplicated elites\n") | |
| 394 | } | |
| 395 | ||
| 396 | nbElites <- min(after, nbElites) | |
| 489 | 397 | # Sort by rank. |
| 490 | elites <- configurations[order(as.numeric(configurations$.RANK.)), , drop = FALSE] | |
| 398 | elites <- configurations[order(configurations$.RANK.), , drop = FALSE] | |
| 491 | 399 | elites <- elites[1:nbElites, , drop = FALSE] |
| 492 | 400 | elites[, ".WEIGHT."] <- ((nbElites - (1:nbElites) + 1) |
| 493 | 401 | / (nbElites * (nbElites + 1) / 2)) |
| 494 | return (elites) | |
| 402 | elites | |
| 495 | 403 | } |
| 496 | 404 | |
| 497 | 405 | #' removeConfigurationsMetaData |
| 507 | 415 | #' @return The same matrix without the "metadata". |
| 508 | 416 | #' |
| 509 | 417 | #' @seealso |
| 510 | #' \code{\link{configurations.print.command}} to print the configurations as command lines. | |
| 511 | #' \code{\link{configurations.print}} to print the configurations as a data frame. | |
| 418 | #' [configurations.print.command()] to print the configurations as command lines. | |
| 419 | #' [configurations.print()] to print the configurations as a data frame. | |
| 512 | 420 | #' |
| 513 | 421 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 514 | 422 | #' @export |
| 516 | 424 | removeConfigurationsMetaData <- function(configurations) |
| 517 | 425 | { |
| 518 | 426 | # Meta-data colnames begin with "." |
| 519 | return (configurations[, grep("^\\.", colnames(configurations), invert = TRUE), | |
| 520 | drop = FALSE]) | |
| 427 | configurations[, grep("^\\.", colnames(configurations), invert = TRUE), | |
| 428 | drop = FALSE] | |
| 521 | 429 | } |
| 522 | 430 | |
| 523 | 431 | #' Print configurations as a data frame |
| 530 | 438 | #' @return None. |
| 531 | 439 | #' |
| 532 | 440 | #' @seealso |
| 533 | #' \code{\link{configurations.print.command}} to print the configurations as command-line strings. | |
| 441 | #' [configurations.print.command()] to print the configurations as command-line strings. | |
| 534 | 442 | #' |
| 535 | 443 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 536 | 444 | #' @export |
| 554 | 462 | #' @return None. |
| 555 | 463 | #' |
| 556 | 464 | #' @seealso |
| 557 | #' \code{\link{configurations.print}} to print the configurations as a data frame. | |
| 465 | #' [configurations.print()] to print the configurations as a data frame. | |
| 558 | 466 | #' |
| 559 | 467 | #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 560 | 468 | #' @export |
| 748 | 656 | formatC(proc.time()["elapsed"] - elapsed, |
| 749 | 657 | format = "f", digits = 2), "\n") |
| 750 | 658 | } |
| 751 | return(list(output = output, error = NULL)) | |
| 659 | # TODO: Return elapsed time so that we can report at the end the total | |
| 660 | # elapsed time taken by irace vs the time taken by the target-runner. | |
| 661 | list(output = output, error = NULL) | |
| 752 | 662 | } |
| 753 | 663 | |
| 754 | 664 | # Safe sampling of vector: |
| 777 | 687 | # } |
| 778 | 688 | |
| 779 | 689 | is.file.extension <- function(filename, ext) |
| 780 | { | |
| 781 | return(substring(filename, nchar(filename) + 1 - nchar(ext)) == ext) | |
| 782 | } | |
| 690 | substring(filename, nchar(filename) + 1 - nchar(ext)) == ext | |
| 783 | 691 | |
| 784 | 692 | # Same as !(x %in% table) |
| 785 | "%!in%" <- function (x, table) match(x, table, nomatch = 0L) == 0L | |
| 693 | "%!in%" <- function(x, table) match(x, table, nomatch = 0L) == 0L | |
| 786 | 694 | |
| 787 | 695 | irace_save_logfile <- function(iraceResults, scenario) |
| 788 | 696 | { |
| 790 | 698 | cwd <- setwd(scenario$execDir) |
| 791 | 699 | # FIXME: Use saveRDS |
| 792 | 700 | # FIXME: Bump to version=3 when we bump the minimum R version to >=3.6 |
| 793 | save (iraceResults, file = scenario$logFile, version = 2) | |
| 794 | setwd (cwd) | |
| 795 | } | |
| 701 | save(iraceResults, file = scenario$logFile, version = 2) | |
| 702 | setwd(cwd) | |
| 703 | } | |
| 704 | ||
| 705 | valid_iracelog <- function(x) | |
| 706 | { | |
| 707 | is.list(x) && ("scenario" %in% names(x)) | |
| 708 | } | |
| 709 | ||
| 710 | #' Read the log file produced by irace (`irace.Rdata`). | |
| 711 | #' | |
| 712 | #' @param filename Filename that contains the log file saved by irace. Example: `irace.Rdata`. | |
| 713 | #' | |
| 714 | #' @param name Optional argument that allows overriding the default name of the object in the file. | |
| 715 | #' | |
| 716 | #' @return (`list()`) | |
| 717 | #' @concept analysis | |
| 718 | #' @export | |
| 719 | read_logfile <- function(filename, name = "iraceResults") | |
| 720 | { | |
| 721 | # If filename is already the iraceResults object, just return it. | |
| 722 | if (valid_iracelog(filename)) return(filename) | |
| 723 | ||
| 724 | if (file.access(filename, mode=4) != 0) | |
| 725 | stop("read_logfile: Cannot read file '", filename, "'") | |
| 726 | ||
| 727 | load(filename) | |
| 728 | iraceResults <- get0(name, inherits=FALSE) | |
| 729 | if (!valid_iracelog(iraceResults)) | |
| 730 | stop("The file '", filename, "' does not contain the '", name, "' object.") | |
| 731 | ||
| 732 | iraceResults | |
| 733 | } | |
| 734 | ||
| 735 | do_nothing <- function(...) invisible() | |
| 0 | 0 | #' irace.version |
| 1 | 1 | #' |
| 2 | #' A character string containing the version of \pkg{irace}. | |
| 3 | #' | |
| 2 | #' A character string containing the version of `irace`. | |
| 4 | 3 | #' @export |
| 5 | irace.version <- '3.4.1.9fcaeaf' | |
| 4 | irace.version <- '3.5.6863679' |
| 0 | # Uses 10 decimal places at most so that there is space for '-', '.', and | |
| 1 | # 'e-NN' within the 16 spaces. | |
| 2 | .irace.format.perf <- "%#16.10g" | |
| 3 | ||
| 4 | .onLoad <- function(libname, pkgname) { | |
| 5 | # FIXME: We would like to use %#16.10g but this causes problems with | |
| 6 | # https://github.com/oracle/fastr/issues/191 | |
| 7 | R_engine <- R.version$engine | |
| 8 | if (!is.null(R_engine) && R_engine == "FastR") | |
| 9 | .irace.format.perf <<- "%16.10g" | |
| 10 | invisible() | |
| 11 | } |
| 0 | 0 | **irace**: Iterated Racing for Automatic Algorithm Configuration |
| 1 | 1 | ================================================================ |
| 2 | 2 | |
| 3 | [](https://cran.r-project.org/package=irace) [](https://CRAN.R-project.org/package=irace) [](https://travis-ci.org/MLopez-Ibanez/irace) | |
| 5 | ||
| 6 | [ [**Homepage**](http://iridia.ulb.ac.be/irace/) ] [ [**User Guide (PDF)**](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) ] | |
| 7 | ||
| 8 | **Maintainers:** [Manuel López-Ibáñez](http://lopez-ibanez.eu), Leslie Pérez Cáceres | |
| 9 | ||
| 10 | **Creators:** [Manuel López-Ibáñez](http://lopez-ibanez.eu), Jérémie Dubois-Lacoste | |
| 3 | <!-- badges: start --> | |
| 4 | [](https://cran.r-project.org/package=irace) [](https://CRAN.R-project.org/package=irace) [](https://github.com/MLopez-Ibanez/irace/actions) | |
| 6 | [](https://app.codecov.io/gh/MLopez-Ibanez/irace?branch=master) | |
| 7 | <!-- badges: end --> | |
| 8 | ||
| 9 | [ [**Homepage**](https://mlopez-ibanez.github.io/irace/) ] [ [**User Guide (PDF)**](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) ] | |
| 10 | ||
| 11 | **Maintainers:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Leslie Pérez Cáceres | |
| 12 | ||
| 13 | **Creators:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Jérémie Dubois-Lacoste | |
| 11 | 14 | |
| 12 | 15 | **Contributors:** Jérémie Dubois-Lacoste, Thomas Stützle, Mauro Birattari, Eric |
| 13 | 16 | Yuan and Prasanna Balaprakash. |
| 30 | 33 | **Relevant literature:** |
| 31 | 34 | |
| 32 | 35 | 1. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and |
| 33 | M. Birattari. [The irace package: Iterated Racing for Automatic Algorithm Configuration.](http://dx.doi.org/10.1016/j.orp.2016.09.002). | |
| 36 | M. Birattari. [The irace package: Iterated Racing for Automatic Algorithm Configuration](http://dx.doi.org/10.1016/j.orp.2016.09.002). | |
| 34 | 37 | *Operations Research Perspectives*, 3:43–58, 2016.<br> |
| 35 | [ [bibtex](http://lopez-ibanez.eu/LopezIbanez_bib.html#LopDubPerStuBir2016irace) | |
| 38 | [ [bibtex](https://lopez-ibanez.eu/LopezIbanez_bib.html#LopDubPerStuBir2016irace) | |
| 36 | 39 | | doi: [10.1016/j.orp.2016.09.002](http://dx.doi.org/10.1016/j.orp.2016.09.002) ] |
| 37 | 40 | |
| 38 | 41 | 2. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro |
| 39 | Birattari. [The irace package, Iterated Race for Automatic Algorithm Configuration](http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-004.pdf). | |
| 42 | Birattari. [The irace package, Iterated Race for Automatic Algorithm Configuration](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf). | |
| 40 | 43 | Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, |
| 41 | 44 | Belgium, 2011.<br> |
| 42 | [ [bibtex](http://iridia.ulb.ac.be/~manuel/LopezIbanez_bib.html#LopDubStu2011irace) | |
| 43 | | [PDF](http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-004.pdf) ] | |
| 44 | ||
| 45 | 3. Manuel López-Ibáñez. [The irace | |
| 46 | software package: A tutorial](http://iridia.ulb.ac.be/irace/files/irace-comex-tutorial.pdf). COMEX Workshop on Practical Automatic Algorithm Configuration, 2014.<br> | |
| 47 | [ [workshop webpage](http://iridia.ulb.ac.be/~manuel/comex_workshop/) | |
| 48 | | [PDF](http://iridia.ulb.ac.be/irace/files/irace-comex-tutorial.pdf) ] | |
| 45 | [ [bibtex](https://iridia-ulb.github.io/references/index.html#LopDubStu2011irace) | |
| 46 | | [PDF](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf) ] | |
| 47 | ||
| 48 | 3. Thomas Stützle and Manuel López-Ibáñez. [Tutorial: Automated algorithm | |
| 49 | configuration and design](https://doi.org/10.1145/3449726.3461404). GECCO | |
| 50 | '21: Proceedings of the Genetic and Evolutionary Computation Conference | |
| 51 | Companion, July 2021. | |
| 52 | [doi:10.1145/3449726.3461404](https://doi.org/10.1145/3449726.3461404) | |
| 49 | 53 | |
| 50 | 54 | |
| 51 | 55 | Requisites |
| 69 | 73 | The following is a quick-start guide. The user guide gives more detailed |
| 70 | 74 | instructions. |
| 71 | 75 | |
| 76 | Quick Start | |
| 77 | =========== | |
| 78 | 1. Install R (with your favourite package manager, and see more details below). | |
| 79 | 2. Install irace. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). | |
| 80 | ```bash | |
| 81 | $ Rscript -e "install.packages('irace', repos='https://cloud.r-project.org')" | |
| 82 | ``` | |
| 83 | ||
| 84 | 3. Add irace to path. For windows user, this step is unfortunately more involved, so please see more detailed instructions below. | |
| 85 | ```bash | |
| 86 | $ export PATH="$(Rscript -e "cat(paste0(system.file(package='irace', 'bin', mustWork=TRUE), ':'))" 2> /dev/null)${PATH}" | |
| 87 | ``` | |
| 88 | Consider adding this line to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` for it to persist between sessions. | |
| 89 | ||
| 90 | 4. You can open the user guide with the following command. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). | |
| 91 | ```bash | |
| 92 | $ Rscript -e "vignette('irace-package')" | |
| 93 | ``` | |
| 94 | ||
| 72 | 95 | Installing R |
| 73 | 96 | ============ |
| 74 | 97 | |
| 94 | 117 | You can install R directly from a CRAN mirror |
| 95 | 118 | (<https://cran.r-project.org/bin/macosx/>). |
| 96 | 119 | |
| 97 | Alternatively, if you use homebrew, you can just brew the R formula | |
| 98 | from the science tap (unfortunately it does not come already bottled | |
| 99 | so you need to have Xcode installed to compile it): | |
| 100 | ||
| 101 | ```bash | |
| 102 | $ brew tap homebrew/science | |
| 103 | $ brew install r | |
| 104 | ``` | |
| 120 | Alternatively, if you use homebrew, you can just do | |
| 121 | ``` | |
| 122 | $ brew install --cask r | |
| 123 | ``` | |
| 124 | ||
| 125 | (Using `brew install r` is not recommended because that will build R from source and you will not be able to use any CRAN binary, possibly resulting in annoying build failiures). | |
| 105 | 126 | |
| 106 | 127 | Once R is installed, you can launch R from the Terminal (or from your |
| 107 | 128 | Applications), and from the R prompt install the irace package. See |
| 115 | 136 | launch the R console and install the irace package from it. See instructions |
| 116 | 137 | below. |
| 117 | 138 | |
| 139 | In addition to using the R console, it might be very useful to add R to PATH so you can run most of the GUN/Linux shell commands without modification in CMD or Powershell. Usually, R is installed in `C:\Program Files\R\R-4.1.3` (the version number depends on your installation). | |
| 140 | ||
| 141 | You should add the following line [to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/) (if you want to use the 64-bit version) | |
| 142 | ``` | |
| 143 | C:\Program Files\R\R-4.1.3\bin\x64 | |
| 144 | ``` | |
| 145 | ||
| 146 | Or, if you are on a 32-bit version | |
| 147 | ``` | |
| 148 | C:\Program Files\R\R-4.1.3\bin\i386 | |
| 149 | ``` | |
| 118 | 150 | |
| 119 | 151 | Installing the irace package |
| 120 | 152 | ============================ |
| 121 | 153 | |
| 122 | There are two methods for installing the [irace](http://iridia.ulb.ac.be/irace) R package on your | |
| 123 | computer: | |
| 154 | There are two methods for installing the | |
| 155 | [irace](https://mlopez-ibanez.github.io/irace/) R package on your computer: | |
| 124 | 156 | |
| 125 | 157 | 1. Install within R (automatic download): |
| 126 | 158 | ```R |
| 155 | 187 | ```R |
| 156 | 188 | $ R |
| 157 | 189 | R> library(irace) |
| 158 | R> system.file(package="irace") | |
| 159 | [1] "~/R/irace" | |
| 190 | R> cat(system.file(package="irace", "bin", mustWork=TRUE), "\n") | |
| 191 | /home/user/R/irace/bin | |
| 160 | 192 | ``` |
| 161 | 193 | The last command tells you the installation directory of `irace`. |
| 162 | 194 | |
| 167 | 199 | `.bash_profile`, `.bashrc` or `.profile`: |
| 168 | 200 | |
| 169 | 201 | ```bash |
| 170 | export IRACE_HOME=~/R/irace/ # Path given by system.file(package="irace") | |
| 171 | export PATH=${IRACE_HOME}/bin/:$PATH | |
| 202 | export IRACE_HOME=/home/user/R/irace/bin/ # Path given by system.file(package="irace", "bin", mustWork=TRUE) | |
| 203 | export PATH=${IRACE_HOME}:$PATH | |
| 172 | 204 | # export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced |
| 173 | 205 | ``` |
| 174 | 206 | |
| 183 | 215 | Windows |
| 184 | 216 | ------- |
| 185 | 217 | |
| 186 | If the installation directory of `irace` is `C:/R/irace/`, you can invoke | |
| 187 | `irace` by opening a terminal (launch the program `cmd.exe`) and executing: | |
| 188 | ||
| 189 | ```bash | |
| 190 | C:\> C:\R\irace\bin\x64\irace.exe --help | |
| 191 | ``` | |
| 192 | ||
| 193 | or if you are in a 32-bits system, executing: | |
| 194 | ||
| 195 | ```bash | |
| 196 | C:\> C:\R\irace\bin\i386\irace.exe --help | |
| 218 | You can find out where the irace binary is installed by running the following in Powershell or CMD: | |
| 219 | ||
| 220 | ```Powershell | |
| 221 | C:\> Rscript -e "cat(gsub('/', '\\\\', system.file(package='irace', 'bin', 'x64', mustWork=TRUE)))" | |
| 222 | ``` | |
| 223 | ||
| 224 | It will output a path, such as `C:\Program Files\R\R-4.1.3\library\irace\bin\x64` (replace `x64` with `i386` if you are on a 32-bit system), which can you [add to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/). | |
| 225 | ||
| 226 | Then running the following should work: | |
| 227 | ```Powershell | |
| 228 | C:\> irace --help | |
| 197 | 229 | ``` |
| 198 | 230 | |
| 199 | 231 | You can also launch irace by opening the R console and executing: |
| 203 | 235 | R> irace.cmdline("--help") |
| 204 | 236 | ``` |
| 205 | 237 | |
| 238 | GitHub (Development version) | |
| 239 | ---------------------------- | |
| 240 | ||
| 241 | If you wish to try the development version, you can install it by executing the | |
| 242 | following commands within the R console: | |
| 243 | ||
| 244 | ```R | |
| 245 | R> install.packages("devtools") | |
| 246 | R> devtools::install_github("MLopez-Ibanez/irace") | |
| 247 | ``` | |
| 248 | ||
| 249 | Python | |
| 250 | ------ | |
| 251 | ||
| 252 | You can use the irace R package from Python using `rpy2`. An example on how to do this is the implementation of [iracepy](https://github.com/auto-optimization/iracepy). | |
| 253 | ||
| 254 | ||
| 206 | 255 | Usage |
| 207 | 256 | ===== |
| 208 | 257 | |
| 212 | 261 | $ cd ~/tuning |
| 213 | 262 | ``` |
| 214 | 263 | |
| 215 | 2. Copy the template and example files to the scenario directory | |
| 216 | ```bash | |
| 217 | $ cp $IRACE_HOME/templates/*.tmpl . | |
| 218 | ``` | |
| 219 | ||
| 220 | where `$IRACE_HOME` is the path to the installation directory of | |
| 221 | `irace`. It can be obtained by doing: | |
| 222 | ||
| 223 | ```R | |
| 224 | $ R | |
| 225 | > library(irace) | |
| 226 | > system.file(package="irace") | |
| 227 | ``` | |
| 228 | ||
| 229 | 3. For each template in your tuning directory, remove the `.tmpl` | |
| 230 | suffix, and modify them following the instructions in each file. In | |
| 231 | particular, | |
| 264 | 2. Initialize your tuning directory with template config files | |
| 265 | ```bash | |
| 266 | $ $IRACE_HOME/bin/irace --init | |
| 267 | ``` | |
| 268 | ||
| 269 | 3. Modify the generated files following the instructions found within each file. In particular, | |
| 232 | 270 | * The scripts `target-runner` and `target-evaluator` (if you need it at all) |
| 233 | 271 | should be executable. The output of `target-runner` (or |
| 234 | 272 | `target-evaluator` if you use a separate evaluation step) is minimized by |
| 242 | 280 | 4. Put the instances in `~/tuning/Instances/`. In addition, you can |
| 243 | 281 | create a file that specifies which instances from that directory |
| 244 | 282 | should be run and which instance-specific parameters to use. See |
| 245 | `scenario.txt.tmpl` and `instances-list.tmpl` for examples. The command | |
| 283 | `scenario.txt` and `instances-list.txt` for examples. The command | |
| 246 | 284 | irace will not attempt to create the execution directory (`execDir`), |
| 247 | 285 | so it must exist before calling irace. The default `execDir` is the |
| 248 | 286 | current directory. |
| 251 | 289 | ```bash |
| 252 | 290 | $ cd ~/tuning/ && $IRACE_HOME/bin/irace |
| 253 | 291 | ``` |
| 254 | ||
| 255 | performs one run of Iterated Race. See the output of `irace --help` for | |
| 256 | additional irace parameters. Command-line parameters override the | |
| 257 | scenario setup specified in the `scenario.txt` file. | |
| 292 | performs one run of Iterated Race. See the output of `irace --help` for | |
| 293 | additional irace parameters. Command-line parameters override the | |
| 294 | scenario setup specified in the `scenario.txt` file. | |
| 258 | 295 | |
| 259 | 296 | |
| 260 | 297 | Many tuning runs in parallel |
| 318 | 355 | Birattari. |
| 319 | 356 | |
| 320 | 357 | |
| 358 | Building an irace standalone container | |
| 359 | ====================================== | |
| 360 | ||
| 361 | Thanks to [Singularity](https://sylabs.io/singularity/), you can build a | |
| 362 | standalone container of `irace` using the file `irace.sindef` which is | |
| 363 | available in the directory `inst/` in the source tarball and github repository | |
| 364 | or, after installing the irace R package, in the installation directory given | |
| 365 | by the R expression `system.file(package="irace")`. After installing | |
| 366 | SingularityCE, the container may be build using: | |
| 367 | ||
| 368 | sudo singularity build irace.sindef irace.sif | |
| 369 | ||
| 370 | and run with: | |
| 371 | ||
| 372 | singularity run irace.sif <arguments> | |
| 373 | ||
| 374 | ||
| 375 | ||
| 321 | 376 | Frequently Asked Questions |
| 322 | 377 | ========================== |
| 323 | 378 | |
Binary diff not shown
Binary diff not shown
| 0 | 0 | #!/bin/sh |
| 1 | 1 | rm -f irace-Ex.R config.* confdefs.h \ |
| 2 | src/*.o src/*.so src/config.h src/symbols.rds \ | |
| 2 | src/*.so src/*.o src/*/*.o src/*.gcno src/*/*.gcno src/config.h src/symbols.rds \ | |
| 3 | 3 | inst/doc/*.blg inst/doc/*.bbl \ |
| 4 | 4 | tests/testthat/*.log tests/testthat/*.Rout tests/testthat/irace.Rdata |
| 5 | 5 | |
| 6 | find . -name '*.orig' -o -name '.Rhistory' | xargs rm -f | |
| 6 | find . -name '*.orig' -o -name '.Rhistory' -o -name '*.rej' | xargs rm -f | |
| 7 | 7 | |
| 8 | 8 | rm -rf autom4te.cache |
| 9 | 9 |
| 59 | 59 | |
| 60 | 60 | usage() { |
| 61 | 61 | cat <<EOF |
| 62 | Usage: $0 N[-M] [EXECDIR] --parallel <NUM> [--machine MACHINE] [IRACE PARAMS] | |
| 62 | Usage: $0 N[-M] [EXECDIR] --parallel NUM [--seed NUM] [--machine MACHINE] [IRACE PARAMS] | |
| 63 | 63 | |
| 64 | 64 | Parameters: |
| 65 | 65 | N an integer giving the number of repetitions of irace |
| 66 | 66 | or a sequence N-M giving which repetitions to redo. |
| 67 | 67 | EXECDIR job M will use EXECDIR-M directory (default: execdir-) |
| 68 | 68 | as the execDir (--exec-dir) parameter of irace. |
| 69 | --parallel <NUM> number of parallel jobs | |
| 69 | --parallel NUM number of parallel jobs (value in scenario.txt is ignored) | |
| 70 | --seed NUM the seed of each run i will be SEED+i-1 (default: 1234567) | |
| 70 | 71 | --machine MACHINE qsub queue type, e.g., opteron6272 |
| 71 | 72 | IRACE PARAMS additional parameters for irace. |
| 72 | 73 | EOF |
| 54 | 54 | exit \$RET |
| 55 | 55 | EOF |
| 56 | 56 | } |
| 57 | ## End of customization | |
| 57 | ## END OF CUSTOMIZATION | |
| 58 | 58 | |
| 59 | 59 | error () { |
| 60 | 60 | echo "$0: error: $@" >&2 |
| 63 | 63 | |
| 64 | 64 | usage() { |
| 65 | 65 | cat <<EOF |
| 66 | Usage: $0 N[-M] [EXECDIR] [IRACE PARAMS] | |
| 66 | Usage: $0 N[-M] [EXECDIR] [--seed NUM] [IRACE PARAMS] | |
| 67 | 67 | |
| 68 | 68 | Parameters: |
| 69 | 69 | N an integer giving the number of repetitions of irace |
| 70 | 70 | or a sequence N-M giving which repetitions to redo. |
| 71 | 71 | EXECDIR job M will use EXECDIR-M directory (default: execdir) |
| 72 | 72 | as the execDir (--exec-dir) parameter of irace. |
| 73 | --seed NUM the seed of each run i will be SEED+i-1 (default: 1234567) | |
| 73 | 74 | IRACE PARAMS additional parameters for irace. |
| 74 | 75 | EOF |
| 75 | 76 | exit 1 |
| 0 | #!/usr/bin/env bash | |
| 1 | ############################################################################### | |
| 2 | # This script launches several runs of irace in parallel on a SLURM cluster. | |
| 3 | # Execute without parameters to see usage. | |
| 4 | ############################################################################### | |
| 5 | set -e | |
| 6 | set -o pipefail | |
| 7 | # Find our own location. | |
| 8 | BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") | |
| 9 | IRACE="$BINDIR/irace" | |
| 10 | # This function launches one run of irace. | |
| 11 | irace_main() { | |
| 12 | # We would like to use $BASHPID here, but OS X version of bash does not | |
| 13 | # support it. | |
| 14 | JOBNAME="irace-$$-$1" | |
| 15 | # You may need to CUSTOMIZE this part according to the setup of your own cluster: https://slurm.schedmd.com | |
| 16 | exec sbatch <(cat<<EOF | |
| 17 | #!/usr/bin/env bash | |
| 18 | # Amount of RAM needed for this job: | |
| 19 | #SBATCH --mem=2gb | |
| 20 | # The time the job will be running: | |
| 21 | #SBATCH --time=60:00:00 | |
| 22 | # To use GPUs you have to request them: | |
| 23 | ##SBATCH --gres=gpu:1 | |
| 24 | #SBATCH --constraint=cal | |
| 25 | # The name to show in queue lists for this job: | |
| 26 | #SBATCH -J $JOBNAME | |
| 27 | #SBATCH --array=1-$N_RUNS | |
| 28 | # Number of desired cpus: | |
| 29 | #SBATCH --cpus-per-task=$N_CPUS | |
| 30 | # Set output and error files | |
| 31 | #SBATCH --error=$EXECDIR/irace.stderr | |
| 32 | #SBATCH --output=$EXECDIR/irace.stdout | |
| 33 | # To load some software (you can show the list with 'module avail'): | |
| 34 | module load R/4.1.0_sin | |
| 35 | run=\$SLURM_ARRAY_TASK_ID | |
| 36 | exec $IRACE --exec-dir=$EXECDIR --seed $RUNSEED $PARAMS | |
| 37 | EOF | |
| 38 | ) | |
| 39 | } | |
| 40 | ## END OF CUSTOMIZATION | |
| 41 | ||
| 42 | error () { | |
| 43 | echo "$0: error: $@" >&2 | |
| 44 | exit 1 | |
| 45 | } | |
| 46 | ||
| 47 | usage() { | |
| 48 | cat <<EOF | |
| 49 | Usage: $0 N[-M] [EXECDIR] --parallel NUM [--seed NUM] [IRACE PARAMS] | |
| 50 | ||
| 51 | Parameters: | |
| 52 | N an integer giving the number of repetitions of irace | |
| 53 | or a sequence N-M giving which repetitions to redo. | |
| 54 | EXECDIR job M will use EXECDIR-M directory (default: execdir) | |
| 55 | as the execDir (--exec-dir) parameter of irace. | |
| 56 | --parallel NUM number of parallel jobs (value in scenario.txt is ignored) | |
| 57 | --seed NUM the seed of each run i will be SEED+i-1 (default: 1234567) | |
| 58 | IRACE PARAMS additional parameters for irace. | |
| 59 | EOF | |
| 60 | exit 1 | |
| 61 | } | |
| 62 | ||
| 63 | # Issue usage if no parameters are given. | |
| 64 | test $# -ge 1 || usage | |
| 65 | ||
| 66 | # Number of repetitions of iRace | |
| 67 | REPETITIONS=$1 | |
| 68 | shift | |
| 69 | START=1 | |
| 70 | if [[ "$REPETITIONS" =~ ^([0-9]+)-([0-9]+)$ ]] ; then | |
| 71 | START=${BASH_REMATCH[1]} | |
| 72 | REPETITIONS=${BASH_REMATCH[2]} | |
| 73 | elif ! [[ "$REPETITIONS" =~ ^[0-9]+$ ]] ; then | |
| 74 | error "number of repetitions must be an integer" | |
| 75 | fi | |
| 76 | ||
| 77 | # execDir (--exec-dir) directory | |
| 78 | EXECDIR_PREFIX=${1:-execdir} | |
| 79 | shift | |
| 80 | SEED=1234567 | |
| 81 | N_CPUS=1 | |
| 82 | PARAMS= | |
| 83 | while [ $# -gt 0 ]; do | |
| 84 | case "$1" in | |
| 85 | --parallel) shift; N_CPUS="$1"; shift;; | |
| 86 | --seed) shift; SEED="$1"; shift;; | |
| 87 | *) PARAMS="$PARAMS $1"; shift;;# terminate case | |
| 88 | esac | |
| 89 | done | |
| 90 | ||
| 91 | # FIXME: Use SBATCH --array=$START-$REPETITIONS | |
| 92 | N_RUNS=1 | |
| 93 | for i in $(seq $START $REPETITIONS); do | |
| 94 | EXECDIR=$(printf '%s-%002d' ${EXECDIR_PREFIX} $i) | |
| 95 | echo "execution directory: $EXECDIR" | |
| 96 | rm -rf $EXECDIR | |
| 97 | mkdir -p $EXECDIR | |
| 98 | let RUNSEED=SEED+i-1 | |
| 99 | irace_main $(printf '%002d' $i) & | |
| 100 | sleep 1 | |
| 101 | done |
| 0 | ## ----include=FALSE------------------------------------------------------- | |
| 0 | ## ----include=FALSE------------------------------------------------------------ | |
| 1 | 1 | library(knitr) |
| 2 | 2 | |
| 3 | 3 | ## ----exampleload,eval=TRUE,include=FALSE---------------------------- |
| 4 | 4 | library("irace") |
| 5 | 5 | load("examples.Rdata") |
| 6 | load("irace-acotsp.Rdata") | |
| 7 | load("log-ablation.Rdata") | |
| 6 | iraceResults <- irace::read_logfile("irace-acotsp.Rdata") | |
| 7 | log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 8 | load(log_ablation_file) | |
| 8 | 9 | options(width = 70) |
| 9 | 10 | |
| 10 | 11 | ## ----R_irace_install, prompt=FALSE, eval=FALSE---------------------- |
| 89 | 90 | # { |
| 90 | 91 | # isreal <- parameters$type[colnames(configuration)] %in% "r" |
| 91 | 92 | # configuration[isreal] <- configuration[isreal] / sum(configuration[isreal]) |
| 93 | # configurations[isreal] <- round(configuration[isreal], digits) | |
| 92 | 94 | # return(configuration) |
| 93 | 95 | # } |
| 94 | 96 | |
| 116 | 118 | print(output) |
| 117 | 119 | |
| 118 | 120 | ## ----testing_r, prompt=FALSE, eval=FALSE---------------------------- |
| 119 | # testing.main(logFile = "./irace.Rdata") | |
| 121 | # testing_fromlog(logFile = "./irace.Rdata", testNbElites = 1) | |
| 120 | 122 | |
| 121 | 123 | ## ----change_recover, prompt=TRUE, eval=FALSE------------------------ |
| 122 | 124 | # load ("~/tuning/irace.Rdata") |
| 141 | 143 | print(iraceResults$allElites) |
| 142 | 144 | |
| 143 | 145 | ## ----get_elites, prompt=TRUE, eval=TRUE, comment=""----------------- |
| 144 | getFinalElites(logFile = "irace-acotsp.Rdata", n = 0) | |
| 146 | getFinalElites("irace-acotsp.Rdata", n = 0) | |
| 145 | 147 | |
| 146 | 148 | ## ----show_iditelites, prompt=TRUE, eval=TRUE, comment=""------------ |
| 147 | 149 | print(iraceResults$iterationElites) |
| 149 | 151 | ## ----get_elite, prompt=TRUE, eval=TRUE, comment=""------------------ |
| 150 | 152 | last <- length(iraceResults$iterationElites) |
| 151 | 153 | id <- iraceResults$iterationElites[last] |
| 152 | getConfigurationById(logFile = "irace-acotsp.Rdata", ids = id) | |
| 154 | getConfigurationById(iraceResults, ids = id) | |
| 153 | 155 | |
| 154 | 156 | ## ----get_experiments, prompt=TRUE, eval=TRUE, comment=""------------ |
| 155 | 157 | # As an example, we use the best configuration found |
| 164 | 166 | # As an example, we get seed and instance of the experiments |
| 165 | 167 | # of the best candidate. |
| 166 | 168 | # Get index of the instances |
| 167 | pair.id <- names(all.exp[!is.na(all.exp)]) | |
| 168 | index <- iraceResults$state$.irace$instancesList[pair.id,"instance"] | |
| 169 | pair.id <- which(!is.na(all.exp)) | |
| 170 | index <- iraceResults$state$.irace$instancesList[pair.id, "instance"] | |
| 169 | 171 | # Obtain the instance names |
| 170 | 172 | iraceResults$scenario$instances[index] |
| 171 | 173 | # Get the seeds |
| 172 | iraceResults$state$.irace$instancesList[index,"seed"] | |
| 174 | iraceResults$state$.irace$instancesList[pair.id, "seed"] | |
| 173 | 175 | |
| 174 | 176 | ## ----get_model, prompt=TRUE, eval=TRUE, comment=""------------------ |
| 175 | 177 | # As an example, we get the model probabilities for the |
| 186 | 188 | # Get the seeds used for testing |
| 187 | 189 | iraceResults$testing$seeds |
| 188 | 190 | |
| 189 | ## ----plot_test, fig.pos="tbp", fig.align="center", fig.height = 4, fig.width = 8, out.width='0.85\\textwidth', fig.cap="Boxplot of the testing results of the best configurations.", prompt=TRUE, eval=TRUE, comment=""---- | |
| 191 | ## ----wilcox_test,prompt=TRUE, eval=TRUE, comment=""----------------- | |
| 190 | 192 | results <- iraceResults$testing$experiments |
| 191 | 193 | # Wilcoxon paired test |
| 192 | 194 | conf <- gl(ncol(results), # number of configurations |
| 193 | 195 | nrow(results), # number of instances |
| 194 | 196 | labels = colnames(results)) |
| 195 | 197 | pairwise.wilcox.test (as.vector(results), conf, paired = TRUE, p.adj = "bonf") |
| 196 | # Plot the results | |
| 197 | configurationsBoxplot (results, ylab = "Solution cost") | |
| 198 | ||
| 199 | ## ----freq, fig.pos="tbp", fig.cap="Parameters sampling frequency.", out.width="\\textwidth",prompt=TRUE, eval=TRUE, comment=""---- | |
| 200 | parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 201 | ||
| 202 | ## ----parcord, fig.pos="tbp", fig.align="center", out.width="0.7\\textwidth", fig.cap="Parallel coordinate plots of the parameters of the configurations in the last two iterations of a run of \\irace.", prompt=FALSE, eval=TRUE, comment=""---- | |
| 203 | # Get last iteration number | |
| 204 | last <- length(iraceResults$iterationElites) | |
| 205 | # Get configurations in the last two iterations | |
| 206 | conf <- getConfigurationByIteration(iraceResults = iraceResults, | |
| 207 | iterations = c(last - 1, last)) | |
| 208 | parallelCoordinatesPlot (conf, iraceResults$parameters, | |
| 209 | param_names = c("algorithm", "alpha", "beta", "rho", "q0"), | |
| 210 | hierarchy = FALSE) | |
| 211 | ||
| 212 | ## ----testEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment=""---- | |
| 198 | ||
| 199 | ## ----conc, prompt=TRUE, eval=TRUE, comment=""----------------------- | |
| 200 | irace:::concordance(iraceResults$testing$experiments) | |
| 201 | ||
| 202 | ## ----testEvo, fig.pos="tb", fig.align="center", out.width='0.7\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment=""---- | |
| 213 | 203 | # Get number of iterations |
| 214 | 204 | iters <- unique(iraceResults$experimentLog[, "iteration"]) |
| 215 | 205 | # Get number of experiments (runs of target-runner) up to each iteration |
| 218 | 208 | # for the best configuration of that iteration. |
| 219 | 209 | elites <- as.character(iraceResults$iterationElites) |
| 220 | 210 | values <- colMeans(iraceResults$testing$experiments[, elites]) |
| 211 | stderr <- function(x) sqrt(var(x)/length(x)) | |
| 212 | err <- apply(iraceResults$testing$experiments[, elites], 2, stderr) | |
| 221 | 213 | plot(fes, values, type = "s", |
| 222 | 214 | xlab = "Number of runs of the target algorithm", |
| 223 | ylab = "Mean value over testing set") | |
| 224 | points(fes, values) | |
| 215 | ylab = "Mean value over testing set", ylim=c(23000000,23500000)) | |
| 216 | points(fes, values, pch=19) | |
| 217 | arrows(fes, values - err, fes, values + err, length=0.05, angle=90, code=3) | |
| 225 | 218 | text(fes, values, elites, pos = 1) |
| 226 | 219 | |
| 227 | 220 | ## ----ablation, prompt=FALSE, eval=FALSE----------------------------- |
| 228 | # ablation(iraceLogFile = "irace.Rdata", | |
| 229 | # src = 1, target = 60, pdf.file = "plot-ablation.pdf") | |
| 230 | ||
| 231 | ## ----testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{ablation()}.", prompt=FALSE, eval=TRUE, echo=FALSE---- | |
| 232 | plotAblation(abLogFile = "log-ablation.Rdata") | |
| 221 | # ablog <- ablation("irace.Rdata", src = 1, target = 60) | |
| 222 | # plotAblation(ablog) | |
| 223 | ||
| 224 | ## ----testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{plotAblation()}.", prompt=FALSE, eval=TRUE, echo=FALSE---- | |
| 225 | logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 226 | plotAblation(logfile) | |
| 227 | ||
| 228 | ## ----ablation_cmdline, prompt=FALSE, eval=TRUE,echo=FALSE----------- | |
| 229 | ablation_cmdline("--help") | |
| 233 | 230 | |
| 234 | 231 | ## ----postsel, prompt=FALSE, eval=FALSE------------------------------ |
| 235 | 232 | # # Execute all elite configurations in the iterations |
| 5 | 5 | %\VignetteDepends{knitr} |
| 6 | 6 | %\VignetteCompiler{knitr} |
| 7 | 7 | \synctex=1 |
| 8 | \RequirePackage{xparse} | |
| 8 | 9 | \RequirePackage[dvipsnames]{xcolor} |
| 9 | 10 | \documentclass[a4paper,english]{article} |
| 11 | \usepackage[T1]{fontenc} | |
| 12 | \usepackage[utf8]{inputenc} | |
| 10 | 13 | \usepackage[a4paper]{geometry} % It saves some pages |
| 11 | \usepackage[utf8]{inputenc} | |
| 12 | \usepackage[T1]{fontenc} | |
| 13 | 14 | \usepackage[english]{babel} |
| 14 | 15 | \usepackage{ifthen} |
| 15 | 16 | \newboolean{Release} |
| 23 | 24 | \usepackage{amsmath,amssymb} |
| 24 | 25 | \usepackage{relsize} |
| 25 | 26 | \usepackage{fancyvrb} |
| 27 | \usepackage{microtype} | |
| 28 | % \texttt{test -- test} keeps the "--" as "--" (and does not convert it to an en dash) | |
| 29 | \DisableLigatures{encoding = T1, family = tt* } | |
| 26 | 30 | \usepackage[hyphens]{url} |
| 27 | 31 | \usepackage{hyperref} |
| 28 | 32 | \usepackage[numbers]{natbib} |
| 97 | 101 | \makeatletter |
| 98 | 102 | \DeclareRobustCommand\code{\bgroup\@makeother\_\@makeother\~\@makeother\$\@noligs\@codex} |
| 99 | 103 | \def\@codex#1{\texorpdfstring% |
| 100 | {{\normalfont\ttfamily\hyphenchar\font=-1 #1}}% | |
| 104 | {{\text{\normalfont\ttfamily\hyphenchar\font=-1 #1}}}% | |
| 101 | 105 | {#1}\egroup} |
| 102 | 106 | \makeatother |
| 103 | 107 | |
| 119 | 123 | \emph{flag:} % |
| 120 | 124 | \ifthenelse{\equal{#1}{}}{}{% |
| 121 | 125 | \code{-#1}~~~\emph{or}~~~}% |
| 122 | \code{-{}-#3} ~~ }% | |
| 126 | \code{--#3} ~~ }% | |
| 123 | 127 | \emph{default:}~\texttt{#4} \\ |
| 124 | 128 | } |
| 125 | 129 | \newcommand{\parameter}[1]{\hyperlink{opt:#1}{\code{#1}}} |
| 212 | 216 | <<exampleload,eval=TRUE,include=FALSE>>= |
| 213 | 217 | library("irace") |
| 214 | 218 | load("examples.Rdata") |
| 215 | load("irace-acotsp.Rdata") | |
| 216 | load("log-ablation.Rdata") | |
| 219 | iraceResults <- irace::read_logfile("irace-acotsp.Rdata") | |
| 220 | log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 221 | load(log_ablation_file) | |
| 217 | 222 | options(width = 70) |
| 218 | 223 | @ |
| 219 | 224 | \newpage |
| 247 | 252 | \end{center} |
| 248 | 253 | % |
| 249 | 254 | More information about \irace is available at |
| 250 | \url{http://iridia.ulb.ac.be/irace}. | |
| 255 | \url{https://mlopez-ibanez.github.io/irace}. | |
| 251 | 256 | |
| 252 | 257 | \subsection{Version} |
| 253 | 258 | The current version of the \irace package is \iraceversion. Previous |
| 544 | 549 | \end{lstlisting} |
| 545 | 550 | %@ |
| 546 | 551 | |
| 547 | \item Copy all the template files from the \IRACEHOME{/templates/} | |
| 548 | directory to the scenario directory. | |
| 549 | %<<copy1,engine='bash',eval=FALSE>>= | |
| 552 | \item Initialize the tuning directory with template config files. On GNU/Linux or OS X, you can do this as follows: | |
| 553 | %<<dir1,engine='bash',eval=FALSE>>= | |
| 550 | 554 | \begin{lstlisting}[style=BashInputStyle] |
| 551 | # $IRACE_HOME is the installation directory of irace. | |
| 552 | cp $IRACE_HOME/templates/*.tmpl ~/tuning/ | |
| 555 | irace --init | |
| 553 | 556 | \end{lstlisting} |
| 554 | 557 | %@ |
| 555 | ||
| 556 | % Remember that \IRACEHOME{} is the path to the installation | |
| 557 | % directory of \irace. It can be obtained in the \aR console with: | |
| 558 | % | |
| 559 | % <<irace_path,eval=FALSE, prompt=FALSE>>= | |
| 560 | % library("irace") | |
| 561 | % system.file(package = "irace") | |
| 562 | % @ | |
| 563 | ||
| 564 | \item For each template in your tuning directory, remove the \code{.tmpl} | |
| 565 | suffix, and modify them following the next steps. | |
| 566 | 558 | |
| 567 | 559 | \item Define the target algorithm parameters to be tuned by following the |
| 568 | 560 | instructions in \code{parameters.txt}. Available parameter types and other |
| 612 | 604 | time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. |
| 613 | 605 | The \code{target-runner} template is written in \proglang{GNU Bash} |
| 614 | 606 | scripting language, which can be executed easily in GNU/Linux and OS X |
| 615 | systems. However, you may use any other programming language. As an example, | |
| 616 | we provide a \proglang{Python} example in the directory | |
| 617 | \IRACEHOME{/examples/python}. % | |
| 607 | systems. However, you may use any other programming language. We provide examples written in \proglang{Python}, \proglang{MATLAB} and other languages in \IRACEHOME{/examples/}. % | |
| 618 | 608 | Follow these instructions to adjust the given \code{target-runner} template |
| 619 | 609 | to your algorithm: |
| 620 | 610 | \begin{enumerate} |
| 655 | 645 | \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} |
| 656 | 646 | \end{center} |
| 657 | 647 | parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} |
| 658 | script must return \textbf{only} one number. In the template example, the result is returned with | |
| 659 | \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the used files are deleted. | |
| 648 | script must print \textbf{only} one number. In the template example, the result is printed with | |
| 649 | \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the generated files are deleted (you may remove that line if you wish to keep them). | |
| 660 | 650 | |
| 661 | 651 | \begin{xwarningbox} |
| 662 | The \code{target-runner} script must be executable. | |
| 652 | The \code{target-runner} script must be an executable file, unless you specify \parameter{targetRunnerLauncher} and \parameter{targetRunnerLauncherArgs}. | |
| 663 | 653 | \end{xwarningbox} |
| 664 | 654 | |
| 665 | 655 | You can test the target runner from the \aR console by checking the scenario as |
| 673 | 663 | |
| 674 | 664 | \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. |
| 675 | 665 | |
| 676 | \item The \irace executable provides an option (\parameter{-{}-check}) to | |
| 666 | \item The \irace executable provides an option (\parameter{--check}) to | |
| 677 | 667 | check that the scenario is correctly defined. We recommend to perform a |
| 678 | 668 | check every time you create a new scenario. When performing the check, |
| 679 | 669 | \irace will verify that the scenario and parameter definitions are |
| 706 | 696 | or directly from the \aR console: |
| 707 | 697 | |
| 708 | 698 | \begin{itemize} |
| 709 | \item{% | |
| 699 | \item | |
| 710 | 700 | \textbf{From the command-line console}, call the command (on Windows, you should execute |
| 711 | 701 | \code{irace.exe}): |
| 712 | 702 | \begin{lstlisting}[style=BashInputStyle] |
| 720 | 710 | properly in the \code{scenario.txt} file using the options described in |
| 721 | 711 | \autoref{sec:irace options}. Most \irace options can be specified |
| 722 | 712 | in the command line or directly in the \code{scenario.txt} file. |
| 723 | } | |
| 724 | \item{ | |
| 713 | ||
| 714 | \item | |
| 725 | 715 | \textbf{From the \aR console}, evaluate: |
| 726 | 716 | |
| 727 | 717 | <<irace_R_exe, eval=FALSE, prompt=FALSE>>= |
| 732 | 722 | scenario = defaultScenario()) |
| 733 | 723 | irace.main(scenario = scenario) |
| 734 | 724 | @ |
| 735 | } | |
| 725 | ||
| 736 | 726 | \end{itemize} |
| 737 | 727 | |
| 738 | 728 | This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on |
| 746 | 736 | |
| 747 | 737 | \subsection{Setup example for ACOTSP}\label{sec:example} |
| 748 | 738 | |
| 749 | The \ACOTSP tuning example can be found in the package installation at | |
| 750 | \IRACEHOME{/examples/acotsp}. | |
| 739 | The \ACOTSP tuning example can be found in the package installation in the folder \IRACEHOME{/examples/acotsp}. | |
| 751 | 740 | % |
| 752 | Additionally, a number of example scenarios can be found in the \code{examples} folder. More | |
| 741 | Other example scenarios can be found in the same folder. More | |
| 753 | 742 | examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). |
| 754 | 743 | |
| 755 | 744 | In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up |
| 756 | 745 | your own scenario, continue to the next section. For this example, we assume |
| 757 | a GNU/Linux system but making the necessary changes in the commands and \parameter{targetRunner}, | |
| 758 | it can be executed in any system that has a \proglang{C} compiler. | |
| 759 | %\MANUEL{I don't think this is true, since the target-runner script needs Bash} | |
| 746 | a GNU/Linux system such as Ubuntu with a working \proglang{C} compiler such as \code{gcc}. | |
| 760 | 747 | To execute this scenario follow these steps: |
| 761 | 748 | |
| 762 | 749 | \begin{enumerate} |
| 771 | 758 | \end{lstlisting} |
| 772 | 759 | %@ |
| 773 | 760 | |
| 774 | \item Download the training instances from \url{http://iridia.ulb.ac.be/irace/} to the \path{~/tuning/} directory. | |
| 761 | \item Download the training instances from \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} to the \path{~/tuning/} directory. | |
| 775 | 762 | \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. |
| 776 | 763 | |
| 777 | 764 | %<<instance0,engine='bash',eval=FALSE>>= |
| 792 | 779 | make |
| 793 | 780 | \end{lstlisting} |
| 794 | 781 | %@ |
| 795 | \item Create a directory for the executable and copy it: | |
| 796 | ||
| 797 | %<<acotsp1,engine='bash',eval=FALSE>>= | |
| 798 | \begin{lstlisting}[style=BashInputStyle] | |
| 799 | mkdir ~/bin/ | |
| 800 | cp ~/tuning/ACOTSP-1.03/acotsp ~/bin/ | |
| 801 | \end{lstlisting} | |
| 802 | %@ | |
| 803 | 782 | |
| 804 | 783 | \item Create a directory for executing the experiments and execute \irace: |
| 805 | 784 | |
| 858 | 837 | option \parameter{digits}. For example, given the default |
| 859 | 838 | number of digits of $4$, the values $0.12345$ and |
| 860 | 839 | $0.12341$ are both rounded to $0.1234$. |
| 861 | % However, the values $0.00001$ and $0.00005$ remain the same. | |
| 862 | Selected real-valued parameters can be optionally sampled on | |
| 840 | Selected real-valued parameters can be optionally sampled on | |
| 863 | 841 | a logarithmic scale (base $e$). |
| 864 | 842 | |
| 865 | 843 | \item \textit{Integer} parameters are numerical parameters that can take only |
| 895 | 873 | defined on them. All fixed parameters must be defined as categorical |
| 896 | 874 | parameters and have a domain of one element. |
| 897 | 875 | |
| 898 | \subsubsection{Conditional parameters} | |
| 876 | \subsubsection{Parameter dependent domains}\label{sec:paramdependant} | |
| 877 | ||
| 878 | Domains that are dependent on the values of other parameters can be specified | |
| 879 | only for numerical parameters (both integer and real). To do so, the dependent | |
| 880 | domain must be expressed in function of another parameter, which must be a | |
| 881 | numerical parameter. The expression that defines a dependency must be written | |
| 882 | between quotes: \code{(value, "expression")} or \code{("expression", value)} or | |
| 883 | \code{("expression", "expression")}. | |
| 884 | ||
| 885 | The expressions can only use the following operators and \aR functions: \code{+}, | |
| 886 | \code{-}, \code{*}, \code{/}, \code{\%\%}, \code{min}, \code{max}, | |
| 887 | \code{round}, \code{floor}, \code{ceiling}, \code{trunc}. If you need to use an | |
| 888 | operator or function not listed here, please contact us. | |
| 889 | ||
| 890 | \begin{xwarningbox} | |
| 891 | The user must ensure that the defined domain is valid at all times since | |
| 892 | \irace currently is not able to detect possible invalid domains based on the expressions | |
| 893 | provided. | |
| 894 | \end{xwarningbox} | |
| 895 | ||
| 896 | If you have a parameter \code{p2} that is just a transformation of another | |
| 897 | \code{p1}, then instead of using a dependent domain (left-hand side of the | |
| 898 | following example), it will be better to create a dummy parameter that controls | |
| 899 | the transformation (right-hand side) and do the transformation within \code{target-runner}. For example: | |
| 900 | % | |
| 901 | \begin{center} | |
| 902 | \begin{minipage}{0.4\linewidth} | |
| 903 | \small\centering% | |
| 904 | \begin{CodeInput}[frame=single] | |
| 905 | # With dependent domains | |
| 906 | p1 "" r (0, 100) | |
| 907 | p2 "" r ("p1", "p1 + 10") | |
| 908 | \end{CodeInput} | |
| 909 | \end{minipage} | |
| 910 | \hspace{\stretch{1}} should be \hspace{\stretch{1}} | |
| 911 | \begin{minipage}{0.4\linewidth} | |
| 912 | \small\centering% | |
| 913 | \begin{CodeInput}[frame=single] | |
| 914 | # With a dummy parameter | |
| 915 | p1 "" r (0, 100) | |
| 916 | p2dum "" r (0, 10) | |
| 917 | \end{CodeInput} | |
| 918 | \end{minipage} | |
| 919 | \end{center} | |
| 920 | % | |
| 921 | and \code{target-runner} will compute $\code{p2} = \code{p2dum} \cdot \code{p1}$. | |
| 922 | ||
| 923 | ||
| 924 | \subsubsection{Conditional parameters}\label{sec:conditional} | |
| 899 | 925 | |
| 900 | 926 | Conditional parameters are active only when others have certain values. These |
| 901 | 927 | dependencies define a hierarchical relation between parameters. For example, |
| 910 | 936 | table. Each line of the table defines a configurable parameter |
| 911 | 937 | % |
| 912 | 938 | \begin{center} |
| 913 | \code{<name>\ <label>\ <type>\ <range>\ [ | <condition>\ ] } | |
| 939 | \code{<name>\ <label>\ <type>\ <domain>\ [ | <condition>\ ] } | |
| 914 | 940 | \end{center} |
| 915 | 941 | where each field is defined as follows: |
| 916 | 942 | % |
| 917 | \begin{center} | |
| 918 | \renewcommand{\arraystretch}{1.2} | |
| 919 | \begin{tabularx}{0.98\linewidth}{@{}rX@{}} | |
| 920 | \code{<name>} & The name of the parameter as an unquoted | |
| 921 | alphanumeric string, e.g., `\code{ants}'.\\ | |
| 943 | \begin{description}[left=5em,itemindent=-0.5em,align=right,itemsep=0pt] | |
| 944 | % \DrawEnumitemLabel | |
| 945 | \item[\code{<name>}] The name of the parameter as an unquoted | |
| 946 | alphanumeric string, e.g., `\code{ants}'. | |
| 922 | 947 | % |
| 923 | \code{<label>}& A \emph{label} for this parameter. This is a | |
| 948 | \item[\code{<label>}] | |
| 949 | A \emph{label} for this parameter. This is a | |
| 924 | 950 | string that will be passed together with the parameter |
| 925 | 951 | to \parameter{targetRunner}. In the default \parameter{targetRunner} |
| 926 | 952 | provided with the package (\autoref{sec:runner}), this is the |
| 927 | 953 | command-line switch used to pass the value of this parameter, for |
| 928 | instance `\code{"-{}-ants "}'.\newline | |
| 954 | instance `\code{"--ants "}'.\newline | |
| 929 | 955 | The value of the parameter is concatenated \emph{without |
| 930 | 956 | separator} to the label when |
| 931 | invoking \parameter{targetRunner}, thus \emph{any whitespace in the label is significant}. Following the same example, when parameter \code{ants} takes value \code{5}, the default targetRunner will pass the parameter as \code{"-{}-ants 5"}.\\ | |
| 932 | % | |
| 933 | \code{{<type>}} & The type of the parameter, either | |
| 957 | invoking \parameter{targetRunner}, thus \emph{any whitespace in the label is significant}. Following the same example, when parameter \code{ants} takes value \code{5}, the default targetRunner will pass the parameter as \code{"--ants 5"}. | |
| 958 | ||
| 959 | \item[\code{{<type>}}] The type of the parameter, either | |
| 934 | 960 | \textit{integer}, \textit{real}, \textit{ordinal} or |
| 935 | 961 | \textit{categorical}, given as a single letter: `\code{i}', |
| 936 | 962 | `\code{r}', `\code{o}' or `\code{c}'. Numerical parameters |
| 937 | 963 | can be sampled using a natural logarithmic scale with '\code{i,log}' and |
| 938 | 964 | '\code{r,log}' (without spaces) for integer and real |
| 939 | parameters, respectively.\\ | |
| 965 | parameters, respectively. | |
| 940 | 966 | % |
| 941 | \code{{<range>}} & The range or set of values of the parameter delimited by | |
| 942 | parentheses. \eg~\code{(0,1)} or \code{(a,b,c,d)}.\\ | |
| 967 | \item[\code{{<domain>}}] The range or set of values of the parameter delimited by | |
| 968 | parentheses, \eg~\code{(0,1)} or \code{(a,b,c,d)}. | |
| 969 | See also parameter dependent domains (\autoref{sec:paramdependant}). | |
| 943 | 970 | % |
| 944 | 971 | |
| 945 | \code{{<condition>}} & An optional \emph{condition} that determines whether the parameter is | |
| 972 | \item[\code{{<condition>}}] An optional \emph{condition} that determines whether the parameter is | |
| 946 | 973 | enabled or disabled, thus making the parameter conditional. If the |
| 947 | 974 | condition evaluates to false, then no value is assigned to this |
| 948 | 975 | parameter, and neither the parameter value nor the corresponding |
| 952 | 979 | The condition may contain the name |
| 953 | 980 | of other parameters as long as the dependency graph does not |
| 954 | 981 | contain any cycle. Otherwise, \irace will detect the cycle and |
| 955 | stop with an error. \\ | |
| 956 | \end{tabularx} | |
| 957 | \end{center} | |
| 982 | stop with an error. | |
| 983 | ||
| 984 | \end{description} | |
| 958 | 985 | |
| 959 | 986 | As an example, Figure~\ref{fig:acotsp_parameters} shows the parameters file |
| 960 | 987 | of the \ACOTSP scenario. |
| 962 | 989 | \begin{figure}[!hbt] |
| 963 | 990 | \small\centering% |
| 964 | 991 | \begin{CodeInput} |
| 965 | # name switch type values [conditions (using R syntax)] | |
| 966 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 967 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 968 | alpha "--alpha " r (0.00, 5.00) | |
| 969 | beta "--beta " r (0.00, 10.00) | |
| 970 | rho "--rho " r (0.01, 1.00) | |
| 971 | ants "--ants " i (5, 100) | |
| 972 | nnls "--nnls " i (5, 50) | localsearch %in% c(1, 2, 3) | |
| 973 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 974 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 975 | rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 976 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 992 | # name switch type values [conditions (using R syntax)] | |
| 993 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 994 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 995 | alpha "--alpha " r (0.00, 5.00) | |
| 996 | beta "--beta " r (0.00, 10.00) | |
| 997 | rho "--rho " r (0.01, 1.00) | |
| 998 | ants "--ants " i (5, 100) | |
| 999 | nnls "--nnls " i (5, 50) | localsearch %in% c(1, 2, 3) | |
| 1000 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 1001 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 1002 | rasrank "--rasranks " i (1, "ants") | algorithm == "ras" | |
| 1003 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 977 | 1004 | \end{CodeInput} |
| 978 | 1005 | \caption{Parameter file (\code{parameters.txt}) for tuning \ACOTSP.}\label{fig:acotsp_parameters} |
| 979 | 1006 | \end{figure} |
| 1008 | 1035 | \code{nbParameters} & An integer, the total number of parameters.\\ |
| 1009 | 1036 | \code{nbFixed} & An integer, the number of parameters with a fixed value.\\ |
| 1010 | 1037 | \code{nbVariable} & Number of variable (i.e., to be tuned) parameters.\\ |
| 1038 | \code{isDependent} & Logical vector that specifies which parameter defines | |
| 1039 | a dependent domain.\\ | |
| 1011 | 1040 | \end{tabularx} |
| 1012 | 1041 | \end{center} |
| 1013 | 1042 | |
| 1057 | 1086 | \code{id.instance} & an alphanumeric string that uniquely identifies an instance;\\ |
| 1058 | 1087 | \code{seed} & seed for the random number generator to be used for this evaluation, ignore the seed for deterministic algorithms;\\ |
| 1059 | 1088 | \code{instance} & string giving the instance to be used for this evaluation;\\ |
| 1060 | \code{bound} & optional execution time bound. Only provided when the \parameter{maxBound} option is set in the scenario, see \autoref{sec:capping};\\ | |
| 1089 | \code{bound} & optional execution time bound. Only provided when the \parameter{boundMax} option is set in the scenario, see \autoref{sec:capping};\\ | |
| 1061 | 1090 | \code{configuration} & the pairs parameter label-value that |
| 1062 | 1091 | describe this candidate configuration. Typically given as command-line |
| 1063 | 1092 | switches to be passed to the executable program.\\ |
| 1430 | 1459 | the condition with \code{as.numeric(a)}. |
| 1431 | 1460 | \end{xwarningbox} |
| 1432 | 1461 | |
| 1462 | If the constraints provided are too strict, \irace may produce the following error: | |
| 1463 | \begin{CodeInput} | |
| 1464 | irace tried 100 times to sample from the model a configuration not forbidden | |
| 1465 | without success, perhaps your constraints are too strict? | |
| 1466 | \end{CodeInput} | |
| 1467 | % | |
| 1468 | In that case, it may be a good idea to reformulate the constraints as conditional parameters (\autoref{sec:conditional}), parameter-dependent domains (\autoref{sec:paramdependant}), repairing the configurations (\autoref{sec:repairconf}) or post-processing within the target-algorithm (\autoref{sec:complex_domains}). | |
| 1469 | ||
| 1470 | ||
| 1433 | 1471 | \subsection{Repairing configurations}\label{sec:repairconf} |
| 1434 | 1472 | |
| 1435 | 1473 | In some problems, the parameter values require complex constraints that cannot |
| 1455 | 1493 | { |
| 1456 | 1494 | isreal <- parameters$type[colnames(configuration)] %in% "r" |
| 1457 | 1495 | configuration[isreal] <- configuration[isreal] / sum(configuration[isreal]) |
| 1496 | configurations[isreal] <- round(configuration[isreal], digits) | |
| 1458 | 1497 | return(configuration) |
| 1459 | 1498 | } |
| 1460 | 1499 | @ |
| 1635 | 1674 | \end{itemize} |
| 1636 | 1675 | |
| 1637 | 1676 | The testing can be also (re-)executed at a later time by using the following |
| 1638 | \aR command: | |
| 1677 | \aR command (but you may need to override \code{testNbElites} and \code{testIterationElites}): | |
| 1639 | 1678 | % |
| 1640 | 1679 | <<testing_r, prompt=FALSE, eval=FALSE>>= |
| 1641 | testing.main(logFile = "./irace.Rdata") | |
| 1680 | testing_fromlog(logFile = "./irace.Rdata", testNbElites = 1) | |
| 1642 | 1681 | @ |
| 1643 | 1682 | % |
| 1644 | 1683 | The above line will load the scenario setup from \parameter{logFile} to |
| 1648 | 1687 | \autoref{sec:output r}. For examples on how to analyse the results see |
| 1649 | 1688 | \autoref{sec:analysis}. |
| 1650 | 1689 | |
| 1651 | Another alternative is to test a specific set of configurations using the command-line option \parameter{-{}-only-test} as follows: | |
| 1690 | Another alternative is to test a specific set of configurations using the command-line option \parameter{--only-test} as follows: | |
| 1652 | 1691 | \begin{lstlisting}[style=BashInputStyle] |
| 1653 | 1692 | irace --only-test configurations.txt |
| 1654 | 1693 | \end{lstlisting} |
| 1849 | 1888 | discarded after the statistical test was performed. |
| 1850 | 1889 | \item \code{Best}: ID of the best configuration according to the instances seen |
| 1851 | 1890 | so far in this race (i.e., not including previous iterations). |
| 1852 | \item \code{Mean best}: Mean of the best configuration across the instances seen so | |
| 1853 | far in this race. | |
| 1891 | \item \code{Mean best}: Mean cost value of the best configuration across the instances seen so | |
| 1892 | far in this race (not globally). Equivalent to the concept of ``iteration-best'' in other algorithms. | |
| 1854 | 1893 | \item \code{Exp so far}: Number of experiments |
| 1855 | 1894 | performed so far. |
| 1856 | 1895 | \item \code{W time}: Wall-clock time spent on this instance. |
| 1923 | 1962 | parameters of all elite configurations found by \irace use: |
| 1924 | 1963 | |
| 1925 | 1964 | <<get_elites, prompt=TRUE, eval=TRUE, comment="">>= |
| 1926 | getFinalElites(logFile = "irace-acotsp.Rdata", n = 0) | |
| 1965 | getFinalElites("irace-acotsp.Rdata", n = 0) | |
| 1927 | 1966 | @ |
| 1928 | 1967 | |
| 1929 | 1968 | \item \code{iterationElites}: A vector containing the best candidate configuration ID of each iteration. The best configuration found corresponds to the last one |
| 1937 | 1976 | <<get_elite, prompt=TRUE, eval=TRUE, comment="">>= |
| 1938 | 1977 | last <- length(iraceResults$iterationElites) |
| 1939 | 1978 | id <- iraceResults$iterationElites[last] |
| 1940 | getConfigurationById(logFile = "irace-acotsp.Rdata", ids = id) | |
| 1979 | getConfigurationById(iraceResults, ids = id) | |
| 1941 | 1980 | @ |
| 1942 | 1981 | |
| 1943 | 1982 | \item \code{rejectedConfigurations}: A vector containing the rejected configurations IDs. These |
| 1973 | 2012 | # As an example, we get seed and instance of the experiments |
| 1974 | 2013 | # of the best candidate. |
| 1975 | 2014 | # Get index of the instances |
| 1976 | pair.id <- names(all.exp[!is.na(all.exp)]) | |
| 1977 | index <- iraceResults$state$.irace$instancesList[pair.id,"instance"] | |
| 2015 | pair.id <- which(!is.na(all.exp)) | |
| 2016 | index <- iraceResults$state$.irace$instancesList[pair.id, "instance"] | |
| 1978 | 2017 | # Obtain the instance names |
| 1979 | 2018 | iraceResults$scenario$instances[index] |
| 1980 | 2019 | # Get the seeds |
| 1981 | iraceResults$state$.irace$instancesList[index,"seed"] | |
| 2020 | iraceResults$state$.irace$instancesList[pair.id, "seed"] | |
| 1982 | 2021 | @ |
| 1983 | 2022 | \item \code{experimentLog}: A matrix with columns \code{iteration, instance, configuration}. This matrix contains the log of all the experiments that \irace |
| 1984 | 2023 | performs during its execution. The \code{instance} column refers to the index of the |
| 2037 | 2076 | configuration is reported first. |
| 2038 | 2077 | |
| 2039 | 2078 | If testing is performed, you can further analyze the resulting best |
| 2040 | configurations by performing statistical tests in \aR or just plotting the | |
| 2041 | results: | |
| 2042 | ||
| 2043 | <<plot_test, fig.pos="tbp", fig.align="center", fig.height = 4, fig.width = 8, out.width='0.85\\textwidth', fig.cap="Boxplot of the testing results of the best configurations.", prompt=TRUE, eval=TRUE, comment="">>= | |
| 2079 | configurations by performing statistical tests in \aR: | |
| 2080 | % FIXME: iraceplot | |
| 2081 | % or just plotting the results: | |
| 2082 | %<<plot_test, fig.pos="tbp", fig.align="center", fig.height = 4, fig.width = 8, out.width='0.85\\textwidth', fig.cap="Boxplot of the testing results of the best configurations.", prompt=TRUE, eval=TRUE, comment="">>= | |
| 2083 | <<wilcox_test,prompt=TRUE, eval=TRUE, comment="">>= | |
| 2044 | 2084 | results <- iraceResults$testing$experiments |
| 2045 | 2085 | # Wilcoxon paired test |
| 2046 | 2086 | conf <- gl(ncol(results), # number of configurations |
| 2047 | 2087 | nrow(results), # number of instances |
| 2048 | 2088 | labels = colnames(results)) |
| 2049 | 2089 | pairwise.wilcox.test (as.vector(results), conf, paired = TRUE, p.adj = "bonf") |
| 2050 | # Plot the results | |
| 2051 | configurationsBoxplot (results, ylab = "Solution cost") | |
| 2052 | @ | |
| 2053 | ||
| 2054 | %%FIXME: should we add something like this? | |
| 2055 | %The Kendall concordance coefficient (\code{W}) and the Spearman's rho can be | |
| 2056 | %applied over data that has the characteristics of the data obtained in the testing, | |
| 2057 | %that is a full matrix where all configurations are executed in all instances. \code{W} | |
| 2058 | %can show if the configurations tested have an homogeneous performance on the used instances | |
| 2059 | %set. If evidence of an heterogeneous scenario found we recommend to make some adjustments | |
| 2060 | %in the \irace options as described in \autoref{sec:} | |
| 2061 | ||
| 2062 | %<<conc, prompt=TRUE, eval=TRUE, comment="">>= | |
| 2063 | %irace:::concordance(iraceResults$testing$experiments) | |
| 2064 | %@ | |
| 2065 | % | |
| 2066 | ||
| 2067 | During the tuning, \irace iteratively updates the sampling models of the | |
| 2068 | parameters to focus on the best regions of the parameter search space. The | |
| 2069 | frequency of the sampled configurations can provide insights on the parameter | |
| 2070 | search space. We provide a function for plotting the | |
| 2071 | frequency of the sampling of a set of configurations. For more information on this function, please see the \aR help, type in the | |
| 2072 | \aR console: \code{?parameterFrequency}. The following example | |
| 2073 | plots the frequency of the parameters sampled during one \irace run: | |
| 2074 | ||
| 2075 | <<freq, fig.pos="tbp", fig.cap="Parameters sampling frequency.", out.width="\\textwidth",prompt=TRUE, eval=TRUE, comment="">>= | |
| 2076 | parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 2077 | @ | |
| 2078 | ||
| 2079 | By using parallel coordinates plots, it is possible to analyze how the parameters | |
| 2080 | interact with each other. For more information on this function, please see the \aR help, type | |
| 2081 | in the \aR console: (\code{?parallelCoordinatesPlot}). | |
| 2082 | The following example shows how to create a parallel | |
| 2083 | coordinate plot of the configurations in the last two iterations of \irace. | |
| 2084 | ||
| 2085 | <<parcord, fig.pos="tbp", fig.align="center", out.width="0.7\\textwidth", fig.cap="Parallel coordinate plots of the parameters of the configurations in the last two iterations of a run of \\irace.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2086 | # Get last iteration number | |
| 2087 | last <- length(iraceResults$iterationElites) | |
| 2088 | # Get configurations in the last two iterations | |
| 2089 | conf <- getConfigurationByIteration(iraceResults = iraceResults, | |
| 2090 | iterations = c(last - 1, last)) | |
| 2091 | parallelCoordinatesPlot (conf, iraceResults$parameters, | |
| 2092 | param_names = c("algorithm", "alpha", "beta", "rho", "q0"), | |
| 2093 | hierarchy = FALSE) | |
| 2094 | @ | |
| 2090 | @ | |
| 2091 | % # Plot the results | |
| 2092 | % configurationsBoxplot (results, ylab = "Solution cost") | |
| 2093 | ||
| 2094 | The Kendall concordance coefficient (\code{W}) and the Spearman's rho can be | |
| 2095 | applied over data that has the characteristics of the data obtained in the testing, | |
| 2096 | that is a full matrix where all configurations are executed in all instances. \code{W} | |
| 2097 | can show if the configurations tested have an homogeneous performance on the used instances | |
| 2098 | set. If evidence of an heterogeneous scenario found we recommend to make some adjustments | |
| 2099 | in the \irace options as described in \autoref{sec:het}. | |
| 2100 | ||
| 2101 | <<conc, prompt=TRUE, eval=TRUE, comment="">>= | |
| 2102 | irace:::concordance(iraceResults$testing$experiments) | |
| 2103 | @ | |
| 2104 | % | |
| 2105 | ||
| 2106 | %% FIXME: iraceplot | |
| 2107 | % During the tuning, \irace iteratively updates the sampling models of the | |
| 2108 | % parameters to focus on the best regions of the parameter search space. The | |
| 2109 | % frequency of the sampled configurations can provide insights on the parameter | |
| 2110 | % search space. We provide a function for plotting the | |
| 2111 | % frequency of the sampling of a set of configurations. For more information on this function, please see the \aR help, type in the | |
| 2112 | % \aR console: \code{?parameterFrequency}. The following example | |
| 2113 | % plots the frequency of the parameters sampled during one \irace run: | |
| 2114 | ||
| 2115 | % <<freq, fig.pos="tbp", fig.cap="Parameters sampling frequency.", out.width="\\textwidth",prompt=TRUE, eval=TRUE, comment="">>= | |
| 2116 | % parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 2117 | % @ | |
| 2118 | ||
| 2119 | %% FIXME: iraceplot | |
| 2120 | % By using parallel coordinates plots, it is possible to analyze how the parameters | |
| 2121 | % interact with each other. For more information on this function, please see the \aR help, type | |
| 2122 | % in the \aR console: (\code{?parallelCoordinatesPlot}). | |
| 2123 | % The following example shows how to create a parallel | |
| 2124 | % coordinate plot of the configurations in the last two iterations of \irace. | |
| 2125 | ||
| 2126 | % <<parcord, fig.pos="tbp", fig.align="center", out.width="0.7\\textwidth", fig.cap="Parallel coordinate plots of the parameters of the configurations in the last two iterations of a run of \\irace.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2127 | % # Get last iteration number | |
| 2128 | % last <- length(iraceResults$iterationElites) | |
| 2129 | % # Get configurations in the last two iterations | |
| 2130 | % conf <- getConfigurationByIteration(iraceResults = iraceResults, | |
| 2131 | % iterations = c(last - 1, last)) | |
| 2132 | % parallelCoordinatesPlot (conf, iraceResults$parameters, | |
| 2133 | % param_names = c("algorithm", "alpha", "beta", "rho", | |
| 2134 | % "q0", "rasrank"), | |
| 2135 | % hierarchy = FALSE) | |
| 2136 | % @ | |
| 2095 | 2137 | |
| 2096 | 2138 | %%% FIXME: This plot is completely misleading. We should either calculate the |
| 2097 | 2139 | %%% performance over the whole training set, or find a way to plot the |
| 2099 | 2141 | %%% different number of instances. Or simply plot a heatmap of instances x |
| 2100 | 2142 | %%% best-of-each iteration. |
| 2101 | 2143 | \hide{It is also possible to plot the performance evolution of the best-so-far configuration over the number of experiments as follows: |
| 2102 | %% <<trainEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Training set performance of the best-so-far configuration over number of experiments.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2144 | %%<<trainEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Training set performance of the best-so-far configuration over number of experiments.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2103 | 2145 | # Get number of iterations |
| 2104 | 2146 | iters <- unique(iraceResults$experimentLog[, "iteration"]) |
| 2105 | 2147 | # Get number of experiments (runs of target-runner) up to each iteration |
| 2119 | 2161 | points(fes, values) |
| 2120 | 2162 | } |
| 2121 | 2163 | |
| 2122 | It is also possible to plot the performance on the test set of the best-so-far configuration over the number of experiments as follows: | |
| 2123 | <<testEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2164 | % FIXME: We should normalize the ranges per instance so since the performance | |
| 2165 | % on different instances is quite different. | |
| 2166 | It is also possible, as shown in Fig.~\ref{fig:testEvo}, to plot the performance on the test set of the best-so-far configuration over the number of experiments as follows: | |
| 2167 | <<testEvo, fig.pos="tb", fig.align="center", out.width='0.7\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2124 | 2168 | # Get number of iterations |
| 2125 | 2169 | iters <- unique(iraceResults$experimentLog[, "iteration"]) |
| 2126 | 2170 | # Get number of experiments (runs of target-runner) up to each iteration |
| 2129 | 2173 | # for the best configuration of that iteration. |
| 2130 | 2174 | elites <- as.character(iraceResults$iterationElites) |
| 2131 | 2175 | values <- colMeans(iraceResults$testing$experiments[, elites]) |
| 2176 | stderr <- function(x) sqrt(var(x)/length(x)) | |
| 2177 | err <- apply(iraceResults$testing$experiments[, elites], 2, stderr) | |
| 2132 | 2178 | plot(fes, values, type = "s", |
| 2133 | 2179 | xlab = "Number of runs of the target algorithm", |
| 2134 | ylab = "Mean value over testing set") | |
| 2135 | points(fes, values) | |
| 2180 | ylab = "Mean value over testing set", ylim=c(23000000,23500000)) | |
| 2181 | points(fes, values, pch=19) | |
| 2182 | arrows(fes, values - err, fes, values + err, length=0.05, angle=90, code=3) | |
| 2136 | 2183 | text(fes, values, elites, pos = 1) |
| 2137 | 2184 | @ |
| 2138 | 2185 | |
| 2139 | 2186 | The \irace package also provides an implementation of the ablation |
| 2140 | 2187 | method~\cite{FawHoos2015ablation}. See \autoref{sec:ablation}. |
| 2188 | ||
| 2189 | Finally, more advanced visualizations of the behavior of \irace are provided by | |
| 2190 | the ACVIZ software package~\cite{DesRitLopPer2021acviz}, which is available at \url{https://github.com/souzamarcelo/acviz}. See an example in Fig.~\ref{fig:acviz}. | |
| 2191 | ||
| 2192 | \begin{figure}[htb] | |
| 2193 | \centering | |
| 2194 | \includegraphics[width=\textwidth]{fig1u-acotsp-instances} | |
| 2195 | \caption{Visualization produced by ACVIZ~\cite{DesRitLopPer2021acviz}.} | |
| 2196 | \label{fig:acviz} | |
| 2197 | \end{figure} | |
| 2198 | ||
| 2199 | ||
| 2141 | 2200 | |
| 2142 | 2201 | %% |
| 2143 | 2202 | %% |
| 2169 | 2228 | must return the evaluation cost together with the execution time (\code{"cost time"}). |
| 2170 | 2229 | |
| 2171 | 2230 | \begin{xwarningbox} |
| 2172 | When the goal is to minimize the computation time of an algorithm, and you wish to use \parameter{maxTime} as the tuning budget, \parameter{targetRunner} must return the time also as the evaluation cost, that is, return the time two times as \code{"time time"}. | |
| 2231 | When the goal is to minimize the computation time of an algorithm, and you wish to use \parameter{maxTime} as the tuning budget, \parameter{targetRunner} must return the time also as the evaluation cost, that is, return the time twice as \code{"time time"}. | |
| 2173 | 2232 | \end{xwarningbox} |
| 2174 | 2233 | |
| 2175 | 2234 | \begin{xwarningbox} |
| 2230 | 2289 | \subsection{Tuning for minimizing computation time} |
| 2231 | 2290 | \label{sec:capping} |
| 2232 | 2291 | |
| 2233 | When using \irace for tuning algorithms that report computation time to | |
| 2234 | reach a target, \parameter{targetRunner} should return the execution time of a configuration instead of solution cost. | |
| 2235 | ||
| 2236 | %Even though \irace can be used for | |
| 2237 | %minimizing computation time, \irace may itself require more time to do so in | |
| 2238 | %its current version than other methods, such as | |
| 2239 | %\code{ParamILS}\footnote{\url{http://www.cs.ubc.ca/labs/beta/Projects/ParamILS/}} | |
| 2240 | %or \code{SMAC}\footnote{\url{http://www.cs.ubc.ca/labs/beta/Projects/SMAC/}}, | |
| 2241 | %since it does not make use of techniques, such as ``adaptive capping'', that | |
| 2242 | %avoid long runs of the target algorithm. | |
| 2243 | %We are currently extending \irace with an adaptive capping mechanism. | |
| 2292 | When using \irace for tuning algorithms that only report computation time to | |
| 2293 | reach a target, \parameter{targetRunner} should return the execution time of a configuration instead of solution cost. When using \parameter{maxTime} as the budget, this means that \parameter{targetRunner} must return twice the execution time since the first value is the minimization objective and the second value is used to track the budget consumed. | |
| 2244 | 2294 | |
| 2245 | 2295 | Starting from version $3.0$, \irace includes an elitist racing procedure that |
| 2246 | 2296 | implements an \textbf{adaptive capping mechanism} \citep{PerLopHooStu2017:lion}. |
| 2334 | 2384 | |
| 2335 | 2385 | |
| 2336 | 2386 | |
| 2337 | \subsection{Hyper-pararameter optimization of machine learning methods} | |
| 2338 | ||
| 2339 | The \irace package can also be used for model selection and hyper-parameter optimization | |
| 2340 | of machine learning methods. For such a task, we recommend the \pkg{mlr} package~\citep{R:mlr}. | |
| 2341 | The following webpage documents how to use \irace for this purpose: \url{https://mlr-org.github.io/mlr-tutorial/devel/html/advanced_tune/index.html} | |
| 2387 | \subsection{Hyper-parameter optimization of machine learning methods} | |
| 2388 | ||
| 2389 | The \irace package can also be used for model selection and hyper-parameter | |
| 2390 | optimization of machine learning (ML) methods. We will next explain a possible | |
| 2391 | setup for one given dataset and using $10$-fold cross-validation | |
| 2392 | (CV). Generalizing to multiple datasets and different resampling strategies, | |
| 2393 | e.g. leave-one-out, is straightforward. | |
| 2394 | ||
| 2395 | First, split the dataset into training, to be used by \irace, and testing, to | |
| 2396 | be used for evaluating the performance of the configuration returned by | |
| 2397 | \irace. A typical split could be $70\%$ and $30\%$, respectively. | |
| 2398 | ||
| 2399 | The training set is used by \irace to perform $10$-fold CV, that is, the data | |
| 2400 | is split into $10$ folds. A single run of the \parameter{targetRunner} will use | |
| 2401 | $9$ folds for training and the remaining fold for validation. Splitting the | |
| 2402 | data into folds can be done at each call of \parameter{targetRunner} or before | |
| 2403 | running \irace, however, it is important that the split is always the same for | |
| 2404 | every call of the \parameter{targetRunner}, i.e., the content of the folds does | |
| 2405 | not change, only which folds are used for training and validation will change. | |
| 2406 | ||
| 2407 | The setup of \irace should be as follows: | |
| 2408 | % | |
| 2409 | \begin{itemize} | |
| 2410 | \item \parameter{trainInstancesFile}\code{="train-instances.txt"}, where this | |
| 2411 | file contains one number per line from $1$ to $10$. This number will tell the | |
| 2412 | \parameter{targetRunner} which fold should be used for validation. | |
| 2413 | \item \parameter{trainInstancesDir}\code{=""}, because the folds are the | |
| 2414 | ``instances'' and you do not have actual instance files. If you want to pass | |
| 2415 | the name of the dataset to the \parameter{targetRunner}, you can specify it | |
| 2416 | either at each line of \code{"train-instances.txt"}, directly in the | |
| 2417 | \parameter{targetRunner}, or as a fixed parameter in the | |
| 2418 | \parameter{parameterFile}. | |
| 2419 | \item \parameter{deterministic}\code{=1} unless it really makes sense to | |
| 2420 | train more than once the same ML model on the same data. If it makes sense, | |
| 2421 | then your \parameter{targetRunner} should use the seed passed by \irace to | |
| 2422 | seed the ML model before training. | |
| 2423 | \item \parameter{sampleInstances}\code{=0} because the folds should already be generated by randomly sampling the dataset. | |
| 2424 | \item \parameter{testType}\code{="t-test} because the performance metrics in | |
| 2425 | ML are typically the mean of the CV results, which assumes that the | |
| 2426 | performance are close to normally distributed. | |
| 2427 | \item \parameter{firstTest}\code{=2} because \irace should discard | |
| 2428 | configurations very aggressively looking for maximum generality. | |
| 2429 | \end{itemize} | |
| 2430 | ||
| 2431 | \noindent% | |
| 2432 | Finally, your \parameter{targetRunner} needs to be able to do the following: | |
| 2433 | \begin{itemize} | |
| 2434 | \item Receive from \irace the hyper-parameter settings, the dataset name and a | |
| 2435 | fold number (the ``instance''). Let us use fold 3 as an example. | |
| 2436 | \item Train the ML model on the whole training set minus fold 3, then validate | |
| 2437 | (score) the model on fold 3 and return the score to \irace (negated if the | |
| 2438 | score must be maximized, because \irace assumes minimization). Since each | |
| 2439 | fold is different, each instance should give a different result. Each row in | |
| 2440 | the table printed by \irace should print something different; otherwise, | |
| 2441 | something is wrong in your setup. | |
| 2442 | \end{itemize} | |
| 2443 | ||
| 2444 | The above is actually 10 times faster than doing 10-fold CV for each call to | |
| 2445 | \parameter{targetRunner}, thus, you should assign to \irace 10 times the budget | |
| 2446 | than what would be assigned to other methods that do a complete 10-fold CV at | |
| 2447 | each step. | |
| 2448 | ||
| 2449 | %% FIXME: They removed the tutorial | |
| 2450 | %% For such a task, we recommend the \pkg{mlr} package~\citep{R:mlr}. | |
| 2451 | %% The following webpage documents how to use \irace for this purpose: \url{https://mlr-org.github.io/mlr-tutorial/devel/html/advanced_tune/index.html} | |
| 2342 | 2452 | |
| 2343 | 2453 | |
| 2344 | 2454 | \subsection{Heterogeneous scenarios} |
| 2361 | 2471 | selection techniques can be used to select the best configuration from the |
| 2362 | 2472 | portfolio when facing a new instance. |
| 2363 | 2473 | |
| 2474 | To make sure \irace is not misled by results on few instances, it may be useful | |
| 2475 | to increase the number of instances executed before doing a statistical test | |
| 2476 | using the option \parameter{firstTest}, \eg \code{--first-test 10} (default | |
| 2477 | value is 5), in order to see more instances before discarding configurations. | |
| 2478 | The option \parameter{elitistNewInstances} in elitist \irace (option | |
| 2479 | \parameter{elitist}) can be used to increase the number of new instances | |
| 2480 | executed in each iteration, \eg \code{--elitist-new-instances 5} (default value | |
| 2481 | is 1). | |
| 2482 | ||
| 2364 | 2483 | If finding an overall good configuration for all the instances is the |
| 2365 | objective, then we recommend that instances are randomly sampled | |
| 2366 | (option \parameter{sampleInstances}), unless one can provide the instances in a | |
| 2484 | objective, then we recommend that instances are randomly sampled (option | |
| 2485 | \parameter{sampleInstances}), unless one can provide the instances in a | |
| 2367 | 2486 | particular order that does not bias the tuning towards any subset. For |
| 2368 | example, let's assume a heterogeneous scenario with two classes of instances. If | |
| 2369 | training instances are not sampled and the first ten instances belong to only | |
| 2370 | one class, the tuning will be biased towards configurations that perform good | |
| 2371 | for those instances. An optimal order would not ever present consecutively two | |
| 2372 | instances of the same type. | |
| 2373 | ||
| 2374 | In addition, it may be useful to increase the number of instances executed | |
| 2375 | before doing a statistical test in order to see more instance classes before | |
| 2376 | discarding configurations. The option \parameter{elitistNewInstances} in elitist | |
| 2377 | \irace (option \parameter{elitist}) can be used to increase the number of new | |
| 2378 | instances executed in each iteration, \eg \code{--elitist-new-instances 5} (default | |
| 2379 | value is 1). For the non-elitist \irace, the option \parameter{firstTest} may be used for the same purpose, \eg \code{--first-test 10} (default value is 5). | |
| 2380 | \MANUEL{And both at the same time?} | |
| 2487 | example, let's assume a heterogeneous scenario with two classes of instances. | |
| 2488 | If training instances are not sampled and the first ten instances belong to | |
| 2489 | only one class, the tuning will be biased towards configurations that perform | |
| 2490 | good for those instances. An optimal order would never present consecutively | |
| 2491 | two instances of the same type. One can make sure \irace sees all instance | |
| 2492 | classes before discarding configurations by carefully | |
| 2493 | specifying the order of instances (disabling \parameter{sampleInstances} and | |
| 2494 | specifying the order in \parameter{trainInstancesFile}) and setting both | |
| 2495 | \parameter{firstTest} and \parameter{eachTest} appropriately. | |
| 2381 | 2496 | |
| 2382 | 2497 | While executing \irace, the homogeneity of the scenario can be observed by |
| 2383 | 2498 | examining the values of Spearman's rank correlation coefficient and Kendall's |
| 2431 | 2546 | them and, therefore, it is more likely to discard good configurations. |
| 2432 | 2547 | |
| 2433 | 2548 | |
| 2434 | \subsection{Complex parameter space constraints} | |
| 2549 | \subsection{Complex parameter space constraints}\label{sec:complex_domains} | |
| 2435 | 2550 | |
| 2436 | 2551 | Some parameters may have complex dependencies. Ideally, parameters should be |
| 2437 | 2552 | defined in the way that is more likely to help the search performed by |
| 2553 | 2668 | values from target. The sequence can be seen as a ``path'' from the source to |
| 2554 | 2669 | the target configuration. This can be used to find new better ``intermediate'' |
| 2555 | 2670 | configurations or to analyse the impact of the parameters in the performance. |
| 2556 | To perform ablation use the \code{ablation} function and specify the IDs of the | |
| 2557 | source and target configurations. By default, the source is taken as the first | |
| 2558 | configuration evaluated by \irace and the target as the best overall | |
| 2559 | configuration found. The argument \code{ab.params} can be used to specify a | |
| 2560 | subset of the parameters considered in the ablation. The option | |
| 2671 | ||
| 2672 | ||
| 2673 | To perform ablation, you can use the \code{ablation} function and specify the | |
| 2674 | IDs of the source and target configurations. By default, the source is taken | |
| 2675 | as the first configuration evaluated by \irace and the target as the best | |
| 2676 | overall configuration found. The argument \code{ab.params} can be used to | |
| 2677 | specify a subset of the parameters considered in the ablation. The option | |
| 2561 | 2678 | \parameter{firstTest} defines how many instances are selected for the |
| 2562 | 2679 | evaluation of configurations, if a different number of instances is required it |
| 2563 | must be specified in the argument \code{n.instance}. If a PDF filename is | |
| 2564 | provided (\code{pdf.file}), a plot will be produced from the ablation results | |
| 2565 | (Fig.~\ref{fig:testAb}). | |
| 2680 | must be specified in the argument \code{n.instance}. Use the function | |
| 2681 | \code{plotAblation} to visualize the ablation results (Fig.~\ref{fig:testAb}). | |
| 2566 | 2682 | |
| 2567 | 2683 | <<ablation, prompt=FALSE, eval=FALSE>>= |
| 2568 | ablation(iraceLogFile = "irace.Rdata", | |
| 2569 | src = 1, target = 60, pdf.file = "plot-ablation.pdf") | |
| 2570 | @ | |
| 2571 | ||
| 2572 | <<testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{ablation()}.", prompt=FALSE, eval=TRUE, echo=FALSE>>= | |
| 2573 | plotAblation(abLogFile = "log-ablation.Rdata") | |
| 2684 | ablog <- ablation("irace.Rdata", src = 1, target = 60) | |
| 2685 | plotAblation(ablog) | |
| 2686 | @ | |
| 2687 | ||
| 2688 | <<testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{plotAblation()}.", prompt=FALSE, eval=TRUE, echo=FALSE>>= | |
| 2689 | logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 2690 | plotAblation(logfile) | |
| 2574 | 2691 | @ |
| 2575 | 2692 | |
| 2576 | 2693 | The function returns a list containing the following elements: |
| 2583 | 2700 | \item \code{best}: Best overall configuration found. |
| 2584 | 2701 | \end{description} |
| 2585 | 2702 | |
| 2703 | We also provide a command-line executable that allows you to perform ablation without launching \aR. It is installed in the same location as the \irace command-line executable and has the following options: | |
| 2704 | <<ablation_cmdline, prompt=FALSE, eval=TRUE,echo=FALSE>>= | |
| 2705 | ablation_cmdline("--help") | |
| 2706 | @ | |
| 2586 | 2707 | |
| 2587 | 2708 | \subsection{Postselection race}\label{sec:postselection} |
| 2588 | 2709 | |
| 2633 | 2754 | Although ablation analysis without surrogates may be more time-consuming, results of the surrogate version may be less accurate than the non-surrogate one. |
| 2634 | 2755 | |
| 2635 | 2756 | The \code{.Rdata} dataset generated by \irace can be used as input for \PyImp. |
| 2636 | We provide a script to translate an \code{irace.Rdata} file into the input format required by \PyImp. | |
| 2637 | The script is available in the \irace package, and can be accessed either through the \aR console (function \code{irace2pyimp}), or via command line (\IRACEHOME{/bin/irace2pyimp}). | |
| 2757 | The package \code{irace2pyimp}\footnote{\url{https://github.com/ndangtt/irace2pyimp}} is able to convert an \code{irace.Rdata} file into the input format required by \PyImp. The conversion can be accessed either through the \aR console (function \code{irace2pyimp}), or via command line (\code{system.file("/bin/irace2pyimp", package="irace2pyimp")}). | |
| 2638 | 2758 | |
| 2639 | 2759 | To see the usage of the executable, please run: \code{irace2pyimp --help}. |
| 2640 | 2760 | For more information on the \aR function \code{irace2pyimp}, type in the \aR console: \code{?irace2pyimp}. |
| 2649 | 2769 | \item \code{features.csv}: a .csv file containing instance features. If no instance features are provided, the index of each instance will be used as a feature. |
| 2650 | 2770 | \end{itemize} |
| 2651 | 2771 | |
| 2652 | \PyImp can then be called using the files listed above as input. Several examples on how to use the script and to call \PyImp can be found at \IRACEHOME{/inst/examples/irace2pyimp/}. | |
| 2772 | \PyImp can then be called using the files listed above as input. Several | |
| 2773 | examples on how to use the script and call \PyImp can be found at | |
| 2774 | \code{system.file("/examples/", package="irace2pyimp")}. | |
| 2653 | 2775 | |
| 2654 | 2776 | %\emph{\PyImp installation note:} At the time when this document is written (December 2019), the latest \PyImp release version (1.0.6) has some bugs. |
| 2655 | 2777 | %Please check on \url{https://pypi.org/project/PyImp/#history} to see if \PyImp version > 1.0.6 has been released. |
| 2813 | 2935 | This is a warning given by \aR when the last line of an input file does not |
| 2814 | 2936 | finish with the newline character. The warning is harmless and can be |
| 2815 | 2937 | ignored. If you want to suppress it, just open the file and press the |
| 2816 | \code{ENTER} key at the end of the last line of the file. | |
| 2938 | \code{ENTER} key at the end of the last line of the file to end the final line with a newline. | |
| 2817 | 2939 | |
| 2818 | 2940 | |
| 2819 | 2941 | \subsection{How are relative filesystem paths interpreted by \irace?}\label{faq:relpaths} |
| 2843 | 2965 | |
| 2844 | 2966 | \subsection{My parameter space is small enough that \irace could generate all possible configurations; however, \irace generates repeated configurations and/or does not generate some of them. Is this a bug?}\label{faq:allconfs} |
| 2845 | 2967 | |
| 2846 | Currently, \irace does not try to detect whether all possible configurations | |
| 2847 | can be evaluated for the given budget, thus, the initial random sampling | |
| 2968 | Typically, \irace is applied to parameter spaces that are much larger than what | |
| 2969 | can be explored within the budget given. Thus, \irace does not try to detect | |
| 2970 | whether all possible configurations can be evaluated for the given budget and | |
| 2971 | it does not waste computation time to check for repeated configurations. Thus, | |
| 2972 | if the parameter space is actually very small, the initial random sampling | |
| 2848 | 2973 | performed by \irace may generate repeated configurations and/or never generate |
| 2849 | some configurations, which is not ideal. The ideal approach in such cases is to | |
| 2850 | provide all configurations explicitly to \irace (\autoref{sec:initial}) and | |
| 2851 | execute a single race (\parameter{nbIterations}\code{=1}) with exactly the | |
| 2852 | number of configurations provided (e.g., | |
| 2853 | \parameter{nbConfigurations}\code{=10}). A future version of \irace will | |
| 2854 | automatically detect this case and switch to basic racing without having to set additional options. | |
| 2974 | some configurations, which is not ideal. If you still want to use | |
| 2975 | (non-iterated) racing, the recommended approach is to provide all | |
| 2976 | configurations explicitly to \irace (\autoref{sec:initial}) and execute a | |
| 2977 | single race (\parameter{nbIterations}\code{=1}) with exactly the number of | |
| 2978 | configurations provided (e.g., \parameter{nbConfigurations}\code{=240}). A | |
| 2979 | future version of \irace may automatically detect this case and switch to | |
| 2980 | non-iterated racing without having to set additional options. Future versions | |
| 2981 | may also implement computationally cheap checks for repeated | |
| 2982 | configurations.\footnote{If you are interested in implementing this, please | |
| 2983 | contact us!} | |
| 2984 | ||
| 2985 | \subsection[On Windows and using target-runner.py (a Python file), I | |
| 2986 | get the error target-runner.py is not executable]{On Windows and using \code{target-runner.py} (a \proglang{Python} file), I | |
| 2987 | get the error ``\code{target-runner.py is not executable}''}\label{faq:py-not-exe} | |
| 2988 | ||
| 2989 | The issue is that \code{.py} files are not executable on their own and you need \code{python.exe} to read the \code{.py} file and execute it. Linux knows how to do this if the first line of the file is ``\code{\#!/usr/bin/python}'', however, Windows doesn't know how to do it. In Windows you have 2 options: | |
| 2990 | \begin{itemize} | |
| 2991 | \item Create a \code{target-runner.bat} file that contains a line similar to (see \path{templates/windows/target-runner.bat}): | |
| 2992 | \begin{lstlisting}[style=BashInputStyle] | |
| 2993 | C:\path\to\python.exe C:\path\to\target-runner.py %instance% %seed% \ | |
| 2994 | %candidate_parameters% 1>%stdout% 2>%stderr% | |
| 2995 | \end{lstlisting} | |
| 2996 | ||
| 2997 | \item Or convert \code{target-runner.py} into an \code{.exe} file, for example, using \code{auto-py-to-exe}\footnote{\url{https://pypi.org/project/auto-py-to-exe/}}, so that you do not need a \code{.bat} file. | |
| 2998 | \end{itemize} | |
| 2999 | ||
| 3000 | \subsection[Error in socketConnection(\ldots) : can not open the connection]{Error in \code{socketConnection("localhost", port = port, server = TRUE, lock = TRUE,} : can not open the connection} | |
| 3001 | ||
| 3002 | This error may arise if you activate the \parameter{parallel} option of \irace and your \parameter{targetRunner} or \parameter{targetEvaluator} tries to setup a parallel cluster or execute code in parallel in a way that interacts badly with the parallel mechanism in \aR. In this case, you need to either investigate yourself if there is a way for the two parallel mechanisms to co-exist or, if that is not possible, disable parallelism in \irace or in your code. Note that packages or software used by your \parameter{targetRunner} may have a parallel mechanism enabled by default and unknown to you. This is definitely NOT a bug in \irace. | |
| 2855 | 3003 | |
| 2856 | 3004 | %% |
| 2857 | 3005 | %% |
| 2868 | 3016 | \section{Resources and contact information} \label{sec:contact} |
| 2869 | 3017 | |
| 2870 | 3018 | More information about the package can be found on the \irace webpage: |
| 2871 | \begin{center} \url{http://iridia.ulb.ac.be/irace/} \end{center} | |
| 3019 | \begin{center} \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} \end{center} | |
| 2872 | 3020 | |
| 2873 | 3021 | For questions and suggestions please contact the development team through |
| 2874 | 3022 | the \irace package Google group: |
| 2879 | 3027 | \begin{center} |
| 2880 | 3028 | \href{mailto:irace-package@googlegroups.com}{irace-package@googlegroups.com} |
| 2881 | 3029 | \end{center} |
| 2882 | ||
| 2883 | %% MANUEL: I think we should use just one channel. If the Google group is the winner, that should be it. | |
| 2884 | %the mailing list: | |
| 2885 | %% LESLIE I agree. | |
| 2886 | %\begin{center} \href{mailto:irace@iridia.ulb.ac.be}{irace@iridia.ulb.ac.be} \end{center} | |
| 2887 | 3030 | |
| 2888 | 3031 | |
| 2889 | 3032 | \section{Acknowledgements} |
Binary diff not shown
| 0 | Additional examples of scenarios can be found in: | |
| 1 | ||
| 2 | Capping Methods for the Automatic Configuration of Optimization Algorithms | |
| 3 | Marcelo de Souza, Marcus Ritt and Manuel López-Ibáñez | |
| 4 | https://github.com/souzamarcelo/supp-cor-capopt | |
| 5 | ||
| 6 | * ACOTSP: ant colony optimization algorithms for the symmetric traveling salesperson problem. | |
| 7 | * HEACOL: hybrid evolutionary algorithm for graph coloring. | |
| 8 | * TSBPP: tabu search for the bin packing problem. | |
| 9 | * HHBQP: hybrid heuristic for unconstrained binary quadratic programming. | |
| 10 | * LKH: a heuristic algorithm for solving the symmetric traveling salesperson problem. | |
| 11 | * SCIP: an exact solver for mixed integer programs for solving the | |
| 12 | combinatorial auction winner determination problem. | |
| 13 | ||
| 14 | ||
| 15 | http://aclib.net/ | |
| 16 | Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, | |
| 17 | Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. AClib: a Benchmark | |
| 18 | Library for Algorithm Configuration. In P. M. Pardalos, M. G. C. Resende, | |
| 19 | C. Vogiatzis, and J. L. Walteros, editors, Learning and Intelligent | |
| 20 | Optimization, 8th International Conference, LION 8, volume 8426 of Lecture | |
| 21 | Notes in Computer Science, pages 36–40. Springer, 2014. |
| 11 | 11 | |
| 12 | 12 | TSP instances and more information is available at: |
| 13 | 13 | |
| 14 | http://iridia.ulb.ac.be/irace/#ACOTSP | |
| 14 | https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html | |
| 15 | 15 | |
| 16 | 16 |
| 17 | 17 | } |
| 18 | 18 | |
| 19 | 19 | |
| 20 | # Path to the ACOTSP software: | |
| 21 | EXE=~/bin/acotsp | |
| 20 | # Path to the ACOTSP executable (this path is relative to the execution directory `execDir` specified in the scenario). | |
| 21 | # EXE="~/bin/acotsp" | |
| 22 | EXE="../ACOTSP-1.03/acotsp" | |
| 22 | 23 | |
| 23 | 24 | # Fixed parameters that should be always passed to ACOTSP. |
| 24 | 25 | # The time to be used is always 10 seconds, and we want only one run: |
| 51 | 51 | # This is an example of reading a number from the output of |
| 52 | 52 | # target-runner. It assumes that the objective value is the first number in |
| 53 | 53 | # the first column of the only line starting with a digit. |
| 54 | if [ -s "${STDOUT}" ]; then | |
| 55 | # You may need to update this to parse the output of your algorithm. | |
| 56 | COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) | |
| 57 | echo "$COST" | |
| 58 | TODELETE=$(comm -23 <(ls -1 c*-${INSTANCEID}-${SEED}.* | sort) <(ls -1 $ALLFILES)) | |
| 59 | ## Comment out if you wish to keep all output files around | |
| 60 | # rm -f $TODELETE | |
| 61 | exit 0 | |
| 62 | else | |
| 54 | if [ ! -s "${STDOUT}" ]; then | |
| 63 | 55 | error "${STDOUT}: No such file or directory" |
| 64 | 56 | fi |
| 57 | # You may need to update this to parse the output of your algorithm. | |
| 58 | COST=$(cat ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) | |
| 59 | echo "$COST" | |
| 60 | ## Comment out if you wish to keep all output files around | |
| 61 | # TODELETE=$(comm -23 <(ls -1 c*-${INSTANCEID}-${SEED}.* | sort) <(ls -1 $ALLFILES)) | |
| 62 | # rm -f $TODELETE | |
| 63 | exit 0 | |
| 65 | 64 | |
| 65 |
| 0 | #!/bin/bash | |
| 1 | ############################################################################### | |
| 2 | # This script is the command that is executed every run. | |
| 3 | # Check the examples in examples/ | |
| 4 | # | |
| 5 | # This script is run in the execution directory (execDir, --exec-dir), | |
| 6 | # the same directory where target-evaluator is executed. Hence, you may | |
| 7 | # need to copy extra files needed by the executable to this directory. | |
| 8 | # | |
| 9 | # | |
| 10 | # PARAMETERS: | |
| 11 | # $1 is the candidate number | |
| 12 | # $2 is the instance id | |
| 13 | # $3 is the seed | |
| 14 | # $4 is the instance name | |
| 15 | # The rest ($* after `shift 4') are parameters to the run | |
| 16 | # | |
| 17 | # RETURN VALUE: | |
| 18 | # This script should print nothing. | |
| 19 | # Exit with 0 if no error, with 1 in case of error | |
| 20 | ############################################################################### | |
| 21 | error() { | |
| 22 | echo "`TZ=UTC date`: $0: error: $@" | |
| 23 | cat $tmpfile | |
| 24 | rm -f $tmpfile | |
| 25 | exit 1 | |
| 26 | } | |
| 27 | ||
| 28 | tmpfile=$(mktemp) | |
| 29 | ||
| 30 | EXE=~/bin/program | |
| 31 | FIXED_PARAMS="" | |
| 32 | ||
| 33 | CANDIDATE=$1 | |
| 34 | INSTANCEID=$2 | |
| 35 | SEED=$3 | |
| 36 | INSTANCE=$4 | |
| 37 | shift 4 || error "Not enough parameters" | |
| 38 | CAND_PARAMS=$* | |
| 39 | ||
| 40 | STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout | |
| 41 | STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr | |
| 42 | STDLOG=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdlog | |
| 43 | ||
| 44 | # modify with the HTCONDOR submission template | |
| 45 | condor_submit 1> $tmpfile <<EOF | |
| 46 | universe = vanilla | |
| 47 | stream_output = True | |
| 48 | stream_error = True | |
| 49 | executable = $EXE | |
| 50 | arguments = ${FIXED_PARAMS} --input $INSTANCE ${CAND_PARAMS} | |
| 51 | log = $STDLOG | |
| 52 | output = $STDOUT | |
| 53 | error = $STDERR | |
| 54 | queue | |
| 55 | EOF | |
| 56 | ||
| 57 | rc=$? | |
| 58 | if [[ $rc == 0 ]]; then | |
| 59 | JOBID=$(cat $STDLOG | grep "Job submitted" | egrep -o '\([0-9]+.' | egrep -o '[0-9]+') | |
| 60 | if ! [[ "$JOBID" =~ ^[_-.@0-9A-Za-z]+$ ]] ; then | |
| 61 | error "$0: cannot parse jobID from the log file!" | |
| 62 | fi | |
| 63 | echo "$JOBID" | |
| 64 | rm -f $tmpfile | |
| 65 | exit $rc | |
| 66 | else | |
| 67 | error "$0: condor_submit failed!" | |
| 68 | fi |
| 0 | # create a directory for saving output | |
| 1 | mkdir -p results | |
| 2 | ||
| 3 | # example of using irace2pyimp with filter conditions and normalisation: | |
| 4 | # - filter tuning data such that only configurations with n_templates_middle<=40 are used, then convert irace.Rdata to PIMP's input format, | |
| 5 | # - the cost values of all configurations are normalised on an instance-basis | |
| 6 | irace2pyimp --out-dir results --normalise instance --filter-conditions "n_templates_middle<=40" --irace-data-file irace.Rdata | |
| 7 | ||
| 8 | # call PyImp (executable name is "pimp") with two modules: forward-selection & ablation analysis | |
| 9 | cd results | |
| 10 | pimp -S scenario.txt -H runhistory.json -M forward-selection | |
| 11 | pimp -S scenario.txt -H runhistory.json -M ablation | |
| 12 | cd .. |
| 0 | nCities | |
| 1 | 1000 | |
| 2 | 1000 | |
| 3 | 1000 | |
| 4 | 1000 | |
| 5 | 1000 | |
| 6 | 1000 | |
| 7 | 1000 | |
| 8 | 1000 | |
| 9 | 1000 | |
| 10 | 1000 | |
| 11 | 1500 | |
| 12 | 1500 | |
| 13 | 1500 | |
| 14 | 1500 | |
| 15 | 1500 | |
| 16 | 1500 | |
| 17 | 1500 | |
| 18 | 1500 | |
| 19 | 1500 | |
| 20 | 1500 | |
| 21 | 2000 | |
| 22 | 2000 | |
| 23 | 2000 | |
| 24 | 2000 | |
| 25 | 2000 | |
| 26 | 2000 | |
| 27 | 2000 | |
| 28 | 2000 | |
| 29 | 2000 | |
| 30 | 2000 | |
| 31 | 2500 | |
| 32 | 2500 | |
| 33 | 2500 | |
| 34 | 2500 | |
| 35 | 2500 | |
| 36 | 2500 | |
| 37 | 2500 | |
| 38 | 2500 | |
| 39 | 2500 | |
| 40 | 2500 | |
| 41 | 3000 | |
| 42 | 3000 | |
| 43 | 3000 | |
| 44 | 3000 | |
| 45 | 3000 | |
| 46 | 3000 | |
| 47 | 3000 | |
| 48 | 3000 | |
| 49 | 3000 | |
| 50 | 3000 |
| 0 | # create a directory for saving output | |
| 1 | mkdir -p results | |
| 2 | ||
| 3 | # example 1: convert irace.Rdata to PyImp's input format, without normalisation | |
| 4 | irace2pyimp --out-dir results --instance-feature-file features.csv --irace-data-file irace.Rdata | |
| 5 | ||
| 6 | # example 2: convert irace.Rdata to PyImp's input format, without normalisation, only configurations with algorithm!='as' are taken | |
| 7 | #irace2pyimp --out-dir results --instance-feature-file features.csv --filter-conditions "algorithm!='as'" --irace-data-file irace.Rdata | |
| 8 | ||
| 9 | # example 3: convert irace.Rdata to PyImp's input format, with normalisation based on instance | |
| 10 | #irace2pyimp --out-dir results --instance-feature-file features.csv --normalise instance --irace-data-file irace.Rdata | |
| 11 | ||
| 12 | # example 4: convert irace.Rdata to PyImp's input format, with normalisation based on feature-vector values | |
| 13 | #irace2pyimp --out-dir results --instance-feature-file features.csv --normalise feature --irace-data-file irace.Rdata | |
| 14 | ||
| 15 | # call PyImp (executable name is "pimp") with all modules (fanova, abalation analysis, and forward selection) | |
| 16 | cd results | |
| 17 | pimp -S scenario.txt -H runhistory.json -T traj_aclib2.json -M all | |
| 18 | cd .. |
| 0 | nCities | |
| 1 | 1000 | |
| 2 | 1000 | |
| 3 | 1000 | |
| 4 | 1000 | |
| 5 | 1000 | |
| 6 | 1000 | |
| 7 | 1000 | |
| 8 | 1000 | |
| 9 | 1000 | |
| 10 | 1000 | |
| 11 | 1500 | |
| 12 | 1500 | |
| 13 | 1500 | |
| 14 | 1500 | |
| 15 | 1500 | |
| 16 | 1500 | |
| 17 | 1500 | |
| 18 | 1500 | |
| 19 | 1500 | |
| 20 | 1500 | |
| 21 | 2000 | |
| 22 | 2000 | |
| 23 | 2000 | |
| 24 | 2000 | |
| 25 | 2000 | |
| 26 | 2000 | |
| 27 | 2000 | |
| 28 | 2000 | |
| 29 | 2000 | |
| 30 | 2000 | |
| 31 | 2500 | |
| 32 | 2500 | |
| 33 | 2500 | |
| 34 | 2500 | |
| 35 | 2500 | |
| 36 | 2500 | |
| 37 | 2500 | |
| 38 | 2500 | |
| 39 | 2500 | |
| 40 | 2500 | |
| 41 | 3000 | |
| 42 | 3000 | |
| 43 | 3000 | |
| 44 | 3000 | |
| 45 | 3000 | |
| 46 | 3000 | |
| 47 | 3000 | |
| 48 | 3000 | |
| 49 | 3000 | |
| 50 | 3000 |
| 0 | # create a directory for saving output | |
| 1 | mkdir -p results | |
| 2 | ||
| 3 | # example 1: convert irace.Rdata to PyImp's input format, without normalisation | |
| 4 | #irace2pyimp --out-dir results --instance-feature-file features.csv --irace-data-file irace.Rdata --ignore-unsupported 0 | |
| 5 | ||
| 6 | # example 2: convert irace.Rdata to PyImp's input format, without normalisation, ignore forbidden constraints & repairConfiguration (irace2pyimp doesn't support those features), then call PyImp (executable name is "pimp") with all modules (fanova, abalation analysis, and forward selection) | |
| 7 | irace2pyimp --out-dir results --instance-feature-file features.csv --irace-data-file irace.Rdata --ignore-unsupported 1 | |
| 8 | cd results | |
| 9 | pimp -S scenario.txt -H runhistory.json -T traj_aclib2.json -M all | |
| 10 | cd .. |
| 0 | % function Result = Main(INSTANCE,SEED,A, B) | |
| 1 | % Using INSTANCE as a seed, this function creates a pseudo-random number | |
| 2 | % drawn from a uniform distribution in the open interval (-1,1) and returns | |
| 3 | % the value of Result, where Result = A*B + r | |
| 4 | function Result = Main(INSTANCE,SEED,A,B) | |
| 5 | ||
| 6 | rng('default'); | |
| 7 | rng(SEED); | |
| 8 | ||
| 9 | minValue = -1; | |
| 10 | maxValue = 1; | |
| 11 | r = (maxValue - minValue) * rand(1) + minValue; | |
| 12 | Result = A * B + r; | |
| 13 | % fprintf('Result for irace=%g\n', Result); | |
| 14 | fid=fopen('Result.txt','wt'); | |
| 15 | fprintf(fid,'%g\n',Result); | |
| 16 | fclose(fid); | |
| 17 | ||
| 18 | end |
| 0 | % function Result = RUN(INSTANCE, A, B) | |
| 1 | % Using INSTANCE as a seed, this function creates a pseudo-random number | |
| 2 | % drawn from a uniform distribution in the open interval (-1,1) and returns | |
| 3 | % the value of Result, where Result = A*B + r | |
| 4 | function Result = RUN(INSTANCE,A,B) | |
| 5 | ||
| 6 | rng('default'); | |
| 7 | rng(INSTANCE); | |
| 8 | ||
| 9 | minValue = -1; | |
| 10 | maxValue = 1; | |
| 11 | r = (maxValue - minValue) * rand(1) + minValue; | |
| 12 | Result = A * B + r; | |
| 13 | fprintf('Result for irace=%g\n', Result); | |
| 14 | end | |
| 15 |
| 0 | ###################################################### -*- mode: r -*- ##### | |
| 1 | ||
| 2 | ## You need to install the matlabr package, for example, | |
| 3 | # install.packages("matlabr") | |
| 4 | require(matlabr, quietly=TRUE) | |
| 5 | # We assume that the matlab .m files are in the same folder as this file. | |
| 6 | # Otherwise, change the next line. (This variable has to start with '.' so that | |
| 7 | # irace does not give an error.) | |
| 8 | .matlab_script_path = getwd() | |
| 9 | # You may need to provide the path to matlab if have_matlab() cannot find it. | |
| 10 | # options(matlab.path="/opt/apps/apps/binapps/matlab/R2020a/bin/") | |
| 11 | ||
| 12 | if (have_matlab()) { | |
| 13 | cat("=== irace === Using matlab as: ", get_matlab(), "\n") | |
| 14 | } else { | |
| 15 | stop("=== irace === MATLAB not found! Maybe not installed or you need to provide the matlab.path in scenario.txt") | |
| 16 | } | |
| 17 | ||
| 18 | ||
| 19 | targetRunner <- function(experiment, scenario) | |
| 20 | { | |
| 21 | matlab_function_call <- "Main(INSTANCE,SEED,A,B)" | |
| 22 | ||
| 23 | debugLevel <- scenario$debugLevel | |
| 24 | configuration.id <- experiment$id.configuration | |
| 25 | instance.id <- experiment$id.instance | |
| 26 | seed <- experiment$seed | |
| 27 | configuration <- experiment$configuration | |
| 28 | instance <- experiment$instance | |
| 29 | ||
| 30 | args <- paste0(names(configuration), "=", configuration, collapse=";") | |
| 31 | exitcode <- matlabr::run_matlab_code( | |
| 32 | # This is the instance and seed | |
| 33 | c(paste0("INSTANCE='", instance, "';SEED=", seed), | |
| 34 | # The configuration parameters, | |
| 35 | args, matlab_function_call), | |
| 36 | paths_to_add=.matlab_script_path, | |
| 37 | verbose= (debugLevel >= 2)) | |
| 38 | ||
| 39 | ## Parse the output (just a single number!) | |
| 40 | # FIXME: Create a uniquely named temporary file instead of Result.txt to | |
| 41 | # allow parallel execution | |
| 42 | output <- scan(file="Result.txt", quiet=TRUE) | |
| 43 | cost <- as.numeric(output) | |
| 44 | ||
| 45 | if (exitcode != 0) { | |
| 46 | cat("=== irace === targetRunner: matlab returned exit code ", exitcode, "\n") | |
| 47 | } | |
| 48 | if (is.na(cost)) { | |
| 49 | cat("=== irace === targetRunner: error parsing numeric result from matlab output\n") | |
| 50 | if (!file.exists(file.path(.matlab_script_path, "Main.m"))) | |
| 51 | cat("=== irace === matlab script: '", file.path(.matlab_script_path, "Main.m"), "' not found!\n") | |
| 52 | } | |
| 53 | return(list(cost = cost, outputRaw = output)) | |
| 54 | } | |
| 55 | ||
| 56 | ## Maximum number of runs (invocations of targetRunner) that will be | |
| 57 | ## performed. It determines the maximum budget of experiments for the | |
| 58 | ## tuning. | |
| 59 | maxExperiments = 100 | |
| 60 | ||
| 61 | ## File that contains the description of the parameters of the target | |
| 62 | ## algorithm. | |
| 63 | # parameterFile = "./parameters.txt" | |
| 64 | ||
| 65 | ## Directory where the programs will be run. | |
| 66 | # execDir = "./" | |
| 67 | ||
| 68 | ## Directory where training instances are located; either absolute path or | |
| 69 | ## relative to current directory. If no trainInstancesFiles is provided, | |
| 70 | ## all the files in trainInstancesDir will be listed as instances. | |
| 71 | trainInstancesDir = "" | |
| 72 | ||
| 73 | ## File that contains a list of training instances and optionally | |
| 74 | ## additional parameters for them. If trainInstancesDir is provided, irace | |
| 75 | ## will search for the files in this folder. | |
| 76 | trainInstancesFile = "instances.txt" | |
| 77 | ||
| 78 | # Add other scenario options for irace. See the user guide. | |
| 79 |
| 0 | ###################################################### -*- mode: r -*- ##### | |
| 1 | ||
| 2 | maxExperiments = 1000 | |
| 3 | ||
| 4 | targetRunner <- function(function(experiment, scenario) | |
| 5 | { | |
| 6 | main.script <- "Main.m" | |
| 7 | ||
| 8 | require(matlabr) | |
| 9 | debugLevel <- scenario$debugLevel | |
| 10 | configuration.id <- experiment$id.configuration | |
| 11 | instance.id <- experiment$id.instance | |
| 12 | seed <- experiment$seed | |
| 13 | configuration <- experiment$configuration | |
| 14 | instance <- experiment$instance | |
| 15 | switches <- experiment$switches | |
| 16 | ||
| 17 | args <- irace::buildCommandLine(configuration, switches) | |
| 18 | args <- strsplit(args, split=" ")[[1]] | |
| 19 | matlablr::run_matlab_code(c(args, paset0("run('", main.script, "')"))) | |
| 20 | cost <- as.numeric(readLines("hv")) | |
| 21 | return(list(cost = cost)) | |
| 22 | } |
| 0 | #!/bin/bash | |
| 1 | ############################################################################### | |
| 2 | # This script is the command that is executed every run. | |
| 3 | # Check the examples in examples/ | |
| 4 | # | |
| 5 | # This target-runner is run in the execution directory (execDir, --exec-dir). | |
| 6 | # | |
| 7 | # | |
| 8 | # PARAMETERS: | |
| 9 | # $1 is the candidate number | |
| 10 | # $2 is the instance ID | |
| 11 | # $3 is the seed | |
| 12 | # $4 is the instance name | |
| 13 | # The rest ($* after `shift 4') are parameters to the run | |
| 14 | # | |
| 15 | # RETURN VALUE: | |
| 16 | # This script should print one numerical value: the cost that must be minimized. | |
| 17 | # Exit with 0 if no error, with 1 in case of error | |
| 18 | ############################################################################### | |
| 19 | error() { | |
| 20 | echo "`TZ=UTC date`: $0: error: $@" | |
| 21 | exit 1 | |
| 22 | } | |
| 23 | ||
| 24 | EXE=matlab | |
| 25 | FIXED_PARAMS="-nosplash -nodesktop -nodisplay" | |
| 26 | ||
| 27 | CANDIDATE=$1 | |
| 28 | INSTANCEID=$2 | |
| 29 | SEED=$3 | |
| 30 | INSTANCE=$4 | |
| 31 | shift 4 || error "Not enough parameters" | |
| 32 | CAND_PARAMS=$* | |
| 33 | ||
| 34 | STDOUT=c${CANDIDATE}-${INSTANCEID}-${SEED}.stdout | |
| 35 | STDERR=c${CANDIDATE}-${INSTANCEID}-${SEED}.stderr | |
| 36 | ||
| 37 | ## If the program just prints a number, we can use 'exec' to avoid | |
| 38 | ## creating another process, but there can be no other commands after exec. | |
| 39 | #exec $EXE ${FIXED_PARAMS} -i $INSTANCE ${CAND_PARAMS} | |
| 40 | #exit 1 | |
| 41 | ||
| 42 | ## Otherwise, save the output to a file, and parse the result from it. | |
| 43 | ## (If you wish to ignore segmentation faults you can use '{}' around | |
| 44 | ## the command.) | |
| 45 | ## This example assumes that CAND_PARAMS contains something like "A=10 B=5" | |
| 46 | $EXE ${FIXED_PARAMS} -r "INSTANCE=${INSTANCEID};SEED=${SEED};${CAND_PARAMS// /;};RUN(INSTANCE,A,B);exit" < RUN.m 1> ${STDOUT} 2> ${STDERR} | |
| 47 | ||
| 48 | ## This may be used to introduce a delay if there are filesystem | |
| 49 | ## issues. | |
| 50 | # SLEEPTIME=1 | |
| 51 | # while [ ! -s "${STDOUT}" ]; do | |
| 52 | # sleep $SLEEPTIME | |
| 53 | # let "SLEEPTIME += 1" | |
| 54 | # done | |
| 55 | ||
| 56 | if [ -s "${STDOUT}" ]; then | |
| 57 | COST=$(cat ${STDOUT} | grep 'Result for irace=' | cut -f2 -d '=') | |
| 58 | echo "$COST" | |
| 59 | rm -f "${STDOUT}" "${STDERR}" | |
| 60 | exit 0 | |
| 61 | else | |
| 62 | error "${STDOUT}: No such file or directory" | |
| 63 | fi |
| 9 | 9 | Multi-Objective Ant Colony Optimization Algorithms. IEEE |
| 10 | 10 | Transactions on Evolutionary Computation, 2012. |
| 11 | 11 | doi:10.1109/TEVC.2011.2182651 |
| 12 | (PDF preprint) http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-003.pdf | |
| 12 | (PDF preprint) https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-003.pdf | |
| 13 | 13 | |
| 14 | 14 | The MOACO framework software can be download from: |
| 15 | 15 | |
| 16 | http://iridia.ulb.ac.be/~manuel/moaco | |
| 16 | http://lopez-ibanez.eu/moaco | |
| 17 | 17 | |
| 18 | 18 | Instances of the bTSP can be generated by concatenating two single-objective TSP |
| 19 | 19 | instances: |
| 22 | 22 | |
| 23 | 23 | TSP instances and more information is available at: |
| 24 | 24 | |
| 25 | http://iridia.ulb.ac.be/irace/ | |
| 25 | https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html | |
| 26 | 26 | |
| 27 | 27 | |
| 28 | ||
| 0 | ::############################################################################## | |
| 1 | :: In Windows, we cannot execute directly .py files so we use this script as | |
| 2 | :: the targetRunner that calls the actual .py file. | |
| 3 | :: Contributed by Levi Ribeiro<azalos13@gmail.com> | |
| 4 | :: | |
| 5 | :: This script is run in the execution directory (execDir, --exec-dir). | |
| 6 | :: | |
| 7 | :: PARAMETERS: | |
| 8 | :: %%1 is the candidate configuration number | |
| 9 | :: %%2 is the instance ID | |
| 10 | :: %%3 is the seed | |
| 11 | :: %%4 is the instance name | |
| 12 | :: The rest are parameters to the target-algorithm | |
| 13 | :: | |
| 14 | :: RETURN VALUE: | |
| 15 | :: This script should print one numerical value: the cost that must be minimized. | |
| 16 | :: Exit with 0 if no error, with 1 in case of error | |
| 17 | ::############################################################################## | |
| 18 | @echo off | |
| 19 | :: If python is in the PATH environment variable, you can call it directly. | |
| 20 | SET "pythonexe=python.exe" | |
| 21 | :: Otherwise, you need to provide its absolute location | |
| 22 | :: SET "pythonexe=C:\bin\python.exe" | |
| 23 | ||
| 24 | :: This script is run in the execution directory (execDir, --exec-dir), which often does not coincide with the directory containing target-runner.py. Either a relative or absolute path should be provided. | |
| 25 | SET "targetrunnerpy=..\target-runner.py" | |
| 26 | set parameters=%* | |
| 27 | %pythonexe% %targetrunnerpy% %parameters% |
Binary diff not shown
Binary diff not shown
Binary diff not shown
| 0 | Bootstrap: library | |
| 1 | From: ubuntu:20.04 | |
| 2 | ||
| 3 | %post | |
| 4 | # Dependencies | |
| 5 | apt -y install software-properties-common | |
| 6 | add-apt-repository universe | |
| 7 | apt -y update | |
| 8 | apt -y dist-upgrade | |
| 9 | apt -y install git make r-base r-cran-jsonlite | |
| 10 | apt clean | |
| 11 | ||
| 12 | # Temporary directory where we are going to build everything. | |
| 13 | tmpdir=$(mktemp -d) | |
| 14 | mkdir -p ${tmpdir} | |
| 15 | ||
| 16 | # Build | |
| 17 | cd ${tmpdir} | |
| 18 | git clone --single-branch --branch master https://github.com/MLopez-Ibanez/irace.git | |
| 19 | cd irace | |
| 20 | make quick-install | |
| 21 | ||
| 22 | # Clean-up | |
| 23 | rm -rf ${tmpdir} | |
| 24 | apt -y purge software-properties-common git make | |
| 25 | apt -y --purge autoremove | |
| 26 | apt -y autoclean | |
| 27 | apt clean | |
| 28 | ||
| 29 | %help | |
| 30 | This container provides an minimal executable version of irace. | |
| 31 | You can run it with `singularity run <irace command line arguments>`. | |
| 32 | e.g. `singularity run --help` | |
| 33 | ||
| 34 | %runscript | |
| 35 | /usr/local/lib/R/site-library/irace/bin/irace $* | |
| 36 | ||
| 37 | %labels | |
| 38 | Authors Johann Dreo | |
| 39 |
| 0 | ## This is an example of specifying instances with a file. | |
| 0 | ## This is an example of specifying a list of instances. | |
| 1 | 1 | |
| 2 | # Each line is an instance relative to trainInstancesDir | |
| 3 | # (see scenario.txt.tmpl) and an optional sequence of instance-specific | |
| 4 | # parameters that will be passed to target-runnerx when invoked on that | |
| 5 | # instance. | |
| 2 | # Each line is an instance relative to trainInstancesDir/testInstancesdir | |
| 3 | # unless set to "" (see scenario.txt.tmpl) and an optional sequence of | |
| 4 | # instance-specific parameters that will be passed to target-runner when | |
| 5 | # invoked on that instance. | |
| 6 | 6 | |
| 7 | 7 | # Empty lines and comments are ignored. |
| 8 | 8 |
| 5 | 5 | ## the parameter (comment it out in this file, and do not give any |
| 6 | 6 | ## value on the command line). |
| 7 | 7 | |
| 8 | ## Directory where the programs will be run. | |
| 9 | # execDir = "./" | |
| 10 | ||
| 8 | 11 | ## File that contains the description of the parameters of the target |
| 9 | 12 | ## algorithm. |
| 10 | 13 | # parameterFile = "./parameters.txt" |
| 11 | 14 | |
| 12 | ## Directory where the programs will be run. | |
| 13 | # execDir = "./" | |
| 15 | ## File that contains a list of logical expressions that cannot be TRUE for | |
| 16 | ## any evaluated configuration. If empty or NULL, do not use forbidden | |
| 17 | ## expressions. | |
| 18 | # forbiddenFile = "" | |
| 19 | ||
| 20 | ## File that contains a table of initial configurations. If empty or NULL, all | |
| 21 | ## initial configurations are randomly generated. | |
| 22 | # configurationsFile = "" | |
| 14 | 23 | |
| 15 | 24 | ## File to save tuning results as an R dataset, either absolute path or |
| 16 | 25 | ## relative to execDir. |
| 17 | 26 | # logFile = "./irace.Rdata" |
| 18 | 27 | |
| 19 | ## Previously saved log file to recover the execution of irace, either | |
| 20 | ## absolute path or relative to the current directory. If empty or NULL, | |
| 21 | ## recovery is not performed. | |
| 22 | # recoveryFile = "" | |
| 23 | ||
| 24 | 28 | ## Directory where training instances are located; either absolute path or |
| 25 | ## relative to current directory. If no trainInstancesFiles is provided, | |
| 26 | ## all the files in trainInstancesDir will be listed as instances. | |
| 29 | ## relative to current directory. If no trainInstancesFiles is provided, all | |
| 30 | ## the files in trainInstancesDir will be listed as instances. | |
| 27 | 31 | # trainInstancesDir = "./Instances" |
| 28 | 32 | |
| 29 | ## File that contains a list of training instances and optionally | |
| 30 | ## additional parameters for them. If trainInstancesDir is provided, irace | |
| 31 | ## will search for the files in this folder. | |
| 33 | ## File that contains a list of training instances and optionally additional | |
| 34 | ## parameters for them. If trainInstancesDir is provided, irace will search | |
| 35 | ## for the files in this folder. | |
| 32 | 36 | # trainInstancesFile = "" |
| 33 | 37 | |
| 34 | ## File that contains a table of initial configurations. If empty or NULL, | |
| 35 | ## all initial configurations are randomly generated. | |
| 36 | # configurationsFile = "" | |
| 37 | ||
| 38 | ## File that contains a list of logical expressions that cannot be TRUE | |
| 39 | ## for any evaluated configuration. If empty or NULL, do not use forbidden | |
| 40 | ## expressions. | |
| 41 | # forbiddenFile = "" | |
| 42 | ||
| 43 | ## Script called for each configuration that executes the target algorithm | |
| 44 | ## to be tuned. See templates. | |
| 38 | ## Randomly sample the training instances or use them in the order given. | |
| 39 | # sampleInstances = 1 | |
| 40 | ||
| 41 | ## Directory where testing instances are located, either absolute or relative | |
| 42 | ## to current directory. | |
| 43 | # testInstancesDir = "" | |
| 44 | ||
| 45 | ## File containing a list of test instances and optionally additional | |
| 46 | ## parameters for them. | |
| 47 | # testInstancesFile = "" | |
| 48 | ||
| 49 | ## Number of elite configurations returned by irace that will be tested if | |
| 50 | ## test instances are provided. | |
| 51 | # testNbElites = 1 | |
| 52 | ||
| 53 | ## Enable/disable testing the elite configurations found at each iteration. | |
| 54 | # testIterationElites = 0 | |
| 55 | ||
| 56 | ## Statistical test used for elimination. The default value selects t-test if | |
| 57 | ## capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman | |
| 58 | ## test), t-test (pairwise t-tests with no correction), t-test-bonferroni | |
| 59 | ## (t-test with Bonferroni's correction for multiple comparisons), t-test-holm | |
| 60 | ## (t-test with Holm's correction for multiple comparisons). | |
| 61 | # testType = "" | |
| 62 | ||
| 63 | ## Number of instances evaluated before the first elimination test. It must be | |
| 64 | ## a multiple of eachTest. | |
| 65 | # firstTest = 5 | |
| 66 | ||
| 67 | ## Number of instances evaluated between elimination tests. | |
| 68 | # eachTest = 1 | |
| 69 | ||
| 70 | ## Executable called for each configuration that executes the target algorithm | |
| 71 | ## to be tuned. See the templates and examples provided. | |
| 45 | 72 | # targetRunner = "./target-runner" |
| 73 | ||
| 74 | ## Executable that will be used to launch the target runner, when targetRunner | |
| 75 | ## cannot be executed directly (.e.g, a Python script in Windows). | |
| 76 | # targetRunnerLauncher = "" | |
| 77 | ||
| 78 | ## Command-line arguments provided to targetRunnerLauncher. The substrings | |
| 79 | ## \{targetRunner\} and \{targetRunnerArgs\} will be replaced by the value of | |
| 80 | ## the option targetRunner and by the arguments usually passed when calling | |
| 81 | ## targetRunner, respectively. Example: "-m {targetRunner --args | |
| 82 | ## {targetRunnerArgs}"}. | |
| 83 | # targetRunnerLauncherArgs = "{targetRunner} {targetRunnerArgs}" | |
| 46 | 84 | |
| 47 | 85 | ## Number of times to retry a call to targetRunner if the call failed. |
| 48 | 86 | # targetRunnerRetries = 0 |
| 49 | 87 | |
| 50 | 88 | ## Optional data passed to targetRunner. This is ignored by the default |
| 51 | ## targetRunner function, but it may be used by custom targetRunner | |
| 52 | ## functions to pass persistent data around. | |
| 89 | ## targetRunner function, but it may be used by custom targetRunner functions | |
| 90 | ## to pass persistent data around. | |
| 53 | 91 | # targetRunnerData = "" |
| 54 | 92 | |
| 55 | 93 | ## Optional R function to provide custom parallelization of targetRunner. |
| 59 | 97 | ## configuration. See templates/target-evaluator.tmpl |
| 60 | 98 | # targetEvaluator = "" |
| 61 | 99 | |
| 100 | ## If the target algorithm is deterministic, configurations will be evaluated | |
| 101 | ## only once per instance. | |
| 102 | # deterministic = 0 | |
| 103 | ||
| 62 | 104 | ## Maximum number of runs (invocations of targetRunner) that will be |
| 63 | ## performed. It determines the maximum budget of experiments for the | |
| 64 | ## tuning. | |
| 105 | ## performed. It determines the maximum budget of experiments for the tuning. | |
| 65 | 106 | # maxExperiments = 0 |
| 66 | 107 | |
| 67 | ## Maximum total execution time in seconds for the executions of | |
| 68 | ## targetRunner. targetRunner must return two values: cost and time. | |
| 108 | ## Maximum total execution time in seconds for the executions of targetRunner. | |
| 109 | ## targetRunner must return two values: cost and time. | |
| 69 | 110 | # maxTime = 0 |
| 70 | 111 | |
| 71 | 112 | ## Fraction (smaller than 1) of the budget used to estimate the mean |
| 72 | 113 | ## computation time of a configuration. Only used when maxTime > 0 |
| 73 | 114 | # budgetEstimation = 0.02 |
| 74 | 115 | |
| 75 | ## Maximum number of decimal places that are significant for numerical | |
| 76 | ## (real) parameters. | |
| 116 | ## Minimum time unit that is still (significantly) measureable. | |
| 117 | # minMeasurableTime = 0.01 | |
| 118 | ||
| 119 | ## Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean | |
| 120 | ## no parallelization. | |
| 121 | # parallel = 0 | |
| 122 | ||
| 123 | ## Enable/disable load-balancing when executing experiments in parallel. | |
| 124 | ## Load-balancing makes better use of computing resources, but increases | |
| 125 | ## communication overhead. If this overhead is large, disabling load-balancing | |
| 126 | ## may be faster. | |
| 127 | # loadBalancing = 1 | |
| 128 | ||
| 129 | ## Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter | |
| 130 | ## parallel is the number of slaves). | |
| 131 | # mpi = 0 | |
| 132 | ||
| 133 | ## Specify how irace waits for jobs to finish when targetRunner submits jobs | |
| 134 | ## to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must | |
| 135 | ## submit jobs to the cluster using, for example, qsub. | |
| 136 | # batchmode = 0 | |
| 137 | ||
| 138 | ## Maximum number of decimal places that are significant for numerical (real) | |
| 139 | ## parameters. | |
| 77 | 140 | # digits = 4 |
| 141 | ||
| 142 | ## Reduce the output generated by irace to a minimum. | |
| 143 | # quiet = 0 | |
| 78 | 144 | |
| 79 | 145 | ## Debug level of the output of irace. Set this to 0 to silence all debug |
| 80 | 146 | ## messages. Higher values provide more verbose debug messages. |
| 81 | 147 | # debugLevel = 0 |
| 82 | 148 | |
| 83 | ## Number of iterations. | |
| 84 | # nbIterations = 0 | |
| 85 | ||
| 86 | ## Number of runs of the target algorithm per iteration. | |
| 87 | # nbExperimentsPerIteration = 0 | |
| 88 | ||
| 89 | ## Randomly sample the training instances or use them in the order given. | |
| 90 | # sampleInstances = 1 | |
| 91 | ||
| 92 | ## Statistical test used for elimination. Default test is always F-test | |
| 93 | ## unless capping is enabled, in which case the default test is t-test. | |
| 94 | ## Valid values are: F-test (Friedman test), t-test (pairwise t-tests with | |
| 95 | ## no correction), t-test-bonferroni (t-test with Bonferroni's correction | |
| 96 | ## for multiple comparisons), t-test-holm (t-test with Holm's correction | |
| 97 | ## for multiple comparisons). | |
| 98 | # testType = "F-test" | |
| 99 | ||
| 100 | ## Number of instances evaluated before the first elimination test. It | |
| 101 | ## must be a multiple of eachTest. | |
| 102 | # firstTest = 5 | |
| 103 | ||
| 104 | ## Number of instances evaluated between elimination tests. | |
| 105 | # eachTest = 1 | |
| 106 | ||
| 107 | ## Minimum number of configurations needed to continue the execution of | |
| 108 | ## each race (iteration). | |
| 109 | # minNbSurvival = 0 | |
| 110 | ||
| 111 | ## Number of configurations to be sampled and evaluated at each iteration. | |
| 112 | # nbConfigurations = 0 | |
| 113 | ||
| 114 | ## Parameter used to define the number of configurations sampled and | |
| 115 | ## evaluated at each iteration. | |
| 116 | # mu = 5 | |
| 117 | ||
| 118 | ## Confidence level for the elimination test. | |
| 119 | # confidence = 0.95 | |
| 120 | ||
| 121 | ## If the target algorithm is deterministic, configurations will be | |
| 122 | ## evaluated only once per instance. | |
| 123 | # deterministic = 0 | |
| 124 | ||
| 125 | ## Seed of the random number generator (by default, generate a random | |
| 126 | ## seed). | |
| 149 | ## Seed of the random number generator (by default, generate a random seed). | |
| 127 | 150 | # seed = NA |
| 128 | 151 | |
| 129 | ## Number of calls to targetRunner to execute in parallel. Values 0 or 1 | |
| 130 | ## mean no parallelization. | |
| 131 | # parallel = 0 | |
| 132 | ||
| 133 | ## Enable/disable load-balancing when executing experiments in parallel. | |
| 134 | ## Load-balancing makes better use of computing resources, but increases | |
| 135 | ## communication overhead. If this overhead is large, disabling | |
| 136 | ## load-balancing may be faster. | |
| 137 | # loadBalancing = 1 | |
| 138 | ||
| 139 | ## Enable/disable MPI. Use Rmpi to execute targetRunner in parallel | |
| 140 | ## (parameter parallel is the number of slaves). | |
| 141 | # mpi = 0 | |
| 142 | ||
| 143 | ## Specify how irace waits for jobs to finish when targetRunner submits | |
| 144 | ## jobs to a batch cluster: sge, pbs, torque or slurm. targetRunner must | |
| 145 | ## submit jobs to the cluster using, for example, qsub. | |
| 146 | # batchmode = 0 | |
| 147 | ||
| 148 | ## Enable/disable the soft restart strategy that avoids premature | |
| 149 | ## convergence of the probabilistic model. | |
| 152 | ## Enable/disable the soft restart strategy that avoids premature convergence | |
| 153 | ## of the probabilistic model. | |
| 150 | 154 | # softRestart = 1 |
| 151 | 155 | |
| 152 | ## Soft restart threshold value for numerical parameters. If NA, NULL or | |
| 153 | ## "", it is computed as 10^-digits. | |
| 156 | ## Soft restart threshold value for numerical parameters. If NA, NULL or "", | |
| 157 | ## it is computed as 10^-digits. | |
| 154 | 158 | # softRestartThreshold = "" |
| 155 | ||
| 156 | ## Directory where testing instances are located, either absolute or | |
| 157 | ## relative to current directory. | |
| 158 | # testInstancesDir = "" | |
| 159 | ||
| 160 | ## File containing a list of test instances and optionally additional | |
| 161 | ## parameters for them. | |
| 162 | # testInstancesFile = "" | |
| 163 | ||
| 164 | ## Number of elite configurations returned by irace that will be tested if | |
| 165 | ## test instances are provided. | |
| 166 | # testNbElites = 1 | |
| 167 | ||
| 168 | ## Enable/disable testing the elite configurations found at each | |
| 169 | ## iteration. | |
| 170 | # testIterationElites = 0 | |
| 171 | 159 | |
| 172 | 160 | ## Enable/disable elitist irace. |
| 173 | 161 | # elitist = 1 |
| 174 | 162 | |
| 175 | ## Number of instances added to the execution list before previous | |
| 176 | ## instances in elitist irace. | |
| 163 | ## Number of instances added to the execution list before previous instances | |
| 164 | ## in elitist irace. | |
| 177 | 165 | # elitistNewInstances = 1 |
| 178 | 166 | |
| 179 | ## In elitist irace, maximum number per race of elimination tests that do | |
| 180 | ## not eliminate a configuration. Use 0 for no limit. | |
| 167 | ## In elitist irace, maximum number per race of elimination tests that do not | |
| 168 | ## eliminate a configuration. Use 0 for no limit. | |
| 181 | 169 | # elitistLimit = 2 |
| 182 | 170 | |
| 183 | ## User-defined R function that takes a configuration generated by irace | |
| 184 | ## and repairs it. | |
| 171 | ## User-defined R function that takes a configuration generated by irace and | |
| 172 | ## repairs it. | |
| 185 | 173 | # repairConfiguration = "" |
| 186 | 174 | |
| 187 | ## Enable the use of adaptive capping, a technique designed for minimizing | |
| 188 | ## the computation time of configurations. This is only available when | |
| 189 | ## elitist is active. | |
| 175 | ## Enable the use of adaptive capping, a technique designed for minimizing the | |
| 176 | ## computation time of configurations. This is only available when elitist is | |
| 177 | ## active. | |
| 190 | 178 | # capping = 0 |
| 191 | 179 | |
| 192 | 180 | ## Measure used to obtain the execution bound from the performance of the |
| 193 | 181 | ## elite configurations: median, mean, worst, best. |
| 194 | 182 | # cappingType = "median" |
| 195 | 183 | |
| 196 | ## Method to calculate the mean performance of elite configurations: | |
| 197 | ## candidate or instance. | |
| 184 | ## Method to calculate the mean performance of elite configurations: candidate | |
| 185 | ## or instance. | |
| 198 | 186 | # boundType = "candidate" |
| 199 | 187 | |
| 200 | ## Maximum execution bound for targetRunner. It must be specified when | |
| 201 | ## capping is enabled. | |
| 188 | ## Maximum execution bound for targetRunner. It must be specified when capping | |
| 189 | ## is enabled. | |
| 202 | 190 | # boundMax = 0 |
| 203 | 191 | |
| 204 | 192 | ## Precision used for calculating the execution time. It must be specified |
| 212 | 200 | ## Replace the configuration cost of bounded executions with boundMax. |
| 213 | 201 | # boundAsTimeout = 1 |
| 214 | 202 | |
| 215 | ## Percentage of the configuration budget used to perform a postselection | |
| 216 | ## race of the best configurations of each iteration after the execution | |
| 217 | ## of irace. | |
| 203 | ## Percentage of the configuration budget used to perform a postselection race | |
| 204 | ## of the best configurations of each iteration after the execution of irace. | |
| 218 | 205 | # postselection = 0 |
| 219 | 206 | |
| 220 | 207 | ## Enable/disable AClib mode. This option enables compatibility with |
| 221 | 208 | ## GenericWrapper4AC as targetRunner script. |
| 222 | 209 | # aclib = 0 |
| 223 | 210 | |
| 211 | ## Maximum number of iterations. | |
| 212 | # nbIterations = 0 | |
| 213 | ||
| 214 | ## Number of runs of the target algorithm per iteration. | |
| 215 | # nbExperimentsPerIteration = 0 | |
| 216 | ||
| 217 | ## Minimum number of configurations needed to continue the execution of each | |
| 218 | ## race (iteration). | |
| 219 | # minNbSurvival = 0 | |
| 220 | ||
| 221 | ## Number of configurations to be sampled and evaluated at each iteration. | |
| 222 | # nbConfigurations = 0 | |
| 223 | ||
| 224 | ## Parameter used to define the number of configurations sampled and evaluated | |
| 225 | ## at each iteration. | |
| 226 | # mu = 5 | |
| 227 | ||
| 228 | ## Confidence level for the elimination test. | |
| 229 | # confidence = 0.95 | |
| 230 | ||
| 224 | 231 | ## END of scenario file |
| 225 | 232 | ############################################################################ |
| 226 | 233 | |
| 20 | 20 | exit 1 |
| 21 | 21 | } |
| 22 | 22 | |
| 23 | EXE=~/bin/program | |
| 24 | FIXED_PARAMS="" | |
| 25 | ||
| 23 | # This parses the arguments given by irace. Do not touch it! | |
| 26 | 24 | CONFIG_ID=$1 |
| 27 | 25 | INSTANCE_ID=$2 |
| 28 | 26 | SEED=$3 |
| 29 | 27 | INSTANCE=$4 |
| 30 | 28 | shift 4 || error "Not enough parameters" |
| 31 | 29 | CONFIG_PARAMS=$* |
| 30 | # End of parsing | |
| 32 | 31 | |
| 32 | ## Find our own location. | |
| 33 | # BINDIR=$(dirname "$(readlink -f "$(type -P $0 || echo $0)")") | |
| 34 | ||
| 35 | # EDIT THIS: Path to your executable | |
| 36 | EXE=~/bin/program | |
| 37 | # EDIT THIS: Specify how parameters are given to your executable | |
| 38 | EXE_PARAMS="--time 5 -i $INSTANCE --seed ${SEED} ${CONFIG_PARAMS}" | |
| 39 | ||
| 40 | if [ ! -x "$(command -v ${EXE})" ]; then | |
| 41 | error "${EXE}: not found or not executable (pwd: $(pwd))" | |
| 42 | fi | |
| 43 | ||
| 44 | # # If the program just prints a number, we can use 'exec' to avoid creating | |
| 45 | # # another process, but there can be no other commands after exec, only exit. | |
| 46 | # exec $EXE ${EXE_PARAMS} | |
| 47 | # exit 1 | |
| 48 | ||
| 49 | # These files are saved in execDir. Saving them in /tmp may be faster. | |
| 33 | 50 | STDOUT=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stdout |
| 34 | 51 | STDERR=c${CONFIG_ID}-${INSTANCE_ID}-${SEED}.stderr |
| 35 | 52 | |
| 36 | if [ ! -x "${EXE}" ]; then | |
| 37 | error "${EXE}: not found or not executable (pwd: $(pwd))" | |
| 38 | fi | |
| 53 | # # Otherwise, save the output to a file, and parse the result from it. (If | |
| 54 | # # you wish to ignore segmentation faults you can use '{}' around the | |
| 55 | # # command.) | |
| 56 | # # If the command below fails, but you are not sure why, it may be useful to | |
| 57 | # # print it before executing using 'echo', as in: | |
| 58 | # echo "$EXE ${EXE_PARAMS} 1> ${STDOUT} 2> ${STDERR}" | |
| 59 | $EXE ${EXE_PARAMS} 1> ${STDOUT} 2> ${STDERR} | |
| 39 | 60 | |
| 40 | # If the program just prints a number, we can use 'exec' to avoid | |
| 41 | # creating another process, but there can be no other commands after exec. | |
| 42 | #exec $EXE ${FIXED_PARAMS} -i $INSTANCE ${CONFIG_PARAMS} | |
| 43 | # exit 1 | |
| 44 | # | |
| 45 | # Otherwise, save the output to a file, and parse the result from it. | |
| 46 | # (If you wish to ignore segmentation faults you can use '{}' around | |
| 47 | # the command.) | |
| 48 | $EXE ${FIXED_PARAMS} -i $INSTANCE --seed ${SEED} ${CONFIG_PARAMS} 1> ${STDOUT} 2> ${STDERR} | |
| 49 | ||
| 50 | # # This may be used to introduce a delay if there are filesystem | |
| 51 | # # issues. | |
| 61 | # # This may be used to introduce a delay if there are filesystem issues. | |
| 52 | 62 | # SLEEPTIME=1 |
| 53 | 63 | # while [ ! -s "${STDOUT}" ]; do |
| 54 | 64 | # sleep $SLEEPTIME |
| 55 | 65 | # let "SLEEPTIME += 1" |
| 56 | 66 | # done |
| 57 | 67 | |
| 58 | # This is an example of reading a number from the output. | |
| 59 | # It assumes that the objective value is the first number in | |
| 60 | # the first column of the last line of the output. | |
| 61 | if [ -s "${STDOUT}" ]; then | |
| 62 | COST=$(tail -n 1 ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) | |
| 63 | echo "$COST" | |
| 64 | rm -f "${STDOUT}" "${STDERR}" | |
| 65 | exit 0 | |
| 66 | else | |
| 68 | if [ ! -s "${STDOUT}" ]; then | |
| 67 | 69 | error "${STDOUT}: No such file or directory" |
| 68 | 70 | fi |
| 71 | # This is an example of reading a number from the output. It assumes that the | |
| 72 | # objective value is the first number in the first column of the last line of | |
| 73 | # the output. | |
| 74 | COST=$(tail -n 1 ${STDOUT} | grep -e '^[[:space:]]*[+-]\?[0-9]' | cut -f1) | |
| 75 | echo "$COST" | |
| 76 | # Comment the following line if you wish to preserve temporary files | |
| 77 | rm -f "${STDOUT}" "${STDERR}" | |
| 78 | exit 0 | |
| 79 |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/argparser.R | |
| 2 | \name{CommandArgsParser} | |
| 3 | \alias{CommandArgsParser} | |
| 4 | \alias{cmdline_usage} | |
| 5 | \title{R6 Class for parsing command-line arguments} | |
| 6 | \usage{ | |
| 7 | cmdline_usage(cmdline_args) | |
| 8 | } | |
| 9 | \arguments{ | |
| 10 | \item{cmdline_args}{Definition of the command-line arguments.} | |
| 11 | } | |
| 12 | \description{ | |
| 13 | R6 Class for parsing command-line arguments | |
| 14 | ||
| 15 | R6 Class for parsing command-line arguments | |
| 16 | ||
| 17 | \code{cmdline_usage()} prints the output of \code{--help} | |
| 18 | } | |
| 19 | \section{Methods}{ | |
| 20 | \subsection{Public methods}{ | |
| 21 | \itemize{ | |
| 22 | \item \href{#method-new}{\code{CommandArgsParser$new()}} | |
| 23 | \item \href{#method-readCmdLineParameter}{\code{CommandArgsParser$readCmdLineParameter()}} | |
| 24 | \item \href{#method-readArg}{\code{CommandArgsParser$readArg()}} | |
| 25 | \item \href{#method-readAll}{\code{CommandArgsParser$readAll()}} | |
| 26 | \item \href{#method-cmdline_usage}{\code{CommandArgsParser$cmdline_usage()}} | |
| 27 | } | |
| 28 | } | |
| 29 | \if{html}{\out{<hr>}} | |
| 30 | \if{html}{\out{<a id="method-new"></a>}} | |
| 31 | \if{latex}{\out{\hypertarget{method-new}{}}} | |
| 32 | \subsection{Method \code{new()}}{ | |
| 33 | \subsection{Usage}{ | |
| 34 | \if{html}{\out{<div class="r">}}\preformatted{CommandArgsParser$new(argv, argsdef)}\if{html}{\out{</div>}} | |
| 35 | } | |
| 36 | ||
| 37 | } | |
| 38 | \if{html}{\out{<hr>}} | |
| 39 | \if{html}{\out{<a id="method-readCmdLineParameter"></a>}} | |
| 40 | \if{latex}{\out{\hypertarget{method-readCmdLineParameter}{}}} | |
| 41 | \subsection{Method \code{readCmdLineParameter()}}{ | |
| 42 | \subsection{Usage}{ | |
| 43 | \if{html}{\out{<div class="r">}}\preformatted{CommandArgsParser$readCmdLineParameter(paramName, default = NULL)}\if{html}{\out{</div>}} | |
| 44 | } | |
| 45 | ||
| 46 | } | |
| 47 | \if{html}{\out{<hr>}} | |
| 48 | \if{html}{\out{<a id="method-readArg"></a>}} | |
| 49 | \if{latex}{\out{\hypertarget{method-readArg}{}}} | |
| 50 | \subsection{Method \code{readArg()}}{ | |
| 51 | \subsection{Usage}{ | |
| 52 | \if{html}{\out{<div class="r">}}\preformatted{CommandArgsParser$readArg(short = "", long = "")}\if{html}{\out{</div>}} | |
| 53 | } | |
| 54 | ||
| 55 | } | |
| 56 | \if{html}{\out{<hr>}} | |
| 57 | \if{html}{\out{<a id="method-readAll"></a>}} | |
| 58 | \if{latex}{\out{\hypertarget{method-readAll}{}}} | |
| 59 | \subsection{Method \code{readAll()}}{ | |
| 60 | \subsection{Usage}{ | |
| 61 | \if{html}{\out{<div class="r">}}\preformatted{CommandArgsParser$readAll()}\if{html}{\out{</div>}} | |
| 62 | } | |
| 63 | ||
| 64 | } | |
| 65 | \if{html}{\out{<hr>}} | |
| 66 | \if{html}{\out{<a id="method-cmdline_usage"></a>}} | |
| 67 | \if{latex}{\out{\hypertarget{method-cmdline_usage}{}}} | |
| 68 | \subsection{Method \code{cmdline_usage()}}{ | |
| 69 | \subsection{Usage}{ | |
| 70 | \if{html}{\out{<div class="r">}}\preformatted{CommandArgsParser$cmdline_usage()}\if{html}{\out{</div>}} | |
| 71 | } | |
| 72 | ||
| 73 | } | |
| 74 | } |
| 1 | 1 | % Please edit documentation in R/ablation.R |
| 2 | 2 | \name{ablation} |
| 3 | 3 | \alias{ablation} |
| 4 | \title{Performs ablation between two configurations.} | |
| 4 | \title{Performs ablation between two configurations (from source to target).} | |
| 5 | 5 | \usage{ |
| 6 | ablation(iraceLogFile = NULL, iraceResults = NULL, src = NULL, | |
| 7 | target = NULL, ab.params = NULL, n.instances = NULL, | |
| 8 | type = "full", seed = 1234567, | |
| 9 | ablationLogFile = "log-ablation.Rdata", pdf.file = NULL, | |
| 10 | pdf.width = 20, mar = c(12, 5, 4, 1), debugLevel = NULL) | |
| 6 | ablation( | |
| 7 | iraceResults, | |
| 8 | src = 1L, | |
| 9 | target = NULL, | |
| 10 | ab.params = NULL, | |
| 11 | type = c("full", "racing"), | |
| 12 | n_instances = 1L, | |
| 13 | seed = 1234567, | |
| 14 | ablationLogFile = "log-ablation.Rdata", | |
| 15 | ... | |
| 16 | ) | |
| 11 | 17 | } |
| 12 | 18 | \arguments{ |
| 13 | \item{iraceLogFile}{Log file created by \pkg{irace}, this file must contain | |
| 14 | the \code{iraceResults} object.} | |
| 19 | \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} | |
| 15 | 20 | |
| 16 | \item{iraceResults}{Object created by \pkg{irace} and saved in | |
| 17 | \code{scenario$logFile}.} | |
| 21 | \item{src, target}{Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as \code{src} and the best configuration found by irace is used as target.} | |
| 18 | 22 | |
| 19 | \item{src, target}{Source and target configuration IDs. If \code{NULL}, then | |
| 20 | the first configuration ever evaluated is used as source and the best | |
| 21 | configuration found is used as target.} | |
| 23 | \item{ab.params}{Specific parameter names to be used for the ablation. They must be in \code{parameters$names}. By default, use all parameters.} | |
| 22 | 24 | |
| 23 | \item{ab.params}{Parameter names to be used for the ablation. They must be | |
| 24 | in parameters$names.} | |
| 25 | \item{type}{Type of ablation to perform: \code{"full"} will execute each configuration on all \code{n_instances} to determine the best-performing one; \code{"racing"} will apply racing to find the best configurations.} | |
| 25 | 26 | |
| 26 | \item{n.instances}{Number of instances to be used for the "full" ablation, | |
| 27 | if not provided firstTest instances are used.} | |
| 27 | \item{n_instances}{(\code{integer(1)}) Number of instances used in \code{"full"} ablation will be \code{n_instances * scenario$firstTest}.} | |
| 28 | 28 | |
| 29 | \item{type}{Type of ablation to perform, "full" will execute all instances | |
| 30 | in the configurations to determine the best performing, "racing" will | |
| 31 | apply racing to find the best configurations.} | |
| 29 | \item{seed}{(\code{integer(1)}) Integer value to use as seed for the random number generation.} | |
| 32 | 30 | |
| 33 | \item{seed}{Numerical value to use as seed for the random number generation.} | |
| 31 | \item{ablationLogFile}{(\code{character(1)}) Log file to save the ablation log. If \code{NULL}, the results are not saved to a file.} | |
| 34 | 32 | |
| 35 | \item{ablationLogFile}{Log file to save the ablation log.} | |
| 36 | ||
| 37 | \item{pdf.file}{Prefix that will be used to save the plot file of the | |
| 38 | ablation results.} | |
| 39 | ||
| 40 | \item{pdf.width}{Width provided to create the pdf file.} | |
| 41 | ||
| 42 | \item{mar}{Vector with the margins for the ablation plot.} | |
| 43 | ||
| 44 | \item{debugLevel}{(\code{integer(1)}) \cr Larger values produce more verbose output.} | |
| 33 | \item{...}{Further arguments to override scenario settings, e.g., \code{debugLevel}, \code{parallel}, etc.} | |
| 45 | 34 | } |
| 46 | 35 | \value{ |
| 47 | 36 | A list containing the following elements: |
| 48 | \describe{ | |
| 49 | \item{configurations}{Configurations tested in the ablation.} | |
| 50 | \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 51 | instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} | |
| 52 | \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 53 | \item{scenario}{Scenario object with the settings used for the experiments.} | |
| 54 | \item{trajectory}{IDs of the best configurations at each step of the ablation.} | |
| 55 | \item{best}{Best configuration found in the experiments.} | |
| 56 | } | |
| 37 | \describe{ | |
| 38 | \item{configurations}{Configurations tested in the ablation.} | |
| 39 | \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 40 | instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} | |
| 41 | \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 42 | \item{scenario}{Scenario object with the settings used for the experiments.} | |
| 43 | \item{trajectory}{IDs of the best configurations at each step of the ablation.} | |
| 44 | \item{best}{Best configuration found in the experiments.} | |
| 45 | } | |
| 57 | 46 | } |
| 58 | 47 | \description{ |
| 59 | 48 | Ablation is a method for analyzing the differences between two configurations. |
| 60 | 49 | } |
| 61 | 50 | \examples{ |
| 62 | 51 | \donttest{ |
| 63 | irace.logfile <- file.path(system.file(package="irace"), "exdata", "sann.rda") | |
| 64 | load(irace.logfile) | |
| 52 | logfile <- system.file(package="irace", "exdata", "sann.rda") | |
| 65 | 53 | # Execute ablation between the first and the best configuration found by irace. |
| 66 | ablation(iraceResults = iraceResults, ablationLogFile = NULL) | |
| 54 | ablog <- ablation(logfile, ablationLogFile = NULL) | |
| 55 | plotAblation(ablog) | |
| 67 | 56 | # Execute ablation between two selected configurations, and selecting only a |
| 68 | 57 | # subset of parameters, directly reading the setup from the irace log file. |
| 69 | ablation(iraceLogFile = irace.logfile, src = 1, target = 10, | |
| 70 | ab.params = c("temp"), ablationLogFile = NULL) | |
| 58 | ablog <- ablation(logfile, src = 1, target = 10, | |
| 59 | ab.params = c("temp"), ablationLogFile = NULL) | |
| 60 | plotAblation(ablog) | |
| 71 | 61 | } |
| 72 | 62 | |
| 73 | 63 | } |
| 75 | 65 | C. Fawcett and H. H. Hoos. Analysing differences between algorithm |
| 76 | 66 | configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016. |
| 77 | 67 | } |
| 68 | \seealso{ | |
| 69 | \code{\link[=plotAblation]{plotAblation()}} | |
| 70 | } | |
| 78 | 71 | \author{ |
| 79 | 72 | Leslie Pérez Cáceres and Manuel López-Ibáñez |
| 80 | 73 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/ablation.R | |
| 2 | \name{ablation_cmdline} | |
| 3 | \alias{ablation_cmdline} | |
| 4 | \title{Launch ablation with command-line options.} | |
| 5 | \usage{ | |
| 6 | ablation_cmdline(argv = commandArgs(trailingOnly = TRUE)) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{argv}{(\code{character()}) \cr The arguments | |
| 10 | provided on the R command line as a character vector, e.g., | |
| 11 | \code{c("-i", "irace.Rdata", "--src", 1)}.} | |
| 12 | } | |
| 13 | \value{ | |
| 14 | A list containing the following elements: | |
| 15 | \describe{ | |
| 16 | \item{configurations}{Configurations tested in the ablation.} | |
| 17 | \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 18 | instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} | |
| 19 | \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 20 | \item{scenario}{Scenario object with the settings used for the experiments.} | |
| 21 | \item{trajectory}{IDs of the best configurations at each step of the ablation.} | |
| 22 | \item{best}{Best configuration found in the experiments.} | |
| 23 | } | |
| 24 | } | |
| 25 | \description{ | |
| 26 | Launch \code{\link[=ablation]{ablation()}} with the same command-line options as the command-line | |
| 27 | executable (\code{ablation.exe} in Windows). | |
| 28 | } | |
| 29 | \details{ | |
| 30 | The function reads the parameters given on the command line | |
| 31 | used to invoke R, launches \code{\link[=ablation]{ablation()}} and possibly \code{\link[=plotAblation]{plotAblation()}}. | |
| 32 | ||
| 33 | List of command-line options:\preformatted{-l,--log-file Path to the (.Rdata) file created by irace from which | |
| 34 | the "iraceResults" object will be loaded. | |
| 35 | -S,--src Source configuration ID. Default: 1. | |
| 36 | -T,--target Target configuration ID. By default the best | |
| 37 | configuration found by irace. | |
| 38 | -P,--params Specific parameter names to be used for the ablation | |
| 39 | (separated with commas). By default use all | |
| 40 | -t,--type Type of ablation to perform: "full" will execute each | |
| 41 | configuration on all "--n-instances" to determine the | |
| 42 | best-performing one; "racing" will apply racing to | |
| 43 | find the best configurations. Default: full. | |
| 44 | -n,--n-instances Number of instances used in "full" ablation will be | |
| 45 | n_instances * scenario$firstTest. Default: 1. | |
| 46 | --seed Integer value to use as seed for the random number | |
| 47 | generation. Default: 1234567. | |
| 48 | -o,--output-file Log file to save the ablation log. If "", the results | |
| 49 | are not saved to a file. Default: log-ablation.Rdata. | |
| 50 | -p,--plot Output filename (.pdf) for the plot. If not given, no | |
| 51 | plot is created. | |
| 52 | -O,--plot-type Type of plot. Supported values are "mean" and | |
| 53 | "boxplot". Default: mean. | |
| 54 | --old-path Old path found in the log-file (.Rdata) given as input | |
| 55 | to be replaced by --new-path. | |
| 56 | --new-path New path to replace the path found in the log-file | |
| 57 | (.Rdata) given as input. | |
| 58 | -e,--exec-dir Directory where the target runner will be run. | |
| 59 | -s,--scenario Scenario file to override the scenario given in the | |
| 60 | log-file (.Rdata) | |
| 61 | --parallel Number of calls to targetRunner to execute in | |
| 62 | parallel. Values 0 or 1 mean no parallelization. | |
| 63 | } | |
| 64 | } | |
| 65 | \examples{ | |
| 66 | # ablation_cmdline("--help") | |
| 67 | ||
| 68 | } | |
| 69 | \author{ | |
| 70 | Manuel López-Ibáñez | |
| 71 | } | |
| 72 | \concept{running} |
| 33 | 33 | irace.logfile <- file.path(system.file(package="irace"), |
| 34 | 34 | "exdata", "irace-acotsp.Rdata") |
| 35 | 35 | load(irace.logfile) |
| 36 | attach(iraceResults) | |
| 36 | allConfigurations <- iraceResults$allConfigurations | |
| 37 | parameters <- iraceResults$parameters | |
| 37 | 38 | apply(allConfigurations[1:10, unlist(parameters$names)], 1, buildCommandLine, |
| 38 | 39 | unlist(parameters$switches)) |
| 39 | 40 |
| 3 | 3 | \alias{checkIraceScenario} |
| 4 | 4 | \title{Test that the given irace scenario can be run.} |
| 5 | 5 | \usage{ |
| 6 | checkIraceScenario(scenario, parameters = NULL) | |
| 6 | checkIraceScenario(scenario, parameters) | |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | 12 | |
| 13 | 13 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter |
| 14 | 14 | space definition. The data structure has to similar to the one returned by the |
| 15 | 15 | function \code{\link{readParameters}}.} |
| 16 | 16 | } |
| 17 | 17 | \value{ |
| 18 | returns \code{TRUE} if succesful and gives an error and returns | |
| 18 | returns \code{TRUE} if successful and gives an error and returns | |
| 19 | 19 | \code{FALSE} otherwise. |
| 20 | 20 | } |
| 21 | 21 | \description{ |
| 22 | \code{checkIraceScenario} tests that the given irace scenario | |
| 23 | can be run by checking the scenario settings provided and trying to run | |
| 24 | the target-algorithm. | |
| 22 | Test that the given irace scenario can be run by checking the scenario | |
| 23 | settings provided and trying to run the target-algorithm. | |
| 25 | 24 | } |
| 26 | 25 | \details{ |
| 27 | Provide the \code{parameters} argument only if the parameter list | |
| 28 | should not be obtained from the parameter file given by the scenario. If | |
| 29 | the parameter list is provided it will not be checked. This function will | |
| 30 | try to execute the target-algorithm. | |
| 26 | If the \code{parameters} argument is missing, then the parameters | |
| 27 | will be read from the file \code{parameterFile} given by \code{scenario}. If | |
| 28 | \code{parameters} is provided, then \code{parameterFile} will not be read. This function will | |
| 29 | try to execute the target-algorithm. | |
| 31 | 30 | } |
| 32 | 31 | \seealso{ |
| 33 | 32 | \describe{ |
| 34 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 35 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 36 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 37 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 33 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 34 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 35 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 36 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 38 | 37 | } |
| 39 | 38 | } |
| 40 | 39 | \author{ |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/readParameters.R | |
| 2 | \name{checkParameters} | |
| 3 | \alias{checkParameters} | |
| 4 | \title{checkParameters} | |
| 5 | \usage{ | |
| 6 | checkParameters(parameters) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter | |
| 10 | space definition. The data structure has to similar to the one returned by the | |
| 11 | function \code{\link{readParameters}}.} | |
| 12 | } | |
| 13 | \description{ | |
| 14 | FIXME: This is incomplete, for now we only repair inputs from previous irace | |
| 15 | versions. | |
| 16 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | 12 | } |
| 13 | 13 | \value{ |
| 14 | 14 | The scenario received as a parameter, possibly corrected. Unset |
| 15 | 15 | scenario settings are set to their default values. |
| 16 | 16 | } |
| 17 | 17 | \description{ |
| 18 | \code{checkScenario} takes a (possibly incomplete) scenario setup of | |
| 19 | \pkg{irace}, checks for errors and transforms it into a valid scenario. | |
| 18 | Checks for errors a (possibly incomplete) scenario setup of | |
| 19 | \pkg{irace} and transforms it into a valid scenario. | |
| 20 | 20 | } |
| 21 | 21 | \details{ |
| 22 | 22 | This function checks that the directories and the file names |
| 28 | 28 | } |
| 29 | 29 | \seealso{ |
| 30 | 30 | \describe{ |
| 31 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 32 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 33 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 34 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 31 | \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} | |
| 32 | \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} | |
| 33 | \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} | |
| 34 | \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} | |
| 35 | 35 | } |
| 36 | 36 | } |
| 37 | 37 | \author{ |
| 20 | 20 | Print configurations as a data frame |
| 21 | 21 | } |
| 22 | 22 | \seealso{ |
| 23 | \code{\link{configurations.print.command}} to print the configurations as command-line strings. | |
| 23 | \code{\link[=configurations.print.command]{configurations.print.command()}} to print the configurations as command-line strings. | |
| 24 | 24 | } |
| 25 | 25 | \author{ |
| 26 | 26 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 21 | 21 | command-line. |
| 22 | 22 | } |
| 23 | 23 | \seealso{ |
| 24 | \code{\link{configurations.print}} to print the configurations as a data frame. | |
| 24 | \code{\link[=configurations.print]{configurations.print()}} to print the configurations as a data frame. | |
| 25 | 25 | } |
| 26 | 26 | \author{ |
| 27 | 27 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/parameterAnalysis.R | |
| 2 | \name{configurationsBoxplot} | |
| 3 | \alias{configurationsBoxplot} | |
| 4 | \title{Creates box plots of the quality of configurations.} | |
| 5 | \usage{ | |
| 6 | configurationsBoxplot(experiments, title = NULL, | |
| 7 | xlabel = "Configuration ID", ylabel = "Configuration cost", | |
| 8 | filename = NULL) | |
| 9 | } | |
| 10 | \arguments{ | |
| 11 | \item{experiments}{Matrix of performance of configurations (columns) over a set of instances (rows).} | |
| 12 | ||
| 13 | \item{title}{(\code{NULL}) Title for the plot.} | |
| 14 | ||
| 15 | \item{xlabel}{Label for the x axis.} | |
| 16 | ||
| 17 | \item{ylabel}{Label for the y axis.} | |
| 18 | ||
| 19 | \item{filename}{(\code{NULL}) Filename prefix to create a pdf file with the plot.} | |
| 20 | } | |
| 21 | \value{ | |
| 22 | Box plot of the performance of the configurations. | |
| 23 | } | |
| 24 | \description{ | |
| 25 | Creates box plots of the quality of configurations. | |
| 26 | } | |
| 27 | \author{ | |
| 28 | Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 29 | } |
| 3 | 3 | \alias{defaultScenario} |
| 4 | 4 | \title{Default scenario settings} |
| 5 | 5 | \usage{ |
| 6 | defaultScenario(scenario = list()) | |
| 6 | defaultScenario(scenario = list(), params_def = .irace.params.def) | |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | ||
| 13 | \item{params_def}{(\code{data.frame()}) \cr Definition of the options accepted by the scenario. This should only be modified by packages that wish to extend \pkg{irace}.} | |
| 12 | 14 | } |
| 13 | 15 | \value{ |
| 14 | 16 | A list indexed by the \pkg{irace} parameter names, |
| 15 | 17 | containing the default values for each parameter, except for those |
| 16 | 18 | already present in the scenario passed as argument. |
| 17 | The scenario list contains the following elements: | |
| 19 | The scenario list contains the following elements: | |
| 18 | 20 | \itemize{ |
| 19 | \item General options: | |
| 20 | \describe{ | |
| 21 | \item{\code{scenarioFile}}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: \code{"./scenario.txt"})} | |
| 22 | \item{\code{execDir}}{Directory where the programs will be run. (Default: \code{"./"})} | |
| 23 | \item{\code{logFile}}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: \code{"./irace.Rdata"})} | |
| 24 | \item{\code{debugLevel}}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: \code{0})} | |
| 25 | \item{\code{seed}}{Seed of the random number generator (by default, generate a random seed). (Default: \code{NA})} | |
| 26 | \item{\code{repairConfiguration}}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: \code{""})} | |
| 27 | \item{\code{postselection}}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: \code{0})} | |
| 28 | \item{\code{aclib}}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: \code{0})} | |
| 29 | } | |
| 30 | \item Elitist \code{irace}: | |
| 31 | \describe{ | |
| 32 | \item{\code{elitist}}{Enable/disable elitist irace. (Default: \code{1})} | |
| 33 | \item{\code{elitistNewInstances}}{Number of instances added to the execution list before previous instances in elitist irace. (Default: \code{1})} | |
| 34 | \item{\code{elitistLimit}}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: \code{2})} | |
| 35 | } | |
| 36 | \item Internal \code{irace} options: | |
| 37 | \describe{ | |
| 38 | \item{\code{nbIterations}}{Number of iterations. (Default: \code{0})} | |
| 39 | \item{\code{nbExperimentsPerIteration}}{Number of runs of the target algorithm per iteration. (Default: \code{0})} | |
| 40 | \item{\code{sampleInstances}}{Randomly sample the training instances or use them in the order given. (Default: \code{1})} | |
| 41 | \item{\code{minNbSurvival}}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: \code{0})} | |
| 42 | \item{\code{nbConfigurations}}{Number of configurations to be sampled and evaluated at each iteration. (Default: \code{0})} | |
| 43 | \item{\code{mu}}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: \code{5})} | |
| 44 | \item{\code{softRestart}}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: \code{1})} | |
| 45 | \item{\code{softRestartThreshold}}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: \code{""})} | |
| 46 | } | |
| 47 | \item Target algorithm parameters: | |
| 48 | \describe{ | |
| 49 | \item{\code{parameterFile}}{File that contains the description of the parameters of the target algorithm. (Default: \code{"./parameters.txt"})} | |
| 50 | \item{\code{forbiddenExps}}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: \code{""})} | |
| 51 | \item{\code{forbiddenFile}}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: \code{""})} | |
| 52 | \item{\code{digits}}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: \code{4})} | |
| 53 | } | |
| 54 | \item Target algorithm execution: | |
| 55 | \describe{ | |
| 56 | \item{\code{targetRunner}}{Script called for each configuration that executes the target algorithm to be tuned. See templates. (Default: \code{"./target-runner"})} | |
| 57 | \item{\code{targetRunnerRetries}}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: \code{0})} | |
| 58 | \item{\code{targetRunnerData}}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: \code{""})} | |
| 59 | \item{\code{targetRunnerParallel}}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: \code{""})} | |
| 60 | \item{\code{targetEvaluator}}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: \code{""})} | |
| 61 | \item{\code{deterministic}}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: \code{0})} | |
| 62 | \item{\code{parallel}}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: \code{0})} | |
| 63 | \item{\code{loadBalancing}}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: \code{1})} | |
| 64 | \item{\code{mpi}}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: \code{0})} | |
| 65 | \item{\code{batchmode}}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque or slurm. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: \code{0})} | |
| 66 | } | |
| 67 | \item Initial configurations: | |
| 68 | \describe{ | |
| 69 | \item{\code{initConfigurations}}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: \code{""})} | |
| 70 | \item{\code{configurationsFile}}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: \code{""})} | |
| 71 | } | |
| 72 | \item Training instances: | |
| 73 | \describe{ | |
| 74 | \item{\code{instances}}{Character vector of the instances to be used in the \code{targetRunner}. (Default: \code{""})} | |
| 75 | \item{\code{trainInstancesDir}}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: \code{"./Instances"})} | |
| 76 | \item{\code{trainInstancesFile}}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: \code{""})} | |
| 77 | } | |
| 78 | \item Tuning budget: | |
| 79 | \describe{ | |
| 80 | \item{\code{maxExperiments}}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: \code{0})} | |
| 81 | \item{\code{maxTime}}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: \code{0})} | |
| 82 | \item{\code{budgetEstimation}}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: \code{0.02})} | |
| 83 | } | |
| 84 | \item Statistical test: | |
| 85 | \describe{ | |
| 86 | \item{\code{testType}}{Statistical test used for elimination. Default test is always \code{F-test} unless \code{capping} is enabled, in which case the default test is \code{t-test}. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: \code{"F-test"})} | |
| 87 | \item{\code{firstTest}}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: \code{5})} | |
| 88 | \item{\code{eachTest}}{Number of instances evaluated between elimination tests. (Default: \code{1})} | |
| 89 | \item{\code{confidence}}{Confidence level for the elimination test. (Default: \code{0.95})} | |
| 90 | } | |
| 91 | \item Adaptive capping: | |
| 92 | \describe{ | |
| 93 | \item{\code{capping}}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: \code{0})} | |
| 94 | \item{\code{cappingType}}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: \code{"median"})} | |
| 95 | \item{\code{boundType}}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: \code{"candidate"})} | |
| 96 | \item{\code{boundMax}}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: \code{0})} | |
| 97 | \item{\code{boundDigits}}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: \code{0})} | |
| 98 | \item{\code{boundPar}}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: \code{1})} | |
| 99 | \item{\code{boundAsTimeout}}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: \code{1})} | |
| 100 | } | |
| 101 | \item Recovery: | |
| 102 | \describe{ | |
| 103 | \item{\code{recoveryFile}}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: \code{""})} | |
| 104 | } | |
| 105 | \item Testing: | |
| 106 | \describe{ | |
| 107 | \item{\code{testInstancesDir}}{Directory where testing instances are located, either absolute or relative to current directory. (Default: \code{""})} | |
| 108 | \item{\code{testInstancesFile}}{File containing a list of test instances and optionally additional parameters for them. (Default: \code{""})} | |
| 109 | \item{\code{testInstances}}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: \code{""})} | |
| 110 | \item{\code{testNbElites}}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: \code{1})} | |
| 111 | \item{\code{testIterationElites}}{Enable/disable testing the elite configurations found at each iteration. (Default: \code{0})} | |
| 112 | } | |
| 21 | \item General options: | |
| 22 | \describe{ | |
| 23 | \item{\code{scenarioFile}}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: \code{"./scenario.txt"})} | |
| 24 | \item{\code{execDir}}{Directory where the programs will be run. (Default: \code{"./"})} | |
| 25 | \item{\code{logFile}}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: \code{"./irace.Rdata"})} | |
| 26 | \item{\code{quiet}}{Reduce the output generated by irace to a minimum. (Default: \code{0})} | |
| 27 | \item{\code{debugLevel}}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: \code{0})} | |
| 28 | \item{\code{seed}}{Seed of the random number generator (by default, generate a random seed). (Default: \code{NA})} | |
| 29 | \item{\code{repairConfiguration}}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: \code{""})} | |
| 30 | \item{\code{postselection}}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: \code{0})} | |
| 31 | \item{\code{aclib}}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: \code{0})} | |
| 32 | } | |
| 33 | \item Elitist \code{irace}: | |
| 34 | \describe{ | |
| 35 | \item{\code{elitist}}{Enable/disable elitist irace. (Default: \code{1})} | |
| 36 | \item{\code{elitistNewInstances}}{Number of instances added to the execution list before previous instances in elitist irace. (Default: \code{1})} | |
| 37 | \item{\code{elitistLimit}}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: \code{2})} | |
| 38 | } | |
| 39 | \item Internal \code{irace} options: | |
| 40 | \describe{ | |
| 41 | \item{\code{sampleInstances}}{Randomly sample the training instances or use them in the order given. (Default: \code{1})} | |
| 42 | \item{\code{softRestart}}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: \code{1})} | |
| 43 | \item{\code{softRestartThreshold}}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: \code{""})} | |
| 44 | \item{\code{nbIterations}}{Maximum number of iterations. (Default: \code{0})} | |
| 45 | \item{\code{nbExperimentsPerIteration}}{Number of runs of the target algorithm per iteration. (Default: \code{0})} | |
| 46 | \item{\code{minNbSurvival}}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: \code{0})} | |
| 47 | \item{\code{nbConfigurations}}{Number of configurations to be sampled and evaluated at each iteration. (Default: \code{0})} | |
| 48 | \item{\code{mu}}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: \code{5})} | |
| 49 | } | |
| 50 | \item Target algorithm parameters: | |
| 51 | \describe{ | |
| 52 | \item{\code{parameterFile}}{File that contains the description of the parameters of the target algorithm. (Default: \code{"./parameters.txt"})} | |
| 53 | \item{\code{forbiddenExps}}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: \code{""})} | |
| 54 | \item{\code{forbiddenFile}}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: \code{""})} | |
| 55 | \item{\code{digits}}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: \code{4})} | |
| 56 | } | |
| 57 | \item Target algorithm execution: | |
| 58 | \describe{ | |
| 59 | \item{\code{targetRunner}}{Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. (Default: \code{"./target-runner"})} | |
| 60 | \item{\code{targetRunnerLauncher}}{Executable that will be used to launch the target runner, when \code{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). (Default: \code{""})} | |
| 61 | \item{\code{targetRunnerLauncherArgs}}{Command-line arguments provided to \code{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \code{targetRunner} and by the arguments usually passed when calling \code{targetRunner}, respectively. Example: \code{"-m {targetRunner} --args {targetRunnerArgs}"}. (Default: \code{"{targetRunner} {targetRunnerArgs}"})} | |
| 62 | \item{\code{targetRunnerRetries}}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: \code{0})} | |
| 63 | \item{\code{targetRunnerData}}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: \code{""})} | |
| 64 | \item{\code{targetRunnerParallel}}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: \code{""})} | |
| 65 | \item{\code{targetEvaluator}}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: \code{""})} | |
| 66 | \item{\code{deterministic}}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: \code{0})} | |
| 67 | \item{\code{parallel}}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: \code{0})} | |
| 68 | \item{\code{loadBalancing}}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: \code{1})} | |
| 69 | \item{\code{mpi}}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: \code{0})} | |
| 70 | \item{\code{batchmode}}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: \code{0})} | |
| 71 | } | |
| 72 | \item Initial configurations: | |
| 73 | \describe{ | |
| 74 | \item{\code{initConfigurations}}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: \code{""})} | |
| 75 | \item{\code{configurationsFile}}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: \code{""})} | |
| 76 | } | |
| 77 | \item Training instances: | |
| 78 | \describe{ | |
| 79 | \item{\code{instances}}{Character vector of the instances to be used in the \code{targetRunner}. (Default: \code{""})} | |
| 80 | \item{\code{trainInstancesDir}}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: \code{"./Instances"})} | |
| 81 | \item{\code{trainInstancesFile}}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: \code{""})} | |
| 82 | } | |
| 83 | \item Tuning budget: | |
| 84 | \describe{ | |
| 85 | \item{\code{maxExperiments}}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: \code{0})} | |
| 86 | \item{\code{maxTime}}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: \code{0})} | |
| 87 | \item{\code{budgetEstimation}}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: \code{0.02})} | |
| 88 | \item{\code{minMeasurableTime}}{Minimum time unit that is still (significantly) measureable. (Default: \code{0.01})} | |
| 89 | } | |
| 90 | \item Statistical test: | |
| 91 | \describe{ | |
| 92 | \item{\code{testType}}{Statistical test used for elimination. The default value selects \code{t-test} if \code{capping} is enabled or \code{F-test}, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: \code{""})} | |
| 93 | \item{\code{firstTest}}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: \code{5})} | |
| 94 | \item{\code{eachTest}}{Number of instances evaluated between elimination tests. (Default: \code{1})} | |
| 95 | \item{\code{confidence}}{Confidence level for the elimination test. (Default: \code{0.95})} | |
| 96 | } | |
| 97 | \item Adaptive capping: | |
| 98 | \describe{ | |
| 99 | \item{\code{capping}}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: \code{0})} | |
| 100 | \item{\code{cappingType}}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: \code{"median"})} | |
| 101 | \item{\code{boundType}}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: \code{"candidate"})} | |
| 102 | \item{\code{boundMax}}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: \code{0})} | |
| 103 | \item{\code{boundDigits}}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: \code{0})} | |
| 104 | \item{\code{boundPar}}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: \code{1})} | |
| 105 | \item{\code{boundAsTimeout}}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: \code{1})} | |
| 106 | } | |
| 107 | \item Recovery: | |
| 108 | \describe{ | |
| 109 | \item{\code{recoveryFile}}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: \code{""})} | |
| 110 | } | |
| 111 | \item Testing: | |
| 112 | \describe{ | |
| 113 | \item{\code{testInstancesDir}}{Directory where testing instances are located, either absolute or relative to current directory. (Default: \code{""})} | |
| 114 | \item{\code{testInstancesFile}}{File containing a list of test instances and optionally additional parameters for them. (Default: \code{""})} | |
| 115 | \item{\code{testInstances}}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: \code{""})} | |
| 116 | \item{\code{testNbElites}}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: \code{1})} | |
| 117 | \item{\code{testIterationElites}}{Enable/disable testing the elite configurations found at each iteration. (Default: \code{0})} | |
| 118 | } | |
| 113 | 119 | } |
| 114 | 120 | } |
| 115 | 121 | \description{ |
| 116 | Return scenario with default values. | |
| 122 | Return scenario object with default values. | |
| 117 | 123 | } |
| 118 | 124 | \seealso{ |
| 119 | 125 | \describe{ |
| 120 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 121 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 122 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 123 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 126 | \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} | |
| 127 | \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} | |
| 128 | \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} | |
| 129 | \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} | |
| 124 | 130 | } |
| 125 | 131 | } |
| 126 | 132 | \author{ |
| 3 | 3 | \alias{getConfigurationById} |
| 4 | 4 | \title{Returns the configurations selected by ID.} |
| 5 | 5 | \usage{ |
| 6 | getConfigurationById(iraceResults = NULL, logFile = NULL, ids, | |
| 7 | drop.metadata = FALSE) | |
| 6 | getConfigurationById(iraceResults, ids, drop.metadata = FALSE) | |
| 8 | 7 | } |
| 9 | 8 | \arguments{ |
| 10 | \item{iraceResults}{Object created by \pkg{irace} and saved in \code{scenario$logFile}.} | |
| 11 | ||
| 12 | \item{logFile}{Log file created by \pkg{irace}, this file must contain the | |
| 13 | \code{iraceResults} object.} | |
| 9 | \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} | |
| 14 | 10 | |
| 15 | 11 | \item{ids}{The id or a vector of ids of the candidates configurations to obtain.} |
| 16 | 12 | |
| 17 | 13 | \item{drop.metadata}{Remove metadata, such the configuration ID and |
| 18 | 14 | the ID of the parent, from the returned configurations. See |
| 19 | \code{\link{removeConfigurationsMetaData}}.} | |
| 15 | \code{\link[=removeConfigurationsMetaData]{removeConfigurationsMetaData()}}.} | |
| 20 | 16 | } |
| 21 | 17 | \value{ |
| 22 | 18 | A data frame containing the elite configurations required. |
| 27 | 23 | \author{ |
| 28 | 24 | Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 29 | 25 | } |
| 26 | \concept{analysis} | |
| 3 | 3 | \alias{getConfigurationByIteration} |
| 4 | 4 | \title{Returns the configurations by the iteration in which they were executed.} |
| 5 | 5 | \usage{ |
| 6 | getConfigurationByIteration(iraceResults = NULL, logFile = NULL, | |
| 7 | iterations, drop.metadata = FALSE) | |
| 6 | getConfigurationByIteration(iraceResults, iterations, drop.metadata = FALSE) | |
| 8 | 7 | } |
| 9 | 8 | \arguments{ |
| 10 | \item{iraceResults}{(\code{NULL}) Object created by \pkg{irace} and saved in \code{scenario$logFile}.} | |
| 9 | \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} | |
| 11 | 10 | |
| 12 | \item{logFile}{(\code{NULL}) Log file created by \pkg{irace}, this file must contain the | |
| 13 | \code{iraceResults} object.} | |
| 14 | ||
| 15 | \item{iterations}{The iteration number or a vector of iteration numbers from where | |
| 11 | \item{iterations}{The iteration number or a vector of iteration numbers from where | |
| 16 | 12 | the configurations should be obtained.} |
| 17 | 13 | |
| 18 | 14 | \item{drop.metadata}{(\code{FALSE}) Remove metadata, such the configuration ID and |
| 28 | 24 | \author{ |
| 29 | 25 | Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 30 | 26 | } |
| 27 | \concept{analysis} | |
| 3 | 3 | \alias{getFinalElites} |
| 4 | 4 | \title{Return the elite configurations of the final iteration.} |
| 5 | 5 | \usage{ |
| 6 | getFinalElites(iraceResults = NULL, logFile = NULL, n = 0, | |
| 7 | drop.metadata = FALSE) | |
| 6 | getFinalElites(iraceResults, n = 0L, drop.metadata = FALSE) | |
| 8 | 7 | } |
| 9 | 8 | \arguments{ |
| 10 | \item{iraceResults}{Object created by \pkg{irace} and saved in \code{scenario$logFile}.} | |
| 9 | \item{iraceResults}{(\code{list()}|\code{character(1)}) \code{iraceResults} object created by \pkg{irace} and typically saved in the log file \code{irace.Rdata}. If a character string is given, then it is interpreted as the path to the log file from which the \code{iraceResults} object will be loaded.} | |
| 11 | 10 | |
| 12 | \item{logFile}{Log file created by \pkg{irace}, this file must contain the | |
| 13 | \code{iraceResults} object.} | |
| 14 | ||
| 15 | \item{n}{Number of elite configurations to return, if \code{n} is larger than the | |
| 16 | number of configurations, then only the existing ones are returned.} | |
| 11 | \item{n}{Number of elite configurations to return, if \code{n} is larger than the | |
| 12 | number of configurations, then only the existing ones are returned. The default (\code{n=0}) returns all of them.} | |
| 17 | 13 | |
| 18 | 14 | \item{drop.metadata}{Remove metadata, such the configuration ID and |
| 19 | 15 | the ID of the parent, from the returned configurations. See |
| 25 | 21 | \description{ |
| 26 | 22 | Return the elite configurations of the final iteration. |
| 27 | 23 | } |
| 24 | \examples{ | |
| 25 | log_file <- system.file("exdata/irace-acotsp.Rdata", package="irace", mustWork=TRUE) | |
| 26 | print(removeConfigurationsMetaData(getFinalElites(log_file, n=1))) | |
| 27 | ||
| 28 | } | |
| 28 | 29 | \author{ |
| 29 | 30 | Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 30 | 31 | } |
| 32 | \concept{analysis} | |
| 35 | 35 | sum(x * x - 10 * cos(2 * pi * x) + 10) |
| 36 | 36 | } |
| 37 | 37 | |
| 38 | ## We generate 200 instances (in this case, weights): | |
| 39 | weights <- rnorm(200, mean = 0.9, sd = 0.02) | |
| 38 | ## We generate 20 instances (in this case, weights): | |
| 39 | weights <- rnorm(20, mean = 0.9, sd = 0.02) | |
| 40 | 40 | |
| 41 | 41 | ## On this set of instances, we are interested in optimizing two |
| 42 | 42 | ## parameters of the SANN algorithm: tmax and temp. We setup the |
| 43 | 43 | ## parameter space as follows: |
| 44 | parameters.table <- ' | |
| 45 | tmax "" i (1, 5000) | |
| 44 | parameters_table <- ' | |
| 45 | tmax "" i,log (1, 5000) | |
| 46 | 46 | temp "" r (0, 100) |
| 47 | 47 | ' |
| 48 | 48 | |
| 49 | 49 | ## We use the irace function readParameters to read this table: |
| 50 | parameters <- readParameters(text = parameters.table) | |
| 50 | parameters <- readParameters(text = parameters_table) | |
| 51 | 51 | |
| 52 | 52 | ## Next, we define the function that will evaluate each candidate |
| 53 | 53 | ## configuration on a single instance. For simplicity, we restrict to |
| 54 | 54 | ## three-dimensional functions and we set the maximum number of |
| 55 | ## iterations of SANN to 5000. | |
| 56 | target.runner <- function(experiment, scenario) | |
| 55 | ## iterations of SANN to 1000. | |
| 56 | target_runner <- function(experiment, scenario) | |
| 57 | 57 | { |
| 58 | 58 | instance <- experiment$instance |
| 59 | 59 | configuration <- experiment$configuration |
| 65 | 65 | return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) |
| 66 | 66 | } |
| 67 | 67 | res <- stats::optim(par,fn, method="SANN", |
| 68 | control=list(maxit=5000 | |
| 68 | control=list(maxit=1000 | |
| 69 | 69 | , tmax = as.numeric(configuration[["tmax"]]) |
| 70 | 70 | , temp = as.numeric(configuration[["temp"]]) |
| 71 | 71 | )) |
| 74 | 74 | ## - 'error' is a string used to report an error |
| 75 | 75 | ## - 'outputRaw' is a string used to report the raw output of calls to |
| 76 | 76 | ## an external program or function. |
| 77 | ## - 'call' is a string used to report how target.runner called the | |
| 77 | ## - 'call' is a string used to report how target_runner called the | |
| 78 | 78 | ## external program or function. |
| 79 | 79 | return(list(cost = res$value)) |
| 80 | 80 | } |
| 81 | 81 | |
| 82 | 82 | ## We define a configuration scenario by setting targetRunner to the |
| 83 | ## function define above, instances to the first 100 random weights, and | |
| 84 | ## a maximum budget of 1000 calls to targetRunner. | |
| 85 | scenario <- list(targetRunner = target.runner, | |
| 86 | instances = weights[1:100], | |
| 87 | maxExperiments = 1000, | |
| 83 | ## function define above, instances to the first 10 random weights, and | |
| 84 | ## a maximum budget of 'maxExperiments' calls to targetRunner. | |
| 85 | scenario <- list(targetRunner = target_runner, | |
| 86 | instances = weights[1:10], | |
| 87 | maxExperiments = 500, | |
| 88 | 88 | # Do not create a logFile |
| 89 | 89 | logFile = "") |
| 90 | 90 | |
| 91 | 91 | ## We check that the scenario is valid. This will also try to execute |
| 92 | ## target.runner. | |
| 92 | ## target_runner. | |
| 93 | 93 | checkIraceScenario(scenario, parameters = parameters) |
| 94 | 94 | |
| 95 | 95 | \donttest{ |
| 96 | 96 | ## We are now ready to launch irace. We do it by means of the irace |
| 97 | 97 | ## function. The function will print information about its |
| 98 | 98 | ## progress. This may require a few minutes, so it is not run by default. |
| 99 | tuned.confs <- irace(scenario = scenario, parameters = parameters) | |
| 99 | tuned_confs <- irace(scenario = scenario, parameters = parameters) | |
| 100 | 100 | |
| 101 | 101 | ## We can print the best configurations found by irace as follows: |
| 102 | configurations.print(tuned.confs) | |
| 102 | configurations.print(tuned_confs) | |
| 103 | 103 | |
| 104 | 104 | ## We can evaluate the quality of the best configuration found by |
| 105 | 105 | ## irace versus the default configuration of the SANN algorithm on |
| 106 | ## the other 100 instances previously generated. | |
| 106 | ## the other 10 instances previously generated. | |
| 107 | 107 | ## To do so, first we apply the default configuration of the SANN |
| 108 | 108 | ## algorithm to these instances: |
| 109 | 109 | test <- function(configuration) |
| 110 | 110 | { |
| 111 | res <- lapply(weights[101:200], | |
| 112 | function(x) target.runner( | |
| 111 | res <- lapply(weights[11:20], | |
| 112 | function(x) target_runner( | |
| 113 | 113 | experiment = list(instance = x, |
| 114 | 114 | configuration = configuration), |
| 115 | 115 | scenario = scenario)) |
| 118 | 118 | default <- test(data.frame(tmax=10, temp=10)) |
| 119 | 119 | ## We extract and apply the winning configuration found by irace |
| 120 | 120 | ## to these instances: |
| 121 | tuned <- test (removeConfigurationsMetaData(tuned.confs[1,])) | |
| 121 | tuned <- test(removeConfigurationsMetaData(tuned_confs[1,])) | |
| 122 | 122 | |
| 123 | 123 | ## Finally, we can compare using a boxplot the quality obtained with the |
| 124 | 124 | ## default parametrization of SANN and the quality obtained with the |
| 129 | 129 | } |
| 130 | 130 | \references{ |
| 131 | 131 | Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, |
| 132 | Thomas Stützle, and Mauro Birattari. The irace package: Iterated | |
| 133 | Racing for Automatic Algorithm Configuration. \emph{Operations Research | |
| 134 | Perspectives}, 2016. \doi{10.1016/j.orp.2016.09.002} | |
| 135 | ||
| 136 | Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro | |
| 137 | Birattari. \emph{The irace package, Iterated Race for Automatic | |
| 138 | Algorithm Configuration}. Technical Report TR/IRIDIA/2011-004, IRIDIA, | |
| 139 | Université Libre de Bruxelles, Belgium, 2011. | |
| 140 | ||
| 141 | Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of | |
| 142 | Multi-Objective Ant Colony Optimization Algorithms. \emph{IEEE Transactions | |
| 143 | on Evolutionary Computation}, 2012. | |
| 132 | Thomas Stützle, and Mauro Birattari. The irace package: Iterated | |
| 133 | Racing for Automatic Algorithm Configuration. \emph{Operations Research | |
| 134 | Perspectives}, 2016. \doi{10.1016/j.orp.2016.09.002} | |
| 135 | ||
| 136 | Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro | |
| 137 | Birattari. \emph{The irace package, Iterated Race for Automatic | |
| 138 | Algorithm Configuration}. Technical Report TR/IRIDIA/2011-004, IRIDIA, | |
| 139 | Université Libre de Bruxelles, Belgium, 2011. | |
| 140 | ||
| 141 | Manuel López-Ibáñez and Thomas Stützle. The Automatic Design of | |
| 142 | Multi-Objective Ant Colony Optimization Algorithms. \emph{IEEE Transactions | |
| 143 | on Evolutionary Computation}, 2012. | |
| 144 | 144 | } |
| 145 | 145 | \seealso{ |
| 146 | 146 | \code{\link{irace.main}} to start \pkg{irace} with a given scenario. |
| 147 | 147 | } |
| 148 | 148 | \author{ |
| 149 | 149 | Maintainers: Manuel López-Ibáñez and Leslie Pérez Cáceres |
| 150 | \email{irace-package@googlegroups.com} | |
| 150 | \email{irace-package@googlegroups.com} | |
| 151 | 151 | } |
| 152 | 152 | \keyword{automatic} |
| 153 | 153 | \keyword{configuration} |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | 12 | |
| 13 | 13 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter |
| 14 | 14 | space definition. The data structure has to similar to the one returned by the |
| 21 | 21 | The data frame has the following columns: |
| 22 | 22 | \itemize{ |
| 23 | 23 | \item \code{.ID.} : Internal id of the candidate configuration. |
| 24 | \item \code{Parameter names} : One column per parameter name in \code{parameters}. | |
| 24 | \item \verb{Parameter names} : One column per parameter name in \code{parameters}. | |
| 25 | 25 | \item \code{.PARENT.} : Internal id of the parent candidate configuration. |
| 26 | 26 | } |
| 27 | 27 | |
| 83 | 83 | } |
| 84 | 84 | } |
| 85 | 85 | \description{ |
| 86 | \code{irace} implements iterated Race. It receives some parameters to be tuned | |
| 87 | and returns the best configurations found, namely, the elite configurations | |
| 88 | obtained from the last iterations (and sorted by rank). | |
| 86 | \code{irace} implements iterated Race. It receives some parameters to be tuned | |
| 87 | and returns the best configurations found, namely, the elite configurations | |
| 88 | obtained from the last iterations (and sorted by rank). | |
| 89 | 89 | } |
| 90 | 90 | \details{ |
| 91 | The function \code{irace} executes the tuning procedure using | |
| 92 | the information provided in \code{scenario} and \code{parameters}. Initially it checks | |
| 93 | the correctness of \code{scenario} and recovers a previous execution if | |
| 94 | \code{scenario$recoveryFile} is set. A R data file log of the execution is created | |
| 95 | in \code{scenario$logFile}. | |
| 91 | The function \code{irace} executes the tuning procedure using | |
| 92 | the information provided in \code{scenario} and \code{parameters}. Initially it checks | |
| 93 | the correctness of \code{scenario} and recovers a previous execution if | |
| 94 | \code{scenario$recoveryFile} is set. A R data file log of the execution is created | |
| 95 | in \code{scenario$logFile}. | |
| 96 | 96 | } |
| 97 | 97 | \examples{ |
| 98 | 98 | \dontrun{ |
| 99 | 99 | parameters <- readParameters("parameters.txt") |
| 100 | scenario <- readScenario(filename = "scenario.txt", | |
| 101 | scenario = defaultScenario()) | |
| 100 | scenario <- readScenario(filename = "scenario.txt") | |
| 102 | 101 | irace(scenario = scenario, parameters = parameters) |
| 103 | 102 | } |
| 104 | 103 | |
| 105 | 104 | } |
| 106 | 105 | \seealso{ |
| 107 | 106 | \describe{ |
| 108 | \item{\code{\link{irace.main}}}{a higher-level command-line interface to \code{irace}.} | |
| 109 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 110 | \item{\code{\link{readParameters}}}{read the target algorithm parameters from a file.} | |
| 111 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 112 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 107 | \item{\code{\link[=irace.main]{irace.main()}}}{a higher-level interface to \code{irace}.} | |
| 108 | \item{\code{\link[=irace.cmdline]{irace.cmdline()}}}{a command-line interface to \code{irace}.} | |
| 109 | \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} | |
| 110 | \item{\code{\link[=readParameters]{readParameters()}}}{read the target algorithm parameters from a file.} | |
| 111 | \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} | |
| 112 | \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} | |
| 113 | 113 | } |
| 114 | 114 | } |
| 115 | 115 | \author{ |
| 116 | 116 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 117 | 117 | } |
| 118 | \concept{running} | |
| 1 | 1 | % Please edit documentation in R/main.R |
| 2 | 2 | \name{irace.cmdline} |
| 3 | 3 | \alias{irace.cmdline} |
| 4 | \title{irace.cmdline} | |
| 4 | \title{Launch \code{irace} with command-line options.} | |
| 5 | 5 | \usage{ |
| 6 | 6 | irace.cmdline(argv = commandArgs(trailingOnly = TRUE)) |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | \item{argv}{(\code{character()}) \cr The arguments | |
| 10 | provided on the R command line as a character vector, e.g., | |
| 9 | \item{argv}{(\code{character()}) \cr The arguments | |
| 10 | provided on the R command line as a character vector, e.g., | |
| 11 | 11 | \code{c("--scenario", "scenario.txt", "-p", "parameters.txt")}. |
| 12 | Using the default value (not providing the parameter) is the | |
| 12 | Using the default value (not providing the parameter) is the | |
| 13 | 13 | easiest way to call \code{irace.cmdline}.} |
| 14 | 14 | } |
| 15 | 15 | \value{ |
| 19 | 19 | The data frame has the following columns: |
| 20 | 20 | \itemize{ |
| 21 | 21 | \item \code{.ID.} : Internal id of the candidate configuration. |
| 22 | \item \code{Parameter names} : One column per parameter name in \code{parameters}. | |
| 22 | \item \verb{Parameter names} : One column per parameter name in \code{parameters}. | |
| 23 | 23 | \item \code{.PARENT.} : Internal id of the parent candidate configuration. |
| 24 | 24 | } |
| 25 | 25 | |
| 81 | 81 | } |
| 82 | 82 | } |
| 83 | 83 | \description{ |
| 84 | \code{irace.cmdline} starts \pkg{irace} using the parameters | |
| 85 | of the command line used to invoke R. | |
| 84 | Calls \code{\link[=irace.main]{irace.main()}} using command-line options, maybe parsed from the | |
| 85 | command line used to invoke R. | |
| 86 | 86 | } |
| 87 | 87 | \details{ |
| 88 | 88 | The function reads the parameters given on the command line |
| 89 | 89 | used to invoke R, finds the name of the scenario file, |
| 90 | initializes the scenario from the file (with the function | |
| 91 | \code{\link{readScenario}}) and possibly from parameters passed on | |
| 92 | the command line. It finally starts \pkg{irace} by calling | |
| 93 | \code{\link{irace.main}}. | |
| 90 | initializes the scenario from the file (with the function | |
| 91 | \code{\link{readScenario}}) and possibly from parameters passed in | |
| 92 | the command line. It finally starts \pkg{irace} by calling | |
| 93 | \code{\link{irace.main}}. | |
| 94 | ||
| 95 | List of command-line options:\preformatted{-h,--help Show this help. | |
| 96 | -v,--version Show irace package version. | |
| 97 | -c,--check Check scenario. | |
| 98 | -i,--init Initialize the working directory with template config | |
| 99 | files. | |
| 100 | --only-test Only test the configurations given in the file passed | |
| 101 | as argument. | |
| 102 | -s,--scenario File that describes the configuration scenario setup | |
| 103 | and other irace settings. Default: ./scenario.txt. | |
| 104 | --exec-dir Directory where the programs will be run. Default: ./. | |
| 105 | -p,--parameter-file File that contains the description of the parameters | |
| 106 | of the target algorithm. Default: ./parameters.txt. | |
| 107 | --forbidden-file File that contains a list of logical expressions that | |
| 108 | cannot be TRUE for any evaluated configuration. If | |
| 109 | empty or NULL, do not use forbidden expressions. | |
| 110 | --configurations-file File that contains a table of initial configurations. | |
| 111 | If empty or NULL, all initial configurations are | |
| 112 | randomly generated. | |
| 113 | -l,--log-file File to save tuning results as an R dataset, either | |
| 114 | absolute path or relative to execDir. Default: | |
| 115 | ./irace.Rdata. | |
| 116 | --recovery-file Previously saved log file to recover the execution of | |
| 117 | irace, either absolute path or relative to the current | |
| 118 | directory. If empty or NULL, recovery is not | |
| 119 | performed. | |
| 120 | --train-instances-dir Directory where training instances are located; | |
| 121 | either absolute path or relative to current directory. | |
| 122 | If no trainInstancesFiles is provided, all the files | |
| 123 | in trainInstancesDir will be listed as instances. | |
| 124 | Default: ./Instances. | |
| 125 | --train-instances-file File that contains a list of training instances and | |
| 126 | optionally additional parameters for them. If | |
| 127 | trainInstancesDir is provided, irace will search for | |
| 128 | the files in this folder. | |
| 129 | --sample-instances Randomly sample the training instances or use them in | |
| 130 | the order given. Default: 1. | |
| 131 | --test-instances-dir Directory where testing instances are located, either | |
| 132 | absolute or relative to current directory. | |
| 133 | --test-instances-file File containing a list of test instances and | |
| 134 | optionally additional parameters for them. | |
| 135 | --test-num-elites Number of elite configurations returned by irace that | |
| 136 | will be tested if test instances are provided. | |
| 137 | Default: 1. | |
| 138 | --test-iteration-elites Enable/disable testing the elite configurations | |
| 139 | found at each iteration. Default: 0. | |
| 140 | --test-type Statistical test used for elimination. The default | |
| 141 | value selects t-test if capping is enabled or F-test, | |
| 142 | otherwise. Valid values are: F-test (Friedman test), | |
| 143 | t-test (pairwise t-tests with no correction), | |
| 144 | t-test-bonferroni (t-test with Bonferroni's correction | |
| 145 | for multiple comparisons), t-test-holm (t-test with | |
| 146 | Holm's correction for multiple comparisons). | |
| 147 | --first-test Number of instances evaluated before the first | |
| 148 | elimination test. It must be a multiple of eachTest. | |
| 149 | Default: 5. | |
| 150 | --each-test Number of instances evaluated between elimination | |
| 151 | tests. Default: 1. | |
| 152 | --target-runner Executable called for each configuration that executes | |
| 153 | the target algorithm to be tuned. See the templates | |
| 154 | and examples provided. Default: ./target-runner. | |
| 155 | --target-runner-launcher Executable that will be used to launch the target | |
| 156 | runner, when targetRunner cannot be executed directly | |
| 157 | (.e.g, a Python script in Windows). | |
| 158 | --target-runner-args Command-line arguments provided to | |
| 159 | targetRunnerLauncher. The substrings \\\{targetRunner\\\} | |
| 160 | and \\\{targetRunnerArgs\\\} will be replaced by the value | |
| 161 | of the option targetRunner and by the arguments | |
| 162 | usually passed when calling targetRunner, | |
| 163 | respectively. Example: "-m \{targetRunner --args | |
| 164 | \{targetRunnerArgs\}"\}. Default: \{targetRunner\} | |
| 165 | \{targetRunnerArgs\}. | |
| 166 | --target-runner-retries Number of times to retry a call to targetRunner if | |
| 167 | the call failed. Default: 0. | |
| 168 | --target-evaluator Optional script or R function that provides a numeric | |
| 169 | value for each configuration. See | |
| 170 | templates/target-evaluator.tmpl | |
| 171 | --deterministic If the target algorithm is deterministic, | |
| 172 | configurations will be evaluated only once per | |
| 173 | instance. Default: 0. | |
| 174 | --max-experiments Maximum number of runs (invocations of targetRunner) | |
| 175 | that will be performed. It determines the maximum | |
| 176 | budget of experiments for the tuning. Default: 0. | |
| 177 | --max-time Maximum total execution time in seconds for the | |
| 178 | executions of targetRunner. targetRunner must return | |
| 179 | two values: cost and time. Default: 0. | |
| 180 | --budget-estimation Fraction (smaller than 1) of the budget used to | |
| 181 | estimate the mean computation time of a configuration. | |
| 182 | Only used when maxTime > 0 Default: 0.02. | |
| 183 | --min-measurable-time Minimum time unit that is still (significantly) | |
| 184 | measureable. Default: 0.01. | |
| 185 | --parallel Number of calls to targetRunner to execute in | |
| 186 | parallel. Values 0 or 1 mean no parallelization. | |
| 187 | Default: 0. | |
| 188 | --load-balancing Enable/disable load-balancing when executing | |
| 189 | experiments in parallel. Load-balancing makes better | |
| 190 | use of computing resources, but increases | |
| 191 | communication overhead. If this overhead is large, | |
| 192 | disabling load-balancing may be faster. Default: 1. | |
| 193 | --mpi Enable/disable MPI. Use Rmpi to execute targetRunner | |
| 194 | in parallel (parameter parallel is the number of | |
| 195 | slaves). Default: 0. | |
| 196 | --batchmode Specify how irace waits for jobs to finish when | |
| 197 | targetRunner submits jobs to a batch cluster: sge, | |
| 198 | pbs, torque, slurm or htcondor. targetRunner must | |
| 199 | submit jobs to the cluster using, for example, qsub. | |
| 200 | Default: 0. | |
| 201 | --digits Maximum number of decimal places that are significant | |
| 202 | for numerical (real) parameters. Default: 4. | |
| 203 | -q,--quiet Reduce the output generated by irace to a minimum. | |
| 204 | Default: 0. | |
| 205 | --debug-level Debug level of the output of irace. Set this to 0 to | |
| 206 | silence all debug messages. Higher values provide more | |
| 207 | verbose debug messages. Default: 0. | |
| 208 | --seed Seed of the random number generator (by default, | |
| 209 | generate a random seed). | |
| 210 | --soft-restart Enable/disable the soft restart strategy that avoids | |
| 211 | premature convergence of the probabilistic model. | |
| 212 | Default: 1. | |
| 213 | --soft-restart-threshold Soft restart threshold value for numerical | |
| 214 | parameters. If NA, NULL or "", it is computed as | |
| 215 | 10^-digits. | |
| 216 | -e,--elitist Enable/disable elitist irace. Default: 1. | |
| 217 | --elitist-new-instances Number of instances added to the execution list | |
| 218 | before previous instances in elitist irace. Default: | |
| 219 | 1. | |
| 220 | --elitist-limit In elitist irace, maximum number per race of | |
| 221 | elimination tests that do not eliminate a | |
| 222 | configuration. Use 0 for no limit. Default: 2. | |
| 223 | --capping Enable the use of adaptive capping, a technique | |
| 224 | designed for minimizing the computation time of | |
| 225 | configurations. This is only available when elitist is | |
| 226 | active. Default: 0. | |
| 227 | --capping-type Measure used to obtain the execution bound from the | |
| 228 | performance of the elite configurations: median, mean, | |
| 229 | worst, best. Default: median. | |
| 230 | --bound-type Method to calculate the mean performance of elite | |
| 231 | configurations: candidate or instance. Default: | |
| 232 | candidate. | |
| 233 | --bound-max Maximum execution bound for targetRunner. It must be | |
| 234 | specified when capping is enabled. Default: 0. | |
| 235 | --bound-digits Precision used for calculating the execution time. It | |
| 236 | must be specified when capping is enabled. Default: 0. | |
| 237 | --bound-par Penalization constant for timed out executions | |
| 238 | (executions that reach boundMax execution time). | |
| 239 | Default: 1. | |
| 240 | --bound-as-timeout Replace the configuration cost of bounded executions | |
| 241 | with boundMax. Default: 1. | |
| 242 | --postselection Percentage of the configuration budget used to perform | |
| 243 | a postselection race of the best configurations of | |
| 244 | each iteration after the execution of irace. Default: | |
| 245 | 0. | |
| 246 | --aclib Enable/disable AClib mode. This option enables | |
| 247 | compatibility with GenericWrapper4AC as targetRunner | |
| 248 | script. Default: 0. | |
| 249 | --iterations Maximum number of iterations. Default: 0. | |
| 250 | --experiments-per-iteration Number of runs of the target algorithm per | |
| 251 | iteration. Default: 0. | |
| 252 | --min-survival Minimum number of configurations needed to continue | |
| 253 | the execution of each race (iteration). Default: 0. | |
| 254 | --num-configurations Number of configurations to be sampled and evaluated | |
| 255 | at each iteration. Default: 0. | |
| 256 | --mu Parameter used to define the number of configurations | |
| 257 | sampled and evaluated at each iteration. Default: 5. | |
| 258 | --confidence Confidence level for the elimination test. Default: | |
| 259 | 0.95. | |
| 260 | } | |
| 261 | } | |
| 262 | \examples{ | |
| 263 | irace.cmdline("--version") | |
| 94 | 264 | } |
| 95 | 265 | \seealso{ |
| 96 | \code{\link{irace.main}} to start \pkg{irace} with a given scenario. | |
| 266 | \code{\link[=irace.main]{irace.main()}} to start \pkg{irace} with a given scenario. | |
| 97 | 267 | } |
| 98 | 268 | \author{ |
| 99 | 269 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 100 | 270 | } |
| 271 | \concept{running} | |
| 3 | 3 | \name{irace.license} |
| 4 | 4 | \alias{irace.license} |
| 5 | 5 | \title{irace.license} |
| 6 | \format{An object of class \code{character} of length 1.} | |
| 6 | \format{ | |
| 7 | An object of class \code{character} of length 1. | |
| 8 | } | |
| 7 | 9 | \usage{ |
| 8 | 10 | irace.license |
| 9 | 11 | } |
| 1 | 1 | % Please edit documentation in R/main.R |
| 2 | 2 | \name{irace.main} |
| 3 | 3 | \alias{irace.main} |
| 4 | \title{irace.main} | |
| 4 | \title{Higher-level interface to launch irace.} | |
| 5 | 5 | \usage{ |
| 6 | irace.main(scenario = defaultScenario(), output.width = 9999L) | |
| 6 | irace.main(scenario, output.width = 9999L) | |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | 12 | |
| 13 | \item{output.width}{(\code{integer(1)}) The width that must be used for the screen | |
| 13 | \item{output.width}{(\code{integer(1)}) The width used for the screen | |
| 14 | 14 | output.} |
| 15 | 15 | } |
| 16 | 16 | \value{ |
| 20 | 20 | The data frame has the following columns: |
| 21 | 21 | \itemize{ |
| 22 | 22 | \item \code{.ID.} : Internal id of the candidate configuration. |
| 23 | \item \code{Parameter names} : One column per parameter name in \code{parameters}. | |
| 23 | \item \verb{Parameter names} : One column per parameter name in \code{parameters}. | |
| 24 | 24 | \item \code{.PARENT.} : Internal id of the parent candidate configuration. |
| 25 | 25 | } |
| 26 | 26 | |
| 82 | 82 | } |
| 83 | 83 | } |
| 84 | 84 | \description{ |
| 85 | \code{irace.main} is a higher-level interface to invoke \code{\link{irace}}. | |
| 85 | Higher-level interface to launch irace. | |
| 86 | 86 | } |
| 87 | 87 | \details{ |
| 88 | The function \code{irace.main} checks the correctness of the | |
| 89 | scenario, prints it, reads the parameter space from | |
| 90 | \code{scenario$parameterFile}, invokes \code{\link{irace}} and | |
| 91 | prints its results in various formatted ways. If you want a | |
| 92 | lower-level interface, please see function \code{\link{irace}}. | |
| 88 | This function checks the correctness of the scenario, reads the | |
| 89 | parameter space from \code{scenario$parameterFile}, invokes \code{\link[=irace]{irace()}}, | |
| 90 | prints its results in various formatted ways, (optionally) calls | |
| 91 | \code{\link[=psRace]{psRace()}} and, finally, evaluates the best configurations on the test | |
| 92 | instances (if provided). If you want a lower-level interface that just | |
| 93 | runs irace, please see function \code{\link[=irace]{irace()}}. | |
| 93 | 94 | } |
| 94 | 95 | \seealso{ |
| 95 | \code{\link{irace.cmdline}} a higher-level command-line interface to | |
| 96 | \code{irace.main}. | |
| 97 | \code{\link{readScenario}} to read the scenario setup from a file. | |
| 98 | \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. | |
| 96 | \code{\link[=irace.cmdline]{irace.cmdline()}} a higher-level command-line interface to | |
| 97 | \code{\link[=irace]{irace()}} | |
| 98 | \code{\link[=readScenario]{readScenario()}} to read the scenario setup from a file. | |
| 99 | \code{\link[=defaultScenario]{defaultScenario()}} to provide a default scenario for \pkg{irace}. | |
| 99 | 100 | } |
| 100 | 101 | \author{ |
| 101 | 102 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 102 | 103 | } |
| 104 | \concept{running} | |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/main.R | |
| 2 | \name{irace.usage} | |
| 3 | \alias{irace.usage} | |
| 4 | \title{irace.usage} | |
| 5 | \usage{ | |
| 6 | irace.usage() | |
| 7 | } | |
| 8 | \description{ | |
| 9 | \code{irace.usage} This function prints all command-line options of \pkg{irace}, | |
| 10 | with the corresponding switches and a short description. | |
| 11 | } | |
| 12 | \author{ | |
| 13 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste | |
| 14 | } |
| 3 | 3 | \name{irace.version} |
| 4 | 4 | \alias{irace.version} |
| 5 | 5 | \title{irace.version} |
| 6 | \format{An object of class \code{character} of length 1.} | |
| 6 | \format{ | |
| 7 | An object of class \code{character} of length 1. | |
| 8 | } | |
| 7 | 9 | \usage{ |
| 8 | 10 | irace.version |
| 9 | 11 | } |
| 10 | 12 | \description{ |
| 11 | A character string containing the version of \pkg{irace}. | |
| 13 | A character string containing the version of \code{irace}. | |
| 12 | 14 | } |
| 13 | 15 | \keyword{datasets} |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/irace2pyimp.R | |
| 2 | \name{irace2pyimp} | |
| 3 | \alias{irace2pyimp} | |
| 4 | \title{Convert an \code{irace.Rdata} file into the format supported by PyImp} | |
| 5 | \usage{ | |
| 6 | irace2pyimp(file = "./irace.Rdata", normalise = "none", | |
| 7 | outdir = "./pyimp-input/", instanceFeatureFile = NA, | |
| 8 | filterConditions = NA, defaultConfigurationID = 1, | |
| 9 | ignoreUnsupported = 0) | |
| 10 | } | |
| 11 | \arguments{ | |
| 12 | \item{file}{(\code{character(1)}) \cr Filename of the \code{.Rdata} file generated by | |
| 13 | irace after a tuning run is finished.} | |
| 14 | ||
| 15 | \item{normalise}{(\code{none} | \code{instance} | \code{feature}) \cr Normalise the cost metric values into \code{[0, 1]} range before converting to PyImp format. Possible values are:\cr | |
| 16 | * \code{none} (default): no normalisation. \cr | |
| 17 | * \code{instance} : normalisation is done per instance. \cr | |
| 18 | * \code{feature} : normalisation is based on features, i.e., instances with the same feature-vector values are grouped together and the normalised cost is calculated per group.} | |
| 19 | ||
| 20 | \item{outdir}{(\code{character(1)}) \cr Directory where all generated files are stored.} | |
| 21 | ||
| 22 | \item{instanceFeatureFile}{(\code{character(1)}) \cr A \code{.csv} file containing instance features (one line per instance, | |
| 23 | sorted in the same order as the list of instances input to irace). The first line contains feature names.} | |
| 24 | ||
| 25 | \item{filterConditions}{Only extract data that satisfies the given conditions. The conditions are in R expression format.} | |
| 26 | ||
| 27 | \item{defaultConfigurationID}{Index of default configuration (starting from 1), used by ablation analysis.} | |
| 28 | ||
| 29 | \item{ignoreUnsupported}{Forbidden configurations and repairConfiguration are not supported by the script. Set ignoreUnsupported=1 to ignore them and proceed with your own risk. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis's path.} | |
| 30 | } | |
| 31 | \description{ | |
| 32 | This function converts an \code{irace.Rdata} file generated by irace into the | |
| 33 | input format supported by the parameter importance analysis tool \code{PyImp} | |
| 34 | (https://github.com/automl/ParameterImportance). | |
| 35 | } | |
| 36 | \details{ | |
| 37 | The generated files include: | |
| 38 | \itemize{ | |
| 39 | \item \code{params.pcs} : a text file containing the parameter space definition. | |
| 40 | \item \code{runhistory.json} : a JSON file containing the list of algorithm configurations evaluated during the tuning and the performance data obtained. | |
| 41 | \item \code{traj_aclib2.json} : a JSON file containing the best configurations after each iteration of irace. The last configuration will be used as the target configuration in ablation analysis. | |
| 42 | \item \code{scenario.txt} : a text file containing the definition of the tuning scenario. | |
| 43 | \item \code{instances.txt} : a text file containing the list of instances. | |
| 44 | \item \code{features.csv} : a text file containing instance features. If no instance features are provided, the index of each instance will be used as a feature. | |
| 45 | } | |
| 46 | } | |
| 47 | \examples{ | |
| 48 | \dontrun{ | |
| 49 | irace2pyimp(file='irace.Rdata', outdir='pyimp-run') | |
| 50 | irace2pyimp(file='irace.Rdata', normalise='feature', | |
| 51 | instanceFeatureFile='feature.csv', filterConditions="algorithm!='mas'") | |
| 52 | } | |
| 53 | cat("See more examples in '", | |
| 54 | file.path(system.file(package="irace"), "examples/irace2pyimp/acotsp/run.sh"), | |
| 55 | "' and in '", | |
| 56 | file.path(system.file(package="irace"), "examples/irace2pyimp/002-TemplateDesign/run.sh"), | |
| 57 | "'\\n") | |
| 58 | ||
| 59 | } | |
| 60 | \author{ | |
| 61 | Nguyen Dang and Manuel López-Ibáñez | |
| 62 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/irace2pyimp.R | |
| 2 | \name{irace2pyimp_cmdline} | |
| 3 | \alias{irace2pyimp_cmdline} | |
| 4 | \title{Command-line interface to irace2pyimp} | |
| 5 | \usage{ | |
| 6 | irace2pyimp_cmdline(argv = commandArgs(trailingOnly = TRUE)) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{argv}{(\code{character()}) \cr Command-line arguments.} | |
| 10 | } | |
| 11 | \value{ | |
| 12 | None. | |
| 13 | } | |
| 14 | \description{ | |
| 15 | This is a command-line interface for calling the \code{\link{irace2pyimp}} function, | |
| 16 | which converts an \code{irace.Rdata} file into the input format supported by the | |
| 17 | parameter importance analysis tool \code{PyImp} | |
| 18 | (https://github.com/automl/ParameterImportance). \cr The best way to use | |
| 19 | this command line interface is to run the script \code{irace-to-pyimp}. \cr To | |
| 20 | see usage of the script, run: irace-to-pyimp --help | |
| 21 | } | |
| 22 | \examples{ | |
| 23 | ||
| 24 | irace2pyimp_cmdline("--help") | |
| 25 | ||
| 26 | } | |
| 27 | \seealso{ | |
| 28 | \code{\link{irace2pyimp}} | |
| 29 | } | |
| 30 | \author{ | |
| 31 | Nguyen Dang and Manuel López-Ibáñez | |
| 32 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/parameterAnalysis.R | |
| 2 | \name{parallelCoordinatesPlot} | |
| 3 | \alias{parallelCoordinatesPlot} | |
| 4 | \title{parallelCoordinatesPlot} | |
| 5 | \usage{ | |
| 6 | parallelCoordinatesPlot(configurations, parameters, | |
| 7 | param_names = parameters$names, hierarchy = TRUE, filename = NULL, | |
| 8 | pdf.width = 14, mar = c(8, 1, 4, 1)) | |
| 9 | } | |
| 10 | \arguments{ | |
| 11 | \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the | |
| 12 | target algorithm (one per row).} | |
| 13 | ||
| 14 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter | |
| 15 | space definition. The data structure has to similar to the one returned by the | |
| 16 | function \code{\link{readParameters}}.} | |
| 17 | ||
| 18 | \item{param_names}{Parameters names that should be included. Default: parameters$names.} | |
| 19 | ||
| 20 | \item{hierarchy}{If \code{TRUE} conditional parameters will be displayed in a different | |
| 21 | plot. Default \code{TRUE}.} | |
| 22 | ||
| 23 | \item{filename}{Filename prefix to generate the plots. If \code{NULL} the plot | |
| 24 | displayed but not saved.} | |
| 25 | ||
| 26 | \item{pdf.width}{Width for the pdf file generated.} | |
| 27 | ||
| 28 | \item{mar}{Margin to use for the plot. See \code{\link{par}}.} | |
| 29 | } | |
| 30 | \value{ | |
| 31 | A set of parallel coordinates plots showing the parameters values. | |
| 32 | If a filename is provided this plots are saved in one or more files. | |
| 33 | } | |
| 34 | \description{ | |
| 35 | \code{parallelCoordinatesPlot} plots a set of parameter configurations in | |
| 36 | parallel coordinates. | |
| 37 | } | |
| 38 | \examples{ | |
| 39 | \donttest{ | |
| 40 | ## To use data obtained by irace | |
| 41 | # First, load the data produced by irace. | |
| 42 | irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") | |
| 43 | load(irace.logfile) | |
| 44 | attach(iraceResults) | |
| 45 | parallelCoordinatesPlot(allConfigurations, parameters, hierarchy = FALSE) | |
| 46 | } | |
| 47 | ||
| 48 | } | |
| 49 | \seealso{ | |
| 50 | \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. | |
| 51 | \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from | |
| 52 | a configurations file. | |
| 53 | } | |
| 54 | \author{ | |
| 55 | Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 56 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/parameterAnalysis.R | |
| 2 | \name{parameterFrequency} | |
| 3 | \alias{parameterFrequency} | |
| 4 | \title{Plot of histogram of parameter values} | |
| 5 | \usage{ | |
| 6 | parameterFrequency(configurations, parameters, rows = 4, cols = 3, | |
| 7 | filename = NULL, pdf.width = 12, col = "gray") | |
| 8 | } | |
| 9 | \arguments{ | |
| 10 | \item{configurations}{(\code{data.frame}) \cr Parameter configurations of the | |
| 11 | target algorithm (one per row).} | |
| 12 | ||
| 13 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter | |
| 14 | space definition. The data structure has to similar to the one returned by the | |
| 15 | function \code{\link{readParameters}}.} | |
| 16 | ||
| 17 | \item{rows}{Number of plots per column.} | |
| 18 | ||
| 19 | \item{cols}{Number of plots per row.} | |
| 20 | ||
| 21 | \item{filename}{Filename prefix to generate the plots. If \code{NULL} the plot | |
| 22 | displayed but not saved.} | |
| 23 | ||
| 24 | \item{pdf.width}{Width for the pdf file generated.} | |
| 25 | ||
| 26 | \item{col}{Color of the bar plot.} | |
| 27 | } | |
| 28 | \description{ | |
| 29 | \code{parameterFrequency} plots the frequency of the parameters values in a | |
| 30 | set of target algorithm configurations. It generates plots showing the | |
| 31 | frequency of parameter values for each parameter, with \code{rows} * | |
| 32 | \code{cols} parameters being shown per plot. If a filename is provided the | |
| 33 | plots are saved in one or more files. | |
| 34 | } | |
| 35 | \examples{ | |
| 36 | \donttest{ | |
| 37 | ## To use data obtained by irace | |
| 38 | ||
| 39 | # First, load the data produced by irace. | |
| 40 | irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") | |
| 41 | load(irace.logfile) | |
| 42 | attach(iraceResults) | |
| 43 | parameterFrequency(allConfigurations, parameters) | |
| 44 | } | |
| 45 | ||
| 46 | } | |
| 47 | \seealso{ | |
| 48 | \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. | |
| 49 | \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from | |
| 50 | a configurations file. | |
| 51 | } | |
| 52 | \author{ | |
| 53 | Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 54 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/path_rel2abs.R | |
| 2 | \name{path_rel2abs} | |
| 3 | \alias{path_rel2abs} | |
| 4 | \title{Converts a relative path to an absolute path. It tries really hard to create | |
| 5 | canonical paths.} | |
| 6 | \usage{ | |
| 7 | path_rel2abs(path, cwd = getwd()) | |
| 8 | } | |
| 9 | \arguments{ | |
| 10 | \item{path}{(\code{character(1)}) Character string representing a relative path.} | |
| 11 | ||
| 12 | \item{cwd}{(\code{character(1)}) Current working directory.} | |
| 13 | } | |
| 14 | \value{ | |
| 15 | (\code{character(1)}) Character string representing the absolute path | |
| 16 | } | |
| 17 | \description{ | |
| 18 | Converts a relative path to an absolute path. It tries really hard to create | |
| 19 | canonical paths. | |
| 20 | } | |
| 21 | \examples{ | |
| 22 | path_rel2abs("..") | |
| 23 | } |
| 3 | 3 | \alias{plotAblation} |
| 4 | 4 | \title{Create plot from an ablation log} |
| 5 | 5 | \usage{ |
| 6 | plotAblation(ab.log = NULL, abLogFile = NULL, pdf.file = NULL, | |
| 7 | pdf.width = 20, type = c("mean", "boxplot"), mar = par("mar"), | |
| 8 | ylab = "Mean configuration cost", ...) | |
| 6 | plotAblation( | |
| 7 | ablog, | |
| 8 | pdf.file = NULL, | |
| 9 | pdf.width = 20, | |
| 10 | type = c("mean", "boxplot"), | |
| 11 | mar = par("mar"), | |
| 12 | ylab = "Mean configuration cost", | |
| 13 | ylim = NULL, | |
| 14 | ... | |
| 15 | ) | |
| 9 | 16 | } |
| 10 | 17 | \arguments{ |
| 11 | \item{ab.log}{Ablation log returned by \code{\link{ablation}}.} | |
| 12 | ||
| 13 | \item{abLogFile}{Rdata file containing the ablation log.} | |
| 18 | \item{ablog}{(\code{list()}|\code{character(1)}) Ablation log object returned by \code{\link[=ablation]{ablation()}}. Alternatively, the path to an \code{.Rdata} file, e.g., \code{"log-ablation.Rdata"}, from which the object will be loaded.} | |
| 14 | 19 | |
| 15 | 20 | \item{pdf.file}{Output filename.} |
| 16 | 21 | |
| 17 | 22 | \item{pdf.width}{Width provided to create the pdf file.} |
| 18 | 23 | |
| 19 | \item{type}{Type of plots. Supported values are \code{"mean"} and | |
| 20 | \code{"boxplot"}.} | |
| 24 | \item{type}{Type of plot. Supported values are \code{"mean"} and \code{"boxplot"}.} | |
| 21 | 25 | |
| 22 | 26 | \item{mar}{Vector with the margins for the ablation plot.} |
| 23 | 27 | |
| 24 | 28 | \item{ylab}{Label of y-axis.} |
| 25 | 29 | |
| 30 | \item{ylim}{Numeric vector of length 2, giving the y coordinates ranges.} | |
| 31 | ||
| 26 | 32 | \item{...}{Further graphical parameters may also be supplied as |
| 27 | arguments. See \code{plot.default}.} | |
| 33 | arguments. See \code{\link[graphics:plot.default]{graphics::plot.default()}}.} | |
| 28 | 34 | } |
| 29 | 35 | \description{ |
| 30 | 36 | Create plot from an ablation log |
| 31 | 37 | } |
| 38 | \examples{ | |
| 39 | logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 40 | plotAblation(ablog = logfile) | |
| 41 | } | |
| 32 | 42 | \seealso{ |
| 33 | \code{\link{ablation}} | |
| 43 | \code{\link[=ablation]{ablation()}} | |
| 34 | 44 | } |
| 35 | 45 | \author{ |
| 36 | 46 | Leslie Pérez Cáceres and Manuel López-Ibáñez |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/readParameters.R | |
| 2 | \name{printParameters} | |
| 3 | \alias{printParameters} | |
| 4 | \title{Print parameter space in the textual format accepted by irace.} | |
| 5 | \usage{ | |
| 6 | printParameters(params, digits = 15L) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{params}{(\code{list()}) Parameter object stored in \code{irace.Rdata} or read with \code{irace::readParameters()}.} | |
| 10 | ||
| 11 | \item{digits}{(\code{integer()}) The desired number of digits after the decimal point for real-valued parameters. Default is 15, but it should be the value in \code{scenario$digits}.} | |
| 12 | } | |
| 13 | \description{ | |
| 14 | FIXME: Dependent parameter bounds are not supported yet. | |
| 15 | } | |
| 16 | \examples{ | |
| 17 | parameters.table <- ' | |
| 18 | # name switch type values [conditions (using R syntax)] | |
| 19 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 20 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 21 | ants "--ants " i,log (5, 100) | |
| 22 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 23 | nnls "--nnls " i (5, 50) | localsearch \%in\% c(1,2,3) | |
| 24 | ' | |
| 25 | parameters <- readParameters(text=parameters.table) | |
| 26 | printParameters(parameters) | |
| 27 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 10 | 10 | settings. The data structure has to be the one returned by the function |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 11 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 12 | 12 | } |
| 13 | 13 | \description{ |
| 14 | 14 | Prints the given scenario |
| 15 | 15 | } |
| 16 | 16 | \seealso{ |
| 17 | 17 | \describe{ |
| 18 | \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} | |
| 19 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 20 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 21 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 18 | \item{\code{\link[=readScenario]{readScenario()}}}{for reading a configuration scenario from a file.} | |
| 19 | \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} | |
| 20 | \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} | |
| 21 | \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} | |
| 22 | 22 | } |
| 23 | 23 | } |
| 24 | 24 | \author{ |
| 3 | 3 | \alias{psRace} |
| 4 | 4 | \title{psRace} |
| 5 | 5 | \usage{ |
| 6 | psRace(iraceLogFile = NULL, iraceResults = NULL, conf.ids = NULL, | |
| 7 | postselection = NULL, max.experiments = NULL, elites = FALSE, | |
| 8 | seed = 1234567) | |
| 6 | psRace( | |
| 7 | iraceLogFile = NULL, | |
| 8 | iraceResults = NULL, | |
| 9 | conf.ids = NULL, | |
| 10 | postselection = NULL, | |
| 11 | max.experiments = NULL, | |
| 12 | elites = FALSE, | |
| 13 | seed = 1234567 | |
| 14 | ) | |
| 9 | 15 | } |
| 10 | 16 | \arguments{ |
| 11 | \item{iraceLogFile}{NULL Log file created by \pkg{irace}, this file must contain the | |
| 17 | \item{iraceLogFile}{NULL Log file created by \pkg{irace}, this file must contain the | |
| 12 | 18 | \code{iraceResults} object.} |
| 13 | 19 | |
| 14 | 20 | \item{iraceResults}{NULL Object created by \pkg{irace} and saved in \code{scenario$logFile}.} |
| 19 | 25 | \item{postselection}{NULL Percentage of the maxExperiments provided in the scenario to be used in the race.} |
| 20 | 26 | |
| 21 | 27 | \item{max.experiments}{NULL Number of experiments available for the race. If NULL budget for the race is set |
| 22 | by the parameter scenario$postselection, which defines the percentage of the total budget of \pkg{irace} | |
| 28 | by the parameter scenario$postselection, which defines the percentage of the total budget of \pkg{irace} | |
| 23 | 29 | (iraceResults$scenario$maxExperiments or iraceResults$scenario$maxTime/iraceResults$state$timeEstimate) to use |
| 24 | for the postselection.} | |
| 30 | for the postselection.} | |
| 25 | 31 | |
| 26 | 32 | \item{elites}{FALSE Flag for selecting configurations. If FALSE, the best configurations of each |
| 27 | 33 | iteration are used for the race. If TRUE, the elite configurtions of each iteration are used for the race.} |
| 30 | 36 | } |
| 31 | 37 | \value{ |
| 32 | 38 | If iraceLogFile is NULL, it returns a list with the following elements: |
| 33 | \describe{ | |
| 34 | \item{configurations}{Configurations used in the race.} | |
| 35 | \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 36 | instances ids from iraceResults$scenario$instances, second column the seed assigned to the instance.} | |
| 37 | \item{maxExperiments}{Maximum number of experiments set for the race.} | |
| 38 | \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 39 | \item{elites}{Best configurations found in the experiments.} | |
| 40 | } | |
| 39 | \describe{ | |
| 40 | \item{configurations}{Configurations used in the race.} | |
| 41 | \item{instances}{A matrix with the instances used in the experiments. First column has the | |
| 42 | instances ids from iraceResults$scenario$instances, second column the seed assigned to the instance.} | |
| 43 | \item{maxExperiments}{Maximum number of experiments set for the race.} | |
| 44 | \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} | |
| 45 | \item{elites}{Best configurations found in the experiments.} | |
| 46 | } | |
| 41 | 47 | If \code{iraceLogFile} is provided this list object will be saved in \code{iraceResults$psrace.log}. |
| 42 | 48 | } |
| 43 | 49 | \description{ |
| 46 | 52 | \examples{ |
| 47 | 53 | \dontrun{ |
| 48 | 54 | # Execute the postselection automatically after irace |
| 55 | scenario <- readScenario(filename="scenario.txt") | |
| 49 | 56 | parameters <- readParameters("parameters.txt") |
| 50 | scenario <- readScenario(filename="scenario.txt", | |
| 51 | scenario=defaultScenario()) | |
| 52 | 57 | # Use 10\% of the total budget |
| 53 | 58 | scenario$postselection <- 0.1 |
| 54 | 59 | irace(scenario=scenario, parameters=parameters) |
| 55 | # Execute the postselection after the execution of \\pkg{irace}. | |
| 60 | # Execute the postselection after the execution of \pkg{irace}. | |
| 56 | 61 | psRace(iraceLogFile="irace.Rdata", max.experiments=120) |
| 57 | 62 | } |
| 58 | 63 | |
| 1 | 1 | % Please edit documentation in R/readConfiguration.R |
| 2 | 2 | \name{readConfigurationsFile} |
| 3 | 3 | \alias{readConfigurationsFile} |
| 4 | \title{readConfigurationsFile} | |
| 4 | \title{Read parameter configurations from a file} | |
| 5 | 5 | \usage{ |
| 6 | 6 | readConfigurationsFile(filename, parameters, debugLevel = 0, text) |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | \item{filename}{(\code{character(1)}) \cr Filename from which the configurations should be read.} | |
| 9 | \item{filename}{(\code{character(1)}) \cr Filename from which the configurations should be read. The contents should be readable by \code{read.table( , header=TRUE)}.} | |
| 10 | 10 | |
| 11 | 11 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter |
| 12 | 12 | space definition. The data structure has to similar to the one returned by the |
| 23 | 23 | the columns correspond to the parameter names in \code{parameters}. |
| 24 | 24 | } |
| 25 | 25 | \description{ |
| 26 | \code{readConfigurationsFile} reads a set of target algorithms configurations | |
| 27 | from a file and puts them in \pkg{irace} format. The configurations are checked | |
| 28 | to match the parameters description provided. | |
| 26 | Reads a set of target-algorithm configurations from a file and puts them in | |
| 27 | \pkg{irace} format. The configurations are checked to match the parameters | |
| 28 | description provided. | |
| 29 | } | |
| 30 | \details{ | |
| 31 | Example of an input file:\preformatted{# This is a comment line | |
| 32 | <param_name_1> <param_name_2> ... | |
| 33 | 0.5 "value_1" ... | |
| 34 | 1.0 "value_2" ... | |
| 35 | 1.2 "value_3" ... | |
| 36 | ... ... | |
| 37 | } | |
| 38 | ||
| 39 | The order of the columns does not necessarily have to be the same | |
| 40 | as in the file containing the definition of the parameters. | |
| 29 | 41 | } |
| 30 | 42 | \seealso{ |
| 31 | \code{\link{readParameters}} to obtain a valid parameter structure from a parameters list. | |
| 43 | \code{\link[=readParameters]{readParameters()}} to obtain a valid parameter structure from a parameters file. | |
| 32 | 44 | } |
| 33 | 45 | \author{ |
| 34 | 46 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 1 | 1 | % Please edit documentation in R/readParameters.R |
| 2 | 2 | \name{readParameters} |
| 3 | 3 | \alias{readParameters} |
| 4 | \title{readParameters} | |
| 4 | \title{Reads the parameters to be tuned by \pkg{irace} from a file or from a | |
| 5 | character string.} | |
| 5 | 6 | \usage{ |
| 6 | 7 | readParameters(file, digits = 4, debugLevel = 0, text) |
| 7 | 8 | } |
| 37 | 38 | \item{\code{nbParameters}}{An integer, the total number of parameters.} |
| 38 | 39 | \item{\code{nbFixed}}{An integer, the number of parameters with a fixed value.} |
| 39 | 40 | \item{\code{nbVariable}}{Number of variable (to be tuned) parameters.} |
| 41 | \item{\code{depends}}{List of character vectors, each vector specifies | |
| 42 | which parameters depend on this one.} | |
| 43 | \item{\code{isDependent}}{Logical vector that specifies which parameter has | |
| 44 | a dependent domain.} | |
| 40 | 45 | } |
| 41 | 46 | } |
| 42 | 47 | \description{ |
| 43 | \code{readParameters} reads the parameters to be tuned by | |
| 44 | \pkg{irace} from a file or directly from a character string. | |
| 48 | Reads the parameters to be tuned by \pkg{irace} from a file or from a | |
| 49 | character string. | |
| 45 | 50 | } |
| 46 | 51 | \details{ |
| 47 | 52 | Either \code{file} or \code{text} must be given. If \code{file} is given, the |
| 64 | 69 | \examples{ |
| 65 | 70 | ## Read the parameters directly from text |
| 66 | 71 | parameters.table <- ' |
| 67 | # name switch type values [conditions (using R syntax)] | |
| 68 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 69 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 70 | alpha "--alpha " r (0.00, 5.00) | |
| 71 | beta "--beta " r (0.00, 10.00) | |
| 72 | rho "--rho " r (0.01, 1.00) | |
| 73 | ants "--ants " i (5, 100) | |
| 74 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 75 | rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 76 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 77 | nnls "--nnls " i (5, 50) | localsearch \%in\% c(1,2,3) | |
| 78 | dlb "--dlb " c (0, 1) | localsearch \%in\% c(1,2,3) | |
| 72 | # name switch type values [conditions (using R syntax)] | |
| 73 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 74 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 75 | alpha "--alpha " r (0.00, 5.00) | |
| 76 | beta "--beta " r (0.00, 10.00) | |
| 77 | rho "--rho " r (0.01, 1.00) | |
| 78 | ants "--ants " i,log (5, 100) | |
| 79 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 80 | rasrank "--rasranks " i (1, "min(ants, 10)") | algorithm == "ras" | |
| 81 | elitistants "--elitistants " i (1, ants) | algorithm == "eas" | |
| 82 | nnls "--nnls " i (5, 50) | localsearch \%in\% c(1,2,3) | |
| 83 | dlb "--dlb " c (0, 1) | localsearch \%in\% c(1,2,3) | |
| 79 | 84 | ' |
| 80 | 85 | parameters <- readParameters(text=parameters.table) |
| 81 | 86 | str(parameters) |
| 1 | 1 | % Please edit documentation in R/readConfiguration.R |
| 2 | 2 | \name{readScenario} |
| 3 | 3 | \alias{readScenario} |
| 4 | \title{readScenario} | |
| 4 | \title{Reads from a file the scenario settings to be used by \pkg{irace}.} | |
| 5 | 5 | \usage{ |
| 6 | readScenario(filename = "", scenario = list()) | |
| 6 | readScenario(filename = "", scenario = list(), params_def = .irace.params.def) | |
| 7 | 7 | } |
| 8 | 8 | \arguments{ |
| 9 | 9 | \item{filename}{(\code{character(1)}) \cr Filename from which the scenario will |
| 10 | 10 | be read. If empty, the default \code{scenarioFile} is used. An example |
| 11 | scenario file is provided in \code{system.file(``package="irace",} | |
| 12 | \code{"templates/scenario.txt.tmpl")}.} | |
| 11 | scenario file is provided in \verb{system.file(``package="irace",} | |
| 12 | \verb{"templates/scenario.txt.tmpl")}.} | |
| 13 | 13 | |
| 14 | 14 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 15 | 15 | settings. The data structure has to be the one returned by the function |
| 16 | \code{\link{defaultScenario}} or \code{\link{readScenario}}. | |
| 16 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}. | |
| 17 | 17 | This is an initial scenario that is overwritten} |
| 18 | ||
| 19 | \item{params_def}{(\code{data.frame()}) \cr Definition of the options accepted by the scenario. This should only be modified by packages that wish to extend \pkg{irace}.} | |
| 18 | 20 | } |
| 19 | 21 | \value{ |
| 20 | 22 | The scenario list read from the file. The scenario settings not |
| 21 | 23 | present in the file are not present in the list, i.e., they are \code{NULL}. |
| 22 | 24 | } |
| 23 | 25 | \description{ |
| 24 | \code{readScenario} reads from a file the scenario settings to be used by | |
| 25 | \pkg{irace}.. | |
| 26 | Reads from a file the scenario settings to be used by \pkg{irace}. | |
| 26 | 27 | } |
| 27 | 28 | \seealso{ |
| 28 | 29 | \describe{ |
| 29 | \item{\code{\link{printScenario}}}{prints the given scenario.} | |
| 30 | \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} | |
| 31 | \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} | |
| 30 | \item{\code{\link[=printScenario]{printScenario()}}}{prints the given scenario.} | |
| 31 | \item{\code{\link[=defaultScenario]{defaultScenario()}}}{returns the default scenario settings of \pkg{irace}.} | |
| 32 | \item{\code{\link[=checkScenario]{checkScenario()}}}{to check that the scenario is valid.} | |
| 32 | 33 | } |
| 33 | 34 | } |
| 34 | 35 | \author{ |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/utils.R | |
| 2 | \name{read_logfile} | |
| 3 | \alias{read_logfile} | |
| 4 | \title{Read the log file produced by irace (\code{irace.Rdata}).} | |
| 5 | \usage{ | |
| 6 | read_logfile(filename, name = "iraceResults") | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{filename}{Filename that contains the log file saved by irace. Example: \code{irace.Rdata}.} | |
| 10 | ||
| 11 | \item{name}{Optional argument that allows overriding the default name of the object in the file.} | |
| 12 | } | |
| 13 | \value{ | |
| 14 | (\code{list()}) | |
| 15 | } | |
| 16 | \description{ | |
| 17 | Read the log file produced by irace (\code{irace.Rdata}). | |
| 18 | } | |
| 19 | \concept{analysis} |
| 1 | 1 | % Please edit documentation in R/readParameters.R |
| 2 | 2 | \name{read_pcs_file} |
| 3 | 3 | \alias{read_pcs_file} |
| 4 | \title{read_pcs_file} | |
| 4 | \title{Read parameters in PCS (AClib) format and write them in irace format.} | |
| 5 | 5 | \usage{ |
| 6 | 6 | read_pcs_file(file, digits = 4, debugLevel = 0, text) |
| 7 | 7 | } |
| 31 | 31 | whose name is \code{file}) in the expected form. See the documentation |
| 32 | 32 | for details. If none of these parameters is given, \pkg{irace} |
| 33 | 33 | will stop with an error. |
| 34 | ||
| 35 | \strong{FIXME:} Forbidden configurations, default configuration and transformations ("log") are currently ignored. See \url{https://github.com/MLopez-Ibanez/irace/issues/31} | |
| 34 | 36 | } |
| 35 | 37 | \examples{ |
| 36 | 38 | ## Read the parameters directly from text |
| 37 | pcs.table <- ' | |
| 38 | # name values [conditions (using R syntax)] | |
| 39 | pcs_table <- ' | |
| 40 | # name domain | |
| 39 | 41 | algorithm {as,mmas,eas,ras,acs}[as] |
| 40 | 42 | localsearch {0, 1, 2, 3}[0] |
| 41 | 43 | alpha [0.00, 5.00][1] |
| 54 | 56 | nnls | localsearch in {1,2,3} |
| 55 | 57 | dlb | localsearch in {1,2,3} |
| 56 | 58 | ' |
| 57 | parameters.table <- read_pcs_file(text=pcs.table) | |
| 58 | parameters <- readParameters(text=parameters.table) | |
| 59 | parameters_table <- read_pcs_file(text=pcs_table) | |
| 60 | cat(parameters_table) | |
| 61 | parameters <- readParameters(text=parameters_table) | |
| 59 | 62 | str(parameters) |
| 60 | 63 | |
| 64 | } | |
| 65 | \references{ | |
| 66 | Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. \strong{AClib: A Benchmark Library for Algorithm Configuration}. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, \emph{Learning and Intelligent Optimization, 8th International Conference, LION 8}, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, Heidelberg, 2014. | |
| 61 | 67 | } |
| 62 | 68 | \author{ |
| 63 | 69 | Manuel López-Ibáñez |
| 20 | 20 | of the configuration without data possibly useless to the user. |
| 21 | 21 | } |
| 22 | 22 | \seealso{ |
| 23 | \code{\link{configurations.print.command}} to print the configurations as command lines. | |
| 24 | \code{\link{configurations.print}} to print the configurations as a data frame. | |
| 23 | \code{\link[=configurations.print.command]{configurations.print.command()}} to print the configurations as command lines. | |
| 24 | \code{\link[=configurations.print]{configurations.print()}} to print the configurations as a data frame. | |
| 25 | 25 | } |
| 26 | 26 | \author{ |
| 27 | 27 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/utils.R | |
| 2 | \name{scenario.update.paths} | |
| 3 | \alias{scenario.update.paths} | |
| 4 | \title{Update filesystem paths of a scenario consistently.} | |
| 5 | \usage{ | |
| 6 | scenario.update.paths(scenario, from, to, fixed = TRUE) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} | |
| 10 | settings. The data structure has to be the one returned by the function | |
| 11 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 12 | ||
| 13 | \item{from}{character string containing a regular expression (or character | |
| 14 | string for \code{fixed = TRUE}) to be matched.} | |
| 15 | ||
| 16 | \item{to}{the replacement string.character string. For \code{fixed = FALSE} | |
| 17 | this can include backreferences \code{"\1"} to \code{"\9"} to | |
| 18 | parenthesized subexpressions of \code{from}.} | |
| 19 | ||
| 20 | \item{fixed}{logical. If \code{TRUE}, \code{from} is a string to be matched | |
| 21 | as is.} | |
| 22 | } | |
| 23 | \value{ | |
| 24 | The updated scenario | |
| 25 | } | |
| 26 | \description{ | |
| 27 | This function should be used to change the filesystem paths stored in a | |
| 28 | scenario object. Useful when moving a scenario from one computer to another. | |
| 29 | } | |
| 30 | \examples{ | |
| 31 | \dontrun{ | |
| 32 | scenario <- readScenario(filename = "scenario.txt") | |
| 33 | scenario <- scenario.update.paths(scenario, from = "/home/manuel/", to = "/home/leslie") | |
| 34 | } | |
| 35 | } | |
| 36 | \seealso{ | |
| 37 | \code{\link[base]{grep}} | |
| 38 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/utils.R | |
| 2 | \name{scenario_update_paths} | |
| 3 | \alias{scenario_update_paths} | |
| 4 | \alias{scenario.update.paths} | |
| 5 | \title{Update filesystem paths of a scenario consistently.} | |
| 6 | \usage{ | |
| 7 | scenario_update_paths(scenario, from, to, fixed = TRUE) | |
| 8 | ||
| 9 | scenario.update.paths(scenario, from, to, fixed = TRUE) | |
| 10 | } | |
| 11 | \arguments{ | |
| 12 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} | |
| 13 | settings. The data structure has to be the one returned by the function | |
| 14 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 15 | ||
| 16 | \item{from}{character string containing a regular expression (or character | |
| 17 | string for \code{fixed = TRUE}) to be matched.} | |
| 18 | ||
| 19 | \item{to}{the replacement string.character string. For \code{fixed = FALSE} | |
| 20 | this can include backreferences \code{"\\1"} to \code{"\\9"} to | |
| 21 | parenthesized subexpressions of \code{from}.} | |
| 22 | ||
| 23 | \item{fixed}{logical. If \code{TRUE}, \code{from} is a string to be matched | |
| 24 | as is.} | |
| 25 | } | |
| 26 | \value{ | |
| 27 | The updated scenario | |
| 28 | } | |
| 29 | \description{ | |
| 30 | This function should be used to change the filesystem paths stored in a | |
| 31 | scenario object. Useful when moving a scenario from one computer to another. | |
| 32 | } | |
| 33 | \examples{ | |
| 34 | \dontrun{ | |
| 35 | scenario <- readScenario(filename = "scenario.txt") | |
| 36 | scenario <- scenario_update_paths(scenario, from = "/home/manuel/", to = "/home/leslie") | |
| 37 | } | |
| 38 | } | |
| 39 | \seealso{ | |
| 40 | \code{\link[base:grep]{base::grep()}} | |
| 41 | } |
| 3 | 3 | \alias{target.evaluator.default} |
| 4 | 4 | \title{target.evaluator.default} |
| 5 | 5 | \usage{ |
| 6 | target.evaluator.default(experiment, num.configurations, all.conf.id, | |
| 7 | scenario, target.runner.call) | |
| 6 | target.evaluator.default( | |
| 7 | experiment, | |
| 8 | num.configurations, | |
| 9 | all.conf.id, | |
| 10 | scenario, | |
| 11 | target.runner.call | |
| 12 | ) | |
| 8 | 13 | } |
| 9 | 14 | \arguments{ |
| 10 | 15 | \item{experiment}{A list describing the experiment. It contains at least: |
| 27 | 32 | |
| 28 | 33 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 29 | 34 | settings. The data structure has to be the one returned by the function |
| 30 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 35 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 31 | 36 | |
| 32 | 37 | \item{target.runner.call}{String describing the call to \code{targetRunner} that |
| 33 | 38 | corresponds to this call to \code{targetEvaluator}. This is used for |
| 1 | 1 | % Please edit documentation in R/race-wrapper.R |
| 2 | 2 | \name{target.runner.default} |
| 3 | 3 | \alias{target.runner.default} |
| 4 | \title{target.runner.default} | |
| 4 | \title{Default \code{targetRunner} function.} | |
| 5 | 5 | \usage{ |
| 6 | 6 | target.runner.default(experiment, scenario) |
| 7 | 7 | } |
| 22 | 22 | |
| 23 | 23 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 24 | 24 | settings. The data structure has to be the one returned by the function |
| 25 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 25 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 26 | 26 | } |
| 27 | 27 | \value{ |
| 28 | 28 | If \code{targetEvaluator} is \code{NULL}, then the \code{targetRunner} |
| 44 | 44 | } |
| 45 | 45 | } |
| 46 | 46 | \description{ |
| 47 | \code{target.runner.default} is the default targetRunner function. | |
| 48 | You can use it as an advanced example of how to create your own targetRunner | |
| 49 | function. | |
| 47 | Use it as an advanced example of how to create your own \code{targetRunner} function. | |
| 50 | 48 | } |
| 51 | 49 | \author{ |
| 52 | 50 | Manuel López-Ibáñez and Jérémie Dubois-Lacoste |
| 1 | 1 | % Please edit documentation in R/testing.R |
| 2 | 2 | \name{testConfigurations} |
| 3 | 3 | \alias{testConfigurations} |
| 4 | \title{testConfigurations} | |
| 4 | \title{Execute the given configurations on the testing instances specified in the | |
| 5 | scenario} | |
| 5 | 6 | \usage{ |
| 6 | 7 | testConfigurations(configurations, scenario, parameters) |
| 7 | 8 | } |
| 11 | 12 | |
| 12 | 13 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} |
| 13 | 14 | settings. The data structure has to be the one returned by the function |
| 14 | \code{\link{defaultScenario}} or \code{\link{readScenario}}.} | |
| 15 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 15 | 16 | |
| 16 | 17 | \item{parameters}{(\code{list()}) \cr Data structure containing the parameter |
| 17 | 18 | space definition. The data structure has to similar to the one returned by the |
| 19 | 20 | } |
| 20 | 21 | \value{ |
| 21 | 22 | A list with the following elements: |
| 22 | \describe{ | |
| 23 | \item{\code{experiments}}{Experiments results.} | |
| 24 | \item{\code{seeds}}{Array of the instance seeds used in the experiments.} | |
| 25 | } | |
| 23 | \describe{ | |
| 24 | \item{\code{experiments}}{Experiments results.} | |
| 25 | \item{\code{seeds}}{Array of the instance seeds used in the experiments.} | |
| 26 | } | |
| 26 | 27 | } |
| 27 | 28 | \description{ |
| 28 | \code{testConfigurations} executes the given configurations on the | |
| 29 | testing instances specified in the scenario. | |
| 29 | Execute the given configurations on the testing instances specified in the | |
| 30 | scenario | |
| 30 | 31 | } |
| 31 | 32 | \details{ |
| 32 | A test instance set must be provided through \code{scenario$testInstances}. | |
| 33 | A test instance set must be provided through \code{scenario[["testInstances"]]}. | |
| 33 | 34 | } |
| 34 | 35 | \seealso{ |
| 35 | \code{\link{testing.main}} | |
| 36 | \code{\link[=testing_fromlog]{testing_fromlog()}} | |
| 36 | 37 | } |
| 37 | 38 | \author{ |
| 38 | 39 | Manuel López-Ibáñez |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/main.R | |
| 2 | \name{testing.main} | |
| 3 | \alias{testing.main} | |
| 4 | \title{testing.main} | |
| 5 | \usage{ | |
| 6 | testing.main(logFile) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{logFile}{Path to the \code{.Rdata} file produced by \pkg{irace}.} | |
| 10 | } | |
| 11 | \value{ | |
| 12 | Boolean. TRUE if the testing ended successfully otherwise, returns | |
| 13 | FALSE. | |
| 14 | } | |
| 15 | \description{ | |
| 16 | \code{testing.main} executes the testing of the target | |
| 17 | algorithm configurations found on an \pkg{irace} execution. | |
| 18 | } | |
| 19 | \details{ | |
| 20 | The function \code{testing.main} loads the \code{logFile} and | |
| 21 | obtains the needed configurations according to the specified test. Use the | |
| 22 | \code{scenario$testNbElites} to test N final elite configurations or use | |
| 23 | \code{scenario$testIterationElites} to test the best configuration of each | |
| 24 | iteration. A test instance set must be provided through | |
| 25 | \code{scenario$testInstancesDir} and \code{testInstancesFile}. | |
| 26 | } | |
| 27 | \seealso{ | |
| 28 | \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. | |
| 29 | } | |
| 30 | \author{ | |
| 31 | Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 32 | } |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/main.R | |
| 2 | \name{testing_fromfile} | |
| 3 | \alias{testing_fromfile} | |
| 4 | \title{Test configurations given an explicit table of configurations and a scenario file} | |
| 5 | \usage{ | |
| 6 | testing_fromfile(filename, scenario) | |
| 7 | } | |
| 8 | \arguments{ | |
| 9 | \item{filename}{Path to a file containing configurations: one configuration | |
| 10 | per line, one parameter per column, parameter names in header.} | |
| 11 | ||
| 12 | \item{scenario}{(\code{list()}) \cr Data structure containing \pkg{irace} | |
| 13 | settings. The data structure has to be the one returned by the function | |
| 14 | \code{\link[=defaultScenario]{defaultScenario()}} or \code{\link[=readScenario]{readScenario()}}.} | |
| 15 | } | |
| 16 | \value{ | |
| 17 | iraceResults | |
| 18 | } | |
| 19 | \description{ | |
| 20 | Executes the testing of an explicit list of configurations given in | |
| 21 | \code{filename} (same format as in \code{\link[=readConfigurationsFile]{readConfigurationsFile()}}). A \code{logFile} is | |
| 22 | created unless disabled in \code{scenario}. This may overwrite an existing one! | |
| 23 | } | |
| 24 | \seealso{ | |
| 25 | \code{\link[=testing_fromlog]{testing_fromlog()}} provides a different interface for testing. | |
| 26 | } | |
| 27 | \author{ | |
| 28 | Manuel López-Ibáñez | |
| 29 | } | |
| 30 | \concept{running} |
| 0 | % Generated by roxygen2: do not edit by hand | |
| 1 | % Please edit documentation in R/main.R | |
| 2 | \name{testing_fromlog} | |
| 3 | \alias{testing_fromlog} | |
| 4 | \title{Test configurations given in \code{.Rdata} file} | |
| 5 | \usage{ | |
| 6 | testing_fromlog( | |
| 7 | logFile, | |
| 8 | testNbElites, | |
| 9 | testIterationElites, | |
| 10 | testInstancesDir, | |
| 11 | testInstancesFile, | |
| 12 | testInstances | |
| 13 | ) | |
| 14 | } | |
| 15 | \arguments{ | |
| 16 | \item{logFile}{Path to the \code{.Rdata} file produced by \pkg{irace}.} | |
| 17 | ||
| 18 | \item{testNbElites}{Number of (final) elite configurations to test. Overrides | |
| 19 | the value found in \code{logFile}.} | |
| 20 | ||
| 21 | \item{testIterationElites}{(\code{logical(1)}) If \code{FALSE}, only the final | |
| 22 | \code{testNbElites} configurations are tested; otherwise, also test the best | |
| 23 | configurations of each iteration. Overrides the value found in \code{logFile}.} | |
| 24 | ||
| 25 | \item{testInstancesDir}{Directory where testing instances are located, either absolute or relative to current directory.} | |
| 26 | ||
| 27 | \item{testInstancesFile}{File containing a list of test instances and optionally additional parameters for them.} | |
| 28 | ||
| 29 | \item{testInstances}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing.} | |
| 30 | } | |
| 31 | \value{ | |
| 32 | Boolean. \code{TRUE} if the testing ended successfully otherwise, \code{FALSE}. | |
| 33 | } | |
| 34 | \description{ | |
| 35 | \code{testing_fromlog} executes the testing of the target algorithm configurations | |
| 36 | found by an \pkg{irace} execution. | |
| 37 | } | |
| 38 | \details{ | |
| 39 | The function \code{testing_fromlog} loads the \code{logFile} and obtains the | |
| 40 | testing setup and configurations to be tested. Within the \code{logFile}, the | |
| 41 | variable \code{scenario$testNbElites} specifies how many final elite | |
| 42 | configurations to test and \code{scenario$testIterationElites} indicates | |
| 43 | whether test the best configuration of each iteration. The values may be | |
| 44 | overridden by setting the corresponding arguments in this function. The | |
| 45 | set of testing instances must appear in \code{scenario[["testInstances"]]}. | |
| 46 | } | |
| 47 | \seealso{ | |
| 48 | \code{\link[=defaultScenario]{defaultScenario()}} to provide a default scenario for \pkg{irace}. | |
| 49 | \code{\link[=testing_fromfile]{testing_fromfile()}} provides a different interface for testing. | |
| 50 | } | |
| 51 | \author{ | |
| 52 | Manuel López-Ibáñez and Leslie Pérez Cáceres | |
| 53 | } | |
| 54 | \concept{running} |
| 0 | 0 | # -*- mode: makefile -*- |
| 1 | 1 | EXE= |
| 2 | iraceEXE= iracebin/irace$(EXE) | |
| 3 | irace2pyimpEXE= iracebin/irace2pyimp$(EXE) | |
| 2 | include Makevars-common | |
| 4 | 3 | |
| 5 | all: $(iraceEXE) $(irace2pyimpEXE) | |
| 6 | 4 | |
| 7 | $(irace2pyimpEXE): iracebin/irace2pyimp.o | |
| 8 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace2pyimp.o -o $@ | |
| 9 | ||
| 10 | $(iraceEXE): iracebin/irace.o | |
| 11 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace.o -o $@ | |
| 12 | ||
| 13 | iracebin/irace.o: iracebin/irace.h | |
| 14 | iracebin/irace2pyimp.o: iracebin/irace.h | |
| 15 | ||
| 16 | clean: | |
| 17 | @$(RM) iracebin/*.o $(irace2pyimpEXE) $(iraceEXE) |
| 0 | iraceEXE= iracebin/irace$(EXE) | |
| 1 | ablationEXE= iracebin/ablation$(EXE) | |
| 2 | ||
| 3 | all: $(iraceEXE) $(ablationEXE) | |
| 4 | ||
| 5 | $(iraceEXE): iracebin/irace.o | |
| 6 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace.o -o $@ | |
| 7 | ||
| 8 | iracebin/irace.o: iracebin/irace.h | |
| 9 | ||
| 10 | $(ablationEXE): iracebin/ablation.o | |
| 11 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/ablation.o -o $@ | |
| 12 | ||
| 13 | iracebin/ablation.o: iracebin/irace.h | |
| 14 | ||
| 15 | clean: | |
| 16 | @$(RM) iracebin/*.o $(iraceEXE) $(ablationEXE) |
| 0 | 0 | # -*- mode: makefile -*- |
| 1 | 1 | EXE=.exe |
| 2 | iraceEXE= iracebin/irace$(EXE) | |
| 3 | irace2pyimpEXE= iracebin/irace2pyimp$(EXE) | |
| 2 | include Makevars-common | |
| 4 | 3 | |
| 5 | all: $(iraceEXE) $(irace2pyimpEXE) | |
| 6 | ||
| 7 | $(irace2pyimpEXE): iracebin/irace2pyimp.o | |
| 8 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace2pyimp.o -o $@ | |
| 9 | ||
| 10 | $(iraceEXE): iracebin/irace.o | |
| 11 | $(CC) $(PKG_CPPFLAGS) $(CFLAGS) iracebin/irace.o -o $@ | |
| 12 | ||
| 13 | iracebin/irace.o: iracebin/irace.h | |
| 14 | iracebin/irace2pyimp.o: iracebin/irace.h | |
| 15 | ||
| 16 | clean: | |
| 17 | @$(RM) iracebin/*.o $(irace2pyimpEXE) $(iraceEXE) |
| 0 | execs <- file.path("iracebin", c("irace","irace2pyimp")) | |
| 0 | execs <- file.path("iracebin", c("irace","ablation")) | |
| 1 | 1 | if (WINDOWS) execs <- paste0(execs, ".exe") |
| 2 | 2 | if (any(file.exists(execs))) { |
| 3 | 3 | dest <- file.path(R_PACKAGE_DIR, paste0('bin', R_ARCH)) |
| 0 | #include <stdlib.h> | |
| 1 | #include <string.h> | |
| 2 | #include <stdio.h> | |
| 3 | #include <stdarg.h> | |
| 4 | #include <unistd.h> | |
| 5 | ||
| 6 | #define R_CODE "library(irace);ablation_cmdline()" | |
| 7 | #include "irace.h" | |
| 8 | ||
| 9 |
| 0 | #include <stdlib.h> | |
| 1 | #include <string.h> | |
| 2 | #include <stdio.h> | |
| 3 | #include <stdarg.h> | |
| 4 | #include <unistd.h> | |
| 5 | ||
| 6 | #define R_CODE "library(irace);irace2pyimp_cmdline()" | |
| 7 | #include "irace.h" | |
| 8 | ||
| 9 | ||
| 10 |
| 0 | parametersFile = "parameters.txt" |
Binary diff not shown
Binary diff not shown
| 0 | param1 "--param1 " i (1, "real") | |
| 1 | param2 "--param2 " i ("real", 10) | |
| 2 | mode "--" c ("x1" ,"x2", "x3") | |
| 3 | real "--paramreal=" r (1.5, 7.5) | |
| 4 | mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") | |
| 5 | #unused "-u " c (1, 2, 10, 20) |
| 0 | param1 "--param1 " i (1, "real") | mode %in% c("x1", "x2") | |
| 1 | param2 "--param2 " i ("real", 10) | mode %in% c("x1", "x3") && real > 2.5 && real <= 3.5 | |
| 2 | mode "--" c ("x1" ,"x2", "x3") | |
| 3 | real "--paramreal=" r (1.5, 7.5) | |
| 4 | mutation "--mutation=" o ("none", "very low", "low", "medium", "high", "very high", "all") | |
| 5 | #unused "-u " c (1, 2, 10, 20) |
| 7 | 7 | |
| 8 | 8 | test_irace_detectCores <- function() |
| 9 | 9 | { |
| 10 | if (identical(Sys.getenv("NOT_CRAN"), "true")) | |
| 10 | if (identical(Sys.getenv("NOT_CRAN"), "true")) { | |
| 11 | x <- Sys.getenv("_R_CHECK_LIMIT_CORES_", "") | |
| 12 | if (nzchar(x) && x == "TRUE") return(2L) | |
| 11 | 13 | return(parallel::detectCores()) |
| 14 | } | |
| 12 | 15 | return(1L) |
| 13 | 16 | } |
| 0 | n_templates_middle [1,50] [1]il | |
| 1 | n_templates_delta [1,25] [8]il | |
| 2 | n_slots_middle [1,50] [2]il | |
| 3 | n_slots_delta [1,25] [22]il | |
| 4 | n_variations_middle [1,50] [34]il | |
| 5 | n_variations_delta [1,25] [21]il | |
| 6 | demand_range_middle [1,51] [17]il | |
| 7 | demand_range_delta [1,26] [3]il | |
| 8 | ||
| 9 | ||
| 10 | #Conditions: |
| 0 | algo=/circa/home/nttd/instance-generation-conjure/instance-generation/experiments/maxint-50-log/002-TemplateDesign/target-runner | |
| 1 | execDir=./ | |
| 2 | deterministic=false | |
| 3 | run_obj=quality | |
| 4 | cutoff_time=1 | |
| 5 | tunerTimeout=999999 | |
| 6 | overall_obj=mean | |
| 7 | paramfile=params.pcs | |
| 8 | instance_file=instances.txt | |
| 9 | feature_file=features.csv |
| 0 | {"cputime": 32, "evaluations": 7, "cost": 0.611593, "incumbent": ["n_templates_middle=1", "n_templates_delta=1", "n_slots_middle=10", "n_slots_delta=11", "n_variations_middle=30", "n_variations_delta=3", "demand_range_middle=23", "demand_range_delta=1"], "total_cpu_time": null, "wallclock_time": 32} |
| 0 | instance,nCities | |
| 1 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-1.tsp,1000 | |
| 2 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-2.tsp,1000 | |
| 3 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-3.tsp,1000 | |
| 4 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-4.tsp,1000 | |
| 5 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-5.tsp,1000 | |
| 6 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-6.tsp,1000 | |
| 7 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-7.tsp,1000 | |
| 8 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-8.tsp,1000 | |
| 9 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-9.tsp,1000 | |
| 10 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-10.tsp,1000 | |
| 11 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-1.tsp,1500 | |
| 12 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-2.tsp,1500 | |
| 13 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-3.tsp,1500 | |
| 14 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-4.tsp,1500 | |
| 15 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-5.tsp,1500 | |
| 16 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-6.tsp,1500 | |
| 17 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-7.tsp,1500 | |
| 18 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-8.tsp,1500 | |
| 19 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-9.tsp,1500 | |
| 20 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-10.tsp,1500 | |
| 21 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-1.tsp,2000 | |
| 22 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-2.tsp,2000 | |
| 23 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-3.tsp,2000 | |
| 24 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-4.tsp,2000 | |
| 25 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-5.tsp,2000 | |
| 26 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-6.tsp,2000 | |
| 27 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-7.tsp,2000 | |
| 28 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-8.tsp,2000 | |
| 29 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-9.tsp,2000 | |
| 30 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-10.tsp,2000 | |
| 31 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-1.tsp,2500 | |
| 32 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-2.tsp,2500 | |
| 33 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-3.tsp,2500 | |
| 34 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-4.tsp,2500 | |
| 35 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-5.tsp,2500 | |
| 36 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-6.tsp,2500 | |
| 37 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-7.tsp,2500 | |
| 38 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-8.tsp,2500 | |
| 39 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-9.tsp,2500 | |
| 40 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-10.tsp,2500 | |
| 41 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-1.tsp,3000 | |
| 42 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-2.tsp,3000 | |
| 43 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-3.tsp,3000 | |
| 44 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-4.tsp,3000 | |
| 45 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-5.tsp,3000 | |
| 46 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-6.tsp,3000 | |
| 47 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-7.tsp,3000 | |
| 48 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-8.tsp,3000 | |
| 49 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-9.tsp,3000 | |
| 50 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-10.tsp,3000 |
| 0 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-1.tsp | |
| 1 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-2.tsp | |
| 2 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-3.tsp | |
| 3 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-4.tsp | |
| 4 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-5.tsp | |
| 5 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-6.tsp | |
| 6 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-7.tsp | |
| 7 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-8.tsp | |
| 8 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-9.tsp | |
| 9 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-10.tsp | |
| 10 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-1.tsp | |
| 11 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-2.tsp | |
| 12 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-3.tsp | |
| 13 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-4.tsp | |
| 14 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-5.tsp | |
| 15 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-6.tsp | |
| 16 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-7.tsp | |
| 17 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-8.tsp | |
| 18 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-9.tsp | |
| 19 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-10.tsp | |
| 20 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-1.tsp | |
| 21 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-2.tsp | |
| 22 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-3.tsp | |
| 23 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-4.tsp | |
| 24 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-5.tsp | |
| 25 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-6.tsp | |
| 26 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-7.tsp | |
| 27 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-8.tsp | |
| 28 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-9.tsp | |
| 29 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-10.tsp | |
| 30 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-1.tsp | |
| 31 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-2.tsp | |
| 32 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-3.tsp | |
| 33 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-4.tsp | |
| 34 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-5.tsp | |
| 35 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-6.tsp | |
| 36 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-7.tsp | |
| 37 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-8.tsp | |
| 38 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-9.tsp | |
| 39 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-10.tsp | |
| 40 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-1.tsp | |
| 41 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-2.tsp | |
| 42 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-3.tsp | |
| 43 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-4.tsp | |
| 44 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-5.tsp | |
| 45 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-6.tsp | |
| 46 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-7.tsp | |
| 47 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-8.tsp | |
| 48 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-9.tsp | |
| 49 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-10.tsp |
| 0 | algorithm {as,mmas,eas,ras,acs} [as] | |
| 1 | localsearch {0,1,2,3} [0] | |
| 2 | alpha [0,5] [0] | |
| 3 | beta [0,10] [0] | |
| 4 | rho [0.01,1] [0.01] | |
| 5 | ants [1,100] [5]i | |
| 6 | nnls [5,50] [5]i | |
| 7 | dlb {0,1} [0] | |
| 8 | q0 [0,1] [0] | |
| 9 | rasrank [1,100] [1]i | |
| 10 | elitistants [1,750] [1]i | |
| 11 | ||
| 12 | ||
| 13 | #Conditions: | |
| 14 | nnls | localsearch in {1, 2, 3} | |
| 15 | dlb | localsearch in {1, 2, 3} | |
| 16 | q0 | algorithm in {acs, mmas} | |
| 17 | rasrank | algorithm in {ras} | |
| 18 | elitistants | algorithm in {eas} |
| 0 | algo=/vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/run-02/target-runner | |
| 1 | execDir=./ | |
| 2 | deterministic=false | |
| 3 | run_obj=quality | |
| 4 | cutoff_time=1 | |
| 5 | tunerTimeout=999999 | |
| 6 | overall_obj=mean | |
| 7 | paramfile=params.pcs | |
| 8 | instance_file=instances.txt | |
| 9 | feature_file=features.csv |
| 0 | {"cputime": 58, "evaluations": 7, "cost": 33371218.4285714, "incumbent": ["algorithm='mmas'", "localsearch='3'", "alpha=3.8", "beta=4.71", "rho=0.27", "ants=39", "nnls=18", "dlb='1'", "q0=0.43"], "total_cpu_time": null, "wallclock_time": 58} | |
| 1 | {"cputime": 58, "evaluations": 10, "cost": 32743608.7, "incumbent": ["algorithm='mmas'", "localsearch='3'", "alpha=3.8", "beta=4.71", "rho=0.27", "ants=39", "nnls=18", "dlb='1'", "q0=0.43"], "total_cpu_time": null, "wallclock_time": 58} | |
| 2 | {"cputime": 382, "evaluations": 13, "cost": 31385289.6923077, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.6", "beta=7.48", "rho=0.43", "ants=28", "nnls=22", "dlb='1'", "q0=0.69"], "total_cpu_time": null, "wallclock_time": 382} | |
| 3 | {"cputime": 492, "evaluations": 15, "cost": 31721217.2666667, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=2", "beta=6.19", "rho=0.52", "ants=36", "nnls=19", "dlb='1'", "q0=0.63"], "total_cpu_time": null, "wallclock_time": 492} | |
| 4 | {"cputime": 636, "evaluations": 19, "cost": 31670552.6842105, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.92", "beta=8.3", "rho=0.65", "ants=38", "nnls=11", "dlb='1'", "q0=0.84"], "total_cpu_time": null, "wallclock_time": 636} | |
| 5 | {"cputime": 649, "evaluations": 21, "cost": 32294021.6666667, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.85", "beta=8.34", "rho=0.46", "ants=32", "nnls=9", "dlb='1'", "q0=0.89"], "total_cpu_time": null, "wallclock_time": 649} | |
| 6 | {"cputime": 674, "evaluations": 23, "cost": 32803908.9565217, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.87", "beta=8.3", "rho=0.43", "ants=26", "nnls=12", "dlb='1'", "q0=0.89"], "total_cpu_time": null, "wallclock_time": 674} | |
| 7 | {"cputime": 682, "evaluations": 23, "cost": 32371424.1304348, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.74", "beta=8.16", "rho=0.49", "ants=36", "nnls=11", "dlb='1'", "q0=0.9"], "total_cpu_time": null, "wallclock_time": 682} |
| 0 | instance,nCities | |
| 1 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-1.tsp,1000 | |
| 2 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-2.tsp,1000 | |
| 3 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-3.tsp,1000 | |
| 4 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-4.tsp,1000 | |
| 5 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-5.tsp,1000 | |
| 6 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-6.tsp,1000 | |
| 7 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-7.tsp,1000 | |
| 8 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-8.tsp,1000 | |
| 9 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-9.tsp,1000 | |
| 10 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-10.tsp,1000 | |
| 11 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-1.tsp,1500 | |
| 12 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-2.tsp,1500 | |
| 13 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-3.tsp,1500 | |
| 14 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-4.tsp,1500 | |
| 15 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-5.tsp,1500 | |
| 16 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-6.tsp,1500 | |
| 17 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-7.tsp,1500 | |
| 18 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-8.tsp,1500 | |
| 19 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-9.tsp,1500 | |
| 20 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-10.tsp,1500 | |
| 21 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-1.tsp,2000 | |
| 22 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-2.tsp,2000 | |
| 23 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-3.tsp,2000 | |
| 24 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-4.tsp,2000 | |
| 25 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-5.tsp,2000 | |
| 26 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-6.tsp,2000 | |
| 27 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-7.tsp,2000 | |
| 28 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-8.tsp,2000 | |
| 29 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-9.tsp,2000 | |
| 30 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-10.tsp,2000 | |
| 31 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-1.tsp,2500 | |
| 32 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-2.tsp,2500 | |
| 33 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-3.tsp,2500 | |
| 34 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-4.tsp,2500 | |
| 35 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-5.tsp,2500 | |
| 36 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-6.tsp,2500 | |
| 37 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-7.tsp,2500 | |
| 38 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-8.tsp,2500 | |
| 39 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-9.tsp,2500 | |
| 40 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-10.tsp,2500 | |
| 41 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-1.tsp,3000 | |
| 42 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-2.tsp,3000 | |
| 43 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-3.tsp,3000 | |
| 44 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-4.tsp,3000 | |
| 45 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-5.tsp,3000 | |
| 46 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-6.tsp,3000 | |
| 47 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-7.tsp,3000 | |
| 48 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-8.tsp,3000 | |
| 49 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-9.tsp,3000 | |
| 50 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-10.tsp,3000 |
| 0 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-1.tsp | |
| 1 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-2.tsp | |
| 2 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-3.tsp | |
| 3 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-4.tsp | |
| 4 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-5.tsp | |
| 5 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-6.tsp | |
| 6 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-7.tsp | |
| 7 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-8.tsp | |
| 8 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-9.tsp | |
| 9 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1000-10.tsp | |
| 10 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-1.tsp | |
| 11 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-2.tsp | |
| 12 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-3.tsp | |
| 13 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-4.tsp | |
| 14 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-5.tsp | |
| 15 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-6.tsp | |
| 16 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-7.tsp | |
| 17 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-8.tsp | |
| 18 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-9.tsp | |
| 19 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/1500-10.tsp | |
| 20 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-1.tsp | |
| 21 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-2.tsp | |
| 22 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-3.tsp | |
| 23 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-4.tsp | |
| 24 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-5.tsp | |
| 25 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-6.tsp | |
| 26 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-7.tsp | |
| 27 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-8.tsp | |
| 28 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-9.tsp | |
| 29 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2000-10.tsp | |
| 30 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-1.tsp | |
| 31 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-2.tsp | |
| 32 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-3.tsp | |
| 33 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-4.tsp | |
| 34 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-5.tsp | |
| 35 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-6.tsp | |
| 36 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-7.tsp | |
| 37 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-8.tsp | |
| 38 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-9.tsp | |
| 39 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/2500-10.tsp | |
| 40 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-1.tsp | |
| 41 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-2.tsp | |
| 42 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-3.tsp | |
| 43 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-4.tsp | |
| 44 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-5.tsp | |
| 45 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-6.tsp | |
| 46 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-7.tsp | |
| 47 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-8.tsp | |
| 48 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-9.tsp | |
| 49 | /vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/data/tsp-rue-1000-3000/3000-10.tsp |
| 0 | algorithm {as,mmas,eas,ras,acs} [as] | |
| 1 | localsearch {0,1,2,3} [0] | |
| 2 | alpha [0,5] [0] | |
| 3 | beta [0,10] [0] | |
| 4 | rho [0.01,1] [0.01] | |
| 5 | ants [1,100] [5]i | |
| 6 | nnls [5,50] [5]i | |
| 7 | dlb {0,1} [0] | |
| 8 | q0 [0,1] [0] | |
| 9 | rasrank [1,100] [1]i | |
| 10 | elitistants [1,750] [1]i | |
| 11 | ||
| 12 | ||
| 13 | #Conditions: | |
| 14 | nnls | localsearch in {1, 2, 3} | |
| 15 | dlb | localsearch in {1, 2, 3} | |
| 16 | q0 | algorithm in {acs, mmas} | |
| 17 | rasrank | algorithm in {ras} | |
| 18 | elitistants | algorithm in {eas} |
| 0 | algo=/vsc-hard-mounts/leuven-data/307/vsc30746/ParameterImportance/tuning-irace/ACOTSP/run-02/target-runner | |
| 1 | execDir=./ | |
| 2 | deterministic=false | |
| 3 | run_obj=quality | |
| 4 | cutoff_time=1 | |
| 5 | tunerTimeout=999999 | |
| 6 | overall_obj=mean | |
| 7 | paramfile=params.pcs | |
| 8 | instance_file=instances.txt | |
| 9 | feature_file=features.csv |
| 0 | {"cputime": 58, "evaluations": 7, "cost": 0.0514369130188286, "incumbent": ["algorithm='mmas'", "localsearch='3'", "alpha=3.8", "beta=4.71", "rho=0.27", "ants=39", "nnls=18", "dlb='1'", "q0=0.43"], "total_cpu_time": null, "wallclock_time": 58} | |
| 1 | {"cputime": 58, "evaluations": 10, "cost": 0.0514506545239323, "incumbent": ["algorithm='mmas'", "localsearch='3'", "alpha=3.8", "beta=4.71", "rho=0.27", "ants=39", "nnls=18", "dlb='1'", "q0=0.43"], "total_cpu_time": null, "wallclock_time": 58} | |
| 2 | {"cputime": 382, "evaluations": 13, "cost": 0.0378041911323998, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.6", "beta=7.48", "rho=0.43", "ants=28", "nnls=22", "dlb='1'", "q0=0.69"], "total_cpu_time": null, "wallclock_time": 382} | |
| 3 | {"cputime": 492, "evaluations": 15, "cost": 0.0346616167298519, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=2", "beta=6.19", "rho=0.52", "ants=36", "nnls=19", "dlb='1'", "q0=0.63"], "total_cpu_time": null, "wallclock_time": 492} | |
| 4 | {"cputime": 636, "evaluations": 19, "cost": 0.060167259522421, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.92", "beta=8.3", "rho=0.65", "ants=38", "nnls=11", "dlb='1'", "q0=0.84"], "total_cpu_time": null, "wallclock_time": 636} | |
| 5 | {"cputime": 649, "evaluations": 21, "cost": 0.0646020648718007, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.85", "beta=8.34", "rho=0.46", "ants=32", "nnls=9", "dlb='1'", "q0=0.89"], "total_cpu_time": null, "wallclock_time": 649} | |
| 6 | {"cputime": 674, "evaluations": 23, "cost": 0.0625681325316403, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.87", "beta=8.3", "rho=0.43", "ants=26", "nnls=12", "dlb='1'", "q0=0.89"], "total_cpu_time": null, "wallclock_time": 674} | |
| 7 | {"cputime": 682, "evaluations": 23, "cost": 0.0599210644937226, "incumbent": ["algorithm='acs'", "localsearch='3'", "alpha=1.74", "beta=8.16", "rho=0.49", "ants=36", "nnls=11", "dlb='1'", "q0=0.9"], "total_cpu_time": null, "wallclock_time": 682} |
| 0 | parameterFile = "parameters.txt" |
| 0 | old_opts = options( | |
| 1 | warnPartialMatchArgs = TRUE, | |
| 2 | warnPartialMatchAttr = TRUE, | |
| 3 | warnPartialMatchDollar = TRUE | |
| 4 | ) | |
| 5 | ||
| 6 | old_opts = lapply(old_opts, function(x) if (is.null(x)) FALSE else x) | |
| 7 |
| 0 | options(old_opts) |
| 4 | 4 | skip_on_cran() |
| 5 | 5 | skip_on_ci() |
| 6 | 6 | skip_on_travis() |
| 7 | skip_on_os("mac") | |
| 7 | 8 | # FIXME: how to check that a python package is installed? |
| 8 | ||
| 9 | 9 | parameters <- readParameters(text = ' |
| 10 | 10 | cost "cost" r (0.1, 1.00) |
| 11 | 11 | runtime "runtime" r (0.1, 1.00)') |
| 0 | context("bug") | |
| 1 | skip_on_cran() | |
| 2 | withr::with_output_sink("test-bad_scenario.Rout", { | |
| 3 | expect_error(irace.cmdline("--scenario bad_scenario.txt"), "unknown variables") | |
| 4 | expect_error(irace.cmdline("--scenario scenario.txt --unknown"), "Unknown command-line options: --unknown") | |
| 5 | }) |
| 0 | context("bug") | |
| 1 | skip_on_cran() | |
| 2 | # https://github.com/MLopez-Ibanez/irace/issues/10 | |
| 3 | withr::with_output_sink("test-bug-10.Rout", { | |
| 4 | ||
| 5 | parameters.txt <- ' | |
| 6 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 7 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 8 | alpha "--alpha " r (0.00, 5.00) | |
| 9 | beta "--beta " r (0.00, 10.00) | |
| 10 | rho "--rho " r (0.01, 1.00) | |
| 11 | ants "--ants " i (5, 100) | |
| 12 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 13 | rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 14 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 15 | nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) | |
| 16 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 17 | ' | |
| 18 | target.runner <- function(experiment, scenario) | |
| 19 | return(list(cost = 100, call = toString(experiment))) | |
| 20 | ||
| 21 | parameters <- irace:::readParameters(text=parameters.txt) | |
| 22 | ||
| 23 | withr::with_options(list(warning=2), { | |
| 24 | scenario <- list(targetRunner = target.runner, | |
| 25 | instances=1:10, | |
| 26 | maxExperiments = 5000, logFile = "", | |
| 27 | deterministic = TRUE, | |
| 28 | elitistNewInstances = 0, | |
| 29 | elitistLimit = 0, | |
| 30 | elitist = 0) | |
| 31 | scenario <- checkScenario (scenario) | |
| 32 | confs <- irace(scenario = scenario, parameters = parameters) | |
| 33 | }) | |
| 34 | }) |
| 0 | context("bug") | |
| 1 | skip_on_cran() | |
| 2 | # https://github.com/MLopez-Ibanez/irace/issues/13 | |
| 3 | withr::with_output_sink("test-bug-13.Rout", { | |
| 4 | ||
| 5 | parameters.txt <- ' | |
| 6 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 7 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 8 | alpha "--alpha " r (0.00, 5.00) | |
| 9 | beta "--beta " r (0.00, 10.00) | |
| 10 | rho "--rho " r (0.01, 1.00) | |
| 11 | ants "--ants " i (5, 100) | |
| 12 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 13 | rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 14 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 15 | nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) | |
| 16 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 17 | ' | |
| 18 | parameters <- irace:::readParameters(text=parameters.txt) | |
| 19 | ||
| 20 | lookup <- readRDS("bug-13-lookup.rds") | |
| 21 | counter <- 0 | |
| 22 | target.runner <- function(experiment, scenario) { | |
| 23 | configuration <- experiment$configuration | |
| 24 | instance <- experiment$instance | |
| 25 | switches <- experiment$switches | |
| 26 | debugLevel <- scenario$debugLevel | |
| 27 | args <- paste(instance, trimws(irace:::buildCommandLine(configuration, switches))) | |
| 28 | pos <- pmatch(args, lookup[,1]) | |
| 29 | if(!is.na(pos)) { | |
| 30 | cost <- as.numeric(lookup[pos, 2]) | |
| 31 | counter <<- counter + 1 | |
| 32 | } else { | |
| 33 | cost <- 10 | |
| 34 | } | |
| 35 | if(debugLevel > 0) { | |
| 36 | cat("# ", args, "\n") | |
| 37 | cat(cost, "\n") | |
| 38 | } | |
| 39 | return(list(cost = cost, call = toString(experiment))) | |
| 40 | } | |
| 41 | ||
| 42 | withr::with_options(list(warning=2), { | |
| 43 | scenario <- list(instances = scan(what="", quiet=TRUE, text=" | |
| 44 | 2000-121.tsp | |
| 45 | 2000-122.tsp | |
| 46 | 2000-123.tsp | |
| 47 | 2000-124.tsp | |
| 48 | 2000-125.tsp | |
| 49 | 2000-126.tsp | |
| 50 | 2000-127.tsp | |
| 51 | 2000-128.tsp | |
| 52 | 2000-129.tsp | |
| 53 | 2000-130.tsp"), | |
| 54 | trainInstancesDir = "", | |
| 55 | trainInstancesFile = "", | |
| 56 | maxExperiments = 3000, | |
| 57 | digits = 3, | |
| 58 | seed = 2357, | |
| 59 | deterministic = 1, | |
| 60 | elitist = 0, | |
| 61 | targetRunner = target.runner, | |
| 62 | debugLevel=3) | |
| 63 | scenario <- checkScenario (scenario) | |
| 64 | confs <- irace(scenario = scenario, parameters = parameters) | |
| 65 | expect_equal(counter, 548) | |
| 66 | }) | |
| 67 | ||
| 68 | }) |
| 5 | 5 | { |
| 6 | 6 | # FIXME: For some unknown reason, this test doesn't work in R 3.6 |
| 7 | 7 | skip_if(grepl("^3.6", getRversion())) |
| 8 | skip_if(grepl("^4", getRversion())) | |
| 8 | 9 | |
| 9 | 10 | scenario <- readRDS(rds_filename)$scenario |
| 10 | 11 | parameters <- readRDS(rds_filename)$parameters |
| 11 | ||
| 12 | ||
| 12 | 13 | scenario$targetRunner <- function(experiment, scenario, filename = rds_filename) { |
| 13 | 14 | saved <- readRDS(filename) |
| 14 | 15 | row <- which(saved$instancesList[, "instance"] == experiment[["id.instance"]] |
| 15 | 16 | & saved$instancesList[, "seed"] == experiment[["seed"]]) |
| 17 | ||
| 16 | 18 | cost <- saved$experiments[row, experiment[["id.configuration"]] ] |
| 17 | if (is.na(cost)) { | |
| 19 | if (length(cost) == 0 || is.na(cost)) { | |
| 18 | 20 | print(row) |
| 19 | 21 | print(experiment) |
| 20 | 22 | } |
| 29 | 31 | skip_on_cran() |
| 30 | 32 | # FIXME: For some unknown reason, this test doesn't work in R 3.6 |
| 31 | 33 | skip_if(grepl("^3.6", getRversion())) |
| 34 | skip_if(grepl("^4", getRversion())) | |
| 32 | 35 | # FIXME: Convert this test to use run_irace_from_rds() |
| 33 | 36 | load("bug_large_new_instances.Rdata", verbose = TRUE) |
| 34 | 37 | |
| 38 | 41 | row <- which(saved_instances_list[, "instance"] == experiment[["id.instance"]] |
| 39 | 42 | & saved_instances_list[, "seed"] == experiment[["seed"]]) |
| 40 | 43 | cost <- saved_experiments[row, experiment[["id.configuration"]] ] |
| 41 | if (is.na(cost)) { | |
| 44 | if (length(cost) == 0 || is.na(cost)) { | |
| 42 | 45 | print(row) |
| 43 | 46 | print(experiment) |
| 44 | 47 | } |
| 99 | 102 | |
| 100 | 103 | test_that("maxim_bug2", { |
| 101 | 104 | skip_on_cran() |
| 102 | ||
| 103 | 105 | run_irace_from_rds("saved_maxim_bug2.rds") |
| 104 | 106 | }) |
| 107 | ||
| 108 | test_that("non-normal", { | |
| 109 | skip_on_cran() | |
| 110 | df <- as.matrix(read.table(text=' | |
| 111 | 0.007858276 0.007934570 0.007949829 | |
| 112 | 0.009384155 0.009521484 0.009719849 | |
| 113 | 0.011520386 0.012100220 0.012176514 | |
| 114 | 0.214721680 0.219146729 0.219512939 | |
| 115 | 0.005462646 0.005462646 0.005523682 | |
| 116 | 0.005035400 0.005035400 0.005035400 | |
| 117 | 0.011001587 0.011367798 0.011962891 | |
| 118 | 0.153244019 0.155212402 1.157546997 | |
| 119 | 0.008636475 0.008789062 0.008666992 | |
| 120 | 0.008239746 0.008499146 0.008560181 | |
| 121 | 0.010848999 0.011077881 0.011703491 | |
| 122 | 0.191589355 0.191879272 0.191589355 | |
| 123 | 0.011245728 0.011245728 0.011245728 | |
| 124 | 0.006454468 0.006454468 0.006561279 | |
| 125 | 0.010330200 0.010894775 0.011032104 | |
| 126 | 0.160491943 0.164169312 0.167068481', header=FALSE)) | |
| 127 | ## colnames(df) <- 1:3 | |
| 128 | ## #df <- log(df) | |
| 129 | ## library(ggplot2) | |
| 130 | ## library(tidyr) | |
| 131 | ## df_long <- as.data.frame(df) | |
| 132 | ## df_long$run <- 1:16 | |
| 133 | ||
| 134 | ## df_long <- df_long %>% gather("config", "y", as.character(1:3)) | |
| 135 | ## ggplot(df_long, aes(df=y)) + | |
| 136 | ## geom_histogram(aes(y=..density..), colour="black", fill="white", binwidth=0.01)+ | |
| 137 | ## geom_density(alpha=.2, fill="#FF6666") + | |
| 138 | ## facet_grid(config ~ .) | |
| 139 | ||
| 140 | ## ggplot(df_long, aes(x=config, y=y)) + geom_boxplot() + geom_jitter() | |
| 141 | ## aggregate(y ~ config, data = df_long, FUN = function(x) c(mean = mean(x), sd = sd(x), median=median(x), | |
| 142 | ## shapiro.test = shapiro.test(x)$p.value) ) | |
| 143 | ||
| 144 | ## t.test(x[,1], x[,2],paired=TRUE) | |
| 145 | ## t.test(x[,1], x[,3],paired=TRUE) | |
| 146 | ||
| 147 | ## wilcox.test(x[,1], x[,2],paired=TRUE) | |
| 148 | ## wilcox.test(x[,1], x[,3],paired=TRUE) | |
| 149 | ||
| 150 | ## print(all(x[,1] <= x[,2])) | |
| 151 | ## print(all(x[,1] <= x[,3])) | |
| 152 | parameters <- readParameters(text='p "" c (1,2,3,4,5)') | |
| 153 | scenario <- list(instances = 1:16, maxExperiments = 50, | |
| 154 | testType="t-test") | |
| 155 | scenario$targetRunner <- function(experiment, scenario, this_df = df) { | |
| 156 | if (experiment[["id.configuration"]] > 3) | |
| 157 | cost <- as.numeric(experiment[["id.configuration"]]) | |
| 158 | else | |
| 159 | cost <- this_df[as.numeric(experiment[["id.instance"]]), | |
| 160 | as.numeric(experiment[["id.configuration"]]) ] | |
| 161 | return(list(cost=cost)) | |
| 162 | } | |
| 163 | confs <- irace(scenario = scenario, parameters = parameters) | |
| 164 | print(confs) | |
| 165 | expect_equal(confs$.ID., 1L) | |
| 166 | }) | |
| 167 | ||
| 105 | 168 | }) # withr::with_output_sink() |
| 106 | ||
| 150 | 150 | irace:::checkTargetFiles(scenario = scenario, parameters = parameters) |
| 151 | 151 | |
| 152 | 152 | confs <- irace(scenario = scenario, parameters = parameters) |
| 153 | best.conf <- getFinalElites(logFile = scenario$logFile, n = 1, | |
| 154 | drop.metadata = TRUE) | |
| 155 | expect_identical(removeConfigurationsMetaData(confs[1, , drop = FALSE]), | |
| 153 | best.conf <- getFinalElites(scenario$logFile, n = 1L, drop.metadata = TRUE) | |
| 154 | expect_identical(removeConfigurationsMetaData(confs[1L, , drop = FALSE]), | |
| 156 | 155 | best.conf) |
| 157 | 156 | } |
| 158 | 157 |
| 0 | context("Test dependencies") | |
| 1 | ||
| 2 | withr::with_output_sink("test-dependencies.Rout", { | |
| 3 | ||
| 4 | test_that("param depend error checking", { | |
| 5 | expect_error_readParameters <- function(text, error) | |
| 6 | expect_error(readParameters(text=text), error) | |
| 7 | ||
| 8 | expect_error_readParameters(' | |
| 9 | p1 "" r (0, 1) | |
| 10 | p2 "" r (p3, 1) | |
| 11 | ', "parameter 'p2' is not valid: 'p3' cannot be found") | |
| 12 | ||
| 13 | expect_error_readParameters(' | |
| 14 | p1 "" c (0, 1) | |
| 15 | p2 "" r (1, p1) | |
| 16 | ', "parameter 'p2' depends on non-numerical parameters") | |
| 17 | ||
| 18 | expect_error_readParameters(' | |
| 19 | p1 "" r (0, 1) | |
| 20 | p2 "" r (1, foobar(p1)) | |
| 21 | ', "parameter 'p2' uses function") | |
| 22 | ||
| 23 | expect_error_readParameters(' | |
| 24 | p1 "" r (0, p3) | |
| 25 | p2 "" r (1, p1) | |
| 26 | p3 "" r (p2, 1) | |
| 27 | ', "cycle detected") | |
| 28 | ||
| 29 | expect_error_readParameters(' | |
| 30 | p1 "" r (0, 1) | |
| 31 | p2 "" i (0.1, p1) | |
| 32 | ', "values must be integers") | |
| 33 | ||
| 34 | }) | |
| 35 | ||
| 36 | ||
| 37 | checkConditionalAndDependency <- function(configuration, parameters) | |
| 38 | { | |
| 39 | namesParameters <- names(parameters$conditions) | |
| 40 | for (p in namesParameters) { | |
| 41 | if (!irace:::conditionsSatisfied(parameters, configuration, p)) { | |
| 42 | expect(is.na(configuration[,p]), | |
| 43 | paste0("Conditional parameter '", p, | |
| 44 | "' is not active but it has a value '", configuration[,p], "' assigned.")) | |
| 45 | } else if (parameters$isDependent[p]) { | |
| 46 | bounds <- irace:::getDependentBound(parameters, p, configuration) | |
| 47 | if (anyNA(bounds)) { | |
| 48 | expect(is.na(configuration[,p]), | |
| 49 | paste0("Dependent parameter '", p, | |
| 50 | "' has a value '", configuration[,p], "' but it should be inactive.")) | |
| 51 | expect_true(anyNA(configuration[parameters$depends[[p]]])) | |
| 52 | } else { | |
| 53 | expect (configuration[,p] >= bounds[1], | |
| 54 | paste0("Parameter '", p, "=", configuration[,p], | |
| 55 | "' does not comply with dependency: ", parameters$depends[[p]], | |
| 56 | " and lower bound: ", bounds[1])) | |
| 57 | expect (configuration[,p] <= bounds[2], | |
| 58 | paste0("Parameter '", p, " = ", configuration[,p], | |
| 59 | "' does not comply with dependency: ", parameters$depends[[p]], | |
| 60 | " and upper bound: ", bounds[2])) | |
| 61 | } | |
| 62 | } | |
| 63 | } | |
| 64 | } | |
| 65 | ||
| 66 | ||
| 67 | test_that("test inactive dependent", { | |
| 68 | parameters <- readParameters(text=' | |
| 69 | p1 "" r (0,1) | |
| 70 | p2 "" r (0, p1) | p1 < 0.5 | |
| 71 | p3 "" r (0, p2) | |
| 72 | ') | |
| 73 | confs <- irace:::sampleUniform(parameters, 50, digits = 2) | |
| 74 | for (i in 1:nrow(confs)) { | |
| 75 | checkConditionalAndDependency(confs[i,], parameters) | |
| 76 | } | |
| 77 | }) | |
| 78 | ||
| 79 | test_that("checkDependencies", { | |
| 80 | ||
| 81 | target.runner <- function(experiment, scenario) | |
| 82 | { | |
| 83 | configuration <- experiment$configuration | |
| 84 | tmax <- as.numeric(configuration[["real"]]) | |
| 85 | if (configuration[["mode"]] %in% c("x1", "x2")) | |
| 86 | temp <- as.numeric(configuration[["param1"]]) | |
| 87 | else | |
| 88 | temp <- 1 | |
| 89 | time <- max(1, abs(rnorm(1, mean=(tmax+temp)/10))) | |
| 90 | return(list(cost = time, time = time, call = toString(experiment))) | |
| 91 | } | |
| 92 | ||
| 93 | test.checkDependencies <- function(parameterFile, ...) | |
| 94 | { | |
| 95 | args <- list(...) | |
| 96 | weights <- rnorm(200, mean = 0.9, sd = 0.02) | |
| 97 | parameters <- readParameters(parameterFile) | |
| 98 | scenario <- list(targetRunner = target.runner, instances = weights, seed = 1234567, maxExperiments=200) | |
| 99 | scenario <- modifyList(scenario, args) | |
| 100 | scenario <- checkScenario (scenario) | |
| 101 | ||
| 102 | nconf <- 100 | |
| 103 | conf <- irace:::sampleUniform(parameters, nconf, 4) | |
| 104 | conf <- cbind(seq(1,nrow(conf)), conf) | |
| 105 | names(conf)[1]<- ".ID." | |
| 106 | ||
| 107 | for (i in 1:nconf) | |
| 108 | checkConditionalAndDependency(conf[i,], parameters) | |
| 109 | ||
| 110 | model <- irace:::initialiseModel(parameters, conf, 4) | |
| 111 | conf2 <- irace:::sampleModel(parameters, conf, model, nconf, 4) | |
| 112 | for (i in 1:nconf) | |
| 113 | checkConditionalAndDependency(conf2[i,], parameters) | |
| 114 | ||
| 115 | confs <- irace(scenario = scenario, parameters = parameters) | |
| 116 | for (i in 1:nrow(confs)) { | |
| 117 | checkConditionalAndDependency(confs[i,], parameters) | |
| 118 | } | |
| 119 | } | |
| 120 | test.checkDependencies(parameterFile="dependencies.txt") | |
| 121 | test.checkDependencies(parameterFile="dependencies2.txt") | |
| 122 | }) | |
| 123 | ||
| 124 | }) # withr::with_output_sink() |
| 0 | 0 | context("Test forbidden") |
| 1 | ||
| 2 | withr::with_output_sink("test-forbidden.Rout", { | |
| 1 | 3 | |
| 2 | 4 | test_that("checkForbidden", { |
| 3 | 5 | |
| 21 | 23 | test.checkForbidden("logparameters.txt") |
| 22 | 24 | }) |
| 23 | 25 | |
| 26 | }) | |
| 0 | context("irace2pyimp") # Avoid bug in testthat 2.0.0 | |
| 1 | test_that("irace2pyimp", { | |
| 2 | skip_on_cran() | |
| 3 | ||
| 4 | check_irace2pyimp_files <- function(outdir, origdir) | |
| 5 | { | |
| 6 | # Print it for debugging if something goes wrong | |
| 7 | cat("outdir: ", outdir, "\n") | |
| 8 | ||
| 9 | outfiles <- c("scenario.txt", "params.pcs", "runhistory.json", "traj_aclib2.json", "instances.txt", "features.csv") | |
| 10 | for (file in outfiles) { | |
| 11 | outfile <- file.path(outdir, file) | |
| 12 | origfile <- file.path(origdir, file) | |
| 13 | if (file.exists(paste0(origfile, ".gz"))) | |
| 14 | origfile <- paste0(origfile, ".gz") | |
| 15 | else | |
| 16 | expect_true(file.exists(origfile)) | |
| 17 | expect_true(file.exists(outfile)) | |
| 18 | expect_equal(readLines(origfile), readLines(outfile)) | |
| 19 | } | |
| 20 | } | |
| 21 | ||
| 22 | # acotsp example 1: convert irace.Rdata to PyImp's input format, without normalisation | |
| 23 | cat(". Testing acotsp_1","\n") | |
| 24 | outdir <- file.path("/home/manu/work/irace/nguyen/tests/testthat/", "irace2pyimp/acotsp_1") | |
| 25 | outdir <- tempdir() | |
| 26 | logfile <- file.path(system.file(package="irace"), "examples", "irace2pyimp", | |
| 27 | "acotsp", "irace.Rdata") | |
| 28 | featuresfile <- file.path(system.file(package="irace"), "examples", | |
| 29 | "irace2pyimp", "acotsp", "features.csv") | |
| 30 | ||
| 31 | irace2pyimp(file = logfile, instanceFeatureFile = featuresfile, outdir = outdir) | |
| 32 | check_irace2pyimp_files(outdir, "./irace2pyimp/acotsp_1/") | |
| 33 | ||
| 34 | # acotsp example 4: convert irace.Rdata to PyImp's input format, with normalisation based on feature-vector values | |
| 35 | cat(". Testing acotsp_4","\n") | |
| 36 | outdir <- tempdir() | |
| 37 | logfile <- file.path(system.file(package="irace"), "examples", "irace2pyimp", | |
| 38 | "acotsp", "irace.Rdata") | |
| 39 | featuresfile <- file.path(system.file(package="irace"), "examples", | |
| 40 | "irace2pyimp", "acotsp", "features.csv") | |
| 41 | ||
| 42 | irace2pyimp(file = logfile, outdir = outdir, | |
| 43 | normalise='feature', instanceFeatureFile = featuresfile) | |
| 44 | check_irace2pyimp_files(outdir, "./irace2pyimp/acotsp_4/") | |
| 45 | ||
| 46 | # 002-TemplateDesign example: using irace2pyimp with filter conditions and normalisation: | |
| 47 | # - filter tuning data such that only configurations with n_templates_middle<=40 are used, then convert irace.Rdata to PIMP's input format, | |
| 48 | # - the cost values of all configurations are normalised on an instance-basis | |
| 49 | cat(". Testing 002-TemplateDesign","\n") | |
| 50 | outdir <- "/home/manu/work/irace/nguyen/tests/testthat/irace2pyimp/002-TemplateDesign" | |
| 51 | outdir <- tempdir() | |
| 52 | logfile <- file.path(system.file(package="irace"), "examples", "irace2pyimp", | |
| 53 | "002-TemplateDesign", "irace.Rdata") | |
| 54 | ||
| 55 | irace2pyimp(file = logfile, outdir = outdir, normalise = "instance", | |
| 56 | filterConditions = "n_templates_middle<=40") | |
| 57 | ||
| 58 | check_irace2pyimp_files(outdir, "./irace2pyimp/002-TemplateDesign") | |
| 59 | }) |
| 14 | 14 | { |
| 15 | 15 | args <- list(...) |
| 16 | 16 | weights <- rnorm(200, mean = 0.9, sd = 0.02) |
| 17 | test_weights <- rnorm(2, mean = 0.9, sd = 0.02) | |
| 17 | 18 | parameters <- readParameters(text = ' |
| 18 | 19 | tmax "" i (1, 50) |
| 19 | 20 | temp "" r (0, 10) |
| 20 | 21 | ') |
| 21 | scenario <- list(targetRunner = target.runner, instances = weights, seed = 1234567) | |
| 22 | scenario <- list(targetRunner = target.runner, | |
| 23 | instances = weights, | |
| 24 | testInstances = test_weights, | |
| 25 | seed = 1234567) | |
| 22 | 26 | scenario <- modifyList(scenario, args) |
| 23 | 27 | scenario <- checkScenario (scenario) |
| 24 | 28 | |
| 25 | 29 | irace:::checkTargetFiles(scenario = scenario, parameters = parameters) |
| 26 | 30 | |
| 27 | 31 | confs <- irace(scenario = scenario, parameters = parameters) |
| 32 | final_ids <- as.character(sort(confs$.ID.[1:scenario$testNbElites])) | |
| 28 | 33 | expect_gt(nrow(confs), 0L) |
| 34 | testing_fromlog(scenario$logFile) | |
| 35 | load(scenario$logFile) | |
| 36 | if (scenario$testIterationElites) { | |
| 37 | # FIXME: We could test here that the correct configurations are tested. | |
| 38 | expect_gte(ncol(iraceResults$testing$experiments), scenario$testNbElites) | |
| 39 | } else { | |
| 40 | test_ids <- sort(colnames(iraceResults$testing$experiments)) | |
| 41 | expect_equivalent(final_ids, test_ids) | |
| 42 | } | |
| 43 | return(confs) | |
| 29 | 44 | } |
| 30 | 45 | |
| 31 | 46 | |
| 32 | test_that("maxTime 500", { | |
| 47 | test_that("maxTime=500 testNbElites=2 testIterationElites=FALSE", { | |
| 33 | 48 | generate.set.seed() |
| 34 | time.irace(maxTime = 500) | |
| 49 | time.irace(maxTime = 500, testNbElites=2) | |
| 35 | 50 | }) |
| 36 | 51 | |
| 37 | test_that("maxTime 500", { | |
| 52 | test_that("maxTime=1111 testNbElites=3 testIterationElites=TRUE", { | |
| 38 | 53 | skip_on_cran() |
| 39 | 54 | generate.set.seed() |
| 40 | time.irace(maxTime = 1111) | |
| 55 | time.irace(maxTime = 1111, testNbElites=3, testIterationElites=TRUE) | |
| 41 | 56 | }) |
| 42 | 57 | |
| 43 | 58 | }) # withr::with_output_sink() |
| 0 | context("path.rel2abs") | |
| 0 | context("path_rel2abs") | |
| 1 | 1 | |
| 2 | test_that("test.path.rel2abs", { | |
| 2 | test_that("test.path_rel2abs", { | |
| 3 | 3 | # Try to set wd; otherwise fail silently. |
| 4 | 4 | old.cwd <- getwd() |
| 5 | 5 | skip_if(is.null(old.cwd)) |
| 40 | 40 | "../../../data" "./" "/data" |
| 41 | 41 | "../../../data" "/tmp/a/b/c/" "/tmp/data" |
| 42 | 42 | "..//a" ".//" "/a" |
| 43 | "R" "/tmp/" "Sys.which" | |
| 43 | 44 | ', stringsAsFactors=FALSE) |
| 44 | 45 | for(i in 1:nrow(testcases)) { |
| 45 | 46 | orig <- testcases[i,1] |
| 46 | 47 | cwd <- testcases[i,2] |
| 47 | res <- irace:::path.rel2abs(testcases[i,1], cwd) | |
| 48 | exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) | |
| 48 | res <- irace:::path_rel2abs(testcases[i,1], cwd) | |
| 49 | if (testcases[i,3] == "Sys.which") { | |
| 50 | exp <- normalizePath(Sys.which(testcases[i,1]), winslash = "/", mustWork = NA) | |
| 51 | } else { | |
| 52 | exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) | |
| 53 | } | |
| 49 | 54 | if (res == exp) { |
| 50 | 55 | # cat("[OK] (", orig, ", ", cwd, ") -> ", res, "\n", sep="") |
| 51 | 56 | } else { |
| 55 | 60 | } |
| 56 | 61 | }) |
| 57 | 62 | |
| 58 | test_that("test.path.rel2abs for windows", { | |
| 63 | test_that("test.path_rel2abs for windows", { | |
| 59 | 64 | |
| 60 | 65 | testcases <- read.table(text=' |
| 61 | 66 | . N:\\\\tmp N:/tmp |
| 162 | 167 | x.r N:/tmp N:/tmp/x.r |
| 163 | 168 | ~/irace/../x.r N:/tmp ~/x.r |
| 164 | 169 | ~/x.r N:/tmp ~/x.r |
| 170 | "R" "/tmp/" "Sys.which" | |
| 165 | 171 | ', stringsAsFactors=FALSE) |
| 166 | 172 | for(i in 1:nrow(testcases)) { |
| 167 | 173 | orig <- testcases[i,1] |
| 168 | 174 | cwd <- testcases[i,2] |
| 169 | res <- irace:::path.rel2abs(testcases[i,1], cwd) | |
| 170 | exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) | |
| 175 | res <- path_rel2abs(testcases[i,1], cwd) | |
| 176 | if (testcases[i,3] == "Sys.which") { | |
| 177 | exp <- normalizePath(Sys.which(testcases[i,1]), winslash = "/", mustWork = NA) | |
| 178 | } else { | |
| 179 | exp <- gsub("\\", "/", path.expand(testcases[i,3]), fixed = TRUE) | |
| 180 | } | |
| 171 | 181 | if (res == exp) { |
| 172 | #cat("[OK] ", i, ": path.rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, "\n", sep="") | |
| 182 | #cat("[OK] ", i, ": path_rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, "\n", sep="") | |
| 173 | 183 | } else { |
| 174 | cat("[FAILED] ", i, ": path.rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, " but expected: ", exp, "\n") | |
| 184 | cat("[FAILED] ", i, ": path_rel2abs(\"", orig, "\", \"", cwd, "\") -> ", res, " but expected: ", exp, "\n") | |
| 175 | 185 | } |
| 176 | 186 | expect_match(res, exp, fixed = TRUE) |
| 177 | 187 | } |
| 0 | context("raceconfs") | |
| 1 | ||
| 2 | withr::with_output_sink("test-raceconfs.Rout", { | |
| 3 | ||
| 4 | parameters.txt <- ' | |
| 5 | launch_method "--launch_method=" c (L-BFGS-B,SLSQP) | |
| 6 | visit_method "--visit_method=" c (DA, DE) | |
| 7 | global_method "--global_method=" c (CMAES,DE) | |
| 8 | ' | |
| 9 | confs.txt <- ' | |
| 10 | launch_method visit_method global_method | |
| 11 | L-BFGS-B DA CMAES | |
| 12 | SLSQP DA CMAES | |
| 13 | L-BFGS-B DE CMAES | |
| 14 | SLSQP DE CMAES | |
| 15 | L-BFGS-B DA DE | |
| 16 | SLSQP DA DE | |
| 17 | L-BFGS-B DE DE | |
| 18 | SLSQP DE DE | |
| 19 | ' | |
| 20 | ||
| 21 | target.runner <- function(experiment, scenario) | |
| 22 | return(list(cost = 100, call = toString(experiment))) | |
| 23 | ||
| 24 | withr::with_options(list(warning=2), { | |
| 25 | parameters <- irace:::readParameters(text=parameters.txt) | |
| 26 | initconfs <- irace:::readConfigurationsFile(text=confs.txt, parameters=parameters) | |
| 27 | scenario <- list(targetRunner = target.runner, | |
| 28 | instances=1:10, | |
| 29 | nbConfigurations=8, | |
| 30 | maxExperiments = 96, logFile = "", | |
| 31 | initConfigurations=initconfs) | |
| 32 | scenario <- checkScenario (scenario) | |
| 33 | confs <- irace(scenario = scenario, parameters = parameters) | |
| 34 | }) | |
| 35 | }) |
| 0 | context("Test read configurations") | |
| 1 | ||
| 2 | withr::with_output_sink("test-readconfs.Rout", { | |
| 3 | ||
| 4 | test_that("checkDuplicates", { | |
| 5 | params <- irace:::readParameters("parameters.txt") | |
| 6 | ||
| 7 | expect_error(irace:::readConfigurationsFile(text=' | |
| 8 | param1 param2 mode real mutation | |
| 9 | 5 NA "x2" 4.0 "low" | |
| 10 | 1 NA "x2" 4.0 "low" | |
| 11 | 5 NA "x2" 4.0 "low" | |
| 12 | NA NA "x3" 4.5 "low" | |
| 13 | ', parameters = params), "Duplicated") | |
| 14 | ||
| 15 | expect_error(irace:::readConfigurationsFile(text=' | |
| 16 | param1 param2 mode real mutation | |
| 17 | 5 NA "x2" 4.0 "low" | |
| 18 | 1 NA "x2" 4.0 "low" | |
| 19 | 5 NA "x2" 4.0 "low" | |
| 20 | 1 NA "x2" 4.0 "low" | |
| 21 | ', parameters = params), "Duplicated") | |
| 22 | ||
| 23 | }) | |
| 24 | ||
| 25 | }) | |
| 26 |
| 74 | 74 | scenario <- checkScenario (scenario) |
| 75 | 75 | |
| 76 | 76 | confs <- irace(scenario = scenario, parameters = parameters) |
| 77 | best.conf <- getFinalElites(logFile = scenario$logFile, n = 1, | |
| 78 | drop.metadata = TRUE) | |
| 77 | best.conf <- getFinalElites(scenario$logFile, n = 1, drop.metadata = TRUE) | |
| 79 | 78 | expect_identical(removeConfigurationsMetaData(confs[1, , drop = FALSE]), |
| 80 | 79 | best.conf) |
| 81 | 80 | } |
| 0 | library(testthat) | |
| 1 | library(irace) | |
| 2 | test_check("irace", reporter = c("summary","check")) | |
| 0 | if (requireNamespace("testthat", quietly = TRUE)) { | |
| 1 | library(testthat) | |
| 2 | library(irace) | |
| 3 | test_check("irace", reporter = c("summary","check")) | |
| 4 | } | |
| 3 | 5 | |
| 6 |
| 0 | # irace 3.5 | |
| 1 | ||
| 2 | ## New features and improvements | |
| 3 | ||
| 4 | * Handling of dependent parameter domains: These should be specified in the | |
| 5 | parameter domain definition and, for now, only numerical parameter can | |
| 6 | define dependent domains. A numerical domain can be dependent on one bound, | |
| 7 | e.g. `(1, "param1*2")`, where the dependent bound can include basic | |
| 8 | arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 9 | ||
| 10 | * The package now provides an `ablation` executable (`ablation.exe` in | |
| 11 | Windows) that makes easier to perform ablation analysis without having any R | |
| 12 | knowledge. | |
| 13 | ||
| 14 | * The interface to functions `ablation()` and `plotAblation()` has been | |
| 15 | simplified. The `ablation()` function now allows overriding scenario | |
| 16 | settings. The `plotAblation()` function will not create the plot if the | |
| 17 | ablation log does not contain a complete ablation. | |
| 18 | (Manuel López-Ibáñez) | |
| 19 | ||
| 20 | * The argument `n.instances` of `ablation()` has been renamed to `n_instances` and it is now a factor that multiplies `scenario$firstTest`. | |
| 21 | (Manuel López-Ibáñez) | |
| 22 | ||
| 23 | * New command-line option `--quiet` to run without producing any output | |
| 24 | except errors (also available as a scenario option). | |
| 25 | (Manuel López-Ibáñez) | |
| 26 | ||
| 27 | * New command-line option `--init` to initialize a scenario. (Deyao Chen) | |
| 28 | ||
| 29 | * Added support for HTCondor cluster framework to `--batchmode`. | |
| 30 | (Filippo Bistaffa) | |
| 31 | ||
| 32 | * `--check` now also check the contents of `configurationsFile` and runs | |
| 33 | configurations provided via `initConfigurations`. | |
| 34 | (Manuel López-Ibáñez, reported by Andreea Avramescu) | |
| 35 | ||
| 36 | * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs` | |
| 37 | to help in cases where the target-runner must be invoked via another | |
| 38 | software with particular options (such as `python.exe` in Windows). | |
| 39 | (Manuel López-Ibáñez) | |
| 40 | ||
| 41 | * New scenario option `minMeasurableTime`. | |
| 42 | (Manuel López-Ibáñez) | |
| 43 | ||
| 44 | * An error is produced if a variable set in the scenario file is not known to | |
| 45 | irace. If your scenario file contains R code, then use variable names | |
| 46 | beginning with a dot `'.'`, which will be ignored by irace. | |
| 47 | (Manuel López-Ibáñez) | |
| 48 | ||
| 49 | * Plotting functions have been moved to the new package | |
| 50 | [iraceplot](https://auto-optimization.github.io/iraceplot/). In particular, | |
| 51 | `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and | |
| 52 | `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by | |
| 53 | `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and | |
| 54 | `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`. | |
| 55 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 56 | ||
| 57 | * The user-guide now contains a detailed section on "Hyper-parameter | |
| 58 | optimization of machine learning methods". | |
| 59 | (Manuel López-Ibáñez) | |
| 60 | ||
| 61 | * When `testType="F-test"` and only two configurations remain, the elimination | |
| 62 | test now uses the pseudo-median estimated by the Wilcoxon signed-rank test | |
| 63 | to decide which configuration is the best one instead of comparing the | |
| 64 | median difference. | |
| 65 | (Manuel López-Ibáñez) | |
| 66 | ||
| 67 | * New functions `testing_fromlog()` and `testing_fromfile()` for independently | |
| 68 | executing the testing phase. The function `testing.main()` was removed as it | |
| 69 | is superseded by the new ones. | |
| 70 | (Manuel López-Ibáñez) | |
| 71 | ||
| 72 | * New function `read_logfile()` to easily read the log file produced by irace. | |
| 73 | (Manuel López-Ibáñez) | |
| 74 | ||
| 75 | * New function `printParameters()` that prints a parameters R object as a valid input text. | |
| 76 | (Manuel López-Ibáñez) | |
| 77 | ||
| 78 | * `irace2pyimp` moved to its own R package. | |
| 79 | (Manuel López-Ibáñez) | |
| 80 | ||
| 81 | * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`. | |
| 82 | (Manuel López-Ibáñez, reported by Johann Dreo) | |
| 83 | ||
| 84 | * `path_rel2abs()` and `checkParameters()` are now exported so that other | |
| 85 | packages may use them. | |
| 86 | (Manuel López-Ibáñez) | |
| 87 | ||
| 88 | * `path_rel2abs()` also searches in system paths. (Manuel López-Ibáñez) | |
| 89 | ||
| 90 | * `readConfigurationsFile()` will now detect duplicated configurations and | |
| 91 | error. (Manuel López-Ibáñez) | |
| 92 | ||
| 93 | * The interface to functions `getFinalElites()`, `getConfigurationById()` and | |
| 94 | `getConfigurationByIteration()` has been simplified. | |
| 95 | ||
| 96 | * The package provides a `irace.sindef` file that may be used for building a | |
| 97 | standalone container of irace using Singularity. See the `README.md` file | |
| 98 | for instructions. (Contributed by Johann Dreo) | |
| 99 | ||
| 100 | * New example `examples/target-runner-python/target-runner-python-win.bat` | |
| 101 | contributed by Levi Ribeiro. | |
| 102 | ||
| 103 | * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters. | |
| 104 | (Manuel López-Ibáñez) | |
| 105 | ||
| 106 | * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez) | |
| 107 | ||
| 108 | ## Fixes | |
| 109 | ||
| 110 | * Correctly handle clear out-performance cases despite strong bi-modality. | |
| 111 | (Reported by Nguyen Dang, | |
| 112 | fixed by Manuel López-Ibáñez) | |
| 113 | ||
| 114 | * Fix error when recovering from a parallel run on Windows. | |
| 115 | (Manuel López-Ibáñez, reported by Tarek Gamal) | |
| 116 | ||
| 117 | * `testNbElites` now controls how many iteration elites are tested when | |
| 118 | `testIterationElites=1`. This is the documented behavior in the user guide. | |
| 119 | (Manuel López-Ibáñez, reported by Marcelo de Souza) | |
| 120 | ||
| 121 | * Fixes to the Matlab example. (Manuel López-Ibáñez) | |
| 122 | ||
| 123 | * The default of `testType` is now set to `t-test` when capping is enabled. | |
| 124 | (Manuel López-Ibáñez, reported by Jovana Radjenovic) | |
| 125 | ||
| 126 | * Fix various issues in the user guide. | |
| 127 | (Manuel López-Ibáñez, reported by Jovana Radjenovic) | |
| 128 | ||
| 129 | * Remove duplicated elites. | |
| 130 | (Manuel López-Ibáñez, reported by Federico Naldini) | |
| 131 | ||
| 132 | * Fix (#7): warnings with partial matched parameters. | |
| 133 | (Manuel López-Ibáñez, reported by Marc Becker) | |
| 134 | ||
| 135 | * Fix (#10): wrong assert with `elitist=0`. (Manuel López-Ibáñez) | |
| 136 | ||
| 137 | * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r). | |
| 138 | ||
| 139 | * Fix (#13): Maximum number configurations immediately rejected reached. | |
| 140 | (Manuel López-Ibáñez) | |
| 141 | ||
| 142 | * Fix: when setting the scenario file in the command-line, `scenarioFile` was | |
| 143 | not set correctly. The correct scenario was used, however, the debug output | |
| 144 | and the value stored in the log / recovery file was wrong. | |
| 145 | (Manuel López-Ibáñez, reported by Richard Schoonhoven) | |
| 146 | ||
| 147 | * With `sampleInstances = FALSE`, elitist irace does not change the order of | |
| 148 | instances already seen. However, if you want to make sure that the order of | |
| 149 | the instances is enforced, you also need to set `elitistNewInstances=0`. | |
| 150 | ||
| 151 | * The function `irace.usage()` was removed. It was not really useful for R | |
| 152 | users as the same result can be obtained by calling | |
| 153 | `irace.cmdline("--help")`. | |
| 154 | (Manuel López-Ibáñez) | |
| 155 | ||
| 156 | ||
| 0 | 157 | # irace 3.4.1 (31/03/2020) |
| 1 | 158 | |
| 2 | 159 | * `NEWS` converted to markdown. |
| 20 | 177 | versions. |
| 21 | 178 | (Manuel López-Ibáñez) |
| 22 | 179 | * Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres) |
| 23 | ||
| 180 | ||
| 24 | 181 | ``` |
| 25 | 182 | value >= 1L && value <= length(possibleValues) is not TRUE |
| 26 | 183 | ``` |
| 29 | 186 | of a script. On Windows, `irace.exe` replaces `irace.bat` |
| 30 | 187 | (Manuel López-Ibáñez) |
| 31 | 188 | |
| 32 | * inst/examples/Spear contains the Spear (SAT solver) configuration scenario. | |
| 189 | * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario. | |
| 33 | 190 | (Manuel López-Ibáñez) |
| 34 | 191 | |
| 35 | * Fixed bug when reporting minimum maxTime required. | |
| 192 | * Fixed bug when reporting minimum `maxTime` required. | |
| 36 | 193 | (Reported by Luciana Salete Buriol, |
| 37 | 194 | fixed by Manuel López-Ibáñez) |
| 38 | 195 | |
| 41 | 198 | ```R |
| 42 | 199 | all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE |
| 43 | 200 | ``` |
| 44 | ||
| 201 | ||
| 45 | 202 | (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez) |
| 46 | 203 | |
| 47 | 204 | |
| 60 | 217 | satisfy `digits` (up to `digits=15`). |
| 61 | 218 | (Manuel López-Ibáñez) |
| 62 | 219 | |
| 63 | * It is possible to specify boundMax without capping. | |
| 220 | * It is possible to specify `boundMax` without capping. | |
| 64 | 221 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) |
| 65 | 222 | |
| 66 | 223 | * `irace --check` will exit with code 1 if the check is unsuccessful |
| 68 | 225 | |
| 69 | 226 | * Print where irace is installed with `--help`. (Manuel López-Ibáñez) |
| 70 | 227 | |
| 71 | * irace will now complain if the output of target-runner or target-evaluator | |
| 228 | * irace will now complain if the output of `target-runner` or `target-evaluator` | |
| 72 | 229 | contains extra lines even if the first line of output is correct. This is to |
| 73 | 230 | avoid parsing the wrong output. Unfortunately, this may break setups that |
| 74 | 231 | relied on this behavior. The solution is to only print the output that irace |
| 81 | 238 | |
| 82 | 239 | * New option `aclib=` (`--aclib 1`) enables compatibility with the |
| 83 | 240 | GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by |
| 84 | AClib (http://aclib.net/). This is EXPERIMENTAL. | |
| 85 | `--aclib 1` also sets digits to 15 for compatibility with AClib defaults. | |
| 241 | AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets | |
| 242 | digits to 15 for compatibility with AClib defaults. | |
| 86 | 243 | (Manuel López-Ibáñez) |
| 87 | 244 | |
| 88 | 245 | * Fix printing of output when capping is enabled. |
| 107 | 264 | * Fix bug in `checkTargetFiles()` (`--check`) with capping. |
| 108 | 265 | (Leslie Pérez Cáceres) |
| 109 | 266 | |
| 110 | * Clarify a few errors/warnings when `maxTime` > 0. | |
| 267 | * Clarify a few errors/warnings when `maxTime > 0`. | |
| 111 | 268 | (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos) |
| 112 | 269 | |
| 113 | 270 | |
| 190 | 347 | evaluation noticeably faster. |
| 191 | 348 | (Manuel López-Ibáñez) |
| 192 | 349 | |
| 193 | * The argument 'experiment' passed to the R function targetRunner does not | |
| 194 | contain anymore an element 'extra.params'. Similarly, the 'scenario' | |
| 195 | structure does not contain anymore the elements 'instances.extra.params' and | |
| 196 | 'testInstances.extra.params'. Any instance-specific parameters values now | |
| 350 | * The argument `'experiment'` passed to the R function `targetRunner` does not | |
| 351 | contain anymore an element `'extra.params'`. Similarly, the `'scenario'` | |
| 352 | structure does not contain anymore the elements `'instances.extra.params'` and | |
| 353 | `'testInstances.extra.params'`. Any instance-specific parameters values now | |
| 197 | 354 | form part of the character string that defines an instance and it is up to |
| 198 | the user-defined targetRunner to parse them appropriately. These changes | |
| 355 | the user-defined `targetRunner` to parse them appropriately. These changes | |
| 199 | 356 | make no difference when targetRunner is an external script, or when |
| 200 | 357 | instances and instance-specific parameter values are read from a file. |
| 201 | 358 | (Manuel López-Ibáñez) |
| 202 | 359 | |
| 203 | 360 | # irace 2.3 |
| 204 | 361 | |
| 205 | * Fix bug that will cause iraceResults$experimentLog to count calls to | |
| 206 | targetEvaluator as experiments, even if no call to targetRunner was | |
| 362 | * Fix bug that will cause `iraceResults$experimentLog` to count calls to | |
| 363 | `targetEvaluator` as experiments, even if no call to `targetRunner` was | |
| 207 | 364 | performed. This does not affect the computation of the budget consumed and, |
| 208 | 365 | thus, it does not affect the termination criteria of irace. The bug triggers |
| 209 | 366 | an assertion that terminates irace, thus no run that was successful with |
| 221 | 378 | |
| 222 | 379 | * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by |
| 223 | 380 | `--batchmode` (`batchmode`). It is now the responsibility of the target-runner |
| 224 | to parse the output of the batch job submission command (e.g., qsub or | |
| 225 | squeue), and return just the job ID. Values supported are: "sge", "torque", | |
| 381 | to parse the output of the batch job submission command (e.g., `qsub` or | |
| 382 | `squeue`), and return just the job ID. Values supported are: "sge", "torque", | |
| 226 | 383 | "pbs" and "slurm". (Manuel López-Ibáñez) |
| 227 | 384 | |
| 228 | 385 | * The option `--parallel` can now be combined with `--batchmode` to limit the |
| 239 | 396 | ```R |
| 240 | 397 | eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE)) |
| 241 | 398 | ``` |
| 242 | ||
| 399 | ||
| 243 | 400 | This feature is VERY experimental and the syntax is likely to change in the |
| 244 | 401 | future. (Manuel López-Ibáñez) |
| 245 | 402 | |
| 256 | 413 | (Manuel López-Ibáñez, Leslie Pérez Cáceres) |
| 257 | 414 | |
| 258 | 415 | * Update manual and vignette with details about the expected arguments and |
| 259 | return value of targetRunner and targetEvaluator. (Manuel López-Ibáñez) | |
| 416 | return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez) | |
| 260 | 417 | |
| 261 | 418 | * Many updates to the User Guide vignette. (Manuel López-Ibáñez) |
| 262 | 419 | |
| 263 | * Fix \dontrun example in irace-package.Rd (Manuel López-Ibáñez) | |
| 420 | * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez) | |
| 264 | 421 | |
| 265 | 422 | * Fix bug: If testInstances contains duplicates, results of testing are not |
| 266 | correctly saved in iraceResults$testing$experiments nor reported correctly | |
| 267 | at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for | |
| 423 | correctly saved in `iraceResults$testing$experiments` nor reported correctly | |
| 424 | at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for | |
| 268 | 425 | each testing instance. These IDs are used for the rownames of |
| 269 | iraceResults$testing$experiments and the names of the scenario$testInstances | |
| 270 | and iraceResults$testing$seeds vectors. (Manuel López-Ibáñez) | |
| 271 | ||
| 272 | * Fix bug where irace keeps retrying the target-runner call even if it | |
| 426 | `iraceResults$testing$experiments` and the names of the | |
| 427 | `scenario$testInstances` | |
| 428 | and `iraceResults$testing$seeds` vectors. (Manuel López-Ibáñez) | |
| 429 | ||
| 430 | * Fix bug where irace keeps retrying the `target-runner` call even if it | |
| 273 | 431 | succeeds. (Manuel López-Ibáñez) |
| 274 | 432 | |
| 275 | 433 | * New command-line parameter |
| 280 | 438 | instances defined by the scenario. Useful if you decide on the testing |
| 281 | 439 | instances only after running irace. (Manuel López-Ibáñez) |
| 282 | 440 | |
| 283 | * Bugfix: When using maxTime != 0, the number of experiments performed may be | |
| 441 | * Bugfix: When using `maxTime != 0`, the number of experiments performed may be | |
| 284 | 442 | miscounted in some cases. (Manuel López-Ibáñez) |
| 285 | 443 | |
| 286 | 444 | |
| 304 | 462 | comparisons), |
| 305 | 463 | - `t-test-holm` (t-test with Holm's correction for multiple comparisons) |
| 306 | 464 | |
| 307 | * MPI does not create log files with --debug-level 0. | |
| 465 | * MPI does not create log files with `--debug-level 0`. | |
| 308 | 466 | (Manuel López-Ibáñez) |
| 309 | 467 | |
| 310 | * For simplicity, the parallel-irace-* scripts do not use an auxiliary | |
| 468 | * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary | |
| 311 | 469 | `tune-main` script. For customizing them, make a copy and edit them |
| 312 | 470 | directly. |
| 313 | 471 | (Manuel López-Ibáñez) |
| 316 | 474 | ``` |
| 317 | 475 | --target-runner-retries : Retry target-runner this many times in case of error. |
| 318 | 476 | ``` |
| 319 | ||
| 477 | ||
| 320 | 478 | |
| 321 | 479 | * We print diversity measures after evaluating on each instance: |
| 322 | 480 | (Leslie Pérez Cáceres) |
| 379 | 537 | |
| 380 | 538 | * The best configurations found, either at the end or at each iteration of an |
| 381 | 539 | irace run, can now be applied to a set of test instances different from the |
| 382 | training instances. See options testInstanceDir, testInstanceFile, | |
| 383 | testNbElites, and testIterationElites. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 384 | ||
| 385 | * The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed. | |
| 540 | training instances. See options `testInstanceDir`, `testInstanceFile`, | |
| 541 | `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) | |
| 542 | ||
| 543 | * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed. | |
| 386 | 544 | See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of |
| 387 | 545 | the new interfaces. |
| 388 | 546 | |
| 390 | 548 | IDs, and numbers are printed in a more human-readable format. |
| 391 | 549 | (Leslie Pérez Cáceres, Manuel López-Ibáñez) |
| 392 | 550 | |
| 393 | * Reduce memory use for very large values of maxExperiments. | |
| 551 | * Reduce memory use for very large values of `maxExperiments`. | |
| 394 | 552 | (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) |
| 395 | 553 | |
| 396 | 554 | * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing |
| 417 | 575 | |
| 418 | 576 | * New configuration options, mainly for R users: |
| 419 | 577 | |
| 420 | - hookRunParallel: Optional R function to provide custom | |
| 421 | parallelization of hook.run. | |
| 422 | ||
| 423 | - hookRunData: Optional data passed to hookRun. This is ignored by | |
| 424 | the default hookRun function, but it may be used by custom hookRun R | |
| 425 | functions to pass persistent data around. | |
| 426 | (Manuel López-Ibáñez) | |
| 578 | - `hookRunParallel`: Optional R function to provide custom | |
| 579 | parallelization of `hook.run`. | |
| 580 | ||
| 581 | - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the | |
| 582 | default `hookRun` function, but it may be used by custom `hookRun` R | |
| 583 | functions to pass persistent data around. (Manuel López-Ibáñez) | |
| 427 | 584 | |
| 428 | 585 | # irace 1.05 |
| 429 | 586 | |
| 448 | 605 | See `--forbidden-file` and `inst/templates/forbidden.tmpl`. |
| 449 | 606 | (Manuel López-Ibáñez) |
| 450 | 607 | |
| 451 | * New option `--recovery-file` (recoveryFile) allows resuming a | |
| 608 | * New option `--recovery-file` (`recoveryFile`) allows resuming a | |
| 452 | 609 | previous irace run. (Leslie Pérez Cáceres) |
| 453 | 610 | |
| 454 | 611 | * The confidence level for the elimination test is now |
| 469 | 626 | * Print elapsed time for calls to hook-run if `debugLevel >=1`. |
| 470 | 627 | (Manuel López-Ibáñez) |
| 471 | 628 | |
| 472 | * `examples/hook-run-python/hook-run`: A multi-purpose hook-run written | |
| 629 | * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written | |
| 473 | 630 | in Python. (Franco Mascia) |
| 474 | 631 | |
| 475 | 632 | * Parallel mode in an SGE cluster (`--sge-cluster`) is more |
| 499 | 656 | |
| 500 | 657 | * More concise output. |
| 501 | 658 | |
| 502 | * The parameters expName and expDescription are now useless and they | |
| 659 | * The parameters `expName` and `expDescription` are now useless and they | |
| 503 | 660 | were removed. |
| 504 | 661 | |
| 505 | 662 | * Faster computation of similar candidates (Jeremie Dubois-Lacoste |
| 506 | 663 | and Leslie Pérez Cáceres). |
| 507 | 664 | |
| 508 | * Fix bug when saving instances in tunerResults$experiments. | |
| 665 | * Fix bug when saving instances in `tunerResults$experiments`. | |
| 509 | 666 | |
| 510 | 667 | * `irace.cmdline ("--help")` does not try to quit R anymore. |
| 511 | 668 | |
Binary diff not shown
Binary diff not shown
| 0 | 0 | #------------------------------------------------------------------------------ |
| 1 | 1 | # irace: An implementation in R of (Elitist) Iterated Racing |
| 2 | # Version: 3.4.9a6f8d4 | |
| 3 | # Copyright (C) 2010-2019 | |
| 2 | # Version: 3.5.6863679 | |
| 3 | # Copyright (C) 2010-2020 | |
| 4 | 4 | # Manuel Lopez-Ibanez <manuel.lopez-ibanez@manchester.ac.uk> |
| 5 | 5 | # Jeremie Dubois-Lacoste |
| 6 | 6 | # Leslie Perez Caceres <leslie.perez.caceres@ulb.ac.be> |
| 13 | 13 | # race: Racing methods for the selection of the best |
| 14 | 14 | # Copyright (C) 2003 Mauro Birattari |
| 15 | 15 | #------------------------------------------------------------------------------ |
| 16 | # installed at: /home/manu/R/x86_64-pc-linux-gnu-library/3.4/irace | |
| 16 | # installed at: /home/manu/R/x86_64-pc-linux-gnu-library/3.6/irace | |
| 17 | 17 | # called with: --parallel 2 |
| 18 | Warning: A default scenario file './scenario.txt' has been found and will be read | |
| 18 | == irace == WARNING: A default scenario file '/home/manu/work/irace/git/examples/vignette-example/scenario.txt' has been found and will be read. | |
| 19 | 19 | # Read 1 configuration(s) from file '/home/manu/work/irace/git/examples/vignette-example/default.txt' |
| 20 | # 2020-03-27 13:00:40 GMT: Initialization | |
| 20 | # 2022-10-22 10:04:33 BST: Initialization | |
| 21 | 21 | # Elitist race |
| 22 | 22 | # Elitist new instances: 1 |
| 23 | 23 | # Elitist limit: 2 |
| 24 | 24 | # nbIterations: 5 |
| 25 | 25 | # minNbSurvival: 5 |
| 26 | 26 | # nbParameters: 11 |
| 27 | # seed: 39201275 | |
| 27 | # seed: 1406598565 | |
| 28 | 28 | # confidence level: 0.95 |
| 29 | 29 | # budget: 1000 |
| 30 | 30 | # mu: 5 |
| 31 | 31 | # deterministic: FALSE |
| 32 | 32 | |
| 33 | # 2020-03-27 13:00:40 GMT: Iteration 1 of 5 | |
| 33 | # 2022-10-22 10:04:33 BST: Iteration 1 of 5 | |
| 34 | 34 | # experimentsUsedSoFar: 0 |
| 35 | 35 | # remainingBudget: 1000 |
| 36 | 36 | # currentBudget: 200 |
| 43 | 43 | ! The test is performed and configurations could be discarded but elite configurations are preserved. |
| 44 | 44 | . All alive configurations are elite and nothing is discarded |
| 45 | 45 | |
| 46 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 47 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 48 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 49 | |x| 1| 33| 20| 23285353.00| 33|00:02:58| NA| NA| NA| | |
| 50 | |x| 2| 33| 3| 23321763.50| 66|00:03:00|+0.96|0.98|0.0121| | |
| 51 | |x| 3| 33| 3| 23400954.67| 99|00:02:56|+0.97|0.98|0.0148| | |
| 52 | |x| 4| 33| 20| 23291781.00| 132|00:03:01|+0.97|0.97|0.0129| | |
| 53 | |-| 5| 3| 20| 23311852.00| 165|00:02:56|+0.10|0.28|0.5572| | |
| 54 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 55 | Best-so-far configuration: 20 mean value: 23311852.00 | |
| 56 | Description of the best-so-far configuration: | |
| 57 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 58 | 20 20 acs 3 4.5477 4.5539 0.3696 8 45 0.2489 1 NA NA NA | |
| 59 | ||
| 60 | # 2020-03-27 13:15:34 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 46 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 47 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 48 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 49 | |x| 1| 33| 14| 23436382.00| 33|00:02:51| NA| NA| NA| | |
| 50 | |x| 2| 33| 11| 23376711.50| 66|00:02:52|+0.94|0.97|0.0066| | |
| 51 | |x| 3| 33| 14| 23336841.67| 99|00:02:51|+0.91|0.94|0.0146| | |
| 52 | |x| 4| 33| 14| 23278031.50| 132|00:02:51|+0.93|0.94|0.0127| | |
| 53 | |-| 5| 4| 14| 23207001.40| 165|00:02:51|-0.08|0.14|0.8158| | |
| 54 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 55 | Best-so-far configuration: 14 mean value: 23207001.40 | |
| 56 | Description of the best-so-far configuration: | |
| 57 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 58 | 14 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA NA | |
| 59 | ||
| 60 | # 2022-10-22 10:18:52 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 61 | 61 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants |
| 62 | 20 acs 3 4.5477 4.5539 0.3696 8 45 0.2489 1 NA NA | |
| 63 | 3 acs 2 2.3623 2.1143 0.0570 7 8 0.6042 0 NA NA | |
| 64 | 29 ras 3 3.2195 1.2291 0.0466 60 20 NA 1 71 NA | |
| 65 | # 2020-03-27 13:15:34 GMT: Iteration 2 of 5 | |
| 62 | 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA | |
| 63 | 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA | |
| 64 | 31 acs 1 2.3896 7.8790 0.3748 34 36 0.9029 1 NA NA | |
| 65 | 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA | |
| 66 | # 2022-10-22 10:18:52 BST: Iteration 2 of 5 | |
| 66 | 67 | # experimentsUsedSoFar: 165 |
| 67 | 68 | # remainingBudget: 835 |
| 68 | 69 | # currentBudget: 208 |
| 69 | # nbConfigurations: 31 | |
| 70 | # Markers: | |
| 71 | x No test is performed. | |
| 72 | c Configurations are discarded only due to capping. | |
| 73 | - The test is performed and some configurations are discarded. | |
| 74 | = The test is performed but no configuration is discarded. | |
| 75 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 76 | . All alive configurations are elite and nothing is discarded | |
| 77 | ||
| 78 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 79 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 80 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 81 | |x| 6| 31| 46| 22862976.00| 31|00:02:42| NA| NA| NA| | |
| 82 | |x| 2| 31| 46| 23034699.50| 59|00:02:22|+0.83|0.92|0.0102| | |
| 83 | |x| 3| 31| 39| 23205201.33| 87|00:02:21|+0.84|0.89|0.0073| | |
| 84 | |x| 1| 31| 39| 23218554.50| 115|00:02:21|+0.87|0.90|0.0061| | |
| 85 | |-| 4| 6| 39| 23177444.20| 143|00:02:21|+0.60|0.68|0.3842| | |
| 86 | |-| 5| 2| 46| 23191355.17| 146|00:00:20|-0.07|0.11|0.5333| | |
| 87 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 88 | Best-so-far configuration: 46 mean value: 23191355.17 | |
| 89 | Description of the best-so-far configuration: | |
| 90 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 91 | 46 46 acs 3 2.785 4.3699 0.3523 53 18 0.0134 1 NA NA 29 | |
| 92 | ||
| 93 | # 2020-03-27 13:28:03 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 70 | # nbConfigurations: 32 | |
| 71 | # Markers: | |
| 72 | x No test is performed. | |
| 73 | c Configurations are discarded only due to capping. | |
| 74 | - The test is performed and some configurations are discarded. | |
| 75 | = The test is performed but no configuration is discarded. | |
| 76 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 77 | . All alive configurations are elite and nothing is discarded | |
| 78 | ||
| 79 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 80 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 81 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 82 | |x| 6| 32| 6| 23326585.00| 32|00:02:41| NA| NA| NA| | |
| 83 | |x| 2| 32| 11| 23327784.00| 60|00:02:21|+0.94|0.97|0.0053| | |
| 84 | |x| 5| 32| 55| 23179654.00| 88|00:02:21|+0.93|0.96|0.0065| | |
| 85 | |x| 3| 32| 55| 23157937.50| 116|00:02:21|+0.92|0.94|0.0068| | |
| 86 | |-| 1| 10| 55| 23231073.00| 144|00:02:21|-0.10|0.12|1.0265| | |
| 87 | |=| 4| 10| 55| 23195411.50| 150|00:00:30|+0.02|0.18|0.9224| | |
| 88 | |=| 7| 10| 57| 23198073.14| 160|00:00:50|+0.04|0.18|0.9016| | |
| 89 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 90 | Best-so-far configuration: 57 mean value: 23198073.14 | |
| 91 | Description of the best-so-far configuration: | |
| 92 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 93 | 57 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA 11 | |
| 94 | ||
| 95 | # 2022-10-22 10:32:21 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 94 | 96 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants |
| 95 | 46 acs 3 2.7850 4.3699 0.3523 53 18 0.0134 1 NA NA | |
| 96 | 39 eas 3 2.1783 6.4126 0.1220 24 20 NA 1 NA 485 | |
| 97 | # 2020-03-27 13:28:03 GMT: Iteration 3 of 5 | |
| 98 | # experimentsUsedSoFar: 311 | |
| 99 | # remainingBudget: 689 | |
| 100 | # currentBudget: 229 | |
| 101 | # nbConfigurations: 30 | |
| 102 | # Markers: | |
| 103 | x No test is performed. | |
| 104 | c Configurations are discarded only due to capping. | |
| 105 | - The test is performed and some configurations are discarded. | |
| 106 | = The test is performed but no configuration is discarded. | |
| 107 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 108 | . All alive configurations are elite and nothing is discarded | |
| 109 | ||
| 110 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 111 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 112 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 113 | |x| 7| 30| 46| 23323269.00| 30|00:02:32| NA| NA| NA| | |
| 114 | |x| 6| 30| 46| 23093122.50| 58|00:02:21|+0.83|0.91|0.0076| | |
| 115 | |x| 3| 30| 46| 23223770.00| 86|00:02:22|+0.79|0.86|0.0098| | |
| 116 | |x| 2| 30| 73| 23193591.50| 114|00:02:21|+0.82|0.86|0.0086| | |
| 117 | |-| 4| 7| 73| 23147713.80| 142|00:02:22|-0.03|0.18|0.8460| | |
| 118 | |=| 5| 7| 73| 23170297.17| 147|00:00:30|-0.02|0.15|0.8472| | |
| 119 | |=| 1| 7| 73| 23178456.00| 152|00:00:30|+0.01|0.15|0.8010| | |
| 120 | |=| 8| 7| 73| 23169326.25| 159|00:00:40|+0.01|0.13|0.8029| | |
| 121 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 122 | Best-so-far configuration: 73 mean value: 23169326.25 | |
| 123 | Description of the best-so-far configuration: | |
| 124 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 125 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 126 | ||
| 127 | # 2020-03-27 13:41:45 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 97 | 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA | |
| 98 | 55 mmas 3 1.0832 5.5967 0.9317 18 46 NA 0 NA NA | |
| 99 | 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA | |
| 100 | 39 acs 2 2.5486 3.7878 0.0262 13 12 0.7621 0 NA NA | |
| 101 | 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA | |
| 102 | # 2022-10-22 10:32:21 BST: Iteration 3 of 5 | |
| 103 | # experimentsUsedSoFar: 325 | |
| 104 | # remainingBudget: 675 | |
| 105 | # currentBudget: 225 | |
| 106 | # nbConfigurations: 32 | |
| 107 | # Markers: | |
| 108 | x No test is performed. | |
| 109 | c Configurations are discarded only due to capping. | |
| 110 | - The test is performed and some configurations are discarded. | |
| 111 | = The test is performed but no configuration is discarded. | |
| 112 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 113 | . All alive configurations are elite and nothing is discarded | |
| 114 | ||
| 115 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 116 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 117 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 118 | |x| 8| 32| 62| 23338867.00| 32|00:02:41| NA| NA| NA| | |
| 119 | |x| 1| 32| 62| 23375111.50| 59|00:02:21|+0.94|0.97|0.0075| | |
| 120 | |x| 5| 32| 62| 23196573.67| 86|00:02:21|+0.95|0.97|0.0068| | |
| 121 | |x| 6| 32| 78| 23240373.50| 113|00:02:21|+0.82|0.87|0.0165| | |
| 122 | |-| 4| 10| 78| 23206857.00| 140|00:02:21|+0.23|0.38|0.7064| | |
| 123 | |!| 3| 10| 62| 23192587.50| 145|00:00:30|+0.20|0.33|0.7410| | |
| 124 | |!| 2| 10| 78| 23195256.86| 150|00:00:30|+0.24|0.35|0.6748| | |
| 125 | |-| 7| 5| 78| 23193481.75| 155|00:00:30|-0.08|0.06|0.8453| | |
| 126 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 127 | Best-so-far configuration: 78 mean value: 23193481.75 | |
| 128 | Description of the best-so-far configuration: | |
| 129 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 130 | 78 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 57 | |
| 131 | ||
| 132 | # 2022-10-22 10:45:59 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 128 | 133 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants |
| 129 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 130 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 131 | 80 acs 3 3.0760 2.7653 0.8385 66 19 0.6784 1 NA NA | |
| 132 | 46 acs 3 2.7850 4.3699 0.3523 53 18 0.0134 1 NA NA | |
| 133 | 74 acs 3 4.4845 0.2285 0.6031 25 28 0.3753 1 NA NA | |
| 134 | # 2020-03-27 13:41:45 GMT: Iteration 4 of 5 | |
| 135 | # experimentsUsedSoFar: 470 | |
| 136 | # remainingBudget: 530 | |
| 137 | # currentBudget: 265 | |
| 134 | 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA | |
| 135 | 62 mmas 3 1.2260 2.0146 0.7659 8 16 NA 0 NA NA | |
| 136 | 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA | |
| 137 | 71 acs 3 1.7574 6.9647 0.6223 13 19 0.4806 0 NA NA | |
| 138 | 72 acs 3 2.0489 7.1428 0.5326 16 6 0.2179 0 NA NA | |
| 139 | # 2022-10-22 10:45:59 BST: Iteration 4 of 5 | |
| 140 | # experimentsUsedSoFar: 480 | |
| 141 | # remainingBudget: 520 | |
| 142 | # currentBudget: 260 | |
| 138 | 143 | # nbConfigurations: 33 |
| 139 | 144 | # Markers: |
| 140 | 145 | x No test is performed. |
| 144 | 149 | ! The test is performed and configurations could be discarded but elite configurations are preserved. |
| 145 | 150 | . All alive configurations are elite and nothing is discarded |
| 146 | 151 | |
| 147 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 148 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 149 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 150 | |x| 9| 33| 96| 23120160.00| 33|00:02:53| NA| NA| NA| | |
| 151 | |x| 8| 33| 100| 23109761.50| 61|00:02:21|+0.85|0.92|0.0020| | |
| 152 | |x| 1| 33| 98| 23153124.67| 89|00:02:21|+0.87|0.92|0.0016| | |
| 153 | |x| 7| 33| 73| 23207028.25| 117|00:02:21|+0.86|0.89|0.0022| | |
| 154 | |-| 5| 14| 96| 23213596.20| 145|00:02:21|-0.09|0.12|1.0140| | |
| 155 | |=| 2| 14| 73| 23212858.67| 154|00:00:50|-0.02|0.15|0.9322| | |
| 156 | |=| 6| 14| 96| 23162964.14| 163|00:00:50|-0.02|0.13|0.9370| | |
| 157 | |=| 4| 14| 73| 23143564.38| 172|00:00:50|-0.00|0.12|0.9160| | |
| 158 | |=| 3| 14| 73| 23167587.67| 181|00:00:50|+0.02|0.13|0.8881| | |
| 159 | |=| 10| 14| 73| 23160000.70| 195|00:01:10|+0.04|0.14|0.8792| | |
| 160 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 161 | Best-so-far configuration: 73 mean value: 23160000.70 | |
| 162 | Description of the best-so-far configuration: | |
| 163 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 164 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 165 | ||
| 166 | # 2020-03-27 13:58:39 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 167 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 168 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 169 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 170 | 80 acs 3 3.0760 2.7653 0.8385 66 19 0.6784 1 NA NA | |
| 171 | 100 acs 3 1.6705 3.5003 0.6537 43 17 0.0799 1 NA NA | |
| 172 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 173 | # 2020-03-27 13:58:39 GMT: Iteration 5 of 5 | |
| 174 | # experimentsUsedSoFar: 665 | |
| 175 | # remainingBudget: 335 | |
| 176 | # currentBudget: 335 | |
| 177 | # nbConfigurations: 35 | |
| 178 | # Markers: | |
| 179 | x No test is performed. | |
| 180 | c Configurations are discarded only due to capping. | |
| 181 | - The test is performed and some configurations are discarded. | |
| 182 | = The test is performed but no configuration is discarded. | |
| 183 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 184 | . All alive configurations are elite and nothing is discarded | |
| 185 | ||
| 186 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 187 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 188 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 189 | |x| 11| 35| 96| 23207142.00| 35|00:03:02| NA| NA| NA| | |
| 190 | |x| 5| 35| 96| 23241093.50| 65|00:02:31|+0.86|0.93|0.0405| | |
| 191 | |x| 7| 35| 73| 23287271.00| 95|00:02:31|+0.78|0.85|0.0807| | |
| 192 | |x| 10| 35| 73| 23238382.75| 125|00:02:32|+0.75|0.82|0.0691| | |
| 193 | |-| 1| 13| 73| 23236188.00| 155|00:02:33|-0.06|0.15|1.0198| | |
| 194 | |=| 6| 13| 96| 23168875.00| 163|00:00:40|-0.06|0.11|0.9995| | |
| 195 | |=| 2| 13| 96| 23175237.43| 171|00:00:40|-0.03|0.12|0.9585| | |
| 196 | |=| 3| 13| 73| 23201712.38| 179|00:00:40|-0.02|0.11|0.9243| | |
| 197 | |=| 9| 13| 73| 23196375.33| 187|00:00:40|+0.00|0.11|0.8977| | |
| 198 | |=| 4| 13| 73| 23173158.10| 195|00:00:40|-0.01|0.09|0.9054| | |
| 199 | |=| 8| 13| 73| 23166999.91| 203|00:00:40|-0.02|0.07|0.9106| | |
| 200 | |=| 12| 13| 73| 23179586.75| 216|00:01:10|+0.00|0.09|0.8827| | |
| 201 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 202 | Best-so-far configuration: 73 mean value: 23179586.75 | |
| 203 | Description of the best-so-far configuration: | |
| 204 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 205 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 206 | ||
| 207 | # 2020-03-27 14:17:06 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 208 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 209 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 210 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 211 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 212 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 213 | 80 acs 3 3.0760 2.7653 0.8385 66 19 0.6784 1 NA NA | |
| 214 | # 2020-03-27 14:17:06 GMT: Iteration 6 of 6 | |
| 215 | # experimentsUsedSoFar: 881 | |
| 216 | # remainingBudget: 119 | |
| 217 | # currentBudget: 119 | |
| 218 | # nbConfigurations: 13 | |
| 219 | # Markers: | |
| 220 | x No test is performed. | |
| 221 | c Configurations are discarded only due to capping. | |
| 222 | - The test is performed and some configurations are discarded. | |
| 223 | = The test is performed but no configuration is discarded. | |
| 224 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 225 | . All alive configurations are elite and nothing is discarded | |
| 226 | ||
| 227 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 228 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 229 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 230 | |x| 13| 13| 84| 22844704.00| 13|00:01:11| NA| NA| NA| | |
| 231 | |x| 11| 13| 153| 23033848.50| 21|00:00:41|+0.25|0.62|0.4405| | |
| 232 | |x| 8| 13| 153| 23046986.67| 29|00:00:40|+0.34|0.56|0.3481| | |
| 233 | |x| 10| 13| 153| 23060902.25| 37|00:00:40|+0.46|0.60|0.3030| | |
| 234 | |-| 3| 6| 84| 23136069.20| 45|00:00:40|-0.05|0.16|0.8665| | |
| 235 | |=| 12| 6| 84| 23156266.67| 46|00:00:10|+0.01|0.17|0.8645| | |
| 236 | |=| 5| 6| 84| 23179812.29| 47|00:00:10|-0.07|0.08|0.9204| | |
| 237 | |=| 2| 6| 84| 23181538.88| 48|00:00:10|-0.04|0.09|0.8945| | |
| 238 | |=| 7| 6| 84| 23200454.11| 49|00:00:10|-0.04|0.07|0.8839| | |
| 239 | |=| 6| 6| 84| 23170620.10| 50|00:00:10|-0.05|0.05|0.8825| | |
| 240 | |=| 9| 6| 73| 23165207.64| 51|00:00:10|-0.07|0.03|0.8916| | |
| 241 | |=| 1| 6| 84| 23177602.67| 52|00:00:10|-0.06|0.03|0.8871| | |
| 242 | |=| 4| 6| 73| 23154530.46| 53|00:00:10|-0.06|0.02|0.8810| | |
| 243 | |=| 14| 6| 73| 23166941.57| 59|00:00:30|-0.05|0.02|0.8779| | |
| 244 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 245 | Best-so-far configuration: 73 mean value: 23166941.57 | |
| 246 | Description of the best-so-far configuration: | |
| 247 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 248 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 249 | ||
| 250 | # 2020-03-27 14:22:51 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 251 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 252 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 253 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 254 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 255 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 256 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 257 | # 2020-03-27 14:22:52 GMT: Iteration 7 of 7 | |
| 258 | # experimentsUsedSoFar: 940 | |
| 259 | # remainingBudget: 60 | |
| 260 | # currentBudget: 60 | |
| 152 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 153 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 154 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 155 | |x| 9| 33| 71| 23073306.00| 33|00:02:51| NA| NA| NA| | |
| 156 | |x| 7| 33| 78| 23137315.50| 61|00:02:21|+0.74|0.87|0.0148| | |
| 157 | |x| 6| 33| 78| 23196511.00| 89|00:02:21|+0.75|0.83|0.0236| | |
| 158 | |x| 1| 33| 78| 23254301.25| 117|00:02:22|+0.76|0.82|0.0233| | |
| 159 | |-| 3| 14| 113| 23225975.00| 145|00:02:21|-0.02|0.18|0.9419| | |
| 160 | |=| 8| 14| 113| 23242633.33| 154|00:00:50|+0.01|0.18|0.9115| | |
| 161 | |=| 4| 14| 113| 23210932.71| 163|00:00:50|-0.03|0.12|0.9145| | |
| 162 | |=| 2| 14| 113| 23212340.75| 172|00:00:50|+0.03|0.15|0.8593| | |
| 163 | |=| 5| 14| 113| 23174747.00| 181|00:00:50|+0.04|0.15|0.8558| | |
| 164 | |=| 10| 14| 78| 23180941.50| 195|00:01:10|+0.04|0.13|0.8711| | |
| 165 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 166 | Best-so-far configuration: 78 mean value: 23180941.50 | |
| 167 | Description of the best-so-far configuration: | |
| 168 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 169 | 78 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA 57 | |
| 170 | ||
| 171 | # 2022-10-22 11:02:50 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 172 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 173 | 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA | |
| 174 | 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA | |
| 175 | 110 mmas 3 1.1100 2.0382 0.6833 15 11 NA 0 NA NA | |
| 176 | 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA | |
| 177 | 91 acs 3 1.5186 5.7143 0.5523 14 26 0.8361 1 NA NA | |
| 178 | # 2022-10-22 11:02:50 BST: Iteration 5 of 5 | |
| 179 | # experimentsUsedSoFar: 675 | |
| 180 | # remainingBudget: 325 | |
| 181 | # currentBudget: 325 | |
| 182 | # nbConfigurations: 34 | |
| 183 | # Markers: | |
| 184 | x No test is performed. | |
| 185 | c Configurations are discarded only due to capping. | |
| 186 | - The test is performed and some configurations are discarded. | |
| 187 | = The test is performed but no configuration is discarded. | |
| 188 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 189 | . All alive configurations are elite and nothing is discarded | |
| 190 | ||
| 191 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 192 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 193 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 194 | |x| 11| 34| 138| 23258118.00| 34|00:02:51| NA| NA| NA| | |
| 195 | |x| 4| 34| 138| 23112149.00| 63|00:02:31|+0.42|0.71|0.5375| | |
| 196 | |x| 7| 34| 127| 23142940.33| 92|00:02:31|+0.55|0.70|0.3724| | |
| 197 | |x| 6| 34| 127| 23173748.75| 121|00:02:31|+0.51|0.63|0.3385| | |
| 198 | |-| 8| 11| 127| 23202123.40| 150|00:02:31|+0.17|0.34|0.7210| | |
| 199 | |=| 3| 11| 127| 23177660.67| 156|00:00:30|+0.12|0.26|0.7576| | |
| 200 | |=| 1| 11| 127| 23216506.14| 162|00:00:30|+0.02|0.16|0.8691| | |
| 201 | |=| 10| 11| 127| 23212485.75| 168|00:00:30|-0.01|0.11|0.8993| | |
| 202 | |=| 5| 11| 127| 23171591.78| 174|00:00:30|-0.02|0.10|0.9045| | |
| 203 | |=| 9| 11| 127| 23162579.40| 180|00:00:30|+0.02|0.11|0.8709| | |
| 204 | |=| 2| 11| 127| 23171928.36| 186|00:00:30|-0.00|0.09|0.8872| | |
| 205 | |=| 12| 11| 138| 23170655.33| 197|00:01:00|+0.02|0.10|0.8676| | |
| 206 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 207 | Best-so-far configuration: 138 mean value: 23170655.33 | |
| 208 | Description of the best-so-far configuration: | |
| 209 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 210 | 138 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 78 | |
| 211 | ||
| 212 | # 2022-10-22 11:19:51 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 213 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 214 | 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA | |
| 215 | 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA | |
| 216 | 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA | |
| 217 | 124 acs 3 1.6169 7.9049 0.7166 20 8 0.2282 1 NA NA | |
| 218 | 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA | |
| 219 | # 2022-10-22 11:19:51 BST: Iteration 6 of 6 | |
| 220 | # experimentsUsedSoFar: 872 | |
| 221 | # remainingBudget: 128 | |
| 222 | # currentBudget: 128 | |
| 223 | # nbConfigurations: 14 | |
| 224 | # Markers: | |
| 225 | x No test is performed. | |
| 226 | c Configurations are discarded only due to capping. | |
| 227 | - The test is performed and some configurations are discarded. | |
| 228 | = The test is performed but no configuration is discarded. | |
| 229 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 230 | . All alive configurations are elite and nothing is discarded | |
| 231 | ||
| 232 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 233 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 234 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 235 | |x| 13| 14| 149| 23168071.00| 14|00:01:10| NA| NA| NA| | |
| 236 | |x| 9| 14| 138| 23147346.50| 23|00:00:50|+0.28|0.64|0.2294| | |
| 237 | |x| 2| 14| 138| 23168709.00| 32|00:00:50|+0.33|0.56|0.1781| | |
| 238 | |x| 1| 14| 138| 23222390.50| 41|00:00:50|+0.35|0.51|0.1570| | |
| 239 | |-| 11| 7| 138| 23229536.00| 50|00:00:50|+0.20|0.36|0.8842| | |
| 240 | |=| 5| 7| 138| 23175555.00| 52|00:00:10|+0.08|0.23|0.8989| | |
| 241 | |=| 8| 7| 138| 23200975.71| 54|00:00:10|+0.02|0.16|0.8807| | |
| 242 | |=| 4| 7| 138| 23171626.25| 56|00:00:10|+0.10|0.21|0.8201| | |
| 243 | |=| 3| 7| 120| 23156247.00| 58|00:00:10|+0.08|0.18|0.8132| | |
| 244 | |=| 10| 7| 138| 23163756.60| 60|00:00:10|+0.04|0.14|0.8714| | |
| 245 | |=| 7| 7| 138| 23169372.45| 62|00:00:10|+0.03|0.11|0.8612| | |
| 246 | |=| 12| 7| 138| 23163537.83| 64|00:00:10|+0.04|0.12|0.8393| | |
| 247 | |=| 6| 7| 138| 23174511.15| 66|00:00:10|-0.01|0.07|0.8624| | |
| 248 | |=| 14| 7| 138| 23178603.86| 73|00:00:40|-0.02|0.05|0.8731| | |
| 249 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 250 | Best-so-far configuration: 138 mean value: 23178603.86 | |
| 251 | Description of the best-so-far configuration: | |
| 252 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 253 | 138 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA 78 | |
| 254 | ||
| 255 | # 2022-10-22 11:26:25 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 256 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 257 | 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA | |
| 258 | 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA | |
| 259 | 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA | |
| 260 | 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA | |
| 261 | 147 acs 3 1.3012 6.1776 0.7881 22 10 0.1276 1 NA NA | |
| 262 | # 2022-10-22 11:26:25 BST: Iteration 7 of 7 | |
| 263 | # experimentsUsedSoFar: 945 | |
| 264 | # remainingBudget: 55 | |
| 265 | # currentBudget: 55 | |
| 261 | 266 | # nbConfigurations: 8 |
| 262 | 267 | # Markers: |
| 263 | 268 | x No test is performed. |
| 267 | 272 | ! The test is performed and configurations could be discarded but elite configurations are preserved. |
| 268 | 273 | . All alive configurations are elite and nothing is discarded |
| 269 | 274 | |
| 270 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 271 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 272 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 273 | |x| 15| 8| 153| 23127030.00| 8|00:00:40| NA| NA| NA| | |
| 274 | |x| 8| 8| 153| 23100146.50| 11|00:00:20|+0.45|0.73|0.5326| | |
| 275 | |x| 7| 8| 153| 23178547.33| 14|00:00:20|+0.58|0.72|0.3910| | |
| 276 | |x| 4| 8| 153| 23137335.50| 17|00:00:20|+0.49|0.62|0.4480| | |
| 277 | |-| 2| 5| 73| 23143737.00| 20|00:00:20|+0.03|0.22|0.7919| | |
| 278 | |.| 10| 5| 73| 23135067.17| 20|00:00:00|+0.05|0.21|0.7855| | |
| 279 | |.| 3| 5| 73| 23167168.14| 20|00:00:00|+0.11|0.24|0.7414| | |
| 280 | |.| 9| 5| 73| 23165482.00| 20|00:00:00|+0.01|0.13|0.7875| | |
| 281 | |.| 13| 5| 73| 23130856.78| 20|00:00:00|+0.04|0.14|0.7526| | |
| 282 | |.| 1| 5| 73| 23140512.00| 20|00:00:00|-0.01|0.09|0.7927| | |
| 283 | |.| 11| 5| 73| 23149282.91| 20|00:00:00|-0.03|0.06|0.8071| | |
| 284 | |.| 5| 5| 73| 23160443.83| 20|00:00:00|-0.04|0.05|0.8124| | |
| 285 | |.| 6| 5| 73| 23140960.46| 20|00:00:00|-0.06|0.02|0.8236| | |
| 286 | |.| 14| 5| 73| 23154340.86| 20|00:00:00|-0.05|0.03|0.8149| | |
| 287 | |.| 12| 5| 73| 23165254.27| 20|00:00:00|-0.06|0.01|0.8340| | |
| 288 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 289 | Best-so-far configuration: 73 mean value: 23165254.27 | |
| 290 | Description of the best-so-far configuration: | |
| 291 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 292 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 293 | ||
| 294 | # 2020-03-27 14:24:53 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 295 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 296 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 297 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 298 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 299 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 300 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 301 | # 2020-03-27 14:24:53 GMT: Iteration 8 of 8 | |
| 302 | # experimentsUsedSoFar: 960 | |
| 303 | # remainingBudget: 40 | |
| 304 | # currentBudget: 40 | |
| 305 | # nbConfigurations: 7 | |
| 306 | # Markers: | |
| 307 | x No test is performed. | |
| 308 | c Configurations are discarded only due to capping. | |
| 309 | - The test is performed and some configurations are discarded. | |
| 310 | = The test is performed but no configuration is discarded. | |
| 311 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 312 | . All alive configurations are elite and nothing is discarded | |
| 313 | ||
| 314 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 315 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 316 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 317 | |x| 16| 7| 119| 23194435.00| 7|00:00:40| NA| NA| NA| | |
| 318 | |x| 4| 7| 73| 23082541.50| 9|00:00:10|+0.75|0.88|0.1412| | |
| 319 | |x| 13| 7| 119| 23004955.67| 11|00:00:10|+0.65|0.77|0.2227| | |
| 320 | |x| 12| 7| 119| 23073183.00| 13|00:00:10|+0.42|0.56|0.4463| | |
| 321 | |-| 7| 5| 119| 23132915.20| 15|00:00:10|+0.06|0.25|0.7478| | |
| 322 | |.| 15| 5| 119| 23136705.00| 15|00:00:00|+0.03|0.19|0.7587| | |
| 323 | |.| 2| 5| 119| 23149094.29| 15|00:00:00|+0.01|0.16|0.7786| | |
| 324 | |.| 9| 5| 119| 23150013.12| 15|00:00:00|-0.04|0.09|0.8037| | |
| 325 | |.| 14| 5| 119| 23170012.78| 15|00:00:00|-0.06|0.06|0.8104| | |
| 326 | |.| 8| 5| 119| 23159085.00| 15|00:00:00|-0.05|0.06|0.7956| | |
| 327 | |.| 11| 5| 119| 23169075.73| 15|00:00:00|-0.06|0.04|0.8042| | |
| 328 | |.| 3| 5| 119| 23186836.08| 15|00:00:00|-0.03|0.05|0.7964| | |
| 329 | |.| 1| 5| 119| 23191912.00| 15|00:00:00|-0.04|0.04|0.8047| | |
| 330 | |.| 6| 5| 119| 23165664.86| 15|00:00:00|-0.05|0.02|0.8043| | |
| 331 | |.| 10| 5| 119| 23160432.07| 15|00:00:00|-0.05|0.02|0.8170| | |
| 332 | |.| 5| 5| 119| 23169025.69| 15|00:00:00|-0.05|0.02|0.8148| | |
| 333 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 334 | Best-so-far configuration: 73 mean value: 23167480.88 | |
| 335 | Description of the best-so-far configuration: | |
| 336 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 337 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 338 | ||
| 339 | # 2020-03-27 14:26:14 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 340 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 341 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 342 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 343 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 344 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 345 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 346 | # 2020-03-27 14:26:15 GMT: Iteration 9 of 9 | |
| 347 | # experimentsUsedSoFar: 975 | |
| 348 | # remainingBudget: 25 | |
| 349 | # currentBudget: 25 | |
| 350 | # nbConfigurations: 6 | |
| 351 | # Markers: | |
| 352 | x No test is performed. | |
| 353 | c Configurations are discarded only due to capping. | |
| 354 | - The test is performed and some configurations are discarded. | |
| 355 | = The test is performed but no configuration is discarded. | |
| 356 | ! The test is performed and configurations could be discarded but elite configurations are preserved. | |
| 357 | . All alive configurations are elite and nothing is discarded | |
| 358 | ||
| 359 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 360 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 361 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 362 | |x| 17| 6| 119| 23372808.00| 6|00:00:30| NA| NA| NA| | |
| 363 | |x| 6| 6| 119| 23098630.00| 7|00:00:10|+0.37|0.69|0.5977| | |
| 364 | |x| 15| 6| 119| 23117638.00| 8|00:00:10|+0.03|0.35|0.6849| | |
| 365 | |x| 2| 6| 73| 23149018.50| 9|00:00:10|-0.11|0.16|0.8444| | |
| 366 | |=| 9| 6| 96| 23145692.00| 10|00:00:10|-0.04|0.17|0.7576| | |
| 367 | |=| 10| 6| 73| 23140245.17| 11|00:00:10|-0.13|0.06|0.8796| | |
| 368 | |=| 1| 6| 96| 23148626.71| 12|00:00:10|-0.13|0.03|0.8959| | |
| 369 | |=| 3| 6| 73| 23178581.75| 13|00:00:10|-0.09|0.05|0.8483| | |
| 370 | |=| 12| 6| 73| 23194077.33| 14|00:00:10|-0.08|0.04|0.8562| | |
| 371 | |=| 14| 6| 96| 23205962.30| 15|00:00:10|-0.06|0.04|0.8402| | |
| 372 | |=| 7| 6| 73| 23219689.91| 16|00:00:10|-0.05|0.04|0.8279| | |
| 373 | |=| 4| 6| 73| 23198399.33| 17|00:00:10|-0.05|0.04|0.8286| | |
| 374 | |=| 16| 6| 73| 23198590.15| 18|00:00:10|-0.02|0.05|0.8003| | |
| 375 | |=| 13| 6| 73| 23173966.21| 19|00:00:10|-0.01|0.06|0.7968| | |
| 376 | |=| 11| 6| 73| 23178167.93| 20|00:00:10|-0.01|0.06|0.7926| | |
| 377 | |=| 8| 6| 73| 23173621.06| 21|00:00:10|-0.02|0.05|0.8028| | |
| 378 | |=| 5| 6| 73| 23180067.71| 22|00:00:10|-0.01|0.05|0.8044| | |
| 379 | +-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ | |
| 380 | Best-so-far configuration: 73 mean value: 23180067.71 | |
| 381 | Description of the best-so-far configuration: | |
| 382 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 383 | 73 73 acs 3 1.4601 7.6777 0.358 17 18 0.2046 1 NA NA 39 | |
| 384 | ||
| 385 | # 2020-03-27 14:29:27 GMT: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 386 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 387 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 388 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 389 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 390 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 391 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 392 | # 2020-03-27 14:29:27 GMT: Stopped because there is not enough budget left to race more than the minimum (5) | |
| 275 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 276 | | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| | |
| 277 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 278 | |x| 15| 8| 120| 23314657.00| 8|00:00:40| NA| NA| NA| | |
| 279 | |x| 12| 8| 127| 23220366.50| 11|00:00:20|+0.24|0.62|0.3659| | |
| 280 | |x| 5| 8| 127| 23095057.67| 14|00:00:20|+0.40|0.60|0.2741| | |
| 281 | |x| 2| 8| 155| 23128548.75| 17|00:00:20|+0.40|0.55|0.2772| | |
| 282 | |-| 1| 6| 155| 23176037.40| 20|00:00:20|+0.15|0.32|0.7974| | |
| 283 | |=| 9| 6| 155| 23163208.83| 21|00:00:10|+0.02|0.18|0.8719| | |
| 284 | |=| 13| 6| 155| 23167865.00| 22|00:00:10|+0.00|0.14|0.8751| | |
| 285 | |=| 4| 6| 155| 23146280.75| 23|00:00:10|+0.02|0.15|0.8457| | |
| 286 | |=| 3| 6| 155| 23137118.56| 24|00:00:10|+0.02|0.13|0.8488| | |
| 287 | |=| 14| 6| 155| 23145861.20| 25|00:00:10|-0.01|0.09|0.8678| | |
| 288 | |=| 7| 6| 155| 23150676.36| 26|00:00:10|-0.01|0.08|0.8466| | |
| 289 | |=| 10| 6| 155| 23149829.50| 27|00:00:10|+0.00|0.09|0.8466| | |
| 290 | |=| 11| 6| 155| 23158077.23| 28|00:00:10|+0.04|0.11|0.8411| | |
| 291 | |=| 6| 6| 155| 23166610.93| 29|00:00:10|+0.04|0.11|0.8286| | |
| 292 | |=| 8| 6| 155| 23180093.40| 30|00:00:10|+0.03|0.09|0.8325| | |
| 293 | |=| 16| 6| 155| 23192615.19| 36|00:00:30|+0.05|0.11|0.8122| | |
| 294 | +-+-----------+-----------+-----------+----------------+-----------+--------+-----+----+------+ | |
| 295 | Best-so-far configuration: 155 mean value: 23192615.19 | |
| 296 | Description of the best-so-far configuration: | |
| 297 | .ID. algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants .PARENT. | |
| 298 | 155 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA 120 | |
| 299 | ||
| 300 | # 2022-10-22 11:30:38 BST: Elite configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): | |
| 301 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 302 | 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA | |
| 303 | 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA | |
| 304 | 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA | |
| 305 | 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA | |
| 306 | 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA | |
| 307 | # 2022-10-22 11:30:38 BST: Stopped because there is not enough budget left to race more than the minimum (5) | |
| 393 | 308 | # You may either increase the budget or set 'minNbSurvival' to a lower value |
| 394 | # Iteration: 10 | |
| 395 | # nbIterations: 10 | |
| 396 | # experimentsUsedSoFar: 997 | |
| 309 | # Iteration: 8 | |
| 310 | # nbIterations: 8 | |
| 311 | # experimentsUsedSoFar: 981 | |
| 397 | 312 | # timeUsed: 0 |
| 398 | # remainingBudget: 3 | |
| 399 | # currentBudget: 3 | |
| 313 | # remainingBudget: 19 | |
| 314 | # currentBudget: 19 | |
| 400 | 315 | # number of elites: 5 |
| 401 | # nbConfigurations: 4 | |
| 316 | # nbConfigurations: 5 | |
| 317 | # Total CPU user time: 9871.301, CPU sys time: 46.151, Wall-clock time: 5165.205 | |
| 402 | 318 | # Best configurations (first number is the configuration ID; listed from best to worst according to the sum of ranks): |
| 403 | 319 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants |
| 404 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 405 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 406 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 407 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 408 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 320 | 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA | |
| 321 | 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA | |
| 322 | 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA | |
| 323 | 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA | |
| 324 | 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA | |
| 409 | 325 | # Best configurations as commandlines (first number is the configuration ID; same order as above): |
| 410 | 73 --acs --localsearch 3 --alpha 1.4601 --beta 7.6777 --rho 0.358 --ants 17 --nnls 18 --q0 0.2046 --dlb 1 | |
| 411 | 119 --acs --localsearch 3 --alpha 1.5931 --beta 5.8926 --rho 0.6227 --ants 37 --nnls 18 --q0 0.0123 --dlb 1 | |
| 412 | 96 --acs --localsearch 3 --alpha 2.5594 --beta 6.9138 --rho 0.5257 --ants 12 --nnls 24 --q0 0.1014 --dlb 1 | |
| 413 | 153 --acs --localsearch 3 --alpha 2.1087 --beta 5.2236 --rho 0.5776 --ants 49 --nnls 14 --q0 0.1293 --dlb 1 | |
| 414 | 84 --acs --localsearch 3 --alpha 2.6558 --beta 1.3544 --rho 0.034 --ants 22 --nnls 15 --q0 0.5287 --dlb 1 | |
| 415 | ||
| 416 | ||
| 417 | # 2020-03-27 14:29:27 GMT: Testing configurations (in no particular order): 20 46 73 119 96 153 84 | |
| 418 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 419 | 20 acs 3 4.5477 4.5539 0.3696 8 45 0.2489 1 NA NA | |
| 420 | 46 acs 3 2.7850 4.3699 0.3523 53 18 0.0134 1 NA NA | |
| 421 | 73 acs 3 1.4601 7.6777 0.3580 17 18 0.2046 1 NA NA | |
| 422 | 119 acs 3 1.5931 5.8926 0.6227 37 18 0.0123 1 NA NA | |
| 423 | 96 acs 3 2.5594 6.9138 0.5257 12 24 0.1014 1 NA NA | |
| 424 | 153 acs 3 2.1087 5.2236 0.5776 49 14 0.1293 1 NA NA | |
| 425 | 84 acs 3 2.6558 1.3544 0.0340 22 15 0.5287 1 NA NA | |
| 326 | 155 --acs --localsearch 3 --alpha 1.7343 --beta 8.2833 --rho 0.7587 --ants 31 --nnls 19 --q0 0.0859 --dlb 1 | |
| 327 | 127 --acs --localsearch 3 --alpha 1.6457 --beta 5.9689 --rho 0.7436 --ants 13 --nnls 17 --q0 0.008 --dlb 1 | |
| 328 | 138 --acs --localsearch 3 --alpha 1.4294 --beta 6.0012 --rho 0.9344 --ants 16 --nnls 21 --q0 0.2217 --dlb 1 | |
| 329 | 120 --acs --localsearch 3 --alpha 1.3726 --beta 6.832 --rho 0.8051 --ants 24 --nnls 16 --q0 0.2054 --dlb 1 | |
| 330 | 149 --acs --localsearch 3 --alpha 1.8645 --beta 8.8636 --rho 0.8623 --ants 31 --nnls 18 --q0 0.0324 --dlb 1 | |
| 331 | ||
| 332 | ||
| 426 | 333 | # Testing of elite configurations: 5 |
| 427 | 334 | # Testing iteration configurations: TRUE |
| 428 | # 2020-03-27 14:35:21 GMT: Testing results (column number is configuration ID in no particular order): | |
| 429 | 20 46 73 119 96 153 84 | |
| 430 | 1t 23467028 23378036 23348394 23368581 23359911 23345562 23368641 | |
| 431 | 2t 23218038 23175305 23187510 23151427 23149285 23111828 23126419 | |
| 432 | 3t 23081105 22968142 23020314 22992964 23128306 23003962 22981597 | |
| 433 | 4t 23209722 23050700 22999730 23016134 23072876 23032503 23062485 | |
| 434 | 5t 23239110 23207351 23148135 23209542 23203112 23211830 23200361 | |
| 435 | 6t 23414662 23480562 23421143 23408180 23478925 23428935 23394469 | |
| 436 | 7t 23378696 23309958 23337527 23315313 23334010 23299940 23332029 | |
| 437 | 8t 23233802 23236143 23224288 23229160 23197809 23225885 23206032 | |
| 438 | 9t 23323703 23346648 23294819 23264655 23333054 23292474 23280391 | |
| 439 | 10t 23791077 23033499 23067827 23039152 23064926 23049826 23060048 | |
| 440 | # 2020-03-27 14:35:21 GMT: Finished testing | |
| 335 | # 2022-10-22 11:30:38 BST: Testing configurations (in no particular order): 14 11 31 6 57 55 39 78 62 85 71 72 113 110 91 138 127 120 124 149 147 155 | |
| 336 | algorithm localsearch alpha beta rho ants nnls q0 dlb rasrank elitistants | |
| 337 | 14 ras 3 3.7374 4.7749 0.1279 5 17 NA 0 37 NA | |
| 338 | 11 acs 3 2.6159 1.0514 0.5986 12 21 0.2211 0 NA NA | |
| 339 | 31 acs 1 2.3896 7.8790 0.3748 34 36 0.9029 1 NA NA | |
| 340 | 6 acs 3 2.2422 7.8826 0.1091 14 40 0.0174 0 NA NA | |
| 341 | 57 acs 3 1.0519 6.5794 0.6649 11 33 0.2283 1 NA NA | |
| 342 | 55 mmas 3 1.0832 5.5967 0.9317 18 46 NA 0 NA NA | |
| 343 | 39 acs 2 2.5486 3.7878 0.0262 13 12 0.7621 0 NA NA | |
| 344 | 78 acs 3 1.5444 6.6646 0.8947 20 18 0.2581 1 NA NA | |
| 345 | 62 mmas 3 1.2260 2.0146 0.7659 8 16 NA 0 NA NA | |
| 346 | 85 acs 3 3.0740 4.9115 0.0734 18 47 0.4206 1 NA NA | |
| 347 | 71 acs 3 1.7574 6.9647 0.6223 13 19 0.4806 0 NA NA | |
| 348 | 72 acs 3 2.0489 7.1428 0.5326 16 6 0.2179 0 NA NA | |
| 349 | 113 acs 3 1.9861 6.6955 0.7803 26 15 0.0512 1 NA NA | |
| 350 | 110 mmas 3 1.1100 2.0382 0.6833 15 11 NA 0 NA NA | |
| 351 | 91 acs 3 1.5186 5.7143 0.5523 14 26 0.8361 1 NA NA | |
| 352 | 138 acs 3 1.4294 6.0012 0.9344 16 21 0.2217 1 NA NA | |
| 353 | 127 acs 3 1.6457 5.9689 0.7436 13 17 0.0080 1 NA NA | |
| 354 | 120 acs 3 1.3726 6.8320 0.8051 24 16 0.2054 1 NA NA | |
| 355 | 124 acs 3 1.6169 7.9049 0.7166 20 8 0.2282 1 NA NA | |
| 356 | 149 acs 3 1.8645 8.8636 0.8623 31 18 0.0324 1 NA NA | |
| 357 | 147 acs 3 1.3012 6.1776 0.7881 22 10 0.1276 1 NA NA | |
| 358 | 155 acs 3 1.7343 8.2833 0.7587 31 19 0.0859 1 NA NA | |
| 359 | # 2022-10-22 11:49:09 BST: Testing results (column number is configuration ID in no particular order): | |
| 360 | 14 11 31 6 57 55 39 78 62 85 71 72 113 110 91 138 127 120 124 149 147 155 | |
| 361 | 1t 23391860 23332695 23537760 23414477 23364455 23485970 23376575 23368630 23370336 23416359 23412859 23402853 23413652 23358323 23443305 23390347 23324399 23359717 23360665 23356925 23301025 23356774 | |
| 362 | 2t 23229539 23167464 23337793 23272324 23147927 23195166 23258960 23114234 23263064 23232588 23183375 23155695 23140836 23206760 23136299 23126209 23117499 23142083 23141203 23118103 23131024 23175108 | |
| 363 | 3t 23184984 23094701 23252927 23171748 22982248 23026437 23038220 23083312 23053955 23082028 23023432 23067583 23011134 23000956 23053830 23047801 22994436 23001269 23028544 22977398 23032405 23032211 | |
| 364 | 4t 23116701 23126420 23232767 23157322 23044199 23063903 23136997 23053899 23115735 23102111 23108032 23010549 23051643 23022518 23074969 23031784 23023703 23031107 23042387 23081150 23015695 23027772 | |
| 365 | 5t 23237519 23373072 23380589 23320207 23217884 23233250 23316907 23211425 23173925 23170016 23276775 23209411 23189520 23219876 23261978 23196505 23197529 23174024 23222521 23185052 23184322 23163568 | |
| 366 | 6t 23481721 23475588 23508760 23479494 23467320 23495025 23549932 23409175 23483799 23423813 23479579 23446369 23410700 23415018 23491880 23444264 23436888 23416332 23419003 23463089 23427505 23394824 | |
| 367 | 7t 23369278 23676371 23494491 23392883 23401441 23380877 23319057 23343664 23389619 23383997 23405404 23350024 23312217 23354092 23396796 23343027 23343136 23331633 23354417 23361221 23324993 23375510 | |
| 368 | 8t 23311915 23288024 23387646 23337527 23249894 23333692 23377228 23215822 23324171 23316955 23267699 23285411 23242464 23268552 23284711 23208403 23233397 23247441 23247049 23209032 23217163 23267663 | |
| 369 | 9t 23358261 23341818 23449508 23430565 23374655 23293488 23476762 23328162 23428572 23284190 23280730 23371465 23318878 23303311 23287065 23288033 23334631 23296373 23256521 23308818 23264005 23299111 | |
| 370 | 10t 23095703 23296750 23286985 23147297 23078823 23051369 23124520 23112076 23066743 23077135 23189020 23065758 23087598 23053962 23028421 23066774 23087940 23041424 23132030 23062589 23014124 23038023 | |
| 371 | # 2022-10-22 11:49:09 BST: Finished testing | |
Binary diff not shown
| 5 | 5 | %\VignetteDepends{knitr} |
| 6 | 6 | %\VignetteCompiler{knitr} |
| 7 | 7 | \synctex=1 |
| 8 | \RequirePackage{xparse} | |
| 8 | 9 | \RequirePackage[dvipsnames]{xcolor} |
| 9 | 10 | \documentclass[a4paper,english]{article} |
| 11 | \usepackage[T1]{fontenc} | |
| 12 | \usepackage[utf8]{inputenc} | |
| 10 | 13 | \usepackage[a4paper]{geometry} % It saves some pages |
| 11 | \usepackage[utf8]{inputenc} | |
| 12 | \usepackage[T1]{fontenc} | |
| 13 | 14 | \usepackage[english]{babel} |
| 14 | 15 | \usepackage{ifthen} |
| 15 | 16 | \newboolean{Release} |
| 23 | 24 | \usepackage{amsmath,amssymb} |
| 24 | 25 | \usepackage{relsize} |
| 25 | 26 | \usepackage{fancyvrb} |
| 27 | \usepackage{microtype} | |
| 28 | % \texttt{test -- test} keeps the "--" as "--" (and does not convert it to an en dash) | |
| 29 | \DisableLigatures{encoding = T1, family = tt* } | |
| 26 | 30 | \usepackage[hyphens]{url} |
| 27 | 31 | \usepackage{hyperref} |
| 28 | 32 | \usepackage[numbers]{natbib} |
| 97 | 101 | \makeatletter |
| 98 | 102 | \DeclareRobustCommand\code{\bgroup\@makeother\_\@makeother\~\@makeother\$\@noligs\@codex} |
| 99 | 103 | \def\@codex#1{\texorpdfstring% |
| 100 | {{\normalfont\ttfamily\hyphenchar\font=-1 #1}}% | |
| 104 | {{\text{\normalfont\ttfamily\hyphenchar\font=-1 #1}}}% | |
| 101 | 105 | {#1}\egroup} |
| 102 | 106 | \makeatother |
| 103 | 107 | |
| 119 | 123 | \emph{flag:} % |
| 120 | 124 | \ifthenelse{\equal{#1}{}}{}{% |
| 121 | 125 | \code{-#1}~~~\emph{or}~~~}% |
| 122 | \code{-{}-#3} ~~ }% | |
| 126 | \code{--#3} ~~ }% | |
| 123 | 127 | \emph{default:}~\texttt{#4} \\ |
| 124 | 128 | } |
| 125 | 129 | \newcommand{\parameter}[1]{\hyperlink{opt:#1}{\code{#1}}} |
| 212 | 216 | <<exampleload,eval=TRUE,include=FALSE>>= |
| 213 | 217 | library("irace") |
| 214 | 218 | load("examples.Rdata") |
| 215 | load("irace-acotsp.Rdata") | |
| 216 | load("log-ablation.Rdata") | |
| 219 | iraceResults <- irace::read_logfile("irace-acotsp.Rdata") | |
| 220 | log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 221 | load(log_ablation_file) | |
| 217 | 222 | options(width = 70) |
| 218 | 223 | @ |
| 219 | 224 | \newpage |
| 247 | 252 | \end{center} |
| 248 | 253 | % |
| 249 | 254 | More information about \irace is available at |
| 250 | \url{http://iridia.ulb.ac.be/irace}. | |
| 255 | \url{https://mlopez-ibanez.github.io/irace}. | |
| 251 | 256 | |
| 252 | 257 | \subsection{Version} |
| 253 | 258 | The current version of the \irace package is \iraceversion. Previous |
| 544 | 549 | \end{lstlisting} |
| 545 | 550 | %@ |
| 546 | 551 | |
| 547 | \item Copy all the template files from the \IRACEHOME{/templates/} | |
| 548 | directory to the scenario directory. | |
| 549 | %<<copy1,engine='bash',eval=FALSE>>= | |
| 552 | \item Initialize the tuning directory with template config files. On GNU/Linux or OS X, you can do this as follows: | |
| 553 | %<<dir1,engine='bash',eval=FALSE>>= | |
| 550 | 554 | \begin{lstlisting}[style=BashInputStyle] |
| 551 | # $IRACE_HOME is the installation directory of irace. | |
| 552 | cp $IRACE_HOME/templates/*.tmpl ~/tuning/ | |
| 555 | irace --init | |
| 553 | 556 | \end{lstlisting} |
| 554 | 557 | %@ |
| 555 | ||
| 556 | % Remember that \IRACEHOME{} is the path to the installation | |
| 557 | % directory of \irace. It can be obtained in the \aR console with: | |
| 558 | % | |
| 559 | % <<irace_path,eval=FALSE, prompt=FALSE>>= | |
| 560 | % library("irace") | |
| 561 | % system.file(package = "irace") | |
| 562 | % @ | |
| 563 | ||
| 564 | \item For each template in your tuning directory, remove the \code{.tmpl} | |
| 565 | suffix, and modify them following the next steps. | |
| 566 | 558 | |
| 567 | 559 | \item Define the target algorithm parameters to be tuned by following the |
| 568 | 560 | instructions in \code{parameters.txt}. Available parameter types and other |
| 612 | 604 | time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. |
| 613 | 605 | The \code{target-runner} template is written in \proglang{GNU Bash} |
| 614 | 606 | scripting language, which can be executed easily in GNU/Linux and OS X |
| 615 | systems. However, you may use any other programming language. As an example, | |
| 616 | we provide a \proglang{Python} example in the directory | |
| 617 | \IRACEHOME{/examples/python}. % | |
| 607 | systems. However, you may use any other programming language. We provide examples written in \proglang{Python}, \proglang{MATLAB} and other languages in \IRACEHOME{/examples/}. % | |
| 618 | 608 | Follow these instructions to adjust the given \code{target-runner} template |
| 619 | 609 | to your algorithm: |
| 620 | 610 | \begin{enumerate} |
| 655 | 645 | \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} |
| 656 | 646 | \end{center} |
| 657 | 647 | parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} |
| 658 | script must return \textbf{only} one number. In the template example, the result is returned with | |
| 659 | \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the used files are deleted. | |
| 648 | script must print \textbf{only} one number. In the template example, the result is printed with | |
| 649 | \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the generated files are deleted (you may remove that line if you wish to keep them). | |
| 660 | 650 | |
| 661 | 651 | \begin{xwarningbox} |
| 662 | The \code{target-runner} script must be executable. | |
| 652 | The \code{target-runner} script must be an executable file, unless you specify \parameter{targetRunnerLauncher} and \parameter{targetRunnerLauncherArgs}. | |
| 663 | 653 | \end{xwarningbox} |
| 664 | 654 | |
| 665 | 655 | You can test the target runner from the \aR console by checking the scenario as |
| 673 | 663 | |
| 674 | 664 | \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. |
| 675 | 665 | |
| 676 | \item The \irace executable provides an option (\parameter{-{}-check}) to | |
| 666 | \item The \irace executable provides an option (\parameter{--check}) to | |
| 677 | 667 | check that the scenario is correctly defined. We recommend to perform a |
| 678 | 668 | check every time you create a new scenario. When performing the check, |
| 679 | 669 | \irace will verify that the scenario and parameter definitions are |
| 706 | 696 | or directly from the \aR console: |
| 707 | 697 | |
| 708 | 698 | \begin{itemize} |
| 709 | \item{% | |
| 699 | \item | |
| 710 | 700 | \textbf{From the command-line console}, call the command (on Windows, you should execute |
| 711 | 701 | \code{irace.exe}): |
| 712 | 702 | \begin{lstlisting}[style=BashInputStyle] |
| 720 | 710 | properly in the \code{scenario.txt} file using the options described in |
| 721 | 711 | \autoref{sec:irace options}. Most \irace options can be specified |
| 722 | 712 | in the command line or directly in the \code{scenario.txt} file. |
| 723 | } | |
| 724 | \item{ | |
| 713 | ||
| 714 | \item | |
| 725 | 715 | \textbf{From the \aR console}, evaluate: |
| 726 | 716 | |
| 727 | 717 | <<irace_R_exe, eval=FALSE, prompt=FALSE>>= |
| 732 | 722 | scenario = defaultScenario()) |
| 733 | 723 | irace.main(scenario = scenario) |
| 734 | 724 | @ |
| 735 | } | |
| 725 | ||
| 736 | 726 | \end{itemize} |
| 737 | 727 | |
| 738 | 728 | This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on |
| 746 | 736 | |
| 747 | 737 | \subsection{Setup example for ACOTSP}\label{sec:example} |
| 748 | 738 | |
| 749 | The \ACOTSP tuning example can be found in the package installation at | |
| 750 | \IRACEHOME{/examples/acotsp}. | |
| 739 | The \ACOTSP tuning example can be found in the package installation in the folder \IRACEHOME{/examples/acotsp}. | |
| 751 | 740 | % |
| 752 | Additionally, a number of example scenarios can be found in the \code{examples} folder. More | |
| 741 | Other example scenarios can be found in the same folder. More | |
| 753 | 742 | examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). |
| 754 | 743 | |
| 755 | 744 | In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up |
| 756 | 745 | your own scenario, continue to the next section. For this example, we assume |
| 757 | a GNU/Linux system but making the necessary changes in the commands and \parameter{targetRunner}, | |
| 758 | it can be executed in any system that has a \proglang{C} compiler. | |
| 759 | %\MANUEL{I don't think this is true, since the target-runner script needs Bash} | |
| 746 | a GNU/Linux system such as Ubuntu with a working \proglang{C} compiler such as \code{gcc}. | |
| 760 | 747 | To execute this scenario follow these steps: |
| 761 | 748 | |
| 762 | 749 | \begin{enumerate} |
| 771 | 758 | \end{lstlisting} |
| 772 | 759 | %@ |
| 773 | 760 | |
| 774 | \item Download the training instances from \url{http://iridia.ulb.ac.be/irace/} to the \path{~/tuning/} directory. | |
| 761 | \item Download the training instances from \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} to the \path{~/tuning/} directory. | |
| 775 | 762 | \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. |
| 776 | 763 | |
| 777 | 764 | %<<instance0,engine='bash',eval=FALSE>>= |
| 792 | 779 | make |
| 793 | 780 | \end{lstlisting} |
| 794 | 781 | %@ |
| 795 | \item Create a directory for the executable and copy it: | |
| 796 | ||
| 797 | %<<acotsp1,engine='bash',eval=FALSE>>= | |
| 798 | \begin{lstlisting}[style=BashInputStyle] | |
| 799 | mkdir ~/bin/ | |
| 800 | cp ~/tuning/ACOTSP-1.03/acotsp ~/bin/ | |
| 801 | \end{lstlisting} | |
| 802 | %@ | |
| 803 | 782 | |
| 804 | 783 | \item Create a directory for executing the experiments and execute \irace: |
| 805 | 784 | |
| 858 | 837 | option \parameter{digits}. For example, given the default |
| 859 | 838 | number of digits of $4$, the values $0.12345$ and |
| 860 | 839 | $0.12341$ are both rounded to $0.1234$. |
| 861 | % However, the values $0.00001$ and $0.00005$ remain the same. | |
| 862 | Selected real-valued parameters can be optionally sampled on | |
| 840 | Selected real-valued parameters can be optionally sampled on | |
| 863 | 841 | a logarithmic scale (base $e$). |
| 864 | 842 | |
| 865 | 843 | \item \textit{Integer} parameters are numerical parameters that can take only |
| 895 | 873 | defined on them. All fixed parameters must be defined as categorical |
| 896 | 874 | parameters and have a domain of one element. |
| 897 | 875 | |
| 898 | \subsubsection{Conditional parameters} | |
| 876 | \subsubsection{Parameter dependent domains}\label{sec:paramdependant} | |
| 877 | ||
| 878 | Domains that are dependent on the values of other parameters can be specified | |
| 879 | only for numerical parameters (both integer and real). To do so, the dependent | |
| 880 | domain must be expressed in function of another parameter, which must be a | |
| 881 | numerical parameter. The expression that defines a dependency must be written | |
| 882 | between quotes: \code{(value, "expression")} or \code{("expression", value)} or | |
| 883 | \code{("expression", "expression")}. | |
| 884 | ||
| 885 | The expressions can only use the following operators and \aR functions: \code{+}, | |
| 886 | \code{-}, \code{*}, \code{/}, \code{\%\%}, \code{min}, \code{max}, | |
| 887 | \code{round}, \code{floor}, \code{ceiling}, \code{trunc}. If you need to use an | |
| 888 | operator or function not listed here, please contact us. | |
| 889 | ||
| 890 | \begin{xwarningbox} | |
| 891 | The user must ensure that the defined domain is valid at all times since | |
| 892 | \irace currently is not able to detect possible invalid domains based on the expressions | |
| 893 | provided. | |
| 894 | \end{xwarningbox} | |
| 895 | ||
| 896 | If you have a parameter \code{p2} that is just a transformation of another | |
| 897 | \code{p1}, then instead of using a dependent domain (left-hand side of the | |
| 898 | following example), it will be better to create a dummy parameter that controls | |
| 899 | the transformation (right-hand side) and do the transformation within \code{target-runner}. For example: | |
| 900 | % | |
| 901 | \begin{center} | |
| 902 | \begin{minipage}{0.4\linewidth} | |
| 903 | \small\centering% | |
| 904 | \begin{CodeInput}[frame=single] | |
| 905 | # With dependent domains | |
| 906 | p1 "" r (0, 100) | |
| 907 | p2 "" r ("p1", "p1 + 10") | |
| 908 | \end{CodeInput} | |
| 909 | \end{minipage} | |
| 910 | \hspace{\stretch{1}} should be \hspace{\stretch{1}} | |
| 911 | \begin{minipage}{0.4\linewidth} | |
| 912 | \small\centering% | |
| 913 | \begin{CodeInput}[frame=single] | |
| 914 | # With a dummy parameter | |
| 915 | p1 "" r (0, 100) | |
| 916 | p2dum "" r (0, 10) | |
| 917 | \end{CodeInput} | |
| 918 | \end{minipage} | |
| 919 | \end{center} | |
| 920 | % | |
| 921 | and \code{target-runner} will compute $\code{p2} = \code{p2dum} \cdot \code{p1}$. | |
| 922 | ||
| 923 | ||
| 924 | \subsubsection{Conditional parameters}\label{sec:conditional} | |
| 899 | 925 | |
| 900 | 926 | Conditional parameters are active only when others have certain values. These |
| 901 | 927 | dependencies define a hierarchical relation between parameters. For example, |
| 910 | 936 | table. Each line of the table defines a configurable parameter |
| 911 | 937 | % |
| 912 | 938 | \begin{center} |
| 913 | \code{<name>\ <label>\ <type>\ <range>\ [ | <condition>\ ] } | |
| 939 | \code{<name>\ <label>\ <type>\ <domain>\ [ | <condition>\ ] } | |
| 914 | 940 | \end{center} |
| 915 | 941 | where each field is defined as follows: |
| 916 | 942 | % |
| 917 | \begin{center} | |
| 918 | \renewcommand{\arraystretch}{1.2} | |
| 919 | \begin{tabularx}{0.98\linewidth}{@{}rX@{}} | |
| 920 | \code{<name>} & The name of the parameter as an unquoted | |
| 921 | alphanumeric string, e.g., `\code{ants}'.\\ | |
| 943 | \begin{description}[left=5em,itemindent=-0.5em,align=right,itemsep=0pt] | |
| 944 | % \DrawEnumitemLabel | |
| 945 | \item[\code{<name>}] The name of the parameter as an unquoted | |
| 946 | alphanumeric string, e.g., `\code{ants}'. | |
| 922 | 947 | % |
| 923 | \code{<label>}& A \emph{label} for this parameter. This is a | |
| 948 | \item[\code{<label>}] | |
| 949 | A \emph{label} for this parameter. This is a | |
| 924 | 950 | string that will be passed together with the parameter |
| 925 | 951 | to \parameter{targetRunner}. In the default \parameter{targetRunner} |
| 926 | 952 | provided with the package (\autoref{sec:runner}), this is the |
| 927 | 953 | command-line switch used to pass the value of this parameter, for |
| 928 | instance `\code{"-{}-ants "}'.\newline | |
| 954 | instance `\code{"--ants "}'.\newline | |
| 929 | 955 | The value of the parameter is concatenated \emph{without |
| 930 | 956 | separator} to the label when |
| 931 | invoking \parameter{targetRunner}, thus \emph{any whitespace in the label is significant}. Following the same example, when parameter \code{ants} takes value \code{5}, the default targetRunner will pass the parameter as \code{"-{}-ants 5"}.\\ | |
| 932 | % | |
| 933 | \code{{<type>}} & The type of the parameter, either | |
| 957 | invoking \parameter{targetRunner}, thus \emph{any whitespace in the label is significant}. Following the same example, when parameter \code{ants} takes value \code{5}, the default targetRunner will pass the parameter as \code{"--ants 5"}. | |
| 958 | ||
| 959 | \item[\code{{<type>}}] The type of the parameter, either | |
| 934 | 960 | \textit{integer}, \textit{real}, \textit{ordinal} or |
| 935 | 961 | \textit{categorical}, given as a single letter: `\code{i}', |
| 936 | 962 | `\code{r}', `\code{o}' or `\code{c}'. Numerical parameters |
| 937 | 963 | can be sampled using a natural logarithmic scale with '\code{i,log}' and |
| 938 | 964 | '\code{r,log}' (without spaces) for integer and real |
| 939 | parameters, respectively.\\ | |
| 965 | parameters, respectively. | |
| 940 | 966 | % |
| 941 | \code{{<range>}} & The range or set of values of the parameter delimited by | |
| 942 | parentheses. \eg~\code{(0,1)} or \code{(a,b,c,d)}.\\ | |
| 967 | \item[\code{{<domain>}}] The range or set of values of the parameter delimited by | |
| 968 | parentheses, \eg~\code{(0,1)} or \code{(a,b,c,d)}. | |
| 969 | See also parameter dependent domains (\autoref{sec:paramdependant}). | |
| 943 | 970 | % |
| 944 | 971 | |
| 945 | \code{{<condition>}} & An optional \emph{condition} that determines whether the parameter is | |
| 972 | \item[\code{{<condition>}}] An optional \emph{condition} that determines whether the parameter is | |
| 946 | 973 | enabled or disabled, thus making the parameter conditional. If the |
| 947 | 974 | condition evaluates to false, then no value is assigned to this |
| 948 | 975 | parameter, and neither the parameter value nor the corresponding |
| 952 | 979 | The condition may contain the name |
| 953 | 980 | of other parameters as long as the dependency graph does not |
| 954 | 981 | contain any cycle. Otherwise, \irace will detect the cycle and |
| 955 | stop with an error. \\ | |
| 956 | \end{tabularx} | |
| 957 | \end{center} | |
| 982 | stop with an error. | |
| 983 | ||
| 984 | \end{description} | |
| 958 | 985 | |
| 959 | 986 | As an example, Figure~\ref{fig:acotsp_parameters} shows the parameters file |
| 960 | 987 | of the \ACOTSP scenario. |
| 962 | 989 | \begin{figure}[!hbt] |
| 963 | 990 | \small\centering% |
| 964 | 991 | \begin{CodeInput} |
| 965 | # name switch type values [conditions (using R syntax)] | |
| 966 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 967 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 968 | alpha "--alpha " r (0.00, 5.00) | |
| 969 | beta "--beta " r (0.00, 10.00) | |
| 970 | rho "--rho " r (0.01, 1.00) | |
| 971 | ants "--ants " i (5, 100) | |
| 972 | nnls "--nnls " i (5, 50) | localsearch %in% c(1, 2, 3) | |
| 973 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 974 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 975 | rasrank "--rasranks " i (1, 100) | algorithm == "ras" | |
| 976 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 992 | # name switch type values [conditions (using R syntax)] | |
| 993 | algorithm "--" c (as,mmas,eas,ras,acs) | |
| 994 | localsearch "--localsearch " c (0, 1, 2, 3) | |
| 995 | alpha "--alpha " r (0.00, 5.00) | |
| 996 | beta "--beta " r (0.00, 10.00) | |
| 997 | rho "--rho " r (0.01, 1.00) | |
| 998 | ants "--ants " i (5, 100) | |
| 999 | nnls "--nnls " i (5, 50) | localsearch %in% c(1, 2, 3) | |
| 1000 | q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" | |
| 1001 | dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) | |
| 1002 | rasrank "--rasranks " i (1, "ants") | algorithm == "ras" | |
| 1003 | elitistants "--elitistants " i (1, 750) | algorithm == "eas" | |
| 977 | 1004 | \end{CodeInput} |
| 978 | 1005 | \caption{Parameter file (\code{parameters.txt}) for tuning \ACOTSP.}\label{fig:acotsp_parameters} |
| 979 | 1006 | \end{figure} |
| 1008 | 1035 | \code{nbParameters} & An integer, the total number of parameters.\\ |
| 1009 | 1036 | \code{nbFixed} & An integer, the number of parameters with a fixed value.\\ |
| 1010 | 1037 | \code{nbVariable} & Number of variable (i.e., to be tuned) parameters.\\ |
| 1038 | \code{isDependent} & Logical vector that specifies which parameter defines | |
| 1039 | a dependent domain.\\ | |
| 1011 | 1040 | \end{tabularx} |
| 1012 | 1041 | \end{center} |
| 1013 | 1042 | |
| 1057 | 1086 | \code{id.instance} & an alphanumeric string that uniquely identifies an instance;\\ |
| 1058 | 1087 | \code{seed} & seed for the random number generator to be used for this evaluation, ignore the seed for deterministic algorithms;\\ |
| 1059 | 1088 | \code{instance} & string giving the instance to be used for this evaluation;\\ |
| 1060 | \code{bound} & optional execution time bound. Only provided when the \parameter{maxBound} option is set in the scenario, see \autoref{sec:capping};\\ | |
| 1089 | \code{bound} & optional execution time bound. Only provided when the \parameter{boundMax} option is set in the scenario, see \autoref{sec:capping};\\ | |
| 1061 | 1090 | \code{configuration} & the pairs parameter label-value that |
| 1062 | 1091 | describe this candidate configuration. Typically given as command-line |
| 1063 | 1092 | switches to be passed to the executable program.\\ |
| 1430 | 1459 | the condition with \code{as.numeric(a)}. |
| 1431 | 1460 | \end{xwarningbox} |
| 1432 | 1461 | |
| 1462 | If the constraints provided are too strict, \irace may produce the following error: | |
| 1463 | \begin{CodeInput} | |
| 1464 | irace tried 100 times to sample from the model a configuration not forbidden | |
| 1465 | without success, perhaps your constraints are too strict? | |
| 1466 | \end{CodeInput} | |
| 1467 | % | |
| 1468 | In that case, it may be a good idea to reformulate the constraints as conditional parameters (\autoref{sec:conditional}), parameter-dependent domains (\autoref{sec:paramdependant}), repairing the configurations (\autoref{sec:repairconf}) or post-processing within the target-algorithm (\autoref{sec:complex_domains}). | |
| 1469 | ||
| 1470 | ||
| 1433 | 1471 | \subsection{Repairing configurations}\label{sec:repairconf} |
| 1434 | 1472 | |
| 1435 | 1473 | In some problems, the parameter values require complex constraints that cannot |
| 1455 | 1493 | { |
| 1456 | 1494 | isreal <- parameters$type[colnames(configuration)] %in% "r" |
| 1457 | 1495 | configuration[isreal] <- configuration[isreal] / sum(configuration[isreal]) |
| 1496 | configurations[isreal] <- round(configuration[isreal], digits) | |
| 1458 | 1497 | return(configuration) |
| 1459 | 1498 | } |
| 1460 | 1499 | @ |
| 1635 | 1674 | \end{itemize} |
| 1636 | 1675 | |
| 1637 | 1676 | The testing can be also (re-)executed at a later time by using the following |
| 1638 | \aR command: | |
| 1677 | \aR command (but you may need to override \code{testNbElites} and \code{testIterationElites}): | |
| 1639 | 1678 | % |
| 1640 | 1679 | <<testing_r, prompt=FALSE, eval=FALSE>>= |
| 1641 | testing.main(logFile = "./irace.Rdata") | |
| 1680 | testing_fromlog(logFile = "./irace.Rdata", testNbElites = 1) | |
| 1642 | 1681 | @ |
| 1643 | 1682 | % |
| 1644 | 1683 | The above line will load the scenario setup from \parameter{logFile} to |
| 1648 | 1687 | \autoref{sec:output r}. For examples on how to analyse the results see |
| 1649 | 1688 | \autoref{sec:analysis}. |
| 1650 | 1689 | |
| 1651 | Another alternative is to test a specific set of configurations using the command-line option \parameter{-{}-only-test} as follows: | |
| 1690 | Another alternative is to test a specific set of configurations using the command-line option \parameter{--only-test} as follows: | |
| 1652 | 1691 | \begin{lstlisting}[style=BashInputStyle] |
| 1653 | 1692 | irace --only-test configurations.txt |
| 1654 | 1693 | \end{lstlisting} |
| 1849 | 1888 | discarded after the statistical test was performed. |
| 1850 | 1889 | \item \code{Best}: ID of the best configuration according to the instances seen |
| 1851 | 1890 | so far in this race (i.e., not including previous iterations). |
| 1852 | \item \code{Mean best}: Mean of the best configuration across the instances seen so | |
| 1853 | far in this race. | |
| 1891 | \item \code{Mean best}: Mean cost value of the best configuration across the instances seen so | |
| 1892 | far in this race (not globally). Equivalent to the concept of ``iteration-best'' in other algorithms. | |
| 1854 | 1893 | \item \code{Exp so far}: Number of experiments |
| 1855 | 1894 | performed so far. |
| 1856 | 1895 | \item \code{W time}: Wall-clock time spent on this instance. |
| 1923 | 1962 | parameters of all elite configurations found by \irace use: |
| 1924 | 1963 | |
| 1925 | 1964 | <<get_elites, prompt=TRUE, eval=TRUE, comment="">>= |
| 1926 | getFinalElites(logFile = "irace-acotsp.Rdata", n = 0) | |
| 1965 | getFinalElites("irace-acotsp.Rdata", n = 0) | |
| 1927 | 1966 | @ |
| 1928 | 1967 | |
| 1929 | 1968 | \item \code{iterationElites}: A vector containing the best candidate configuration ID of each iteration. The best configuration found corresponds to the last one |
| 1937 | 1976 | <<get_elite, prompt=TRUE, eval=TRUE, comment="">>= |
| 1938 | 1977 | last <- length(iraceResults$iterationElites) |
| 1939 | 1978 | id <- iraceResults$iterationElites[last] |
| 1940 | getConfigurationById(logFile = "irace-acotsp.Rdata", ids = id) | |
| 1979 | getConfigurationById(iraceResults, ids = id) | |
| 1941 | 1980 | @ |
| 1942 | 1981 | |
| 1943 | 1982 | \item \code{rejectedConfigurations}: A vector containing the rejected configurations IDs. These |
| 1973 | 2012 | # As an example, we get seed and instance of the experiments |
| 1974 | 2013 | # of the best candidate. |
| 1975 | 2014 | # Get index of the instances |
| 1976 | pair.id <- names(all.exp[!is.na(all.exp)]) | |
| 1977 | index <- iraceResults$state$.irace$instancesList[pair.id,"instance"] | |
| 2015 | pair.id <- which(!is.na(all.exp)) | |
| 2016 | index <- iraceResults$state$.irace$instancesList[pair.id, "instance"] | |
| 1978 | 2017 | # Obtain the instance names |
| 1979 | 2018 | iraceResults$scenario$instances[index] |
| 1980 | 2019 | # Get the seeds |
| 1981 | iraceResults$state$.irace$instancesList[index,"seed"] | |
| 2020 | iraceResults$state$.irace$instancesList[pair.id, "seed"] | |
| 1982 | 2021 | @ |
| 1983 | 2022 | \item \code{experimentLog}: A matrix with columns \code{iteration, instance, configuration}. This matrix contains the log of all the experiments that \irace |
| 1984 | 2023 | performs during its execution. The \code{instance} column refers to the index of the |
| 2037 | 2076 | configuration is reported first. |
| 2038 | 2077 | |
| 2039 | 2078 | If testing is performed, you can further analyze the resulting best |
| 2040 | configurations by performing statistical tests in \aR or just plotting the | |
| 2041 | results: | |
| 2042 | ||
| 2043 | <<plot_test, fig.pos="tbp", fig.align="center", fig.height = 4, fig.width = 8, out.width='0.85\\textwidth', fig.cap="Boxplot of the testing results of the best configurations.", prompt=TRUE, eval=TRUE, comment="">>= | |
| 2079 | configurations by performing statistical tests in \aR: | |
| 2080 | % FIXME: iraceplot | |
| 2081 | % or just plotting the results: | |
| 2082 | %<<plot_test, fig.pos="tbp", fig.align="center", fig.height = 4, fig.width = 8, out.width='0.85\\textwidth', fig.cap="Boxplot of the testing results of the best configurations.", prompt=TRUE, eval=TRUE, comment="">>= | |
| 2083 | <<wilcox_test,prompt=TRUE, eval=TRUE, comment="">>= | |
| 2044 | 2084 | results <- iraceResults$testing$experiments |
| 2045 | 2085 | # Wilcoxon paired test |
| 2046 | 2086 | conf <- gl(ncol(results), # number of configurations |
| 2047 | 2087 | nrow(results), # number of instances |
| 2048 | 2088 | labels = colnames(results)) |
| 2049 | 2089 | pairwise.wilcox.test (as.vector(results), conf, paired = TRUE, p.adj = "bonf") |
| 2050 | # Plot the results | |
| 2051 | configurationsBoxplot (results, ylab = "Solution cost") | |
| 2052 | @ | |
| 2053 | ||
| 2054 | %%FIXME: should we add something like this? | |
| 2055 | %The Kendall concordance coefficient (\code{W}) and the Spearman's rho can be | |
| 2056 | %applied over data that has the characteristics of the data obtained in the testing, | |
| 2057 | %that is a full matrix where all configurations are executed in all instances. \code{W} | |
| 2058 | %can show if the configurations tested have an homogeneous performance on the used instances | |
| 2059 | %set. If evidence of an heterogeneous scenario found we recommend to make some adjustments | |
| 2060 | %in the \irace options as described in \autoref{sec:} | |
| 2061 | ||
| 2062 | %<<conc, prompt=TRUE, eval=TRUE, comment="">>= | |
| 2063 | %irace:::concordance(iraceResults$testing$experiments) | |
| 2064 | %@ | |
| 2065 | % | |
| 2066 | ||
| 2067 | During the tuning, \irace iteratively updates the sampling models of the | |
| 2068 | parameters to focus on the best regions of the parameter search space. The | |
| 2069 | frequency of the sampled configurations can provide insights on the parameter | |
| 2070 | search space. We provide a function for plotting the | |
| 2071 | frequency of the sampling of a set of configurations. For more information on this function, please see the \aR help, type in the | |
| 2072 | \aR console: \code{?parameterFrequency}. The following example | |
| 2073 | plots the frequency of the parameters sampled during one \irace run: | |
| 2074 | ||
| 2075 | <<freq, fig.pos="tbp", fig.cap="Parameters sampling frequency.", out.width="\\textwidth",prompt=TRUE, eval=TRUE, comment="">>= | |
| 2076 | parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 2077 | @ | |
| 2078 | ||
| 2079 | By using parallel coordinates plots, it is possible to analyze how the parameters | |
| 2080 | interact with each other. For more information on this function, please see the \aR help, type | |
| 2081 | in the \aR console: (\code{?parallelCoordinatesPlot}). | |
| 2082 | The following example shows how to create a parallel | |
| 2083 | coordinate plot of the configurations in the last two iterations of \irace. | |
| 2084 | ||
| 2085 | <<parcord, fig.pos="tbp", fig.align="center", out.width="0.7\\textwidth", fig.cap="Parallel coordinate plots of the parameters of the configurations in the last two iterations of a run of \\irace.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2086 | # Get last iteration number | |
| 2087 | last <- length(iraceResults$iterationElites) | |
| 2088 | # Get configurations in the last two iterations | |
| 2089 | conf <- getConfigurationByIteration(iraceResults = iraceResults, | |
| 2090 | iterations = c(last - 1, last)) | |
| 2091 | parallelCoordinatesPlot (conf, iraceResults$parameters, | |
| 2092 | param_names = c("algorithm", "alpha", "beta", "rho", "q0"), | |
| 2093 | hierarchy = FALSE) | |
| 2094 | @ | |
| 2090 | @ | |
| 2091 | % # Plot the results | |
| 2092 | % configurationsBoxplot (results, ylab = "Solution cost") | |
| 2093 | ||
| 2094 | The Kendall concordance coefficient (\code{W}) and the Spearman's rho can be | |
| 2095 | applied over data that has the characteristics of the data obtained in the testing, | |
| 2096 | that is a full matrix where all configurations are executed in all instances. \code{W} | |
| 2097 | can show if the configurations tested have an homogeneous performance on the used instances | |
| 2098 | set. If evidence of an heterogeneous scenario found we recommend to make some adjustments | |
| 2099 | in the \irace options as described in \autoref{sec:het}. | |
| 2100 | ||
| 2101 | <<conc, prompt=TRUE, eval=TRUE, comment="">>= | |
| 2102 | irace:::concordance(iraceResults$testing$experiments) | |
| 2103 | @ | |
| 2104 | % | |
| 2105 | ||
| 2106 | %% FIXME: iraceplot | |
| 2107 | % During the tuning, \irace iteratively updates the sampling models of the | |
| 2108 | % parameters to focus on the best regions of the parameter search space. The | |
| 2109 | % frequency of the sampled configurations can provide insights on the parameter | |
| 2110 | % search space. We provide a function for plotting the | |
| 2111 | % frequency of the sampling of a set of configurations. For more information on this function, please see the \aR help, type in the | |
| 2112 | % \aR console: \code{?parameterFrequency}. The following example | |
| 2113 | % plots the frequency of the parameters sampled during one \irace run: | |
| 2114 | ||
| 2115 | % <<freq, fig.pos="tbp", fig.cap="Parameters sampling frequency.", out.width="\\textwidth",prompt=TRUE, eval=TRUE, comment="">>= | |
| 2116 | % parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) | |
| 2117 | % @ | |
| 2118 | ||
| 2119 | %% FIXME: iraceplot | |
| 2120 | % By using parallel coordinates plots, it is possible to analyze how the parameters | |
| 2121 | % interact with each other. For more information on this function, please see the \aR help, type | |
| 2122 | % in the \aR console: (\code{?parallelCoordinatesPlot}). | |
| 2123 | % The following example shows how to create a parallel | |
| 2124 | % coordinate plot of the configurations in the last two iterations of \irace. | |
| 2125 | ||
| 2126 | % <<parcord, fig.pos="tbp", fig.align="center", out.width="0.7\\textwidth", fig.cap="Parallel coordinate plots of the parameters of the configurations in the last two iterations of a run of \\irace.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2127 | % # Get last iteration number | |
| 2128 | % last <- length(iraceResults$iterationElites) | |
| 2129 | % # Get configurations in the last two iterations | |
| 2130 | % conf <- getConfigurationByIteration(iraceResults = iraceResults, | |
| 2131 | % iterations = c(last - 1, last)) | |
| 2132 | % parallelCoordinatesPlot (conf, iraceResults$parameters, | |
| 2133 | % param_names = c("algorithm", "alpha", "beta", "rho", | |
| 2134 | % "q0", "rasrank"), | |
| 2135 | % hierarchy = FALSE) | |
| 2136 | % @ | |
| 2095 | 2137 | |
| 2096 | 2138 | %%% FIXME: This plot is completely misleading. We should either calculate the |
| 2097 | 2139 | %%% performance over the whole training set, or find a way to plot the |
| 2099 | 2141 | %%% different number of instances. Or simply plot a heatmap of instances x |
| 2100 | 2142 | %%% best-of-each iteration. |
| 2101 | 2143 | \hide{It is also possible to plot the performance evolution of the best-so-far configuration over the number of experiments as follows: |
| 2102 | %% <<trainEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Training set performance of the best-so-far configuration over number of experiments.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2144 | %%<<trainEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Training set performance of the best-so-far configuration over number of experiments.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2103 | 2145 | # Get number of iterations |
| 2104 | 2146 | iters <- unique(iraceResults$experimentLog[, "iteration"]) |
| 2105 | 2147 | # Get number of experiments (runs of target-runner) up to each iteration |
| 2119 | 2161 | points(fes, values) |
| 2120 | 2162 | } |
| 2121 | 2163 | |
| 2122 | It is also possible to plot the performance on the test set of the best-so-far configuration over the number of experiments as follows: | |
| 2123 | <<testEvo, fig.pos="tbp", fig.align="center", out.width='0.75\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2164 | % FIXME: We should normalize the ranges per instance so since the performance | |
| 2165 | % on different instances is quite different. | |
| 2166 | It is also possible, as shown in Fig.~\ref{fig:testEvo}, to plot the performance on the test set of the best-so-far configuration over the number of experiments as follows: | |
| 2167 | <<testEvo, fig.pos="tb", fig.align="center", out.width='0.7\\textwidth', fig.cap="Testing set performance of the best-so-far configuration over number of experiments. Label of each point is the configuration ID.", prompt=FALSE, eval=TRUE, comment="">>= | |
| 2124 | 2168 | # Get number of iterations |
| 2125 | 2169 | iters <- unique(iraceResults$experimentLog[, "iteration"]) |
| 2126 | 2170 | # Get number of experiments (runs of target-runner) up to each iteration |
| 2129 | 2173 | # for the best configuration of that iteration. |
| 2130 | 2174 | elites <- as.character(iraceResults$iterationElites) |
| 2131 | 2175 | values <- colMeans(iraceResults$testing$experiments[, elites]) |
| 2176 | stderr <- function(x) sqrt(var(x)/length(x)) | |
| 2177 | err <- apply(iraceResults$testing$experiments[, elites], 2, stderr) | |
| 2132 | 2178 | plot(fes, values, type = "s", |
| 2133 | 2179 | xlab = "Number of runs of the target algorithm", |
| 2134 | ylab = "Mean value over testing set") | |
| 2135 | points(fes, values) | |
| 2180 | ylab = "Mean value over testing set", ylim=c(23000000,23500000)) | |
| 2181 | points(fes, values, pch=19) | |
| 2182 | arrows(fes, values - err, fes, values + err, length=0.05, angle=90, code=3) | |
| 2136 | 2183 | text(fes, values, elites, pos = 1) |
| 2137 | 2184 | @ |
| 2138 | 2185 | |
| 2139 | 2186 | The \irace package also provides an implementation of the ablation |
| 2140 | 2187 | method~\cite{FawHoos2015ablation}. See \autoref{sec:ablation}. |
| 2188 | ||
| 2189 | Finally, more advanced visualizations of the behavior of \irace are provided by | |
| 2190 | the ACVIZ software package~\cite{DesRitLopPer2021acviz}, which is available at \url{https://github.com/souzamarcelo/acviz}. See an example in Fig.~\ref{fig:acviz}. | |
| 2191 | ||
| 2192 | \begin{figure}[htb] | |
| 2193 | \centering | |
| 2194 | \includegraphics[width=\textwidth]{fig1u-acotsp-instances} | |
| 2195 | \caption{Visualization produced by ACVIZ~\cite{DesRitLopPer2021acviz}.} | |
| 2196 | \label{fig:acviz} | |
| 2197 | \end{figure} | |
| 2198 | ||
| 2199 | ||
| 2141 | 2200 | |
| 2142 | 2201 | %% |
| 2143 | 2202 | %% |
| 2169 | 2228 | must return the evaluation cost together with the execution time (\code{"cost time"}). |
| 2170 | 2229 | |
| 2171 | 2230 | \begin{xwarningbox} |
| 2172 | When the goal is to minimize the computation time of an algorithm, and you wish to use \parameter{maxTime} as the tuning budget, \parameter{targetRunner} must return the time also as the evaluation cost, that is, return the time two times as \code{"time time"}. | |
| 2231 | When the goal is to minimize the computation time of an algorithm, and you wish to use \parameter{maxTime} as the tuning budget, \parameter{targetRunner} must return the time also as the evaluation cost, that is, return the time twice as \code{"time time"}. | |
| 2173 | 2232 | \end{xwarningbox} |
| 2174 | 2233 | |
| 2175 | 2234 | \begin{xwarningbox} |
| 2230 | 2289 | \subsection{Tuning for minimizing computation time} |
| 2231 | 2290 | \label{sec:capping} |
| 2232 | 2291 | |
| 2233 | When using \irace for tuning algorithms that report computation time to | |
| 2234 | reach a target, \parameter{targetRunner} should return the execution time of a configuration instead of solution cost. | |
| 2235 | ||
| 2236 | %Even though \irace can be used for | |
| 2237 | %minimizing computation time, \irace may itself require more time to do so in | |
| 2238 | %its current version than other methods, such as | |
| 2239 | %\code{ParamILS}\footnote{\url{http://www.cs.ubc.ca/labs/beta/Projects/ParamILS/}} | |
| 2240 | %or \code{SMAC}\footnote{\url{http://www.cs.ubc.ca/labs/beta/Projects/SMAC/}}, | |
| 2241 | %since it does not make use of techniques, such as ``adaptive capping'', that | |
| 2242 | %avoid long runs of the target algorithm. | |
| 2243 | %We are currently extending \irace with an adaptive capping mechanism. | |
| 2292 | When using \irace for tuning algorithms that only report computation time to | |
| 2293 | reach a target, \parameter{targetRunner} should return the execution time of a configuration instead of solution cost. When using \parameter{maxTime} as the budget, this means that \parameter{targetRunner} must return twice the execution time since the first value is the minimization objective and the second value is used to track the budget consumed. | |
| 2244 | 2294 | |
| 2245 | 2295 | Starting from version $3.0$, \irace includes an elitist racing procedure that |
| 2246 | 2296 | implements an \textbf{adaptive capping mechanism} \citep{PerLopHooStu2017:lion}. |
| 2334 | 2384 | |
| 2335 | 2385 | |
| 2336 | 2386 | |
| 2337 | \subsection{Hyper-pararameter optimization of machine learning methods} | |
| 2338 | ||
| 2339 | The \irace package can also be used for model selection and hyper-parameter optimization | |
| 2340 | of machine learning methods. For such a task, we recommend the \pkg{mlr} package~\citep{R:mlr}. | |
| 2341 | The following webpage documents how to use \irace for this purpose: \url{https://mlr-org.github.io/mlr-tutorial/devel/html/advanced_tune/index.html} | |
| 2387 | \subsection{Hyper-parameter optimization of machine learning methods} | |
| 2388 | ||
| 2389 | The \irace package can also be used for model selection and hyper-parameter | |
| 2390 | optimization of machine learning (ML) methods. We will next explain a possible | |
| 2391 | setup for one given dataset and using $10$-fold cross-validation | |
| 2392 | (CV). Generalizing to multiple datasets and different resampling strategies, | |
| 2393 | e.g. leave-one-out, is straightforward. | |
| 2394 | ||
| 2395 | First, split the dataset into training, to be used by \irace, and testing, to | |
| 2396 | be used for evaluating the performance of the configuration returned by | |
| 2397 | \irace. A typical split could be $70\%$ and $30\%$, respectively. | |
| 2398 | ||
| 2399 | The training set is used by \irace to perform $10$-fold CV, that is, the data | |
| 2400 | is split into $10$ folds. A single run of the \parameter{targetRunner} will use | |
| 2401 | $9$ folds for training and the remaining fold for validation. Splitting the | |
| 2402 | data into folds can be done at each call of \parameter{targetRunner} or before | |
| 2403 | running \irace, however, it is important that the split is always the same for | |
| 2404 | every call of the \parameter{targetRunner}, i.e., the content of the folds does | |
| 2405 | not change, only which folds are used for training and validation will change. | |
| 2406 | ||
| 2407 | The setup of \irace should be as follows: | |
| 2408 | % | |
| 2409 | \begin{itemize} | |
| 2410 | \item \parameter{trainInstancesFile}\code{="train-instances.txt"}, where this | |
| 2411 | file contains one number per line from $1$ to $10$. This number will tell the | |
| 2412 | \parameter{targetRunner} which fold should be used for validation. | |
| 2413 | \item \parameter{trainInstancesDir}\code{=""}, because the folds are the | |
| 2414 | ``instances'' and you do not have actual instance files. If you want to pass | |
| 2415 | the name of the dataset to the \parameter{targetRunner}, you can specify it | |
| 2416 | either at each line of \code{"train-instances.txt"}, directly in the | |
| 2417 | \parameter{targetRunner}, or as a fixed parameter in the | |
| 2418 | \parameter{parameterFile}. | |
| 2419 | \item \parameter{deterministic}\code{=1} unless it really makes sense to | |
| 2420 | train more than once the same ML model on the same data. If it makes sense, | |
| 2421 | then your \parameter{targetRunner} should use the seed passed by \irace to | |
| 2422 | seed the ML model before training. | |
| 2423 | \item \parameter{sampleInstances}\code{=0} because the folds should already be generated by randomly sampling the dataset. | |
| 2424 | \item \parameter{testType}\code{="t-test} because the performance metrics in | |
| 2425 | ML are typically the mean of the CV results, which assumes that the | |
| 2426 | performance are close to normally distributed. | |
| 2427 | \item \parameter{firstTest}\code{=2} because \irace should discard | |
| 2428 | configurations very aggressively looking for maximum generality. | |
| 2429 | \end{itemize} | |
| 2430 | ||
| 2431 | \noindent% | |
| 2432 | Finally, your \parameter{targetRunner} needs to be able to do the following: | |
| 2433 | \begin{itemize} | |
| 2434 | \item Receive from \irace the hyper-parameter settings, the dataset name and a | |
| 2435 | fold number (the ``instance''). Let us use fold 3 as an example. | |
| 2436 | \item Train the ML model on the whole training set minus fold 3, then validate | |
| 2437 | (score) the model on fold 3 and return the score to \irace (negated if the | |
| 2438 | score must be maximized, because \irace assumes minimization). Since each | |
| 2439 | fold is different, each instance should give a different result. Each row in | |
| 2440 | the table printed by \irace should print something different; otherwise, | |
| 2441 | something is wrong in your setup. | |
| 2442 | \end{itemize} | |
| 2443 | ||
| 2444 | The above is actually 10 times faster than doing 10-fold CV for each call to | |
| 2445 | \parameter{targetRunner}, thus, you should assign to \irace 10 times the budget | |
| 2446 | than what would be assigned to other methods that do a complete 10-fold CV at | |
| 2447 | each step. | |
| 2448 | ||
| 2449 | %% FIXME: They removed the tutorial | |
| 2450 | %% For such a task, we recommend the \pkg{mlr} package~\citep{R:mlr}. | |
| 2451 | %% The following webpage documents how to use \irace for this purpose: \url{https://mlr-org.github.io/mlr-tutorial/devel/html/advanced_tune/index.html} | |
| 2342 | 2452 | |
| 2343 | 2453 | |
| 2344 | 2454 | \subsection{Heterogeneous scenarios} |
| 2361 | 2471 | selection techniques can be used to select the best configuration from the |
| 2362 | 2472 | portfolio when facing a new instance. |
| 2363 | 2473 | |
| 2474 | To make sure \irace is not misled by results on few instances, it may be useful | |
| 2475 | to increase the number of instances executed before doing a statistical test | |
| 2476 | using the option \parameter{firstTest}, \eg \code{--first-test 10} (default | |
| 2477 | value is 5), in order to see more instances before discarding configurations. | |
| 2478 | The option \parameter{elitistNewInstances} in elitist \irace (option | |
| 2479 | \parameter{elitist}) can be used to increase the number of new instances | |
| 2480 | executed in each iteration, \eg \code{--elitist-new-instances 5} (default value | |
| 2481 | is 1). | |
| 2482 | ||
| 2364 | 2483 | If finding an overall good configuration for all the instances is the |
| 2365 | objective, then we recommend that instances are randomly sampled | |
| 2366 | (option \parameter{sampleInstances}), unless one can provide the instances in a | |
| 2484 | objective, then we recommend that instances are randomly sampled (option | |
| 2485 | \parameter{sampleInstances}), unless one can provide the instances in a | |
| 2367 | 2486 | particular order that does not bias the tuning towards any subset. For |
| 2368 | example, let's assume a heterogeneous scenario with two classes of instances. If | |
| 2369 | training instances are not sampled and the first ten instances belong to only | |
| 2370 | one class, the tuning will be biased towards configurations that perform good | |
| 2371 | for those instances. An optimal order would not ever present consecutively two | |
| 2372 | instances of the same type. | |
| 2373 | ||
| 2374 | In addition, it may be useful to increase the number of instances executed | |
| 2375 | before doing a statistical test in order to see more instance classes before | |
| 2376 | discarding configurations. The option \parameter{elitistNewInstances} in elitist | |
| 2377 | \irace (option \parameter{elitist}) can be used to increase the number of new | |
| 2378 | instances executed in each iteration, \eg \code{--elitist-new-instances 5} (default | |
| 2379 | value is 1). For the non-elitist \irace, the option \parameter{firstTest} may be used for the same purpose, \eg \code{--first-test 10} (default value is 5). | |
| 2380 | \MANUEL{And both at the same time?} | |
| 2487 | example, let's assume a heterogeneous scenario with two classes of instances. | |
| 2488 | If training instances are not sampled and the first ten instances belong to | |
| 2489 | only one class, the tuning will be biased towards configurations that perform | |
| 2490 | good for those instances. An optimal order would never present consecutively | |
| 2491 | two instances of the same type. One can make sure \irace sees all instance | |
| 2492 | classes before discarding configurations by carefully | |
| 2493 | specifying the order of instances (disabling \parameter{sampleInstances} and | |
| 2494 | specifying the order in \parameter{trainInstancesFile}) and setting both | |
| 2495 | \parameter{firstTest} and \parameter{eachTest} appropriately. | |
| 2381 | 2496 | |
| 2382 | 2497 | While executing \irace, the homogeneity of the scenario can be observed by |
| 2383 | 2498 | examining the values of Spearman's rank correlation coefficient and Kendall's |
| 2431 | 2546 | them and, therefore, it is more likely to discard good configurations. |
| 2432 | 2547 | |
| 2433 | 2548 | |
| 2434 | \subsection{Complex parameter space constraints} | |
| 2549 | \subsection{Complex parameter space constraints}\label{sec:complex_domains} | |
| 2435 | 2550 | |
| 2436 | 2551 | Some parameters may have complex dependencies. Ideally, parameters should be |
| 2437 | 2552 | defined in the way that is more likely to help the search performed by |
| 2553 | 2668 | values from target. The sequence can be seen as a ``path'' from the source to |
| 2554 | 2669 | the target configuration. This can be used to find new better ``intermediate'' |
| 2555 | 2670 | configurations or to analyse the impact of the parameters in the performance. |
| 2556 | To perform ablation use the \code{ablation} function and specify the IDs of the | |
| 2557 | source and target configurations. By default, the source is taken as the first | |
| 2558 | configuration evaluated by \irace and the target as the best overall | |
| 2559 | configuration found. The argument \code{ab.params} can be used to specify a | |
| 2560 | subset of the parameters considered in the ablation. The option | |
| 2671 | ||
| 2672 | ||
| 2673 | To perform ablation, you can use the \code{ablation} function and specify the | |
| 2674 | IDs of the source and target configurations. By default, the source is taken | |
| 2675 | as the first configuration evaluated by \irace and the target as the best | |
| 2676 | overall configuration found. The argument \code{ab.params} can be used to | |
| 2677 | specify a subset of the parameters considered in the ablation. The option | |
| 2561 | 2678 | \parameter{firstTest} defines how many instances are selected for the |
| 2562 | 2679 | evaluation of configurations, if a different number of instances is required it |
| 2563 | must be specified in the argument \code{n.instance}. If a PDF filename is | |
| 2564 | provided (\code{pdf.file}), a plot will be produced from the ablation results | |
| 2565 | (Fig.~\ref{fig:testAb}). | |
| 2680 | must be specified in the argument \code{n.instance}. Use the function | |
| 2681 | \code{plotAblation} to visualize the ablation results (Fig.~\ref{fig:testAb}). | |
| 2566 | 2682 | |
| 2567 | 2683 | <<ablation, prompt=FALSE, eval=FALSE>>= |
| 2568 | ablation(iraceLogFile = "irace.Rdata", | |
| 2569 | src = 1, target = 60, pdf.file = "plot-ablation.pdf") | |
| 2570 | @ | |
| 2571 | ||
| 2572 | <<testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{ablation()}.", prompt=FALSE, eval=TRUE, echo=FALSE>>= | |
| 2573 | plotAblation(abLogFile = "log-ablation.Rdata") | |
| 2684 | ablog <- ablation("irace.Rdata", src = 1, target = 60) | |
| 2685 | plotAblation(ablog) | |
| 2686 | @ | |
| 2687 | ||
| 2688 | <<testAb, fig.pos="htb!", fig.align="center", out.width="0.75\\textwidth", fig.cap="Example of plot generated by \\code{plotAblation()}.", prompt=FALSE, eval=TRUE, echo=FALSE>>= | |
| 2689 | logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") | |
| 2690 | plotAblation(logfile) | |
| 2574 | 2691 | @ |
| 2575 | 2692 | |
| 2576 | 2693 | The function returns a list containing the following elements: |
| 2583 | 2700 | \item \code{best}: Best overall configuration found. |
| 2584 | 2701 | \end{description} |
| 2585 | 2702 | |
| 2703 | We also provide a command-line executable that allows you to perform ablation without launching \aR. It is installed in the same location as the \irace command-line executable and has the following options: | |
| 2704 | <<ablation_cmdline, prompt=FALSE, eval=TRUE,echo=FALSE>>= | |
| 2705 | ablation_cmdline("--help") | |
| 2706 | @ | |
| 2586 | 2707 | |
| 2587 | 2708 | \subsection{Postselection race}\label{sec:postselection} |
| 2588 | 2709 | |
| 2633 | 2754 | Although ablation analysis without surrogates may be more time-consuming, results of the surrogate version may be less accurate than the non-surrogate one. |
| 2634 | 2755 | |
| 2635 | 2756 | The \code{.Rdata} dataset generated by \irace can be used as input for \PyImp. |
| 2636 | We provide a script to translate an \code{irace.Rdata} file into the input format required by \PyImp. | |
| 2637 | The script is available in the \irace package, and can be accessed either through the \aR console (function \code{irace2pyimp}), or via command line (\IRACEHOME{/bin/irace2pyimp}). | |
| 2757 | The package \code{irace2pyimp}\footnote{\url{https://github.com/ndangtt/irace2pyimp}} is able to convert an \code{irace.Rdata} file into the input format required by \PyImp. The conversion can be accessed either through the \aR console (function \code{irace2pyimp}), or via command line (\code{system.file("/bin/irace2pyimp", package="irace2pyimp")}). | |
| 2638 | 2758 | |
| 2639 | 2759 | To see the usage of the executable, please run: \code{irace2pyimp --help}. |
| 2640 | 2760 | For more information on the \aR function \code{irace2pyimp}, type in the \aR console: \code{?irace2pyimp}. |
| 2649 | 2769 | \item \code{features.csv}: a .csv file containing instance features. If no instance features are provided, the index of each instance will be used as a feature. |
| 2650 | 2770 | \end{itemize} |
| 2651 | 2771 | |
| 2652 | \PyImp can then be called using the files listed above as input. Several examples on how to use the script and to call \PyImp can be found at \IRACEHOME{/inst/examples/irace2pyimp/}. | |
| 2772 | \PyImp can then be called using the files listed above as input. Several | |
| 2773 | examples on how to use the script and call \PyImp can be found at | |
| 2774 | \code{system.file("/examples/", package="irace2pyimp")}. | |
| 2653 | 2775 | |
| 2654 | 2776 | %\emph{\PyImp installation note:} At the time when this document is written (December 2019), the latest \PyImp release version (1.0.6) has some bugs. |
| 2655 | 2777 | %Please check on \url{https://pypi.org/project/PyImp/#history} to see if \PyImp version > 1.0.6 has been released. |
| 2813 | 2935 | This is a warning given by \aR when the last line of an input file does not |
| 2814 | 2936 | finish with the newline character. The warning is harmless and can be |
| 2815 | 2937 | ignored. If you want to suppress it, just open the file and press the |
| 2816 | \code{ENTER} key at the end of the last line of the file. | |
| 2938 | \code{ENTER} key at the end of the last line of the file to end the final line with a newline. | |
| 2817 | 2939 | |
| 2818 | 2940 | |
| 2819 | 2941 | \subsection{How are relative filesystem paths interpreted by \irace?}\label{faq:relpaths} |
| 2843 | 2965 | |
| 2844 | 2966 | \subsection{My parameter space is small enough that \irace could generate all possible configurations; however, \irace generates repeated configurations and/or does not generate some of them. Is this a bug?}\label{faq:allconfs} |
| 2845 | 2967 | |
| 2846 | Currently, \irace does not try to detect whether all possible configurations | |
| 2847 | can be evaluated for the given budget, thus, the initial random sampling | |
| 2968 | Typically, \irace is applied to parameter spaces that are much larger than what | |
| 2969 | can be explored within the budget given. Thus, \irace does not try to detect | |
| 2970 | whether all possible configurations can be evaluated for the given budget and | |
| 2971 | it does not waste computation time to check for repeated configurations. Thus, | |
| 2972 | if the parameter space is actually very small, the initial random sampling | |
| 2848 | 2973 | performed by \irace may generate repeated configurations and/or never generate |
| 2849 | some configurations, which is not ideal. The ideal approach in such cases is to | |
| 2850 | provide all configurations explicitly to \irace (\autoref{sec:initial}) and | |
| 2851 | execute a single race (\parameter{nbIterations}\code{=1}) with exactly the | |
| 2852 | number of configurations provided (e.g., | |
| 2853 | \parameter{nbConfigurations}\code{=10}). A future version of \irace will | |
| 2854 | automatically detect this case and switch to basic racing without having to set additional options. | |
| 2974 | some configurations, which is not ideal. If you still want to use | |
| 2975 | (non-iterated) racing, the recommended approach is to provide all | |
| 2976 | configurations explicitly to \irace (\autoref{sec:initial}) and execute a | |
| 2977 | single race (\parameter{nbIterations}\code{=1}) with exactly the number of | |
| 2978 | configurations provided (e.g., \parameter{nbConfigurations}\code{=240}). A | |
| 2979 | future version of \irace may automatically detect this case and switch to | |
| 2980 | non-iterated racing without having to set additional options. Future versions | |
| 2981 | may also implement computationally cheap checks for repeated | |
| 2982 | configurations.\footnote{If you are interested in implementing this, please | |
| 2983 | contact us!} | |
| 2984 | ||
| 2985 | \subsection[On Windows and using target-runner.py (a Python file), I | |
| 2986 | get the error target-runner.py is not executable]{On Windows and using \code{target-runner.py} (a \proglang{Python} file), I | |
| 2987 | get the error ``\code{target-runner.py is not executable}''}\label{faq:py-not-exe} | |
| 2988 | ||
| 2989 | The issue is that \code{.py} files are not executable on their own and you need \code{python.exe} to read the \code{.py} file and execute it. Linux knows how to do this if the first line of the file is ``\code{\#!/usr/bin/python}'', however, Windows doesn't know how to do it. In Windows you have 2 options: | |
| 2990 | \begin{itemize} | |
| 2991 | \item Create a \code{target-runner.bat} file that contains a line similar to (see \path{templates/windows/target-runner.bat}): | |
| 2992 | \begin{lstlisting}[style=BashInputStyle] | |
| 2993 | C:\path\to\python.exe C:\path\to\target-runner.py %instance% %seed% \ | |
| 2994 | %candidate_parameters% 1>%stdout% 2>%stderr% | |
| 2995 | \end{lstlisting} | |
| 2996 | ||
| 2997 | \item Or convert \code{target-runner.py} into an \code{.exe} file, for example, using \code{auto-py-to-exe}\footnote{\url{https://pypi.org/project/auto-py-to-exe/}}, so that you do not need a \code{.bat} file. | |
| 2998 | \end{itemize} | |
| 2999 | ||
| 3000 | \subsection[Error in socketConnection(\ldots) : can not open the connection]{Error in \code{socketConnection("localhost", port = port, server = TRUE, lock = TRUE,} : can not open the connection} | |
| 3001 | ||
| 3002 | This error may arise if you activate the \parameter{parallel} option of \irace and your \parameter{targetRunner} or \parameter{targetEvaluator} tries to setup a parallel cluster or execute code in parallel in a way that interacts badly with the parallel mechanism in \aR. In this case, you need to either investigate yourself if there is a way for the two parallel mechanisms to co-exist or, if that is not possible, disable parallelism in \irace or in your code. Note that packages or software used by your \parameter{targetRunner} may have a parallel mechanism enabled by default and unknown to you. This is definitely NOT a bug in \irace. | |
| 2855 | 3003 | |
| 2856 | 3004 | %% |
| 2857 | 3005 | %% |
| 2868 | 3016 | \section{Resources and contact information} \label{sec:contact} |
| 2869 | 3017 | |
| 2870 | 3018 | More information about the package can be found on the \irace webpage: |
| 2871 | \begin{center} \url{http://iridia.ulb.ac.be/irace/} \end{center} | |
| 3019 | \begin{center} \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} \end{center} | |
| 2872 | 3020 | |
| 2873 | 3021 | For questions and suggestions please contact the development team through |
| 2874 | 3022 | the \irace package Google group: |
| 2879 | 3027 | \begin{center} |
| 2880 | 3028 | \href{mailto:irace-package@googlegroups.com}{irace-package@googlegroups.com} |
| 2881 | 3029 | \end{center} |
| 2882 | ||
| 2883 | %% MANUEL: I think we should use just one channel. If the Google group is the winner, that should be it. | |
| 2884 | %the mailing list: | |
| 2885 | %% LESLIE I agree. | |
| 2886 | %\begin{center} \href{mailto:irace@iridia.ulb.ac.be}{irace@iridia.ulb.ac.be} \end{center} | |
| 2887 | 3030 | |
| 2888 | 3031 | |
| 2889 | 3032 | \section{Acknowledgements} |
| 0 | 0 | |
| 1 | 1 | |
| 2 | @preamble{{\providecommand{\MaxMinAntSystem}{{$\cal MAX$--$\cal MIN$} {A}nt {S}ystem} } # {\providecommand{\Rpackage}[1]{#1} } # {\providecommand{\SoftwarePackage}[1]{#1} } # {\providecommand{\proglang}[1]{#1} }} | |
| 2 | @preamble{{\providecommand{\MaxMinAntSystem}{{$\cal MAX$--$\cal MIN$} {Ant} {System}} } # {\providecommand{\rpackage}[1]{{#1}} } # {\providecommand{\softwarepackage}[1]{{#1}} } # {\providecommand{\proglang}[1]{{#1}} }} | |
| 3 | 3 | |
| 4 | 4 | @string{and = { and }} |
| 5 | 5 | |
| 6 | 6 | @string{lncs = {Lecture Notes in Computer Science}} |
| 7 | 7 | |
| 8 | @string{add-berlin = { Berlin, Germany }} | |
| 9 | ||
| 10 | @string{add-cham = { Cham, Switzerland }} | |
| 11 | ||
| 12 | @string{add-heidelberg = { Heidelberg }} | |
| 13 | ||
| 8 | 14 | @string{aaaip-pub = {{AAAI} Press}} |
| 9 | 15 | |
| 10 | 16 | @string{ieeep = {IEEE Press}} |
| 13 | 19 | |
| 14 | 20 | @string{springer = {Springer}} |
| 15 | 21 | |
| 16 | @string{springer-berlin-ad = {Berlin, Germany}} | |
| 17 | ||
| 18 | @string{springer-lncs = {Springer, Heidelberg, Germany}} | |
| 19 | ||
| 20 | @string{add-cham = {Cham, Switzerland}} | |
| 21 | ||
| 22 | 22 | @string{iridia = {IRIDIA, Universit{\'e} Libre de Bruxelles, Belgium}} |
| 23 | 23 | |
| 24 | @string{cec06 = {Proceedings of the 2006 Congress on Evolutionary | |
| 25 | Computation (CEC 2006)}} | |
| 24 | @string{proc_of = {Proceedings of }} | |
| 25 | ||
| 26 | @string{proc_of_the = proc_of # { the }} | |
| 27 | ||
| 28 | @string{aaai = proc_of_the # {{AAAI} Conference on Artificial Intelligence}} | |
| 29 | ||
| 30 | @string{cec = { Congress on Evolutionary Computation (CEC}} | |
| 31 | ||
| 32 | @string{cec06 = proc_of_the # {2006} # cec # { 2006)}} | |
| 33 | ||
| 34 | @string{icml = { International Conference on Machine Learning, {ICML} }} | |
| 35 | ||
| 36 | @string{icml2014 = proc_of_the # {31st} # icml # {2014}} | |
| 26 | 37 | |
| 27 | 38 | @string{acm-cs = {{ACM} Computing Surveys}} |
| 28 | 39 | |
| 38 | 49 | |
| 39 | 50 | @string{battiti = { Roberto Battiti }} |
| 40 | 51 | |
| 52 | @string{biedenkapp = { Biedenkapp, Andr{\'e} }} | |
| 53 | ||
| 41 | 54 | @string{birattari = { Mauro Birattari }} |
| 42 | 55 | |
| 43 | @string{bischl_b = { Bernd Bischl }} | |
| 44 | ||
| 45 | 56 | @string{chiarandini = { Marco Chiarandini }} |
| 46 | 57 | |
| 58 | @string{desouza = { Marcelo {De Souza} }} | |
| 59 | ||
| 47 | 60 | @string{dubois-lacoste = { J{\'e}r{\'e}mie Dubois-Lacoste }} |
| 48 | 61 | |
| 49 | 62 | @string{fawcett = { Chris Fawcett }} |
| 58 | 71 | |
| 59 | 72 | @string{leyton-brown = { Kevin Leyton-Brown }} |
| 60 | 73 | |
| 74 | @string{lindauer_m = { Marius Thomas Lindauer }} | |
| 75 | ||
| 61 | 76 | @string{lopez-ibanez = { Manuel L{\'o}pez-Ib{\'a}{\~n}ez }} |
| 62 | 77 | |
| 63 | 78 | @string{mcgeoch_cc = { Catherine C. McGeoch }} |
| 66 | 81 | |
| 67 | 82 | @string{pardalos = { Panos M. Pardalos }} |
| 68 | 83 | |
| 69 | @string{perez_l = { Leslie {P{\'e}rez C{\'a}ceres} }} | |
| 84 | @string{perez_l = { P{\'e}rez C{\'a}ceres, Leslie}} | |
| 70 | 85 | |
| 71 | 86 | @string{preuss_m = { Mike Preuss }} |
| 72 | 87 | |
| 73 | @string{schneider = { Marius Schneider }} | |
| 88 | @string{ritt = { Marcus Ritt}} | |
| 89 | ||
| 90 | @string{schneider_m = { Marius Schneider }} | |
| 74 | 91 | |
| 75 | 92 | @string{schoenauer = { Marc Schoenauer }} |
| 76 | 93 | |
| 79 | 96 | @string{yuan_z = { Zhi Yuan }} |
| 80 | 97 | |
| 81 | 98 | @inproceedings{BieLinEggFraFawHoo2017, |
| 82 | author = {Biedenkapp, Andr{\'e} and Lindauer, Marius and Eggensperger, | |
| 99 | author = biedenkapp # and # lindauer_m # and # {Eggensperger, | |
| 83 | 100 | Katharina} # and # hutter # and # fawcett # and # hoos, |
| 84 | 101 | title = {Efficient Parameter Importance Analysis via Ablation with |
| 85 | 102 | Surrogates}, |
| 92 | 109 | title = {{F}-Race and Iterated {F}-Race: An Overview}, |
| 93 | 110 | pages = {311--336}, |
| 94 | 111 | crossref = {BarChiPaqPre2010emaoa}, |
| 95 | keywords = {F-race, iterated F-race, irace, tuning} | |
| 112 | keywords = {F-race, iterated F-race, irace, tuning}, | |
| 113 | doi = {10.1007/978-3-642-02538-9_13} | |
| 114 | } | |
| 115 | ||
| 116 | @article{DesRitLopPer2021acviz, | |
| 117 | author = desouza # and # ritt # and # lopez-ibanez # and # perez_l, | |
| 118 | title = {{\softwarepackage{ACVIZ}}: A Tool for the Visual Analysis of | |
| 119 | the Configuration of Algorithms with {\rpackage{irace}}}, | |
| 120 | journal = orp, | |
| 121 | year = 2021, | |
| 122 | doi = {10.1016/j.orp.2021.100186}, | |
| 123 | supplement = {https://zenodo.org/record/4714582}, | |
| 124 | abstract = {This paper introduces acviz, a tool that helps to analyze the | |
| 125 | automatic configuration of algorithms with irace. It provides | |
| 126 | a visual representation of the configuration process, | |
| 127 | allowing users to extract useful information, e.g. how the | |
| 128 | configurations evolve over time. When test data is available, | |
| 129 | acviz also shows the performance of each configuration on the | |
| 130 | test instances. Using this visualization, users can analyze | |
| 131 | and compare the quality of the resulting configurations and | |
| 132 | observe the performance differences on training and test | |
| 133 | instances.}, | |
| 134 | volume = 8, | |
| 135 | pages = 100186 | |
| 96 | 136 | } |
| 97 | 137 | |
| 98 | 138 | @article{FawHoos2015ablation, |
| 108 | 148 | |
| 109 | 149 | @incollection{FonPaqLop06:hypervolume, |
| 110 | 150 | author = fonseca # and # paquete # and # lopez-ibanez, |
| 111 | title = {An improved dimension\hspace{0pt}-\hspace{0pt}sweep | |
| 151 | title = {An improved dimension-\hspace{0pt}sweep | |
| 112 | 152 | algorithm for the hypervolume indicator}, |
| 113 | 153 | crossref = {CEC2006}, |
| 114 | 154 | pages = {1157--1163}, |
| 132 | 172 | |
| 133 | 173 | @incollection{HutHooLey2013lion, |
| 134 | 174 | author = hutter # and # hoos # and # leyton-brown, |
| 135 | title = {Identifying key algorithm parameters and instance features | |
| 136 | using forward selection}, | |
| 175 | title = {Identifying Key Algorithm Parameters and Instance Features | |
| 176 | using Forward Selection}, | |
| 137 | 177 | crossref = {LION2013}, |
| 138 | 178 | pages = {364--381}, |
| 139 | 179 | doi = {10.1007/978-3-642-44973-4_40}, |
| 152 | 192 | |
| 153 | 193 | @article{HutHooLeyStu2009jair, |
| 154 | 194 | author = hutter # and # hoos # and # leyton-brown # and # stuetzle, |
| 155 | title = {{ParamILS:} An Automatic Algorithm Configuration | |
| 156 | Framework}, | |
| 195 | title = {{\softwarepackage{ParamILS}:} An Automatic Algorithm Configuration Framework}, | |
| 157 | 196 | journal = jair, |
| 158 | 197 | year = 2009, |
| 159 | 198 | volume = 36, |
| 160 | 199 | pages = {267--306}, |
| 161 | month = oct | |
| 200 | month = oct, | |
| 201 | doi = {10.1613/jair.2861} | |
| 162 | 202 | } |
| 163 | 203 | |
| 164 | 204 | @techreport{IRIDIA-2004-001, |
| 171 | 211 | |
| 172 | 212 | @article{LopDubPerStuBir2016irace, |
| 173 | 213 | author = lopez-ibanez # and # dubois-lacoste # and # perez_l # and # stuetzle # and # birattari, |
| 174 | title = {The {\Rpackage{irace}} package: Iterated Racing for Automatic | |
| 214 | title = {The {\rpackage{irace}} Package: Iterated Racing for Automatic | |
| 175 | 215 | Algorithm Configuration}, |
| 176 | 216 | journal = orp, |
| 177 | 217 | year = 2016, |
| 183 | 223 | |
| 184 | 224 | @techreport{LopDubStu2011irace, |
| 185 | 225 | author = lopez-ibanez # and # dubois-lacoste # and # stuetzle # and # birattari, |
| 186 | title = {The {\Rpackage{irace}} package, Iterated Race for Automatic | |
| 226 | title = {The {\rpackage{irace}} package, Iterated Race for Automatic | |
| 187 | 227 | Algorithm Configuration}, |
| 188 | 228 | institution = iridia, |
| 189 | 229 | year = 2011, |
| 228 | 268 | |
| 229 | 269 | @incollection{PerLopHooStu2017:lion, |
| 230 | 270 | author = perez_l # and # lopez-ibanez # and # hoos # and # stuetzle, |
| 231 | title = {An experimental study of adaptive capping in irace}, | |
| 271 | title = {An Experimental Study of Adaptive Capping in {\rpackage{irace}}}, | |
| 232 | 272 | crossref = {LION2017}, |
| 233 | 273 | pages = {235--250}, |
| 234 | 274 | pdf = {PerLopHooStu2017lion.pdf}, |
| 235 | 275 | doi = {10.1007/978-3-319-69404-7_17}, |
| 236 | 276 | supplement = {http://iridia.ulb.ac.be/supp/IridiaSupp2016-007/} |
| 237 | } | |
| 238 | ||
| 239 | @manual{R:mlr, | |
| 240 | title = {{\Rpackage{mlr}}: Machine Learning in \proglang{R}}, | |
| 241 | author = bischl_b # and # {Michel Lang and Jakob Bossek and Leonard Judt and Jakob Richter and Tobias Kuehn and Erich Studerus}, | |
| 242 | year = 2013, | |
| 243 | note = {\proglang{R} package}, | |
| 244 | url = {http://cran.r-project.org/package=mlr} | |
| 245 | 277 | } |
| 246 | 278 | |
| 247 | 279 | @incollection{SchHoo2012quanti, |
| 250 | 282 | crossref = {LION2012}, |
| 251 | 283 | keywords = {Quantifying Homogeneity; Empirical Analysis; |
| 252 | 284 | Parameter Optimization; Algorithm Configuration}, |
| 253 | author = schneider # and # hoos, | |
| 285 | author = schneider_m # and # hoos, | |
| 254 | 286 | pages = {190--204}, |
| 255 | 287 | doi = {10.1007/978-3-642-34413-8_14} |
| 256 | 288 | } |
| 257 | 289 | |
| 258 | 290 | @book{AAAI2017, |
| 259 | booktitle = {AAAI Conference on Artificial Intelligence}, | |
| 291 | booktitle = aaai, | |
| 260 | 292 | editor = {Satinder P. Singh and Shaul Markovitch}, |
| 261 | 293 | title = {Proceedings of the Thirty-First {AAAI} Conference on |
| 262 | 294 | Artificial Intelligence, February 4-9, 2017, San Francisco, |
| 272 | 304 | booktitle = {Experimental Methods for the Analysis of |
| 273 | 305 | Optimization Algorithms}, |
| 274 | 306 | publisher = springer, |
| 275 | address = springer-berlin-ad, | |
| 307 | address = add-berlin, | |
| 276 | 308 | year = 2010, |
| 277 | 309 | editor = bartz-beielstein # and # chiarandini # and # paquete # and # preuss_m |
| 278 | 310 | } |
| 290 | 322 | |
| 291 | 323 | @proceedings{ICML2014, |
| 292 | 324 | editor = {Xing, Eric P. and Jebara, Tony}, |
| 293 | title = {Proceedings of the 31th International Conference on Machine | |
| 325 | title = {Proceedings of the 31st International Conference on Machine | |
| 294 | 326 | Learning, {ICML} 2014, Beijing, China, 21-26 June 2014}, |
| 295 | booktitle = {Proceedings of the 31th International Conference on Machine | |
| 296 | Learning}, | |
| 327 | booktitle = icml2014, | |
| 297 | 328 | volume = 32, |
| 298 | 329 | year = 2014, |
| 299 | 330 | url = {http://jmlr.org/proceedings/papers/v32/} |
| 306 | 337 | International Conference, LION 6}, |
| 307 | 338 | year = 2012, |
| 308 | 339 | series = lncs, |
| 309 | publisher = springer-lncs, | |
| 340 | publisher = springer, | |
| 341 | address = add-heidelberg, | |
| 310 | 342 | editor = hamadi # and # schoenauer, |
| 311 | 343 | volume = 7219 |
| 312 | 344 | } |
| 320 | 352 | series = lncs, |
| 321 | 353 | editor = pardalos # and # {G. Nicosia}, |
| 322 | 354 | volume = 7997, |
| 323 | publisher = springer-lncs | |
| 355 | publisher = springer, | |
| 356 | address = add-heidelberg | |
| 324 | 357 | } |
| 325 | 358 | |
| 326 | 359 | @book{LION2017, |
| 10 | 10 | |
| 11 | 11 | \defparameter[c]{-{}-check}{check}{}% |
| 12 | 12 | Check that the scenario and parameter definitions are correct and test the execution of the target algorithm. See \autoref{sec:execution}. |
| 13 | ||
| 14 | \defparameter[i]{-{}-init}{init}{}% | |
| 15 | Initialize the working directory with the template config files. This copies the files in \code{\$IRACE_HOME/templates} to the working directory without overwriting the files with the same names as those of the template files. | |
| 13 | 16 | |
| 14 | 17 | \defparameter[s]{scenarioFile}{scenario}{./scenario.txt}% |
| 15 | 18 | File that contains the scenario setup and other irace options. All options listed in this section can be included in this file. See \IRACEHOME{/templates/} for an example. Relative file-system paths specified in the scenario file are relative to the scenario file itself. |
| 67 | 70 | |
| 68 | 71 | \subsection[Internal irace options]{Internal \irace options} |
| 69 | 72 | \begin{description} |
| 73 | \defparameter{sampleInstances}{sample-instances}{1}% | |
| 74 | Enable/disable the sampling of the training instances. If the option \parameter{sampleInstances} is disabled, the instances are used in the order provided in the \parameter{trainInstancesFile} or in the order they are read from the \parameter{trainInstancesDir} when\parameter{trainInstancesFile} is not provided. For more information about training instances see \autoref{sec:training}. | |
| 75 | ||
| 76 | \defparameter{softRestart}{soft-restart}{1}% | |
| 77 | Enable/disable the soft-restart strategy that avoids premature convergence of the probabilistic model. When a sampled configuration is \emph{similar} to its parent configuration, the probabilistic model of these configurations is soft restarted. The soft-restart mechanism is explained in the \irace paper~\citep{LopDubPerStuBir2016irace}. The similarity of categorical and ordinal parameters is given by the hamming distance, and the option \parameter{softRestartThreshold} defines the similarity of numerical parameters. | |
| 78 | ||
| 79 | \defparameter{softRestartThreshold}{soft-restart-threshold}{}% | |
| 80 | Soft restart threshold value for numerical parameters. By default, it is computed as $10^{-digits}$, where \parameter{digits} corresponds to the \irace option explained in this section. | |
| 81 | ||
| 70 | 82 | \defparameter{nbIterations}{iterations}{0}% |
| 71 | Minimum number of iterations to be executed. Each iteration involves the generation of new configurations and the use of racing to select the best configurations. By default (with 0), \irace calculates the minimum number of iterations as $\Niter = \lfloor 2 + \log_{2}\Nparam \rfloor$, where $\Nparam$ is the number of non-fixed parameters to be tuned. We recommend to use the default value. | |
| 83 | Maximum number of iterations to be executed. Each iteration involves the generation of new configurations and the use of racing to select the best configurations. By default (with 0), \irace calculates a \emph{minimum} number of iterations as $\Niter = \lfloor 2 + \log_{2}\Nparam \rfloor$, where $\Nparam$ is the number of non-fixed parameters to be tuned. Setting this parameter may make \irace stop sooner than it should without using all the available budget. We recommend to use the default value. | |
| 72 | 84 | |
| 73 | 85 | \defparameter{nbExperimentsPerIteration}{experiments-per-iteration}{0}% |
| 74 | 86 | Number of runs of the target algorithm per iteration. By default (when equal to 0), this value changes for each iteration and depends on the iteration index and the remaining budget. Further details are provided in the \irace paper~\citep{LopDubPerStuBir2016irace}. |
| 75 | 87 | We recommend to use the default value. |
| 76 | 88 | |
| 77 | \defparameter{sampleInstances}{sample-instances}{1}% | |
| 78 | Enable/disable the sampling of the training instances. If the option \parameter{sampleInstances} is disabled, the instances are used in the order provided in the \parameter{trainInstancesFile} or in the order they are read from the \parameter{trainInstancesDir} when\parameter{trainInstancesFile} is not provided. For more information about training instances see \autoref{sec:training}. | |
| 79 | ||
| 80 | 89 | \defparameter{minNbSurvival}{min-survival}{0}% |
| 81 | 90 | Minimum number of configurations needed to continue the execution of each race (iteration). If the number of configurations alive in the race is not larger than this value, the current iteration will stop and a new iteration will start, even if there is budget left to continue the current race. By default (when equal to 0), the value is calculated automatically as $\lfloor 2 + \log_{2}\Nparam \rfloor$, where $\Nparam$ is the number of non-fixed parameters to be tuned. |
| 82 | 91 | |
| 87 | 96 | \defparameter{mu}{mu}{5}% |
| 88 | 97 | Parameter used to define the number of configurations to be sampled and evaluated at each iteration. The number of configurations will be calculated such that there is enough budget in each race to evaluate all configurations on at least $\mu + \min(5,j)$ training instances, where $j$ is the index of the current iteration. The value of $\mu$ will be adjusted to never be lower than the value of \parameter{firstTest}. We recommend to use the default value and, if needed, adjust \parameter{firstTest}and \parameter{eachTest}, instead. |
| 89 | 98 | |
| 90 | \defparameter{softRestart}{soft-restart}{1}% | |
| 91 | Enable/disable the soft-restart strategy that avoids premature convergence of the probabilistic model. When a sampled configuration is \emph{similar} to its parent configuration, the probabilistic model of these configurations is soft restarted. The soft-restart mechanism is explained in the \irace paper~\citep{LopDubPerStuBir2016irace}. The similarity of categorical and ordinal parameters is given by the hamming distance, and the option \parameter{softRestartThreshold} defines the similarity of numerical parameters. | |
| 92 | ||
| 93 | \defparameter{softRestartThreshold}{soft-restart-threshold}{}% | |
| 94 | Soft restart threshold value for numerical parameters. By default, it is computed as $10^{-digits}$, where \parameter{digits} corresponds to the \irace option explained in this section. | |
| 95 | ||
| 96 | 99 | \end{description} |
| 97 | 100 | |
| 98 | 101 | \subsection[Target algorithm parameters]{Target algorithm parameters} |
| 111 | 114 | \subsection[Target algorithm execution]{Target algorithm execution} |
| 112 | 115 | \begin{description} |
| 113 | 116 | \defparameter{targetRunner}{target-runner}{./target-runner}% |
| 114 | This option defines a script or an \aR function that evaluates a configuration of the target algorithm on a particular instance. See \autoref{sec:runner} for details. | |
| 117 | Executable or \aR function that evaluates a configuration of the target algorithm on a particular instance. See \autoref{sec:runner} for details. | |
| 118 | ||
| 119 | \defparameter{targetRunnerLauncher}{target-runner-launcher}{}% | |
| 120 | Executable that will be used to launch the target runner, when \parameter{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). | |
| 121 | ||
| 122 | \defparameter{targetRunnerLauncherArgs}{target-runner-args}{{targetRunner} {targetRunnerArgs}}% | |
| 123 | Command-line arguments provided to \parameter{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \parameter{targetRunner} and by the arguments usually passed when calling \parameter{targetRunner}, respectively. Example: \code{"-m \{targetRunner\} --args \{targetRunnerArgs\}"}. | |
| 115 | 124 | |
| 116 | 125 | \defparameter{targetRunnerRetries}{target-runner-retries}{0}% |
| 117 | Number of times to retry a call to \parameter{targetRunner} if the call failed. | |
| 126 | Number of times to retry a call to \parameter{targetRunner} if the call failed. | |
| 118 | 127 | |
| 119 | 128 | \defparameter{targetRunnerData}{}{}% |
| 120 | 129 | Optional data passed to \parameter{targetRunner}. This is ignored by the default \parameter{targetRunner} function, but it may be used by custom \parameter{targetRunner} functions to pass persistent data around. |
| 141 | 150 | Enable/disable use of \pkg{Rmpi} to execute the \parameter{targetRunner} in parallel using MPI protocol. When \parameter{mpi} is enabled, the option \parameter{parallel} is the number of slave nodes. See \autoref{sec:parallel}. |
| 142 | 151 | |
| 143 | 152 | \defparameter{batchmode}{batchmode}{0}% |
| 144 | Specify how irace waits for jobs to finish when \parameter{targetRunner} submits jobs to a batch cluster: \code{sge}, \code{pbs}, \code{torque} or \code{slurm} (\parameter{targetRunner} must submit jobs to the cluster using. for example, \code{qsub}). See \autoref{sec:parallel}. | |
| 153 | Specify how irace waits for jobs to finish when \parameter{targetRunner} submits jobs to a batch cluster: \code{sge}, \code{pbs}, \code{torque}, \code{slurm} or \code{htcondor} (\parameter{targetRunner} must submit jobs to the cluster using. for example, \code{qsub}). See \autoref{sec:parallel}. | |
| 145 | 154 | |
| 146 | 155 | \end{description} |
| 147 | 156 | |
| 180 | 189 | \defparameter{budgetEstimation}{budget-estimation}{0.02}% |
| 181 | 190 | Fraction (smaller than 1) of the budget used to estimate the mean execution time of a configuration. Only used when \parameter{maxTime} $> 0$. See \autoref{sec:budget}. |
| 182 | 191 | |
| 192 | \defparameter{minMeasurableTime}{min-measurable-time}{0.01}% | |
| 193 | Minimum time unit that is still (significantly) measureable. | |
| 194 | ||
| 183 | 195 | \end{description} |
| 184 | 196 | |
| 185 | 197 | \subsection[Statistical test]{Statistical test} |
| 186 | 198 | \begin{description} |
| 187 | \defparameter{testType}{test-type}{F-test}% | |
| 199 | \defparameter{testType}{test-type}{}% | |
| 188 | 200 | Specifies the statistical test used for elimination: |
| 189 | 201 | \begin{itemize} |
| 190 | 202 | \item[] \code{F-test} (Friedman test) |
| 195 | 207 | We recommend to not use corrections for multiple comparisons because the test typically becomes too strict and the search stagnates. |
| 196 | 208 | See \autoref{sec:stat test} for details about choosing the statistical test most appropriate for your scenario. |
| 197 | 209 | \begin{xwarningbox} |
| 198 | The default setting of \parameter{testType} is \code{F-test} unless the \parameter{capping} option is enabled in which case, the default setting is defined as \code{t-test}. | |
| 210 | The default setting of \parameter{testType} is \code{F-test} unless the \parameter{capping} option is enabled in which case, the default setting is \code{t-test}. | |
| 199 | 211 | \end{xwarningbox} |
| 200 | 212 | |
| 201 | 213 | \defparameter{firstTest}{first-test}{5}% |