diff --git a/.Rinstignore b/.Rinstignore index 163a613..627fd2d 100644 --- a/.Rinstignore +++ b/.Rinstignore @@ -1,2 +1,4 @@ vignettes/section/irace-options.tex ^GenericWrapper4AC +\.github +_snaps diff --git a/BUGS b/BUGS index 2fc083e..bb027a4 100644 --- a/BUGS +++ b/BUGS @@ -1,112 +1,8 @@ ------------------------------------------------------------*- mode: text -*- -#12 ------------------------------------------------------------ +Please report bugs here: https://github.com/MLopez-Ibanez/irace/issues -irace --check or checkIraceScenario do not work with targetRunnerParallel if -the user does not use target.runner at all. - -#11 ------------------------------------------------------------ - -The semantics of recovery are not clear. Should it recover the exact parameters -given to the original run? If so, then moving to a new execdir will mess up -things. Should it allow changing parameters after recovery (that is, what to do -if it finds a new scenario file with different values or it is passed -additional command-line parameters?). - -One possible behavior could be: Recover everything as given originally; if new -settings are given, either by a configuration file or command-line, override -them; for settings that are still set to default values in the original, -recompute the default (because overriden settings may influence the default of -other settings). Report everything that is restored and overriden. - -This means recovery should happen much earlier, before reading -command-line/scenario file. - - -#9 ------------------------------------------------------------ - -By default -# trainInstancesDir = "./Instances/" -# testInstancesDir = "" - -So not specifying any directory, like: - -## File with a list of instances and (optionally) parameters. -## If empty or NULL, do not use a file. -trainInstancesFile = "instances-train.txt" - -## File containing a list of test instances and optionally additional -## parameters for them. If empty or NULL, do not use a file. -testInstancesFile = "instances-test.txt" - -defaults to for train instances to Instances/train-instance-1 ... -but to ./test-instance-1 ... for test instances, which is unexpected. - -# FIXME: Use NA as default value for testInstancesFile/testInstancesDir. -If testInstancesFile != NA and testInstancesDir == NA, -then make testInstancesDir = trainInstancesDir. Maybe give a warning when doing this. - - -#8 ------------------------------------------------------------ - -We do not re-evaluate the elitist (using which.exe), but we do call -target.evaluator on them. See this code in race.R: - - # Execute commands - if (length(which.exe) > 0) { - which.exps <- which(which.alive %in% which.exe) - target.output[which.exps] <- execute.experiments (experiments[which.exps], scenario) - } - irace.assert(!any(sapply(target.output, is.null))) - - # target.evaluator may be NULL. If so, target.output must - # contain the right output already. - if (!is.null(.irace$target.evaluator)) - target.output <- execute.evaluator (experiments, scenario, target.output, - configurations[which.alive, ".ID."]) - -The reason for that is, in some scenarios, we want to normalize the output of -the algorithm, so when we evaluate new configurations, there is a need to -re-normalize the output of the elites as well. So we need to call -target-evaluator again. - -However, when recovering, the files needed by target.evaluator are likely -lost. We would need to call again target-runner. - -This is perhaps a design mistake: We could simply do the normalization internally in irace or create a separate target script (target-re-evaluator?) that only people needing something like re-normalization will use. - -As a short-term fix, we could implement something like: - -If recovering and target-evaluator is in use, keep elites as elites, but re-run target-runner on them. (Perhaps we need a new .irace$ flag saying: first race from recovery). - - -#6 ------------------------------------------------------------ - -Sometimes system2 does not say what exactly failed or what command was -run when something failed, it just prints: "error in running command". -Can we provide more details? - - -#2 ------------------------------------------------------------ - -A parameter like: - -a "" c (0, 5, 10, 20) - -and a condition like a > 10 will return TRUE when a is 5, because a -is actually "5" so the comparison is lexicographic. How to solve this? - - - Fix 1. Check if a can be converted to numeric, then force it before - evaluating conditions. However, we cannot know for sure if the user - actually wants to convert a to numeric or not. Imagine a = "+3", then - a == "+3" would fail! - - - Fix 2. Force users to use quotes when defining categorical and - ordinal parameters. Hopefully this will make obvious that these - values are strings and cannot be compared with other numbers. - -This is already documented in the user-guide. - +This file is only kept for historical purposes. ############################################################### # FIXED bugs diff --git a/DESCRIPTION b/DESCRIPTION index cf0b416..2f1bcbb 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -7,7 +7,7 @@ settings given a set of instances of an optimization problem. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari (2016) . -Version: 3.4.1 +Version: 3.5 Authors@R: c(person("Manuel", "López-Ibáñez", role = c("aut", "cre"), email = "manuel.lopez-ibanez@manchester.ac.uk", comment = c(ORCID = "0000-0001-9974-1295")), @@ -21,19 +21,18 @@ Depends: R (>= 3.2.0) Imports: stats, utils, compiler, R6 Suggests: Rmpi (>= 0.6.0), parallel, knitr, testthat, withr, mlr (>= - 2.15.0), ParamHelpers, devtools + 2.15.0), ParamHelpers, devtools, covr VignetteBuilder: knitr License: GPL (>= 2) -URL: http://iridia.ulb.ac.be/irace, +URL: https://mlopez-ibanez.github.io/irace/, https://github.com/MLopez-Ibanez/irace BugReports: https://github.com/MLopez-Ibanez/irace/issues ByteCompile: yes -LazyData: yes Encoding: UTF-8 -RoxygenNote: 6.1.1 +RoxygenNote: 7.1.1 SystemRequirements: GNU make NeedsCompilation: yes -Packaged: 2020-03-31 08:07:18 UTC; manu +Packaged: 2022-10-23 13:44:18 UTC; manu Author: Manuel López-Ibáñez [aut, cre] (), Jérémie Dubois-Lacoste [aut], @@ -45,4 +44,4 @@ Nguyen Dang [ctb] Maintainer: Manuel López-Ibáñez Repository: CRAN -Date/Publication: 2020-03-31 13:40:03 UTC +Date/Publication: 2022-10-23 14:55:06 UTC diff --git a/MD5 b/MD5 index df0225d..84e48af 100644 --- a/MD5 +++ b/MD5 @@ -1,181 +1,177 @@ -8f9f201288b16a339b6f57273a707d1f *BUGS -90442f63f91cc4c296eef4b18123d897 *DESCRIPTION -929993335cf3a237b7d27e9c1c1a0486 *NAMESPACE -da2fd1953cd33a19dfc3f9d78a435863 *NEWS.md -f728d8206d29e524728b98b5e5c98bc4 *R/ablation.R -47187f0811097d94d2f3abcf83e97dca *R/argparser.R -05a84aa85867928116ca1da97c6bc56f *R/cluster.R -77a6154db7d6d3202a998bffd562cf7b *R/generation.R -1570a3c84a2eabf122b979f12e47a11f *R/irace-options.R -019a192de8c07048f5b05b6294a42fa1 *R/irace-package.R -21751b754f699b991754b2d5310ba37a *R/irace.R -8eed0fa02814e45224e5d3b37e91b038 *R/irace2pyimp.R -f55a403c2ea36bad4fb84f79096acfa1 *R/main.R -477a890824f2689de002705efa5d5e7b *R/model.R -607c6e1c1944510033038d794664c271 *R/parameterAnalysis.R -e403df46a7a45f16e35b104948f42b02 *R/parameterExploration.R -b5f0dad93e467221992a310ab8ab80d3 *R/race-wrapper.R -d585220e5670cc23ce454f67a2353a7a *R/race.R -9745a0ec7a762830726686a84ef8d5f2 *R/readConfiguration.R -fc1fc0c4096fabbfbd1af7f46860ba76 *R/readParameters.R -4c0cd07ce0136e79a6dc457ddb15f57f *R/testing.R -58e5c86ea54797458928b7171e1beaed *R/tnorm.R -2f7cc13b106b43c8c01d1531a12a5c4d *R/utils.R -49ef85c2dd41fbca000ecccd19ea0807 *R/version.R -87efc0b972f0c2596be88c2b8bf5238a *README.md -8bba10219a86dfabf2c028f10e24c361 *build/partial.rdb -6d005f12fcd5b771eb3775e7e6bb9ae3 *build/vignette.rds -5c3dba114664cbebb390e2912437dd22 *cleanup +69a04f8f4c0abbd39178a3d0c7f3ce10 *BUGS +5045e033d6ef0ac2cadce585596446bd *DESCRIPTION +a63f58132973b19bf43020d7a5f15a8d *NAMESPACE +038a593734b0b4fcccde699be7b7ce20 *NEWS.md +b1f319dfbac17e70234c634a2490b632 *R/ablation.R +9c80cd3c08b5cb818c94fd65f81907a2 *R/argparser.R +5e50e779e5017e05cd2fa56b684f6c06 *R/cluster.R +a0a9019a8fff086a6b04aba345855c11 *R/generation.R +92e15c99068d5698c8b5f396abf50212 *R/irace-options.R +76968ff9efe28f18642b4d4a3163ae66 *R/irace-package.R +b56a08ad22985062d39a85653a4f4e4f *R/irace.R +915e1242e234fc7b6ec17a0793781a3c *R/main.R +fef3c5475713f5f09a5580f3225ab771 *R/model.R +d1301df655fad850e615a3f62d536179 *R/parameterAnalysis.R +86ead6121525dc45cf04a3f54ead0c19 *R/parameterExploration.R +068f4659da9911b4e24e68c81ba0614f *R/path_rel2abs.R +213f69a714e687be269920999074c87c *R/race-wrapper.R +816656412e80a1f170fc8938d968c25d *R/race.R +ee9dff2757fd0537a3f43759518bc37b *R/readConfiguration.R +bc45aa88d1d4ff5d05a22778123b18a4 *R/readParameters.R +519ff55d3b1b32a21363ad4e4bbfa133 *R/testing.R +bc04ea89773012f517adc374a6cc6cd9 *R/timer.R +55b4b74d608899de8109a8a136490e96 *R/tnorm.R +4fd568645b95fe328c7d2843442f1c89 *R/utils.R +4904572120e0067c272b3b2335d5a8d1 *R/version.R +2a118b7354fdedff5c74246d8423e5ad *R/zzz.R +559d78ef5ba4df1b8666192ac468df59 *README.md +92d3ace7e0265aaf74b473cadbcf470a *build/partial.rdb +a852616f9564563cd4aadf7640345486 *build/vignette.rds +76b4951f93d078c5a1598e781fd9e670 *cleanup 4fb959d1be3d68345289530627355ca9 *inst/CITATION f694f4bed8633608f1914f539c2dfee4 *inst/bin/parallel-irace -2b6592cf4f79610bdbcc504a4e70bc28 *inst/bin/parallel-irace-mpi -89f5edb040e5d72e0b93ea460b11d42d *inst/bin/parallel-irace-qsub -6033d97deebd3a6c8e4b1bd4b7433b8e *inst/doc/irace-package.R -c6f6dde37bededb32d96fa6889891b2a *inst/doc/irace-package.Rnw -25df86126dd2756b08d7f257caafbc1d *inst/doc/irace-package.pdf +36995f96e183cdd8e7e25207c4d95394 *inst/bin/parallel-irace-mpi +404f816e7b4fdf2a142c65c931f300ce *inst/bin/parallel-irace-qsub +8eda33b154a7c9cff1a90c0e168da70e *inst/bin/parallel-irace-slurm +1263d7dbd890bf549235501fbaf60c05 *inst/doc/irace-package.R +c2e0a6fcffab3c32cdfc05ff2d5aa126 *inst/doc/irace-package.Rnw +afa8bf9cb475e5f5f0cde602756d274b *inst/doc/irace-package.pdf +6f5adad387e00e68a7802e8d543b18ce *inst/examples/README.md 067fe2821c16cf8262dece7a4c36144e *inst/examples/Spear/README 4a9044d0341a321c352ec901cb97a8f0 *inst/examples/Spear/parameters-cat.txt cb2da2a31c96ffc3a4274e9a8ef3832a *inst/examples/Spear/parameters-mixed.txt ed518f04134d2ca0f90b7393467b4e15 *inst/examples/Spear/scenario.txt 039051117ef71079e1d2ff106d6c4e7a *inst/examples/Spear/target-runner -f2c52ac9107d5cfcd53901d5c7ef94e9 *inst/examples/acotsp/README +d24870bcca1038abb7a62d97df77cbd2 *inst/examples/acotsp/README d14e5bb16ddb374e204cabfc0be4545d *inst/examples/acotsp/default.txt 0f16709b42a05455f08906576ba3ee18 *inst/examples/acotsp/forbidden.txt 45771f3ceccfe39d9568abda93d7df24 *inst/examples/acotsp/parameters-acotsp.txt cc17eef505606543c0d5536f2c34b079 *inst/examples/acotsp/scenario.txt -60be927405606a9e2f2668039ec1fde3 *inst/examples/acotsp/target-runner +addd8e7ae0e2ca926476aa9b9e13cb08 *inst/examples/acotsp/target-runner b7d067aba94e5b7fc1e92237a6f0a1b4 *inst/examples/batchmode-cluster/README d960f69eeadff7405e4452f8f50c710a *inst/examples/batchmode-cluster/irace-sge-cluster -1f0d72ccb5c598003e563fee4e06dcb4 *inst/examples/batchmode-cluster/target-evaluator +0b3da55eb761846dcf247a23060ffb0a *inst/examples/batchmode-cluster/target-evaluator +cb7e8b282835fd66d0745aa16dfdecbf *inst/examples/batchmode-cluster/target-runner-htcondor 682e5b6a084e3e94e0ae045b60de8f9f *inst/examples/batchmode-cluster/target-runner-pbs 3cd7828292c3927406d34868ad9266b0 *inst/examples/batchmode-cluster/target-runner-sge 40fbfe617cda99828251f4b2128a4076 *inst/examples/batchmode-cluster/target-runner-slurm 3523d53e5d616ae1343cbb787b703b5c *inst/examples/hypervolume/README 5d16974368bcb97b8dcf9594b19152f2 *inst/examples/hypervolume/target-evaluator 74406c35bc1a89a6e202f9aa27f48817 *inst/examples/hypervolume/target-runner -70ca30740880fa79fc905341f90dfc09 *inst/examples/irace2pyimp/002-TemplateDesign/irace.Rdata -6a0d04e2c0edf0c32c442bb98678a4e4 *inst/examples/irace2pyimp/002-TemplateDesign/run.sh -902005160f28d6d22be7c517b6ffb04d *inst/examples/irace2pyimp/acotsp-forbidden/features.csv -25bdee597b648c0b2bdf94f8e3634d33 *inst/examples/irace2pyimp/acotsp-forbidden/irace.Rdata -397ac9afae6495e8602cc0af0c0037ee *inst/examples/irace2pyimp/acotsp-forbidden/run.sh -902005160f28d6d22be7c517b6ffb04d *inst/examples/irace2pyimp/acotsp/features.csv -e76df5487875f5c1900196d164809e6d *inst/examples/irace2pyimp/acotsp/irace.Rdata -25f67c1549f30889eb348d6a6e6a3ff1 *inst/examples/irace2pyimp/acotsp/run.sh -3e4174afe9c5d85bcecb1ba765e94b40 *inst/examples/matlab/RUN.m -ef092e94145f01eea192cef2f1cdf3d8 *inst/examples/matlab/parameters.txt -6200a5e84d27116627d145d0ecbd0ece *inst/examples/matlab/scenario2.txt -4401349ba53ea145723c8393cb6ababb *inst/examples/matlab/target-runner -a7eb13541113d2c2b51f6c71e57dbe7c *inst/examples/moaco/README +7a908b35eff27e6ed9fe12d5936f8018 *inst/examples/matlab/Main.m +3b0332e02daabf31651a5a0d81ba830a *inst/examples/matlab/instances.txt +570fbc605de77856756be963e4fb8c91 *inst/examples/matlab/parameters.txt +a7c2b75263639028f078e5096a8c2be0 *inst/examples/matlab/scenario.txt +117ed65e989d5b1d901431e35c392b00 *inst/examples/moaco/README 6bd5d5787a551af4e396b6681960681f *inst/examples/moaco/parameters.txt 047f8de89d4747cd3526a4c7e11075c3 *inst/examples/moaco/scenario.txt 5157ef3fc2efb9230f76aef1cfe73291 *inst/examples/moaco/target-evaluator e250b38c64ed8030bc71ee66ec4f8941 *inst/examples/moaco/target-runner de151ff3fce62d302ea77b93c8a5c92b *inst/examples/target-runner-python/target-runner-acotsp.py 5ab7fbf168799bc1c7a54c9b8549a159 *inst/examples/target-runner-python/target-runner-advanced.py +fc5b988b87d11f4e54a72ced66b72fb2 *inst/examples/target-runner-python/target-runner-python-win.bat 2286cd7e0e526fa1526ea38a612d476b *inst/examples/target-runner-python/trivial/instances.txt e3f5b6aee118bd70096440b0d648e812 *inst/examples/target-runner-python/trivial/parameters.txt e35dda63b5e6a98c1c15b392693ac633 *inst/examples/target-runner-python/trivial/scenario.txt 3dcae77aa87c659dff7dd85639ed2e27 *inst/examples/target-runner-python/trivial/target-runner.py -4dd3073b98f2b4723f68ebcb4eeace03 *inst/exdata/irace-acotsp.Rdata -a86b3efa1a616252fbefff57796ea8b5 *inst/exdata/sann.rda +fdf194d8f19dada2867eff7a0695139d *inst/exdata/irace-acotsp.Rdata +15d6164b0d369f64e77ed992d0ec3385 *inst/exdata/log-ablation.Rdata +38aec57159de03f0ff4d710331413041 *inst/exdata/sann.rda +a523dd89b6ec7ba2adc26e2ed3a3a179 *inst/irace.sindef 5718f32d8359ec715f2d8bc1a0ec02b3 *inst/templates/configurations.txt.tmpl 635cce18d757a309cf38afafd9ebda9d *inst/templates/forbidden.txt.tmpl -2733cea5852f923b71bc3086b5d04592 *inst/templates/instances-list.txt.tmpl +af9376428b2c2d911710fd6e51ecf784 *inst/templates/instances-list.txt.tmpl 261e74e936e2217cb612278c5d231887 *inst/templates/parameters.txt.tmpl -9c73861922818cb0a328735cfa033332 *inst/templates/scenario.txt.tmpl +65bde26c9a3d3b154c68a90381fe889e *inst/templates/scenario.txt.tmpl 0185c79ae7b6f5994f89da433acd7b94 *inst/templates/target-evaluator.tmpl -94feb17231e156778d4e28ea3f45f1ca *inst/templates/target-runner.tmpl +ce1e0647e76140eac038f0a1e0f30d2a *inst/templates/target-runner.tmpl 4850d7fbd68fbc7c06af53d3c510734f *inst/templates/windows/target-runner.bat -79e960ad8a32ad97019252d27cef0f85 *man/ablation.Rd -f7c1bfa57a8d880f6c0fe00d2c84c0d4 *man/buildCommandLine.Rd -1fedf949064f213cb77f96163204ae1e *man/checkIraceScenario.Rd -b3b549648e43a6e29ba0a9b9ec9bb80d *man/checkScenario.Rd -852a7b0994e6e5000f5760213f5c3b9e *man/configurations.print.Rd -b0a694214053c371c5a8d0847fbdad53 *man/configurations.print.command.Rd -ffa323a082c23e3383246e6fd1a0bf83 *man/configurationsBoxplot.Rd -625fcb458cc61915be09c476fcfde5ba *man/defaultScenario.Rd -9adffac2d7742ba4fc2747034f2692f2 *man/getConfigurationById.Rd -e5e93fa55f64b5b5b5edf401319939f5 *man/getConfigurationByIteration.Rd -e0253c1596549c13a10a29b5c86f4215 *man/getFinalElites.Rd -449aa819717c8ae99831b0861d36d92f *man/irace-package.Rd -9ed0683b875e228e95ba34ccc4ff8db5 *man/irace.Rd -0b166a31d7d1f224695714e5526d9d9c *man/irace.cmdline.Rd -72e86d95156bc51a47f00b8f9d477265 *man/irace.license.Rd -2ffe037626ebe04abc9cb2038b61d751 *man/irace.main.Rd -2ade5885c5cf1fb824d8df9cdd67160b *man/irace.usage.Rd -a740f833d3612fedf1e33d4bf64ef75a *man/irace.version.Rd -b8dade7a46d233dacfda4398f2f7bb5f *man/irace2pyimp.Rd -1988aeacc8c81ba9734e4cd7715feb66 *man/irace2pyimp_cmdline.Rd -fec6572fb0e35b89e77627333b0a4f71 *man/parallelCoordinatesPlot.Rd -c9f68cd60c69d04e3a29804e32fe5cec *man/parameterFrequency.Rd -44b2288280813116a2c1bb598fbb34d6 *man/plotAblation.Rd -31acaee9cb001ea0df185df8b60a21cb *man/printScenario.Rd -430e7a0ed7ae11d1b8b67376a2642559 *man/psRace.Rd -c3299a7e0324778ba81bad825e192501 *man/readConfigurationsFile.Rd -b5032e1d497cedeac151f24e68039460 *man/readParameters.Rd -1de19eb53f010a4d76089623502ed92d *man/readScenario.Rd -17f8b6b6965e8888dd556d2aabf0f221 *man/read_pcs_file.Rd -f8c4ca3085c228adc5c1387f6661c27c *man/removeConfigurationsMetaData.Rd -ba808bcb9c03f9e8872669bbc7c4452c *man/scenario.update.paths.Rd -05a86b94b4bd798628b916ae1bc60d39 *man/target.evaluator.default.Rd -1946a8c451d554ea18de6acb1e53e406 *man/target.runner.default.Rd -26530cb06e42ce4114cfd085af70b723 *man/testConfigurations.Rd -6900bec9835be9cfcc765d20ad500b27 *man/testing.main.Rd -96d2e468dfc0ea6d9ff783c9245ffc9e *src/Makevars -b225187001f851aea8395d09173a6583 *src/Makevars.win -30d0a08b1017171d15bdc9ebc332d159 *src/install.libs.R +994adc0a852364fdcea6b274bbc4ba47 *man/CommandArgsParser.Rd +6b0dde5d934c2c16a9555cc6305bc8c1 *man/ablation.Rd +078aae4ce581e120cd576971577f5e12 *man/ablation_cmdline.Rd +c1211469bba0c10e00fa80c47a384d7a *man/buildCommandLine.Rd +d412ab580b7ba52bfc02370a93fe99eb *man/checkIraceScenario.Rd +7a1d06d2da2f300a7eed9fe508838824 *man/checkParameters.Rd +04e6a8b6f89522fe45d545285d6c3f4f *man/checkScenario.Rd +7e5aa8f3c75eaedc7d8e19a902864403 *man/configurations.print.Rd +14e033558771c9106b429f53bfb72320 *man/configurations.print.command.Rd +c10db4cc76effd892d2d71f48d67c292 *man/defaultScenario.Rd +cd7f7baaa8734863c8f7a62ae97a045b *man/getConfigurationById.Rd +9a72690609110ba97808c4284a4121ee *man/getConfigurationByIteration.Rd +147fc1fd2f1fe41f7d19efa46d981955 *man/getFinalElites.Rd +ab24f5f4834c020a9e13d5d0b27456d9 *man/irace-package.Rd +9fda11a0c9063c4d668b58fbd06970ed *man/irace.Rd +fc7d521a068ac95854f7aa0cd07fa8eb *man/irace.cmdline.Rd +8fb6cd730c0c3f68f950035aab00f116 *man/irace.license.Rd +bfee6562847f9202097c048aea8fc38e *man/irace.main.Rd +c61fcfc9965c72d1792982c4ac05487b *man/irace.version.Rd +f851af63937121b12ba3818fc89d9cc4 *man/path_rel2abs.Rd +88e40f9047a1e78663eb5bb714d5862c *man/plotAblation.Rd +1e954c71985f87b3d84d20832e64ba8e *man/printParameters.Rd +1fa672e70f48ba69c3c48a9c634c199c *man/printScenario.Rd +0866f93c1c8b41b8ddf20cda0d2b37f8 *man/psRace.Rd +3933746918dd3372aef6825c963c2f6f *man/readConfigurationsFile.Rd +6416a3ab3b2c54c915f5297beba44166 *man/readParameters.Rd +fd4014da95182daa5f4e036d42716aff *man/readScenario.Rd +f38a57a9b8da01e0cb6e227853a7bbd2 *man/read_logfile.Rd +0013e857af07f83b4c3657a1a88525cb *man/read_pcs_file.Rd +fe958fdb0536802ef73a17e55b3c2ddb *man/removeConfigurationsMetaData.Rd +8d2c92c883e214f8ac76f4a4d63a750f *man/scenario_update_paths.Rd +f4388a116d42c37a4bc6d8f64cb281bd *man/target.evaluator.default.Rd +7c150396daf48a28b3dae46ecbeca84a *man/target.runner.default.Rd +8499f76c5044d2aa52df9bcf03cacbb5 *man/testConfigurations.Rd +269490247d3ceb4ea52130c545f5ee00 *man/testing_fromfile.Rd +a5e75a420330b3074709671da1e3d0e7 *man/testing_fromlog.Rd +a886f8cb75354603c3204d2cebc07888 *src/Makevars +7906feceff998d2b2309454c9cbb2f10 *src/Makevars-common +ca5fd252fe6221d98e26e1eab7b8d9c6 *src/Makevars.win +b9f27579825988f77dd3bd9cc80449c1 *src/install.libs.R +4fb82b74a5fcc75be149ef7b2ee9b2a7 *src/iracebin/ablation.c 58ea452025942c848ac07b11f57427ea *src/iracebin/irace.c f6939d43ee41ad7076812ca9404cf4e8 *src/iracebin/irace.h -156cf581cb11fb3f1827f81530593ae0 *src/iracebin/irace2pyimp.c -0b87b4635dc04ba67212afa13fa019b1 *tests/testthat.R +4c3b685dcd16cc3d2091da956863d966 *tests/testthat.R +b735d8cdd247647949863dc0db5242c5 *tests/testthat/bad_scenario.txt +d2a04968b98d1dc030087ed5a3973a24 *tests/testthat/bug-13-lookup.rds 392472d97647beac3650ff564bd64455 *tests/testthat/bug_large_new_instances-confs.txt -3e458de2e4dd97fc94a3c3831d72deca *tests/testthat/bug_large_new_instances.Rdata +c37af8399644c4c02083ce46b2a72631 *tests/testthat/bug_large_new_instances.Rdata 0c70d9f2df1bdaae7a5427383deae61f *tests/testthat/configurations.txt +5392765ea169c97660c1a8f5e38962d5 *tests/testthat/dependencies.txt +72b9700b9daeb38f2b3d56b07e761337 *tests/testthat/dependencies2.txt a96f6d23df9bca94fd33c47b8fe54835 *tests/testthat/dummy_wrapper.py db236f7a1008af30779f40caf07060f7 *tests/testthat/forbidden.txt -5500ec952b6f8dd4b6472c20eff359a0 *tests/testthat/helper-common.R -edde2fb3f78e939587d0648fd74d83e5 *tests/testthat/irace2pyimp/002-TemplateDesign/features.csv -9b48de17b419b215a12be667a0ea0c72 *tests/testthat/irace2pyimp/002-TemplateDesign/instances.txt -70e839a3df66606579446e283067da69 *tests/testthat/irace2pyimp/002-TemplateDesign/params.pcs -365683596b87b0c2b409c548f0a426a0 *tests/testthat/irace2pyimp/002-TemplateDesign/runhistory.json.gz -a7f86b05a53f425857cfc03186aed9c9 *tests/testthat/irace2pyimp/002-TemplateDesign/scenario.txt -9f77d33d5c3f24a30ea673dc093dfd3a *tests/testthat/irace2pyimp/002-TemplateDesign/traj_aclib2.json -cece95303d20fd3bd1580fdd7740e2f6 *tests/testthat/irace2pyimp/acotsp_1/features.csv -4a996468edd9f178280824b6ef814b83 *tests/testthat/irace2pyimp/acotsp_1/instances.txt -94c9b9d0a582ae85c22367dac120d440 *tests/testthat/irace2pyimp/acotsp_1/params.pcs -da20c33a03f92e4dfb2cb55b58253f0d *tests/testthat/irace2pyimp/acotsp_1/runhistory.json.gz -e409ace6c8879c98144e09f6cf1ed82a *tests/testthat/irace2pyimp/acotsp_1/scenario.txt -f3bbc537d647447eefdb23ca1c03a87c *tests/testthat/irace2pyimp/acotsp_1/traj_aclib2.json -cece95303d20fd3bd1580fdd7740e2f6 *tests/testthat/irace2pyimp/acotsp_4/features.csv -4a996468edd9f178280824b6ef814b83 *tests/testthat/irace2pyimp/acotsp_4/instances.txt -94c9b9d0a582ae85c22367dac120d440 *tests/testthat/irace2pyimp/acotsp_4/params.pcs -f8b5cb41296b3abeeb4da342f174f5a1 *tests/testthat/irace2pyimp/acotsp_4/runhistory.json.gz -e409ace6c8879c98144e09f6cf1ed82a *tests/testthat/irace2pyimp/acotsp_4/scenario.txt -283ffe2d398e7e3615983559f29cba69 *tests/testthat/irace2pyimp/acotsp_4/traj_aclib2.json +e53bf0109c13ffc1734dc5d66c77caac *tests/testthat/helper-common.R 5a08d4053b2e1733ac189074f84a62cc *tests/testthat/logparameters.txt 1e4c15b1cb77a4738b5431e83f59cf33 *tests/testthat/parameters.txt 34ecd3091ca49fabe295aaa76720e449 *tests/testthat/saved_maxim_bug.rds bfcde32a37798cf3fa4a0cd3a47933c9 *tests/testthat/saved_maxim_bug2.rds -4fd4da106e4a97ca4ac34a1e94f99394 *tests/testthat/test-GenericWrapper4AC.R -396ac6d04ed9c0f755adbf11e7259544 *tests/testthat/test-bugs.R -e1df5cacd43a5f253eed310cbdb09b8c *tests/testthat/test-capping.R -d603700d9179ad177a49d1020362637b *tests/testthat/test-forbidden.R -59f1c6fa5b4ca51785df8648cfbcbc6a *tests/testthat/test-irace2pyimp.R -d9b37c6098677a850f7d148454056426 *tests/testthat/test-maxTime.R +802b1377a53b451f13bc190dd692eb9e *tests/testthat/scenario.txt +fa5811cf38687ea0ba22857d07e43708 *tests/testthat/setup.R +390e7a8b3437a82b11f614fd1c1976f1 *tests/testthat/teardown.R +f738821c6a6716067263dcb035a8f499 *tests/testthat/test-GenericWrapper4AC.R +c76a2bb7c81016dd0769ea1463db40f9 *tests/testthat/test-bad_scenario.R +9001476440addf62d0bddf2d83e66391 *tests/testthat/test-bug-10.R +e72a8ebe9175644123b6939ac21d9e79 *tests/testthat/test-bug-13.R +53a1fc7b10969c0a91f94585abe9a7ea *tests/testthat/test-bugs.R +968331f372e347ce39e0d3da72f58c70 *tests/testthat/test-capping.R +74f00ef9d7219ebcf467f6e63f1de3d1 *tests/testthat/test-dependencies.R +158ae05d47f940a435ee6770ec56ae86 *tests/testthat/test-forbidden.R +a86211242111165f2e32a8a9105bdc54 *tests/testthat/test-maxTime.R 56754b543d36c2c63459a4befbd50c03 *tests/testthat/test-mlr.R -7f1baf58cc9b20a1a2a4156a3b1b4395 *tests/testthat/test-path.R -5053ba8defc4c86962e3b8b84d4103ff *tests/testthat/test-sann-irace.R +1dbfbc6930ca68aac00a2ecf119b0b0a *tests/testthat/test-path.R +827ba08b388c13ad94d7c47ec643b48b *tests/testthat/test-raceconfs.R +386a21c6965ec4ee93c55c6b2d4dff5d *tests/testthat/test-readconfs.R +c6dbfe891bc46013458c5858ce22e7bf *tests/testthat/test-sann-irace.R 54aba5c7c5e6cacd5d809cef03a4e4ac *tests/testthat/test-similar.R 8ebd654ea8317ed80746c261c4e90b5d *tests/testthat/test-targetRunnerParallel.R 30e04b3da8a10ca41b7457a0f553d059 *tests/testthat/test-targeteval.R -da2fd1953cd33a19dfc3f9d78a435863 *vignettes/NEWS.txt +038a593734b0b4fcccde699be7b7ce20 *vignettes/NEWS.txt 289ab2a15fa9dca53bb4a94dff1817e3 *vignettes/Warning-icon.png -c8940c257353249fb90b2435afe5f1af *vignettes/examples.Rdata -60f556ce3296211bb986740daba10d91 *vignettes/irace-acotsp-stdout.txt -4dd3073b98f2b4723f68ebcb4eeace03 *vignettes/irace-acotsp.Rdata -c6f6dde37bededb32d96fa6889891b2a *vignettes/irace-package.Rnw -02f58d0f325d4de0c5961854092e5cb1 *vignettes/irace-package.bib +29ce28ad39dd3182608f02197ec1ed45 *vignettes/examples.Rdata +4784f9bce400ae4624ca874739cd7c4b *vignettes/fig1u-acotsp-instances.pdf +df916e74a88ae7e49d09735b04c82cd8 *vignettes/irace-acotsp-stdout.txt +fdf194d8f19dada2867eff7a0695139d *vignettes/irace-acotsp.Rdata +c2e0a6fcffab3c32cdfc05ff2d5aa126 *vignettes/irace-package.Rnw +003c544a7fe10a9552d74ac6bdc0a1ac *vignettes/irace-package.bib f44025a8b672c55f7c68873d309336ad *vignettes/irace-scheme.pdf 0b5a399b2478c44749ab63304194f6a7 *vignettes/light-bulb-icon.png -eb7d9a9a7d61a14e2377e09ee0b9450d *vignettes/log-ablation.Rdata -90c0dfc0fa6e1da79e2a602ea0085fc4 *vignettes/section/irace-options.tex +9d6ff066c6a773ca2b8c41999f684e0b *vignettes/section/irace-options.tex diff --git a/NAMESPACE b/NAMESPACE index 80c58c4..ee817fd 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -1,12 +1,15 @@ # Generated by roxygen2: do not edit by hand +export(CommandArgsParser) export(ablation) +export(ablation_cmdline) export(buildCommandLine) export(checkIraceScenario) +export(checkParameters) export(checkScenario) +export(cmdline_usage) export(configurations.print) export(configurations.print.command) -export(configurationsBoxplot) export(defaultScenario) export(getConfigurationById) export(getConfigurationByIteration) @@ -15,41 +18,38 @@ export(irace.cmdline) export(irace.license) export(irace.main) -export(irace.usage) export(irace.version) -export(irace2pyimp) -export(irace2pyimp_cmdline) -export(parallelCoordinatesPlot) -export(parameterFrequency) +export(path_rel2abs) export(plotAblation) +export(printParameters) export(printScenario) export(psRace) export(readConfigurationsFile) export(readParameters) export(readScenario) +export(read_logfile) export(read_pcs_file) export(removeConfigurationsMetaData) export(scenario.update.paths) +export(scenario_update_paths) export(target.evaluator.default) export(target.runner.default) export(testConfigurations) -export(testing.main) +export(testing_fromfile) +export(testing_fromlog) import(compiler) import(stats) import(utils) importFrom(R6,R6Class) -importFrom(grDevices,cairo_pdf) importFrom(grDevices,dev.new) importFrom(grDevices,dev.off) importFrom(grDevices,pdf) -importFrom(grDevices,rgb) importFrom(graphics,abline) importFrom(graphics,axis) importFrom(graphics,barplot) importFrom(graphics,boxplot) importFrom(graphics,bxp) importFrom(graphics,grid) -importFrom(graphics,hist) importFrom(graphics,lines) importFrom(graphics,matplot) importFrom(graphics,mtext) diff --git a/NEWS.md b/NEWS.md index 0b078ce..4f97659 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,160 @@ +# irace 3.5 + +## New features and improvements + + * Handling of dependent parameter domains: These should be specified in the + parameter domain definition and, for now, only numerical parameter can + define dependent domains. A numerical domain can be dependent on one bound, + e.g. `(1, "param1*2")`, where the dependent bound can include basic + arithmetic operators. (Leslie Pérez Cáceres, Manuel López-Ibáñez) + + * The package now provides an `ablation` executable (`ablation.exe` in + Windows) that makes easier to perform ablation analysis without having any R + knowledge. + + * The interface to functions `ablation()` and `plotAblation()` has been + simplified. The `ablation()` function now allows overriding scenario + settings. The `plotAblation()` function will not create the plot if the + ablation log does not contain a complete ablation. + (Manuel López-Ibáñez) + + * The argument `n.instances` of `ablation()` has been renamed to `n_instances` and it is now a factor that multiplies `scenario$firstTest`. + (Manuel López-Ibáñez) + + * New command-line option `--quiet` to run without producing any output + except errors (also available as a scenario option). + (Manuel López-Ibáñez) + + * New command-line option `--init` to initialize a scenario. (Deyao Chen) + + * Added support for HTCondor cluster framework to `--batchmode`. + (Filippo Bistaffa) + + * `--check` now also check the contents of `configurationsFile` and runs + configurations provided via `initConfigurations`. + (Manuel López-Ibáñez, reported by Andreea Avramescu) + + * New scenario options `targetRunnerLauncher` and `targetRunnerLauncherArgs` + to help in cases where the target-runner must be invoked via another + software with particular options (such as `python.exe` in Windows). + (Manuel López-Ibáñez) + + * New scenario option `minMeasurableTime`. + (Manuel López-Ibáñez) + + * An error is produced if a variable set in the scenario file is not known to + irace. If your scenario file contains R code, then use variable names + beginning with a dot `'.'`, which will be ignored by irace. + (Manuel López-Ibáñez) + + * Plotting functions have been moved to the new package + [iraceplot](https://auto-optimization.github.io/iraceplot/). In particular, + `configurationsBoxplot()` is replaced by `iraceplot::boxplot_training()` and + `iraceplot::boxplot_test()`; `parallelCoordinatesPlot()` is replaced by + `iraceplot::parallel_cat()` and `iraceplot::parallel_coord()`; and + `parameterFrequency()` is replaced by `iraceplot::sampling_frequency()`. + (Leslie Pérez Cáceres, Manuel López-Ibáñez) + + * The user-guide now contains a detailed section on "Hyper-parameter + optimization of machine learning methods". + (Manuel López-Ibáñez) + + * When `testType="F-test"` and only two configurations remain, the elimination + test now uses the pseudo-median estimated by the Wilcoxon signed-rank test + to decide which configuration is the best one instead of comparing the + median difference. + (Manuel López-Ibáñez) + + * New functions `testing_fromlog()` and `testing_fromfile()` for independently + executing the testing phase. The function `testing.main()` was removed as it + is superseded by the new ones. + (Manuel López-Ibáñez) + + * New function `read_logfile()` to easily read the log file produced by irace. + (Manuel López-Ibáñez) + + * New function `printParameters()` that prints a parameters R object as a valid input text. + (Manuel López-Ibáñez) + + * `irace2pyimp` moved to its own R package. + (Manuel López-Ibáñez) + + * Generating the file `irace.Rdata` may be disabled by setting `logFile=""`. + (Manuel López-Ibáñez, reported by Johann Dreo) + + * `path_rel2abs()` and `checkParameters()` are now exported so that other + packages may use them. + (Manuel López-Ibáñez) + + * `path_rel2abs()` also searches in system paths. (Manuel López-Ibáñez) + + * `readConfigurationsFile()` will now detect duplicated configurations and + error. (Manuel López-Ibáñez) + + * The interface to functions `getFinalElites()`, `getConfigurationById()` and + `getConfigurationByIteration()` has been simplified. + + * The package provides a `irace.sindef` file that may be used for building a + standalone container of irace using Singularity. See the `README.md` file + for instructions. (Contributed by Johann Dreo) + + * New example `examples/target-runner-python/target-runner-python-win.bat` + contributed by Levi Ribeiro. + + * New helper script in `bin/parallel-irace-slurm` to launch `irace` in [SLURM](https://slurm.schedmd.com/) computer clusters. + (Manuel López-Ibáñez) + + * Rename `scenario.update.paths()` to `scenario_update_paths()`. The old name is deprecated. (Manuel López-Ibáñez) + +## Fixes + + * Correctly handle clear out-performance cases despite strong bi-modality. + (Reported by Nguyen Dang, + fixed by Manuel López-Ibáñez) + + * Fix error when recovering from a parallel run on Windows. + (Manuel López-Ibáñez, reported by Tarek Gamal) + + * `testNbElites` now controls how many iteration elites are tested when + `testIterationElites=1`. This is the documented behavior in the user guide. + (Manuel López-Ibáñez, reported by Marcelo de Souza) + + * Fixes to the Matlab example. (Manuel López-Ibáñez) + + * The default of `testType` is now set to `t-test` when capping is enabled. + (Manuel López-Ibáñez, reported by Jovana Radjenovic) + + * Fix various issues in the user guide. + (Manuel López-Ibáñez, reported by Jovana Radjenovic) + + * Remove duplicated elites. + (Manuel López-Ibáñez, reported by Federico Naldini) + + * Fix (#7): warnings with partial matched parameters. + (Manuel López-Ibáñez, reported by Marc Becker) + + * Fix (#10): wrong assert with `elitist=0`. (Manuel López-Ibáñez) + + * Fix (#12): irace can be run with [FastR](https://www.graalvm.org/22.1/docs/getting-started/#run-r). + + * Fix (#13): Maximum number configurations immediately rejected reached. + (Manuel López-Ibáñez) + + * Fix: when setting the scenario file in the command-line, `scenarioFile` was + not set correctly. The correct scenario was used, however, the debug output + and the value stored in the log / recovery file was wrong. + (Manuel López-Ibáñez, reported by Richard Schoonhoven) + + * With `sampleInstances = FALSE`, elitist irace does not change the order of + instances already seen. However, if you want to make sure that the order of + the instances is enforced, you also need to set `elitistNewInstances=0`. + + * The function `irace.usage()` was removed. It was not really useful for R + users as the same result can be obtained by calling + `irace.cmdline("--help")`. + (Manuel López-Ibáñez) + + # irace 3.4.1 (31/03/2020) * `NEWS` converted to markdown. @@ -21,7 +178,7 @@ versions. (Manuel López-Ibáñez) * Fix invalid assert with ordered parameters: (Leslie Pérez Cáceres) - + ``` value >= 1L && value <= length(possibleValues) is not TRUE ``` @@ -30,10 +187,10 @@ of a script. On Windows, `irace.exe` replaces `irace.bat` (Manuel López-Ibáñez) - * inst/examples/Spear contains the Spear (SAT solver) configuration scenario. + * `inst/examples/Spear` contains the Spear (SAT solver) configuration scenario. (Manuel López-Ibáñez) - * Fixed bug when reporting minimum maxTime required. + * Fixed bug when reporting minimum `maxTime` required. (Reported by Luciana Salete Buriol, fixed by Manuel López-Ibáñez) @@ -42,7 +199,7 @@ ```R all(apply(!is.na(elite.data$experiments), 1, any)) is not TRUE ``` - + (Reported by Maxim Buzdalov, fixed by Manuel López-Ibáñez) @@ -61,7 +218,7 @@ satisfy `digits` (up to `digits=15`). (Manuel López-Ibáñez) - * It is possible to specify boundMax without capping. + * It is possible to specify `boundMax` without capping. (Leslie Pérez Cáceres, Manuel López-Ibáñez) * `irace --check` will exit with code 1 if the check is unsuccessful @@ -69,7 +226,7 @@ * Print where irace is installed with `--help`. (Manuel López-Ibáñez) - * irace will now complain if the output of target-runner or target-evaluator + * irace will now complain if the output of `target-runner` or `target-evaluator` contains extra lines even if the first line of output is correct. This is to avoid parsing the wrong output. Unfortunately, this may break setups that relied on this behavior. The solution is to only print the output that irace @@ -82,8 +239,8 @@ * New option `aclib=` (`--aclib 1`) enables compatibility with the GenericWrapper4AC (https://github.com/automl/GenericWrapper4AC/) used by - AClib (http://aclib.net/). This is EXPERIMENTAL. - `--aclib 1` also sets digits to 15 for compatibility with AClib defaults. + AClib (http://aclib.net/). This is EXPERIMENTAL. `--aclib 1` also sets + digits to 15 for compatibility with AClib defaults. (Manuel López-Ibáñez) * Fix printing of output when capping is enabled. @@ -108,7 +265,7 @@ * Fix bug in `checkTargetFiles()` (`--check`) with capping. (Leslie Pérez Cáceres) - * Clarify a few errors/warnings when `maxTime` > 0. + * Clarify a few errors/warnings when `maxTime > 0`. (Manuel López-Ibáñez, suggested by Haroldo Gambini Santos) @@ -191,20 +348,20 @@ evaluation noticeably faster. (Manuel López-Ibáñez) - * The argument 'experiment' passed to the R function targetRunner does not - contain anymore an element 'extra.params'. Similarly, the 'scenario' - structure does not contain anymore the elements 'instances.extra.params' and - 'testInstances.extra.params'. Any instance-specific parameters values now + * The argument `'experiment'` passed to the R function `targetRunner` does not + contain anymore an element `'extra.params'`. Similarly, the `'scenario'` + structure does not contain anymore the elements `'instances.extra.params'` and + `'testInstances.extra.params'`. Any instance-specific parameters values now form part of the character string that defines an instance and it is up to - the user-defined targetRunner to parse them appropriately. These changes + the user-defined `targetRunner` to parse them appropriately. These changes make no difference when targetRunner is an external script, or when instances and instance-specific parameter values are read from a file. (Manuel López-Ibáñez) # irace 2.3 - * Fix bug that will cause iraceResults$experimentLog to count calls to - targetEvaluator as experiments, even if no call to targetRunner was + * Fix bug that will cause `iraceResults$experimentLog` to count calls to + `targetEvaluator` as experiments, even if no call to `targetRunner` was performed. This does not affect the computation of the budget consumed and, thus, it does not affect the termination criteria of irace. The bug triggers an assertion that terminates irace, thus no run that was successful with @@ -222,8 +379,8 @@ * The option `--sge-cluster` (`sgeCluster`) was removed and replaced by `--batchmode` (`batchmode`). It is now the responsibility of the target-runner - to parse the output of the batch job submission command (e.g., qsub or - squeue), and return just the job ID. Values supported are: "sge", "torque", + to parse the output of the batch job submission command (e.g., `qsub` or + `squeue`), and return just the job ID. Values supported are: "sge", "torque", "pbs" and "slurm". (Manuel López-Ibáñez) * The option `--parallel` can now be combined with `--batchmode` to limit the @@ -240,7 +397,7 @@ ```R eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE)) ``` - + This feature is VERY experimental and the syntax is likely to change in the future. (Manuel López-Ibáñez) @@ -257,20 +414,21 @@ (Manuel López-Ibáñez, Leslie Pérez Cáceres) * Update manual and vignette with details about the expected arguments and - return value of targetRunner and targetEvaluator. (Manuel López-Ibáñez) + return value of `targetRunner` and `targetEvaluator`. (Manuel López-Ibáñez) * Many updates to the User Guide vignette. (Manuel López-Ibáñez) - * Fix \dontrun example in irace-package.Rd (Manuel López-Ibáñez) + * Fix `\dontrun` example in `irace-package.Rd` (Manuel López-Ibáñez) * Fix bug: If testInstances contains duplicates, results of testing are not - correctly saved in iraceResults$testing$experiments nor reported correctly - at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for + correctly saved in `iraceResults$testing$experiments` nor reported correctly + at the end of a run. Now unique IDs of the form `1t, 2t, ...` are used for each testing instance. These IDs are used for the rownames of - iraceResults$testing$experiments and the names of the scenario$testInstances - and iraceResults$testing$seeds vectors. (Manuel López-Ibáñez) - - * Fix bug where irace keeps retrying the target-runner call even if it + `iraceResults$testing$experiments` and the names of the + `scenario$testInstances` + and `iraceResults$testing$seeds` vectors. (Manuel López-Ibáñez) + + * Fix bug where irace keeps retrying the `target-runner` call even if it succeeds. (Manuel López-Ibáñez) * New command-line parameter @@ -281,7 +439,7 @@ instances defined by the scenario. Useful if you decide on the testing instances only after running irace. (Manuel López-Ibáñez) - * Bugfix: When using maxTime != 0, the number of experiments performed may be + * Bugfix: When using `maxTime != 0`, the number of experiments performed may be miscounted in some cases. (Manuel López-Ibáñez) @@ -305,10 +463,10 @@ comparisons), - `t-test-holm` (t-test with Holm's correction for multiple comparisons) - * MPI does not create log files with --debug-level 0. + * MPI does not create log files with `--debug-level 0`. (Manuel López-Ibáñez) - * For simplicity, the parallel-irace-* scripts do not use an auxiliary + * For simplicity, the `parallel-irace-*` scripts do not use an auxiliary `tune-main` script. For customizing them, make a copy and edit them directly. (Manuel López-Ibáñez) @@ -317,7 +475,7 @@ ``` --target-runner-retries : Retry target-runner this many times in case of error. ``` - + * We print diversity measures after evaluating on each instance: (Leslie Pérez Cáceres) @@ -380,10 +538,10 @@ * The best configurations found, either at the end or at each iteration of an irace run, can now be applied to a set of test instances different from the - training instances. See options testInstanceDir, testInstanceFile, - testNbElites, and testIterationElites. (Leslie Pérez Cáceres, Manuel López-Ibáñez) - - * The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed. + training instances. See options `testInstanceDir`, `testInstanceFile`, + `testNbElites`, and `testIterationElites`. (Leslie Pérez Cáceres, Manuel López-Ibáñez) + + * The R interfaces of `hookRun`, `hookEvaluate` and `hookRunParallel` have changed. See `help(hook.run.default)` and `help(hook.evaluate.default)` for examples of the new interfaces. @@ -391,7 +549,7 @@ IDs, and numbers are printed in a more human-readable format. (Leslie Pérez Cáceres, Manuel López-Ibáñez) - * Reduce memory use for very large values of maxExperiments. + * Reduce memory use for very large values of `maxExperiments`. (Manuel López-Ibáñez, thanks to Federico Caselli for identifying the issue) * New option `--load-balancing` (`loadBalancing`) for disabling load-balancing @@ -418,13 +576,12 @@ * New configuration options, mainly for R users: - - hookRunParallel: Optional R function to provide custom - parallelization of hook.run. - - - hookRunData: Optional data passed to hookRun. This is ignored by - the default hookRun function, but it may be used by custom hookRun R - functions to pass persistent data around. - (Manuel López-Ibáñez) + - `hookRunParallel`: Optional R function to provide custom + parallelization of `hook.run`. + + - `hookRunData`: Optional data passed to `hookRun`. This is ignored by the + default `hookRun` function, but it may be used by custom `hookRun` R + functions to pass persistent data around. (Manuel López-Ibáñez) # irace 1.05 @@ -449,7 +606,7 @@ See `--forbidden-file` and `inst/templates/forbidden.tmpl`. (Manuel López-Ibáñez) - * New option `--recovery-file` (recoveryFile) allows resuming a + * New option `--recovery-file` (`recoveryFile`) allows resuming a previous irace run. (Leslie Pérez Cáceres) * The confidence level for the elimination test is now @@ -470,7 +627,7 @@ * Print elapsed time for calls to hook-run if `debugLevel >=1`. (Manuel López-Ibáñez) - * `examples/hook-run-python/hook-run`: A multi-purpose hook-run written + * `examples/hook-run-python/hook-run`: A multi-purpose `hook-run` written in Python. (Franco Mascia) * Parallel mode in an SGE cluster (`--sge-cluster`) is more @@ -500,13 +657,13 @@ * More concise output. - * The parameters expName and expDescription are now useless and they + * The parameters `expName` and `expDescription` are now useless and they were removed. * Faster computation of similar candidates (Jeremie Dubois-Lacoste and Leslie Pérez Cáceres). - * Fix bug when saving instances in tunerResults$experiments. + * Fix bug when saving instances in `tunerResults$experiments`. * `irace.cmdline ("--help")` does not try to quit R anymore. diff --git a/R/ablation.R b/R/ablation.R index 87e810a..dfd0645 100644 --- a/R/ablation.R +++ b/R/ablation.R @@ -1,3 +1,128 @@ +.ablation.params.def <- read.table(header=TRUE, stringsAsFactors = FALSE, text=" +name type short long default description +iraceResults p -l --log-file NA 'Path to the (.Rdata) file created by irace from which the \"iraceResults\" object will be loaded.' +src i -S --src 1 'Source configuration ID.' +target i -T --target NA 'Target configuration ID. By default the best configuration found by irace.' +ab.params s -P --params '' 'Specific parameter names to be used for the ablation (separated with commas). By default use all' +type s -t --type 'full' 'Type of ablation to perform: \"full\" will execute each configuration on all \"--n-instances\" to determine the best-performing one; \"racing\" will apply racing to find the best configurations.' +n_instances i -n --n-instances 1 'Number of instances used in \"full\" ablation will be n_instances * scenario$firstTest.' +seed i '' --seed 1234567 'Integer value to use as seed for the random number generation.' +ablationLogFile p -o --output-file 'log-ablation.Rdata' 'Log file to save the ablation log. If \"\", the results are not saved to a file.' +plot s -p --plot '' 'Output filename (.pdf) for the plot. If not given, no plot is created.' +plot_type s -O --plot-type 'mean' 'Type of plot. Supported values are \"mean\" and \"boxplot\".' +old_path p '' --old-path NA 'Old path found in the log-file (.Rdata) given as input to be replaced by --new-path.' +new_path p '' --new-path NA 'New path to replace the path found in the log-file (.Rdata) given as input.' +execDir p -e --exec-dir NA 'Directory where the target runner will be run.' +scenarioFile p -s --scenario NA 'Scenario file to override the scenario given in the log-file (.Rdata)' +parallel i '' --parallel NA 'Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.' +") + +cat_ablation_license <- function() +{ + ablation_license <- +'#------------------------------------------------------------------------------ +# ablation: An implementation in R of Ablation Analysis +# Version: __VERSION__ +# Copyright (C) 2020--2022 +# Manuel Lopez-Ibanez +# Leslie Perez Caceres +# +# This is free software, and you are welcome to redistribute it under certain +# conditions. See the GNU General Public License for details. There is NO +# WARRANTY; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +#------------------------------------------------------------------------------ +' + cat(sub("__VERSION__", irace.version, ablation_license, fixed=TRUE)) +} + +#' Launch ablation with command-line options. +#' +#' Launch [ablation()] with the same command-line options as the command-line +#' executable (`ablation.exe` in Windows). +#' +#' @param argv (`character()`) \cr The arguments +#' provided on the R command line as a character vector, e.g., +#' `c("-i", "irace.Rdata", "--src", 1)`. +#' +#' @details The function reads the parameters given on the command line +#' used to invoke R, launches [ablation()] and possibly [plotAblation()]. +#' +#' List of command-line options: +#' ```{r echo=FALSE,comment=NA} +#' cmdline_usage(.ablation.params.def) +#' ``` +#' @template ret_ablog +#' @examples +#' # ablation_cmdline("--help") +#' +#' @author Manuel López-Ibáñez +#' @concept running +#' @export +ablation_cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) +{ + op <- options(width = 9999L) # Do not wrap the output. + on.exit(options(op), add = TRUE) + + cat_ablation_license() + cat ("# installed at: ", system.file(package="irace"), "\n", + "# called with: ", paste(argv, collapse = " "), "\n", sep = "") + parser <- CommandArgsParser$new(argv = argv, argsdef = .ablation.params.def) + if (!is.null(parser$readArg (short = "-h", long = "--help"))) { + parser$cmdline_usage() + return(invisible(NULL)) + } + + if (!is.null(parser$readArg (short = "-v", long = "--version"))) { + print(citation(package="irace")) + return(invisible(NULL)) + } + + params <- parser$readAll() + # TODO: Send the other options to the irace command-line parser so the user + # can override options in scenario via the command-line. + if (length(parser$argv) > 0) + stop("Unknown command-line options: ", paste(parser$argv, collapse = " ")) + + if (is.null(params$iraceResults)) { + irace.error("You must provide the path to the '.Rdata' file that contains the 'iraceResults' object generated by irace.") + return(invisible(NULL)) + } + iraceResults <- read_logfile(params$iraceResults) + if (is.null(params$old_path) != is.null(params$new_path)) { + irace.error("To update paths you must provide both --old-path and --new-path.") + return(invisible(NULL)) + } else if (!is.null(params$old_path)) { + iraceResults$scenario <- scenario_update_paths(iraceResults$scenario, params$old_path, params$new_path) + } + if (!is.null(params$scenarioFile)) { + scenario <- readScenario(params$scenarioFile) + } + if (is_null_or_empty_or_na(trim(params$ablationLogFile))) { + params$ablationLogFile <- NULL + } + for (p in c("execDir", "parallel")) { + if (!is.null(params[[p]])) scenario[[p]] <- params[[p]] + } + + if (!is.null(params$ablationLogFile)) + params$ablationLogFile <- path_rel2abs(params$ablationLogFile) + + if (!is.null(params$ab.params)) + params$ab.params <- sapply(strsplit(params$ab.params, ",")[[1]], trimws, USE.NAMES=FALSE) + + ablog <- do.call(ablation, + args = c(list(iraceResults = iraceResults, + src = params$src, target = params$target, + ab.params = params$ab.params, type = params$type, + n_instances = params$n_instances, seed = params$seed, + ablationLogFile = params$ablationLogFile), + scenario)) + if (!is.null(params[["plot"]]) || base::interactive()) { + plotAblation(ablog, pdf.file = params[["plot"]], type = params[["plot_type"]]) + } + invisible(ablog) +} + ## This function fixes dependent parameters when a parameter value has been ## changed. fixDependenciesWithReference <- function(configuration, ref.configuration, parameters) @@ -51,129 +176,134 @@ configurations <- rbind.data.frame(configurations, new.configuration) } rownames(configurations) <- NULL - return (list(configurations=configurations, changed.params=changed.params)) -} - -#' Performs ablation between two configurations. + list(configurations=configurations, changed.params=changed.params) +} + +report_duplicated_results <- function(experiments, configurations) +{ + x <- t(experiments) + x <- x[duplicated(x) | duplicated(x, fromLast = TRUE), , drop=FALSE] + if (nrow(x) == 0L) return(NULL) # No duplicates + dups <- split(rownames(x), apply(x, 1, paste0, collapse="")) + names(dups) <- NULL + for (g in dups) { + cat("Warning: The following configuration produced the same results (the different parameters had not effect):\n") + print(configurations[configurations$.ID. %in% g, , drop=FALSE]) + } + dups +} + +#' Performs ablation between two configurations (from source to target). #' #' @description Ablation is a method for analyzing the differences between two configurations. -#' -#' @param iraceLogFile Log file created by \pkg{irace}, this file must contain -#' the \code{iraceResults} object. -#' @param iraceResults Object created by \pkg{irace} and saved in -#' \code{scenario$logFile}. -#' @param src,target Source and target configuration IDs. If \code{NULL}, then -#' the first configuration ever evaluated is used as source and the best -#' configuration found is used as target. -#' @param ab.params Parameter names to be used for the ablation. They must be -#' in parameters$names. -#' @param n.instances Number of instances to be used for the "full" ablation, -#' if not provided firstTest instances are used. -#' @param type Type of ablation to perform, "full" will execute all instances -#' in the configurations to determine the best performing, "racing" will -#' apply racing to find the best configurations. -#' @param seed Numerical value to use as seed for the random number generation. -#' @param ablationLogFile Log file to save the ablation log. -#' @param pdf.file Prefix that will be used to save the plot file of the -#' ablation results. -#' @param pdf.width Width provided to create the pdf file. -#' @param mar Vector with the margins for the ablation plot. -#' @template arg_debuglevel +#' +#' @template arg_iraceresults +#' @param src,target Source and target configuration IDs. By default, the first configuration ever evaluated (ID 1) is used as `src` and the best configuration found by irace is used as target. +#' @param ab.params Specific parameter names to be used for the ablation. They must be in `parameters$names`. By default, use all parameters. +#' @param type Type of ablation to perform: `"full"` will execute each configuration on all `n_instances` to determine the best-performing one; `"racing"` will apply racing to find the best configurations. +#' @param n_instances (`integer(1)`) Number of instances used in `"full"` ablation will be `n_instances * scenario$firstTest`. +#' @param seed (`integer(1)`) Integer value to use as seed for the random number generation. +#' @param ablationLogFile (`character(1)`) Log file to save the ablation log. If `NULL`, the results are not saved to a file. +#' @param ... Further arguments to override scenario settings, e.g., `debugLevel`, `parallel`, etc. #' #' @references #' C. Fawcett and H. H. Hoos. Analysing differences between algorithm #' configurations through ablation. Journal of Heuristics, 22(4):431–458, 2016. #' -#' @return A list containing the following elements: -#' \describe{ -#' \item{configurations}{Configurations tested in the ablation.} -#' \item{instances}{A matrix with the instances used in the experiments. First column has the -#' instances IDs from \code{iraceResults$scenario$instances}, second column the seed assigned to the instance.} -#' \item{experiments}{A matrix with the results of the experiments (columns are configurations, rows are instances).} -#' \item{scenario}{Scenario object with the settings used for the experiments.} -#' \item{trajectory}{IDs of the best configurations at each step of the ablation.} -#' \item{best}{Best configuration found in the experiments.} -#' } -#' +#' @template ret_ablog +#' @seealso [plotAblation()] #' @examples #' \donttest{ -#' irace.logfile <- file.path(system.file(package="irace"), "exdata", "sann.rda") -#' load(irace.logfile) +#' logfile <- system.file(package="irace", "exdata", "sann.rda") #' # Execute ablation between the first and the best configuration found by irace. -#' ablation(iraceResults = iraceResults, ablationLogFile = NULL) +#' ablog <- ablation(logfile, ablationLogFile = NULL) +#' plotAblation(ablog) #' # Execute ablation between two selected configurations, and selecting only a #' # subset of parameters, directly reading the setup from the irace log file. -#' ablation(iraceLogFile = irace.logfile, src = 1, target = 10, -#' ab.params = c("temp"), ablationLogFile = NULL) +#' ablog <- ablation(logfile, src = 1, target = 10, +#' ab.params = c("temp"), ablationLogFile = NULL) +#' plotAblation(ablog) #' } #' #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez #' @export -#FIXME: Add check for the inputs! -ablation <- function(iraceLogFile = NULL, iraceResults = NULL, - src = NULL, target = NULL, - ab.params = NULL, n.instances = NULL, - type = "full", seed = 1234567, - ablationLogFile = "log-ablation.Rdata", - pdf.file = NULL, pdf.width = 20, mar = c(12,5,4,1), - debugLevel = NULL) -{ +ablation <- function(iraceResults, src = 1L, target = NULL, + ab.params = NULL, type = c("full", "racing"), + n_instances = 1L, seed = 1234567, + ablationLogFile = "log-ablation.Rdata", ...) +{ + # Input check + if (missing(iraceResults) || is.null(iraceResults)) + stop("You must provide an 'iraceResults' object generated by irace or the path to the '.Rdata' file that contains this object.") + + type <- match.arg(type) + if (n_instances != 1L && type == "racing") + stop("'n_instances' has no effect when type == 'racing'") + + if (!is.null(ablationLogFile)) + file.check(ablationLogFile, writeable = TRUE, text = 'logFile') + + save_ablog <- function(complete) { + ablog <- list(changes = changes, + configurations = all_configurations, + experiments = results, + instances = instances, + scenario = scenario, + trajectory = trajectory, + best = best.configuration, + complete = complete) + if (!is.null(ablationLogFile)) save(ablog, file = ablationLogFile, version = 2) + ablog + } + # FIXME: The previous seed needs to be saved and restored at the end. set.seed(seed) - - # Input check - if (is.null(iraceLogFile) && is.null(iraceResults)) - irace.error("You must provide a Rdata file or an iraceResults object.") - if (!(type %in% c("full", "racing"))) - irace.error("Type of ablation", type, "not recognized.") - # Load the data of the log file - if (!is.null(iraceLogFile)) load(iraceLogFile) - - if (is.null(src)) src <- 1 + iraceResults <- read_logfile(iraceResults) if (is.null(target)) target <- iraceResults$iterationElites[length(iraceResults$iterationElites)] - irace.note ("Starting ablation from ", src, " to ", target, " :\n# Seed:", seed, "\n") + irace.note ("Starting ablation from ", src, " to ", target, "\n# Seed: ", seed, "\n") if (src %!in% iraceResults$allConfigurations$.ID.) - irace.error("Source configuration ID (", src, ") cannot be found") + stop("Source configuration ID (", src, ") cannot be found") if (target %!in% iraceResults$allConfigurations$.ID.) - irace.error("Target configuration ID (", target, ") cannot be found") + stop("Target configuration ID (", target, ") cannot be found") src.configuration <- iraceResults$allConfigurations[src, , drop = FALSE] target.configuration <- iraceResults$allConfigurations[target, , drop = FALSE] parameters <- iraceResults$parameters scenario <- iraceResults$scenario - - if (!is.null(ablationLogFile)) scenario$logFile <- ablationLogFile - if (!is.null(debugLevel)) scenario$debugLevel <- debugLevel - + scenario_args <- list(...) + if (length(scenario_args) > 0L) { + unknown_scenario_args <- setdiff(names(scenario_args), names(scenario)) + if (length(unknown_scenario_args) > 0L) + irace.error("Unknown scenario settings given: ", paste0(unknown_scenario_args, collapse=", ")) + scenario <- modifyList(scenario, scenario_args) + } + scenario$logFile <- "" scenario <- checkScenario (scenario) - startParallel(scenario) on.exit(stopParallel(), add = TRUE) - # LESLIE: we should decide how to select the instances to perform the ablation - # for now we generate an instace list with one instance. - ## MANUEL: I think this should be obtained from parameters like - ## instanceFile/instanceDir/instance and re-use the code from irace. And by - ## default, it should use something like firstTest random training instances. - ## LESLIE: Ok by default we use firstTest and I added a parameter to change it in case needed. - if (type == "racing") - n.instances <- length(scenario$instances) - else if (is.null(n.instances)) - n.instances <- scenario$firstTest - instances <- generateInstances(scenario, n.instances) + n_instances <- if (type == "racing") length(scenario$instances) else n_instances * scenario$firstTest + instances <- generateInstances(scenario, n_instances) .irace$instancesList <- instances # Select the parameters used for ablation if (is.null(ab.params)) { - ab.params <- parameters$names + ab.params <- parameters$names } else if (!all(ab.params %in% parameters$names)) { - irace.error("Some of the parameters provided are not defined in the parameters object.") - } + irace.error("Some of the parameters provided (", paste0(setdiff(ab.params, parameters$names), collapse=", "), + ") are not defined in the parameter space.") + } + + cat("# Source configuration (row number is ID):\n") + configurations.print(src.configuration) + cat("# Target configuration (row number is ID):\n") + configurations.print(target.configuration) + # Select parameters that are different in both configurations neq.params <- which(src.configuration[,ab.params] != target.configuration[,ab.params]) @@ -181,12 +311,9 @@ irace.error("Candidates are equal considering the parameters selected\n") param.names <- colnames(src.configuration[,ab.params])[neq.params] - cat("# Configurations (row number is ID):\n") - configurations.print(rbind(src.configuration, target.configuration)) - # FIXME: Do we really need to override the ID? src.configuration$.ID. <- best.id <- 1 - best.configuration <- all.configurations <- src.configuration + best.configuration <- all_configurations <- src.configuration # Execute source and target configurations. ## FIXME: We may already have these experiments in the logFile! @@ -197,7 +324,12 @@ seeds = instances[, "seed"], scenario = scenario, bounds = scenario$boundMax) - irace.note("Executing source and target configurations on the given instances...\n") + # Define variables needed + trajectory <- 1 + names(trajectory) <- "source" + # FIXME: changes should only store the changed parameters. + changes <- list() + irace.note("Executing source and target configurations on the given instances (", nrow(instances), ")...\n") target.output <- execute.experiments(experiments, scenario) if (!is.null(scenario$targetEvaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, @@ -207,15 +339,9 @@ results <- matrix(NA, ncol = 1, nrow = nrow(instances), dimnames = list(seq(1,nrow(instances)), 1)) results[,1] <- output[1:nrow(instances)] - lastres <- output[(nrow(instances)+1):(2*nrow(instances))] - - # Define variables needed - trajectory <- 1 - names(trajectory) <- "source" - changes <- list() - - # Start ablation + lastres <- output[(nrow(instances)+1):(2 * nrow(instances))] step <- 1 + ablog <- save_ablog(complete = FALSE) while (length(param.names) > 1) { # Generate ablation configurations cat("# Generating configurations (row number is ID):", param.names,"\n") @@ -230,11 +356,10 @@ # New configurations ids ## FIXME: These should be generated with respect to the logFile to make ## sure we don't have duplicate IDs. - new.ids <- seq(max(all.configurations$.ID.) + 1, - max(all.configurations$.ID.) + nrow(aconfigurations)) - aconfigurations[,".ID."] <- new.ids + aconfigurations[,".ID."] <- seq(max(all_configurations$.ID.) + 1, + max(all_configurations$.ID.) + nrow(aconfigurations)) configurations.print(aconfigurations, metadata = FALSE) - all.configurations <- rbind(all.configurations, aconfigurations) + all_configurations <- rbind(all_configurations, aconfigurations) # Set variables for the racing procedure if (scenario$capping) { @@ -250,13 +375,11 @@ scenario$elitist <- FALSE .irace$next.instance <- 1 } - - # All instances for the firstTest for the full - if (type == "full") scenario$firstTest <- nrow(instances) - + irace.note("Ablation (", type, ") of ", nrow(aconfigurations), " configurations on ", nrow(instances), " instances.\n") - # MANUEL: Is this racing or just running the configurations? + # Force the race to see all instances in "full" mode + if (type == "full") scenario$firstTest <- nrow(instances) race.output <- race(maxExp = nrow(aconfigurations) * nrow(instances), minSurvival = 1, elite.data = elite.data, @@ -266,35 +389,28 @@ elitistNewInstances = 0) results <- merge.matrix (results, race.output$experiments) - # Save temp log - ab.log <- list(configurations = all.configurations, - instances = instances, - experiments = results, - scenario = scenario, - trajectory = trajectory, - changes = changes) - if (!is.null(ablationLogFile)) - save(ab.log, file = ablationLogFile, version = 2) + # Save log + ablog <- save_ablog(complete = FALSE) # Get the best configuration based on the criterion of irace # MANUEL: Doesn't race.output already give you all this info??? cranks <- overall.ranks(results[,aconfigurations$.ID.,drop=FALSE], scenario$testType) - best.ids <- which.min(cranks) - cand.mean <- colMeans(results[,aconfigurations$.ID.,drop=FALSE], na.rm=TRUE) + best_id <- which.min(cranks)[1] + # cand.mean <- colMeans(results[,aconfigurations$.ID.,drop=FALSE], na.rm=TRUE) changes[[step]] <- ab.aux$changed.params - best.change <- changes[[step]][[best.ids[1]]] - trajectory <- c(trajectory, aconfigurations[best.ids[1], ".ID."]) + best.change <- changes[[step]][[best_id]] + trajectory <- c(trajectory, aconfigurations[best_id, ".ID."]) # Report best # FIXME: This ID does not actually match the configuration ID # The race already reports the best. - #cat("# Best configuration ID:", best.ids[1], "mean:", cand.mean[best.ids[1]],"\n") - for(i in seq_along(best.change)) { - cat("# ", best.change[i], best.configuration[,best.change[i]], "->", - aconfigurations[best.ids[1], best.change[i]], "\n") + cat("# Best changed parameters:\n") + for (i in seq_along(best.change)) { + cat("#", best.change[i], ":", best.configuration[,best.change[i]], "->", + aconfigurations[best_id, best.change[i]], "\n") } - best.configuration <- aconfigurations[best.ids[1],,drop=FALSE] + best.configuration <- aconfigurations[best_id,,drop=FALSE] best.id <- best.configuration$.ID. param.names <- param.names[!(param.names %in% best.change)] step <- step + 1 @@ -302,8 +418,8 @@ # Add last configuration and its results # FIXME: This may be overriding the ID of an existing configuration!!! - target.configuration$.ID. <- max(all.configurations$.ID.) + 1 - all.configurations <- rbind(all.configurations, target.configuration) + target.configuration$.ID. <- max(all_configurations$.ID.) + 1 + all_configurations <- rbind(all_configurations, target.configuration) results <- cbind(results, matrix(lastres, ncol = 1, dimnames=list(seq(1, nrow(instances)), target.configuration$.ID.))) @@ -311,29 +427,23 @@ # Get the overall best cranks <- overall.ranks(results[,trajectory, drop=FALSE], scenario$testType) - best.ids <- which.min(cranks) - best.configuration <- all.configurations[trajectory[best.ids[1]],,drop=FALSE] + best_id <- which.min(cranks)[1] + ## FIXME: At this point, the rownames of all_configurations does not match + ## all_configurations$.ID. That is confusing and a potential source of + ## bugs. Instead of fixing it here, we should not generate the discrepancy + ## ever. + best.configuration <- all_configurations[trajectory[best_id],,drop=FALSE] irace.note("Final best configuration:\n") configurations.print(best.configuration) - + + # Check for duplicated results: + report_duplicated_results(results, all_configurations) + # LESLIE: If we use racing we can have a matrix of results that is not # complete, how should we do the plots? # MANUEL: Do not plot anything that was discarded - # Save final log. - ab.log <- list(configurations = all.configurations, - instances = instances, - experiments = results, - scenario = scenario, - trajectory = trajectory, - best = best.configuration) - - if (!is.null(ablationLogFile)) - save(ab.log, file = ablationLogFile, version = 2) - - plotAblation(ab.log = ab.log, pdf.file = pdf.file, - pdf.width = pdf.width, mar = mar, main = "Ablation") - return(ab.log) + save_ablog(complete = TRUE) } ablation.labels <- function(trajectory, configurations) @@ -355,37 +465,39 @@ #' Create plot from an ablation log #' -#' @param ab.log Ablation log returned by \code{\link{ablation}}. -#' @param abLogFile Rdata file containing the ablation log. +#' @param ablog (`list()`|`character(1)`) Ablation log object returned by [ablation()]. Alternatively, the path to an `.Rdata` file, e.g., `"log-ablation.Rdata"`, from which the object will be loaded. + #' @param pdf.file Output filename. #' @param pdf.width Width provided to create the pdf file. -#' @param type Type of plots. Supported values are \code{"mean"} and -#' \code{"boxplot"}. +#' @param type Type of plot. Supported values are `"mean"` and `"boxplot"`. #' @param mar Vector with the margins for the ablation plot. #' @param ylab Label of y-axis. +#' @param ylim Numeric vector of length 2, giving the y coordinates ranges. #' @param ... Further graphical parameters may also be supplied as -#' arguments. See \code{plot.default}. +#' arguments. See [graphics::plot.default()]. #' #' @author Leslie Pérez Cáceres and Manuel López-Ibáñez -#' @seealso \code{\link{ablation}} +#' @seealso [ablation()] +#' @examples +#' logfile <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") +#' plotAblation(ablog = logfile) #' @export -plotAblation <- function (ab.log = NULL, abLogFile = NULL, - pdf.file = NULL, pdf.width = 20, +plotAblation <- function (ablog, pdf.file = NULL, pdf.width = 20, type = c("mean", "boxplot"), mar = par("mar"), - ylab = "Mean configuration cost", ...) + ylab = "Mean configuration cost", ylim = NULL, + ...) { type <- match.arg(type) - if (is.null(ab.log)) { - if (is.null(abLogFile)) - irace.error("You must provide a log file or an ablation log object") - else { - load(abLogFile) - if (is.null(ab.log)) - irace.error("abLogFile '", abLogFile, "' does not contain ab.log") - } - } - + if (missing(ablog) || is.null(ablog)) { + irace.error("You must provide an 'ablog' object generated by ablation() or the path to the '.Rdata' file that contains this object.") + } + + ablog <- read_logfile(ablog, name = "ablog") + if (!ablog$complete) + stop("The ablog shows that the ablation procedure did not complete cleanly and only contains partial information") + + if (!is.null(pdf.file)) { if (!is.file.extension(pdf.file, ".pdf")) pdf.file <- paste0(pdf.file, ".pdf") @@ -395,27 +507,29 @@ on.exit(dev.off(), add = TRUE) } - trajectory <- ab.log$trajectory - configurations <- ab.log$configurations + trajectory <- ablog$trajectory + configurations <- ablog$configurations # Generate labels # FIXME: allow overriding these labels. labels <- ablation.labels(trajectory, configurations) inches_to_lines <- (par("mar") / par("mai"))[1] lab.width <- max(strwidth(labels, units = "inches")) * inches_to_lines - old.par <- par(mar = mar + c(lab.width - 2, 0, 0, 0), cex.axis = 1) - on.exit(par(old.par), add = TRUE) - - experiments <- ab.log$experiments - - # FIXME: We should also show the other alternatives at each step not just the + old.par <- par(mar = c(lab.width + 2.1, 4.1, 0.1, 0.1), cex.axis = 1) + if (!is.null(pdf.file)) + on.exit(par(old.par), add = TRUE) + + experiments <- ablog$experiments + + # FIXME: We could also show the other alternatives at each step not just the # one selected. See Leonardo's thesis. - ylim <- NULL if (type == "boxplot") { bx <- boxplot(experiments[, trajectory], plot=FALSE) - ylim <- range(ylim, bx$stats[is.finite(bx$stats)], - bx$out[is.finite(bx$out)], - bx$conf[is.finite(bx$conf)]) + if (is.null(ylim)) { + ylim <- range(bx$stats[is.finite(bx$stats)], + bx$out[is.finite(bx$out)], + bx$conf[is.finite(bx$conf)]) + } } costs.avg <- colMeans(experiments[, trajectory]) diff --git a/R/argparser.R b/R/argparser.R index b312b5f..ba29d22 100644 --- a/R/argparser.R +++ b/R/argparser.R @@ -1,3 +1,6 @@ +#' R6 Class for parsing command-line arguments +#' +#' @export CommandArgsParser <- R6::R6Class("CommandArgsParser", cloneable = FALSE, list( argv = NULL, argsdef = NULL, @@ -8,16 +11,21 @@ argv <- strsplit(trim(argv), " +")[[1]] } self$argv <- argv + required_colnames <- c("name", "short", "long", "type", "default") + if (any(required_colnames %!in% colnames(argsdef))) { + stop("argsdef must contain the column names: ", paste0(required_colnames, collapse=", ")) + } self$argsdef <- argsdef + rownames(self$argsdef) <- argsdef$name + self }, readCmdLineParameter = function (paramName, default = NULL) { - value <- self$readArg (short = self$argsdef[paramName, "short"], - long = self$argsdef[paramName,"long"]) + value <- self$readArg(short = self$argsdef[paramName, "short"], + long = self$argsdef[paramName,"long"]) if (is.null(value)) { value <- if (is.null(default)) self$argsdef[paramName, "default"] else default - } else if (is.na(value) && self$argsdef[paramName,"type"] != 'x' ) { - irace.error ("option '", self$argsdef[paramName,"long"], - "' requires an argument\n") + } else if (is.na(value) && self$argsdef[paramName,"type"] != 'x') { + stop("option '", self$argsdef[paramName,"long"],"' requires an argument", call. = FALSE) } return(value) }, @@ -27,31 +35,69 @@ readArg = function(short = "", long = "") { argv <- self$argv pos <- c() - if (length (short) > 0) { - pos <- grep (paste0("^", short, "$"), argv) - if (length (pos) == 0) { - pos <- grep (paste0("^", short, "="), argv) + if (length(short) > 0) { + # FIXME: use match() + pos <- grep(paste0("^", short, "$"), argv) + if (length(pos) == 0) { + # FIXME: use pmatch() + pos <- grep(paste0("^", short, "="), argv) } } - if (length (long) > 0 && length (pos) == 0) { - pos <- grep (paste0("^", long, "$"), argv) - if (length (pos) == 0) { - pos <- grep (paste0("^", long, "="), argv) + if (length(long) > 0 && length(pos) == 0) { + pos <- grep(paste0("^", long, "$"), argv) + if (length(pos) == 0) { + pos <- grep(paste0("^", long, "="), argv) } } - if (length (pos) == 0) { - return (NULL) - } else if (length(pos) > 0) { + if (length(pos) == 0) { + return(NULL) + } else if(length(pos) > 0) { # Allow repeated parameters pos <- max(pos) } - value <- unlist(strsplit (argv[pos], '=', fixed = TRUE))[2] + value <- unlist(strsplit(argv[pos], '=', fixed = TRUE))[2] if (is.null (value) || is.na(value)) { value <- argv[pos + 1] self$argv <- argv[-(pos + 1)] } self$argv <- self$argv[-pos] return (value) + }, + readAll = function() { + params <- list() + for (param in self$argsdef$name[self$argsdef$type != 'x']) { + value <- self$readCmdLineParameter(paramName = param) + if (is.na(value) || (length(value) > 0 && value == "")) value <- NULL + params[[param]] <- value + } + params + }, + cmdline_usage = function(){ + irace::cmdline_usage(self$argsdef) }) - ) +) + +#' `cmdline_usage()` prints the output of `--help` +#' +#' @param cmdline_args Definition of the command-line arguments. +#' +#' @rdname CommandArgsParser +#' @export +cmdline_usage <- function(cmdline_args) +{ + for (i in seq_len(nrow(cmdline_args))) { + short <- cmdline_args[i,"short"] + long <- cmdline_args[i,"long"] + desc <- cmdline_args[i,"description"] + if (desc == "" || (short == "" && long == "")) next + if (short != "") short <- paste0(short,",") + default <- cmdline_args[i,"default"] + if (!is_null_or_empty_or_na(default)) { + desc <- paste0(desc, " Default: ", default, ".") + } + cat(sep = "\n", strwrap(desc, width = 80, + initial = sprintf("%3s%-20s ", short, long), + exdent = 25)) + } +} diff --git a/R/cluster.R b/R/cluster.R index cced101..ffed41c 100644 --- a/R/cluster.R +++ b/R/cluster.R @@ -1,17 +1,11 @@ ### Submit/wait for jobs in batch clusters. sge.job.finished <- function(jobid) -{ - return(system (paste0("qstat -j ", jobid), - ignore.stdout = TRUE, ignore.stderr = TRUE, - intern = FALSE, wait = TRUE)) -} + system (paste0("qstat -j ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, + intern = FALSE, wait = TRUE) pbs.job.finished <- function(jobid) -{ - return(system (paste0("qstat ", jobid), - ignore.stdout = TRUE, ignore.stderr = TRUE, - intern = FALSE, wait = TRUE)) -} + system (paste0("qstat ", jobid), ignore.stdout = TRUE, ignore.stderr = TRUE, + intern = FALSE, wait = TRUE) torque.job.finished <- function(jobid) { @@ -32,7 +26,7 @@ # output. If the 5th token in the last line is a 'C', then the job has # terminated and its output files can be processed. Otherwise the job is not # completed (queued, running, exiting...) - return(any(grepl(paste0(jobid, ".*\\sC\\s"), output))) + any(grepl(paste0(jobid, ".*\\sC\\s"), output)) } slurm.job.finished <- function(jobid) @@ -44,7 +38,14 @@ if (!is.null(attr(output, "status"))) return(TRUE) # If may return zero, but the job is not in the system anymore because it # completed. This is different from the Torque case. - return (!any(grepl(paste0("\\s", jobid, "\\s"), output))) + !any(grepl(paste0("\\s", jobid, "\\s"), output)) +} + +htcondor.job.finished <- function(jobid) +{ + output <- suppressWarnings(system2("condor_q", jobid, stdout = TRUE, stderr = TRUE)) + # Check if job is still in the queue, otherwise it is considered finished + !any(grepl(paste0("ID:\\s", jobid), output)) } ## Launch a job with qsub and return its jobID. This function does not @@ -52,29 +53,18 @@ ## invokes qsub and returns a jobID. target.runner.qsub <- function(experiment, scenario) { - debugLevel <- scenario$debugLevel - configuration.id <- experiment$id.configuration - instance.id <- experiment$id.instance - seed <- experiment$seed - configuration <- experiment$configuration - instance <- experiment$instance - switches <- experiment$switches - - targetRunner <- scenario$targetRunner - if (as.logical(file.access(targetRunner, mode = 1))) { - irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") - } - - args <- paste(configuration.id, instance.id, seed, instance, - buildCommandLine(configuration, switches)) - output <- runcommand(targetRunner, args, configuration.id, debugLevel) + debugLevel <- scenario$debugLevel + res <- run_target_runner(experiment, scenario) + cmd <- res$cmd + output <- res$output + args <- res$args jobID <- NULL outputRaw <- output$output err.msg <- output$error if (is.null(err.msg)) { # We cannot use parse.output because that tries to convert to numeric. - if (scenario$debugLevel >= 2) { cat (outputRaw, sep = "\n") } + if (debugLevel >= 2) { cat (outputRaw, sep = "\n") } # Initialize output as raw. If it is empty stays like this. # strsplit crashes if outputRaw == character(0) if (length(outputRaw) > 0) { @@ -85,8 +75,8 @@ jobID <- NULL } } - return(list(jobID = jobID, error = err.msg, outputRaw = outputRaw, - call = paste(targetRunner, args))) + list(jobID = jobID, error = err.msg, outputRaw = outputRaw, + call = paste(cmd, args)) } cluster.lapply <- function(X, scenario, poll.time = 2) @@ -99,6 +89,7 @@ pbs = pbs.job.finished, torque = torque.job.finished, slurm = slurm.job.finished, + htcondor = htcondor.job.finished, irace.error ("Invalid value of scenario$batchmode = ", scenario$batchmode)) # Parallel controls how many jobs we send at once. Some clusters have low @@ -131,5 +122,5 @@ } } } - return(output) + output } diff --git a/R/generation.R b/R/generation.R index 9ef2670..5091545 100644 --- a/R/generation.R +++ b/R/generation.R @@ -18,7 +18,7 @@ return(v) } -new.empty.configuration <- function(parameters) +new_empty_configuration <- function(parameters) { newConfigurationsColnames <- c(names(parameters$conditions), ".PARENT.") return(setNames(as.list(rep(NA, length(newConfigurationsColnames))), @@ -39,22 +39,52 @@ } } +## Calculates the parameter bounds when parameters domain is dependent +getDependentBound <- function(parameters, param, configuration) +{ + values <- parameters$domain[[param]] + if (is.expression(values)) { + # Depends contains parameters that enable param and parameters that define + # its domain. If this is a partial configuration, we need only the latter. + # Use names() here in case the configuration is simply a list. + deps <- intersect(names(configuration), parameters$depends[[param]]) + # If it depends on a parameter that is disabled, then this is disabled. + if (anyNA(configuration[deps])) return(NA) + + values <- sapply(values, eval, configuration) + irace.assert(all(is.finite(values))) + # Value gets truncated (defined from robotics initial requirements) + if (parameters$types[param] == "i") values <- as.integer(values) + if (values[1] > values[2]) { + irace.error ("Invalid domain (", paste0(values, collapse=", "), + ") generated for parameter '", param, + "' that depends on parameters (", + paste0(parameters$depends[[param]], collapse=", "), + "). This is NOT a bug in irace. Check the definition of these parameters.") + } + } + + return(values) +} + ### Uniform sampling for the initial generation sampleUniform <- function (parameters, nbConfigurations, digits, forbidden = NULL, repair = NULL) { + if (is.null(repair)) repair <- function(c, p, d) c + namesParameters <- names(parameters$conditions) newConfigurations <- as.data.frame(matrix(nrow = nbConfigurations, ncol = length(namesParameters) + 1, dimnames = list(NULL, c(namesParameters, ".PARENT.")) - )) - empty.configuration <- new.empty.configuration(parameters) + ), stringsAsFactors=FALSE) + empty_configuration <- new_empty_configuration(parameters) for (idxConfiguration in seq_len(nbConfigurations)) { forbidden.retries <- 0 while (forbidden.retries < 100) { - configuration <- empty.configuration + configuration <- empty_configuration for (p in seq_along(namesParameters)) { currentParameter <- namesParameters[p] if (!conditionsSatisfied(parameters, configuration, currentParameter)) { @@ -69,8 +99,11 @@ # We don't even need to sample, there is only one possible value ! newVal <- get.fixed.value (currentParameter, parameters) # The parameter is not a fixed and should be sampled - } else if (currentType %in% c("i", "r")) { - newVal <- sample.unif(currentParameter, parameters, currentType, digits) + } else if (currentType %in% c("i","r")) { + domain <- getDependentBound(parameters, currentParameter, configuration) + newVal <- sample_unif(currentType, domain, + transf = parameters$transform[[currentParameter]], + digits) } else { irace.assert(currentType %in% c("c","o")) possibleValues <- parameters$domain[[currentParameter]] @@ -79,9 +112,7 @@ configuration[[p]] <- newVal } configuration <- as.data.frame(configuration, stringsAsFactors=FALSE) - if (!is.null(repair)) { - configuration <- repair(configuration, parameters, digits) - } + configuration <- repair(configuration, parameters, digits) if (is.null(forbidden) || nrow(checkForbidden(configuration, forbidden)) == 1) { @@ -104,6 +135,8 @@ nbNewConfigurations, digits, forbidden = NULL, repair = NULL) { + if (is.null(repair)) repair <- function(c, p, d) c + if (nbNewConfigurations <= 0) { irace.error ("The number of configurations to generate appears to be negative or zero.") } @@ -112,8 +145,8 @@ as.data.frame(matrix(nrow = nbNewConfigurations, ncol = length(namesParameters) + 1, dimnames = list(NULL, c(namesParameters, ".PARENT.")) - )) - empty.configuration <- new.empty.configuration(parameters) + ), stringsAsFactors=FALSE) + empty_configuration <- new_empty_configuration(parameters) for (idxConfiguration in seq_len(nbNewConfigurations)) { forbidden.retries <- 0 @@ -123,7 +156,7 @@ prob = eliteConfigurations[[".WEIGHT."]]) eliteParent <- eliteConfigurations[indexEliteParent, ] idEliteParent <- eliteParent[[".ID."]] - configuration <- empty.configuration + configuration <- empty_configuration configuration[[".PARENT."]] <- idEliteParent # Sample a value for every parameter of the new configuration. @@ -135,7 +168,7 @@ currentType <- parameters$types[[currentParameter]] if (!conditionsSatisfied(parameters, configuration, currentParameter)) { # Some conditions are unsatisfied. - # Should be useless, NA is ?always? assigned when matrix created + # Should be useless, NA is (always?) assigned when matrix created newVal <- NA } else if (isFixed(currentParameter, parameters)) { @@ -143,17 +176,27 @@ newVal <- get.fixed.value (currentParameter, parameters) # The parameter is not a fixed and should be sampled } else if (currentType %in% c("i", "r")) { + domain <- getDependentBound(parameters, currentParameter, configuration) mean <- as.numeric(eliteParent[currentParameter]) - # If there is not value we obtain it from the model + # If there is not value we obtain it from the model or the mean obtained is + # not in the current domain, this can happen when using dependent domains if (is.na(mean)) mean <- model[[currentParameter]][[as.character(idEliteParent)]][2] - if (is.na(mean)) { + if (is.na(mean) || !inNumericDomain(mean, domain)) { # The elite parent does not have any value for this parameter, # let's sample uniformly. - newVal <- sample.unif(currentParameter, parameters, currentType, digits) - + newVal <- sample_unif(currentType, domain, + transf = parameters$transform[[currentParameter]], digits) } else { stdDev <- model[[currentParameter]][[as.character(idEliteParent)]][1] - newVal <- sample.norm(mean, stdDev, currentParameter, parameters, currentType, digits) + # If parameters are dependent standard deviation must be computed + # based on the current domain + if (parameters$isDependent[currentParameter]) { + # Conditions should be satisfied for the parameter, thus domain cannot be NA + stdDev <- (domain[2] - domain[1]) * stdDev + } + newVal <- sample_norm(mean, stdDev, currentType, domain, + transf = parameters$transform[[currentParameter]], + digits) } } else if (currentType == "o") { possibleValues <- paramDomain(currentParameter, parameters) @@ -171,7 +214,7 @@ stdDev <- model[[currentParameter]][[as.character(idEliteParent)]] # Sample with truncated normal distribution as an integer. - # See sample.norm() for an explanation. + # See sample_norm() for an explanation. newValAsInt <- floor(rtnorm(1, mean + 0.5, stdDev, lower = 1, upper = length(possibleValues) + 1L)) @@ -197,9 +240,8 @@ } configuration <- as.data.frame(configuration, stringsAsFactors = FALSE) - if (!is.null(repair)) { - configuration <- repair(configuration, parameters, digits) - } + configuration <- repair(configuration, parameters, digits) + if (is.null(forbidden) || nrow(checkForbidden(configuration, forbidden)) == 1) { newConfigurations[idxConfiguration,] <- configuration @@ -271,12 +313,11 @@ # except for the case of log-transformed negative domains, where we have to # translate by -0.5. # -numeric.value.round <- function(type, value, lowerBound, upperBound, digits) +numeric_value_round <- function(type, value, lowerBound, upperBound, digits) { irace.assert(is.finite(value)) if (type == "i") { value <- floor(value) - upperBound <- upperBound - 1L # undo the above for the assert # The probability of this happening is very small, but it could happen. if (value == upperBound + 1L) value <- upperBound @@ -288,11 +329,16 @@ } # Sample value for a numerical parameter. -sample.unif <- function(param, parameters, type, digits = NULL) -{ - lowerBound <- paramLowerBound(param, parameters) - upperBound <- paramUpperBound(param, parameters) - transf <- parameters$transform[[param]] +sample_unif <- function(type, domain, transf, digits) +{ + # Dependent domains could be not available because of inactivity of parameters + # on which they are depedent. In this case, the dependent parameter becomes + # not active and we return NA. + if (anyNA(domain)) return(NA) + + lowerBound <- domain[1] + upperBound <- domain[2] + if (type == "i") { # +1 for correct rounding before floor() upperBound <- 1L + upperBound @@ -303,15 +349,21 @@ } else { value <- runif(1, min = lowerBound, max = upperBound) } - value <- numeric.value.round(type, value, lowerBound, upperBound, digits) + # We use original upperBound, not the +1L for 'i'. + value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) return(value) } -sample.norm <- function(mean, sd, param, parameters, type, digits = NULL) -{ - lowerBound <- paramLowerBound(param, parameters) - upperBound <- paramUpperBound(param, parameters) - transf <- parameters$transform[[param]] +sample_norm <- function(mean, sd, type, domain, transf, digits) +{ + # Dependent domains could be not available because of inactivity of parameters + # on which they are depedent. In this case, the dependent parameter becomes + # not active and we return NA. + if (anyNA(domain)) return(NA) + + lowerBound <- domain[1] + upperBound <- domain[2] + if (type == "i") { upperBound <- 1L + upperBound # Because negative domains are log-transformed to positive domains. @@ -325,7 +377,7 @@ } else { value <- rtnorm(1, mean, sd, lowerBound, upperBound) } - - value <- numeric.value.round(type, value, lowerBound, upperBound, digits) + # We use original upperBound, not the +1L for 'i'. + value <- numeric_value_round(type, value, lowerBound, upperBound = domain[2], digits) return(value) } diff --git a/R/irace-options.R b/R/irace-options.R index 6134a88..3a7b942 100644 --- a/R/irace-options.R +++ b/R/irace-options.R @@ -3,91 +3,99 @@ # Variables that do not have a command-line option have description == "" # Types are b(oolean), i(nteger), s(tring), r(eal), p(ath), x (R object or no value) # FIXME: Add special type for R functions. -# FIXME: For i and r add their range. -.irace.params.def <- structure(list(name = c(".help", ".version", ".check", ".onlytest", -"scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", -"instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", -"configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", -"targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", -"targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", -"digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", -"sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", -"nbConfigurations", "mu", "confidence", "deterministic", "seed", -"parallel", "loadBalancing", "mpi", "batchmode", "softRestart", -"softRestartThreshold", "testInstancesDir", "testInstancesFile", -"testInstances", "testNbElites", "testIterationElites", "elitist", -"elitistNewInstances", "elitistLimit", "repairConfiguration", -"capping", "cappingType", "boundType", "boundMax", "boundDigits", -"boundPar", "boundAsTimeout", "postselection", "aclib"), type = c("x", -"x", "x", "p", "p", "p", "p", "p", "p", "s", "x", "p", "p", "p", -"x", "p", "p", "i", "x", "x", "p", "i", "i", "r", "i", "i", "i", -"i", "b", "s", "i", "i", "i", "i", "i", "r", "b", "i", "i", "b", -"b", "s", "b", "r", "p", "p", "x", "i", "b", "b", "i", "i", "x", -"b", "s", "s", "i", "i", "i", "b", "r", "b"), short = c("-h", -"-v", "-c", "", "-s", "-p", "", "-l", "", "", "", "", "", "", +# FIXME: For i and r add their domain. +.irace.params.def <- structure(list(name = c(".help", ".version", ".check", ".init", +".onlytest", "scenarioFile", "execDir", "parameterFile", "forbiddenExps", +"forbiddenFile", "initConfigurations", "configurationsFile", +"logFile", "recoveryFile", "instances", "trainInstancesDir", +"trainInstancesFile", "sampleInstances", "testInstancesDir", +"testInstancesFile", "testInstances", "testNbElites", "testIterationElites", +"testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", +"targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", +"targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", +"maxTime", "budgetEstimation", "minMeasurableTime", "parallel", +"loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", +"seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", +"elitistLimit", "repairConfiguration", "capping", "cappingType", +"boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", +"postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", +"minNbSurvival", "nbConfigurations", "mu", "confidence"), type = c("x", +"x", "x", "x", "p", "p", "p", "p", "x", "p", "x", "p", "p", "p", +"s", "p", "p", "b", "p", "p", "x", "i", "b", "s", "i", "i", "p", +"p", "s", "i", "x", "x", "p", "b", "i", "i", "r", "r", "i", "b", +"b", "s", "i", "b", "i", "i", "b", "r", "b", "i", "i", "x", "b", +"s", "s", "i", "i", "i", "b", "r", "b", "i", "i", "i", "i", "i", +"r"), short = c("-h", "-v", "-c", "-i", "", "-s", "", "-p", "", +"", "", "", "-l", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", -"", "", "", "", "", "", "", "", "", "", "", "", "", "", "", "", -"", "", "", "-e", "", "", "", "", "", "", "", "", "", "", "", -""), long = c("--help", "--version", "--check", "--only-test", -"--scenario", "--parameter-file", "--exec-dir", "--log-file", -"--recovery-file", "", "", "--train-instances-dir", "--train-instances-file", -"--configurations-file", "", "--forbidden-file", "--target-runner", -"--target-runner-retries", "", "", "--target-evaluator", "--max-experiments", -"--max-time", "--budget-estimation", "--digits", "--debug-level", -"--iterations", "--experiments-per-iteration", "--sample-instances", -"--test-type", "--first-test", "--each-test", "--min-survival", -"--num-configurations", "--mu", "--confidence", "--deterministic", -"--seed", "--parallel", "--load-balancing", "--mpi", "--batchmode", -"--soft-restart", "--soft-restart-threshold", "--test-instances-dir", -"--test-instances-file", "", "--test-num-elites", "--test-iteration-elites", +"", "", "", "-q", "", "", "", "", "-e", "", "", "", "", "", "", +"", "", "", "", "", "", "", "", "", "", "", ""), long = c("--help", +"--version", "--check", "--init", "--only-test", "--scenario", +"--exec-dir", "--parameter-file", "", "--forbidden-file", "", +"--configurations-file", "--log-file", "--recovery-file", "", +"--train-instances-dir", "--train-instances-file", "--sample-instances", +"--test-instances-dir", "--test-instances-file", "", "--test-num-elites", +"--test-iteration-elites", "--test-type", "--first-test", "--each-test", +"--target-runner", "--target-runner-launcher", "--target-runner-args", +"--target-runner-retries", "", "", "--target-evaluator", "--deterministic", +"--max-experiments", "--max-time", "--budget-estimation", "--min-measurable-time", +"--parallel", "--load-balancing", "--mpi", "--batchmode", "--digits", +"--quiet", "--debug-level", "--seed", "--soft-restart", "--soft-restart-threshold", "--elitist", "--elitist-new-instances", "--elitist-limit", "", "--capping", "--capping-type", "--bound-type", "--bound-max", "--bound-digits", "--bound-par", "--bound-as-timeout", "--postselection", -"--aclib"), default = c(NA, NA, NA, "", "./scenario.txt", "./parameters.txt", -"./", "./irace.Rdata", "", "", "", "./Instances", "", "", "", -"", "./target-runner", "0", "", "", "", "0", "0", "0.02", "4", -"0", "0", "0", "1", "F-test", "5", "1", "0", "0", "5", "0.95", -"0", NA, "0", "1", "0", "0", "1", "", "", "", "", "1", "0", "1", -"1", "2", "", "0", "median", "candidate", "0", "0", "1", "1", -"0", "0"), description = c("Show this help.", "Show irace package version.", -"Check scenario.", "Only test the configurations given in the file passed as argument.", +"--aclib", "--iterations", "--experiments-per-iteration", "--min-survival", +"--num-configurations", "--mu", "--confidence"), default = c(NA, +NA, NA, "", "", "./scenario.txt", "./", "./parameters.txt", "", +"", "", "", "./irace.Rdata", "", "", "./Instances", "", "1", +"", "", "", "1", "0", "", "5", "1", "./target-runner", "", "{targetRunner} {targetRunnerArgs}", +"0", "", "", "", "0", "0", "0", "0.02", "0.01", "0", "1", "0", +"0", "4", "0", "0", NA, "1", "", "1", "1", "2", "", "0", "median", +"candidate", "0", "0", "1", "1", "0", "0", "0", "0", "0", "0", +"5", "0.95"), domain = c(NA, NA, NA, NA, NA, NA, NA, NA, NA, +NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, "F-test,t-test,t-test-holm,t-test-bonferroni", +NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, +NA, "sge,pbs,torque,slurm,htcondor", NA, NA, NA, NA, NA, NA, +NA, NA, NA, NA, NA, "median,mean,worst,best", "instance,candidate", +NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA), description = c("Show this help.", +"Show irace package version.", "Check scenario.", "Initialize the working directory with template config files.", +"Only test the configurations given in the file passed as argument.", "File that describes the configuration scenario setup and other irace settings.", -"File that contains the description of the parameters of the target algorithm.", -"Directory where the programs will be run.", "File to save tuning results as an R dataset, either absolute path or relative to execDir.", +"Directory where the programs will be run.", "File that contains the description of the parameters of the target algorithm.", +"", "File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions.", +"", "File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated.", +"File to save tuning results as an R dataset, either absolute path or relative to execDir.", "Previously saved log file to recover the execution of irace, either absolute path or relative to the current directory. If empty or NULL, recovery is not performed.", -"", "", "Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances.", +"", "Directory where training instances are located; either absolute path or relative to current directory. If no trainInstancesFiles is provided, all the files in trainInstancesDir will be listed as instances.", "File that contains a list of training instances and optionally additional parameters for them. If trainInstancesDir is provided, irace will search for the files in this folder.", -"File that contains a table of initial configurations. If empty or NULL, all initial configurations are randomly generated.", -"", "File that contains a list of logical expressions that cannot be TRUE for any evaluated configuration. If empty or NULL, do not use forbidden expressions.", -"Script called for each configuration that executes the target algorithm to be tuned. See templates.", +"Randomly sample the training instances or use them in the order given.", +"Directory where testing instances are located, either absolute or relative to current directory.", +"File containing a list of test instances and optionally additional parameters for them.", +"", "Number of elite configurations returned by irace that will be tested if test instances are provided.", +"Enable/disable testing the elite configurations found at each iteration.", +"Statistical test used for elimination. The default value selects t-test if capping is enabled or F-test, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons).", +"Number of instances evaluated before the first elimination test. It must be a multiple of eachTest.", +"Number of instances evaluated between elimination tests.", "Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided.", +"Executable that will be used to launch the target runner, when targetRunner cannot be executed directly (.e.g, a Python script in Windows).", +"Command-line arguments provided to targetRunnerLauncher. The substrings \\{targetRunner\\} and \\{targetRunnerArgs\\} will be replaced by the value of the option targetRunner and by the arguments usually passed when calling targetRunner, respectively. Example: \"-m {targetRunner --args {targetRunnerArgs}\"}.", "Number of times to retry a call to targetRunner if the call failed.", "Optional data passed to targetRunner. This is ignored by the default targetRunner function, but it may be used by custom targetRunner functions to pass persistent data around.", "Optional R function to provide custom parallelization of targetRunner.", "Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl", +"If the target algorithm is deterministic, configurations will be evaluated only once per instance.", "Maximum number of runs (invocations of targetRunner) that will be performed. It determines the maximum budget of experiments for the tuning.", "Maximum total execution time in seconds for the executions of targetRunner. targetRunner must return two values: cost and time.", "Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when maxTime > 0", -"Maximum number of decimal places that are significant for numerical (real) parameters.", -"Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages.", -"Number of iterations.", "Number of runs of the target algorithm per iteration.", -"Randomly sample the training instances or use them in the order given.", -"Statistical test used for elimination. Default test is always F-test unless capping is enabled, in which case the default test is t-test. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons).", -"Number of instances evaluated before the first elimination test. It must be a multiple of eachTest.", -"Number of instances evaluated between elimination tests.", "Minimum number of configurations needed to continue the execution of each race (iteration).", -"Number of configurations to be sampled and evaluated at each iteration.", -"Parameter used to define the number of configurations sampled and evaluated at each iteration.", -"Confidence level for the elimination test.", "If the target algorithm is deterministic, configurations will be evaluated only once per instance.", -"Seed of the random number generator (by default, generate a random seed).", +"Minimum time unit that is still (significantly) measureable.", "Number of calls to targetRunner to execute in parallel. Values 0 or 1 mean no parallelization.", "Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster.", "Enable/disable MPI. Use Rmpi to execute targetRunner in parallel (parameter parallel is the number of slaves).", -"Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque or slurm. targetRunner must submit jobs to the cluster using, for example, qsub.", +"Specify how irace waits for jobs to finish when targetRunner submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. targetRunner must submit jobs to the cluster using, for example, qsub.", +"Maximum number of decimal places that are significant for numerical (real) parameters.", +"Reduce the output generated by irace to a minimum.", "Debug level of the output of irace. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages.", +"Seed of the random number generator (by default, generate a random seed).", "Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model.", "Soft restart threshold value for numerical parameters. If NA, NULL or \"\", it is computed as 10^-digits.", -"Directory where testing instances are located, either absolute or relative to current directory.", -"File containing a list of test instances and optionally additional parameters for them.", -"", "Number of elite configurations returned by irace that will be tested if test instances are provided.", -"Enable/disable testing the elite configurations found at each iteration.", "Enable/disable elitist irace.", "Number of instances added to the execution list before previous instances in elitist irace.", "In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit.", "User-defined R function that takes a configuration generated by irace and repairs it.", @@ -99,37 +107,43 @@ "Penalization constant for timed out executions (executions that reach boundMax execution time).", "Replace the configuration cost of bounded executions with boundMax.", "Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace.", -"Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script." -)), .Names = c("name", "type", "short", "long", "default", "description" -), row.names = c(".help", ".version", ".check", ".onlytest", -"scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", -"instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", -"configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", -"targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", -"targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", -"digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", -"sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", -"nbConfigurations", "mu", "confidence", "deterministic", "seed", -"parallel", "loadBalancing", "mpi", "batchmode", "softRestart", -"softRestartThreshold", "testInstancesDir", "testInstancesFile", -"testInstances", "testNbElites", "testIterationElites", "elitist", -"elitistNewInstances", "elitistLimit", "repairConfiguration", +"Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script.", +"Maximum number of iterations.", "Number of runs of the target algorithm per iteration.", +"Minimum number of configurations needed to continue the execution of each race (iteration).", +"Number of configurations to be sampled and evaluated at each iteration.", +"Parameter used to define the number of configurations sampled and evaluated at each iteration.", +"Confidence level for the elimination test.")), row.names = c(".help", +".version", ".check", ".init", ".onlytest", "scenarioFile", "execDir", +"parameterFile", "forbiddenExps", "forbiddenFile", "initConfigurations", +"configurationsFile", "logFile", "recoveryFile", "instances", +"trainInstancesDir", "trainInstancesFile", "sampleInstances", +"testInstancesDir", "testInstancesFile", "testInstances", "testNbElites", +"testIterationElites", "testType", "firstTest", "eachTest", "targetRunner", +"targetRunnerLauncher", "targetRunnerLauncherArgs", "targetRunnerRetries", +"targetRunnerData", "targetRunnerParallel", "targetEvaluator", +"deterministic", "maxExperiments", "maxTime", "budgetEstimation", +"minMeasurableTime", "parallel", "loadBalancing", "mpi", "batchmode", +"digits", "quiet", "debugLevel", "seed", "softRestart", "softRestartThreshold", +"elitist", "elitistNewInstances", "elitistLimit", "repairConfiguration", "capping", "cappingType", "boundType", "boundMax", "boundDigits", -"boundPar", "boundAsTimeout", "postselection", "aclib"), class = "data.frame") -.irace.params.names <- c("scenarioFile", "parameterFile", "execDir", "logFile", "recoveryFile", -"instances", "initConfigurations", "trainInstancesDir", "trainInstancesFile", -"configurationsFile", "forbiddenExps", "forbiddenFile", "targetRunner", -"targetRunnerRetries", "targetRunnerData", "targetRunnerParallel", -"targetEvaluator", "maxExperiments", "maxTime", "budgetEstimation", -"digits", "debugLevel", "nbIterations", "nbExperimentsPerIteration", -"sampleInstances", "testType", "firstTest", "eachTest", "minNbSurvival", -"nbConfigurations", "mu", "confidence", "deterministic", "seed", -"parallel", "loadBalancing", "mpi", "batchmode", "softRestart", -"softRestartThreshold", "testInstancesDir", "testInstancesFile", -"testInstances", "testNbElites", "testIterationElites", "elitist", -"elitistNewInstances", "elitistLimit", "repairConfiguration", -"capping", "cappingType", "boundType", "boundMax", "boundDigits", -"boundPar", "boundAsTimeout", "postselection", "aclib") +"boundPar", "boundAsTimeout", "postselection", "aclib", "nbIterations", +"nbExperimentsPerIteration", "minNbSurvival", "nbConfigurations", +"mu", "confidence"), class = "data.frame") +.irace.params.names <- c("scenarioFile", "execDir", "parameterFile", "forbiddenExps", +"forbiddenFile", "initConfigurations", "configurationsFile", +"logFile", "recoveryFile", "instances", "trainInstancesDir", +"trainInstancesFile", "sampleInstances", "testInstancesDir", +"testInstancesFile", "testInstances", "testNbElites", "testIterationElites", +"testType", "firstTest", "eachTest", "targetRunner", "targetRunnerLauncher", +"targetRunnerLauncherArgs", "targetRunnerRetries", "targetRunnerData", +"targetRunnerParallel", "targetEvaluator", "deterministic", "maxExperiments", +"maxTime", "budgetEstimation", "minMeasurableTime", "parallel", +"loadBalancing", "mpi", "batchmode", "digits", "quiet", "debugLevel", +"seed", "softRestart", "softRestartThreshold", "elitist", "elitistNewInstances", +"elitistLimit", "repairConfiguration", "capping", "cappingType", +"boundType", "boundMax", "boundDigits", "boundPar", "boundAsTimeout", +"postselection", "aclib", "nbIterations", "nbExperimentsPerIteration", +"minNbSurvival", "nbConfigurations", "mu", "confidence") ## FIXME: If these values are special perhaps they should be saved in $state ? .irace.params.recover <- c("instances", "seed", "testInstances", # We need this because this data may mutate diff --git a/R/irace-package.R b/R/irace-package.R index ec8e995..05f23ab 100644 --- a/R/irace-package.R +++ b/R/irace-package.R @@ -6,8 +6,8 @@ #' @docType package #' @import stats utils compiler #' @importFrom R6 R6Class -#' @importFrom grDevices dev.new dev.off pdf cairo_pdf rgb -#' @importFrom graphics abline axis barplot boxplot hist lines matplot mtext par plot points strwidth text bxp grid +#' @importFrom grDevices dev.new dev.off pdf +#' @importFrom graphics abline axis barplot boxplot lines matplot mtext par plot points strwidth text bxp grid #' #' #' @details License: GPL (>= 2) @@ -59,25 +59,25 @@ #' sum(x * x - 10 * cos(2 * pi * x) + 10) #' } #' -#' ## We generate 200 instances (in this case, weights): -#' weights <- rnorm(200, mean = 0.9, sd = 0.02) +#' ## We generate 20 instances (in this case, weights): +#' weights <- rnorm(20, mean = 0.9, sd = 0.02) #' #' ## On this set of instances, we are interested in optimizing two #' ## parameters of the SANN algorithm: tmax and temp. We setup the #' ## parameter space as follows: -#' parameters.table <- ' -#' tmax "" i (1, 5000) +#' parameters_table <- ' +#' tmax "" i,log (1, 5000) #' temp "" r (0, 100) #' ' #' #' ## We use the irace function readParameters to read this table: -#' parameters <- readParameters(text = parameters.table) +#' parameters <- readParameters(text = parameters_table) #' #' ## Next, we define the function that will evaluate each candidate #' ## configuration on a single instance. For simplicity, we restrict to #' ## three-dimensional functions and we set the maximum number of -#' ## iterations of SANN to 5000. -#' target.runner <- function(experiment, scenario) +#' ## iterations of SANN to 1000. +#' target_runner <- function(experiment, scenario) #' { #' instance <- experiment$instance #' configuration <- experiment$configuration @@ -89,7 +89,7 @@ #' return(weight * f_rastrigin(x) + (1 - weight) * f_rosenbrock(x)) #' } #' res <- stats::optim(par,fn, method="SANN", -#' control=list(maxit=5000 +#' control=list(maxit=1000 #' , tmax = as.numeric(configuration[["tmax"]]) #' , temp = as.numeric(configuration[["temp"]]) #' )) @@ -98,42 +98,42 @@ #' ## - 'error' is a string used to report an error #' ## - 'outputRaw' is a string used to report the raw output of calls to #' ## an external program or function. -#' ## - 'call' is a string used to report how target.runner called the +#' ## - 'call' is a string used to report how target_runner called the #' ## external program or function. #' return(list(cost = res$value)) #' } #' #' ## We define a configuration scenario by setting targetRunner to the -#' ## function define above, instances to the first 100 random weights, and -#' ## a maximum budget of 1000 calls to targetRunner. -#' scenario <- list(targetRunner = target.runner, -#' instances = weights[1:100], -#' maxExperiments = 1000, +#' ## function define above, instances to the first 10 random weights, and +#' ## a maximum budget of 'maxExperiments' calls to targetRunner. +#' scenario <- list(targetRunner = target_runner, +#' instances = weights[1:10], +#' maxExperiments = 500, #' # Do not create a logFile #' logFile = "") #' #' ## We check that the scenario is valid. This will also try to execute -#' ## target.runner. +#' ## target_runner. #' checkIraceScenario(scenario, parameters = parameters) #' #' \donttest{ #' ## We are now ready to launch irace. We do it by means of the irace #' ## function. The function will print information about its #' ## progress. This may require a few minutes, so it is not run by default. -#' tuned.confs <- irace(scenario = scenario, parameters = parameters) +#' tuned_confs <- irace(scenario = scenario, parameters = parameters) #' #' ## We can print the best configurations found by irace as follows: -#' configurations.print(tuned.confs) +#' configurations.print(tuned_confs) #' #' ## We can evaluate the quality of the best configuration found by #' ## irace versus the default configuration of the SANN algorithm on -#' ## the other 100 instances previously generated. +#' ## the other 10 instances previously generated. #' ## To do so, first we apply the default configuration of the SANN #' ## algorithm to these instances: #' test <- function(configuration) #' { -#' res <- lapply(weights[101:200], -#' function(x) target.runner( +#' res <- lapply(weights[11:20], +#' function(x) target_runner( #' experiment = list(instance = x, #' configuration = configuration), #' scenario = scenario)) @@ -143,7 +143,7 @@ #' ## We extract and apply the winning configuration found by irace #' ## to these instances: -#' tuned <- test (removeConfigurationsMetaData(tuned.confs[1,])) +#' tuned <- test(removeConfigurationsMetaData(tuned_confs[1,])) #' #' ## Finally, we can compare using a boxplot the quality obtained with the #' ## default parametrization of SANN and the quality obtained with the @@ -156,3 +156,5 @@ #' NULL +# Prefix for printing messages to the user. +.msg.prefix <- "== irace == " diff --git a/R/irace.R b/R/irace.R index 7e949c9..d9c7143 100644 --- a/R/irace.R +++ b/R/irace.R @@ -25,6 +25,7 @@ # .Random.seed and .irace are special for (name in setdiff(names(iraceResults$state), c(".Random.seed", ".irace"))) assign(name, iraceResults$state[[name]]) + # FIXME: Check that irace.version matches and warn if not. assign(".Random.seed", iraceResults$state$.Random.seed, .GlobalEnv) for (name in ls(iraceResults$state$.irace)) assign(name, get(name, envir = iraceResults$state$.irace), envir = .irace) @@ -50,8 +51,8 @@ selected <- 1:nrow(configurations) for (i in seq_along(param.names)) { param <- param.names[i] - lower <- parameters$domain[[param]][1] - upper <- parameters$domain[[param]][2] + x.domain <- getDependentBound(parameters, param, x) + x.range <- diff(x.domain) X <- x[[param]] # FIXME: Since at the end we select a subset of configurations, we could use selected here. @@ -69,7 +70,20 @@ # FIXME: Why is this updating d[j]? It seems that if the difference is # large for one configuration, then it will be assumed to be large for # the rest. - d[j] <- max(d[j], abs((as.numeric(X) - as.numeric(Y)) / (upper - lower))) + if (parameters$isDependent[param]) { + # Compare depedent domains by normalising their values to their own ranges first + # and calculating the difference. (When possible) + y.domain <- getDependentBound(parameters, param, configurations[selected[j],]) + y.range <- diff(x.domain) + dx <- ifelse (x.range == 0, 0, (as.numeric(X) - x.domain[1]) / x.range) + dy <- ifelse (y.range == 0, 0, (as.numeric(Y) - y.domain[1]) / y.range) + + d[j] <- max(d[j], abs(dx - dy)) + } else { + # FIXME: We should calculate (X - x.domain[1]) / x.range once for all configurations + # and all parameters, then calculate the differences using vectorization. + d[j] <- max(d[j], abs((as.numeric(X) - as.numeric(Y)) / x.range)) + } if (d[j] > threshold) isSimilar.mat[j,i] <- FALSE } } @@ -127,7 +141,7 @@ ## filtering them out: configurations <- configurations [keepIdx, , drop=FALSE] ## filtering their strings out (to use them to define blocks): - strings <- strings [keepIdx] + strings <- strings[keepIdx] ## if everything is already filtered out, return if (nrow(configurations) == 0) { @@ -191,10 +205,7 @@ ## Number of iterations. -computeNbIterations <- function(nbParameters) -{ - return (2 + log2(nbParameters)) -} +computeNbIterations <- function(nbParameters) (2 + log2(nbParameters)) ## Computational budget at each iteration. computeComputationalBudget <- function(remainingBudget, indexIteration, @@ -219,10 +230,7 @@ ## Termination of a race at each iteration. The race will stop if the ## number of surviving configurations is equal or less than this number. -computeTerminationOfRace <- function(nbParameters) -{ - return (2 + log2(nbParameters)) -} +computeTerminationOfRace <- function(nbParameters) (2 + log2(nbParameters)) ## Compute the minimum budget required, and exit early in case the ## budget given by the user is insufficient. @@ -330,10 +338,15 @@ } else { requireNamespace("parallel", quietly = TRUE) if (.Platform$OS.type == 'windows' && is.null(.irace$cluster)) { + # FIXME: makeCluster does not print the output generated by the workers + # on Windows. We need to use the future package for that: + # https://stackoverflow.com/questions/56501937/how-to-print-from-clusterapply .irace$cluster <- parallel::makeCluster(parallel) + if (scenario$debugLevel >= 1) irace.note("makeCluster initialized for ", parallel, " jobs.") # In Windows, this needs to be exported, or we get: ## Error in checkForRemoteErrors(val) : ## 2 nodes produced errors; first error: could not find function "target.runner" + parallel::clusterExport(.irace$cluster, ls(environment(startParallel)), envir=environment(startParallel)) parallel::clusterExport(.irace$cluster, list("target.runner"), envir=.irace) # In addition, we export the global environment because the user may # have defined stuff there. There must be a better way to do this, but @@ -347,7 +360,7 @@ stopParallel <- function() { if (!is.null(.irace$cluster)) { - parallel::stopCluster(.irace$cluster) + try(parallel::stopCluster(.irace$cluster), silent=TRUE) .irace$cluster <- NULL } } @@ -356,6 +369,7 @@ { # We need to do this here to use/recover .Random.seed later. if (is.na(scenario$seed)) { + # FIXME: We should store this seed in state not in scenario. We should not modify scenario. scenario$seed <- trunc(runif(1, 1, .Machine$integer.max)) } set.seed(scenario$seed) @@ -363,7 +377,7 @@ irace.note("RNGkind: ", paste0(RNGkind(), collapse = " "), "\n") irace.note(".Random.seed: ", paste0(.Random.seed, collapse = ", "), "\n") } - return(scenario) + scenario } ## Generate instances + seed. @@ -380,14 +394,13 @@ # Sample instances index in groups (ntimes) sindex <- as.vector(sapply(rep(length(instances), ntimes), sample.int, replace = FALSE)) } else { - sindex <- rep(1:length(instances), ntimes) + sindex <- rep(1L:length(instances), ntimes) } # Sample seeds. # 2147483647 is the maximum value for a 32-bit signed integer. # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. - tmp <- data.frame (instance = sindex, - seed = sample.int(2147483647, size = ntimes * length(instances), replace = TRUE)) - return(tmp) + data.frame(instance = sindex, + seed = sample.int(2147483647L, size = length(sindex), replace = TRUE), stringsAsFactors=FALSE) } addInstances <- function(scenario, instancesList, n.instances) @@ -396,7 +409,7 @@ if (is.null.or.empty(instancesList)) instancesList <- generateInstances(scenario, n.instances) # If deterministic, we have already added all instances. - else if (! scenario$deterministic) + else if (!scenario$deterministic) instancesList <- rbind(instancesList, generateInstances(scenario, n.instances)) # FIXME: Something is adding rownames. Clear them to avoid future problems. @@ -469,7 +482,7 @@ scenario$configurationsFile, "'.") cat("# Adding", nrow(initConfigurations), "initial configuration(s)\n") if (scenario$debugLevel >= 2) - print(as.data.frame(scenario$initConfigurations, stringAsFactor = FALSE), digits=15) + print(as.data.frame(scenario$initConfigurations, stringsAsFactors = FALSE), digits=15) } else { initConfigurations <- confs_from_file } @@ -491,49 +504,102 @@ allConfigurations <- as.data.frame(matrix(ncol = length(configurations.colnames), nrow = 0, - dimnames = list(NULL, configurations.colnames))) - } - return(allConfigurations) + dimnames = list(NULL, configurations.colnames)), + stringsAsFactors=FALSE) + } + allConfigurations } #' irace #' -#' \code{irace} implements iterated Race. It receives some parameters to be tuned +#' `irace` implements iterated Race. It receives some parameters to be tuned #' and returns the best configurations found, namely, the elite configurations #' obtained from the last iterations (and sorted by rank). #' #' @template arg_scenario #' @template arg_parameters #' -#' @details The function \code{irace} executes the tuning procedure using -#' the information provided in \code{scenario} and \code{parameters}. Initially it checks -#' the correctness of \code{scenario} and recovers a previous execution if -#' \code{scenario$recoveryFile} is set. A R data file log of the execution is created -#' in \code{scenario$logFile}. +#' @details The function `irace` executes the tuning procedure using +#' the information provided in `scenario` and `parameters`. Initially it checks +#' the correctness of `scenario` and recovers a previous execution if +#' `scenario$recoveryFile` is set. A R data file log of the execution is created +#' in `scenario$logFile`. #' #' @template return_irace #' @examples #' \dontrun{ #' parameters <- readParameters("parameters.txt") -#' scenario <- readScenario(filename = "scenario.txt", -#' scenario = defaultScenario()) +#' scenario <- readScenario(filename = "scenario.txt") #' irace(scenario = scenario, parameters = parameters) #' } #' #' @seealso #' \describe{ -#' \item{\code{\link{irace.main}}}{a higher-level command-line interface to \code{irace}.} -#' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} -#' \item{\code{\link{readParameters}}}{read the target algorithm parameters from a file.} -#' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} -#' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} +#' \item{[irace.main()]}{a higher-level interface to `irace`.} +#' \item{[irace.cmdline()]}{a command-line interface to `irace`.} +#' \item{[readScenario()]}{for reading a configuration scenario from a file.} +#' \item{[readParameters()]}{read the target algorithm parameters from a file.} +#' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} +#' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste +#' @concept running #' @export irace <- function(scenario, parameters) -{ - catInfo <- function(..., verbose = TRUE) { + irace_common(scenario, parameters, simple = TRUE) + +irace_common <- function(scenario, parameters, simple, output.width = 9999L) +{ + if (!simple) { + op <- options(width = output.width) # Do not wrap the output. + on.exit(options(op), add = TRUE) + } + scenario <- checkScenario(scenario) + debugLevel <- scenario$debugLevel + + if (debugLevel >= 1) { + op.debug <- options(warning.length = 8170, + error = if (interactive()) utils::recover + else irace.dump.frames) + on.exit(options(op.debug), add = TRUE) + printScenario (scenario) + } + + if (missing(parameters)) { + # Read parameters definition + parameters <- readParameters (file = scenario$parameterFile, + digits = scenario$digits, + debugLevel = debugLevel) + } else { + parameters <- checkParameters(parameters) + } + + eliteConfigurations <- irace_run(scenario = scenario, parameters = parameters) + if (simple) return(eliteConfigurations) + + if (!scenario$quiet) { + cat("# Best configurations (first number is the configuration ID;", + " listed from best to worst according to the ", + test.type.order.str(scenario$testType), "):\n", sep = "") + configurations.print(eliteConfigurations) + + cat("# Best configurations as commandlines (first number is the configuration ID; same order as above):\n") + configurations.print.command (eliteConfigurations, parameters) + } + + if (scenario$postselection > 0) + psRace(iraceLogFile=scenario$logFile, postselection=scenario$postselection, elites=TRUE) + + testing_fromlog(logFile = scenario$logFile) + + invisible(eliteConfigurations) +} + +irace_run <- function(scenario, parameters) +{ + quiet <- scenario$quiet + catInfo <- if (quiet) do_nothing else function(..., verbose = TRUE) { irace.note (..., "\n") if (verbose) { cat ("# Iteration: ", indexIteration, "\n", @@ -541,36 +607,47 @@ "# experimentsUsedSoFar: ", experimentsUsedSoFar, "\n", "# timeUsed: ", timeUsed, "\n", "# remainingBudget: ", remainingBudget, "\n", - "# currentBudget: ", currentBudget, "\n", + "# currentBudget: ", currentBudget, "\n", "# number of elites: ", nrow(eliteConfigurations), "\n", "# nbConfigurations: ", nbConfigurations, "\n", sep = "") } } - - scenario <- checkScenario(defaultScenario(scenario)) + + irace_finish <- function(iraceResults, scenario, reason) { + elapsed <- timer$elapsed() + if (!quiet) + cat("# Total CPU user time: ", elapsed["user"], ", CPU sys time: ", elapsed["system"], + ", Wall-clock time: ", elapsed["wallclock"], "\n", sep="") + iraceResults$state$elapsed = elapsed + iraceResults$state$completed = reason + irace_save_logfile(iraceResults, scenario) + iraceResults$state$eliteConfigurations + } + + timer <- Timer$new() + debugLevel <- scenario$debugLevel # Recover state from file? - if (!is.null(scenario$recoveryFile)) { + if (!is.null.or.empty(scenario$recoveryFile)) { irace.note ("Resuming from file: '", scenario$recoveryFile,"'\n") recoverFromFile(scenario$recoveryFile) # We call checkScenario again to fix any inconsistencies in the recovered # data. + # FIXME: Do not call checkScenario earlier and instead do the minimum to check recoveryFile. + scenario <- checkScenario(scenario) firstRace <- FALSE - scenario <- checkScenario(scenario) + stopParallel() startParallel(scenario) on.exit(stopParallel(), add = TRUE) - } else { # Do not recover firstRace <- TRUE - scenario <- irace.init (scenario) + scenario <- irace.init(scenario) forbiddenExps <- scenario$forbiddenExps - debugLevel <- scenario$debugLevel # Set options controlling debug level. # FIXME: This should be the other way around, the options set the debugLevel. options(.race.debug.level = debugLevel) options(.irace.debug.level = debugLevel) - # Create a data frame of all configurations ever generated. allConfigurations <- allConfigurationsInit(scenario, parameters) nbUserConfigurations <- nrow(allConfigurations) @@ -588,7 +665,7 @@ ) model <- NULL nbConfigurations <- 0 - eliteConfigurations <- data.frame() + eliteConfigurations <- data.frame(stringsAsFactors=FALSE) nbIterations <- ifelse (scenario$nbIterations == 0, computeNbIterations(parameters$nbVariable), @@ -652,8 +729,8 @@ output <- do.experiments(configurations = allConfigurations[next.configuration:nconfigurations, ], ninstances = ninstances, scenario = scenario, parameters = parameters) iraceResults$experimentLog <- rbind(iraceResults$experimentLog, - cbind(rep(0, nrow(output$experimentLog)), - output$experimentLog)) + # These experiments are assigned iteration 0 + cbind(iteration=0L, output$experimentLog)) iraceResults$experiments <- merge.matrix (iraceResults$experiments, output$experiments) @@ -687,7 +764,7 @@ } else { nconfigurations <- min(1024, nconfigurations + new.conf) } - } + } # end of while(TRUE) if (length(rejectedIDs) > 0) { irace.note ("Immediately rejected configurations: ", @@ -764,11 +841,11 @@ if (scenario$capping) paste0("# capping: ", scenario$cappingType, "\n", "# type bound: ", scenario$boundType, "\n", - "# maxBound: ", scenario$boundMax, "\n", + "# boundMax: ", scenario$boundMax, "\n", "# par bound: ", scenario$boundPar, "\n", "# bound digits: ", scenario$boundDigits, "\n") else if (!is.null(scenario$boundMax)) - paste0("# maxBound: ", scenario$boundMax, "\n"), + paste0("# boundMax: ", scenario$boundMax, "\n"), verbose = FALSE) @@ -790,28 +867,25 @@ boundEstimate = boundEstimate, rejectedIDs = rejectedIDs, forbiddenExps = forbiddenExps, - completed = list(flag=FALSE, msg="")) - # Consistency checks - irace.assert(sum(!is.na(iraceResults$experiments)) == experimentsUsedSoFar) - irace.assert(nrow(iraceResults$experimentLog) == experimentsUsedSoFar) - + completed = "Incomplete") ## Save to the log file iraceResults$allConfigurations <- allConfigurations irace_save_logfile(iraceResults, scenario) + # Consistency checks + irace.assert(nrow(iraceResults$experimentLog) == experimentsUsedSoFar) + # With elitist=0 we may re-run the same configuration on the same (instance,seed) pair + # FIXME: This assert is failing when sampleInstances=FALSE because we are re-executing again an instance that was executed at the start of the race. + if (FALSE && scenario$elitist) + irace.assert(sum(!is.na(iraceResults$experiments)) == experimentsUsedSoFar) + if (remainingBudget <= 0) { catInfo("Stopped because budget is exhausted") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Budget exhausted" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Budget exhausted")) } if (scenario$maxTime > 0 && timeUsed >= scenario$maxTime) { catInfo("Stopped because time budget is exhausted") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Time budget exhausted" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Time budget exhausted")) } if (indexIteration > nbIterations) { @@ -821,10 +895,7 @@ if (debugLevel >= 1) { catInfo("Limit of iterations reached", verbose = FALSE) } - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Limit of iterations reached" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Limit of iterations reached")) } } # Compute the current budget (nb of experiments for this iteration), @@ -868,10 +939,7 @@ } else { catInfo("Stopped because ", "there is not enough budget to enforce the value of nbConfigurations.") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Not enough budget to enforce the value of nbConfigurations" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Not enough budget to enforce the value of nbConfigurations")) } } @@ -880,10 +948,7 @@ catInfo("Stopped because there is not enough budget left to race more than ", "the minimum (", minSurvival,")\n", "# You may either increase the budget or set 'minNbSurvival' to a lower value") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Not enough budget to race more than the minimum configurations" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race more than the minimum configurations")) } @@ -901,10 +966,7 @@ catInfo("Stopped because ", "there is not enough budget left to race newly sampled configurations") #(number of elites + 1) * (mu + min(5, indexIteration)) > remainingBudget" - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Not enough budget left to race newly sampled configurations" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Not enough budget left to race newly sampled configurations")) } if (scenario$elitist) { @@ -914,18 +976,12 @@ + nrow(eliteConfigurations) * min(scenario$elitistNewInstances, max(scenario$mu, scenario$firstTest)) > currentBudget) { catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Not enough budget to race all configurations up to the first test (or mu)" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) } } else if (nbConfigurations * max(scenario$mu, scenario$firstTest) > currentBudget) { catInfo("Stopped because there is not enough budget left to race all configurations up to the first test (or mu)") - iraceResults$state$completed$flag = TRUE - iraceResults$state$completed$msg = "Not enough budget to race all configurations up to the first test (or mu)" - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) + return(irace_finish(iraceResults, scenario, reason = "Not enough budget to race all configurations up to the first test (or mu)")) } catInfo("Iteration ", indexIteration, " of ", nbIterations, "\n", @@ -959,7 +1015,7 @@ allConfigurations <- rbind(allConfigurations, newConfigurations) rownames(allConfigurations) <- allConfigurations$.ID. raceConfigurations <- allConfigurations[allConfigurations$.ID. %!in% rejectedIDs, , drop = FALSE] - } else if (nbNewConfigurations < 0) { + } else if (nbNewConfigurations <= 0) { # We let the user know that not all configurations will be used. if (nbUserConfigurations > nbConfigurations) { catInfo("Only ", nbConfigurations, @@ -982,14 +1038,13 @@ nbNewConfigurations <- nbConfigurations - nrow(eliteConfigurations) # Update the model based on elites configurations - if (debugLevel >= 1) { irace.note("Update model\n") } + if (debugLevel >= 1) irace.note("Update model\n") model <- updateModel(parameters, eliteConfigurations, model, indexIteration, nbIterations, nbNewConfigurations, scenario) - if (debugLevel >= 2) { printModel (model) } + if (debugLevel >= 2) printModel (model) + if (debugLevel >= 1) + irace.note("Sample ", nbNewConfigurations, " configurations from model\n") - if (debugLevel >= 1) { - irace.note("Sample ", nbNewConfigurations, " configurations from model\n") - } newConfigurations <- sampleModel(parameters, eliteConfigurations, model, nbNewConfigurations, digits = scenario$digits, @@ -1116,12 +1171,13 @@ # FIXME: Since we only actually keep the alive ones, we don't need # to carry around rejected ones in raceResults$configurations. This # would reduce overhead. - eliteConfigurations <- extractElites(raceResults$configurations, + eliteConfigurations <- extractElites(scenario, parameters, + raceResults$configurations, min(raceResults$nbAlive, minSurvival)) irace.note("Elite configurations (first number is the configuration ID;", " listed from best to worst according to the ", test.type.order.str(scenario$testType), "):\n") - configurations.print(eliteConfigurations, metadata = debugLevel >= 1) + if (!quiet) configurations.print(eliteConfigurations, metadata = debugLevel >= 1) iraceResults$iterationElites <- c(iraceResults$iterationElites, eliteConfigurations$.ID.[1]) iraceResults$allElites[[indexIteration]] <- eliteConfigurations$.ID. @@ -1146,9 +1202,5 @@ irace.print.memUsed() } } - # This code is actually never executed because we return above. - # Leslie: adding this just in case - iraceResults$state$completed$flag = TRUE - irace_save_logfile(iraceResults, scenario) - return (eliteConfigurations) -} + irace.internal.error("This code is actually never executed because we return above") +} diff --git a/R/irace2pyimp.R b/R/irace2pyimp.R deleted file mode 100644 index aede2e9..0000000 --- a/R/irace2pyimp.R +++ /dev/null @@ -1,584 +0,0 @@ -# Convert an irace.Rdata file into input format supported by the parameter -# importance analysis tool PyImp -# (https://github.com/automl/ParameterImportance). -# -# PyImp is developed by the AutoML-Freiburgh group (https://github.com/automl) - -single_quote <- function(s) paste0("'",s,"'") -double_quote <- function(s) paste0('"',s,'"') - -.irace2pyimp_header <- -'#------------------------------------------------------------------------------ -# irace2pyimp converts an irace.Rdata file into input format supported by the -# parameter importance analysis tool PyImp (https://github.com/automl/ParameterImportance) -# -# irace2pyimp was developed by Nguyen Dang -# -# PyImp is developed by the AutoML-Freiburg-Hannover group (https://www.automl.org/) -# -# To see usage: irace2pyimp --help -# See examples in `$IRACE_HOME/inst/examples/irace2pyimp/acotsp/run.sh` -# and in `$IRACE_HOME/inst/examples/irace2pyimp/002-TemplateDesign/run.sh` -# -# irace2pyimp can also be called as an R function instead. To see usage of the -# function, run in R console: -# > library(irace) -# > ?irace2pyimp -# -# The generated files include: -# * `params.pcs` : a text file containing the parameter space definition. -# * `runhistory.json` : a JSON file containing the list of algorithm configurations -# evaluated during the tuning and the performance data obtained. -# * `traj_aclib2.json` : a JSON file containing the best configurations after -# each iteration of irace. -# * `scenario.txt` : a text file containing the definition of the tuning scenario. -# * `instances.txt` : a text file containing the list of instances. -# * `features.csv` : a text file containing instance features. -#------------------------------------------------------------------------------ -\n' - -# list of command line arguments -.irace2pyimp_args <- read.table(header=TRUE, stringsAsFactors = FALSE, row.names = "name", text=' -name type short long default description -.help x "-h" "--help" NA "Show this help." -normalise s "-n" "--normalise" "none" "Normalise the cost metric values into the range of [0,1] before converting to PyImp format. Values: {none(default), instance, feature}, which correspond to no-normalisation, normalisation based on instance, and normalisation based on features, respectively." -outDir s "-o" "--out-dir" "./pyimp-input" "Directory where all generated data are stored. Default: ./pyimp-input" -instanceFeatureFile s "-f" "--instance-feature-file" NA " A .csv file containing instance features (one line per instance, sorted in the same order as the list of instances input to irace). The first line contains feature names. By default, instance index is used as a feature." -filterConditions s "-c" "--filter-conditions" "NA" "Only extract configurations that satisfies the given conditions. The conditions are in R expression format. Default: no filter" -defaultConfigurationID i "-i" "--default-configuration-index" 1 "Index of default configuration (starting from 1), used by ablation analysis. Default value: 1" -iraceRdataFile s "-r" "--irace-data-file" NA "The log .Rdata file generated by irace" -ignoreUnsupported b "-ignore" "--ignore-unsupported" 0 "Forbidden configurations and repairConfiguration are not supported by the script. Set this flag to 1 to ignore them and proceed with your own risk. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis\'s path. Values: {0,1}" -') - -load_irace_rdata <- function(filename) -{ - load(filename) - - if ('iraceResults' %!in% ls()) { - irace.error(filename, " is not a valid .Rdata file generated by irace") - } - - # Some fields are not present in the old version of irace, so we assign them as new version's default values - if ('capping' %!in% names(iraceResults$scenario)) - iraceResults$scenario$capping <- FALSE - - return(iraceResults) -} - -filter_data <- function(rdata, conditions, defaultConfigurationID) -{ - # - remove all configurations not satistfying conditions in: - # + rdata$allConfigurations - # + rdata$iterationElites - # + rdata$experiments & rdata$experimentLog - # - also re-assign default configuration if necessary - - conditions <- parse(text=conditions) - - # update allConfigurations - allConfigurations <- rdata$allConfigurations - allConfigurations <- allConfigurations[eval(conditions, allConfigurations), , drop = FALSE] - allConfigurations <- allConfigurations[order(allConfigurations$.ID.), , drop = FALSE] - - # update defaultConfigurationID - if (defaultConfigurationID %!in% allConfigurations$.ID.) { - irace.warning("default configuration '", defaultConfigurationID, - "' does not satisfy condition '", as.character(conditions), - "', thus setting configuration ", allConfigurations$.ID.[1], - " as default configuration instead.") - defaultConfigurationID <- allConfigurations$.ID.[1] - } - - # update iterationElites - removed <- setdiff(rdata$iterationElites, allConfigurations$.ID.) - if (length(removed) > 0) - irace.warning("elite configuration(s) ", paste0(removed, collapse=", "), - " did not satisfy condition ", as.character(conditions), - ' and were eliminated') - - iterationElites <- setdiff(rdata$iterationElites, removed) - - # update experiments - experiments <- rdata$experiments[, allConfigurations$.ID., drop = FALSE] - - # update experimentLog - experimentLog <- rdata$experimentLog[rdata$experimentLog[,'configuration'] %in% allConfigurations$.ID., ,drop = FALSE] - - rdata$allConfigurations <- allConfigurations - rdata$experiments <- experiments - rdata$experimentLog <- experimentLog - rdata$iterationElites <- iterationElites - rdata$defaultConfigurationID <- defaultConfigurationID - return(rdata) -} - -generate_feature_file <- function(file = "features.csv", instances, instanceFeatureFile) -{ - # Generate features.csv - cat(sep="", "Generating feature file '", file, "' ...\n") - - #--- features.csv ----- - tFeatures <- data.frame(instance=instances) - - # if no instance features are provided, instance index will be the only - # feature. - if (is.null.or.na(instanceFeatureFile)) { - tFeatures$id <- 1:length(instances) - - } else {# otherwise, add features - if (!file.exists(instanceFeatureFile)) { - irace.error("Instance feature file '", instanceFeatureFile,"' does not exists.") - } - features <- read.table(instanceFeatureFile, header=TRUE) - if (nrow(features) != length(instances)) { - irace.error("The number of instances (", nrow(features), ") in '", instanceFeatureFile, - "' does not match the instance list given to irace (", length(instances),")") - } - tFeatures <- cbind(tFeatures, features) - } - - # write to features.csv - write.csv(tFeatures,file = file, row.names=FALSE, quote=FALSE) - return (tFeatures) -} - -remove_fixed_parameters <- function(rdata) -{ - # remove fixed parameters, as we don't need them in the analyses - parameters <- rdata$parameters - - # update parameters - fixedParams <- names(which(parameters$isFixed)) - if (length(fixedParams) > 0) { - cat(sep="", "Removing fixed parameters: ", paste0(fixedParams, collapse=", "), " ...\n") - varParams <- names(which(!parameters$isFixed)) - # Go over 'names','types','switches','hierarchy', etc and keep only - # varParams (except for elements that contain a single value as - # $nbParameters) - parameters <- lapply(parameters, function(x) { - if (length(x) == 1) return(x) - else return(x[varParams]) - }) - parameters$nbParameters <- parameters$nbParameters - length(fixedParams) - parameters <- rdata$parameters <- parameters - rdata$allConfigurations <- rdata$allConfigurations[, varParams] - } - return (rdata) -} - -generate_pcs_file <- function(file, parameters, defaultConfiguration) -{ - #---- generate param file in SMAC's format ------ - cat(sep="", "Generating parameter definition file '", file, "' ...\n") - - # param types, ranges, and default value - paramlines <- c() - for (param in parameters$names) { - type <- parameters$types[[param]] - domain <- parameters$domain[[param]] - isLogTransform <- FALSE - if ('transform' %in% names(parameters) && parameters$transform[[param]]=='log') - isLogTransform <- TRUE - - val <- defaultConfiguration[[param]] - if (is.na(val)) val <- domain[1] - - if (type %in% c('r','i')) { - line <- paste0(param, ' [', domain[1], ',',domain[2],'] [', val, ']', - if (type == 'i') 'i' else '', - if (isLogTransform) 'l' else '') - } else { - line <- paste0(param, ' {', paste(domain, collapse = ','), '} [', val, ']') - } - paramlines <- c(paramlines, line) - } - - # param conditions - conditionlines <- c() - for (param in parameters$names) { - # FIXME: This will not work if we byte-compile the conditions - irace.assert(!is.bytecode(parameters$conditions[[param]])) - conditions <- as.character(parameters$conditions[[param]]) - # Convert to SMAC's param format. - if (conditions != "TRUE" && conditions != "FALSE") { - conditions <- gsub("%in%", "in", conditions) # remove % - conditions <- gsub("\"","", conditions) # remove double quote - conditions <- gsub("\'","", conditions) # remove single quote - - # replace c() by {} - conditions <- gsub("c[[:space:]]*\\(([^()]+)\\)", "{\\1}", conditions) - - # remove "(" and ")" at the beginning and end of the condition - conditions <- trimws(conditions) - conditionlines <- c(conditionlines, paste0(param," | ", conditions)) - } - } - lines <- c(paramlines, '\n', '#Conditions:', conditionlines) - # FIXME, TODO: forbidden conditions? - # nguyen: SMAC's parameter space only allows particular forbidden constraints in the format of {parameter_name_1=value_1, ..., parameter_name_N=value_N}, while irace allows more general forbidden constraints. I'm not sure how to deal with it at the moment... - - writeLines(lines, file) -} - -process_experiments <- function(rdata, normalise, tFeatures) -{ - #---- reformat rdata$experiments and add extra information into it ---# - cat(sep="", "Preprocessing experiment data...\n") - experiments <- rdata$experiments - - # instance_id and seed - tInstanceSeeds <- rdata$state$.irace$instancesList - colnames(tInstanceSeeds)[colnames(tInstanceSeeds)=='instance'] <- 'instance_id' - tInstanceSeeds <- cbind(instance_seed_id = 1:nrow(tInstanceSeeds), tInstanceSeeds) - - # add instance names into tInstanceSeeds - instances <- rdata$scenario$instances - tInstances <- data.frame(instance_id = 1:length(instances), instance=instances) - tInstanceSeeds <- merge(tInstanceSeeds, tInstances, by = c('instance_id')) - - # melt experiments - experiments <- cbind(instance_seed_id = 1:nrow(experiments), experiments) - experiments <- na.omit(reshape(as.data.frame(experiments), direction='long', - idvar='instance_seed_id', - times=colnames(experiments)[2:ncol(experiments)], - timevar='candidate_id', - varying=list(colnames(experiments)[2:ncol(experiments)]), - v.names='cost')) - rownames(experiments) <- NULL - - # add instance names and instance_seed_id into experiments - experiments <- merge(experiments, tInstanceSeeds, by='instance_seed_id') - - # normalise cost values if necessary - if (normalise %!in% c('none','instance','feature')) - irace.error("Invalid normalisation method '", normalise,"'") - - if (normalise != 'none' && rdata$scenario$capping) { - irace.warning("normalisation for capped data is not tested and can be buggy.") - } - if (normalise != 'none') { - t1 <- experiments # experiments table with minCost and maxCost for normalisation - - # get minCost and maxCost over each instance - if (normalise == 'instance'){ - tMinCost <- aggregate(cost~instance_id, t1, FUN=min) - colnames(tMinCost)[2] <- 'minCost' - tMaxCost <- aggregate(cost~instance_id, t1, FUN=max) - colnames(tMaxCost)[2] <- 'maxCost' - t1 <- merge(t1, tMinCost, by='instance_id') - t1 <- merge(t1, tMaxCost, by='instance_id') - } else { # get minCost and maxCost over each feature vector (i.e., over many instances with the same feature values) - t2 <- merge(t1, tFeatures, by='instance') - featureNames <- colnames(tFeatures)[-c(1)] - featureVals <- as.list(t2[, featureNames, drop=FALSE]) - tMinCost <- aggregate(t2$cost, by=featureVals, min) - colnames(tMinCost)[colnames(tMinCost)=='x'] <- 'minCost' - tMaxCost <- aggregate(t2$cost, by=featureVals, max) - colnames(tMaxCost)[colnames(tMaxCost)=='x'] <- 'maxCost' - t2 <- merge(t2, tMinCost, by=featureNames) - t2 <- merge(t2, tMaxCost, by=featureNames) - t1 <- t2[, names(t1) %!in% featureNames] - } - - # calculate normalised cost - # TODO: allow user-defined normalisation method - t1$cost <- (t1$cost - t1$minCost) / (t1$maxCost - t1$minCost) - - # remove rows with normalisedCost == Inf - t1 <- t1[is.finite(t1$cost), ] - experiments <- t1[, names(t1) %!in% c('minCost','maxCost')] - } - - # add extra information on iteration index & bound (when irace's capping is enabled) - tLog <- data.frame(rdata$experimentLog) - colnames(tLog)[colnames(tLog)=='instance'] <- 'instance_seed_id' - colnames(tLog)[colnames(tLog)=='configuration'] <- 'candidate_id' - tLog$candidate_id <- as.integer(tLog$candidate_id) - log_columns <- c('instance_seed_id','candidate_id','iteration') - if (('capping' %in% names(rdata$scenario)) && rdata$scenario$capping) { - log_columns <- c(log_columns, 'time', 'bound') - } - experiments <- merge(experiments, tLog[, log_columns], - by=c('instance_seed_id','candidate_id')) - return(experiments) -} - -generate_runhistory <- function(file = "runhistory.json", rdata, experiments) -{ - #------- generate runhistory.json ------# - cat(sep="", "Generating runhistory JSON file '", file, "' ...\n") - - # TODO: If we ever use the json R package for something else, we could use it - # here as well, however, in runhistory.json, values are written with/without - # quotes depending on data types. Therefore, we need to make sure they are - # written correctly. - - # Configurations: The first columns is .ID., and the last column is - # .PARENT. We are extracting parameter names only. - ls_param_names <- setdiff(colnames(rdata$allConfigurations), c(".ID.", ".PARENT.")) - output <- paste0('{', double_quote('configs'), ': {\n') - for (row_id in seq_len(nrow(rdata$allConfigurations))) { - cand <- rdata$allConfigurations[row_id, ] - cand_id <- cand$.ID. - ls_params_vals <- sapply(ls_param_names, function(pname) { - val <- cand[[pname]] - if (is.na(val)) return("") - if (rdata$parameters$types[[pname]] == 'c') - val <- double_quote(val) - return(paste0(double_quote(pname), ': ', val)) - }) - if (row_id > 1) - output <- paste0(output, ',\n') - ls_params_vals <- ls_params_vals[ls_params_vals != ""] - output <- paste0(output, double_quote(cand_id),': {', paste0(ls_params_vals, collapse = ', '),'}') - } - output <- paste0(output, '}') - - # data - output <- paste0(output, ',\n', double_quote('data'), ': [\n') - experiments <- experiments[order(experiments$candidate_id,experiments$instance_seed_id),] # order rows in experiments, so that runhistory.json is always the same across different runs - for (row_id in seq_len(nrow(experiments))) { - row <- experiments[row_id,] - cost <- row$cost - - if (('capping' %in% rdata$scenario) && rdata$scenario$capping) { - time <- row$time - if (row$time >= row$bound){ - if (row$bound < (rdata$scenario$boundMax - 0.01)) - status <- 'StatusType.CAPPED' - else - status <- 'StatusType.TIMEOUT' - } else - status <- 'StatusType.SUCCESS' - - } else { - time <- 0.9 - status <- 'StatusType.SUCCESS' - } - - if (row_id > 1) - output <- paste0(output, ',\n') - #rs = [[conf_id, inst, seed], [cost, time, status, {}]] - s1 <- paste(row$candidate_id, double_quote(row$instance), row$seed, sep=', ') - s2 <- paste(as.character(cost), as.character(time), paste('{',double_quote('__enum__'),': ', double_quote(status),'}'), '{}', sep=', ') - output <- paste0(output,'[[', s1, '], [', s2, ']]') - } - output <- paste0(output,'\n]}\n') - cat(output, file = file) -} - -generate_trajectory <- function(file = 'traj_aclib2.json', rdata, experiments) -{ - #-------------------- traj_aclib2.json ------------------------------# - cat(sep="", "Generating '", file, "' ...\n") - - # Configurations: The first columns is .ID., and the last column is - # .PARENT. We are extracting parameter names only. - ls_param_names <- setdiff(colnames(rdata$allConfigurations), c(".ID.", ".PARENT.")) - output <- "" - for (iterationId in seq_along(rdata$iterationElites)) { - confId <- rdata$iterationElites[iterationId] - # if this elite was eliminated due to filterConditions, ignore it - t1 <- experiments[(experiments$candidate_id == confId) & (experiments$iteration <= iterationId), ] - - # cpu time - cpu_time <- paste0(double_quote('cputime'),': ', confId) - total_cpu_time <- paste0(double_quote('total_cpu_time'),': null') - wallclock_time <- paste0(double_quote('wallclock_time'),': ', confId) - # evaluations - evaluations <- paste0(double_quote('evaluations'),': ', nrow(t1)) - # cost - cost <- paste0(double_quote('cost'),': ', mean(t1$cost)) - # configuration string - cand <- rdata$allConfigurations[confId, ] - ls_params_vals <- sapply(ls_param_names, function(pname) { - val <- cand[[pname]] - if (is.na(val)) return("") - if (rdata$parameters$types[[pname]] == 'c') - val <- single_quote(val) - # FIMXE: if type is 'c', we do single_quote(val) two times? - # nguyen: One single_quote is indeed sufficient! I've updated the next line accordingly :) - return(double_quote(paste0(pname, '=', val))) - }) - ls_params_vals <- ls_params_vals[ls_params_vals != ""] - configuration_string <- paste0(double_quote("incumbent"),': [', - paste0(ls_params_vals, collapse = ', '),']') - # combine everything - output <- paste0(output, '{', - paste(cpu_time, evaluations, cost, configuration_string, total_cpu_time, wallclock_time, sep=', '),'}\n') - } - cat(output, file = file) -} - -generate_scenario_file <- function(file = "scenario.txt", rdata) -{ - #------ create scenario file ------- - cat(sep="", "Generating scenario file '", file, "' ...\n") - lss <- list() - lss[['algo']] <- rdata$scenario$targetRunner - lss[['execDir']] <- './' - if (rdata$scenario$deterministic) - lss[['deterministic']] <- 'true' - else - lss[['deterministic']] <- 'false' - if (rdata$scenario$capping) { - lss[['run_obj']] <- 'runtime' - lss[['cutoff_time']] <- rdata$scenario$boundMax - lss[['overall_obj']] <- paste0('par',rdata$scenario$boundPar) - } else { - lss[['run_obj']] <- 'quality' - lss[['cutoff_time']] <- 1 - } - lss[['tunerTimeout']] <- 999999 - lss[['overall_obj']] <- 'mean' - lss[['paramfile']] <- 'params.pcs' - lss[['instance_file']] <- 'instances.txt' - lss[['feature_file']] <- 'features.csv' - writeLines(paste0(names(lss), "=", lss), file) -} - - -#' Command-line interface to irace2pyimp -#' -#' This is a command-line interface for calling the [`irace2pyimp`] function, -#' which converts an `irace.Rdata` file into the input format supported by the -#' parameter importance analysis tool `PyImp` -#' (https://github.com/automl/ParameterImportance). \cr The best way to use -#' this command line interface is to run the script `irace-to-pyimp`. \cr To -#' see usage of the script, run: irace-to-pyimp --help -#' -#' @param argv (`character()`) \cr Command-line arguments. -#' -#' @return None. -#' -#' @examples -#' -#' irace2pyimp_cmdline("--help") -#' -#' @seealso -#' [`irace2pyimp`] -#' -#' @author Nguyen Dang and Manuel López-Ibáñez -#' @export -#' @md -irace2pyimp_cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) -{ - parser <- CommandArgsParser$new(argv = argv, argsdef = .irace2pyimp_args) - - if (!is.null(parser$readArg(short = "-h", long = "--help"))) { - cat(.irace2pyimp_header) - cmdline_usage(.irace2pyimp_args) - return(invisible(NULL)) - } - argvalues <- lapply(rownames(.irace2pyimp_args), parser$readCmdLineParameter) - names(argv) <- rownames(.irace2pyimp_args) - - irace2pyimp(file = argvalues$iraceRdataFile, normalise = argvalues$normalise, - outdir = argvalues$outDir, - instanceFeatureFile = argvalues$instanceFeatureFile, - filterConditions = argvalues$filterConditions, - defaultConfigurationID = argvalues$defaultConfigurationID, - ignoreUnsupported=argvalues$ignoreUnsupported) -} - -#' Convert an `irace.Rdata` file into the format supported by PyImp -#' -#' This function converts an `irace.Rdata` file generated by irace into the -#' input format supported by the parameter importance analysis tool `PyImp` -#' (https://github.com/automl/ParameterImportance). - -#' -#' The generated files include: -#' -#' * `params.pcs` : a text file containing the parameter space definition. -#' * `runhistory.json` : a JSON file containing the list of algorithm configurations evaluated during the tuning and the performance data obtained. -#' * `traj_aclib2.json` : a JSON file containing the best configurations after each iteration of irace. The last configuration will be used as the target configuration in ablation analysis. -#' * `scenario.txt` : a text file containing the definition of the tuning scenario. -#' * `instances.txt` : a text file containing the list of instances. -#' * `features.csv` : a text file containing instance features. If no instance features are provided, the index of each instance will be used as a feature. -#' -#' @param file (`character(1)`) \cr Filename of the `.Rdata` file generated by -#' irace after a tuning run is finished. -#' -#' @param normalise (`none` | `instance` | `feature`) \cr Normalise the cost metric values into `[0, 1]` range before converting to PyImp format. Possible values are:\cr -#' * `none` (default): no normalisation. \cr -#' * `instance` : normalisation is done per instance. \cr -#' * `feature` : normalisation is based on features, i.e., instances with the same feature-vector values are grouped together and the normalised cost is calculated per group. -#' -#' @param outdir (`character(1)`) \cr Directory where all generated files are stored. -#' @param instanceFeatureFile (`character(1)`) \cr A `.csv` file containing instance features (one line per instance, -#' sorted in the same order as the list of instances input to irace). The first line contains feature names. -#' @param filterConditions Only extract data that satisfies the given conditions. The conditions are in R expression format. -#' @param defaultConfigurationID Index of default configuration (starting from 1), used by ablation analysis. -#' @param ignoreUnsupported Forbidden configurations and repairConfiguration are not supported by the script. Set ignoreUnsupported=1 to ignore them and proceed with your own risk. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis's path. -#' -#' @examples -#' \dontrun{ -#' irace2pyimp(file='irace.Rdata', outdir='pyimp-run') -#' irace2pyimp(file='irace.Rdata', normalise='feature', -#' instanceFeatureFile='feature.csv', filterConditions="algorithm!='mas'") -#' } -#' cat("See more examples in '", -#' file.path(system.file(package="irace"), "examples/irace2pyimp/acotsp/run.sh"), -#' "' and in '", -#' file.path(system.file(package="irace"), "examples/irace2pyimp/002-TemplateDesign/run.sh"), -#' "'\n") -#' -#' @author Nguyen Dang and Manuel López-Ibáñez -#' @export -#' @md -irace2pyimp <- function(file='./irace.Rdata', normalise='none', outdir='./pyimp-input/', - instanceFeatureFile=NA, filterConditions=NA, defaultConfigurationID=1, ignoreUnsupported=0) -{ - if (str_sub(outdir, start=-1) != "/") outdir <- paste0(outdir, "/") - # create output dir if it doesn't exist - if (!file.exists(outdir)) dir.create(outdir, recursive = TRUE) - - rdata <- load_irace_rdata(file) - - # check forbidden constraint and repairConfiguration - if ((ignoreUnsupported==0) && (!is.null.or.empty(rdata$scenario$forbiddenExps) || !is.null.or.empty(rdata$scenario$repairConfiguration))){ - irace.error("Forbidden constraints and/or repairConfiguration are not yet supported. To ignore them and proceed with your own risk, please set '--ignore-unsupported 1' (command line), or 'ignoreUnsupported=1' (R console) while calling irace2pyimp. This may cause some unwanted behaviours, e.g., forbidden configurations may appear in ablation analysis's path.") - } - # print warning about forbidden constraint and repairConfiguration before proceeding - if (!is.null.or.empty(rdata$scenario$forbiddenExps) || !is.null.or.empty(rdata$scenario$repairConfiguration)){ - irace.warning("Forbidden constraints and repairConfiguration are ignored. Proceeding with your own risk...") - } - - ops <- options(scipen=999) # Disable scientific notation - on.exit(options(ops)) - - # filter data - if (!is.na(filterConditions) && trimws(filterConditions) != '') { - rdata <- filter_data(rdata = rdata, conditions = filterConditions, - defaultConfigurationID = defaultConfigurationID) - defaultConfigurationID <- rdata$defaultConfigurationID - } - - # Generate instances.txt and features.csv - #--- instances.txt ---- - cat(sep="", "Generating instance list file '", paste0(outdir, 'instances.txt'), "' ...\n") - writeLines(rdata$scenario$instances, paste0(outdir, 'instances.txt')) - - tFeatures <- generate_feature_file(file = paste0(outdir, 'features.csv'), - instances = rdata$scenario$instances, - instanceFeatureFile = instanceFeatureFile) - - # remove fixed parameters - rdata <- remove_fixed_parameters(rdata) - - # generate params.pcs - generate_pcs_file(file = paste0(outdir, 'params.pcs'), - parameters = rdata$parameters, - defaultConfiguration = rdata$allConfigurations[defaultConfigurationID,]) - - # generate runhistory.json and traj_aclib2.json - experiments <- process_experiments(rdata, normalise, tFeatures) - - generate_runhistory(file = paste0(outdir, 'runhistory.json'), rdata = rdata, experiments = experiments) - generate_trajectory(file = paste0(outdir, 'traj_aclib2.json'), rdata = rdata, experiments = experiments) - - # generate scenario.txt - generate_scenario_file(file = paste0(outdir, "scenario.txt"), rdata = rdata) -} - diff --git a/R/main.R b/R/main.R index 1e6a811..a466aa4 100644 --- a/R/main.R +++ b/R/main.R @@ -53,227 +53,210 @@ # Copyright (C) 2003 Mauro Birattari #------------------------------------------------------------------------------ ' -cat.irace.license <- function() +cat_irace_license <- function() { cat(sub("__VERSION__", irace.version, irace.license, fixed=TRUE)) } -cmdline_usage <- function(cmdline_args) -{ - for (i in seq_len(nrow(cmdline_args))) { - short <- cmdline_args[i,"short"] - long <- cmdline_args[i,"long"] - desc <- cmdline_args[i,"description"] - if (desc == "" || (short == "" && long == "")) next - cat(sep = "\n", strwrap(desc, width = 80, - initial = sprintf("%2s %-20s ", short, long), - exdent = 25)) - } -} - -#' irace.usage -#' -#' \code{irace.usage} This function prints all command-line options of \pkg{irace}, -#' with the corresponding switches and a short description. -#' -#' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @export -irace.usage <- function () -{ - cat.irace.license() - cat ("# installed at: ", system.file(package="irace"), "\n", sep = "") - cmdline_usage(.irace.params.def) -} - -#' irace.main -#' -#' \code{irace.main} is a higher-level interface to invoke \code{\link{irace}}. -#' +#' Higher-level interface to launch irace. +#' #' @template arg_scenario #' -#' @param output.width (\code{integer(1)}) The width that must be used for the screen -#' output. -#' -#' @details The function \code{irace.main} checks the correctness of the -#' scenario, prints it, reads the parameter space from -#' \code{scenario$parameterFile}, invokes \code{\link{irace}} and -#' prints its results in various formatted ways. If you want a -#' lower-level interface, please see function \code{\link{irace}}. +#' @param output.width (\code{integer(1)}) The width used for the screen +#' output. +#' +#' @details This function checks the correctness of the scenario, reads the +#' parameter space from \code{scenario$parameterFile}, invokes [irace()], +#' prints its results in various formatted ways, (optionally) calls +#' [psRace()] and, finally, evaluates the best configurations on the test +#' instances (if provided). If you want a lower-level interface that just +#' runs irace, please see function [irace()]. #' #' @templateVar return_invisible TRUE #' @template return_irace #' @seealso -#' \code{\link{irace.cmdline}} a higher-level command-line interface to -#' \code{irace.main}. -#' \code{\link{readScenario}} to read the scenario setup from a file. -#' \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. +#' [irace.cmdline()] a higher-level command-line interface to +#' [irace()] +#' [readScenario()] to read the scenario setup from a file. +#' [defaultScenario()] to provide a default scenario for \pkg{irace}. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @export -irace.main <- function(scenario = defaultScenario(), output.width = 9999L) -{ - op <- options(width = output.width) # Do not wrap the output. - on.exit(options(op), add = TRUE) - - scenario <- checkScenario (scenario) - debug.level <- scenario$debugLevel - - if (debug.level >= 1) { - op.debug <- options(warning.length = 8170, - error = if (interactive()) utils::recover - else irace.dump.frames) - on.exit(options(op.debug), add = TRUE) - printScenario (scenario) - } - - # Read parameters definition - parameters <- readParameters (file = scenario$parameterFile, - digits = scenario$digits, - debugLevel = debug.level) - - if (debug.level >= 2) { irace.note("Parameters have been read\n") } - - eliteConfigurations <- irace (scenario = scenario, parameters = parameters) - - cat("# Best configurations (first number is the configuration ID;", - " listed from best to worst according to the ", - test.type.order.str(scenario$testType), "):\n", sep = "") - configurations.print(eliteConfigurations) - - cat("# Best configurations as commandlines (first number is the configuration ID; same order as above):\n") - configurations.print.command (eliteConfigurations, parameters) - - if (scenario$postselection > 0) - psRace(iraceLogFile=scenario$logFile, postselection=scenario$postselection, elites=TRUE) - - if (length(eliteConfigurations) > 0 && - (scenario$testIterationElites != 0 || scenario$testNbElites != 0)) - testing.main(logFile = scenario$logFile) - - invisible(eliteConfigurations) -} - -#' testing.main -#' -#' \code{testing.main} executes the testing of the target -#' algorithm configurations found on an \pkg{irace} execution. -#' -#' @param logFile Path to the \code{.Rdata} file produced by \pkg{irace}. -#' -#' @return Boolean. TRUE if the testing ended successfully otherwise, returns -#' FALSE. -#' -#' @details The function \code{testing.main} loads the \code{logFile} and -#' obtains the needed configurations according to the specified test. Use the -#' \code{scenario$testNbElites} to test N final elite configurations or use -#' \code{scenario$testIterationElites} to test the best configuration of each -#' iteration. A test instance set must be provided through -#' \code{scenario$testInstancesDir} and \code{testInstancesFile}. -#' -#' @seealso -#' \code{\link{defaultScenario}} to provide a default scenario for \pkg{irace}. +#' @concept running +#' @export +irace.main <- function(scenario, output.width = 9999L) + irace_common(scenario = scenario, simple=FALSE, output.width = output.width) + +#' Test configurations given in `.Rdata` file +#' +#' `testing_fromlog` executes the testing of the target algorithm configurations +#' found by an \pkg{irace} execution. +#' +#' @param logFile Path to the `.Rdata` file produced by \pkg{irace}. +#' +#' @param testNbElites Number of (final) elite configurations to test. Overrides +#' the value found in `logFile`. +#' +#' @param testIterationElites (`logical(1)`) If `FALSE`, only the final +#' `testNbElites` configurations are tested; otherwise, also test the best +#' configurations of each iteration. Overrides the value found in `logFile`. +#' +#' @param testInstancesDir Directory where testing instances are located, either absolute or relative to current directory. +#' +#' @param testInstancesFile File containing a list of test instances and optionally additional parameters for them. +#' +#' @param testInstances Character vector of the instances to be used in the `targetRunner` when executing the testing. +#' +#' @return Boolean. `TRUE` if the testing ended successfully otherwise, `FALSE`. +#' +#' @details The function `testing_fromlog` loads the `logFile` and obtains the +#' testing setup and configurations to be tested. Within the `logFile`, the +#' variable `scenario$testNbElites` specifies how many final elite +#' configurations to test and `scenario$testIterationElites` indicates +#' whether test the best configuration of each iteration. The values may be +#' overridden by setting the corresponding arguments in this function. The +#' set of testing instances must appear in `scenario[["testInstances"]]`. +#' +#' @seealso [defaultScenario()] to provide a default scenario for \pkg{irace}. +#' [testing_fromfile()] provides a different interface for testing. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres -#' @export -testing.main <- function(logFile) +#' @concept running +#' @export +testing_fromlog <- function(logFile, testNbElites, testIterationElites, + testInstancesDir, testInstancesFile, testInstances) { if (is.null.or.empty(logFile)) { irace.note("No logFile provided to perform the testing of configurations. Skipping testing.\n") return(FALSE) } + iraceResults <- read_logfile(logFile) + scenario <- iraceResults[["scenario"]] + parameters <- iraceResults[["parameters"]] + instances_changed <- FALSE - file.check(logFile, readable = TRUE, text = "irace log file") - - load (logFile) - scenario <- iraceResults$scenario - parameters <- iraceResults$parameters - - if (is.null.or.empty(scenario$testInstances)) { + if (!missing(testNbElites)) + scenario[["testNbElites"]] <- testNbElites + if (!missing(testIterationElites)) + scenario$testIterationElites <- testIterationElites + + if (!missing(testInstances)) + scenario[["testInstances"]] <- testInstances + + if (!missing(testInstancesDir)) { + scenario$testInstancesDir <- testInstancesDir + instances_changed <- TRUE + } + if (!missing(testInstancesFile)) { + scenario$testInstancesFile <- testInstancesFile + instances_changed <- TRUE + } + + cat("\n\n# Testing of elite configurations:", scenario$testNbElites, + "\n# Testing iteration configurations:", scenario$testIterationElites,"\n") + if (scenario$testNbElites <= 0) return (FALSE) + + # If they are already setup, don't change them. + if (instances_changed || is.null.or.empty(scenario[["testInstances"]])) { + scenario <- setup_test_instances(scenario) + if (is.null.or.empty(scenario[["testInstances"]])) { + irace.note("No test instances, skip testing\n") + return(FALSE) + } } - # Get configurations - testing.id <- c() + # Get configurations that will be tested if (scenario$testIterationElites) - testing.id <- c(testing.id, iraceResults$iterationElites) - if (scenario$testNbElites > 0) { + testing_id <- sapply(iraceResults$allElites, function(x) + x[1:min(length(x), scenario$testNbElites)]) + else { tmp <- iraceResults$allElites[[length(iraceResults$allElites)]] - testing.id <- c(testing.id, tmp[1:min(length(tmp), scenario$testNbElites)]) - } - testing.id <- unique(testing.id) - configurations <- iraceResults$allConfigurations[testing.id, , drop=FALSE] - - cat(" \n\n") - irace.note ("Testing configurations (in no particular order): ", paste(testing.id, collapse=" "), "\n") - configurations.print(configurations) - cat("# Testing of elite configurations:", scenario$testNbElites, - "\n# Testing iteration configurations:", scenario$testIterationElites,"\n") - - iraceResults$testing <- testConfigurations(configurations, scenario, parameters) - - # FIXME : We should print the seeds also. As an additional column? - irace.note ("Testing results (column number is configuration ID in no particular order):\n") - print(iraceResults$testing$experiments) - irace_save_logfile (iraceResults, scenario) - - irace.note ("Finished testing\n") + testing_id <- tmp[1:min(length(tmp), scenario$testNbElites)] + } + testing_id <- unique.default(unlist(testing_id)) + configurations <- iraceResults$allConfigurations[testing_id, , drop=FALSE] + + irace.note ("Testing configurations (in no particular order): ", paste(testing_id, collapse=" "), "\n") + testing_common(configurations, scenario, parameters, iraceResults) return(TRUE) } -testing.cmdline <- function(filename, scenario) +#' Test configurations given an explicit table of configurations and a scenario file +#' +#' Executes the testing of an explicit list of configurations given in +#' `filename` (same format as in [readConfigurationsFile()]). A `logFile` is +#' created unless disabled in `scenario`. This may overwrite an existing one! +#' +#' @param filename Path to a file containing configurations: one configuration +#' per line, one parameter per column, parameter names in header. +#' +#' @template arg_scenario +#' +#' @return iraceResults +#' +#' @seealso [testing_fromlog()] provides a different interface for testing. +#' +#' @author Manuel López-Ibáñez +#' @concept running +#' @export +testing_fromfile <- function(filename, scenario) { irace.note ("Checking scenario\n") scenario <- checkScenario(scenario) - printScenario(scenario) + if (!scenario$quiet) printScenario(scenario) irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") parameters <- readParameters (file = scenario$parameterFile, digits = scenario$digits) - allConfigurations <- readConfigurationsFile (filename, parameters) - allConfigurations <- cbind(.ID. = 1:nrow(allConfigurations), - allConfigurations, + configurations <- readConfigurationsFile (filename, parameters) + configurations <- cbind(.ID. = 1:nrow(configurations), + configurations, .PARENT. = NA) - rownames(allConfigurations) <- allConfigurations$.ID. - num <- nrow(allConfigurations) - allConfigurations <- checkForbidden(allConfigurations, scenario$forbiddenExps) - if (nrow(allConfigurations) < num) { - cat("# Warning: some of the configurations in the configurations file were forbidden", - "and, thus, discarded\n") - } - + rownames(configurations) <- configurations$.ID. + num <- nrow(configurations) + configurations <- checkForbidden(configurations, scenario$forbiddenExps) + if (nrow(configurations) < num) { + irace.warning("Some of the configurations in the configurations file were forbidden", + "and, thus, discarded") + } # To save the logs iraceResults <- list(scenario = scenario, irace.version = irace.version, parameters = parameters, - allConfigurations = allConfigurations) + allConfigurations = configurations) irace.note ("Testing configurations (in the order given as input): \n") - configurations.print(allConfigurations) - iraceResults$testing <- testConfigurations(allConfigurations, scenario, parameters) - + iraceResults <- testing_common(configurations, scenario, parameters, iraceResults) + return(iraceResults) +} + +testing_common <- function(configurations, scenario, parameters, iraceResults) +{ + verbose <- !scenario$quiet + if (verbose) configurations.print(configurations) + iraceResults$testing <- testConfigurations(configurations, scenario, parameters) + irace_save_logfile (iraceResults, scenario) # FIXME : We should print the seeds also. As an additional column? irace.note ("Testing results (column number is configuration ID in no particular order):\n") - print(iraceResults$testing$experiments) - irace_save_logfile(iraceResults, scenario) + if (verbose) print(iraceResults$testing$experiments) irace.note ("Finished testing\n") return(iraceResults) } #' Test that the given irace scenario can be run. #' -#' @description \code{checkIraceScenario} tests that the given irace scenario -#' can be run by checking the scenario settings provided and trying to run -#' the target-algorithm. +#' Test that the given irace scenario can be run by checking the scenario +#' settings provided and trying to run the target-algorithm. #' #' @template arg_scenario #' @template arg_parameters #' -#' @return returns \code{TRUE} if succesful and gives an error and returns +#' @return returns \code{TRUE} if successful and gives an error and returns #' \code{FALSE} otherwise. #' -#' @details Provide the \code{parameters} argument only if the parameter list -#' should not be obtained from the parameter file given by the scenario. If -#' the parameter list is provided it will not be checked. This function will +#' @details If the `parameters` argument is missing, then the parameters +#' will be read from the file `parameterFile` given by `scenario`. If +#' `parameters` is provided, then `parameterFile` will not be read. This function will #' try to execute the target-algorithm. #' #' @seealso @@ -286,26 +269,27 @@ #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export -checkIraceScenario <- function(scenario, parameters = NULL) +checkIraceScenario <- function(scenario, parameters) { irace.note ("Checking scenario\n") scenario$debugLevel <- 2 scenario <- checkScenario(scenario) - printScenario(scenario) + if (!scenario$quiet) printScenario(scenario) - if (is.null(parameters)) { + if (missing(parameters)) { irace.note("Reading parameter file '", scenario$parameterFile, "'.\n") parameters <- readParameters (file = scenario$parameterFile, digits = scenario$digits, debugLevel = 2) } else if (!is.null.or.empty(scenario$parameterFile)) { - cat("# Parameters provided by user.\n", - "# Parameter file '", scenario$parameterFile, "' will be ignored\n", sep = "") - } - - irace.note("Checking target execution.\n") + if (!scenario$quiet) + cat("# checkIraceScenario(): 'parameters' provided by user. ", + "Parameter file '", scenario$parameterFile, "' will be ignored\n", sep = "") + } + checkParameters(parameters) + irace.note("Checking target runner.\n") if (checkTargetFiles(scenario = scenario, parameters = parameters)) { - irace.note("Check succesful.\n") + irace.note("Check successful.\n") return(TRUE) } else { irace.error("Check unsuccessful.\n") @@ -313,11 +297,28 @@ } } - -#' irace.cmdline -#' -#' \code{irace.cmdline} starts \pkg{irace} using the parameters -#' of the command line used to invoke R. +init <- function() +{ + irace.note("Initializing working directory...\n") + libPath <- system.file(package = "irace") + tmplFiles <- list.files(file.path(libPath, "templates")) + for (file in tmplFiles) { + if (grepl(".tmpl", file) && (file != "target-evaluator.tmpl")) { + newFile <- gsub(".tmpl", "", file) + if ((file == "target-runner.tmpl") && .Platform$OS.type == 'windows') { + file.copy(file.path(libPath, "templates", "windows", "target-runner.bat"), file.path(getwd(), "target-runner.bat"), overwrite = FALSE) + } else { + file.copy(file.path(libPath, "templates", file), file.path(getwd(), newFile), overwrite = FALSE) + } + } + } +} + + +#' Launch `irace` with command-line options. +#' +#' Calls [irace.main()] using command-line options, maybe parsed from the +#' command line used to invoke R. #' #' @param argv (\code{character()}) \cr The arguments #' provided on the R command line as a character vector, e.g., @@ -328,35 +329,50 @@ #' @details The function reads the parameters given on the command line #' used to invoke R, finds the name of the scenario file, #' initializes the scenario from the file (with the function -#' \code{\link{readScenario}}) and possibly from parameters passed on +#' \code{\link{readScenario}}) and possibly from parameters passed in #' the command line. It finally starts \pkg{irace} by calling #' \code{\link{irace.main}}. #' +#' List of command-line options: +#' ```{r echo=FALSE,comment=NA} +#' cmdline_usage(.irace.params.def) +#' ``` +#' #' @templateVar return_invisible TRUE #' @template return_irace #' #' @seealso -#' \code{\link{irace.main}} to start \pkg{irace} with a given scenario. -#' +#' [irace.main()] to start \pkg{irace} with a given scenario. +#' @examples +#' irace.cmdline("--version") #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @export -irace.cmdline <- function(argv = commandArgs (trailingOnly = TRUE)) +#' @concept running +#' @export +irace.cmdline <- function(argv = commandArgs(trailingOnly = TRUE)) { parser <- CommandArgsParser$new(argv = argv, argsdef = .irace.params.def) + quiet <- !is.null(parser$readArg (short = "-q", long = "--quiet")) + if (quiet) { + op <- options(.irace.quiet = TRUE) + on.exit(options(op)) + } else { + cat_irace_license() + cat("# installed at: ", system.file(package="irace"), "\n", + "# called with: ", paste(argv, collapse = " "), "\n", sep = "") + } if (!is.null(parser$readArg (short = "-h", long = "--help"))) { - irace.usage() + parser$cmdline_usage() return(invisible(NULL)) } - - if (!is.null(parser$readArg (short = "-v", long = "--version"))) { - cat.irace.license() - cat ("# installed at: ", system.file(package="irace"), "\n", sep = "") + if (!is.null(parser$readArg(short = "-v", long = "--version"))) { print(citation(package="irace")) return(invisible(NULL)) } - cat.irace.license() - cat ("# installed at: ", system.file(package="irace"), "\n", - "# called with: ", paste(argv, collapse = " "), "\n", sep = "") + + if (!is.null(parser$readArg(short = "-i", long = "--init"))) { + init() + return(invisible(NULL)) + } # Read the scenario file and the command line scenarioFile <- parser$readCmdLineParameter ("scenarioFile", default = "") @@ -366,6 +382,7 @@ parser$readCmdLineParameter(paramName = param, default = scenario[[param]]) } + if (quiet) scenario$quiet <- TRUE # Check scenario if (!is.null(parser$readArg (short = "-c", long = "--check"))) { @@ -376,12 +393,12 @@ # Only do testing testFile <- parser$readArg (long = "--only-test") if (!is.null(testFile)) { - return(invisible(testing.cmdline(testFile, scenario))) + return(invisible(testing_fromfile(testFile, scenario))) } if (length(parser$argv) > 0) { irace.error ("Unknown command-line options: ", paste(parser$argv, collapse = " ")) } - - irace.main(scenario) -} + + irace_common(scenario = scenario, simple=FALSE) +} diff --git a/R/model.R b/R/model.R index 3458ae8..cbf0dab 100644 --- a/R/model.R +++ b/R/model.R @@ -26,6 +26,7 @@ irace.assert(type == "o") value <- (nbValues - 1) / 2 } + param <- list() for (indexConfig in seq_len(nbConfigurations)) { idCurrentConfig <- as.character(configurations[indexConfig, ".ID."]) @@ -153,9 +154,9 @@ model[[param]][[id]] <- probVector / sum(probVector) } else { if (type == "i" || type == "r") { - value <- init.model.numeric(param, parameters) - # We keep the value of the configuration as last known - value[2] <- configurations[id, param] + value <- c(init.model.numeric(param, parameters), + # We keep the value of the configuration as last known + configurations[id, param]) } else { irace.assert(type == "o") value <- (length(parameters$domain[[param]]) - 1) / 2 @@ -171,18 +172,25 @@ } # Initialise model in case of numerical variables. -# it retuns an array size 2, first number indicates the +# it retuns an array size 2, first number indicates the # standard deviation and second the last known value (initially NA) init.model.numeric <- function(param, parameters) { - lower <- paramLowerBound(param, parameters) - upper <- paramUpperBound(param, parameters) + # Dependent parameters define the standard deviation as + # a portion of the size of the domain interval. In this case, + # 0.5 indicates half of the interval, equivalent to + # (domain[2] - domain[1]) * 0.5 + if (parameters$isDependent[[param]]) { + return(0.5) + } + transf <- parameters$transform[[param]] if (transf == "log") { - lower <- 0 - upper <- 1 + domain <- c(0,1) + } else { + domain <- parameters$domain[[param]] } - value <- (upper - lower) / 2.0 + value <- (domain[2] - domain[1]) / 2.0 irace.assert(is.finite(value)) - return(c(value, NA)) + return(value) } diff --git a/R/parameterAnalysis.R b/R/parameterAnalysis.R index 691c2dc..2d7bfbd 100644 --- a/R/parameterAnalysis.R +++ b/R/parameterAnalysis.R @@ -1,363 +1,33 @@ -#' Plot of histogram of parameter values -#' -#' \code{parameterFrequency} plots the frequency of the parameters values in a -#' set of target algorithm configurations. It generates plots showing the -#' frequency of parameter values for each parameter, with \code{rows} * -#' \code{cols} parameters being shown per plot. If a filename is provided the -#' plots are saved in one or more files. -#' -#' @template arg_configurations -#' @template arg_parameters -#' @param rows Number of plots per column. -#' @param cols Number of plots per row. -#' @param filename Filename prefix to generate the plots. If \code{NULL} the plot -#' displayed but not saved. -#' @param pdf.width Width for the pdf file generated. -#' @param col Color of the bar plot. -#' -#' @examples -#' \donttest{ -#' ## To use data obtained by irace -#' -#' # First, load the data produced by irace. -#' irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") -#' load(irace.logfile) -#' attach(iraceResults) -#' parameterFrequency(allConfigurations, parameters) -#' } -#' -#' @seealso -#' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. -#' \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from -#' a configurations file. -#' -#' @author Manuel López-Ibáñez and Leslie Pérez Cáceres -#' @export -# TODO: -# * change slightly background of conditional parameters -# -# * accept a configuration (e.g, best one) and mark its values of the best -# configuration somehow (bold?) -# -# * print number of NAs in numerical parameters within the plot "# NA = Total -# (Percentage%)" -# -# * export this function and add an R manual page, perhaps with example. -# -# * how to resize the Window when filename == NULL to have a rows/cols aspect ratio. -# -# * use ggplot2 ? -# -# * add tests! -#parameterFrequency(iraceResults$allConfigurations, iraceResults$parameters) -#best <- colnames(experiments)[which(apply(experiments, 2, function(x) { sum(!is.na(x))}) > 5)] -#parameterFrequency(allConfigurations[best,], parameters) -parameterFrequency <- function(configurations, parameters, - rows = 4, cols = 3, - filename = NULL, - pdf.width = 12, - col = "gray") -{ - xlab <- "values" - ylab.cat <- "Frequency" - ylab.num <- "Probability density" - - def.par <- par(no.readonly = TRUE) # save default, for resetting... - on.exit(par(def.par), add = TRUE) #- reset to default - - configurations <- removeConfigurationsMetaData(configurations) - - param.names <- as.character(parameters$names) - nparams <- parameters$nbVariable - nlines <- ceiling(nparams / cols) - if (nlines < rows) rows <- nlines - - nplot <- 1 - cplot <- 1 - if (!is.null(filename)) { - # Remove possible ".pdf" extension. - filename <- sub(".pdf", "", filename, fixed = TRUE) - pdf(file = paste0(filename, "-", cplot, ".pdf"), onefile = TRUE, width = pdf.width) - on.exit(dev.off(), add = TRUE) - } - par(mfrow=c(rows,cols), mar=0.1 + c(4,3,3,1)) - - for (param.name in param.names) { - - if (parameters$isFixed[[param.name]]){ - cat("Skipping fixed parameter:", param.name, "\n") - next - } else { - cat("Plotting:", param.name, "\n") - } - - if (nplot > rows * cols) { - cat("Make new plot\n") - cplot <- cplot + 1 - if (!is.null(filename)) { - dev.off() - pdf(file = paste0(filename, "-", cplot, ".pdf"), onefile = TRUE, width = pdf.width) - } else { - dev.new() - } - par(mfrow=c(rows, cols)) - nplot <- 1 - } - - data <- configurations[, param.name] - type <- parameters$types[[param.name]] - domain <- parameters$domain[[param.name]] - if (type %in% c("c", "o")) { - data <- factor(data, domain) - data <- addNA(data, ifany = TRUE) - levels(data)[is.na(levels(data))] <- "" - data <- table(data) - barplot(data, main = param.name, xlab = xlab, ylab = ylab.cat, col = col) - } else if (type %in% c("i", "r")) { - data <- data[!is.na(data)] - if (length(data) == 0) { - cat("All values are NA for: ", param.name, ", skipping plot\n") - next - } else { - hist(data, xlim = domain, prob = TRUE, - main = param.name, xlab = xlab, ylab = ylab.num, col = col) - if (length(data) > 1) { - lines(density(data), col = "blue", lwd = 2) - } else { - abline(v = data[1], col = "blue", lwd = 2) - } - } - } - nplot <- nplot + 1 - } -} - -# Function parcoordlabel plots parallel coordinates for categorical and -# numerical configurations. Idea from -# http://stackoverflow.com/questions/23553210/add-ticks-to-parcoord-parallel-coordinates-plot -parcoordlabel <- function (configurations, parameters, col = "green", lty = 1, - lblcol="blue", title="Parameters parallel coordinates", ...) -{ - replace.cat <- function(y, vals){ - x <- rep(NA, length(y)) - for(i in 1:length(vals)) - x[y %in% vals[i]] <- i - return(x) - } - - replace.na <- function(x,r) { - x <- unlist(x) - x <- (x-r[1])/(r[2]-r[1]) - x[is.na(x)] <- 1 - return(x) - } - - add.level <- function(x, bound, type){ - if (type == "i" || type == "r"){ - x <- c(x, x[length(x)] + (x[2]-x[1])) - } else { - x <- c(x, x[length(x)] + x[length(x)]/length(bound)) - } - return(x) - } - - param.names <- colnames(configurations) - bound.num <- parameters$domain - # Categorical -> Numerical - for (i in 1:ncol(configurations)) { - pname <- param.names[i] - if (parameters$types[[pname]] %in% c("c","o")) { - configurations[,i]<- as.numeric(replace.cat(configurations[,i], bound.num[[pname]])) - bound.num[[pname]] <- seq(1,length(bound.num[[pname]])) - } - } - - # Intervals - pr <- lapply(bound.num[param.names], pretty) - # add extra level for NA values - for(param in param.names) - pr[[param]] <- add.level( pr[[param]], bound.num[[param]], parameters$types[[param]]) - # Total range - rx <- lapply(pr, range, na.rm = TRUE) - - # Values x plot - configurations <- mapply(replace.na, as.data.frame(configurations), rx) - - matplot(1:ncol(configurations), t(configurations), type = "l", col = col, - lty = lty, xlab = "", ylab = "", ylim=c(0,1), axes = FALSE, main=title, ...) - - axis(1, at = 1:ncol(configurations), labels = colnames(configurations), las=2) - - for (i in 1:ncol(configurations)) { - pnames <- param.names[i] - lines(c(i, i), c(0, 1), col = "grey70") - if(parameters$types[[param.names[i]]]=="c"){ - labels <- c(parameters$domain[[param.names[i]]], "NA") - }else{ - labels <- pr[[pnames]] - labels[length(labels)] <- "" - } - text(c(i, i), seq(0,1,length.out=length(labels)), labels = labels, xpd = NA, col=lblcol) - } - invisible() -} - -#' parallelCoordinatesPlot -#' -#' \code{parallelCoordinatesPlot} plots a set of parameter configurations in -#' parallel coordinates. -#' -#' @template arg_configurations -#' @template arg_parameters -#' @param param_names Parameters names that should be included. Default: parameters$names. -#' @param hierarchy If \code{TRUE} conditional parameters will be displayed in a different -#' plot. Default \code{TRUE}. -#' @param filename Filename prefix to generate the plots. If \code{NULL} the plot -#' displayed but not saved. -#' @param pdf.width Width for the pdf file generated. -#' @param mar Margin to use for the plot. See \code{\link{par}}. -#' -#' @return A set of parallel coordinates plots showing the parameters values. -#' If a filename is provided this plots are saved in one or more files. -#' -#' @examples -#' \donttest{ -#' ## To use data obtained by irace -#' # First, load the data produced by irace. -#' irace.logfile <- file.path(system.file(package="irace"), "exdata", "irace-acotsp.Rdata") -#' load(irace.logfile) -#' attach(iraceResults) -#' parallelCoordinatesPlot(allConfigurations, parameters, hierarchy = FALSE) -#' } -#' -#' @seealso -#' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters file. -#' \code{\link{readConfigurationsFile}} to obtain a set of target algorithm configurations from -#' a configurations file. -#' -#' @author Manuel López-Ibáñez and Leslie Pérez Cáceres -#' @export -# TODO: -# * add color scheme -# -## Filters and prepare data to plots parallel coordinate of configurations -## configurations: structure as in iraceResults -## parameters: structure as in iraceResults -## param_names: selection of parameters names that must be plotted -## hierarchy: separate plots according to dependencies of parameters (one level) -## mar: margin for the plots -## filename: prefix to save pdf files -parallelCoordinatesPlot <- - function(configurations, parameters, param_names=parameters$names, - hierarchy = TRUE, filename = NULL, pdf.width = 14 , mar = c(8,1,4,1)) -{ - getDependency <- function() { - sdep <- list() - independent <- c() - for (param in param_names) { - ## FIXME: If we ever byte-compile conditions, we need to use - ## all.vars(.Internal(disassemble(condition))[[3]][[1]] - constraint <- all.vars(parameters$conditions[[param]]) - if (length(constraint) < 1) { - independent <- unique(c(independent, param)) - next - } - - for(cc in constraint) { - if(is.null(sdep[[cc]]) || is.na(sdep[[cc]])) - sdep[[cc]] <- param - else - sdep[[cc]] <- c(sdep[[cc]], param) - } - } - return(list(independent=independent, dependencies=sdep)) - } - - - configurations <- configurations[, grep("^\\.", colnames(configurations), invert=TRUE), drop = FALSE] - configurations <- configurations[, param_names] - - if(hierarchy){ - aux <- getDependency() - if(length(aux$independent) >= 2){ - indconfigurations <- configurations[, aux$independent] - if(!is.null(filename)) - pdf(file = paste0(filename,"_independent.pdf"), width=pdf.width) - else dev.new() - par(mar=mar) - parcoordlabel(indconfigurations, parameters, title="Independent parameter parallel coordinates") - if(!is.null(filename)) dev.off() - }else{ - cat("Skipping independent parameters",aux$independent,"\n") - } - - dep <- aux$dependencies - dnames <- names(dep) - for(i in seq_along(dep)){ - if(length(dep[[i]]) < 2){ - cat("Skipping parameters",dnames[i],"\n") - next - } - depconfigurations <- configurations[, dep[[i]]] - depconfigurations <- depconfigurations[!apply(apply(depconfigurations, 1, is.na), 2, all),] - - if(nrow(depconfigurations) < 2 ){ - cat("Skipping parameter",dnames[i],"\n") - next - } - if(!is.null(filename)) - pdf(file = paste0(filename,"_",dnames[i],".pdf"), width=pdf.width) - else dev.new() - par(mar=mar) - parcoordlabel(depconfigurations, parameters, - title = paste0("Parameters of dependent of ", dnames[i], " parallel coordinates")) - if (!is.null(filename)) dev.off() - } - - }else{ - if(!is.null(filename)) - pdf(file = paste0(filename,".pdf"), width=pdf.width) - #else dev.new() - par(mar=mar) - parcoordlabel(configurations, parameters) - if(!is.null(filename)) dev.off() - } - -} - #' Return the elite configurations of the final iteration. #' -#' @param iraceResults Object created by \pkg{irace} and saved in \code{scenario$logFile}. -#' @param logFile Log file created by \pkg{irace}, this file must contain the -#' \code{iraceResults} object. +#' @template arg_iraceresults #' @param n Number of elite configurations to return, if \code{n} is larger than the -#' number of configurations, then only the existing ones are returned. +#' number of configurations, then only the existing ones are returned. The default (\code{n=0}) returns all of them. #' @param drop.metadata Remove metadata, such the configuration ID and #' the ID of the parent, from the returned configurations. See #' \code{\link{removeConfigurationsMetaData}}. #' #' @return A data frame containing the elite configurations required. #' +#' @examples +#' log_file <- system.file("exdata/irace-acotsp.Rdata", package="irace", mustWork=TRUE) +#' print(removeConfigurationsMetaData(getFinalElites(log_file, n=1))) #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres +#' @concept analysis #' @export -getFinalElites <- function(iraceResults = NULL, logFile = NULL, n = 0, - drop.metadata = FALSE) +getFinalElites <- function(iraceResults, n = 0L, drop.metadata = FALSE) { - if (is.null(iraceResults)) { - if (is.null(logFile)) - stop("You must supply either 'iraceResults' or 'logFile' argument.\n") - else - load(logFile) - } - + if (missing(iraceResults)) stop("argument 'iraceResults' is missing") + iraceResults <- read_logfile(iraceResults) + last.elites <- iraceResults$allElites[[length(iraceResults$allElites)]] if (n == 0) n <- length(last.elites) if (length(last.elites) < n) { - cat("Only", length(last.elites), "configurations available, reducing n,\n") + cat("Only", length(last.elites), "configurations available, reducing n,") n <- length(last.elites) } last.elites <- last.elites[1:n] @@ -372,41 +42,27 @@ #' Returns the configurations selected by ID. #' -#' @param iraceResults Object created by \pkg{irace} and saved in \code{scenario$logFile}. -#' @param logFile Log file created by \pkg{irace}, this file must contain the -#' \code{iraceResults} object. +#' @template arg_iraceresults #' @param ids The id or a vector of ids of the candidates configurations to obtain. #' @param drop.metadata Remove metadata, such the configuration ID and #' the ID of the parent, from the returned configurations. See -#' \code{\link{removeConfigurationsMetaData}}. +#' [removeConfigurationsMetaData()]. #' #' @return A data frame containing the elite configurations required. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres +#' @concept analysis #' @export -## Get configuration(s) by the id(s). -## iraceResults: object created by irace and saved in scenario$logFile. -## iraceLog: log file created by irace, this file must contain the iraceResults object. -## ids: the id or a vector of ids of the candidates configurations to obtain. -## drop.metadata: Remove the internal identifier and parent identifier from the returned -## configurations data frame. -## * iraceResults or iraceLog must be provided, in case both are give iraceResults will be used. -## This function returns a data frame containing the selected candidate configurations -getConfigurationById <- function(iraceResults = NULL, logFile = NULL, - ids, drop.metadata = FALSE) +getConfigurationById <- function(iraceResults, ids, drop.metadata = FALSE) { - if (is.null(iraceResults)) { - if (is.null(logFile)) - stop("You must supply either iraceResults or iraceLog argument.\n") - else - load(logFile) - } - - if (length(ids) < 1) stop("You must provide at least one configuration id.\n") + if (missing(iraceResults)) stop("argument 'iraceResults' is missing") + iraceResults <- read_logfile(iraceResults) + + if (length(ids) < 1) stop("You must provide at least one configuration id.") selection <- iraceResults$allConfigurations[,".ID."] %in% ids - if (length(selection) < 1) stop("No configuration found with id", ids,".\n") + if (length(selection) < 1) stop("No configuration found with id", ids,".") configurations <-iraceResults$allConfigurations[selection, , drop = FALSE] @@ -417,9 +73,7 @@ #' Returns the configurations by the iteration in which they were executed. #' -#' @param iraceResults (\code{NULL}) Object created by \pkg{irace} and saved in \code{scenario$logFile}. -#' @param logFile (\code{NULL}) Log file created by \pkg{irace}, this file must contain the -#' \code{iraceResults} object. +#' @template arg_iraceresults #' @param iterations The iteration number or a vector of iteration numbers from where #' the configurations should be obtained. #' @param drop.metadata (\code{FALSE}) Remove metadata, such the configuration ID and @@ -429,27 +83,16 @@ #' @return A data frame containing the elite configurations required. #' #' @author Manuel López-Ibáñez and Leslie Pérez Cáceres +#' @concept analysis #' @export -## Get configuration(s) by the iteration in which they were executed. -## iraceResults: object created by irace and saved in scenario$logFile. -## iraceLog: log file created by irace, this file must contain the iraceResults object. -## iterations: the iteration or a vector of iterations from where the configurations should be obtained. -## drop.metadata: Remove the internal identifier and parent identifier from the returned -## configurations data frame. -## * iraceResults or iraceLog must be provided, in case both are give iraceResults will be used. -## This function returns a data frame containing the selected candidate configurations -getConfigurationByIteration <- function(iraceResults = NULL, logFile = NULL, +getConfigurationByIteration <- function(iraceResults, iterations, drop.metadata = FALSE) { - if (is.null(iraceResults)) { - if (is.null(logFile)) - stop("You must supply either iraceResults or iraceLog argument.\n") - else - load(logFile) - } - + if (missing(iraceResults)) stop("argument 'iraceResults' is missing") + iraceResults <- read_logfile(iraceResults) + if (length(iterations) < 1) - stop("You must provide at least one configuration id.\n") + stop("You must provide at least one configuration id.") # To silence warning. iteration <- NULL @@ -459,7 +102,7 @@ selection <- iraceResults$allConfigurations[,".ID."] %in% ids - if (length(selection) < 1) stop("No configuration found with id", ids,".\n") + if (length(selection) < 1) stop("No configuration found with id", ids,".") configurations <- iraceResults$allConfigurations[selection, , drop=FALSE] @@ -468,69 +111,3 @@ return(configurations) } -#' Creates box plots of the quality of configurations. -#' -#' @param experiments Matrix of performance of configurations (columns) over a set of instances (rows). -#' @param title (\code{NULL}) Title for the plot. -#' @param xlabel Label for the x axis. -#' @param ylabel Label for the y axis. -#' @param filename (\code{NULL}) Filename prefix to create a pdf file with the plot. -#' -#' @return Box plot of the performance of the configurations. -#' -#' @author Manuel López-Ibáñez and Leslie Pérez Cáceres -#' @export -configurationsBoxplot <- function(experiments, title = NULL, - xlabel = "Configuration ID", - ylabel = "Configuration cost", - filename = NULL) -{ - plot.jitter.points <- function(x, y, factor = 10 / x, pch = 20, ...) - points(jitter(rep(x, length(y)), factor = factor), - y, pch = pch, col = rgb(0,0,0,.2), ...) - - if (any(colSums(is.na(experiments)) > 0)) - cat("Warning: There are NA values in the experiment results provided.\n") - - data.labels <- colnames(experiments) - if (is.null(data.labels)) data.labels <- 1:ncol(experiments) - - # These parameters could be exposed to configurate the plot - - if (!is.null(filename)) { - filename <- paste0(filename, ".pdf.") - cat("Creating file", filename,"\n") - # MANUEL: Why cairo_pdf = - cairo_pdf(filename = filename, width=20, height=8) - on.exit(dev.off(), add = TRUE) - plot.mar <- c(7,11,4,1) - plot.lwd <- 5 - cex.axis <- 3 - cex.main <- 3 - x.add <- 2 - } else { - plot.mar <- c(2.5,9,4,1) - plot.lwd <- 2 - cex.axis <- 1 - cex.main <- 1 - x.add <- 0 - } - - if (is.null(title)) plot.mar[3] <- 1 - - old.par <- par(las=1, mar=plot.mar, cex.axis=cex.axis, cex.main=cex.main, lwd=plot.lwd) - on.exit(old.par, add = TRUE) - - boxplot(experiments, main = title, xaxt = "n", outline = TRUE) - for (i in 1:ncol(experiments)) { - plot.jitter.points (i, experiments[,i], cex = 1.5 * cex.axis) - } - - # X axis - axis(1, at=c(1:length(data.labels)), labels=data.labels, line = -0.5 + x.add, - tick=FALSE, las=1, cex.axis=cex.axis) - mtext(xlabel, side=1, line=1.5 + 2*x.add, cex=cex.axis, las=0) - # Y axis - mtext(ylabel, side=2, line=5+1.8*x.add, cex=cex.axis, las=0) -} - diff --git a/R/parameterExploration.R b/R/parameterExploration.R index 493fbf4..89b20b4 100644 --- a/R/parameterExploration.R +++ b/R/parameterExploration.R @@ -30,9 +30,8 @@ #' @examples #' \dontrun{ #' # Execute the postselection automatically after irace +#' scenario <- readScenario(filename="scenario.txt") #' parameters <- readParameters("parameters.txt") -#' scenario <- readScenario(filename="scenario.txt", -#' scenario=defaultScenario()) #' # Use 10% of the total budget #' scenario$postselection <- 0.1 #' irace(scenario=scenario, parameters=parameters) @@ -42,8 +41,6 @@ #' #' @author Leslie Pérez Cáceres #' @export -# This function executes a post selection race -# elites: test all elites psRace <- function(iraceLogFile=NULL, iraceResults=NULL, conf.ids=NULL, postselection=NULL, max.experiments=NULL, elites=FALSE, seed=1234567) { @@ -51,13 +48,13 @@ if (is.null(iraceLogFile) && is.null(iraceResults)) irace.error("You must provide a Rdata file or an iraceResults object.") - irace.note ("Stating post-selection:\n# Seed:", seed, "\n") + irace.note ("Starting post-selection:\n# Seed:", seed, "\n") if (!is.null(iraceLogFile)) cat("# Log file:",iraceLogFile,"\n") # Load the data of the log file if (!is.null(iraceLogFile)) - load(iraceLogFile) + iraceResults <- read_logfile(iraceLogFile) parameters <- iraceResults$parameters scenario <- iraceResults$scenario @@ -66,7 +63,7 @@ if (!is.null(conf.ids)) { if (!all(conf.ids %in% iraceResults$allConfigurations$.ID.)) irace.error("Configuration ids provided", conf.ids,"cannot be found in the configurations.") - configurations <- iraceResults$allConfigurations[iraceResults$allConfigurations$.ID.%in% conf.ids,,drop=FALSE] + configurations <- iraceResults$allConfigurations[iraceResults$allConfigurations$.ID.%in% conf.ids,,drop=FALSE] } else { which.elites <- if (elites) unlist(iraceResults$allElites) else iraceResults$iterationElites which.elites <- unique(which.elites) @@ -113,7 +110,8 @@ elitistNewInstances = 0) experiments <- race.output$experiments - elite.configurations <- extractElites(race.output$configurations, + elite.configurations <- extractElites(scenario, parameters, + race.output$configurations, min(race.output$nbAlive, 1)) irace.note("Elite configurations (first number is the configuration ID;", " listed from best to worst according to the ", diff --git a/R/path_rel2abs.R b/R/path_rel2abs.R new file mode 100644 index 0000000..357e3b0 --- /dev/null +++ b/R/path_rel2abs.R @@ -0,0 +1,127 @@ +#' Converts a relative path to an absolute path. It tries really hard to create +#' canonical paths. +#' +#' @param path (`character(1)`) Character string representing a relative path. +#' @param cwd (`character(1)`) Current working directory. +#' +#' @return (`character(1)`) Character string representing the absolute path +#' +#' @examples +#' path_rel2abs("..") +#' @export +path_rel2abs <- function (path, cwd = getwd()) +{ + # Keep doing gsub as long as x keeps changing. + gsub.all <- function(pattern, repl, x, ...) { + repeat { + newx <- gsub(pattern, repl, x, ...) + if (newx == x) return(newx) + x <- newx + } + } + # FIXME: Why NA and not FALSE? + irace_normalize_path <- function(path) + suppressWarnings(normalizePath(path, winslash = "/", mustWork = NA)) + + if (is.null.or.na(path)) { + return (NULL) + } else if (path == "") { + return ("") + } + + # Using .Platform$file.sep is too fragile. Better just use "/" everywhere. + s <- "/" + + # Possibly expand ~/path to /home/user/path. + path <- path.expand(path) + # Remove winslashes if given. + path <- gsub("\\", s, path, fixed = TRUE) + + # Detect a Windows drive + windrive.regex <- "^[A-Za-z]:" + windrive <- "" + if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { + m <- regexpr(windrive.regex, path) + windrive <- regmatches(path, m) + path <- sub(windrive.regex, "", path) + } + + # Change "/./" to "/" to get a canonical form + path <- gsub.all(paste0(s, ".", s), s, path, fixed = TRUE) + # Change "//" to "/" to get a canonical form + path <- gsub(paste0(s, s, "+"), s, path) + # Change "/.$" to "/" to get a canonical form + path <- sub(paste0(s, "\\.$"), s, path) + # Drop final "/" + path <- sub(paste0(s, "$"), "", path) + if (path == "") path <- s + + # Prefix the current cwd to the path if it doesn't start with + # / \\ or whatever separator. + if (path == "." || !startsWith(path, s)) { + # There is no need to normalize cwd if it was returned by getwd() + if (!missing(cwd)) { + # Recurse to get absolute cwd + cwd <- path_rel2abs(cwd) + } + + # Speed-up the most common cases. + # If it is just "." + if (path == ".") return (irace_normalize_path(cwd)) + + # If it does not contain separators at all and does not start with ".." + if (!startsWith(path, "..") && !grepl(s, path)) { + # it may be a command in the path. + sys_path <- suppressWarnings(Sys.which(path)) + if (nchar(sys_path) > 0 + && file.access(sys_path, mode = 1) == 0 + && !file.info(sys_path)$isdir) { + return(irace_normalize_path(as.vector(sys_path))) + } + } + + # Remove "./" from the start of path. + path <- sub(paste0("^\\.", s), "", path) + # Make it absolute but avoid doubling s + if (substring(cwd, nchar(cwd)) == s) path <- paste0(cwd, path) + else path <- paste0(cwd, s, path) + # If it is just a path without ".." inside + if (!grepl(paste0(s,"\\.\\."), path)) { + return (irace_normalize_path(path)) + } + # Detect a Windows drive + if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { + m <- regexpr(windrive.regex, path) + windrive <- regmatches(path, m) + path <- sub(windrive.regex, "", path) + } + } + # else + + # Change "/x/.." to "/" to get a canonical form + prevdir.regex <- paste0(s, "[^", s,"]+", s, "\\.\\.") + repeat { + # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" + tmp <- sub(paste0(prevdir.regex, s), s, path) + if (tmp == path) break + path <- tmp + } + # Handle "/something/..$" to "/" that is, when ".." is the last thing in the path. + path <- sub(paste0(prevdir.regex, "$"), s, path) + + # Handle "^/../../.." to "/" that is, going up at the root just returns the root. + repeat { + # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" + tmp <- sub(paste0("^", s, "\\.\\.", s), s, path) + if (tmp == path) break + path <- tmp + } + # Handle "^/..$" to "/" that is, when ".." is the last thing in the path. + path <- sub(paste0("^", s, "\\.\\.$"), s, path) + # Add back Windows drive, if any. + path <- paste0(windrive, path) + + # We use normalizePath, which will further simplify the path if + # the path exists. + irace_normalize_path(path) +} diff --git a/R/race-wrapper.R b/R/race-wrapper.R index 9bcb6c7..fde78b4 100644 --- a/R/race-wrapper.R +++ b/R/race-wrapper.R @@ -34,12 +34,12 @@ #' irace.logfile <- file.path(system.file(package="irace"), #' "exdata", "irace-acotsp.Rdata") #' load(irace.logfile) -#' attach(iraceResults) +#' allConfigurations <- iraceResults$allConfigurations +#' parameters <- iraceResults$parameters #' apply(allConfigurations[1:10, unlist(parameters$names)], 1, buildCommandLine, #' unlist(parameters$switches)) #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export buildCommandLine <- function(values, switches) { @@ -78,11 +78,11 @@ target.evaluator.call = NULL) { if (!is.null(target.evaluator.call)) { - err.msg <- paste0(err.msg, "\n", .irace.prefix, + err.msg <- paste0(err.msg, "\n", .msg.prefix, "The call to targetEvaluator was:\n", target.evaluator.call) } if (!is.null(target.runner.call)) { - err.msg <- paste0(err.msg, "\n", .irace.prefix, + err.msg <- paste0(err.msg, "\n", .msg.prefix, "The call to targetRunner was:\n", target.runner.call) } if (is.null(output$outputRaw)) { @@ -101,9 +101,9 @@ " Try to run the command(s) above from the execution directory '", scenario$execDir, "' to investigate the issue. See also Appendix B (targetRunner troubleshooting checklist) of the User Guide (https://cran.r-project.org/package=irace/vignettes/irace-package.pdf).") } - irace.error(err.msg, "\n", .irace.prefix, + irace.error(err.msg, "\n", .msg.prefix, "The output was:\n", paste(output$outputRaw, collapse = "\n"), - "\n", .irace.prefix, advice.txt) + "\n", .msg.prefix, advice.txt) } check.output.target.evaluator <- function (output, scenario, target.runner.call = NULL) @@ -146,7 +146,7 @@ scenario, target.runner.call) check.output.target.evaluator (output, scenario, target.runner.call = target.runner.call) # Fix too small time. - output$time <- if (is.null(output$time)) NA else max(output$time, 0.01) + output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) return (output) } @@ -194,7 +194,6 @@ #' #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export target.evaluator.default <- function(experiment, num.configurations, all.conf.id, scenario, target.runner.call) @@ -212,8 +211,7 @@ } cwd <- setwd (scenario$execDir) - # FIXME: I think we don't even need to paste the args, since system2 handles this by itself. - args <- paste(configuration.id, instance.id, seed, instance, num.configurations, all.conf.id) + args <- c(configuration.id, instance.id, seed, instance, num.configurations, all.conf.id) output <- runcommand(targetEvaluator, args, configuration.id, debugLevel) setwd (cwd) @@ -234,7 +232,7 @@ } return(list(cost = cost, time = time, error = err.msg, outputRaw = output$output, - call = paste(targetEvaluator, args))) + call = paste(targetEvaluator, args, collapse=" "))) } check.output.target.runner <- function (output, scenario) @@ -290,7 +288,7 @@ target.error (err.msg, output, scenario, target.runner.call = output$call) } # Fix too small time. - output$time <- if (is.null(output$time)) NA else max(output$time, 0.01) + output$time <- if (is.null(output$time)) NA else max(output$time, scenario$minMeasurableTime) return (output) } @@ -302,7 +300,7 @@ doit <- function(experiment, scenario) { x <- target.runner(experiment, scenario) - return (check.output.target.runner (x, scenario)) + return (check.output.target.runner(x, scenario)) } retries <- scenario$targetRunnerRetries @@ -354,6 +352,40 @@ target.runner.aclib <- function(experiment, scenario) { + debugLevel <- scenario$debugLevel + res <- run_target_runner(experiment, scenario) + cmd <- res$cmd + output <- res$output + args <- res$args + + err.msg <- output$error + if (is.null(err.msg)) { + return(c(parse.aclib.output(output$output), + list(outputRaw = output$output, call = paste(cmd, args)))) + } + + list(cost = NULL, time = NULL, error = err.msg, + outputRaw = output$output, call = paste(cmd, args)) +} + +check_launcher_args <- function(targetRunnerLauncherArgs) +{ + if (!grepl("{targetRunner}", targetRunnerLauncherArgs, fixed=TRUE)) + irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunner}'") + if (!grepl("{targetRunnerArgs}", targetRunnerLauncherArgs, fixed=TRUE)) + irace.error("targetRunnerLauncherArgs '", targetRunnerLauncherArgs, "' must contain '{targetRunnerArgs}'") +} + +process_launcher_args <- function(targetRunnerLauncherArgs, targetRunner, args) +{ + check_launcher_args(targetRunnerLauncherArgs) + targetRunnerLauncherArgs <- gsub("{targetRunner}", targetRunner, targetRunnerLauncherArgs, fixed=TRUE) + targetRunnerLauncherArgs <- gsub("{targetRunnerArgs}", args, targetRunnerLauncherArgs, fixed=TRUE) + targetRunnerLauncherArgs +} + +run_target_runner <- function(experiment, scenario) +{ configuration.id <- experiment$id.configuration instance.id <- experiment$id.instance seed <- experiment$seed @@ -362,39 +394,43 @@ switches <- experiment$switches bound <- experiment$bound - debugLevel <- scenario$debugLevel targetRunner <- scenario$targetRunner - if (as.logical(file.access(targetRunner, mode = 1))) { - irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") - } - - has_value <- !is.na(configuration) - # [] [] ... [--cutoff ] [--instance ] - # [--seed ] --config [-param_name_1 value_1] [-param_name_2 value_2] ... - args <- paste("--instance", instance, "--seed", seed, "--config", - paste0("-", switches[has_value], " ", configuration[has_value], - collapse = " ")) - if (!is.null.or.na(bound)) - args <- paste("--cutoff", bound, args) - - output <- runcommand(targetRunner, args, configuration.id, debugLevel) - - err.msg <- output$error - if (is.null(err.msg)) { - return(c(parse.aclib.output (output$output), - list(outputRaw = output$output, call = paste(targetRunner, args)))) - } - - return(list(cost = NULL, time = NULL, error = err.msg, - outputRaw = output$output, call = paste(targetRunner, args))) -} - - -#' target.runner.default + debugLevel <- scenario$debugLevel + + if (scenario$aclib) { + has_value <- !is.na(configuration) + # [] [] ... [--cutoff ] [--instance ] + # [--seed ] --config [-param_name_1 value_1] [-param_name_2 value_2] ... + args <- paste("--instance", instance, "--seed", seed, "--config", + paste0("-", switches[has_value], " ", configuration[has_value], + collapse = " ")) + if (!is.null.or.na(bound)) + args <- paste("--cutoff", bound, args) + } else { + args <- paste(configuration.id, instance.id, seed, instance, bound, + buildCommandLine(configuration, switches)) + } + + targetRunnerLauncher <- scenario$targetRunnerLauncher + if (is.null.or.empty(scenario$targetRunnerLauncher)) { + if (as.logical(file.access(targetRunner, mode = 1))) { + irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") + } + output <- runcommand(targetRunner, args, configuration.id, debugLevel) + return(list(cmd=targetRunner, output=output, args=args)) + } + if (as.logical(file.access(targetRunnerLauncher, mode = 1))) { + irace.error ("targetRunnerLauncher ", shQuote(targetRunnerLauncher), " cannot be found or is not executable!\n") + } + + args <- process_launcher_args(scenario$targetRunnerLauncherArgs, targetRunner, args) + output <- runcommand(targetRunnerLauncher, args, configuration.id, debugLevel) + return(list(cmd=targetRunnerLauncher, output=output, args=args)) +} + +#' Default `targetRunner` function. #' -#' `target.runner.default` is the default targetRunner function. -#' You can use it as an advanced example of how to create your own targetRunner -#' function. +#' Use it as an advanced example of how to create your own `targetRunner` function. #' #' @param experiment A list describing the experiment. It contains at least: #' \describe{ @@ -431,32 +467,19 @@ #' #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export target.runner.default <- function(experiment, scenario) { - configuration.id <- experiment$id.configuration - instance.id <- experiment$id.instance - seed <- experiment$seed - configuration <- experiment$configuration - instance <- experiment$instance - switches <- experiment$switches - bound <- experiment$bound - - debugLevel <- scenario$debugLevel - targetRunner <- scenario$targetRunner - if (as.logical(file.access(targetRunner, mode = 1))) { - irace.error ("targetRunner ", shQuote(targetRunner), " cannot be found or is not executable!\n") - } - - args <- paste(configuration.id, instance.id, seed, instance, bound, - buildCommandLine(configuration, switches)) - output <- runcommand(targetRunner, args, configuration.id, debugLevel) - + res <- run_target_runner(experiment, scenario) + cmd <- res$cmd + output <- res$output + args <- res$args + + debugLevel <- scenario$debugLevel cost <- time <- NULL err.msg <- output$error if (is.null(err.msg)) { - v.output <- parse.output(output$output, verbose = (scenario$debugLevel >= 2)) + v.output <- parse.output(output$output, verbose = (debugLevel >= 2)) if (length(v.output) > 2) { err.msg <- "The output of targetRunner should not be more than two numbers!" } else if (length(v.output) == 1) { @@ -470,9 +493,9 @@ time <- v.output[2] } } - return(list(cost = cost, time = time, - error = err.msg, outputRaw = output$output, - call = paste(targetRunner, args))) + list(cost = cost, time = time, + error = err.msg, outputRaw = output$output, + call = paste(cmd, args, collapse = " ")) } execute.experiments <- function(experiments, scenario) @@ -570,14 +593,14 @@ scenario = scenario, target.runner = target.runner) } - - return(target.output) + target.output } execute.evaluator <- function(experiments, scenario, target.output, configurations.id) { ## FIXME: We do not need the configurations.id argument: - # configurations.id <- sapply(experiments, function(x) x[["id.configuration"]]) + irace.assert(isTRUE(all.equal(configurations.id, + sapply(experiments, getElement, "id.configuration")))) all.conf.id <- paste(configurations.id, collapse = " ") ## Evaluate configurations sequentially diff --git a/R/race.R b/R/race.R index d85371b..e60ce70 100644 --- a/R/race.R +++ b/R/race.R @@ -78,9 +78,7 @@ # which.alive nor which.exe which.alive, which.exe, parameters, scenario) { - debugLevel <- scenario$debugLevel - - irace.assert (isTRUE(parameters$nbParameters > 0)) + irace.assert (parameters$nbVariable > 0) irace.assert (length(parameters$names) == parameters$nbParameters) # FIXME: Accessing 'seed' and 'instance' should be moved to createExperimentList. @@ -109,7 +107,7 @@ if (!is.null(scenario$targetEvaluator)) target.output <- execute.evaluator (experiments, scenario, target.output, configurations[, ".ID."]) - return(target.output) + target.output } aux2.friedman <- function(y, I, alive, conf.level = 0.95) @@ -160,31 +158,29 @@ if (no.alive == 2) { best <- NULL ranks <- NULL + dropped.any <- TRUE + PVAL <- 0 # If only 2 configurations are left, switch to Wilcoxon - V1 <- results[, which.alive[1]] - V2 <- results[, which.alive[2]] - PVAL <- wilcox.test(V1, V2, paired = TRUE, exact = FALSE)$p.value - if (!is.nan(PVAL) && !is.na(PVAL) && PVAL < 1 - conf.level) { - dropped.any <- TRUE - if (median(V1 - V2) < 0) { - best <- which.alive[1] - ranks <- c(1,2) - alive[which.alive[2]] <- FALSE - } else { - best <- which.alive[2] - ranks <- c(2,1) - alive[which.alive[1]] <- FALSE - } + V1 <- results[, which.alive[1]] + V2 <- results[, which.alive[2]] + + # Avoid the test if the answer is obvious + if (all(V1 <= V2)) { + ranks <- c(1,2) + } else if (all(V2 <= V1)) { + ranks <- c(2,1) } else { - dropped.any <- FALSE - if (median(V1 - V2) < 0) { - best <- which.alive[1] - ranks <- c(1,2) - } else { - best <- which.alive[2] - ranks <- c(2,1) - } - } + res <- wilcox.test(V1, V2, paired = TRUE, conf.int = TRUE) + PVAL <- res$p.value + irace.assert(!is.nan(PVAL) & !is.na(PVAL)) + if (PVAL >= 1 - conf.level) dropped.any <- FALSE + # We use the pseudo median estimated by the test. + ranks <- if (res$estimate <= 0) c(1,2) else c(2,1) + } + best <- which.alive[ranks[1]] + if (dropped.any) + alive[which.alive[ranks[2]]] <- FALSE + irace.assert(which.alive[which.min(ranks)] == best) return(list(best = best, ranks = ranks, alive = alive, dropped.any = dropped.any, p.value = PVAL)) } else { @@ -193,12 +189,11 @@ } } -aux.ttest <- function(results, alive, which.alive, conf.level, - adjust = c("none","bonferroni","holm")) +aux.one_ttest <- function(results, alive, which.alive, conf.level, + adjust = c("none","bonferroni","holm")) { adjust <- match.arg(adjust) irace.assert(sum(alive) == length(which.alive)) - results <- results[, which.alive] means <- colMeans(results) best <- which.min(means) @@ -211,12 +206,16 @@ for (j in which_test) { PVAL <- pvals[j] if (PVAL == 1.0) next - results_j <- results[ , j] + results_j <- results[, j] # t.test may fail if the data in each group is almost constant. Hence, we # surround the call in a try() and we initialize p with 1 if the means are # equal or zero if they are different if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next - try(PVAL <- t.test(results_best, results_j, paired = TRUE)$p.value) + # The t.test may fail if the data are not normal despite one configuration + # clearly dominating the other. + if (all(results_best <= results_j)) next + try(PVAL <- t.test(results_best, results_j, alternative = "less", + paired = TRUE)$p.value) irace.assert(!is.nan(PVAL) & !is.na(PVAL)) pvals[j] <- PVAL } @@ -229,56 +228,118 @@ dropped.any = dropped_any, p.value = min(pvals))) } -race.init.instances <- function(deterministic, max.instances) -{ - if (.irace$next.instance <= max.instances) { - race.instances <- .irace$next.instance : max.instances - } else { - # This may happen if the scenario is deterministic and we would need - # more instances than what we have. - irace.assert(deterministic) - race.instances <- 1 : max.instances - } - return(race.instances) -} - -elitrace.init.instances <- function(race.env, deterministic, max.instances) +aux.ttest <- function(results, alive, which.alive, conf.level, + adjust = c("none","bonferroni","holm")) +{ + adjust <- match.arg(adjust) + irace.assert(sum(alive) == length(which.alive)) + + results <- results[, which.alive] + means <- colMeans(results) + # FIXME: break ties using median or ranks? + best <- which.min(means) + mean_best <- means[best] + pvals <- sapply(means, function(x) as.numeric(isTRUE( + all.equal.numeric(mean_best[[1]], x[[1]], check.attributes = FALSE)))) + results_best <- results[, best] + var_best <- var(results_best) + which_test <- which(pvals < 1.0) + for (j in which_test) { + PVAL <- pvals[j] + if (PVAL == 1.0) next + results_j <- results[, j] + # t.test may fail if the data in each group is almost constant. Hence, we + # surround the call in a try() and we initialize p with 1 if the means are + # equal or zero if they are different + if (min(var(results_best), var(results_j)) < 10 * .Machine$double.eps) next + # The t.test may fail if the data are not normal despite one configuration + # clearly dominating the other. + if (all(results_best <= results_j)) next + try(PVAL <- t.test(results_best, results_j, paired = TRUE)$p.value) + irace.assert(!is.nan(PVAL) & !is.na(PVAL)) + pvals[j] <- PVAL + } + pvals <- p.adjust(pvals, method = adjust) + dropj <- which.alive[pvals < 1.0 - conf.level] + dropped_any <- length(dropj) > 0 + irace.assert(all(alive[dropj])) + alive[dropj] <- FALSE + return(list(best = which.alive[best], ranks = means, alive = alive, + dropped.any = dropped_any, p.value = min(pvals))) +} + +no_elitrace.init.instances <- function(deterministic, max_instances) { # if next.instance == 1 then this is the first iteration. - if (.irace$next.instance != 1) { - new.instances <- NULL - last.new <- .irace$next.instance + race.env$elitistNewInstances - 1 - # Do we need to add new instances? - if (race.env$elitistNewInstances > 0) { - if (last.new > max.instances) { - # This may happen if the scenario is deterministic and we would need - # more instances than what we have. - irace.assert(deterministic) - if (.irace$next.instance <= max.instances) { - # Add all instances that we have not seen yet as new ones. - last.new <- max.instances - new.instances <- .irace$next.instance : last.new - } # else new.instances remains NULL and last.new remains > number of instances. - # We need to update this because the value is used below and now there - # may be fewer than expected, even zero. - race.env$elitistNewInstances <- length(new.instances) - } else { + # If deterministic consider all (do not resample). + if (.irace$next.instance == 1 || deterministic) return(1:max_instances) + irace.assert(.irace$next.instance < max_instances) + return(.irace$next.instance : max_instances) +} + +elitrace.init.instances <- function(race.env, deterministic, max_instances, sampleInstances) +{ + # if next.instance == 1 then this is the first iteration. + if (.irace$next.instance == 1) return(1:max_instances) # Consider all + + new.instances <- NULL + last.new <- .irace$next.instance + race.env$elitistNewInstances - 1 + # Do we need to add new instances? + if (race.env$elitistNewInstances > 0) { + if (last.new > max_instances) { + # This may happen if the scenario is deterministic and we would need + # more instances than what we have. + irace.assert(deterministic) + if (.irace$next.instance <= max_instances) { + # Add all instances that we have not seen yet as new ones. + last.new <- max_instances new.instances <- .irace$next.instance : last.new - } - } - future.instances <- NULL - if ((last.new + 1) <= max.instances) { - future.instances <- (last.new + 1) : max.instances - } - # new.instances + past.instances + future.instances - race.instances <- c(new.instances, sample.int(.irace$next.instance - 1), - future.instances) - } else { - race.instances <- race.init.instances(deterministic, max.instances) - } - return(race.instances) -} - + } # else new.instances remains NULL and last.new remains > number of instances. + # We need to update this because the value is used below and now there + # may be fewer than expected, even zero. + race.env$elitistNewInstances <- length(new.instances) + } else { + new.instances <- .irace$next.instance : last.new + } + } + future.instances <- NULL + if ((last.new + 1) <= max_instances) { + future.instances <- (last.new + 1) : max_instances + } + # new.instances + past.instances + future.instances + # FIXME: we should sample taking into account the block-size, so we sample blocks, not instances. + past_instances <- if (sampleInstances) + sample.int(.irace$next.instance - 1) else + 1:.irace$next.instance + c(new.instances, past_instances, future.instances) +} + +table_hline <- function(widths) { + s <- "+" + for(w in widths) { + s <- paste0(s, strrep("-", w), "+") + } + return(paste0(s, "\n")) +} +table_sprint <- function(text, widths) { + s <- "|" + for (i in seq_along(text)) { + s <- paste0(s, sprintf("%*s", widths[i], text[i]), "|") + } + return(paste0(s, "\n")) +} +.nocap_table_fields_width <- c(1, 11, 11, 11, 16, 11, 8, 5, 4, 6) +.nocap_colum_names <- c(" ", "Instance", "Alive", "Best", "Mean best", "Exp so far", + "W time", "rho", "KenW", "Qvar") +.capping_table_fields_width <- c(1, 11, 8, 11, 11, 16, 11, 8, 5, 4, 6) +.capping_colum_names <- c(" ", "Instance", "Bound", "Alive", "Best", "Mean best", "Exp so far", + "W time", "rho", "KenW", "Qvar") +capping_hline <- table_hline(.capping_table_fields_width) +capping_header <- table_sprint(.capping_colum_names, .capping_table_fields_width) +nocap_hline <- table_hline(.nocap_table_fields_width) +nocap_header <- table_sprint(.nocap_colum_names, .nocap_table_fields_width) + +# FIXME: Depending on capping here is ugly. We should simply set-up the correct printing functions at the start of race(). race.print.header <- function(capping) { cat(sep = "", "# Markers: @@ -287,16 +348,11 @@ - The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. - . All alive configurations are elite and nothing is discarded\n") - cat(sep = "", if (!capping) " -+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ -| | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| -+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+ -" else " -+-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+ -| | Instance| Bound| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| -+-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+ -") + . All alive configurations are elite and nothing is discarded\n\n") + if (capping) + cat(sep = "", capping_hline, capping_header, capping_hline) + else + cat(sep = "", nocap_hline, nocap_header, nocap_hline) } race.print.task <- function(res.symb, Results, @@ -313,21 +369,25 @@ if (now <= start) return("00:00:00") elapsed <- difftime(now, start, units = "secs") # FIXME: Maybe better and faster if we only print seconds? - return(format(.POSIXct(elapsed, tz="GMT"), "%H:%M:%S")) + format(.POSIXct(elapsed, tz="GMT"), "%H:%M:%S") } # FIXME: This is the mean of the best, but perhaps it should # be the sum of ranks in the case of test == friedman? mean_best <- mean(Results[, best]) time_str <- elapsed_wctime_str(Sys.time(), start.time) cat(sprintf("|%s|%11d|", res.symb, instance)) - if (capping) cat(sprintf("%8.2f|", if(is.null(bound)) NA else bound)) - cat(sprintf("%11d|%11d|%#15.10g|%11d|%s", + if (capping) { + if (is.null(bound)) cat(" NA|") else cat(sprintf("%8.2f|", bound)) + } + cat(sprintf(paste0("%11d|%11d|", .irace.format.perf, "|%11d|%s"), sum(alive), id.best, mean_best, experimentsUsed, time_str)) if (current.task > 1 && sum(alive) > 1) { conc <- concordance(Results[1:current.task, alive, drop = FALSE]) qvar <- dataVariance(Results[1:current.task, alive, drop = FALSE]) - cat(sprintf("|%+#4.2f|%.2f|%.4f|\n", conc$spearman.rho, conc$kendall.w, qvar)) + # FIXME: We would like to use %+#4.2f but this causes problems with + # https://github.com/oracle/fastr/issues/191 + cat(sprintf("|%+4.2f|%.2f|%.4f|\n", conc$spearman.rho, conc$kendall.w, qvar)) } else { cat("| NA| NA| NA|\n") } @@ -336,13 +396,11 @@ race.print.footer <- function(bestconf, mean.best, break.msg, debug.level, capping = FALSE) { cat(sep = "", - if (!capping) - "+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+\n" - else - "+-+-----------+--------+-----------+-----------+---------------+-----------+--------+-----+----+------+\n", + if (capping) capping_hline else nocap_hline, if (debug.level >= 1) paste0("# Stopped because ", break.msg, "\n"), sprintf("Best-so-far configuration: %11d", bestconf[1, ".ID."]), - sprintf(" mean value: %#15.10g", mean.best), "\n", + " mean value: ", + sprintf(.irace.format.perf, mean.best), "\n", "Description of the best-so-far configuration:\n") configurations.print(bestconf, metadata = TRUE) cat("\n") @@ -429,8 +487,9 @@ ## that exceed the maximum execution time (boundMax) applyPAR <- function(results, boundMax, boundPar) { + # We do not want to change Inf or -Inf because those represent rejection. if (boundPar != 1) - results[results >= boundMax] <- boundMax * boundPar + results[is.finite(results) & results >= boundMax] <- boundMax * boundPar return(results) } @@ -443,13 +502,12 @@ final.execution.bound <- function(experimentsTime, elites, no.configurations, current.task, which.exe, scenario) { - # FIXME: Make this a scenario parameter. The same parameter used in - # race-wrapper when parsing target.runner output. - minMeasurableTime <- 0.01 - - final.bounds <- rep(scenario$boundMax, no.configurations) - total.time <- current.task * scenario$boundMax - elite.bound <- scenario$boundMax + minMeasurableTime <- scenario$minMeasurableTime + boundMax <- scenario$boundMax + # FIXME: should we use an adjusted boundMax + final.bounds <- rep(boundMax, no.configurations) + total.time <- current.task * boundMax + elite.bound <- boundMax # Elite candidates can have NA values due to the rejection if (length(elites) > 0) elites <- elites[!is.na(experimentsTime[current.task,elites])] @@ -463,17 +521,17 @@ if (scenario$boundType == "instance") { elite.bound <- instanceBound(experimentsTime[current.task, elites], type = scenario$cappingType) - final.bounds[which.exe] <- min(elite.bound + minMeasurableTime, scenario$boundMax) + final.bounds[which.exe] <- min(elite.bound + minMeasurableTime, boundMax) final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) } else { elite.bound <- executionBound(experimentsTime[1:current.task, elites, drop = FALSE], type = scenario$cappingType) - elite.bound <- min(elite.bound, scenario$boundMax) + elite.bound <- min(elite.bound, boundMax) # FIXME: This minMeasurableTime should be a scenario setting and it # should be the same value that we use in check.output.target.runner total.time <- (current.task * elite.bound) + minMeasurableTime time.left <- total.time - colSums(experimentsTime[1:current.task, which.exe, drop = FALSE], na.rm = TRUE) - final.bounds[which.exe] <- sapply(time.left, min, scenario$boundMax) + final.bounds[which.exe] <- sapply(time.left, min, boundMax) # We round up the bounds up to the specified number of digits. This may # be necessary if the target-algorithm does not support higher precision. final.bounds[which.exe] <- ceiling.digits(final.bounds[which.exe], scenario$boundDigits) @@ -563,6 +621,8 @@ first.test <- scenario$firstTest each.test <- scenario$eachTest elitist <- scenario$elitist + capping <- scenario$capping + quiet <- scenario$quiet no.configurations <- nrow(configurations) experimentLog <- matrix(nrow = 0, ncol = 4, dimnames = list(NULL, c("instance", "configuration", "time", "bound"))) @@ -596,10 +656,11 @@ if (elitist) race.instances <- elitrace.init.instances(race.env, scenario$deterministic, - max.instances = nrow(.irace$instancesList)) + max_instances = nrow(.irace$instancesList), + sampleInstances = scenario$sampleInstances) else - race.instances <- race.init.instances(scenario$deterministic, - max.instances = nrow(.irace$instancesList)) + race.instances <- no_elitrace.init.instances(scenario$deterministic, + max_instances = nrow(.irace$instancesList)) no.tasks <- length(race.instances) # Initialize some variables... @@ -631,7 +692,7 @@ nrow = elite.safe, ncol = no.configurations, dimnames = list(elite.instances.ID, configurations.ID)) - if (scenario$capping) + if (capping) experimentsTime <- matrix(NA, nrow = elite.safe, ncol = no.configurations, @@ -641,14 +702,14 @@ Results[rownames(elite.data[["experiments"]]), colnames(elite.data[["experiments"]])] <- elite.data[["experiments"]] - if (scenario$capping) { + if (capping) { experimentsTime[rownames(elite.data[["time"]]), colnames(elite.data[["time"]])] <- elite.data[["time"]] } # Preliminary execution of elite configurations to calculate # the execution bound of initial configurations (capping only). - if (scenario$capping && elitistNewInstances != 0) { + if (capping && elitistNewInstances != 0) { # FIXME: This should go into its own function. n.elite <- ncol(elite.data[["experiments"]]) which.elites <- which(rep(TRUE, n.elite)) @@ -717,12 +778,14 @@ best <- 0 race.ranks <- c() no.elimination <- 0 # number of tasks without elimination. - - race.print.header(scenario$capping) + if (!quiet) + race.print.header(capping) # Test that all instances that have been previously seen have been evaluated # by at least one configuration. all_elite_instances_evaluated <- function() { + if (!elitist) + return(TRUE) return(all(apply(!is.na(Results[ as.character(seq_len(.irace$next.instance - 1)), alive, drop=FALSE]), 1, any))) @@ -730,7 +793,7 @@ # Start main loop break.msg <- NULL - for (current.task in seq_len (no.tasks)) { + for (current.task in seq_len(no.tasks)) { which.alive <- which(alive) nbAlive <- length(which.alive) which.exe <- which.alive @@ -749,12 +812,13 @@ # criterion is disabled) ## MANUEL: So what is the reason to not immediately terminate here? Is ## there a reason to continue? - race.print.task(".", Results[1:current.task, , drop = FALSE], - race.instances[current.task], - current.task, alive = alive, - configurations[best, ".ID."], - best = best, experimentsUsed, Sys.time(), - bound = NA, scenario$capping) + if (!quiet) + race.print.task(".", Results[1:current.task, , drop = FALSE], + race.instances[current.task], + current.task, alive = alive, + configurations[best, ".ID."], + best = best, experimentsUsed, Sys.time(), + bound = NA, capping) next } } @@ -768,12 +832,12 @@ ## there are instances previously seen that have not been evaluated on any ## alive configuration. if (current.task > first.test) { - #if ((current.task > first.test && !scenario$capping) + #if ((current.task > first.test && !capping) # MANUEL: This is new and I'm not sure what it does. # LESLIE: When using capping, we dont finish any race until all # previous instances have been executed (this makes sure that all non-elite # configurations execute all the previous instances) - # || (scenario$capping && (current.task > elite.safe))) { + # || (capping && (current.task > elite.safe))) { # MANUEL: How is this even possible? # LESLIE: It can be that the capping eliminate all but one configuration # (which should be an elite one) after we finish the new instances to be evaluated, @@ -826,7 +890,7 @@ if (nrow(Results) < current.task) { Results <- rbind(Results, rep(NA, ncol(Results))) rownames(Results) <- race.instances[1:nrow(Results)] - if (scenario$capping) { + if (capping) { experimentsTime <- rbind(experimentsTime, rep(NA, ncol(experimentsTime))) rownames(experimentsTime) <- race.instances[1:nrow(experimentsTime)] } @@ -839,7 +903,7 @@ # Calculate bounds for executing if needed. which.elite.exe <- intersect(which.exe, which(is.elite > 0)) irace.assert(setequal(which.elite.exe, which(is.elite & is.na(Results[current.task,])))) - if (scenario$capping) { + if (capping) { # Pre-execute elite configurations that are not yet executed in the current instance. if (length(which.elite.exe)) { # FIXME: This should go into its own function @@ -884,14 +948,16 @@ elite.safe <- update.elite.safe(Results, is.elite) } which.exe <- setdiff(which.exe, which.elite.exe) + # FIXME: There is similar code above. if (length(which.exe) == 0L) { is.elite <- update.is.elite(is.elite, which.elite.exe) - race.print.task(".", Results[1:current.task, , drop = FALSE], - race.instances[current.task], - current.task, alive = alive, - configurations[best, ".ID."], - best = best, experimentsUsed, start.time, - bound = NA, scenario$capping) + if (!quiet) + race.print.task(".", Results[1:current.task, , drop = FALSE], + race.instances[current.task], + current.task, alive = alive, + configurations[best, ".ID."], + best = best, experimentsUsed, start.time, + bound = NA, capping) next } } @@ -925,7 +991,7 @@ irace.assert(length(vcost) == length(which.exps)) # Set max execution bound to timed out executions which have execution # times smaller than boundMax and implement parX if required - if (scenario$capping) { + if (capping) { vcost <- applyPAR(vcost, boundMax = scenario$boundMax, boundPar = scenario$boundPar) if (scenario$boundAsTimeout) vcost[(vcost >= final.bounds[which.exps]) & (vcost < scenario$boundMax)] <- scenario$boundMax @@ -937,7 +1003,7 @@ irace.assert(length(which.exps) == length(which.exe)) vtimes <- unlist(lapply(output[which.exps], "[[", "time")) irace.assert(length(vtimes) == length(which.exe)) - if (scenario$capping) { + if (capping) { # Correct higher execution times. experimentsTime[current.task, which.exps] <- pmin(vtimes, final.bounds[which.exps]) } @@ -981,7 +1047,7 @@ ## Dominance elimination (Capping only). # The second condition can be false if we eliminated via immediate # rejection - if (scenario$capping && sum(alive) > minSurvival) { + if (capping && sum(alive) > minSurvival) { irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) cap.alive <- dom.elim(Results[1:current.task, , drop = FALSE], # Get current elite configurations @@ -1025,9 +1091,9 @@ # It may happen that the capping and the test eliminate together all # configurations. In that case, we only trust the capping elimination. - if (scenario$capping && !any(alive)) { + if (capping && !any(alive)) { if (scenario$debugLevel >= 2) { - irace.note("Warning: Elimination tests have eliminated all configurations, keeping the capping results.\n") + irace.warning("Elimination tests have eliminated all configurations, keeping the capping results.\n") irace.note("Alive according to capping:", which(cap.alive), "\n") irace.note("Alive according to test:", which(test.alive), "\n") } @@ -1069,14 +1135,14 @@ # Remove the ranks of those that are not alive anymore race.ranks <- race.ranks[which.alive] irace.assert(length(race.ranks) == sum(alive)) - - race.print.task(res.symb, Results[1:current.task, , drop = FALSE], - race.instances[current.task], - current.task, alive = alive, - configurations[best, ".ID."], - best = best, experimentsUsed, start.time, - bound = elite.bound, scenario$capping) - + if (!quiet) + race.print.task(res.symb, Results[1:current.task, , drop = FALSE], + race.instances[current.task], + current.task, alive = alive, + configurations[best, ".ID."], + best = best, experimentsUsed, start.time, + bound = elite.bound, capping) + if (elitist) { # Compute number of statistical tests without eliminations. irace.assert(!any(is.elite > 0) == (current.task >= elite.safe)) @@ -1091,18 +1157,19 @@ } } - # Adding this given that when ncandidates = minsuvival+1 + if (is.null(break.msg)) + break.msg <- paste0("all instances (", no.tasks, ") evaluated") + + # Adding this given that when ncandidates = minsurvival+1 # and there one elite configuration that gets discarded in the new instances # execution the race is finished with no executions. # FIXME: we should handle this better, maybe allowing irace to handle no elite # in irace() - # MANUEL: I still don't understand how we reach this error if there are non-rejected elites. - if (current.task == 1 && !any(is.elite > 0)) + # MANUEL: Leslie, how can we reach this error in normal circumstances? + # Can we handle this better? + if (current.task == 1 && !any(is.elite > 0)) irace.error ("Maximum number configurations immediately rejected reached!") - - if (is.null(break.msg)) - break.msg <- paste0("all instances (", no.tasks, ") evaluated") - + # All instances that are not new in this race must have been evaluated by at # least one configuration. irace.assert(all_elite_instances_evaluated(), @@ -1114,14 +1181,15 @@ race.ranks <- overall.ranks(Results[, alive, drop = FALSE], stat.test = stat.test) best <- which.alive[which.min(race.ranks)] - race.print.footer(bestconf = configurations[best, , drop = FALSE], - # FIXME: This is the mean of the best, but perhaps it - # should be the sum of ranks in the case of test == - # friedman? - mean.best = mean(Results[, best]), - break.msg = break.msg, debug.level = scenario$debugLevel, - capping = scenario$capping) - + if (!quiet) + race.print.footer(bestconf = configurations[best, , drop = FALSE], + # FIXME: This is the mean of the best, but perhaps it + # should be the sum of ranks in the case of test == + # friedman? + mean.best = mean(Results[, best]), + break.msg = break.msg, debug.level = scenario$debugLevel, + capping = capping) + nbAlive <- sum(alive) configurations$.ALIVE. <- as.logical(alive) # Assign the proper ranks in the configurations data.frame. diff --git a/R/readConfiguration.R b/R/readConfiguration.R index 7da7009..ea1985c 100644 --- a/R/readConfiguration.R +++ b/R/readConfiguration.R @@ -1,10 +1,10 @@ -#' readConfigurationsFile -#' -#' `readConfigurationsFile` reads a set of target algorithms configurations -#' from a file and puts them in \pkg{irace} format. The configurations are checked -#' to match the parameters description provided. -#' -#' @param filename (`character(1)`) \cr Filename from which the configurations should be read. +#' Read parameter configurations from a file +#' +#' Reads a set of target-algorithm configurations from a file and puts them in +#' \pkg{irace} format. The configurations are checked to match the parameters +#' description provided. +#' +#' @param filename (`character(1)`) \cr Filename from which the configurations should be read. The contents should be readable by `read.table( , header=TRUE)`. #' @template arg_parameters #' @template arg_debuglevel #' @template arg_text @@ -13,25 +13,24 @@ #' Each row of the data frame is a candidate configuration, #' the columns correspond to the parameter names in `parameters`. #' +#' @details +#' Example of an input file: +#' ``` +#' # This is a comment line +#' ... +#' 0.5 "value_1" ... +#' 1.0 "value_2" ... +#' 1.2 "value_3" ... +#' ... ... +#' ``` +#' The order of the columns does not necessarily have to be the same +#' as in the file containing the definition of the parameters. +#' #' @seealso -#' \code{\link{readParameters}} to obtain a valid parameter structure from a parameters list. +#' [readParameters()] to obtain a valid parameter structure from a parameters file. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export -## Read some configurations from a file. -## Example of an input file, -## it should be readable with read.table( , header=TRUE). -## ------------------------------- -## -## 1 "value_1" -## 2 "value_2" -## # This is a comment line -## 3 "value_3" -## ------------------------------- -## The order of the columns does not necessarily have to be the same -## as in the file containing the definition of the parameters. -## ## FIXME: What about digits? readConfigurationsFile <- function(filename, parameters, debugLevel = 0, text) { @@ -51,7 +50,7 @@ # Print the table that has been read. cat("# Read ", nbConfigurations, " configuration(s) from file '", filename, "'\n", sep="") if (debugLevel >= 2) { - print(as.data.frame(configurationTable, stringAsFactor = FALSE), digits=15) + print(as.data.frame(configurationTable, stringsAsFactor = FALSE), digits=15) } namesParameters <- names(parameters$conditions) @@ -157,7 +156,12 @@ } } } - return (configurationTable) + if (anyDuplicated(configurationTable)) { + irace.error("Duplicated configurations in file ", filename, " :\n", + paste0(capture.output( + configurationTable[duplicated(configurationTable), , drop=FALSE]), "\n")) + } + configurationTable } # FIXME: It may be faster to create a single expression that concatenates all # the elements of forbidden using '|' @@ -174,7 +178,7 @@ ## would be faster to break as soon as nrow(configurations) < 1 } #print(nrow(configurations)) - return(configurations) + configurations } compile.forbidden <- function(x) @@ -189,10 +193,10 @@ # We expect that there will be undefined variables, since the expressions # will be evaluated within a data.frame later. - exp <- compiler::compile(substitute(is.na(x) | !(x), list(x = x)), + expr <- compiler::compile(substitute(is.na(x) | !(x), list(x = x)), options = list(suppressUndefined=TRUE)) - attr(exp, "source") <- as.character(as.expression(x)) - return(exp) + attr(expr, "source") <- as.character(as.expression(x)) + expr } readForbiddenFile <- function(filename) @@ -206,9 +210,11 @@ # FIXME: Instead of a list, we should generate a single expression that is # the logical-OR of all elements of the list. - + # First we would need to handle the "is.na(x) | !(x)" case here. + # Maybe: sapply(forbiddenExps, function(x) substitute(is.na(x) | !(x), list(x=x))) + # x <- parse(text=paste0("(", paste0(forbiddenExps,collapse=")||("), ")")) # Byte-compile them. - return(sapply(forbiddenExps, compile.forbidden)) + sapply(forbiddenExps, compile.forbidden) } buildForbiddenExp <- function(configurations, parameters) @@ -227,13 +233,10 @@ } exps <- parse(text = lines) # print(exps) - return(sapply(exps, compile.forbidden)) + sapply(exps, compile.forbidden) } -#' readScenario -#' -#' `readScenario` reads from a file the scenario settings to be used by -#' \pkg{irace}.. +#' Reads from a file the scenario settings to be used by \pkg{irace}. #' #' @param filename (`character(1)`) \cr Filename from which the scenario will #' be read. If empty, the default `scenarioFile` is used. An example @@ -242,25 +245,26 @@ #' @templateVar arg_appendix This is an initial scenario that is overwritten #' for every setting specified in the file to be read. #' @template arg_scenario -#' +#' @template arg_params_def +#' #' @return The scenario list read from the file. The scenario settings not #' present in the file are not present in the list, i.e., they are `NULL`. #' #' @seealso #' \describe{ -#' \item{\code{\link{printScenario}}}{prints the given scenario.} -#' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} -#' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} +#' \item{[printScenario()]}{prints the given scenario.} +#' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} +#' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export -readScenario <- function(filename = "", scenario = list()) +readScenario <- function(filename = "", scenario = list(), + params_def = .irace.params.def) { # This function allows recursively including scenario files. - envir <- environment() - include.scenario <- function(rfilename, topfile = filename, envir. = envir) + scenario_env <- new.env() + include.scenario <- function(rfilename, topfile = filename, envir. = scenario_env) { if (!file.exists (rfilename)) { irace.error ("The scenario file ", shQuote(rfilename), " included from ", @@ -280,22 +284,25 @@ # First find out which file... if (filename == "") { - filename <- .irace.params.def["scenarioFile","default"] + filename <- path_rel2abs(params_def["scenarioFile","default"]) if (file.exists(filename)) { - cat("Warning: A default scenario file", shQuote(filename), - "has been found and will be read\n") + irace.warning("A default scenario file ", shQuote(filename), + " has been found and will be read.") } else { irace.error ("Not scenario file given (use ", - .irace.params.def["scenarioFile", "short"], " or ", - .irace.params.def["scenarioFile", "long"], + params_def["scenarioFile", "short"], " or ", + params_def["scenarioFile", "long"], ") and no default scenario file ", shQuote(filename), " has been found.") } - } + } else { + filename <- path_rel2abs(filename) + } + if (file.exists (filename)) { debug.level <- getOption(".irace.debug.level", default = 0) if (debug.level >= 1) - cat ("# Reading scenario file", shQuote(filename), ".......") + cat("# Reading scenario file", shQuote(filename), ".......") # chdir = TRUE to allow recursive sourcing. handle.source.error <- function(e) { irace.error("Reading scenario file ", shQuote(filename), @@ -304,34 +311,72 @@ return(NULL) } withCallingHandlers( - tryCatch(source(filename, local = TRUE, chdir = TRUE), + tryCatch(source(filename, local = scenario_env, chdir = TRUE), error = handle.source.error, warning = handle.source.error)) if (debug.level >= 1) cat (" done!\n") } else { irace.error ("The scenario file ", shQuote(filename), " does not exist.") } + ## Read scenario file variables. + scenario[["scenarioFile"]] <- filename # If these are given and relative, they should be relative to the # scenario file (except logFile, which is relative to execDir). - pathParams <- setdiff(.irace.params.def[.irace.params.def[, "type"] == "p", + pathParams <- setdiff(params_def[params_def[, "type"] == "p", "name"], "logFile") - for (param in .irace.params.names) { - if (exists (param, inherits = FALSE)) { - value <- get(param, inherits = FALSE) + params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] + + for (param in params_names) { + if (exists (param, envir = scenario_env, inherits = FALSE)) { + value <- get(param, envir = scenario_env, inherits = FALSE) if (!is.null.or.empty(value) && is.character(value) && (param %in% pathParams)) { - value <- path.rel2abs(value, cwd = dirname(filename)) + value <- path_rel2abs(value, cwd = dirname(filename)) } scenario[[param]] <- value } } - return (scenario) + unknown_scenario_vars <- setdiff(ls(scenario_env), params_names) + if (length(unknown_scenario_vars) > 0) { + # We only accept variables that match irace.params.names and if the user + # wants to define their own, they should use names starting with ".", which + # are ignored by ls() + irace.error("Scenario file ", shQuote(filename), " contains unknown variables: ", + paste0(unknown_scenario_vars, collapse=", "), + "\nMAKE SURE NO VARIABLE NAME IS MISSPELL (for example, 'parameterFile' is correct, while 'parametersFile' is not)", + "\nIf you wish to use your own variables in the scenario file, use names beginning with a dot `.'") + } + scenario } + +setup_test_instances <- function(scenario) +{ + if (is.null.or.empty(scenario[["testInstances"]])) { + if (!is.null.or.empty(scenario$testInstancesDir) || + !is.null.or.empty(scenario$testInstancesFile)) { + scenario$testInstancesDir <- path_rel2abs(scenario$testInstancesDir) + if (!is.null.or.empty(scenario$testInstancesFile)) { + scenario$testInstancesFile <- path_rel2abs(scenario$testInstancesFile) + } + scenario[["testInstances"]] <- + readInstances(instancesDir = scenario$testInstancesDir, + instancesFile = scenario$testInstancesFile) + } else { + scenario[["testInstances"]] <- NULL + } + } + if (!is.null(scenario[["testInstances"]]) + && is.null(names(scenario[["testInstances"]]))) { + # Create unique IDs for testInstances + names(scenario[["testInstances"]]) <- paste0(1:length(scenario[["testInstances"]]), "t") + } + scenario +} #' Check and correct the given scenario #' -#' `checkScenario` takes a (possibly incomplete) scenario setup of -#' \pkg{irace}, checks for errors and transforms it into a valid scenario. +#' Checks for errors a (possibly incomplete) scenario setup of +#' \pkg{irace} and transforms it into a valid scenario. #' #' @template arg_scenario #' @@ -347,14 +392,13 @@ #' #' @seealso #' \describe{ -#' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} -#' \item{\code{\link{printScenario}}}{prints the given scenario.} -#' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} -#' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} +#' \item{[readScenario()]}{for reading a configuration scenario from a file.} +#' \item{[printScenario()]}{prints the given scenario.} +#' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} +#' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export ## FIXME: This function should only do checks and return TRUE/FALSE. There ## should be other function that does the various transformations. @@ -362,7 +406,10 @@ { quote.param <- function(name) { - return(paste0("'", name, "' (", .irace.params.def[name, "long"], ")")) + if (.irace.params.def[name, "long"] != "") { + return(paste0("'", name, "' (", .irace.params.def[name, "long"], ")")) + } + paste0("'", name, "'") } as.boolean.param <- function(x, name) @@ -371,21 +418,23 @@ if (is.na (x) || (x != 0 && x != 1)) { irace.error (quote.param(name), " must be either 0 or 1.") } - return(as.logical(x)) - } - - check.valid.param <- function(x, valid) + as.logical(x) + } + + check.valid.param <- function(x) { + cat(x, "\n") + cat(.irace.params.def[x, "domain"], "\n") + valid <- trimws(strsplit(.irace.params.def[x, "domain"],",",fixed=TRUE)[[1]]) if (scenario[[x]] %!in% valid) { irace.error ("Invalid value '", scenario[[x]], "' of ", quote.param(x), ", valid values are: ", paste0(valid, collapse = ", ")) } } - - + # Fill possible unset (NULL) with default settings. - scenario <- defaultScenario (scenario) + scenario <- defaultScenario(scenario) # Duplicated entries will cause confusion. dups <- anyDuplicated(names(scenario)) @@ -397,36 +446,37 @@ for (p in boolParams) { scenario[[p]] <- as.boolean.param (scenario[[p]], p) } - + options(.irace.quiet = scenario$quiet) ## Check that everything is fine with external parameters # Check that the files exist and are readable. - scenario$parameterFile <- path.rel2abs(scenario$parameterFile) + scenario$parameterFile <- path_rel2abs(scenario$parameterFile) # We don't read parameterFile here because the user may give the parameters # explicitly. And it is validated in readParameters anyway. - scenario$execDir <- path.rel2abs(scenario$execDir) + scenario$execDir <- path_rel2abs(scenario$execDir) file.check (scenario$execDir, isdir = TRUE, text = paste0("execution directory ", quote.param("execDir"))) options(.irace.execdir = scenario$execDir) if (!is.null.or.empty(scenario$logFile)) { - scenario$logFile <- path.rel2abs(scenario$logFile, cwd = scenario$execDir) - file.check(scenario$logFile, writeable = TRUE, - text = quote.param('logFile')) + scenario$logFile <- path_rel2abs(scenario$logFile, cwd = scenario$execDir) + file.check(scenario$logFile, writeable = TRUE, text = quote.param('logFile')) } else { - scenario$logFile <- NULL + # We cannot use NULL because defaultScenario() would override it. + scenario$logFile <- "" } if (!is.null.or.empty(scenario$recoveryFile)) { - scenario$recoveryFile <- path.rel2abs(scenario$recoveryFile) + scenario$recoveryFile <- path_rel2abs(scenario$recoveryFile) file.check(scenario$recoveryFile, readable = TRUE, text = paste0("recovery file ", quote.param("recoveryFile"))) - if (!is.null.or.empty(scenario$logFile) + if (!is.null.or.empty(scenario$logFile) # Must have been set "" above. && scenario$recoveryFile == scenario$logFile) { - irace.error("log file and recovery file should be different ('", + irace.error("log file and recovery file should be different '", scenario$logFile, "'") } } else { - scenario$recoveryFile <- NULL + # We cannot use NULL because defaultScenario() would override it. + scenario$recoveryFile <- "" } if (is.null.or.empty(scenario$targetRunnerParallel)) { @@ -451,11 +501,18 @@ .irace$target.runner <- bytecompile(scenario$targetRunner) } else if (is.null(scenario$targetRunnerParallel)) { if (is.character(scenario$targetRunner)) { - scenario$targetRunner <- path.rel2abs(scenario$targetRunner) - file.check (scenario$targetRunner, executable = TRUE, - text = paste0("target runner ", quote.param("targetRunner"))) + scenario$targetRunner <- path_rel2abs(scenario$targetRunner) .irace$target.runner <- if (scenario$aclib) target.runner.aclib else target.runner.default + if (is.null.or.empty(scenario$targetRunnerLauncher)) { + file.check (scenario$targetRunner, executable = TRUE, + text = paste0("target runner ", quote.param("targetRunner"))) + } else { + scenario$targetRunnerLauncher <- path_rel2abs(scenario$targetRunnerLauncher) + file.check (scenario$targetRunnerLauncher, executable = TRUE, + text = paste0("target runner launcher ", quote.param("targetRunnerLauncher"))) + check_launcher_args(scenario$targetRunnerLauncherArgs) + } } else { irace.error(quote.param ('targetRunner'), " must be a function or an executable program") } @@ -468,21 +525,21 @@ scenario$targetEvaluator <- get.function(scenario$targetEvaluator) .irace$target.evaluator <- bytecompile(scenario$targetEvaluator) } else if (is.character(scenario$targetEvaluator)) { - scenario$targetEvaluator <- path.rel2abs(scenario$targetEvaluator) + scenario$targetEvaluator <- path_rel2abs(scenario$targetEvaluator) file.check (scenario$targetEvaluator, executable = TRUE, text = "target evaluator") .irace$target.evaluator <- target.evaluator.default } else { - irace.error("'targetEvaluator' must be a function or an executable program") + irace.error(quote.param('targetEvaluator'), " must be a function or an executable program") } irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) # Training instances if (is.null.or.empty(scenario$instances)) { - scenario$trainInstancesDir <- path.rel2abs(scenario$trainInstancesDir) + scenario$trainInstancesDir <- path_rel2abs(scenario$trainInstancesDir) if (!is.null.or.empty(scenario$trainInstancesFile)) { - scenario$trainInstancesFile <- path.rel2abs(scenario$trainInstancesFile) + scenario$trainInstancesFile <- path_rel2abs(scenario$trainInstancesFile) } if (is.null.or.empty(scenario$trainInstancesDir) && is.null.or.empty(scenario$trainInstancesFile)) @@ -496,29 +553,11 @@ } # Testing instances - if (is.null.or.empty(scenario$testInstances)) { - if (!is.null.or.empty(scenario$testInstancesDir) || - !is.null.or.empty(scenario$testInstancesFile)) { - scenario$testInstancesDir <- path.rel2abs(scenario$testInstancesDir) - if (!is.null.or.empty(scenario$testInstancesFile)) { - scenario$testInstancesFile <- path.rel2abs(scenario$testInstancesFile) - } - scenario$testInstances <- - readInstances(instancesDir = scenario$testInstancesDir, - instancesFile = scenario$testInstancesFile) - } else { - scenario$testInstances <- NULL - } - } - if (!is.null(scenario$testInstances) - && is.null(names(scenario$testInstances))) { - # Create unique IDs for testInstances - names(scenario$testInstances) <- paste0(1:length(scenario$testInstances), "t") - } - + scenario <- setup_test_instances(scenario) + # Configurations file if (!is.null.or.empty(scenario$configurationsFile)) { - scenario$configurationsFile <- path.rel2abs(scenario$configurationsFile) + scenario$configurationsFile <- path_rel2abs(scenario$configurationsFile) file.check (scenario$configurationsFile, readable = TRUE, text = "configurations file") # We cannot read the configurations here because we need the parameters. @@ -535,18 +574,18 @@ # the user specified them explicitly. if (is.null.or.empty(scenario$forbiddenExps) && !is.null.or.empty(scenario$forbiddenFile)) { - scenario$forbiddenFile <- path.rel2abs(scenario$forbiddenFile) + scenario$forbiddenFile <- path_rel2abs(scenario$forbiddenFile) file.check (scenario$forbiddenFile, readable = TRUE, text = "forbidden configurations file") scenario$forbiddenExps <- readForbiddenFile(scenario$forbiddenFile) - cat("# ", length(scenario$forbiddenExps), - " expression(s) specifying forbidden configurations read from '", - scenario$forbiddenFile, "'\n", sep = "") + irace.note(length(scenario$forbiddenExps), + " expression(s) specifying forbidden configurations read from '", + scenario$forbiddenFile, "'\n") } # Make it NULL if it is "" or NA # FIXME: If it is a non-empty vector of strings, parse them as above. - if (is.null.or.empty(scenario$forbiddenExps) || is.null.or.na(scenario$forbiddenExps)) + if (is_null_or_empty_or_na(scenario$forbiddenExps)) scenario$forbiddenExps <- NULL # We have characters everywhere, set to the right types to avoid @@ -577,7 +616,7 @@ if (scenario$mu < scenario$firstTest) { if (scenario$debugLevel >= 1) { - cat("Warning: Assuming 'mu = firstTest' because 'mu' cannot be lower than 'firstTest'\n") + irace.warning("Assuming 'mu = firstTest' because 'mu' cannot be lower than 'firstTest'\n") } scenario$mu <- scenario$firstTest } @@ -639,12 +678,12 @@ " must be larger than 1 when mpi is enabled.") } + if (is.null.or.empty(scenario$batchmode)) scenario$batchmode <- 0 if (scenario$batchmode != 0) { scenario$batchmode <- tolower(scenario$batchmode) - # FIXME: We should encode options in the large table in main.R - check.valid.param("batchmode", valid = c("sge", "pbs", "torque", "slurm")) + check.valid.param("batchmode") } # Currently batchmode requires a targetEvaluator if (scenario$batchmode != 0 && is.null(scenario$targetEvaluator)) { @@ -663,25 +702,18 @@ if (scenario$boundMax <= 0) irace.error("When capping == TRUE, boundMax (", scenario$boundMax, ") must be > 0") - check.valid.param("cappingType", - valid = c("median", "mean", "worst", "best")) - check.valid.param("boundType", valid = c("instance", "candidate")) - + check.valid.param("cappingType") + check.valid.param("boundType") if (scenario$boundPar < 1) - irace.error("Invalid value boundPar (", scenario$boundPar, - ") must be >= 1") - } else { # no capping - if (scenario$boundMax <= 0 || is.na(scenario$boundMax)) - scenario$boundMax <- NULL - } - - - - if (is.null.or.empty(scenario$testType) || - is.null.or.na(scenario$testType) || - scenario$testType =="") { - if (scenario$capping) scenario$testType <- "t-test" - else scenario$testType <- "f-test" + irace.error("Invalid value (", scenario$boundPar, ") ", + quote.param("boundPar"), " must be >= 1") + } else if (scenario$boundMax <= 0 || is.na(scenario$boundMax)) { # no capping + scenario$boundMax <- NULL + } + + if (is_null_or_empty_or_na(scenario$testType)) { + if (scenario$capping) scenario$testType <- "t-test" + else scenario$testType <- "f-test" } scenario$testType <- @@ -694,10 +726,8 @@ "t.holm" = "t.holm", "t-test-bonferroni" =, # Fall-through, "t.bonferroni" = "t.bonferroni", - check.valid.param("testType", - c("F-test, t-test, t-test-holm, t-test-bonferroni"))) - - return (scenario) + check.valid.param("testType")) + scenario } #' Prints the given scenario @@ -706,19 +736,19 @@ #' #' @seealso #' \describe{ -#' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} -#' \item{\code{\link{printScenario}}}{prints the given scenario.} -#' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} -#' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} +#' \item{[readScenario()]}{for reading a configuration scenario from a file.} +#' \item{[printScenario()]}{prints the given scenario.} +#' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} +#' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export printScenario <- function(scenario) { + params_names <- .irace.params.names cat("## irace scenario:\n") - for (param in .irace.params.names) { + for (param in params_names) { if (param == "forbiddenExps") extra <- paste0(" = expression(", paste0(collapse=", ", sapply(scenario[[param]], attr, "source")), ")") @@ -730,9 +760,11 @@ #' Default scenario settings #' -#' Return scenario with default values. +#' Return scenario object with default values. #' #' @template arg_scenario +#' +#' @template arg_params_def #' #' @return A list indexed by the \pkg{irace} parameter names, #' containing the default values for each parameter, except for those @@ -742,127 +774,133 @@ #' \itemize{ #' \item General options: #' \describe{ -#' \item{\code{scenarioFile}}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: \code{"./scenario.txt"})} -#' \item{\code{execDir}}{Directory where the programs will be run. (Default: \code{"./"})} -#' \item{\code{logFile}}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: \code{"./irace.Rdata"})} -#' \item{\code{debugLevel}}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: \code{0})} -#' \item{\code{seed}}{Seed of the random number generator (by default, generate a random seed). (Default: \code{NA})} -#' \item{\code{repairConfiguration}}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: \code{""})} -#' \item{\code{postselection}}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: \code{0})} -#' \item{\code{aclib}}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: \code{0})} -#' } -#' \item Elitist \code{irace}: -#' \describe{ -#' \item{\code{elitist}}{Enable/disable elitist irace. (Default: \code{1})} -#' \item{\code{elitistNewInstances}}{Number of instances added to the execution list before previous instances in elitist irace. (Default: \code{1})} -#' \item{\code{elitistLimit}}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: \code{2})} -#' } -#' \item Internal \code{irace} options: -#' \describe{ -#' \item{\code{nbIterations}}{Number of iterations. (Default: \code{0})} -#' \item{\code{nbExperimentsPerIteration}}{Number of runs of the target algorithm per iteration. (Default: \code{0})} -#' \item{\code{sampleInstances}}{Randomly sample the training instances or use them in the order given. (Default: \code{1})} -#' \item{\code{minNbSurvival}}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: \code{0})} -#' \item{\code{nbConfigurations}}{Number of configurations to be sampled and evaluated at each iteration. (Default: \code{0})} -#' \item{\code{mu}}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: \code{5})} -#' \item{\code{softRestart}}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: \code{1})} -#' \item{\code{softRestartThreshold}}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: \code{""})} +#' \item{`scenarioFile`}{Path of the file that describes the configuration scenario setup and other irace settings. (Default: `"./scenario.txt"`)} +#' \item{`execDir`}{Directory where the programs will be run. (Default: `"./"`)} +#' \item{`logFile`}{File to save tuning results as an R dataset, either absolute path or relative to execDir. (Default: `"./irace.Rdata"`)} +#' \item{`quiet`}{Reduce the output generated by irace to a minimum. (Default: `0`)} +#' \item{`debugLevel`}{Debug level of the output of \code{irace}. Set this to 0 to silence all debug messages. Higher values provide more verbose debug messages. (Default: `0`)} +#' \item{`seed`}{Seed of the random number generator (by default, generate a random seed). (Default: `NA`)} +#' \item{`repairConfiguration`}{User-defined R function that takes a configuration generated by irace and repairs it. (Default: `""`)} +#' \item{`postselection`}{Percentage of the configuration budget used to perform a postselection race of the best configurations of each iteration after the execution of irace. (Default: `0`)} +#' \item{`aclib`}{Enable/disable AClib mode. This option enables compatibility with GenericWrapper4AC as targetRunner script. (Default: `0`)} +#' } +#' \item Elitist `irace`: +#' \describe{ +#' \item{`elitist`}{Enable/disable elitist irace. (Default: `1`)} +#' \item{`elitistNewInstances`}{Number of instances added to the execution list before previous instances in elitist irace. (Default: `1`)} +#' \item{`elitistLimit`}{In elitist irace, maximum number per race of elimination tests that do not eliminate a configuration. Use 0 for no limit. (Default: `2`)} +#' } +#' \item Internal `irace` options: +#' \describe{ +#' \item{`sampleInstances`}{Randomly sample the training instances or use them in the order given. (Default: `1`)} +#' \item{`softRestart`}{Enable/disable the soft restart strategy that avoids premature convergence of the probabilistic model. (Default: `1`)} +#' \item{`softRestartThreshold`}{Soft restart threshold value for numerical parameters. If \code{NA}, \code{NULL} or \code{""}, it is computed as \code{10^-digits}. (Default: `""`)} +#' \item{`nbIterations`}{Maximum number of iterations. (Default: `0`)} +#' \item{`nbExperimentsPerIteration`}{Number of runs of the target algorithm per iteration. (Default: `0`)} +#' \item{`minNbSurvival`}{Minimum number of configurations needed to continue the execution of each race (iteration). (Default: `0`)} +#' \item{`nbConfigurations`}{Number of configurations to be sampled and evaluated at each iteration. (Default: `0`)} +#' \item{`mu`}{Parameter used to define the number of configurations sampled and evaluated at each iteration. (Default: `5`)} #' } #' \item Target algorithm parameters: #' \describe{ -#' \item{\code{parameterFile}}{File that contains the description of the parameters of the target algorithm. (Default: \code{"./parameters.txt"})} -#' \item{\code{forbiddenExps}}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: \code{""})} -#' \item{\code{forbiddenFile}}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: \code{""})} -#' \item{\code{digits}}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: \code{4})} +#' \item{`parameterFile`}{File that contains the description of the parameters of the target algorithm. (Default: `"./parameters.txt"`)} +#' \item{`forbiddenExps`}{Vector of R logical expressions that cannot evaluate to \code{TRUE} for any evaluated configuration. (Default: `""`)} +#' \item{`forbiddenFile`}{File that contains a list of logical expressions that cannot be \code{TRUE} for any evaluated configuration. If empty or \code{NULL}, do not use forbidden expressions. (Default: `""`)} +#' \item{`digits`}{Maximum number of decimal places that are significant for numerical (real) parameters. (Default: `4`)} #' } #' \item Target algorithm execution: #' \describe{ -#' \item{\code{targetRunner}}{Script called for each configuration that executes the target algorithm to be tuned. See templates. (Default: \code{"./target-runner"})} -#' \item{\code{targetRunnerRetries}}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: \code{0})} -#' \item{\code{targetRunnerData}}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: \code{""})} -#' \item{\code{targetRunnerParallel}}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: \code{""})} -#' \item{\code{targetEvaluator}}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: \code{""})} -#' \item{\code{deterministic}}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: \code{0})} -#' \item{\code{parallel}}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: \code{0})} -#' \item{\code{loadBalancing}}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: \code{1})} -#' \item{\code{mpi}}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: \code{0})} -#' \item{\code{batchmode}}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque or slurm. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: \code{0})} +#' \item{`targetRunner`}{Executable called for each configuration that executes the target algorithm to be tuned. See the templates and examples provided. (Default: `"./target-runner"`)} +#' \item{`targetRunnerLauncher`}{Executable that will be used to launch the target runner, when \code{targetRunner} cannot be executed directly (.e.g, a Python script in Windows). (Default: `""`)} +#' \item{`targetRunnerLauncherArgs`}{Command-line arguments provided to \code{targetRunnerLauncher}. The substrings \code{\{targetRunner\}} and \code{\{targetRunnerArgs\}} will be replaced by the value of the option \code{targetRunner} and by the arguments usually passed when calling \code{targetRunner}, respectively. Example: \code{"-m {targetRunner} --args {targetRunnerArgs}"}. (Default: `"{targetRunner} {targetRunnerArgs}"`)} +#' \item{`targetRunnerRetries`}{Number of times to retry a call to \code{targetRunner} if the call failed. (Default: `0`)} +#' \item{`targetRunnerData`}{Optional data passed to \code{targetRunner}. This is ignored by the default \code{targetRunner} function, but it may be used by custom \code{targetRunner} functions to pass persistent data around. (Default: `""`)} +#' \item{`targetRunnerParallel`}{Optional R function to provide custom parallelization of \code{targetRunner}. (Default: `""`)} +#' \item{`targetEvaluator`}{Optional script or R function that provides a numeric value for each configuration. See templates/target-evaluator.tmpl (Default: `""`)} +#' \item{`deterministic`}{If the target algorithm is deterministic, configurations will be evaluated only once per instance. (Default: `0`)} +#' \item{`parallel`}{Number of calls to \code{targetRunner} to execute in parallel. Values \code{0} or \code{1} mean no parallelization. (Default: `0`)} +#' \item{`loadBalancing`}{Enable/disable load-balancing when executing experiments in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Default: `1`)} +#' \item{`mpi`}{Enable/disable MPI. Use \code{Rmpi} to execute \code{targetRunner} in parallel (parameter \code{parallel} is the number of slaves). (Default: `0`)} +#' \item{`batchmode`}{Specify how irace waits for jobs to finish when \code{targetRunner} submits jobs to a batch cluster: sge, pbs, torque, slurm or htcondor. \code{targetRunner} must submit jobs to the cluster using, for example, \code{qsub}. (Default: `0`)} #' } #' \item Initial configurations: #' \describe{ -#' \item{\code{initConfigurations}}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: \code{""})} -#' \item{\code{configurationsFile}}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: \code{""})} +#' \item{`initConfigurations`}{Data frame describing initial configurations (usually read from a file using \code{readConfigurations}). (Default: `""`)} +#' \item{`configurationsFile`}{File that contains a table of initial configurations. If empty or \code{NULL}, all initial configurations are randomly generated. (Default: `""`)} #' } #' \item Training instances: #' \describe{ -#' \item{\code{instances}}{Character vector of the instances to be used in the \code{targetRunner}. (Default: \code{""})} -#' \item{\code{trainInstancesDir}}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: \code{"./Instances"})} -#' \item{\code{trainInstancesFile}}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: \code{""})} +#' \item{`instances`}{Character vector of the instances to be used in the \code{targetRunner}. (Default: `""`)} +#' \item{`trainInstancesDir`}{Directory where training instances are located; either absolute path or relative to current directory. If no \code{trainInstancesFiles} is provided, all the files in \code{trainInstancesDir} will be listed as instances. (Default: `"./Instances"`)} +#' \item{`trainInstancesFile`}{File that contains a list of training instances and optionally additional parameters for them. If \code{trainInstancesDir} is provided, \code{irace} will search for the files in this folder. (Default: `""`)} #' } #' \item Tuning budget: #' \describe{ -#' \item{\code{maxExperiments}}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: \code{0})} -#' \item{\code{maxTime}}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: \code{0})} -#' \item{\code{budgetEstimation}}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: \code{0.02})} +#' \item{`maxExperiments`}{Maximum number of runs (invocations of \code{targetRunner}) that will be performed. It determines the maximum budget of experiments for the tuning. (Default: `0`)} +#' \item{`maxTime`}{Maximum total execution time in seconds for the executions of \code{targetRunner}. \code{targetRunner} must return two values: cost and time. (Default: `0`)} +#' \item{`budgetEstimation`}{Fraction (smaller than 1) of the budget used to estimate the mean computation time of a configuration. Only used when \code{maxTime} > 0 (Default: `0.02`)} +#' \item{`minMeasurableTime`}{Minimum time unit that is still (significantly) measureable. (Default: `0.01`)} #' } #' \item Statistical test: #' \describe{ -#' \item{\code{testType}}{Statistical test used for elimination. Default test is always \code{F-test} unless \code{capping} is enabled, in which case the default test is \code{t-test}. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: \code{"F-test"})} -#' \item{\code{firstTest}}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: \code{5})} -#' \item{\code{eachTest}}{Number of instances evaluated between elimination tests. (Default: \code{1})} -#' \item{\code{confidence}}{Confidence level for the elimination test. (Default: \code{0.95})} +#' \item{`testType`}{Statistical test used for elimination. The default value selects \code{t-test} if \code{capping} is enabled or \code{F-test}, otherwise. Valid values are: F-test (Friedman test), t-test (pairwise t-tests with no correction), t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons). (Default: `""`)} +#' \item{`firstTest`}{Number of instances evaluated before the first elimination test. It must be a multiple of \code{eachTest}. (Default: `5`)} +#' \item{`eachTest`}{Number of instances evaluated between elimination tests. (Default: `1`)} +#' \item{`confidence`}{Confidence level for the elimination test. (Default: `0.95`)} #' } #' \item Adaptive capping: #' \describe{ -#' \item{\code{capping}}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: \code{0})} -#' \item{\code{cappingType}}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: \code{"median"})} -#' \item{\code{boundType}}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: \code{"candidate"})} -#' \item{\code{boundMax}}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: \code{0})} -#' \item{\code{boundDigits}}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: \code{0})} -#' \item{\code{boundPar}}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: \code{1})} -#' \item{\code{boundAsTimeout}}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: \code{1})} +#' \item{`capping`}{Enable the use of adaptive capping, a technique designed for minimizing the computation time of configurations. This is only available when \code{elitist} is active. (Default: `0`)} +#' \item{`cappingType`}{Measure used to obtain the execution bound from the performance of the elite configurations.\itemize{\item median: Median performance of the elite configurations.\item mean: Mean performance of the elite configurations.\item best: Best performance of the elite configurations.\item worst: Worst performance of the elite configurations.} (Default: `"median"`)} +#' \item{`boundType`}{Method to calculate the mean performance of elite configurations.\itemize{\item candidate: Mean execution times across the executed instances and the current one.\item instance: Execution time of the current instance.} (Default: `"candidate"`)} +#' \item{`boundMax`}{Maximum execution bound for \code{targetRunner}. It must be specified when capping is enabled. (Default: `0`)} +#' \item{`boundDigits`}{Precision used for calculating the execution time. It must be specified when capping is enabled. (Default: `0`)} +#' \item{`boundPar`}{Penalization constant for timed out executions (executions that reach \code{boundMax} execution time). (Default: `1`)} +#' \item{`boundAsTimeout`}{Replace the configuration cost of bounded executions with \code{boundMax}. (Default: `1`)} #' } #' \item Recovery: #' \describe{ -#' \item{\code{recoveryFile}}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: \code{""})} +#' \item{`recoveryFile`}{Previously saved log file to recover the execution of \code{irace}, either absolute path or relative to the current directory. If empty or \code{NULL}, recovery is not performed. (Default: `""`)} #' } #' \item Testing: #' \describe{ -#' \item{\code{testInstancesDir}}{Directory where testing instances are located, either absolute or relative to current directory. (Default: \code{""})} -#' \item{\code{testInstancesFile}}{File containing a list of test instances and optionally additional parameters for them. (Default: \code{""})} -#' \item{\code{testInstances}}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: \code{""})} -#' \item{\code{testNbElites}}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: \code{1})} -#' \item{\code{testIterationElites}}{Enable/disable testing the elite configurations found at each iteration. (Default: \code{0})} +#' \item{`testInstancesDir`}{Directory where testing instances are located, either absolute or relative to current directory. (Default: `""`)} +#' \item{`testInstancesFile`}{File containing a list of test instances and optionally additional parameters for them. (Default: `""`)} +#' \item{`testInstances`}{Character vector of the instances to be used in the \code{targetRunner} when executing the testing. (Default: `""`)} +#' \item{`testNbElites`}{Number of elite configurations returned by irace that will be tested if test instances are provided. (Default: `1`)} +#' \item{`testIterationElites`}{Enable/disable testing the elite configurations found at each iteration. (Default: `0`)} #' } #' } # __IRACE_OPTIONS__END__ #' #' @seealso #' \describe{ -#' \item{\code{\link{readScenario}}}{for reading a configuration scenario from a file.} -#' \item{\code{\link{printScenario}}}{prints the given scenario.} -#' \item{\code{\link{defaultScenario}}}{returns the default scenario settings of \pkg{irace}.} -#' \item{\code{\link{checkScenario}}}{to check that the scenario is valid.} +#' \item{[readScenario()]}{for reading a configuration scenario from a file.} +#' \item{[printScenario()]}{prints the given scenario.} +#' \item{[defaultScenario()]}{returns the default scenario settings of \pkg{irace}.} +#' \item{[checkScenario()]}{to check that the scenario is valid.} #' } #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export -defaultScenario <- function(scenario = list()) -{ - if (!is.null(names(scenario)) - && !all(names(scenario) %in% .irace.params.names)) { +defaultScenario <- function(scenario = list(), + params_def = .irace.params.def) +{ + params_names <- params_def[!startsWith(params_def[,"name"], "."), "name"] + if (is.null(names(scenario))) { + scenario <- setNames(as.list(params_def[params_names,"default"]), params_names) + } else if (!all(names(scenario) %in% params_names)) { irace.error("Unknown scenario parameters: ", - paste(names(scenario)[which(!names(scenario) - %in% .irace.params.names)], + paste(names(scenario)[!(names(scenario) %in% params_names)], collapse = ", ")) - } - - for (k in .irace.params.names) { - if (is.null.or.na(scenario[[k]])) { - scenario[[k]] <- .irace.params.def[k, "default"] - } - } - return (scenario) + } else { + for (k in params_names) { + if (is.null.or.na(scenario[[k]])) { + scenario[[k]] <- params_def[k, "default"] + } + } + } + scenario } readInstances <- function(instancesDir = NULL, instancesFile = NULL) @@ -879,8 +917,7 @@ instances <- sub("#.*$", "", instances) # Remove comments instances <- sub("^[[:space:]]+", "", instances) # Remove leading whitespace instances <- instances[instances != ""] # Delete empty lines - # FIXME: is.null.or.empty should handle length() == 0. - if (is.null.or.empty(instances) || length(instances) == 0) + if (is.null.or.empty(instances)) irace.error("No instances found in '", instancesFile, "' (whitespace and comments starting with '#' are ignored)") if (!is.null.or.empty(instancesDir)) @@ -895,26 +932,30 @@ if (length (instances) == 0) irace.error("No instances found in `", instancesDir, "'") } - - return(instances) + instances } ## Check targetRunner execution checkTargetFiles <- function(scenario, parameters) { - result <- TRUE ## Create two random configurations conf.id <- c("testConfig1", "testConfig2") configurations <- sampleUniform(parameters, length(conf.id), digits = scenario$digits, forbidden = scenario$forbiddenExps, repair = scenario$repairConfiguration) - configurations <- cbind (.ID. = conf.id, configurations) - + configurations <- cbind(.ID. = conf.id, configurations, stringsAsFactors=FALSE) + + # Read initial configurations provided by the user. + initConfigurations <- allConfigurationsInit(scenario, parameters) + if (nrow(initConfigurations) > 0L) { + irace.assert(all(colnames(configurations) == colnames(initConfigurations))) + configurations <- rbind(configurations, initConfigurations) + } bounds <- rep(scenario$boundMax, nrow(configurations)) instances.ID <- if (scenario$sampleInstances) - sample.int(length(scenario$instances), 1) else 1 + sample.int(length(scenario$instances), 1L) else 1L experiments <- createExperimentList( configurations, parameters, instances = scenario$instances, instances.ID = instances.ID, seeds = 1234567, scenario, bounds = bounds) @@ -925,6 +966,7 @@ # FIXME: Create a function try.call(err.msg,warn.msg, fun, ...) # Executing targetRunner cat("# Executing targetRunner (", nrow(configurations), "times)...\n") + result <- TRUE output <- withCallingHandlers( tryCatch(execute.experiments(experiments, scenario), error = function(e) { @@ -945,7 +987,8 @@ } irace.assert(is.null(scenario$targetEvaluator) == is.null(.irace$target.evaluator)) - + if (!result) return(FALSE) + if (!is.null(scenario$targetEvaluator)) { cat("# Executing targetEvaluator...\n") output <- withCallingHandlers( @@ -966,7 +1009,7 @@ print(output, digits = 15) } } - return(result) + result } diff --git a/R/readParameters.R b/R/readParameters.R index dfaf154..f57e143 100644 --- a/R/readParameters.R +++ b/R/readParameters.R @@ -1,7 +1,5 @@ -#' readParameters -#' -#' `readParameters` reads the parameters to be tuned by -#' \pkg{irace} from a file or directly from a character string. +#' Reads the parameters to be tuned by \pkg{irace} from a file or from a +#' character string. #' #' @param file (`character(1)`) \cr Filename containing the definitions of #' the parameters to be tuned. @@ -9,7 +7,7 @@ #' parameters. #' @template arg_debuglevel #' @template arg_text -#' +#' #' @return A list containing the definitions of the parameters read. The list is #' structured as follows: #' \describe{ @@ -29,6 +27,10 @@ #' \item{`nbParameters`}{An integer, the total number of parameters.} #' \item{`nbFixed`}{An integer, the number of parameters with a fixed value.} #' \item{`nbVariable`}{Number of variable (to be tuned) parameters.} +#' \item{`depends`}{List of character vectors, each vector specifies +#' which parameters depend on this one.} +#' \item{`isDependent`}{Logical vector that specifies which parameter has +#' a dependent domain.} #' } #' #' @details Either `file` or `text` must be given. If `file` is given, the @@ -51,26 +53,25 @@ #' @examples #' ## Read the parameters directly from text #' parameters.table <- ' -#' # name switch type values [conditions (using R syntax)] -#' algorithm "--" c (as,mmas,eas,ras,acs) -#' localsearch "--localsearch " c (0, 1, 2, 3) -#' alpha "--alpha " r (0.00, 5.00) -#' beta "--beta " r (0.00, 10.00) -#' rho "--rho " r (0.01, 1.00) -#' ants "--ants " i (5, 100) -#' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" -#' rasrank "--rasranks " i (1, 100) | algorithm == "ras" -#' elitistants "--elitistants " i (1, 750) | algorithm == "eas" -#' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) -#' dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) +#' # name switch type values [conditions (using R syntax)] +#' algorithm "--" c (as,mmas,eas,ras,acs) +#' localsearch "--localsearch " c (0, 1, 2, 3) +#' alpha "--alpha " r (0.00, 5.00) +#' beta "--beta " r (0.00, 10.00) +#' rho "--rho " r (0.01, 1.00) +#' ants "--ants " i,log (5, 100) +#' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" +#' rasrank "--rasranks " i (1, "min(ants, 10)") | algorithm == "ras" +#' elitistants "--elitistants " i (1, ants) | algorithm == "eas" +#' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) +#' dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) #' ' #' parameters <- readParameters(text=parameters.table) #' str(parameters) #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste -#' @md #' @export -readParameters <- function(file, digits = 4, debugLevel = 0, text) +readParameters <- function (file, digits = 4, debugLevel = 0, text) { if (missing(file) && !missing(text)) { filename <- strcat("text=", deparse(substitute(text))) @@ -167,7 +168,7 @@ # The following line detects cycles if (child == rootParam) irace.error("A cycle detected in subordinate parameters! ", - "Check definition of conditions.\n", + "Check definition of conditions and/or dependent domains.\n", "One parameter of this cycle is '", rootParam, "'") # The following line detects a missing definition @@ -197,6 +198,12 @@ transform.domain <- function(transf, domain, type) { if (transf == "") return(transf) + + # We do not support transformation of dependent parameters, yet + # TODO: think about dependent domain transfomation + if (is.expression(domain)) + irace.error("Parameter domain transformations are not yet available for", + " dependent parameter domains.") lower <- domain[1] upper <- domain[2] @@ -217,14 +224,50 @@ } irace.internal.error("unrecognized transformation type '", transf, "'") } - + + # Checks that variables in the expressions are within + # the parameters names. + check_parameter_dependencies <- function (parameters) { + for (p in names(Filter(length, parameters$depends))) { + vars <- parameters$depends[[p]] + flag <- vars %in% parameters$names + if (!all(flag)) { + irace.error ("Domain (", paste0(parameters$domain[[p]], collapse=", "), + ") of parameter '", p, "' is not valid: '", + paste0(vars[!flag], collapse=", "), + "' cannot be found in the scenario parameters: ", + paste0(parameters$names, collapse=" , ")," .") + } + flag <- parameters$types[vars] %in% c("i", "r") + if (!all(flag)) { + irace.error ("Domain of parameter '", p, "' depends on non-numerical", + " parameters: ", paste0(vars[!flag], collapse=", "), " .") + } + + # Supported operations for dependent domains + allowed.fx <- c("+", "-", "*", "/", "%%", "min", "max", "round", "floor", "ceiling", "trunc") + fx <- setdiff(all.names(parameters$domain[[p]], unique=TRUE), + all.vars(parameters$domain[[p]], unique=TRUE)) + flag <- fx %in% allowed.fx + if (!all(flag)) { + irace.error ("Domain of parameter '", p, "' uses function(s) ", + "not yet supported by irace: ", + paste0(fx[!flag], collapse=", "), " .") + } + } + return(TRUE) + } + parameters <- list(names = character(0), types = character(0), switches = character(0), domain = list(), conditions = list(), isFixed = logical(0), - transform = list()) + # FIXME: This has to be a list because we assign + # attributes to elements. + transform = list(), + isDependent = logical(0)) conditions <- list() lines <- readLines(con = file) @@ -282,43 +325,55 @@ } ## Match param.value (delimited by parenthesis) - result <- field.match (line, "\\([^)]+\\)", delimited = TRUE, sep = "") + # Regexp to detect dependent domains of the type ("min(p1)", 100) + result <- field.match (line, "\\([^|]+\\)", delimited = TRUE, sep = "") param.value <- result$match line <- result$line - if (is.null (result$match)) { + if (is.null (param.value)) { errReadParameters (filename, nbLines, line, "Allowed values must be a list within parenthesis") } - param.value <- string2vector(param.value) + # For numerical parameters domains could be dependent + # thus, we keep the string values in a variable + # for example (10, param1+2) + param.value.str <- string2vector(param.value) if (param.type %in% c("r","i")) { - param.value <- suppressWarnings(as.numeric(param.value)) - if (any(is.na(param.value)) || length(param.value) != 2) { + # For dependent domains param.value will be NA (we will parse + # it later) + param.value <- suppressWarnings(as.numeric(param.value.str)) + if (length(param.value) != 2) { errReadParameters (filename, nbLines, NULL, "incorrect numeric range (", result$match, ") for parameter '", param.name, "'") - } else if (param.value[1] > param.value[2]) { - errReadParameters (filename, nbLines, NULL, - "lower bound must be smaller than upper bound in numeric range (", - result$match, ") for parameter '", param.name, "'") } if (param.type == "r") { # FIXME: Given (0.01,0.99) and digits=1, this produces (0, 1), which is # probably not what the user wants. param.value <- round(param.value, digits = digits) - } else if (param.type == "i" && !all(is.wholenumber(param.value))) { + } else if (param.type == "i" && any(!is.wholenumber(param.value[!is.na(param.value)]))) { + errReadParameters (filename, nbLines, NULL, + "for parameter type 'i' values must be integers (", + result$match, ") for parameter '", param.name, "'") + } + + # Time to parse dependent domains or check values + if (any(is.na(param.value))) { + try(param.value[is.na(param.value)] <- parse(text=param.value.str[is.na(param.value)])) + } else if (param.value[1] >= param.value[2]) { errReadParameters (filename, nbLines, NULL, - "for parameter type 'i' values must be integers (", + "lower bound must be smaller than upper bound in numeric range (", result$match, ") for parameter '", param.name, "'") } - + param.transform <- transform.domain(param.transform, param.value, param.type) if (is.null(param.transform)) { errReadParameters (filename, nbLines, NULL, "The domain of parameter '", param.name, "' of type 'log' cannot contain zero") } } else { + param.value <- param.value.str if (anyDuplicated(param.value)) { dups <- duplicated(param.value) errReadParameters (filename, nbLines, NULL, @@ -385,11 +440,30 @@ "check that the parameter file is not empty") } + # Generate dependency flag + # FIXME: check if we really need this vector + parameters$isDependent <- sapply(parameters$domain, is.expression) + + names(parameters$types) <- + names(parameters$switches) <- + names(parameters$domain) <- + names(parameters$isFixed) <- + names(parameters$transform) <- + names(parameters$isDependent) <- parameters$names + # Obtain the variables in each condition ## FIXME: In R 3.2, all.vars does not work with byte-compiled expressions, ## thus we do not byte-compile them; but we could use ## all.vars(.Internal(disassemble(condition))[[3]][[1]]) - parameters$depends <- lapply(conditions, all.vars) + ## LESLIE: should we make then an all.vars in utils.R so we can + ## use it without problems? + parameters$depends <- lapply(parameters$domain, all.vars) + # Check that dependencies are ok + check_parameter_dependencies(parameters) + # Merge dependencies and conditions + parameters$depends <- Map(c, parameters$depends, lapply(conditions, all.vars)) + parameters$depends <- lapply(parameters$depends, unique) + # Sort parameters in 'conditions' in the proper order according to # conditions hierarchyLevel <- sapply(parameters$names, treeLevel, @@ -397,19 +471,11 @@ parameters$hierarchy <- hierarchyLevel parameters$conditions <- conditions[order(hierarchyLevel)] - names(parameters$types) <- - names(parameters$switches) <- - names(parameters$domain) <- - names(parameters$isFixed) <- - names(parameters$hierarchy) <- - names(parameters$transform) <- parameters$names + names(parameters$hierarchy) <- parameters$names # Print the hierarchy vector: if (debugLevel >= 1) { cat ("# --- Parameters Hierarchy ---\n") - print(paste0(names(parameters$hierarchy))) - print(parameters$hierarchy) - print(sapply(parameters$depends, paste0, collapse=", ")) print(data.frame(Parameter = paste0(names(parameters$hierarchy)), Level = parameters$hierarchy, "Depends on" = sapply(parameters$depends, paste0, collapse=", "), @@ -422,12 +488,13 @@ parameters$nbParameters <- length(parameters$names) parameters$nbFixed <- sum(parameters$isFixed == TRUE) parameters$nbVariable <- sum(parameters$isFixed == FALSE) - if (debugLevel >= 2) print(parameters, digits = 15) - return (parameters) + if (debugLevel >= 2) { + print(parameters, digits = 15) + irace.note("Parameters have been read\n") + } + parameters } -#' read_pcs_file -#' #' Read parameters in PCS (AClib) format and write them in irace format. #' #' @param file (`character(1)`) \cr Filename containing the definitions of @@ -447,10 +514,15 @@ #' for details. If none of these parameters is given, \pkg{irace} #' will stop with an error. #' +#' **FIXME:** Forbidden configurations, default configuration and transformations ("log") are currently ignored. See +#' +#' @references +#' Frank Hutter, Manuel López-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brown, and Thomas Stützle. **AClib: A Benchmark Library for Algorithm Configuration**. In P. M. Pardalos, M. G. C. Resende, C. Vogiatzis, and J. L. Walteros, editors, _Learning and Intelligent Optimization, 8th International Conference, LION 8_, volume 8426 of Lecture Notes in Computer Science, pages 36–40. Springer, Heidelberg, 2014. +#' #' @examples #' ## Read the parameters directly from text -#' pcs.table <- ' -#' # name values [conditions (using R syntax)] +#' pcs_table <- ' +#' # name domain #' algorithm {as,mmas,eas,ras,acs}[as] #' localsearch {0, 1, 2, 3}[0] #' alpha [0.00, 5.00][1] @@ -469,12 +541,12 @@ #' nnls | localsearch in {1,2,3} #' dlb | localsearch in {1,2,3} #' ' -#' parameters.table <- read_pcs_file(text=pcs.table) -#' parameters <- readParameters(text=parameters.table) +#' parameters_table <- read_pcs_file(text=pcs_table) +#' cat(parameters_table) +#' parameters <- readParameters(text=parameters_table) #' str(parameters) #' #' @author Manuel López-Ibáñez -#' @md #' @export read_pcs_file <- function(file, digits = 4, debugLevel = 0, text) { @@ -564,5 +636,59 @@ } irace.error("unrecognized line: ", line) } - return(output) + output } + +#' checkParameters +#' +#' FIXME: This is incomplete, for now we only repair inputs from previous irace +#' versions. +#' +#' @template arg_parameters +#' @export +checkParameters <- function(parameters) +{ + if (is.null(parameters$isDependent)) { + parameters$isDependent <- sapply(parameters$domain, is.expression) + names(parameters$isDependent) <- parameters$names + } + parameters +} + +#' Print parameter space in the textual format accepted by irace. +#' +#' FIXME: Dependent parameter bounds are not supported yet. +#' +#' @param params (`list()`) Parameter object stored in `irace.Rdata` or read with `irace::readParameters()`. +#' +#' @param digits (`integer()`) The desired number of digits after the decimal point for real-valued parameters. Default is 15, but it should be the value in `scenario$digits`. +#' +#' @examples +#' parameters.table <- ' +#' # name switch type values [conditions (using R syntax)] +#' algorithm "--" c (as,mmas,eas,ras,acs) +#' localsearch "--localsearch " c (0, 1, 2, 3) +#' ants "--ants " i,log (5, 100) +#' q0 "--q0 " r (0.0, 1.0) | algorithm == "acs" +#' nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) +#' ' +#' parameters <- readParameters(text=parameters.table) +#' printParameters(parameters) +#' @export +printParameters <- function(params, digits = 15L) +{ + names_len <- max(nchar(params$names)) + switches_len <- max(nchar(params$switches)) + 2 + for (name in params$names) { + switch <- paste0('"', params$switches[[name]], '"') + type <- params$types[[name]] + transf <- params$transform[[name]] + domain <- params$domain[[name]] + if (type == "r") domain <- formatC(domain, digits=digits, format="f", drop0trailing=TRUE) + domain <- paste0('(', paste0(domain, collapse=","), ')') + condition <- params$conditions[[name]] + condition <- if (isTRUE(condition)) "" else paste0(" | ", condition) + if (!is.null(transf) && transf != "") type <- paste0(type, ",", transf) + cat(sprintf('%*s %*s %s %-15s%s\n', -names_len, name, -switches_len, switch, type, domain, condition)) + } +} diff --git a/R/testing.R b/R/testing.R index a5f19db..e1f456e 100644 --- a/R/testing.R +++ b/R/testing.R @@ -1,8 +1,6 @@ -#' testConfigurations +#' Execute the given configurations on the testing instances specified in the +#' scenario #' -#' \code{testConfigurations} executes the given configurations on the -#' testing instances specified in the scenario. -#' #' @template arg_configurations #' @template arg_scenario #' @template arg_parameters @@ -13,10 +11,10 @@ #' \item{\code{seeds}}{Array of the instance seeds used in the experiments.} #' } #' -#' @details A test instance set must be provided through \code{scenario$testInstances}. +#' @details A test instance set must be provided through `scenario[["testInstances"]]`. #' #' @seealso -#' \code{\link{testing.main}} +#' [testing_fromlog()] #' #' @author Manuel López-Ibáñez #' @export @@ -24,16 +22,16 @@ { # We need to set up a default scenario (and repeat all checks) in case # we are called directly instead of being called after executing irace. - scenario <- checkScenario(defaultScenario(scenario)) + scenario <- checkScenario(scenario) - testInstances <- scenario$testInstances + testInstances <- scenario[["testInstances"]] instances.ID <- names(testInstances) if (length(testInstances) == 0) irace.error("No test instances given") if (is.null(instances.ID)) irace.error("testInstances must have names") # 2147483647 is the maximum value for a 32-bit signed integer. # We use replace = TRUE, because replace = FALSE allocates memory for each possible number. - ## FIXME: scenario$testInstances and scenario$instances behave differently, + ## FIXME: scenario[["testInstances"]] and scenario$instances behave differently, ## we should unify them so that the seeds are also saved in scenario. instanceSeed <- sample.int(2147483647, size = length(testInstances), replace = TRUE) names(instanceSeed) <- instances.ID diff --git a/R/timer.R b/R/timer.R new file mode 100644 index 0000000..c788711 --- /dev/null +++ b/R/timer.R @@ -0,0 +1,18 @@ +Timer <- R6::R6Class("Timer", cloneable = FALSE, list( + start = NULL, + initialize = function() { + self$start <- proc.time() + self + }, + elapsed = function() { + x <- proc.time() - self$start + if (!is.na(x[4L])) + x[1L] <- x[1L] + x[4L] + if (!is.na(x[5L])) + x[2L] <- x[2L] + x[5L] + x <- x[1L:3L] + names(x) <- c("user", "system", "wallclock") + x + }) +) + diff --git a/R/tnorm.R b/R/tnorm.R index 07e65a8..bf17193 100644 --- a/R/tnorm.R +++ b/R/tnorm.R @@ -43,13 +43,13 @@ rtnorm <- function (n, mean = 0, sd = 1, lower = -Inf, upper = Inf) { if (length(n) > 1) n <- length(n) - mean <- rep(mean, length=n) - sd <- rep(sd, length=n) - lower <- rep(lower, length=n) - upper <- rep(upper, length=n) + mean <- rep_len(mean, n) + sd <- rep_len(sd, n) + lower <- rep_len(lower, n) + upper <- rep_len(upper, n) lower <- (lower - mean) / sd ## Algorithm works on mean 0, sd 1 scale upper <- (upper - mean) / sd - ind <- seq(length=n) + ind <- seq_len(n) ret <- numeric(n) ## Different algorithms depending on where upper/lower limits lie. alg <- ifelse( diff --git a/R/utils.R b/R/utils.R index e5ad6d1..63cefa8 100644 --- a/R/utils.R +++ b/R/utils.R @@ -9,8 +9,6 @@ options(error = if (interactive()) utils::recover else quote(utils::dump.frames("iracedump", TRUE))) } - -.irace.prefix <- "== irace == " irace.print.memUsed <- function(objects) { @@ -42,7 +40,8 @@ # cannot help the user to understand why the program failed. irace.warning <- function(...) { - cat(sep="", "WARNING: ", ..., "\n") + if (getOption(".irace.quiet", default=FALSE)) return() + cat(sep="", .msg.prefix, "WARNING: ", ..., "\n") } # Print a user-level fatal error message, when the calling context @@ -53,11 +52,14 @@ # value allowed up to R 3.0.2 op <- options(warning.length = 8170) on.exit(options(op)) - stop (.irace.prefix, ..., call. = FALSE) -} - -## utils::dump.frames is broken and cannot be used with bquote, so we need a wrapper. -## See help(dump.frames) + stop (.msg.prefix, ..., call. = FALSE) +} + +## utils::dump.frames is broken and cannot be used with bquote, so we need a wrapper. When irace crashes, it generates a file "iracedump.rda". To debug the crash use: +## R> load("iracedump.rda") +## R> debugger(iracedump) +## +## See help(dump.frames) for more details. irace.dump.frames <- function() { execDir <- getOption(".irace.execdir") @@ -80,7 +82,7 @@ { .irace.bug.report <- paste0("An unexpected condition occurred. ", - "Please report this bug to the authors of the irace package ") + "Please report this bug to the authors of the irace package ") op <- options(warning.length = 8170, error = if (interactive()) utils::recover @@ -89,9 +91,9 @@ # 6 to not show anything below irace.assert() bt <- capture.output(traceback(6)) warnings() - stop (.irace.prefix, paste0(..., collapse = "\n"), + stop (.msg.prefix, paste0(..., collapse = "\n"), paste0(bt, collapse= "\n"), "\n", - .irace.prefix, "\n", .irace.bug.report, call. = FALSE) + .msg.prefix, "\n", .irace.bug.report, call. = FALSE) invisible() } @@ -104,7 +106,7 @@ msg <- paste0(deparse(mc), " is not TRUE\n") if (!is.null(eval.after)) { msg.after <- eval.parent(capture.output(eval.after)) - msg <- paste0(msg, "\n", msg.after) + msg <- paste0(msg, "\n", paste0(msg.after, collapse="\n")) } irace.internal.error(msg) invisible() @@ -112,6 +114,9 @@ irace.note <- function(...) { + # FIXME: If this was a function within an irace object, we could replace it + # when using quiet. + if (getOption(".irace.quiet", default=FALSE)) return() cat ("# ", format(Sys.time(), usetz=TRUE), ": ", paste0(..., collapse = ""), sep = "") } @@ -127,7 +132,7 @@ if (!is.character(file) || is.null.or.empty(file)) { irace.error (text, " ", shQuote(file), " is not a vaild filename") } - file <- path.rel2abs(file) + file <- path_rel2abs(file) ## The above should remove the trailing separator if present for windows OS ## compatibility, except when we have just C:/, where the trailing separator ## must remain. @@ -204,6 +209,11 @@ (length(x) == 0) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") } +is_null_or_empty_or_na <- function(x) +{ + (length(x) == 0) || is.na.nowarn(x) || (length(x) == 1 && !suppressWarnings(is.na(x)) && is.character(x) && x == "") +} + is.function.name <- function(FUN) { # FIXME: Is there a simpler way to do this check? @@ -242,117 +252,6 @@ strcat <- function(...) { do.call(paste0, args = list(..., collapse = NULL)) -} - -# Function to convert a relative to an absolute path. CWD is the -# working directory to complete relative paths. It tries really hard -# to create canonical paths. -path.rel2abs <- function (path, cwd = getwd()) -{ - # Keep doing gsub as long as x keeps changing. - gsub.all <- function(pattern, repl, x, ...) { - repeat { - newx <- gsub(pattern, repl, x, ...) - if (newx == x) return(newx) - x <- newx - } - } - irace.normalize.path <- function(path) { - return(suppressWarnings(normalizePath(path, winslash = "/", mustWork = NA))) - } - - if (is.null.or.na(path)) { - return (NULL) - } else if (path == "") { - return ("") - } - # Using .Platform$file.sep is too fragile. Better just use "/" everywhere. - s <- "/" - - # Possibly expand ~/path to /home/user/path. - path <- path.expand(path) - # Remove winslashes if given. - path <- gsub("\\", s, path, fixed = TRUE) - - # Detect a Windows drive - windrive.regex <- "^[A-Za-z]:" - windrive <- "" - if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { - m <- regexpr(windrive.regex, path) - windrive <- regmatches(path, m) - path <- sub(windrive.regex, "", path) - } - - - # Change "/./" to "/" to get a canonical form - path <- gsub.all(paste0(s, ".", s), s, path, fixed = TRUE) - # Change "//" to "/" to get a canonical form - path <- gsub(paste0(s, s, "+"), s, path) - # Change "/.$" to "/" to get a canonical form - path <- sub(paste0(s, "\\.$"), s, path) - # Drop final "/" - path <- sub(paste0(s, "$"), "", path) - if (path == "") path <- s - - # Prefix the current cwd to the path if it doesn't start with - # / \\ or whatever separator. - if (path == "." || !grepl(paste0("^",s), path)) { - # There is no need to normalize cwd if it was returned by getwd() - if (!missing(cwd)) { - # Recurse to get absolute cwd - cwd <- path.rel2abs(cwd) - } - - # Speed-up the most common cases. - # If it is just "." - if (path == ".") { - return (irace.normalize.path(cwd)) - } - # Remove "./" from the start of path. - path <- sub(paste0("^\\.", s), "", path) - # Make it absolute but avoid doubling s - if (substring(cwd, nchar(cwd)) == s) path <- paste0(cwd, path) - else path <- paste0(cwd, s, path) - # If it is just a path without ".." inside - if (!grepl(paste0(s,"\\.\\."), path)) { - return (irace.normalize.path(path)) - } - # Detect a Windows drive - if (grepl(paste0(windrive.regex, "($|", s, ")"), path)) { - m <- regexpr(windrive.regex, path) - windrive <- regmatches(path, m) - path <- sub(windrive.regex, "", path) - } - } - # else - - # Change "/x/.." to "/" to get a canonical form - prevdir.regex <- paste0(s, "[^", s,"]+", s, "\\.\\.") - repeat { - # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" - tmp <- sub(paste0(prevdir.regex, s), s, path) - if (tmp == path) break - path <- tmp - } - # Handle "/something/..$" to "/" that is, when ".." is the last thing in the path. - path <- sub(paste0(prevdir.regex, "$"), s, path) - - # Handle "^/../../.." to "/" that is, going up at the root just returns the root. - repeat { - # We need to do it one by one so "a/b/c/../../../" is not converted to "a/b/../" - tmp <- sub(paste0("^", s, "\\.\\.", s), s, path) - if (tmp == path) break - path <- tmp - } - # Handle "^/..$" to "/" that is, when ".." is the last thing in the path. - path <- sub(paste0("^", s, "\\.\\.$"), s, path) - - # Add back Windows drive, if any. - path <- paste0(windrive, path) - - # We use normalizePath, which will further simplify the path if - # the path exists. - return (irace.normalize.path(path)) } #' Update filesystem paths of a scenario consistently. @@ -362,88 +261,81 @@ #' #' @template arg_scenario #' @param from character string containing a regular expression (or character -#' string for \code{fixed = TRUE}) to be matched. -#' @param to the replacement string.character string. For \code{fixed = FALSE} -#' this can include backreferences \code{"\1"} to \code{"\9"} to -#' parenthesized subexpressions of \code{from}. -#' @param fixed logical. If \code{TRUE}, \code{from} is a string to be matched +#' string for `fixed = TRUE`) to be matched. +#' @param to the replacement string.character string. For `fixed = FALSE` +#' this can include backreferences `"\1"` to `"\9"` to +#' parenthesized subexpressions of `from`. +#' @param fixed logical. If `TRUE`, `from` is a string to be matched #' as is. #' @return The updated scenario #' @examples #' \dontrun{ #' scenario <- readScenario(filename = "scenario.txt") -#' scenario <- scenario.update.paths(scenario, from = "/home/manuel/", to = "/home/leslie") +#' scenario <- scenario_update_paths(scenario, from = "/home/manuel/", to = "/home/leslie") #' } -#' @seealso \code{\link[base]{grep}} +#' @seealso [base::grep()] #' @export -scenario.update.paths <- function(scenario, from, to, fixed = TRUE) +scenario_update_paths <- function(scenario, from, to, fixed = TRUE) { pathParams <- .irace.params.def[.irace.params.def[, "type"] == "p", "name"] # Only consider the ones that actually appear in scenario. pathParams <- intersect(pathParams, names(scenario)) scenario[pathParams] <- lapply(scenario[pathParams], sub, pattern = from, replacement = to, fixed = fixed) - return(scenario) + scenario +} + +#' @rdname scenario_update_paths +#' @export +scenario.update.paths <- function(scenario, from, to, fixed = TRUE) +{ + .Deprecated("scenario_update_paths") + scenario_update_paths(scenario=scenario, from=from, to=to, fixed=fixed) } # This function is used to trim potentially large strings for printing, since # the maximum error/warning length is 8170 characters (R 3.0.2) strlimit <- function(str, limit = 5000) { - if (nchar(str) > limit) { - return(paste0(substr(str, 1, limit - 3), "...")) - } + if (nchar(str) > limit) return(paste0(substr(str, 1, limit - 3), "...")) return(str) } test.type.order.str <- function(test.type) { - return (switch(test.type, - friedman = "sum of ranks", - t.none =, # Fall-throught - t.holm =, # Fall-throught - t.bonferroni = "mean value", - irace.internal.error ("test.type.order.str() Invalid value '", - test.type, "' of test.type"))) + switch(test.type, + friedman = "sum of ranks", + t.none =, # Fall-throught + t.holm =, # Fall-throught + t.bonferroni = "mean value", + irace.internal.error ("test.type.order.str() Invalid value '", + test.type, "' of test.type")) } trim.leading <- function(str) -{ - return (sub('^[[:space:]]+', '', str)) ## white space, POSIX-style -} + sub('^[[:space:]]+', '', str) ## white space, POSIX-style + trim.trailing <- function(str) -{ - return (sub('[[:space:]]+$', '', str)) ## white space, POSIX-style -} + sub('[[:space:]]+$', '', str) ## white space, POSIX-style + # remove leading and trailing white space characters -trim <- function(str) -{ - return (trim.trailing(trim.leading(str))) -} +trim <- function(str) trim.trailing(trim.leading(str)) isFixed <- function (paramName, parameters) -{ - return (as.logical(parameters$isFixed[paramName])) -} + as.logical(parameters$isFixed[paramName]) paramDomain <- function (paramName, parameters) -{ - return (parameters$domain[[paramName]]) -} + parameters$domain[[paramName]] paramLowerBound <- function (paramName, parameters) -{ - return (as.numeric(parameters$domain[[paramName]][1])) -} + as.numeric(parameters$domain[[paramName]][1]) paramUpperBound <- function (paramName, parameters) -{ - return (as.numeric(parameters$domain[[paramName]][2])) -} - -nbParam <- function (parameters) -{ - return (length(parameters$names)) -} + as.numeric(parameters$domain[[paramName]][2]) + + +inNumericDomain <- function(value, domain) (value >= domain[1] && value <= domain[2]) + +nbParam <- function (parameters) length(parameters$names) ## This function takes two matrices x and y and merges them such that the ## resulting matrix z has: @@ -481,18 +373,34 @@ # Input: the configurations with the .RANK. field filled. # the number of elites wished # Output: nbElites elites, sorted by ranks, with the weights assigned. -extractElites <- function(configurations, nbElites) -{ +extractElites <- function(scenario, parameters, configurations, nbElites) +{ + # Keep only alive configurations. + ## FIXME: Shouldn't this be done by the caller? + configurations <- configurations[configurations$.ALIVE., , drop = FALSE] if (nbElites < 1) { - ## FIXME: Should this be an error or should we handle it in some other way? - irace.error("nbElites is lower or equal to zero.") - } + irace.internal.error("nbElites is lower or equal to zero.") + } + # Remove duplicated. Duplicated configurations may be generated, however, it + # is too slow to check at generation time. Nevertheless, we can check now + # since we typically have very few elites. + ## FIXME: Use a variant of similarConfigurations. + configurations <- configurations[order(configurations$.ID.), , drop = FALSE] + before <- nrow(configurations) + configurations <- configurations[!duplicated(removeConfigurationsMetaData(configurations)), + , drop = FALSE] + after <- nrow(configurations) + if (after < before && scenario$debugLevel >= 1) { + irace.note("Dropped ", before - after, " duplicated elites\n") + } + + nbElites <- min(after, nbElites) # Sort by rank. - elites <- configurations[order(as.numeric(configurations$.RANK.)), , drop = FALSE] + elites <- configurations[order(configurations$.RANK.), , drop = FALSE] elites <- elites[1:nbElites, , drop = FALSE] elites[, ".WEIGHT."] <- ((nbElites - (1:nbElites) + 1) / (nbElites * (nbElites + 1) / 2)) - return (elites) + elites } #' removeConfigurationsMetaData @@ -508,8 +416,8 @@ #' @return The same matrix without the "metadata". #' #' @seealso -#' \code{\link{configurations.print.command}} to print the configurations as command lines. -#' \code{\link{configurations.print}} to print the configurations as a data frame. +#' [configurations.print.command()] to print the configurations as command lines. +#' [configurations.print()] to print the configurations as a data frame. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export @@ -517,8 +425,8 @@ removeConfigurationsMetaData <- function(configurations) { # Meta-data colnames begin with "." - return (configurations[, grep("^\\.", colnames(configurations), invert = TRUE), - drop = FALSE]) + configurations[, grep("^\\.", colnames(configurations), invert = TRUE), + drop = FALSE] } #' Print configurations as a data frame @@ -531,7 +439,7 @@ #' @return None. #' #' @seealso -#' \code{\link{configurations.print.command}} to print the configurations as command-line strings. +#' [configurations.print.command()] to print the configurations as command-line strings. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export @@ -555,7 +463,7 @@ #' @return None. #' #' @seealso -#' \code{\link{configurations.print}} to print the configurations as a data frame. +#' [configurations.print()] to print the configurations as a data frame. #' #' @author Manuel López-Ibáñez and Jérémie Dubois-Lacoste #' @export @@ -749,7 +657,9 @@ formatC(proc.time()["elapsed"] - elapsed, format = "f", digits = 2), "\n") } - return(list(output = output, error = NULL)) + # TODO: Return elapsed time so that we can report at the end the total + # elapsed time taken by irace vs the time taken by the target-runner. + list(output = output, error = NULL) } # Safe sampling of vector: @@ -778,12 +688,10 @@ # } is.file.extension <- function(filename, ext) -{ - return(substring(filename, nchar(filename) + 1 - nchar(ext)) == ext) -} + substring(filename, nchar(filename) + 1 - nchar(ext)) == ext # Same as !(x %in% table) -"%!in%" <- function (x, table) match(x, table, nomatch = 0L) == 0L +"%!in%" <- function(x, table) match(x, table, nomatch = 0L) == 0L irace_save_logfile <- function(iraceResults, scenario) { @@ -791,6 +699,38 @@ cwd <- setwd(scenario$execDir) # FIXME: Use saveRDS # FIXME: Bump to version=3 when we bump the minimum R version to >=3.6 - save (iraceResults, file = scenario$logFile, version = 2) - setwd (cwd) -} + save(iraceResults, file = scenario$logFile, version = 2) + setwd(cwd) +} + +valid_iracelog <- function(x) +{ + is.list(x) && ("scenario" %in% names(x)) +} + +#' Read the log file produced by irace (`irace.Rdata`). +#' +#' @param filename Filename that contains the log file saved by irace. Example: `irace.Rdata`. +#' +#' @param name Optional argument that allows overriding the default name of the object in the file. +#' +#' @return (`list()`) +#' @concept analysis +#' @export +read_logfile <- function(filename, name = "iraceResults") +{ + # If filename is already the iraceResults object, just return it. + if (valid_iracelog(filename)) return(filename) + + if (file.access(filename, mode=4) != 0) + stop("read_logfile: Cannot read file '", filename, "'") + + load(filename) + iraceResults <- get0(name, inherits=FALSE) + if (!valid_iracelog(iraceResults)) + stop("The file '", filename, "' does not contain the '", name, "' object.") + + iraceResults +} + +do_nothing <- function(...) invisible() diff --git a/R/version.R b/R/version.R index e9b68ea..9e312b0 100644 --- a/R/version.R +++ b/R/version.R @@ -1,6 +1,5 @@ #' irace.version #' -#' A character string containing the version of \pkg{irace}. -#' +#' A character string containing the version of `irace`. #' @export -irace.version <- '3.4.1.9fcaeaf' +irace.version <- '3.5.6863679' diff --git a/R/zzz.R b/R/zzz.R new file mode 100644 index 0000000..8322c5c --- /dev/null +++ b/R/zzz.R @@ -0,0 +1,12 @@ +# Uses 10 decimal places at most so that there is space for '-', '.', and +# 'e-NN' within the 16 spaces. +.irace.format.perf <- "%#16.10g" + +.onLoad <- function(libname, pkgname) { + # FIXME: We would like to use %#16.10g but this causes problems with + # https://github.com/oracle/fastr/issues/191 + R_engine <- R.version$engine + if (!is.null(R_engine) && R_engine == "FastR") + .irace.format.perf <<- "%16.10g" + invisible() +} diff --git a/README.md b/README.md index 4d04506..995b9d6 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,17 @@ **irace**: Iterated Racing for Automatic Algorithm Configuration ================================================================ -[![CRAN Status](https://www.r-pkg.org/badges/version-last-release/irace)](https://cran.r-project.org/package=irace) [![CRAN Downloads](https://cranlogs.r-pkg.org/badges/grand-total/irace)](https://CRAN.R-project.org/package=irace) [![Travis Build -Status](https://travis-ci.org/MLopez-Ibanez/irace.svg?branch=master)](https://travis-ci.org/MLopez-Ibanez/irace) - -[ [**Homepage**](http://iridia.ulb.ac.be/irace/) ] [ [**User Guide (PDF)**](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) ] - -**Maintainers:** [Manuel López-Ibáñez](http://lopez-ibanez.eu), Leslie Pérez Cáceres - -**Creators:** [Manuel López-Ibáñez](http://lopez-ibanez.eu), Jérémie Dubois-Lacoste + +[![CRAN Status](https://www.r-pkg.org/badges/version-last-release/irace)](https://cran.r-project.org/package=irace) [![CRAN Downloads](https://cranlogs.r-pkg.org/badges/grand-total/irace)](https://CRAN.R-project.org/package=irace) [![R build +status](https://github.com/MLopez-Ibanez/irace/workflows/R-CMD-check/badge.svg)](https://github.com/MLopez-Ibanez/irace/actions) +[![Codecov test coverage](https://codecov.io/gh/MLopez-Ibanez/irace/branch/master/graph/badge.svg)](https://app.codecov.io/gh/MLopez-Ibanez/irace?branch=master) + + +[ [**Homepage**](https://mlopez-ibanez.github.io/irace/) ] [ [**User Guide (PDF)**](https://cran.r-project.org/package=irace/vignettes/irace-package.pdf) ] + +**Maintainers:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Leslie Pérez Cáceres + +**Creators:** [Manuel López-Ibáñez](https://lopez-ibanez.eu/), Jérémie Dubois-Lacoste **Contributors:** Jérémie Dubois-Lacoste, Thomas Stützle, Mauro Birattari, Eric Yuan and Prasanna Balaprakash. @@ -31,22 +34,23 @@ **Relevant literature:** 1. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and - M. Birattari. [The irace package: Iterated Racing for Automatic Algorithm Configuration.](http://dx.doi.org/10.1016/j.orp.2016.09.002). + M. Birattari. [The irace package: Iterated Racing for Automatic Algorithm Configuration](http://dx.doi.org/10.1016/j.orp.2016.09.002). *Operations Research Perspectives*, 3:43–58, 2016.
- [ [bibtex](http://lopez-ibanez.eu/LopezIbanez_bib.html#LopDubPerStuBir2016irace) + [ [bibtex](https://lopez-ibanez.eu/LopezIbanez_bib.html#LopDubPerStuBir2016irace) | doi: [10.1016/j.orp.2016.09.002](http://dx.doi.org/10.1016/j.orp.2016.09.002) ] 2. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro - Birattari. [The irace package, Iterated Race for Automatic Algorithm Configuration](http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-004.pdf). + Birattari. [The irace package, Iterated Race for Automatic Algorithm Configuration](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf). Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, Belgium, 2011.
- [ [bibtex](http://iridia.ulb.ac.be/~manuel/LopezIbanez_bib.html#LopDubStu2011irace) - | [PDF](http://iridia.ulb.ac.be/IridiaTrSeries/IridiaTr2011-004.pdf) ] - - 3. Manuel López-Ibáñez. [The irace - software package: A tutorial](http://iridia.ulb.ac.be/irace/files/irace-comex-tutorial.pdf). COMEX Workshop on Practical Automatic Algorithm Configuration, 2014.
- [ [workshop webpage](http://iridia.ulb.ac.be/~manuel/comex_workshop/) - | [PDF](http://iridia.ulb.ac.be/irace/files/irace-comex-tutorial.pdf) ] + [ [bibtex](https://iridia-ulb.github.io/references/index.html#LopDubStu2011irace) + | [PDF](https://iridia.ulb.ac.be/IridiaTrSeries/link/IridiaTr2011-004.pdf) ] + + 3. Thomas Stützle and Manuel López-Ibáñez. [Tutorial: Automated algorithm + configuration and design](https://doi.org/10.1145/3449726.3461404). GECCO + '21: Proceedings of the Genetic and Evolutionary Computation Conference + Companion, July 2021. + [doi:10.1145/3449726.3461404](https://doi.org/10.1145/3449726.3461404) Requisites @@ -70,6 +74,25 @@ The following is a quick-start guide. The user guide gives more detailed instructions. +Quick Start +=========== +1. Install R (with your favourite package manager, and see more details below). +2. Install irace. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). +```bash + $ Rscript -e "install.packages('irace', repos='https://cloud.r-project.org')" +``` + +3. Add irace to path. For windows user, this step is unfortunately more involved, so please see more detailed instructions below. +```bash + $ export PATH="$(Rscript -e "cat(paste0(system.file(package='irace', 'bin', mustWork=TRUE), ':'))" 2> /dev/null)${PATH}" +``` +Consider adding this line to your `~/.bashrc`, `~/.zshrc`, or `~/.profile` for it to persist between sessions. + +4. You can open the user guide with the following command. This command works on CMD and Powershell with R added to PATH (see detailed instructions below). +```bash + $ Rscript -e "vignette('irace-package')" +``` + Installing R ============ @@ -95,14 +118,12 @@ You can install R directly from a CRAN mirror (). -Alternatively, if you use homebrew, you can just brew the R formula -from the science tap (unfortunately it does not come already bottled -so you need to have Xcode installed to compile it): - -```bash - $ brew tap homebrew/science - $ brew install r -``` +Alternatively, if you use homebrew, you can just do +``` + $ brew install --cask r +``` + +(Using `brew install r` is not recommended because that will build R from source and you will not be able to use any CRAN binary, possibly resulting in annoying build failiures). Once R is installed, you can launch R from the Terminal (or from your Applications), and from the R prompt install the irace package. See @@ -116,12 +137,23 @@ launch the R console and install the irace package from it. See instructions below. +In addition to using the R console, it might be very useful to add R to PATH so you can run most of the GUN/Linux shell commands without modification in CMD or Powershell. Usually, R is installed in `C:\Program Files\R\R-4.1.3` (the version number depends on your installation). + +You should add the following line [to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/) (if you want to use the 64-bit version) +``` +C:\Program Files\R\R-4.1.3\bin\x64 +``` + +Or, if you are on a 32-bit version +``` +C:\Program Files\R\R-4.1.3\bin\i386 +``` Installing the irace package ============================ -There are two methods for installing the [irace](http://iridia.ulb.ac.be/irace) R package on your -computer: +There are two methods for installing the +[irace](https://mlopez-ibanez.github.io/irace/) R package on your computer: 1. Install within R (automatic download): ```R @@ -156,8 +188,8 @@ ```R $ R R> library(irace) - R> system.file(package="irace") - [1] "~/R/irace" + R> cat(system.file(package="irace", "bin", mustWork=TRUE), "\n") + /home/user/R/irace/bin ``` The last command tells you the installation directory of `irace`. @@ -168,8 +200,8 @@ `.bash_profile`, `.bashrc` or `.profile`: ```bash - export IRACE_HOME=~/R/irace/ # Path given by system.file(package="irace") - export PATH=${IRACE_HOME}/bin/:$PATH + export IRACE_HOME=/home/user/R/irace/bin/ # Path given by system.file(package="irace", "bin", mustWork=TRUE) + export PATH=${IRACE_HOME}:$PATH # export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced ``` @@ -184,17 +216,17 @@ Windows ------- -If the installation directory of `irace` is `C:/R/irace/`, you can invoke -`irace` by opening a terminal (launch the program `cmd.exe`) and executing: - -```bash - C:\> C:\R\irace\bin\x64\irace.exe --help -``` - -or if you are in a 32-bits system, executing: - -```bash - C:\> C:\R\irace\bin\i386\irace.exe --help +You can find out where the irace binary is installed by running the following in Powershell or CMD: + +```Powershell + C:\> Rscript -e "cat(gsub('/', '\\\\', system.file(package='irace', 'bin', 'x64', mustWork=TRUE)))" +``` + +It will output a path, such as `C:\Program Files\R\R-4.1.3\library\irace\bin\x64` (replace `x64` with `i386` if you are on a 32-bit system), which can you [add to PATH](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/). + +Then running the following should work: +```Powershell + C:\> irace --help ``` You can also launch irace by opening the R console and executing: @@ -204,6 +236,23 @@ R> irace.cmdline("--help") ``` +GitHub (Development version) +---------------------------- + +If you wish to try the development version, you can install it by executing the +following commands within the R console: + +```R + R> install.packages("devtools") + R> devtools::install_github("MLopez-Ibanez/irace") +``` + +Python +------ + +You can use the irace R package from Python using `rpy2`. An example on how to do this is the implementation of [iracepy](https://github.com/auto-optimization/iracepy). + + Usage ===== @@ -213,23 +262,12 @@ $ cd ~/tuning ``` -2. Copy the template and example files to the scenario directory -```bash - $ cp $IRACE_HOME/templates/*.tmpl . -``` - - where `$IRACE_HOME` is the path to the installation directory of - `irace`. It can be obtained by doing: - -```R - $ R - > library(irace) - > system.file(package="irace") -``` - -3. For each template in your tuning directory, remove the `.tmpl` - suffix, and modify them following the instructions in each file. In - particular, +2. Initialize your tuning directory with template config files +```bash + $ $IRACE_HOME/bin/irace --init +``` + +3. Modify the generated files following the instructions found within each file. In particular, * The scripts `target-runner` and `target-evaluator` (if you need it at all) should be executable. The output of `target-runner` (or `target-evaluator` if you use a separate evaluation step) is minimized by @@ -243,7 +281,7 @@ 4. Put the instances in `~/tuning/Instances/`. In addition, you can create a file that specifies which instances from that directory should be run and which instance-specific parameters to use. See - `scenario.txt.tmpl` and `instances-list.tmpl` for examples. The command + `scenario.txt` and `instances-list.txt` for examples. The command irace will not attempt to create the execution directory (`execDir`), so it must exist before calling irace. The default `execDir` is the current directory. @@ -252,10 +290,9 @@ ```bash $ cd ~/tuning/ && $IRACE_HOME/bin/irace ``` - - performs one run of Iterated Race. See the output of `irace --help` for - additional irace parameters. Command-line parameters override the - scenario setup specified in the `scenario.txt` file. + performs one run of Iterated Race. See the output of `irace --help` for + additional irace parameters. Command-line parameters override the + scenario setup specified in the `scenario.txt` file. Many tuning runs in parallel @@ -319,6 +356,24 @@ Birattari. +Building an irace standalone container +====================================== + +Thanks to [Singularity](https://sylabs.io/singularity/), you can build a +standalone container of `irace` using the file `irace.sindef` which is +available in the directory `inst/` in the source tarball and github repository +or, after installing the irace R package, in the installation directory given +by the R expression `system.file(package="irace")`. After installing +SingularityCE, the container may be build using: + + sudo singularity build irace.sindef irace.sif + +and run with: + + singularity run irace.sif + + + Frequently Asked Questions ========================== diff --git a/build/partial.rdb b/build/partial.rdb index 1a6ce10..f2bc9dd 100644 Binary files a/build/partial.rdb and b/build/partial.rdb differ diff --git a/build/vignette.rds b/build/vignette.rds index c2e5cca..51566a7 100644 Binary files a/build/vignette.rds and b/build/vignette.rds differ diff --git a/cleanup b/cleanup index 8d6905a..f463623 100755 --- a/cleanup +++ b/cleanup @@ -1,10 +1,10 @@ #!/bin/sh rm -f irace-Ex.R config.* confdefs.h \ - src/*.o src/*.so src/config.h src/symbols.rds \ + src/*.so src/*.o src/*/*.o src/*.gcno src/*/*.gcno src/config.h src/symbols.rds \ inst/doc/*.blg inst/doc/*.bbl \ tests/testthat/*.log tests/testthat/*.Rout tests/testthat/irace.Rdata -find . -name '*.orig' -o -name '.Rhistory' | xargs rm -f +find . -name '*.orig' -o -name '.Rhistory' -o -name '*.rej' | xargs rm -f rm -rf autom4te.cache diff --git a/inst/bin/parallel-irace-mpi b/inst/bin/parallel-irace-mpi index 9caf2fd..a30ae30 100755 --- a/inst/bin/parallel-irace-mpi +++ b/inst/bin/parallel-irace-mpi @@ -60,14 +60,15 @@ usage() { cat < [--machine MACHINE] [IRACE PARAMS] +Usage: $0 N[-M] [EXECDIR] --parallel NUM [--seed NUM] [--machine MACHINE] [IRACE PARAMS] Parameters: N an integer giving the number of repetitions of irace or a sequence N-M giving which repetitions to redo. EXECDIR job M will use EXECDIR-M directory (default: execdir-) as the execDir (--exec-dir) parameter of irace. - --parallel number of parallel jobs + --parallel NUM number of parallel jobs (value in scenario.txt is ignored) + --seed NUM the seed of each run i will be SEED+i-1 (default: 1234567) --machine MACHINE qsub queue type, e.g., opteron6272 IRACE PARAMS additional parameters for irace. EOF diff --git a/inst/bin/parallel-irace-qsub b/inst/bin/parallel-irace-qsub index e5ba718..185db58 100755 --- a/inst/bin/parallel-irace-qsub +++ b/inst/bin/parallel-irace-qsub @@ -55,7 +55,7 @@ exit \$RET EOF } -## End of customization +## END OF CUSTOMIZATION error () { echo "$0: error: $@" >&2 @@ -64,13 +64,14 @@ usage() { cat <&2 + exit 1 +} + +usage() { + cat <>= library("irace") load("examples.Rdata") -load("irace-acotsp.Rdata") -load("log-ablation.Rdata") +iraceResults <- irace::read_logfile("irace-acotsp.Rdata") +log_ablation_file <- file.path(system.file(package="irace"), "exdata", "log-ablation.Rdata") +load(log_ablation_file) options(width = 70) @ \newpage @@ -248,7 +253,7 @@ \end{center} % More information about \irace is available at -\url{http://iridia.ulb.ac.be/irace}. +\url{https://mlopez-ibanez.github.io/irace}. \subsection{Version} The current version of the \irace package is \iraceversion. Previous @@ -545,25 +550,12 @@ \end{lstlisting} %@ -\item Copy all the template files from the \IRACEHOME{/templates/} -directory to the scenario directory. -%<>= +\item Initialize the tuning directory with template config files. On GNU/Linux or OS X, you can do this as follows: +%<>= \begin{lstlisting}[style=BashInputStyle] -# $IRACE_HOME is the installation directory of irace. -cp $IRACE_HOME/templates/*.tmpl ~/tuning/ +irace --init \end{lstlisting} %@ - -% Remember that \IRACEHOME{} is the path to the installation -% directory of \irace. It can be obtained in the \aR console with: -% -% <>= -% library("irace") -% system.file(package = "irace") -% @ - -\item For each template in your tuning directory, remove the \code{.tmpl} - suffix, and modify them following the next steps. \item Define the target algorithm parameters to be tuned by following the instructions in \code{parameters.txt}. Available parameter types and other @@ -613,9 +605,7 @@ time (\code{cost [time]}). When the \parameter{maxTime} option is used, returning \code{time} is mandatory. The \code{target-runner} template is written in \proglang{GNU Bash} scripting language, which can be executed easily in GNU/Linux and OS X - systems. However, you may use any other programming language. As an example, - we provide a \proglang{Python} example in the directory - \IRACEHOME{/examples/python}. % + systems. However, you may use any other programming language. We provide examples written in \proglang{Python}, \proglang{MATLAB} and other languages in \IRACEHOME{/examples/}. % Follow these instructions to adjust the given \code{target-runner} template to your algorithm: \begin{enumerate} @@ -656,11 +646,11 @@ \code{COST=\$(cat \$\{STDOUT\} | grep -e '\^{}[[:space:]]*[+-]\textbackslash{}?[0-9]' | cut -f1)} \end{center} parses the output of your algorithm to obtain the result from the last line. The \code{target-runner} - script must return \textbf{only} one number. In the template example, the result is returned with - \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the used files are deleted. + script must print \textbf{only} one number. In the template example, the result is printed with + \code{echo "\$COST"} (assuming \parameter{maxExperiments} is used) and the generated files are deleted (you may remove that line if you wish to keep them). \begin{xwarningbox} - The \code{target-runner} script must be executable. + The \code{target-runner} script must be an executable file, unless you specify \parameter{targetRunnerLauncher} and \parameter{targetRunnerLauncherArgs}. \end{xwarningbox} You can test the target runner from the \aR console by checking the scenario as @@ -674,7 +664,7 @@ \item \textit{Optional}: Modify the \code{target-evaluator} file. This is rarely needed and the \code{target-runner} template does not use it. \autoref{sec:evaluator} explains when a \parameter{targetEvaluator} is needed and how to define it. - \item The \irace executable provides an option (\parameter{-{}-check}) to + \item The \irace executable provides an option (\parameter{--check}) to check that the scenario is correctly defined. We recommend to perform a check every time you create a new scenario. When performing the check, \irace will verify that the scenario and parameter definitions are @@ -707,7 +697,7 @@ or directly from the \aR console: \begin{itemize} -\item{% +\item \textbf{From the command-line console}, call the command (on Windows, you should execute \code{irace.exe}): \begin{lstlisting}[style=BashInputStyle] @@ -721,8 +711,8 @@ properly in the \code{scenario.txt} file using the options described in \autoref{sec:irace options}. Most \irace options can be specified in the command line or directly in the \code{scenario.txt} file. -} -\item{ + +\item \textbf{From the \aR console}, evaluate: <>= @@ -733,7 +723,7 @@ scenario = defaultScenario()) irace.main(scenario = scenario) @ -} + \end{itemize} This will perform one run of \irace. See the output of \code{irace --help} in the command-line or \code{irace.usage()} in \aR for quick information on @@ -747,17 +737,14 @@ \subsection{Setup example for ACOTSP}\label{sec:example} -The \ACOTSP tuning example can be found in the package installation at - \IRACEHOME{/examples/acotsp}. +The \ACOTSP tuning example can be found in the package installation in the folder \IRACEHOME{/examples/acotsp}. % -Additionally, a number of example scenarios can be found in the \code{examples} folder. More +Other example scenarios can be found in the same folder. More examples of tuning scenarios can be found in the Algorithm Configuration Library (AClib, \url{http://www.aclib.net/}). In this section, we describe how to execute the \ACOTSP scenario. If you wish to start setting up your own scenario, continue to the next section. For this example, we assume -a GNU/Linux system but making the necessary changes in the commands and \parameter{targetRunner}, -it can be executed in any system that has a \proglang{C} compiler. -%\MANUEL{I don't think this is true, since the target-runner script needs Bash} +a GNU/Linux system such as Ubuntu with a working \proglang{C} compiler such as \code{gcc}. To execute this scenario follow these steps: \begin{enumerate} @@ -772,7 +759,7 @@ \end{lstlisting} %@ -\item Download the training instances from \url{http://iridia.ulb.ac.be/irace/} to the \path{~/tuning/} directory. +\item Download the training instances from \url{https://iridia.ulb.ac.be/supp/IridiaSupp2016-003/index.html} to the \path{~/tuning/} directory. \item Create the instance directory (\eg~\path{~/tuning/Instances}) and decompress the instance files on it. %<>= @@ -793,14 +780,6 @@ make \end{lstlisting} %@ -\item Create a directory for the executable and copy it: - -%<>= -\begin{lstlisting}[style=BashInputStyle] -mkdir ~/bin/ -cp ~/tuning/ACOTSP-1.03/acotsp ~/bin/ -\end{lstlisting} -%@ \item Create a directory for executing the experiments and execute \irace: @@ -859,8 +838,7 @@ option \parameter{digits}. For example, given the default number of digits of $4$, the values $0.12345$ and $0.12341$ are both rounded to $0.1234$. - % However, the values $0.00001$ and $0.00005$ remain the same. - Selected real-valued parameters can be optionally sampled on + Selected real-valued parameters can be optionally sampled on a logarithmic scale (base $e$). \item \textit{Integer} parameters are numerical parameters that can take only @@ -896,7 +874,55 @@ defined on them. All fixed parameters must be defined as categorical parameters and have a domain of one element. -\subsubsection{Conditional parameters} +\subsubsection{Parameter dependent domains}\label{sec:paramdependant} + +Domains that are dependent on the values of other parameters can be specified +only for numerical parameters (both integer and real). To do so, the dependent +domain must be expressed in function of another parameter, which must be a +numerical parameter. The expression that defines a dependency must be written +between quotes: \code{(value, "expression")} or \code{("expression", value)} or +\code{("expression", "expression")}. + +The expressions can only use the following operators and \aR functions: \code{+}, +\code{-}, \code{*}, \code{/}, \code{\%\%}, \code{min}, \code{max}, +\code{round}, \code{floor}, \code{ceiling}, \code{trunc}. If you need to use an +operator or function not listed here, please contact us. + +\begin{xwarningbox} +The user must ensure that the defined domain is valid at all times since +\irace currently is not able to detect possible invalid domains based on the expressions +provided. +\end{xwarningbox} + +If you have a parameter \code{p2} that is just a transformation of another +\code{p1}, then instead of using a dependent domain (left-hand side of the +following example), it will be better to create a dummy parameter that controls +the transformation (right-hand side) and do the transformation within \code{target-runner}. For example: +% +\begin{center} +\begin{minipage}{0.4\linewidth} +\small\centering% +\begin{CodeInput}[frame=single] +# With dependent domains +p1 "" r (0, 100) +p2 "" r ("p1", "p1 + 10") +\end{CodeInput} +\end{minipage} +\hspace{\stretch{1}} should be \hspace{\stretch{1}} +\begin{minipage}{0.4\linewidth} +\small\centering% +\begin{CodeInput}[frame=single] +# With a dummy parameter +p1 "" r (0, 100) +p2dum "" r (0, 10) +\end{CodeInput} +\end{minipage} +\end{center} +% +and \code{target-runner} will compute $\code{p2} = \code{p2dum} \cdot \code{p1}$. + + +\subsubsection{Conditional parameters}\label{sec:conditional} Conditional parameters are active only when others have certain values. These dependencies define a hierarchical relation between parameters. For example, @@ -911,39 +937,40 @@ table. Each line of the table defines a configurable parameter % \begin{center} -\code{\