New Upstream Release - python-ml-collections
Ready changes
Summary
Merged new upstream version: 0.1.1 (was: 0.1.0).
Resulting package
Built on 2022-03-20T13:30 (took 18m28s)
The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:
apt install -t fresh-releases python3-ml-collections
Lintian Result
Diff
diff --git a/AUTHORS b/AUTHORS
new file mode 100644
index 0000000..243e960
--- /dev/null
+++ b/AUTHORS
@@ -0,0 +1,9 @@
+# This is the list of ML Collections's significant contributors.
+#
+# This does not necessarily list everyone who has contributed code,
+# especially since many employees of one corporation may be contributing.
+# To see the full list of contributors, see the revision history in
+# source control.
+
+DeepMind Technologies Limited
+Google LLC
\ No newline at end of file
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 0000000..7a4a3ea
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
\ No newline at end of file
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..c4ef03e
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,2 @@
+include requirements-test.txt
+include requirements.txt
diff --git a/PKG-INFO b/PKG-INFO
index 0e0b10b..b185bde 100644
--- a/PKG-INFO
+++ b/PKG-INFO
@@ -1,606 +1,11 @@
Metadata-Version: 2.1
Name: ml_collections
-Version: 0.1.0
+Version: 0.1.1
Summary: ML Collections is a library of Python collections designed for ML usecases.
Home-page: https://github.com/google/ml_collections
Author: ML Collections Authors
Author-email: ml-collections@google.com
License: Apache 2.0
-Description: # ML Collections
-
- ML Collections is a library of Python Collections designed for ML use cases.
-
- ## ConfigDict
-
- The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
- structures with dot access to nested elements. Together, they are supposed to be
- used as a main way of expressing configurations of experiments and models.
-
- This document describes example usage of `ConfigDict`, `FrozenConfigDict`,
- `FieldReference`.
-
- ### Features
-
- * Dot-based access to fields.
- * Locking mechanism to prevent spelling mistakes.
- * Lazy computation.
- * FrozenConfigDict() class which is immutable and hashable.
- * Type safety.
- * "Did you mean" functionality.
- * Human readable printing (with valid references and cycles), using valid YAML
- format.
- * Fields can be passed as keyword arguments using the `**` operator.
- * There are two exceptions to the strong type-safety of the ConfigDict. `int`
- values can be passed in to fields of type `float`. In such a case, the value
- is type-converted to a `float` before being stored. Similarly, all string
- types (including Unicode strings) can be stored in fields of type `str` or
- `unicode`.
-
- ### Basic Usage
-
- ```python
- import ml_collections
-
- cfg = ml_collections.ConfigDict()
- cfg.float_field = 12.6
- cfg.integer_field = 123
- cfg.another_integer_field = 234
- cfg.nested = ml_collections.ConfigDict()
- cfg.nested.string_field = 'tom'
-
- print(cfg.integer_field) # Prints 123.
- print(cfg['integer_field']) # Prints 123 as well.
-
- try:
- cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.
- except TypeError as e:
- print(e)
-
- cfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.
- cfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.
-
- print(cfg)
- ```
-
- ### FrozenConfigDict
-
- A `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:
-
- ```python
- import ml_collections
-
- initial_dictionary = {
- 'int': 1,
- 'list': [1, 2],
- 'tuple': (1, 2, 3),
- 'set': {1, 2, 3, 4},
- 'dict_tuple_list': {'tuple_list': ([1, 2], 3)}
- }
-
- cfg = ml_collections.ConfigDict(initial_dictionary)
- frozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)
-
- print(frozen_dict.tuple) # Prints tuple (1, 2, 3)
- print(frozen_dict.list) # Prints tuple (1, 2)
- print(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}
- print(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)
-
- frozen_cfg = ml_collections.FrozenConfigDict(cfg)
- print(frozen_cfg == frozen_dict) # True
- print(hash(frozen_cfg) == hash(frozen_dict)) # True
-
- try:
- frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.
- except AttributeError as e:
- print(e)
-
- # Converting between `FrozenConfigDict` and `ConfigDict`:
- thawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)
- print(thawed_frozen_cfg == cfg) # True
- frozen_cfg_to_cfg = frozen_dict.as_configdict()
- print(frozen_cfg_to_cfg == cfg) # True
- ```
-
- ### FieldReferences and placeholders
-
- A `FieldReference` is useful for having multiple fields use the same value. It
- can also be used for [lazy computation](#lazy-computation).
-
- You can use `placeholder()` as a shortcut to create a `FieldReference` (field)
- with a `None` default value. This is useful if a program uses optional
- configuration fields.
-
- ```python
- import ml_collections
- from ml_collections.config_dict import config_dict
-
- placeholder = ml_collections.FieldReference(0)
- cfg = ml_collections.ConfigDict()
- cfg.placeholder = placeholder
- cfg.optional = config_dict.placeholder(int)
- cfg.nested = ml_collections.ConfigDict()
- cfg.nested.placeholder = placeholder
-
- try:
- cfg.optional = 'tom' # Raises Type error as this field is an integer.
- except TypeError as e:
- print(e)
-
- cfg.optional = 1555 # Works fine.
- cfg.placeholder = 1 # Changes the value of both placeholder and
- # nested.placeholder fields.
-
- print(cfg)
- ```
-
- Note that the indirection provided by `FieldReference`s will be lost if accessed
- through a `ConfigDict`.
-
- ```python
- import ml_collections
-
- placeholder = ml_collections.FieldReference(0)
- cfg.field1 = placeholder
- cfg.field2 = placeholder # This field will be tied to cfg.field1.
- cfg.field3 = cfg.field1 # This will just be an int field initialized to 0.
- ```
-
- ### Lazy computation
-
- Using a `FieldReference` in a standard operation (addition, subtraction,
- multiplication, etc...) will return another `FieldReference` that points to the
- original's value. You can use `FieldReference.get()` to execute the operations
- and get the reference's computed value, and `FieldReference.set()` to change the
- original reference's value.
-
- ```python
- import ml_collections
-
- ref = ml_collections.FieldReference(1)
- print(ref.get()) # Prints 1
-
- add_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten
- add_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer
-
- print(add_ten) # Prints 11
- print(add_ten_lazy.get()) # Prints 11 because ref's value is 1
-
- # Addition is lazily computed for FieldReferences so changing ref will change
- # the value that is used to compute add_ten.
- ref.set(5)
- print(add_ten) # Prints 11
- print(add_ten_lazy.get()) # Prints 15 because ref's value is 5
- ```
-
- If a `FieldReference` has `None` as its original value, or any operation has an
- argument of `None`, then the lazy computation will evaluate to `None`.
-
- We can also use fields in a `ConfigDict` in lazy computation. In this case a
- field will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.reference_field = ml_collections.FieldReference(1)
- config.integer_field = 2
- config.float_field = 2.5
-
- # No lazy evaluatuations because we didn't use get_ref()
- config.no_lazy = config.integer_field * config.float_field
-
- # This will lazily evaluate ONLY config.integer_field
- config.lazy_integer = config.get_ref('integer_field') * config.float_field
-
- # This will lazily evaluate ONLY config.float_field
- config.lazy_float = config.integer_field * config.get_ref('float_field')
-
- # This will lazily evaluate BOTH config.integer_field and config.float_Field
- config.lazy_both = (config.get_ref('integer_field') *
- config.get_ref('float_field'))
-
- config.integer_field = 3
- print(config.no_lazy) # Prints 5.0 - It uses integer_field's original value
-
- print(config.lazy_integer) # Prints 7.5
-
- config.float_field = 3.5
- print(config.lazy_float) # Prints 7.0
- print(config.lazy_both) # Prints 10.5
- ```
-
- #### Changing lazily computed values
-
- Lazily computed values in a ConfigDict can be overridden in the same way as
- regular values. The reference to the `FieldReference` used for the lazy
- computation will be lost and all computations downstream in the reference graph
- will use the new value.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.reference = 1
- config.reference_0 = config.get_ref('reference') + 10
- config.reference_1 = config.get_ref('reference') + 20
- config.reference_1_0 = config.get_ref('reference_1') + 100
-
- print(config.reference) # Prints 1.
- print(config.reference_0) # Prints 11.
- print(config.reference_1) # Prints 21.
- print(config.reference_1_0) # Prints 121.
-
- config.reference_1 = 30
-
- print(config.reference) # Prints 1 (unchanged).
- print(config.reference_0) # Prints 11 (unchanged).
- print(config.reference_1) # Prints 30.
- print(config.reference_1_0) # Prints 130.
- ```
-
- #### Cycles
-
- You cannot create cycles using references. Fortunately
- [the only way](#changing-lazily-computed-values) to create a cycle is by
- assigning a computed field to one that *is not* the result of computation. This
- is forbidden:
-
- ```python
- import ml_collections
- from ml_collections.config_dict import config_dict
-
- config = ml_collections.ConfigDict()
- config.integer_field = 1
- config.bigger_integer_field = config.get_ref('integer_field') + 10
-
- try:
- # Raises a MutabilityError because setting config.integer_field would
- # cause a cycle.
- config.integer_field = config.get_ref('bigger_integer_field') + 2
- except config_dict.MutabilityError as e:
- print(e)
- ```
-
- ### Advanced usage
-
- Here are some more advanced examples showing lazy computation with different
- operators and data types.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.float_field = 12.6
- config.integer_field = 123
- config.list_field = [0, 1, 2]
-
- config.float_multiply_field = config.get_ref('float_field') * 3
- print(config.float_multiply_field) # Prints 37.8
-
- config.float_field = 10.0
- print(config.float_multiply_field) # Prints 30.0
-
- config.longer_list_field = config.get_ref('list_field') + [3, 4, 5]
- print(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]
-
- config.list_field = [-1]
- print(config.longer_list_field) # Prints [-1, 3, 4, 5]
-
- # Both operands can be references
- config.ref_subtraction = (
- config.get_ref('float_field') - config.get_ref('integer_field'))
- print(config.ref_subtraction) # Prints -113.0
-
- config.integer_field = 10
- print(config.ref_subtraction) # Prints 0.0
- ```
-
- ### Equality checking
-
- You can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`
- and `FrozenConfigDict` objects.
-
- ```python
- import ml_collections
-
- dict_1 = {'list': [1, 2]}
- dict_2 = {'list': (1, 2)}
- cfg_1 = ml_collections.ConfigDict(dict_1)
- frozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)
- frozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)
-
- # True because FrozenConfigDict converts lists to tuples
- print(frozen_cfg_1.items() == frozen_cfg_2.items())
- # False because == distinguishes the underlying difference
- print(frozen_cfg_1 == frozen_cfg_2)
-
- # False because == distinguishes these types
- print(frozen_cfg_1 == cfg_1)
- # But eq_as_configdict() treats both as ConfigDict, so these are True:
- print(frozen_cfg_1.eq_as_configdict(cfg_1))
- print(cfg_1.eq_as_configdict(frozen_cfg_1))
- ```
-
- ### Equality checking with lazy computation
-
- Equality checks see if the computed values are the same. Equality is satisfied
- if two sets of computations are different as long as they result in the same
- value.
-
- ```python
- import ml_collections
-
- cfg_1 = ml_collections.ConfigDict()
- cfg_1.a = 1
- cfg_1.b = cfg_1.get_ref('a') + 2
-
- cfg_2 = ml_collections.ConfigDict()
- cfg_2.a = 1
- cfg_2.b = cfg_2.get_ref('a') * 3
-
- # True because all computed values are the same
- print(cfg_1 == cfg_2)
- ```
-
- ### Locking and copying
-
- Here is an example with `lock()` and `deepcopy()`:
-
- ```python
- import copy
- import ml_collections
-
- cfg = ml_collections.ConfigDict()
- cfg.integer_field = 123
-
- # Locking prohibits the addition and deletion of new fields but allows
- # modification of existing values.
- cfg.lock()
- try:
- cfg.integer_field = 124 # Raises AttributeError and suggests valid field.
- except AttributeError as e:
- print(e)
- with cfg.unlocked():
- cfg.integer_field = 1555 # Works fine too.
-
- # Get a copy of the config dict.
- new_cfg = copy.deepcopy(cfg)
- new_cfg.integer_field = -123 # Works fine.
-
- print(cfg)
- ```
-
- ### Dictionary attributes and initialization
-
- ```python
- import ml_collections
-
- referenced_dict = {'inner_float': 3.14}
- d = {
- 'referenced_dict_1': referenced_dict,
- 'referenced_dict_2': referenced_dict,
- 'list_containing_dict': [{'key': 'value'}],
- }
-
- # We can initialize on a dictionary
- cfg = ml_collections.ConfigDict(d)
-
- # Reference structure is preserved
- print(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True
-
- # And the dict attributes have been converted to ConfigDict
- print(type(cfg.referenced_dict_1)) # ConfigDict
-
- # However, the initialization does not look inside of lists, so dicts inside
- # lists are not converted to ConfigDict
- print(type(cfg.list_containing_dict[0])) # dict
- ```
-
- ### More Examples
-
- TODO(mohitreddy): Add links for examples.
-
- For more examples, take a look at these `ml_collections/config_dict/examples/`.
-
- For examples and gotchas specifically about initializing a ConfigDict, see
- `ml_collections/config_dict/examples/config_dict_initialization.py`.
-
- ## Config Flags
-
- This library adds flag definitions to `absl.flags` to handle config files. It
- does not wrap `absl.flags` so if using any standard flag definitions alongside
- config file flags, users must also import `absl.flags`.
-
- Currently, this module adds two new flag types, namely `DEFINE_config_file`
- which accepts a path to a Python file that generates a configuration, and
- `DEFINE_config_dict` which accepts a configuration directly. Configurations are
- dict-like structures (see [ConfigDict](#configdict)) whose nested elements
- can be overridden using special command-line flags. See the examples below
- for more details.
-
- ### Usage
-
- Use `ml_collections.config_flags` alongside `absl.flags`. For
- example:
-
- `script.py`:
-
- ```python
- from absl import app
- from absl import flags
-
- from ml_collections.config_flags import config_flags
-
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_file('my_config')
-
- def main(_):
- print(FLAGS.my_config)
-
- if __name__ == '__main__':
- app.run()
- ```
-
- `config.py`:
-
- ```python
- # Note that this is a valid Python script.
- # get_config() can return an arbitrary dict-like object. However, it is advised
- # to use ml_collections.ConfigDict.
- # See ml_collections/config_dict/examples/config_dict_basic.py
-
- import ml_collections
-
- def get_config():
- config = ml_collections.ConfigDict()
- config.field1 = 1
- config.field2 = 'tom'
- config.nested = ml_collections.ConfigDict()
- config.nested.field = 2.23
- config.tuple = (1, 2, 3)
- return config
- ```
-
- Now, after running:
-
- ```bash
- python script.py -- --my_config=config.py \
- --my_config.field1=8 \
- --my_config.nested.field=2.1 \
- --my_config.tuple='(1, 2, (1, 2))'
- ```
-
- we get:
-
- ```
- field1: 8
- field2: tom
- nested:
- field: 2.1
- tuple: !!python/tuple
- - 1
- - 2
- - !!python/tuple
- - 1
- - 2
- ```
-
- Usage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main
- difference is the configuration is defined in `script.py` instead of in a
- separate file.
-
- `script.py`:
-
- ```python
- from absl import app
- from absl import flags
-
- import ml_collections
- from ml_collections.config_flags import config_flags
-
- config = ml_collections.ConfigDict()
- config.field1 = 1
- config.field2 = 'tom'
- config.nested = ml_collections.ConfigDict()
- config.nested.field = 2.23
- config.tuple = (1, 2, 3)
-
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_dict('my_config', config)
-
- def main(_):
- print(FLAGS.my_config)
-
- if __name__ == '__main__':
- app.run()
- ```
-
- `config_file` flags are compatible with the command-line flag syntax. All the
- following options are supported for non-boolean values in configurations:
-
- * `-(-)config.field=value`
- * `-(-)config.field value`
-
- Options for boolean values are slightly different:
-
- * `-(-)config.boolean_field`: set boolean value to True.
- * `-(-)noconfig.boolean_field`: set boolean value to False.
- * `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or
- `False`.
-
- Note that `-(-)config.boolean_field value` is not supported.
-
- ### Parameterising the get_config() function
-
- It's sometimes useful to be able to pass parameters into `get_config`, and
- change what is returned based on this configuration. One example is if you are
- grid searching over parameters which have a different hierarchical structure -
- the flag needs to be present in the resulting ConfigDict. It would be possible
- to include the union of all possible leaf values in your ConfigDict,
- but this produces a confusing config result as you have to remember which
- parameters will actually have an effect and which won't.
-
- A better system is to pass some configuration, indicating which structure of
- ConfigDict should be returned. An example is the following config file:
-
- ```python
- import ml_collections
-
- def get_config(config_string):
- possible_structures = {
- 'linear': ml_collections.ConfigDict({
- 'model_constructor': 'snt.Linear',
- 'model_config': ml_collections.ConfigDict({
- 'output_size': 42,
- }),
- 'lstm': ml_collections.ConfigDict({
- 'model_constructor': 'snt.LSTM',
- 'model_config': ml_collections.ConfigDict({
- 'hidden_size': 108,
- })
- })
- }
-
- return possible_structures[config_string]
- ```
-
- The value of `config_string` will be anything that is to the right of the first
- colon in the config file path, if one exists. If no colon exists, no value is
- passed to `get_config` (producing a TypeError if `get_config` expects a value.)
-
- The above example can be run like:
-
- ```bash
- python script.py -- --config=path_to_config.py:linear \
- --config.model_config.output_size=256
- ```
-
- or like:
-
- ```bash
- python script.py -- --config=path_to_config.py:lstm \
- --config.model_config.hidden_size=512
- ```
-
- ### Additional features
-
- * Loads any valid python script which defines `get_config()` function
- returning any python object.
- * Automatic locking of the loaded object, if the loaded object defines a
- callable `.lock()` method.
- * Supports command-line overriding of arbitrarily nested values in dict-like
- objects (with key/attribute based getters/setters) of the following types:
- * `types.IntType` (integer)
- * `types.FloatType` (float)
- * `types.BooleanType` (bool)
- * `types.StringType` (string)
- * `types.TupleType` (tuple)
- * Overriding is type safe.
- * Overriding of `TupleType` can be done by passing in the `tuple` as a string
- (see the example in the [Usage](#usage) section).
- * The overriding `tuple` object can be of a different size and have different
- types than the original. Nested tuples are also supported.
-
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
@@ -613,3 +18,611 @@ Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=2.6
Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: AUTHORS
+
+# ML Collections
+
+ML Collections is a library of Python Collections designed for ML use cases.
+
+[](https://ml-collections.readthedocs.io/en/latest/?badge=latest)
+[](https://badge.fury.io/py/ml-collections)
+[](https://github.com/google/ml_collections/actions?query=workflow%3A%22Python+package%22)
+
+## ConfigDict
+
+The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
+structures with dot access to nested elements. Together, they are supposed to be
+used as a main way of expressing configurations of experiments and models.
+
+This document describes example usage of `ConfigDict`, `FrozenConfigDict`,
+`FieldReference`.
+
+### Features
+
+* Dot-based access to fields.
+* Locking mechanism to prevent spelling mistakes.
+* Lazy computation.
+* FrozenConfigDict() class which is immutable and hashable.
+* Type safety.
+* "Did you mean" functionality.
+* Human readable printing (with valid references and cycles), using valid YAML
+ format.
+* Fields can be passed as keyword arguments using the `**` operator.
+* There are two exceptions to the strong type-safety of the ConfigDict. `int`
+ values can be passed in to fields of type `float`. In such a case, the value
+ is type-converted to a `float` before being stored. Similarly, all string
+ types (including Unicode strings) can be stored in fields of type `str` or
+ `unicode`.
+
+### Basic Usage
+
+```python
+import ml_collections
+
+cfg = ml_collections.ConfigDict()
+cfg.float_field = 12.6
+cfg.integer_field = 123
+cfg.another_integer_field = 234
+cfg.nested = ml_collections.ConfigDict()
+cfg.nested.string_field = 'tom'
+
+print(cfg.integer_field) # Prints 123.
+print(cfg['integer_field']) # Prints 123 as well.
+
+try:
+ cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.
+except TypeError as e:
+ print(e)
+
+cfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.
+cfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.
+
+print(cfg)
+```
+
+### FrozenConfigDict
+
+A `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:
+
+```python
+import ml_collections
+
+initial_dictionary = {
+ 'int': 1,
+ 'list': [1, 2],
+ 'tuple': (1, 2, 3),
+ 'set': {1, 2, 3, 4},
+ 'dict_tuple_list': {'tuple_list': ([1, 2], 3)}
+}
+
+cfg = ml_collections.ConfigDict(initial_dictionary)
+frozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)
+
+print(frozen_dict.tuple) # Prints tuple (1, 2, 3)
+print(frozen_dict.list) # Prints tuple (1, 2)
+print(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}
+print(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)
+
+frozen_cfg = ml_collections.FrozenConfigDict(cfg)
+print(frozen_cfg == frozen_dict) # True
+print(hash(frozen_cfg) == hash(frozen_dict)) # True
+
+try:
+ frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.
+except AttributeError as e:
+ print(e)
+
+# Converting between `FrozenConfigDict` and `ConfigDict`:
+thawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)
+print(thawed_frozen_cfg == cfg) # True
+frozen_cfg_to_cfg = frozen_dict.as_configdict()
+print(frozen_cfg_to_cfg == cfg) # True
+```
+
+### FieldReferences and placeholders
+
+A `FieldReference` is useful for having multiple fields use the same value. It
+can also be used for [lazy computation](#lazy-computation).
+
+You can use `placeholder()` as a shortcut to create a `FieldReference` (field)
+with a `None` default value. This is useful if a program uses optional
+configuration fields.
+
+```python
+import ml_collections
+from ml_collections.config_dict import config_dict
+
+placeholder = ml_collections.FieldReference(0)
+cfg = ml_collections.ConfigDict()
+cfg.placeholder = placeholder
+cfg.optional = config_dict.placeholder(int)
+cfg.nested = ml_collections.ConfigDict()
+cfg.nested.placeholder = placeholder
+
+try:
+ cfg.optional = 'tom' # Raises Type error as this field is an integer.
+except TypeError as e:
+ print(e)
+
+cfg.optional = 1555 # Works fine.
+cfg.placeholder = 1 # Changes the value of both placeholder and
+ # nested.placeholder fields.
+
+print(cfg)
+```
+
+Note that the indirection provided by `FieldReference`s will be lost if accessed
+through a `ConfigDict`.
+
+```python
+import ml_collections
+
+placeholder = ml_collections.FieldReference(0)
+cfg.field1 = placeholder
+cfg.field2 = placeholder # This field will be tied to cfg.field1.
+cfg.field3 = cfg.field1 # This will just be an int field initialized to 0.
+```
+
+### Lazy computation
+
+Using a `FieldReference` in a standard operation (addition, subtraction,
+multiplication, etc...) will return another `FieldReference` that points to the
+original's value. You can use `FieldReference.get()` to execute the operations
+and get the reference's computed value, and `FieldReference.set()` to change the
+original reference's value.
+
+```python
+import ml_collections
+
+ref = ml_collections.FieldReference(1)
+print(ref.get()) # Prints 1
+
+add_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten
+add_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer
+
+print(add_ten) # Prints 11
+print(add_ten_lazy.get()) # Prints 11 because ref's value is 1
+
+# Addition is lazily computed for FieldReferences so changing ref will change
+# the value that is used to compute add_ten.
+ref.set(5)
+print(add_ten) # Prints 11
+print(add_ten_lazy.get()) # Prints 15 because ref's value is 5
+```
+
+If a `FieldReference` has `None` as its original value, or any operation has an
+argument of `None`, then the lazy computation will evaluate to `None`.
+
+We can also use fields in a `ConfigDict` in lazy computation. In this case a
+field will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.reference_field = ml_collections.FieldReference(1)
+config.integer_field = 2
+config.float_field = 2.5
+
+# No lazy evaluatuations because we didn't use get_ref()
+config.no_lazy = config.integer_field * config.float_field
+
+# This will lazily evaluate ONLY config.integer_field
+config.lazy_integer = config.get_ref('integer_field') * config.float_field
+
+# This will lazily evaluate ONLY config.float_field
+config.lazy_float = config.integer_field * config.get_ref('float_field')
+
+# This will lazily evaluate BOTH config.integer_field and config.float_Field
+config.lazy_both = (config.get_ref('integer_field') *
+ config.get_ref('float_field'))
+
+config.integer_field = 3
+print(config.no_lazy) # Prints 5.0 - It uses integer_field's original value
+
+print(config.lazy_integer) # Prints 7.5
+
+config.float_field = 3.5
+print(config.lazy_float) # Prints 7.0
+print(config.lazy_both) # Prints 10.5
+```
+
+#### Changing lazily computed values
+
+Lazily computed values in a ConfigDict can be overridden in the same way as
+regular values. The reference to the `FieldReference` used for the lazy
+computation will be lost and all computations downstream in the reference graph
+will use the new value.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.reference = 1
+config.reference_0 = config.get_ref('reference') + 10
+config.reference_1 = config.get_ref('reference') + 20
+config.reference_1_0 = config.get_ref('reference_1') + 100
+
+print(config.reference) # Prints 1.
+print(config.reference_0) # Prints 11.
+print(config.reference_1) # Prints 21.
+print(config.reference_1_0) # Prints 121.
+
+config.reference_1 = 30
+
+print(config.reference) # Prints 1 (unchanged).
+print(config.reference_0) # Prints 11 (unchanged).
+print(config.reference_1) # Prints 30.
+print(config.reference_1_0) # Prints 130.
+```
+
+#### Cycles
+
+You cannot create cycles using references. Fortunately
+[the only way](#changing-lazily-computed-values) to create a cycle is by
+assigning a computed field to one that *is not* the result of computation. This
+is forbidden:
+
+```python
+import ml_collections
+from ml_collections.config_dict import config_dict
+
+config = ml_collections.ConfigDict()
+config.integer_field = 1
+config.bigger_integer_field = config.get_ref('integer_field') + 10
+
+try:
+ # Raises a MutabilityError because setting config.integer_field would
+ # cause a cycle.
+ config.integer_field = config.get_ref('bigger_integer_field') + 2
+except config_dict.MutabilityError as e:
+ print(e)
+```
+
+### Advanced usage
+
+Here are some more advanced examples showing lazy computation with different
+operators and data types.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.float_field = 12.6
+config.integer_field = 123
+config.list_field = [0, 1, 2]
+
+config.float_multiply_field = config.get_ref('float_field') * 3
+print(config.float_multiply_field) # Prints 37.8
+
+config.float_field = 10.0
+print(config.float_multiply_field) # Prints 30.0
+
+config.longer_list_field = config.get_ref('list_field') + [3, 4, 5]
+print(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]
+
+config.list_field = [-1]
+print(config.longer_list_field) # Prints [-1, 3, 4, 5]
+
+# Both operands can be references
+config.ref_subtraction = (
+ config.get_ref('float_field') - config.get_ref('integer_field'))
+print(config.ref_subtraction) # Prints -113.0
+
+config.integer_field = 10
+print(config.ref_subtraction) # Prints 0.0
+```
+
+### Equality checking
+
+You can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`
+and `FrozenConfigDict` objects.
+
+```python
+import ml_collections
+
+dict_1 = {'list': [1, 2]}
+dict_2 = {'list': (1, 2)}
+cfg_1 = ml_collections.ConfigDict(dict_1)
+frozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)
+frozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)
+
+# True because FrozenConfigDict converts lists to tuples
+print(frozen_cfg_1.items() == frozen_cfg_2.items())
+# False because == distinguishes the underlying difference
+print(frozen_cfg_1 == frozen_cfg_2)
+
+# False because == distinguishes these types
+print(frozen_cfg_1 == cfg_1)
+# But eq_as_configdict() treats both as ConfigDict, so these are True:
+print(frozen_cfg_1.eq_as_configdict(cfg_1))
+print(cfg_1.eq_as_configdict(frozen_cfg_1))
+```
+
+### Equality checking with lazy computation
+
+Equality checks see if the computed values are the same. Equality is satisfied
+if two sets of computations are different as long as they result in the same
+value.
+
+```python
+import ml_collections
+
+cfg_1 = ml_collections.ConfigDict()
+cfg_1.a = 1
+cfg_1.b = cfg_1.get_ref('a') + 2
+
+cfg_2 = ml_collections.ConfigDict()
+cfg_2.a = 1
+cfg_2.b = cfg_2.get_ref('a') * 3
+
+# True because all computed values are the same
+print(cfg_1 == cfg_2)
+```
+
+### Locking and copying
+
+Here is an example with `lock()` and `deepcopy()`:
+
+```python
+import copy
+import ml_collections
+
+cfg = ml_collections.ConfigDict()
+cfg.integer_field = 123
+
+# Locking prohibits the addition and deletion of new fields but allows
+# modification of existing values.
+cfg.lock()
+try:
+ cfg.integer_field = 124 # Raises AttributeError and suggests valid field.
+except AttributeError as e:
+ print(e)
+with cfg.unlocked():
+ cfg.integer_field = 1555 # Works fine too.
+
+# Get a copy of the config dict.
+new_cfg = copy.deepcopy(cfg)
+new_cfg.integer_field = -123 # Works fine.
+
+print(cfg)
+```
+
+### Dictionary attributes and initialization
+
+```python
+import ml_collections
+
+referenced_dict = {'inner_float': 3.14}
+d = {
+ 'referenced_dict_1': referenced_dict,
+ 'referenced_dict_2': referenced_dict,
+ 'list_containing_dict': [{'key': 'value'}],
+}
+
+# We can initialize on a dictionary
+cfg = ml_collections.ConfigDict(d)
+
+# Reference structure is preserved
+print(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True
+
+# And the dict attributes have been converted to ConfigDict
+print(type(cfg.referenced_dict_1)) # ConfigDict
+
+# However, the initialization does not look inside of lists, so dicts inside
+# lists are not converted to ConfigDict
+print(type(cfg.list_containing_dict[0])) # dict
+```
+
+### More Examples
+
+For more examples, take a look at
+[`ml_collections/config_dict/examples/`](https://github.com/google/ml_collections/tree/master/ml_collections/config_dict/examples)
+
+For examples and gotchas specifically about initializing a ConfigDict, see
+[`ml_collections/config_dict/examples/config_dict_initialization.py`](https://github.com/google/ml_collections/blob/master/ml_collections/config_dict/examples/config_dict_initialization.py).
+
+## Config Flags
+
+This library adds flag definitions to `absl.flags` to handle config files. It
+does not wrap `absl.flags` so if using any standard flag definitions alongside
+config file flags, users must also import `absl.flags`.
+
+Currently, this module adds two new flag types, namely `DEFINE_config_file`
+which accepts a path to a Python file that generates a configuration, and
+`DEFINE_config_dict` which accepts a configuration directly. Configurations are
+dict-like structures (see [ConfigDict](#configdict)) whose nested elements
+can be overridden using special command-line flags. See the examples below
+for more details.
+
+### Usage
+
+Use `ml_collections.config_flags` alongside `absl.flags`. For
+example:
+
+`script.py`:
+
+```python
+from absl import app
+from absl import flags
+
+from ml_collections.config_flags import config_flags
+
+FLAGS = flags.FLAGS
+config_flags.DEFINE_config_file('my_config')
+
+def main(_):
+ print(FLAGS.my_config)
+
+if __name__ == '__main__':
+ app.run(main)
+```
+
+`config.py`:
+
+```python
+# Note that this is a valid Python script.
+# get_config() can return an arbitrary dict-like object. However, it is advised
+# to use ml_collections.ConfigDict.
+# See ml_collections/config_dict/examples/config_dict_basic.py
+
+import ml_collections
+
+def get_config():
+ config = ml_collections.ConfigDict()
+ config.field1 = 1
+ config.field2 = 'tom'
+ config.nested = ml_collections.ConfigDict()
+ config.nested.field = 2.23
+ config.tuple = (1, 2, 3)
+ return config
+```
+
+Now, after running:
+
+```bash
+python script.py --my_config=config.py \
+ --my_config.field1=8 \
+ --my_config.nested.field=2.1 \
+ --my_config.tuple='(1, 2, (1, 2))'
+```
+
+we get:
+
+```
+field1: 8
+field2: tom
+nested:
+ field: 2.1
+tuple: !!python/tuple
+- 1
+- 2
+- !!python/tuple
+ - 1
+ - 2
+```
+
+Usage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main
+difference is the configuration is defined in `script.py` instead of in a
+separate file.
+
+`script.py`:
+
+```python
+from absl import app
+from absl import flags
+
+import ml_collections
+from ml_collections.config_flags import config_flags
+
+config = ml_collections.ConfigDict()
+config.field1 = 1
+config.field2 = 'tom'
+config.nested = ml_collections.ConfigDict()
+config.nested.field = 2.23
+config.tuple = (1, 2, 3)
+
+FLAGS = flags.FLAGS
+config_flags.DEFINE_config_dict('my_config', config)
+
+def main(_):
+ print(FLAGS.my_config)
+
+if __name__ == '__main__':
+ app.run()
+```
+
+`config_file` flags are compatible with the command-line flag syntax. All the
+following options are supported for non-boolean values in configurations:
+
+* `-(-)config.field=value`
+* `-(-)config.field value`
+
+Options for boolean values are slightly different:
+
+* `-(-)config.boolean_field`: set boolean value to True.
+* `-(-)noconfig.boolean_field`: set boolean value to False.
+* `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or
+ `False`.
+
+Note that `-(-)config.boolean_field value` is not supported.
+
+### Parameterising the get_config() function
+
+It's sometimes useful to be able to pass parameters into `get_config`, and
+change what is returned based on this configuration. One example is if you are
+grid searching over parameters which have a different hierarchical structure -
+the flag needs to be present in the resulting ConfigDict. It would be possible
+to include the union of all possible leaf values in your ConfigDict,
+but this produces a confusing config result as you have to remember which
+parameters will actually have an effect and which won't.
+
+A better system is to pass some configuration, indicating which structure of
+ConfigDict should be returned. An example is the following config file:
+
+```python
+import ml_collections
+
+def get_config(config_string):
+ possible_structures = {
+ 'linear': ml_collections.ConfigDict({
+ 'model_constructor': 'snt.Linear',
+ 'model_config': ml_collections.ConfigDict({
+ 'output_size': 42,
+ }),
+ 'lstm': ml_collections.ConfigDict({
+ 'model_constructor': 'snt.LSTM',
+ 'model_config': ml_collections.ConfigDict({
+ 'hidden_size': 108,
+ })
+ })
+ }
+
+ return possible_structures[config_string]
+```
+
+The value of `config_string` will be anything that is to the right of the first
+colon in the config file path, if one exists. If no colon exists, no value is
+passed to `get_config` (producing a TypeError if `get_config` expects a value.)
+
+The above example can be run like:
+
+```bash
+python script.py -- --config=path_to_config.py:linear \
+ --config.model_config.output_size=256
+```
+
+or like:
+
+```bash
+python script.py -- --config=path_to_config.py:lstm \
+ --config.model_config.hidden_size=512
+```
+
+### Additional features
+
+* Loads any valid python script which defines `get_config()` function
+ returning any python object.
+* Automatic locking of the loaded object, if the loaded object defines a
+ callable `.lock()` method.
+* Supports command-line overriding of arbitrarily nested values in dict-like
+ objects (with key/attribute based getters/setters) of the following types:
+ * `types.IntType` (integer)
+ * `types.FloatType` (float)
+ * `types.BooleanType` (bool)
+ * `types.StringType` (string)
+ * `types.TupleType` (tuple)
+* Overriding is type safe.
+* Overriding of `TupleType` can be done by passing in the `tuple` as a string
+ (see the example in the [Usage](#usage) section).
+* The overriding `tuple` object can be of a different size and have different
+ types than the original. Nested tuples are also supported.
+
+## Authors
+* Sergio Gómez Colmenarejo - sergomez@google.com
+* Wojciech Marian Czarnecki - lejlot@google.com
+* Nicholas Watters
+* Mohit Reddy - mohitreddy@google.com
+
+
diff --git a/README.md b/README.md
index 18c2a89..e0a6d1f 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,10 @@
ML Collections is a library of Python Collections designed for ML use cases.
+[](https://ml-collections.readthedocs.io/en/latest/?badge=latest)
+[](https://badge.fury.io/py/ml-collections)
+[](https://github.com/google/ml_collections/actions?query=workflow%3A%22Python+package%22)
+
## ConfigDict
The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
@@ -390,12 +394,11 @@ print(type(cfg.list_containing_dict[0])) # dict
### More Examples
-TODO(mohitreddy): Add links for examples.
-
-For more examples, take a look at these `ml_collections/config_dict/examples/`.
+For more examples, take a look at
+[`ml_collections/config_dict/examples/`](https://github.com/google/ml_collections/tree/master/ml_collections/config_dict/examples)
For examples and gotchas specifically about initializing a ConfigDict, see
-`ml_collections/config_dict/examples/config_dict_initialization.py`.
+[`ml_collections/config_dict/examples/config_dict_initialization.py`](https://github.com/google/ml_collections/blob/master/ml_collections/config_dict/examples/config_dict_initialization.py).
## Config Flags
@@ -430,7 +433,7 @@ def main(_):
print(FLAGS.my_config)
if __name__ == '__main__':
- app.run()
+ app.run(main)
```
`config.py`:
@@ -456,10 +459,10 @@ def get_config():
Now, after running:
```bash
-python script.py -- --my_config=config.py \
- --my_config.field1=8 \
- --my_config.nested.field=2.1 \
- --my_config.tuple='(1, 2, (1, 2))'
+python script.py --my_config=config.py \
+ --my_config.field1=8 \
+ --my_config.nested.field=2.1 \
+ --my_config.tuple='(1, 2, (1, 2))'
```
we get:
@@ -592,3 +595,9 @@ python script.py -- --config=path_to_config.py:lstm \
(see the example in the [Usage](#usage) section).
* The overriding `tuple` object can be of a different size and have different
types than the original. Nested tuples are also supported.
+
+## Authors
+* Sergio Gómez Colmenarejo - sergomez@google.com
+* Wojciech Marian Czarnecki - lejlot@google.com
+* Nicholas Watters
+* Mohit Reddy - mohitreddy@google.com
diff --git a/debian/changelog b/debian/changelog
index 67cf967..f0d6dd1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+python-ml-collections (0.1.1-1) UNRELEASED; urgency=low
+
+ * New upstream release.
+ * Drop patch fix-test.patch, present upstream.
+
+ -- Debian Janitor <janitor@jelmer.uk> Sun, 20 Mar 2022 13:12:16 -0000
+
python-ml-collections (0.1.0-2) unstable; urgency=medium
* Team upload.
diff --git a/debian/patches/fix-test.patch b/debian/patches/fix-test.patch
deleted file mode 100644
index 2535f15..0000000
--- a/debian/patches/fix-test.patch
+++ /dev/null
@@ -1,67 +0,0 @@
-Description: Cherry pick upstream changes to fix tests
-Author: Nilesh Patra <nilesh@debian.org>
-Forwarded: not-needed
-Last-Update: 2021-08-01
---- a/ml_collections/config_dict/tests/config_dict_test.py
-+++ b/ml_collections/config_dict/tests/config_dict_test.py
-@@ -1,4 +1,4 @@
--# Copyright 2020 The ML Collections Authors.
-+# Copyright 2021 The ML Collections Authors.
- #
- # Licensed under the Apache License, Version 2.0 (the "License");
- # you may not use this file except in compliance with the License.
-@@ -16,7 +16,8 @@
- """Tests for ml_collections.ConfigDict."""
-
- import abc
--import collections as python_collections
-+from collections import abc as collections_abc
-+import functools
- import json
- import pickle
- import sys
-@@ -57,7 +58,7 @@
- _test_object = _TestClass()
-
-
--class _TestClassNoStr(): # pylint: disable=old-style-class
-+class _TestClassNoStr():
- pass
-
-
-@@ -589,7 +590,7 @@
- hash(cfg)
-
- # Ensure Python realizes ConfigDict is not hashable.
-- self.assertNotIsInstance(cfg, python_collections.Hashable)
-+ self.assertNotIsInstance(cfg, collections_abc.Hashable)
-
- def testDidYouMeanFeature(self):
- """Tests 'did you mean' suggestions."""
-@@ -675,7 +676,7 @@
- def testDeYaml(self):
- """Tests YAML deserialization."""
- cfg = _get_test_config_dict()
-- deyamled = yaml.load(cfg.to_yaml())
-+ deyamled = yaml.load(cfg.to_yaml(), yaml.UnsafeLoader)
- self.assertEqual(cfg, deyamled)
-
- def testJSONConversion(self):
-@@ -723,7 +724,7 @@
- cfg_dict.r1 = field
- cfg_dict.r2 = field
-
-- cfg_load = yaml.load(repr(cfg_dict))
-+ cfg_load = yaml.load(repr(cfg_dict), yaml.UnsafeLoader)
-
- # Test FieldReferences are preserved
- cfg_load['r1'].set(2)
-@@ -1331,7 +1332,7 @@
- This checks backward compatibility of deserialisation.
- """
- cfg = ml_collections.ConfigDict(dict(a=1))
-- self.assertTrue(yaml.load(cfg.to_yaml())._convert_dict)
-+ self.assertTrue(yaml.load(cfg.to_yaml(), yaml.UnsafeLoader)._convert_dict)
-
- def testRecursiveRename(self):
- """Test recursive_rename.
diff --git a/debian/patches/noRequirements.patch b/debian/patches/noRequirements.patch
index ce17e70..9689f2b 100644
--- a/debian/patches/noRequirements.patch
+++ b/debian/patches/noRequirements.patch
@@ -2,9 +2,11 @@ Description: Do not check requirements, for they will already be available for o
Author: Steffen Moeller <moeller@debian.org>
Forwarded: not-needed
Last-Update: 2021-08-01
---- a/setup.py
-+++ b/setup.py
-@@ -19,9 +19,9 @@
+Index: python-ml-collections/setup.py
+===================================================================
+--- python-ml-collections.orig/setup.py
++++ python-ml-collections/setup.py
+@@ -19,9 +19,9 @@ from setuptools import find_namespace_pa
from setuptools import setup
@@ -16,8 +18,8 @@ Last-Update: 2021-08-01
+# return fp.read().splitlines()
- _VERSION = '0.1.0'
-@@ -39,8 +39,8 @@
+ _VERSION = '0.1.1'
+@@ -39,8 +39,8 @@ setup(
license='Apache 2.0',
# Contained modules and scripts.
packages=find_namespace_packages(exclude=['*_test.py']),
diff --git a/debian/patches/series b/debian/patches/series
index 5272e4e..e60c0f7 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,2 +1 @@
noRequirements.patch
-fix-test.patch
diff --git a/docs/conf.py b/docs/conf.py
new file mode 100644
index 0000000..f9630e6
--- /dev/null
+++ b/docs/conf.py
@@ -0,0 +1,83 @@
+# Copyright 2021 The ML Collections Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+
+# -- Project information -----------------------------------------------------
+
+project = 'ml_collections'
+copyright = '2020, The ML Collection Authors'
+author = 'The ML Collection Authors'
+
+# The full version, including alpha/beta/rc tags
+release = '0.1.0'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+ 'sphinx.ext.autodoc',
+ 'sphinx.ext.autosummary',
+ 'sphinx.ext.intersphinx',
+ 'sphinx.ext.mathjax',
+ 'sphinx.ext.napoleon',
+ 'sphinx.ext.viewcode',
+ 'nbsphinx',
+ 'recommonmark',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+autosummary_generate = True
+
+master_doc = 'index'
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages. See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
\ No newline at end of file
diff --git a/ml_collections.egg-info/PKG-INFO b/ml_collections.egg-info/PKG-INFO
index 1f16ec9..c7a69aa 100644
--- a/ml_collections.egg-info/PKG-INFO
+++ b/ml_collections.egg-info/PKG-INFO
@@ -1,606 +1,11 @@
Metadata-Version: 2.1
Name: ml-collections
-Version: 0.1.0
+Version: 0.1.1
Summary: ML Collections is a library of Python collections designed for ML usecases.
Home-page: https://github.com/google/ml_collections
Author: ML Collections Authors
Author-email: ml-collections@google.com
License: Apache 2.0
-Description: # ML Collections
-
- ML Collections is a library of Python Collections designed for ML use cases.
-
- ## ConfigDict
-
- The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
- structures with dot access to nested elements. Together, they are supposed to be
- used as a main way of expressing configurations of experiments and models.
-
- This document describes example usage of `ConfigDict`, `FrozenConfigDict`,
- `FieldReference`.
-
- ### Features
-
- * Dot-based access to fields.
- * Locking mechanism to prevent spelling mistakes.
- * Lazy computation.
- * FrozenConfigDict() class which is immutable and hashable.
- * Type safety.
- * "Did you mean" functionality.
- * Human readable printing (with valid references and cycles), using valid YAML
- format.
- * Fields can be passed as keyword arguments using the `**` operator.
- * There are two exceptions to the strong type-safety of the ConfigDict. `int`
- values can be passed in to fields of type `float`. In such a case, the value
- is type-converted to a `float` before being stored. Similarly, all string
- types (including Unicode strings) can be stored in fields of type `str` or
- `unicode`.
-
- ### Basic Usage
-
- ```python
- import ml_collections
-
- cfg = ml_collections.ConfigDict()
- cfg.float_field = 12.6
- cfg.integer_field = 123
- cfg.another_integer_field = 234
- cfg.nested = ml_collections.ConfigDict()
- cfg.nested.string_field = 'tom'
-
- print(cfg.integer_field) # Prints 123.
- print(cfg['integer_field']) # Prints 123 as well.
-
- try:
- cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.
- except TypeError as e:
- print(e)
-
- cfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.
- cfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.
-
- print(cfg)
- ```
-
- ### FrozenConfigDict
-
- A `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:
-
- ```python
- import ml_collections
-
- initial_dictionary = {
- 'int': 1,
- 'list': [1, 2],
- 'tuple': (1, 2, 3),
- 'set': {1, 2, 3, 4},
- 'dict_tuple_list': {'tuple_list': ([1, 2], 3)}
- }
-
- cfg = ml_collections.ConfigDict(initial_dictionary)
- frozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)
-
- print(frozen_dict.tuple) # Prints tuple (1, 2, 3)
- print(frozen_dict.list) # Prints tuple (1, 2)
- print(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}
- print(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)
-
- frozen_cfg = ml_collections.FrozenConfigDict(cfg)
- print(frozen_cfg == frozen_dict) # True
- print(hash(frozen_cfg) == hash(frozen_dict)) # True
-
- try:
- frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.
- except AttributeError as e:
- print(e)
-
- # Converting between `FrozenConfigDict` and `ConfigDict`:
- thawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)
- print(thawed_frozen_cfg == cfg) # True
- frozen_cfg_to_cfg = frozen_dict.as_configdict()
- print(frozen_cfg_to_cfg == cfg) # True
- ```
-
- ### FieldReferences and placeholders
-
- A `FieldReference` is useful for having multiple fields use the same value. It
- can also be used for [lazy computation](#lazy-computation).
-
- You can use `placeholder()` as a shortcut to create a `FieldReference` (field)
- with a `None` default value. This is useful if a program uses optional
- configuration fields.
-
- ```python
- import ml_collections
- from ml_collections.config_dict import config_dict
-
- placeholder = ml_collections.FieldReference(0)
- cfg = ml_collections.ConfigDict()
- cfg.placeholder = placeholder
- cfg.optional = config_dict.placeholder(int)
- cfg.nested = ml_collections.ConfigDict()
- cfg.nested.placeholder = placeholder
-
- try:
- cfg.optional = 'tom' # Raises Type error as this field is an integer.
- except TypeError as e:
- print(e)
-
- cfg.optional = 1555 # Works fine.
- cfg.placeholder = 1 # Changes the value of both placeholder and
- # nested.placeholder fields.
-
- print(cfg)
- ```
-
- Note that the indirection provided by `FieldReference`s will be lost if accessed
- through a `ConfigDict`.
-
- ```python
- import ml_collections
-
- placeholder = ml_collections.FieldReference(0)
- cfg.field1 = placeholder
- cfg.field2 = placeholder # This field will be tied to cfg.field1.
- cfg.field3 = cfg.field1 # This will just be an int field initialized to 0.
- ```
-
- ### Lazy computation
-
- Using a `FieldReference` in a standard operation (addition, subtraction,
- multiplication, etc...) will return another `FieldReference` that points to the
- original's value. You can use `FieldReference.get()` to execute the operations
- and get the reference's computed value, and `FieldReference.set()` to change the
- original reference's value.
-
- ```python
- import ml_collections
-
- ref = ml_collections.FieldReference(1)
- print(ref.get()) # Prints 1
-
- add_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten
- add_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer
-
- print(add_ten) # Prints 11
- print(add_ten_lazy.get()) # Prints 11 because ref's value is 1
-
- # Addition is lazily computed for FieldReferences so changing ref will change
- # the value that is used to compute add_ten.
- ref.set(5)
- print(add_ten) # Prints 11
- print(add_ten_lazy.get()) # Prints 15 because ref's value is 5
- ```
-
- If a `FieldReference` has `None` as its original value, or any operation has an
- argument of `None`, then the lazy computation will evaluate to `None`.
-
- We can also use fields in a `ConfigDict` in lazy computation. In this case a
- field will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.reference_field = ml_collections.FieldReference(1)
- config.integer_field = 2
- config.float_field = 2.5
-
- # No lazy evaluatuations because we didn't use get_ref()
- config.no_lazy = config.integer_field * config.float_field
-
- # This will lazily evaluate ONLY config.integer_field
- config.lazy_integer = config.get_ref('integer_field') * config.float_field
-
- # This will lazily evaluate ONLY config.float_field
- config.lazy_float = config.integer_field * config.get_ref('float_field')
-
- # This will lazily evaluate BOTH config.integer_field and config.float_Field
- config.lazy_both = (config.get_ref('integer_field') *
- config.get_ref('float_field'))
-
- config.integer_field = 3
- print(config.no_lazy) # Prints 5.0 - It uses integer_field's original value
-
- print(config.lazy_integer) # Prints 7.5
-
- config.float_field = 3.5
- print(config.lazy_float) # Prints 7.0
- print(config.lazy_both) # Prints 10.5
- ```
-
- #### Changing lazily computed values
-
- Lazily computed values in a ConfigDict can be overridden in the same way as
- regular values. The reference to the `FieldReference` used for the lazy
- computation will be lost and all computations downstream in the reference graph
- will use the new value.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.reference = 1
- config.reference_0 = config.get_ref('reference') + 10
- config.reference_1 = config.get_ref('reference') + 20
- config.reference_1_0 = config.get_ref('reference_1') + 100
-
- print(config.reference) # Prints 1.
- print(config.reference_0) # Prints 11.
- print(config.reference_1) # Prints 21.
- print(config.reference_1_0) # Prints 121.
-
- config.reference_1 = 30
-
- print(config.reference) # Prints 1 (unchanged).
- print(config.reference_0) # Prints 11 (unchanged).
- print(config.reference_1) # Prints 30.
- print(config.reference_1_0) # Prints 130.
- ```
-
- #### Cycles
-
- You cannot create cycles using references. Fortunately
- [the only way](#changing-lazily-computed-values) to create a cycle is by
- assigning a computed field to one that *is not* the result of computation. This
- is forbidden:
-
- ```python
- import ml_collections
- from ml_collections.config_dict import config_dict
-
- config = ml_collections.ConfigDict()
- config.integer_field = 1
- config.bigger_integer_field = config.get_ref('integer_field') + 10
-
- try:
- # Raises a MutabilityError because setting config.integer_field would
- # cause a cycle.
- config.integer_field = config.get_ref('bigger_integer_field') + 2
- except config_dict.MutabilityError as e:
- print(e)
- ```
-
- ### Advanced usage
-
- Here are some more advanced examples showing lazy computation with different
- operators and data types.
-
- ```python
- import ml_collections
-
- config = ml_collections.ConfigDict()
- config.float_field = 12.6
- config.integer_field = 123
- config.list_field = [0, 1, 2]
-
- config.float_multiply_field = config.get_ref('float_field') * 3
- print(config.float_multiply_field) # Prints 37.8
-
- config.float_field = 10.0
- print(config.float_multiply_field) # Prints 30.0
-
- config.longer_list_field = config.get_ref('list_field') + [3, 4, 5]
- print(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]
-
- config.list_field = [-1]
- print(config.longer_list_field) # Prints [-1, 3, 4, 5]
-
- # Both operands can be references
- config.ref_subtraction = (
- config.get_ref('float_field') - config.get_ref('integer_field'))
- print(config.ref_subtraction) # Prints -113.0
-
- config.integer_field = 10
- print(config.ref_subtraction) # Prints 0.0
- ```
-
- ### Equality checking
-
- You can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`
- and `FrozenConfigDict` objects.
-
- ```python
- import ml_collections
-
- dict_1 = {'list': [1, 2]}
- dict_2 = {'list': (1, 2)}
- cfg_1 = ml_collections.ConfigDict(dict_1)
- frozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)
- frozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)
-
- # True because FrozenConfigDict converts lists to tuples
- print(frozen_cfg_1.items() == frozen_cfg_2.items())
- # False because == distinguishes the underlying difference
- print(frozen_cfg_1 == frozen_cfg_2)
-
- # False because == distinguishes these types
- print(frozen_cfg_1 == cfg_1)
- # But eq_as_configdict() treats both as ConfigDict, so these are True:
- print(frozen_cfg_1.eq_as_configdict(cfg_1))
- print(cfg_1.eq_as_configdict(frozen_cfg_1))
- ```
-
- ### Equality checking with lazy computation
-
- Equality checks see if the computed values are the same. Equality is satisfied
- if two sets of computations are different as long as they result in the same
- value.
-
- ```python
- import ml_collections
-
- cfg_1 = ml_collections.ConfigDict()
- cfg_1.a = 1
- cfg_1.b = cfg_1.get_ref('a') + 2
-
- cfg_2 = ml_collections.ConfigDict()
- cfg_2.a = 1
- cfg_2.b = cfg_2.get_ref('a') * 3
-
- # True because all computed values are the same
- print(cfg_1 == cfg_2)
- ```
-
- ### Locking and copying
-
- Here is an example with `lock()` and `deepcopy()`:
-
- ```python
- import copy
- import ml_collections
-
- cfg = ml_collections.ConfigDict()
- cfg.integer_field = 123
-
- # Locking prohibits the addition and deletion of new fields but allows
- # modification of existing values.
- cfg.lock()
- try:
- cfg.integer_field = 124 # Raises AttributeError and suggests valid field.
- except AttributeError as e:
- print(e)
- with cfg.unlocked():
- cfg.integer_field = 1555 # Works fine too.
-
- # Get a copy of the config dict.
- new_cfg = copy.deepcopy(cfg)
- new_cfg.integer_field = -123 # Works fine.
-
- print(cfg)
- ```
-
- ### Dictionary attributes and initialization
-
- ```python
- import ml_collections
-
- referenced_dict = {'inner_float': 3.14}
- d = {
- 'referenced_dict_1': referenced_dict,
- 'referenced_dict_2': referenced_dict,
- 'list_containing_dict': [{'key': 'value'}],
- }
-
- # We can initialize on a dictionary
- cfg = ml_collections.ConfigDict(d)
-
- # Reference structure is preserved
- print(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True
-
- # And the dict attributes have been converted to ConfigDict
- print(type(cfg.referenced_dict_1)) # ConfigDict
-
- # However, the initialization does not look inside of lists, so dicts inside
- # lists are not converted to ConfigDict
- print(type(cfg.list_containing_dict[0])) # dict
- ```
-
- ### More Examples
-
- TODO(mohitreddy): Add links for examples.
-
- For more examples, take a look at these `ml_collections/config_dict/examples/`.
-
- For examples and gotchas specifically about initializing a ConfigDict, see
- `ml_collections/config_dict/examples/config_dict_initialization.py`.
-
- ## Config Flags
-
- This library adds flag definitions to `absl.flags` to handle config files. It
- does not wrap `absl.flags` so if using any standard flag definitions alongside
- config file flags, users must also import `absl.flags`.
-
- Currently, this module adds two new flag types, namely `DEFINE_config_file`
- which accepts a path to a Python file that generates a configuration, and
- `DEFINE_config_dict` which accepts a configuration directly. Configurations are
- dict-like structures (see [ConfigDict](#configdict)) whose nested elements
- can be overridden using special command-line flags. See the examples below
- for more details.
-
- ### Usage
-
- Use `ml_collections.config_flags` alongside `absl.flags`. For
- example:
-
- `script.py`:
-
- ```python
- from absl import app
- from absl import flags
-
- from ml_collections.config_flags import config_flags
-
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_file('my_config')
-
- def main(_):
- print(FLAGS.my_config)
-
- if __name__ == '__main__':
- app.run()
- ```
-
- `config.py`:
-
- ```python
- # Note that this is a valid Python script.
- # get_config() can return an arbitrary dict-like object. However, it is advised
- # to use ml_collections.ConfigDict.
- # See ml_collections/config_dict/examples/config_dict_basic.py
-
- import ml_collections
-
- def get_config():
- config = ml_collections.ConfigDict()
- config.field1 = 1
- config.field2 = 'tom'
- config.nested = ml_collections.ConfigDict()
- config.nested.field = 2.23
- config.tuple = (1, 2, 3)
- return config
- ```
-
- Now, after running:
-
- ```bash
- python script.py -- --my_config=config.py \
- --my_config.field1=8 \
- --my_config.nested.field=2.1 \
- --my_config.tuple='(1, 2, (1, 2))'
- ```
-
- we get:
-
- ```
- field1: 8
- field2: tom
- nested:
- field: 2.1
- tuple: !!python/tuple
- - 1
- - 2
- - !!python/tuple
- - 1
- - 2
- ```
-
- Usage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main
- difference is the configuration is defined in `script.py` instead of in a
- separate file.
-
- `script.py`:
-
- ```python
- from absl import app
- from absl import flags
-
- import ml_collections
- from ml_collections.config_flags import config_flags
-
- config = ml_collections.ConfigDict()
- config.field1 = 1
- config.field2 = 'tom'
- config.nested = ml_collections.ConfigDict()
- config.nested.field = 2.23
- config.tuple = (1, 2, 3)
-
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_dict('my_config', config)
-
- def main(_):
- print(FLAGS.my_config)
-
- if __name__ == '__main__':
- app.run()
- ```
-
- `config_file` flags are compatible with the command-line flag syntax. All the
- following options are supported for non-boolean values in configurations:
-
- * `-(-)config.field=value`
- * `-(-)config.field value`
-
- Options for boolean values are slightly different:
-
- * `-(-)config.boolean_field`: set boolean value to True.
- * `-(-)noconfig.boolean_field`: set boolean value to False.
- * `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or
- `False`.
-
- Note that `-(-)config.boolean_field value` is not supported.
-
- ### Parameterising the get_config() function
-
- It's sometimes useful to be able to pass parameters into `get_config`, and
- change what is returned based on this configuration. One example is if you are
- grid searching over parameters which have a different hierarchical structure -
- the flag needs to be present in the resulting ConfigDict. It would be possible
- to include the union of all possible leaf values in your ConfigDict,
- but this produces a confusing config result as you have to remember which
- parameters will actually have an effect and which won't.
-
- A better system is to pass some configuration, indicating which structure of
- ConfigDict should be returned. An example is the following config file:
-
- ```python
- import ml_collections
-
- def get_config(config_string):
- possible_structures = {
- 'linear': ml_collections.ConfigDict({
- 'model_constructor': 'snt.Linear',
- 'model_config': ml_collections.ConfigDict({
- 'output_size': 42,
- }),
- 'lstm': ml_collections.ConfigDict({
- 'model_constructor': 'snt.LSTM',
- 'model_config': ml_collections.ConfigDict({
- 'hidden_size': 108,
- })
- })
- }
-
- return possible_structures[config_string]
- ```
-
- The value of `config_string` will be anything that is to the right of the first
- colon in the config file path, if one exists. If no colon exists, no value is
- passed to `get_config` (producing a TypeError if `get_config` expects a value.)
-
- The above example can be run like:
-
- ```bash
- python script.py -- --config=path_to_config.py:linear \
- --config.model_config.output_size=256
- ```
-
- or like:
-
- ```bash
- python script.py -- --config=path_to_config.py:lstm \
- --config.model_config.hidden_size=512
- ```
-
- ### Additional features
-
- * Loads any valid python script which defines `get_config()` function
- returning any python object.
- * Automatic locking of the loaded object, if the loaded object defines a
- callable `.lock()` method.
- * Supports command-line overriding of arbitrarily nested values in dict-like
- objects (with key/attribute based getters/setters) of the following types:
- * `types.IntType` (integer)
- * `types.FloatType` (float)
- * `types.BooleanType` (bool)
- * `types.StringType` (string)
- * `types.TupleType` (tuple)
- * Overriding is type safe.
- * Overriding of `TupleType` can be done by passing in the `tuple` as a string
- (see the example in the [Usage](#usage) section).
- * The overriding `tuple` object can be of a different size and have different
- types than the original. Nested tuples are also supported.
-
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
@@ -613,3 +18,611 @@ Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=2.6
Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: AUTHORS
+
+# ML Collections
+
+ML Collections is a library of Python Collections designed for ML use cases.
+
+[](https://ml-collections.readthedocs.io/en/latest/?badge=latest)
+[](https://badge.fury.io/py/ml-collections)
+[](https://github.com/google/ml_collections/actions?query=workflow%3A%22Python+package%22)
+
+## ConfigDict
+
+The two classes called `ConfigDict` and `FrozenConfigDict` are "dict-like" data
+structures with dot access to nested elements. Together, they are supposed to be
+used as a main way of expressing configurations of experiments and models.
+
+This document describes example usage of `ConfigDict`, `FrozenConfigDict`,
+`FieldReference`.
+
+### Features
+
+* Dot-based access to fields.
+* Locking mechanism to prevent spelling mistakes.
+* Lazy computation.
+* FrozenConfigDict() class which is immutable and hashable.
+* Type safety.
+* "Did you mean" functionality.
+* Human readable printing (with valid references and cycles), using valid YAML
+ format.
+* Fields can be passed as keyword arguments using the `**` operator.
+* There are two exceptions to the strong type-safety of the ConfigDict. `int`
+ values can be passed in to fields of type `float`. In such a case, the value
+ is type-converted to a `float` before being stored. Similarly, all string
+ types (including Unicode strings) can be stored in fields of type `str` or
+ `unicode`.
+
+### Basic Usage
+
+```python
+import ml_collections
+
+cfg = ml_collections.ConfigDict()
+cfg.float_field = 12.6
+cfg.integer_field = 123
+cfg.another_integer_field = 234
+cfg.nested = ml_collections.ConfigDict()
+cfg.nested.string_field = 'tom'
+
+print(cfg.integer_field) # Prints 123.
+print(cfg['integer_field']) # Prints 123 as well.
+
+try:
+ cfg.integer_field = 'tom' # Raises TypeError as this field is an integer.
+except TypeError as e:
+ print(e)
+
+cfg.float_field = 12 # Works: `Int` types can be assigned to `Float`.
+cfg.nested.string_field = u'bob' # `String` fields can store Unicode strings.
+
+print(cfg)
+```
+
+### FrozenConfigDict
+
+A `FrozenConfigDict`is an immutable, hashable type of `ConfigDict`:
+
+```python
+import ml_collections
+
+initial_dictionary = {
+ 'int': 1,
+ 'list': [1, 2],
+ 'tuple': (1, 2, 3),
+ 'set': {1, 2, 3, 4},
+ 'dict_tuple_list': {'tuple_list': ([1, 2], 3)}
+}
+
+cfg = ml_collections.ConfigDict(initial_dictionary)
+frozen_dict = ml_collections.FrozenConfigDict(initial_dictionary)
+
+print(frozen_dict.tuple) # Prints tuple (1, 2, 3)
+print(frozen_dict.list) # Prints tuple (1, 2)
+print(frozen_dict.set) # Prints frozenset {1, 2, 3, 4}
+print(frozen_dict.dict_tuple_list.tuple_list[0]) # Prints tuple (1, 2)
+
+frozen_cfg = ml_collections.FrozenConfigDict(cfg)
+print(frozen_cfg == frozen_dict) # True
+print(hash(frozen_cfg) == hash(frozen_dict)) # True
+
+try:
+ frozen_dict.int = 2 # Raises TypeError as FrozenConfigDict is immutable.
+except AttributeError as e:
+ print(e)
+
+# Converting between `FrozenConfigDict` and `ConfigDict`:
+thawed_frozen_cfg = ml_collections.ConfigDict(frozen_dict)
+print(thawed_frozen_cfg == cfg) # True
+frozen_cfg_to_cfg = frozen_dict.as_configdict()
+print(frozen_cfg_to_cfg == cfg) # True
+```
+
+### FieldReferences and placeholders
+
+A `FieldReference` is useful for having multiple fields use the same value. It
+can also be used for [lazy computation](#lazy-computation).
+
+You can use `placeholder()` as a shortcut to create a `FieldReference` (field)
+with a `None` default value. This is useful if a program uses optional
+configuration fields.
+
+```python
+import ml_collections
+from ml_collections.config_dict import config_dict
+
+placeholder = ml_collections.FieldReference(0)
+cfg = ml_collections.ConfigDict()
+cfg.placeholder = placeholder
+cfg.optional = config_dict.placeholder(int)
+cfg.nested = ml_collections.ConfigDict()
+cfg.nested.placeholder = placeholder
+
+try:
+ cfg.optional = 'tom' # Raises Type error as this field is an integer.
+except TypeError as e:
+ print(e)
+
+cfg.optional = 1555 # Works fine.
+cfg.placeholder = 1 # Changes the value of both placeholder and
+ # nested.placeholder fields.
+
+print(cfg)
+```
+
+Note that the indirection provided by `FieldReference`s will be lost if accessed
+through a `ConfigDict`.
+
+```python
+import ml_collections
+
+placeholder = ml_collections.FieldReference(0)
+cfg.field1 = placeholder
+cfg.field2 = placeholder # This field will be tied to cfg.field1.
+cfg.field3 = cfg.field1 # This will just be an int field initialized to 0.
+```
+
+### Lazy computation
+
+Using a `FieldReference` in a standard operation (addition, subtraction,
+multiplication, etc...) will return another `FieldReference` that points to the
+original's value. You can use `FieldReference.get()` to execute the operations
+and get the reference's computed value, and `FieldReference.set()` to change the
+original reference's value.
+
+```python
+import ml_collections
+
+ref = ml_collections.FieldReference(1)
+print(ref.get()) # Prints 1
+
+add_ten = ref.get() + 10 # ref.get() is an integer and so is add_ten
+add_ten_lazy = ref + 10 # add_ten_lazy is a FieldReference - NOT an integer
+
+print(add_ten) # Prints 11
+print(add_ten_lazy.get()) # Prints 11 because ref's value is 1
+
+# Addition is lazily computed for FieldReferences so changing ref will change
+# the value that is used to compute add_ten.
+ref.set(5)
+print(add_ten) # Prints 11
+print(add_ten_lazy.get()) # Prints 15 because ref's value is 5
+```
+
+If a `FieldReference` has `None` as its original value, or any operation has an
+argument of `None`, then the lazy computation will evaluate to `None`.
+
+We can also use fields in a `ConfigDict` in lazy computation. In this case a
+field will only be lazily evaluated if `ConfigDict.get_ref()` is used to get it.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.reference_field = ml_collections.FieldReference(1)
+config.integer_field = 2
+config.float_field = 2.5
+
+# No lazy evaluatuations because we didn't use get_ref()
+config.no_lazy = config.integer_field * config.float_field
+
+# This will lazily evaluate ONLY config.integer_field
+config.lazy_integer = config.get_ref('integer_field') * config.float_field
+
+# This will lazily evaluate ONLY config.float_field
+config.lazy_float = config.integer_field * config.get_ref('float_field')
+
+# This will lazily evaluate BOTH config.integer_field and config.float_Field
+config.lazy_both = (config.get_ref('integer_field') *
+ config.get_ref('float_field'))
+
+config.integer_field = 3
+print(config.no_lazy) # Prints 5.0 - It uses integer_field's original value
+
+print(config.lazy_integer) # Prints 7.5
+
+config.float_field = 3.5
+print(config.lazy_float) # Prints 7.0
+print(config.lazy_both) # Prints 10.5
+```
+
+#### Changing lazily computed values
+
+Lazily computed values in a ConfigDict can be overridden in the same way as
+regular values. The reference to the `FieldReference` used for the lazy
+computation will be lost and all computations downstream in the reference graph
+will use the new value.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.reference = 1
+config.reference_0 = config.get_ref('reference') + 10
+config.reference_1 = config.get_ref('reference') + 20
+config.reference_1_0 = config.get_ref('reference_1') + 100
+
+print(config.reference) # Prints 1.
+print(config.reference_0) # Prints 11.
+print(config.reference_1) # Prints 21.
+print(config.reference_1_0) # Prints 121.
+
+config.reference_1 = 30
+
+print(config.reference) # Prints 1 (unchanged).
+print(config.reference_0) # Prints 11 (unchanged).
+print(config.reference_1) # Prints 30.
+print(config.reference_1_0) # Prints 130.
+```
+
+#### Cycles
+
+You cannot create cycles using references. Fortunately
+[the only way](#changing-lazily-computed-values) to create a cycle is by
+assigning a computed field to one that *is not* the result of computation. This
+is forbidden:
+
+```python
+import ml_collections
+from ml_collections.config_dict import config_dict
+
+config = ml_collections.ConfigDict()
+config.integer_field = 1
+config.bigger_integer_field = config.get_ref('integer_field') + 10
+
+try:
+ # Raises a MutabilityError because setting config.integer_field would
+ # cause a cycle.
+ config.integer_field = config.get_ref('bigger_integer_field') + 2
+except config_dict.MutabilityError as e:
+ print(e)
+```
+
+### Advanced usage
+
+Here are some more advanced examples showing lazy computation with different
+operators and data types.
+
+```python
+import ml_collections
+
+config = ml_collections.ConfigDict()
+config.float_field = 12.6
+config.integer_field = 123
+config.list_field = [0, 1, 2]
+
+config.float_multiply_field = config.get_ref('float_field') * 3
+print(config.float_multiply_field) # Prints 37.8
+
+config.float_field = 10.0
+print(config.float_multiply_field) # Prints 30.0
+
+config.longer_list_field = config.get_ref('list_field') + [3, 4, 5]
+print(config.longer_list_field) # Prints [0, 1, 2, 3, 4, 5]
+
+config.list_field = [-1]
+print(config.longer_list_field) # Prints [-1, 3, 4, 5]
+
+# Both operands can be references
+config.ref_subtraction = (
+ config.get_ref('float_field') - config.get_ref('integer_field'))
+print(config.ref_subtraction) # Prints -113.0
+
+config.integer_field = 10
+print(config.ref_subtraction) # Prints 0.0
+```
+
+### Equality checking
+
+You can use `==` and `.eq_as_configdict()` to check equality among `ConfigDict`
+and `FrozenConfigDict` objects.
+
+```python
+import ml_collections
+
+dict_1 = {'list': [1, 2]}
+dict_2 = {'list': (1, 2)}
+cfg_1 = ml_collections.ConfigDict(dict_1)
+frozen_cfg_1 = ml_collections.FrozenConfigDict(dict_1)
+frozen_cfg_2 = ml_collections.FrozenConfigDict(dict_2)
+
+# True because FrozenConfigDict converts lists to tuples
+print(frozen_cfg_1.items() == frozen_cfg_2.items())
+# False because == distinguishes the underlying difference
+print(frozen_cfg_1 == frozen_cfg_2)
+
+# False because == distinguishes these types
+print(frozen_cfg_1 == cfg_1)
+# But eq_as_configdict() treats both as ConfigDict, so these are True:
+print(frozen_cfg_1.eq_as_configdict(cfg_1))
+print(cfg_1.eq_as_configdict(frozen_cfg_1))
+```
+
+### Equality checking with lazy computation
+
+Equality checks see if the computed values are the same. Equality is satisfied
+if two sets of computations are different as long as they result in the same
+value.
+
+```python
+import ml_collections
+
+cfg_1 = ml_collections.ConfigDict()
+cfg_1.a = 1
+cfg_1.b = cfg_1.get_ref('a') + 2
+
+cfg_2 = ml_collections.ConfigDict()
+cfg_2.a = 1
+cfg_2.b = cfg_2.get_ref('a') * 3
+
+# True because all computed values are the same
+print(cfg_1 == cfg_2)
+```
+
+### Locking and copying
+
+Here is an example with `lock()` and `deepcopy()`:
+
+```python
+import copy
+import ml_collections
+
+cfg = ml_collections.ConfigDict()
+cfg.integer_field = 123
+
+# Locking prohibits the addition and deletion of new fields but allows
+# modification of existing values.
+cfg.lock()
+try:
+ cfg.integer_field = 124 # Raises AttributeError and suggests valid field.
+except AttributeError as e:
+ print(e)
+with cfg.unlocked():
+ cfg.integer_field = 1555 # Works fine too.
+
+# Get a copy of the config dict.
+new_cfg = copy.deepcopy(cfg)
+new_cfg.integer_field = -123 # Works fine.
+
+print(cfg)
+```
+
+### Dictionary attributes and initialization
+
+```python
+import ml_collections
+
+referenced_dict = {'inner_float': 3.14}
+d = {
+ 'referenced_dict_1': referenced_dict,
+ 'referenced_dict_2': referenced_dict,
+ 'list_containing_dict': [{'key': 'value'}],
+}
+
+# We can initialize on a dictionary
+cfg = ml_collections.ConfigDict(d)
+
+# Reference structure is preserved
+print(id(cfg.referenced_dict_1) == id(cfg.referenced_dict_2)) # True
+
+# And the dict attributes have been converted to ConfigDict
+print(type(cfg.referenced_dict_1)) # ConfigDict
+
+# However, the initialization does not look inside of lists, so dicts inside
+# lists are not converted to ConfigDict
+print(type(cfg.list_containing_dict[0])) # dict
+```
+
+### More Examples
+
+For more examples, take a look at
+[`ml_collections/config_dict/examples/`](https://github.com/google/ml_collections/tree/master/ml_collections/config_dict/examples)
+
+For examples and gotchas specifically about initializing a ConfigDict, see
+[`ml_collections/config_dict/examples/config_dict_initialization.py`](https://github.com/google/ml_collections/blob/master/ml_collections/config_dict/examples/config_dict_initialization.py).
+
+## Config Flags
+
+This library adds flag definitions to `absl.flags` to handle config files. It
+does not wrap `absl.flags` so if using any standard flag definitions alongside
+config file flags, users must also import `absl.flags`.
+
+Currently, this module adds two new flag types, namely `DEFINE_config_file`
+which accepts a path to a Python file that generates a configuration, and
+`DEFINE_config_dict` which accepts a configuration directly. Configurations are
+dict-like structures (see [ConfigDict](#configdict)) whose nested elements
+can be overridden using special command-line flags. See the examples below
+for more details.
+
+### Usage
+
+Use `ml_collections.config_flags` alongside `absl.flags`. For
+example:
+
+`script.py`:
+
+```python
+from absl import app
+from absl import flags
+
+from ml_collections.config_flags import config_flags
+
+FLAGS = flags.FLAGS
+config_flags.DEFINE_config_file('my_config')
+
+def main(_):
+ print(FLAGS.my_config)
+
+if __name__ == '__main__':
+ app.run(main)
+```
+
+`config.py`:
+
+```python
+# Note that this is a valid Python script.
+# get_config() can return an arbitrary dict-like object. However, it is advised
+# to use ml_collections.ConfigDict.
+# See ml_collections/config_dict/examples/config_dict_basic.py
+
+import ml_collections
+
+def get_config():
+ config = ml_collections.ConfigDict()
+ config.field1 = 1
+ config.field2 = 'tom'
+ config.nested = ml_collections.ConfigDict()
+ config.nested.field = 2.23
+ config.tuple = (1, 2, 3)
+ return config
+```
+
+Now, after running:
+
+```bash
+python script.py --my_config=config.py \
+ --my_config.field1=8 \
+ --my_config.nested.field=2.1 \
+ --my_config.tuple='(1, 2, (1, 2))'
+```
+
+we get:
+
+```
+field1: 8
+field2: tom
+nested:
+ field: 2.1
+tuple: !!python/tuple
+- 1
+- 2
+- !!python/tuple
+ - 1
+ - 2
+```
+
+Usage of `DEFINE_config_dict` is similar to `DEFINE_config_file`, the main
+difference is the configuration is defined in `script.py` instead of in a
+separate file.
+
+`script.py`:
+
+```python
+from absl import app
+from absl import flags
+
+import ml_collections
+from ml_collections.config_flags import config_flags
+
+config = ml_collections.ConfigDict()
+config.field1 = 1
+config.field2 = 'tom'
+config.nested = ml_collections.ConfigDict()
+config.nested.field = 2.23
+config.tuple = (1, 2, 3)
+
+FLAGS = flags.FLAGS
+config_flags.DEFINE_config_dict('my_config', config)
+
+def main(_):
+ print(FLAGS.my_config)
+
+if __name__ == '__main__':
+ app.run()
+```
+
+`config_file` flags are compatible with the command-line flag syntax. All the
+following options are supported for non-boolean values in configurations:
+
+* `-(-)config.field=value`
+* `-(-)config.field value`
+
+Options for boolean values are slightly different:
+
+* `-(-)config.boolean_field`: set boolean value to True.
+* `-(-)noconfig.boolean_field`: set boolean value to False.
+* `-(-)config.boolean_field=value`: `value` is `true`, `false`, `True` or
+ `False`.
+
+Note that `-(-)config.boolean_field value` is not supported.
+
+### Parameterising the get_config() function
+
+It's sometimes useful to be able to pass parameters into `get_config`, and
+change what is returned based on this configuration. One example is if you are
+grid searching over parameters which have a different hierarchical structure -
+the flag needs to be present in the resulting ConfigDict. It would be possible
+to include the union of all possible leaf values in your ConfigDict,
+but this produces a confusing config result as you have to remember which
+parameters will actually have an effect and which won't.
+
+A better system is to pass some configuration, indicating which structure of
+ConfigDict should be returned. An example is the following config file:
+
+```python
+import ml_collections
+
+def get_config(config_string):
+ possible_structures = {
+ 'linear': ml_collections.ConfigDict({
+ 'model_constructor': 'snt.Linear',
+ 'model_config': ml_collections.ConfigDict({
+ 'output_size': 42,
+ }),
+ 'lstm': ml_collections.ConfigDict({
+ 'model_constructor': 'snt.LSTM',
+ 'model_config': ml_collections.ConfigDict({
+ 'hidden_size': 108,
+ })
+ })
+ }
+
+ return possible_structures[config_string]
+```
+
+The value of `config_string` will be anything that is to the right of the first
+colon in the config file path, if one exists. If no colon exists, no value is
+passed to `get_config` (producing a TypeError if `get_config` expects a value.)
+
+The above example can be run like:
+
+```bash
+python script.py -- --config=path_to_config.py:linear \
+ --config.model_config.output_size=256
+```
+
+or like:
+
+```bash
+python script.py -- --config=path_to_config.py:lstm \
+ --config.model_config.hidden_size=512
+```
+
+### Additional features
+
+* Loads any valid python script which defines `get_config()` function
+ returning any python object.
+* Automatic locking of the loaded object, if the loaded object defines a
+ callable `.lock()` method.
+* Supports command-line overriding of arbitrarily nested values in dict-like
+ objects (with key/attribute based getters/setters) of the following types:
+ * `types.IntType` (integer)
+ * `types.FloatType` (float)
+ * `types.BooleanType` (bool)
+ * `types.StringType` (string)
+ * `types.TupleType` (tuple)
+* Overriding is type safe.
+* Overriding of `TupleType` can be done by passing in the `tuple` as a string
+ (see the example in the [Usage](#usage) section).
+* The overriding `tuple` object can be of a different size and have different
+ types than the original. Nested tuples are also supported.
+
+## Authors
+* Sergio Gómez Colmenarejo - sergomez@google.com
+* Wojciech Marian Czarnecki - lejlot@google.com
+* Nicholas Watters
+* Mohit Reddy - mohitreddy@google.com
+
+
diff --git a/ml_collections.egg-info/SOURCES.txt b/ml_collections.egg-info/SOURCES.txt
index 1bbcf2e..07ab224 100644
--- a/ml_collections.egg-info/SOURCES.txt
+++ b/ml_collections.egg-info/SOURCES.txt
@@ -1,5 +1,11 @@
+AUTHORS
+LICENSE
+MANIFEST.in
README.md
+requirements-test.txt
+requirements.txt
setup.py
+docs/conf.py
ml_collections/__init__.py
ml_collections.egg-info/PKG-INFO
ml_collections.egg-info/SOURCES.txt
@@ -25,12 +31,14 @@ ml_collections/config_flags/__init__.py
ml_collections/config_flags/config_flags.py
ml_collections/config_flags/tuple_parser.py
ml_collections/config_flags/examples/config.py
+ml_collections/config_flags/examples/define_config_dataclass_basic.py
ml_collections/config_flags/examples/define_config_dict_basic.py
ml_collections/config_flags/examples/define_config_file_basic.py
ml_collections/config_flags/examples/examples_test.py
ml_collections/config_flags/examples/parameterised_config.py
ml_collections/config_flags/tests/config_overriding_test.py
ml_collections/config_flags/tests/configdict_config.py
+ml_collections/config_flags/tests/dataclass_overriding_test.py
ml_collections/config_flags/tests/fieldreference_config.py
ml_collections/config_flags/tests/ioerror_config.py
ml_collections/config_flags/tests/mini_config.py
diff --git a/ml_collections.egg-info/requires.txt b/ml_collections.egg-info/requires.txt
index 6f289bb..32b75f6 100644
--- a/ml_collections.egg-info/requires.txt
+++ b/ml_collections.egg-info/requires.txt
@@ -1,7 +1,10 @@
-absl-py
PyYAML
-six
+absl-py
contextlib2
+six
[:python_version < "3.5"]
typing
+
+[:python_version < "3.7"]
+dataclasses
diff --git a/ml_collections.egg-info/top_level.txt b/ml_collections.egg-info/top_level.txt
index 50b72e9..759af71 100644
--- a/ml_collections.egg-info/top_level.txt
+++ b/ml_collections.egg-info/top_level.txt
@@ -1 +1,4 @@
+build
+dist
+docs
ml_collections
diff --git a/ml_collections/__init__.py b/ml_collections/__init__.py
index f2096bb..1264616 100644
--- a/ml_collections/__init__.py
+++ b/ml_collections/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/__init__.py b/ml_collections/config_dict/__init__.py
index 5f448ed..a0dbc1e 100644
--- a/ml_collections/config_dict/__init__.py
+++ b/ml_collections/config_dict/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/config_dict.py b/ml_collections/config_dict/config_dict.py
index 2027e60..554b78e 100644
--- a/ml_collections/config_dict/config_dict.py
+++ b/ml_collections/config_dict/config_dict.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -23,12 +23,14 @@ configurations of experiments and models.
import abc
import collections
+from collections import abc as collections_abc
import contextlib
import difflib
import functools
import inspect
import json
import operator
+from typing import Any, Mapping, Optional
from absl import logging
@@ -62,6 +64,11 @@ class JSONDecodeError(Exception):
_NoneType = type(None)
+def _is_callable_type(field_type):
+ """Tries to ensure: `_is_callable_type(type(obj)) == callable(obj)`."""
+ return any('__call__' in c.__dict__ for c in field_type.__mro__)
+
+
def _is_type_safety_violation(value, field_type):
"""Helper function for type safety exceptions.
@@ -78,8 +85,11 @@ def _is_type_safety_violation(value, field_type):
# Allow None to override and be overridden by any type.
if value is None or field_type == _NoneType:
return False
+ elif isinstance(value, field_type):
+ return False
else:
- return not isinstance(value, field_type)
+ # A callable can overridde a callable.
+ return not (callable(value) and _is_callable_type(field_type))
def _safe_cast(value, field_type, type_safe=False):
@@ -157,30 +167,29 @@ class _Op(collections.namedtuple('_Op', ['fn', 'args'])):
@functools.total_ordering
-class FieldReference(object):
+class FieldReference:
"""Reference to a configuration element.
- Typed configuration element that can take a None default value. Example:
+ Typed configuration element that can take a None default value. Example::
- from ml_collections import config_dict
+ from ml_collections import config_dict
- cfg_field = config_dict.FieldReference(0)
- cfg = config_dict.ConfigDict({
- 'optional': configdict.FieldReference(None, field_type=str)
- 'field': cfg_field,
- 'nested': {'field': cfg_field}
- })
+ cfg_field = config_dict.FieldReference(0)
+ cfg = config_dict.ConfigDict({
+ 'optional': configdict.FieldReference(None, field_type=str)
+ 'field': cfg_field,
+ 'nested': {'field': cfg_field}
+ })
- with self.assertRaises(TypeError):
- cfg.optional = 10 # Raises an error because it's defined as an
- # intfield.
+ with self.assertRaises(TypeError):
+ cfg.optional = 10 # Raises an error because it's defined as an
+ # intfield.
- cfg.field = 1 # Changes the value of both cfg.field and cfg.nested.field.
- print(cfg)
+ cfg.field = 1 # Changes the value of both cfg.field and cfg.nested.field.
+ print(cfg)
- This class also supports lazy computation. Example:
+ This class also supports lazy computation. Example::
- ```python
ref = config_dict.FieldReference(0)
# Using ref in a standard operation returns another FieldReference. The new
@@ -193,7 +202,6 @@ class FieldReference(object):
ref.set(-2) # change ref's value again
print(ref_plus_ten.get()) # Prints 8 because ref's value is -2
- ```
"""
def __init__(self, default, field_type=None, op=None, required=False):
@@ -387,6 +395,9 @@ class FieldReference(object):
def identity(self):
return self._apply_op(lambda x: x)
+ def attr(self, attr_name):
+ return self._apply_op(operator.attrgetter(attr_name))
+
def __add__(self, other):
return self._apply_op(operator.add, other)
@@ -409,10 +420,10 @@ class FieldReference(object):
return self._apply_op(rmul)
def __div__(self, other):
- return self._apply_op(operator.div, other)
+ return self._apply_op(operator.truediv, other)
def __rdiv__(self, other):
- rdiv = functools.partial(operator.div, other)
+ rdiv = functools.partial(operator.truediv, other)
return self._apply_op(rdiv)
def __truediv__(self, other):
@@ -548,7 +559,7 @@ def _configdict_fill_seed(seed, initial_dictionary, visit_map=None):
seed.__setattr__(key, value)
-class ConfigDict(object):
+class ConfigDict:
# pylint: disable=line-too-long
"""Base class for configuration objects used in DeepMind.
@@ -558,7 +569,7 @@ class ConfigDict(object):
- it has dot-based access as well as dict-style key access,
- it is type safe (once a value is set one cannot change its type).
- Typical usage eaxmple:
+ Typical usage example::
from ml_collections import config_dict
@@ -571,23 +582,23 @@ class ConfigDict(object):
print(cfg)
- Config dictionaries can also be used to pass named arguments to functions:
+ Config dictionaries can also be used to pass named arguments to functions::
- from ml_collections import config_dict
+ from ml_collections import config_dict
- def print_point(x, y):
- print "({},{})".format(x, y)
+ def print_point(x, y):
+ print "({},{})".format(x, y)
- point = config_dict.ConfigDict()
- point.x = 1
- point.y = 2
- print_point(**point)
+ point = config_dict.ConfigDict()
+ point.x = 1
+ point.y = 2
+ print_point(**point)
Note that, depending on your use case, it may be easier to use the `create`
- function in this package to construct a `ConfigDict`:
+ function in this package to construct a `ConfigDict`::
- from ml_collections.config_dict import config_dict
- point = config_dict.create(x=1, y=2)
+ from ml_collections.config_dict import config_dict
+ point = config_dict.create(x=1, y=2)
Differently from standard `dicts`, `ConfigDicts` also have the nice property
that iterating over them is deterministic, in a fashion similar to
@@ -595,20 +606,28 @@ class ConfigDict(object):
"""
# pylint: enable=line-too-long
+ # Loosen the static type checking requirements.
+ _HAS_DYNAMIC_ATTRIBUTES = True
+
def __init__(
- self, initial_dictionary=None, type_safe=True, convert_dict=True):
+ self,
+ initial_dictionary: Optional[Mapping[str, Any]] = None,
+ type_safe: bool = True,
+ convert_dict: bool = True,
+ ):
"""Creates an instance of ConfigDict.
Warning: In most cases, this faithfully reproduces the reference structure
of initial_dictionary, even if initial_dictionary is self-referencing.
However, unexpected behavior occurs if self-references are contained within
- list, tuple, or custom types. For example:
- d = {}
- d['a'] = d
- d['b'] = [d]
- cd = ConfigDict(d)
- cd.a # refers to cd, type ConfigDict. Expected behavior.
- cd.b # refers to d, type dict. Unexpected behavior.
+ list, tuple, or custom types. For example::
+
+ d = {}
+ d['a'] = d
+ d['b'] = [d]
+ cd = ConfigDict(d)
+ cd.a # refers to cd, type ConfigDict. Expected behavior.
+ cd.b # refers to d, type dict. Unexpected behavior.
Warning: FieldReference values may be changed. If initial_dictionary
contains a FieldReference with a value of type dict or FrozenConfigDict,
@@ -616,16 +635,19 @@ class ConfigDict(object):
Args:
initial_dictionary: May be one of the following:
- 1) dict. In this case, all values of initial_dictionary that are
- dictionaries are also be converted to ConfigDict. However,
- dictionaries within values of non-dict type are untouched.
- 2) ConfigDict. In this case, all attributes are uncopied, and only the
- top-level object (self) is re-addressed. This is the same behavior
- as Python dict, list, and tuple.
- 3) FrozenConfigDict. In this case, initial_dictionary is converted to
- a ConfigDict version of the initial dictionary for the
- FrozenConfigDict (reversing any mutability changes FrozenConfigDict
- made).
+
+ 1) dict. In this case, all values of initial_dictionary that are
+ dictionaries are also be converted to ConfigDict. However,
+ dictionaries within values of non-dict type are untouched.
+
+ 2) ConfigDict. In this case, all attributes are uncopied, and only the
+ top-level object (self) is re-addressed. This is the same behavior
+ as Python dict, list, and tuple.
+
+ 3) FrozenConfigDict. In this case, initial_dictionary is converted to
+ a ConfigDict version of the initial dictionary for the
+ FrozenConfigDict (reversing any mutability changes FrozenConfigDict
+ made).
type_safe: If set to True, once an attribute value is assigned, its type
cannot be overridden without .ignore_type() context manager
(default: True).
@@ -651,7 +673,7 @@ class ConfigDict(object):
initial_dictionary.is_type_safe)
@property
- def is_type_safe(self):
+ def is_type_safe(self) -> bool:
"""Returns True if config dict is type safe."""
return self._type_safe
@@ -660,7 +682,7 @@ class ConfigDict(object):
"""Returns True if it is converting dicts to ConfigDict automatically."""
return self._convert_dict
- def lock(self):
+ def lock(self) -> 'ConfigDict':
"""Locks object, preventing user from adding new fields.
Returns:
@@ -680,11 +702,11 @@ class ConfigDict(object):
return self
@property
- def is_locked(self):
+ def is_locked(self) -> bool:
"""Returns True if object is locked."""
return self._locked
- def unlock(self):
+ def unlock(self) -> 'ConfigDict':
"""Grants user the ability to add new fields to ConfigDict.
In most cases, the unlocked() context manager should be preferred to the
@@ -701,7 +723,7 @@ class ConfigDict(object):
element.unlock()
return self
- def get(self, key, default=None):
+ def get(self, key: str, default=None):
"""Returns value if key is present, or a user defined value otherwise."""
try:
return self[key]
@@ -725,19 +747,17 @@ class ConfigDict(object):
def get_oneway_ref(self, key):
"""Returns a one-way FieldReference.
- Example:
+ Example::
- ```
- cfg = collections.ConfigDict(dict(a=1))
- cfg.b = cfg.get_oneway_ref('a')
+ cfg = ml_collections.ConfigDict(dict(a=1))
+ cfg.b = cfg.get_oneway_ref('a')
- cfg.a = 2
- print(cfg.b) # 2
+ cfg.a = 2
+ print(cfg.b) # 2
- cfg.b = 3
- print(cfg.a) # 2 (would have been 3 if using get_ref())
- print(cfg.b) # 3
- ```
+ cfg.b = 3
+ print(cfg.a) # 2 (would have been 3 if using get_ref())
+ print(cfg.b) # 3
Args:
key: Key for field we want to reference.
@@ -855,7 +875,7 @@ class ConfigDict(object):
request)
return message
- def __delitem__(self, key):
+ def __delitem__(self, key: str):
if self.is_locked:
raise KeyError('This ConfigDict is locked, you have to unlock it before '
'trying to delete a field.')
@@ -872,7 +892,7 @@ class ConfigDict(object):
except KeyError as e:
raise KeyError(self._generate_did_you_mean_message(key, str(e)))
- def __getitem__(self, key):
+ def __getitem__(self, key: str):
if '.' in key:
# As per the check in __setitem__ above, keys cannot contain dots.
# Hence, we can use dots to do recursive calls.
@@ -888,15 +908,25 @@ class ConfigDict(object):
except KeyError as e:
raise KeyError(self._generate_did_you_mean_message(key, str(e)))
- def __contains__(self, key):
+ def __contains__(self, key: str):
return key in self._fields
- def __repr__(self):
- return yaml.dump(self.to_dict(preserve_field_references=True),
- default_flow_style=False)
-
- def __str__(self):
- return yaml.dump(self.to_dict())
+ def __repr__(self) -> str:
+ # We want __repr__ to always run without throwing an exception,
+ # even if the config dict is not YAML serialisable.
+ try:
+ return yaml.dump(self.to_dict(preserve_field_references=True),
+ default_flow_style=False)
+ except Exception: # pylint: disable=broad-except
+ return repr(self.to_dict())
+
+ def __str__(self) -> str:
+ # We want __str__ to always run without throwing an exception,
+ # even if the config dict is not YAML serialisable.
+ try:
+ return yaml.dump(self.to_dict())
+ except Exception: # pylint: disable=broad-except
+ return str(self.to_dict())
def keys(self):
"""Returns the sorted list of all the keys defined in a config."""
@@ -963,11 +993,11 @@ class ConfigDict(object):
"""Type-invariant equals.
This is like `__eq__`, except it does not distinguish `FrozenConfigDict`
- from `ConfigDict`. For example:
+ from `ConfigDict`. For example::
- cd = ConfigDict()
- fcd = FrozenConfigDict()
- fcd.eq_as_configdict(cd) # Returns True
+ cd = ConfigDict()
+ fcd = FrozenConfigDict()
+ fcd.eq_as_configdict(cd) # Returns True
Args:
other: Object to compare self to.
@@ -986,6 +1016,23 @@ class ConfigDict(object):
def to_yaml(self, **kwargs):
"""Returns a YAML representation of the object.
+ ConfigDict serializes types of fields as well as the values of fields
+ themselves. Deserializing the YAML representation hence requires using
+ YAML's UnsafeLoader:
+
+ ```
+ yaml.load(cfg.to_yaml(), Loader=yaml.UnsafeLoader)
+ ```
+
+ or equivalently:
+
+ ```
+ yaml.unsafe_load(cfg.to_yaml())
+ ```
+
+ Please see the PyYAML documentation and https://msg.pyyaml.org/load for more
+ details on the consequences of this.
+
Args:
**kwargs: Keyword arguments for yaml.dump.
@@ -1143,6 +1190,8 @@ class ConfigDict(object):
"""
visit_map = visit_map or {}
config_dict_copy = self.__class__()
+ super(ConfigDict, config_dict_copy).__setattr__('_convert_dict',
+ self.convert_dict)
visit_map[id(self)] = config_dict_copy
for key, value in six.iteritems(self._fields):
@@ -1204,9 +1253,9 @@ class ConfigDict(object):
if isinstance(field, ConfigDict):
managers.append(field.ignore_type)
# Recursively add elements in nested collections.
- elif isinstance(field, collections.Mapping):
+ elif isinstance(field, collections_abc.Mapping):
fields.extend(six.iteritems(field))
- elif isinstance(field, (collections.Sequence, collections.Set)):
+ elif isinstance(field, (collections_abc.Sequence, collections_abc.Set)):
fields.extend(field)
super(ConfigDict, self).__setattr__('_type_safe', False)
@@ -1260,7 +1309,7 @@ class ConfigDict(object):
if isinstance(other, ConfigDict):
iteritems_kwargs['preserve_field_references'] = True
- for key, value in six.iteritems(other, **iteritems_kwargs):
+ for key, value in six.iteritems(other, **iteritems_kwargs): # pytype: disable=wrong-keyword-args
if key not in self:
self[key] = value
elif isinstance(self._fields[key], ConfigDict):
@@ -1294,55 +1343,55 @@ class ConfigDict(object):
def update_from_flattened_dict(self, flattened_dict, strip_prefix=''):
"""In-place updates values taken from a flattened dict.
- This allows a potentially nested source `ConfigDict` of the following form:
+ This allows a potentially nested source `ConfigDict` of the following form::
- cfg = ConfigDict({
- 'a': 1,
- 'b': {
- 'c': {
- 'd': 2
- }
- }
- })
+ cfg = ConfigDict({
+ 'a': 1,
+ 'b': {
+ 'c': {
+ 'd': 2
+ }
+ }
+ })
to be updated from a dict containing paths navigating to child items, of the
- following form:
+ following form::
- updates = {
- 'a': 2,
- 'b.c.d': 3
- }
+ updates = {
+ 'a': 2,
+ 'b.c.d': 3
+ }
This filters `paths_dict` to only contain paths starting with
`strip_prefix` and strips the prefix when applying the update.
- For example, consider we have the following values returned as flags:
-
- flags = {
- 'flag1': x,
- 'flag2': y,
- 'config': 'some_file.py',
- 'config.a.b': 1,
- 'config.a.c': 2
- }
-
- config = ConfigDict({
- 'a': {
- 'b': 0,
- 'c': 0
- }
- })
+ For example, consider we have the following values returned as flags::
- config.update_from_flattened_dict(flags, 'config.')
+ flags = {
+ 'flag1': x,
+ 'flag2': y,
+ 'config': 'some_file.py',
+ 'config.a.b': 1,
+ 'config.a.c': 2
+ }
- Then we will now have:
+ config = ConfigDict({
+ 'a': {
+ 'b': 0,
+ 'c': 0
+ }
+ })
- config = ConfigDict({
- 'a': {
- 'b': 1,
- 'c': 2
- }
- })
+ config.update_from_flattened_dict(flags, 'config.')
+
+ Then we will now have::
+
+ config = ConfigDict({
+ 'a': {
+ 'b': 1,
+ 'c': 2
+ }
+ })
Args:
flattened_dict: A mapping (key path) -> value.
@@ -1618,13 +1667,17 @@ class FrozenConfigDict(ConfigDict):
Args:
initial_dictionary: May be one of the following:
- 1) dict. In this case all values of initial_dictionary that are
- dictionaries are also converted to FrozenConfigDict. If there are
- dictionaries contained in lists or tuples, an error is raised.
- 2) ConfigDict. In this case all ConfigDict attributes are also
- converted to FrozenConfigDict.
- 3) FrozenConfigDict. In this case all attributes are uncopied, and
- only the top-level object (self) is re-addressed.
+
+ 1) dict. In this case all values of initial_dictionary that are
+ dictionaries are also converted to FrozenConfigDict. If there are
+ dictionaries contained in lists or tuples, an error is raised.
+
+ 2) ConfigDict. In this case all ConfigDict attributes are also
+ converted to FrozenConfigDict.
+
+ 3) FrozenConfigDict. In this case all attributes are uncopied, and
+ only the top-level object (self) is re-addressed.
+
type_safe: See ConfigDict documentation. Note that this only matters
if the FrozenConfigDict is converted to ConfigDict at some point.
"""
@@ -1836,23 +1889,23 @@ def create(**kwargs):
"""Creates a `ConfigDict` with the given named arguments as key-value pairs.
This allows for simple dictionaries whose elements can be accessed directly
- using field access:
+ using field access::
- from ml_collections.config_dict import config_dict
- point = config_dict.create(x=1, y=2)
- print(point.x, point.y)
+ from ml_collections.config_dict import config_dict
+ point = config_dict.create(x=1, y=2)
+ print(point.x, point.y)
- This is particularly useful for compactly writing nested configurations:
+ This is particularly useful for compactly writing nested configurations::
- config = config_dict.create(
- data=config_dict.create(
- game='freeway',
- frame_size=100),
- model=config_dict.create(num_hidden=1000))
+ config = config_dict.create(
+ data=config_dict.create(
+ game='freeway',
+ frame_size=100),
+ model=config_dict.create(num_hidden=1000))
The reason for the existence of this function is that it simplifies the
code required for the majority of the use cases of `ConfigDict`, compared
- to using either `ConfigDict` or `namedtuple`s. Examples of such use cases
+ to using either `ConfigDict` or `namedtuple's`. Examples of such use cases
include training script configuration, and returning multiple named values.
Args:
@@ -1868,11 +1921,11 @@ def create(**kwargs):
def placeholder(field_type, required=False):
"""Defines an entry in a ConfigDict that has no value yet.
- Example:
+ Example::
- config = configdict.create(
- batch_size = configdict.placeholder(int),
- frame_shape = configdict.placeholder(tf.TensorShape))
+ config = configdict.create(
+ batch_size = configdict.placeholder(int),
+ frame_shape = configdict.placeholder(tf.TensorShape))
Args:
field_type: type of value.
@@ -1888,18 +1941,18 @@ def placeholder(field_type, required=False):
def required_placeholder(field_type):
"""Defines an entry in a ConfigDict with unknown but required value.
- Example:
+ Example::
- config = configdict.create(
- batch_size = configdict.required_placeholder(int))
+ config = configdict.create(
+ batch_size = configdict.required_placeholder(int))
- try:
- print(config.batch_size)
- except RequiredValueError:
- pass
+ try:
+ print(config.batch_size)
+ except RequiredValueError:
+ pass
- config.batch_size = 10
- print(config.batch_size) # 10
+ config.batch_size = 10
+ print(config.batch_size) # 10
Args:
field_type: type of value.
diff --git a/ml_collections/config_dict/examples/config.py b/ml_collections/config_dict/examples/config.py
index b57a68a..a1034d7 100644
--- a/ml_collections/config_dict/examples/config.py
+++ b/ml_collections/config_dict/examples/config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/config_dict_advanced.py b/ml_collections/config_dict/examples/config_dict_advanced.py
index 4e40cff..d8f0f04 100644
--- a/ml_collections/config_dict/examples/config_dict_advanced.py
+++ b/ml_collections/config_dict/examples/config_dict_advanced.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -21,14 +21,10 @@ safety, iteration over fields, checking for a particular field, unpacking with
"""
from absl import app
-from absl import flags
from ml_collections.config_flags import config_flags
import yaml
-
-FLAGS = flags.FLAGS
-
-config_flags.DEFINE_config_file(
+_CONFIG = config_flags.DEFINE_config_file(
'my_config',
default='ml_collections/config_dict/examples/config.py')
@@ -49,13 +45,13 @@ def print_section(name):
def main(_):
# Config is already loaded in FLAGS.my_config due to the logic hidden
# in app.run().
- config = FLAGS.my_config
+ config = _CONFIG.value
print_section('Printing config.')
- print(FLAGS.my_config)
+ print(config)
# Config is of our type ConfigDict.
- print('Type of the config {}'.format(type(FLAGS.my_config)))
+ print('Type of the config {}'.format(type(config)))
# By default it is locked, thus you cannot add new fields.
# This prevents you from misspelling your attribute name.
@@ -122,7 +118,7 @@ def main(_):
# Note: __repr__ (not __str__) is the recommended representation, as it
# preserves FieldReferences and placeholders.
print_section('Loading dictionary from string representation.')
- dictionary = yaml.load(repr(config))
+ dictionary = yaml.load(repr(config), yaml.UnsafeLoader)
print('dict["object_reference"]["dict"]["dict"]["float"]={}'.format(
dictionary['object_reference']['dict']['dict']['float']))
diff --git a/ml_collections/config_dict/examples/config_dict_basic.py b/ml_collections/config_dict/examples/config_dict_basic.py
index 87cfa37..00750af 100644
--- a/ml_collections/config_dict/examples/config_dict_basic.py
+++ b/ml_collections/config_dict/examples/config_dict_basic.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/config_dict_initialization.py b/ml_collections/config_dict/examples/config_dict_initialization.py
index ee64dd5..501a4f1 100644
--- a/ml_collections/config_dict/examples/config_dict_initialization.py
+++ b/ml_collections/config_dict/examples/config_dict_initialization.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/config_dict_lock.py b/ml_collections/config_dict/examples/config_dict_lock.py
index 47c8a88..a22be91 100644
--- a/ml_collections/config_dict/examples/config_dict_lock.py
+++ b/ml_collections/config_dict/examples/config_dict_lock.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/config_dict_placeholder.py b/ml_collections/config_dict/examples/config_dict_placeholder.py
index a58904d..ce71d41 100644
--- a/ml_collections/config_dict/examples/config_dict_placeholder.py
+++ b/ml_collections/config_dict/examples/config_dict_placeholder.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/examples_test.py b/ml_collections/config_dict/examples/examples_test.py
index 6133091..3c91cdd 100644
--- a/ml_collections/config_dict/examples/examples_test.py
+++ b/ml_collections/config_dict/examples/examples_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/field_reference.py b/ml_collections/config_dict/examples/field_reference.py
index 5899c52..91e4394 100644
--- a/ml_collections/config_dict/examples/field_reference.py
+++ b/ml_collections/config_dict/examples/field_reference.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/examples/frozen_config_dict.py b/ml_collections/config_dict/examples/frozen_config_dict.py
index b500ca8..ea5ffdf 100644
--- a/ml_collections/config_dict/examples/frozen_config_dict.py
+++ b/ml_collections/config_dict/examples/frozen_config_dict.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_dict/tests/config_dict_test.py b/ml_collections/config_dict/tests/config_dict_test.py
index 4b1fedf..482835b 100644
--- a/ml_collections/config_dict/tests/config_dict_test.py
+++ b/ml_collections/config_dict/tests/config_dict_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -16,7 +16,8 @@
"""Tests for ml_collections.ConfigDict."""
import abc
-import collections as python_collections
+from collections import abc as collections_abc
+import functools
import json
import pickle
import sys
@@ -57,7 +58,7 @@ class _TestClass(six.with_metaclass(abc.ABCMeta, object)):
_test_object = _TestClass()
-class _TestClassNoStr(): # pylint: disable=old-style-class
+class _TestClassNoStr():
pass
@@ -290,6 +291,28 @@ class ConfigDictTest(parameterized.TestCase):
self.assertEqual(cfg.long_field, int_value)
self.assertIsInstance(cfg.long_field, expected)
+ def testOverrideCallable(self):
+ """Test that overriding a callable with a callable works."""
+
+ class SomeClass:
+
+ def __init__(self, x, power=1):
+ self.y = x**power
+
+ def factory(self, x):
+ return SomeClass(self.y + x)
+
+ fn1 = SomeClass
+ fn2 = lambda x: SomeClass(x, power=2)
+ fn3 = functools.partial(SomeClass, power=3)
+ fn4 = SomeClass(4.0).factory
+ cfg = ml_collections.ConfigDict()
+ for orig in [fn1, fn2, fn3, fn4]:
+ for new in [fn1, fn2, fn3, fn4]:
+ cfg.fn_field = orig
+ cfg.fn_field = new
+ self.assertEqual(cfg.fn_field, new)
+
def testOverrideFieldReference(self):
"""Test overriding with FieldReference objects."""
cfg = ml_collections.ConfigDict()
@@ -589,7 +612,7 @@ class ConfigDictTest(parameterized.TestCase):
hash(cfg)
# Ensure Python realizes ConfigDict is not hashable.
- self.assertNotIsInstance(cfg, python_collections.Hashable)
+ self.assertNotIsInstance(cfg, collections_abc.Hashable)
def testDidYouMeanFeature(self):
"""Tests 'did you mean' suggestions."""
@@ -675,7 +698,7 @@ class ConfigDictTest(parameterized.TestCase):
def testDeYaml(self):
"""Tests YAML deserialization."""
cfg = _get_test_config_dict()
- deyamled = yaml.load(cfg.to_yaml())
+ deyamled = yaml.load(cfg.to_yaml(), yaml.UnsafeLoader)
self.assertEqual(cfg, deyamled)
def testJSONConversion(self):
@@ -723,7 +746,7 @@ class ConfigDictTest(parameterized.TestCase):
cfg_dict.r1 = field
cfg_dict.r2 = field
- cfg_load = yaml.load(repr(cfg_dict))
+ cfg_load = yaml.load(repr(cfg_dict), yaml.UnsafeLoader)
# Test FieldReferences are preserved
cfg_load['r1'].set(2)
@@ -1317,6 +1340,21 @@ class ConfigDictUpdateTest(absltest.TestCase):
self.assertIsInstance(cfg.a, dict)
self.assertIsInstance(cfg.a['b'], dict)
+ def testConvertDictInCopyAndResolveReferences(self):
+ """Test conversion, or not, of dict in copy and resolve references."""
+ cfg = ml_collections.ConfigDict()
+ cfg.a = dict(b=dict(c=0))
+ copied_cfg = cfg.copy_and_resolve_references()
+ self.assertIsInstance(copied_cfg.a, ml_collections.ConfigDict)
+ self.assertIsInstance(copied_cfg.a.b, ml_collections.ConfigDict)
+
+ cfg = ml_collections.ConfigDict(convert_dict=False)
+ cfg.a = dict(b=dict(c=0))
+ copied_cfg = cfg.copy_and_resolve_references()
+ self.assertNotIsInstance(copied_cfg.a, ml_collections.ConfigDict)
+ self.assertIsInstance(copied_cfg.a, dict)
+ self.assertIsInstance(copied_cfg.a['b'], dict)
+
def testConvertDictTypeCompat(self):
"""Test that automatic conversion to ConfigDict doesn't trigger type errors."""
cfg = ml_collections.ConfigDict()
@@ -1331,7 +1369,7 @@ class ConfigDictUpdateTest(absltest.TestCase):
This checks backward compatibility of deserialisation.
"""
cfg = ml_collections.ConfigDict(dict(a=1))
- self.assertTrue(yaml.load(cfg.to_yaml())._convert_dict)
+ self.assertTrue(yaml.load(cfg.to_yaml(), yaml.UnsafeLoader)._convert_dict)
def testRecursiveRename(self):
"""Test recursive_rename.
diff --git a/ml_collections/config_dict/tests/field_reference_test.py b/ml_collections/config_dict/tests/field_reference_test.py
index cbe37e8..9f8d449 100644
--- a/ml_collections/config_dict/tests/field_reference_test.py
+++ b/ml_collections/config_dict/tests/field_reference_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -532,6 +532,41 @@ class FieldReferenceTest(parameterized.TestCase):
self._test_unary_operator(initial_value, operator.neg, true_value,
new_initial_value, new_true_value)
+ @parameterized.parameters(
+ {
+ 'initial_value': config_dict.create(attribute=2),
+ 'true_value': 2,
+ 'new_initial_value': config_dict.create(attribute=3),
+ 'new_true_value': 3,
+ },
+ {
+ 'initial_value': config_dict.create(attribute={'a': 1}),
+ 'true_value': config_dict.create(a=1),
+ 'new_initial_value': config_dict.create(attribute={'b': 1}),
+ 'new_true_value': config_dict.create(b=1),
+ },
+ {
+ 'initial_value':
+ ml_collections.FieldReference(config_dict.create(attribute=2)),
+ 'true_value':
+ ml_collections.FieldReference(2),
+ 'new_initial_value':
+ config_dict.create(attribute=3),
+ 'new_true_value':
+ 3,
+ },
+ {
+ 'initial_value': config_dict.placeholder(config_dict.ConfigDict),
+ 'true_value': None,
+ 'new_initial_value': config_dict.create(attribute=3),
+ 'new_true_value': 3,
+ },
+ )
+ def testAttr(self, initial_value, true_value, new_initial_value,
+ new_true_value):
+ self._test_unary_operator(initial_value, lambda x: x.attr('attribute'),
+ true_value, new_initial_value, new_true_value)
+
@parameterized.parameters(
{
'initial_value': 3,
diff --git a/ml_collections/config_dict/tests/frozen_config_dict_test.py b/ml_collections/config_dict/tests/frozen_config_dict_test.py
index 156d8b5..ceb45da 100644
--- a/ml_collections/config_dict/tests/frozen_config_dict_test.py
+++ b/ml_collections/config_dict/tests/frozen_config_dict_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -15,7 +15,7 @@
# Lint as: python3
"""Tests for ml_collections.FrozenConfigDict."""
-import collections as python_collections
+from collections import abc as collections_abc
import copy
import pickle
@@ -357,7 +357,7 @@ class FrozenConfigDictTest(absltest.TestCase):
hash(ml_collections.FrozenConfigDict(list_to_tuple)))
# Ensure Python realizes FrozenConfigDict is hashable
- self.assertIsInstance(_test_frozenconfigdict(), python_collections.Hashable)
+ self.assertIsInstance(_test_frozenconfigdict(), collections_abc.Hashable)
def testUnhashableType(self):
"""Ensures __hash__() fails if FrozenConfigDict has unhashable value."""
diff --git a/ml_collections/config_flags/__init__.py b/ml_collections/config_flags/__init__.py
index 619e74b..14d619f 100644
--- a/ml_collections/config_flags/__init__.py
+++ b/ml_collections/config_flags/__init__.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
# Lint as: python 3
"""Config flags module."""
+from .config_flags import DEFINE_config_dataclass
from .config_flags import DEFINE_config_dict
from .config_flags import DEFINE_config_file
+from .config_flags import get_config_filename
+from .config_flags import get_override_values
-__all__ = ("DEFINE_config_dict", "DEFINE_config_file")
+__all__ = (
+ "DEFINE_config_dataclass",
+ "DEFINE_config_dict",
+ "DEFINE_config_file",
+ "get_config_filename",
+ "get_override_values",
+)
diff --git a/ml_collections/config_flags/config_flags.py b/ml_collections/config_flags/config_flags.py
index c8a5195..6f35848 100644
--- a/ml_collections/config_flags/config_flags.py
+++ b/ml_collections/config_flags/config_flags.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -13,7 +13,7 @@
# limitations under the License.
# Lint as: python 3
-"""Complex configs commmand line parser."""
+"""Configuration commmand line parser."""
import errno
import imp
@@ -21,9 +21,11 @@ import os
import re
import sys
import traceback
+from typing import Any, Dict, Generic, List, MutableMapping, Optional, Tuple, Type, TypeVar
from absl import flags
from absl import logging
+import dataclasses
import ml_collections
from ml_collections.config_flags import tuple_parser
import six
@@ -75,12 +77,12 @@ class UnparsedFlagError(flags.Error):
def DEFINE_config_file( # pylint: disable=g-bad-name
- name,
- default=None,
- help_string='path to config file.',
- flag_values=FLAGS,
- lock_config=True,
- **kwargs):
+ name: str,
+ default: Optional[str] = None,
+ help_string: str = 'path to config file.',
+ flag_values: flags.FlagValues = FLAGS,
+ lock_config: bool = True,
+ **kwargs) -> flags.FlagHolder:
r"""Defines flag for `ConfigDict` files compatible with absl flags.
The flag's value should be a path to a valid python file which contains a
@@ -100,95 +102,78 @@ def DEFINE_config_file( # pylint: disable=g-bad-name
Typical usage example:
- `script.py`:
+ `script.py`::
- ```python
- ...
- from absl import flags
- from ml_collections.config_flags import config_flags
+ from absl import flags
+ from ml_collections.config_flags import config_flags
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_file('my_config')
- ...
+ FLAGS = flags.FLAGS
+ config_flags.DEFINE_config_file('my_config')
- print(FLAGS.my_config)
- ```
+ print(FLAGS.my_config)
- `config.py`:
+ `config.py`::
- ```python
- def get_config():
- return {
- 'field1': 1,
- 'field2': 'tom',
- 'nested': {
- 'field': 2.23,
- },
- }
- ```
+ def get_config():
+ return {
+ 'field1': 1,
+ 'field2': 'tom',
+ 'nested': {
+ 'field': 2.23,
+ },
+ }
- The following command:
+ The following command::
- ```
- python script.py -- --my_config=config.py
- --my_config.field1 8
- --my_config.nested.field=2.1
- ```
+ python script.py -- --my_config=config.py
+ --my_config.field1 8
+ --my_config.nested.field=2.1
- will print:
+ will print::
- ```
- {'field1': 8, 'field2': 'tom', 'nested': {'field': 2.1}}
- ```
+ {'field1': 8, 'field2': 'tom', 'nested': {'field': 2.1}}
It is possible to parameterise the get_config function, allowing it to
return a differently structured result for different occasions. This is
particularly useful when setting up hyperparameter sweeps across various
network architectures.
- `parameterised_config.py`:
-
- ```python
- def get_config(config_string):
- possible_configs = {
- 'mlp': {
- 'constructor': 'snt.nets.MLP',
- 'config': {
- 'output_sizes': (128, 128, 1),
- }
- },
- 'lstm': {
- 'constructor': 'snt.LSTM',
- 'config': {
- 'hidden_size': 128,
- 'forget_bias': 1.0,
- }
- }
- }
- return possible_configs[config_string]
- ```
+ `parameterised_config.py`::
+
+ def get_config(config_string):
+ possible_configs = {
+ 'mlp': {
+ 'constructor': 'snt.nets.MLP',
+ 'config': {
+ 'output_sizes': (128, 128, 1),
+ }
+ },
+ 'lstm': {
+ 'constructor': 'snt.LSTM',
+ 'config': {
+ 'hidden_size': 128,
+ 'forget_bias': 1.0,
+ }
+ }
+ }
+ return possible_configs[config_string]
If a colon is present in the command line override for the config file,
everything to the right of the colon is passed into the get_config function.
- The following command lines will both function correctly:
+ The following command lines will both function correctly::
- ```
- python script.py -- --my_config=parameterised_config.py:mlp
- --my_config.config.output_sizes="(256,256,1)"
- ```
+ python script.py -- --my_config=parameterised_config.py:mlp
+ --my_config.config.output_sizes="(256,256,1)"
- ```
- python script.py -- --my_config=parameterised_config.py:lstm
- --my_config.config.hidden_size=256
- ```
+
+ python script.py -- --my_config=parameterised_config.py:lstm
+ --my_config.config.hidden_size=256
The following will produce an error, as the hidden_size flag does not
- exist when the "mlp" config_string is provided.
+ exist when the "mlp" config_string is provided::
- ```
- python script.py -- --my_config=parameterised_config.py:mlp
- --my_config.config.hidden_size=256
- ```
+ python script.py -- --my_config=parameterised_config.py:mlp
+ --my_config.config.hidden_size=256
Args:
name: Flag name, optionally including extra config after a colon.
@@ -200,6 +185,9 @@ def DEFINE_config_file( # pylint: disable=g-bad-name
lock_config: If set to True, loaded config will be locked through calling
.lock() method on its instance (if it exists). (default: True)
**kwargs: Optional keyword arguments passed to Flag constructor.
+
+ Returns:
+ a handle to defined flag.
"""
parser = _ConfigFileParser(name=name, lock_config=lock_config)
flag = _ConfigFlag(
@@ -214,17 +202,17 @@ def DEFINE_config_file( # pylint: disable=g-bad-name
# Get the module name for the frame at depth 1 in the call stack.
module_name = sys._getframe(1).f_globals.get('__name__', None) # pylint: disable=protected-access
module_name = sys.argv[0] if module_name == '__main__' else module_name
- flags.DEFINE_flag(flag, flag_values, module_name=module_name)
+ return flags.DEFINE_flag(flag, flag_values, module_name=module_name)
def DEFINE_config_dict( # pylint: disable=g-bad-name
- name,
- config,
- help_string='ConfigDict instance.',
- flag_values=FLAGS,
- lock_config=True,
- **kwargs):
- """Defines flag for inline `ConfigDict`s compatible with absl flags.
+ name: str,
+ config: ml_collections.ConfigDict,
+ help_string: str = 'ConfigDict instance.',
+ flag_values: flags.FlagValues = FLAGS,
+ lock_config: bool = True,
+ **kwargs) -> flags.FlagHolder:
+ """Defines flag for inline `ConfigDict's` compatible with absl flags.
Similar to `DEFINE_config_file` except the flag's value should be a
`ConfigDict` instead of a path to a file containing a `ConfigDict`. After the
@@ -233,46 +221,39 @@ def DEFINE_config_dict( # pylint: disable=g-bad-name
Typical usage example:
- `script.py`:
+ `script.py`::
- ```python
- ...
- from absl import flags
+ from absl import flags
- import ml_collections
- from ml_collections.config_flags import config_flags
+ import ml_collections
+ from ml_collections.config_flags import config_flags
- config = ml_collections.ConfigDict({
- 'field1': 1,
- 'field2': 'tom',
- 'nested': {
- 'field': 2.23,
- }
- })
+ config = ml_collections.ConfigDict({
+ 'field1': 1,
+ 'field2': 'tom',
+ 'nested': {
+ 'field': 2.23,
+ }
+ })
- FLAGS = flags.FLAGS
- config_flags.DEFINE_config_dict('my_config', config)
- ...
+ FLAGS = flags.FLAGS
+ config_flags.DEFINE_config_dict('my_config', config)
+ ...
- print(FLAGS.my_config)
- ```
+ print(FLAGS.my_config)
- The following command:
+ The following command::
- ```
- python script.py -- --my_config.field1 8
- --my_config.nested.field=2.1
- ```
+ python script.py -- --my_config.field1 8
+ --my_config.nested.field=2.1
- will print:
+ will print::
- ```
- field1: 8
- field2: tom
- nested: {field: 2.1}
- ```
+ field1: 8
+ field2: tom
+ nested: {field: 2.1}
Args:
name: Flag name.
@@ -284,6 +265,9 @@ def DEFINE_config_dict( # pylint: disable=g-bad-name
lock_config: If set to True, loaded config will be locked through calling
.lock() method on its instance (if it exists). (default: True)
**kwargs: Optional keyword arguments passed to Flag constructor.
+
+ Returns:
+ a handle to defined flag.
"""
if not isinstance(config, ml_collections.ConfigDict):
raise TypeError('config should be a ConfigDict')
@@ -300,10 +284,107 @@ def DEFINE_config_dict( # pylint: disable=g-bad-name
# Get the module name for the frame at depth 1 in the call stack.
module_name = sys._getframe(1).f_globals.get('__name__', None) # pylint: disable=protected-access
module_name = sys.argv[0] if module_name == '__main__' else module_name
- flags.DEFINE_flag(flag, flag_values, module_name=module_name)
+ return flags.DEFINE_flag(flag, flag_values, module_name=module_name)
+
+
+# Note that we would add a bound to constrain this to be a dataclass, except
+# that dataclasses don't have a specific base class, and structural typing for
+# attributes is currently (2021Q1) not supported in pytype (b/150927776).
+_T = TypeVar('_T')
+
+
+class _TypedFlagHolder(flags.FlagHolder, Generic[_T]):
+ """A typed wrapper for a FlagHolder."""
+
+ def __init__(self, flag: flags.FlagHolder, ctor: Type[_T]):
+ self._flag = flag
+ self._ctor = ctor
+
+ @property
+ def value(self) -> _T:
+ return self._ctor(**self._flag.value)
+
+ @property
+ def default(self) -> _T:
+ return self._ctor(**self._flag.default)
+
+ @property
+ def name(self) -> str:
+ return self._flag.name
+
+
+def DEFINE_config_dataclass( # pylint: disable=invalid-name
+ name: str,
+ config: _T,
+ help_string: str = 'Configuration object. Must be a dataclass.',
+ flag_values: flags.FlagValues = FLAGS,
+ **kwargs,
+) -> _TypedFlagHolder[_T]:
+ """Defines a typed (dataclass) flag-overrideable configuration.
+
+ Similar to `DEFINE_config_dict` except `config` should be a `dataclass`.
+
+ Args:
+ name: Flag name.
+ config: A user-defined configuration object. Must be built via `dataclass`.
+ help_string: Help string to display when --helpfull is called.
+ flag_values: FlagValues instance used for parsing.
+ **kwargs: Optional keyword arguments passed to Flag constructor.
+ Returns:
+ A handle to the defined flag.
+ """
+
+ if not dataclasses.is_dataclass(config):
+ raise ValueError('Configuration object must be a `dataclass`.')
+
+ # Convert to configdict *without* recursing into leaf node(s).
+ # If our config contains dataclasses (or other types) as fields, we want to
+ # preserve them; dataclasses.asdict recursively turns all fields into dicts.
+ dictionary = {field.name: getattr(config, field.name)
+ for field in dataclasses.fields(config)}
+ config_dict = ml_collections.ConfigDict(initial_dictionary=dictionary)
+
+ # Define the flag.
+ config_flag = DEFINE_config_dict(
+ name,
+ config=config_dict,
+ help_string=help_string,
+ flag_values=flag_values,
+ **kwargs,
+ )
+
+ return _TypedFlagHolder(flag=config_flag, ctor=config.__class__)
+
+
+def get_config_filename(config_flag) -> str: # pylint: disable=g-bad-name
+ """Returns the path to the config file given the config flag.
+
+ Args:
+ config_flag: The flag instance obtained from FLAGS, e.g. FLAGS['config'].
+
+ Returns:
+ the path to the config file.
+ """
+ if not is_config_flag(config_flag):
+ raise TypeError('expect a config flag, found {}'.format(type(config_flag)))
+ return config_flag.config_filename
+
+
+def get_override_values(config_flag) -> Dict[str, Any]: # pylint: disable=g-bad-name
+ """Returns a flat dict containing overridden values from the config flag.
+
+ Args:
+ config_flag: The flag instance obtained from FLAGS, e.g. FLAGS['config'].
+
+ Returns:
+ a flat dict containing overridden values from the config flag.
+ """
+ if not is_config_flag(config_flag):
+ raise TypeError('expect a config flag, found {}'.format(type(config_flag)))
+ return config_flag.override_values
-class _IgnoreFileNotFoundAndCollectErrors(object):
+class _IgnoreFileNotFoundAndCollectErrors:
"""Helps recording "file not found" exceptions when loading config.
Usage:
@@ -315,14 +396,14 @@ class _IgnoreFileNotFoundAndCollectErrors(object):
"""
def __init__(self):
- self._attempts = [] # type: List[Tuple[Tuple[Text, Text], IOError]]
+ self._attempts = [] # type: List[Tuple[Tuple[str, str], IOError]]
def Attempt(self, description, path):
"""Creates a context manager that routes exceptions to this class."""
self._current_attempt = (description, path)
ignore_errors = self
- class _ContextManager(object):
+ class _ContextManager:
def __enter__(self):
return self
@@ -346,7 +427,7 @@ class _IgnoreFileNotFoundAndCollectErrors(object):
for attempt, e in self._attempts)
-def _LoadConfigModule(name, path):
+def _LoadConfigModule(name: str, path: str):
"""Loads a script from external file specified by path.
Unprefixed path is looked for in the current working directory using
@@ -377,7 +458,7 @@ def _LoadConfigModule(name, path):
name, ignoring_errors.DescribeAttempts()))
-class _ErrorConfig(object):
+class _ErrorConfig:
"""Dummy ConfigDict that raises an error on any attribute access."""
def __init__(self, error):
@@ -531,7 +612,7 @@ class _ConfigFlag(flags.Flag):
if self._IsConfigSpecified(sys.argv):
self.default = default
else:
- super(_ConfigFlag, self)._set_default(default)
+ super(_ConfigFlag, self)._set_default(default) # pytype: disable=attribute-error
self.default_as_str = "'{}'".format(default)
def _parse(self, argument):
@@ -593,7 +674,8 @@ class _ConfigFlag(flags.Flag):
FLAGS['my_config'].config_filename
- will output 'ml_collections/config_flags/tests/configdict_config.py'
+ will output
+ 'ml_collections/config_flags/tests/configdict_config.py'
```
Returns:
@@ -684,7 +766,7 @@ def is_config_flag(flag): # pylint: disable=g-bad-name
return isinstance(flag, _ConfigFlag)
-class _ConfigFieldParser(object):
+class _ConfigFieldParser(flags.ArgumentParser):
"""Parser with config update after parsing.
This class-based wrapper creates a new object, which uses
@@ -693,7 +775,12 @@ class _ConfigFieldParser(object):
the config object.
"""
- def __init__(self, parser, path, config, override_values):
+ def __init__(
+ self,
+ parser: flags.ArgumentParser,
+ path: str,
+ config: ml_collections.ConfigDict,
+ override_values: MutableMapping[str, Any]):
"""Creates new parser with callback, using existing one to perform parsing.
Args:
@@ -718,6 +805,13 @@ class _ConfigFieldParser(object):
self._override_values[self._path] = value
return value
+ def flag_type(self) -> str:
+ return self._parser.flag_type()
+
+ @property
+ def syntactic_help(self) -> str:
+ return self._parser.syntactic_help
+
def _ExtractIndicesFromStep(step):
"""Separates identifier from indexes. Example: id[1][10] -> 'id', [1, 10]."""
@@ -782,7 +876,7 @@ def _TakeStep(current, step):
return _AccessConfig(current, field, indices)
-def GetValue(path, config):
+def GetValue(path: str, config: ml_collections.ConfigDict):
"""Gets value of a single field."""
current = config
for step in path.split('.'):
@@ -817,7 +911,7 @@ def GetTypes(paths, config):
return [GetType(path, config) for path in paths]
-def SetValue(path, config, value):
+def SetValue(path: str, config: ml_collections.ConfigDict, value):
"""Sets value of a single field."""
current = config
steps = path.split('.')
diff --git a/ml_collections/config_flags/examples/config.py b/ml_collections/config_flags/examples/config.py
index f647495..25a020e 100644
--- a/ml_collections/config_flags/examples/config.py
+++ b/ml_collections/config_flags/examples/config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/examples/define_config_dataclass_basic.py b/ml_collections/config_flags/examples/define_config_dataclass_basic.py
new file mode 100644
index 0000000..a285bc9
--- /dev/null
+++ b/ml_collections/config_flags/examples/define_config_dataclass_basic.py
@@ -0,0 +1,54 @@
+# Copyright 2021 The ML Collections Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Lint as: python 3
+r"""Example of basic DEFINE_config_dataclass usage.
+
+To run this example:
+python define_config_dataclass_basic.py -- --my_config.field1=8 \
+ --my_config.nested.field=2.1 --my_config.tuple='(1, 2, (1, 2))'
+"""
+
+from typing import Any, Mapping, Sequence
+
+from absl import app
+
+import dataclasses
+from ml_collections.config_flags import config_flags
+
+
+@dataclasses.dataclass
+class MyConfig:
+ field1: int
+ field2: str
+ nested: Mapping[str, Any]
+ tuple: Sequence[int]
+
+
+config = MyConfig(
+ field1=1,
+ field2='tom',
+ nested={'field': 2.23},
+ tuple=(1, 2, 3),
+)
+
+_CONFIG = config_flags.DEFINE_config_dataclass('my_config', config)
+
+
+def main(_):
+ print(_CONFIG.value)
+
+
+if __name__ == '__main__':
+ app.run(main)
diff --git a/ml_collections/config_flags/examples/define_config_dict_basic.py b/ml_collections/config_flags/examples/define_config_dict_basic.py
index 984b1a3..eb9cd89 100644
--- a/ml_collections/config_flags/examples/define_config_dict_basic.py
+++ b/ml_collections/config_flags/examples/define_config_dict_basic.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -13,7 +13,7 @@
# limitations under the License.
# Lint as: python 3
-r"""Example of basic DEFINE_flag_dict usage.
+r"""Example of basic DEFINE_config_dict usage.
To run this example:
python define_config_dict_basic.py -- --my_config_dict.field1=8 \
@@ -21,7 +21,6 @@ python define_config_dict_basic.py -- --my_config_dict.field1=8 \
"""
from absl import app
-from absl import flags
import ml_collections
from ml_collections.config_flags import config_flags
@@ -33,12 +32,11 @@ config.nested = ml_collections.ConfigDict()
config.nested.field = 2.23
config.tuple = (1, 2, 3)
-FLAGS = flags.FLAGS
-config_flags.DEFINE_config_dict('my_config_dict', config)
+_CONFIG = config_flags.DEFINE_config_dict('my_config_dict', config)
def main(_):
- print(FLAGS.my_config_dict)
+ print(_CONFIG.value)
if __name__ == '__main__':
diff --git a/ml_collections/config_flags/examples/define_config_file_basic.py b/ml_collections/config_flags/examples/define_config_file_basic.py
index 009489e..eeb68e0 100644
--- a/ml_collections/config_flags/examples/define_config_file_basic.py
+++ b/ml_collections/config_flags/examples/define_config_file_basic.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -32,16 +32,14 @@ python define_config_dict_basic.py -- \
# pylint: enable=line-too-long
from absl import app
-from absl import flags
from ml_collections.config_flags import config_flags
-FLAGS = flags.FLAGS
-config_flags.DEFINE_config_file('my_config')
+_CONFIG = config_flags.DEFINE_config_file('my_config')
def main(_):
- print(FLAGS.my_config)
+ print(_CONFIG.value)
if __name__ == '__main__':
diff --git a/ml_collections/config_flags/examples/examples_test.py b/ml_collections/config_flags/examples/examples_test.py
index b9aac67..09d7a47 100644
--- a/ml_collections/config_flags/examples/examples_test.py
+++ b/ml_collections/config_flags/examples/examples_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/examples/parameterised_config.py b/ml_collections/config_flags/examples/parameterised_config.py
index d72c966..3a9383e 100644
--- a/ml_collections/config_flags/examples/parameterised_config.py
+++ b/ml_collections/config_flags/examples/parameterised_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/config_overriding_test.py b/ml_collections/config_flags/tests/config_overriding_test.py
index 381e9f8..7249a5e 100644
--- a/ml_collections/config_flags/tests/config_overriding_test.py
+++ b/ml_collections/config_flags/tests/config_overriding_test.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -132,7 +132,7 @@ class _ConfigFlagTestCase(object):
def assert_equal_configs(self, config1, config2):
"""Checks if two configs are identical."""
self.assert_subset_configs(config1, config2)
- self.assert_subset_configs(config2, config1)
+ self.assert_subset_configs(config2, config1) # pylint: disable=arguments-out-of-order
class ConfigFileFlagTest(_ConfigFlagTestCase, parameterized.TestCase):
@@ -695,7 +695,8 @@ class ConfigFileFlagTest(_ConfigFlagTestCase, parameterized.TestCase):
_TEST_CONFIG_FILE, integer_override,
dictfloat_override))
- config.update_from_flattened_dict(values['test_config'].override_values)
+ config.update_from_flattened_dict(
+ config_flags.get_override_values(values['test_config']))
self.assertEqual(config['integer'], integer_override)
self.assertEqual(config['float'], original_float)
self.assertEqual(config['dict']['float'], dictfloat_override)
@@ -708,7 +709,8 @@ class ConfigFileFlagTest(_ConfigFlagTestCase, parameterized.TestCase):
def testConfigPath(self, config_file):
"""Test access to saved config file path."""
values = _parse_flags('./program --test_config={}'.format(config_file))
- self.assertEqual(values['test_config'].config_filename, config_file)
+ self.assertEqual(config_flags.get_config_filename(values['test_config']),
+ config_file)
def _simple_config():
diff --git a/ml_collections/config_flags/tests/configdict_config.py b/ml_collections/config_flags/tests/configdict_config.py
index 60defc2..969c01d 100644
--- a/ml_collections/config_flags/tests/configdict_config.py
+++ b/ml_collections/config_flags/tests/configdict_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/dataclass_overriding_test.py b/ml_collections/config_flags/tests/dataclass_overriding_test.py
new file mode 100644
index 0000000..8076d03
--- /dev/null
+++ b/ml_collections/config_flags/tests/dataclass_overriding_test.py
@@ -0,0 +1,86 @@
+# Copyright 2021 The ML Collections Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for config_flags used in conjunction with DEFINE_config_dataclass."""
+
+import shlex
+import sys
+from typing import Mapping, Optional, Sequence
+
+from absl import flags
+from absl.testing import absltest
+import dataclasses
+from ml_collections import config_flags
+
+
+#####
+# Simple dummy configuration classes.
+@dataclasses.dataclass
+class MyModelConfig:
+ foo: int
+ bar: Sequence[str]
+ baz: Optional[Mapping[str, str]] = None
+
+
+@dataclasses.dataclass
+class MyConfig:
+ my_model: MyModelConfig
+ baseline_model: MyModelConfig
+
+
+_CONFIG = MyConfig(
+ my_model=MyModelConfig(
+ foo=3,
+ bar=['a', 'b'],
+ baz={'foo': 'bar'},
+ ),
+ baseline_model=MyModelConfig(
+ foo=55,
+ bar=['c', 'd'],
+ ),
+)
+
+# Define the flag.
+_CONFIG_FLAG = config_flags.DEFINE_config_dataclass('config', _CONFIG)
+
+
+class TypedConfigFlagsTest(absltest.TestCase):
+
+ def test_instance(self):
+ config = _CONFIG_FLAG.value
+ self.assertIsInstance(config, MyConfig)
+ self.assertEqual(config.my_model, _CONFIG.my_model)
+ self.assertEqual(_CONFIG, config)
+
+ def test_flag_overrides(self):
+
+ # Set up some flag overrides.
+ old_argv = list(sys.argv)
+ sys.argv = shlex.split(
+ './program foo.py --test_config.baseline_model.foo=99')
+ flag_values = flags.FlagValues()
+
+ # Define a config dataclass flag.
+ test_config = config_flags.DEFINE_config_dataclass(
+ 'test_config', _CONFIG, flag_values=flag_values)
+
+ # Inject the flag overrides.
+ flag_values(sys.argv)
+ sys.argv = old_argv
+
+ # Did the value get overridden?
+ self.assertEqual(test_config.value.baseline_model.foo, 99)
+
+if __name__ == '__main__':
+ absltest.main()
diff --git a/ml_collections/config_flags/tests/fieldreference_config.py b/ml_collections/config_flags/tests/fieldreference_config.py
index 7c270ce..5954975 100644
--- a/ml_collections/config_flags/tests/fieldreference_config.py
+++ b/ml_collections/config_flags/tests/fieldreference_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/ioerror_config.py b/ml_collections/config_flags/tests/ioerror_config.py
index 58d9280..c27d7f6 100644
--- a/ml_collections/config_flags/tests/ioerror_config.py
+++ b/ml_collections/config_flags/tests/ioerror_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/mini_config.py b/ml_collections/config_flags/tests/mini_config.py
index 7c445f2..51377fe 100644
--- a/ml_collections/config_flags/tests/mini_config.py
+++ b/ml_collections/config_flags/tests/mini_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/mock_config.py b/ml_collections/config_flags/tests/mock_config.py
index b3132e3..696f288 100644
--- a/ml_collections/config_flags/tests/mock_config.py
+++ b/ml_collections/config_flags/tests/mock_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/parameterised_config.py b/ml_collections/config_flags/tests/parameterised_config.py
index a71edc5..df8ebfe 100644
--- a/ml_collections/config_flags/tests/parameterised_config.py
+++ b/ml_collections/config_flags/tests/parameterised_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/typeerror_config.py b/ml_collections/config_flags/tests/typeerror_config.py
index c01ebcf..a392a6f 100644
--- a/ml_collections/config_flags/tests/typeerror_config.py
+++ b/ml_collections/config_flags/tests/typeerror_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tests/valueerror_config.py b/ml_collections/config_flags/tests/valueerror_config.py
index 55a8520..126d187 100644
--- a/ml_collections/config_flags/tests/valueerror_config.py
+++ b/ml_collections/config_flags/tests/valueerror_config.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/ml_collections/config_flags/tuple_parser.py b/ml_collections/config_flags/tuple_parser.py
index 08de218..6e7df3d 100644
--- a/ml_collections/config_flags/tuple_parser.py
+++ b/ml_collections/config_flags/tuple_parser.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
diff --git a/requirements-test.txt b/requirements-test.txt
new file mode 100644
index 0000000..932a895
--- /dev/null
+++ b/requirements-test.txt
@@ -0,0 +1 @@
+mock
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..0c9f81f
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,6 @@
+absl-py
+PyYAML
+six
+contextlib2
+dataclasses;python_version<'3.7'
+typing;python_version<'3.5'
diff --git a/setup.py b/setup.py
index 14bf6e6..24af8e3 100644
--- a/setup.py
+++ b/setup.py
@@ -1,4 +1,4 @@
-# Copyright 2020 The ML Collections Authors.
+# Copyright 2021 The ML Collections Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -24,7 +24,7 @@ def _parse_requirements(requirements_txt_path):
return fp.read().splitlines()
-_VERSION = '0.1.0'
+_VERSION = '0.1.1'
setup(