New Upstream Release - natsort
Ready changes
Summary
Merged new upstream version: 8.3.1 (was: 8.2.0).
Resulting package
Built on 2023-04-08T10:13 (took 6m25s)
The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:
apt install -t fresh-releases python-natsort-docapt install -t fresh-releases python3-natsort
Lintian Result
Diff
diff --git a/.github/workflows/code-quality.yml b/.github/workflows/code-quality.yml
index 89d31f2..0fb0e8e 100644
--- a/.github/workflows/code-quality.yml
+++ b/.github/workflows/code-quality.yml
@@ -15,12 +15,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
- python-version: '3.6'
+ python-version: '3.8'
- name: Install black
run: pip install black
@@ -33,12 +33,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
- python-version: '3.6'
+ python-version: '3.8'
- name: Install Flake8
run: pip install flake8 flake8-import-order flake8-bugbear pep8-naming
@@ -51,12 +51,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
- python-version: '3.6'
+ python-version: '3.8'
- name: Install MyPy
run: pip install mypy hypothesis pytest pytest-mock fastnumbers
@@ -69,12 +69,12 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
- python-version: '3.6'
+ python-version: '3.8'
- name: Install Validators
run: pip install twine check-manifest
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
index e5e4f1d..70c3730 100644
--- a/.github/workflows/deploy.yml
+++ b/.github/workflows/deploy.yml
@@ -12,20 +12,21 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
python-version: 3.9
- name: Build Source Distribution and Wheel
run: |
pip install wheel
- python setup.py sdist --format=gztar bdist_wheel
+ python setup.py sdist --format=gztar
+ pip wheel . -w dist
- name: Publish to PyPI
- uses: pypa/gh-action-pypi-publish@master
+ uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.pypi_token_password }}
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index d78fdf9..64065ce 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -15,7 +15,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
- python-version: [3.6, 3.7, 3.8, 3.9, "3.10"]
+ python-version: [3.7, 3.8, 3.9, "3.10", "3.11"]
os: [ubuntu-latest]
extras: [false]
include:
@@ -25,10 +25,10 @@ jobs:
steps:
- name: Checkout code
- uses: actions/checkout@v2
+ uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
- uses: actions/setup-python@v2
+ uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
@@ -58,4 +58,25 @@ jobs:
run: coverage xml
- name: Upload to CodeCov
- uses: codecov/codecov-action@v1
+ uses: codecov/codecov-action@v3
+
+ test-bsd:
+ name: Test on FreeBSD
+ runs-on: macos-12
+
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v3
+
+ - name: Install and Run Tests
+ uses: vmactions/freebsd-vm@v0
+ with:
+ prepare: |
+ pkg install -y python3
+
+ run: |
+ python3 -m venv .venv
+ source .venv/bin/activate.csh
+ pip install --upgrade pip
+ pip install pytest pytest-mock hypothesis
+ python -m pytest --hypothesis-profile=slow-tests
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9e1b99a..dee5589 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,18 +1,65 @@
Unreleased
---
+[8.3.1] - 2023-03-01
+---
+
+### Fixed
+- Broken test on FreeBSD due to a broken `locale.strxfrm`.
+ **This change has no effect outside fixing tests**
+ (Issue [#161](https://github.com/SethMMorton/natsort/issues/161))
+
+[8.3.0] - 2023-02-27
+---
+
+### Added
+- The `PRESORT` option to the `ns` enum to attain consistent
+ sort order in certain corner cases (Issue
+ [#149](https://github.com/SethMMorton/natsort/issues/149))
+- Logic to ensure `None` and NaN are sorted in a consistent order
+ (Issue [#149](https://github.com/SethMMorton/natsort/issues/149))
+- Explict Python 3.11 support
+
+### Changed
+- Only convert to `str` if necessary in `os_sorted`
+ ([@Dobatymo](https://github.com/Dobatymo), issues
+ [#157](https://github.com/SethMMorton/natsort/issues/157) and
+ [#158](https://github.com/SethMMorton/natsort/issues/158))
+- Attempt to use new `fastnumbers` functionality if available
+- Move non-API documentation to the GitHub wiki
+
+### Removed
+
+- Support for EOL Python 3.6
+
+[8.2.0] - 2022-09-01
+---
+
+### Changed
+- Auto-coerce `pathlib.Path` objects to `str` since it is the least astonishing
+ behavior ([@Gilthans](https://github.com/Gilthans), issues #152, #153)
+- Reduce strictness of type hints to avoid over-constraining client code
+ (issues #154, #155)
+
+[8.1.0] - 2022-01-30
+---
+
+### Changed
+- When using `ns.PATH`, only split off a maximum of two suffixes from
+ a file name (issues #145, #146).
+
[8.0.2] - 2021-12-14
---
### Fixed
-- Bug where sorting paths fail if one of the paths is '.'.
+- Bug where sorting paths fail if one of the paths is '.' (issues #142, #143)
[8.0.1] - 2021-12-10
---
### Fixed
- Compose unicode characters when using locale to ensure sorting is correct
- across all locales.
+ across all locales (issues #140, #141)
[8.0.0] - 2021-11-03
---
@@ -52,7 +99,7 @@ Unreleased
### Changed
- MacOS unit tests run on native Python
- - Treate `None` like `NaN` internally to avoid `TypeError` (issue #117)
+ - Treat `None` like `NaN` internally to avoid `TypeError` (issue #117)
- No longer fail tests every time a new Python version is released (issue #122)
### Fixed
@@ -76,7 +123,7 @@ Unreleased
- Release checklist in `RELEASING.md` ([@hugovk](https://github.com/hugovk), issue #106)
### Changed
- - Updated auxillary shell scripts to be written in python, and added
+ - Updated auxiliary shell scripts to be written in python, and added
ability to call these from `tox`
- Improved Travis-CI experience
- Update testing dependency versions
@@ -281,7 +328,7 @@ Unreleased
because the new factory function paradigm eliminates most `if` branches
during execution). For the most cases, the code is 30-40% faster than version 4.0.4.
If using `ns.LOCALE` or `humansorted`, the code is 1100% faster than version 4.0.4
- - Improved clarity of documentaion with regards to locale-aware sorting
+ - Improved clarity of documentation with regards to locale-aware sorting
### Deprecated
- `ns.TYPESAFE` option as it is now always on (due to a new
@@ -427,7 +474,7 @@ Unreleased
- `reverse` option to `natsorted` & co. to make it's API more
similar to the builtin 'sorted'
- More unit tests
- - Auxillary test code that helps in profiling and stress-testing
+ - Auxiliary test code that helps in profiling and stress-testing
- Support for coveralls.io
### Changed
@@ -574,7 +621,7 @@ a pipeline by which to filter
---
### Added
- - Tests into the natsort.py file iteself
+ - Tests into the natsort.py file itself
### Changed
- Reorganized directory structure
@@ -590,6 +637,10 @@ a pipeline by which to filter
- Sorting algorithm to support floats (including exponentials) and basic version number support
<!---Comparison links-->
+[8.3.1]: https://github.com/SethMMorton/natsort/compare/8.3.0...8.3.1
+[8.3.0]: https://github.com/SethMMorton/natsort/compare/8.2.0...8.3.0
+[8.2.0]: https://github.com/SethMMorton/natsort/compare/8.1.0...8.2.0
+[8.1.0]: https://github.com/SethMMorton/natsort/compare/8.0.2...8.1.0
[8.0.2]: https://github.com/SethMMorton/natsort/compare/8.0.1...8.0.2
[8.0.1]: https://github.com/SethMMorton/natsort/compare/8.0.0...8.0.1
[8.0.0]: https://github.com/SethMMorton/natsort/compare/7.2.0...8.0.0
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 05f8492..696d60d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -7,11 +7,16 @@ robust algorithm. Contributions that change the public API of
less usable after the contribution and is backwards-compatible (unless there is
a good reason not to be).
+Located in the `dev/` folder is development collateral such as formatting and
+patching scripts. The only development collateral not in the `dev/`
+folder are those files that are expected to exist in the the top-level directory
+(such as `setup.py`, `tox.ini`, and CI configuration). All of these scripts
+can either be run with the python stdandard library, or have hooks in `tox`.
+
I do not have strong opinions on how one should contribute, so
I have copy/pasted some text verbatim from the
[Contributor's Guide](http://docs.python-requests.org/en/latest/dev/contributing/) section of
-[Kenneth Reitz's](http://docs.python-requests.org/en/latest/dev/contributing/)
-excellent [requests](https://github.com/kennethreitz/requests) library in
+the [requests](https://github.com/kennethreitz/requests) library in
lieu of coming up with my own.
> ### Steps for Submitting Code
@@ -27,7 +32,7 @@ lieu of coming up with my own.
> - Make your change.
> - Run the entire test suite again, confirming that all tests pass including the
ones you just added.
-> - Send a GitHub Pull Request to the main repository's master branch.
+> - Send a GitHub Pull Request to the main repository's main branch.
GitHub Pull Requests are the expected method of code collaboration on this project.
> ### Documentation Contributions
@@ -40,5 +45,3 @@ lieu of coming up with my own.
> When contributing documentation, please do your best to follow the style of the
documentation files. This means a soft-limit of 79 characters wide in your text
files and a semi-formal, yet friendly and approachable, prose style.
-
-> When presenting Python code, use single-quoted strings ('hello' instead of "hello").
diff --git a/LICENSE b/LICENSE
index ec76a82..cc66046 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,4 +1,4 @@
-Copyright (c) 2012-2021 Seth M. Morton
+Copyright (c) 2012-2023 Seth M. Morton
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
diff --git a/MANIFEST.in b/MANIFEST.in
index a4e0ee4..27d7981 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -2,6 +2,7 @@ include LICENSE
include CHANGELOG.md
include tox.ini
include RELEASING.md
+recursive-include mypy_stubs *.pyi
graft dev
graft docs
graft natsort
diff --git a/README.rst b/README.rst
index fb9c081..f1af055 100644
--- a/README.rst
+++ b/README.rst
@@ -8,23 +8,26 @@ natsort
:target: https://pypi.org/project/natsort/
.. image:: https://img.shields.io/pypi/l/natsort.svg
- :target: https://github.com/SethMMorton/natsort/blob/master/LICENSE
+ :target: https://github.com/SethMMorton/natsort/blob/main/LICENSE
.. image:: https://github.com/SethMMorton/natsort/workflows/Tests/badge.svg
:target: https://github.com/SethMMorton/natsort/actions
-.. image:: https://codecov.io/gh/SethMMorton/natsort/branch/master/graph/badge.svg
+.. image:: https://codecov.io/gh/SethMMorton/natsort/branch/main/graph/badge.svg
:target: https://codecov.io/gh/SethMMorton/natsort
+.. image:: https://img.shields.io/pypi/dw/natsort.svg
+ :target: https://pypi.org/project/natsort/
+
Simple yet flexible natural sorting in Python.
- Source Code: https://github.com/SethMMorton/natsort
- Downloads: https://pypi.org/project/natsort/
- Documentation: https://natsort.readthedocs.io/
- - `Examples and Recipes <https://natsort.readthedocs.io/en/master/examples.html>`_
- - `How Does Natsort Work? <https://natsort.readthedocs.io/en/master/howitworks.html>`_
- - `API <https://natsort.readthedocs.io/en/master/api.html>`_
+ - `Examples and Recipes`_
+ - `How Does Natsort Work?`_
+ - `API`_
- `Quick Description`_
- `Quick Examples`_
@@ -34,11 +37,10 @@ Simple yet flexible natural sorting in Python.
- `Installation`_
- `How to Run Tests`_
- `How to Build Documentation`_
- - `Deprecation Schedule`_
+ - `Dropped Deprecated APIs`_
- `History`_
-**NOTE**: Please see the `Deprecation Schedule`_ section for changes in
-``natsort`` version 7.0.0.
+**NOTE**: Please see the `Dropped Deprecated APIs`_ section for changes.
Quick Description
-----------------
@@ -57,10 +59,10 @@ Notice that it has the order ('1', '10', '2') - this is because the list is
being sorted in lexicographical order, which sorts numbers like you would
letters (i.e. 'b', 'ba', 'c').
-``natsort`` provides a function ``natsorted`` that helps sort lists
+`natsort`_ provides a function `natsorted()`_ that helps sort lists
"naturally" ("naturally" is rather ill-defined, but in general it means
sorting based on meaning and not computer code point).
-Using ``natsorted`` is simple:
+Using `natsorted()`_ is simple:
.. code-block:: pycon
@@ -69,14 +71,13 @@ Using ``natsorted`` is simple:
>>> natsorted(a)
['1 ft 5 in', '2 ft 7 in', '2 ft 11 in', '7 ft 6 in', '10 ft 2 in']
-``natsorted`` identifies numbers anywhere in a string and sorts them
-naturally. Below are some other things you can do with ``natsort``
-(also see the `examples <https://natsort.readthedocs.io/en/master/examples.html>`_
-for a quick start guide, or the
-`api <https://natsort.readthedocs.io/en/master/api.html>`_ for complete details).
+`natsorted()`_ identifies numbers anywhere in a string and sorts them
+naturally. Below are some other things you can do with `natsort`_
+(also see the `Examples and Recipes`_ for a quick start guide, or the
+`API`_ for complete details).
-**Note**: ``natsorted`` is designed to be a drop-in replacement for the
-built-in ``sorted`` function. Like ``sorted``, ``natsorted``
+**Note**: `natsorted()`_ is designed to be a drop-in replacement for the
+built-in `sorted()`_ function. Like `sorted()`_, `natsorted()`_
`does not sort in-place`. To sort a list and assign the output to the same
variable, you must explicitly assign the output to a variable:
@@ -103,19 +104,19 @@ Quick Examples
- `Locale-Aware Sorting (or "Human Sorting")`_
- `Further Customizing Natsort`_
- `Sorting Mixed Types`_
-- `Handling Bytes on Python 3`_
+- `Handling Bytes`_
- `Generating a Reusable Sorting Key and Sorting In-Place`_
- `Other Useful Things`_
Sorting Versions
++++++++++++++++
-``natsort`` does not actually *comprehend* version numbers.
+`natsort`_ does not actually *comprehend* version numbers.
It just so happens that the most common versioning schemes are designed to
work with standard natural sorting techniques; these schemes include
``MAJOR.MINOR``, ``MAJOR.MINOR.PATCH``, ``YEAR.MONTH.DAY``. If your data
conforms to a scheme like this, then it will work out-of-the-box with
-``natsorted`` (as of ``natsort`` version >= 4.0.0):
+`natsorted()`_ (as of `natsort`_ version >= 4.0.0):
.. code-block:: pycon
@@ -124,14 +125,14 @@ conforms to a scheme like this, then it will work out-of-the-box with
['version-1.9', 'version-1.10', 'version-1.11', 'version-2.0']
If you need to versions that use a more complicated scheme, please see
-`these examples <https://natsort.readthedocs.io/en/master/examples.html#rc-sorting>`_.
+`these version sorting examples`_.
Sort Paths Like My File Browser (e.g. Windows Explorer on Windows)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-Prior to ``natsort`` version 7.1.0, it was a common request to be able to
-sort paths like Windows Explorer. As of ``natsort`` 7.1.0, the function
-``os_sorted`` has been added to provide users the ability to sort
+Prior to `natsort`_ version 7.1.0, it was a common request to be able to
+sort paths like Windows Explorer. As of `natsort`_ 7.1.0, the function
+`os_sorted()`_ has been added to provide users the ability to sort
in the order that their file browser might sort (e.g Windows Explorer on
Windows, Finder on MacOS, Dolphin/Nautilus/Thunar/etc. on Linux).
@@ -145,17 +146,17 @@ Windows, Finder on MacOS, Dolphin/Nautilus/Thunar/etc. on Linux).
Output will be different depending on the operating system you are on.
For users **not** on Windows (e.g. MacOS/Linux) it is **strongly** recommended
-to also install `PyICU <https://pypi.org/project/PyICU>`_, which will help
-``natsort`` give results that match most file browsers. If this is not installed,
-it will fall back on Python's built-in ``locale`` module and will give good
+to also install `PyICU`_, which will help
+`natsort`_ give results that match most file browsers. If this is not installed,
+it will fall back on Python's built-in `locale`_ module and will give good
results for most input, but will give poor results for special characters.
Sorting by Real Numbers (i.e. Signed Floats)
++++++++++++++++++++++++++++++++++++++++++++
-This is useful in scientific data analysis (and was
-the default behavior of ``natsorted`` for ``natsort``
-version < 4.0.0). Use the ``realsorted`` function:
+This is useful in scientific data analysis (and was the default behavior
+of `natsorted()`_ for `natsort`_ version < 4.0.0). Use the `realsorted()`_
+function:
.. code-block:: pycon
@@ -176,7 +177,7 @@ Locale-Aware Sorting (or "Human Sorting")
This is where the non-numeric characters are also ordered based on their
meaning, not on their ordinal value, and a locale-dependent thousands
separator and decimal separator is accounted for in the number.
-This can be achieved with the ``humansorted`` function:
+This can be achieved with the `humansorted()`_ function:
.. code-block:: pycon
@@ -193,9 +194,8 @@ This can be achieved with the ``humansorted`` function:
['apple15', 'apple14,689', 'Apple', 'banana', 'Banana']
You may find you need to explicitly set the locale to get this to work
-(as shown in the example).
-Please see `locale issues <https://natsort.readthedocs.io/en/master/locale_issues.html>`_ and the
-`Optional Dependencies`_ section below before using the ``humansorted`` function.
+(as shown in the example). Please see `locale issues`_ and the
+`Optional Dependencies`_ section below before using the `humansorted()`_ function.
Further Customizing Natsort
+++++++++++++++++++++++++++
@@ -219,7 +219,7 @@ bitwise OR operator (``|``). For example,
True
All of the available customizations can be found in the documentation for
-`the ns enum <https://natsort.readthedocs.io/en/master/api.html#natsort.ns>`_.
+`the ns enum`_.
You can also add your own custom transformation functions with the ``key``
argument. These can be used with ``alg`` if you wish.
@@ -233,44 +233,40 @@ argument. These can be used with ``alg`` if you wish.
Sorting Mixed Types
+++++++++++++++++++
-You can mix and match ``int``, ``float``, and ``str`` (or ``unicode``) types
-when you sort:
+You can mix and match `int`_, `float`_, and `str`_ types when you sort:
.. code-block:: pycon
>>> a = ['4.5', 6, 2.0, '5', 'a']
>>> natsorted(a)
[2.0, '4.5', '5', 6, 'a']
- >>> # On Python 2, sorted(a) would return [2.0, 6, '4.5', '5', 'a']
- >>> # On Python 3, sorted(a) would raise an "unorderable types" TypeError
+ >>> # sorted(a) would raise an "unorderable types" TypeError
-Handling Bytes on Python 3
-++++++++++++++++++++++++++
+Handling Bytes
+++++++++++++++
-``natsort`` does not officially support the `bytes` type on Python 3, but
-convenience functions are provided that help you decode to `str` first:
+`natsort`_ does not officially support the `bytes`_ type, but
+convenience functions are provided that help you decode to `str`_ first:
.. code-block:: pycon
>>> from natsort import as_utf8
>>> a = [b'a', 14.0, 'b']
- >>> # On Python 2, natsorted(a) would would work as expected.
- >>> # On Python 3, natsorted(a) would raise a TypeError (bytes() < str())
+ >>> # natsorted(a) would raise a TypeError (bytes() < str())
>>> natsorted(a, key=as_utf8) == [14.0, b'a', 'b']
True
>>> a = [b'a56', b'a5', b'a6', b'a40']
- >>> # On Python 2, natsorted(a) would would work as expected.
- >>> # On Python 3, natsorted(a) would return the same results as sorted(a)
+ >>> # natsorted(a) would return the same results as sorted(a)
>>> natsorted(a, key=as_utf8) == [b'a5', b'a6', b'a40', b'a56']
True
Generating a Reusable Sorting Key and Sorting In-Place
++++++++++++++++++++++++++++++++++++++++++++++++++++++
-Under the hood, ``natsorted`` works by generating a custom sorting
-key using ``natsort_keygen`` and then passes that to the built-in
-``sorted``. You can use the ``natsort_keygen`` function yourself to
-generate a custom sorting key to sort in-place using the ``list.sort``
+Under the hood, `natsorted()`_ works by generating a custom sorting
+key using `natsort_keygen()`_ and then passes that to the built-in
+`sorted()`_. You can use the `natsort_keygen()`_ function yourself to
+generate a custom sorting key to sort in-place using the `list.sort()`_
method.
.. code-block:: pycon
@@ -286,53 +282,49 @@ method.
All of the algorithm customizations mentioned in the
`Further Customizing Natsort`_ section can also be applied to
-``natsort_keygen`` through the *alg* keyword option.
+`natsort_keygen()`_ through the *alg* keyword option.
Other Useful Things
+++++++++++++++++++
- recursively descend into lists of lists
- automatic unicode normalization of input data
- - `controlling the case-sensitivity <https://natsort.readthedocs.io/en/master/examples.html#case-sort>`_
- - `sorting file paths correctly <https://natsort.readthedocs.io/en/master/examples.html#path-sort>`_
- - `allow custom sorting keys <https://natsort.readthedocs.io/en/master/examples.html#custom-sort>`_
- - `accounting for units <https://natsort.readthedocs.io/en/master/examples.html#accounting-for-units-when-sorting>`_
+ - `controlling the case-sensitivity`_
+ - `sorting file paths correctly`_
+ - `allow custom sorting keys`_
+ - `accounting for units`_
FAQ
---
-How do I debug ``natsort.natsorted()``?
- The best way to debug ``natsorted()`` is to generate a key using ``natsort_keygen()``
- with the same options being passed to ``natsorted``. One can take a look at
+How do I debug `natsorted()`_?
+ The best way to debug `natsorted()`_ is to generate a key using `natsort_keygen()`_
+ with the same options being passed to `natsorted()`_. One can take a look at
exactly what is being done with their input using this key - it is highly
- recommended
- to `look at this issue describing how to debug <https://github.com/SethMMorton/natsort/issues/13#issuecomment-50422375>`_
- for *how* to debug, and also to review the
- `How Does Natsort Work? <https://natsort.readthedocs.io/en/master/howitworks.html>`_
- page for *why* ``natsort`` is doing that to your data.
+ recommended to `look at this issue describing how to debug`_ for *how* to debug,
+ and also to review the `How Does Natsort Work?`_ page for *why* `natsort`_ is
+ doing that to your data.
If you are trying to sort custom classes and running into trouble, please
take a look at https://github.com/SethMMorton/natsort/issues/60. In short,
custom classes are not likely to be sorted correctly if one relies
on the behavior of ``__lt__`` and the other rich comparison operators in
their custom class - it is better to use a ``key`` function with
- ``natsort``, or use the ``natsort`` key as part of your rich comparison
+ `natsort`_, or use the `natsort`_ key as part of your rich comparison
operator definition.
-``natsort`` gave me results I didn't expect, and it's a terrible library!
+`natsort`_ gave me results I didn't expect, and it's a terrible library!
Did you try to debug using the above advice? If so, and you still cannot figure out
- the error, then please `file an issue <https://github.com/SethMMorton/natsort/issues/new>`_.
+ the error, then please `file an issue`_.
-How *does* ``natsort`` work?
- If you don't want to read `How Does Natsort Work? <https://natsort.readthedocs.io/en/master/howitworks.html>`_,
+How *does* `natsort`_ work?
+ If you don't want to read `How Does Natsort Work?`_,
here is a quick primer.
- ``natsort`` provides a `key function <https://docs.python.org/3/howto/sorting.html#key-functions>`_
- that can be passed to `list.sort() <https://docs.python.org/3/library/stdtypes.html#list.sort>`_
- or `sorted() <https://docs.python.org/3/library/functions.html#sorted>`_ in order to
- modify the default sorting behavior. This key is generated on-demand with
- the key generator ``natsort.natsort_keygen()``. ``natsort.natsorted()``
- is essentially a wrapper for the following code:
+ `natsort`_ provides a `key function`_ that can be passed to `list.sort()`_
+ or `sorted()`_ in order to modify the default sorting behavior. This key
+ is generated on-demand with the key generator `natsort_keygen()`_.
+ `natsorted()`_ is essentially a wrapper for the following code:
.. code-block:: pycon
@@ -341,35 +333,36 @@ How *does* ``natsort`` work?
>>> sorted(['1', '10', '2'], key=natsort_key)
['1', '2', '10']
- Users can further customize ``natsort`` sorting behavior with the ``key``
+ Users can further customize `natsort`_ sorting behavior with the ``key``
and/or ``alg`` options (see details in the `Further Customizing Natsort`_
section).
- The key generated by ``natsort_keygen`` *always* returns a ``tuple``. It
+ The key generated by `natsort_keygen()`_ *always* returns a `tuple`_. It
does so in the following way (*some details omitted for clarity*):
1. Assume the input is a string, and attempt to split it into numbers and
non-numbers using regular expressions. Numbers are then converted into
- either ``int`` or ``float``.
+ either `int`_ or `float`_.
2. If the above fails because the input is not a string, assume the input
- is some other sequence (e.g. ``list`` or ``tuple``), and recursively
+ is some other sequence (e.g. `list`_ or `tuple`_), and recursively
apply the key to each element of the sequence.
3. If the above fails because the input is not iterable, assume the input
- is an ``int`` or ``float``, and just return the input in a ``tuple``.
+ is an `int`_ or `float`_, and just return the input in a `tuple`_.
- Because a ``tuple`` is always returned, a ``TypeError`` should not be common
- unless one tries to do something odd like sort an ``int`` against a ``list``.
+ Because a `tuple`_ is always returned, a `TypeError`_ should not be common
+ unless one tries to do something odd like sort an `int`_ against a `list`_.
Shell script
------------
-``natsort`` comes with a shell script called ``natsort``, or can also be called
-from the command line with ``python -m natsort``.
+`natsort`_ comes with a shell script called `natsort`_, or can also be called
+from the command line with ``python -m natsort``. Check out the
+`shell script wiki documentation`_ for more details.
Requirements
------------
-``natsort`` requires Python 3.6 or greater.
+`natsort`_ requires Python 3.7 or greater.
Optional Dependencies
---------------------
@@ -378,20 +371,18 @@ fastnumbers
+++++++++++
The most efficient sorting can occur if you install the
-`fastnumbers <https://pypi.org/project/fastnumbers>`_ package
+`fastnumbers`_ package
(version >=2.0.0); it helps with the string to number conversions.
-``natsort`` will still run (efficiently) without the package, but if you need
+`natsort`_ will still run (efficiently) without the package, but if you need
to squeeze out that extra juice it is recommended you include this as a
-dependency. ``natsort`` will not require (or check) that
-`fastnumbers <https://pypi.org/project/fastnumbers>`_ is installed
-at installation.
+dependency. `natsort`_ will not require (or check) that
+`fastnumbers`_ is installed at installation.
PyICU
+++++
-It is recommended that you install `PyICU <https://pypi.org/project/PyICU>`_
-if you wish to sort in a locale-dependent manner, see
-https://natsort.readthedocs.io/en/master/locale_issues.html for an explanation why.
+It is recommended that you install `PyICU`_ if you wish to sort in a
+locale-dependent manner, see this page on `locale issues`_ for an explanation why.
Installation
------------
@@ -403,10 +394,8 @@ Use ``pip``!
$ pip install natsort
If you want to install the `Optional Dependencies`_, you can use the
-`"extras" notation <https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras>`_
-at installation time to install those dependencies as well - use ``fast`` for
-`fastnumbers <https://pypi.org/project/fastnumbers>`_ and ``icu`` for
-`PyICU <https://pypi.org/project/PyICU>`_.
+`"extras" notation`_ at installation time to install those dependencies as
+well - use ``fast`` for `fastnumbers`_ and ``icu`` for `PyICU`_.
.. code-block:: console
@@ -418,11 +407,10 @@ at installation time to install those dependencies as well - use ``fast`` for
How to Run Tests
----------------
-Please note that ``natsort`` is NOT set-up to support ``python setup.py test``.
+Please note that `natsort`_ is NOT set-up to support ``python setup.py test``.
-The recommended way to run tests is with `tox <https://tox.readthedocs.io/en/latest/>`_.
-After installing ``tox``, running tests is as simple as executing the following
-in the ``natsort`` directory:
+The recommended way to run tests is with `tox`_. After installing ``tox``,
+running tests is as simple as executing the following in the `natsort`_ directory:
.. code-block:: console
@@ -437,7 +425,7 @@ with ``tox --listenvs``.
How to Build Documentation
--------------------------
-If you want to build the documentation for ``natsort``, it is recommended to
+If you want to build the documentation for `natsort`_, it is recommended to
use ``tox``:
.. code-block:: console
@@ -446,29 +434,10 @@ use ``tox``:
This will place the documentation in ``build/sphinx/html``.
-Deprecation Schedule
---------------------
-
-Dropped Python 3.4 and Python 3.5 Support
-+++++++++++++++++++++++++++++++++++++++++
-
-``natsort`` version 8.0.0 dropped support for Python < 3.6.
-
-Dropped Python 2.7 Support
-++++++++++++++++++++++++++
-
-``natsort`` version 7.0.0 dropped support for Python 2.7.
-
-The version 6.X branch will remain as a "long term support" branch where bug
-fixes are applied so that users who cannot update from Python 2.7 will not be
-forced to use a buggy ``natsort`` version (bug fixes will need to be requested;
-by default only the 7.X branch will be updated).
-New features would not be added to version 6.X, only bug fixes.
-
Dropped Deprecated APIs
-+++++++++++++++++++++++
+-----------------------
-In ``natsort`` version 6.0.0, the following APIs and functions were removed
+In `natsort`_ version 6.0.0, the following APIs and functions were removed
- ``number_type`` keyword argument (deprecated since 3.4.0)
- ``signed`` keyword argument (deprecated since 3.4.0)
@@ -490,7 +459,7 @@ can run your code with the following flag
$ python -Wdefault::DeprecationWarning my-code.py
-By default ``DeprecationWarnings`` are not shown, but this will cause them
+By default `DeprecationWarnings`_ are not shown, but this will cause them
to be shown. Alternatively, you can just set the environment variable
``PYTHONWARNINGS`` to "default::DeprecationWarning" and then run your code.
@@ -502,6 +471,42 @@ Seth M. Morton
History
-------
-Please visit the changelog
-`on GitHub <https://github.com/SethMMorton/natsort/blob/master/CHANGELOG.md>`_ or
-`in the documentation <https://natsort.readthedocs.io/en/master/changelog.html>`_.
+Please visit the changelog `on GitHub`_ or `in the documentation`_.
+
+.. _natsort: https://natsort.readthedocs.io/en/stable/index.html
+.. _natsorted(): https://natsort.readthedocs.io/en/stable/api.html#natsort.natsorted
+.. _natsort_keygen(): https://natsort.readthedocs.io/en/stable/api.html#natsort.natsort_keygen
+.. _realsorted(): https://natsort.readthedocs.io/en/stable/api.html#natsort.realsorted
+.. _humansorted(): https://natsort.readthedocs.io/en/stable/api.html#natsort.humansorted
+.. _os_sorted(): https://natsort.readthedocs.io/en/stable/api.html#natsort.os_sorted
+.. _the ns enum: https://natsort.readthedocs.io/en/stable/api.html#natsort.ns
+.. _fastnumbers: https://github.com/SethMMorton/fastnumbers
+.. _sorted(): https://docs.python.org/3/library/functions.html#sorted
+.. _list.sort(): https://docs.python.org/3/library/stdtypes.html#list.sort
+.. _key function: https://docs.python.org/3/howto/sorting.html#key-functions
+.. _locale: https://docs.python.org/3/library/locale.html
+.. _int: https://docs.python.org/3/library/functions.html#int
+.. _float: https://docs.python.org/3/library/functions.html#float
+.. _str: https://docs.python.org/3/library/stdtypes.html#str
+.. _bytes: https://docs.python.org/3/library/stdtypes.html#bytes
+.. _list: https://docs.python.org/3/library/stdtypes.html#list
+.. _tuple: https://docs.python.org/3/library/stdtypes.html#tuple
+.. _TypeError: https://docs.python.org/3/library/exceptions.html#TypeError
+.. _DeprecationWarnings: https://docs.python.org/3/library/exceptions.html#DeprecationWarning
+.. _"extras" notation: https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras
+.. _PyICU: https://pypi.org/project/PyICU
+.. _tox: https://tox.readthedocs.io/en/latest/
+.. _Examples and Recipes: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes
+.. _How Does Natsort Work?: https://github.com/SethMMorton/natsort/wiki/How-Does-Natsort-Work%3F
+.. _API: https://natsort.readthedocs.io/en/stable/api.html
+.. _on GitHub: https://github.com/SethMMorton/natsort/blob/main/CHANGELOG.md
+.. _in the documentation: https://natsort.readthedocs.io/en/stable/changelog.html
+.. _file an issue: https://github.com/SethMMorton/natsort/issues/new
+.. _look at this issue describing how to debug: https://github.com/SethMMorton/natsort/issues/13#issuecomment-50422375
+.. _controlling the case-sensitivity: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes#controlling-case-when-sorting
+.. _sorting file paths correctly: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes#sort-os-generated-paths
+.. _allow custom sorting keys: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes#using-a-custom-sorting-key
+.. _accounting for units: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes#accounting-for-units-when-sorting
+.. _these version sorting examples: https://github.com/SethMMorton/natsort/wiki/Examples-and-Recipes#sorting-more-expressive-versioning-schemes
+.. _locale issues: https://github.com/SethMMorton/natsort/wiki/Possible-Issues-with-natsort.humansorted-or-ns.LOCALE
+.. _shell script wiki documentation: https://github.com/SethMMorton/natsort/wiki/Shell-Script
\ No newline at end of file
diff --git a/RELEASING.md b/RELEASING.md
index 1c6932f..9f510c6 100644
--- a/RELEASING.md
+++ b/RELEASING.md
@@ -1,8 +1,8 @@
# Release Checklist
-- [ ] Get master to the appropriate code release state.
- [Travis CI](https://travis-ci.com/SethMMorton/natsort) must be passing:
- [](https://travis-ci.com/SethMMorton/natsort)
+- [ ] Get main to the appropriate code release state.
+ [GitHub Actions](https://github.com/SethMMorton/natsort/actions) must be passing:
+ [](https://github.com/SethMMorton/natsort/actions)
- [ ] Ensure that the `CHANGELOG.md` includes the changes made since last release.
Please follow the style outlined in https://keepachangelog.com/.
@@ -23,16 +23,13 @@
git push
```
-- [ ] Check that the [Travis CI build](https://travis-ci.com/SethMMorton/natsort) has
- deployed correctly to [the test PyPI](https://test.pypi.org/project/natsort/#history).
-
- [ ] Push the tag:
```bash
git push --tags
```
-- [ ] Check that the tagged [Travis CI build](https://travis-ci.com/SethMMorton/natsort) has
+- [ ] Check that the tagged [GitHub Actions build](https://github.com/SethMMorton/natsort/actions) has
deployed correctly to [PyPI](https://pypi.org/project/natsort/#history).
- [ ] Check installation:
diff --git a/debian/changelog b/debian/changelog
index 466e38b..d82fbe2 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,10 @@
-natsort (8.0.2-3) UNRELEASED; urgency=medium
+natsort (8.3.1-1) UNRELEASED; urgency=medium
* Update standards version to 4.6.2, no changes needed.
+ * New upstream release.
+ * New upstream release.
- -- Debian Janitor <janitor@jelmer.uk> Fri, 06 Jan 2023 05:03:37 -0000
+ -- Debian Janitor <janitor@jelmer.uk> Sat, 08 Apr 2023 10:07:49 -0000
natsort (8.0.2-2) unstable; urgency=medium
diff --git a/docs/api.rst b/docs/api.rst
index 39d7cec..ddf6964 100644
--- a/docs/api.rst
+++ b/docs/api.rst
@@ -83,8 +83,8 @@ Convenience Functions
.. _bytes_help:
-Help With Bytes On Python 3
-+++++++++++++++++++++++++++
+Help With Bytes
++++++++++++++++
The official stance of :mod:`natsort` is to not support `bytes` for
sorting; there is just too much that can go wrong when trying to automate
@@ -121,7 +121,7 @@ the corresponding regular expression to locate numbers will be returned.
Help With Type Hinting
++++++++++++++++++++++
-If you need to explictly specify the types that natsort accepts or returns
+If you need to explicitly specify the types that natsort accepts or returns
in your code, the following types have been exposed for your convenience.
+--------------------------------+----------------------------------------------------------------------------------------+
diff --git a/docs/conf.py b/docs/conf.py
index f19e176..ab9c3ee 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,7 +59,7 @@ copyright = "2014, Seth M. Morton"
# built documents.
#
# The full version, including alpha/beta/rc tags.
-release = "8.0.2"
+release = "8.3.1"
# The short X.Y version.
version = ".".join(release.split(".")[0:2])
diff --git a/docs/examples.rst b/docs/examples.rst
index a0e3a01..b372ca7 100644
--- a/docs/examples.rst
+++ b/docs/examples.rst
@@ -6,422 +6,5 @@
Examples and Recipes
====================
-If you want more detailed examples than given on this page, please see
-https://github.com/SethMMorton/natsort/tree/master/tests.
-
-.. contents::
- :local:
-
-Basic Usage
------------
-
-In the most basic use case, simply import :func:`~natsorted` and use
-it as you would :func:`sorted`:
-
-.. code-block:: pycon
-
- >>> a = ['2 ft 7 in', '1 ft 5 in', '10 ft 2 in', '2 ft 11 in', '7 ft 6 in']
- >>> sorted(a)
- ['1 ft 5 in', '10 ft 2 in', '2 ft 11 in', '2 ft 7 in', '7 ft 6 in']
- >>> from natsort import natsorted, ns
- >>> natsorted(a)
- ['1 ft 5 in', '2 ft 7 in', '2 ft 11 in', '7 ft 6 in', '10 ft 2 in']
-
-Sort Version Numbers
---------------------
-
-As of :mod:`natsort` version >= 4.0.0, :func:`~natsorted` will work for
-well-behaved version numbers, like ``MAJOR.MINOR.PATCH``.
-
-.. _rc_sorting:
-
-Sorting More Expressive Versioning Schemes
-++++++++++++++++++++++++++++++++++++++++++
-
-By default, if you wish to sort versions that are not as simple as
-``MAJOR.MINOR.PATCH`` (or similar), you may not get the results you expect:
-
-.. code-block:: pycon
-
- >>> a = ['1.2', '1.2rc1', '1.2beta2', '1.2beta1', '1.2alpha', '1.2.1', '1.1', '1.3']
- >>> natsorted(a)
- ['1.1', '1.2', '1.2.1', '1.2alpha', '1.2beta1', '1.2beta2', '1.2rc1', '1.3']
-
-To make the '1.2' pre-releases come before '1.2.1', you need to use the
-following recipe:
-
-.. code-block:: pycon
-
- >>> natsorted(a, key=lambda x: x.replace('.', '~'))
- ['1.1', '1.2', '1.2alpha', '1.2beta1', '1.2beta2', '1.2rc1', '1.2.1', '1.3']
-
-If you also want '1.2' after all the alpha, beta, and rc candidates, you can
-modify the above recipe:
-
-.. code-block:: pycon
-
- >>> natsorted(a, key=lambda x: x.replace('.', '~')+'z')
- ['1.1', '1.2alpha', '1.2beta1', '1.2beta2', '1.2rc1', '1.2', '1.2.1', '1.3']
-
-Please see `this issue <https://github.com/SethMMorton/natsort/issues/13>`_ to
-see why this works.
-
-Sorting Rigorously Defined Versioning Schemes (e.g. SemVer or PEP 440)
-""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
-
-If you know you are using a versioning scheme that follows a well-defined format
-for which there is third-party module support, you should use those modules
-to assist in sorting. Some examples might be
-`PEP 440 <https://packaging.pypa.io/en/latest/version>`_ or
-`SemVer <https://python-semver.readthedocs.io/en/latest/api.html>`_.
-
-If we are being honest, using these methods to parse a version means you don't
-need to use :mod:`natsort` - you should probably just use :func:`sorted`
-directly. Here's an example with SemVer:
-
-.. code-block:: pycon
-
- >>> from semver import VersionInfo
- >>> a = ['3.4.5-pre.1', '3.4.5', '3.4.5-pre.2+build.4']
- >>> sorted(a, key=VersionInfo.parse)
- ['3.4.5-pre.1', '3.4.5-pre.2+build.4', '3.4.5']
-
-.. _path_sort:
-
-Sort OS-Generated Paths
------------------------
-
-In some cases when sorting file paths with OS-Generated names, the default
-:mod:`~natsorted` algorithm may not be sufficient. In cases like these,
-you may need to use the ``ns.PATH`` option:
-
-.. code-block:: pycon
-
- >>> a = ['./folder/file (1).txt',
- ... './folder/file.txt',
- ... './folder (1)/file.txt',
- ... './folder (10)/file.txt']
- >>> natsorted(a)
- ['./folder (1)/file.txt', './folder (10)/file.txt', './folder/file (1).txt', './folder/file.txt']
- >>> natsorted(a, alg=ns.PATH)
- ['./folder/file.txt', './folder/file (1).txt', './folder (1)/file.txt', './folder (10)/file.txt']
-
-Locale-Aware Sorting (Human Sorting)
-------------------------------------
-
-.. note::
- Please read :ref:`locale_issues` before using ``ns.LOCALE``, :func:`humansorted`,
- or :func:`index_humansorted`.
-
-You can instruct :mod:`natsort` to use locale-aware sorting with the
-``ns.LOCALE`` option. In addition to making this understand non-ASCII
-characters, it will also properly interpret non-'.' decimal separators
-and also properly order case. It may be more convenient to just use
-the :func:`humansorted` function:
-
-.. code-block:: pycon
-
- >>> from natsort import humansorted
- >>> import locale
- >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
- 'en_US.UTF-8'
- >>> a = ['Apple', 'corn', 'Corn', 'Banana', 'apple', 'banana']
- >>> natsorted(a, alg=ns.LOCALE)
- ['apple', 'Apple', 'banana', 'Banana', 'corn', 'Corn']
- >>> humansorted(a)
- ['apple', 'Apple', 'banana', 'Banana', 'corn', 'Corn']
-
-You may find that if you do not explicitly set the locale your results may not
-be as you expect... I have found that it depends on the system you are on.
-If you use `PyICU <https://pypi.org/project/PyICU>`_ (see below) then
-you should not need to do this.
-
-.. _case_sort:
-
-Controlling Case When Sorting
------------------------------
-
-For non-numbers, by default :mod:`natsort` used ordinal sorting (i.e.
-it sorts by the character's value in the ASCII table). For example:
-
-.. code-block:: pycon
-
- >>> a = ['Apple', 'corn', 'Corn', 'Banana', 'apple', 'banana']
- >>> natsorted(a)
- ['Apple', 'Banana', 'Corn', 'apple', 'banana', 'corn']
-
-There are times when you wish to ignore the case when sorting,
-you can easily do this with the ``ns.IGNORECASE`` option:
-
-.. code-block:: pycon
-
- >>> natsorted(a, alg=ns.IGNORECASE)
- ['Apple', 'apple', 'Banana', 'banana', 'corn', 'Corn']
-
-Note thats since Python's sorting is stable, the order of equivalent
-elements after lowering the case is the same order they appear in the
-original list.
-
-Upper-case letters appear first in the ASCII table, but many natural
-sorting methods place lower-case first. To do this, use
-``ns.LOWERCASEFIRST``:
-
-.. code-block:: pycon
-
- >>> natsorted(a, alg=ns.LOWERCASEFIRST)
- ['apple', 'banana', 'corn', 'Apple', 'Banana', 'Corn']
-
-It may be undesirable to have the upper-case letters grouped together
-and the lower-case letters grouped together; most would expect all
-"a"s to bet together regardless of case, and all "b"s, and so on. To
-achieve this, use ``ns.GROUPLETTERS``:
-
-.. code-block:: pycon
-
- >>> natsorted(a, alg=ns.GROUPLETTERS)
- ['Apple', 'apple', 'Banana', 'banana', 'Corn', 'corn']
-
-You might combine this with ``ns.LOWERCASEFIRST`` to get what most
-would expect to be "natural" sorting:
-
-.. code-block:: pycon
-
- >>> natsorted(a, alg=ns.G | ns.LF)
- ['apple', 'Apple', 'banana', 'Banana', 'corn', 'Corn']
-
-Customizing Float Definition
-----------------------------
-
-You can make :func:`~natsorted` search for any float that would be
-a valid Python float literal, such as 5, 0.4, -4.78, +4.2E-34, etc.
-using the ``ns.FLOAT`` key. You can disable the exponential component
-of the number with ``ns.NOEXP``.
-
-.. code-block:: pycon
-
- >>> a = ['a50', 'a51.', 'a+50.4', 'a5.034e1', 'a+50.300']
- >>> natsorted(a, alg=ns.FLOAT)
- ['a50', 'a5.034e1', 'a51.', 'a+50.300', 'a+50.4']
- >>> natsorted(a, alg=ns.FLOAT | ns.SIGNED)
- ['a50', 'a+50.300', 'a5.034e1', 'a+50.4', 'a51.']
- >>> natsorted(a, alg=ns.FLOAT | ns.SIGNED | ns.NOEXP)
- ['a5.034e1', 'a50', 'a+50.300', 'a+50.4', 'a51.']
-
-For convenience, the ``ns.REAL`` option is provided which is a shortcut
-for ``ns.FLOAT | ns.SIGNED`` and can be used to sort on real numbers.
-This can be easily accessed with the :func:`~realsorted` convenience
-function. Please note that the behavior of the :func:`~realsorted` function
-was the default behavior of :func:`~natsorted` for :mod:`natsort`
-version < 4.0.0:
-
-.. code-block:: pycon
-
- >>> natsorted(a, alg=ns.REAL)
- ['a50', 'a+50.300', 'a5.034e1', 'a+50.4', 'a51.']
- >>> from natsort import realsorted
- >>> realsorted(a)
- ['a50', 'a+50.300', 'a5.034e1', 'a+50.4', 'a51.']
-
-.. _custom_sort:
-
-Using a Custom Sorting Key
---------------------------
-
-Like the built-in ``sorted`` function, ``natsorted`` can accept a custom
-sort key so that:
-
-.. code-block:: pycon
-
- >>> from operator import attrgetter, itemgetter
- >>> a = [['a', 'num4'], ['b', 'num8'], ['c', 'num2']]
- >>> natsorted(a, key=itemgetter(1))
- [['c', 'num2'], ['a', 'num4'], ['b', 'num8']]
- >>> class Foo:
- ... def __init__(self, bar):
- ... self.bar = bar
- ... def __repr__(self):
- ... return "Foo('{}')".format(self.bar)
- >>> b = [Foo('num3'), Foo('num5'), Foo('num2')]
- >>> natsorted(b, key=attrgetter('bar'))
- [Foo('num2'), Foo('num3'), Foo('num5')]
-
-.. _unit_sorting:
-
-Accounting for Units When Sorting
-+++++++++++++++++++++++++++++++++
-
-:mod:`natsort` does not come with any pre-built mechanism to sort units,
-but you can write your own `key` to do this. Below, I will demonstrate sorting
-imperial lengths (e.g. feet an inches), but of course you can extend this to any
-set of units you need. This example is based on code
-`from this issue <https://github.com/SethMMorton/natsort/issues/100#issuecomment-530659310>`_,
-and uses the function :func:`natsort.numeric_regex_chooser` to build a regular
-expression that will parse numbers in the same manner as :mod:`natsort` itself.
-
-.. code-block:: pycon
-
- >>> import re
- >>> import natsort
- >>>
- >>> # Define how each unit will be transformed
- >>> conversion_mapping = {
- ... "in": 1,
- ... "inch": 1,
- ... "inches": 1,
- ... "ft": 12,
- ... "feet": 12,
- ... "foot": 12,
- ... }
- >>>
- >>> # This regular expression searches for numbers and units
- >>> all_units = "|".join(conversion_mapping.keys())
- >>> float_re = natsort.numeric_regex_chooser(natsort.FLOAT | natsort.SIGNED)
- >>> unit_finder = re.compile(r"({})\s*({})".format(float_re, all_units), re.IGNORECASE)
- >>>
- >>> def unit_replacer(matchobj):
- ... """
- ... Given a regex match object, return a replacement string where units are modified
- ... """
- ... number = matchobj.group(1)
- ... unit = matchobj.group(2)
- ... new_number = float(number) * conversion_mapping[unit]
- ... return "{} in".format(new_number)
- ...
- >>> # Demo time!
- >>> data = ['1 ft', '5 in', '10 ft', '2 in']
- >>> [unit_finder.sub(unit_replacer, x) for x in data]
- ['12.0 in', '5.0 in', '120.0 in', '2.0 in']
- >>>
- >>> natsort.natsorted(data, key=lambda x: unit_finder.sub(unit_replacer, x))
- ['2 in', '5 in', '1 ft', '10 ft']
-
-Generating a Natsort Key
-------------------------
-
-If you need to sort a list in-place, you cannot use :func:`~natsorted`; you
-need to pass a key to the :meth:`list.sort` method. The function
-:func:`~natsort_keygen` is a convenient way to generate these keys for you:
-
-.. code-block:: pycon
-
- >>> from natsort import natsort_keygen
- >>> a = ['a50', 'a51.', 'a50.4', 'a5.034e1', 'a50.300']
- >>> natsort_key = natsort_keygen(alg=ns.FLOAT)
- >>> a.sort(key=natsort_key)
- >>> a
- ['a50', 'a50.300', 'a5.034e1', 'a50.4', 'a51.']
-
-:func:`~natsort_keygen` has the same API as :func:`~natsorted` (minus the
-`reverse` option).
-
-Sorting Multiple Lists According to a Single List
--------------------------------------------------
-
-Sometimes you have multiple lists, and you want to sort one of those
-lists and reorder the other lists according to how the first was sorted.
-To achieve this you could use the :func:`~index_natsorted` in combination
-with the convenience function
-:func:`~order_by_index`:
-
-.. code-block:: pycon
-
- >>> from natsort import index_natsorted, order_by_index
- >>> a = ['a2', 'a9', 'a1', 'a4', 'a10']
- >>> b = [4, 5, 6, 7, 8]
- >>> c = ['hi', 'lo', 'ah', 'do', 'up']
- >>> index = index_natsorted(a)
- >>> order_by_index(a, index)
- ['a1', 'a2', 'a4', 'a9', 'a10']
- >>> order_by_index(b, index)
- [6, 4, 7, 5, 8]
- >>> order_by_index(c, index)
- ['ah', 'hi', 'do', 'lo', 'up']
-
-Returning Results in Reverse Order
-----------------------------------
-
-Just like the :func:`sorted` built-in function, you can supply the
-``reverse`` option to return the results in reverse order:
-
-.. code-block:: pycon
-
- >>> a = ['a2', 'a9', 'a1', 'a4', 'a10']
- >>> natsorted(a, reverse=True)
- ['a10', 'a9', 'a4', 'a2', 'a1']
-
-Sorting Bytes on Python 3
--------------------------
-
-Python 3 is rather strict about comparing strings and bytes, and this
-can make it difficult to deal with collections of both. Because of the
-challenge of guessing which encoding should be used to decode a bytes
-array to a string, :mod:`natsort` does *not* try to guess and automatically
-convert for you; in fact, the official stance of :mod:`natsort` is to
-not support sorting bytes. Instead, some decoding convenience functions
-have been provided to you (see :ref:`bytes_help`) that allow you to
-provide a codec for decoding bytes through the ``key`` argument that
-will allow :mod:`natsort` to convert byte arrays to strings for sorting;
-these functions know not to raise an error if the input is not a byte
-array, so you can use the key on any arbitrary collection of data.
-
-.. code-block:: pycon
-
- >>> from natsort import as_ascii
- >>> a = [b'a', 14.0, 'b']
- >>> # On Python 2, natsorted(a) would would work as expected.
- >>> # On Python 3, natsorted(a) would raise a TypeError (bytes() < str())
- >>> natsorted(a, key=as_ascii) == [14.0, b'a', 'b']
- True
-
-Additionally, regular expressions cannot be run on byte arrays, making it
-so that :mod:`natsort` cannot parse them for numbers. As a result, if you
-run :mod:`natsort` on a list of bytes, you will get results that are like
-Python's default sorting behavior. Of course, you can use the decoding
-functions to solve this:
-
-.. code-block:: pycon
-
- >>> from natsort import as_utf8
- >>> a = [b'a56', b'a5', b'a6', b'a40']
- >>> natsorted(a) # doctest: +SKIP
- [b'a40', b'a5', b'a56', b'a6']
- >>> natsorted(a, key=as_utf8) == [b'a5', b'a6', b'a40', b'a56']
- True
-
-If you need a codec different from ASCII or UTF-8, you can use
-:func:`decoder` to generate a custom key:
-
-.. code-block:: pycon
-
- >>> from natsort import decoder
- >>> a = [b'a56', b'a5', b'a6', b'a40']
- >>> natsorted(a, key=decoder('latin1')) == [b'a5', b'a6', b'a40', b'a56']
- True
-
-Sorting a Pandas DataFrame
---------------------------
-
-Starting from Pandas version 1.1.0, the
-`sorting methods accept a "key" argument <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_values.html>`_,
-so you can simply pass :func:`natsort_keygen` to the sorting methods and sort:
-
-.. code-block:: python
-
- import pandas as pd
- from natsort import natsort_keygen
- s = pd.Series(['2 ft 7 in', '1 ft 5 in', '10 ft 2 in', '2 ft 11 in', '7 ft 6 in'])
- s.sort_values(key=natsort_keygen())
- # 1 1 ft 5 in
- # 0 2 ft 7 in
- # 3 2 ft 11 in
- # 4 7 ft 6 in
- # 2 10 ft 2 in
- # dtype: object
-
-Similarly, if you need to sort the index there is
-`sort_index <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.sort_index.html>`_
-of a DataFrame.
-
-If you are on an older version of Pandas, check out please check out
-`this answer on StackOverflow <https://stackoverflow.com/a/29582718/1399279>`_
-for ways to do this without the ``key`` argument to ``sort_values``.
+This page has been moved to the
+`natsort wiki <https://github.com/SethMMorton/natsort/wiki/How-Does-Natsort-Work%3F>`_.
diff --git a/docs/howitworks.rst b/docs/howitworks.rst
index c60bbeb..4617a1b 100644
--- a/docs/howitworks.rst
+++ b/docs/howitworks.rst
@@ -6,1118 +6,11 @@
How Does Natsort Work?
======================
-.. contents::
- :local:
-
-:mod:`natsort` works by breaking strings into smaller sub-components (numbers
-or everything else), and returning these components in a tuple. Sorting
-tuples in Python is well-defined, and this fact is used to sort the input
-strings properly. But how does one break a string into sub-components?
-And what does one do to those components once they are split? Below I
-will explain the algorithm that was chosen for the :mod:`natsort` module,
-and some of the thinking that went into those design decisions. I will
-also mention some of the stumbling blocks I ran into because
-`getting sorting right is surprisingly hard`_.
-
-If you are impatient, you can skip to :ref:`tldr1` for the algorithm
-in the simplest case, and :ref:`tldr2`
-to see what extra code is needed to handle special cases.
-
-First, How Does Natural Sorting Work At a High Level?
------------------------------------------------------
-
-If I want to compare '2 ft 7 in' to '2 ft 11 in', I might do the following
-
-.. code-block:: pycon
-
- >>> '2 ft 7 in' < '2 ft 11 in'
- False
-
-We as humans know that the above should be true, but why does Python think it
-is false? Here is how it is performing the comparison:
-
-::
-
- '2' <=> '2' ==> equal, so keep going
- ' ' <=> ' ' ==> equal, so keep going
- 'f' <=> 'f' ==> equal, so keep going
- 't' <=> 't' ==> equal, so keep going
- ' ' <=> ' ' ==> equal, so keep going
- '7' <=> '1' ==> different, use result of '7' < '1'
-
-'7' evaluates as greater than '1' so the statement is false. When sorting, if
-a value is less than another it is placed first, so in our above example
-'2 ft 11 in' would end up before '2 ft 7 in', which is not correct. What to do?
-
-The best way to handle this is to break the string into sub-components
-of numbers and non-numbers, and then convert the numeric parts into
-:func:`float` or :func:`int` types. This will force Python to
-actually understand the context of what it is sorting and then "do the
-right thing." Luckily, it handles sorting lists of strings right
-out-of-the-box, so the only hard part is actually making this string-to-list
-transformation and then Python will handle the rest.
-
-::
-
- '2 ft 7 in' ==> (2, ' ft ', 7, ' in')
- '2 ft 11 in' ==> (2, ' ft ', 11, ' in')
-
-When Python compares the two, it roughly follows the below logic:
-
-::
-
- 2 <=> 2 ==> equal, so keep going
- ' ft ' <=> ' ft ' ==> a string is a special type of sequence - evaluate each character individually
- ||
- -->
- ' ' <=> ' ' ==> equal, so keep going
- 'f' <=> 'f' ==> equal, so keep going
- 't' <=> 't' ==> equal, so keep going
- ' ' <=> ' ' ==> equal, so keep going
- <== Back to parent sequence
- 7 <=> 11 ==> different, use the result of 7 < 11
-
-Clearly, seven is less than eleven, so our comparison is as we expect, and we
-would get the sorting order we wanted.
-
-At its heart, :mod:`natsort` is simply a tool to break strings into tuples,
-turning numbers in strings (i.e. ``'79'``) into *ints* and *floats* as it does this.
-
-Natsort's Approach
-------------------
-
-.. contents::
- :local:
-
-Decomposing Strings Into Sub-Components
-+++++++++++++++++++++++++++++++++++++++
-
-The first major hurtle to overcome is to decompose the string into
-sub-components. Remarkably, this turns out to be the easy part, owing mostly
-to Python's easy access to regular expressions. Breaking an arbitrary string
-based on a pattern is pretty straightforward.
-
-.. code-block:: pycon
-
- >>> import re
- >>> re.split(r'(\d+)', '2 ft 11 in')
- ['', '2', ' ft ', '11', ' in']
-
-Clear (assuming you can read regular expressions) and concise.
-
-The reason I began developing :mod:`natsort` in the first place was because I
-needed to handle the natural sorting of strings containing *real numbers*, not
-just unsigned integers as the above example contains. By real numbers, I mean
-those like ``-45.4920E-23``. :mod:`natsort` can handle just about any number
-definition; to that end, here are all the regular expressions used in
-:mod:`natsort`:
-
-.. code-block:: pycon
-
- >>> unsigned_int = r'([0-9]+)'
- >>> signed_int = r'([-+]?[0-9]+)'
- >>> unsigned_float = r'((?:[0-9]+\.?[0-9]*|\.[0-9]+)(?:[eE][-+]?[0-9]+)?)'
- >>> signed_float = r'([-+]?(?:[0-9]+\.?[0-9]*|\.[0-9]+)(?:[eE][-+]?[0-9]+)?)'
- >>> unsigned_float_no_exponent = r'((?:[0-9]+\.?[0-9]*|\.[0-9]+))'
- >>> signed_float_no_exponent = r'([-+]?(?:[0-9]+\.?[0-9]*|\.[0-9]+))'
-
-Note that ``"inf"`` and ``"nan"`` are deliberately omitted from the float
-definition because you wouldn't want (for example) ``"banana"`` to be converted
-into ``['ba', 'nan', 'a']``, Let's see an example:
-
-.. code-block:: pycon
-
- >>> re.split(signed_float, 'The mass of 3 electrons is 2.732815068E-30 kg')
- ['The mass of ', '3', ' electrons is ', '2.732815068E-30', ' kg']
-
-.. note::
-
- It is a bit of a lie to say the above are the complete regular expressions. In the
- actual code there is also handling for non-ASCII unicode characters (such as ⑦),
- but I will ignore that aspect of :mod:`natsort` in this discussion.
-
-Now, when the user wants to change the definition of a number, it is as easy as
-changing the pattern supplied to the regular expression engine.
-
-Choosing the right default is hard, though (well, in this case it shouldn't
-have been but I was rather thick-headed). In retrospect, it should have been
-obvious that since essentially all the code examples I had/have seen for
-natural sorting were for *unsigned integers*, I should have made the default
-definition of a number an *unsigned integer*. But, in the brash days of my
-youth I assumed that since my use case was real numbers, everyone else would
-be happier sorting by real numbers; so, I made the default definition of a
-number a *signed float with exponent*. `This astonished`_ `a lot`_ `of people`_
-(`and some people aren't very nice when they are astonished`_).
-Starting with :mod:`natsort` version 4.0.0 the default number definition was
-changed to an *unsigned integer* which satisfies the "least astonishment"
-principle, and I have not heard a complaint since.
-
-Coercing Strings Containing Numbers Into Numbers
-++++++++++++++++++++++++++++++++++++++++++++++++
-
-There has been some debate on Stack Overflow as to what method is best to
-coerce a string to a number if it can be coerced, and leaving it alone otherwise
-(see `this one for coercion`_ and `this one for checking`_ for some high traffic questions),
-but it mostly boils down to two different solutions, shown here:
-
-.. code-block:: pycon
-
- >>> def coerce_try_except(x):
- ... try:
- ... return int(x)
- ... except ValueError:
- ... return x
- ...
- >>> def coerce_regex(x):
- ... # Note that precompiling the regex is more performant,
- ... # but I do not show that here for clarity's sake.
- ... return int(x) if re.match(r'[-+]?\d+$', x) else x
- ...
-
-Here are some timing results run on my machine:
-
-.. code-block:: pycon
-
- In [0]: numbers = list(map(str, range(100))) # A list of numbers as strings
-
- In [1]: not_numbers = ['banana' + x for x in numbers]
-
- In [2]: %timeit [coerce_try_except(x) for x in numbers]
- 10000 loops, best of 3: 51.1 µs per loop
-
- In [3]: %timeit [coerce_try_except(x) for x in not_numbers]
- 1000 loops, best of 3: 289 µs per loop
-
- In [4]: %timeit [coerce_regex(x) for x in not_numbers]
- 10000 loops, best of 3: 67.6 µs per loop
-
- In [5]: %timeit [coerce_regex(x) for x in numbers]
- 10000 loops, best of 3: 123 µs per loop
-
-What can we learn from this? The ``try: except`` method (arguably the most
-"pythonic" of the solutions) is best for numeric input, but performs over 5X
-slower for non-numeric input. Conversely, the regular expression method, though
-slower than ``try: except`` for both input types, is more efficient for
-non-numeric input than for input that can be converted to an ``int``. Further,
-even though the regular expression method is slower for both input types, it is
-always at least twice as fast as the worst case for the ``try: except``.
-
-Why do I care? Shouldn't I just pick a method and not worry about it? Probably.
-However, I am very conscious about the performance of :mod:`natsort`, and want
-it to be a true drop-in replacement for :func:`sorted` without having to incur
-a performance penalty. For the purposes of :mod:`natsort`, there is no clear
-winner between the two algorithms - the data being passed to this function will
-likely be a mix of numeric and non-numeric string content. Do I use the
-``try: except`` method and hope the speed gains on numbers will offset the
-non-number performance, or do I use regular expressions and take the more
-stable performance?
-
-It turns out that within the context of :mod:`natsort`, some assumptions can be
-made that make a hybrid approach attractive. Because all strings are pre-split
-into numeric and non-numeric content *before* being passed to this coercion
-function, the assumption can be made that *if a string begins with a digit or a
-sign, it can be coerced into a number*.
-
-.. code-block:: pycon
-
- >>> def coerce_to_int(x):
- ... if x[0] in '0123456789+-':
- ... try:
- ... return int(x)
- ... except ValueError:
- ... return x
- ... else:
- ... return x
- ...
-
-So how does this perform compared to the standard coercion methods?
-
-.. code-block:: pycon
-
- In [6]: %timeit [coerce_to_int(x) for x in numbers]
- 10000 loops, best of 3: 71.6 µs per loop
-
- In [7]: %timeit [coerce_to_int(x) for x in not_numbers]
- 10000 loops, best of 3: 26.4 µs per loop
-
-The hybrid method eliminates most of the time wasted on numbers checking
-that it is in fact a number before passing to :func:`int`, and eliminates
-the time wasted in the exception stack for input that is not a number.
-
-That's as fast as we can get, right? In pure Python, probably. At least, it's
-close. But because I am crazy and a glutton for punishment, I decided to see
-if I could get any faster writing a C extension. It's called
-`fastnumbers`_ and contains a C implementation of the above coercion functions
-called :func:`fast_int`. How does it fair? Pretty well.
-
-.. code-block:: pycon
-
- In [8]: %timeit [fast_int(x) for x in numbers]
- 10000 loops, best of 3: 30.9 µs per loop
-
- In [9]: %timeit [fast_int(x) for x in not_numbers]
- 10000 loops, best of 3: 30 µs per loop
-
-During development of :mod:`natsort`, I wanted to ensure that using it did not
-get in the way of a user's program by introducing a performance penalty to
-their code. To that end, I do not feel like my adventures down the rabbit hole
-of optimization of coercion functions was a waste; I can confidently look users
-in the eye and say I considered every option in ensuring :mod:`natsort` is as
-efficient as possible. This is why if `fastnumbers`_ is installed it will be
-used for this step, and otherwise the hybrid method will be used.
-
-.. note::
-
- Modifying the hybrid coercion function for floats is straightforward.
-
- .. code-block:: pycon
-
- >>> def coerce_to_float(x):
- ... if x[0] in '.0123456789+-' or x.lower().lstrip()[:3] in ('nan', 'inf'):
- ... try:
- ... return float(x)
- ... except ValueError:
- ... return x
- ... else:
- ... return x
- ...
-
-.. _tldr1:
-
-TL;DR 1 - The Simple "No Special Cases" Algorithm
-+++++++++++++++++++++++++++++++++++++++++++++++++
-
-At this point, our :mod:`natsort` algorithm is essentially the following:
-
-.. code-block:: pycon
-
- >>> import re
- >>> def natsort_key(x, as_float=False, signed=False):
- ... if as_float:
- ... regex = signed_float if signed else unsigned_float
- ... else:
- ... regex = signed_int if signed else unsigned_int
- ... split_input = re.split(regex, x)
- ... split_input = filter(None, split_input) # removes null strings
- ... coerce = coerce_to_float if as_float else coerce_to_int
- ... return tuple(coerce(s) for s in split_input)
- ...
-
-I have written the above for clarity and not performance.
-This pretty much matches `most natural sort solutions for python on Stack Overflow`_
-(except the above includes customization of the definition of a number).
+This page has been moved to the
+`natsort wiki <https://github.com/SethMMorton/natsort/wiki/How-Does-Natsort-Work%3F>`_.
Special Cases Everywhere!
-------------------------
-.. contents::
- :local:
-
-.. image:: special_cases_everywhere.jpg
-
-If what I described in :ref:`TL;DR 1 <tldr1>` were
-all that :mod:`natsort` needed to
-do then there probably wouldn't be much need for a third-party module, right?
-Probably. But it turns out that in real-world data there are a lot of
-special cases that need to be handled, and in true `80%/20%`_ fashion, the
-majority of the code in :mod:`natsort` is devoted to handling special cases
-like those described below.
-
-Sorting Filesystem Paths
-++++++++++++++++++++++++
-
-`The first major special case I encountered was sorting filesystem paths`_
-(if you go to the link, you will see I didn't handle it well for a year...
-this was before I fully realized how much functionality I could really add
-to :mod:`natsort`). Let's apply the :func:`natsort_key` from above to some
-filesystem paths that you might see being auto-generated from your operating
-system:
-
-.. code-block:: pycon
-
- >>> paths = ['Folder (10)/file.tar.gz',
- ... 'Folder/file.tar.gz',
- ... 'Folder (1)/file (1).tar.gz',
- ... 'Folder (1)/file.tar.gz']
- >>> sorted(paths, key=natsort_key)
- ['Folder (1)/file (1).tar.gz', 'Folder (1)/file.tar.gz', 'Folder (10)/file.tar.gz', 'Folder/file.tar.gz']
-
-Well that's not right! What is ``'Folder/file.tar.gz'`` doing at the end?
-It has to do with the numerical ASCII code assigned to the space and
-``/`` characters in the `ASCII table`_. According to the `ASCII table`_, the
-space character (number 32) comes before the ``/`` character (number 47). If
-we remove the common prefix in all of the above strings (``'Folder'``), we
-can see why this happens:
-
-.. code-block:: pycon
-
- >>> ' (1)/file.tar.gz' < '/file.tar.gz'
- True
- >>> ' ' < '/'
- True
-
-This isn't very convenient... how do we solve it? We can split the path
-across the path separators and then sort. A convenient way do to this is
-with the :data:`Path.parts <pathlib.PurePath.parts>` property from
-:mod:`pathlib`:
-
-.. code-block:: pycon
-
- >>> import pathlib
- >>> sorted(paths, key=lambda x: tuple(natsort_key(s) for s in pathlib.Path(x).parts))
- ['Folder/file.tar.gz', 'Folder (1)/file (1).tar.gz', 'Folder (1)/file.tar.gz', 'Folder (10)/file.tar.gz']
-
-Almost! It seems like there is some funny business going on in the final
-filename component as well. We can solve that nicely and quickly with
-:data:`Path.suffixes <pathlib.PurePath.suffixes>` and :data:`Path.stem
-<pathlib.PurePath.stem>`.
-
-.. code-block:: pycon
-
- >>> def decompose_path_into_components(x):
- ... path_split = list(pathlib.Path(x).parts)
- ... # Remove the final filename component from the path.
- ... final_component = pathlib.Path(path_split.pop())
- ... # Split off all the extensions.
- ... suffixes = final_component.suffixes
- ... stem = final_component.name.replace(''.join(suffixes), '')
- ... # Remove the '.' prefix of each extension, and make that
- ... # final component a list of the stem and each suffix.
- ... final_component = [stem] + [x[1:] for x in suffixes]
- ... # Replace the split final filename component.
- ... path_split.extend(final_component)
- ... return path_split
- ...
- >>> def natsort_key_with_path_support(x):
- ... return tuple(natsort_key(s) for s in decompose_path_into_components(x))
- ...
- >>> sorted(paths, key=natsort_key_with_path_support)
- ['Folder/file.tar.gz', 'Folder (1)/file.tar.gz', 'Folder (1)/file (1).tar.gz', 'Folder (10)/file.tar.gz']
-
-This works because in addition to breaking the input by path separators,
-the final filename component is separated from its extensions as well.
-*Then*, each of these separated components is sent to the
-:mod:`natsort` algorithm, so the result is a tuple of tuples. Once that
-is done, we can see how comparisons can be done in the expected manner.
-
-.. code-block:: pycon
-
- >>> a = natsort_key_with_path_support('Folder (1)/file (1).tar.gz')
- >>> a
- (('Folder (', 1, ')'), ('file (', 1, ')'), ('tar',), ('gz',))
- >>>
- >>> b = natsort_key_with_path_support('Folder/file.tar.gz')
- >>> b
- (('Folder',), ('file',), ('tar',), ('gz',))
- >>>
- >>> a > b
- True
-
-Comparing Different Types on Python 3
-+++++++++++++++++++++++++++++++++++++
-
-`The second major special case I encountered was sorting of different types`_.
-If you are on Python 2 (i.e. legacy Python), this mostly doesn't matter *too*
-much since it uses an arbitrary heuristic to allow traditionally un-comparable
-types to be compared (such as comparing ``'a'`` to ``1``). However, on Python 3
-(i.e. Python) it simply won't let you perform such nonsense, raising a
-:exc:`TypeError` instead.
-
-You can imagine that a module that breaks strings into tuples of numbers and
-strings is walking a dangerous line if it does not have special handling for
-comparing numbers and strings. My imagination was not so great at first.
-Let's take a look at all the ways this can fail with real-world data.
-
-.. code-block:: pycon
-
- >>> def natsort_key_with_poor_real_number_support(x):
- ... split_input = re.split(signed_float, x)
- ... split_input = filter(None, split_input) # removes null strings
- ... return tuple(coerce_to_float(s) for s in split_input)
- >>>
- >>> sorted([5, '4'], key=natsort_key_with_poor_real_number_support)
- Traceback (most recent call last):
- ...
- TypeError: ...
- >>>
- >>> sorted(['12 apples', 'apples'], key=natsort_key_with_poor_real_number_support)
- Traceback (most recent call last):
- ...
- TypeError: ...
- >>>
- >>> sorted(['version5.3.0', 'version5.3rc1'], key=natsort_key_with_poor_real_number_support)
- Traceback (most recent call last):
- ...
- TypeError: ...
-
-Let's break these down.
-
-#. The integer ``5`` is sent to ``re.split`` which expects only strings
- or bytes, which is a no-no.
-#. ``natsort_key_with_poor_real_number_support('12 apples') < natsort_key_with_poor_real_number_support('apples')``
- is the same as ``(12.0, ' apples') < ('apples',)``, and thus a number gets
- compared to a string [#f1]_ which also is a no-no.
-#. This one scores big on the astonishment scale, especially if one
- accidentally uses signed integers or real numbers when they mean
- to use unsigned integers.
- ``natsort_key_with_poor_real_number_support('version5.3.0') < natsort_key_with_poor_real_number_support('version5.3rc1')``
- is the same as ``('version', 5.3, 0.0) < ('version', 5.3, 'rc', 1.0)``,
- so in the third element a number gets compared to a string, once again
- the same old no-no. (The same would happen with ``'version5-3'`` and
- ``'version5-a'``, which would become ``('version', 5, -3)`` and
- ``('version', 5, '-a')``).
-
-As you might expect, the solution to the first issue is to wrap the
-``re.split`` call in a ``try: except:`` block and handle the number specially
-if a :exc:`TypeError` is raised. The second and third cases *could* be handled
-in a "special case" manner, meaning only respond and do something different
-if these problems are detected. But a less error-prone method is to ensure
-that the data is correct-by-construction, and this can be done by ensuring
-that the returned tuples *always* start with a string, and then alternate
-in a string-number-string-number-string pattern; this can be achieved by
-adding an empty string wherever the pattern is not followed [#f2]_. This ends
-up working out pretty nicely because empty strings are always "less" than
-any non-empty string, and we typically want numbers to come before strings.
-
-Let's take a look at how this works out.
-
-.. code-block:: pycon
-
- >>> from natsort.utils import sep_inserter
- >>> list(sep_inserter(iter(['apples']), ''))
- ['apples']
- >>>
- >>> list(sep_inserter(iter([12, ' apples']), ''))
- ['', 12, ' apples']
- >>>
- >>> list(sep_inserter(iter(['version', 5, -3]), ''))
- ['version', 5, '', -3]
- >>>
- >>> from natsort import natsort_keygen, ns
- >>> natsort_key_with_good_real_number_support = natsort_keygen(alg=ns.REAL)
- >>>
- >>> sorted([5, '4'], key=natsort_key_with_good_real_number_support)
- ['4', 5]
- >>>
- >>> sorted(['12 apples', 'apples'], key=natsort_key_with_good_real_number_support)
- ['12 apples', 'apples']
- >>>
- >>> sorted(['version5.3.0', 'version5.3rc1'], key=natsort_key_with_good_real_number_support)
- ['version5.3.0', 'version5.3rc1']
-
-How the "good" version works will be given in
-`TL;DR 2 - Handling Crappy, Real-World Input`_.
-
-Handling NaN
-++++++++++++
-
-`A rather unexpected special case I encountered was sorting collections containing NaN`_.
-Let's see what happens when you try to sort a plain old list of numbers when there
-is a **NaN** floating around in there.
-
-.. code-block:: pycon
-
- >>> danger = [7, float('nan'), 22.7, 19, -14, 59.123, 4]
- >>> sorted(danger)
- [7, nan, -14, 4, 19, 22.7, 59.123]
-
-Clearly that isn't correct, and for once it isn't my fault!
-`It's hard to compare floating point numbers`_. By definition, **NaN** is unorderable
-to any other number, and is never equal to any other number, including itself.
-
-.. code-block:: pycon
-
- >>> nan = float('nan')
- >>> 5 > nan
- False
- >>> 5 < nan
- False
- >>> 5 == nan
- False
- >>> 5 != nan
- True
- >>> nan == nan
- False
- >>> nan != nan
- True
-
-The implication of all this for us is that if there is an **NaN** in the
-data-set we are trying to sort, the data-set will end up being sorted in
-two separate yet individually sorted sequences - the one *before* the **NaN**,
-and the one *after*. This is because the ``<`` operation that is used
-to sort always returns :const:`False` with **NaN**.
-
-Because :mod:`natsort` aims to sort sequences in a way that does not surprise
-the user, keeping this behavior is not acceptable (I don't require my users
-to know how **NaN** will behave in a sorting algorithm). The simplest way to
-satisfy the "least astonishment" principle is to substitute **NaN** with
-some other value. But what value is *least* astonishing? I chose to replace
-**NaN** with :math:`-\infty` so that these poorly behaved elements always
-end up at the front where the users will most likely be alerted to their
-presence.
-
-.. code-block:: pycon
-
- >>> def fix_nan(x):
- ... if x != x: # only true for NaN
- ... return float('-inf')
- ... else:
- ... return x
- ...
-
-Let's check out :ref:`TL;DR 2 <tldr2>` to see how this can be
-incorporated into the simple key function from :ref:`TL;DR 1 <tldr1>`.
-
-.. _tldr2:
-
-TL;DR 2 - Handling Crappy, Real-World Input
-+++++++++++++++++++++++++++++++++++++++++++
-
-Let's see how our elegant key function from :ref:`TL;DR 1 <tldr1>` has
-become bastardized in order to support handling mixed real-world data
-and user customizations.
-
-.. code-block:: pycon
-
- >>> def natsort_key(x, as_float=False, signed=False, as_path=False):
- ... if as_float:
- ... regex = signed_float if signed else unsigned_float
- ... else:
- ... regex = signed_int if signed else unsigned_int
- ... try:
- ... if as_path:
- ... x = decompose_path_into_components(x) # Decomposes into list of strings
- ... # If this raises a TypeError, input is not a string.
- ... split_input = re.split(regex, x)
- ... except TypeError:
- ... try:
- ... # Does this need to be applied recursively (list-of-list)?
- ... return tuple(map(natsort_key, x))
- ... except TypeError:
- ... # Must be a number
- ... ret = ('', fix_nan(x)) # Maintain string-number-string pattern
- ... return (ret,) if as_path else ret # as_path returns tuple-of-tuples
- ... else:
- ... split_input = filter(None, split_input) # removes null strings
- ... # Note that the coerce_to_int/coerce_to_float functions
- ... # are also modified to use the fix_nan function.
- ... if as_float:
- ... coerced_input = (coerce_to_float(s) for s in split_input)
- ... else:
- ... coerced_input = (coerce_to_int(s) for s in split_input)
- ... return tuple(sep_inserter(coerced_input, ''))
- ...
-
-And this doesn't even show handling :class:`bytes` type! Notice that we have
-to do non-obvious things like modify the return form of numbers when ``as_path``
-is given, just to avoid comparing strings and numbers for the case in which a
-user provides input like ``['/home/me', 42]``.
-
-Let's take it out for a spin!
-
-.. code-block:: pycon
-
- >>> danger = [7, float('nan'), 22.7, '19', '-14', '59.123', 4]
- >>> sorted(danger, key=lambda x: natsort_key(x, as_float=True, signed=True))
- [nan, '-14', 4, 7, '19', 22.7, '59.123']
- >>>
- >>> paths = ['Folder (1)/file.tar.gz',
- ... 'Folder/file.tar.gz',
- ... 123456]
- >>> sorted(paths, key=lambda x: natsort_key(x, as_path=True))
- [123456, 'Folder/file.tar.gz', 'Folder (1)/file.tar.gz']
-
-Here Be Dragons: Adding Locale Support
---------------------------------------
-
-.. contents::
- :local:
-
-Probably the most challenging special case I had to handle was getting
-:mod:`natsort` to handle sorting the non-numerical parts of input
-correctly, and also allowing it to sort the numerical bits in different
-locales. This was in no way what I originally set out to do with this
-library, so I was
-`caught a bit off guard when the request was initially made`_.
-I discovered the :mod:`locale` library, and assumed that if it's part of
-Python's StdLib there can't be too many dragons, right?
-
-.. admonition:: INCOMPLETE LIST OF DRAGONS
-
- - https://github.com/SethMMorton/natsort/issues/21
- - https://github.com/SethMMorton/natsort/issues/22
- - https://github.com/SethMMorton/natsort/issues/23
- - https://github.com/SethMMorton/natsort/issues/36
- - https://github.com/SethMMorton/natsort/issues/44
- - https://bugs.python.org/issue2481
- - https://bugs.python.org/issue23195
- - https://stackoverflow.com/questions/3412933/python-not-sorting-unicode-properly-strcoll-doesnt-help
- - https://stackoverflow.com/questions/22203550/sort-dictionary-by-key-using-locale-collation
- - https://stackoverflow.com/questions/33459384/unicode-character-not-in-range-when-calling-locale-strxfrm
- - https://stackoverflow.com/questions/36431810/sort-numeric-lines-with-thousand-separators
- - https://stackoverflow.com/questions/45734562/how-can-i-get-a-reasonable-string-sorting-with-python
-
-These can be summed up as follows:
-
-#. :mod:`locale` is a thin wrapper over your operating system's *locale*
- library, so if *that* is broken (like it is on BSD and OSX) then
- :mod:`locale` is broken in Python.
-#. Because of a bug in legacy Python (i.e. Python 2), there is no uniform
- way to use the :mod:`locale` sorting functionality between legacy Python
- and Python 3.
-#. People have differing opinions of how capitalization should affect word
- order.
-#. There is no built-in way to handle locale-dependent thousands separators
- and decimal points *robustly*.
-#. Proper handling of Unicode is complicated.
-#. Proper handling of :mod:`locale` is complicated.
-
-Easily over half of the code in :mod:`natsort` is in some way dealing with some
-aspect of :mod:`locale` or basic case handling. It would have been impossible
-to get right without a `really good`_ `testing strategy`_.
-
-Don't expect any more TL;DR's... if you want to see how all this is fully
-incorporated into the :mod:`natsort` algorithm then please take a look
-`at the code`_. However, I will hint at how specific steps are taken in
-each section.
-
-Let's see how we can handle some of the dragons, one-by-one.
-
-Basic Case Control Support
-++++++++++++++++++++++++++
-
-Without even thinking about the mess that is adding :mod:`locale` support,
-:mod:`natsort` can introduce support for controlling how case is interpreted.
-
-First, let's take a look at how it is sorted by default (due to
-where characters lie on the `ASCII table`_).
-
-.. code-block:: pycon
-
- >>> a = ['Apple', 'corn', 'Corn', 'Banana', 'apple', 'banana']
- >>> sorted(a)
- ['Apple', 'Banana', 'Corn', 'apple', 'banana', 'corn']
-
-All uppercase letters come before lowercase letters in the `ASCII table`_,
-so all capitalized words appear first. Not everyone agrees that this
-is the correct order. Some believe that the capitalized words should
-be last (``['apple', 'banana', 'corn', 'Apple', 'Banana', 'Corn']``).
-Some believe that both the lowercase and uppercase versions
-should appear together
-(``['Apple', 'apple', 'Banana', 'banana', 'Corn', 'corn']``).
-Some believe that both should be true ☹. Some people don't care at all [#f3]_.
-
-Solving the first case (I call it *LOWERCASEFIRST*) is actually pretty
-easy... just call the :meth:`str.swapcase` method on the input.
-
-.. code-block:: pycon
-
- >>> sorted(a, key=lambda x: x.swapcase())
- ['apple', 'banana', 'corn', 'Apple', 'Banana', 'Corn']
-
-The last (i call it *IGNORECASE*) should be super easy, right?
-Simply call :meth:`str.lowercase` on the input. This will work but may
-not always give the correct answer on non-latin character sets. It's
-a good thing that in Python 3.3
-:meth:`str.casefold` was introduced, which does a better job of removing
-all case information from unicode characters in
-non-latin alphabets.
-
-.. code-block:: pycon
-
- >>> def remove_case(x):
- ... try:
- ... return x.casefold()
- ... except AttributeError: # Legacy Python backwards compatibility
- ... return x.lowercase()
- ...
- >>> sorted(a, key=remove_case)
- ['Apple', 'apple', 'Banana', 'banana', 'corn', 'Corn']
-
-The middle case (I call it *GROUPLETTERS*) is less straightforward.
-The most efficient way to handle this is to duplicate each character
-with its lowercase version and then the original character.
-
-.. code-block:: pycon
-
- >>> import itertools
- >>> def groupletters(x):
- ... return ''.join(itertools.chain.from_iterable((remove_case(y), y) for y in x))
- ...
- >>> groupletters('Apple')
- 'aAppppllee'
- >>> groupletters('apple')
- 'aappppllee'
- >>> sorted(a, key=groupletters)
- ['Apple', 'apple', 'Banana', 'banana', 'Corn', 'corn']
-
-The effect of this is that both ``'Apple'`` and ``'apple'`` are
-placed adjacent to each other because their transformations both begin
-with ``'a'``, and then the second character can be used to order them
-appropriately with respect to each other.
-
-There's a problem with this, though. Within the context of :mod:`natsort`
-we are trying to correctly sort numbers and those should be left alone.
-
-.. code-block:: pycon
-
- >>> a = ['Apple5', 'apple', 'Apple4E10', 'Banana']
- >>> sorted(a, key=lambda x: natsort_key(x, as_float=True))
- ['Apple5', 'Apple4E10', 'Banana', 'apple']
- >>> sorted(a, key=lambda x: natsort_key(groupletters(x), as_float=True))
- ['Apple4E10', 'Apple5', 'apple', 'Banana']
- >>> groupletters('Apple4E10')
- 'aAppppllee44eE1100'
-
-We messed up the numbers! Looks like :func:`groupletters` needs to be applied
-*after* the strings are broken into their components. I'm not going to show
-how this is done here, but basically it requires applying the function in
-the ``else:`` block of :func:`coerce_to_int`/:func:`coerce_to_float`.
-
-.. code-block:: pycon
-
- >>> better_groupletters = natsort_keygen(alg=ns.GROUPLETTERS | ns.REAL)
- >>> better_groupletters('Apple4E10')
- ('aAppppllee', 40000000000.0)
- >>> sorted(a, key=better_groupletters)
- ['Apple5', 'Apple4E10', 'apple', 'Banana']
-
-Of course, applying both *LOWERCASEFIRST* and *GROUPLETTERS* is just
-a matter of turning on both functions.
-
-Basic Unicode Support
-+++++++++++++++++++++
-
-Unicode is hard and complicated. Here's an example.
-
-.. code-block:: pycon
-
- >>> b = [b'\x66', b'\x65', b'\xc3\xa9', b'\x65\xcc\x81', b'\x61', b'\x7a']
- >>> a = [x.decode('utf8') for x in b]
- >>> a # doctest: +SKIP
- ['f', 'e', 'é', 'é', 'a', 'z']
- >>> sorted(a) # doctest: +SKIP
- ['a', 'e', 'é', 'f', 'z', 'é']
-
-There are more than one way to represent the character 'é' in Unicode.
-In fact, many characters have multiple representations. This is a challenge
-because comparing the two representations would return ``False`` even though
-they *look* the same.
-
-.. code-block:: pycon
-
- >>> a[2] == a[3]
- False
-
-Alas, since characters are compared based on the numerical value of their
-representation, sorting Unicode often gives unexpected results (like seeing
-'é' come both *before* and *after* 'z').
-
-The original approach that :mod:`natsort` took with respect to non-ASCII
-Unicode characters was to say "just use
-the :mod:`locale` or :mod:`PyICU` library" and then cross it's fingers
-and hope those libraries take care of it. As you will find in the following
-sections, that comes with its own baggage, and turned out to not always work
-anyway (see https://stackoverflow.com/q/45734562/1399279). A more robust
-approach is to handle the Unicode out-of-the-box without invoking a
-heavy-handed library like :mod:`locale` or :mod:`PyICU`.
-To do this, we must use *normalization*.
-
-To fully understand Unicode normalization,
-`check out some official Unicode documentation`_.
-Just kidding... that's too much text. The following StackOverflow answers do
-a good job at explaining Unicode normalization in simple terms:
-https://stackoverflow.com/a/7934397/1399279 and
-https://stackoverflow.com/a/7931547/1399279. Put simply, normalization
-ensures that Unicode characters with multiple representations are in
-some canonical and consistent representation so that (for example) comparisons
-of the characters can be performed in a sane way. The following discussion
-assumes you at least read the StackOverflow answers.
-
-Looking back at our 'é' example, we can see that the two versions were
-constructed with the byte strings ``b'\xc3\xa9'`` and ``b'\x65\xcc\x81'``.
-The former representation is actually
-`LATIN SMALL LETTER E WITH ACUTE <https://www.fileformat.info/info/unicode/char/e9/index.htm>`_
-and is a single character in the Unicode standard. This is known as the
-*compressed form* and corresponds to the 'NFC' normalization scheme.
-The latter representation is actually the letter 'e' followed by
-`COMBINING ACUTE ACCENT <https://www.fileformat.info/info/unicode/char/0301/index.htm>`_
-and so is two characters in the Unicode standard. This is known as the
-*decompressed form* and corresponds to the 'NFD' normalization scheme.
-Since the first character in the decompressed form is actually the letter 'e',
-when compared to other ASCII characters it fits where you might expect.
-Unfortunately, all Unicode compressed form characters come after the
-ASCII characters and so they always will be placed after 'z' when sorting.
-
-It seems that most Unicode data is stored and shared in the compressed form
-which makes it challenging to sort. This can be solved by normalizing all
-incoming Unicode data to the decompressed form ('NFD') and *then* sorting.
-
-.. code-block:: pycon
-
- >>> import unicodedata
- >>> c = [unicodedata.normalize('NFD', x) for x in a]
- >>> c # doctest: +SKIP
- ['f', 'e', 'é', 'é', 'a', 'z']
- >>> sorted(c) # doctest: +SKIP
- ['a', 'e', 'é', 'é', 'f', 'z']
-
-Huzzah! Sane sorting without having to resort to :mod:`locale`!
-
-Using Locale to Compare Strings
-+++++++++++++++++++++++++++++++
-
-The :mod:`locale` module is actually pretty cool, and provides lowly
-spare-time programmers like myself a way to handle the daunting task
-of proper locale-dependent support of their libraries and utilities.
-Having said that, it can be a bit of a bear to get right,
-`although they do point out in the documentation that it will be painful to use`_.
-Aside from the caveats spelled out in that link, it turns out that just
-comparing strings with :mod:`locale` in a cross-platform and
-cross-python-version manner is not as straightforward as one might hope.
-
-First, how to use :mod:`locale` to compare strings? It's actually
-pretty straightforward. Simply run the input through the :mod:`locale`
-transformation function :func:`locale.strxfrm`.
-
-.. code-block:: pycon
-
- >>> import locale, sys
- >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
- 'en_US.UTF-8'
- >>> a = ['a', 'b', 'ä']
- >>> sorted(a)
- ['a', 'b', 'ä']
- >>> # The below fails on OSX, so don't run doctest on darwin.
- >>> is_osx = sys.platform == 'darwin'
- >>> sorted(a, key=locale.strxfrm) if not is_osx else ['a', 'ä', 'b']
- ['a', 'ä', 'b']
- >>>
- >>> a = ['apple', 'Banana', 'banana', 'Apple']
- >>> sorted(a, key=locale.strxfrm) if not is_osx else ['apple', 'Apple', 'banana', 'Banana']
- ['apple', 'Apple', 'banana', 'Banana']
-
-It turns out that locale-aware sorting groups numbers in the same
-way as turning on *GROUPLETTERS* and *LOWERCASEFIRST*.
-The trick is that you have to apply :func:`locale.strxfrm` only to non-numeric
-characters; otherwise, numbers won't be parsed properly. Therefore, it must
-be applied as part of the :func:`coerce_to_int`/:func:`coerce_to_float`
-functions in a manner similar to :func:`groupletters`.
-
-As you might have guessed, there is a small problem.
-It turns out the there is a bug in the legacy Python implementation of
-:func:`locale.strxfrm` that causes it to outright fail for :func:`unicode`
-input (https://bugs.python.org/issue2481). :func:`locale.strcoll` works,
-but is intended for use with ``cmp``, which does not exist in current Python
-implementations. Luckily, the :func:`functools.cmp_to_key` function
-makes :func:`locale.strcoll` behave like :func:`locale.strxfrm`.
-
-Handling Broken Locale On OSX
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-But what if the underlying *locale* implementation that :mod:`locale`
-relies upon is simply broken? It turns out that the *locale* library on
-OSX (and other BSD systems) is broken (and for some reason has never been
-fixed?), and so :mod:`locale` does not work as expected.
-
-How do I define doesn't work as expected?
-
-.. code-block:: pycon
-
- >>> a = ['apple', 'Banana', 'banana', 'Apple']
- >>> sorted(a)
- ['Apple', 'Banana', 'apple', 'banana']
- >>>
- >>> sorted(a, key=locale.strxfrm) if is_osx else sorted(a)
- ['Apple', 'Banana', 'apple', 'banana']
-
-IT'S SORTING AS IF :func:`locale.stfxfrm` WAS NEVER USED!! (and it's worse
-once non-ASCII characters get thrown into the mix.) I'm really not
-sure why this is considered OK for the OSX/BSD maintainers to not fix,
-but it's more than frustrating for poor developers who have been dragged
-into the *locale* game kicking and screaming. *<deep breath>*.
-
-So, how to deal with this situation? There are two ways to do so.
-
-#. Detect if :mod:`locale` is sorting incorrectly (i.e. ``dumb``) by seeing
- if ``'A'`` is sorted before ``'a'`` (incorrect) or not.
-
- .. code-block:: pycon
-
- >>> # This is genuinely the name of this function.
- >>> # See natsort.compat.locale.py
- >>> def dumb_sort():
- ... return locale.strxfrm('A') < locale.strxfrm('a')
- ...
-
- If a ``dumb`` *locale* implementation is found, then automatically
- turn on *LOWERCASEFIRST* and *GROUPLETTERS*.
-#. Use an alternate library if installed. `ICU <http://site.icu-project.org/>`_
- is a great and powerful library that has a pretty decent Python port
- called (you guessed it) `PyICU <https://pypi.org/project/PyICU/>`_.
- If a user has this library installed on their computer, :mod:`natsort`
- chooses to use that instead of :mod:`locale`. With a little bit of
- planning, one can write a set of wrapper functions that call
- the correct library under the hood such that the business logic never
- has to know what library is being used (see `natsort.compat.locale.py`_).
-
-Let me tell you, this little complication really makes a challenge of testing
-the code, since one must set up different environments on different operating
-systems in order to test all possible code paths. Not to mention that
-certain checks *will* fail for certain operating systems and environments
-so one must be diligent in either writing the tests not to fail, or ignoring
-those tests when on offending environments.
-
-Handling Locale-Aware Numbers
-+++++++++++++++++++++++++++++
-
-`Thousands separator support`_ is a problem that I knew would someday be
-requested but had decided to push off until a rainy day. One day it finally
-rained, and I decided to tackle the problem.
-
-So what is the problem? Consider the number ``1,234,567`` (assuming the
-``','`` is the thousands separator). Try to run that through :func:`int`
-and you will get a :exc:`ValueError`. To handle this properly the thousands
-separators must be removed.
-
-.. code-block:: pycon
-
- >>> float('1,234,567'.replace(',', ''))
- 1234567.0
-
-What if, in our current locale, the thousands separator is ``'.'`` and
-the ``','`` is the decimal separator (like for the German locale *de_DE*)?
-
-.. code-block:: pycon
-
- >>> float('1.234.567'.replace('.', '').replace(',', '.'))
- 1234567.0
- >>> float('1.234.567,89'.replace('.', '').replace(',', '.'))
- 1234567.89
-
-This is pretty much what :func:`locale.atoi` and :func:`locale.atof` do
-under the hood. So what's the problem? Why doesn't :mod:`natsort` just
-use this method under its hood?
-Well, let's take a look at what would happen if we send some possible
-:mod:`natsort` input through our the above function:
-
-.. code-block:: pycon
-
- >>> natsort_key('1,234 apples, please.'.replace(',', ''))
- ('', 1234, ' apples please.')
- >>> natsort_key('Sir, €1.234,50 please.'.replace('.', '').replace(',', '.'), as_float=True)
- ('Sir. €', 1234.5, ' please')
-
-Any character matching the thousands separator was dropped, and anything
-matching the decimal separator was changed to ``'.'``! If these characters
-were critical to how your data was ordered, this would break :mod:`natsort`.
-
-The first solution one might consider would be to first decompose the
-input into sub-components (like we did for the *GROUPLETTERS* method
-above) and then only apply these transformations on the number components.
-This is a chicken-and-egg problem, though, because *we cannot appropriately
-separate out the numbers because of the thousands separators and
-non-'.' decimal separators* (well, at least not without making multiple
-passes over the data which I do not consider to be a valid option).
-
-Regular expressions to the rescue! With regular expressions, we can
-remove the thousands separators and change the decimal separator only
-when they are actually within a number. Once the input has been
-pre-processed with this regular expression, all the infrastructure
-shown previously will work.
-
-Beware, these regular expressions will make your eyes bleed.
-
-.. code-block:: pycon
-
- >>> decimal = ',' # Assume German locale, so decimal separator is ','
- >>> # Look-behind assertions cannot accept range modifiers, so instead of i.e.
- >>> # (?<!\.[0-9]{1,3}) I have to repeat the look-behind for 1, 2, and 3.
- >>> nodecimal = r'(?<!{dec}[0-9])(?<!{dec}[0-9]{{2}})(?<!{dec}[0-9]{{3}})'.format(dec=decimal)
- >>> strip_thousands = r'''
- ... (?<=[0-9]{{1}}) # At least 1 number
- ... (?<![0-9]{{4}}) # No more than 3 numbers
- ... {nodecimal} # Cannot follow decimal
- ... {thou} # The thousands separator
- ... (?=[0-9]{{3}} # Three numbers must follow
- ... ([^0-9]|$) # But a non-number after that
- ... )
- ... '''.format(nodecimal=nodecimal, thou=re.escape('.')) # Thousands separator is '.' in German locale.
- ...
- >>> re.sub(strip_thousands, '', 'Sir, €1.234,50 please.', flags=re.X)
- 'Sir, €1234,50 please.'
- >>>
- >>> # The decimal point must be preceded by a number or after
- >>> # a number. This option only needs to be performed in the
- >>> # case when the decimal separator for the locale is not '.'.
- >>> switch_decimal = r'(?<=[0-9]){decimal}|{decimal}(?=[0-9])'
- >>> switch_decimal = switch_decimal.format(decimal=decimal)
- >>> re.sub(switch_decimal, '.', 'Sir, €1234,50 please.', flags=re.X)
- 'Sir, €1234.50 please.'
- >>>
- >>> natsort_key('Sir, €1234.50 please.', as_float=True)
- ('Sir, €', 1234.5, ' please.')
-
-Final Thoughts
---------------
-
-My hope is that users of :mod:`natsort` never have to think about or worry
-about all the bookkeeping or any of the details described above, and that using
-:mod:`natsort` seems to magically "just work". For those of you who
-took the time to read this engineering description, I hope it has enlightened
-you to some of the issues that can be encountered when code is released
-into the wild and has to accept "real-world data", or to what happens
-to developers who naïvely make bold assumptions that are counter to
-what the rest of the world assumes.
-
-.. rubric:: Footnotes
-
-.. [#f1]
- *"But if you hadn't removed the leading empty string from re.split this
- wouldn't have happened!!"* I can hear you saying. Well, that's true. I don't
- have a *great* reason for having done that except that in an earlier
- non-optimal incarnation of the algorithm I needed to it, and it kind of
- stuck, and it made other parts of the code easier if the assumption that
- there were no empty strings was valid.
-.. [#f2]
- I'm not going to show how this is implemented in this document,
- but if you are interested you can look at the code to
- :func:`sep_inserter` in `util.py`_.
-.. [#f3]
- Handling each of these is straightforward, but coupled with the rapidly
- fracturing execution paths presented in :ref:`TL;DR 2 <tldr2>` one can
- imagine this will get out of hand quickly. If you take a look at
- `natsort.py`_ and `util.py`_ you can observe that to avoid this I take
- a more functional approach to construting the :mod:`natsort` algorithm
- as opposed to the procedural approach illustrated in
- :ref:`TL;DR 1 <tldr1>` and :ref:`TL;DR 2 <tldr2>`.
-
-.. _ASCII table: https://www.asciitable.com/
-.. _getting sorting right is surprisingly hard: http://www.compciv.org/guides/python/fundamentals/sorting-collections-with-sorted/
-.. _This astonished: https://github.com/SethMMorton/natsort/issues/19
-.. _a lot: https://stackoverflow.com/questions/29548742/python-natsort-sort-strings-recursively
-.. _of people: https://stackoverflow.com/questions/24045348/sort-set-of-numbers-in-the-form-xx-yy-in-python
-.. _and some people aren't very nice when they are astonished:
- https://github.com/xolox/python-naturalsort/blob/ed3e6b6ffaca3bdea3b76e08acbb8bd2a5fee463/README.rst#why-another-natsort-module
-.. _fastnumbers: https://github.com/SethMMorton/fastnumbers
-.. _as part of my testing: https://github.com/SethMMorton/natsort/blob/master/test_natsort/slow_splitters.py
-.. _this one for coercion: https://stackoverflow.com/questions/736043/checking-if-a-string-can-be-converted-to-float-in-python
-.. _this one for checking: https://stackoverflow.com/questions/354038/how-do-i-check-if-a-string-is-a-number-float
-.. _most natural sort solutions for python on Stack Overflow: https://stackoverflow.com/q/4836710/1399279
-.. _80%/20%: https://en.wikipedia.org/wiki/Pareto_principle
-.. _The first major special case I encountered was sorting filesystem paths: https://github.com/SethMMorton/natsort/issues/3
-.. _The second major special case I encountered was sorting of different types: https://github.com/SethMMorton/natsort/issues/7
-.. _A rather unexpected special case I encountered was sorting collections containing NaN:
- https://github.com/SethMMorton/natsort/issues/27
-.. _It's hard to compare floating point numbers: http://www.drdobbs.com/cpp/its-hard-to-compare-floating-point-numbe/240149806
-.. _caught a bit off guard when the request was initially made: https://github.com/SethMMorton/natsort/issues/14
-.. _at the code: https://github.com/SethMMorton/natsort/tree/master/natsort
-.. _natsort.py: https://github.com/SethMMorton/natsort/blob/master/natsort/natsort.py
-.. _util.py: https://github.com/SethMMorton/natsort/blob/master/natsort/util.py
-.. _although they do point out in the documentation that it will be painful to use:
- https://docs.python.org/3/library/locale.html#background-details-hints-tips-and-caveats
-.. _natsort.compat.locale.py: https://github.com/SethMMorton/natsort/blob/master/natsort/compat/locale.py
-.. _Thousands separator support: https://github.com/SethMMorton/natsort/issues/36
-.. _really good: https://hypothesis.readthedocs.io/en/latest/
-.. _testing strategy: https://docs.pytest.org/en/latest/
-.. _check out some official Unicode documentation: https://unicode.org/reports/tr15/
+This page has been moved to the
+`natsort wiki <https://github.com/SethMMorton/natsort/wiki/How-Does-Natsort-Work%3F#special-cases-everywhere>`_.
diff --git a/docs/locale_issues.rst b/docs/locale_issues.rst
index f51ab27..3539904 100644
--- a/docs/locale_issues.rst
+++ b/docs/locale_issues.rst
@@ -6,92 +6,5 @@
Possible Issues with :func:`~natsort.humansorted` or ``ns.LOCALE``
==================================================================
-Being Locale-Aware Means Both Numbers and Non-Numbers
------------------------------------------------------
-
-In addition to modifying how characters are sorted, ``ns.LOCALE`` will take
-into account locale-dependent thousands separators (and locale-dependent
-decimal separators if ``ns.FLOAT`` is enabled). This means that if you are in a
-locale that uses commas as the thousands separator, a number like
-``123,456`` will be interpreted as ``123456``. If this is not what you want,
-you may consider using ``ns.LOCALEALPHA`` which will only enable locale-aware
-sorting for non-numbers (similarly, ``ns.LOCALENUM`` enables locale-aware
-sorting only for numbers).
-
-Regenerate Key With :func:`~natsort.natsort_keygen` After Changing Locale
--------------------------------------------------------------------------
-
-When :func:`~natsort.natsort_keygen` is called it returns a key function that
-hard-codes the provided settings. This means that the key returned when
-``ns.LOCALE`` is used contains the settings specifed by the locale
-*loaded at the time the key is generated*. If you change the locale,
-you should regenerate the key to account for the new locale.
-
-Corollary: Do Not Reuse :func:`~natsort.natsort_keygen` After Changing Locale
-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-If you change locale, the old function will not work as expected.
-The :mod:`locale` library works with a global state. When
-:func:`~natsort.natsort_keygen` is called it does the best job that it can to
-make the returned function as static as possible and independent of the global
-state, but the :func:`locale.strxfrm` function must access this global state to
-work; therefore, if you change locale and use ``ns.LOCALE`` then you should
-discard the old key.
-
-.. note:: If you use `PyICU`_ then you may be able to reuse keys after changing
- locale.
-
-The :mod:`locale` Module From the StdLib Has Issues
----------------------------------------------------
-
-:mod:`natsort` will use `PyICU`_ for :func:`~natsort.humansorted` or
-``ns.LOCALE`` if it is installed. If not, it will fall back on the
-:mod:`locale` library from the Python stdlib. If you do not have `PyICU`_
-installed, please keep the following known problems and issues in mind.
-
-.. note:: Remember, if you have `PyICU`_ installed you shouldn't need to worry
- about any of these.
-
-Explicitly Set the Locale Before Using ``ns.LOCALE``
-++++++++++++++++++++++++++++++++++++++++++++++++++++
-
-I have found that unless you explicitly set a locale, the sorted order may not
-be what you expect. Setting this is straightforward
-(in the below example I use 'en_US.UTF-8', but you should use your
-locale):
-
-.. code-block:: pycon
-
- >>> import locale
- >>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
- 'en_US.UTF-8'
-
-.. _bug_note:
-
-The :mod:`locale` Module Is Broken on Mac OS X
-++++++++++++++++++++++++++++++++++++++++++++++
-
-It's not Python's fault, but the OS... the locale library for BSD-based systems
-(of which Mac OS X is one) is broken. See the following links:
-
- - https://stackoverflow.com/questions/3412933/python-not-sorting-unicode-properly-strcoll-doesnt-help
- - https://bugs.python.org/issue23195
- - https://github.com/SethMMorton/natsort/issues/21 (contains instructons on installing)
- - https://stackoverflow.com/questions/33459384/unicode-character-not-in-range-when-calling-locale-strxfrm
- - https://github.com/SethMMorton/natsort/issues/34
-
-Of course, installing `PyICU`_ fixes this, but if you don't want to or cannot
-install this there is some hope.
-
- 1. As of ``natsort`` version 4.0.0, ``natsort`` is configured
- to compensate for a broken ``locale`` library. When sorting non-numbers
- it will handle case as you expect, but it will still not be able to
- comprehend non-ASCII characters properly. Additionally, it has
- a built-in lookup table of thousands separators that are incorrect
- on OS X/BSD (but is possible it is not complete... please file an
- issue if you see it is not complete)
- 2. Use "\*.ISO8859-1" locale (i.e. 'en_US.ISO8859-1') rather than
- "\*.UTF-8" locale. I have found that these have fewer issues than
- "UTF-8", but your mileage may vary.
-
-.. _PyICU: https://pypi.org/project/PyICU
+This page has been moved to the
+`natsort wiki <https://github.com/SethMMorton/natsort/wiki/Possible-Issues-with-natsort.humansorted-or-ns.LOCALE>`_.
diff --git a/docs/shell.rst b/docs/shell.rst
index 0d7d3c9..bf40874 100644
--- a/docs/shell.rst
+++ b/docs/shell.rst
@@ -6,153 +6,5 @@
Shell Script
============
-The ``natsort`` shell script is automatically installed when you install
-:mod:`natsort` with pip.
-
-Below is the usage and some usage examples for the ``natsort`` shell script.
-
-Usage
------
-
-.. code-block::
-
- usage: natsort [-h] [--version] [-p] [-f LOW HIGH] [-F LOW HIGH] [-e EXCLUDE]
- [-r] [-t {digit,int,float,version,ver}] [--nosign] [--noexp]
- [--locale]
- [entries [entries ...]]
-
- Performs a natural sort on entries given on the command-line.
- A natural sort sorts numerically then alphabetically, and will sort
- by numbers in the middle of an entry.
-
- positional arguments:
- entries The entries to sort. Taken from stdin if nothing is
- given on the command line.
-
- optional arguments:
- -h, --help show this help message and exit
- --version show program's version number and exit
- -p, --paths Interpret the input as file paths. This is not
- strictly necessary to sort all file paths, but in
- cases where there are OS-generated file paths like
- "Folder/" and "Folder (1)/", this option is needed to
- make the paths sorted in the order you expect
- ("Folder/" before "Folder (1)/").
- -f LOW HIGH, --filter LOW HIGH
- Used for keeping only the entries that have a number
- falling in the given range.
- -F LOW HIGH, --reverse-filter LOW HIGH
- Used for excluding the entries that have a number
- falling in the given range.
- -e EXCLUDE, --exclude EXCLUDE
- Used to exclude an entry that contains a specific
- number.
- -r, --reverse Returns in reversed order.
- -t {digit,int,float,version,ver,real,f,i,r,d},
- --number-type {digit,int,float,version,ver,real,f,i,r,d},
- --number_type {digit,int,float,version,ver,real,f,i,r,d}
- Choose the type of number to search for. "float" will
- search for floating-point numbers. "int" will only
- search for integers. "digit", "version", and "ver" are
- synonyms for "int"."real" is a shortcut for "float"
- with --sign. "i" and "d" are synonyms for "int", "f"
- is a synonym for "float", and "r" is a synonym for
- "real".The default is int.
- --nosign Do not consider "+" or "-" as part of a number, i.e.
- do not take sign into consideration. This is the
- default.
- -s, --sign Consider "+" or "-" as part of a number, i.e. take
- sign into consideration. The default is unsigned.
- --noexp Do not consider an exponential as part of a number,
- i.e. 1e4, would be considered as 1, "e", and 4, not as
- 10000. This only effects the --number-type=float.
- -l, --locale Causes natsort to use locale-aware sorting. You will
- get the best results if you install PyICU.
-
-Description
------------
-
-``natsort`` was originally written to aid in computational chemistry
-research so that it would be easy to analyze large sets of output files
-named after the parameter used:
-
-.. code-block:: console
-
- $ ls *.out
- mode1000.35.out mode1243.34.out mode744.43.out mode943.54.out
-
-(Obviously, in reality there would be more files, but you get the idea.) Notice
-that the shell sorts in lexicographical order. This is the behavior of programs like
-``find`` as well as ``ls``. The problem is passing these files to an
-analysis program causes them not to appear in numerical order, which can lead
-to bad analysis. To remedy this, use ``natsort``:
-
-.. code-block:: console
-
- $ natsort *.out
- mode744.43.out
- mode943.54.out
- mode1000.35.out
- mode1243.34.out
- $ natsort -t r *.out | xargs your_program
-
-``-t r`` is short for ``--number-type real``. You can also place natsort in
-the middle of a pipe:
-
-.. code-block:: console
-
- $ find . -name "*.out" | natsort -t r | xargs your_program
-
-To sort version numbers, use the default ``--number-type``:
-
-.. code-block:: console
-
- $ ls *
- prog-1.10.zip prog-1.9.zip prog-2.0.zip
- $ natsort *
- prog-1.9.zip
- prog-1.10.zip
- prog-2.0.zip
-
-In general, all ``natsort`` shell script options mirror the :func:`~natsorted`
-API, with notable exception of the ``--filter``, ``--reverse-filter``, and ``--exclude``
-options. These three options are used as follows:
-
-.. code-block:: console
-
- $ ls *.out
- mode1000.35.out mode1243.34.out mode744.43.out mode943.54.out
- $ natsort -t r *.out -f 900 1100 # Select only numbers between 900-1100
- mode943.54.out
- mode1000.35.out
- $ natsort -t r *.out -F 900 1100 # Select only numbers NOT between 900-1100
- mode744.43.out
- mode1243.34.out
- $ natsort -t r *.out -e 1000.35 # Exclude 1000.35 from search
- mode744.43.out
- mode943.54.out
- mode1243.34.out
-
-If you are sorting paths with OS-generated filenames, you may require the
-``--paths``/``-p`` option:
-
-.. code-block:: console
-
- $ find . ! -path . -type f
- ./folder/file (1).txt
- ./folder/file.txt
- ./folder (1)/file.txt
- ./folder (10)/file.txt
- ./folder (2)/file.txt
- $ find . ! -path . -type f | natsort
- ./folder (1)/file.txt
- ./folder (2)/file.txt
- ./folder (10)/file.txt
- ./folder/file (1).txt
- ./folder/file.txt
- $ find . ! -path . -type f | natsort -p
- ./folder/file.txt
- ./folder/file (1).txt
- ./folder (1)/file.txt
- ./folder (2)/file.txt
- ./folder (10)/file.txt
+This page has been moved to the
+`natsort wiki <https://github.com/SethMMorton/natsort/wiki/Shell-Script>`_.
diff --git a/mypy_stubs/icu.pyi b/mypy_stubs/icu.pyi
new file mode 100644
index 0000000..f7ac204
--- /dev/null
+++ b/mypy_stubs/icu.pyi
@@ -0,0 +1,24 @@
+from typing import overload
+
+@overload
+def Locale() -> str: ...
+@overload
+def Locale(x: str) -> str: ...
+
+class UCollAttribute:
+ NUMERIC_COLLATION: int
+
+class UCollAttributeValue:
+ ON: int
+
+class DecimalFormatSymbols:
+ kGroupingSeparatorSymbol: int
+ kDecimalSeparatorSymbol: int
+ def __init__(self, locale: str) -> None: ...
+ def getSymbol(self, symbol: int) -> str: ...
+
+class Collator:
+ @classmethod
+ def createInstance(cls, locale: str) -> Collator: ...
+ def getSortKey(self, source: str) -> bytes: ...
+ def setAttribute(self, attr: int, value: int) -> None: ...
diff --git a/natsort/__init__.py b/natsort/__init__.py
index 0c3a07d..420c2dd 100644
--- a/natsort/__init__.py
+++ b/natsort/__init__.py
@@ -23,7 +23,7 @@ from natsort.natsort import (
from natsort.ns_enum import NSType, ns
from natsort.utils import KeyType, NatsortInType, NatsortOutType, chain_functions
-__version__ = "8.0.2"
+__version__ = "8.3.1"
__all__ = [
"natsort_key",
diff --git a/natsort/compat/fake_fastnumbers.py b/natsort/compat/fake_fastnumbers.py
index 5d44605..430345a 100644
--- a/natsort/compat/fake_fastnumbers.py
+++ b/natsort/compat/fake_fastnumbers.py
@@ -4,7 +4,7 @@ This module is intended to replicate some of the functionality
from the fastnumbers module in the event that module is not installed.
"""
import unicodedata
-from typing import Callable, FrozenSet, Optional, Union
+from typing import Callable, FrozenSet, Union
from natsort.unicode_numbers import decimal_chars
@@ -35,11 +35,10 @@ StrOrFloat = Union[str, float]
StrOrInt = Union[str, int]
-# noinspection PyIncorrectDocstring
def fast_float(
x: str,
- key: Callable[[str], StrOrFloat] = lambda x: x,
- nan: Optional[StrOrFloat] = None,
+ key: Callable[[str], str] = lambda x: x,
+ nan: float = float("inf"),
_uni: Callable[[str, StrOrFloat], StrOrFloat] = unicodedata.numeric,
_nan_inf: FrozenSet[str] = NAN_INF,
_first_char: FrozenSet[str] = POTENTIAL_FIRST_CHAR,
@@ -56,7 +55,7 @@ def fast_float(
String to attempt to convert to a float.
key : callable
Single-argument function to apply to *x* if conversion fails.
- nan : object
+ nan : float
Value to return instead of NaN if NaN would be returned.
Returns
@@ -67,7 +66,7 @@ def fast_float(
if x[0] in _first_char or x.lstrip()[:3] in _nan_inf:
try:
ret = float(x)
- return nan if nan is not None and ret != ret else ret
+ return nan if ret != ret else ret
except ValueError:
try:
return _uni(x, key(x)) if len(x) == 1 else key(x)
@@ -80,10 +79,9 @@ def fast_float(
return key(x)
-# noinspection PyIncorrectDocstring
def fast_int(
x: str,
- key: Callable[[str], StrOrInt] = lambda x: x,
+ key: Callable[[str], str] = lambda x: x,
_uni: Callable[[str, StrOrInt], StrOrInt] = unicodedata.digit,
_first_char: FrozenSet[str] = POTENTIAL_FIRST_CHAR,
) -> StrOrInt:
diff --git a/natsort/compat/fastnumbers.py b/natsort/compat/fastnumbers.py
index 049030d..f37ee84 100644
--- a/natsort/compat/fastnumbers.py
+++ b/natsort/compat/fastnumbers.py
@@ -4,11 +4,17 @@ Interface for natsort to access fastnumbers functions without
having to worry if it is actually installed.
"""
import re
+from typing import Callable, Iterable, Iterator, Tuple, Union
-__all__ = ["fast_float", "fast_int"]
+StrOrFloat = Union[str, float]
+StrOrInt = Union[str, int]
+__all__ = ["try_float", "try_int"]
-def is_supported_fastnumbers(fastnumbers_version: str) -> bool:
+
+def is_supported_fastnumbers(
+ fastnumbers_version: str, minimum: Tuple[int, int, int] = (2, 0, 0)
+) -> bool:
match = re.match(
r"^(\d+)\.(\d+)(\.(\d+))?([ab](\d+))?$",
fastnumbers_version,
@@ -22,7 +28,7 @@ def is_supported_fastnumbers(fastnumbers_version: str) -> bool:
(major, minor, patch) = match.group(1, 2, 4)
- return (int(major), int(minor), int(patch)) >= (2, 0, 0)
+ return (int(major), int(minor), int(patch)) >= minimum
# If the user has fastnumbers installed, they will get great speed
@@ -34,5 +40,35 @@ try:
# Require >= version 2.0.0.
if not is_supported_fastnumbers(fn_ver):
raise ImportError # pragma: no cover
+
+ # For versions of fastnumbers with mapping capability, use that
+ if is_supported_fastnumbers(fn_ver, (5, 0, 0)):
+ del fast_float, fast_int
+ from fastnumbers import try_float, try_int
except ImportError:
from natsort.compat.fake_fastnumbers import fast_float, fast_int # type: ignore
+
+# Re-map the old-or-compatibility functions fast_float/fast_int to the
+# newer API of try_float/try_int. If we already imported try_float/try_int
+# then there is nothing to do.
+if "try_float" not in globals():
+
+ def try_float( # type: ignore[no-redef] # noqa: F811
+ x: Iterable[str],
+ map: bool,
+ nan: float = float("inf"),
+ on_fail: Callable[[str], str] = lambda x: x,
+ ) -> Iterator[StrOrFloat]:
+ assert map is True
+ return (fast_float(y, nan=nan, key=on_fail) for y in x)
+
+
+if "try_int" not in globals():
+
+ def try_int( # type: ignore[no-redef] # noqa: F811
+ x: Iterable[str],
+ map: bool,
+ on_fail: Callable[[str], str] = lambda x: x,
+ ) -> Iterator[StrOrInt]:
+ assert map is True
+ return (fast_int(y, key=on_fail) for y in x)
diff --git a/natsort/compat/locale.py b/natsort/compat/locale.py
index b4c5356..d802194 100644
--- a/natsort/compat/locale.py
+++ b/natsort/compat/locale.py
@@ -38,21 +38,21 @@ try: # noqa: C901
# If using icu, get the locale from the current global locale,
def get_icu_locale() -> str:
- try:
- return cast(str, icu.Locale(".".join(getlocale())))
- except TypeError: # pragma: no cover
- return cast(str, icu.Locale())
+ language_code, encoding = getlocale()
+ if language_code is None or encoding is None: # pragma: no cover
+ return icu.Locale()
+ return icu.Locale(f"{language_code}.{encoding}")
def get_strxfrm() -> TrxfmFunc:
- return cast(TrxfmFunc, icu.Collator.createInstance(get_icu_locale()).getSortKey)
+ return icu.Collator.createInstance(get_icu_locale()).getSortKey
def get_thousands_sep() -> str:
sep = icu.DecimalFormatSymbols.kGroupingSeparatorSymbol
- return cast(str, icu.DecimalFormatSymbols(get_icu_locale()).getSymbol(sep))
+ return icu.DecimalFormatSymbols(get_icu_locale()).getSymbol(sep)
def get_decimal_point() -> str:
sep = icu.DecimalFormatSymbols.kDecimalSeparatorSymbol
- return cast(str, icu.DecimalFormatSymbols(get_icu_locale()).getSymbol(sep))
+ return icu.DecimalFormatSymbols(get_icu_locale()).getSymbol(sep)
except ImportError:
import locale
@@ -75,10 +75,11 @@ except ImportError:
# characters are incorrectly blank. Here is a lookup table of the
# corrections I am aware of.
if dumb_sort():
- try:
- loc = ".".join(locale.getlocale())
- except TypeError: # No locale loaded, default to ','
+ language_code, encoding = locale.getlocale()
+ if language_code is None or encoding is None:
+ # No locale loaded, default to ','
return ","
+ loc = f"{language_code}.{encoding}"
return {
"de_DE.ISO8859-15": ".",
"es_ES.ISO8859-1": ".",
diff --git a/natsort/natsort.py b/natsort/natsort.py
index 9f34bc1..2325443 100644
--- a/natsort/natsort.py
+++ b/natsort/natsort.py
@@ -9,6 +9,7 @@ The majority of the "work" is defined in utils.py.
import platform
from functools import partial
from operator import itemgetter
+from pathlib import PurePath
from typing import (
Any,
Callable,
@@ -18,42 +19,27 @@ from typing import (
Optional,
Sequence,
Tuple,
- Union,
+ TypeVar,
cast,
- overload,
)
import natsort.compat.locale
from natsort import utils
from natsort.ns_enum import NSType, NS_DUMB, ns
-from natsort.utils import (
- KeyType,
- MaybeKeyType,
- NatsortInType,
- NatsortOutType,
- StrBytesNum,
- StrBytesPathNum,
-)
+from natsort.utils import NatsortInType, NatsortOutType
# Common input and output types
-Iter_ns = Iterable[NatsortInType]
-Iter_any = Iterable[Any]
-List_ns = List[NatsortInType]
-List_any = List[Any]
-List_int = List[int]
+T = TypeVar("T")
+NatsortInTypeT = TypeVar("NatsortInTypeT", bound=NatsortInType)
# The type that natsort_key returns
NatsortKeyType = Callable[[NatsortInType], NatsortOutType]
# Types for os_sorted
-OSSortInType = Iterable[Optional[StrBytesPathNum]]
-OSSortOutType = Tuple[Union[StrBytesNum, Tuple[StrBytesNum, ...]], ...]
-OSSortKeyType = Callable[[Optional[StrBytesPathNum]], OSSortOutType]
-Iter_path = Iterable[Optional[StrBytesPathNum]]
-List_path = List[StrBytesPathNum]
+OSSortKeyType = Callable[[NatsortInType], NatsortOutType]
-def decoder(encoding: str) -> Callable[[NatsortInType], NatsortInType]:
+def decoder(encoding: str) -> Callable[[Any], Any]:
"""
Return a function that can be used to decode bytes to unicode.
@@ -94,7 +80,7 @@ def decoder(encoding: str) -> Callable[[NatsortInType], NatsortInType]:
return partial(utils.do_decoding, encoding=encoding)
-def as_ascii(s: NatsortInType) -> NatsortInType:
+def as_ascii(s: Any) -> Any:
"""
Function to decode an input with the ASCII codec, or return as-is.
@@ -117,7 +103,7 @@ def as_ascii(s: NatsortInType) -> NatsortInType:
return utils.do_decoding(s, "ascii")
-def as_utf8(s: NatsortInType) -> NatsortInType:
+def as_utf8(s: Any) -> Any:
"""
Function to decode an input with the UTF-8 codec, or return as-is.
@@ -141,8 +127,8 @@ def as_utf8(s: NatsortInType) -> NatsortInType:
def natsort_keygen(
- key: MaybeKeyType = None, alg: NSType = ns.DEFAULT
-) -> NatsortKeyType:
+ key: Optional[Callable[[Any], NatsortInType]] = None, alg: NSType = ns.DEFAULT
+) -> Callable[[Any], NatsortOutType]:
"""
Generate a key to sort strings and numbers naturally.
@@ -252,26 +238,12 @@ natsort_keygen
"""
-@overload
-def natsorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_ns:
- ...
-
-
-@overload
-def natsorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_any:
- ...
-
-
def natsorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_any:
+) -> List[T]:
"""
Sorts an iterable naturally.
@@ -316,29 +288,17 @@ def natsorted(
['num2', 'num3', 'num5']
"""
+ if alg & ns.PRESORT:
+ seq = sorted(seq, reverse=reverse, key=str)
return sorted(seq, reverse=reverse, key=natsort_keygen(key, alg))
-@overload
-def humansorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_ns:
- ...
-
-
-@overload
def humansorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_any:
- ...
-
-
-def humansorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_any:
+) -> List[T]:
"""
Convenience function to properly sort non-numeric characters.
@@ -390,26 +350,12 @@ def humansorted(
return natsorted(seq, key, reverse, alg | ns.LOCALE)
-@overload
-def realsorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_ns:
- ...
-
-
-@overload
def realsorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_any:
- ...
-
-
-def realsorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_any:
+) -> List[T]:
"""
Convenience function to properly sort signed floats.
@@ -462,26 +408,12 @@ def realsorted(
return natsorted(seq, key, reverse, alg | ns.REAL)
-@overload
-def index_natsorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
-@overload
-def index_natsorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
def index_natsorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_int:
+) -> List[int]:
"""
Determine the list of the indexes used to sort the input sequence.
@@ -537,40 +469,28 @@ def index_natsorted(
['baz', 'foo', 'bar']
"""
- newkey: KeyType
+ newkey: Callable[[Tuple[int, T]], NatsortInType]
if key is None:
newkey = itemgetter(1)
else:
- def newkey(x: Any) -> NatsortInType:
- return cast(KeyType, key)(itemgetter(1)(x))
+ def newkey(x: Tuple[int, T]) -> NatsortInType:
+ return cast(Callable[[T], NatsortInType], key)(itemgetter(1)(x))
# Pair the index and sequence together, then sort by element
index_seq_pair = [(x, y) for x, y in enumerate(seq)]
+ if alg & ns.PRESORT:
+ index_seq_pair.sort(reverse=reverse, key=lambda x: str(itemgetter(1)(x)))
index_seq_pair.sort(reverse=reverse, key=natsort_keygen(newkey, alg))
return [x for x, _ in index_seq_pair]
-@overload
-def index_humansorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
-@overload
-def index_humansorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
def index_humansorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_int:
+) -> List[int]:
"""
This is a wrapper around ``index_natsorted(seq, alg=ns.LOCALE)``.
@@ -619,26 +539,12 @@ def index_humansorted(
return index_natsorted(seq, key, reverse, alg | ns.LOCALE)
-@overload
-def index_realsorted(
- seq: Iter_ns, key: None = None, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
-@overload
def index_realsorted(
- seq: Iter_any, key: KeyType, reverse: bool = False, alg: NSType = ns.DEFAULT
-) -> List_int:
- ...
-
-
-def index_realsorted(
- seq: Iter_any,
- key: MaybeKeyType = None,
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
reverse: bool = False,
alg: NSType = ns.DEFAULT,
-) -> List_int:
+) -> List[int]:
"""
This is a wrapper around ``index_natsorted(seq, alg=ns.REAL)``.
@@ -683,10 +589,9 @@ def index_realsorted(
return index_natsorted(seq, key, reverse, alg | ns.REAL)
-# noinspection PyShadowingBuiltins,PyUnresolvedReferences
def order_by_index(
seq: Sequence[Any], index: Iterable[int], iter: bool = False
-) -> Iter_any:
+) -> Iterable[Any]:
"""
Order a given sequence by an index sequence.
@@ -764,15 +669,18 @@ def numeric_regex_chooser(alg: NSType) -> str:
return utils.regex_chooser(alg).pattern[1:-1]
-def _split_apply(v: Any, key: MaybeKeyType = None) -> Iterator[str]:
+def _split_apply(
+ v: Any, key: Optional[Callable[[T], NatsortInType]] = None
+) -> Iterator[str]:
if key is not None:
v = key(v)
- return utils.path_splitter(str(v))
+ if not isinstance(v, (str, PurePath)):
+ v = str(v)
+ return utils.path_splitter(v)
# Choose the implementation based on the host OS
if platform.system() == "Windows":
-
from ctypes import wintypes, windll # type: ignore
from functools import cmp_to_key
@@ -781,13 +689,15 @@ if platform.system() == "Windows":
_windows_sort_cmp.restype = wintypes.INT
_winsort_key = cmp_to_key(_windows_sort_cmp)
- def os_sort_keygen(key: MaybeKeyType = None) -> OSSortKeyType:
+ def os_sort_keygen(
+ key: Optional[Callable[[Any], NatsortInType]] = None
+ ) -> Callable[[Any], NatsortOutType]:
return cast(
- OSSortKeyType, lambda x: tuple(map(_winsort_key, _split_apply(x, key)))
+ Callable[[Any], NatsortOutType],
+ lambda x: tuple(map(_winsort_key, _split_apply(x, key))),
)
else:
-
# For UNIX-based platforms, ICU performs MUCH better than locale
# at replicating the file explorer's sort order. We will use
# ICU's ability to do basic natural sorting as it also better
@@ -802,15 +712,16 @@ else:
except ImportError:
# No ICU installed
- def os_sort_keygen(key: MaybeKeyType = None) -> OSSortKeyType:
- return cast(
- OSSortKeyType,
- natsort_keygen(key=key, alg=ns.LOCALE | ns.PATH | ns.IGNORECASE),
- )
+ def os_sort_keygen(
+ key: Optional[Callable[[Any], NatsortInType]] = None
+ ) -> Callable[[Any], NatsortOutType]:
+ return natsort_keygen(key=key, alg=ns.LOCALE | ns.PATH | ns.IGNORECASE)
else:
# ICU installed
- def os_sort_keygen(key: MaybeKeyType = None) -> OSSortKeyType:
+ def os_sort_keygen(
+ key: Optional[Callable[[Any], NatsortInType]] = None
+ ) -> Callable[[Any], NatsortOutType]:
loc = natsort.compat.locale.get_icu_locale()
collator = icu.Collator.createInstance(loc)
collator.setAttribute(
@@ -857,26 +768,19 @@ os_sort_keygen
"""
-@overload
-def os_sorted(seq: Iter_path, key: None = None, reverse: bool = False) -> List_path:
- ...
-
-
-@overload
-def os_sorted(seq: Iter_any, key: KeyType, reverse: bool = False) -> List_any:
- ...
-
-
def os_sorted(
- seq: Iter_any, key: MaybeKeyType = None, reverse: bool = False
-) -> List_any:
+ seq: Iterable[T],
+ key: Optional[Callable[[T], NatsortInType]] = None,
+ reverse: bool = False,
+ presort: bool = False,
+) -> List[T]:
"""
Sort elements in the same order as your operating system's file browser
.. warning::
The resulting function will generate results that will be
- differnt depending on your platform. This is intentional.
+ different depending on your platform. This is intentional.
On Windows, this will sort with the same order as Windows Explorer.
@@ -892,7 +796,7 @@ def os_sorted(
special characters this will give correct results, but once
special characters are added you should lower your expectations.
- It is *strongly* reccommended to have :mod:`pyicu` installed on
+ It is *strongly* recommended to have :mod:`pyicu` installed on
MacOS/Linux if you want correct sort results.
It does *not* take into account if a path is a directory or a file
@@ -911,6 +815,10 @@ def os_sorted(
Return the list in reversed sorted order. The default is
`False`.
+ presort : {{True, False}}, optional
+ Equivalent to adding ``ns.PRESORT``, see :class:`ns` for
+ documentation. The default is `False`.
+
Returns
-------
out : list
@@ -926,4 +834,6 @@ def os_sorted(
This will implicitly coerce all inputs to str before collating.
"""
- return sorted(seq, key=os_sort_keygen(key), reverse=reverse)
+ if presort:
+ seq = sorted(seq, reverse=reverse, key=str)
+ return sorted(seq, reverse=reverse, key=os_sort_keygen(key))
diff --git a/natsort/ns_enum.py b/natsort/ns_enum.py
index c147909..02f970f 100644
--- a/natsort/ns_enum.py
+++ b/natsort/ns_enum.py
@@ -114,6 +114,14 @@ class ns(enum.IntEnum): # noqa: N801
treat these as +Infinity and place them after all the other numbers.
By default, an NaN be treated as -Infinity and be placed first.
Note that this ``None`` is treated like NaN internally.
+ PRESORT, PS
+ Sort the input as strings before sorting with the `nasort`
+ algorithm. This can help eliminate inconsistent sorting in cases
+ where two different strings represent the same number. For example,
+ "a1" and "a01" both are internally represented as ("a", "1), so
+ without `PRESORT` the order of these two values would depend on
+ the order they appeared in the input (because Python's `sorted`
+ is a stable sorting algorithm).
Notes
-----
@@ -143,6 +151,7 @@ class ns(enum.IntEnum): # noqa: N801
NANLAST = NL = 1 << next(_counter)
COMPATIBILITYNORMALIZE = CN = 1 << next(_counter)
NUMAFTER = NA = 1 << next(_counter)
+ PRESORT = PS = 1 << next(_counter)
# Following were previously options but are now defaults.
DEFAULT = 0
diff --git a/natsort/unicode_numbers.py b/natsort/unicode_numbers.py
index 7800fd2..94cf0ff 100644
--- a/natsort/unicode_numbers.py
+++ b/natsort/unicode_numbers.py
@@ -24,7 +24,7 @@ for a in numeric_hex:
# The digit characters are a subset of the numerals.
digit_chars = [a for a in numeric_chars if unicodedata.digit(a, None) is not None]
-# The decimal characters are a subset of the numberals
+# The decimal characters are a subset of the numerals
# (probably of the digits, but let's be safe).
decimal_chars = [a for a in numeric_chars if unicodedata.decimal(a, None) is not None]
diff --git a/natsort/unicode_numeric_hex.py b/natsort/unicode_numeric_hex.py
index c1e789d..a4fce03 100644
--- a/natsort/unicode_numeric_hex.py
+++ b/natsort/unicode_numeric_hex.py
@@ -1519,6 +1519,16 @@ numeric_hex = (
0x16A67,
0x16A68,
0x16A69,
+ 0x16AC0,
+ 0x16AC1,
+ 0x16AC2,
+ 0x16AC3,
+ 0x16AC4,
+ 0x16AC5,
+ 0x16AC6,
+ 0x16AC7,
+ 0x16AC8,
+ 0x16AC9,
0x16B50,
0x16B51,
0x16B52,
diff --git a/natsort/utils.py b/natsort/utils.py
index 8d56b06..dd8c39f 100644
--- a/natsort/utils.py
+++ b/natsort/utils.py
@@ -53,6 +53,7 @@ from typing import (
Match,
Optional,
Pattern,
+ TYPE_CHECKING,
Tuple,
Union,
cast,
@@ -60,7 +61,7 @@ from typing import (
)
from unicodedata import normalize
-from natsort.compat.fastnumbers import fast_float, fast_int
+from natsort.compat.fastnumbers import try_float, try_int
from natsort.compat.locale import (
StrOrBytes,
get_decimal_point,
@@ -70,9 +71,28 @@ from natsort.compat.locale import (
from natsort.ns_enum import NSType, NS_DUMB, ns
from natsort.unicode_numbers import digits_no_decimals, numeric_no_decimals
+if TYPE_CHECKING:
+ from typing_extensions import Protocol
+else:
+ Protocol = object
+
#
# Pre-define a slew of aggregate types which makes the type hinting below easier
#
+
+
+class SupportsDunderLT(Protocol):
+ def __lt__(self, __other: Any) -> bool:
+ ...
+
+
+class SupportsDunderGT(Protocol):
+ def __gt__(self, __other: Any) -> bool:
+ ...
+
+
+Sortable = Union[SupportsDunderLT, SupportsDunderGT]
+
StrToStr = Callable[[str], str]
AnyCall = Callable[[Any], Any]
@@ -83,45 +103,33 @@ BytesTransform = Union[BytesTuple, NestedBytesTuple]
BytesTransformer = Callable[[bytes], BytesTransform]
# For the number transform factory
-NumType = Union[float, int]
-MaybeNumType = Optional[NumType]
-NumTuple = Tuple[StrOrBytes, NumType]
-NestedNumTuple = Tuple[NumTuple]
-StrNumTuple = Tuple[Tuple[str], NumTuple]
-NestedStrNumTuple = Tuple[StrNumTuple]
-MaybeNumTransform = Union[NumTuple, NestedNumTuple, StrNumTuple, NestedStrNumTuple]
-MaybeNumTransformer = Callable[[MaybeNumType], MaybeNumTransform]
+BasicTuple = Tuple[Any, ...]
+NestedAnyTuple = Tuple[BasicTuple, ...]
+AnyTuple = Union[BasicTuple, NestedAnyTuple]
+NumTransform = AnyTuple
+NumTransformer = Callable[[Any], NumTransform]
# For the string component transform factory
StrBytesNum = Union[str, bytes, float, int]
-StrTransformer = Callable[[str], StrBytesNum]
+StrTransformer = Callable[[Iterable[str]], Iterator[StrBytesNum]]
# For the final data transform factory
-TwoBlankTuple = Tuple[Tuple[()], Tuple[()]]
-TupleOfAny = Tuple[Any, ...]
-TupleOfStrAnyPair = Tuple[Tuple[str], TupleOfAny]
-FinalTransform = Union[TwoBlankTuple, TupleOfAny, TupleOfStrAnyPair]
+FinalTransform = AnyTuple
FinalTransformer = Callable[[Iterable[Any], str], FinalTransform]
-# For the string parsing factory
-StrSplitter = Callable[[str], Iterable[str]]
-StrParser = Callable[[str], FinalTransform]
-
-# For the path splitter
PathArg = Union[str, PurePath]
MatchFn = Callable[[str], Optional[Match]]
+# For the string parsing factory
+StrSplitter = Callable[[str], Iterable[str]]
+StrParser = Callable[[PathArg], FinalTransform]
+
# For the path parsing factory
PathSplitter = Callable[[PathArg], Tuple[FinalTransform, ...]]
# For the natsort key
-StrBytesPathNum = Union[str, bytes, float, int, PurePath]
-NatsortInType = Union[
- Optional[StrBytesPathNum], Iterable[Union[Optional[StrBytesPathNum], Iterable[Any]]]
-]
-NatsortOutType = Tuple[
- Union[StrBytesNum, Tuple[Union[StrBytesNum, Tuple[Any, ...]], ...]], ...
-]
+NatsortInType = Optional[Sortable]
+NatsortOutType = Tuple[Sortable, ...]
KeyType = Callable[[Any], NatsortInType]
MaybeKeyType = Optional[KeyType]
@@ -260,7 +268,7 @@ def natsort_key(
key: None,
string_func: Union[StrParser, PathSplitter],
bytes_func: BytesTransformer,
- num_func: MaybeNumTransformer,
+ num_func: NumTransformer,
) -> NatsortOutType:
...
@@ -271,7 +279,7 @@ def natsort_key(
key: KeyType,
string_func: Union[StrParser, PathSplitter],
bytes_func: BytesTransformer,
- num_func: MaybeNumTransformer,
+ num_func: NumTransformer,
) -> NatsortOutType:
...
@@ -281,7 +289,7 @@ def natsort_key(
key: MaybeKeyType,
string_func: Union[StrParser, PathSplitter],
bytes_func: BytesTransformer,
- num_func: MaybeNumTransformer,
+ num_func: NumTransformer,
) -> NatsortOutType:
"""
Key to sort strings and numbers naturally.
@@ -321,7 +329,7 @@ def natsort_key(
--------
parse_string_factory
parse_bytes_factory
- parse_number_factory
+ parse_number_or_none_factory
"""
@@ -329,26 +337,17 @@ def natsort_key(
if key is not None:
val = key(val)
- # Assume the input are strings, which is the most common case
- try:
- return string_func(cast(str, val))
- except (TypeError, AttributeError):
-
- # If bytes type, use the bytes_func
- if type(val) in (bytes,):
- return bytes_func(cast(bytes, val))
-
- # Otherwise, assume it is an iterable that must be parsed recursively.
- # Do not apply the key recursively.
- try:
- return tuple(
- natsort_key(x, None, string_func, bytes_func, num_func)
- for x in cast(Iterable[Any], val)
- )
-
- # If that failed, it must be a number.
- except TypeError:
- return num_func(cast(NumType, val))
+ if isinstance(val, (str, PurePath)):
+ return string_func(val)
+ elif isinstance(val, bytes):
+ return bytes_func(val)
+ elif isinstance(val, Iterable):
+ # Must be parsed recursively, but do not apply the key recursively.
+ return tuple(
+ natsort_key(x, None, string_func, bytes_func, num_func) for x in val
+ )
+ else: # Anything else goes here
+ return num_func(val)
def parse_bytes_factory(alg: NSType) -> BytesTransformer:
@@ -386,7 +385,7 @@ def parse_bytes_factory(alg: NSType) -> BytesTransformer:
def parse_number_or_none_factory(
alg: NSType, sep: StrOrBytes, pre_sep: str
-) -> MaybeNumTransformer:
+) -> NumTransformer:
"""
Create a function that will format a number (or None) into a tuple.
@@ -418,10 +417,23 @@ def parse_number_or_none_factory(
nan_replace = float("+inf") if alg & ns.NANLAST else float("-inf")
def func(
- val: MaybeNumType, _nan_replace: float = nan_replace, _sep: StrOrBytes = sep
- ) -> NumTuple:
+ val: Any,
+ _nan_replace: float = nan_replace,
+ _sep: StrOrBytes = sep,
+ reverse: bool = nan_replace == float("+inf"),
+ ) -> BasicTuple:
"""Given a number, place it in a tuple with a leading null string."""
- return _sep, (_nan_replace if val != val or val is None else val)
+ # Add a trailing string numbers equaling _nan_replace. This will make
+ # the ordering between None NaN, and the NaN replacement value...
+ # None comes first, then NaN, then the replacement value.
+ if val != val:
+ return _sep, _nan_replace, "3" if reverse else "1"
+ elif val is None:
+ return _sep, _nan_replace, "2"
+ elif val == _nan_replace:
+ return _sep, _nan_replace, "1" if reverse else "3"
+ else:
+ return _sep, val
# Return the function, possibly wrapping in tuple if PATH is selected.
if alg & ns.PATH and alg & ns.UNGROUPLETTERS and alg & ns.LOCALEALPHA:
@@ -493,7 +505,11 @@ def parse_string_factory(
normalize_input = _normalize_input_factory(alg)
compose_input = _compose_input_factory(alg) if alg & ns.LOCALEALPHA else _no_op
- def func(x: str) -> FinalTransform:
+ def func(x: PathArg) -> FinalTransform:
+ if isinstance(x, PurePath):
+ # While paths are technically not strings, it is natural for them
+ # to be treated the same.
+ x = str(x)
# Apply string input transformation function and return to x.
# Original function is usually a no-op, but some algorithms require it
# to also be the transformation function.
@@ -502,7 +518,7 @@ def parse_string_factory(
c = compose_input(b) # Decompose unicode if using LOCALE
d = splitter(c) # Split string into components.
e = filter(None, d) # Remove empty strings.
- f = map(component_transform, e) # Apply transform on components.
+ f = component_transform(e) # Apply transform on components.
g = sep_inserter(f, sep) # Insert '' between numbers.
return final_transform(g, original) # Apply the final transform.
@@ -685,14 +701,14 @@ def string_component_transform_factory(alg: NSType) -> StrTransformer:
func_chain.append(get_strxfrm())
# Return the correct chained functions.
- kwargs: Dict[str, Union[float, Callable[[str], StrOrBytes]]]
- kwargs = {"key": chain_functions(func_chain)} if func_chain else {}
+ kwargs: Dict[str, Union[float, Callable[[str], StrOrBytes], bool]]
+ kwargs = {"on_fail": chain_functions(func_chain)} if func_chain else {}
+ kwargs["map"] = True
if alg & ns.FLOAT:
- # noinspection PyTypeChecker
kwargs["nan"] = nan_val
- return cast(Callable[[str], StrOrBytes], partial(fast_float, **kwargs))
+ return cast(StrTransformer, partial(try_float, **kwargs))
else:
- return cast(Callable[[str], StrOrBytes], partial(fast_int, **kwargs))
+ return cast(StrTransformer, partial(try_int, **kwargs))
def final_data_transform_factory(
@@ -725,7 +741,7 @@ def final_data_transform_factory(
"""
if alg & ns.UNGROUPLETTERS and alg & ns.LOCALEALPHA:
swap = alg & NS_DUMB and alg & ns.LOWERCASEFIRST
- transform = cast(StrToStr, methodcaller("swapcase")) if swap else _no_op
+ transform = cast(StrToStr, methodcaller("swapcase") if swap else _no_op)
def func(
split_val: Iterable[NatsortInType],
@@ -831,11 +847,11 @@ def do_decoding(s: bytes, encoding: str) -> str:
@overload
-def do_decoding(s: NatsortInType, encoding: str) -> NatsortInType:
+def do_decoding(s: Any, encoding: str) -> Any:
...
-def do_decoding(s: NatsortInType, encoding: str) -> NatsortInType:
+def do_decoding(s: Any, encoding: str) -> Any:
"""
Helper to decode a *bytes* object, or return the object as-is.
@@ -852,9 +868,9 @@ def do_decoding(s: NatsortInType, encoding: str) -> NatsortInType:
*s* if *s* was not *bytes*.
"""
- try:
- return cast(bytes, s).decode(encoding)
- except (AttributeError, TypeError):
+ if isinstance(s, bytes):
+ return s.decode(encoding)
+ else:
return s
@@ -893,16 +909,21 @@ def path_splitter(
path_parts = []
base = str(s)
- # Now, split off the file extensions until we reach a decimal number at
- # the beginning of the suffix or there are no more extensions.
- suffixes = PurePath(base).suffixes
- try:
- digit_index = next(i for i, x in enumerate(reversed(suffixes)) if _d_match(x))
- except StopIteration:
- pass
- else:
- digit_index = len(suffixes) - digit_index
- suffixes = suffixes[digit_index:]
-
+ # Now, split off the file extensions until
+ # - we reach a decimal number at the beginning of the suffix
+ # - more than two suffixes have been seen
+ # - a suffix is more than five characters (including leading ".")
+ # - there are no more extensions
+ suffixes = []
+ for i, suffix in enumerate(reversed(PurePath(base).suffixes)):
+ if _d_match(suffix) or i > 1 or len(suffix) > 5:
+ break
+ suffixes.append(suffix)
+ suffixes.reverse()
+
+ # Remove the suffixes from the base component
base = base.replace("".join(suffixes), "")
- return filter(None, ichain(path_parts, [base], suffixes))
+ base_component = [base] if base else []
+
+ # Join all path comonents in an iterator
+ return filter(None, ichain(path_parts, base_component, suffixes))
diff --git a/setup.cfg b/setup.cfg
index c97939e..42ea3b9 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,5 +1,5 @@
[bumpversion]
-current_version = 8.0.2
+current_version = 8.3.1
commit = True
tag = True
tag_name = {new_version}
@@ -12,7 +12,7 @@ description = Simple yet flexible natural sorting in Python.
long_description = file: README.rst
long_description_content_type = text/x-rst
license = MIT
-license_file = LICENSE
+license_files = LICENSE
classifiers =
Development Status :: 5 - Production/Stable
Intended Audience :: Developers
@@ -25,11 +25,11 @@ classifiers =
Natural Language :: English
Programming Language :: Python
Programming Language :: Python :: 3
- Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
+ Programming Language :: Python :: 3.11
Topic :: Scientific/Engineering :: Information Analysis
Topic :: Utilities
Topic :: Text Processing
@@ -64,6 +64,4 @@ exclude =
.venv
[mypy]
-
-[mypy-icu]
-ignore_missing_imports = True
+mypy_path = mypy_stubs
diff --git a/setup.py b/setup.py
index e74e817..ddff592 100644
--- a/setup.py
+++ b/setup.py
@@ -4,10 +4,10 @@ from setuptools import find_packages, setup
setup(
name="natsort",
- version="8.0.2",
+ version="8.3.1",
packages=find_packages(),
entry_points={"console_scripts": ["natsort = natsort.__main__:main"]},
- python_requires=">=3.6",
+ python_requires=">=3.7",
extras_require={"fast": ["fastnumbers >= 2.0.0"], "icu": ["PyICU >= 1.0.0"]},
package_data={"": ["py.typed"]},
zip_safe=False,
diff --git a/tests/test_fake_fastnumbers.py b/tests/test_fake_fastnumbers.py
index 574f7cf..6324c64 100644
--- a/tests/test_fake_fastnumbers.py
+++ b/tests/test_fake_fastnumbers.py
@@ -4,7 +4,7 @@ Test the fake fastnumbers module.
"""
import unicodedata
-from math import isnan
+from math import isinf
from typing import Union, cast
from hypothesis import given
@@ -62,10 +62,10 @@ def test_fast_float_returns_nan_alternate_if_nan_option_is_given() -> None:
def test_fast_float_converts_float_string_to_float_example() -> None:
assert fast_float("45.8") == 45.8
assert fast_float("-45") == -45.0
- assert fast_float("45.8e-2", key=len) == 45.8e-2
- assert isnan(cast(float, fast_float("nan")))
- assert isnan(cast(float, fast_float("+nan")))
- assert isnan(cast(float, fast_float("-NaN")))
+ assert fast_float("45.8e-2", key=lambda x: x.upper()) == 45.8e-2
+ assert isinf(cast(float, fast_float("nan")))
+ assert isinf(cast(float, fast_float("+nan")))
+ assert isinf(cast(float, fast_float("-NaN")))
assert fast_float("۱۲.۱۲") == 12.12
assert fast_float("-۱۲.۱۲") == -12.12
@@ -85,12 +85,12 @@ def test_fast_float_leaves_string_as_is(x: str) -> None:
def test_fast_float_with_key_applies_to_string_example() -> None:
- assert fast_float("invalid", key=len) == len("invalid")
+ assert fast_float("invalid", key=lambda x: x.upper()) == "INVALID"
@given(text().filter(not_a_float).filter(bool))
def test_fast_float_with_key_applies_to_string(x: str) -> None:
- assert fast_float(x, key=len) == len(x)
+ assert fast_float(x, key=lambda x: x.upper()) == x.upper()
def test_fast_int_leaves_float_string_as_is_example() -> None:
@@ -126,9 +126,9 @@ def test_fast_int_leaves_string_as_is(x: str) -> None:
def test_fast_int_with_key_applies_to_string_example() -> None:
- assert fast_int("invalid", key=len) == len("invalid")
+ assert fast_int("invalid", key=lambda x: x.upper()) == "INVALID"
@given(text().filter(not_an_int).filter(bool))
def test_fast_int_with_key_applies_to_string(x: str) -> None:
- assert fast_int(x, key=len) == len(x)
+ assert fast_int(x, key=lambda x: x.upper()) == x.upper()
diff --git a/tests/test_natsorted.py b/tests/test_natsorted.py
index 4a64a27..bd1cc39 100644
--- a/tests/test_natsorted.py
+++ b/tests/test_natsorted.py
@@ -4,7 +4,9 @@ Here are a collection of examples of how this module can be used.
See the README or the natsort homepage for more details.
"""
+import math
from operator import itemgetter
+from pathlib import PurePosixPath
from typing import List, Tuple, Union
import pytest
@@ -84,6 +86,15 @@ def test_natsorted_can_sort_as_version_numbers() -> None:
assert natsorted(given) == expected
+def test_natsorted_can_sorts_paths_same_as_strings() -> None:
+ paths = [
+ PurePosixPath("a/1/something"),
+ PurePosixPath("a/2/something"),
+ PurePosixPath("a/10/something"),
+ ]
+ assert [str(p) for p in natsorted(paths)] == natsorted([str(p) for p in paths])
+
+
@pytest.mark.parametrize(
"alg, expected",
[
@@ -100,19 +111,29 @@ def test_natsorted_handles_mixed_types(
@pytest.mark.parametrize(
- "alg, expected, slc",
+ "alg, expected",
[
- (ns.DEFAULT, [float("nan"), 5, "25", 1e40], slice(1, None)),
- (ns.NANLAST, [5, "25", 1e40, float("nan")], slice(None, 3)),
+ (ns.DEFAULT, [float("nan"), None, float("-inf"), 5, "25", 1e40, float("inf")]),
+ (ns.NANLAST, [float("-inf"), 5, "25", 1e40, float("inf"), None, float("nan")]),
],
)
-def test_natsorted_handles_nan(
- alg: NSType, expected: List[Union[str, float, int]], slc: slice
+def test_natsorted_consistent_ordering_with_nan_and_friends(
+ alg: NSType, expected: List[Union[str, float, None, int]]
) -> None:
- given: List[Union[str, float, int]] = ["25", 5, float("nan"), 1e40]
- # The slice is because NaN != NaN
- # noinspection PyUnresolvedReferences
- assert natsorted(given, alg=alg)[slc] == expected[slc]
+ sentinel = math.pi
+ expected = [sentinel if x != x else x for x in expected]
+ given: List[Union[str, float, None, int]] = [
+ float("inf"),
+ float("-inf"),
+ "25",
+ 5,
+ float("nan"),
+ 1e40,
+ None,
+ ]
+ result = natsorted(given, alg=alg)
+ result = [sentinel if x != x else x for x in result]
+ assert result == expected
def test_natsorted_with_mixed_bytes_and_str_input_raises_type_error() -> None:
@@ -182,6 +203,21 @@ def test_natsorted_handles_numbers_and_filesystem_paths_simultaneously() -> None
assert natsorted(given, alg=ns.PATH) == expected
+def test_natsorted_path_extensions_heuristic() -> None:
+ # https://github.com/SethMMorton/natsort/issues/145
+ given = [
+ "Try.Me.Bug - 09 - One.Two.Three.[text].mkv",
+ "Try.Me.Bug - 07 - One.Two.5.[text].mkv",
+ "Try.Me.Bug - 08 - One.Two.Three[text].mkv",
+ ]
+ expected = [
+ "Try.Me.Bug - 07 - One.Two.5.[text].mkv",
+ "Try.Me.Bug - 08 - One.Two.Three[text].mkv",
+ "Try.Me.Bug - 09 - One.Two.Three.[text].mkv",
+ ]
+ assert natsorted(given, alg=ns.PATH) == expected
+
+
@pytest.mark.parametrize(
"alg, expected",
[
@@ -342,3 +378,32 @@ def test_natsorted_sorts_mixed_ascii_and_non_ascii_numbers() -> None:
"street ۱۲",
]
assert natsorted(given, alg=ns.IGNORECASE) == expected
+
+
+def test_natsort_sorts_consistently_with_presort() -> None:
+ # Demonstrate the problem:
+ # Sorting is order-dependent for values that have different
+ # string representations are equiavlent numerically.
+ given = ["a01", "a1.4500", "a1", "a1.45"]
+ expected = ["a01", "a1", "a1.4500", "a1.45"]
+ result = natsorted(given, alg=ns.FLOAT)
+ assert result == expected
+
+ given = ["a1", "a1.45", "a01", "a1.4500"]
+ expected = ["a1", "a01", "a1.45", "a1.4500"]
+ result = natsorted(given, alg=ns.FLOAT)
+ assert result == expected
+
+ # The solution - use "presort" which will sort the
+ # input by its string representation before sorting
+ # with natsorted, which gives consitent results even
+ # if the numeric representation is identical
+ expected = ["a01", "a1", "a1.45", "a1.4500"]
+
+ given = ["a01", "a1.4500", "a1", "a1.45"]
+ result = natsorted(given, alg=ns.FLOAT | ns.PRESORT)
+ assert result == expected
+
+ given = ["a1", "a1.45", "a01", "a1.4500"]
+ result = natsorted(given, alg=ns.FLOAT | ns.PRESORT)
+ assert result == expected
diff --git a/tests/test_natsorted_convenience.py b/tests/test_natsorted_convenience.py
index 0b2cd75..81bdf5c 100644
--- a/tests/test_natsorted_convenience.py
+++ b/tests/test_natsorted_convenience.py
@@ -88,6 +88,13 @@ def test_index_natsorted_applies_key_function_before_sorting() -> None:
assert index_natsorted(given, key=itemgetter(1)) == expected
+def test_index_natsorted_can_presort() -> None:
+ expected = [2, 0, 3, 1]
+ given = ["a1", "a1.4500", "a01", "a1.45"]
+ result = index_natsorted(given, alg=ns.FLOAT | ns.PRESORT)
+ assert result == expected
+
+
def test_index_realsorted_is_identical_to_index_natsorted_with_real_alg(
float_list: List[str],
) -> None:
diff --git a/tests/test_ns_enum.py b/tests/test_ns_enum.py
index 7a30718..c950812 100644
--- a/tests/test_ns_enum.py
+++ b/tests/test_ns_enum.py
@@ -18,6 +18,7 @@ from natsort import ns
("NANLAST", 0x0400),
("COMPATIBILITYNORMALIZE", 0x0800),
("NUMAFTER", 0x1000),
+ ("PRESORT", 0x2000),
("DEFAULT", 0x0000),
("INT", 0x0000),
("UNSIGNED", 0x0000),
@@ -42,6 +43,7 @@ from natsort import ns
("NL", 0x0400),
("CN", 0x0800),
("NA", 0x1000),
+ ("PS", 0x2000),
],
)
def test_ns_enum(given: str, expected: int) -> None:
diff --git a/tests/test_os_sorted.py b/tests/test_os_sorted.py
index d0ecc79..c29c110 100644
--- a/tests/test_os_sorted.py
+++ b/tests/test_os_sorted.py
@@ -3,7 +3,6 @@
Testing for the OS sorting
"""
import platform
-from typing import cast
import natsort
import pytest
@@ -44,7 +43,14 @@ def test_os_sorted_misc_no_fail() -> None:
def test_os_sorted_key() -> None:
given = ["foo0", "foo2", "goo1"]
expected = ["foo0", "goo1", "foo2"]
- result = natsort.os_sorted(given, key=lambda x: cast(str, x).replace("g", "f"))
+ result = natsort.os_sorted(given, key=lambda x: x.replace("g", "f"))
+ assert result == expected
+
+
+def test_os_sorted_can_presort() -> None:
+ given = ["a1", "a01"]
+ expected = ["a01", "a1"]
+ result = natsort.os_sorted(given, presort=True)
assert result == expected
diff --git a/tests/test_parse_number_function.py b/tests/test_parse_number_function.py
index e5f417d..24ee714 100644
--- a/tests/test_parse_number_function.py
+++ b/tests/test_parse_number_function.py
@@ -7,7 +7,7 @@ import pytest
from hypothesis import given
from hypothesis.strategies import floats, integers
from natsort.ns_enum import NSType, ns
-from natsort.utils import MaybeNumTransformer, parse_number_or_none_factory
+from natsort.utils import NumTransformer, parse_number_or_none_factory
@pytest.mark.usefixtures("with_locale_en_us")
@@ -20,9 +20,9 @@ from natsort.utils import MaybeNumTransformer, parse_number_or_none_factory
(ns.PATH | ns.UNGROUPLETTERS | ns.LOCALE, lambda x: ((("xx",), ("", x)),)),
],
)
-@given(x=floats(allow_nan=False) | integers())
+@given(x=floats(allow_nan=False, allow_infinity=False) | integers())
def test_parse_number_factory_makes_function_that_returns_tuple(
- x: Union[float, int], alg: NSType, example_func: MaybeNumTransformer
+ x: Union[float, int], alg: NSType, example_func: NumTransformer
) -> None:
parse_number_func = parse_number_or_none_factory(alg, "", "xx")
assert parse_number_func(x) == example_func(x)
@@ -32,10 +32,20 @@ def test_parse_number_factory_makes_function_that_returns_tuple(
"alg, x, result",
[
(ns.DEFAULT, 57, ("", 57)),
- (ns.DEFAULT, float("nan"), ("", float("-inf"))), # NaN transformed to -infinity
- (ns.NANLAST, float("nan"), ("", float("+inf"))), # NANLAST makes it +infinity
- (ns.DEFAULT, None, ("", float("-inf"))), # None transformed to -infinity
- (ns.NANLAST, None, ("", float("+inf"))), # NANLAST makes it +infinity
+ (
+ ns.DEFAULT,
+ float("nan"),
+ ("", float("-inf"), "1"),
+ ), # NaN transformed to -infinity
+ (
+ ns.NANLAST,
+ float("nan"),
+ ("", float("+inf"), "3"),
+ ), # NANLAST makes it +infinity
+ (ns.DEFAULT, None, ("", float("-inf"), "2")), # None transformed to -infinity
+ (ns.NANLAST, None, ("", float("+inf"), "2")), # NANLAST makes it +infinity
+ (ns.DEFAULT, float("-inf"), ("", float("-inf"), "3")),
+ (ns.NANLAST, float("+inf"), ("", float("+inf"), "1")),
],
)
def test_parse_number_factory_treats_nan_and_none_special(
diff --git a/tests/test_parse_string_function.py b/tests/test_parse_string_function.py
index 653a065..d2d33a4 100644
--- a/tests/test_parse_string_function.py
+++ b/tests/test_parse_string_function.py
@@ -7,7 +7,7 @@ from typing import Any, Callable, Iterable, List, Tuple, Union
import pytest
from hypothesis import given
from hypothesis.strategies import floats, integers, lists, text
-from natsort.compat.fastnumbers import fast_float
+from natsort.compat.fastnumbers import try_float
from natsort.ns_enum import NSType, NS_DUMB, ns
from natsort.utils import (
FinalTransform,
@@ -46,7 +46,7 @@ def parse_string_func_factory(alg: NSType) -> StrParser:
sep,
NumRegex.int_nosign().split,
input_transform,
- fast_float,
+ lambda x: try_float(x, map=True),
final_transform,
)
diff --git a/tests/test_string_component_transform_factory.py b/tests/test_string_component_transform_factory.py
index 99df7ea..40b4d34 100644
--- a/tests/test_string_component_transform_factory.py
+++ b/tests/test_string_component_transform_factory.py
@@ -5,9 +5,9 @@ from functools import partial
from typing import Any, Callable, FrozenSet, Union
import pytest
-from hypothesis import example, given
+from hypothesis import assume, example, given
from hypothesis.strategies import floats, integers, text
-from natsort.compat.fastnumbers import fast_float, fast_int
+from natsort.compat.fastnumbers import try_float, try_int
from natsort.compat.locale import get_strxfrm
from natsort.ns_enum import NSType, NS_DUMB, ns
from natsort.utils import groupletters, string_component_transform_factory
@@ -32,34 +32,54 @@ def no_null(x: str) -> bool:
return "\0" not in x
+def input_is_ok_with_locale(x: str) -> bool:
+ """Ensure this input won't cause locale.strxfrm to barf"""
+ # On FreeBSD, locale.strxfrm raises an OSError on input like 'Å'.
+ # You read that right - an *OSError* for invalid input.
+ # We cannot really fix that, so we just filter out any value
+ # that could cause locale.strxfrm to barf with this function.
+ try:
+ get_strxfrm()(x)
+ except OSError:
+ return False
+ else:
+ return True
+
+
@pytest.mark.parametrize(
"alg, example_func",
[
- (ns.INT, fast_int),
- (ns.DEFAULT, fast_int),
- (ns.FLOAT, partial(fast_float, nan=float("-inf"))),
- (ns.FLOAT | ns.NANLAST, partial(fast_float, nan=float("+inf"))),
- (ns.GROUPLETTERS, partial(fast_int, key=groupletters)),
- (ns.LOCALE, partial(fast_int, key=lambda x: get_strxfrm()(x))),
+ (ns.INT, partial(try_int, map=True)),
+ (ns.DEFAULT, partial(try_int, map=True)),
+ (ns.FLOAT, partial(try_float, map=True, nan=float("-inf"))),
+ (ns.FLOAT | ns.NANLAST, partial(try_float, map=True, nan=float("+inf"))),
+ (ns.GROUPLETTERS, partial(try_int, map=True, on_fail=groupletters)),
+ (ns.LOCALE, partial(try_int, map=True, on_fail=lambda x: get_strxfrm()(x))),
(
ns.GROUPLETTERS | ns.LOCALE,
- partial(fast_int, key=lambda x: get_strxfrm()(groupletters(x))),
+ partial(
+ try_int, map=True, on_fail=lambda x: get_strxfrm()(groupletters(x))
+ ),
),
(
NS_DUMB | ns.LOCALE,
- partial(fast_int, key=lambda x: get_strxfrm()(groupletters(x))),
+ partial(
+ try_int, map=True, on_fail=lambda x: get_strxfrm()(groupletters(x))
+ ),
),
(
ns.GROUPLETTERS | ns.LOCALE | ns.FLOAT | ns.NANLAST,
partial(
- fast_float,
- key=lambda x: get_strxfrm()(groupletters(x)),
+ try_float,
+ map=True,
+ on_fail=lambda x: get_strxfrm()(groupletters(x)),
nan=float("+inf"),
),
),
],
)
@example(x=float("nan"))
+@example(x="Å")
@given(
x=integers()
| floats()
@@ -70,8 +90,10 @@ def test_string_component_transform_factory(
x: Union[str, float, int], alg: NSType, example_func: Callable[[str], Any]
) -> None:
string_component_transform_func = string_component_transform_factory(alg)
+ x = str(x)
+ assume(input_is_ok_with_locale(x)) # handle broken locale lib on BSD.
try:
- assert string_component_transform_func(str(x)) == example_func(str(x))
+ assert list(string_component_transform_func(x)) == list(example_func(x))
except ValueError as e: # handle broken locale lib on BSD.
if "is not in range" not in str(e):
raise
diff --git a/tests/test_unicode_numbers.py b/tests/test_unicode_numbers.py
index eb71125..be867ee 100644
--- a/tests/test_unicode_numbers.py
+++ b/tests/test_unicode_numbers.py
@@ -34,7 +34,9 @@ def test_decimal_chars_contains_only_valid_unicode_decimal_characters() -> None:
assert unicodedata.decimal(a, None) is not None
-def test_numeric_chars_contains_all_valid_unicode_numeric_and_digit_characters() -> None:
+def test_numeric_chars_contains_all_valid_unicode_numeric_and_digit_characters() -> (
+ None
+):
set_numeric_chars = set(numeric_chars)
set_digit_chars = set(digit_chars)
set_decimal_chars = set(decimal_chars)
@@ -67,7 +69,8 @@ This can be addressed by running dev/generate_new_unicode_numbers.py with the cu
version of Python.
It would be much appreciated if you would submit a Pull Request to the natsort
repository (https://github.com/SethMMorton/natsort) with the resulting change.
-"""
+""",
+ stacklevel=2,
)
diff --git a/tests/test_utils.py b/tests/test_utils.py
index bb229b9..b140682 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -6,7 +6,7 @@ import pathlib
import string
from itertools import chain
from operator import neg as op_neg
-from typing import List, Pattern, Union
+from typing import List, Pattern, Tuple, Union
import pytest
from hypothesis import given
@@ -155,9 +155,26 @@ def test_path_splitter_splits_path_string_by_sep(x: List[str]) -> None:
assert tuple(utils.path_splitter(z)) == tuple(pathlib.Path(z).parts)
-def test_path_splitter_splits_path_string_by_sep_and_removes_extension_example() -> None:
- given = "/this/is/a/path/file.x1.10.tar.gz"
- expected = (os.sep, "this", "is", "a", "path", "file.x1.10", ".tar", ".gz")
+@pytest.mark.parametrize(
+ "given, expected",
+ [
+ (
+ "/this/is/a/path/file.x1.10.tar.gz",
+ (os.sep, "this", "is", "a", "path", "file.x1.10", ".tar", ".gz"),
+ ),
+ (
+ "/this/is/a/path/file.x1.10.tar",
+ (os.sep, "this", "is", "a", "path", "file.x1.10", ".tar"),
+ ),
+ (
+ "/this/is/a/path/file.x1.threethousand.tar",
+ (os.sep, "this", "is", "a", "path", "file.x1.threethousand", ".tar"),
+ ),
+ ],
+)
+def test_path_splitter_splits_path_string_by_sep_and_removes_extension_example(
+ given: str, expected: Tuple[str, ...]
+) -> None:
assert tuple(utils.path_splitter(given)) == tuple(expected)
diff --git a/tox.ini b/tox.ini
index eb8b615..3fa2378 100644
--- a/tox.ini
+++ b/tox.ini
@@ -5,8 +5,8 @@
[tox]
envlist =
- flake8, mypy, py36, py37, py38, py39, py310
-# Other valid evironments are:
+ flake8, mypy, py37, py38, py39, py310, py311
+# Other valid environments are:
# docs
# release
# clean
@@ -25,7 +25,6 @@ deps =
pytest-cov
pytest-mock
hypothesis
- semver
extras =
{env:WITH_EXTRAS:}
commands =
@@ -46,7 +45,8 @@ deps =
check-manifest
twine
commands =
- {envpython} setup.py sdist bdist_wheel
+ {envpython} setup.py sdist
+ pip wheel . -w dist
flake8
check-manifest --ignore ".github*,*.md,.coveragerc"
twine check dist/*
@@ -59,7 +59,8 @@ deps =
hypothesis
pytest
pytest-mock
- fastnumbers
+ fastnumbers>=5.0.1
+ typing_extensions
commands =
mypy --strict natsort tests
skip_install = true
@@ -104,9 +105,8 @@ skip_install = true
# Get GitHub actions to run the correct tox environment
[gh-actions]
python =
- 3.5: py35
- 3.6: py36
3.7: py37
3.8: py38
3.9: py39
3.10: py310
+ 3.11: py311
Debdiff
[The following lists of changes regard files as different if they have different names, permissions or owners.]
Files in second set of .debs but not in first
-rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/PKG-INFO -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/dependency_links.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/entry_points.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/not-zip-safe -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/requires.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.3.1.egg-info/top_level.txt
Files in first set of .debs but not in second
-rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/PKG-INFO -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/dependency_links.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/entry_points.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/not-zip-safe -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/requires.txt -rw-r--r-- root/root /usr/lib/python3/dist-packages/natsort-8.0.2.egg-info/top_level.txt -rw-r--r-- root/root /usr/share/doc/python-natsort-doc/html/_images/special_cases_everywhere.jpg
No differences were encountered between the control files of package python-natsort-doc
No differences were encountered between the control files of package python3-natsort