Description
-------------------------------------------------------------------------- pathos: parallel graph management and execution in heterogeneous computing -------------------------------------------------------------------------- About the Pathos Framework ========================== ``pathos`` is a framework for heterogeneous computing. It provides a consistent high-level interface for configuring and launching parallel computations across heterogeneous resources. ``pathos`` provides configurable launchers for parallel and distributed computing, where each launcher contains the syntactic logic to configure and launch jobs in an execution environment. Examples of launchers that plug into ``pathos`` are: a queue-less MPI-based launcher (in ``pyina``), a ssh-based launcher (in ``pathos``), and a multi-process launcher (in ``multiprocess``). ``pathos`` provides a consistent interface for parallel and/or distributed versions of ``map`` and ``apply`` for each launcher, thus lowering the barrier for users to extend their code to parallel and/or distributed resources. The guiding design principle behind ``pathos`` is that ``map`` and ``apply`` should be drop-in replacements in otherwise serial code, and thus switching to one or more of the ``pathos`` launchers is all that is needed to enable code to leverage the selected parallel or distributed computing resource. This not only greatly reduces the time to convert a code to parallel, but it also enables a single code-base to be maintained instead of requiring parallel, serial, and distributed versions of a code. ``pathos`` maps can be nested, thus hierarchical heterogeneous computing is possible by merely selecting the desired hierarchy of ``map`` and ``pipe`` (``apply``) objects. The ``pathos`` framework is composed of several interoperating packages: - ``dill``: serialize all of Python - ``pox``: utilities for filesystem exploration and automated builds - ``klepto``: persistent caching to memory, disk, or database - ``multiprocess``: better multiprocessing and multithreading in Python - ``ppft``: distributed and parallel Python - ``pyina``: MPI parallel ``map`` and cluster scheduling - ``pathos``: graph management and execution in heterogeneous computing About Pathos ============ The ``pathos`` package provides a few basic tools to make parallel and distributed computing more accessible to the end user. The goal of ``pathos`` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. ``pathos`` provides methods for configuring, launching, monitoring, and controlling a service on a remote host. One of the most basic features of ``pathos`` is the ability to configure and launch a RPC-based service on a remote host. ``pathos`` seeds the remote host with the ``portpicker`` script, which allows the remote host to inform the localhost of a port that is available for communication. Beyond the ability to establish a RPC service, and then post requests, is the ability to launch code in parallel. Unlike parallel computing performed at the node level (typically with MPI), ``pathos`` enables the user to launch jobs in parallel across heterogeneous distributed resources. ``pathos`` provides distributed ``map`` and ``pipe`` algorithms, where a mix of local processors and distributed workers can be selected. ``pathos`` also provides a very basic automated load balancing service, as well as the ability for the user to directly select the resources. The high-level ``pool.map`` interface, yields a ``map`` implementation that hides the RPC internals from the user. With ``pool.map``, the user can launch their code in parallel, and as a distributed service, using standard Python and without writing a line of server or parallel batch code. RPC servers and communication in general is known to be insecure. However, instead of attempting to make the RPC communication itself secure, ``pathos`` provides the ability to automatically wrap any distributes service or communication in a ssh-tunnel. Ssh is a universally trusted method. Using ssh-tunnels, ``pathos`` has launched several distributed calculations on national lab clusters, and to date has performed test calculations that utilize node-to-node communication between several national lab clusters and a user's laptop. ``pathos`` allows the user to configure and launch at a very atomistic level, through raw access to ssh and scp. ``pathos`` is the core of a Python framework for heterogeneous computing. ``pathos`` is in active development, so any user feedback, bug reports, comments, or suggestions are highly appreciated. A list of issues is located at https://github.com/uqfoundation/pathos/issues, with a legacy list maintained at https://uqfoundation.github.io/project/pathos/query. Major Features ============== ``pathos`` provides a configurable distributed parallel ``map`` interface to launching RPC service calls, with: - a ``map`` interf
Release History
| Version | Changes | Urgency | Date |
|---|---|---|---|
| 0.3.5 | Imported from PyPI (0.3.5) | Low | 4/21/2026 |
| 0.3.4 | # 0.3.4 Release Notes `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, >=3.8` `- dill, >=0.4.0` `- pox, >=0.3.6` `- ppft, >=1.7.7` `- multiprocess, >=0.70.18` `pathos` is licensed under 3-clause BSD | Low | 4/17/2025 |
| 0.3.3 | # 0.3.3 Release Notes `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, >=3.8` `- dill, >=0.3.9` `- pox, >=0.3.5` `- ppft, >=1.7.6.9` `- multiprocess, >=0.70.17` `pathos` is licensed under 3-clause B | Low | 9/30/2024 |
| 0.3.2 | # 0.3.2 Release Notes `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, >=3.8` `- dill, >=0.3.8` `- pox, >=0.3.4` `- ppft, >=1.7.6.8` `- multiprocess, >=0.70.16` `pathos` is licensed under 3-clause B | Low | 1/28/2024 |
| pathos-0.3.1 | # pathos 0.3.1 Release Notes `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, >=3.7` `- dill, >=0.3.7` `- pox, >=0.3.3` `- ppft, >=1.7.6.7` `- multiprocess, >=0.70.15` `pathos` is licensed under 3-c | Low | 7/23/2023 |
| pathos-0.3.0 | # pathos 0.3.0 Release Notes `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, >=3.7` `- dill, >=0.3.6` `- pox, >=0.3.2` `- ppft, >=1.7.6.6` `- multiprocess, >=0.70.14` `pathos` is licensed under 3-c | Low | 10/24/2022 |
| pathos-0.2.9 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `pip`: `$ pip install pathos` `pathos` requires: `- python or pypy, ==2.7 or >=3.7` `- dill, >=0.3.5.1` `- pox, >=0.3.1` `- ppft, >=1.7.6.5` `- multiprocess, >=0.70.13` `pathos` is licensed under 3-clause BSD: ``` | Low | 5/24/2022 |
| pathos-0.2.8 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version == 2.7 *or* version >= 3.6` `- dill, version >= 0.3.4` `- pox, version >= 0.3.0` `- ppft, version >= 1.6.6.4` `- multiprocess >= 0.70 | Low | 6/14/2021 |
| pathos-0.2.7 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version == 2.7 *or* version >= 3.5` `- dill, version >= 0.3.3` `- pox, version >= 0.2.9` `- ppft, version >= 1.6.6.3` `- multiprocess >= 0.70 | Low | 11/2/2020 |
| pathos-0.2.6 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version == 2.7 *or* version >= 3.5` `- dill, version >= 0.3.2` `- pox, version >= 0.2.8` `- ppft, version >= 1.6.6.2` `- multiprocess >= 0.70 | Low | 6/16/2020 |
| pathos-0.2.5 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.3.1` `- pox, version >= 0.2.7` `- ppft, version >= 1.6.6.1` `- multiprocess >= 0.70. | Low | 9/28/2019 |
| pathos-0.2.4 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.3.0` `- pox, version >= 0.2.6` `- ppft, version >= 1.6.6.1` `- multiprocess >= 0.70. | Low | 6/27/2019 |
| pathos-0.2.3 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.2.9` `- pox, version >= 0.2.5` `- ppft, version >= 1.6.4.9` `- multiprocess >= 0.70. | Low | 1/22/2019 |
| pathos-0.2.2.1 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.2.8.2` `- pox, version >= 0.2.4` `- ppft, version >= 1.6.4.8` `- multiprocess >= 0.7 | Low | 7/5/2018 |
| pathos-0.2.2 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ pip install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.2.8.1` `- pox, version >= 0.2.4` `- ppft, version >= 1.6.4.8` `- multiprocess >= 0.7 | Low | 6/21/2018 |
| pathos-0.2.1 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ easy_install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.2.7` `- pox, version >= 0.2.3` `- ppft, version >= 1.6.4.7` `- multiprocess >= 0.70 | Low | 7/7/2017 |
| pathos-0.2.0 | `pathos` provides a few basic tools to make parallel and distributed computing more accessable to the end user. The goal of `pathos` is to enable the user to extend their own code to parallel and distributed computing with minimal refactoring. `pathos` installs with `easy_install` or `pip`: `$ easy_install pathos` `pathos` requires: `- python, version >=2.6 *or* version >= 3.1` `- dill, version >= 0.2.5` `- pox, version >= 0.2.2` `- ppft, version >= 1.6.4.5` `- multiprocess >= 0.70.4` Optio | Low | 7/18/2016 |