Can There Be Too Much Parallelism?

title: Can There Be Too Much Parallelism?
use_katex: False
class: title-slide

# Can There Be Too Much Parallelism?

.larger[Thomas J. Fan] 
<a href="https://www.github.com/thomasjpfan" target="_blank" class="title-link">@thomasjpfan</a>
<a class="this-talk-link", href="https://github.com/thomasjpfan/scipy-2023-too-parallel" target="_blank">

![:scale 15%](images/QuansightLabs_logo_V2.png)

This talk on Github: thomasjpfan/scipy-2023-too-parallel</a>

---

# Yes 👀

---

# User?

![:scale 100%](images/python-libraries.png)

---

# Developer?

![:scale 100%](images/python-libraries.png)

---

# My Perspective

.g[
.g-6[
## Parallelism in Scikit-learn
- BLAS through SciPy
- OpenMP + Cython
- Python Multi-Threading
- Python Multi-Processing
]
.g-6[
![](images/scikit-learn-better.svg)
]
]

---

# Scope

---

.g.g-middle[
.g-6.larger[
## APIs 💻
## Interactions 🥂
## Defaults 🌈
]
.g-6[
![](images/stethoscope.jpg)
]
]

---

# APIs 💻

![:scale 70%](images/plane.jpg)

---

.g.g-middle[
.g-8[

# Environment Variables 🌲

.larger[
- **OpenMP**: `OMP_NUM_THREADS`
- **MKL**: `MKL_NUM_THREADS`
- **OpenBLAS**: `OPENBLAS_NUM_THREADS`
]
]
.g-4[
![](images/common_env.jpg)
]
]

---

.g.g-middle[
.g-8[

# Environment Variables 🌲

.larger[
- **OpenMP**: `OMP_NUM_THREADS`
- **MKL**: `MKL_NUM_THREADS`
- **OpenBLAS**: `OPENBLAS_NUM_THREADS`
- **Polars**: `POLARS_MAX_THREADS`
- **Numba**: `NUMBA_NUM_THREADS`
- **macOS accelerate**: `VECLIB_MAXIMUM_THREADS`
- **numexpr**: `NUMEXPR_NUM_THREADS`
]
]
.g-4[
![](images/common_env.jpg)
]
]

---

# Global Configuration 🌎

.g.g-middle[
.g-8[
.larger[
- `torch.set_num_threads`
- `numba.set_num_threads`
- `threadpoolctl.threadpool_limits`
- `cv.setNumThreads`
]
]
.g-4[
![](images/globe.jpg)
]
]

---

# Block Configuration 🧱
## `threadpoolctl`

```python
from threadpoolctl import threadpool_limits
import numpy as np

*with threadpool_limits(limits=2):
    a = np.random.randn(1000, 1000)
    a_squared = a @ a
```

---

# Call-site ☎️

.g.g-middle[
.g-8[
.larger[
- **scikit-learn**: `n_jobs`
- **SciPy**: `workers`
- **PyTorch DataLoader**: `num_workers`
- **Python**: `max_workers`
]

]
.g-4[
![](images/phone.jpg)
]
]

---

.g.g-middle[
.g-6[
# APIs 💻
.larger[
- Environment Variables 🌲
- Global Configuration 🌎
- Block Configuration 🧱
- Call-site ☎️
]
]
.g-6.g-center[
![:scale 80%](images/plane.jpg)
]
]

---

# Proposal: Consistent APIs 🔮

.g[
.g-6[
## Now
```
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export POLARS_MAX_THREADS=1
export NUMEXPR_NUM_THREADS=1
```
]
.g-6[
]
]

---

# Proposal: Consistent APIs 🔮

.g[
.g-6[
## Now
```
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export POLARS_MAX_THREADS=1
export NUMEXPR_NUM_THREADS=1
```
]
.g-6[
## Future 🚀
### Pragmatic
```
export OMP_NUM_THREADS=1
```
]
]

---

# Proposal: Consistent APIs 🔮

### Better ☀️

```
export GOTO_NUM_THREADS=1
```
]
]

---

# Proposal 🔮

## Recognize more threadpools in `threadpoolctl`

---

# Proposal 🔮

.g[
.g-6[
## Now
- **scikit-learn**: `n_jobs`
- **SciPy**: `workers`
- **PyTorch DataLoader**: `num_workers`
- **Python**: `max_workers`
]
.g-6[
]
]

---

# Proposal 🔮

.g[
.g-6[
## Now
- **scikit-learn**: `n_jobs`
- **SciPy**: `workers`
- **PyTorch DataLoader**: `num_workers`
- **Python**: `max_workers`
]
.g-6[
## Future 🚀
- Everyone uses `workers`
]
]

---

## Interactions 🥂

![:scale 80%](images/network.jpg)

---

# Oversubscription 💥
## Python + native threading 🐍 + 🧵

```python
from scipy import optimize

optimize.brute(
*    computation_that_uses_8_cores, ...
*    workers=8
)
```

---

# Current workarounds 🩹
## Dask ![:scale 10%](images/dask-logo.svg)

![:scale 80%](images/dask-oversub.jpg)

[Source](https://docs.dask.org/en/stable/array-best-practices.html#avoid-oversubscribing-threads)

---

![:scale 20%](images/ray-logo.png)
![](images/ray-oversub.jpg)

[Source](https://docs.ray.io/en/latest/serve/scaling-and-resource-allocation.html#configuring-parallelism-with-omp-num-threads)

---

# PyTorch's DataLoader

.g.g-middle[
.g-8[
```python
from torch.utils.data import DataLoader

dl = DataLoader(..., num_workers=8)

# torch/utils/data/_utils/worker.py
def _worker_loop(...):
    ...
*   torch.set_num_threads(1)
```
]
.g-4.center[
![](images/pytorch.tiff)
]
]

[Source]()

---

# scikit-learn

.g.g-middle[
.g-8[
```python
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV
from sklearn.ensemble import HalvingRandomSearchCV

*clf = HistGradientBoostingClassifier()
search = HalvingGridSearchCV(
    clf,
    param_distributions,
*   n_jos=8
)

search.fit(X, y)
```
]
.g-4.center[
![:scale 90%](images/scikit-learn-better.svg)
]
]

---

# Multiple Parallel Abstractions 🧵 + 🧶

- Python multiprocessing using `fork` + GCC OpenMP: **stalls**

- Intel OpenMP + LLVM OpenMP on Linux: **stalls**

- Multiple OpenBLAS libraries: **sometimes slower**

- Read more at: [thomasjpfan.github.io/parallelism-python-libraries-design/](https://thomasjpfan.github.io/parallelism-python-libraries-design/)

---

# Multiple Parallel Abstractions 🧵 + 🧶
## Using more than one parallel backends 🤯

![](images/combining-native+multi.jpg)

Sources: [polars](https://pola-rs.github.io/polars-book/user-guide/howcani/multiprocessing.html),
[numba](https://numba.pydata.org/numba-doc/latest/user/threading-layer.html),
[scikit-learn](https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux),
[pandas](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#caveats)

---

# Proposal: Catch issues early 🔮

.g.g-middle[
.g-10[
![](images/numba-fork-openmp.jpg)
]
.g-2.center[
![](images/numba.jpg)
]
]
[Source](https://github.com/numba/numba/blob/249c8ff3206928b486346443ec148508f8c25f8e/numba/np/ufunc/omppool.cpp#L119-L121)

## Not a full solution 🩹

---

# Multiple Native threading libraries 🧵 + 🧶

[Source](https://www.slideshare.net/RalfGommers/parallelism-in-a-numpybased-program)

---

# Multiple Native threading libraries 🧵 + 🧶
## CPU Waiting ⏳

```python
for n_iter in range(100):
    UV = U @ V.T             # Use OpenBLAS with pthreads
    compute_with_openmp(UV)  # Use OpenMP
```

[xianyi/OpenBLAS#3187](https://github.com/xianyi/OpenBLAS/issues/3187)

---

# Current Workaround 🩹

.g.g-middle[
.g-6[
## Conda-forge + OpenMP
]
.g-6.center[
![](images/conda-forge-square.png)
]
]

---

# Current Workaround  🩹

[Source](https://www.slideshare.net/RalfGommers/parallelism-in-a-numpybased-program)

---

# Proposal 🔮

## Ship PyPI wheels for OpenMP

![:scale 80%](images/intel-openmp.jpg)

## Not a full solution 🩹

.g.g-center.g-middle[
.g-6[
![:scale 70%](images/polars.tiff)
]
.g-6[
![:scale 30%](images/scipy.svg)
]
]

---

# Defaults 🌈

![:scale 85%](images/path.jpg)

---

# NumPy

.g.g-middle[
.g-8[
```python
import numpy as np

out = np.sum(A_array, axis=1)
```
]
.g-4.center[
![](images/numpy.tiff)
]
]

.alert.bold.center[🐌 One Core 🐌]

---

# NumPy matmul

.g.g-middle[
.g-8[
```python
import numpy as np

out = A_array @ B_array
```
]
.g-4.center[
![](images/numpy.tiff)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# NumPy matmul (Configuration)

```python
out = A_array @ B_array
```
]
.g-4.center[
![](images/numpy.tiff)
]
]

---

# NumPy matmul (Configuration)

```python
out = A_array @ B_array
```

## `threadpoolctl`

```python
from threadpoolctl import threadpool_limits

with threadpool_limits(limits=1):
    out = A_array @ B_array
```
]
.g-4.center[
![](images/numpy.tiff)
]
]

---

# PyTorch

*out = torch.sum(A_tensor, axis=1)
```
]
.g-4.center[
![](images/pytorch.tiff)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# PyTorch (Configuration)

```python
with threadpool_limits(limits=2):
    out = torch.sum(A_tensor, axis=1)
```
]
.g-4.center[
![](images/pytorch.tiff)
]
]

---

# PyTorch (Configuration)

```python
with threadpool_limits(limits=2):
    out = torch.sum(A_tensor, axis=1)
```

- PyTorch function

```python
import torch
*torch.set_num_threads(2)

out = torch.sum(A_tensor, axis=1)
```
]
.g-4.center[
![](images/pytorch.tiff)
]
]

---

# pandas apply

.g[
.g-8[
```python
import pandas as pd
df = pd.DataFrame(np.random.randn(10_000, 100))
roll = df.rolling(100)

*out = roll.mean()
```
]
.g-4.center[
![](images/pandas-logo.tiff)
]
]

.alert.bold.center[🐌 One Core 🐌]

---

# pandas apply + numba

.g.g-middle[
.g-8[
```python
import pandas as pd

df = pd.DataFrame(np.random.randn(10_000, 100))
roll = df.rolling(100)

out = roll.mean(
*    engine="numba",
*    engine_kwargs={"parallel": True},
)
```
[Read more](https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#pandas-numba-engine)
]
.g-4.center[
![](images/pandas-logo.tiff)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# pandas apply + numba (Configuration)

.g[
.g-8[
- Environment variable: `NUMBA_NUM_THREADS`
]
.g-4.center[
![](images/pandas-logo.tiff)
]
]

---

# pandas apply + numba (Configuration)

- Numba function

```python
import numba
*numba.set_num_threads(2)

out = roll.mean(engine="numba", engine_kwargs={"parallel": True})
```
]
.g-4.center[
![](images/pandas-logo.tiff)
]
]

---

# LogisticRegression

.g.g-middle[
.g-8[
```python
from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression().fit(...)

*log_reg.predict(X)
```
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# LogisticRegression (Configuration)

.g[
.g-8[
- Environment variable: `OMP_NUM_THREADS`
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

---

# LogisticRegression (Configuration)

- `threadpoolctl`

```python
*with threadpool_limits(limits=2):
    log_reg.predict(X)
```
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

---

# HistGradientBoostingClassifier

hist = HistGradientBoostingClassifier()

hist.fit(X, y)
```
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# HistGradientBoostingClassifier (Configuration)

.g[
.g-8[
- Environment variable: `OMP_NUM_THREADS`
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

---

# HistGradientBoostingClassifier (Configuration)

- `threadpoolctl`

```python
*with threadpool_limits(limits=2):
    hist.predict(X)
```
]
.g-4.center[
![](images/scikit-learn-better.svg)
]
]

---

# polars

.g.g-middle[
.g-8[
```python
out = (
    pl.scan_csv(...)
    .filter(pl.col("sepal_length") > 5)
    .groupby("species")
    .agg(pl.col("sepal_width").mean())
    .collect()
)
```
]
.g-4.center[
![](images/polars.tiff)
]
]

.success.bold.center[🏎️ All Cores 🏎️]

---

# polars (Configuration)

.g.g-middle[
.g-8[
- Environment variable: `POLARS_MAX_THREADS`

```python
out = (
    pl.scan_csv(...)
    .filter(pl.col("sepal_length") > 5)
    ...
)
```
]
.g-4.center[
![](images/polars.tiff)
]
]

---

# Defaults 🌈

![:scale 90%](images/defaults.svg)

---

.g.g-middle[
.g-6[
# Proposal 🔮
## Agree on a default? 😅
]
.g-6[
![](images/tools.jpg)
]
]

---

# Proposal 🔮
## Libraries document how to configure parallelism

![:scale 70%](images/manual.jpg)

---

.g.g-middle[
.g-6.larger[
## APIs 💻
## Interactions 🥂
## Defaults 🌈
]
.g-6[
![](images/stethoscope.jpg)
]
]

---

# Can There Be Too Much Parallelism?

![:scale 15%](images/QuansightLabs_logo_V2.png)

This talk on Github: thomasjpfan/scipy-2023-too-parallel</a>

---

# Appendix 🪄

---

# Python GIL + Parallelism? 👀

- Python Multi-threading: Release the GIL
- Python Multi-processing: Each process gets it's own GIL
- Native multi-threading: Release the GIL

---

# PEP 684 🔮: Sub-Interpreters
## Need to explore, it could work ☀️

---

# PEP 703 🔮: No-GIL
## Also Promising, but harder lift for Python