Python Packaging

Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.

When to Use This Skill

• Creating Python libraries for distribution
• Building command-line tools with entry points
• Publishing packages to PyPI or private repositories
• Setting up Python project structure
• Creating installable packages with dependencies
• Building wheels and source distributions
• Versioning and releasing Python packages
• Creating namespace packages
• Implementing package metadata and classifiers

Core Concepts

1. Package Structure

• Source layout: src/package_name/ (recommended)
• Flat layout: package_name/ (simpler but less flexible)
• Package metadata: pyproject.toml, setup.py, or setup.cfg
• Distribution formats: wheel (.whl) and source distribution (.tar.gz)

2. Modern Packaging Standards

• PEP 517/518: Build system requirements
• PEP 621: Metadata in pyproject.toml
• PEP 660: Editable installs
• pyproject.toml: Single source of configuration

3. Build Backends

• setuptools: Traditional, widely used
• hatchling: Modern, opinionated
• flit: Lightweight, for pure Python
• poetry: Dependency management + packaging

4. Distribution

• PyPI: Python Package Index (public)
• TestPyPI: Testing before production
• Private repositories: JFrog, AWS CodeArtifact, etc.

Quick Start

Minimal Package Structure

my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── my_package/
│       ├── __init__.py
│       └── module.py
└── tests/
    └── test_module.py

Minimal pyproject.toml

toml
1[build-system]
2requires = ["setuptools>=61.0"]
3build-backend = "setuptools.build_meta"
4
5[project]
6name = "my-package"
7version = "0.1.0"
8description = "A short description"
9authors = [{name = "Your Name", email = "you@example.com"}]
10readme = "README.md"
11requires-python = ">=3.8"
12dependencies = [
13    "requests>=2.28.0",
14]
15
16[project.optional-dependencies]
17dev = [
18    "pytest>=7.0",
19    "black>=22.0",
20]

Package Structure Patterns

Pattern 1: Source Layout (Recommended)

my-package/
├── pyproject.toml
├── README.md
├── LICENSE
├── .gitignore
├── src/
│   └── my_package/
│       ├── __init__.py
│       ├── core.py
│       ├── utils.py
│       └── py.typed          # For type hints
├── tests/
│   ├── __init__.py
│   ├── test_core.py
│   └── test_utils.py
└── docs/
    └── index.md

Advantages:

• Prevents accidentally importing from source
• Cleaner test imports
• Better isolation

pyproject.toml for source layout:

toml
1[tool.setuptools.packages.find]
2where = ["src"]

Pattern 2: Flat Layout

my-package/
├── pyproject.toml
├── README.md
├── my_package/
│   ├── __init__.py
│   └── module.py
└── tests/
    └── test_module.py

Simpler but:

• Can import package without installing
• Less professional for libraries

Pattern 3: Multi-Package Project

project/
├── pyproject.toml
├── packages/
│   ├── package-a/
│   │   └── src/
│   │       └── package_a/
│   └── package-b/
│       └── src/
│           └── package_b/
└── tests/

Complete pyproject.toml Examples

Pattern 4: Full-Featured pyproject.toml

toml
1[build-system]
2requires = ["setuptools>=61.0", "wheel"]
3build-backend = "setuptools.build_meta"
4
5[project]
6name = "my-awesome-package"
7version = "1.0.0"
8description = "An awesome Python package"
9readme = "README.md"
10requires-python = ">=3.8"
11license = {text = "MIT"}
12authors = [
13    {name = "Your Name", email = "you@example.com"},
14]
15maintainers = [
16    {name = "Maintainer Name", email = "maintainer@example.com"},
17]
18keywords = ["example", "package", "awesome"]
19classifiers = [
20    "Development Status :: 4 - Beta",
21    "Intended Audience :: Developers",
22    "License :: OSI Approved :: MIT License",
23    "Programming Language :: Python :: 3",
24    "Programming Language :: Python :: 3.8",
25    "Programming Language :: Python :: 3.9",
26    "Programming Language :: Python :: 3.10",
27    "Programming Language :: Python :: 3.11",
28    "Programming Language :: Python :: 3.12",
29]
30
31dependencies = [
32    "requests>=2.28.0,<3.0.0",
33    "click>=8.0.0",
34    "pydantic>=2.0.0",
35]
36
37[project.optional-dependencies]
38dev = [
39    "pytest>=7.0.0",
40    "pytest-cov>=4.0.0",
41    "black>=23.0.0",
42    "ruff>=0.1.0",
43    "mypy>=1.0.0",
44]
45docs = [
46    "sphinx>=5.0.0",
47    "sphinx-rtd-theme>=1.0.0",
48]
49all = [
50    "my-awesome-package[dev,docs]",
51]
52
53[project.urls]
54Homepage = "https://github.com/username/my-awesome-package"
55Documentation = "https://my-awesome-package.readthedocs.io"
56Repository = "https://github.com/username/my-awesome-package"
57"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
58Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
59
60[project.scripts]
61my-cli = "my_package.cli:main"
62awesome-tool = "my_package.tools:run"
63
64[project.entry-points."my_package.plugins"]
65plugin1 = "my_package.plugins:plugin1"
66
67[tool.setuptools]
68package-dir = {"" = "src"}
69zip-safe = false
70
71[tool.setuptools.packages.find]
72where = ["src"]
73include = ["my_package*"]
74exclude = ["tests*"]
75
76[tool.setuptools.package-data]
77my_package = ["py.typed", "*.pyi", "data/*.json"]
78
79# Black configuration
80[tool.black]
81line-length = 100
82target-version = ["py38", "py39", "py310", "py311"]
83include = '\.pyi?$'
84
85# Ruff configuration
86[tool.ruff]
87line-length = 100
88target-version = "py38"
89
90[tool.ruff.lint]
91select = ["E", "F", "I", "N", "W", "UP"]
92
93# MyPy configuration
94[tool.mypy]
95python_version = "3.8"
96warn_return_any = true
97warn_unused_configs = true
98disallow_untyped_defs = true
99
100# Pytest configuration
101[tool.pytest.ini_options]
102testpaths = ["tests"]
103python_files = ["test_*.py"]
104addopts = "-v --cov=my_package --cov-report=term-missing"
105
106# Coverage configuration
107[tool.coverage.run]
108source = ["src"]
109omit = ["*/tests/*"]
110
111[tool.coverage.report]
112exclude_lines = [
113    "pragma: no cover",
114    "def __repr__",
115    "raise AssertionError",
116    "raise NotImplementedError",
117]

Pattern 5: Dynamic Versioning

toml
1[build-system]
2requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
3build-backend = "setuptools.build_meta"
4
5[project]
6name = "my-package"
7dynamic = ["version"]
8description = "Package with dynamic version"
9
10[tool.setuptools.dynamic]
11version = {attr = "my_package.__version__"}
12
13# Or use setuptools-scm for git-based versioning
14[tool.setuptools_scm]
15write_to = "src/my_package/_version.py"

In init.py:

python
1# src/my_package/__init__.py
2__version__ = "1.0.0"
3
4# Or with setuptools-scm
5from importlib.metadata import version
6__version__ = version("my-package")

Command-Line Interface (CLI) Patterns

Pattern 6: CLI with Click

python
1# src/my_package/cli.py
2import click
3
4@click.group()
5@click.version_option()
6def cli():
7    """My awesome CLI tool."""
8    pass
9
10@cli.command()
11@click.argument("name")
12@click.option("--greeting", default="Hello", help="Greeting to use")
13def greet(name: str, greeting: str):
14    """Greet someone."""
15    click.echo(f"{greeting}, {name}!")
16
17@cli.command()
18@click.option("--count", default=1, help="Number of times to repeat")
19def repeat(count: int):
20    """Repeat a message."""
21    for i in range(count):
22        click.echo(f"Message {i + 1}")
23
24def main():
25    """Entry point for CLI."""
26    cli()
27
28if __name__ == "__main__":
29    main()

Register in pyproject.toml:

toml
1[project.scripts]
2my-tool = "my_package.cli:main"

Usage:

bash
1pip install -e .
2my-tool greet World
3my-tool greet Alice --greeting="Hi"
4my-tool repeat --count=3

Pattern 7: CLI with argparse

python
1# src/my_package/cli.py
2import argparse
3import sys
4
5def main():
6    """Main CLI entry point."""
7    parser = argparse.ArgumentParser(
8        description="My awesome tool",
9        prog="my-tool"
10    )
11
12    parser.add_argument(
13        "--version",
14        action="version",
15        version="%(prog)s 1.0.0"
16    )
17
18    subparsers = parser.add_subparsers(dest="command", help="Commands")
19
20    # Add subcommand
21    process_parser = subparsers.add_parser("process", help="Process data")
22    process_parser.add_argument("input_file", help="Input file path")
23    process_parser.add_argument(
24        "--output", "-o",
25        default="output.txt",
26        help="Output file path"
27    )
28
29    args = parser.parse_args()
30
31    if args.command == "process":
32        process_data(args.input_file, args.output)
33    else:
34        parser.print_help()
35        sys.exit(1)
36
37def process_data(input_file: str, output_file: str):
38    """Process data from input to output."""
39    print(f"Processing {input_file} -> {output_file}")
40
41if __name__ == "__main__":
42    main()

Building and Publishing

Pattern 8: Build Package Locally

bash
1# Install build tools
2pip install build twine
3
4# Build distribution
5python -m build
6
7# This creates:
8# dist/
9#   my-package-1.0.0.tar.gz (source distribution)
10#   my_package-1.0.0-py3-none-any.whl (wheel)
11
12# Check the distribution
13twine check dist/*

Pattern 9: Publishing to PyPI

bash
1# Install publishing tools
2pip install twine
3
4# Test on TestPyPI first
5twine upload --repository testpypi dist/*
6
7# Install from TestPyPI to test
8pip install --index-url https://test.pypi.org/simple/ my-package
9
10# If all good, publish to PyPI
11twine upload dist/*

Using API tokens (recommended):

bash
1# Create ~/.pypirc
2[distutils]
3index-servers =
4    pypi
5    testpypi
6
7[pypi]
8username = __token__
9password = pypi-...your-token...
10
11[testpypi]
12username = __token__
13password = pypi-...your-test-token...

Pattern 10: Automated Publishing with GitHub Actions

yaml
1# .github/workflows/publish.yml
2name: Publish to PyPI
3
4on:
5  release:
6    types: [created]
7
8jobs:
9  publish:
10    runs-on: ubuntu-latest
11
12    steps:
13      - uses: actions/checkout@v3
14
15      - name: Set up Python
16        uses: actions/setup-python@v4
17        with:
18          python-version: "3.11"
19
20      - name: Install dependencies
21        run: |
22          pip install build twine
23
24      - name: Build package
25        run: python -m build
26
27      - name: Check package
28        run: twine check dist/*
29
30      - name: Publish to PyPI
31        env:
32          TWINE_USERNAME: __token__
33          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
34        run: twine upload dist/*

Advanced Patterns

Pattern 11: Including Data Files

toml
1[tool.setuptools.package-data]
2my_package = [
3    "data/*.json",
4    "templates/*.html",
5    "static/css/*.css",
6    "py.typed",
7]

Accessing data files:

python
1# src/my_package/loader.py
2from importlib.resources import files
3import json
4
5def load_config():
6    """Load configuration from package data."""
7    config_file = files("my_package").joinpath("data/config.json")
8    with config_file.open() as f:
9        return json.load(f)
10
11# Python 3.9+
12from importlib.resources import files
13
14data = files("my_package").joinpath("data/file.txt").read_text()

Pattern 12: Namespace Packages

For large projects split across multiple repositories:

# Package 1: company-core
company/
└── core/
    ├── __init__.py
    └── models.py

# Package 2: company-api
company/
└── api/
    ├── __init__.py
    └── routes.py

Do NOT include init.py in the namespace directory (company/):

toml
1# company-core/pyproject.toml
2[project]
3name = "company-core"
4
5[tool.setuptools.packages.find]
6where = ["."]
7include = ["company.core*"]
8
9# company-api/pyproject.toml
10[project]
11name = "company-api"
12
13[tool.setuptools.packages.find]
14where = ["."]
15include = ["company.api*"]

Usage:

python
1# Both packages can be imported under same namespace
2from company.core import models
3from company.api import routes

Pattern 13: C Extensions

toml
1[build-system]
2requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]
3build-backend = "setuptools.build_meta"
4
5[tool.setuptools]
6ext-modules = [
7    {name = "my_package.fast_module", sources = ["src/fast_module.c"]},
8]

Or with setup.py:

python
1# setup.py
2from setuptools import setup, Extension
3
4setup(
5    ext_modules=[
6        Extension(
7            "my_package.fast_module",
8            sources=["src/fast_module.c"],
9            include_dirs=["src/include"],
10        )
11    ]
12)

Version Management

Pattern 14: Semantic Versioning

python
1# src/my_package/__init__.py
2__version__ = "1.2.3"
3
4# Semantic versioning: MAJOR.MINOR.PATCH
5# MAJOR: Breaking changes
6# MINOR: New features (backward compatible)
7# PATCH: Bug fixes

Version constraints in dependencies:

toml
1dependencies = [
2    "requests>=2.28.0,<3.0.0",  # Compatible range
3    "click~=8.1.0",              # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)
4    "pydantic>=2.0",             # Minimum version
5    "numpy==1.24.3",             # Exact version (avoid if possible)
6]

Pattern 15: Git-Based Versioning

toml
1[build-system]
2requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
3build-backend = "setuptools.build_meta"
4
5[project]
6name = "my-package"
7dynamic = ["version"]
8
9[tool.setuptools_scm]
10write_to = "src/my_package/_version.py"
11version_scheme = "post-release"
12local_scheme = "dirty-tag"

Creates versions like:

• 1.0.0 (from git tag)
• 1.0.1.dev3+g1234567 (3 commits after tag)

Testing Installation

Pattern 16: Editable Install

bash
1# Install in development mode
2pip install -e .
3
4# With optional dependencies
5pip install -e ".[dev]"
6pip install -e ".[dev,docs]"
7
8# Now changes to source code are immediately reflected

Pattern 17: Testing in Isolated Environment

bash
1# Create virtual environment
2python -m venv test-env
3source test-env/bin/activate  # Linux/Mac
4# test-env\Scripts\activate  # Windows
5
6# Install package
7pip install dist/my_package-1.0.0-py3-none-any.whl
8
9# Test it works
10python -c "import my_package; print(my_package.__version__)"
11
12# Test CLI
13my-tool --help
14
15# Cleanup
16deactivate
17rm -rf test-env

Documentation

Pattern 18: README.md Template

markdown
1# My Package
2
3[![PyPI version](https://badge.fury.io/py/my-package.svg)](https://pypi.org/project/my-package/)
4[![Python versions](https://img.shields.io/pypi/pyversions/my-package.svg)](https://pypi.org/project/my-package/)
5[![Tests](https://github.com/username/my-package/workflows/Tests/badge.svg)](https://github.com/username/my-package/actions)
6
7Brief description of your package.
8
9## Installation
10
11```bash
12pip install my-package

Quick Start

python
1from my_package import something
2
3result = something.do_stuff()

Features

• Feature 1
• Feature 2
• Feature 3

Documentation

Full documentation: https://my-package.readthedocs.io

Development

bash
1git clone https://github.com/username/my-package.git
2cd my-package
3pip install -e ".[dev]"
4pytest

License

MIT

## Common Patterns

### Pattern 19: Multi-Architecture Wheels

```yaml
# .github/workflows/wheels.yml
name: Build wheels

on: [push, pull_request]

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]

    steps:
      - uses: actions/checkout@v3

      - name: Build wheels
        uses: pypa/cibuildwheel@v2.16.2

      - uses: actions/upload-artifact@v3
        with:
          path: ./wheelhouse/*.whl

Pattern 20: Private Package Index

bash
1# Install from private index
2pip install my-package --index-url https://private.pypi.org/simple/
3
4# Or add to pip.conf
5[global]
6index-url = https://private.pypi.org/simple/
7extra-index-url = https://pypi.org/simple/
8
9# Upload to private index
10twine upload --repository-url https://private.pypi.org/ dist/*

File Templates

.gitignore for Python Packages

gitignore
1# Build artifacts
2build/
3dist/
4*.egg-info/
5*.egg
6.eggs/
7
8# Python
9__pycache__/
10*.py[cod]
11*$py.class
12*.so
13
14# Virtual environments
15venv/
16env/
17ENV/
18
19# IDE
20.vscode/
21.idea/
22*.swp
23
24# Testing
25.pytest_cache/
26.coverage
27htmlcov/
28
29# Distribution
30*.whl
31*.tar.gz

MANIFEST.in

# MANIFEST.in
include README.md
include LICENSE
include pyproject.toml

recursive-include src/my_package/data *.json
recursive-include src/my_package/templates *.html
recursive-exclude * __pycache__
recursive-exclude * *.py[co]

Checklist for Publishing

• Code is tested (pytest passing)
• Documentation is complete (README, docstrings)
• Version number updated
• CHANGELOG.md updated
• License file included
• pyproject.toml is complete
• Package builds without errors
• Installation tested in clean environment
• CLI tools work (if applicable)
• PyPI metadata is correct (classifiers, keywords)
• GitHub repository linked
• Tested on TestPyPI first
• Git tag created for release

Resources

• Python Packaging Guide: https://packaging.python.org/
• PyPI: https://pypi.org/
• TestPyPI: https://test.pypi.org/
• setuptools documentation: https://setuptools.pypa.io/
• build: https://pypa-build.readthedocs.io/
• twine: https://twine.readthedocs.io/

Best Practices Summary

1. Use src/ layout for cleaner package structure
2. Use pyproject.toml for modern packaging
3. Pin build dependencies in build-system.requires
4. Version appropriately with semantic versioning
5. Include all metadata (classifiers, URLs, etc.)
6. Test installation in clean environments
7. Use TestPyPI before publishing to PyPI
8. Document thoroughly with README and docstrings
9. Include LICENSE file
10. Automate publishing with CI/CD