Skip to content
Eon-Labs KB

pip: Checking and Installing Missing Python Modules in Your Virtual Environment

Source: Notion | Last edited: 2024-09-26 | ID: 3609eb27-4dd...

What you’ll do

Section titled “What you’ll do”

You will learn how to check for missing Python modules in your current virtual environment and install them if necessary, using a script that respects .gitignore and ensures the use of the virtual environment’s pip.

What you’ll learn

Section titled “What you’ll learn”
  • How to identify missing Python modules in your project.
  • How to install missing modules using pip within your virtual environment.
  • How to ensure that the script respects .gitignore to avoid unnecessary checks.
  • How to use rich for enhanced console output and logging.

What you’ll need

Section titled “What you’ll need”

Why

Section titled “Why”

The motivation behind this script is to identify and install any missing Python modules in your current virtual environment. This ensures that your project dependencies are up-to-date and that your development environment is consistent with your project’s requirements. By using python -m pip, the script ensures that the pip command is executed within the virtual environment, avoiding potential conflicts with the system Python environment.

What

Section titled “What”

This script performs the following tasks:

  1. Checks if required packages (rich, requests, pathspec, setuptools) are installed and installs them if they are missing.
  2. Analyzes all Python files in the current directory (respecting .gitignore) to find all imported modules.
  3. Checks if these modules are installed in the virtual environment.
  4. Generates a summary of installed modules, modules that need to be installed, and internal or non-PyPI modules.
  5. Provides installation commands for missing modules.

Who

Section titled “Who”

This script is useful for:

  • Developers who want to ensure that all necessary Python modules are installed in their virtual environment.
  • Teams working on shared projects to maintain consistent development environments.
  • Anyone who wants to automate the process of checking and installing project dependencies.

When

Section titled “When”

Use this script:

  • When setting up a new development environment.
  • After pulling changes from a remote repository to ensure all dependencies are installed.
  • Periodically, to ensure that your environment remains consistent with your project’s requirements.

Where

Section titled “Where”

This script should be run in the root directory of your Python project. It will analyze all Python files in the current directory and its subdirectories, respecting the .gitignore file to avoid unnecessary checks.

How

Section titled “How”

Follow these steps to use the script:

python -c "
import subprocess
import sys
import os


def install_package(package_name):
    print(f'Installing {package_name}...')
    subprocess.check_call([sys.executable, '-m', 'pip', 'install', package_name])


# List of required packages
required_packages = ['rich', 'requests', 'pathspec', 'setuptools']


# Install required packages if they're not already installed
for package in required_packages:
    try:
        __import__(package)
    except ImportError:
        install_package(package)


# Now that we've ensured the packages are installed, we can import them
import pkg_resources
import requests
from rich.console import Console
from rich.table import Table
from rich import box
import pathspec
from rich.logging import RichHandler
from rich.traceback import install as rich_install
import ast
import logging
import importlib


console = Console()


# Mapping of import names to PyPI package names
import_to_pypi = {
    'talib': 'TA-Lib',
    # Add more mappings here if needed
}


def install_or_upgrade_package(package_name):
    try:
        # Check if the package is installed
        dist = pkg_resources.get_distribution(package_name)
        installed_version = dist.version
        console.print(f'[bold green]{package_name}[/bold green] is installed with version {installed_version}')


        # Check the latest version available on PyPI
        response = requests.get(f'<https://pypi.org/pypi/{package_name}/json>')
        response.raise_for_status()
        latest_version = response.json()['info']['version']
        console.print(f'Latest version of [bold green]{package_name}[/bold green] available is {latest_version}')


        if installed_version != latest_version:
            console.print(f'Upgrading [bold green]{package_name}[/bold green] to version {latest_version}...')
            subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', package_name])
        else:
            console.print(f'[bold green]{package_name}[/bold green] is already up-to-date.')
    except pkg_resources.DistributionNotFound:
        console.print(f'[bold red]{package_name}[/bold red] is not installed. Installing...')
        subprocess.check_call([sys.executable, '-m', 'pip', 'install', package_name])
    except requests.RequestException as e:
        console.print(f'[bold red]Error checking latest version of {package_name}: {e}[/bold red]')


def is_module_on_pypi(module_name):
    pypi_name = import_to_pypi.get(module_name, module_name)
    try:
        response = requests.get(f'<https://pypi.org/pypi/{pypi_name}/json>')
        response.raise_for_status()
        return True
    except requests.RequestException:
        return False


def find_internal_modules():
    internal_modules = set()
    for root, dirs, files in os.walk('.'):
        for file in files:
            if file.endswith('.py'):
                module_name = os.path.splitext(file)[0]
                internal_modules.add(module_name)
        for dir in dirs:
            if os.path.exists(os.path.join(root, dir, '__init__.py')):
                internal_modules.add(dir)
    return internal_modules


# Ensure necessary packages are installed and up-to-date
necessary_packages = ['pathspec', 'rich', 'requests', 'setuptools']
for package in necessary_packages:
    install_or_upgrade_package(package)


# Setup rich logging
rich_install(show_locals=True)
logging.basicConfig(level='DEBUG', format='%(message)s', datefmt='[%X]', handlers=[RichHandler(rich_tracebacks=True, markup=True)])
logger = logging.getLogger('rich')


def get_imports(node):
    if isinstance(node, ast.Import):
        return [n.name for n in node.names]
    elif isinstance(node, ast.ImportFrom):
        return [node.module] if node.module else []
    return []


def analyze_file(file_path):
    logger.debug(f'Analyzing file: {file_path}')
    try:
        with open(file_path, 'r') as file:
            content = file.read()
        tree = ast.parse(content)
        imports = set(module for node in ast.walk(tree) for module in get_imports(node))
        logger.info(f'Found {len(imports)} unique imports in {file_path}')
        return imports
    except Exception as e:
        logger.error(f'Error analyzing {file_path}: {str(e)}')
        return set()


# Load .gitignore patterns
gitignore_path = '.gitignore'
if os.path.exists(gitignore_path):
    with open(gitignore_path, 'r') as f:
        gitignore_patterns = f.read().splitlines()
    spec = pathspec.PathSpec.from_lines('gitwildmatch', gitignore_patterns)
else:
    spec = pathspec.PathSpec([])


all_imports = {}
analyzed_files = 0
for root, dirs, files in os.walk('.'):
    # Filter out ignored directories
    dirs[:] = [d for d in dirs if not spec.match_file(os.path.join(root, d))]
    for file in files:
        file_path = os.path.join(root, file)
        if file.endswith('.py') and not spec.match_file(file_path):
            imports = analyze_file(file_path)
            for imp in imports:
                if imp not in all_imports:
                    all_imports[imp] = []
                all_imports[imp].append(file_path)
            analyzed_files += 1


logger.info(f'Total unique imports across all .py files: {len(all_imports)}')


installed_modules = set()
not_installed_modules = set()
internal_modules = set()
used_installed_modules = {}


# Find internal modules first
internal_module_names = find_internal_modules()


if all_imports:
    logger.debug('All imports found:')
    for imp in sorted(all_imports.keys()):
        logger.debug(imp)
        print(imp, file=sys.stdout)  # Print to stdout


    logger.debug('Checking module installation status:')
    for imp in sorted(all_imports.keys()):
        if imp.split('.')[0] in internal_module_names:
            logger.debug(f'{imp} is an internal module')
            internal_modules.add(imp)
        else:
            try:
                importlib.import_module(imp.split('.')[0])
                logger.debug(f'{imp} is installed')
                installed_modules.add(imp)
                if imp not in used_installed_modules:
                    used_installed_modules[imp] = []
                used_installed_modules[imp].extend(all_imports[imp])
            except ImportError:
                if is_module_on_pypi(imp.split('.')[0]):
                    logger.debug(f'{imp} is not installed')
                    print(f'{imp} may need to be installed', file=sys.stdout)  # Print to stdout
                    not_installed_modules.add(imp)
                else:
                    logger.debug(f'{imp} is an internal or non-PyPI module')
                    internal_modules.add(imp)
else:
    logger.warning('No imports found in any .py files')


# Get the list of standard library modules
if sys.version_info >= (3, 10):
    stdlib_modules = sys.stdlib_module_names
else:
    stdlib_modules = set([
        'abc', 'aifc', 'antigravity', 'argparse', 'array', 'ast', 'asynchat', 'asyncio', 'asyncore', 'atexit', 'audioop', 'base64', 'bdb', 'binascii', 'binhex', 'bisect', 'builtins', 'bz2', 'cProfile', 'calendar', 'cgi', 'cgitb', 'chunk', 'cmath', 'cmd', 'code', 'codecs', 'codeop', 'collections', 'colorsys', 'compileall', 'concurrent', 'configparser', 'contextlib', 'contextvars', 'copy', 'copyreg', 'crypt', 'csv', 'ctypes', 'curses', 'dataclasses', 'datetime', 'dbm', 'decimal', 'difflib', 'dis', 'distutils', 'doctest', 'email', 'encodings', 'ensurepip', 'enum', 'errno', 'faulthandler', 'fcntl', 'filecmp', 'fileinput', 'fnmatch', 'formatter', 'fractions', 'ftplib', 'functools', 'gc', 'getopt', 'getpass', 'gettext', 'glob', 'graphlib', 'gzip', 'hashlib', 'heapq', 'hmac', 'html', 'http', 'imaplib', 'imghdr', 'imp', 'importlib', 'inspect', 'io', 'ipaddress', 'itertools', 'json', 'keyword', 'lib2to3', 'linecache', 'locale', 'logging', 'lzma', 'mailbox', 'mailcap', 'marshal', 'math', 'mimetypes', 'mmap', 'modulefinder', 'msilib', 'msvcrt', 'multiprocessing', 'netrc', 'nntplib', 'numbers', 'operator', 'optparse', 'os', 'pathlib', 'pdb', 'pickle', 'pickletools', 'pipes', 'pkgutil', 'platform', 'plistlib', 'poplib', 'posix', 'pprint', 'profile', 'pstats', 'pty', 'pwd', 'py_compile', 'pyclbr', 'pydoc', 'pydoc_data', 'pyexpat', 'queue', 'quopri', 'random', 're', 'readline', 'reprlib', 'resource', 'rlcompleter', 'runpy', 'sched', 'secrets', 'select', 'selectors', 'shelve', 'shlex', 'shutil', 'signal', 'site', 'smtpd', 'smtplib', 'sndhdr', 'socket', 'socketserver', 'sqlite3', 'sre_compile', 'sre_constants', 'sre_parse', 'ssl', 'stat', 'statistics', 'string', 'stringprep', 'struct', 'subprocess', 'sunau', 'symbol', 'symtable', 'sys', 'sysconfig', 'tabnanny', 'tarfile', 'telnetlib', 'tempfile', 'termios', 'test', 'textwrap', 'threading', 'time', 'timeit', 'tkinter', 'token', 'tokenize', 'trace', 'traceback', 'tracemalloc', 'tty', 'turtle', 'turtledemo', 'types', 'typing', 'unicodedata', 'unittest', 'urllib', 'uu', 'uuid', 'venv', 'warnings', 'wave', 'weakref', 'webbrowser', 'winreg', 'winsound', 'wsgiref', 'xdrlib', 'xml', 'xmlrpc', 'zipapp', 'zipfile', 'zipimport', 'zlib', 'zoneinfo'
    ])


# Final Summary
console.print('\\n--- [bold cyan]Final Summary[/bold cyan] ---')
console.print(f'Total .py files analyzed: [bold yellow]{analyzed_files}[/bold yellow]')
console.print(f'Total unique imports found: [bold yellow]{len(all_imports)}[/bold yellow]')
console.print(f'Installed modules: [bold green]{len(installed_modules)}[/bold green]')
console.print(f'Modules that may need to be installed: [bold red]{len(not_installed_modules)}[/bold red]')
console.print(f'Internal or non-PyPI modules: [bold magenta]{len(internal_modules)}[/bold magenta]')


if used_installed_modules:
    table = Table(title='[bold green]Used and Installed Modules[/bold green]', box=box.SIMPLE)
    table.add_column('Module', style='bold green')
    table.add_column('Used in', style='bold cyan')
    for module in sorted(used_installed_modules.keys()):
        if module.split('.')[0] not in stdlib_modules:
            file_list = '\\n'.join([f'  - {file_path}' for file_path in used_installed_modules[module]])
            table.add_row(module, file_list)
    console.print(table)


if not_installed_modules:
    table = Table(title='[bold red]Modules that may need to be installed[/bold red]', box=box.SIMPLE)
    table.add_column('Module', style='bold red')
    table.add_column('Used in', style='bold cyan')
    printed_modules = set()
    install_commands = []
    for module in sorted(not_installed_modules):
        base_module = module.split('.')[0]
        if base_module not in printed_modules:
            install_commands.append(f'[bold yellow]python -m pip install {base_module}[/bold yellow]')
            printed_modules.add(base_module)
        file_list = '\\n'.join([f'  - {file_path}' for file_path in all_imports[module]])
        table.add_row(module, file_list)
    console.print(table)
    console.print('\\n'.join(install_commands))


if internal_modules:
    table = Table(title='[bold magenta]Internal or non-PyPI modules[/bold magenta]', box=box.SIMPLE)
    table.add_column('Module', style='bold magenta')
    table.add_column('Used in', style='bold cyan')
    for module in sorted(internal_modules):
        file_list = '\\n'.join([f'  - {file_path}' for file_path in all_imports[module]])
        table.add_row(module, file_list)
    console.print(table)


if not not_installed_modules and not internal_modules:
    console.print('\\n[bold green]All modules are installed.[/bold green]')
"

Additional Purposes

Section titled “Additional Purposes”

Based on the nature of the script, additional purposes include:

  • Dependency Management: Ensuring that all dependencies are installed and up-to-date, which is crucial for maintaining a stable development environment.
  • Project Onboarding: Simplifying the process for new developers joining the project by automating the setup of the required environment.
  • Continuous Integration: Integrating this script into CI pipelines to automatically check and install missing dependencies before running tests or deploying the application. By following this tutorial, you can ensure that your Python project has all the necessary modules installed, making your development process smoother and more efficient.

Sample output of the snippet on the terminal

Section titled “Sample output of the snippet on the terminal”

image