Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Access sysconfig paths through pyodide config CLI #14

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

## Changed

- The `pyodide config list` command now displays additional config variables
derived from the Python `sysconfig` module. These variables may also be accessed
through `pyodide config get <variable name>` for use in out-of-tree builds.
[#14](https://github.com/pyodide/pyodide-build/pull/14)

## [0.27.3] - 2024/07/17

- It is now possible to override `_f2c_fixes.py` file, with `_f2c_fixes_wrapper` variable.
Expand Down
11 changes: 10 additions & 1 deletion pyodide_build/cli/config.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import sysconfig

import typer

from ..build_env import get_build_environment_vars, get_pyodide_root, init_environment
Expand All @@ -19,6 +21,13 @@
"meson_cross_file": "MESON_CROSS_FILE",
}

# Load the values of sysconfig.get_paths() with a
# "pyodide_sysconfig_" prefix to differentiate them
# from other configuration variables
PYODIDE_CONFIGS.update(
{f"pyodide_sysconfig_{k}": v for k, v in sysconfig.get_paths().items()}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are only eight values in sysconfig.get_paths(), so, I think it is okay to list all of them, which will be more explicit and make the code more readable.

)


@app.callback(no_args_is_help=True)
def callback() -> None:
Expand Down Expand Up @@ -52,7 +61,7 @@ def get_config(
),
) -> None:
"""
Get a value of a single config variable used in pyodide
Get a value of a single config variable used in Pyodide
"""
configs = _get_configs()

Expand Down
16 changes: 16 additions & 0 deletions pyodide_build/config.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import os
import subprocess
import sysconfig
from collections.abc import Mapping
from pathlib import Path
from types import MappingProxyType
Expand Down Expand Up @@ -29,8 +30,15 @@ def __init__(self, pyodide_root: Path):
**self._load_makefile_envs(),
**self._load_config_file(Path.cwd(), os.environ),
**self._load_config_from_env(os.environ),
**self._load_sysconfig_paths(),
}

def _load_sysconfig_paths(self) -> Mapping[str, str]:
# Load the values of sysconfig.get_paths() with a
# "pyodide_sysconfig_" prefix to differentiate them
# from other configuration variables
return {f"pyodide_sysconfig_{k}": v for k, v in sysconfig.get_paths().items()}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would retrieve the host Python's sysconfig values, not those in the cross build env. We should parse these values from the _sysconfigdata__emscripten_wasm32-emscripten.py file in the xbuildenv.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where should I retrieve this file from the cross-build environment? I have noted that I can get the file when I do pyodide xbuildenv install, but if there's already a programmatic way to access this, I can use that instead of writing my own methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the code we locate the sysconfig file. Since it uses the variables in the Makefile.envs, you'll need to modify it a bit to use it in the config manager to prevent cyclic import.

sysconfigdata_name = get_build_flag("SYSCONFIG_NAME")
sysconfigdata_path = (
Path(get_build_flag("TARGETINSTALLDIR"))
/ f"sysconfigdata/{sysconfigdata_name}.py"
)

I think you'll also need to read the sysconfig.py and how it retrieves the value from the sysconfig file, as we cannot directly use sysconfig module here.


def _load_default_config(self) -> Mapping[str, str]:
return {
k: _environment_substitute_str(
Expand Down Expand Up @@ -173,6 +181,10 @@ def to_env(self) -> dict[str, str]:
"_f2c_fixes_wrapper": "_F2C_FIXES_WRAPPER",
}

BUILD_KEY_TO_VAR.update(
{f"pyodide_sysconfig_{k}": v for k, v in sysconfig.get_paths().items()}
)

BUILD_VAR_TO_KEY = {v: k for k, v in BUILD_KEY_TO_VAR.items()}

# Configuration keys that can be overridden by the user.
Expand Down Expand Up @@ -205,6 +217,10 @@ def to_env(self) -> dict[str, str]:
"_f2c_fixes_wrapper": "",
}

DEFAULT_CONFIG.update(
{f"pyodide_sysconfig_{k}": v for k, v in sysconfig.get_paths().items()}
)

# Default configs that are computed from other values (often from Makefile.envs)
# TODO: Remove dependency on Makefile.envs
DEFAULT_CONFIG_COMPUTED: dict[str, str] = {
Expand Down
Loading