peps/pep-0499/index.html

342 lines
25 KiB
HTML
Raw Permalink Blame History

This file contains ambiguous Unicode characters!

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="color-scheme" content="light dark">
<title>PEP 499 python -m foo should also bind &#39;foo&#39; in sys.modules | peps.python.org</title>
<link rel="shortcut icon" href="../_static/py.png">
<link rel="canonical" href="https://peps.python.org/pep-0499/">
<link rel="stylesheet" href="../_static/style.css" type="text/css">
<link rel="stylesheet" href="../_static/mq.css" type="text/css">
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" media="(prefers-color-scheme: light)" id="pyg-light">
<link rel="stylesheet" href="../_static/pygments_dark.css" type="text/css" media="(prefers-color-scheme: dark)" id="pyg-dark">
<link rel="alternate" type="application/rss+xml" title="Latest PEPs" href="https://peps.python.org/peps.rss">
<meta property="og:title" content='PEP 499 python -m foo should also bind &#39;foo&#39; in sys.modules | peps.python.org'>
<meta property="og:type" content="website">
<meta property="og:url" content="https://peps.python.org/pep-0499/">
<meta property="og:site_name" content="Python Enhancement Proposals (PEPs)">
<meta property="og:image" content="https://peps.python.org/_static/og-image.png">
<meta property="og:image:alt" content="Python PEPs">
<meta property="og:image:width" content="200">
<meta property="og:image:height" content="200">
<meta name="description" content="Python Enhancement Proposals (PEPs)">
<meta name="theme-color" content="#3776ab">
</head>
<body>
<svg xmlns="http://www.w3.org/2000/svg" style="display: none;">
<symbol id="svg-sun-half" viewBox="0 0 24 24" pointer-events="all">
<title>Following system colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="9"></circle>
<path d="M12 3v18m0-12l4.65-4.65M12 14.3l7.37-7.37M12 19.6l8.85-8.85"></path>
</svg>
</symbol>
<symbol id="svg-moon" viewBox="0 0 24 24" pointer-events="all">
<title>Selected dark colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path stroke="none" d="M0 0h24v24H0z" fill="none"></path>
<path d="M12 3c.132 0 .263 0 .393 0a7.5 7.5 0 0 0 7.92 12.446a9 9 0 1 1 -8.313 -12.454z"></path>
</svg>
</symbol>
<symbol id="svg-sun" viewBox="0 0 24 24" pointer-events="all">
<title>Selected light colour scheme</title>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<circle cx="12" cy="12" r="5"></circle>
<line x1="12" y1="1" x2="12" y2="3"></line>
<line x1="12" y1="21" x2="12" y2="23"></line>
<line x1="4.22" y1="4.22" x2="5.64" y2="5.64"></line>
<line x1="18.36" y1="18.36" x2="19.78" y2="19.78"></line>
<line x1="1" y1="12" x2="3" y2="12"></line>
<line x1="21" y1="12" x2="23" y2="12"></line>
<line x1="4.22" y1="19.78" x2="5.64" y2="18.36"></line>
<line x1="18.36" y1="5.64" x2="19.78" y2="4.22"></line>
</svg>
</symbol>
</svg>
<script>
document.documentElement.dataset.colour_scheme = localStorage.getItem("colour_scheme") || "auto"
</script>
<section id="pep-page-section">
<header>
<h1>Python Enhancement Proposals</h1>
<ul class="breadcrumbs">
<li><a href="https://www.python.org/" title="The Python Programming Language">Python</a> &raquo; </li>
<li><a href="../pep-0000/">PEP Index</a> &raquo; </li>
<li>PEP 499</li>
</ul>
<button id="colour-scheme-cycler" onClick="setColourScheme(nextColourScheme())">
<svg aria-hidden="true" class="colour-scheme-icon-when-auto"><use href="#svg-sun-half"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-dark"><use href="#svg-moon"></use></svg>
<svg aria-hidden="true" class="colour-scheme-icon-when-light"><use href="#svg-sun"></use></svg>
<span class="visually-hidden">Toggle light / dark / auto colour theme</span>
</button>
</header>
<article>
<section id="pep-content">
<h1 class="page-title">PEP 499 <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">-m</span> <span class="pre">foo</span></code> should also bind <code class="docutils literal notranslate"><span class="pre">'foo'</span></code> in <code class="docutils literal notranslate"><span class="pre">sys.modules</span></code></h1>
<dl class="rfc2822 field-list simple">
<dt class="field-odd">Author<span class="colon">:</span></dt>
<dd class="field-odd">Cameron Simpson &lt;cs&#32;&#97;t&#32;cskk.id.au&gt;, Chris Angelico &lt;rosuav&#32;&#97;t&#32;gmail.com&gt;, Joseph Jevnik &lt;joejev&#32;&#97;t&#32;gmail.com&gt;</dd>
<dt class="field-even">BDFL-Delegate<span class="colon">:</span></dt>
<dd class="field-even">Alyssa Coghlan</dd>
<dt class="field-odd">Status<span class="colon">:</span></dt>
<dd class="field-odd"><abbr title="Inactive draft that may be taken up again at a later time">Deferred</abbr></dd>
<dt class="field-even">Type<span class="colon">:</span></dt>
<dd class="field-even"><abbr title="Normative PEP with a new feature for Python, implementation change for CPython or interoperability standard for the ecosystem">Standards Track</abbr></dd>
<dt class="field-odd">Created<span class="colon">:</span></dt>
<dd class="field-odd">07-Aug-2015</dd>
<dt class="field-even">Python-Version<span class="colon">:</span></dt>
<dd class="field-even">3.10</dd>
</dl>
<hr class="docutils" />
<section id="contents">
<details><summary>Table of Contents</summary><ul class="simple">
<li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#proposal">Proposal</a></li>
<li><a class="reference internal" href="#considerations-and-prerequisites">Considerations and Prerequisites</a><ul>
<li><a class="reference internal" href="#pickling-modules">Pickling Modules</a></li>
<li><a class="reference internal" href="#a-normal-module-s-name-is-no-longer-canonical">A Normal Modules <code class="docutils literal notranslate"><span class="pre">__name__</span></code> Is No Longer Canonical</a></li>
</ul>
</li>
<li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li>
<li><a class="reference internal" href="#open-questions">Open Questions</a><ul>
<li><a class="reference internal" href="#pickle-compatibility">Pickle compatibility</a></li>
<li><a class="reference internal" href="#projects-that-special-case-main">Projects that special-case <code class="docutils literal notranslate"><span class="pre">__main__</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#background">Background</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
</details></section>
<section id="pep-deferral">
<h2><a class="toc-backref" href="#pep-deferral" role="doc-backlink">PEP Deferral</a></h2>
<p>The implementation of this PEP isnt currently expected to be ready for the
Python 3.9 feature freeze in April 2020, so it has been deferred 12 months to
Python 3.10.</p>
</section>
<section id="abstract">
<h2><a class="toc-backref" href="#abstract" role="doc-backlink">Abstract</a></h2>
<p>When a module is used as a main program on the Python command line,
such as by:</p>
<blockquote>
<div>python -m module.name …</div></blockquote>
<p>it is easy to accidentally end up with two independent instances
of the module if that module is again imported within the program.
This PEP proposes a way to fix this problem.</p>
<p>When a module is invoked via Pythons -m option the module is bound
to <code class="docutils literal notranslate"><span class="pre">sys.modules['__main__']</span></code> and its <code class="docutils literal notranslate"><span class="pre">.__name__</span></code> attribute is set to
<code class="docutils literal notranslate"><span class="pre">'__main__'</span></code>.
This enables the standard “main program” boilerplate code at the
bottom of many modules, such as:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="n">main</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">))</span>
</pre></div>
</div>
<p>However, when the above command line invocation is used it is a
natural inference to presume that the module is actually imported
under its official name <code class="docutils literal notranslate"><span class="pre">module.name</span></code>,
and therefore that if the program again imports that name
then it will obtain the same module instance.</p>
<p>That actuality is that the module was imported only as <code class="docutils literal notranslate"><span class="pre">'__main__'</span></code>.
Another import will obtain a distinct module instance, which can
lead to confusing bugs,
all stemming from having two instances of module global objects:
one in each module.</p>
<p>Examples include:</p>
<dl class="simple">
<dt>module level data structures</dt><dd>Some modules provide features such as caches or registries
as module level global variables,
typically private.
A second instance of a module creates a second data structure.
If that structure is a cache
such as in the <code class="docutils literal notranslate"><span class="pre">re</span></code> module
then two caches exist leading to wasteful memory use.
If that structure is a shared registry
such as a mapping of values to handlers
then it is possible to register a handler to one registry
and to try to use it via the other registry, where it is unknown.</dd>
<dt>sentinels</dt><dd>The standard test for a sentinel value provided by a module
is the identity comparison using <code class="docutils literal notranslate"><span class="pre">is</span></code>,
as this avoids unreliable “looks like” comparisons
such as equality which can both mismatch two values as “equal”
(for example being zeroish)
or raise a <code class="docutils literal notranslate"><span class="pre">TypeError</span></code> when the objects are incompatible.
When there are two instances of a module
there are two sentinel instances
and only one will be recognised via <code class="docutils literal notranslate"><span class="pre">is</span></code>.</dd>
<dt>classes</dt><dd>With two modules
there are duplicate class definitions of any classes provided.
All operations which depend on recognising these classes
and subclasses of these are prone to failure
depending where the reference class
(from one of the modules) is obtained
and where the comparison class or instance is obtained.
This impacts <code class="docutils literal notranslate"><span class="pre">isinstance</span></code>, <code class="docutils literal notranslate"><span class="pre">issubclass</span></code>
and also <code class="docutils literal notranslate"><span class="pre">try</span></code>/<code class="docutils literal notranslate"><span class="pre">except</span></code> constructs.</dd>
</dl>
</section>
<section id="proposal">
<h2><a class="toc-backref" href="#proposal" role="doc-backlink">Proposal</a></h2>
<p>It is suggested that to fix this situation all that is needed is a
simple change to the way the <code class="docutils literal notranslate"><span class="pre">-m</span></code> option is implemented: in addition
to binding the module object to <code class="docutils literal notranslate"><span class="pre">sys.modules['__main__']</span></code>, it is also
bound to <code class="docutils literal notranslate"><span class="pre">sys.modules['module.name']</span></code>.</p>
<p>Alyssa (Nick) Coghlan has suggested that this is as simple as modifying the
<code class="docutils literal notranslate"><span class="pre">runpy</span></code> modules <code class="docutils literal notranslate"><span class="pre">_run_module_as_main</span></code> function as follows:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">main_globals</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="s2">&quot;__main__&quot;</span><span class="p">]</span><span class="o">.</span><span class="vm">__dict__</span>
</pre></div>
</div>
<p>to instead be:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">main_module</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="s2">&quot;__main__&quot;</span><span class="p">]</span>
<span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">mod_spec</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">main_module</span>
<span class="n">main_globals</span> <span class="o">=</span> <span class="n">main_module</span><span class="o">.</span><span class="vm">__dict__</span>
</pre></div>
</div>
<p>Joseph Jevnik has pointed out that modules which are packages already
do something very similar to this proposal:
the __init__.py file is bound to the modules canonical name
and the __main__.py file is bound to “__main__”.
As such, the double import issue does not occur.
Therefore, this PEP proposes to affect only simple non-package modules.</p>
</section>
<section id="considerations-and-prerequisites">
<h2><a class="toc-backref" href="#considerations-and-prerequisites" role="doc-backlink">Considerations and Prerequisites</a></h2>
<section id="pickling-modules">
<h3><a class="toc-backref" href="#pickling-modules" role="doc-backlink">Pickling Modules</a></h3>
<p>Alyssa has mentioned <a class="reference external" href="http://bugs.python.org/issue19702">issue 19702</a> which proposes (quoted from the issue):</p>
<ul class="simple">
<li>runpy will ensure that when __main__ is executed via the import
system, it will also be aliased in sys.modules as __spec__.name</li>
<li>if __main__.__spec__ is set, pickle will use __spec__.name rather
than __name__ to pickle classes, functions and methods defined in
__main__</li>
<li>multiprocessing is updated appropriately to skip creating __mp_main__
in child processes when __main__.__spec__ is set in the parent
process</li>
</ul>
<p>The first point above covers this PEPs specific proposal.</p>
</section>
<section id="a-normal-module-s-name-is-no-longer-canonical">
<h3><a class="toc-backref" href="#a-normal-module-s-name-is-no-longer-canonical" role="doc-backlink">A Normal Modules <code class="docutils literal notranslate"><span class="pre">__name__</span></code> Is No Longer Canonical</a></h3>
<p>Chris Angelico points out that it becomes possible to import a
module whose <code class="docutils literal notranslate"><span class="pre">__name__</span></code> is not what you gave to “import”, since
“__main__” is now present at “module.name”, so a subsequent
<code class="docutils literal notranslate"><span class="pre">import</span> <span class="pre">module.name</span></code> finds it already present.
Therefore, <code class="docutils literal notranslate"><span class="pre">__name__</span></code> is no longer the canonical name for some normal imports.</p>
<p>Some counter arguments follow:</p>
<ul class="simple">
<li>As of <a class="pep reference internal" href="../pep-0451/" title="PEP 451 A ModuleSpec Type for the Import System">PEP 451</a> a modules canonical name is stored at <code class="docutils literal notranslate"><span class="pre">__spec__.name</span></code>.</li>
<li>Very little code should actually care about <code class="docutils literal notranslate"><span class="pre">__name__</span></code> being the canonical name
and any that does should arguably be updated to consult <code class="docutils literal notranslate"><span class="pre">__spec__.name</span></code>
with fallback to <code class="docutils literal notranslate"><span class="pre">__name__</span></code> for older Pythons, should that be relevant.
This is true even if this PEP is not approved.</li>
<li>Should this PEP be approved,
it becomes possible to introspect a module by its canonical name
and ask “was this the main program?” by inferring from <code class="docutils literal notranslate"><span class="pre">__name__</span></code>.
This was not previously possible.</li>
</ul>
<p>The glaring counter example is the standard “am I the main program?” boilerplate,
where <code class="docutils literal notranslate"><span class="pre">__name__</span></code> is expected to be “__main__”.
This PEP explicitly preserves that semantic.</p>
</section>
</section>
<section id="reference-implementation">
<h2><a class="toc-backref" href="#reference-implementation" role="doc-backlink">Reference Implementation</a></h2>
<p><a class="reference external" href="https://bugs.python.org/issue36375">BPO 36375</a> is the issue tracker entry
for the PEPs reference implementation, with the current draft PR being
available <a class="reference external" href="https://github.com/python/cpython/pull/12490">on GitHub</a>.</p>
</section>
<section id="open-questions">
<h2><a class="toc-backref" href="#open-questions" role="doc-backlink">Open Questions</a></h2>
<p>This proposal does raise some backwards compatibility concerns, and these will
need to be well understood, and either a deprecation process designed, or clear
porting guidelines provided.</p>
<section id="pickle-compatibility">
<h3><a class="toc-backref" href="#pickle-compatibility" role="doc-backlink">Pickle compatibility</a></h3>
<p>If no changes are made to the pickle module, then pickles that were previously
being written with the correct module name (due to a dual import) may start
being written with <code class="docutils literal notranslate"><span class="pre">__main__</span></code> as their module name instead, and hence fail
to be loaded correctly by other projects.</p>
<p>Scenarios to be checked:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">script.py</span></code> writing, <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">-m</span> <span class="pre">script</span></code> reading</li>
<li><code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">-m</span> <span class="pre">script</span></code> writing, <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">script.py</span></code> reading</li>
<li><code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">-m</span> <span class="pre">script</span></code> writing, <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">some_other_app.py</span></code> reading</li>
<li><code class="docutils literal notranslate"><span class="pre">old_python</span> <span class="pre">-m</span> <span class="pre">script</span></code> writing, <code class="docutils literal notranslate"><span class="pre">new_python</span> <span class="pre">-m</span> <span class="pre">script</span></code> reading</li>
<li><code class="docutils literal notranslate"><span class="pre">new_python</span> <span class="pre">-m</span> <span class="pre">script</span></code> writing, <code class="docutils literal notranslate"><span class="pre">old_python</span> <span class="pre">-m</span> <span class="pre">script</span></code> reading</li>
</ul>
</section>
<section id="projects-that-special-case-main">
<h3><a class="toc-backref" href="#projects-that-special-case-main" role="doc-backlink">Projects that special-case <code class="docutils literal notranslate"><span class="pre">__main__</span></code></a></h3>
<p>In order to get the regression test suite to pass, the current reference
implementation had to patch <code class="docutils literal notranslate"><span class="pre">pdb</span></code> to avoid destroying its own global
namespace.</p>
<p>This suggests there may be a broader compatibility issue where some scripts are
relying on direct execution and import giving different namespaces (just as
package execution keeps the two separate by executing the <code class="docutils literal notranslate"><span class="pre">__main__</span></code>
submodule in the <code class="docutils literal notranslate"><span class="pre">__main__</span></code> namespace, while the package name references
the <code class="docutils literal notranslate"><span class="pre">__init__</span></code> file as usual.</p>
</section>
</section>
<section id="background">
<h2><a class="toc-backref" href="#background" role="doc-backlink">Background</a></h2>
<p><a class="reference external" href="https://mail.python.org/pipermail/python-list/2015-August/694905.html">I tripped over this issue</a> while debugging a main program via a
module which tried to monkey patch a named module, that being the
main program module. Naturally, the monkey patching was ineffective
as it imported the main module by name and thus patched the second
module instance, not the running module instance.</p>
<p>However, the problem has been around as long as the <code class="docutils literal notranslate"><span class="pre">-m</span></code> command
line option and is encountered regularly, if infrequently, by others.</p>
<p>In addition to <a class="reference external" href="http://bugs.python.org/issue19702">issue 19702</a>, the discrepancy around <code class="docutils literal notranslate"><span class="pre">__main__</span></code>
is alluded to in <a class="pep reference internal" href="../pep-0451/" title="PEP 451 A ModuleSpec Type for the Import System">PEP 451</a> and a similar proposal (predating <a class="pep reference internal" href="../pep-0451/" title="PEP 451 A ModuleSpec Type for the Import System">PEP 451</a>)
is described in <a class="pep reference internal" href="../pep-0395/" title="PEP 395 Qualified Names for Modules">PEP 395</a> under
<a class="pep reference internal" href="../pep-0395/#fixing-dual-imports-of-the-main-module" title="PEP 395 Qualified Names for Modules § Fixing dual imports of the main module">Fixing dual imports of the main module</a>.</p>
</section>
<section id="copyright">
<h2><a class="toc-backref" href="#copyright" role="doc-backlink">Copyright</a></h2>
<p>This document has been placed in the public domain.</p>
</section>
</section>
<hr class="docutils" />
<p>Source: <a class="reference external" href="https://github.com/python/peps/blob/main/peps/pep-0499.rst">https://github.com/python/peps/blob/main/peps/pep-0499.rst</a></p>
<p>Last modified: <a class="reference external" href="https://github.com/python/peps/commits/main/peps/pep-0499.rst">2023-10-11 12:05:51 GMT</a></p>
</article>
<nav id="pep-sidebar">
<h2>Contents</h2>
<ul>
<li><a class="reference internal" href="#pep-deferral">PEP Deferral</a></li>
<li><a class="reference internal" href="#abstract">Abstract</a></li>
<li><a class="reference internal" href="#proposal">Proposal</a></li>
<li><a class="reference internal" href="#considerations-and-prerequisites">Considerations and Prerequisites</a><ul>
<li><a class="reference internal" href="#pickling-modules">Pickling Modules</a></li>
<li><a class="reference internal" href="#a-normal-module-s-name-is-no-longer-canonical">A Normal Modules <code class="docutils literal notranslate"><span class="pre">__name__</span></code> Is No Longer Canonical</a></li>
</ul>
</li>
<li><a class="reference internal" href="#reference-implementation">Reference Implementation</a></li>
<li><a class="reference internal" href="#open-questions">Open Questions</a><ul>
<li><a class="reference internal" href="#pickle-compatibility">Pickle compatibility</a></li>
<li><a class="reference internal" href="#projects-that-special-case-main">Projects that special-case <code class="docutils literal notranslate"><span class="pre">__main__</span></code></a></li>
</ul>
</li>
<li><a class="reference internal" href="#background">Background</a></li>
<li><a class="reference internal" href="#copyright">Copyright</a></li>
</ul>
<br>
<a id="source" href="https://github.com/python/peps/blob/main/peps/pep-0499.rst">Page Source (GitHub)</a>
</nav>
</section>
<script src="../_static/colour_scheme.js"></script>
<script src="../_static/wrap_tables.js"></script>
<script src="../_static/sticky_banner.js"></script>
</body>
</html>