Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xdoctest][task skip_no_reason 8-11] add skip reason for jit sample code #57676

Merged
merged 3 commits into from
Sep 27, 2023
Merged

[xdoctest][task skip_no_reason 8-11] add skip reason for jit sample code #57676

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ded3539
[xdoctest][task skip_no_reason 8-11] add skip reason for jit sample code
SigureMo Sep 24, 2023
16d896c
fix some api name
SigureMo Sep 24, 2023
2dd17c0
Merge branch 'develop' into xdoctest/skip-reason/jit
SigureMo Sep 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 26 additions & 21 deletions python/paddle/jit/api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ def _extract_vars(inputs, result_list, err_tag='inputs'):
_extract_vars(var, result_list, err_tag)
else:
raise TypeError(
"The type of 'each element of {}' in paddle.jit.TracedLayer.trace must be base.Variable, but received {}.".format(
"The type of 'each element of {}' in paddle.jit.api.TracedLayer.trace must be base.Variable, but received {}.".format(
err_tag, type(inputs)
)
)
Expand Down Expand Up @@ -129,6 +129,7 @@ def _dygraph_to_static_func_(dygraph_func):
Examples:
.. code-block:: python

>>> # doctest: +SKIP('`paddle.jit.dygraph_to_static_func` can not run in xdoctest')
>>> import paddle
>>> from paddle.jit.api import dygraph_to_static_func

Expand Down Expand Up @@ -264,7 +265,7 @@ def to_static(
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> from paddle.jit import to_static

Expand Down Expand Up @@ -361,7 +362,7 @@ def not_to_static(func=None):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle

>>> @paddle.jit.not_to_static
Expand Down Expand Up @@ -647,7 +648,6 @@ def _get_output_vars(outputs, output_spec, with_hook=False):
# 1. Expected cases:
# - paddle.jit.save
# - paddle.static.save_inference_model
# - paddle.base.io.save_inference_model
# 2. Error cases:
# - paddle.save: no .pdmodel for prefix
# - paddle.static.save: no .pdiparams but .pdparams exists
Expand Down Expand Up @@ -729,6 +729,7 @@ def _register_save_pre_hook(hook):
Examples:
.. code-block:: python

>>> # doctest: +SKIP('`paddle.jit.api.to_static` can not run in xdoctest')
>>> import numpy as np
>>> import paddle

Expand Down Expand Up @@ -876,7 +877,7 @@ def save(layer, path, input_spec=None, **configs):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> # example 1: save layer
>>> import numpy as np
>>> import paddle
Expand Down Expand Up @@ -1335,7 +1336,7 @@ def load(path, **configs):
:api_attr: imperative

Load model saved by ``paddle.jit.save`` or ``paddle.static.save_inference_model`` or
paddle 1.x API ``paddle.base.io.save_inference_model`` as ``paddle.jit.TranslatedLayer``,
paddle 1.x API ``paddle.static.save_inference_model`` as ``paddle.jit.TranslatedLayer``,
then performing inference or fine-tune training.

.. note::
Expand Down Expand Up @@ -1368,7 +1369,7 @@ def load(path, **configs):
.. code-block:: python
:name: code-example1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

奇怪，code-example2 一直不过，但本地可以过 @megemini

image

看着像多进程导致的问题？但我用的是最新版 convert-doctest 呀

另外 TracedLayer 三个报错先不用管，TracedLayer 目前已经不用了

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

嗯 ... ... 之前其实一直担心，使用多进程可能会对 paddle 本身有一定副作用 ～

这里 paddle 内部也用了多进程（应该是加载数据用的），不能嵌套进程，即使按照报错信息不用 daemon，也会因为不能 pickle 而报错 ～

有个办法，加个全局 directive，让这个用例不在子进程运行，看看行不？

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里 paddle 内部也用了多进程（应该是加载数据用的），不能嵌套进程，即使按照报错信息不用 daemon，也会因为不能 pickle 而报错 ～

可是 convert-doctest 为什么没问题呢？这俩按理说是一样的呀？

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我在 aistudio 上也报错 ～

你是在本地 mac 运行的？

不同系统对于 multiprocessing 处理不一样 ... ...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

paddle 目前强制用的 fork：

Paddle/python/paddle/reader/decorator.py

Lines 29 to 42 in 395ffbf

# On macOS, the 'spawn' start method is now the default in Python3.8 multiprocessing,
# Paddle is currently unable to solve this, so forces the process to start using
# the 'fork' start method.
#
# TODO: This solution is not good, because the fork start method could lead to
# crashes of the subprocess. Figure out how to make 'spawn' work.
#
# For more details, please refer to
# https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
# https://bugs.python.org/issue33725
if sys.version_info >= (3, 8) and sys.platform == 'darwin':
fork_context = multiprocessing.get_context('fork')
else:
fork_context = multiprocessing

出错的 _DataLoaderIterMultiProcess 也是用的默认的 multiprocessing.Process，linux 下是 fork，mac 下是 spawn ～

fork is fast, unsafe, and maybe bloated，另参考 Fork vs Spawn in Python Multiprocessing, 所以当时我是强制用的 spawn。

我试过用 fork，虽然快，但是问题更多 ... ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

你是在本地 mac 运行的？

嗯嗯对

有个办法，加个全局 directive，让这个用例不在子进程运行，看看行不？

那看样子只能这样了～

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR： #57692 🤔


>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import numpy as np
>>> import paddle
>>> import paddle.nn as nn
Expand Down Expand Up @@ -1454,7 +1455,7 @@ def load(path, **configs):
>>> train(loaded_layer, loader, loss_fn, adam)


2. Load model saved by ``paddle.base.io.save_inference_model`` then performing and fine-tune training.
2. Load model saved by ``paddle.static.save_inference_model`` then performing and fine-tune training.

.. code-block:: python
:name: code-example2
Expand Down Expand Up @@ -1522,8 +1523,12 @@ def load(path, **configs):
... )

>>> model_path = "fc.example.model"
>>> paddle.base.io.save_inference_model(
>>> model_path, ["image"], [pred], exe)
>>> paddle.static.save_inference_model(
... model_path,
... [image],
... [pred],
... exe
... )

>>> # 2. load model

Expand Down Expand Up @@ -1689,7 +1694,7 @@ def trace(layer, inputs):

>>> layer = ExampleLayer()
>>> in_var = paddle.uniform(shape=[2, 3], dtype='float32')
>>> out_dygraph, static_layer = paddle.jit.TracedLayer.trace(layer, inputs=[in_var])
>>> out_dygraph, static_layer = paddle.jit.api.TracedLayer.trace(layer, inputs=[in_var])

>>> # run the static graph model using Executor inside
>>> out_static_graph = static_layer([in_var])
Expand All @@ -1703,7 +1708,7 @@ def trace(layer, inputs):
"""
assert isinstance(
layer, Layer
), "The type of 'layer' in paddle.jit.TracedLayer.trace must be paddle.nn.Layer, but received {}.".format(
), "The type of 'layer' in paddle.jit.api.TracedLayer.trace must be paddle.nn.Layer, but received {}.".format(
type(layer)
)
outs, prog, feed, fetch, parameters = _trace(layer, inputs)
Expand Down Expand Up @@ -1739,7 +1744,7 @@ def set_strategy(self, build_strategy=None, exec_strategy=None):
>>> layer = ExampleLayer()
>>> in_var = paddle.uniform(shape=[2, 3], dtype='float32')

>>> out_dygraph, static_layer = paddle.jit.TracedLayer.trace(layer, inputs=[in_var])
>>> out_dygraph, static_layer = paddle.jit.api.TracedLayer.trace(layer, inputs=[in_var])

>>> build_strategy = paddle.static.BuildStrategy()
>>> build_strategy.enable_inplace = True
Expand All @@ -1754,12 +1759,12 @@ def set_strategy(self, build_strategy=None, exec_strategy=None):
assert self._compiled_program is None, "Cannot set strategy after run"
assert isinstance(
build_strategy, (type(None), BuildStrategy)
), "The type of 'build_strategy' in paddle.jit.TracedLayer.set_strategy must be base.BuildStrategy, but received {}.".format(
), "The type of 'build_strategy' in paddle.jit.api.TracedLayer.set_strategy must be base.BuildStrategy, but received {}.".format(
type(build_strategy)
)
assert isinstance(
exec_strategy, (type(None), ExecutionStrategy)
), "The type of 'exec_strategy' in paddle.jit.TracedLayer.set_strategy must be base.ExecutionStrategy, but received {}.".format(
), "The type of 'exec_strategy' in paddle.jit.api.TracedLayer.set_strategy must be base.ExecutionStrategy, but received {}.".format(
type(exec_strategy)
)
self._build_strategy = build_strategy
Expand Down Expand Up @@ -1845,7 +1850,7 @@ def save_inference_model(self, path, feed=None, fetch=None, **kwargs):
>>> in_var = paddle.to_tensor(in_np)
>>> layer = ExampleLayer()

>>> out_dygraph, static_layer = paddle.jit.TracedLayer.trace(layer, inputs=[in_var])
>>> out_dygraph, static_layer = paddle.jit.api.TracedLayer.trace(layer, inputs=[in_var])
>>> static_layer.save_inference_model(save_dirname, feed=[0], fetch=[0])

>>> paddle.enable_static()
Expand All @@ -1864,35 +1869,35 @@ def save_inference_model(self, path, feed=None, fetch=None, **kwargs):
path,
"path",
str,
"paddle.jit.TracedLayer.save_inference_model",
"paddle.jit.api.TracedLayer.save_inference_model",
)
check_type(
feed,
"feed",
(type(None), list),
"paddle.jit.TracedLayer.save_inference_model",
"paddle.jit.api.TracedLayer.save_inference_model",
)
if isinstance(feed, list):
for f in feed:
check_type(
f,
"each element of feed",
int,
"paddle.jit.TracedLayer.save_inference_model",
"paddle.jit.api.TracedLayer.save_inference_model",
)
check_type(
fetch,
"fetch",
(type(None), list),
"paddle.jit.TracedLayer.save_inference_model",
"paddle.jit.api.TracedLayer.save_inference_model",
)
if isinstance(fetch, list):
for f in fetch:
check_type(
f,
"each element of fetch",
int,
"paddle.jit.TracedLayer.save_inference_model",
"paddle.jit.api.TracedLayer.save_inference_model",
)
clip_extra = kwargs.get('clip_extra', True)
# path check
Expand Down
2 changes: 1 addition & 1 deletion python/paddle/jit/dy2static/convert_call_func.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ def convert_call(func):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> from paddle.jit.dy2static import Call

Expand Down
14 changes: 7 additions & 7 deletions python/paddle/jit/dy2static/program_translator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -559,7 +559,7 @@ def rollback(self):
Example::
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle

>>> class Net(paddle.nn.Layer):
Expand Down Expand Up @@ -906,7 +906,7 @@ def concrete_program(self):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> from paddle.jit import to_static
>>> from paddle.static import InputSpec
Expand Down Expand Up @@ -1778,7 +1778,7 @@ def enable(self, enable_to_static):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> def func(x):
... if paddle.mean(x) > 0:
Expand Down Expand Up @@ -1821,7 +1821,7 @@ def get_output(self, dygraph_func, *args, **kwargs):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> def func(x):
... if paddle.mean(x) > 0:
Expand Down Expand Up @@ -1904,7 +1904,7 @@ def get_func(self, dygraph_func):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> def func(x):
... if paddle.mean(x) > 0:
Expand Down Expand Up @@ -1953,7 +1953,7 @@ def get_program(self, dygraph_func, *args, **kwargs):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> def func(x):
... if paddle.mean(x) > 0:
Expand Down Expand Up @@ -2020,7 +2020,7 @@ def get_code(self, dygraph_func):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import paddle
>>> def func(x):
... if paddle.mean(x) > 0:
Expand Down
4 changes: 2 additions & 2 deletions python/paddle/jit/translated_layer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1316,7 +1316,7 @@ class TranslatedLayer(layers.Layer):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import numpy as np
>>> import paddle
>>> import paddle.nn as nn
Expand Down Expand Up @@ -1526,7 +1526,7 @@ def program(self, method_name='forward'):
Examples:
.. code-block:: python

>>> # doctest: +SKIP
>>> # doctest: +SKIP('`paddle.jit.to_static` can not run in xdoctest')
>>> import numpy as np
>>> import paddle
>>> from paddle import nn
Expand Down