A question about how to read a csv file and output it

[qwerfdg]

Hi!
My environment are centos6.7, python3.7, and pymars0.9.0. When I run the demo you give in zhihu https://www.zhihu.com/question/320961999 , something seems to be wrong.
Here are my codes. I think maybe 'execute()' leads to the problem.

import mars.dataframe as md                                             

df = md.read_csv('bin_end.csv')

df.execute()

Here are the error logs.

/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py:2064: UserWarning: 
No session found, local session will be created in background, it may take a while before execution. If you want to new a local session by yourself, run code below:

import mars

mars.new_session()

  warnings.warn(warning_msg)
Traceback (most recent call last):
  File "my.py", line 5, in <module>
    df.execute()
  File "/usr/local/lib/python3.7/site-packages/mars/core/entity/tileables.py", line 462, in execute
    result = self.data.execute(session=session, **kw)
  File "/usr/local/lib/python3.7/site-packages/mars/core/entity/executable.py", line 144, in execute
    return execute(self, session=session, **kw)
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1888, in execute
    session = get_default_or_create(**(new_session_kwargs or dict()))
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 2065, in get_default_or_create
    session = new_session("127.0.0.1", init_local=True, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 2040, in new_session
    address, session_id=session_id, backend=backend, new=new, **kwargs
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1636, in init
    isolated_session = fut.result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 848, in init
    address, timeout=timeout, backend=backend, **kwargs
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 91, in new_cluster_in_isolation
    await cluster.start()
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 219, in start
    await self._start_service()
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 274, in _start_service
    config=self._config,
  File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/service.py", line 84, in start_worker
    NodeRole.WORKER, config, address=address, mark_ready=mark_ready
  File "/usr/local/lib/python3.7/site-packages/mars/services/core.py", line 174, in start_services
    await asyncio.gather(*[inst.start() for inst in instances])
  File "/usr/local/lib/python3.7/site-packages/mars/services/cluster/worker/service.py", line 68, in start
    address=address,
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/api.py", line 27, in create_actor
    return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 112, in create_actor
    return self._process_result_message(result)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 525, in create_actor
    await self._run_coro(message.message_id, actor.__post_create__())
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
    return await coro
  File "/usr/local/lib/python3.7/site-packages/mars/services/cluster/procinfo.py", line 41, in __post_create__
    allocate_strategy=ProcessIndex(index),
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/api.py", line 27, in create_actor
    return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 112, in create_actor
    return self._process_result_message(result)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
    raise message.as_instanceof_cause()
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 939, in create_actor
    result = await self.call(address, new_create_actor_message)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 402, in call
    return await self._caller.call(self._router, dest_address, message)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/core.py", line 95, in call
    await client.send(message)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/communication/base.py", line 258, in send
    return await self.channel.send(message)
  File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/communication/socket.py", line 73, in send
    buffers = await serializer.run()
  File "/usr/local/lib/python3.7/site-packages/mars/serialization/aio.py", line 85, in run
    return await self._get_buffers()
  File "/usr/local/lib/python3.7/site-packages/mars/serialization/aio.py", line 41, in _get_buffers
    self._obj, spawn_threshold=DEFAULT_SPAWN_THRESHOLD
  File "mars/serialization/core.pyx", line 766, in serialize_with_spawn
  File "mars/serialization/core.pyx", line 660, in mars.serialization.core._serialize_with_stack
  File "mars/serialization/core.pyx", line 598, in mars.serialization.core._serial_single
  File "mars/serialization/core.pyx", line 258, in mars.serialization.core.PickleSerializer.serial
  File "mars/serialization/core.pyx", line 223, in mars.serialization.core.pickle_buffers
  File "/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 640, in dump
    raise pickle.PicklingError(msg) from e
pickle5.pickle.PicklingError: [address=127.0.0.1:33346, pid=30517] Could not pickle object as excessively deep recursion required.

[qinxuye]

Is your data file sensitive? can you share the minimal data to help us to reproduce the error?

[qwerfdg]

Is your data file sensitive? can you share the minimal data to help us to reproduce the error?

It is just a randomly generated test data set.

38628
959698
553744
22201
191673
943405
526206
941645
678506
838648

[DEVdv2006]

It seems like there's an issue with the serialization process, possibly due to the interaction between pymars and the underlying communication mechanisms.

check the compablity of different libraries and update if necessary also instead of directly trying execute() try different approaches provided by pymars