Is Python 4 true multi-core and multi-thread compatible like JAVA?

comment:

It is verification reading the blog of Python3.

Maybe

In Python2 & 3, because it is a fake multi-core & multi-thread, it is verified that it is different from a true multi-core & multi-thread like JAVA or Go lang (Go language).



True multi-core & multi-thread support with uvloop?

It may be possible to say.

aon CEO Masahiro Ishizuka

tel: 042-559-8638

iPhone: 070-3861-5011



Find out why asyncio is using POSIX threads
 2017-03-31
https://nwpct1.hatenablog.com/entry/asyncio-posix-threads

Shared.



Tokibito-sensei (id: nullpobug) called out to the office to play, so we went to the open collector. I had a chat with Mr. aodag (id: aodag), but I remembered what I was worried about before, so I asked him.

I was worried
While preparing the presentation material for a study session, I prepared a code example that sends an HTTP request to a server using asyncio and aiohttp.

import aiohttp
import asyncio

async def fetch (l, url):
    async with aiohttp.ClientSession (loop = l) as session:
        async with session.get (url) as response:
            return await response.text ()


async def main (l, url, num):
    tasks = [fetch (l, url) for _ in range (num)]
    return await asyncio.gather (* tasks)


if __name__ == '__main__':
    loop = asyncio.get_event_loop ()
    results = loop.run_until_complete (main (loop, 'http: // localhost: 8000', 3))
    for r in results:
        print (r)
PyCharm has a function to display the "Concurrency Diagram", and you can check the movement of threads and processes. The behavior of threads when this code is executed is as follows.

f: id: nwpct1: 20170330234242p: plain

For some reason, concurrent.futures.ThreadPoolExecutor has appeared. I tried to find out what was written in the document, but because I couldn't find any such descriptions, I gave up and asked Aodag and Tokibito.

socket.getaddrinfo
He found the cause in tens of minutes. Since socket.getaddrinfo is executed synchronously, it seems that the cpython implementation does not execute this asynchronously but instead executes it in multiple threads using concurrent.futures.ThreadPoolExecutor.

https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/asyncio/base_events.py#L666-L673
https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/asyncio/base_events.py#L627-L636
A sample using aioredis was prepared as a code sample in which getaddrinfo was not used. Connect to a locally built Redis server with a UNIX domain socket.

import asyncio
import aioredis

async def connection_example (key):
    conn = await aioredis.create_connection ('/ tmp / redis.sock')
    return await conn.execute ('GET', key)


async def main (num):
    tasks = [connection_example ('my-key') for _ in range (num)]
    return await asyncio.gather (* tasks)


if __name__ == '__main__':
    loop = asyncio.get_event_loop ()
    results = loop.run_until_complete (main (3))
    for r in results:
        print (r)
By the way, redis config is ↓.

daemonize no
pidfile /var/run/redis.pid
unixsocket /tmp/redis.sock
unixsocketperm 700
logfile ""
databases 1
Looking at the Concurrency Diagram at this time,

f: id: nwpct1: 20170331163806p: plain

Certainly no threads have been created.

Access to an external Redis server

On the other hand, if you prepare a Redis server outside and connect it (this time I used arukas.io),

import asyncio
import aioredis

async def connection_example (key):
    conn = await aioredis.create_connection (
        ('seaof-xxx-xxx.arukascloud.io', 311390),
        db = 0, password = 'xxxxxxxxxxxxxxx')
    return await conn.execute ('GET', key)


async def main (num):
    tasks = [connection_example ('my-key') for _ in range (num)]
    return await asyncio.gather (* tasks)


if __name__ == '__main__':
    loop = asyncio.get_event_loop ()
    results = loop.run_until_complete (main (3))
    for r in results:
        print (r)
When executed, it is as follows.

f: id: nwpct1: 20170331163620p: plain

It seems that a worker thread is created after all.

How many worker threads are created for ThreadPoolExecutor?
The Concurrency Diagram when the number of simultaneous executions in Semaphore is limited to 3 is as follows.

f: id: nwpct1: 20170331164208p: plain

Apparently, Thread in ThreadPoolExecutor is not reused. The extent to which it was created was described in the documentation under ThreadPoolExecutor.

If max_workers is None or not specified, the default value is the number of processors on the machine multiplied by 5.

17.4. Concurrent.futures – Concurrent task execution — Python 3.6.1 documentation

I send about 30 requests for a trial (Semaphore is 3).

f: id: nwpct1: 20170331164432p: plain

Up to 20 were generated, and it was confirmed that they were reused thereafter. Although the upper limit of the worker thread cannot be changed due to the implementation, it is unlikely that there will be any problems.

uvloop
As for the uvloop, I was told that I was working hard without using POSIX threads.

https://github.com/MagicStack/uvloop/blob/ee3f34cdd5bd265981d3d2e188d05b3f14d8d035/uvloop/loop.pyx#L625-L663
import uvloop

# Omitted
loop = uvloop.new_event_loop ()
asyncio.set_event_loop (loop)
When executed,

f: id: nwpct1: 20170330232815p: plain

Oh, really.

in conclusion
Aodag-sensei and tokibito-sensei are amazing ... I was reading and leaving the document humid. I'm also a member of society from tomorrow, so I'm going to work hard for two people.

An open collector with Mr. aodag and Mr. tokibito seems to be looking for a job. Thank you very much!