2020年1月24日金曜日

How many data can be processed per second by Python's Sanic framework?



 You can tell by looking at the page. It is 283,645 per second in JSON serialization.

However, that number has little practical meaning.

First, a typical Web server performs processes such as user authentication, database access, and HTML creation, so the processing time is much longer. It simply means that Sanic's processing time is negligible.

Also, there are not many cases where a web application needs more than 1000 processes per second. First, a web server such as Niginx can process and cache static files. If you do so, it is unlikely that your company's internal system of 10,000 employees will need to process 1000 jobs per second.

Even now, Dgango and Flask are mainly used because they are generally sufficient in many cases. If you have a problem, it's usually better to use the Go language, which is faster and has more parallel processing capabilities than using Sanic in Python. Note that this does not mean that Sanic is not used, but that if the number of processes per second becomes very large, Python itself is often not suitable for that process. .
Views : 80 times · Requested by Masahiro Ishizuka
Appreciate
share



Recommended order
all

Thank you for your answer.
> In general, it is often better to use the Go language, which is faster and has more parallel processing functions than using Sanic in Python.

I see, together


Conclusion

  •  The GIL has ensured that multithreading does not benefit from multicore.

If this blog is right, and we consider it together, Python is a programming language that doesn't take advantage of multi-core CPUs. If you don't need AI (artificial intelligence) or machine learning libraries, Go lang is better than Python. We hope that either the framework or the formal support of multi-core CPUs will be resolved in the future. There is also an AI, machine learning and numerical analysis library developed by Google called TensorFlow for Go lang . This API can be used in Go lang, but it is still only Python that can build and learn models, so I hope that it will be improved in the future.
Reply

In terms of GIL, it is usually not a problem because heavy processing can be eliminated by writing it in C / C ++ or using a library written in C / C ++. However, if the number of operations per second becomes very large, processing collisions cannot be avoided, and Python becomes inefficient. So I only recommend using Go. If it's not large, developing with Python alone is not a problem. For large scales, it may be necessary to calculate and test whether Python alone can withstand it.

As for machine learning, training uses a lot of data to do a lot of calculations. In this regard, Python is good at Go and not good at Go, so Go will never be able to build and learn machine learning models.

On the other hand, in the process using the learned result, the calculation is light because only the result is used. If you have to deal with a lot of processing here, there is an API for Go because Go is more suitable for the reasons mentioned above.

Both TesorFlow and Go have been developed by Google, but it's a good idea to consider the characteristics of Python and Go, so it won't be unified.
Reply
Appreciate

Show question
Introduction of respondents

1 follower you knowTakeo Yamazaki

Worked as a freelancer 2013–present

Majored in Science at Kyoto University

0–0 who lived in Tokushima

Contents viewed: 344.6k times This month: 26.9k times
Active in 1 space

----
Is Python 4 truly multi-core & multi-thread compatible?
comment:
It is verification reading the blog of Python3.
Maybe
In Python2 & 3, because it is a fake multi-core & multi-thread, it is verified that it is different from a true multi-core & multi-thread like JAVA or Go lang (Go language).

True multi-core & multi-thread support with uvloop?
It may be possible to say.
aon CEO Masahiro Ishizuka
tel: 042-559-8638
iPhone: 070-3861-5011

Find out why asyncio is using POSIX threads
  

Shared.

Tokibito-sensei ( id: nullpobug ) called out to the office to play, so we went to the open collector. I had a chat with Mr. aodag ( id: aodag ), but I remembered what I was worried about before.

I was worried

While preparing a presentation for a study session, I prepared a code example that sends an HTTP request to a server using asyncio and aiohttp.
  import aiohttp
 import asyncio

 async def fetch (l, url):
     async with aiohttp.ClientSession (loop = l) as session:
         async with session.get (url) as response:
             return await response.text ()


 async def main (l, url, num):
     tasks = [fetch (l, url) for _ in range (num)]
     return await asyncio.gather (* tasks)


 if __name__ == '__main__' :
     loop = asyncio.get_event_loop ()
     results = loop.run_until_complete (main (loop, 'http: // localhost: 8000' , 3 ))
     for r in results:
         print (r)
PyCharm has a function to display a "Concurrency Diagram", and you can check the movement of threads and processes. The behavior of threads when this code is executed is as follows.
f:id:nwpct1:20170330234242p:plain
For some reason, concurrent.futures.ThreadPoolExecutor is appearing. I tried to find out what was written in the document, but because I couldn't find any such descriptions, I gave up and asked Aodag and Tokibito.

socket.getaddrinfo

He found the cause in tens of minutes. socket.getaddrinfo is executed synchronously, it socket.getaddrinfo that the cpython implementation does not execute this asynchronously, but executes concurrent.futures.ThreadPoolExecutor multiple threads.
A sample using aioredis was prepared as a code sample in which getaddrinfo was not used. Connect to a locally built Redis server with a UNIX domain socket.
  import asyncio
 import aioredis

 async def connection_example (key):
     conn = await aioredis.create_connection ( '/tmp/redis.sock' )
     return await conn.execute ( 'GET' , key)


 async def main (num):
     tasks = [connection_example ( 'my-key' ) for _ in range (num)]
     return await asyncio.gather (* tasks)


 if __name__ == '__main__' :
     loop = asyncio.get_event_loop ()
     results = loop.run_until_complete (main ( 3 ))
     for r in results:
         print (r)
By the way, redis config is ↓.
  daemonize no
 pidfile /var/run/redis.pid
 unixsocket /tmp/redis.sock
 unixsocketperm 700
 logfile ""
 databases 1 
Looking at the Concurrency Diagram at this time,
f:id:nwpct1:20170331163806p:plain
Certainly no threads have been created.
Access to an external Redis server
On the other hand, if you prepare a Redis server outside and connect it (this time I used arukas.io ),
  import asyncio
 import aioredis

 async def connection_example (key):
     conn = await aioredis.create_connection (
         ( 'seaof-xxx-xxx.arukascloud.io' , 311390 ),
         db = 0 , password = 'xxxxxxxxxxxxxxx' )
     return await conn.execute ( 'GET' , key)


 async def main (num):
     tasks = [connection_example ( 'my-key' ) for _ in range (num)]
     return await asyncio.gather (* tasks)


 if __name__ == '__main__' :
     loop = asyncio.get_event_loop ()
     results = loop.run_until_complete (main ( 3 ))
     for r in results:
         print (r)
When executed,
f:id:nwpct1:20170331163620p:plain
It seems that a worker thread is created after all.

How many worker threads are created for ThreadPoolExecutor?

The Concurrency Diagram when the number of simultaneous executions in Semaphore is limited to 3 is as follows.
f:id:nwpct1:20170331164208p:plain
Apparently, Thread in ThreadPoolExecutor is not reused. The extent to which it was created was described in the documentation under ThreadPoolExecutor.
If max_workers is None or not specified, the default value is the number of processors on the machine multiplied by 5.
17.4. Concurrent.futures – Concurrent task execution — Python 3.6.1 documentation
I send about 30 requests for a trial (Semaphore is 3).
f:id:nwpct1:20170331164432p:plain
Up to 20 were generated, and it was confirmed that they were reused thereafter. Although the upper limit of the worker thread cannot be changed due to the implementation, it is unlikely that there will be any problems.

uvloop

As for the uvloop, it was confirmed that they worked hard without using POSIX threads.
  import uvloop

 # Omitted
 loop = uvloop.new_event_loop ()
 asyncio.set_event_loop (loop)
When executed,
f:id:nwpct1:20170330232815p:plain
Oh, really.

in conclusion

Aodag-sensei and tokibito-sensei are amazing ... I was reading and leaving the document humid. I'm also a member of society from tomorrow, so I'm going to work hard for two people.
An open collector with Mr. aodag and Mr. tokibito seems to be looking for a job. Thank you very much!








0 コメント:

コメントを投稿