Python FAQ - How does Python perform tasks in multiple threads, such as downloading a large number of pictures

中文阅读

• Problem Description
• Example
• Reference

Problem Description

In Python programming, it is often necessary to use multi-threaded execution to improve efficiency. Search engines will give many kinds of answers, but which one is the most simple and effective?

Let’s take Python multi-thread downloading pictures as an example to show how Python uses multi-threading to perform tasks.

Example

When you need to download multiple images, using multithreading can speed up the download process. You can use the ThreadPoolExecutor class in Python’s concurrent.futures library to achieve this. The ThreadPoolExecutor class provides a simple interface for creating a thread pool and then submitting tasks to the thread pool. Here is an example code that uses the ThreadPoolExecutor class to download multiple images:

import requests
from concurrent.futures import ThreadPoolExecutor

def download_image(url):
    response = requests.get(url)
    filename = url.split("/")[-1]
    with open(filename, "wb") as f:
        f.write(response.content)

if __name__ == "__main__":
    urls = [
        "https://example.com/image1.jpg",
        "https://example.com/image2.jpg",
        "https://example.com/image3.jpg",
        "https://example.com/image4.jpg",
        "https://example.com/image5.jpg"
    ]
    with ThreadPoolExecutor(max_workers=5) as executor:
        executor.map(download_image, urls)

In this example, we defined a download_image function that takes a URL as an argument and then uses the requests library to download the image and save it to the local file system. We also defined a list urls that contains multiple URLs. Finally, we used the ThreadPoolExecutor class to create a thread pool with a maximum of 5 working threads and applied the download_image function to each URL in the urls list using the map method.

Reference

• https://www.pythontutorial.net/python-concurrency/python-threadpoolexecutor/
• https://www.yisu.com/zixun/773580.html
• https://blog.csdn.net/wulong710/article/details/109435945
• https://www.jianshu.com/p/6d6e4f745c27
• https://docs.python.org/zh-cn/3/library/concurrent.futures.html
• https://www.cnblogs.com/tujia/p/13565799.html