Is synchronization with multiprocessor required in python?

Bobo

When using code like this

def execute_run(list_out): 
   ... do something 

pool = ThreadPoolExecutor(6)
for i in list1:
    for j in list2:
            pool.submit(myfunc, list_out)
pool.join()

Assuming threads modify list_out, do they operate in a synchronous fashion?

Jimmy Lane

If your goal is to compute something in a multiprocessing fashion, it's best not to share state. I recommend that you use simple multiprocessingmap whenever possible :

from multiprocessing import Pool

input_list = []
for i in list1:
    for j in list2:
        input_list.append((i, j))

p = Pool()
result_list = p.map(do_something, input_list)

mapSimilar to a for loop:

def naive_map(input_list, do_something):
   result = []
   for i in input_list:
     result.append(do_something(i))
   return result

So if you want to use a function that accepts multiple arguments, you can use a lambda function to unpack the tuple.

  >> def your_function(v1, v2):
  >>        return v1+v2
  >> f = lambda (x,y): your_function(x, y)
  >> map(f, [(1,2),(3,4),(5,6)])
       [3, 7, 11]

Related

Is synchronization with multiprocessor required in python?

Bobo When using code like this def execute_run(list_out): ... do something pool = ThreadPoolExecutor(6) for i in list1: for j in list2: pool.submit(myfunc, list_out) pool.join() Assuming threads modify list_out, do they operate in a sync

Threading and Multiprocessor Python

Rawad Sabbar ： The following script uses python threads: import logging import threading import time def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name) if ____name____ ==

Threading and Multiprocessor Python

Rawad Sabbar ： The following script uses python threads: import logging import threading import time def thread_function(name): logging.info("Thread %s: starting", name) time.sleep(2) logging.info("Thread %s: finishing", name) if ____name____ ==

With ConcurrentHashMap, when is synchronization required?

For each position: I have a ConcurrentHashMap where I do the following: sequences = new ConcurrentHashMap<Class<?>, AtomicLong>(); if(!sequences.containsKey(table)) { synchronized (sequences) { if(!sequences.containsKey(table)) initial

With ConcurrentHashMap, when is synchronization required?

For each position: I have a ConcurrentHashMap where I do the following: sequences = new ConcurrentHashMap<Class<?>, AtomicLong>(); if(!sequences.containsKey(table)) { synchronized (sequences) { if(!sequences.containsKey(table)) initial

Synchronization is required in getters and setters

snow leopard Probably a very stupid question. Just wanted to confirm my understanding. class Test { private volatile String id; public void setID(String id) { this.id = id; } public String getID() {

Synchronization is required in asynchronous code

Cyrillic I'm trying to create a factory for an angular module that returns a JSON object received through angular $http.get(). In the for's callback function success(), I am trying to assign an object to a variable products. It seems to productsbe returned fro

Synchronization is required in getters and setters

snow leopard Probably a very stupid question. Just wanted to confirm my understanding. class Test { private volatile String id; public void setID(String id) { this.id = id; } public String getID() {

With ConcurrentHashMap, when is synchronization required?

For each position: I have a ConcurrentHashMap where I do the following: sequences = new ConcurrentHashMap<Class<?>, AtomicLong>(); if(!sequences.containsKey(table)) { synchronized (sequences) { if(!sequences.containsKey(table)) initial

Synchronization is required in getters and setters

snow leopard Probably a very stupid question. Just wanted to confirm my understanding. class Test { private volatile String id; public void setID(String id) { this.id = id; } public String getID() {

Only one thread is required for synchronization

Chapter 135 If I have a class that extends Thread with static methods (this is very simplified): public class MyThread extends Thread { private static long SLEEP_INT = 30000; private static Map<Integer, String> myData; //every 30 seconds, update

When using synchronization, atomic references are not required

Maxim Dmitriev The code is from the Java Concurrency Guidelines by Fred Long . I understand that a set of atomic operations is not atomic. Therefore, the following code does not meet the requirements. To find the code, see page 23. public class Adder { pr

When using synchronization, atomic references are not required

Maxim Dmitriev The code is from the Java Concurrency Guidelines by Fred Long . I understand that a set of atomic operations is not atomic. Therefore, the following code does not meet the requirements. To find the code, see page 23. public class Adder { pr

Multiprocessor Scheduling

Angelo Regarding Linux 2.4 and Unix SRV4, I don't understand if a realtime process can preempt a timeshared process, otherwise the timeshared process will overtake the timeshared process. I think so because that's why realtime scheduling was introduced. Angelo

Multiprocessor Scheduling

Angelo Regarding Linux 2.4 and Unix SRV4, I don't understand if a realtime process can preempt a timeshared process, otherwise the timeshared process will overtake the timeshared process. I think so because that's why realtime scheduling was introduced. Angelo

Does Python have synchronization?

dfrankow： The synchronization man page says: sync() causes all buffered modifications to file metadata and data to be written to the underlying filesystem. Can Python make calls? PS not fsync , I see. bbayles： Python 3.3 has os.sync, see docs . The source conf

Python multiprocessing synchronization

fine particles I have a function "function" that I want to call 10 times using 2 by 5 cpus with multiprocessing. So I need a way to synchronize the process described in the code below. Is it possible to do this without using a multiprocessing pool? If I do thi