Training/openMP - Revision history

Pysdlb at 11:43, 15 November 2022

2022-11-15T11:43:13Z

Pysdlb at 15:38, 8 November 2022

2022-11-08T15:38:02Z

Pysdlb at 15:02, 24 April 2018

2018-04-24T15:02:16Z

Pysdlb at 14:45, 24 April 2018

2018-04-24T14:45:27Z

Pysdlb at 14:43, 24 April 2018

2018-04-24T14:43:57Z

Pysdlb at 14:31, 24 April 2018

2018-04-24T14:31:55Z

Pysdlb at 14:31, 24 April 2018

2018-04-24T14:31:20Z

Pysdlb at 14:30, 24 April 2018

2018-04-24T14:30:50Z

Pysdlb at 09:41, 24 April 2018

2018-04-24T09:41:27Z

Pysdlb at 08:26, 24 April 2018

2018-04-24T08:26:08Z

@@ Line 2: / Line 2: @@
 ==Introduction==
-openMP allows threaded programming across a shared memory system, so on our HPC, this means utilizing more than one processing core across one computing node.
+OpenMP allows threaded programming across a shared memory system, so on our HPC, this means utilizing more than one processing core across one computing node.
 A shared memory computer consists of a number of processing cores together with some memory.
@@ Line 14: / Line 14: @@
 ==Programming model==
-Within the idea of the shared memory model, we use the idea of '''threads''' which can share memory with all the other threads. These also have the following characteristics:
+Within the idea of the shared memory model, we use the idea of '''threads''' which can share the memory with all the other threads. These also have the following characteristics:
 * Private data can only be accessed by the thread owning it
@@ Line 135: / Line 135: @@
 ==Parallel loops==
-Loops are the main source of parallelism in many applications. If the iterations of a loop are independent (can be done in any order) then we can share out the iterations between different threads. openMP has a native call to do this efficiently:
+Loops are the main source of parallelism in many applications. If the iterations of a loop are independent (can be done in any order) then we can share out the iterations between different threads. OpenMP has a native call to do this efficiently:
@@ Line 165: / Line 165: @@
 </pre>
-There is a side effect of threading is called '''False sharing''' which can cause poor scaling. If independent data elements happen to sit on the same cache line, each update will cause the cache lines to “slosh back and forth” between threads hence each time the variable has to be re-loaded from main memory.
+There is a side effect of threading is called '''False sharing''' which can cause poor scaling. If independent data elements happen to sit on the same cache line, each update will cause the cache lines to “slosh back and forth” between threads hence each time the variable has to be re-loaded from the main memory.
 One solution is to pad arrays so elements are on distinct cache lines, another method is to allow single threads to operate on a cache line completely. A better way is to re-write the program so this effect is avoided, an example is given below:
@@ Line 311: / Line 311: @@
-== Further Information ==
+==Next Steps==
 {|

@@ Line 2: / Line 2: @@
 ==Introduction==
-openMP allows threaded programming across a shared memory system, so on our HPC this means utilizing more than one processing core across one computing node.
+openMP allows threaded programming across a shared memory system, so on our HPC, this means utilizing more than one processing core across one computing node.
 A shared memory computer consists of a number of processing cores together with some memory.
@@ Line 14: / Line 14: @@
 ==Programming model==
-Within the idea of the shared memory model we use the idea of '''threads''' which can share memory by all the other threads. These also have the following characteristics:
+Within the idea of the shared memory model, we use the idea of '''threads''' which can share memory with all the other threads. These also have the following characteristics:
 * Private data can only be accessed by the thread owning it
-* Each thread can run simultaneous to other threads but also asynchronously, so we need to be careful of race conditions.
+* Each thread can run simultaneously with other threads but also asynchronously, so we need to be careful of race conditions.
 * Usually we have one thread per processing core, although there may be hardware support for more (e.g. hyper-threading)
 ==Thread Synchronization==
-As previously mention threads execute asynchronous;y, which means each thread proceeds through program instruction independently of other threads.
+As previously mention threads execute asynchronously, which means each thread proceeds through program instruction independently of other threads.
-Although this makes for a very flexible system we must be very careful about the actions on shared variables occur in the correct order.
+Although this makes for a very flexible system we must be very careful about the actions on shared variables occurring in the correct order.
-* e.g. If we access a variable to read on thread 1 before thread 2 has written to it we will cause a program crash, likewise is updates to shared variables are accessed by different threads at the same time, one of the updates may get overwritten.
+* e.g. If we access a variable to read on thread 1 before thread 2 has written to it we will cause a program crash, likewise if updates to shared variables are accessed by different threads at the same time, one of the updates may get overwritten.
 To prevent this happen we must either use variables that are independent of the different threads (ie different parts of an array) or perform some sort of synchronization within the code so different threads get to the same point at the same time.
@@ Line 59: / Line 59: @@
 </pre>
-Lets look at the extra components to make this a parallel threaded program
+Let us look at the extra components to make this a parallel-threaded program
 * We have an openMP include file '''(#include <omp.h>''')
@@ Line 80: / Line 80: @@
 </pre>
-* Not very coherent but remember the threads all executed at different times, that is asynchronously and this is why we must be very careful about communicating between different threads of the same program.
+* Not very coherent but remember the threads all executed at different times, that is asynchronous and this is why we must be very careful about communicating between different threads of the same program.
 ==Second threaded program==
-Although the previous program is threaded it does not represent a real world example
+Although the previous program is threaded it does not represent a real-world example
 <pre>
@@ Line 103: / Line 103: @@
 Here each thread executes the same code independently, the only thing different is the omp thread ID is also passed to each call of pooh(ID,A).
-All threads again wait of the end of the closing parallel region brace to finish before proceeding (i.e. a synchronization barrier).
+All threads again wait for the end of the closing parallel region brace to finish before proceeding (i.e. a synchronization barrier).
-In this program we always expect 4 threads to be given to us by the underlying operating system. Unfortunately this may not happen and we will allocated what the scheduler is prepared to offer. This could cause us serious program difficulties if we rely on a fixed number of threads every time.
+In this program we always expect 4 threads to be given to us by the underlying operating system. Unfortunately, this may not happen and we will be allocated what the scheduler is prepared to offer. This could cause us serious program difficulties if we rely on a fixed number of threads every time.
 We must call the  openMP library (at runtime) how many threads we actually got, this is done with the following code:
@@ Line 149: / Line 149: @@
 </pre>
-This is a much neater way and allows the compiler to perform optimizations automatically (unless otherwise stated). The variable loop is made 'private' by each thread by default. Also all threads have to wait at the end of the parallel loop before proceeding past the end of this region.
+This is a much neater way and allows the compiler to perform optimizations automatically (unless otherwise stated). The variable loop is made 'private' by each thread by default. Also, all threads have to wait at the end of the parallel loop before proceeding past the end of this region.
 One OpenMP shortcut is to put the '''parallel pragma''' in the '''parallel for''' part, which just makes the code more readable.
@@ Line 165: / Line 165: @@
 </pre>
-There is an side effect of threading is called '''False sharing''' which can cause poor scaling. If independent data elements happen to sit on the same cache line, each update will cause the cache lines to “slosh back and forth” between threads hence each time the variable have to be re-loaded from main memory.
+There is a side effect of threading is called '''False sharing''' which can cause poor scaling. If independent data elements happen to sit on the same cache line, each update will cause the cache lines to “slosh back and forth” between threads hence each time the variable has to be re-loaded from main memory.
-One solution is to pad arrays so elements are on distinct cache lines, another method is to allow single threads to operate on a cache line completely. A better way is to re-write the program so this effect is avoiding, an example is given below:
+One solution is to pad arrays so elements are on distinct cache lines, another method is to allow single threads to operate on a cache line completely. A better way is to re-write the program so this effect is avoided, an example is given below:
 <pre>
@@ Line 202: / Line 202: @@
 * Here we create a scalar local to each thread to accumulate partial sums
 * No array, no false sharing
-* Within the parallel region we create a '''#pragma omp''' critical region to allow on one thread at a time to enter to perform the summation of the result of pi.
+* Within the parallel region we create a '''#pragma omp''' critical region to allow one thread at a time to enter to perform the summation of the result of pi.
-This solution will scale much better than using an array based design. Again because the above code of often seen in loops, openMP provides a reduction tool as well:
+This solution will scale much better than using an array-based design. Again because the above code of often seen in loops, openMP provides a reduction tool as well:
 <pre>
@@ Line 312: / Line 312: @@
 == Further Information ==
 {|

← Older revision		Revision as of 15:02, 24 April 2018
Line 301:		Line 301:
	\|}		\|}

		+	==Performance threading==
		+
		+	* Avoid false sharing possibilities, this will cause the greatest limitation to scaling
		+	* Does your program justify the overhead of threads, if you're parallelising out only a small '''for..loop''' the overhead cannot be justified and will increase processing time.
		+	* Best choice of schedule might change with the system
		+	* Minimize synchronisation, use nowait where practical
		+	* Locality, must systems are NUMA, modify your loop nest, or change loop order to get better cache behaviour

@@ Line 8: / Line 8: @@
 the whole memory system.
-* every processing core can read and write all memory locations in the system
+* Every processing core can read and write all memory locations in the system
-* one logical memory space
+* One logical memory space
-* all cores refer to a memory location using the same address
+* All cores refer to a memory location using the same address
 ==Programming model==
@@ Line 18: / Line 18: @@
 * Private data can only be accessed by the thread owning it
 * Each thread can run simultaneous to other threads but also asynchronously, so we need to be careful of race conditions.
-* Usually we have one thread per processing core, although there maybe hardware support for more (e.g. hyper-threading)
+* Usually we have one thread per processing core, although there may be hardware support for more (e.g. hyper-threading)
 ==Thread Synchronization==

@@ Line 229: / Line 229: @@
      #pragma omp barrier
      othercode();
 </pre>

@@ Line 249: / Line 249: @@
 {| class="wikitable"
-| style="width:25%" | <Strong>Package</Strong>
+| style="width:25%" | <Strong>OMP Construct</Strong>
 | style="width:75%" | <Strong>Description</Strong>
 |-

@@ Line 297: / Line 297: @@
 #pragma omp taskwait
 | tasks including the data environment for tasks.
 |-
 |}

← Older revision		Revision as of 14:30, 24 April 2018
Line 245:		Line 245:
	}		}
	</pre>		</pre>
		+
		+	==openMP pragmas==
		+
		+	{\| class="wikitable"
		+	\| style="width:25%" \| <Strong>Package</Strong>
		+	\| style="width:75%" \| <Strong>Description</Strong>
		+	\|-
		+	\| #pragma omp parallel
		+	\| parallel region, teams of threads, structured block, interleaved execution
		+	across threads
		+	\|-
		+	\| int omp_get_thread_num()
		+	int omp_get_num_threads()
		+	\| Create threads with a parallel region and split up the work using the
		+	number of threads and thread ID
		+	\|-
		+	\| double omp_get_wtime()
		+	\| speedup and Amdahl's law.
		+	False Sharing and other performance issues
		+	\|-
		+	\| setenv OMP_NUM_THREADS N
		+	\| internal control variables. Setting the default number of threads with an
		+	environment variable
		+	\|-
		+	\| #pragma omp barrier
		+	#pragma omp critical
		+	\| Synchronization and race conditions. Revisit interleaved execution.
		+	\|-
		+	\| #pragma omp for
		+	#pragma omp parallel for
		+	\| worksharing, parallel loops, loop carried dependencies
		+	\|-
		+	\| reduction(op:list)
		+	\| reductions of values across a team of threads
		+	\|-
		+	\| schedule(dynamic [,chunk])
		+	schedule (static [,chunk])
		+	\| Loop schedules, loop overheads and load balance
		+	\|-
		+	\| private(list), firstprivate(list), shared(list)
		+	\| Data environment
		+	\|-
		+	\| nowait
		+	\| disabling implied barriers on workshare constructs, the high cost of
		+	barriers. The flush concept (but not the concept)
		+	\|-
		+	\| #pragma omp single
		+	\| workshare with a single thread
		+	\|-
		+	\| #pragma omp task
		+	#pragma omp taskwait
		+	\| tasks including the data environment for tasks.
		+	\|-
		+	\|
		+	\|
		+	\|-
		+	\|
		+	\|
		+	\|-
		+	\|
		+	\|
		+	\|-
		+	\|
		+	\|
		+	\|-
		+	\|
		+	\|
		+	\|-
		+	\|}

@@ Line 125: / Line 125: @@
 </pre>
-* Each thread calls pooh(ID,A) for ID = 0 to nthrds-1
+* Each thread calls pooh(ID,A) for ID = 0 to '''nthrds-1'''
 ==Parallel loops==
-Loops are the main source of parallelism in many applications. If the iterations of a loop are independent (can be done in any order) then we can share out the iterations between different threads.
+Loops are the main source of parallelism in many applications. If the iterations of a loop are independent (can be done in any order) then we can share out the iterations between different threads. openMP has a native call to do this efficiently:
-* e.g. if we have two threads and the loop
+This solution will scale much better than using an array based design. Again because the above code of often seen in loops, openMP provides a reduction tool as well:
 <pre>
-for (i=0; i<100; i++)
+double ave=0.0, A[MAX];
+{
-     a[i] += b[i];
+     somecode();
+}
 </pre>
-we could do iteration 0-49 on one thread and iterations 50-99 on the

← Older revision		Revision as of 08:26, 24 April 2018
Line 104:		Line 104:

	All threads again wait of the end of the closing parallel region brace to finish before proceeding (i.e. a synchronization barrier).		All threads again wait of the end of the closing parallel region brace to finish before proceeding (i.e. a synchronization barrier).
		+
		+	In this program we always expect 4 threads to be given to us by the underlying operating system. Unfortunately this may not happen and we will allocated what the scheduler is prepared to offer. This could cause us serious program difficulties if we rely on a fixed number of threads every time.
		+
		+	We must call the openMP library (at runtime) how many threads we actually got, this is done with the following code:
		+
		+	<pre>
		+	#include<omp.h>
		+	#include<stdio.h>
		+	int main()
		+	{
		+	double A[1000];
		+	omp_set_num_threads(4);
		+	#pragma omp parallel
		+	{
		+	int ID = omp_get_thread_num();
		+	int nthrds = omp_get_num_threads();
		+	pooh(ID, A);
		+	}
		+	}
		+	</pre>
		+
		+	* Each thread calls pooh(ID,A) for ID = 0 to nthrds-1

	==Parallel loops==		==Parallel loops==