docs/source/parallel-simulations.rst - ndnSIM - Gitiles

 How to speed up simulations by parallel execution
 -------------------------------------------------

 A way to speed up your simulations is to run them in parallel taking advantage of the power of
 all the processors and the memory availability of your machine. This can be done by using the
 Message Passing Interface (MPI) along with the distributed simulator class `provided by NS-3
 <http://www.nsnam.org/docs/models/html/distributed.html#mpi-for-distributed-simulation>`_.

 To make use of MPI, the network topology needs to be partitioned in a proper way, as the
 potential speedup will not be able to exceed the number of topology partitions. However, it
 should be noted that dividing the simulation for distributed purposes in NS-3 can only occur
 across point-to-point links.  Currently, only the applications running on a node can be
 executed in a separate logical processor, while the whole network topology will be created in
 each parallel execution.  Lastly, MPI requires the exchange of messages among the logical
 processors, thus imposing a communication overhead during the execution time.

 Designing a parallel simulation scenario
 ----------------------------------------

 In order to run simulation scenarios using MPI, all you need is to partition your network
 topology in a proper way.  That is to say, to maximize benefits of the parallelization, you
 need to equally distribute the workload for each logical processor.

 The full topology will always be created in each parallel execution (on each "rank" in MPI
 terms), regardless of the individual node system IDs.  Only the applications are specific to a
 rank. For example, consider node 1 on logical processor (LP) 1 and node 2 on LP 2, with a
 traffic generator on node 1. Both node 1 and node 2 will be created on both LP 1 and LP 2;
 however, the traffic generator will only be installed on LP 1. While this is not optimal for
 memory efficiency, it does simplify routing, since all current routing implementations in ns-3
 will work with distributed simulation.

 For more information, you can take a look at the `NS-3 MPI documentation
 <http://www.nsnam.org/docs/models/html/distributed.html#mpi-for-distributed-simulation>`_.

 Compiling and running ndnSIM with MPI support
 ---------------------------------------------

 - Install MPI

     On Ubuntu:

     .. code-block:: bash

         sudo apt-get install openmpi-bin openmpi-common openmpi-doc libopenmpi-dev

     On Fedora:

     .. code-block:: bash

         sudo yum install openmpi openmpi-devel

     On OS X with HomeBrew:

     .. code-block:: bash

         brew install open-mpi

 - Compile ndnSIM with MPI support

     You can compile ndnSIM with MPI support using ./waf configure by adding the parameter
     ``--enable-mpi`` along with the parameters of your preference. For example, to configure
     with examples and MPI support in optimized mode:

     .. code-block:: bash

         cd <ns-3-folder>
         ./waf configure -d optimized --enable-examples --enable-mpi

 - Run ndnSIM with MPI support

     To run a simulation scenario using MPI, you need to type:

     .. code-block:: bash

         mpirun -np <number_of_processors> ./waf --run=<scenario_name>


 .. _simple scenario with MPI support:

 Simple parallel scenario using MPI
 ----------------------------------

 This scenario simulates a network topology consisting of two nodes in parallel. Each node
 is assigned to a dedicated logical processor.

 The default parallel synchronization strategy implemented in the DistributedSimulatorImpl
 class is based on a globally synchronized algorithm using an MPI collective operation to
 synchronize simulation time across all LPs. A second synchronization strategy based on local
 communication and null messages is implemented in the NullMessageSimulatorImpl class, For
 the null message strategy the global all to all gather is not required; LPs only need to
 communication with LPs that have shared point-to-point links. The algorithm to use is
 controlled by which the ns-3 global value SimulatorImplementationType.

 The strategy can be selected according to the value of nullmsg. If nullmsg is true, then
 the local communication strategy is selected. If nullmsg is false, then the globally
 synchronized strategy is selected.  This parameter can be passed either as a command line
 argument or by directly modifying the simulation scenario.

 The best algorithm to use is dependent on the communication and event scheduling pattern for
 the application. In general, null message synchronization algorithms will scale better due
 to local communication scaling better than a global all-to-all gather that is required by
 DistributedSimulatorImpl. There are two known cases where the global synchronization performs
 better. The first is when most LPs have point-to-point link with most other LPs, in other
 words the LPs are nearly fully connected. In this case the null message algorithm will
 generate more message passing traffic than the all-to-all gather. A second case where the
 global all-to-all gather is more efficient is when there are long periods of simulation time
 when no events are occurring. The all-to-all gather algorithm is able to quickly determine
 then next event time globally. The nearest neighbor behavior of the null message algorithm
 will require more communications to propagate that knowledge; each LP is only aware of
 neighbor next event times.

 The following code represents all that is necessary to run such this simple parallel scenario

 .. literalinclude:: ../../examples/ndn-simple-mpi.cpp
    :language: c++
    :linenos:
    :lines: 22-35,71-
    :emphasize-lines: 41-44, 54-58, 78-79, 89-90

 If this code is placed into ``scratch/ndn-simple-mpi.cpp`` or NS-3 is compiled with examples
 enabled, you can compare runtime on one and two CPUs using the following commands::

     # 1 CPU
     mpirun -np 1 ./waf --run=ndn-simple-mpi

     # 2 CPUs
     mpirun -np 2 ./waf --run=ndn-simple-mpi


 The following table summarizes 9 executions on OS X 10.10 and 2.3 GHz Intel Core i7 a single
 CPU, on two CPUs with global synchronization, and on two CPUs with null message
 synchronization:

 +-------------+-----------------+------------------+----------------+
 | # of CPUs   | Real time, s    | User time, s     | System time, s |
 +=============+=================+==================+================+
 | 1           | 20.9 +- 0.14    | 20.6 +- 0.13     | 0.2 +- 0.01    |
 +-------------+-----------------+------------------+----------------+
 | 2 (global)  | 11.1 +- 0.13    | 21.9 +- 0.24     | 0.2 +- 0.02    |
 +-------------+-----------------+------------------+----------------+
 | 2 (nullmsg) | 11.4 +- 0.12    | 22.4 +- 0.21     | 0.2 +- 0.02    |
 +-------------+-----------------+------------------+----------------+

 Note that MPI not always will result in simulation speedup and can actually result in
 performance degradation.  This means that either network is not properly partitioned or the
 simulation cannot take advantage of the partitioning (e.g., the simulation time is dominated by
 the application on one node).
	How to speed up simulations by parallel execution
	-------------------------------------------------

	A way to speed up your simulations is to run them in parallel taking advantage of the power of
	all the processors and the memory availability of your machine. This can be done by using the
	Message Passing Interface (MPI) along with the distributed simulator class `provided by NS-3
	<http://www.nsnam.org/docs/models/html/distributed.html#mpi-for-distributed-simulation>`_.

	To make use of MPI, the network topology needs to be partitioned in a proper way, as the
	potential speedup will not be able to exceed the number of topology partitions. However, it
	should be noted that dividing the simulation for distributed purposes in NS-3 can only occur
	across point-to-point links. Currently, only the applications running on a node can be
	executed in a separate logical processor, while the whole network topology will be created in
	each parallel execution. Lastly, MPI requires the exchange of messages among the logical
	processors, thus imposing a communication overhead during the execution time.

	Designing a parallel simulation scenario
	----------------------------------------

	In order to run simulation scenarios using MPI, all you need is to partition your network
	topology in a proper way. That is to say, to maximize benefits of the parallelization, you
	need to equally distribute the workload for each logical processor.

	The full topology will always be created in each parallel execution (on each "rank" in MPI
	terms), regardless of the individual node system IDs. Only the applications are specific to a
	rank. For example, consider node 1 on logical processor (LP) 1 and node 2 on LP 2, with a
	traffic generator on node 1. Both node 1 and node 2 will be created on both LP 1 and LP 2;
	however, the traffic generator will only be installed on LP 1. While this is not optimal for
	memory efficiency, it does simplify routing, since all current routing implementations in ns-3
	will work with distributed simulation.

	For more information, you can take a look at the `NS-3 MPI documentation
	<http://www.nsnam.org/docs/models/html/distributed.html#mpi-for-distributed-simulation>`_.

	Compiling and running ndnSIM with MPI support
	---------------------------------------------

	- Install MPI

	On Ubuntu:

	.. code-block:: bash

	sudo apt-get install openmpi-bin openmpi-common openmpi-doc libopenmpi-dev

	On Fedora:

	.. code-block:: bash

	sudo yum install openmpi openmpi-devel

	On OS X with HomeBrew:

	.. code-block:: bash

	brew install open-mpi

	- Compile ndnSIM with MPI support

	You can compile ndnSIM with MPI support using ./waf configure by adding the parameter
	``--enable-mpi`` along with the parameters of your preference. For example, to configure
	with examples and MPI support in optimized mode:

	.. code-block:: bash

	cd <ns-3-folder>
	./waf configure -d optimized --enable-examples --enable-mpi

	- Run ndnSIM with MPI support

	To run a simulation scenario using MPI, you need to type:

	.. code-block:: bash

	mpirun -np <number_of_processors> ./waf --run=<scenario_name>


	.. _simple scenario with MPI support:

	Simple parallel scenario using MPI
	----------------------------------

	This scenario simulates a network topology consisting of two nodes in parallel. Each node
	is assigned to a dedicated logical processor.

	The default parallel synchronization strategy implemented in the DistributedSimulatorImpl
	class is based on a globally synchronized algorithm using an MPI collective operation to
	synchronize simulation time across all LPs. A second synchronization strategy based on local
	communication and null messages is implemented in the NullMessageSimulatorImpl class, For
	the null message strategy the global all to all gather is not required; LPs only need to
	communication with LPs that have shared point-to-point links. The algorithm to use is
	controlled by which the ns-3 global value SimulatorImplementationType.

	The strategy can be selected according to the value of nullmsg. If nullmsg is true, then
	the local communication strategy is selected. If nullmsg is false, then the globally
	synchronized strategy is selected. This parameter can be passed either as a command line
	argument or by directly modifying the simulation scenario.

	The best algorithm to use is dependent on the communication and event scheduling pattern for
	the application. In general, null message synchronization algorithms will scale better due
	to local communication scaling better than a global all-to-all gather that is required by
	DistributedSimulatorImpl. There are two known cases where the global synchronization performs
	better. The first is when most LPs have point-to-point link with most other LPs, in other
	words the LPs are nearly fully connected. In this case the null message algorithm will
	generate more message passing traffic than the all-to-all gather. A second case where the
	global all-to-all gather is more efficient is when there are long periods of simulation time
	when no events are occurring. The all-to-all gather algorithm is able to quickly determine
	then next event time globally. The nearest neighbor behavior of the null message algorithm
	will require more communications to propagate that knowledge; each LP is only aware of
	neighbor next event times.

	The following code represents all that is necessary to run such this simple parallel scenario

	.. literalinclude:: ../../examples/ndn-simple-mpi.cpp
	:language: c++
	:linenos:
	:lines: 22-35,71-
	:emphasize-lines: 41-44, 54-58, 78-79, 89-90

	If this code is placed into ``scratch/ndn-simple-mpi.cpp`` or NS-3 is compiled with examples
	enabled, you can compare runtime on one and two CPUs using the following commands::

	# 1 CPU
	mpirun -np 1 ./waf --run=ndn-simple-mpi

	# 2 CPUs
	mpirun -np 2 ./waf --run=ndn-simple-mpi


	The following table summarizes 9 executions on OS X 10.10 and 2.3 GHz Intel Core i7 a single
	CPU, on two CPUs with global synchronization, and on two CPUs with null message
	synchronization:

	+-------------+-----------------+------------------+----------------+
	\| # of CPUs \| Real time, s \| User time, s \| System time, s \|
	+=============+=================+==================+================+
	\| 1 \| 20.9 +- 0.14 \| 20.6 +- 0.13 \| 0.2 +- 0.01 \|
	+-------------+-----------------+------------------+----------------+
	\| 2 (global) \| 11.1 +- 0.13 \| 21.9 +- 0.24 \| 0.2 +- 0.02 \|
	+-------------+-----------------+------------------+----------------+
	\| 2 (nullmsg) \| 11.4 +- 0.12 \| 22.4 +- 0.21 \| 0.2 +- 0.02 \|
	+-------------+-----------------+------------------+----------------+

	Note that MPI not always will result in simulation speedup and can actually result in
	performance degradation. This means that either network is not properly partitioned or the
	simulation cannot take advantage of the partitioning (e.g., the simulation time is dominated by
	the application on one node).