Languages and message passing
Data parallel programming languages
This is a superset of Fortran-77. Fortran-90 provides significant new
facilities some of which, such as array syntax, make it easier for a
compiler to determine that operations may be carried out concurrently.
Compilers available from NAG, IBM, DEC, Cray Research, and others.
One of the most significant aspects of Fortran-90.
- Arrays can be declared ALLOCATABLE. Compiler will defer allocating
memory space to the array until execution.
- Assignments may be made to entire arrays without use of subscripts.
- Array section references - in which part of an array may be assigned to all or part of another array. The arrays must be conformable.
Array section references
INTEGER, DIMENSION(1:100) :: array1
INTEGER, DIMENSION(1:50) :: array2
INTEGER, DIMENSION(1:100) :: array3, array4
array1(1:50) = 5
array1(51:100) = 25
array2 = arr(1:50)
array3(51:100) = array2
array3(1:50) = array1(51:100)
array(1:100:2) = array2
Array Intrinsic Functions
- Vector and matrix multiplication; for example, DOT_PRODUCT, MATMUL
- Array reduction functions; for example, ALL, MAXVAL, SUM
- Array inquiry functions; for example, SIZE, SHAPE
- Array construction functions; for example, MERGE, SPREAD
Other features include
- Control Structures.
- Procedure interface blocks.
- Derived data types.
High Performance Fortran (HPF)
HPF is the latest set of extensions to the Fortran language and is still
under development. Although the published specification has not yet
been adopted by a Standards body almost all parallel machine vendors
have announced implementation efforts.
A minimal starting language defined to encourage early releases of
compilers with HPF features.
Goals of HPF
Some goals of HPF, as laid down by the HPF Forum, are:
- To support data parallel programming.
- To enable high performance on parallel computers while not impeding
performance on other machines.
- To allow relatively easy conversion from existing sequential code to
This is executing the same operations (either synchronously or
asynchronously) in parallel on different sets of data. HPF expresses
data parallelism in several ways.
There are also constructs to describe operations that can be performed
in parallel if the computer has the resources:
- Fortran-90 array expressions and assignments.
- Array intrinsics defined in Fortran-90.
- Array library functions defined in the HPF library.
The FORALL statement.
FORALL (I = 0:9)
X(I+1) = X(I)
HPF can describe how data is to be divided among the processors in a
- DISTRIBUTE - is a directive that describes how an array is divided
and distributed to processors in a regular way.
- ALIGN - is a directive that describes how two arrays "line up".
- There are other directives, including dynamic versions of the above.
In the message passing model, parallel processes do not have direct
access to each others' local memory. Communication is achieved through
exchange of data, using SEND and RECEIVE primitives.
- Point-to-point communication - always involves exactly two
processes. One process SENDS and the other RECEIVES.
- Collective communication involves every process in a specified
Parallel Virtual Machine (PVM) is a software package that allows a
heterogeneous network of computers (parallel, vector, and serial) to
appear as a single concurrent computational resource - virtual machine.
- PVM supports heterogeneity at the application, machine, and network level.
- PVM has libraries that contain routines for initiating processes on
other machines, for communicating between processes, and changing the
configuration of the virtual machine.
- PVM provides interfaces to programs in C and Fortran.
- Multiple users can configure overlapping virtual machines.
Uses system buffers at SEND and RECEIVE ends to pack and unpack data.
However, recent versions e.g. Cray T3D avoid this.
CALL PVMFSEND(TID, MSGTAG, INFO)
This is an asynchronous send - no synchronous version.
CALL PVMFRECV(TID, MSGTAG, BUFID)
This is a blocking receive.
Tasks can broadcast messages to groups whether or not they belong to
CALL PVMFBCAST(GROUP, MSGTAG, INFO)
Availability of PVM
Several vendors are supplying and supporting optimized versions of PVM for their multi-processor systems, including Cray Research, IBM, Convex, Intel, SGI, and DEC.
- Public domain
- Available on netlib, which is a software distribution service
set up on the Internet.
- Available by Email to: email@example.com
with the message: send index from pvm3
- or anonymous ftp from: cs.utk.edu pub/xnetlib
Message Passing Interface (MPI) is a new library specification for
message-passing, proposed as a standard by a broadly based committee of
vendors, implementors, and users. MPI provides source-code portability
for message-passing programs while allowing efficient vendor
implementations. Features of MPI are:
- Support for separately compiled libraries.
- A great deal of functionality based on experience from other
- To allow for implementations that can be used in heterogeneous
- Interfaces to programs in C and Fortran.
There are four SEND modes.
CALL MPI_SEND( BUF, COUNT, DATATYPE, DEST, TAG, COMM, IERROR)
CALL MPI_RECV(BUF, COUNT, DATATYPE, SOURCE, TAG, COMM, STATUS, IERROR)
- This involves every process in the specified communicator.
- There exists a number of routines to distribute data amongst processes.
- Plus others
Availability of MPI
For example, the Argonne National Laboratory /Mississippi State
University implementation. Available by anonymous ftp from
- Implementations from most major vendors of parallel machines.
- Available public domain.
info.mcs.anl.gov in pub/mpi
Submitted by Mark Johnston,
last updated on 21 February 1995.