1.4.2 Spreading vectors, matrix rows, and matrix columns

          A related operation is the spreading (duplication) of vectors, matrix rows, and matrix columns within rows or columns of nodes. The result is that all rows of nodes or all columns of nodes own a copy of the row or column vector.

Spreading a vector within rows (columns) of nodes:   Notice that gathering a vector within rows (columns) to a specified column (row) of nodes can be used to yield a column (row) vector. If we wish to have a copy of this column (row) vector within each column (row) of nodes, we need merely collect   the vector within rows (columns) of nodes, as illustrated in Figure 1.4.

Spreading a matrix row (column) within columns (rows) of nodes:   Given a matrix row, we often require a copy of this matrix row to exist within each row of nodes, an operation that we will call spreading a matrix row within columns of nodes. One approach is to redistribute the matrix row like the inducing vector distribution, and spreading the resulting vector, requiring a scatter followed by a collect, both within columns. Since in [, ] it is shown that a scatter followed by a collect is equivalent to a broadcast,   broadcasting the pieces of the matrix row within columns can be viewed as a shortcut. Similarly, spreading a matrix column within rows of nodes can be accomplished by broadcasting the appropriate pieces of the matrix column within rows of nodes.

Spreading a matrix row (column) within rows (columns) of nodes:     Spreading a matrix row within rows of nodes is logically equivalent to redistributing the matrix row like the inducing vector distribution, followed by a spreading of the vector within rows of nodes. Notice that this requires a scatter within columns of nodes, followed by a collect within rows of nodes. Spreading a matrix column within columns of nodes can be accomplished similarly.

It is important to note that the above observations expose a systematic approach to the required data movement. It naturally exposes the inducing vector distribution as an intermediate step through which redistribution of rows and columns of matrices can be implemented in a building-block approach. More on this in Chapter .