It wants an array with 3 columns, and one row for each edge. The columns are the red, green, and blue components of the color for the edge which corresponds to the row.
A = ones(4) - diag([1 1 1 1]);
G = graph(A);
h = plot(G);
h.EdgeColor = lines(G.numedges);
But you could create the array some other way.