The order of the elements in memory in python, pytorch, numpy, c++ etc. are in row-major ordering:
[ first, second
third, forth ]
While in matlab, fortran, etc. the order is column major:
[ first, third
second, fourth ]
For higher dimensional tensors, this means elements are ordered from the last dimension to the first.
You can easily visualize it using torch.arange followed by .view:
a = torch.arange(24)
a.view(2,3,4)
Results with
tensor([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]])
As you can see the elements are ordered first by row (last dimension), then by column, and finally by the first dimension.
When you reshape a tensor, you do not change the underlying order of the elements, only the shape of the tensor. However, if you permute a tensor - you change the underlying order of the elements.
Look at the difference between a.view(3,2,4) and a.permute(0,1,2) - the shape of the resulting two tensors is the same, but not the ordering of elements:
In []: a.view(3,2,4)
Out[]:
tensor([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23]]])
In []: a.permute(1,0,2)
Out[]:
tensor([[[ 0, 1, 2, 3],
[12, 13, 14, 15]],
[[ 4, 5, 6, 7],
[16, 17, 18, 19]],
[[ 8, 9, 10, 11],
[20, 21, 22, 23]]])
Answer from Shai on Stack OverflowThe order of the elements in memory in python, pytorch, numpy, c++ etc. are in row-major ordering:
[ first, second
third, forth ]
While in matlab, fortran, etc. the order is column major:
[ first, third
second, fourth ]
For higher dimensional tensors, this means elements are ordered from the last dimension to the first.
You can easily visualize it using torch.arange followed by .view:
a = torch.arange(24)
a.view(2,3,4)
Results with
tensor([[[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [16, 17, 18, 19], [20, 21, 22, 23]]])
As you can see the elements are ordered first by row (last dimension), then by column, and finally by the first dimension.
When you reshape a tensor, you do not change the underlying order of the elements, only the shape of the tensor. However, if you permute a tensor - you change the underlying order of the elements.
Look at the difference between a.view(3,2,4) and a.permute(0,1,2) - the shape of the resulting two tensors is the same, but not the ordering of elements:
In []: a.view(3,2,4)
Out[]:
tensor([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23]]])
In []: a.permute(1,0,2)
Out[]:
tensor([[[ 0, 1, 2, 3],
[12, 13, 14, 15]],
[[ 4, 5, 6, 7],
[16, 17, 18, 19]],
[[ 8, 9, 10, 11],
[20, 21, 22, 23]]])
Your example, using numpy methods (I don't have tensorflow installed):
In [559]: x = np.array([
...: [
...: [[0.4495, 0.2356],
...: [0.4069, 0.2361],
...: [0.4224, 0.2362]],
...:
...: [[0.4357, 0.6762],
...: [0.4370, 0.6779],
...: [0.4406, 0.6663]]
...: ],
...: [
...: [[0.5796, 0.4047],
...: [0.5655, 0.4080],
...: [0.5431, 0.4035]],
...:
...: [[0.5338, 0.6255],
...: [0.5335, 0.6266],
...: [0.5204, 0.6396]]
...: ]
...: ])
In [560]: x.shape
Out[560]: (2, 2, 3, 2)
In [562]: s1=np.concatenate((x[0,0],x[0,1]), axis=1)
In [563]: s2=np.concatenate((x[1,0],x[1,1]), axis=1)
In [564]: s1.shape
Out[564]: (3, 4)
In [565]: new =np.concatenate((s1,s2), axis=0)
In [566]: new.shape
Out[566]: (6, 4)
In [567]: new.reshape(2,3,4)
Out[567]:
array([[[0.4495, 0.2356, 0.4357, 0.6762],
[0.4069, 0.2361, 0.437 , 0.6779],
[0.4224, 0.2362, 0.4406, 0.6663]],
[[0.5796, 0.4047, 0.5338, 0.6255],
[0.5655, 0.408 , 0.5335, 0.6266],
[0.5431, 0.4035, 0.5204, 0.6396]]])
numpy has a stack that joins arrays on a new axis, so we can skip the last concatenate and reshape with
np.stack((s1,s2)) # or
np.array((s1,s2))
The direct way to get there is to swap the middle 2 dimensions:
In [569]: x.transpose(0,2,1,3).shape
Out[569]: (2, 3, 2, 2)
In [571]: x.transpose(0,2,1,3).reshape(2,3,4)
Out[571]:
array([[[0.4495, 0.2356, 0.4357, 0.6762],
[0.4069, 0.2361, 0.437 , 0.6779],
[0.4224, 0.2362, 0.4406, 0.6663]],
[[0.5796, 0.4047, 0.5338, 0.6255],
[0.5655, 0.408 , 0.5335, 0.6266],
[0.5431, 0.4035, 0.5204, 0.6396]]])
reshape can be used to combine 'adjacent' dimensions, but doesn't reorder the underlying data. That is x.ravel() remains the same with reshape. While reshape of (2,2,3,2) to (2,3,4) is allowed, the apparent order of values probably is not what you want. That might be easier to see if you try to reshape
In [572]: np.arange(6).reshape(2,3)
Out[572]:
array([[0, 1, 2],
[3, 4, 5]])
In [573]: _.reshape(3,2)
Out[573]:
array([[0, 1],
[2, 3],
[4, 5]])
compare that with a transpose:
In [574]: np.arange(6).reshape(2,3).transpose(1,0)
Out[574]:
array([[0, 3],
[1, 4],
[2, 5]])
The transpose/swap that I did in [569] may be hard to understand. There are enough different ways of reordering dimensions, that it's hard to generalize.
What does view() do in PyTorch?
Where is Tensor.view documented?
Does changing a view change the original tensor?
Yes, it does behave like -1 in numpy.reshape(), i.e. the actual value for this dimension will be inferred so that the number of elements in the view matches the original number of elements.
For instance:
import torch
x = torch.arange(6)
print(x.view(3, -1)) # inferred size will be 2 as 6 / 3 = 2
# tensor([[ 0., 1.],
# [ 2., 3.],
# [ 4., 5.]])
print(x.view(-1, 6)) # inferred size will be 1 as 6 / 6 = 1
# tensor([[ 0., 1., 2., 3., 4., 5.]])
print(x.view(1, -1, 2)) # inferred size will be 3 as 6 / (1 * 2) = 3
# tensor([[[ 0., 1.],
# [ 2., 3.],
# [ 4., 5.]]])
# print(x.view(-1, 5)) # throw error as there's no int N so that 5 * N = 6
# RuntimeError: invalid argument 2: size '[-1 x 5]' is invalid for input with 6 elements
print(x.view(-1, -1, 3)) # throw error as only one dimension can be inferred
# RuntimeError: invalid argument 1: only one dimension can be inferred
I love the answer that Benjamin gives https://stackoverflow.com/a/50793899/1601580
Yes, it does behave like -1 in numpy.reshape(), i.e. the actual value for this dimension will be inferred so that the number of elements in the view matches the original number of elements.
but I think the weird case edge case that might not be intuitive for you (or at least it wasn't for me) is when calling it with a single -1 i.e. tensor.view(-1).
My guess is that it works exactly the same way as always except that since you are giving a single number to view it assumes you want a single dimension. If you had tensor.view(-1, Dnew) it would produce a tensor of two dimensions/indices but would make sure the first dimension to be of the correct size according to the original dimension of the tensor. Say you had (D1, D2) you had Dnew=D1*D2 then the new dimension would be 1.
For real examples with code you can run:
import torch
x = torch.randn(1, 5)
x = x.view(-1)
print(x.size())
x = torch.randn(2, 4)
x = x.view(-1, 8)
print(x.size())
x = torch.randn(2, 4)
x = x.view(-1)
print(x.size())
x = torch.randn(2, 4, 3)
x = x.view(-1)
print(x.size())
output:
torch.Size([5])
torch.Size([1, 8])
torch.Size([8])
torch.Size([24])
History/Context
I feel a good example (common case early on in pytorch before the flatten layer was official added was this common code):
class Flatten(nn.Module):
def forward(self, input):
# input.size(0) usually denotes the batch size so we want to keep that
return input.view(input.size(0), -1)
for sequential. In this view x.view(-1) is a weird flatten layer but missing the squeeze (i.e. adding a dimension of 1). Adding this squeeze or removing it is usually important for the code to actually run.
Example2
if you are wondering what x.view(-1) does it flattens the vector. Why? Because it has to construct a new view with only 1 dimension and infer the dimension -- so it flattens it. In addition it seems this operation avoids the very nasty bugs .resize() brings since the order of the elements seems to be respected. Fyi, pytorch now has this op for flattening: https://pytorch.org/docs/stable/generated/torch.flatten.html
#%%
"""
Summary: view(-1, ...) keeps the remaining dimensions as give and infers the -1 location such that it respects the
original view of the tensor. If it's only .view(-1) then it only has 1 dimension given all the previous ones so it ends
up flattening the tensor.
ref: my answer https://stackoverflow.com/a/66500823/1601580
"""
import torch
x = torch.arange(6)
print(x)
x = x.reshape(3, 2)
print(x)
print(x.view(-1))
output
tensor([0, 1, 2, 3, 4, 5])
tensor([[0, 1],
[2, 3],
[4, 5]])
tensor([0, 1, 2, 3, 4, 5])
see the original tensor is returned!