How to add parameters in module class in pytorch custom model?

Question

I tried to find the answer but I can't.

I make a custom deep learning model using pytorch. For example,

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.nn_layers = nn.ModuleList()
        self.layer = nn.Linear(2,3).double()
        torch.nn.init.xavier_normal_(self.layer.weight)

        self.bias = torch.nn.Parameter(torch.randn(3))

        self.nn_layers.append(self.layer)

    def forward(self, x):
        activation = torch.tanh
        output = activation(self.layer(x)) + self.bias

        return output

If I print

model = Net()
print(list(model.parameters()))

it does not contains model.bias, so optimizer = optimizer.Adam(model.parameters()) does not update model.bias. How can I go through this? Thanks!

If you are subclassing module see the solution posted here. discuss.pytorch.org/t/…. Test with print(list(model.parameters())). — Just_Alex
– Just_Alex, Commented Jan 17, 2022 at 6:06

Shai · Accepted Answer · 2022-08-18 05:57:54Z

30

You need to register your parameters:

self.register_parameter(name='bias', param=torch.nn.Parameter(torch.randn(3)))

Update:
In more recent versions of PyTorch, you no longer need to explicitly register_parameter, it's enough to set a member of your nn.Module with nn.Parameter to "notify" pytorch that this variable should be treated as a trainable parameter:

self.bias = torch.nn.Parameter(torch.randn(3))

Please note that is you want to have more complex data structures of parameters (e.g., lists, etc.) you should use dedicated containers like torch.nn.ParameterList or torch.nn.ParameterDict.

edited Aug 18, 2022 at 5:57

answered Dec 8, 2019 at 10:07

Shai

115k39 gold badges259 silver badges398 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

CSH Over a year ago

Thank you! I think ``` self.register_parameter(name='bias', param=torch.nn.Parameter(self.bias)) ``` does not work. Is there a way to get rid of 'name' part? I think this will be troublesome when save and load the model.

Shai Over a year ago

@CSH by using name='bias' when registering the parameter you implicitly creates self.bias to be used in your Net class. you do not need to explicitly assign self.bias for it to exist.

CSH Over a year ago

OMG. What a convenient function! Thank you for your helps!

NeoZoom.lua Over a year ago

It should be torch.nn.parameter.Parameter now.

Collectives™ on Stack Overflow

How to add parameters in module class in pytorch custom model?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related