A high-dimensional linear experiment is performed below
Suppose our true equations are:
Assume the number of features is 200, and there are 20 training samples and 20 test samples.
Simulated data sets
num_train,num_test = 10,10 num_features = 200 true_w = ((num_features,1),dtype=torch.float32) * 0.01 true_b = (0.5) samples = (0,1,(num_train+num_test,num_features)) noise = (0,0.01,(num_train+num_test,1)) labels = (true_w) + true_b + noise train_samples, train_labels= samples[:num_train],labels[:num_train] test_samples, test_labels = samples[num_train:],labels[num_train:]
Define a loss function with a regular term
def loss_function(predict,label,w,lambd):
loss = (predict - label) ** 2
loss = () + lambd * (w**2).mean()
return loss
Methods of Drawing
def semilogy(x_val,y_val,x_label,y_label,x2_val,y2_val,legend):
(figsize=(3,3))
(x_label)
(y_label)
(x_val,y_val)
if x2_val and y2_val:
(x2_val,y2_val)
(legend)
()
Fitting and plotting
def fit_and_plot(train_samples,train_labels,test_samples,test_labels,num_epoch,lambd):
w = (0,1,(train_samples.shape[-1],1),requires_grad=True)
b = (0.,requires_grad=True)
optimizer = ([w,b],lr=0.05)
train_loss = []
test_loss = []
for epoch in range(num_epoch):
predict = train_samples.matmul(w) + b
epoch_train_loss = loss_function(predict,train_labels,w,lambd)
optimizer.zero_grad()
epoch_train_loss.backward()
()
test_predict = test_sapmles.matmul(w) + b
epoch_test_loss = loss_function(test_predict,test_labels,w,lambd)
train_loss.append(epoch_train_loss.item())
test_loss.append(epoch_test_loss.item())
semilogy(range(1,num_epoch+1),train_loss,'epoch','loss',range(1,num_epoch+1),test_loss,['train','test'])
It can be noticed that the model with the addition of the regular term does drop the loss on the test set
Above is the Python deep learning pyTorch weight decay and L2 paradigm regularization analysis of the details, more information about Python pyTorch weight and L2 paradigm regularization please pay attention to my other related articles!