Estimating mixture of Gaussian models in Pytorch

www3

I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows:

import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True).to(device)
means = torch.tensor(np.random.randn(8,2),requires_grad=True).to(device)#torch.randn(8,2,requires_grad=True).to(device)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True).to(device)
mix = D.Categorical(weights)
comp = D.Independent(D.Normal(means,stdevs), 1)
gmm = D.MixtureSameFamily(mix, comp)

num_iter = 10001#30001
num_iter2 = 200001
loss_max1 = 100
for i in range(num_iter):
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    optimizer1.zero_grad()
    loss2.backward()
    optimizer1.step()

The error I get is:

0
8.089411823514835
Traceback (most recent call last):

  File "/home/cameron/AnacondaProjects/gmm.py", line 183, in <module>
    loss2.backward()

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

As you can see, the model ran for 1 iteration.

e.g. Mr.

There is an ordering problem in your code, because you are creating the Gaussian mixture model outside the training loop and calculating the loss then when the Gaussian mixture model will try to use the initial value of the parameters you set in defining the model, but optimizer1.step()have modified that value so even if You set loss2.backward(retain_graph=True)it up and you still get the error:RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

The solution to this problem is to create a new Gaussian mixture model as long as you update the parameters, the example code works as expected:

import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True)
means = torch.tensor(np.random.randn(8,2),requires_grad=True)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True)

parameters = [weights, means, stdevs]
optimizer1 = optim.SGD(parameters, lr=0.001, momentum=0.9)

num_iter = 10001
for i in range(num_iter):
    mix = D.Categorical(weights)
    comp = D.Independent(D.Normal(means,stdevs), 1)
    gmm = D.MixtureSameFamily(mix, comp)

    optimizer1.zero_grad()
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    loss2.backward()
    optimizer1.step()

    print(i, loss2)

Related

Estimating mixture of Gaussian models in Pytorch

www3 I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows: impo

Estimating mixture of Gaussian models in Pytorch

www3 I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows: impo

Estimating mixture of Gaussian models in Pytorch

www3 I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows: impo

Estimating mixture of Gaussian models in Pytorch

www3 I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows: impo

Estimating mixture of Gaussian models in Pytorch

www3 I actually want to estimate the normalized flow with a Gaussian mixture as the base distribution, so I'm a bit torch bound. However, you can reproduce my error in code just by estimating the mixture of Gaussian models in torch. My code is as follows: impo

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understand the concept of Gaussian mixture models

StuckInPhD I am trying to understand GMM by reading online resources. I have implemented clustering using K-Means and am looking at GMM vs K-means comparison. This is what I understand, please let me know if my concept is wrong: GMM is like KNN in the sense th

Understand the concept of Gaussian mixture models

StuckInPhD I am trying to understand GMM by reading online resources. I have implemented clustering using K-Means and am looking at GMM vs K-means comparison. This is what I understand, please let me know if my concept is wrong: GMM is like KNN in the sense th

Gaussian Mixture Models for Pixel Clustering

Huckleberry Finn I have a small aerial image where human experts have marked the different terrain visible in the image. For example, an image can contain vegetation, rivers, rocky mountains, farmland, etc. Each image can have one or more of these marked areas

Understand the concept of Gaussian mixture models

StuckInPhD I am trying to understand GMM by reading online resources. I have implemented clustering using K-Means and am looking at GMM vs K-means comparison. This is what I understand, please let me know if my concept is wrong: GMM is like KNN in the sense th

Gaussian Mixture Models for Pixel Clustering

Huckleberry Finn I have a small aerial image where human experts have marked the different terrain visible in the image. For example, an image can contain vegetation, rivers, rocky mountains, farmland, etc. Each image can have one or more of these marked areas

MATLAB code for many Gaussian mixture models

Ali Bodaghi I have applied the gaussmix function in the Voicebox MATLAB tool to calculate the GMM. However, when I run it for 512 GMM components, the code gives me errors. No_of_Clusters = 512; No_of_Iterations = 10; [m_ubm1,v_ubm1,w_ubm1]=gaussmix(feature,[],

Motion is recorded using only Gaussian mixture models

Genevieve I am using this example on a Gaussian mixture model . I have a video showing a car in motion, but it's on a less busy street. A few cars flew by every now and then, but for the most part, there was no movement in the background. It gets very tedious

Gaussian Mixture Modeling Matlab

Ashwin Shank Im using a Gaussian mixture model to estimate the log-likelihood function (parameters are estimated by the EM algorithm) Im using Matlab ... My data size is: 17991402*1...17991402 1D data points: When I run gmdistribution.fit(X, 2) I get the desir

Gaussian Mixture Modeling Matlab

Ashwin Shank Im using a Gaussian mixture model to estimate the log-likelihood function (parameters are estimated by the EM algorithm) Im using Matlab ... My data size is: 17991402*1...17991402 1D data points: When I run gmdistribution.fit(X, 2) I get the desir

Holes in a Gaussian Mixture Plot

Alex Gaspare I am trying to plot a Gaussian mixture model using Matlab. I am using the following code/data: p = [0.048544095760874664 , 0.23086205172287944 , 0.43286598287228106 ,0.1825503345829704 , 0.10517753506099443]; meanVectors(:,1) = [1.356437538131880

Gaussian Mixture Model for Image Histogram

Dotted glass I am trying to do automatic image segmentation of different regions of a 2D MR image based on pixel intensity values. The first step is to implement a Gaussian mixture model on the histogram of the image. I need to plot the resulting Gaussian obta

Gaussian Mixture Model for Image Histogram

Dotted glass I am trying to do automatic image segmentation of different regions of a 2D MR image based on pixel intensity values. The first step is to implement a Gaussian mixture model on the histogram of the image. I need to plot the resulting Gaussian obta