Implementing Gaussian Mixture MLE using optim() in R

User 2007598

I am trying to implement MLE for mixture of Gaussians in R using optim() using R's native dataset (Geyser from MASS). My code is as follows. The problem is that optim works fine, but returns the original parameters I passed to it, and also says it has converged. I'd appreciate it if you could point me in the direction I'm going off. My expectation is that it will at least produce different results, if not very different.

library(ggplot2)
library(MASS)
data("geyser")
externaldata=geyser$waiting
x.vector=externaldata


MLE.x= function(combined.vector)
{ combined.vector=bigvec
  x.vector = externaldata
  K = k #capital K inside this MLE function, small K defined in the global environment
  prob.vector = combined.vector[1:K] 
  mu.vector =combined.vector[(K+1):((2*K))]
  sd.vector=combined.vector[(2*K+1):(3*K)]
  prob=matrix(rep(prob.vector,length(x.vector)),byrow=TRUE,nrow = length(x.vector))
  mu.sigma=cbind(mu.vector,sd.vector)
  x.by.K=matrix(nrow = length(x.vector), ncol = k)
  for (i in 1:K){
    x.by.K[,i]=dnorm(x.vector,mu.sigma[i,1],mu.sigma[i,2])
  }
  prob.mat=x.by.K*prob
  density=apply(prob.mat,1,sum)
  log.density=sum(-log(density))
  return(log.density)
}



## k=2 set ##
meanvec=c(50,80)
sigmavec=c(5,5)
k=2
probvec=c(1/3,2/3)
bigvec=c(probvec,meanvec,sigmavec)
est.k2.MLE=MLE.x(bigvec)
z=optim(bigvec,
        fn=MLE.x,
        method = "L-BFGS-B")
z


#### k=3 set #####
meanvec=c(50,70,80)
sigmavec=c(5,5,5)
k=3
probvec=rep(1/3,3)
bigvec=c(probvec,meanvec,sigmavec)
est.k3.MLE=MLE.x(bigvec)
z=optim(bigvec,
        fn=MLE.x,
        method = "BFGS")
z
Asachet

Delete the first line of the MLE.x function.

Since its argument has been replaced by the global variable "bigvec", it will always return the same. So the MLE can't converge, I guess you'd rather hit the maximum number of iterations. You can check by accessing z$convergencethe return value represented by z optim. This will be an integer code. 0 means everything is fine, 1 means the maximum number of iterations has been reached. Other values ​​are different error codes.

However, as you pointed out in the comments, the code still doesn't work. I can't see any errors, so I added the following snippet at the end of MLE.x:

if(any(is.na(density))) {
    browser()
  } else {
    log.density
  }

What it does is, if there are some NAs (or NaNs), we call it browser(), which is a very handy debugging tool: it stops the code and opens the console so we can explore the environment. Otherwise, we return log.density.

Then, I ran the code and saw that when the density is NA, it starts the console instead of failing:

You can see:

Browse[1]> head(x.by.K)
     [,1]       [,2]
[1,]  NaN 0.01032407
[2,]  NaN 0.01152576
[3,]  NaN 0.01183521
[4,]  NaN 0.01032407
[5,]  NaN 0.01107446
[6,]  NaN 0.01079706

The first column of x.by.K is NaN...so dnorm returns NaN...

Browse[1]> mu.sigma
     mu.vector sd.vector
[1,]  64.70180 -20.13726
[2,]  61.89559  33.34679

Here's the problem: SD is -20, which doesn't work well...

Browse[1]> combined.vector
[1] 1267.90677 1663.42604   64.70180   61.89559  -20.13726   33.34679

But this is the input of MLE.x.

There I just showed you how to debug your code :)

So what is happening is that during the optimization routine, parameters 5 and 6 take negative values, which will cause dnorm to fail. Why are they not negative? Optim doesn't know these should stay positive!

So you have to find a way to do constrained optimization with SD > 0 constraints.

But you shouldn't actually do this, and think about what you want to do, since I'm not sure why you're fitting a univariate Gaussian model.

Related

Implementing Gaussian Mixture MLE using optim() in R

User 2007598 I am trying to implement MLE for mixture of Gaussians in R using optim() using R's native dataset (Geyser from MASS). My code is as follows. The problem is that optim works fine, but returns the original parameters I passed to it, and also says it

Implementing Gaussian Mixture MLE using optim() in R

User 2007598 I am trying to implement MLE for mixture of Gaussians in R using optim() using R's native dataset (Geyser from MASS). My code is as follows. The problem is that optim works fine, but returns the original parameters I passed to it, and also says it

Gaussian mixture modeling with mle2/optim

CodeGuy I have mle2developed a mockup here to demonstrate the problem. x1I generate a sum of values from two separate Gaussian distributions x2, combine them together in form x=c(x1,x2), and then create an MLE that attempts to reclassify xthe values as belongi

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians, and was wondering if there was an easy way to automatically plot the estimated kernel density of the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Equivalent of Matlab "fit" of Gaussian mixture model in R?

Dentist_Not edible I have some time series data that looks like this: x <- c(0.5833, 0.95041, 1.722, 3.1928, 3.941, 5.1202, 6.2125, 5.8828, 4.3406, 5.1353, 3.8468, 4.233, 5.8468, 6.1872, 6.1245, 7.6262, 8.6887, 7.7549, 6.9805, 4.3217, 3.0347, 2.4026, 1.9317,

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians, and was wondering if there was an easy way to automatically plot the estimated kernel density of the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians, and was wondering if there was an easy way to automatically plot the estimated kernel density of the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians, and was wondering if there was an easy way to automatically plot the estimated kernel density of the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians and was wondering if there was an easy way to automatically plot the estimated kernel density for the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians and was wondering if there was an easy way to automatically plot the estimated kernel density for the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians and was wondering if there was an easy way to automatically plot the estimated kernel density for the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Plot Gaussian mixture in R with ggplot2

Mr. Zen I'm approximating a distribution with a mixture of Gaussians and was wondering if there was an easy way to automatically plot the estimated kernel density for the entire (1D) dataset as the sum of the component densities, similar to this way using ggpl

Clustering images using a Gaussian mixture model

ninja I want to cluster a binary image using GMM (Gaussian Mixture Model) and also want to plot the cluster centroids on the binary image itself. I used this as a reference : http://in.mathworks.com/help/stats/gaussian-mixture-models.html Here is my initial co

Clustering images using a Gaussian mixture model

ninja I want to cluster a binary image using GMM (Gaussian Mixture Model) and also want to plot the cluster centroids on the binary image itself. I used this as a reference : http://in.mathworks.com/help/stats/gaussian-mixture-models.html Here is my initial co

Clustering images using a Gaussian mixture model

ninja I want to cluster a binary image using GMM (Gaussian Mixture Model) and also want to plot the cluster centroids on the binary image itself. I used this as a reference : http://in.mathworks.com/help/stats/gaussian-mixture-models.html Here is my initial co

Motion is recorded using only Gaussian mixture models

Genevieve I am using this example on a Gaussian mixture model . I have a video showing a car in motion, but it's on a less busy street. A few cars flew by every now and then, but for the most part, there was no movement in the background. It gets very tedious

Problems implementing MLE estimation in the bbmle package (for R)

will I am trying to validate the MLEs for $\alpha$ , $\beta$ and $\lambda$ obtained for the Logistic-Lomax distribution by Zubair et al. in their paper titled A Study of Logistic-Lomax Distribution when using Dataset 1 . The paper uses the following code to do

How to make a histogram using sklearn with 1D Gaussian mixture?

Théré Hernandez I want to do a histogram with mixed 1D Gaussian images. Thanks Meng for the photos. My histogram looks like this: I have a file with a lot of data (4,000,000 numbers) in columns: 1.727182 1.645300 1.619943 1.709263 1.614427 1.522313 I'm using

How to make a histogram using sklearn with 1D Gaussian mixture?

Théré Hernandez I want to do a histogram with mixed 1D Gaussian images. Thanks Meng for the photos. My histogram looks like this: I have a file with a lot of data (4,000,000 numbers) in columns: 1.727182 1.645300 1.619943 1.709263 1.614427 1.522313 I'm using

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Understanding Gaussian Mixture Models

Hansner I'm trying to understand the results of the scikit-learn Gaussian Mixture Model implementation. See the example below: #!/opt/local/bin/python import numpy as np import matplotlib.pyplot as plt from sklearn.mixture import GaussianMixture # Define simp

Gaussian Mixture Modeling Matlab

Ashwin Shank Im using a Gaussian mixture model to estimate the log-likelihood function (parameters are estimated by the EM algorithm) Im using Matlab ... My data size is: 17991402*1...17991402 1D data points: When I run gmdistribution.fit(X, 2) I get the desir

Gaussian Mixture Modeling Matlab

Ashwin Shank Im using a Gaussian mixture model to estimate the log-likelihood function (parameters are estimated by the EM algorithm) Im using Matlab ... My data size is: 17991402*1...17991402 1D data points: When I run gmdistribution.fit(X, 2) I get the desir

Holes in a Gaussian Mixture Plot

Alex Gaspare I am trying to plot a Gaussian mixture model using Matlab. I am using the following code/data: p = [0.048544095760874664 , 0.23086205172287944 , 0.43286598287228106 ,0.1825503345829704 , 0.10517753506099443]; meanVectors(:,1) = [1.356437538131880