Density estimation could be a complex problem in the statistics literature. The density function in R provides 7 kernel density estimations. You'd better search online to find relevant documents for those seven methods.
Refer to "Computational Statistics Handbook With Matlab"
The chapter of Probability density estimation gives a brief introduction to kernel density estimation. Also gives the codes on how to implement in Matlab.