How to determine the number of bins and the edge length based on the density of each bin. (Bins most likely are not uniform.)

Non-uniform bin density Hi,

I have been trying to figure out how to write a function to bin a sample of data together based on its density (Number of Occurrence/ Length of edge). But there are not alot of examples out there.

The output would give a vector of edges where both :

1) the number of bins are given by how many are required to group data that have different density by a threshold (maybe 40%?)

2) and the length of the edges are determined by if the adjacent data groups have similar density. (Similar density are grouped together, but if the neighboring bin is 40% more or less in density, it would require another bin).

So to illustrate my point, below is a simple example:

I have data values that ranges from 1 to 10 and I have 10 observations of it where x=[1,2,3,4,5,5,5,6, 6,7];

x would result in a range with edges that are [1,5,6,7,8], so there are four states just because the bins represent different density clusters.

Just to mention my actual data is continuous, any help is appreciated.

Regards,

Tresno

[tsan toso]

Just to continue from the above example. I thought of a preliminary algorithm for large data samples:

1. Sort data in ascending order.

2. Group data where at least a group has 10 elements

3. Calculate and compare density to group similar ones together.

I got stuck on the 3rd point. Where I am not sure how to effectively group them. My obstacle comes from if the density increases slowly, but gradually e.g. Density: 1,2,3,4,5,6,7,8,9,10

Where do I call it break and say that one group has a different density from another.

Answer 1

[Image Analyst]

Try using kmeans clustering in the stats toolbox. For an example, see http://www.mathworks.com/products/demos/image/color_seg_k/ipexhistology.html

[tsan toso]

Thanks for the feedback. I dont totally understand Kmeans clustering, but doesnt the method require you to provide the number of bins on the onset before the data is processed? Instead of automatically determining how many bins are required to properly represent the data groups based on their difference in density?

Additionally, I thought the objective of Kmeans was to group data together that provides the least variance amongst the bins? I am not clear of how it would relate to the density.

Thanks any clarification is appreciated. :)

Sorry I just appended additional comments to the question as above.

[Image Analyst]

Actually I think you're right. There are clustering methods that can figure out how many clusters you should have based on some hints/criteria you give it, but I don't remember those off the top of my head. You should search for "upsupervised cluster*" to learn about those methods. I think the stats toolbox should have something like that - you mgiht call them and ask them. And once they've identified the clusters in your data then, of course, you'll know the histogram bin edge locations for the clusters also because it's the same thing/concept.

[Image Analyst]

Try this clustering app by Brett Shoelson: http://www.mathworks.com/matlabcentral/fileexchange/35014-clusterdata

[tsan toso]

Hey thanks for the feedback. Did look at the app, however, it seems to cluster groups of data based on how close they are to each other (not exactly based on the sparsity of the data)...would ask if Brett if he have any thoughts.

Thanks again.