What is the best way to separate foreground image and background image?

image1

image2

Based on the image2, how can I get only the human image by converting all the background pixels into black or white? I try manual thresholding however I can't get smooth foreground image and the mask image will varied(due to illumination) since i'm working with image sequence. And I try to get rare pixels on the human images only (by subtracting the background image and foreground image (HSV)) as shown in image1. However I have no idea on how to extract only the obvious pixels region.