HLSL Shader for Background Image Subtraction

Question

I am trying to get the HLSL Pixel Shader for Silverlight to work to subtract the background image from the video image. Can someone suggest a more complex algorithm than I use because my algorithm does not do it right?

float Tolerance : register(C1); SamplerState ImageSampler : register(S0); SamplerState BackgroundSampler : register(S1); struct VS_INPUT { float4 Position : POSITION; float4 Diffuse : COLOR0; float2 UV0 : TEXCOORD0; float2 UV1 : TEXCOORD1; }; struct VS_OUTPUT { float4 Position : POSITION; float4 Color : COLOR0; float2 UV : TEXCOORD0; }; float4 PS( VS_OUTPUT input ) : SV_Target { float4 color = tex2D( ImageSampler, input.UV ); float4 background = tex2D( BackgroundSampler, input.UV); if (abs(background.r - color.r) <= Tolerance && abs(background.g - color.g) <= Tolerance && abs(background.b - color.b) <= Tolerance) { color.rgba = 0; } return color; } 

To see an example of this, you need a computer with a webcam:

• Go to http://xmldocs.net/alphavideo/background.html
• Click [Start Recording].
• Remove your body from the scene and click [Background Capture].
• Then move your body back to the scene and use the slider to set the Toleance value to the shader.

Answer 1

EDIT

A single pixel is not useful for such a task due to noise. Thus, the essence of the algorithm should be to measure the similarity between blocks of pixels. Prescription pseudo-code (based on correlation measurement):

  Divide image into N x M grid
 For each N, M cell in grid:
    correlation = correlation_between (signal_pixels_of (N, M),
                                      background_pixels_of (N, M)
                                     );
    if (correlation> threshold)
       show_background_cell (N, M)
    else
       show_signal_cell (N, M)

This is a serial pseudo code, but it can easily be converted to an HLSL shader. Simply, each pixel determines which block of pixel it belongs to, and then measures the correlation between the respective blocks. And based on this correlation, the current pixel is shown or hidden.

Try this approach, good luck!