I fixed your jsfiddle, let it go now:
http://jsfiddle.net/5Sxrq/2/
The problems were:
margin-top:100px;
margin-left:80px;
if you want to use fields you have to subtract it from offsets
boxOff = $('#box').offset();
mouseOffX = e.pageX - boxOff.left;
mouseOffY = e.pageY - boxOff.top;
mouseOffX mouseOffY, mousedown, mousemove
EDIT:
:
http://jsfiddle.net/5Sxrq/3/