Extracting hand articulations from monocular depth images using curvature scale space descriptors

Shao-fan WANG,Chun LI,De-hui KONG,Bao-cai YIN

Front. Inform. Technol. Electron. Eng. 2016, 17 (1): 41-54. DOI: 10.1631/FITEE.1500126

Abstract

HTML

PDF (2840KB)

We propose a framework of hand articulation detection from a monocular depth image using curvature scale space (CSS) descriptors. We extract the hand contour from an input depth image, and obtain the fingertips and finger-valleys of the contour using the local extrema of a modified CSS map of the contour. Then we recover the undetected fingertips according to the local change of depths of points in the interior of the contour. Compared with traditional appearance-based approaches using either angle detectors or convex hull detectors, the modified CSS descriptor extracts the fingertips and finger-valleys more precisely since it is more robust to noisy or corrupted data; moreover, the local extrema of depths recover the fingertips of bending fingers well while traditional appearance-based approaches hardly work without matching models of hands. Experimental results show that our method captures the hand articulations more precisely compared with three state-of-the-art appearance-based approaches.

The CSS contour map can effectively detect straight or slightly bending fingers, but is ineffective for bending fingers. To recover bending fingers whose fingertips do not appear on the hand contour, we propose angle thresholds for recovering bending non-thumb fingertips. We first determine whether the thumb is bending according to the existence of the fingertip within the [15%]n-th$\sim[25\%]n$-th points of the whole hand contour, where n denotes the number of contour points ordered from thumb to little. If the thumb is bending, we roughly determine the finger-root of the thumb by the [20%]n-th point of the hand contour. Then we recover the fingertips of bending non-thumb and leave the recovery of the fingertip of the thumb at the end of this subsection (if it is missing). We connect each detected finger-root to the palm center and compute the angle of the ith non-thumb finger-root with respect to the finger-root of the thumb, denoted by $\theta_i$, $1\leq i\leq s$, where $s\leq4$ denotes the number of detected non-thumb fingers (Fig. 3). We also denote the following four intervals of the angles, each of which is associated with a non-thumb finger:

Once the ith non-thumb finger is bending, we select the ray $\boldsymbol{L}$ starting from the palm center at the middle angle (i.e., $(d_i+d_{i+1})/2+(45-\textrm{depth}(\textbf{pc}))d_0$) of the corresponding interval, and compute the intersections of $\boldsymbol{L}$ and the hand contour. The missing finger-root is determined by the nearest intersection to the palm center (Usually there is only one intersection between $\boldsymbol{L}$ and the contour. However, when $\boldsymbol{L}$ has a bias direction it may intersect with the contour points of other fingers. In this case the nearest point to the palm center is the correct finger-root). The missing fingertip is then determined by one of the points of the line segment between the finger-root and the palm center which achieves the greatest directional derivative of depths along the counter-direction of $\boldsymbol{L}$. We illustrate the whole procedure in the case of a straight thumb in Fig. 3.