The scalable extension of H.264/AVC, known as scalable video coding or SVC, is currently the main focus of the Joint Video Team’s work. In its present working draft, the higher level syntax of SVC follows the design principles of H.264/AVC. Self-contained network abstraction layer units (NAL units) form natural entities for packetization. The SVC specification is by no means finalized yet, but nevertheless the work towards an optimized RTP payload format has already started. RFC 3984, the RTP payload specification for H.264/AVC has been taken as a starting point, but it became quickly clear that the scalable features of SVC require adaptation in at least the areas of capability/operation point signaling and documentation of the extended NAL unit header. This paper first gives an overview of the history of scalable video coding, and then reviews the video coding layer (VCL) and NAL of the latest SVC draft specification. Finally, it discusses different aspects of the draft SVC RTP payload format, including the design criteria, use cases, signaling and payload structure.

Standard-compatible multiple description coding (MDC) and layered coding (LC) are efficient ways to ensure erasure resilient, scalable transmission of encoded multimedia sources via RTP, allowing a gradual degradation of the application quality with increasing packet loss rate and decreasing bandwidth/throughput on the network. In this paper we review the standard-compatible framework proposed to IETF. Alternative techniques such as robust source coding and channel coding techniques (ARQ: automatic repeat request, FEC: forward error correction) are presented; their integration into the proposed framework is also discussed. The performances of MDC and LC either coupled with channel coding or not, are summarized by reference to current literature. Typical cases and examples are illustrated.

Scalable video coding (SVC), as the Scalable Extension of H.264/AVC, is an ongoing international video coding standard designed for network adaptive or device adaptive applications and also offers high coding efficiency. However, packet losses often occur over unreliable networks even for base layer of SVC and have severe impact on the playback quality of compressed video. Until now, no literature has discussed error concealment support for standard SVC bit-stream. In this paper, we provide robust and effective error concealment techniques for SVC with spatial scalability. Experimental results showed that the proposed methods provide substantial improvement, both subjectively and objectively, without a significant complexity overhead.

In this paper, we propose a multi-source multi-path video streaming system for supporting high quality concurrent video-on-demand (VoD) services over wireless mesh networks (WMNs), and leverage forward error correction to enhance the error resilience of the system. By taking wireless interference into consideration, we present a more realistic networking model to capture the characteristics of WMNs and then design a route selection scheme using a joint rate/interference-distortion optimization framework to help the system optimally select concurrent streaming paths. We mathematically formulate such a route selection problem, and solve it heuristically using genetic algorithm. Simulation results demonstrate the effectiveness of our proposed scheme.

We describe a system for multipoint videoconferencing that offers extremely low end-to-end delay, low cost and complexity, and high scalability, alongside standard features associated with high-end solutions such as rate matching and personal video layout. The system accommodates heterogeneous receivers and networks based on the Internet Protocol and relies on scalable video coding to provide a coded representation of a source video signal at multiple temporal and spatial resolutions as well as quality levels. These are represented by distinct bitstream components which are created at each end-user encoder. Depending on the specific conferencing environment, some or all of these components are transmitted to a Scalable Video Conferencing Server (SVCS). The SVCS redirects these components to one or more recipients depending on, e.g., the available network conditions and user preferences. The scalable aspect of the video coding technique allows the system to adapt to different network conditions, and also accommodates different end-user requirements (e.g., a user may elect to view another user at a high or low spatial resolution). Performance results concerning flexibility, video quality and delay of the system are presented using the Joint Scalable Video Model (JSVM) of the forthcoming SVC (H.264 Annex G) standard, demonstrating that scalable coding outperforms existing state-of-the-art systems and offers the right platform for building next-generation multipoint videoconferencing systems.

A file format for storage of scalable video is proposed in this paper. A generic model is presented to enable a codec independent description of scalable video stream. The relationships, especially the dependencies, among sub-streams in a scalable video stream are specified sufficiently and effectively in the proposed model. Complying with the presented scalable video stream model, the file format for scalable video is proposed based on ISO Base Media File Format, which is simple and flexible enough to address the demands of scalable video application as well as the non-scalable ones.

The support for multiple video streams in an ad-hoc wireless network requires appropriate routing and rate allocation measures ascertaining the set of links for transmitting each stream and the encoding rate of the video to be delivered over the chosen links. The routing and rate allocation procedures impact the sustained quality of each video stream measured as the mean squared error (MSE) distortion at the receiver, and the overall network congestion in terms of queuing delay per link. We study the trade-off between these two competing objectives in a convex optimization formulation, and discuss both centralized and distributed solutions for joint routing and rate allocation for multiple streams. For each stream, the optimal allocated rate strikes a balance between the selfish motive of minimizing video distortion and the global good of minimizing network congestions, while the routes are chosen over the least-congested links in the network. In addition to detailed analysis, network simulation results using ns-2 are presented for studying the optimal choice of parameters and to confirm the effectiveness of the proposed measures.

Application Layer Multicast (ALM) can greatly reduce the load of a server by leveraging the outgoing bandwidth of the participating nodes. However, most proposed ALM schemes become quite complicated and lose bandwidth efficiency if they try to deal with networks that are significantly heterogeneous or time-varying. In earlier work, we proposed MutualCast, an ALM scheme with fully connected mesh that quickly adapts to the time-varying networks, while achieving provably optimal throughput performance. In this paper, we study how MutualCast can be paired with adaptive rate control for streaming media. Specifically, we combine Optimal Rate Control (ORC), our earlier control-theoretical framework for quality adaptation, with the MutualCast delivery scheme. Using multiple bit rate video content, we show that the proposed system can gracefully adjust the common quality received at all the nodes while maintaining a continuous streaming experience at each, even when the network undergoes severe, uncorrelated bandwidth fluctuations at different peer nodes.

This paper presents a streaming system using scalable video coding based on H.264/AVC. The system provides a congestion control algorithm supported by channel bandwidth estimation of the client. It uses retransmission only for packets of the base layer to disburden the congested network. The bandwidth estimation allows for adjusting the transmission rate quickly to the current available bandwidth of the network. Compared to binomial congestion control, the proposed system allows for shorter start-up times and data rate adaptation. The paper describes the components of this streaming system and the results of experiments showing that the proposed approach works effectively for streaming video.

We are interested in providing Video-on-Demand (VoD) streaming service to a large population of clients using peer-to-peer (P2P) approach. Given the asynchronous demands from multiple clients, continuously changing of the buffered contents, and the continuous video display requirement, how to collaborate with potential partners to get expected data for future content delivery are very important and challenging. In this paper, we develop a novel scheduling algorithm based on deadline-aware network coding (DNC) to fully exploit the network resource for efficient VoD service. DNC generalizes the existing network coding (NC) paradigm, an elegant solution for ubiquitous data distribution. Yet, with deadline awareness, DNC improves the network throughput and meanwhile avoid missing the play deadline in high probability, which is a major deficiency of the conventional NC. Extensive simulation results demonstrated that DNC achieves high streaming continuity even in tight network conditions.

Scalable video coding (SVC) is the most promising video format for applications of collaborative communication. MPEG-21 standard has newly emerged to enable the interoperability of multimedia delivery in heterogeneous environments. In this paper we study adaptation of SVC bitstream in the context of MPEG-21 multimedia framework. For interfacing SVC bitstream with MPEG-21 based adaptation system, we propose three SVC specific adaptation operators. Based on our previous work with multidimensional video adaptation, we present an effective approach, using MPEG-21 DIA AdaptationQoS description tool, to model QoS control for SVC adaptation. Then we show how the operator values could be computed from that representation. For the actual adaptation at bitstream level, we propose a procedure to remove the unnecessary NAL units from an SVC bitstream. The result of this study enables QoS management of SVC streaming in an efficient and standardized manner.

The author designed two algorithms for distributed cooperation among multiple video streams sharing common communication resources. The algorithms take advantage of an optimization framework that characterizes video packets such that joint resource allocation can be implemented not only over the packets of a single stream, but also across packets of different streams. The first algorithm enables collaboration among multiple video senders in an 802.11 CSMA/CA wireless network such that their joint performance is maximized. Via the algorithm, the users cooperatively establish transmission priorities based on the assigned characterizations of their video packets. The second technique allows for low-complexity joint bandwidth adaptation of multiple video streams at intermediate network nodes in the Internet in order to maximize the overall network performance. The author analyzes the advantages of the proposed algorithms over conventional solutions employed in such scenarios. It is shown that depending on system parameters such as available network data rate the proposed techniques can provide substantial gains in end-to-end performance.

Future multimedia communication systems have to support the user’s needs, the terminal capabilities, the content specification and the underlying networking technologies. The related protocols and applications must be designed from this integration perspective in a cross-layer centric manner. In this paper, we propose an implementation of a streaming service (e.g., Television over IP service) with a unified QoS management concept that enables an IP driven integration of different system components (terminal, user, content, and network). The MPEG-21 framework is used to provide a common support for implementing and managing the end-to-end QoS. The main focus of this paper is on the architecture design, protocols specification and implementation evaluation. Performance evaluations using PSNR and SSIM objective video quality metrics show the benefit of the proposed MPEG-21-enabled cross-layer adaptation.

Real-time video transport over wireless Internet faces many challenges due to the heterogeneous environment including wireline and wireless networks. A robust network condition classification algorithm using multiple end-to-end metrics and Support Vector Machine (SVM) is proposed to classify different network events and model the transition pattern of network conditions. End-to-end Quality-of-Service (QoS) mechanisms like congestion control, error control, and power control can benefit from the network condition information and react to different network situations appropriately. The proposed network condition classification algorithm uses SVM as a classifier to cluster different end-to-end metrics such as end-to-end delay, delay jitter, throughput and packet loss-rate for the UDP traffic with TCP-friendly Rate Control (TFRC), which is used for video transport. The algorithm is also flexible for classifying different numbers of states representing different levels of network events such as wireline congestion and wireless channel loss. Simulation results using network simulator 2 (ns2) showed the effectiveness of the proposed scheme.

We solve the problem of uplink video streaming in CDMA cellular networks by jointly designing the rate control and scheduling algorithms. In the pricing-based distributed rate control algorithm, the base station announces a price for the per unit average rate it can support, and the mobile devices choose their desired average transmission rates by balancing their video quality and cost of transmission. Each mobile device then determines the specific video frames to transmit by a video summarization process. In the time-division-multiplexing (TDM) scheduling algorithm, the base station collects the information on frames to be transmitted from all devices within the current time window, sorts them in increasing order of deadlines, and schedules the transmissions in a TDM fashion. This joint algorithm takes advantage of the multi-user content diversity, and maximizes the network total utility (i.e., minimize the network total distortion), while satisfying the delivery deadline constraints. Simulations showed that the proposed algorithm significantly outperforms the constant rate provision algorithm.

With the rapid growth of wireless broadband technologies, such as WLAN and WiMAX, quality streaming video contents are available through portable devices anytime, anywhere. The layered multicast system using scalable video codecs has been proposed as an efficient architecture for video dissemination taking account of user and link diversities. However, in the wired/wireless combined best-effort based heterogeneous IP networks which provide more fluctuation in available bandwidth and end-to-end delay, the performance of streaming systems has been greatly degraded due to frequent packet loss, resulting from either wired congestion or wireless fading/shadowing. In this paper, we present a real-time embedded packet train probing scheme for estimating end-to-end available bandwidth so as to accomplish effective congestion and error control. This is facilitated by effective classification of packet loss sources, delay trend detection algorithm and flexible transmission rate of packets. Under the proper wireless channel modelling and estimation, our layered structure can allow appropriate subscription of video layers and adaptively insert necessary amount of forward error correction (FEC) packets so as to achieve QoS optimized system for scalable video multicasting.

Sports video appeals to large audiences due to its high commercial potentials. Automatically extracting useful semantic information and generating highlight summary from sports video to facilitate user’s accessing requirements is an important problem, especially in the forthcoming broadband mobile communication and the need for users to access their multimedia information of interest from anywhere at anytime with their most convenient digital equipments. In this paper, a system to generate highlight summaries oriented for mobile applications is introduced, which includes highlight extraction and video adaptation. In this system, several highlight extraction techniques are provided for field sports video and racket sports video by using multi-modal information. To enhance user’s viewing experience and save bandwidth, 3D animation from highlight segment is also generated. As an important procedure to make video analysis results universally applicable, video transcoding techniques are applied to adapt the video for mobile communication environment and user preference. Experimental results are encouraging and show the advantage and feasibility of the system for multimedia content personalization, enhancement and adaptation to meet different user preference and network/device requirements.

The authors propose a novel method for transporting multi-view videos that aims to keep the bandwidth requirements on both end-users and servers as low as possible. The method is based on application layer multicast, where each end point receives only a selected number of views required for rendering video from its current viewpoint at any given time. The set of selected videos changes in real time as the user’s viewpoint changes because of head or eye movements. Techniques for reducing the black-outs during fast viewpoint changes were investigated. The performance of the approach was studied through network experiments.

This paper proposes an adaptive joint source and channel coding scheme for H.264 video multicast over wireless LAN which takes into account the user topology changes and varying channel conditions of multiple users, and dynamically allocates available bandwidth between source coding and channel coding, with the goal to optimize the overall system performance. In particular, source resilience and error correction are considered jointly in the scheme to achieve the optimal performance. And a channel estimation algorithm based on the average packet loss rate and the variance of packet loss rate is proposed also. Two overall performance criteria for video multicast are investigated and experimental results are presented to show the improvement obtained by the scheme.

Wireless Local Area Networks (WLANs) such as IEEE 802.11a/g and Hiperlan/2 utilise numerous transmission modes, each providing different throughputs and reliability levels. Many link adaptation algorithms proposed in the literature either maximise the error-free data throughput based on channel conditions or are based on the number of failed transmissions. However, these algorithms do not take into account the content of the data stream and strongly rely on the use of Automatic Repeat Requests (ARQs). Low latency video applications such as real-time video transmission may require no retransmission, or only a limited number of retransmissions. Moreover, completely error-free communication is not essential, especially if robust video compression techniques are applied. In such scenarios, improved decoded video quality can be obtained with a video stream transmitted at a higher bit rate using a higher link speed but with some degree of transmission error, rather than an error-free video stream at a lower bit rate using a lower link speed. In this work, we investigate a link adaptation scheme that improves the Quality of Service (QoS) for video transmission, based on the overall received video quality (Peak Signal to Noise Ratio, PSNR), rather than by maximising the error-free throughput. We also study a practical link adaptation approach that uses PER thresholds at the PHY layer. An empirical study showed that thresholds for switching from one mode to another are much lower (almost error free) than those currently used by throughput based schemes. We show that traditional link adaptation strategies are not appropriate for real-time video transmission with no retransmission. Simulation results using the H.264 video compression standard over IEEE 802.11a are presented.

In this work, a new method to deal with the unconnected pixels in motion compensated temporal filtering (MCTF) is presented, which is designed to improve the performance of 3D lifted wavelet coding. Furthermore, multiple description scalable coding (MDSC) is investigated, and novel MDSC schemes based on 3D wavelet coding are proposed, using the lifting implementation of temporal filtering. The proposed MDSC schemes can avoid the mismatch problem in multiple description video coding, and have high scalability and robustness of video transmission. Experimental results showed that the proposed schemes are feasible and adequately effective.

In this paper, we propose a practical design and implementation of network-adaptive high definition (HD) MPEG-2 video streaming combined with cross-layered channel monitoring (CLM) over the IEEE 802.11a wireless local area network (WLAN). For wireless channel monitoring, we adopt a cross-layered approach, where an access point (AP) periodically measures lower layers such as medium access control (MAC) and physical (PHY) transmission information (e.g., MAC layer loss rate) and then sends the monitored information to the streaming server application. The adaptive streaming server with the CLM scheme reacts more quickly and efficiently to the fluctuating wireless channel than the end-to-end application-layer monitoring (E2EM) scheme. The streaming server dynamically performs priority-based frame dropping to adjust the sending rate according to the measured wireless channel condition. For this purpose, the proposed streaming system nicely provides frame-based prioritized packetization by using a real-time stream parsing module. Various evaluation results over an IEEE 802.11a WLAN testbed are provided to verify the intended Quality of Service (QoS) adaptation capability. Experimental results showed that the proposed system can mitigate the quality degradation of video streaming due to the fluctuations of time-varying channel.

The authors propose a scheme for Systematic Lossy Error Protection (SLEP) of an H.264/AVC compressed video bit stream, using standard compatible features such as redundant slices, and flexible macroblock ordering. The systematic portion consists of a conventional H.264/AVC bit stream. For error resilience, an additional Wyner-Ziv bit stream is also transmitted. The Wyner-Ziv bit stream allows the decoding of a coarsely quantized description of the original video signal, and is efficiently generated by using H.264/AVC redundant slices in conjunction with Reed-Solomon coding. The Wyner-Ziv bit stream is decoded in order to recover the redundant video descriptions, which are used in lieu of portions lost from the original video signal due to channel errors. SLEP allows the video quality to degrade gracefully with worsening channel conditions, and provides a flexible trade-off between the achieved error resilience and the coarseness of the redundant description. The performance can be improved especially for low motion video sequences, by applying SLEP to a region-of-interest in the video frame, using flexible macroblock ordering (FMO). Experimental results provided for two video transmission scenarios, demonstrate the advantages of SLEP over forward error correction (FEC) as an error resilience scheme.

This paper describes how to use an Adaptive Forward Error Correction (AFEC) algorithm to efficiently protect the critical data areas of compressed headers and UDP-Lite packets for data transport in a radio link layer of a wireless connection. Augmented with RoHC and UDP-Lite, for H.264 video transmissions over wireless channels in a heterogeneous wired-wireless environment, the erroneous packet payloads can be useful and better able to cope with lost packets (native UDP case), by adopting some of the erasure and error resilient modes in H.264. The context transfer during inter/intra handover is also discussed. Simulations demonstrated that the proposed scenario significantly improves the PSNR performance and video quality.

In this paper we propose a novel method for video quality prediction using video classification. In essence, our approach can serve two goals: (1) To measure the video quality of compressed video sequences without referencing to the original uncompressed videos, i.e., to realize No-Reference (NR) video quality evaluation; (2) To predict quality scores for uncompressed video sequences at various bitrates without actually encoding them. The use of our approach can help realize video streaming with ideal Quality of Service (QoS). Our approach is a low complexity solution, which is specially suitable for application to mobile video streaming where the resources at the handsets are scarce.

We address the problem of joint path selection and rate allocation in multipath wireless streaming, in order to optimize a media specific quality of service. We leverage on the existence of multiple parallel wireless services, in order to enhance the received video quality at a wireless client. An optimization problem is proposed, aimed at minimizing a video distortion metric based on sequence-dependent parameters, and transmission channel characteristics, for a given wireless network infrastructure. Even if joint optimal path selection and rate allocation is in general an NP complete problem, an in-depth analysis of the media distortion evolution allows defining a low complexity optimal streaming strategy, under reasonable network assumptions. In particular, we show that a greedy allocation of rates along paths with increasing error probability leads to an optimal solution. We argue that a network path should not be chosen for transmission, unless all other available paths with lower error probability have been chosen. Moreover, the chosen paths should be used at their maximum end-to-end bandwidth. These results are demonstrated for both independent network paths, and non-disjoint channel segments, in generic network topologies. Simulation results showed that the optimal rate allocation carefully trades off total encoding/transmission rate, with the end-to-end transmission error probability and the number of chosen paths. In many cases, the optimal rate allocation provides more than 20% improvement in received video quality, compared to heuristic-based algorithms.

We propose a Rate-Distortion (RD) optimized strategy for frame-dropping and scheduling of multi-user conversational and streaming videos. We consider a scenario where conversational and streaming videos share the forwarding resources at a network node. Two buffers are setup on the node to temporarily store the packets for these two types of video applications. For streaming video, a big buffer is used as the associated delay constraint of the application is moderate and a very small buffer is used for conversational video to ensure that the forwarding delay of every packet is limited. A scheduler is located behind these two buffers that dynamically assigns transmission slots on the outgoing link to the two buffers. Rate-distortion side information is used to perform RD-optimized frame dropping in case of node overload. Sharing the data rate on the outgoing link between the conversational and the streaming videos is done either based on the fullness of the two associated buffers or on the mean incoming rates of the respective videos. Simulation results showed that our proposed RD-optimized frame dropping and scheduling approach provides significant improvements in performance over the popular priority-based random dropping (PRD) technique.

The integration of reliable Video-on-Demand (VoD) broadcasting schemes in the DVB-h transmission system is studied, exemplary for Pyramid Broadcasting (PB). Sophisticated VoD broadcasting schemes such as PB allow receivers to tune into the ongoing transmission of a video-stream at arbitrary time, while still being able to receive the multimedia sequence from the beginning to end, after a short initial playout latency. Raptor coding, integrated in the FLUTE protocol, is combined with the traditional PB scheme in order to provide high service and presentation reliability. We give a short overview on the DVB-h transmission system as well as on the FLUTE protocol. We present and discuss options for the integration of VoD broadcasting schemes in combination with Raptor coding. We achieve backward-compatibility even with terminals not supporting Raptor coding. Simulation results show the benefits of the discussed VoD scheme compared to existing carousel approaches in DVB-h.

In this paper, we present an innovative design of multiple description coding with spatial-temporal hybrid interpolation (MDC-STHI) for peer-to-peer (P2P) video streaming. MDC can be effective in P2P networks because the nature of overlay routing makes path diversity more feasible. However, most MDC schemes require a redesign of video coding systems and are not cost-effective for wide deployment. We base our work on multiple state video coding, a form of MDC that can utilize standard codecs. Two quarter-sized video bit streams are generated as redundancies and embedded in the original-sized streams. With MDC-STHI, the nodes in P2P network can adjust the streaming traffic to satisfy the constraints of their devices and network environment. By design, the redundancies are used to compensate for missing frames, and can also be streamed independently to fulfill certain needs of low rate, low resolution applications. For better error concealment, optimal weights for spatial and temporal interpolation are determined at the source, quantized, and included in redundancies.