The Design and Implementation of a Networked Real-time Video
Surveillance System Based on FPGA
Weijun Zhang*, Yujie Dun†, Weixiang Shi*
*School of Mechanical Engineering, Xi’an Jiaotong University, China, zhwj408@126.com
†
School of Electronic & Information Engineering, Xi’an Jiaotong University, China, dunyj@mail.xjtu.edu.cn
Keywords: Video Surveillance, MPEG-4, FPGA, Ethernet.
Abstract
Video surveillance systems are widely needed nowadays.
Typically the system can be realized in platforms such as
DSPs, FPGAs, ASICs, or the combination of them. In this
paper, we have developed a framework of a networked real-
time video surveillance system, and implemented it using two
ML402 FPGA development boards provided by Xilinx and a
custom video interface daughter board. The designed system
utilized Ethernet as its connection between encoding side and
decoding side, thus could be easily expanded to multipoint-to-
point or multipoint-to-multipoint Ethernet services that would
be required by different applications. The codec was based on
Xilinx MPEG-4 Part 2 Simple Profile Encoder and Decoder
IP cores with appropriate configuration in the designed
system, which effectively reduced the development time and
ensured the real-time encoding and decoding procedure.
Furthermore, the system can also be viewed as a system-level
video algorithm verification platform.
1 Introduction
Networked video surveillance systems have been deployed in
more and more places with the decreasing costs of camera
and recording equipments, technology advances in video
compression, and infrastructure improvements of IP network
that reduce the cost of transporting video data over large
distances [1].
ASICs with video codec functions, DSP processors, and
FPGAs can all be adopted in the networked video
surveillance systems. As all know, hardware solutions like
ASICs are usually less flexible and difficult to keep up with
the ever-changing new video codec standards, whereas
software solutions like DSPs always can’t meet the
performance requirements. FPGA-based systems can achieve
the uniform of the two, which can realize the hardware/
software integration with flexible tradeoffs. To some extent,
one can say that the FPGA solution makes the hardware soft
and the software hard.
Especially in video surveillance systems, the many distributed
sensing devices need not only fine performance with
relatively lower costs, but also the updating capacity.
Transmission bandwidth is also precious, to make the most of
2008
c The Institution of Engineering and Technology
Printed and published by the IET,
Michael Faraday House, Six Hills Way, Stevenage, Herts SG1 2AY
it, compression schemes have steadily improved. Using the
high computational horsepower required for complex
encoding or decoding compression algorithms such as
MPEG-4 and MPEG-compressed video transcoding,
achieving optimal balance of power, performance, and cost is
a significant challenge for video surveillance systems.
Besides, the widening range of requirements (including
advanced object detection, motion detection, object tracking,
etc.) is of the challenges to video surveillance architectures
[2]. Due to these, FPGA has become a preferred selection.
In this paper, we have developed a framework of a networked
real-time video surveillance system based on FPGA and
implemented it with the development board provided by
Xilinx. The hardware architecture, video capture, video codec
and network transmission mechanism in the designed system
are completely suitable for MPEG-4 video and can easily be
updated or ported to other video coding standards.
The paper is organized as follows: Section 2 gives the general
system framework and the implementation block diagram.
The hardware architecture and its realization are discussed in
Section 3. Section 4 introduces the software architecture and
the main control flow. System performance and tests are
shown in Section 5, followed by a conclusion in Section 6.
2 System framework
Typically, a network surveillance system comprises three
parts: the encoding side that captures video via camera and
converts the video into MPEG-4, H.264, or other format
stream; the decoding side that decodes the compressed stream
back into video frames and may perform further processing
such as noise reduction and object detection; and the transport
network connects the encoding side (namely the multipoint
terminals) and decoding side (usually the control centre or
control rooms in different places). The discussion in this
paper is mainly focused on the design and implementation of
the encoding and decoding sides.
The encoding side performs functions including:
(1) Capturing video using a video camera;
(2) Converting video to appropriate format needed by the
encoder;
(3) Encoding the video to an MPEG-4 format bitstream;
(4) Encapsulating the bitstream into UDP packets and
transmitting them over Ethernet.
508 VIE 08
The decoding side performs functions including:
(1) Receiving UDP packets of compressed video from
Ethernet and unpacking them to bitstream;
(2) Decoding the bitstream of MPEG-4 format to raw video
stream;
(3) Converting the raw video stream to appropriate format
needed by monitor and outputting to be displayed.
In our design, the system has been implemented in FPGA,
which allows parts of the system to be easily migrated
between hardware and software implementations for optimal
performance and resource use. The design was based on the
embedded CPU – Xilinx MicroBlaze, an established soft
processor core. It is based on a 32-bit RISC architecture and
is a preferred choice in smaller or simpler projects where
CPU power is not essential [3]. In addition, it is almost the
best choice for Xilinx FPGAs without an embedded hard
processor, just as what we used in our design, the Virtex-4
SX35 FPGA chip on the ML402 board. Figure 1 depicts the
architecture of the designed networked real-time video
surveillance system.
Figure 1: System architecture
The system has the following features:
• Both the encoding side and the decoding side were
implemented on the same type of FPGA boards;
• Both systems were implemented based on the method of
hardware-software co-design;
• MicroBlaze soft core processor is utilized in the system
with IBM OPB (On-Chip Peripheral Bus) bus interface to
other modules;
• Real-time encoder and decoder is ensured by the proper
uses of the Xilinx codec IP cores;
• UDP/IP is used as the transmission protocol;
The system modules were designed to cover most of the
algorithm accelerators that ensured the real-time video
encoding and decoding. To reduce the system implementation
difficulties, our design employed some of the existing design
modules. Figure 2 depicts the pre-verified modules (grey
blocks) from Xilinx and the custom modules (white blocks)
which were developed to construct the entire networked video
surveillance system.
Figure 2: System modules
Hardware Boards and IP cores make up of the hardware part
of the system, while software applications and drivers
compose the software part. The following two sections will
discuss the two parts in detail separately.
3 Hardware platform
Hardware implementation mainly includes the camera
interface design, video encoder and decoder realization based
509
on MPEG-4 codec IP cores provided by Xilinx, and Ethernet
transmission.
3.1 Camera interface
The camera interface was used to control a camera to convert
the real-world image into video stream, and then transmit the
raw video data to the successive module of the video
processing pipeline. It was composed of a custom video
interface daughter board and the custom IP core within the
main FPGA of ML402 mother board. The daughter board
uses the video analog decoder ASIC SAA7114H to transfer
the video image from analog (PAL of CVBS) to digital
(YUV4:2:0) according to the CCIR.601 standard. The two
boards are connected together through the expansion I/O
headers on the ML402 development board.
Figures 4 and 5 illustrate the block diagrams for the MPEG-4
Part 2 Simple Profile Encoder and Decoder cores respectively.
Hardware-based, pipelined architectures were used for these
implementations, with a host interface provided on the
encoder for software-controlled rate control. With an included
memory controller, the raw captured sequence for the encoder
and the reconstructed frames for the decoder were stored in an
off-chip memory for fast, low-latency access to the pixel data.
A simple FIFO interface was provided for communicating the
compressed bitstreams with the decoder custom-built for a
user-specified number of bitstreams. A system interface was
also included to allow for maximum controllability and
observability [5, 6].

After the video interface daughter board was power on and

reset, the registers of the SAA7114H were configured by the
MicroBlaze processor through the camera interface module.
By using VPO bus synchronous signals which are coming
from SAA7114H (including filed synchronous signal IGPV,
horizontal synchronous signal IGPH, valid image data output
signal IDQ, clock output signal ICLK and odd/even judging
signal IGP1), video data was written into a FIFO of camera
interface from the beginning of an even field [4]. At the mean
time, the video should be pre-processed according to the
encoder’s needs, mainly including the de-interlacing and
conversion from raster serial data to Macroblock data.
MicroBlaze processor can configure the camera interface with
FSL bus. Figure 3 gives the block of camera interface.
Figure 4: Internal encoder architecture

Figure 3: Block diagram of the camera interface

3.2 Video codec

Video codec used in this system was based on the ISO/IEC
MPEG-4 video compression standard. In this system, the
encoder at one end of the Ethernet compressed the input video
Figure 5: Internal decoder architecture
stream into MPEG-4 video format bit stream, and the decoder
performed the inverse operation.
With the support of existing codec IP cores, the main task was
the implementation of integration the IP cores into our
The Xilinx MPEG-4 decoder core can be built with a scalable,
MicroBlaze system, i.e., the codec IP cores need to be
multi-stream interface customized for the application and
wrapped to comply with the OPB bus protocol and imported
system requirements, while both the MPEG-4 encoder and
to the system. Xilinx provides a module named OPB IPIF to
decoder are also capable of servicing a user-specified
facilitate the connection of custom IP cores to the OPB bus.
maximum frame size.

510
In addition to facilitating OPB attachment, the IPIF provides
additional optional services. These services, FIFOs, DMA,
Scatter Gather (automated DMA), software reset, interrupt
support and OPB bus-master access, are placed in the IPIF to
standardize functionality that is common to many IP cores
and to reduce IP development effort. For our system, we just
used the simple register read and write functions to control
and monitor the MPEG-4 SP codec IP cores.
3.3 Transmission via Ethernet
To transmit the compressed bitstrem over Ethernet, it is
needed to carry out the corresponding operation to facilitate
this procedure. In this system, the opb Ethernet MAC
controller was used and the UDP/IP protocol was
implemented in the MicroBlaze processor.
4 Software architecture
The system partition of hardware and software ensures that
the software as host interface can focus on the data
communication as well as flow control for video tasks, while
the platform hardware modules deal with local or Ethernet
data operations and video codec functionality. Therefore, the
software abstraction layer covers the feature of data exchange
and video task flow control for hardware performance. In the
designed system, software architecture of both encoding side
and decoding side perform these two functions to ensure the
whole system work [8].
The controls of encoding side and decoding side mainly
include:
• Initialization of the modules in the system;
• Cooperation of the whole system;
• Schedule and management;
• Configure the camera and encoder/decoder according to
the API requirement;
• Control the data stream direction, i.e. retrieve data from
encoder, Ethernet MAC, and feed data to decoder, and the
display monitor.
In order to perform module functions in parallel, i.e. video
capturing, coding and sending at the encoding side, and video
retrieving, decoding and displaying at the decoding side,
buffer pairs and FIFOs were adopted in our design. An
example is that one buffer is used to store data from capture
of the current frame; another is to store the previous frame.
The flow control was carried out by monitoring the status of
modules and taking corresponding operations.
Figures 6 and 7 give the main control flowchart of the
encoding side and decoding side.

In detail, the encoding side software includes two parts:

• Drivers of camera interface, MPEG-4 encoder, System
ACE, UART, EMAC, etc.;
• Controls of encoding and transporting.

Likewise, the decoding side software includes two parts:

• Drivers of MPEG-4 decoder, System ACE, UART,
EMAC, DDR Sdram, etc.;
• Controls of transporting and decoding.

All the drivers and controls mentioned above were

implemented and integrated into the designed system. Besides
the module drivers provided by the vendor, we realized the
custom module drivers along with the modules themselves.
Controls of encoding side and decoding side were designed
based on these drivers and the partition mechanism of
hardware and software.

Drivers are the interfaces between software and hardware.

Implementations of custom drivers are on the basis of the IP
realization and fundamental hardware access methods
provided by the given system architecture. In our system,
drivers of the MPEG-4 encoder/decoder and the camera
interface were realized through serials of access functions to
those dedicate registers provided by the hardware IP cores.
The access functions take effects by performing proper
XIo_In8/16/32 and XIo_Out8/16/32 functions provided by Figure 6: Software flow chart of encoding side
the MicroBlaze system.

511

Figure 7: Software flow chart of decoding side
5 System performance and tests
As it was said before, the networked video surveillance
system has been tested on two verification board based on
Xilinx Virtex4-SX FPGA, namely ML402, and a video
interface daughter board. All the system modules were
embedded in FPGA, with the aid of other interface
peripherals such as Ethernet, UART, DDR sdram, ZBT ram,
system ACE, etc.
The whole system can perform 4CIF format real time video
encoding and decoding at 25fps.
Figure 8: Testing system
Table 1 and 2 show the FPGA resource usage and the size of
software segments for the overall system in the Virtex-4
SX35 FPGA targeted.
Slices 12700 out of 15360
Encoder Side Block RAMS 128 out of 192
DSP48s 27 out of 192
Slices 10264 out of 15360
Decoder Side Block RAMS 94 out of 192
DSP48s 46 out of 192
Table 1: FPGA Resource Usage Table
.text .data .bss .dec .hex
Encoder side 45266 3908 6980 56187 0xDB7B
Decoder side 34763 292 21616 56671 0xDD5F
Table 2: segments size of the programs
The design offers portability to different platforms due to the
facts that the hardware modules were all developed with
industry standard bus interface such as OPB and the control
software and drivers were written in C.
6 Conclusion
We have successfully designed and implemented an FPGA-
based networked real time video surveillance system with real
time video capture, video encoding and decoding and network
transmission. The test results have shown real time codec
performance and good system architecture, which are
essential to the networked video surveillance system. Since
the FPGA-based system architecture is very flexible and easy
reconfigurable, the system can also be viewed as a system-
level video algorithm verification platform and is suitable to
be used for H.264 and future video codec standards.
References
[1] Mark Oliver, “Next-generation surveillance system
design”, http://www.dspdesignline.com/howto/206106791,
(2008)
[2] Guohua Wei, Dee Zhang, Shanshan Wu, Yulin Gao.
“Design and Implementation of an IP-Based Intelligent Video
Surveillance System”, In Proceedings of the 8th International
Conference on Signal Processing, 2006
[3] Micorblaze Reference Guide EDK
http://www.xilinx.com/ipcenter/catalog/logicore/docs/microbl
aze_risc_32bit_proc_final.pdf
[4] SAA7114 datasheet, Philips Semiconductors, 2006
[5] ISO/IEC 14996-2, Information technology-coding of
audio-visual objects-part 2: Visual, 1998
[6] MPEG-4 Simple Profile Encoder v1.1 Data Sheet
http://china.xilinx.com/support/documentation/ip_documentat
ion/mpeg_4_encoder_ds511.pdf
[7] MPEG-4 Simple Profile Decoder v1.1 Data Sheet
http://china.xilinx.com/support/documentation/ip_documentat
ion/mpeg_4_decoder_ds338.pdf
[8] Yifeng Qiu; Badawy, W.; Turney, R.; “A Prototyping Co-
design Platform with A Simplified Architecture for Video
Codec Implementation, Circuits and Systems,” 2007. ISCAS
2007. IEEE International Symposium on 27-30 May 2007
Page(s):1220 - 1224
512