Design and Implementation of vsStor: A Large-

Scale Video Surveillance Storage System

Abstract

An ever-increasing number of installed surveillance cameras, higher network

bandwidth requirements and larger storage space consumption are major considerations
when designing a video surveillance storage system. Traditional filesystems are not tailored
for receiving and storage of large scale video streams. This project attempts to present vsStor,
a PB-scale, network-based video surveillance storage system which is used for storing
thousands of streams. Another major characteristics of vsStor is the scale-out architecture
design which allows the performance to grow by adding more hardware. We make use of
clustered architecture for connecting various machines and rely on the webservice mechanism
for communication between them.

Problem Identification

High network bandwidth requirement and larger storage space consumption are the
major considerations while designing a video surveillance storage system. We need an
efficient solution to store the large volume of surveillance videos without compromising the
network bandwidth efficiency.

Literature Review

There are some works on the storage of surveillance videos, but to the best of our knowledge,
the problem of receiving and storing high concurrent video streams into a PB-scale storage
system has not been addressed in the literature. The reason for that we believe it is only in
recent years, many governments have embraced smart city initiatives, leading to more
cameras are installed and connected to the network
 D. Rodriguez-Silva, L. Adkinson-Orellana, F. Gonz'lez-Castano, I. Armino-
Franco and D. Gonz'lez-Martinez, "Video Surveillance Based on Cloud
Storage"
This paper outlines a video surveillance system based on Cloud Computing that
securely stores multimedia streams in the cloud storage provider Amazon S3.
Drawbacks:
Keeping data stored in-house or in the cloud is debatable for a circumstance, but as a
rule of thumb, sensitive information should never been put in public clouds, even
security measures are in place
 S. Wang, J. Yang, Y. Zhao, A. Cai, and S.Z. Li, "A surveillance video analysis
and storage scheme for scalable synopsis browsing"
This paper proposes a framework for efficient storing and browsing of surveillance
video based on video synopsis
Drawbacks:
The paper emphases on video analytics instead of video storage
 E. Jaspers, R. Wijnhoven, R. Albers, J. Nesvadba, J. Lukkien, A. Sinitsyn, et al.,
"CANDELA–storage, analysis and retrieval of video content in distributed
systems"
This paper is a project focuses on the integration of video content analysis into a
storage and retrieval system to support fast search.
Drawbacks:
It is not designed to scale to a huge dataset
 Z. Sun, Q. Zhang, Y. Li, and Y. Tan, "Dppdl: a dynamic partialparallel data
layout for green video surveillance storage"
This paper is an energy efficient video surveillance storage system which dynamically
allocates the storage space with appropriate degree of partial parallelism according to
performance requirement
Drawbacks:
For a large scale live video storage system, the number of disks which have the chance
to spin down is limited
 P. Pillai, Y. Ke, and J. Campbell, "Multi-fidelity storage"
This paper introduces a storage system which stores data at multiple fidelity levels and
supports both selective discard of image frames, and selective lossless and lossy
transformations to reduce disk space consumption
Drawbacks:
 This storage system is not designed to use in big data environment
 T. Zhang, A. Chowdhery, P.V. Bahl, K. Jamieson, and S. Banerjee, "The design
and implementation of a wireless video surveillance system"
This paper proposes a wireless video surveillance system
Drawbacks:
This solution is easy to deploy but it is undesirable for large camera clusters as it has
a limited geographical range
 D. Borthakur, "HDFS architecture guide"
Hadoop Distributed File System (HDFS) is an opensource implementation of the Google
File System (GFS).
Drawbacks:
HDFS is not designed to handle large concurrent video streams

Proposed System and Advantages

This project presents vsStor, a large-scale video surveillance storage system which is
used for receiving and storing thousands of concurrent video streams. vsStor is a not
traditional (i.e. POSIX-compliant) filesystem, therefore, it’s not designed to provide normal
file services for existing and legacy applications. Instead, vsStor runs on traditional
filesystems, so it is easier to implement and less error-prone.

The main contributions of this work are:

1) Design and implement a video storage system which can handle thousands of video
streams concurrently.

2) Storage of metadata in database and on disks allows the system to scale to PB-
level which is crucial in Big Data environment

Advantages

 Horizontally scalable. Hence the storage space literally won’t run out of memory.
 Network bandwidth not compromised
 Confidential data isn’t exposed to any public cloud
 Can be scalable to very huge data sets.
Objectives and Methodology

Objectives

 To design and implement vsStor, a large-scale video surveillance storage system which
is used for receiving and storing thousands of video streams.
 To provide an ability for the users to scale the system horizontally when needed
 To provide an efficient means for accessing the stored media across the clustered
environment.
 To provide an ability to the users to know the health information of each of the machine
in the cluster
 To provide a means to the user to enable or disable the machine in the cluster at any
instance of time.

Methodology

vsStor is built on commodity hardware and consists of multiple nodes acted as clients and/or
servers
There are four roles (components) inside vsStor:

 scheduler,
 metadata server,
 data servers, and
 reader(s).

At the edge of the system, there are senders which are the bridges between the cameras and
the vsStor. The system is designed to support high availability (HA), therefore, key nodes (i.e.
scheduler and metadata server) are configured as master/slave.

The scheduler is the kernel of the storage system, its processing power directly affects the
system’s throughput. The core function of the scheduler is store-and-forward: it accepts up
to 10K TCP connections from senders, receives incoming data, caches it in memory, calculates
erasure codes forwards data and encoded data to data nodes, and finally updates database

Data nodes store metadata, video data and parity (redundancy) data. Based on the current
design, a .data file on a data node is the partial video data or parity data of a one-hour long
video stream.

Metadata server is a MySQL database. To minimize metadata overhead on the metadata

server, inspired by the CPU’s L1/L2 cache concept, the database stores only a subset of
metadata.

The reader provides service to users for video download/playback

Hardware Requirements

Processor Intel Core i5 or AMD FX 8 core series with clock speed of 2.4 GHz
or above
RAM 2GB or above
Hard disk 40 GB or above
Input device Keyboard or mouse or compatible pointing devices
Display XGA (1024*768 pixels) or higher resolution monitor with 32 bit
color settings
Miscellaneous USB Interface, Power adapter, etc
Software Requirements

Operating System Windows

Programming Language – Core Java, Advanced Java, J2EE, Map Reduce Framework, MVC
Backend Framework
Programming language - Bootstrap Framework, HTML, CSS, JavaScript, Ajax, JQuery
Frontend
Development Eclipse Oxygen IDE
environment
Application Server Apache Tomcat v9.0
Database MySQL