Comparison algorithms and techniques
Good morning.
My name is Cristian Bursasiu and I applied for the Software
Engineering Master as I believe that I reached the point in my career
where a formal study of this field is required.
In the following minutes I will present the project proposal that I
would like to accomplish by the end of this master in the event that I
am accepted as a student.
The study is called Comparison algorithms and techniques and is
related to the well known comparison tools, or diff tools that
software engineers use with various occasions.
Agenda
The presentation will contain
- a short introduction,
- some issues related to content comparison
- a few software engineering challenges
- and a few algorithmic / theoretical challenges regarding
content comparison that have to do with specialization and machine
learning
- Q&A
- In the final there is the bibliography that I used to
compose this presentation.
Introduction
There are a lot of tools for file/directory comparison on the market
and the majority of them have similar features, results and
performance regarding file and directory comparison.
Usually such a product will compare directory content in a simple,
straight forward manner: it usually displays files that are missing
from one of the directory or the other, files that are the same and
files that are modified – all this is based on file name matching.
Usually the current tools do not “observe” the rename of a file or
the moving of a file to a (or another) subdirectory.
Regarding file comparison, current tools only threat text files in a
general way, with disregard to the higher logic of the information in
the compared content.
Form my software engineering experience and from information
gathered from other engineers, I can say that there is a considerable
interest regarding diff tools especially in older projects where the
size and complexity of the code base is considerable.
Comparison issues 1
Imagine that we have two identical C++ files and in one of the files
we move a line of code several lines down. The result of the
comparison will shows an extra line in the first file (in the place
where the line used to be) and an extra line in the second file (in the
place where the line was moved). Therefore the application does not
realize that the line was moved so it displays 2 modifications instead
of one.
Comparison issues 2
Another example could be if you rename a variable - the diff tool
will recognizes a line with differences for each use of the variable
instead of only one difference – a variable was renamed.
I would like to mention that different coding rules could lead to
major differences in what a current diff tool report even if the
logical/useful differences are minimal.
Comparison issues 3
There are many other examples of modification that a current diff
tool will not notice – mainly because the current tools only operate
at text level and not at the higher logic of the file.
More complex issue arise if a part of the code was moved and then
modified – also the amount of the modification applied to the code
is also a factor in the likelihood that the corresponding pieces of
code can be matched. This remark is also applicable to directory
comparison – we can have files that are moved and renamed an
modified so the degree of the content modifications will determine
the ability of an application to correctly identify matches.
Software engineering challenges
In order to implement a better comparison tool I foresee some
engineering difficulties that I would like to point out:
A good engineering approach is to build an expandable, open
framework that will easily permit third party or future adaptation –
this can be achieved by providing a plug-in system that will cover as
much as possible of the foreseen future development.
Another issue that I would like to study are ways to present higher
logic comparison results to the user without removing the possibility
to fallback to the text level comparison/editing. The result
presentation issue is also interesting to consider/investigate if the
application would be able to compare other types of files like
images, MS word documents, visual studio project files, etc.
A feature that current comparison tools do not approach is file
specialization: Images, Video, Code (language specific), XML, etc. The
plug-in system could prove an effective solution for specialization
even for the specialized presentation issues.
Algorithmic / theoretical challenges 1
Amongst the more academic problem that can be foreseen
regarding content comparison are the ones related to algorithms:
For comparing images I believe that there are some fields that I
should study/investigate for applicable algorithms and techniques:
neural networks, image processing, face recognition, AI, etc.
For comparing source code I believe that some algorithms must be
found/investigated/experimented with regarding matching and
identifying code snippets. Also, code analysis techniques could be
useful in order to match files or code snippets,
Algorithmic / theoretical challenges 2
More advanced techniques can be searched for in the domain of
machine learning.
Neural networks should provide the right place to look for a quick
way of comparing images.
The iCub cognitive architecture is the result of a detailed design
process founded on the developmental psychology and
neurophysiology of humans, capturing much of what is known about
the neuroscience of action, perception, and cognition. This
framework should be studied in order to adopt engineering
information for a better integration of various concepts/modules of
the project.
Study of pattern recognition along with some ways of extracting
higher level information – like code analysis/parsing – could lead to
a better code snippet matching and/or file matching.
There are also some of the newer machine learning domains or sub-
domains that could prove useful in improving the performance and
results of the comparison methods and techniques – computational
neuroscience I believe to be one of them.