Real time shape recognition from video feed

Just to keep myself occupied during some "empty" days between courses, I decided to implement a real time shape recognition system in Matlab. Most of the ideas are based on "Shape Recognition and Pose Estimation for Mobile Augmented Reality" by Nate Hagbi et al. However I didn't implement the pose estimation part since I run out of time. Moreover I encountered some difficulties regarding easier parts of the system (oh yeah image retrieval is not my field per se).

   So the idea behind it is simple: locate the most prominent shape (through filtering such as adaptive thresholding), compute its convex hull and perform matching via a bag-of-words approach. Therefore the system firstly needs to be trained. I use SIFT features on a series of projective transformations of the original shape, similar to "Visual Similarity based 3D Shape Retrieval Using Bag-of-Features" by Zhouhui Lian et al. A demonstration is presented below:

Real time shape recognition using Matlab from Jim B. on Vimeo.

Retrieve analysis (MFCC & Chroma) from EchoNest

EchoNest "offers an incredible array of music data and services for developers to build amazing apps and experiences". Me actually, haven't used EchoNest for nothing more than extracting MFCC and chroma features from songs. The feature extraction procedure is a "black box", however the motivation for employing it is obvious: EchoNest can act as benchmark tool.

    I had initially built a simple webpage for uploading and retrieving single songs. Eventually I realized that this is not efficient for lots of tracks. So I built a Matlab interface/function that performs batch processing.

GU interface

This program requires WGET but apart from that it's rather simple. It is based on Ellis' urlreadpost function; I simply modified some parts to fit my needs. There you have it:

Retrieve EchoNest analysis (sendspace link). Remember that you need an EchoNest API key.

Geometric Pattern Recogntion

For the course in "Pattern Recognition" me and a classmate worked on the task of identifying the rigid motion between two sets of points. In other words we aimed to calculate the scaling, rotation and transposition factors that mapped a point set A to a point set B. The methods that should be used were:

1. Exact point matching
2. Approximate matching (alignment algorithm)
3. Voronoi diagrams (by Shane O'Sullivan)

The user interface in Matlab

We implemented only the two first though. We employed O'Sullivan's code for computing the Voronoi diagram but its structure was inadequate for further usage. Additionally we implemented Hausdorff distance calculation. We used Matlab for the interface and C++ for the matching/distance algorithms.

Interface for 3D sound manipulation in PD

3D representation and control of the soundfield (Ambisonic in particular) was one of the topics I worked on during my studies in Music Technology. The aim of my work was to identify the optimal controller for Ambisonic sound manipulation, hence I built an interface that would let me perform certain Fitt's law experiments. Eventually it turned out to be a nice visualization tool with a lot of potential. Demo and screen shots of the interface are shown below:

3D representation of the soundfield using PD from Jim B. on Vimeo.

YouTube audio retrieval for Python

I am currently working on a project that requires audio retrieval from YouTube videos. So I wanted a batch processing program, that given a string query "Artist name - song title", it would retrieve the audio from the first N matches.

    The following figure shows the GUI for the code I wrote. You can just define the query string and the number of the retrieved videos (their audio actually).

Regarding the dependencies:

• gdata
• urllib

The whole code is based on youtube_dl, which I modified so as to act a callable function, rather as a command prompt program. The script requires the ffprobe and ffmpeg executables.

Linking Post’s Correspondence Problem and ACT-R to Turing Machines

For the final part of the "Models of Computation" course I had to write a paper that would link ACT-R and Post's Correspondence Problem to Turing machines. However the idea behind it was to actually find the connection between ACT-R and PCP. This was not an easy task since PCP is undecidable and ACT-R requires a formalization of a problem-solving procedure. Hence I decided to give a brief, abstract ACT-R model implementation of the heuristic pruning method for solving PCP instances.

Abstract

The aim of this paper to present the relationship between Post’s Correspondence Problem, ACT-R and Turing Machines. For both cases we will simulate the elementary operations of a Turing machine, thus proving constructively their direct analogy, while also their similar capability of computing any functions which can be computed in principle. Moreover, this paper briefly attempts to link Post’s Correspondence Problem to ACT-R using a similar procedure.

Linking Post's Correspondence Problem and ACT-R to Turing Machines

Most of the ideas and methods presented in this paper do not belong to me (just in case people complain). I've cited everyone as it is proper.

ACT-R

My second presentation for the course of "Models of Computation" investigated the ACT-R framework. This is actually a  very popular and interesting model of human cognition. It simulates, with the employment of modules, the functionality of fundamental brain parts.

    The idea behind the presentation was to introduce the model, then link it to Turing Machines and optionally to Post's Correspondence Problem. Since the connection of ACT-R to PCP is not possible, the corresponding slides are small in number. In order to compensate for that I investigated and presented the connection of ACT-R to Neural Networks (ACT-RN model).

ACT-R

1. HolgerSchultheis, "Computational and Explanatory Power of Cognitive Architectures: The Case of ACT-R “.J. R. Anderson, D. Bothell, M. D. Byrne “An Integrated Theory of the Mind”, (2004), Psychological Review, Vol. 111, No. 4,
2. John R. Anderson, “How Can the Human Mind Occur in the Physical Universe?” (2007),Oxford University Press
3. C. Labiere, John R. Anderson, “A Connectionism Implementation of the ACT-R Production System” (1993),In Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society