Audio Features in RDF?   Why bother?

Audio Features in RDF !

What do we gain?

• Modularity
• Extensibility
• Interoperability
• Queryability

We have seen what these mean...

But, there are standards for describing audio content ...?

Some existing standards

Arguably, these all describe some aspects of music or audio:

• AES31-3 (audio workflow and project exchange)
• OMF/AAF (media workflow and project exchange)
• MXF (a metadata carrier specification)
• MIDI (transmission and storage of symbolic data)
• MPEG-7 (content based descriptors)
• SDIF (content based descriptors for parametric sound modelling)
• ACE XML (content based descriptors for classification)
• Music XML (score descriptor)
• Metri XML (score language)
• SMDL (score descriptor)
• + lots of others developed in industry and academia

Problems

Audio Feature Ontology

Audio Feature Ontology at a glance

Audio Feature Ontology

Key points:

• Features represented by Events or Signals
• Timelines link things together

Audio Feature Ontology

How complete is it ?

• Covers many common feature types
• Extensible as the need arises
• Base types from subsumed ontologies can always be used if a concept is not yet defined


This question isn't very meaningful anyway ...

AF Examples (1): Onsets

A note onset on the signal timeline:

@prefix tl: <http://purl.org/NET/c4dm/timeline.owl#>.
@prefix af: <http://purl.org/ontology/af/>.
	
:signal_timeline a tl:Timeline .
:onset_23 a af:Onset;
	    event:time [ 
	        a tl:Instant ;
	        tl:timeline :signal_timeline ;
	        tl:at "PT1.710S"^^xsd:duration ;
	    ] .

AF Examples (2): Segments

A key segment:

	
:signal_timeline a tl:Timeline .
:key_segment_1 a af:Segment;
		rdfs:label """Bb major""" ;
		af:feature "11" ;
		event:time [
			a tl:Interval ;
			tl:timeline :signal_timeline ;
			tl:start "PT30.1S";
			tl:duration "PT200S";
		] .

AF Examples (3): Dense features

Describing a Chromagram: first express the temporal relations between the audio and the features.

:signal_timeline a tl:Timeline .
:feature_timeline a tl:DiscreteTimeLine .

:feature_timeline_map a tl:UniformSamplingWindowingMap ;
    tl:rangeTimeLine :feature_timeline ;
    tl:domainTimeLine :signal_timeline ;
    tl:sampleRate "44100"^^xsd:int ;
    tl:windowLength "65536"^^xsd:int ;
    tl:hopSize "8192"^^xsd:int .

AF Examples (3): Dense features

Describing a Chromagram: express the Chromagram signal itself.

@prefix mo: <http://purl.org/ontology/mo/>.

:feature a af:Chromagram ;
    mo:time [
        a tl:Interval ;
        tl:timeline :feature_timeline ;
    ] ;
	af:dimensions "12 0" ;
	af:value "0.00701192 0.00501109 ... 0.00446376" .

Is that all?

We can use MO to link feature data to recordings and other stuff on the Semantic Web

Wouldn't it be useful to describe feature extractors and their parameters ?

• Vamp plugins and the
• Vamp plugin and transform ontologies

allow you to do just that!

Let's briefly look at Vamp Plugins first!

Vamp Plugins

Recall: Vamp plugins take audio input and return structured data (but not RDF!).

Vamp Plugins

Key points:

• Vamp is a C++ audio feature extraction API
• Vamp plugins require a host program to run (e.g. Sonic Visualiser)
• Vamp plugins encapsulate feature extractor algorithms
• Vamp plugins can be written in C, C++, Python (using Vampy)
• You need a suitable host to produce RDF output (Sonic Annotator)

VamPy: An easy way to write Vamp plugins

Vampy is a Python wrapper for the Vamp Plugin API

Key Features:

• Quick prototyping of feature extraction in a high-level interpreted language
  no need to compile the plugins
• Two-way NumPy support
  Writing a plugin is much like programming in Matlab...
• Dynamic type inference
  you don't need to master the C++ API very well

Vampy example plugins available here.

VamPy Example

The process() function, where the feature extraction happens. Given a super-class computing MFCCs, one could write:

	def process(self,inputbuffers,timestamp):
		fftsize = self.m_blockSize
		complexSpectrum = inputbuffers[0]
		magnitudeSpectrum = abs(complexSpectrum)[0:fftsize/2]

		melSpectrum = self.warpSpectrum(magnitudeSpectrum)
		melCepstrum = self.getMFCCs(melSpectrum,cn=True)

		outputs = FeatureSet()
		outputs[0] = Feature(melCepstrum)
		outputs[1] = Feature(melSpectrum)
		return outputs

Vamp Plugin Ontology

Back to ontologies :-)

• Vamp Plugin Ontology:
  Links the results with a plugin and the enclosed algorithm that computed them.
• Vamp Transform Ontology:
  Allows to express the parameters (e.g. window size) that were used to obtain a particular set of results.

Plugins, parameters and results are linked, and described using the same format!

Let's look at the Transform Ontology in more detail.

Vamp Transform Ontology

An open-ended model to describe an unlimited set of parameters.

Vamp Transform Ontology Example (1)

Setting up a Chromagram plugin. The basic settings:

@prefix xsd: <http://www.w3.org/2001/XMLSchema#>.
@prefix vamp: <http://purl.org/ontology/vamp/>.
@prefix : <#>.

:transform a vamp:Transform ;
	vamp:plugin  ;
	vamp:step_size "8192"^^xsd:int ; 
	vamp:block_size "65536"^^xsd:int ; 
	vamp:output  ;

Vamp Transform Ontology Example (2)

Setting up a Chromagram plugin. Parameter bindings:

    vamp:parameter_binding [
        vamp:parameter [ vamp:identifier "tuning" ] ;
        vamp:value "440"^^xsd:float ;
    ] ;
    vamp:parameter_binding [
        vamp:parameter [ vamp:identifier "minpitch" ] ;
        vamp:value "12"^^xsd:float ;
    ] ;
    vamp:parameter_binding [
        vamp:parameter [ vamp:identifier "maxpitch" ] ;
        vamp:value "96"^^xsd:float ;
    ] .

LV2 plugins: using RDF in audio processing?

LV2 is the successor of the LADSPA audio processor plugin API for Linux.

An RDF schema is an integral part of the LV2 plugin API
Static plugin properties (such as the type of the plugin, number of ports, etc..)
don't have to be compiled into the binary, rather, they are described in RDF
• Conceptually different from Vamp where RDF is not mandatory, but a useful extension.
• A tutorial on LV2 programming can be found here

A quick recoup

So far we looked at:

Audio Feature Ontology
Vamp plugins for feature extraction
Vamp plugin ontologies

Let's see how we produce and consume audio features in RDF.

Sonic Annotator

A command line Vamp plugin host that outputs RDF

Key features:

A program for analysing large collections available locally, or on the Web.
It can read a very wide range of audio file formats.
Reads Vamp plugin configuration in RDF
Returns the features in RDF linked with the configuration and editorial data (if available)

Note: Sonic Annotator produces RDF conforming the AF model, however its expressiveness is not yet fully utilised.

Using Sonic Annotator

(1) Create an RDF transform skeleton:

$ sonic-annotator -s vamp:vamp-example-plugins:fixedtempo:tempo > test.n3

(2) Edit the file if necessary and run the feature extractor:

$ sonic-annotator -t test.n3 audio_file.wav -w rdf --rdf-stdout

This will dump the results on the standard output.

A detailed tutorial is available here.

Sonic Visualiser

A Vamp host, Audio Player and Feature Visualiser

Sonic Visualiser

Sonic Visualiser can import audio features expressed using the AF Ontology.

Sonic Visualiser can be used as/for:

• A graphical Vamp plugin host
• Visualise content based audio features
• Manual audio annotation tool
• Synchronised playback of audio and annotations

And a lot more ...

CLAM Aggregator

CLAM is an audio analysis software framework which includes tools for
music annotation, such as CLAM Annotator.

Putting it all together

An eco-system for using Semantic Web technologies in audio analysis:

Putting it all together

SAWA: A proof of concept application on the Web that brings these technologies together.

SAWA: An easy way to get started

Key features:

• An on-line Vamp plugin host
• Upload an audio file to be analysed
• Choose and configure algorithms
• Get the results in RDF which can be linked and published, or viewed in Sonic Visualiser

SAWA: An easy way to get started

How it's built:

• A Pythonic Web server
• UI: generated from plugin descriptors using the Vamp Plugin Ontology
• Configuration: generated from data using the Vamp Transform Ontology
• Feature Extractor Engine: Sonic Annotator
• Returns data using the Audio Feature Ontology

Demos

(1). Using SAWA and Sonic Visualiser

• Analyse audio using SAWA and Vamp plugins
• Load and visualise using Sonic Visualiser

Image credits

ezdyelectricity

Chris Cannam: Vamp Plugin Programmer's Guide.

Sonic Annotator tutorial.