GR/M84077/01 INDIVIDUAL GRANT REVIEW REPORT
HyStream - Applying Open
Hypermedia to Multimedia Streams
Institution: University of Southampton
Department: Electronics and Computer Science
Starts: Feb 1 2000 Ends: Jan 31 2002
EPSRC Reference: GR/M84077/01
Principal Investigator: Professor DC De Roure
Other
Investigators: Dr L Moreau, Professor W Hall
Background
Research context
At the time of writing the proposal, the research community was coming to regard ‘navigational hypermedia’ (a la WWW) as a solved problem, but this was almost entirely with text and image documents and it predated widespread use of broadband Internet – hypermedia had not been properly applied in the context of multimedia applications, especially temporal media. Furthermore, these applications stood to gain from more effective navigational models, during both content production and delivery. This is still the case, and is also true of pervasive computing applications, which we touched on here and subsequently explored in other projects.
The principal goal of the project was to apply the ‘open hypermedia’ model to temporal media. This model, whereby the hypermedia link information is managed separately from the content, has become well established in the hypermedia research community: it is a powerful model, particularly for authoring and personalised delivery, but it demands more sophisticated architectures than the Web. Many such architectures have been explored, with much of the work based in Southampton. In particular, Southampton’s flavour of open hypermedia includes the facility to link on document content rather than locations, which enables hyperstructures to be reused with new content.
The highly interactive multimedia applications that we envisaged were of
interest to BT, who wished to find out whether their next generation networks
would support these next-generation applications – an obvious issue being rapid
switching of streams.
In considering temporal media we distinguish three different scenarios
which raise different issues: streaming from stored media, vs a live
media stream vs a live bi-directional stream (e.g. a multiway
videoconference). A second axis became
apparent during the project: from hypermedia links to metadata to ontologically
principled metadata; i.e. the increasing richness of metadata, through use of
semantic web technologies. This space
is depicted in figure 1. HyStream
started at the bottom left (i.e. supplying links for streams from multimedia
servers) and took us towards live metadata.
The followup project, CoAKTinG, extends this to realtime metadata
associated with realtime streams (e.g from devices) using ontologies.
Figure 1: Project evolution
Project context
The HyStream project was a £100k two-person-year project which commenced in February 2000 and completed in January 2002; BT Exact were the industrial collaborator. It was one of seven projects selected for funding in response to the fifth and final call for proposals in the Multimedia And Networking Applications (MNA) Programme, a managed programme within the EPSRC IT and CS programme, which started in 1995 with a total EPSRC budget of £5.5M. The HyStream proposal was written in March 1999.
The project adopted a multiagent system approach, drawing on our experience of a previous MNA project (“Usability and Integration of Distributed Multimedia Systems using Agents”, led by Wendy Hall, with De Roure and Hugh Davis) and using the Southampton Framework for Agent Research (SoFAR), developed by Luc Moreau. One of the software components was developed in conjunction with “LINKME: Distributed Link Services for mobile environments” (GR/M25919/01) and the EQUATOR IRC (GR/N15986/01).
The HyStream research results and technologies feed into a new project “CoAKTinG: Collaborative Advanced Knowledge Technologies in the Grid” (GR/R85143/01), due to start in summer 2002.
Key Advances and Supporting Methodology
The achievements of the project were as follows:
1. Design and development of a simple protocol for timely delivery of links, synchronised with temporal media from multimedia servers; the protocol is described in [3]. This extended the prior work in which links were transported separately but not in a temporally significant manner (reported in [7]).
2. Instantiation of this protocol within an agent communication language (i.e. as performatives within a multiagent framework) using FOHM (the Fundamental Open Hypermedia Model) [6]. This was published at ACM Autonomous Agents 2001 [3], where it was regarded as significant in that it brought together agent based computing and multimedia research.
3. Implementation of a demonstrator within a multiagent framework, and integration with other agents. This was demonstrated at ACM Autonomous Agents 2001 and Hypertext 2001 [9].
4. Proof-of-concept of generic linking using melodic pitch contours. The pitch contour engine [2] was part of the study reported in [1] in conjunction with BT and was subsequently integrated with demonstrators (see [10] in particular).
5. Implementation of demonstrators using stored (library music archive) and live (student radio station) content (the latter with metadata generated at the studio).
6. Generalisation from link streams to continuous metadata (published in ACM HyperText 2001 [5]), using semantic web technologies (notably RDF). This has become a significant intellectual step forward – it has demonstrated that our experience of open hypermedia takes on a particular relevance in the context of the semantic web.
7. Application to live metadata from ‘smart spaces’ (published in OHS workshop [8] and at the EPSRC PGNET event [4]). This eventually led to CoAKTinG [11].
8. Further consideration of hyperstructure and musical structure (to be published in ACM HyperText 2002 [10]).
Here is a
review against the original objectives:
1.
Extend the open hypermedia philosophy
to unicast and multicast temporal media streams by building a demonstrator with
existing components over IPv6.
Several
demonstrators were built, using both IPv4 and IPv6 (and hybrid versions). Screenshots are available on the project
web site
http://www.iam.ecs.soton.ac.uk/projects/hystream
2.
Incorporate content-based techniques
in this framework, with a case study to identify application requirements.
The melodic pitch contour engine was incorporated into the HyStream framework and an informal evaluation conducted. It was also made available as a service within the agent framework. It was demonstrated using queries based on fragments of MIDI files but also live MIDI input (over IPv4 and IPv6). A web services version of the engine is currently under development.
3.
Study the implications for emerging
Internet protocols and Web standards and make input to those processes;
investigate the implications of branching media on Quality of Service
implementation.
Web Standards activities have been informed by this project: Southampton was represented in XML Linking Language (XLink) Recommendation 1.0 (June 27th 2001), and De Roure is also a member of the Web-Ontology (WebOnt) Working Group. The study of implications of branching media on QoS implementation has not occurred within the project period due to changes at BT, but will occur instead through later Southampton research activities.
4.
Develop and specify a new open
hypermedia protocol and model and exercise this through the demonstrator
The link
transport protocol was developed and implemented in demonstrators. In particular, it was instantiated within
the agent communication languages of the multiagent framework. FOHM (the Fundamental Open Hypermedia Model)
was used, and HyStream effectively provided early bindings for this model.
As part of the MNA programme, HyStream has investigated the following issues which were identified for the programme and in the fifth call:
· Distributed multimedia databases - how to describe, store and retrieve multimedia information in a database.
· Tools and techniques needed for constructing and dynamically reconfiguring value added services and applications from existing components and lower level services.
· Multimedia teleservices including provision for group-based applications and synchronisation services.
Project Plan Review
The following table shows the workplan submitted to the MNA Coordinator after funding was announced (CDLR is the Centre for Digital Libraries Research).
|
|
Workpackage |
Leader |
Tasks |
|
WP
1 |
Initial
Demonstrator |
De
Roure |
1.
Port existing link services to IPv6 |
|
WP
2 |
CBN
Trial |
De
Roure |
1.
Incorporate CBN using existing servers |
|
WP
3 |
BT
Phase 1 |
BT |
1.
Implement demonstrator on BT test network |
|
WP
4 |
New
model |
Moreau |
1.Extend
open hypermedia model to include streams |
|
WP
5 |
Final
demonstrator |
De
Roure |
1.
Develop demonstrator based on new architecture and protocol |
|
WP
6 |
BT
Phase 2 |
BT |
1.
Implement demonstrator on BT test network |
The table is an accurate reflection of the tasks except WP3 and WP6 did not occur at BT within the project as they were not in a position to provide the QoS-enabled test network. WP2 occurred within our own lab, and WP6 will occur in due course as the lab network becomes QoS-enabled. This has only affected the investigation of branching media on QoS-enabled networks – it has had no impact on the other objectives, and in fact has enabled us to take the project further forward with respect to metadata and semantic web technologies (which underpin CoAKTinG). BT participated fully in project meetings and their contribution to the project was very useful.
The first research assistant (Don Cruickshank) was on the project for 18 months, having been responsible for the HyStream protocol design, the implementation within the agent framework, and the initial demonstator.. He left to take a longer contract on a European project, and we replaced him with a new researcher (Kevin Page) for the last 4 months of the project; Page generalised the linking model to live metadata. In between we had a summer student (Clive Minnican, an undergraduate student) build one of the demonstrators (with the library music archive). The project also contributed to the salary of a research fellow (Danius Michaelides) who built the contextual link server (implementing the FOHM model), which is also used in the LINKME and EQUATOR projects.
Research Impact and Benefits to Society
HyStream has been our only research project working with audio, and with music in particular. Single handedly it has delivered three ACM conference papers (one of which was nominated for best paper award) and contributed to two others. The journal paper, which reflects the generic linking aspect, was largely based on the work of a PhD student who was separately funded but informally related to the project, and his software was used by the project. We have established the UK as the major base of this activity internationally, in particular through the large project presence at ACM Hypertext 2001.
Explanation of Expenditure
The project expenditure has been consistent with the original budget. The two research assistants were supplemented by the summer student and partial support of the research fellow in order to bring appropriate skills to the project, a combination which has probably been more effective than a single researcher in post throughout the project.
In addition to conferences, the project staff gave two presentations at the 2nd Annual PostGraduate Symposium on The Convergence of Telecommunications, Networking & Broadcasting, which was sponsored by EPSRC. The HyStream project also contributed to World Wide Web Consortium membership.
Further Research or Dissemination Activities
HyStream feeds directly into CoAKTinG, an e-Science project within the Advanced Knowledge Technologies IRC, due to start this summer. In CoAKTinG, smart spaces (be they videoconferencing rooms or laboratories) are linked by a messaging framework which is used to convey events from one space to another in order to support the distributed collaboration; the events are expressed as metadata according to ontologies to be developed within the project. The two CoAKTinG partners (Edinburgh AIAI and OU KMI) bring complementary technologies to HyStream, and once again BT Exact is a collaborator. BT have also supported a Case Award in this area. A second collaborator (Napp Pharmaceuticals) provides an alternative dissemination route for these technologies.
The open hypermedia model is also in use in the Equator IRC, where it is used as part of a subproject called ‘City’. There is considerable potential for HyStream technologies within Equator, and plans are in preparation for experimenting with musical scenarios later in 2002. It will be interesting to apply the HyStream ideas in linking physical and virtual worlds.
We are actively seeking opportunities to promote and exploit the HyStream approach, and are currently speaking with radio stations with internet broadcasts.
References
[1] David De Roure, Nigel Walker, Leslie Carr. Investigating Link Service Infrastructures. In Proceedings of ACM Hypertext 2000, pages 67-76. ACM Press. May 2000.
[2] D.C. De Roure, S.G. Blackburn. Content-based navigation of music using melodic pitch contours. Multimedia Systems, 8, 3, 190-200, October 2000.
[3] Don Cruickshank, Luc Moreau, and David De Roure. Architectural Design of a Multi-Agent System for Handling Metadata Streams. In The fifth ACM International Conference on Autonomous Agents, pages 505-512, Montreal, Canada, May 2001.
[4] Kevin Page, Ben Juby, Richard Beales, David De Roure. Continuous Metadata. In Proceedings of the 2nd Annual PostGraduate Symposium on The Convergence of Telecommunications, Networking & Broadcasting, pages 264-9, June 2001.
[5] Kevin R. Page, Don Cruickshank, David De Roure. It’s About Time: Link Streams as Continuous Metadata. In: The Twelfth ACM Conference on Hypertext and Hypermedia (Hypertext '01), pages 93-102. Aarhus, Denmark, August 2001.
[6] Mark J. Weal, Gareth V. Hughes, David E. Millard, and Luc Moreau. Open hypermedia as a navigational interface to ontological information spaces. In Proceedings of the Twelveth ACM Conference on Hypertext and Hypermedia HT'01, pages 227-236. Aarhus, Denmark, August 2001.
[7] Neil Ridgway, David De Roure. FOHM+RTSP: Applying Open Hypermedia and Temporal Linking to Audio Streams. The Seventh International Workshop on Open Hypermedia Systems (OHS7) Lecture Notes in Computer Science (LNCS 2266). Springer Verlag. August 2001.
[8] Richard Beales, Don Cruickshank, David De Roure, Nick Gibbins, Ben Juby, Danius Michaelides, Kevin R. Page. The Pipeline of Enrichment: Supporting Link Creation for Continuous Media. The Seventh International Workshop on Open Hypermedia Systems (OHS7) Lecture Notes in Computer Science (LNCS 2266) Springer Verlag. August 2001.
[9] Don Cruickshank, David De Roure, Gareth Hughes, Kevin Page, David Millard, Luc Moreau, Mark Weal. Using Continuous Metadata to Aid Navigation of Ontological and Temporal Information Spaces. Posters and Demonstrations Proceedings of the Twelth ACM Conference on Hypertext and Hypermedia (Hypertext '01), page 17. ACM August 2001.
[10] David C. De Roure, Don G. Cruickshank, Danius T. Michaelides, Kevin R. Page, Mark J. Weal. On Hyperstructure and Musical Structure. ACM Hypertext 2002 (to appear).
[11] Simon Buckingham Shum, David De Roure, Marc Eisenstadt, Nigel Shadbolt and Austin Tate. CoAKTinG: Collaborative Advanced Knowledge Technologies in the Grid. Workshop on Advanced Collaborative Environments WACE 2002 (to appear).