The Tyrex team aims at developing a vision of a web where content is enhanced and protected, applications made easier to build, maintain and secure. It seeks to open new horizons for the development of the web, enhancing its potential, effectiveness, and dependability. In particular, we aim at making significant contributions by obtaining fundamental results, by building advanced experimental applications showcasing these results and by contributing to web standards. The challenging part here is that contributing to each of these lines of work requires progress in all of them simultaneously.
The scientific goal that we want to reach is to establish a unifying framework for designing advanced (robust, flexible, rich, novel) web applications. More precisely it means building a new generation of multimedia and augmented reality applications with new design foundations. The challenging part is to propose means to combine in an easy and compositional manner rich content, augmented reality and data gathered dynamically from the environment and collected from various sensors, cameras, etc. We propose to address this objective along two complementary directions:
modeling, which consists in capturing various aspects of document processing, data and communication in a unifying model, and whose difficult part involves taking into account the specificities of the web, which require new programming models and supporting theoretical tools that do not exist today.
analysis, verification and optimization, which consist in guaranteeing safety and efficiency properties, and whose hard part consists in dealing with problems close to the frontier of decidability, and therefore in finding useful trade-offs between programming ease, expressivity, complexity, succinctness, algorithmic techniques and effective implementations.
From a technology perspective, the generalized use of mobile terminals is having a profound impact on the way users perceive and interact with their environment. The ubiquitous use of search engines capable of producing results in fractions of a second raised the expectations level to a much higher degree. Users do now expect relevant information to be made available to them instantly and directly by context-sensitivity to the environment itself. However, the information that needs to be processed is becoming more and more complex compared to the one found in the traditional web. In order to unlock the potential introduced by this new generation of the web, a radical rethinking of how web information is produced, organized and processed is necessary.
Until now, content rendering on the web has mainly been based on supporting media formats separately. It is still notably the case in HTML5 where, for instance, vector graphics, mathematical content, audio and video are supported only as isolated media types, even though they are more and more present in web pages.
With the increasing use of web content in mobile terminals, we also need to take into account highly dynamic information flowing from sensors (positioning and orientation moves) and cameras. To reach that goal, web development platforms need to ease the manipulation of such content with carefully designed programming interfaces and by developing supporting integrative methods. More precisely, we will focus on the following aspects: (1) the design of rich content models (2) physical environment modeling and integration (3) native streams support and (4) cross-platform programming abstractions.
Our objective is to investigate the concrete design of robust and efficient mobile web applications capable of processing heterogeneous data in a simple manner. Mobile web applications are among the most complex pieces of code, as they involve both: 1) web services able to efficiently gather and process relevant data from the network , and 2) a front-end, tightly integrated with the terminal, which communicates with them.
For a long time, most multimedia web pages have isolated continuous content behind the fences of add-ons or plug-ins, thus preventing real interaction between these contents and the rest of their host page or the whole web. In addition, the many interactive features that are available with discrete content have no equivalent within plug-ins, where users are limited to the same level of control they have with a VCR.
New models are required to represent the many dimensions of such a rich content. Ideally, such models should keep the aspects of traditional documents that have proven useful, and extend them with the specificities of the web environment and continuous contents. The key issue here is to allow all these aspects to be present simultaneously for representing a single document. This would allow document models to be rich and versatile enough to offer many possibilities to a broad range of applications handling multimedia contents.
To be used in real applications, such multimedia document models have to be instantiated in actual formats and languages. As documents have to be part of the web, these formats must interoperate with existing web formats. They could be extensions of existing formats, or new languages that share as many features as possible with the existing ones. The goal is not to create a separate web for multimedia content, but to seamlessly extend the web as we know it.
XML was created for representing documents and data on the web in a secure and rigorous way. XML is now the ground on which web formats are built. If we want to propose new formats for the web, they have to be based on XML, and we need to make sure new applications will be able to take advantage of these formats. It is therefore crucial to better understand how XML structures can be handled, and what are the theoretical tools that may help to develop an effective framework for processing XML structures.
This is of course an ambitious and long-term goal that requires intermediate steps. The first specialized languages for handling XML structures were transformation-oriented (XSLT, XQuery, XDuce, CDuce, etc.). Typically, programs written in these languages read an XML structure and produce another XML structure as their output, after performing some transformations. Query languages can also be considered as behaving that way. So, the transformation paradigm is an interesting intermediate step towards general XML processing. Actually, a number of applications can be built as transformations: document formatting (XSLT was initially developed as part of the XSL formatting language), filtering, merging, conversion, re-purposing, data query, etc.
A major component in an XML transformation language is the part that allows a programmer to select, in the input structure, the data of interest for a given transformation. We thus focused on this part of XML processing languages and we studied, in particular, the XPath language, which is used in a variety of other languages for XML (XSLT, XQuery, XML Schemas). We also studied the CSS Selectors which play a similar role in the CSS language for style sheets. The main goal of this work is to find the theoretical tools and formalisms that are needed for static analysis of XPath expressions, in order to help programmers develop better and more reliable code for XML data and documents.
This work on XML has been recently extended to RDF and its query language SPARQL, in order to extend to the semantic web the results achieved for the web of documents and data.
Before they can be processed, multimedia XML documents have to be created. A significant part of web documents are generated by programs from other documents and data (see XML processing above), but another part is created by human authors using authoring tools. For multimedia formats to be really used, it is important that authoring tools be available.
Our work in the area of multimedia authoring tools aims at developing editing techniques for creating rich multimedia documents that take advantage of the many new dimensions of multimedia formats. The challenge is to keep these tools simple enough for average web users. Methods used for static, textual documents do not work for dynamic, multimedia web resources. New approaches have to be developed and experimented.
Research in this area is strongly connected with software development projects, with the goal of creating real tools that can be deployed on the web and that real users can use.
For the previous three objectives, we have chosen Augmented Reality as an application domain that helps us focus our work in accordance with application requirements.
To recreate or augment our perception of the real world, all modalities may be involved. For visual perception, the media that come to mind are text, graphics, photographs, video (live or recorded). But augmented reality is not restricted to the visual space. The auditory space also contributes to re-creating or extending the user environment. Moreover, the visual and auditory spaces are connected: events happening in one space often have consequences in the other, and all this is synchronized.
The geographical space is important in augmented environments. The location of the users in the real or virtual world plays a key role, as well as the moves they make. This involves mobility and specific kinds of information, such as maps or points of interest (PoIs). A number of information resources required to build augmented environments are available on the web. Applications then not only have to capture a lot of information about the local environment of their user (mainly through various sensors), but also need to access additional information on the web.
All these features of augmented environments are very demanding for the other activities in the team. They require all kinds of multimedia information, that they have to combine. This information has to be processed efficiently and safely, often in real time, and it also, for a significant part, has to be created by human users.
Pierre Genevès, CNRS researcher, received the bronze medal of CNRS in a ceremony organized on october 4th, 2013, in Grenoble. This medal was awarded by the CNRS INS2I institute.
Extensible Markup Language (XML) has gained considerable interest from industry, and plays now a central role in modern information system infrastructures. In particular, XML is the key technology for describing, storing, and exchanging a wide variety of data on the web. The essence of XML consists in organizing information in tree-tagged structures conforming to some constraints which are expressed using type languages such as DTDs, XML Schemas, and Relax NG.
There still exist important obstacles in XML programming, especially performance and reliability. Programmers are given two options: domain-specific languages such as XSLT, or general-purpose languages augmented with XML application programming interfaces such as the Document Object Model (DOM). Neither of these options is a satisfactory answer to performance and reliability issues, nor is there even a trade-off between the two. As a consequence, new paradigms are being proposed which all have the aim of incorporating XML data as first-class constructs in programming languages. The hope is to build a new generation of tools that are capable of taking reliability and performance into account at compile time.
One of the major challenges in this line of research is to develop automated and tractable techniques for ensuring static type safety and optimization of programs. To this end, there is a need to solve some basic reasoning tasks that involve very complex constructions such as XML types (regular tree types) and powerful navigational primitives (XPath expressions or CSS selectors). In particular, every future compiler of XML programs will have to routinely solve problems such as:
XPath query emptiness in the presence of a schema: if one can decide at compile time that a query is not satisfiable, then subsequent bound computations can be avoided
query equivalence, which is important for query reformulation and optimization
path type-checking, for ensuring at compile time that invalid documents can never arise as the output of XML processing code.
All these problems are known to be computationally heavy (when decidable), and the related algorithms are often tricky.
We have developed an XML/XPath static analyzer based on a new logic of finite trees. This analyzer consists of:
compilers that allow XML types, XPath queries, and CSS selectors to be translated into this logic
an optimized logical solver for testing satisfiability of a formula of this logic.
The benefit of these compilers is that they allow one to reduce all the problems listed above, and many others too, to logical satisfiability. This approach has a couple of important practical advantages. First of all, one can use the satisfiability algorithm to solve all of these problems. More importantly, one could easily explore new variants of these problems, generated for example by the presence of different kinds of type or schema information, with no need to devise a new algorithm for each variant.
We have participated in the international endeavor for defining a standard multimedia document format for the web that accommodates the constraints of different types of terminals. SMIL is the main outcome of this work. It focuses on a modular and scalable XML format that combines efficiently the different dimensions of a multimedia web document: synchronization, layout and linking. Our current work on multimedia formats follows the same trend.
With the advent of HTML5 and its support in all popular browsers, HTML is becoming an important multimedia language. Video and audio can now be embedded in HTML pages without worrying about the availability of plugins. However, animation and synchronization of a HTML5 page still require programming skills. To address this issue, we are developing a scheduler that allows HTML documents to be animated and synchronized in a purely declarative way. This work is based on the SMIL Timing and Synchronization module and the SMIL Timesheets specification. The scheduler is implemented in JavaScript, which makes it usable in any browser. Timesheets can also be used with other XML document languages, such as SVG for instance.
Audio is the poor relation in the web format family. Most contents on the web may be represented in a structured way, such as text in HTML or XML, graphics in SVG, or mathematics in MathML, but sound was left aside with low-level representations that basically only encode the audio signal. Our work on audio formats aims at allowing sound to be on a par with other contents, in such a way it could be easily combined with them in rich multimedia documents that can then be processed safely in advanced applications. More specifically, we have participated in IAsig (Interactive Audio special interest group), an international initiative for creating a new format for interactive audio called iXMF (Interactive eXtensible Music Format). We are now developing A2ML, an XML format for embedded interactive audio, deriving from well-established formats such as iXMF and SMIL. We use it in augmented environments (see section ), where virtual, interactive, 3D sounds are combined with the real sonic environment.
Regarding discrete media objects in multimedia documents, popular document languages such as HTML can represent a very broad range of documents, because they contain very general elements that can be used in many different situations. This advantage comes at the price of a low level of semantics attached to the structure. The concepts of microformats and semantic HTML were proposed to tackle this weakness. More recently, RDFa and microdata were introduced with the same goal. These formats add semantics to web pages while taking advantage of the existing HTML infrastructure. With this approach new applications can be deployed smoothly on the web, but authors of web pages have very little help for creating and encoding this kind of semantic markup.
Multimedia documents are considered through several kinds of structures: logical organization, layout, time, linking, animations. We are working on techniques that allow authors of such documents to manipulate all these structures in homogeneous environments. The main objective is to support new advances in document formats without making the authoring task more complex. The key idea is to present simultaneously several views of the document, each view putting the emphasis on a particular structure, and to allow authors to manipulate each view directly and efficiently. As the various structures of a document are not independent from each other, views are“synchronized” to reflect in each of them the consequences of every change made in a particular view. The XML markup, although it can be accessed at any time, is handled by the tools, and authors do not have to worry about syntactical issues.
The term Augmented Environments refers collectively to ubiquitous computing, context-aware computing, and intelligent environments. The goal of our research on these environments is to introduce personal Augmented Reality (AR) devices, taking advantage of their embedded sensors. We believe that personal AR devices such as mobile phones or tablets will play a central role in augmented environments. These environments offer the possibility of using ubiquitous computation, communication, and sensing to enable the presentation of context-sensitive information and services to the user.
AR applications often rely on 3D content and employ specialized hardware and computer vision techniques for both tracking and scene reconstruction. Our approach tries to seek a balance between these traditional AR contexts and what has come to be known as mobile AR browsing. It first acknowledges that mobile augmented environment browsing does not require that 3D content be the primary means of authoring. It provides instead a method for HTML5 and audio content to be authored, positioned in the surrounding environments and manipulated as freely as in modern web browsers.
Many service providers of augmented environments desire to create innovative services. Accessibility of buildings is one example we are involved in. However, service providers often have to strongly rely on experience, intuition, and tacit knowledge due to lack of tools on which to base a scientific approach. Augmented environments offer the required rigorous approach that enables Evidence-Based Services (EBS) if adequate tools for AR technologies are designed. Service cooperation through exchange of normalized real-time data or data logs is one of these tools, together with sensor data streams fusion inside an AR mobile browser. EBS can improve the performance of real-world sensing, and conversely EBS models authoring and service operation can be facilitated by real-world sensing.
The applications we use to elaborate and validate our concepts are pedestrian navigation for visually impaired people and applications for cultural heritage visits. On the authoring side, we are interested in interactive indoor modeling, audio mobile mixing, and formats for Points of Interest. Augmented environment services we consider are, among others, behavior analysis for accessibility, location services, and indoor geographical information services.
Broadly speaking, the main application domain of our research is the web and its numerous applications. This includes the recent evolutions of the web, with a special attention paid to the mobile web, the multimedia web, and the web as a platform for applications. The goal of our research is to enable new multimedia and mobile applications that can be deployed easily on the web, taking advantage of the existing infrastructure and the latest advances in web technology.
More specifically, our work this year has focused on two main application domains: web development and pedestrian navigation.
Current content representation practices and programming methods for the web remain severely limited. Designing web applications is becoming increasingly complex as it relies more and more on a jungle of programming languages, tools and data formats, each targeted toward the different application layers (presentation, application and storage). This often yields complex and opaque applications organized in silos, which are costly, inefficient, hard to maintain and evolve, and vulnerable to errors. In addition, the communication aspects are often handled independently via remote service invocations and represent another source of complexity and vulnerability.
Most research activities in Tyrex address these issues and try to cope with the fundamental aspects of web applications (advanced content, data and communication) by studying rich document formats, data models and communication patterns, to offer correction guarantees and flexibility in the application architecture. For instance, applications need to be checked, optimized and managed as a whole while leveraging on the consistency of their individual components and data fragments.
A number of factors are changing our thinking of an accessible town, namely the open data movement exemplified by OpenStreetMap, MEMS sensors embedded in mobile phones (accelerometers, magnetometers, gyroscopes), web and Augmented Reality technologies, increase in processing power of mobile phones. All these changes are allowing us to build energy efficient urban pedestrian navigation systems.
Richer and more precise AR mobile applications in such fields as cultural heritage visits, outdoor games, or guidance of people with disabilities will be enabled by MMG navigation, i.e. the joint use of micro, macro and global navigation.
Micro-navigation builds upon embedded software's ability to create a greater awareness of the immediate environment, using texture-based tracking or vision algorithms and relating this information to map and IMU data. Micro-navigation includes avoiding obstacles, locating a clear path in the proximate surroundings or at a complex crossing, finding objects and providing absolute positioning using known landmarks or beacons. Micro-navigation works at a precision level of a few centimeters by using predefined landmarks.
Macro-navigation refers to the actions required to find a route in a larger, not immediately perceptible environment, and builds upon carefully designed pedestrian ways incorporating speech instructions, audio guidance, environmental queries and IMU instructions among other things. Macro-navigation works at a precision level of one step using carefully designed routes with map-matching instructions.
Global navigation is based on an absolute global localization system like the GPS. Its precision is that of a few meters if used in an adequate geographical environment where data from external sensors is accessible. It can be used to bootstrap macro-navigation.
There is a duality relation between micro-navigation and macro-navigation. Micro-navigation is based on a localization system giving an absolute position which makes it possible to compute a relative position with respect to the planned route. Macro-navigation is based on a localization system giving a relative position which allows to compute an absolute position on the route through a process called map-matching. As a consequence, this two kinds of navigation complement and enhance each other.
The XML Reasoning Solver is a tool for the static analysis of XPath queries and XML schemas based on the latest theoretical advances . It allows automated verification of properties that are expressed as logical formulas over trees. A logical formula may for instance express structural constraints or navigation properties (like e.g. path existence and node selection) in finite trees.
The tool can solve many fundamental XML problems such as satisfiability of XPath expressions in the presence of XML schemas, containment and equivalence of XPath expressions, and many other problems that can be formulated with XPath expressions and schemas (DTDs, XML Schemas, Relax-NG).
The system is implemented in Java and uses symbolic techniques (binary decision diagrams) in order to enhance its performance. It is capable of comparing path expressions in the presence of real-world DTDs (such as the W3C SMIL and XHTML language recommendations, for instance). The cost ranges from a few milliseconds, for comparison of XPath queries without tree types, to several seconds for queries under very large and heavily recursive type constraints such as the XHTML DTD. These measurements shed light for the first time on the cost of solving static analysis problems in practice. Furthermore, the analyzer generates XML counter-examples that allow program defects to be reproduced independently from the analyzer.
We have introduced the first system capable of statically verifying properties of a given cascading style sheet (CSS) over the whole set of documents to which this stylesheet applies . The system is composed of a set of parsers for reading the CSS and schema files (XML Schema, Relax NG, or DTD) together with a text file corresponding to the problem description as a logical formula. We have developed a compiler that translates CSS files into their logical representations. Then, the solver takes the overall problem formulation and checks it for satisfiability.
We have started the development of an XQuery IDE with a web interface. This prototype integrates static analyses performed by the solver inside a development environment suited for XQuery programmers.
The goal of the ClaireCourseMakeris to provide straightforward editing tools for structuring, annotating and timeline-based authoring of continuous content such as audio or video. Even if it can be used for any content, it is mainly devoted to synchronize pedagogical material (video, slides, chaptering, etc.) in order to provide rich media online courses à la MOOC. The underlying technology is standard-based and uses the open source JavaScript Popcorn library and Popcorn Maker web application by Mozilla.
The result is a wysiwyg web-based authoring tool which benefits from all the generic features of Popcorn and the specific services that cope with chaptering and synchronization needs.
ClaireCourseMaker is the direct follow-up tool of the Timesheet library which is a cross-browser JavaScript implementation for scheduling the dynamic behavior of HTML5 content that can be described with declarative SMIL markup (SMIL Timing and Synchronization, SMIL Timesheets).
ClaireCourseMaker is developed in collaboration with the OpenClassrooms company in the context of the Claire project (see section ).
The MAUDL library (Mobile AUDio Language) is an evolution of the ARIA library whose primary target was games on mobile devices.
Augmented Reality Audio applications use sound objects to create a soundscape. A sound object is a time structure of audio chunks whose duration is on the time scale of 100 ms to several seconds. These sound objects have heterogeneous and time-varying properties. In order to describe Interactive Audio (IA) contents, we created MAUDL, an XML language inspired by iXMF that is well adapted to the design of dynamic soundtracks for navigation systems.
MAUDL prevents audio information overwhelming through categorization at the declarative level and the use of priority queues at the execution level. This takes account of speed when walking, and of rapid hand gestures when interrogating the environment, for example. MAUDL can be used as an authoring time interchange file format for interactive mobile applications or as a runtime file format that is actually loaded through the web and played directly in the device. MAUDL is a cue-oriented interactive audio system, where audio services are requested using named events and the system's response to each event is determined by the audio artist.
The library has been implemented in C++ and now supports different mobile operating systems such as Android and iOS. MAUDL has been widely used in the first demonstrator of the VENTURI project consisting of a mobile augmented reality game.
A virtual 3D audio pointer provides an intuitive guide to the user of a mobile navigation application, reducing the need for cognitive work when compared to vocal instructions. We have built such a pointer using the MAUDL language. It gives the user the azimuth using HRTF spatialized audio cues, with additional hints taking the form of variations in the sound used. It allows superposing other kinds of audio contents, such as voice while the pointer is active, to indicate distance for example. This audio object is suitable for different sorts of navigation systems, such as POI browsers, self-guided audio tours, or applications for following predefined routes.
As the 3D audio pointer is based on MAUDL this technology is also available for both platforms, iOS and Android. It has been used by Metaio for the micro-navigation which is part of the second VENTURI demonstrator. The use case was to help a visually impaired person to find a box on a shelf with computer vision and 3D audio rendering.
PDRTrack is a localization utility running on iOS or Android smartphones used for recording and playing data sets (accelerometer, gyroscope, barometer and magnetometer values) to find out the effect of different pedometer and map matching parameters and their result on localization accuracy. This application uses the PDR library, written in C++ and developed by the team, which provides the user's location in real time based on the interpretation of sensor readings. Three main modules have been designed to build this localization system:
a pedometer that estimates the distance the user has walked and his speed
a motion manager that enables data set recording and simulation but also the creation of virtual sensors or filters (e.g gyroscope drift compensation, linear acceleration, altimeter)
a map-matching algorithm that provides a new location based on a given OpenStreetMap file description and the current user's trajectory
The PDR library has been shared to the VENTURI consortium for the first part of the second year demonstrator: guiding a visually impaired person from Fondazione Bruno Kessler's bus stop to the building entrance. Others partners have used this localization system for retrieving a scale factor needed for the computer vision part (i.e SLAM).
GPS navigation systems, when used in an urban environment, are limited in precision and can only give instructions at the level of the street and not of the pavement. GPS is also limited to outdoor navigation and requires some delicate transitioning system when switching to another positioning system to perform indoor navigation.
IXE is an open source urban pedestrian navigation system based on Inertial Measurement Units (IMU) and running on mobile phones with onboard geographic data and a routing engine. With IXE, the distinction between indoor and outdoor is blurred as an IMU-based location engine can run indoor and outdoor. IXE allows augmented reality queries on customized embedded geographical data. Queries on route nodes or POIs, on ways and relations are predefined for efficiency and quality of information.
Following the web paradigm, IXE is a browser for XML documents describing navigation networks: by using the micro-format concept, one can define inside OpenStreetMap a complex format for pedestrian navigation networks allowing navigation at the level of pavements or corridors. The big advantage of doing this instead of defining new XML languages is that we can use the standard OpenStreetMap editor JOSM to create navigation networks in a short amount of time.
The purpose of the IXE browser is to read these OSM documents and to generate from them visible or audible navigation information. IXE works on any mobile phone running under iOS or Android. Its heart is composed of three engines, one for dead-reckoning navigation, one for interactive audio and the last one for Augmented Reality visual information, allowing quick reconfiguration for extremely varied applications.
IXE can be used for accessible navigation allowing independent living for people with disabilities.
IXE Android is an enhanced version of our iOS navigation demonstrator. It uses our latest work on the localization positioning system such as PDR, GPS, user and NFC. This application is based on predetermined walks described in a XML format extending OpenStreetMap for navigation purpose, everybody can create and share their walks. In order to create a new walk, the author has to follow specifications described in part 6.3.1. We added some functionalities to the open source application Java OpenStreetMap Editor to enhance walk authoring for the IXE app.
Modeling and authoring web content including rich media and interactions is still an open problem. We have drawn a reference state of the art of this area in . We have also contributed to the cultural heritage domain through the experimentation of a multimedia production chain for digitized theatre performances based on semantic annotations .
In the context of the Claire project (see section ), the results we obtained in 2013 in this area are:
providing a data model which is format agnostic to cope with existing and future rendering systems. More specifically, we specified a chaptering component that includes the structuration and navigation features for continuous media such as video.
prototyping a web environment for authoring such rich media (see section . This authoring services are built as a contribution of the Mozilla Popcorn Maker project (Popcorn Maker).
experimenting this environment for the production of the multimedia part of the first MOOC developed by OpenClassrooms, our main partner in this project.
In the area of XML processing, we obtained new results in several directions:
We showed how to translate Schematron descriptions into the tree logic ;
We built the first IDE equipped with path reasoning capabilities ;
We showed that a whole class of logical combinators (or “macros”) can be used as an intermediate language between the query language and the logical language . This provides a gain in terms of succinctness for the logical formalism.
We continued our work on a novel technique and a tool for the static type-checking of XQuery programs, using backward type inference.
We made preliminary investigations on how to support backward navigation axes in the static type checking for XQuery .
In a joint work with the Exmo team, we benchmarked solvers for deciding the problem of query containment for fragments of SPARQL .
We briefly review these results below.
One major concept in web development using XML is validation: checking whether some document instance fulfills structural constraints described by some schema. Over the last few years, there has been a growing debate about XML validation, and two main schools of thought emerged about the way it should be done. On the one hand, some advocate the use of validation with respect to complete grammar-based descriptions such as DTDs and XML Schemas. On the other hand, motivated by a need for greater flexibility, others argue for no validation at all, or prefer the use of lightweight constraint languages such as Schematron with the aim of validating only required constraints, while making schema descriptions more compositional and more reusable.
We built a compiler for Schematron . This compiler takes a Schematron description as input and generates the corresponding constraints as a logical formula. We showed that validators used in each of these approaches share the same theoretical foundations, meaning that the two approaches are far from being incompatible. Our findings include that modal logic can be seen as a unifying formal ground for the construction of robust and efficient validators and static analyzers using any of these schema description techniques. This reconciles the two approaches from both a theoretical and a practical perspective, therefore facilitating any combination of them.
One of the challenges in web development is to help achieving a good level of quality in terms of code size and runtime performance, for popular domain-specific languages such as XQuery, XSLT, and XML Schema. We presented the first IDE augmented with static detection of inconsistent XPath expressions that assists the programmer for simplifying the development and debugging of any application involving XPath expressions . The tool is based on newly developed formal verification techniques based on expressive modal logics, which are now mature enough to be introduced in the process of software development. We further develop this idea in the context of XQuery for which we introduce an analysis for identifying and eliminating dead code automatically. This proof of concept aims at illustrating the benefits of equipping modern IDEs with reasoning capabilities.
A popular technique in the static analysis for query languages relies on the construction of compilers that effectively translate queries into logical formulas. These formulas are then solved for satisfiability using an off-the-shelf satisfiability solver. A critical aspect in this approach is the size of the obtained logical formula, since it constitutes a factor that affects the combined complexity of the global approach.
We showed that a whole class of logical combinators (or “macros”) can be used as an intermediate language between the query language and the logical language . Those logical combinators provide an exponential gain in succinctness over the corresponding explicit logical representation, yet preserve the typical exponential time complexity of the subsequent logical decision procedure. This opens the way for solving a wide range of problems such as satisfiability and containment for expressive query languages in exponential-time, even though their direct formulation into the underlying logic results in an exponential blowup of the formula size, yielding an incorrectly presumed two-exponential time complexity. We illustrated this from a very practical point of view on a few examples such as numerical occurrence constraints and tree frontier properties, which are concrete problems found in the XML world.
We have continued our work on the design of a novel technique for static type-checking of XQuery programs based on backward type inference. The tool looks for errors in the program by jointly analyzing the source code of the program, input and output schemas that respectively describe the sets of documents admissible as input and as output of the program. The crux and the novelty of our results reside in the joint use of backward type inference and a two-way logic to represent inferred tree type portions. This allowed us to design and implement a type-checker for XQuery which is more precise and supports a larger fragment of XQuery than the approaches previously proposed in the literature; in particular compared to the only few actually implemented static type-checkers such as the one in Galax. The whole system uses compilers and a satisfiability solver for deciding containment for two-way regular tree expressions. Our tool takes an XQuery program and two schemas
XQuery is a functional language dedicated to XML data querying and manipulation. As opposed to other W3C-standardized languages for XML (e.g. XSLT), it has been intended to feature strong static typing. Currently, however, some expressions of the language cannot be statically typed with any precision. We argue that this is due to a discrepancy between the semantics of the language and its type algebra: namely, the values of the language are (possibly inner) tree nodes, which may have siblings and ancestors in the data. The types on the other hand are regular tree types, as usual in the XML world: they describe sets of trees. The type associated to a node then corresponds to the subtree whose root is that node and contains no information about the rest of the data. This makes navigational expressions using `backward axes,' which return e.g. the siblings of a node, impossible to type.
In , we discussed how to solve this discrepancy and proposed a compromise: to use extended types representing possibly inner tree nodes in some key parts of a program, and to cut out the subtrees from their original context in the rest.
Querying the semantic web is mainly done through the sparql language or its extensions through paths and entailment regimes. Query containment is the problem of deciding if the answers to a query are included in those of another query for any queried database , . This problem is very important for query optimization purposes. In the SPARQL context, it can be equally useful for distributing federated queries or for implementing schema-based access control. In order to experimentally assess implementation strengths and limitations, we provided a first SPARQL containment test benchmark. We studied the query demographics on DBPedia logs to design benchmarks for relevant query containment solvers. We tested available solvers on their domain of applicability on three different benchmark suites . (i) tested solutions are overall functionally correct, (ii) in spite of its complexity, SPARQL query containment is practicable for acyclic queries, (iii) state-of-the-art solvers are at an early stage both in terms of capability and implementation.
This work has been developed in collaboration with the Exmo team.
The benchmarks, results and software are available at http://
The concept of Mixed Reality comes from the fact that the real-virtual dichotomy is not sharp. Augmented Reality (AR) mode refers to all cases in which the auditory or visual display of a real environment is augmented by virtual sound or graphic objects. Pedestrian navigation is one of the numerous applications that fit into this field. Depending on the real speed of the user and on the real environment in which he moves (inside a building, ...), the system is augmented with synthetic audio instructions and points of interest. OpenStreetMap format has been extended to support navigation authoring and information related to the various passive or active location providers supported by IXE such as PDR, GPS and NFC.
We defined a cue-based XML language (A2ML, for Advanced Audio Markup Language) using SMIL for internal and external synchronization of sound objects. A2ML is specified by a RELAX-NG grammar. A rule-based selector mechanism allows defining style sheets for OpenStreetMap (OSM) elements. This auditory display together with TTS makes our IXE browser accessible to visually impaired people. Format and Delivery for Mixed Reality Content IXE is based on an extended OSM data format with triggering zones, relations or groups with specific semantics and nodes or POIs whose URIs refer to content expressed in HTML5 and A2ML. Content delivery can be of two types, push or pull. Push content is coming from POIs which trigger when the user enters a new zone. This kind of content is very useful for navigation. We support it through a triggering specification that is inserted in the OpenStreetMap document. We use style sheets with rules to specify both the audio and visual rendering of the various types of OSM nodes. Pull content allows users to search detailed information about the artifacts that are located in the content referenced by the POI. Most of the time, this content is described using HTML5 and A2ML.
Pedestrian navigation can be done with several sensors. GPS locations are better for outside locations, PDR is useful to guide people indoor, but we can also use NFC tags, user proprioception, wifi... Our researches focus on a smart fusion of providers depending on sensor accuracy and on the context in which the person moves. We start by using Kalman Filter to smooth locations and disable jumps during the walk. These algorithms have been successfully tested during Venturi Y2 demo.
We worked on offline map rendering around two solutions. The first one is based on an open source Android project called Mapsforge; it provides a tile generation mechanism from a given OpenStreetMap file and a tile caching system for fast rendering on mobile devices. We mainly enhanced the open source project by increasing the zoom level limitation (21 by default) to 24 for displaying indoor maps. The other solution on which we worked is SVG-oriented and based on OpenLayers (dedicated to web browsers). As the rendering uses SVG we are no longer limited by a maximum zoom level. On the other hand, the SVG drawing has to be fully designed by the author, as we don't support, for the time being, SVG file generation from an OpenStreetMap document. These two approaches are different and their uses depend on the desired level of customization of the rendering (generated automatically or manually).
CLAIRE
Title: Community Learning through Adaptive and Interactive multichannel Resources for Education
Call: Technologies for e-education
Duration: March 2012 - February 2014
Coordinator: OpenClassrooms, ex-SimpleIT
Others partners: LIRIS
See also: http://www.projet-claire.fr/
Abstract: Project CLAIRE aims at developing an open-source tool for collaborative authoring in an e-learning environment (Learning Content Management System), targeting teachers and students in high-school and universities. Its innovative features include:
a platform for collaborative structured editing of rich meda and "semantic" content, e.g.: tools for chaptering video, and for generating interactive evaluation tests
processes for continuous enhancement of content, e.g.: social annotation, behaviour analysis, accessible multi-support publishing, e.g.: web, PDF, ODT, LaTeX, smartphones, tablets.
Datalyse
Title: Entrepôt Intelligent pour Big Data hétérogènes. Investissements d’Avenir Développement de l’Economie Numérique.
Call: Cloud Computing, num 3 – Big Data.
Duration: May 2013 - November 2016
Coordinator: Business & Decision Eolas
Others partners: Groupement des Mousquetaires, Inria Saclay (OAK EPC), LIG (Hadas and Erods teams), LIRMM (Montpellier), LIFL (Lille).
See also: http://www.datalyse.fr/
Abstract: Project Datalyse aims at designing and deploying an infrastructure for big data storage, collection, certification, integration, categorization, enrichment and sharing over very large heterogeneous data sets. It relies on an industrial platform, to be made available on the cloud, and focuses on three flagship applications, showcasing three uses of big data over different data sets:
Data Center Monitoring: The goal of this application is to provide features such as traceability, reporting, optimization and analysis of abnormal behaviour regarding energy efficiency and security issues. The application will be built with an existing application called ScopeBR (Eolas) and will be deployed in two different green data centers, those of Eolas and GDF SUEZ.
“Territoire de données ouvertes et liées”: This application aims at extracting and provisioning public open data collected from the city of Grenoble and its suburbs. The goal is to make public data available to third-party application developers and to federate local actors around a single platform.
Real-time Business Intelligence for the management and processing of points of sale: this application will focus on real-time data analytics and will be deployed within “Groupement des Mousquetaires” in support of their business intelligence platforms.
Typex
Title: Typeful certified XML: integrating language, logic, and data-oriented best practices
Call: Programme Blanc
Duration: January 2012 - December 2014
Coordinator: PPS (CNRS - Paris 7 Diderot)
Others partners: LRI (Orsay)
See also: http://typex.lri.fr
Abstract: The highly ambitious and final goal of this project is to produce a new generation of XML programming languages stemming from the synergy of integrating three approaches into a unique framework:
a logical approach based on solvers
a programming language (PL) approach
a data-oriented approach
These languages will feature precise and polymorphic type systems that merge PL typing techniques with logical-solver-based type inference. They will be implemented efficiently using the latest research on tree automata and formally certified using modern theorem prover technology. They will offer the capacity to specify and formally verify invariants, business rules, and data integrity, and will have a direct and immediate impact on standardization processes.
Title: immersiVe ENhancemenT of User-woRld Interactions
Type: Cooperation (ICT)
Call: FP7-ICT-20111.5 Networked Media and Search Systems
Instrument: Specific Targeted Research Project (STREP)
Duration: October 2011 - September 2014
Coordinator: Fondazione Bruno Kessler (Italy)
Others partners: Fraunhofer Heinrich Hertz Institute (Germany), ST Microelectronics (Italy), ST-Ericsson (France), Metaio (Germany), e-Diam Interactive (Spain), Sony-Ericsson (Sweden)
See also: https://venturi.fbk.eu/
Abstract: Venturi aims to create a pervasive Augmented Reality paradigm, where available information will be presented in a user- rather than device-specific way. The goal is to create an experience that is always present whilst never obstructing. Venturi will exploit, optimize and extend current and next generation mobile platforms; verifying platform and QoE performance through life-enriching use cases and applications to ensure device-to-user continuity.
C. Roisin is a member of the steering committee of the ACM Symposium on Document Engineering.
C. Roisin is a member of Inria Grenoble-Rhône-Alpes Committee for selecting PhD students by CORDI-S grants.
C. Roisin is a member of LIG Committee for selecting Grenoble INP and UJF AGIR projects.
C. Roisin is a member of OpenClassrooms Pedagogical Committee.
N. Layaïda is a member of Inria Grenoble-Rhône-Alpes Committee for Employment of Scientists: Inria hiring of Post-docs, Delegates and Secondments.
N. Layaïda is a member of Scientific committee of Persyval-Lab (Labex), Advanced Data Mining.
N. Layaïda is a member of the Evaluation committee of Minalogic cluster (Pôle de compétitivité mondial Grenoble Rhône-Alpes - Micro nanotechnologies et Logiciel).
J. Lemordant is a member of the steering committee of the European FP7 VENTURI project.
P. Genevès and N. Layaïda are program committee members for the 13th ACM Symposium on Document Engineering (DocEng'13).
P. Genevès is a permanent member of Inria Grenoble-Rhône-Alpes Committee for Technological Development: Inria hiring of Research Engineers.
N. Gesbert is a member of COST Action IC1201, « Behavioural Types for Reliable Large-Scale Software Systems ». He attended a meeting in Rome where he gave a talk about session types in a java-like language. He also participated in writing a state-of-the-art report about behavioural types.
P. Genevès has been a member of the COST Action IC0901, entitled “Rich-Model Toolkit - An Infrastructure for Reliable Computer Systems”. In particular he has been invited to the SVARM 2013 workshop co-located with POPL 2013, and nominated for representing France in decisions taken concerning this COST Action.
P. Genevès gave an invited talk about “A Tree Logic and an Application for the Analysis of Cascading Stylesheets” at the Laboratoire de Recherche en Informatique in Orsay in February 2013.
P. Genevès gave an invited talk about “Static Analysis for Data-Centric Web Programming” at the Qatar Computing Research Institute (QCRI), in Doha, Qatar, in November 2013.
In 2013, P. Genevès has been a referee for the following international journals: ACM Transactions on Database Systems and Theoretical Computer Science; and for the ACM symposium on User interface software and technology (UIST) 2013.
In 2013, N. Gesbert has been a referee for the following international journals: Theoretical Computer Science, and Mathematical Structures in Computer Science; as well as for the European Symposium on Programming (ESOP) 2013.
In 2013, C. Roisin has been a referee for the following international conferences: ACM Document Engineering, and Springer Multimedia Systems.
Licence: Computer Networks, 100h/year, L2, University of Grenoble – C. Roisin
Licence: Logic, 67h30/year, L3, University of Grenoble – N. Gesbert
Licence: Algorithmics, 60h/year, L3, University of Grenoble – N. Gesbert
Licence: Academic-side tutoring of an apprentice, 10h/year, University of Grenoble – N. Gesbert
Master: Logic, 22h30/year, M1, University of Grenoble – N. Gesbert
Master: Web development, 22h30/year, M1, University of Grenoble – N. Gesbert
Master: Software analysis, conception and validation, 45h/year, M1, University of Grenoble – N. Gesbert
Master: Foundations for XML: logics and automata, 18h/year, M2 (Mosig), University of Grenoble – P. Genevès
Master: Fundamentals of Mobile Frameworks and Applications, 50h/year, M1, Polytech, University Joseph Fourier of Grenoble – J. Lemordant
Master: Fundamentals of Mobile Frameworks and Applications, 50h/year, M2, University Scientific and technologic of Hanoi – J. Lemordant
Master: Fundamentals of Mobile Frameworks and Applications, 36h/year, M1, MIAGE, University Joseph Fourier of Grenoble – J. Lemordant
Master: Fundamentals of Mobile Frameworks and Applications, 36h/year, M2, University Pierre Mendes-France of Grenoble – J. Lemordant
HdR: N. Layaïda, Representation and analyses of web content and processing, Université de Grenoble, 23 April 2013.
PhD in progress: D. Graux, Large scale evaluation of semantic web queries, since November 2013, N. Layaïda and P. Genevès
PhD in progress: N. Guido, Satisfiability for Expressive
PhD in progress: M. Junedi, Query-Update Independence Analysis under Constraints , since October 2012, C. Roisin and N. Layaïda
J. Lemordant has participated in several events to popularize recent results in the areas of accessible pedestrian navigation, namely conferences at the Osaka City University.
J. Lemordant has participated in several events to popularize recent results in the areas of pedestrian navigation and mixed reality, namely the 6th European eAccessibility Forum and the State Of The Map 2013 conference.
P. Genevès has contributed to the creation of a video for disseminating the obtained research results. This video was displayed during the ceremony when he received the CNRS Bronze Medal. Intended for a very wide audience, it presents the societal challenges addressed, introduces the obtained research results and their applications.
J. Lemordant, T. Michel and M. Razafimahazo showcased indoor and outdoor navigation to Indian journalists (Education Times, Economic Times, Hindustan Times, The Hindu, NDTV) and have been interviewed by Monideepa Banerjee (NDTV). Grenoble, October 2013.
J. Lemordant, T. Michel and M. Razafimahazo have participated to Experimenta 2013 to present their results on the 3D audio integrated in the first year VENTURI demonstrator. Grenoble, October 2013.
M. Razafimahazo has participated to Museomix and developed in 12 hours an iOS application called Museomix#Gre based on TYREX technologies for the Dauphinois Museum. It consists of an interactive map of the museum and visitors were able to understand the concept of virtual reality by being immersed in 5 different panoramas taken during the event. Grenoble, November 2013.