Section: New Results

Experimental Evaluation

The Reproducibility'17 workshop

Participant: Damien Saucez.

Recently, the ACM highlighted that the lack of reproducibility tended to be general in computer science and proposed normalised artifact reviewing and badging definitions (Artifact Review and Badging, https://www.acm.org/publications/policies/artifact-review-badging, December 2017) with the hope that the various ACM communities would perform artifact reviews based on these definition. We organized a special workshop on reproducibility in conjunction with the ACM SIGCOMM 2017 conference to produce a set of recommendations on how to assess the reproducibility of research published in ACM SIGCOMM’related conferences and journals and ways to promote reproducibility. The proceedings of the workshop is available in [27] and we have produced a set of recommendations to the community in [34] with the following conclusions:

The workshop pointed out that there are several hurdles concerning reproducibility, namely the absence of incentives and the bad habit that our community has grown accustomed to. This is evident in the current typical review process which is not adapted to handle reproducibility. Furthermore, there is no general way to share and preserve artifacts (and related documentation), every author does it in their own way.

The workshop focused on the two most important points to be tackled, namely, i) how to provide incentives for reproducible papers and ii) how to share artifacts.

For the first, a promising approach is to put in place a Reproducibility Committee, which will run in parallel with the normal Technical Program Committee of conferences and workshops, which will assess the level of reproducibility of papers accepted for publication by the TPC. Such approach will solve some of the privacy and anonymity issues while reducing the volume of work for the reviewers that volunteer in assessing the reproducibility level.

For the second, a gradual approach has been suggested. The ACM digital library has been suggested as place to start sharing artifacts, which will be also identified via a DOI number. Beside the artifact itself it is important to share all of the meta-information necessary to actually reproduce prior work, as well as a way to provide feedback in order to make the community learn which meta-information is actually important and build guidelines on how to provide such information.

Towards Realistic Software-Defined Wireless Networking Experiments

Participants: Mohamed Naoufal Mahfoudi, Walid Dabbous, Thierry Turletti.

Software-Defined Wireless Networking (SDWN) is an emerging approach based on decoupling radio control functions from the radio data plane through programmatic interfaces. Despite diverse ongoing efforts to realize the vision of SDWN, many questions remain open from multiple perspectives such as means to rapid prototype and experiment candidate software solutions applicable to real world deployments. To this end, emulation of SDWN has the potential to boost research and development efforts by re-using existing protocol and application stacks while mimicking the behavior of real wireless networks. In this work, we provided an in-depth discussion on that matter focusing on the Mininet-WiFi emulator design to fill a gap in the experimental platform space. We showcased the applicability of our emulator in an SDN wireless context by illustrating the support of a number of use cases aiming to address the question on how far we can go in realistic SDWN experiments, including comparisons to the results obtained in a wireless testbed. Finally, we discussed the ability to replay packet-level and radio signal traces captured in the real testbed towards a virtual yet realistic emulation environment in support of SDWN research. This works has been published in a Special Issue on Software Defined Wireless Networks of the Computer Journal [15].

ORION: Orientation Estimation Using Commodity Wi-Fi

Participants: Mohamed Naoufal Mahfoudi, Thierry Turletti, Thierry Parmentelat, Walid Dabbous.

With MIMO, Wi-Fi led the way to the adoption of antenna array signal processing techniques for finegrained localization using commodity hardware. These techniques, previously exclusive to specific domains of applications, open the road to reach beyond localization, and now allow to consider estimating the device’s orientation in space, that once required other sources of information. Wi-Fi’s popularity and the availability of metrics related to channel propagation (CSI), makes it a candidate readily available for experimentation. We have recently proposed the ORION system to estimate the orientation (heading and yaw) of a MIMO Wi-Fi equipped object, relying on a joint estimation of the angle of arrival and the angle of departure. Although the CSI’s phase data is plagued by several phase inconsistencies, we demonstrate that an appropriate phase compensation strategy significantly improves estimation accuracy. By feeding the estimation to a Kalman filter, we further improve the overall system accuracy, and lay the ground for an efficient tracking. Our technique allows estimating orientations within high precision. The results of the study were presented at a IEEE specialized workshop on Network Localization on Navigation [22].

Lessons Learned while Trying to Reproduce the OpenRF Experiment

Participants: Mohamed Naoufal Mahfoudi, Thierry Turletti, Thierry Parmentelat, Walid Dabbous.

Evaluating and comparing performance of wireless systems, like for any other scientific area, requires the ability to reproduce experimental results. In this work, we described the specific issues that we encountered when focusing on reproducing the experiments described in a paper related to wireless systems. We selected the OpenRF paper published in SIGCOMM 2013, a very interesting research work allowing to perform beamforming on commodity WiFi devices. We illustrated how reproducibility is strongly dependent on the used hardware, and why an extensive knowledge of the used hardware and its design is necessary. On the basis of this experience, we proposed some recommendations and lessons for the design of reproducible wireless experiments. This work has been presented at the ACM SIGCOMM 2017 Reproducibility Workshop in August 2017 [21].

Deploying a 5G network in less than 5 minutes

Participants: Mohamed Naoufal Mahfoudi, Thierry Parmentelat, Thierry Turletti, Walid Dabbous.

We proposed a demonstration run on R2lab, an anechoic chamber located at Inria Sophia Antipolis, France. This demonstration consists in deploying a standalone 5G network in less than 5 minutes. All the network components (base station, subscriber management, serving and packet gateways, network traac analyzers) were run automatically using the nepi-ng experiment orchestration tool. Download and upload performance to the Internet from a commercial phone located in the anechoic chamber are shown. This demo has been presented at the ACM SIGCOMM conference in August 2017 [33].