Experimentation users-group [Xug] Meetings

Biology, medicine, physics, astrophysics, chemistry: all these scientific domains need to process large amount of data with more and more complex software systems. For achieving reproducible science, there are several challenges ahead involving multidisciplinary collaboration and socio-technical innovation with software at the center of the problem. Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results. I am seeing this problem as a manifestation of deep software variability: many factors (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results, up to the point it can dramatically change the conclusions of some scientific studies. In this talk, I argue that deep software variability is a threat and also an opportunity for reproducible science. I first outline some works about (deep) software variability, reporting on preliminary evidence of complex interactions between variability layers. I then link the ongoing works on variability modelling and deep software variability in the quest for reproducible science.

In this XUG session, I hope to foster discussions about reproducible science in general, but also about challenges related to (deep) software (variability).

Dear Anonymous Experimenters,

You are welcome to our next session and we'll have a tutorial on Simgrid (see below) Note that this tutorial will take place on site (Inria Rennes center)

Please register here: https://notes.inria.fr/8Jf8nzaaRV270ZVd_eDg_w#

— XUG seminar // https://xug.gitlabpages.inria.fr/meetings

Tuesday 5. April, 4pm - 5:30pm

Petri/Turing Room (Inria - Rennes) —

SimGrid (https://simgrid.org/) is a toolkit that provides core functionalities for the simulation of distributed applications in heterogeneous distributed environments. It enables the sound and reproducible simulation of (Data)Grids, P2P, Clouds, Clusters and HPC, allowing multi-domain studies. It can be used either to simulate algorithms and prototypes of applications, or to emulate real MPI applications through the virtualization of their communication, or to formally assess algorithms and applications that can run in the framework. This project is the result of an international collaboration lasting since more than 20 years. It was used as a scientific instrument in about 500 scientific papers from all continents.

In this tutorial, you will learn how SimGrid can be used to address the classical challenges in the study of distributed systems. A large part will be dedicated to the practical manipulations, through hands-on activities.

Participants are only required to know either C++ or Python as a programming language, and to have a working Docker installation on their laptop.

When small changes in an analysis code lead to different results, what does it tell us on our research? In this talk we will discuss reproducibility in the field of brain imaging (or neuroimaging). The goal of brain imaging is to provide in-vivo measures of the human brain to better understand how the brain is structured, connected and functions. Neuroimaging studies are characterized by a very large analysis space and, to build their analyses, practitioners must choose between different software, software versions, algorithms, parameters, etc. For many years, those choices have been considered as "implementation details" but evidence is growing that the exact choices of analysis strategy can lead to different and sometimes contradictory results.

In this talk, we will review examples from the literature that demonstrated a lack of reproducibility in brain imaging studies under different contexts. We will discuss how realization of the prevalence of irreproducibility has led to changes in research culture – in particular around data sharing –. Finally we will open the discussion to think more specifically about irreproducibility related to changes in the analysis code and how those can be interpreted.

Slides are available here: docs/S04E05-Maumet-XUG_beyond_computational_reproducibility.pdf

Distributed digital infrastructures for computation and analytics are now evolving towards an interconnected ecosystem allowing complex applications to be executed from IoT Edge devices to the HPC Cloud (aka the Computing Continuum, the Digital Continuum, or the Transcontinuum). Understanding and optimizing end-to-end performance of applications in such a complex continuum is challenging. This breaks down to reconciling many, typically contradicting application requirements and constraints with low-level infrastructure design choices. One important challenge is to accurately reproduce relevant behaviors of a given application workflow and representative settings of the physical infrastructure underlying this complex continuum.

In this talk, I will introduce a rigorous methodology for such a process and its implementation in E2Clab, a framework supporting the reproducible and complete experimental cycle across the Edge-to-Cloud Continuum: (i) the configuration of the experimental environment, libraries and frameworks; (ii) the mapping between the application parts and machines on the Edge, Fog and Cloud; (iii) the deployment of the application on the infrastructure; (iv) the automated execution and optimization; and (v) the gathering of experiment metrics.

Next, I will illustrate our methodology by optimizing Pl@ntNet, a world-wide plant identification application deployed on the Grid'5000 testbed, showing that our framework allows one to understand and improve performance.

Finally, I will present how our methodology can be generalized to other applications in the Edge-to-Cloud Continuum. Lastly, I will briefly highlight the current work on adding support to the FIT IoT LAB testbed in E2Clab and the research challenges on enabling provenance capture in Edge-to-Cloud experiments as a support to the computational reproducibility.

Slides are available here: docs/S04E04-rosendo.pdf

• When Tuesday 16., 23. and 30. november from 4pm to 5:30pm / Lipari Room (see general agenda below)
• Subscriptions https://notes.inria.fr/aATnaCtXRX-zLWfzlQ9E3w# (please add your email to one or more sessions)
• Prerequisites
  • An active G5k account[1] with a setup allowing SSH connections to the platform[2]
  • A laptop with WIFI access to the internet from the Inria meeting room (e.g Internal WIFI, Eduroam)

[1]: https://www.grid5000.fr/w/Grid5000:Get_an_account

[2]: https://www.grid5000.fr/w/Getting_Started#Connecting_for_the_first_time

These 3 sessions are open to everyone and will let you walkthrough and practice with the platform (see below). Note that for PHD students: these sessions can be used as 'heures de formation'.

I also encourage you to come with your own use case so that we can discuss it (and help implementing it).

• [Session 1] Getting started / Computational jobs -> when you need to run a function and are interested in the results. We'll cover the basic mecanisms offered by the testbed in this context: resource selection / different jobs types / SPMD jobs / production queue / GPUs…
• [Session 2] Experiment scripting -> when you need to add some automation to your experiments We'll discover the Grid'5000 REST API and some of tools[1] to start automating (Jupyter notebooks, Execo, python-grid5000 …)
• [Session 3] Deployment of custom applications and control of the environment -> When you need to experiment with a custom app/deployment We'll cover some of the advanced features of Grid'5000: VLAN networks, several NIC supports, Network emulation, Instrumentation of the deployment… We'll use EnOSlib to practice with all the above.

[1]: https://www.grid5000.fr/w/Grid5000:Software

What kind of platform for what kind of FOG experiment ?

Let's take Hamid's talk (summary below) and his experience about FOG computing environment as an opportunity to bootstrap discussions around experimental evaluation of FOG systems.

– Reproducible data stream processing (DSP) experiments seem achievable with some efforts; but reproducible Fog computing testbed for running stream processing experiments is another story! In this talk, I will present how we set up an experimental DSP+Fog testbed over a RaspberryPI cluster, and how we use Ansible to automate installation and deployment of Docker, Kubernetes, Apache Flink, Prometheus, and Grafana on our testbed.

I also will briefly explain how we use this testbed to run a stream processing application in Flink and how we measure Flink’s performance metrics in order to automatically scale the system. Finally, I will share my experiences about technical problems as well as remaining challenges in terms of testbed and automation. –

Slides are available here: docs/S03E03-hamid-arkian.pdf

– Given a configuration, can humans know in advance the size, the compilation time, or the boot time of a Linux kernel? Owing to the huge complexity of Linux (there are more than 15000 options with hard constraints and subtle interactions), machines should rather assist contributors and integrators in mastering the configuration space of the kernel. In this talk, I will introduce TuxML an OSS tool based on Docker/Python to massively gather data about thousands of kernel configurations. I will describe how 200K+ configurations have been automatically built with IGRIDA. I will briefly explain how machine learning can exploit this information to predict properties of unseen Linux configurations, with different use cases (identification of influential/buggy options, finding of small kernels, etc.) The vision is that a continuous understanding of the configuration space is undoubtedly beneficial for the Linux community, yet several technical challenges remain in terms of infrastructure and automation.

I will particularly share my experience related to the technical difficulties faced when gathering data, a shared concern of the XuG group –

Slides are available here: docs/S03-E02-mathieu-acher-tuxML-XuG.pdf

– Measuring the duration of a matrix product is harder than you think

The matrix product algorithm is widely used in a lot of applications, especially in the world of HPC. While its asymptotic complexity only depends on the sizes of the matrices, the actual duration may vary significantly depending on how you measure it. In this talk, I will present the troubles we had to get reproducible performance measures of such a simple workload. –

Slides are available here: docs/S03E01-tom-cornebize.pdf

– Reproducible research on microarchitectural attacks

Reproducible research is generally speaking a good idea, but can be a bit of a nightmare when dealing with hardware. In this presentation, I will talk about reproducibility in security, and more precisely the reproducibility of attacks on microarchitecture. We will see the good, the bad, and the ugly of trying to reproduce these attacks, and the magic solutions I would like to see in this domain (a girl can dream). –

Slides are available here: docs/S02E03-clementine_maurice.pdf

Seminar: Introduction to R for your data analysis and visualisations

• Session 1: Tuesday 28. January 2020, 9 am - 12 am
• Session 2: Thursday 30. January 2020, 2 pm - 5 pm
• Where: Noirmoutier Room (A 030)

Have you ever felt lost for your data analysis and visualisations ? The following seminar/tutorial by Emmanuelle Becker is for you !

* The session offers a short introduction to the use of R to manipulate and combine data tables, and to the ggplot2 library to visualize data.

Keywords:

• R, tidyverse libraries (tidyr, dplyr, reshape2, ggplot2),
• Classical plots (barplot, boxplot, pies, histograms, curves) with or without facetting.

*

Misc. notes:

• The number of participants is limited to 10 for this first session.
• Laptops aren't mandatory: the workstations of the Noirmoutier room will be used for the hands-on exercices.
• For PhD students, this seminar can be validated as doctoral courses

Slides and materials are available here: https://gitlab.inria.fr/ebecker/formation-2020-introduction-to-r-for-data-manipulation-and-data-visualisation

– Are Android malware detection scanners robust to newest adversarial attacks ?

To answer this question, we generate thousands of variants of an existing malware by combining multiple known dissimulations techniques and submitting them to state of the art detection scanners. That said, we are able to generate more than 12000 functional variants of a single malware. Generating this high number of malware variants in a reasonable amount of time implies scalability issues.

More precisely, crafting a new variant requires to manipulate a lot of binary files, to compile Java and native (C/C++) source code, to decompile dex bytecode, to compress and decompress many ressource files, etc. Finally, each generated variant must be dynamically tested to ensure a correct execution but also statically analyzed to extract the necessary data needed to test detection scanners.

I propose to give a short introduction of my research topic, then to present how I manage to perform these experiments at scale using Igrida and finally a few words on how I store and analyze all this data. –

Dear Anonymous Experimenters,

We will have our first meeting for the current academic year:

• Who: Anyone interested in experimenting (more especially Phd students)
• Why: https://xug.gitlabpages.inria.fr/meetings/
• When: 11-07-19 // 11am - 12am
• Where: Houat C217

In this session we'll discover a framework to evaluate FOG applications with an application to earthquake detection. This is a follow-up of an experimental campaign started by Pedro Silva (KerData) during last season of the XUG meetings. The volcano has erupted and things needs to be scaled to handle the activity of thousands of sensors !

Before that we'll have a short presentation of the past meetings and we'll set some objectives for the upcoming ones (tutorials, seminars, collaborations …) — https://xug.gitlabpages.inria.fr/meetings/

Pedro gave us his advice to avoid to work in "warrior/berserk mode".

Slides are available here: docs/S02E01-pedro_silva.pdf

Pedro needs to manage a large infrastructure of distributed data-analytics. Networks sensors (layer 0) at the edge send metrics to the fog nodes (layer 1) that also send data to the cloud (layer 2) for traditional Big Data/Machine learning computations. Pedro is interested in knowing the performance of the data processing pipeline accros the three layers with different data processing ratio in the different layers. For this purpose he needs to control a complex software stack and the network characteristics (emulation). His platform is instrumented with probes which report various metrics for every single component (resource metric, traffic, latency between the different processing stage…).

Experimentation-wise the lessons-learnt from Pedro are (see the slides for more info):

• know your objectives: work in isolation, define clearly the objectives and hypothesis at each step
• know your software stack: complex software stacks have a lot of tunable parameters. In addition your experimental setup may influence some of the observed behaviour.
• know your data: understand how load should be generated and what impact on the system it has.
• know your result expectations: the first graphs you'll draw are probably wrong. Confront deeply what was expected and what was observed.
• know your scale: scaling is probably not straightforward – especially with complex software stacks composed of many internal interactions (This highlights the needs to isolate each part of the experiment and test in isolation)

• online version: https://xug.gitlabpages.inria.fr/nix-tutorial/
• sources: https://gitlab.inria.fr/xug/nix-tutorial

Nix[1] is a package manager with good properties such as software reconstructability, reproducibility and portability. This session will introduce Nix and its main concepts, and see why it is a strong candidate to package your experimentation software stack.

To register please add your name to the list here: https://pad.inria.fr/p/np_9rY1G3ITFz3l0PZL_nix_tutorial Or send me an email: matthieu.simonin@inria.fr

The seminar will alternate between concepts presentation and hands on sessions. The following will be covered:

Overview of Nix

In this section, the attendees will learn the basics of Nix and the associated commands for installing, upgrading, and removing packages.

Development Workflow

We start by creating a simple package, and we show how developers can leverage Nix in their coding sessions.

Experiment Packaging

In order to build an experimental workflow, we use the good properties of Nix, namely explicit build instructions, software environments description and (re)constructability.

Working with Notebooks

This section continues the experiment packaging, by adding different notebooks technologies (Jupyter and Rmarkdown) to our experimental workflow.

Experiment deployment (on g5k)

In this section, we introduce different caching techniques available with Nix to export your software stack from machines to machines.

NB: Apart from the following requirements, no prior knowledge is expected in any of the tools used. Prerequisites:

In order to run the examples on Grid'5000 you'll need

a WIFI access [2]

an active Grid'5000 account [3]

to access Grid'5000 using SSH [4]

(optional) In order to run the examples on your local machine you'll need

a WIFI access [2]

a compatible OS: Linux (i686, x86₆₄, aarch64) or macOS (x86₆₄)

an recent kernel (with usernamespaces) and root access

some disk space (> 2GB)

[1]: https://nixos.org/nix/ [2]: https://wiki.inria.fr/support/Wifi [3]: https://www.grid5000.fr/w/Grid5000:Get_an_account [4]: https://www.grid5000.fr/w/Getting_Started

Dear Anonymous Experimenters,

We will have our fourth user-group meeting on the 29th of May / 11 a.m - 12 a.m / F202 - Lipari

For this session, we'll shift from a presentation + discussions session to a more interactive discussion between us.

Starting from undone/unfinished yet or a planned experiment of a presenter we'll discuss the experimental protocol, tools, good practices that might be envisioned for this specific experiment.

The expected outcome of the session is:

• for the presenter: a greater confidence on his experimentation plan
• for all experimenters: an increased knowledge on experimenting in a repeatable/replicable/reproducible way.

Pedro Silva is a postdoc in KerData and he will be our presenter. His plan is to run an experiment involving stream processing over a fog infrastructure. Let's make his experiment a great one !

By the way, if you want to be involved as a presenter, let me know :)

Next meeting

We will have our third users-group meeting on the 25th of April / 11 a.m. - 12 a.m. / Lipari F202

In this session Millian (Myriads postdoc) will give feedback about his PhD experimentation practice. Here is a brief abstract.

Experimenting with simulation campaigns may look easy at first sight, as simulation is reproducible. Many problems may however arise from the experiment design itself or from the complexity of the software involved in the experiment. In practice, non-reproducible simulation results are common =/.

As an example, this presentation will talk about the evaluation of resource management policies to save energy while keeping "good" performance. The focus will not be on the scientific result but on the design of the software required to conduct such experiments. The decisions that have been made about the following topics will be detailed and discussed.

• How to organize the code and data related to the experiment?
  • Definition/storage of the studied parameters.
  • Execution of the different processes (on your laptop and/or on a cluster).
  • Result storage and aggregation.
  • Result analysis and visualization.
• How to escape version/dependency hell?
• How to tend towards full repeatability?
• What about software reuse?

Slides are available here: docs/2019-04-25-mpoquet_feedback.pdf

Announcement

Our group has now a webpage! https://xug.gitlabpages.inria.fr/meetings/

You'll find there the planning and some notes about the meetings.

Next meeting

We'll have our second users-group meeting on the 27th of March / 11 a.m. - 12 a.m. / Sicile F302

In this session Javier and Matthieu, both engineers, will share with us some aspects of their experimentation practice. Please find below a brief abstract:

Performing experimental campaigns using distributed software can be sometimes a tedious task. You can think of your own distributed software that needs to be scaled and tested, or of a large software stacks that are part of your experimentation process.

We'll show you concrete examples of performance evaluations that have been performed in a geo-distributed context. First we'll cover the general idea of the campaigns and their main characteristics (size, number of parameters, …). Then the examples will serve us to illustrate the successes and difficulties we faced in terms of:

• scaling the experiments
• calibrating them
• and automating them

• slides are available: docs/2019-03-27.pdf

We'll have our second users-group meeting on the

28th of February / 11 a.m. / Lipari (F202)

In this session, we'll be given a concrete example of an experimental campaign by Loïc Guégan.

Loïc is PhD student in Myriads team. He will share with us how he crafted his experimentation about evaluating the performance and accuracy of two simulators (SimGrid and NS3). He'll show us how he maintained his laboratory notebook (based on org-mode) and how he orchestrated the numerous parameters he had to explore (on Grid'5000).

We'll have the opportunity to discuss more generally if the methodology used above can fit your specific use case.

• What we've done
  • Loïc has presented the experimentation side of its work.
  • Scientific question: Energy models of Simgrid VS NS3 (Ecofen module)
  • Loïc maintains a notebook in org-mode which contains everything about the paper:
    • notes, more formal textual part – later exported as the paper
    • scripts for data provenance and data analysis (see below) – extracted (tangled) as executable scripts
  • Data provenance workflow:
    • For a given, topology (communicating entities and links), node and link caracteristics (e.g half/full-duplex), a given simulator
    • Run the simulation
    • Collect metrics + metadata (what exact command has been run), environment capture (what physical node / caracteristics)
    • Repeat until all the parameters have been explored
  • Experimentation dimension:
    • 1440 parameters to explore
    • Each parameter run is independant
    • The volume of the collected data is reasonnable (fit into the G5K home)
    • Note: we haven't spoken about the experiment calibration
  • Data provenance implementation
    • Master bash script
      • Distribute parameters to run to machines (according to a max parallelism setting)
      • Generate the reservation command on the frontend (oarsub command) that launches the worker processes on the nodes
      • Logs of the exectution is written by the worker process back into the home dir of G5K
  • Data analysis
    • R scripts, ran in the local machine, after the logs are retrieved from the frontend
• What we've decided
  • Since the above experimentation corresponds to a massively parallel execution of jobs, we'll probably try to find someone (hum hum) who can talk about experimentation that requires orchestration.

As announced during the D1 day, we are starting a users group on experimentation. The idea is to share good practices about experimentation, and solve together the problems of each participant!

If you are currently (or plan to become) a user of Grid'5000 / Igrida / SimGrid, join us! Free entrance for everybody!

First meeting will take place on January 24 from 3pm to 4pm in Lipari room (F building, Orange floor).

This first meeting will last one hour max and will center around initial discussions among participants on their experimental habits and issues. We then expect a monthly meeting on these topics.

Btw, Martin's D1 day slides are here : http://people.irisa.fr/Martin.Quinson/blog/2018/1128/how_to_fail_an_experiment.pdf

• What we've done
  • We mainly went around the tables
  • then we brainstorm on how to proceed in the short term for the group
• What we've decided
  • We'll need to organize a Grid'5000 tutorial for beginners
  • To start a presentation cycle with volunteers to speak about their experimentation. This should help us to find a common vocabulary and start to build some comon knowledge on experimentation. Volonteers are invited to speak about
    • their use of experimentation (simulation, in-vivo experimentation …)
    • what methodology they follow – what tools ?
    • what was painful / painless