Extreme CFD workshop - User contributions [en]

Ecfd:ecfd 4th edition

2021-04-02T10:41:39Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

[4] Alexandre J. Chorin. '''Numerical solution of the Navier-Stokes equations'''. ''Math. Comp.'', 22:745–762, 1968. [https://doi.org/10.1090/S0025-5718-1968-0242392-2]

[5] Gerard L. G. Sleijpen, Diederick R. Fokkema. '''BiCGStab(ℓ) for linear equations involving unsymmetric matrices with complex spectrum'''. ''Electron. Trans. Numer. Anal.'', 1:11–32, 1993. [http://etna.mcs.kent.edu/volumes/1993-2000/vol1/abstract.php?vol=1&pages=11-32]

=== Turbulent flows - P. Bénard, CORIA ===

''Participants: P. Bénard, P. Bénez, S. Gremmo, F. Houtin-Mongrolle, S. Meynet, E. Muller (CORIA), A. Barge, G. Balarac (LEGI), A. Viré (TUDelft), P. Tene Hedje, U. Vigny, L. Bricteux (UMONS)''

Turbulent flows are involved in many applications. Its understanding and control is a major task to improve system design and efficiency. The intrinsic multi scale characteristics of turbulent flows makes CFD an interesting predictive tool to tackle these challenges. This project gathered four sub-projects related to Turbulent Flows. These activities have direct application in wind turbines flows or heat exchangers.

* '''Sub-project 1: Atmospheric turbulent wind modeling via recycling method (E. Muller, S. Meynet, U. Vigny, F. Houtin-Mongrolle, P. Bénard)'''
The main objective of this sub-project was to take advantage of the recently developed recycling boundary condition in the YALES2 library to apply it on atmospheric turbulent flows. Such method can become an alternative to usual synthetic turbulence injection for such application. Some questions were tackled as the obtained mean velocity profile, the velocity fluctuation level and computational cost. The results showed a promising strategy but further investigations must be performed on the mesh resolution influence and floor wall model coupling.

* '''Sub-project 2: Model development and validation for vertical axis wind turbine (I. Tsetoglou, P. Tene Hedje, P. Bénez, F. Houtin-Mongrolle, P. Bénard)'''

* '''Sub-project 3: Multiphysics modelling of wind turbines (S. Gremmo, F. Houtin-Mongrolle, E. Muller, A. Viré)'''

* '''Sub-project 4: Advanced tools for turbulent flows study (A. Barge, S. Meynet, F. Houtin-Mongrolle, G. Balarac, P. Bénard)'''

* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

Ecfd:ecfd 4th edition

2021-04-02T10:41:10Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

[4] Alexandre J. Chorin. '''Numerical solution of the Navier-Stokes equations'''. ''Math. Comp.'', 22:745–762, 1968. [https://doi.org/10.1090/S0025-5718-1968-0242392-2]

[5] Gerard L. G. Sleijpen, Diederick R. Fokkema. '''BiCGStab(ℓ) for linear equations involving unsymmetric matrices with complex spectrum'''. ''Electron. Trans. Numer. Anal.'', 1:11–32, 1993.
[http://etna.mcs.kent.edu/volumes/1993-2000/vol1/abstract.php?vol=1&pages=11-32]

=== Turbulent flows - P. Bénard, CORIA ===

''Participants: P. Bénard, P. Bénez, S. Gremmo, F. Houtin-Mongrolle, S. Meynet, E. Muller (CORIA), A. Barge, G. Balarac (LEGI), A. Viré (TUDelft), P. Tene Hedje, U. Vigny, L. Bricteux (UMONS)''

Turbulent flows are involved in many applications. Its understanding and control is a major task to improve system design and efficiency. The intrinsic multi scale characteristics of turbulent flows makes CFD an interesting predictive tool to tackle these challenges. This project gathered four sub-projects related to Turbulent Flows. These activities have direct application in wind turbines flows or heat exchangers.

* '''Sub-project 1: Atmospheric turbulent wind modeling via recycling method (E. Muller, S. Meynet, U. Vigny, F. Houtin-Mongrolle, P. Bénard)'''
The main objective of this sub-project was to take advantage of the recently developed recycling boundary condition in the YALES2 library to apply it on atmospheric turbulent flows. Such method can become an alternative to usual synthetic turbulence injection for such application. Some questions were tackled as the obtained mean velocity profile, the velocity fluctuation level and computational cost. The results showed a promising strategy but further investigations must be performed on the mesh resolution influence and floor wall model coupling.

* '''Sub-project 2: Model development and validation for vertical axis wind turbine (I. Tsetoglou, P. Tene Hedje, P. Bénez, F. Houtin-Mongrolle, P. Bénard)'''

* '''Sub-project 3: Multiphysics modelling of wind turbines (S. Gremmo, F. Houtin-Mongrolle, E. Muller, A. Viré)'''

* '''Sub-project 4: Advanced tools for turbulent flows study (A. Barge, S. Meynet, F. Houtin-Mongrolle, G. Balarac, P. Bénard)'''

* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

Ecfd:ecfd 4th edition

2021-04-01T14:26:10Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

[4] Alexandre J. Chorin. '''Numerical solution of the Navier-Stokes equations'''. ''Math. Comp.'', 22:745–762, 1968. [https://doi.org/10.1090/S0025-5718-1968-0242392-2]

[5] Gerard L. G. Sleijpen, Diederick R. Fokkema. '''BiCGStab(ℓ) for linear equations involving unsymmetric matrices with complex spectrum'''. ''Electron. Trans. Numer. Anal.'', 1:11–32, 1993.

=== Turbulent flows - P. Bénard, CORIA ===
* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

Ecfd:ecfd 4th edition

2021-04-01T14:24:25Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

[4] Alexandre J. Chorin. '''Numerical solution of the Navier-Stokes equations'''. ''Math. Comp.'', 22:745–762, 1968. [https://doi.org/10.1090/S0025-5718-1968-0242392-2]

[5] Gerard L. G. Sleijpen, Diederick R. Fokkema. '''BiCGStab(ℓ) for linear equations involving unsymmetric matrices with complex spectrum'''. ''Electron. Trans. Numer. Anal.'', 1993, 1:11–32.

=== Turbulent flows - P. Bénard, CORIA ===
* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

Ecfd:ecfd 4th edition

2021-04-01T14:17:02Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

[4] Alexandre J. Chorin. '''Numerical solution of the Navier-Stokes equations'''. ''Math. Comp.'', 22:745–762, 1968. [https://doi.org/10.1090/S0025-5718-1968-0242392-2]

=== Turbulent flows - P. Bénard, CORIA ===
* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

Ecfd:ecfd 4th edition

2021-04-01T14:14:04Z

Sahut: /* Numerics - G. Lartigue, CORIA */

{{DISPLAYTITLE: ECFD workshop, 4th edition, 2021}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:logo_ecfd4.png | center | thumb | 300px | ECFD4 workshop logo.]]
|}
* Virtual event from '''22nd to 26th of March 2021'''
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, UMONS, UVM, VUB), HPC center/experts (GENCI, IDRIS, NVIDIA, HPE) and industry (Safran, Ariane Group).
* Web TV: [https://webtv.insa-rouen.fr/channels/#ecfd4 https://webtv.insa-rouen.fr/channels/#ecfd4]

== News ==

Annoncements on Linkedin
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_the-4th-extreme-computational-fluid-dynamics-activity-6777492300546236416-njcV '''First annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_2nd-day-of-extreme-computational-fluid-dynamics-activity-6780117155796017152-Epr5 '''Second day annoncement''']
* [https://www.linkedin.com/posts/christelle-piechurski-429b3925_4th-day-of-ecfd4-starting-with-a-plenary-activity-6781048446448140288-wvlz '''Fourth day annoncement''']


== Objectives ==

* Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
* Identify the technological barriers of exaflopic CFD via numerical experiments
* Identify industrial needs and challenges in high-performance computing
* Propose action plans to add to the development roadmaps of the AVBP and YALES2 codes

== Agenda ==

[[File:Agenda ECFDW4.png | 800px | CFDW4 agenda]]

==== Plénière 1 ====
Lundi 22/03/2021 9h00-9h20

'''Introduction (organisation, agenda semaine, etc.)'''

''V. Moureau (CORIA), G. Balarac (LEGI), C. Piechurski (GENCI)''

==== Plénière 2 ====

Lundi 22/03/2021 9h20-11h20

'''Présentation des projets du workshop et Présentation des thématiques du hackathon'''

''Responsables de projets''

==== Plénière 3 ====

Lundi 22/03/2021 11h20-12h00

'''Contrat de Progrès Jean Zay: Véhicule d'accompagnement des utilisateurs au portage des applications sur les nouvelles technologies'''

''P.-F. Lavallée (IDRIS)''

==== Plénière 4 ====

Mardi 23/03/2021 9h00-10h00

'''Evolution de la programmation GPU – CUDA, OpenACC, Standard Langages (C++, Fortran)'''

''F. Courteille (NVIDIA)''

==== Plénière 5 ====

Mercredi 24/03/2021 13h00-14h00

'''Le portage applicatif sur GPU de AVBP et Yales 2: Concrêtement comment cela se matérialise?'''

''G. Staffelbach (CERFACS) & V. Moureau (CORIA)''

==== Plénière 6 ====

Jeudi 25/03/2021 9h00-10h00

'''Approche et démarche pour accompagner le portage d'un code sur GPU NVIDIA'''

''P.-E. Bernard (HPE)''

==== Plénière 7 ====

Vendredi 26/03/2021 9h00-10h00

'''Roadmaps YALES2 & AVBP'''

''V. Moureau (CORIA) & N. Odier (CERFACS)''

==== Plénière 8 ====

Vendredi 26/03/2021 15h00-17h00

'''Wrap-up : présentation des résultats et conclusion générale'''

''Responsables de projets + V. Moureau (CORIA)''

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - B. Cuenot, CERFACS ===
* H2 and alternative fuels combustion
* turbulent combustion modeling

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

''Participants: G. Balarac and M. Bernard (LEGI), Y. Dubief (Vermont U.), U. Vigny and L. Bricteux (Mons U.), A. Grenouilloux, S. Meynet and P. Bernard (CORIA), R. Mercier and J. Leparoux (Safran Tech), P. Mohanamuraly, G. Staffelbach and N. Odier (CERFACS))''

Mesh adaptation is now an essential procedure to be able toi perform numerical simulations in complex geometries. The aim of mesh adaptation is to be able to define an "objective" mesh allowing the best compromise between accuracy and computational cost, with a reproducibility property, i.e. independent of the user. This project gathered thus six sub-projects related to static and dynamic mesh adaptation, with the main objectives to improve mesh adaptation capabilities of codes (sub-projects 1 and 2), to allow automatic mesh convergence (sub-projects 3 and 4), and to perform dynamic mesh adaptation for specific cases (sub-projects 5 and 6).

* '''Sub-project 1: Coupling TreeAdapt / AVBP (P. Mohanamuraly, G. Staffelbach)'''
The main objective of this sub-project was to couple the TreeAdapt library with the AVBP code. TreeAdapt is a library based on the partitioning library TreePart. This allows a hierarchical topology-aware massively parallel, online interface for unstructured mesh adaption. During the workshop the one-way coupling with AVBP has been performed with success and the two-way coupling has been started.

* '''Sub-project 2: New features in YALES2 (A. Grenouilloux, S. Meynet, M. Bernard, R. Mercier):'''
The main objectives of this sub-project was to develop in YALES2 (i) anisotropic mesh adaptation and (ii) a new partitioning algorithm for a more performant mesh adaptation procedure. To allow anisotropic mesh adaptation a new metric definition based on a tensor at cells has been proposed. The new partitioning has been developed to create halos around bad quality cells and to ensure contiguity.

* '''Sub-project 3: Criteria based on statistical quantities for static mesh adaptation in LES (G. Balarac, N. Odier, A. Grenouilloux):'''
The main objective of this sub-project was to develop a strategy for automatic mesh convergence based on statistical quantities. The proposed strategy is independent of the flwo case and of the user. It is defined to guarantee that the energy balance of the overall system is independent of the mesh. This strategy combine criteria already proposed by Benard et al. (2015) and Daviller et al. (2017).

* '''Sub-project 4: Automated Mesh Convergence plugin re-integration (R. Mercier, J. Leparoux, A. Grenouilloux):'''
The main objective of this sub-project was to integrate the Automated Mesh Convergence (AMC) plugin developed by Safran Tech in YALES2 distribution. This was done with success during the workshop. Moreover, additional criteria were integrated. In particular, the y_plus criterion from Duprat law (A. Grenouilloux PhD) was considered to be able to control cells size in boundary layers.

* '''Sub-project 5: Dynamic mesh adaptation for DNS/LES of isolated vortices (L. Bricteux, G. Balarac):'''
The main objective of this sub-project was to develop dynamic mesh adaptation strategy for simulation of isolated vortices, and to compare with DNS on static mesh, or with vortex methods. A well docuimented test case of a 2D vortice has been considered. Criteria based on the Palinstrophy have been proposed with success, allowing to perform simulation with dynamic mesh adaptation having the same accuracy as reference methods.

* '''Sub-project 6: Dynamic mesh adaptation for non-statistically stationary turbulence (U. Vigny, L. Bricteux, Y. Dubief, P. Benard):'''
The main objective of this sub-project was to test dynamic mesh adaptation strategies for flow configurations where statistical quantities are unavailable (conversely to SP3), and where various vortices on a broad range of scales exist (conversely to SP5). Various quantities based on velocity gradient, Q criterion, or passive scalar have been tested. But no unified strategy has been proposed yet. A procedure has been initiated based on a multiobjective genetic algorithm (GA) to identify the optimum dynamic mesh adaptation parameters to minimize computational cost and maximize solution quality.

=== Multi-phase flows - M. Cailler, SAFRAN TECH and V. Moureau, CORIA ===
''Participants: G. Ghigliotti, G. Sahut, S. Pertant (LEGI), Y. Dubief (Vermont U.), S. Mendez (IMAG), R. Mercier, M. Cailler, J. Leparoux (Safran Tech), F. Pecquery, C. Merlin (ARIANE GROUP), V. Moureau, R. Janodet, I. Tsetoglou, P. Benez, Y. Atmani (CORIA)''

The modeling of two-phase flows has always been a tedious task because of the differences in thermo-physical properties between the fluids. While two-phase flow numerics based on interface capturing methods have reached maturity for simple thermodynamics, the focus in this field is now on how to deal with multi-physics. Most of the sub-projects of this event have addressed this need.

* '''Sub-project 1: Thermodynamics for two-phase flows (Y. Atmani, F. Pecquery, M. Cailler, C. Merlin, G. Sahut, S. Pertant, V. Moureau)'''
The main objective of this sub-project was to continue the development in YALES2 of the conservative transport of scalars in two-phase flows using a two-fluid approach. To this aim, new data structures for the "discontinuous scalars" have been derived in order to include various equations of state. The transport of the discontinuous scalars has also been augmented with dilatation. The calculation of surface tension has also been coupled to the scalars in order to start the modeling of Marangoni effects.

* '''Sub-project 2: Contact angle/triple line (S. Pertant, G. Sahut, G. Ghigliotti, C. Merlin, V. Moureau)'''
In this sub-project, the boiling solver of YALES2 has been coupled to the contact angle model of Wang & Desjardins 2018 based on the accurate conservative levelset framework. The discontinuous scalar transport has also been added to the boiling solver. With these new features, the solver has been used to perform the first simulation of nucleate boiling with dynamic mesh adaptation. The merging of the contact angle model into master has also progressed during the event.

* '''Sub-project 3: Heat-flux modeling for two-fluid conservative method (Y. Atmani, R. Janodet, M. Cailler, V. Moureau)'''
This sub-project aimed at improving the heat flux model used at the interface in the two-fluid scalar transport framework in YALES2. The work consisted in evaluating the heat flux at the interface instead in the volume. The interface is here materialized by the intersection of the level set iso-surface with the edges of the mesh. The flux is thus evaluated at this intersection and then extended in the volume where it is used to compute the various terms in the transport and reinitialization equations. These developments have been tested successfully for the transport of a 2D water droplet in hot air.

* '''Sub-project 4: Two-phase flows with polymers (Y. Dubief, S. Mendez, V. Moureau)'''
In this sub-project, the FENE-P model, which as been merged into the master branch of YALES2, has been revisited. While the stiff integration of the non-linear spring, which represents the polymer dynamics, is very efficient and accurate at maximum stretch, the Gibbs phenomenon occurs at zero-stretch and leads to negative values of the trace of the conformation tensor. A new form of the non-linear spring has been derived and tested which prevents the conformation tensor trace to become negative. This new model has been implemented and tested successfully for the flow behind a 2D cylinder.

=== Numerics - G. Lartigue, CORIA ===
''Participants: Ghislain LARTIGUE and Vincent MOUREAU (CORIA), Manuel BERNARD and Guillaume BALARAC (LEGI), Nicolas ODIER and Benjamin MARTIN (CERFACS)''

This project gathered four sub-projects related to Numerical Methods. Most of these activities are related to the use of high-order schemes presented in [1] in the context of Finite-Volumes Method.

* '''Sub-project 1 (N. Odier, B. Martin, G. Lartigue):''' The main objective of this sub-project was to implement a '''High-Order Finite-Volume method in the Cell-Vertex compressible code AVBP'''.

During the workshop, we focused on the improvement of the convective flux computation. Using the high-order reconstruction of [1], we were able to express the conservative variables as high-order polynomials within each nodal volume. In the case of the Euler equations, a specific scheme is required to compute the correct numerical flux crossing each edge. At the edge, the polynomials obtained for the two neighbouring nodes are not continuous and it hence corresponds to a Riemann problem. Two implementations of the numerical fluxes have been tested: a Roe solver and a Lax-Wendroff scheme. Both schemes achieve third order accuracy on regular and distorted triangular grids. The Roe scheme has also been tested on hybrid triangular/quadrilateral grids, with a resulting second order accuracy in accordance with the high-order reconstruction theory of [1].

* '''Sub-project 2 (M. Bernard, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to work on the use of '''High-Order Finite-Volume method to solve the Poisson Equation in the incompressible code YALES2'''.

In the context of projection method, a special attention needs to be paid to the accuracy of the coupling between pressure and velocity fields.
To achieve this goal, the keystone is to be able to solve efficiently the Poisson problem for the pressure.
During the workshop, we focused on resolution of a generic Poisson problem by use of conjugated gradient algorithm (CJ).
Idea was to use, at each iteration of the CG, the high-order Laplacian operator recently developed on the basis of high-order schemes [1].
This high-order Laplacian operator shows a better accuracy than the classical one used in YALES2 (SIMPLEX [3])
However, its usage during conjugated gradient algorithm does not improve the accuracy of the solution of the Poisson problem.
Further investigations are ongoing to evaluate the potential improvement on the correction of the velocity field with the pressure arising from the inversion of the high-order Laplacian operator.

* '''Sub-project 3 (G. Sahut, G. Balarac, G. Lartigue):''' The main objective of this sub-project was to implement a '''URANS method with a semi-implicit solver in YALES2'''.

In view of the development of a hybrid URANS/LES solver in YALES2, we have extended a Semi-Implicit method to a Full-Implicit method by adding a second time derivative in the incompressible Navier-Stokes equations. Within the classical projection method [4], an implicit scheme is used to compute the velocity predictor. The resulting linear system is inverted using the BiCGStab(2) linear solver [5].
The method has been validated on a periodic flow in a 2D channel with varying section, with a CFL number of 100, where an artificial forcing term is applied in the Navier-Stokes equations to move the fluid back and forth. A comparison with the Explicit method of YALES2 (ICS solver) at CFL = 0.9 shows that this new Full-Implicit method is able to recover the proper velocity profile at high CFL numbers such as CFL = 100. This very important result demonstrates the stability of the Full-Implicit method at high CFL numbers. Furthermore, a comparison with the Semi-Implicit method shows that, while the Semi-Implicit method is able to recover the proper velocity profile at CFL=5, it fails at CFL=100, contrary to the Full-Implicit method. This result shows the benefit of these developments, i.e., the ability to simulate unsteady flows accurately at high CFL numbers.

* '''Sub-project 4 (G. Lartigue, V. Moureau):''' The main objective of this sub-project was to '''improve the precision and robustness of the Laplacian Operator in YALES2'''.

There is two class of operators in YALES2: '''ROBUST''' (a.k.a. PAIR_BASED and IGNORE_SKEWNESS) and '''PRECISE''' (a.k.a. SIMPLEX). It has been shown that for operators with constant coefficients (as in ICS and VDS solvers), the '''PRECISE''' approach is unconditionally stable and must be used in all situations. However, in the SPS solver, the density variations across a pair of vertex can lead to a non-PSD operator. A major achievement of the workshop was to propose an hybrid operator that mixes both operators to achieve both precision and robustness. This operator will be implemented in a near future.

* '''Discussion (All):''' A two-hours discussion on Tuesday afternoon have been dedicated to the analysis of the paper [2]. This paper deals with an optimal way of mixing a robust low-order numerical scheme with high-order scheme. The major interest of this mixing technique is that it preserves the boundedness of the solution with a so-called convex-limiting. This is similar to WENO techniques but it relies on the resolution of the interface Riemann

[1] Manuel Bernard, Ghislain Lartigue, Guillaume Balarac, Vincent Moureau, Guillaume Puigt. '''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes'''. ''International Journal for Numerical Methods in Fluids'', Wiley, 2020, 92 (11), pp.1551-1583. [https://onlinelibrary.wiley.com/doi/10.1002/fld.4839] [https://hal.archives-ouvertes.fr/hal-02558814v2]

[2] Jean-Luc Guermond, Bojan Popov, Ignacio Tomas. '''Invariant domain preserving discretization-independent schemes and convex limiting for hyperbolic systems'''. ''Comput. Methods Appl. Mech. Engrg''. 347 (2019) 143–175. [https://www.math.tamu.edu/~guermond/PUBLICATIONS/guermond_popov_tomas_CMAME_2019.pdf]

[3] Ruben Specogna, Francesco Trevisan. '''A discrete geometric approach to solving time independent Schrödinger equation'''. '''Journal of Computational Physics''' 2011, 1370-1381. [https://www.sciencedirect.com/science/article/pii/S0021999110006091]

=== Turbulent flows - P. Bénard, CORIA ===
* turbulence injection
* wall modeling
* rotor modeling for wind or hydro turbines applications
* advanced post processing for unsteady turbulence

=== User experience - R. Mercier, SAFRAN TECH ===
''Participants: Julien Leparoux and Renaud Mercier, Safran Tech, Adrien Grenouilloux and Pierre Bénard, CORIA''

This project gathered three sub-projects related to user-experience and associated tools. Limited efforts could be made during the week because of an important implication of participants in the Mesh project especially.
* '''Sub-project 1 - Automated regression tests:''' The aim of this sub-project was to improve portability of the first version of y2_non_reg tool, demonstrate on test cases and improve reports and documentation. Nothing was achieved during the week but work will be definitely performed at Safran this year on this subject.
* '''Sub-project 2 - Statistics convergence tool:''' The aim of this sub-project was to demonstrate the interest of a recent tool developed by A. Grenouilloux and identify recommendations on SPS and VDS cases. Improvements of the tool and features evolutions were also targeted. During the week, a python script has been used to test convergence of QC2 criterion on a LEGI test case. The script is about to be pushed on master branch.
* '''Sub-project 3 - An “advisor” tool for the setup of large runs:''' The aim of this sub-project was to develop a tool to answer frequently asked questions on the choices of some YALES2 parameters such as dump partitionning. A first version of the tool was obtained during the week.

=== Fluid structure interaction - S. Mendez, IMAG ===
''Participants: Thomas Fabbri and Guillaume Balarac, LEGI, Barthélémy Thibaud and Simon Mendez, IMAG, Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA''

This project gathered three sub-projects related to fluid-structure interactions (FSI). Their common feature was the FSI solver from YALES2, which is based on a partitioned approach. The FSI solver couples an Arbitrary Lagrangian-Eulerian solver for predicting the fluid motion in a moving domain (FSI_ALE) and a solver for structural dynamics (FSI_SMS), which are both YALES2 solvers. The FSI solver has been initiated by Thomas Fabbri (LEGI, Grenoble) and the objectives of ECFD4 were to optimize it and generalize its use among several teams, by improving its performances, demonstrating its versatility and adding multiphysics effects. All the projects made interesting progree and will continue over the newt weeks/months.
* '''Sub-project 1 (Thomas Fabbri and Guillaume Balarac, LEGI):''' The aim of this sub-project was '''to decrease the time spent in computing the fluid grid deformation''', which is currently the most expensive part of the calculation. The strategy is to solve a deformation field on a coarse mesh and apply it to a fine mesh after interpolation. Many pieces exist in YALES2 related to such a task (using several grids, performing interpolations...), but they are currently not appropriate for this application. The work performed during the workshop consisted in identifying the different subroutines of interest and start coding the method. Many parts of the method are functional and the next step is to properly compbine them and test its efficiency.
* '''Sub-project 2 (Barthélémy Thibaud and Simon Mendez, IMAG):''' The aim of this sub-project was '''to validate the FSI solver in the case of a flexible valve''' bent by a pulsatile flow. A proper workflow (sequence of runs) has been defined during the week to be able to run this simulation and the first results are extremely promising, with already fair comparisons with the reference results from the literature. This workshop has also contributed in enhacing the experience of the solver at IMAG.
* '''Sub-project 3 (Likhitha Ramesh Reddy and Axelle Viré, TU Delft and Pierre Bénard, CORIA):''' The long-term aim of this sub-project is to perform simulations of the flow around floating wind turbines, which constitutes a huge challenge, as it gathers the difficulties of wind tubines flows, two-phase flows, and fluid-structure interactions between a fluid and a solid. During the workshop, the aim was '''to progress on two aspects: the use of the two-phase flow solver of YALES2, SPS, in a moving domain (coupling SPS and ALE) and the coupling with FSI'''. Both tasks were tackled: preliminary validation simulations were performed for the SPS-ALE solver, and the strategy to couple the SPS-ALE solver with the FSI has been clearly identified within the group.
* '''Common work:''' TU Delft (Sub-project 3) needs to perform FSI without deformation of the structure, so that the coupling with the SMS solver may not be indispensable. Tests were performed to study the ability of the SMS to work in a regime of very stiff material to mimic rigid bodies, and first tests were very convincing. In the future however, it is planned to implement a rigid-body motion solver in YALES2 as an alternative to SMS. This task gathers the four teams of the project and is a clear shared objective of the next months.
* '''Bugs and cleaning:''' minor bugs were identified in the FSI solver, mostly related to options rarely used. There were corrected and pushed in the YALES2 gitlab.
* '''Documentation:''' the information shared between participants for the use and understanding of the SMS and FSI solvers has been directly gathered in the YALES2 wiki.

=== GENCI Hackathon - G. Staffelbach, CERFACS ===

Participants : V. Moureau (CORIA), P. Bégou (LEGI), J. Legaux, G. Staffelbach (CERFACS), L. Stuber, F. Courteille (NVIDIA), T. Braconnier, P.E Bernard (HPE).

GPU acceleration is the keystone towards exascale computing as evidenced by the top500 where two thirds of the top50 systems are now accelerated. Within this workshop the objective was to reevaluate the performance of both AVBP and YALES2 following their initial port under a contrat de progrés between GENCI and HPE with the support of IDRIS conducted in 2019. Then update as much as possible the codes to todays versions, assess new porting and optimisation possibilities and carry them out when possible.

'''YALES2'''
The YALES2 solver has evolved immensely since the 2019 port and most of the time was spent merging and updated the code to todays standards. Two updated branches with the current source code have been released idris/openacc_node2pair et idris/openacc_pair2node and profiling and optimisation tools have been tested on CORIA and LEGI platforms.
In parallel, using the CVODE GPU-enabled library to accelerate the chemistry solver in YALES2 was investigated. This proved more complex than anticipated as the library did not build as is with the latest release of the NVIDIA SDK. This issue was promptly solved with the help of NVIDIA. Coupling YALES2 with the accelerated library seems to require more extensive knowledge in OpenACC and CUDA, the team is highly motivated to pursue this train of though and will probably participate to the IDRIS hackathon initiative in May 2020 to continue this effort.

'''AVBP'''
Efforts to port AVBP to GPU have continued through an second grand challenge on the JEANZAY system targeting the port of a complex industrial type combustion chamber (DGENCC). In preparation for this workshop, the new models required for the DGENCC simulation were ported to GPU and performance analysis was undertaken. A new branch WIP/GC_JZ2 is currently available allowing for the accelerated simulation of this type of workflow.
Under the guidance of NVIDIA and HPE, optimisation venues have been identified:
* removal of extended temporary arrays.
* remplacement of implicit vector assignements.
* Collapsable compute driven loops.

Integrating this efforts in some of the kernels has yieled a 4.2 acceleration between a full cpu compute node with 40 cascade lake cores and a the accelerated counter part using 4 NVIDIA V100 GPUs. Further more the case has been strong scaling tested up to 1024 gpus with excellent performance.

File:Ecfd3 final project16.pdf

2020-02-01T15:17:38Z

Sahut:

Ecfd:ecfd 3rd edition

2020-01-31T08:31:10Z

Sahut: /* Project #16: Development of a RANS solver in YALES2 */

ECFD workshop, 3rd edition, 2020

== Sponsors ==

[[File:ecfd3_sponsors.png|center|frameless|800px]]

== Participants ==

[[File:ecfd3_participants.png|center|frameless|1000px]]

== Flyer ==

* [[media:ecfd3_flyer.pdf | Flyer]]

== Presentations ==

* [[media:ecfd3_intro.pdf | Introduction workshop]]
* [[media:ecfd3_intro_genci.pdf | Introduction GENCI]]
* [[media:ecfd3_avbp_roadmap_HPC.pdf | Roadmap AVBP (HPC)]]
* [[media:ecfd3_yales2_roadmap.pdf | Roadmap YALES2]]

== Project achievements ==

=== Project #1: Hackathon GENCI/ATOS/AMD/CERFACS on AVBP ===

''C. Piechurski (GENCI), S. Jauré (ATOS), B. Pajot (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G. Staffelbach (CERFACS), J. Legaux (CERFACS)''

We ported the AVBP solver to the AMD Rome system available at GENCI -TGCC ( IRENE Joliot Curie).
Characterisation of the application on the architecture showed a 1/3 performance dependency to bandwidth and 2/3 to compute.
Strong scaling performance up to 130k cores was measured with openmpi and provided an acceleration of 75% without optimisations.
Weak scaling up to 32k MPI ranks suggests that decimation of the processes by a factor 2 improves computational efficiency by up to 30%.
This suggests a trade off between mpi imbalance and decimation is possible if imbalance is higher than 30% to improve time to solution.

Currently Openmpi offers the best perfofrmance, intelmpi is still a bit unstable.

During the Hackathon we also introduced colour based cache blocking using ColPack in the code in order to use OpenMP without critical sections.
On a 2x18 core Skylake processor the new implementation offered similar speedup using full threading versus full MPI with the best trade off being 4 MPI and 9 threads per MPI.
On AMD Rome, Full threading did not offer much acceleration and needs to be inversigated but 8 MPI and 16 threads per MPI seem quite promising.

[[media:ecfd3_final_project1.pdf | Final presentation of project #1]]

=== Project #2: Hackathon GENCI/ATOS/AMD/CORIA on YALES2 ===
''C. Piechurski (GENCI), S. Jauré (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G.Lartigue (CORIA), F. Gava (CORIA), P. Begou (LEGI)''

[[media:ecfd3_final_project2.pdf | Final presentation of project #2]]

=== Project #3: Implementation of a secondary atomization model in YALES2 ===

''C. G. Guillamon (Safran Tech), L .Voivenel (Safran Tech), R. Mercier (Safran Tech)''

In Lagrangian simulations, droplets are transported in an eulerian mesh following ballistic motion. For non-reactive environments, droplets might break due to the aerodynamic interaction. In this work, we implement in YALES2 a secondary atomization model known as Taylor-Analogy Breakup (TAB). This model considers an analogy between a droplet and a second-order mechanical system, hence making possible to determine the breakup behavious by means of Newton's second law.

Another model, the stochastic breakup model by Gorokhovski, is also suggested for future work and will be implemented in YALES2.

[[media:ecfd3_final_project3.pdf | Final presentation of project #3]]

=== Project #4: Application to combustion and lubrication applications ===

[[media:ecfd3_final_project4.pdf | Final presentation of project #4]]

=== Project #5: Jet-in-crossflow par une méthode d’interface diffuse ===

[[media:ecfd3_final_project5.pdf | Final presentation of project #5]]

=== Project #6: Accurate numerical predicti􏴇on of vorti􏴇cal flows using AMR ===

[[media:ecfd3_final_project6.pdf | Final presentation of project #6]]

=== Project #7: Modélisation de parois pour la simulation des grandes échelles ===

[[media:ecfd3_final_project7.pdf | Final presentation of project #7]]

=== Project #8: Implémentation du calcul de la distance à une interface liquide-gaz proche d’une paroi sur maillage non structuré 3D avec YALES2 ===

[[media:ecfd3_final_project8.pdf | Final presentation of project #8]]

=== Project #9: Remeshed particle method at high Schmidt and Reynolds number ===

''S. Santoso (LJK), J.-B. Lagaert (Math Orsay), G.Balarac (LEGI)''

We study the advection of a scalar function in turbulent flows with a multimesh method. The finite volume method is used to solve Navier-Stokes equations on an unstructured mesh (YALES2). The advection equation is solved with remeshed particle method on a cartesian mesh. In the context of parallel computing, we face a very unbalanced problem since a large number of particles are created in a very fine meshed zone. Our strategy to load-balance the problem is to give a weight to every element group which is equal to the density of particle.

[[media:ecfd3_final_project9.pdf | Final presentation of project #9]]

=== Project #10: Adaptive mesh refinement for turbulent premixed combustion ===
''W. Agostinelli, O. Dounia, , T. Jaravel, O. Vermorel

The objective of the project was to evaluate the potential of adaptive mesh refinement (AMR) for premixed combustion in unsteady systems. Three target cases were identified: a semi-vented deflagration with laminar to turbulent transition, a planar detonation wave, and a bluff-body stabilized burner subjected to thermoacoustic oscillations. The simulations were performed with AVBP and coupled to the AMR implementation of YALES2. Several metrics and remeshing criterions were developed to identify and correctly resolve both the combustion wave front and the turbulent flow. The comparison of numerical results with reference simulations showed that the main features of the physics could be recovered with a significant speed-up in term of computational cost.

[[media:ecfd3_final_project10.pdf | Final presentation of project #10]]

=== Project #11: Multiphysics coupling for wind turbine wake modeling ===

''F.Houtin-Mongrolle (CORIA), B. Duboc (SGRE), P. Benard (CORIA)''

The goal of this project was to evaluate the coupling of YALES2 (flow solver) and BHawC(Aero-Servo-Elastic solver).

[[media:ecfd3_final_project11.pdf | Final presentation of project #11]]

=== Project #12: Stability of a semi-implicit compressible cavitation solver ===

[[media:ecfd3_final_project12.pdf | Final presentation of project #12]]

=== Project #13: DNS of droplet dynamics and evaporation : comparison between structured and unstructured solvers ===

[[media:ecfd3_final_project13.pdf | Final presentation of project #13]]

=== Project #14: Méthode d'ordre élevé ===
''M. Bernard (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project14.pdf | Final presentation of project #14]]

=== Project #15: Utilisation d’éléments finis du second ordre dans le SMS ===
''T. Fabbri (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project15.pdf | Final presentation of project #15]]

=== Project #16: Development of a RANS solver in YALES2 ===
''G. Sahut (LEGI), G. Balarac (LEGI), V. Moureau (CORIA), G. Lartigue (CORIA), P. Bénard (CORIA), A. Grenouilloux (CORIA)''

While the accuracy of LES usually approaches the one of DNS, LES are still too time-consuming for daily use in industrial applications. In this context, we started the development of a RANS solver in YALES2. We are first only interested in the steady state of the solution. In order to remove the CFL constraint, we developed, implemented and validated an implicit projection method for the resolution of the Navier-Stokes equations without turbulence models. The method is based on the implicitation of the velocity predictor ; the Poisson equation and the correction step of the velocity are then solved and applied as in the explicit incompressible solver. We validated the method on a stationary 2D Poiseuille flow with periodic boundary conditions: the simulation runs fine for CFL and Fourier numbers which are inaccessible with the explicit incompressible solver. The advection-diffusion equation for scalars has also been implicited and will be used to add turbulence models to the new implicit incompressible solver developped during this Workshop. More complex boundary conditions will also be addressed in a near future.

[[media:ecfd3_final_project16.pdf | Final presentation of project #16]]

=== Project #17: COUPLING OF A FLUID PLASMA SOLVER WITH A LAGRANGIAN SOLVER FOR THE MODELING OF DUSTY ===

[[media:ecfd3_final_project17.pdf | Final presentation of project #17]]

=== Project #18: L’Evaporo O Maıtre ===

[[media:ecfd3_final_project18.pdf | Final presentation of project #18]]

=== Project #19: The Clone Wars ===

[[media:ecfd3_final_project19.pdf | Final presentation of project #19]]

=== Project #20: Stiff complex fluid simulation with YALES2 ===
''Sam Whitmore, Yves Dubief, M2CE, University of Vermont''

The objective was to simulate (1) ionized gases and (2) polymer solutions in flows using YALES2. Both problems are challenging owing to their stiff thermodynamics (1) or polymer dynamics (2). Significant gains were achieved in the implementation of the respective models thanks to the stiff integrator library CVODE. The plasma flow demonstrated an increase in time step of two orders of magnitude compared to previous implementation of the plasma chemistry in the variable density solver. Polymer models are notoriously prone to numerical instability. Again the use of CVODE showed equivalent if not superior stability of the solution at a fraction of the cost of commonly employed algorithms designed to address the stiffness of the problem.

[[media:ecfd3_final_project20.pdf | Final presentation of project #20]]

=== Project #21: AVBP Dense Gases ===

[[media:ecfd3_final_project21.pdf | Final presentation of project #21]]

=== Project #22: Numerical prediction of wind turbine wakes using AMR ===

[[media:ecfd3_final_project22.pdf | Final presentation of project #22]]

Ecfd:ecfd 3rd edition

2020-01-31T08:26:25Z

Sahut: /* Project #16: Development of a RANS solver in YALES2 */

ECFD workshop, 3rd edition, 2020

== Sponsors ==

[[File:ecfd3_sponsors.png|center|frameless|800px]]

== Participants ==

[[File:ecfd3_participants.png|center|frameless|1000px]]

== Flyer ==

* [[media:ecfd3_flyer.pdf | Flyer]]

== Presentations ==

* [[media:ecfd3_intro.pdf | Introduction workshop]]
* [[media:ecfd3_intro_genci.pdf | Introduction GENCI]]
* [[media:ecfd3_avbp_roadmap_HPC.pdf | Roadmap AVBP (HPC)]]
* [[media:ecfd3_yales2_roadmap.pdf | Roadmap YALES2]]

== Project achievements ==

=== Project #1: Hackathon GENCI/ATOS/AMD/CERFACS on AVBP ===

''C. Piechurski (GENCI), S. Jauré (ATOS), B. Pajot (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G. Staffelbach (CERFACS), J. Legaux (CERFACS)''

We ported the AVBP solver to the AMD Rome system available at GENCI -TGCC ( IRENE Joliot Curie).
Characterisation of the application on the architecture showed a 1/3 performance dependency to bandwidth and 2/3 to compute.
Strong scaling performance up to 130k cores was measured with openmpi and provided an acceleration of 75% without optimisations.
Weak scaling up to 32k MPI ranks suggests that decimation of the processes by a factor 2 improves computational efficiency by up to 30%.
This suggests a trade off between mpi imbalance and decimation is possible if imbalance is higher than 30% to improve time to solution.

Currently Openmpi offers the best perfofrmance, intelmpi is still a bit unstable.

During the Hackathon we also introduced colour based cache blocking using ColPack in the code in order to use OpenMP without critical sections.
On a 2x18 core Skylake processor the new implementation offered similar speedup using full threading versus full MPI with the best trade off being 4 MPI and 9 threads per MPI.
On AMD Rome, Full threading did not offer much acceleration and needs to be inversigated but 8 MPI and 16 threads per MPI seem quite promising.

[[media:ecfd3_final_project1.pdf | Final presentation of project #1]]

=== Project #2: Hackathon GENCI/ATOS/AMD/CORIA on YALES2 ===
''C. Piechurski (GENCI), S. Jauré (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G.Lartigue (CORIA), F. Gava (CORIA), P. Begou (LEGI)''

[[media:ecfd3_final_project2.pdf | Final presentation of project #2]]

=== Project #3: Implementation of a secondary atomization model in YALES2 ===

''C. G. Guillamon (Safran Tech), L .Voivenel (Safran Tech), R. Mercier (Safran Tech)''

In Lagrangian simulations, droplets are transported in an eulerian mesh following ballistic motion. For non-reactive environments, droplets might break due to the aerodynamic interaction. In this work, we implement in YALES2 a secondary atomization model known as Taylor-Analogy Breakup (TAB). This model considers an analogy between a droplet and a second-order mechanical system, hence making possible to determine the breakup behavious by means of Newton's second law.

Another model, the stochastic breakup model by Gorokhovski, is also suggested for future work and will be implemented in YALES2.

[[media:ecfd3_final_project3.pdf | Final presentation of project #3]]

=== Project #4: Application to combustion and lubrication applications ===

[[media:ecfd3_final_project4.pdf | Final presentation of project #4]]

=== Project #5: Jet-in-crossflow par une méthode d’interface diffuse ===

[[media:ecfd3_final_project5.pdf | Final presentation of project #5]]

=== Project #6: Accurate numerical predicti􏴇on of vorti􏴇cal flows using AMR ===

[[media:ecfd3_final_project6.pdf | Final presentation of project #6]]

=== Project #7: Modélisation de parois pour la simulation des grandes échelles ===

[[media:ecfd3_final_project7.pdf | Final presentation of project #7]]

=== Project #8: Implémentation du calcul de la distance à une interface liquide-gaz proche d’une paroi sur maillage non structuré 3D avec YALES2 ===

[[media:ecfd3_final_project8.pdf | Final presentation of project #8]]

=== Project #9: Remeshed particle method at high Schmidt and Reynolds number ===

''S. Santoso (LJK), J.-B. Lagaert (Math Orsay), G.Balarac (LEGI)''

We study the advection of a scalar function in turbulent flows with a multimesh method. The finite volume method is used to solve Navier-Stokes equations on an unstructured mesh (YALES2). The advection equation is solved with remeshed particle method on a cartesian mesh. In the context of parallel computing, we face a very unbalanced problem since a large number of particles are created in a very fine meshed zone. Our strategy to load-balance the problem is to give a weight to every element group which is equal to the density of particle.

[[media:ecfd3_final_project9.pdf | Final presentation of project #9]]

=== Project #10: Adaptive mesh refinement for turbulent premixed combustion ===
''W. Agostinelli, O. Dounia, , T. Jaravel, O. Vermorel

The objective of the project was to evaluate the potential of adaptive mesh refinement (AMR) for premixed combustion in unsteady systems. Three target cases were identified: a semi-vented deflagration with laminar to turbulent transition, a planar detonation wave, and a bluff-body stabilized burner subjected to thermoacoustic oscillations. The simulations were performed with AVBP and coupled to the AMR implementation of YALES2. Several metrics and remeshing criterions were developed to identify and correctly resolve both the combustion wave front and the turbulent flow. The comparison of numerical results with reference simulations showed that the main features of the physics could be recovered with a significant speed-up in term of computational cost.

[[media:ecfd3_final_project10.pdf | Final presentation of project #10]]

=== Project #11: Multiphysics coupling for wind turbine wake modeling ===

''F.Houtin-Mongrolle (CORIA), B. Duboc (SGRE), P. Benard (CORIA)''

The goal of this project was to evaluate the coupling of YALES2 (flow solver) and BHawC(Aero-Servo-Elastic solver).

[[media:ecfd3_final_project11.pdf | Final presentation of project #11]]

=== Project #12: Stability of a semi-implicit compressible cavitation solver ===

[[media:ecfd3_final_project12.pdf | Final presentation of project #12]]

=== Project #13: DNS of droplet dynamics and evaporation : comparison between structured and unstructured solvers ===

[[media:ecfd3_final_project13.pdf | Final presentation of project #13]]

=== Project #14: Méthode d'ordre élevé ===
''M. Bernard (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project14.pdf | Final presentation of project #14]]

=== Project #15: Utilisation d’éléments finis du second ordre dans le SMS ===
''T. Fabbri (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project15.pdf | Final presentation of project #15]]

=== Project #16: Development of a RANS solver in YALES2 ===
''G. Sahut (LEGI), G. Balarac (LEGI), V. Moureau (CORIA), G. Lartigue (CORIA), P. Bénard (CORIA), A. Grenouilloux (CORIA)''

While the accuracy of LES usually approaches the one of DNS, LES are still too time-consuming for daily use in industrial applications. In this context, we started the development of a RANS solver in YALES2. We are first only interested in the steady state of the solution. In order to remove the CFL constraint, we developed, implemented and validated an implicit projection method for the resolution of the Navier-Stokes equations without turbulence models. The method is based on the implicitation of the velocity predictor ; the Poisson equation and the correction step of the velocity are then solved and applied as in the classical incompressible solver. We validated the method on a stationary 2D Poiseuille flow: the simulation runs fine for CFL and Fourier numbers which are inaccessible with the classical incompressible solver.

[[media:ecfd3_final_project16.pdf | Final presentation of project #16]]

=== Project #17: COUPLING OF A FLUID PLASMA SOLVER WITH A LAGRANGIAN SOLVER FOR THE MODELING OF DUSTY ===

[[media:ecfd3_final_project17.pdf | Final presentation of project #17]]

=== Project #18: L’Evaporo O Maıtre ===

[[media:ecfd3_final_project18.pdf | Final presentation of project #18]]

=== Project #19: The Clone Wars ===

[[media:ecfd3_final_project19.pdf | Final presentation of project #19]]

=== Project #20: Stiff complex fluid simulation with YALES2 ===
''Sam Whitmore, Yves Dubief, M2CE, University of Vermont''

The objective was to simulate (1) ionized gases and (2) polymer solutions in flows using YALES2. Both problems are challenging owing to their stiff thermodynamics (1) or polymer dynamics (2). Significant gains were achieved in the implementation of the respective models thanks to the stiff integrator library CVODE. The plasma flow demonstrated an increase in time step of two orders of magnitude compared to previous implementation of the plasma chemistry in the variable density solver. Polymer models are notoriously prone to numerical instability. Again the use of CVODE showed equivalent if not superior stability of the solution at a fraction of the cost of commonly employed algorithms designed to address the stiffness of the problem.

[[media:ecfd3_final_project20.pdf | Final presentation of project #20]]

=== Project #21: AVBP Dense Gases ===

[[media:ecfd3_final_project21.pdf | Final presentation of project #21]]

=== Project #22: Numerical prediction of wind turbine wakes using AMR ===

[[media:ecfd3_final_project22.pdf | Final presentation of project #22]]

Ecfd:ecfd 3rd edition

2020-01-31T08:24:14Z

Sahut: /* Project #16: Development of a RANS solver in YALES2 */

ECFD workshop, 3rd edition, 2020

== Sponsors ==

[[File:ecfd3_sponsors.png|center|frameless|800px]]

== Participants ==

[[File:ecfd3_participants.png|center|frameless|1000px]]

== Flyer ==

* [[media:ecfd3_flyer.pdf | Flyer]]

== Presentations ==

* [[media:ecfd3_intro.pdf | Introduction workshop]]
* [[media:ecfd3_intro_genci.pdf | Introduction GENCI]]
* [[media:ecfd3_avbp_roadmap_HPC.pdf | Roadmap AVBP (HPC)]]
* [[media:ecfd3_yales2_roadmap.pdf | Roadmap YALES2]]

== Project achievements ==

=== Project #1: Hackathon GENCI/ATOS/AMD/CERFACS on AVBP ===

''C. Piechurski (GENCI), S. Jauré (ATOS), B. Pajot (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G. Staffelbach (CERFACS), J. Legaux (CERFACS)''

We ported the AVBP solver to the AMD Rome system available at GENCI -TGCC ( IRENE Joliot Curie).
Characterisation of the application on the architecture showed a 1/3 performance dependency to bandwidth and 2/3 to compute.
Strong scaling performance up to 130k cores was measured with openmpi and provided an acceleration of 75% without optimisations.
Weak scaling up to 32k MPI ranks suggests that decimation of the processes by a factor 2 improves computational efficiency by up to 30%.
This suggests a trade off between mpi imbalance and decimation is possible if imbalance is higher than 30% to improve time to solution.

Currently Openmpi offers the best perfofrmance, intelmpi is still a bit unstable.

During the Hackathon we also introduced colour based cache blocking using ColPack in the code in order to use OpenMP without critical sections.
On a 2x18 core Skylake processor the new implementation offered similar speedup using full threading versus full MPI with the best trade off being 4 MPI and 9 threads per MPI.
On AMD Rome, Full threading did not offer much acceleration and needs to be inversigated but 8 MPI and 16 threads per MPI seem quite promising.

[[media:ecfd3_final_project1.pdf | Final presentation of project #1]]

=== Project #2: Hackathon GENCI/ATOS/AMD/CORIA on YALES2 ===
''C. Piechurski (GENCI), S. Jauré (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G.Lartigue (CORIA), F. Gava (CORIA), P. Begou (LEGI)''

[[media:ecfd3_final_project2.pdf | Final presentation of project #2]]

=== Project #3: Implementation of a secondary atomization model in YALES2 ===

[[media:ecfd3_final_project3.pdf | Final presentation of project #3]]

''C. G. Guillamon (Safran Tech), L .Voivenel (Safran Tech), R. Mercier (Safran Tech)''

In Lagrangian simulations, droplets are transported in an eulerian mesh following ballistic motion. For non-reactive environments, droplets might break due to the aerodynamic interaction. In this work, we implement in YALES2 a secondary atomization model known as Taylor-Analogy Breakup (TAB). This model considers an analogy between a droplet and a second-order mechanical system, hence making possible to determine the breakup behavious by means of Newton's second law.

=== Project #4: Application to combustion and lubrication applications ===

[[media:ecfd3_final_project4.pdf | Final presentation of project #4]]

=== Project #5: Jet-in-crossflow par une méthode d’interface diffuse ===

[[media:ecfd3_final_project5.pdf | Final presentation of project #5]]

=== Project #6: Accurate numerical predicti􏴇on of vorti􏴇cal flows using AMR ===

[[media:ecfd3_final_project6.pdf | Final presentation of project #6]]

=== Project #7: Modélisation de parois pour la simulation des grandes échelles ===

[[media:ecfd3_final_project7.pdf | Final presentation of project #7]]

=== Project #8: Implémentation du calcul de la distance à une interface liquide-gaz proche d’une paroi sur maillage non structuré 3D avec YALES2 ===

[[media:ecfd3_final_project8.pdf | Final presentation of project #8]]

=== Project #9: Remeshed particle method at high Schmidt and Reynolds number ===

''S. Santoso (LJK), J.-B. Lagaert (Math Orsay), G.Balarac (LEGI)''

We study the advection of a scalar function in turbulent flows with a multimesh method. The finite volume method is used to solve Navier-Stokes equations on an unstructured mesh (YALES2). The advection equation is solved with remeshed particle method on a cartesian mesh. In the context of parallel computing, we face a very unbalanced problem since a large number of particles are created in a very fine meshed zone. Our strategy to load-balance the problem is to give a weight to every element group which is equal to the density of particle.

[[media:ecfd3_final_project9.pdf | Final presentation of project #9]]

=== Project #10: Adaptive mesh refinement for turbulent premixed combustion ===
''W. Agostinelli, O. Dounia, , T. Jaravel, O. Vermorel

The objective of the project was to evaluate the potential of adaptive mesh refinement (AMR) for premixed combustion in unsteady systems. Three target cases were identified: a semi-vented deflagration with laminar to turbulent transition, a planar detonation wave, and a bluff-body stabilized burner subjected to thermoacoustic oscillations. The simulations were performed with AVBP and coupled to the AMR implementation of YALES2. Several metrics and remeshing criterions were developed to identify and correctly resolve both the combustion wave front and the turbulent flow. The comparison of numerical results with reference simulations showed that the main features of the physics could be recovered with a significant speed-up in term of computational cost.

[[media:ecfd3_final_project10.pdf | Final presentation of project #10]]

=== Project #11: Multiphysics coupling for wind turbine wake modeling ===

''F.Houtin-Mongrolle (CORIA), B. Duboc (SGRE), P. Benard (CORIA)''

The goal of this project was to evaluate the coupling of YALES2 (flow solver) and BHawC(Aero-Servo-Elastic solver).

[[media:ecfd3_final_project11.pdf | Final presentation of project #11]]

=== Project #12: Stability of a semi-implicit compressible cavitation solver ===

[[media:ecfd3_final_project12.pdf | Final presentation of project #12]]

=== Project #13: DNS of droplet dynamics and evaporation : comparison between structured and unstructured solvers ===

[[media:ecfd3_final_project13.pdf | Final presentation of project #13]]

=== Project #14: Méthode d'ordre élevé ===
''M. Bernard (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project14.pdf | Final presentation of project #14]]

=== Project #15: Utilisation d’éléments finis du second ordre dans le SMS ===
''T. Fabbri (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)''

[[media:ecfd3_final_project15.pdf | Final presentation of project #15]]

=== Project #16: Development of a RANS solver in YALES2 ===
''G. Sahut (LEGI), G. Balarac (LEGI), V. Moureau (CORIA), G. Lartigue (CORIA), P. Bénard (CORIA), A. Grenouilloux (CORIA)''

While the accuracy of LES usually approaches the one of DNS, LES are still too time-consuming for daily use in industrial applications. In this context, we started the development of a RANS solver in YALES2. We are first only interested in the steady state of the solution. In order to remove the CFL constraint, we developed, implemented and validated an implicit projection method for the resolution of the Navier-Stokes equations. The method is based on the implicitation of the velocity predictor ; the Poisson equation and the correction step of the velocity are then solved and applied as in the classical incompressible solver. We validated the method on a stationary 2D Poiseuille flow: the simulation runs fine for CFL and Fourier numbers which are inaccessible with the classical incompressible solver.

[[media:ecfd3_final_project16.pdf | Final presentation of project #16]]

=== Project #17: COUPLING OF A FLUID PLASMA SOLVER WITH A LAGRANGIAN SOLVER FOR THE MODELING OF DUSTY ===

[[media:ecfd3_final_project17.pdf | Final presentation of project #17]]

=== Project #18: L’Evaporo O Maıtre ===

[[media:ecfd3_final_project18.pdf | Final presentation of project #18]]

=== Project #19: The Clone Wars ===

[[media:ecfd3_final_project19.pdf | Final presentation of project #19]]

=== Project #20: Stiff complex fluid simulation with YALES2 ===
''Sam Whitmore, Yves Dubief, M2CE, University of Vermont''

The objective was to simulate (1) ionized gases and (2) polymer solutions in flows using YALES2. Both problems are challenging owing to their stiff thermodynamics (1) or polymer dynamics (2). Significant gains were achieved in the implementation of the respective models thanks to the stiff integrator library CVODE. The plasma flow demonstrated an increase in time step of two orders of magnitude compared to previous implementation of the plasma chemistry in the variable density solver. Polymer models are notoriously prone to numerical instability. Again the use of CVODE showed equivalent if not superior stability of the solution at a fraction of the cost of commonly employed algorithms designed to address the stiffness of the problem.

[[media:ecfd3_final_project20.pdf | Final presentation of project #20]]

=== Project #21: AVBP Dense Gases ===

[[media:ecfd3_final_project21.pdf | Final presentation of project #21]]

=== Project #22: Numerical prediction of wind turbine wakes using AMR ===

[[media:ecfd3_final_project22.pdf | Final presentation of project #22]]