Extreme CFD workshop - User contributions [en]

Ecfd:ecfd 8th edition

2025-02-10T13:50:07Z

Bioche: /* User Experience & Data - L. Korzeczek, GDTECH */

{{DISPLAYTITLE: ECFD workshop, 8th edition, 2025}}

== Description ==

* Event from '''27th of January to 7th of February 2025'''
* Location: [https://www.sport-normandie.fr/le-centre/le-site-de-houlgate Centre Sportif de Normandie], Houlgate, near Caen (14)
* Two types of sessions:
** common technical presentations: roadmaps, specific points
** mini-workshops. Potential workshops are listed below
* Free of charge
* Participants from academics, HPC center/experts and industry are welcome
* The number of participants is limited to 68.

* Objectives
** Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
** Identify the technological barriers of exaflopic CFD via numerical experiments
** Identify industrial needs and challenges in high-performance computing
** Propose action plans to add to the development roadmaps of the CFD codes
* Organizers
** Guillaume Balarac (LEGI), Simon Mendez (IMAG), Pierre Bénard, Vincent Moureau, Léa Voivenel (CORIA).

[[File:ecfd8.png|600px|link=https://ecfd.coria-cfd.fr/index.php/Ecfd:ecfd_8th_edition]]

[[File:Acknowledgments_ecfd8.png|text-bottom|600px]]

== News ==

* 23/10/2024: First announcement of the '''8th Extreme CFD Workshop & Hackathon''' !
* 22/11/2024: Deadline to submit your project

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

To come...

== Projects ==

=== Hackathon GENCI - P. Begou, LEGI ===
This ECFD8 GENCI Hackathon was a rich event, involving 4 differents CFD codes (AVBP, ParaDIGM, SONICS and YALES2) using various paradigms (C++/cuda/hip, Fortran/OpenMP/OpenACC) with several SDKs (AMD, Cray/HPE, Nvidia, Gnu) on a large range of GPU architectures (Nvidia A100, GH100, AMD instinct Mi210, Mi250, Mi300). This two-week event benefited from a high level support from three HPC mentors, two on-site from AMD (J. Noudohouenou and A. Tsetoglou) and one remote from CINES (M. Boudaoud).

==== H1 - ParaDIGM and SONICS on GPU, B. Maugars, G. Staffelbach, R.Cazalbou and B. Michel (ONERA)====

==== H2 - AVBP GPU offloading based on OpenMP, M.Ghenai, L. Legaux and A. Dauptain (CERFACS) ====

This hackathon provided a valuable opportunity to work on GPU offloading for AVBP. In the past, significant efforts were made to offload the entire AVBP code to GPUs. OpenACC was the primary strategy chosen, mainly due to access to NVIDIA's support, along with the availability of both software and hardware. This approach achieved good scalability performance.
Recently, with the deployment of new supercomputers like ADASTRA at CINES, some issues have emerged when running AVBP on AMD GPUs, including both MI250 and MI300. The closed-source nature of the Cray environment has also prevented CERFACS from deploying AVBP on local MI210 GPUs.
This hackathon was a great opportunity to address these challenges by exploring a new approach using OpenMP. An automatic translation tool was used to convert approximately 2,700 OpenACC directives to OpenMP, with each directive manually verified and fine-tuned afterward. AVBP with OpenMP had already been tested on NVIDIA GPUs, and during this hackathon, the focus was on extending support to AMD GPUs.
Two compilers were used: Cray and the newly released AFAR from AMD. With the support of AMD and CINES, a working environment for compiling AVBP was set up, and performance-related issues were identified. Additionally, two mini-apps were used for benchmarking. One of them achieved a 2.5× speedup when compiled with AFAR compared to Cray.
The next steps involve adapting the code to address necessary modifications, such as fixing issues related to Fortran indirections, and continuing performance evaluations with mini-apps. Further comparisons will be conducted using both compilers against results obtained with NVIDIA’s NVHPC.

==== H3 - YALES2 GPU from OpenACC to OpenMP, P. Bégou (LEGI), V. Moureau, G. Lartigue (CORIA) and R. Dubois (IMAG) ====
This Hackathon focuses on running Yales2 code on AMD Instinct Mi250 and Mi300 GPUs of the Adastra supercomputer (CINES).
Previously, a first solver in the Yales2 CFD code was successfully ported on the GPU accelerators of the Jean-Zay supercomputer (IDRIS) using Nvidia SDK but difficulties remain on Adastra AMD GPUs, mainly related to the available development tools. High compilation time and the impossibility to use debug flags at compile time as soon as OpenACC is enabled are a real challenge when tracking errors. The current project is to evaluate a freshly deployed version (at the begining of the workshop) of the AMD Fortran compiler. This requires moving to OpenMP paradigm, starting from scratch since the OpenACC branch has largely diverged from the master one while tracking spurious remaining bugs.
If the AMD compiler is able to build the cpu version of Yales2 "out of the box" (wich is not the case for Cray Fortran), the compilation time for each file is significantly higher. However, setting up a 2 stages dynamic compilation process allows for high parallelism that is not possible with Cray Fortran 18 and the library build time drops from nearly 2 hours (Cray Fortran 18) to 17 minutes (Amd Fortran compiler).
Large kernels have been ported from OpenACC to OpenMP, raising some difficulties when offloading intrinsics functions or using strutures attributes in kernels loops. These limitations were also known in the previous OpenACC work. The goal was mainly to check the correctness of the results. The offloading of the complex data structure of Yales2 code was then investigated. Here again some limitations of the "young" compiler were discovered and workarounds were implemented. Several reproducers were built during this ECFD8 and provided to developpers by the 2 on-site AMD engineers.
Preliminary tests on micro-applications show good performances of the generated binaries proving that this compiler could be a serious alternative on AMD GPUs and the goal is now to focus on this SDK in an OpenMP strategy while checking the portablility of this new implementation in Nvidia, Cray/HPE (and Gnu ?) environments.

=== Mesh adaptation - A. Grenouilloux, ONERA & G. Balarac, LEGI ===

=== Numerics - M. Bernard, LEGI & G. Lartigue, CORIA ===

==== N1 - Traction open boundary condition ====

==== N2 - Treatment of Inlet conditions in High-Order solver. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====
In the context of node-centered Finite Volumes Method, spacial accuracy of a numerical scheme depends on ability to evaluate accurately fluxes through interface of each control volume (CV). Such accurate evaluation is not straightforward, especially when dealing with distorted grids. This project follows the work of [1] where fluxes use pointwise quantities, which are reconstructed from integrated quantities advanced in time. During the previous edition of the ECFD, a new data structure has been developed to store data at location of the boundary conditions facelets, with application to wall boundary conditions. During this 8th edition of the ECFD, we used the same data structure, but dedicated to the treatment of inlet conditions.
The inlet condition is then either imposed directly at facelets center, or at nodes position them extrapolated to facelets center by use of Taylor expansion. For this later solution, high-order treatment requires the successive derivatives to be computed in the plane of the boundary condition. This is not done yet, leading for the moment to low accuracy results but the framework is ready for upcoming implementation.

[1] ''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes, , Bernard et. al., IJNMF 2020''

==== N3 - Conservative mesh-to-mesh interpolation. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====

Mesh to mesh interpolations occur quite often in CFD simulations : in the context of adaptative mesh convergence or in the case of dynamic mesh adaptation for for example.
Quality of the solution on the destination grid will depend on the characteristics of the interpolation method.
In this project, we did not focus on accuracy of the interpolation method but rather on conservativity characteristics.
A conservative interpolation ensures that the integral of the data on the source grid is exactly retrieved on the destination grid.
This property is highly interesting when dealing with scalar quantities or phase indicators, whose values should remained bounded.
In the context of nodes centered Finite Volume schemes, the methodology we used consists in (i) reconstructing element quantity from average nodal quantities on source grid.
Then, for a cell of the destination mesh, (ii) computing the geometrical intersection between cells of source and destination meshes to evaluate to evaluate the rate of quantities they.
Eventually, (iii) redistributing the solution from elements to control volumes of the destination mesh.
The overall process is fully conservative as it is based on geometrical intersection of locally integrated quantities.
The methodology as been implemented and tested on a few basic configurations and the conservativity is retrieved.

==== N4 - Determination of timestep in semi-implicit solver. T. Berthelon (LEGI), G. Balarac (LEGI), H. Lam (LEGI), M. El Moatamid (CORIA) ====
In order to reduce the computation time associated with incompressible LES simulations, an implicit time integration, based on BDF schemes, has been developed within the ICS solver. This integration eliminates the stability constraints associated with explicit schemes, and therefore opens up the question of the appropriate choice of time step.
In parallel, recent work has been carried out on meshing criteria in LES. The strategy in place consists of adapting the mesh by distinguishing two zones:
- "DNS" zones, where the meshing criterion is based on an estimate of the adimensioned spatial error.
- "LES" zones, where the meshing criterion is based on Kolmorogov theory.
During this project, the spatial criteria were extended to include temporal criteria. In the "DNS" zones, the time step is chosen using an estimate of the temporal error of the BDF scheme judiciously scaled to match the spatial error. In the "LES" zones, the time step is chosen using a scaling law associated with fully developed turbulence.
The new time step selection strategy has been tested on the case of a turbulent jet and leads to an accuracy equivalent to the explicit case while reducing the simulation return time by a factor of nearly 3.

Another aspect of this project was to integrate certain implicit temporal schemes (C-N and SDIRK) recently developed by Mr. El Moatamid into the incompressible solver.

==== N5 - Local timestep. T. Berthelon (LEGI), M. Bernard (LEGI), G. Balarac (LEGI) ====
RANS modelling has recently been developed within the YALES2 library. With this modeling strategy, the objective is to reach as quick as possible a steady state.
During this project, we investigate the use of a local time step to reduce the time to solution of steady computation in the incompressible solver.
This implies solving a variable-coefficient Poisson equation. Encouraging results were obtained in the simple case of "Couette plan" flow artificially constrained by a mesh variation. In fact, the use of local time-step reduce drastically the time to solution on this configuration. This method needs to be tested on real RANS case.

==== N6 - Distributed version of DOROTHY ====

==== N7 - Implicit time advancement for low-Reynolds number flows with particles. S. Mendez, C. Raveleau (IMAG), M. El Moatamid, V. Moureau (CORIA) ====
IMAG runs numerous simulations of red blood cells under flow. Those simulations are at low Reynolds number (0.001 to 1.0, typically). Splitting of the time advancement is used to treat the diffusion terms implicitly, albeit with an important numerical cost: implicit diffusion is 50 to 60% of the computational cost. Recently, M. El Moatamid implemented a genral framework to deal with implicit time advancement for scalars. In this project, the general method has been transposed to the advancement of the velocity field in the ICS and RBC solvers of YALES2/YALES2BIO. This enables testing various linear solvers (GMRES based). However, such solvers do not decrease the CPU time compared to the existing implementation. However, while working on this, it was identified that residual recycling was not activated in the current implementation of the implicit diffusion. This sped up the implicit diffusion cost by 35%, for a total gain of 20% for the computation. In addition to this achievement, moving to the framework coded by Moncef will have other beneficial side effects: we anticipate simplifying the implementation, with an easier merging between YALES2BIO and YALES2. The method will also be implemented in the electrosatic solver, for which the Poisson problem should benefit from the new GMRES-based solvers. In addition, this project highlights the importance of improving the treatment of stiff source terms in the red blood cells simulations, to be able to overcome the current limitation in time step due to those term and have a chance to benefit from higher-order time schemes, efficient at high Fourier numbers.

==== N8 - Boundary Element Method in Yales2 ====

=== Turbulence - L. Voivenel, CORIA & P. Bénard, CORIA ===

==== T1 - FSI-1D strategy for internal flows====

Many applications developed at Safran Aerosystem are based on internal turbulent flows coupled to a moving body. 2 cases were studied during this ECFD:

'''Case 1 (Incompressible flow)''': Translation of a piston subjected to a pressure difference in a pipe.

The challenges of this case are twofold: the small gap between the piston and the pipe and the large pressure gradient across the piston (>100bar). During the 1st week of ECFD, the CLIB (Conservative Lagrangian Immersed Boundary) solver was tested on this case. The study showed that the solver was unable to ensure the impermeability of the solid under these pressure conditions. In the rest of the study, a porous medium following Darcy's law will be added to the penalty force of the immersed solid to fully satisfy the impermeability of the piston.

'''Case 2 (Compressible flow)''': Rotation of a butterfly in a discharge vane.

The coupling between the ECS (Explicit Compressible Solver) and ALE (Arbitrary Lagrangian Solver) solvers having recently been implemented, this strategy was tested to model the opening of the valve by rotation of the butterfly. The challenge here lies in the small gap between the bottom of the butterfly and the vane casing. To limit the simulation cost, the gap is meshed with 1 element. In this case, MMG succeeded in adapting the mesh up to a critical angle at which the gap becomes too small (Work In Progress).

==== T2 - Dynamic Smagorinsky in Dorothy ====

==== T3 - Turbulence injection strategy for compressible flows ====

==== T4 - Improve wind farm modeling and simulation workflow ====
The YALES2 library includes an advanced modular implementation of the Actuator Line Method (ALM). This approach remains state-of-the-art when performing an LES-based analysis of a wind turbine wake. The method also provides an accurate assessment of the aerodynamic loads applied on the turbine. Still, applying this method to investigate a wind farm flow can be challenging, both in terms of computational cost and simulation setup. For instance, an inadequate management of the wind turbine individual modeling parts in a HPC context can end up being the main bottleneck of the simulation. From another perspective, a wind farm is usually composed of more than 50 wind turbines. For such a case, setting up all YALES2 required inputs manually can be very tedious and error-prone. This project thus mainly aimed to optimize the YALES2 ALM implementation and the user experience around it. Additionally, a cost-effective alternative to the ALM when modeling wind farm flows, namely the Rotating Actuator Disk Method (ADM-R), has been implemented for further investigations.

'''WP1''': Improve Actuator set rotor modelling
* Parallel processing of the ''actuator sets'' used to model the wind turbines
(Felix)

* Rotating Actuator Disk Model (ADM-R):
According to the usual guidelines, the mesh requirements of the ALM, to profit entirely from its reachable accuracy, can be difficult to achieve or even unaffordable when simulating a wind farm flow, especially from the industrial point of view. Alternatives are available in the literature for this kind of application. Likely, the methods from the Actuator Disk family are the most prominent ones. Several kinds of implementation exist, which mostly differ by their capability to include the wake rotation. During the workshop, a new method from the Rotating Actuator Disk kind has been implemented and underwent an early validation on a single turbine setup. Applications to wind farm flows will follow.

'''WP2''': Improve tools User Experience

Three Python tools have been developed or improved :
*The first tool is the wind farm previsualisation tool, 'y2_wind_previsualization', which is used before the calculation run. This provides an interactive HTML interface for viewing global data for each turbine on the farm (position, hub height, yaw angle, etc.). The tool traces all of these via the parsing of the input file.
* The second tool is for duplicating rotor templates for a wind farm (`y2_wind_duplication`). This tool was developed in the previous ECFD, but this time it has been refactored and incorporated into the y2tools package.
* The third and final tool is a post-processing tool for the temporal processing of global wind turbine simulation metrics (Thrust, Power, etc.), `y2_post_wind`. This tool generates an interactive HTML plot of time-dependent global quantities.

==== T5 - Improve atmospheric inflow turbulence ====
Atmospheric inflow turbulence is generated using the precursor database method. A half-channel flow driven by a pressure gradient is used to obtain the inflow which is used as inlet boundary condition for the wind turbine simulation domain. This project aimed to improve the whole methodology, from generation to injection.

* WP1: Improve inflow generation
Anand: pressure controller

* WP2: Improve injection methodology (method A)
The previous workflow used plane probes in the ASCII format to sample the flow. The COWIT2 toolbox was used to convert the file into turbulence box (.man format). While functioning, this methodology had two major flaws. First the probe files are heavy ~O(10Go). Second, the method requires a lot of human effort, allowing numerous sources of errors.
During this workshop, a new methodology has been developed. First, the probes are generated using the HDF5 format (now available for all probe types), leading to lighter file ~O(1Go). Second, Y2_tools is used to read HDF5 format (working for probes and temporals). HDF5 file is then converted into a Look-up Table. Finally, the Look-up Table is read directly by YALES2 as a boundary conditions.

* WP3: Improve injection methodology (method B)
Even though improvements achieved in WP2 prove to be very handy while removing many potential human errors, injecting a turbulent inflow through wind boxes ('offline' precursor approach) can sometimes remain cumbersome for several reasons: (1) no periodicity is enforced in the streamwise direction of those boxes, (2) potential high memory consumption, and (3) the boxes need to be moved to other cores whenever a mesh adaptation occurs. An alternative consists in co-simulating the precursor flow and the flow of interest (refered as the 'successor' simulation) at the same time ('online' precursor approach). The inlet boundary condition for the successor flow is then obtained by mapping the outflow of the precursor domain. During the workshop, some work has been initiated to implement this kind of coupling using the CWIPI library, for which YALES2 provides already an interface.

==== T6 - FSI model in Dorothy ====

=== Two Phase Flow - J. Leparoux, SAFRAN & J. Carmona, CORIA ===

==== TP1 - Towards very small contact angles in Nucleate boiling ====

Participants: Henri Lam (LEGI), Mohammad Umair (LEGI), Manuel Bernard (LEGI), Robin Barbera (LEGI) and Giovanni Ghigliotti (LPSC)

The solver nucleate boiling had difficulties to run at very small contact angles (with angles below 30°). A modified version of the reinitialization has been implemented during the ECFD. It has been tested successfully on the spray solver where no mass transfer occurs, improving the quality of the interface position without significant increase in the computational time. Then, this new reinitialisation has been tested for nucleate boiling with great improvements. Now simulations of nucleate boiling at very small contact angle (10°) can be performed.

==== TP2 - Modeling spray-film interactions ====

Participants: Nicolas Gasnier (EM2C-SafranTech), Julien Leparoux (SafranTech), Mehdi Helal (CORIA-SafranTech) and Julien Carmona (CORIA)

==== TP3 - High-fidelity two-phase flow simulations of the purge of a fuel feed line ====

Participants: Thomas LAROCHE (Safran HE), Romain JANODET (Safran AE), Julien Leparoux (Safran Tech) and Melody Cailler (Safran Tech)

==== TP4 - Volume of Fluid solver in YALES2 ====

Participants: Léa Voivenel (CORIA), Julien Carmona (CORIA), Mehdi Helal (CORIA), Pierre Portais (CORIA), Julien Leparoux (Safran Tech), Mélody Cailler (Safran Tech) and Nicolas Gasnier (EM2C / Safran Tech)

==== TP5 - Implement a local operator to distribute the solid volume of a particle over multiple cells ====

Participants: Théo Ndereyimana (Université de Sherbrooke), Stéphane Moreau (Université de Sherbrooke)

==== TP6 - Complex thermodynamics in sloshing tanks ====

Participants: C. Merlin (AGS), D. Fouquet (CORIA), V. Moureau (CORIA), J. Carmona (CORIA) and G. Lartigue (CORIA)

=== Combustion - Y. Bechane, CORIA & S. Dillon, SAFRAN & K. Bioche, CORIA ===

==== C1 - LES of the thermal degradation of a composite material ====
Participants: A. Grenouilloux (ONERA), K. Bioche (CORIA), N. Dellinger (ONERA) and R. Letournel (SafranTech)

The FIRE test bed is an experimental air-propane burner operated by ONERA. It is dedicated to the study of the thermal degradation of composite materials. This project concerned the implementation of a three-solver coupling methodology to simulate the dynamics of the impinging flame. The methodology considered is based on the coupling between the variable density solver (VDS) and the radiative solver (RDS) of the massively parallel library YALES2 and the solver dedicated to the degradation of composite materials, MoDeTheC, developed by ONERA. Given the typical test times of the order of tens of seconds, a methodology based on 2D axisymmetric calculations was considered. Various tests were performed to determine the optimal coupling frequency between solvers. Cases dedicated to the injection of pyrolysis gasses were set up, with the aim of simulating the auto-ignition phenomenon. Comparisons with experimental data are presented.

==== C2 - Flame stabilization by NRP plasma discharge ====

==== C3 - Extending and validating a generalized formalism of virtual chemistry ====

==== C4 - Turbulent combustion model for NOx prediction ====

==== C5 - Towards 3D simulation of detonation combustion ====

==== C6 - Flame stabilitity of flame-holders within reheat conditions ====

==== C7 - Thermal radiation in oxyflames ====

==== C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 ====

Participants: M. Laignel (CORIA), G. Lartigue (CORIA), K. Bioche (CORIA) and V. Moureau (CORIA)

In numerical simulations of reacting flows, one of the most computationally intensive tasks is the evaluation of source terms resulting from chemical reactions in the species transport equations. This step can account for up to 90% of the total simulation cost , depending on the complexity of the kinetic mechanism involved. To reduce this cost, various techniques such as mechanism reduction, virtual chemistry, etc. have been explored. However, the emergence of GPUs as powerful accelerators offers a promising alternative by providing massive parallelism. Despite their potential, GPUs often require significant adaptation of CPU-based codes. This project aims to address this challenge by taking a first step towards a hybrid CPU/GPU framework for reactive flow simulations. Specifically, the focus is on coupling Y2 with the updated version of the stiff time integration solver (CVODE), which is compatible with GPU (CUDA, HIP, OpenMP). The ultimate goal is to establish a foundation for hybrid computations by implementing and testing the updated solver on simplified test cases.

==== C9 - Soots numerical modeling ====

==== C10 - TECERACT : Tabulated chemistry generator for aeronautical combustion ====

==== C11 - Exploring efficient tabulation strategies for detailed chemistry ====

==== C12 - Dynamic sub-grid-scale modelling of multi-regime flame wrinkling ====

==== C13 - LES of a semi-industrial burner using a non-adiabatic virtual chemical scheme ====

=== User Experience & Data - L. Korzeczek, GDTECH ===

==== U1 - Low-fidelity (RANS) rotor/stator simulations, application to Kaplan Turbine - Y. Lakrifi, G. Balarac (LEGI), R. Mercier (SAFRAN), V. Moureau (CORIA) ====

==== U2 - Coupling PyTorch/YALES2, combustion cartesian look-up tables - J. Leparoux, N. Treleaven, S. Dillon (SAFRAN), K. Bioche, G. Lartigue (CORIA) ====

Participants: Julien Leparoux (Safran Tech), Kévin Bioche (CORIA), Ghislain Lartigue (CORIA), Nicholas Treleaven (Safran Tech)

Neural Networks offer a promising alternative to Cartesian look-up tables for combustion simulations, reducing memory footprint. In this project, we investigated how to integrate an NN model for real-time inference in the YALES2 platform, exploring two approaches: a Python interface and a Fortran Torch binding (using FTorch[https://github.com/Cambridge-ICCS/FTorch]). We validated that the model remains accurate when embedded online and identified improvements for robustness. Inference costs were evaluated on a Mac M3 and the Austral cluster, revealing a strong dependency on data volume. To optimize efficiency, we propose grouping cells at the processor level.

==== U3 - Yales2 Trame Editor, toward a fully featured graphical user interface for YALES2 - L. Korzeczek, S. Meynet (GDTECH), J. Leparoux, M. Cailler (SAFRAN) ====

Ecfd:ecfd 8th edition

2025-02-10T13:43:43Z

Bioche: /* H2 - AVBP GPU offloading based on OpenMP, M.Ghenai, L. Legaux and A. Dauptain (CERFACS) */

{{DISPLAYTITLE: ECFD workshop, 8th edition, 2025}}

== Description ==

* Event from '''27th of January to 7th of February 2025'''
* Location: [https://www.sport-normandie.fr/le-centre/le-site-de-houlgate Centre Sportif de Normandie], Houlgate, near Caen (14)
* Two types of sessions:
** common technical presentations: roadmaps, specific points
** mini-workshops. Potential workshops are listed below
* Free of charge
* Participants from academics, HPC center/experts and industry are welcome
* The number of participants is limited to 68.

* Objectives
** Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
** Identify the technological barriers of exaflopic CFD via numerical experiments
** Identify industrial needs and challenges in high-performance computing
** Propose action plans to add to the development roadmaps of the CFD codes
* Organizers
** Guillaume Balarac (LEGI), Simon Mendez (IMAG), Pierre Bénard, Vincent Moureau, Léa Voivenel (CORIA).

[[File:ecfd8.png|600px|link=https://ecfd.coria-cfd.fr/index.php/Ecfd:ecfd_8th_edition]]

[[File:Acknowledgments_ecfd8.png|text-bottom|600px]]

== News ==

* 23/10/2024: First announcement of the '''8th Extreme CFD Workshop & Hackathon''' !
* 22/11/2024: Deadline to submit your project

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

To come...

== Projects ==

=== Hackathon GENCI - P. Begou, LEGI ===
This ECFD8 GENCI Hackathon was a rich event, involving 4 differents CFD codes (AVBP, ParaDIGM, SONICS and YALES2) using various paradigms (C++/cuda/hip, Fortran/OpenMP/OpenACC) with several SDKs (AMD, Cray/HPE, Nvidia, Gnu) on a large range of GPU architectures (Nvidia A100, GH100, AMD instinct Mi210, Mi250, Mi300). This two-week event benefited from a high level support from three HPC mentors, two on-site from AMD (J. Noudohouenou and A. Tsetoglou) and one remote from CINES (M. Boudaoud).

==== H1 - ParaDIGM and SONICS on GPU, B. Maugars, G. Staffelbach, R.Cazalbou and B. Michel (ONERA)====

==== H2 - AVBP GPU offloading based on OpenMP, M.Ghenai, L. Legaux and A. Dauptain (CERFACS) ====

This hackathon provided a valuable opportunity to work on GPU offloading for AVBP. In the past, significant efforts were made to offload the entire AVBP code to GPUs. OpenACC was the primary strategy chosen, mainly due to access to NVIDIA's support, along with the availability of both software and hardware. This approach achieved good scalability performance.
Recently, with the deployment of new supercomputers like ADASTRA at CINES, some issues have emerged when running AVBP on AMD GPUs, including both MI250 and MI300. The closed-source nature of the Cray environment has also prevented CERFACS from deploying AVBP on local MI210 GPUs.
This hackathon was a great opportunity to address these challenges by exploring a new approach using OpenMP. An automatic translation tool was used to convert approximately 2,700 OpenACC directives to OpenMP, with each directive manually verified and fine-tuned afterward. AVBP with OpenMP had already been tested on NVIDIA GPUs, and during this hackathon, the focus was on extending support to AMD GPUs.
Two compilers were used: Cray and the newly released AFAR from AMD. With the support of AMD and CINES, a working environment for compiling AVBP was set up, and performance-related issues were identified. Additionally, two mini-apps were used for benchmarking. One of them achieved a 2.5× speedup when compiled with AFAR compared to Cray.
The next steps involve adapting the code to address necessary modifications, such as fixing issues related to Fortran indirections, and continuing performance evaluations with mini-apps. Further comparisons will be conducted using both compilers against results obtained with NVIDIA’s NVHPC.

==== H3 - YALES2 GPU from OpenACC to OpenMP, P. Bégou (LEGI), V. Moureau, G. Lartigue (CORIA) and R. Dubois (IMAG) ====
This Hackathon focuses on running Yales2 code on AMD Instinct Mi250 and Mi300 GPUs of the Adastra supercomputer (CINES).
Previously, a first solver in the Yales2 CFD code was successfully ported on the GPU accelerators of the Jean-Zay supercomputer (IDRIS) using Nvidia SDK but difficulties remain on Adastra AMD GPUs, mainly related to the available development tools. High compilation time and the impossibility to use debug flags at compile time as soon as OpenACC is enabled are a real challenge when tracking errors. The current project is to evaluate a freshly deployed version (at the begining of the workshop) of the AMD Fortran compiler. This requires moving to OpenMP paradigm, starting from scratch since the OpenACC branch has largely diverged from the master one while tracking spurious remaining bugs.
If the AMD compiler is able to build the cpu version of Yales2 "out of the box" (wich is not the case for Cray Fortran), the compilation time for each file is significantly higher. However, setting up a 2 stages dynamic compilation process allows for high parallelism that is not possible with Cray Fortran 18 and the library build time drops from nearly 2 hours (Cray Fortran 18) to 17 minutes (Amd Fortran compiler).
Large kernels have been ported from OpenACC to OpenMP, raising some difficulties when offloading intrinsics functions or using strutures attributes in kernels loops. These limitations were also known in the previous OpenACC work. The goal was mainly to check the correctness of the results. The offloading of the complex data structure of Yales2 code was then investigated. Here again some limitations of the "young" compiler were discovered and workarounds were implemented. Several reproducers were built during this ECFD8 and provided to developpers by the 2 on-site AMD engineers.
Preliminary tests on micro-applications show good performances of the generated binaries proving that this compiler could be a serious alternative on AMD GPUs and the goal is now to focus on this SDK in an OpenMP strategy while checking the portablility of this new implementation in Nvidia, Cray/HPE (and Gnu ?) environments.

=== Mesh adaptation - A. Grenouilloux, ONERA & G. Balarac, LEGI ===

=== Numerics - M. Bernard, LEGI & G. Lartigue, CORIA ===

==== N1 - Traction open boundary condition ====

==== N2 - Treatment of Inlet conditions in High-Order solver. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====
In the context of node-centered Finite Volumes Method, spacial accuracy of a numerical scheme depends on ability to evaluate accurately fluxes through interface of each control volume (CV). Such accurate evaluation is not straightforward, especially when dealing with distorted grids. This project follows the work of [1] where fluxes use pointwise quantities, which are reconstructed from integrated quantities advanced in time. During the previous edition of the ECFD, a new data structure has been developed to store data at location of the boundary conditions facelets, with application to wall boundary conditions. During this 8th edition of the ECFD, we used the same data structure, but dedicated to the treatment of inlet conditions.
The inlet condition is then either imposed directly at facelets center, or at nodes position them extrapolated to facelets center by use of Taylor expansion. For this later solution, high-order treatment requires the successive derivatives to be computed in the plane of the boundary condition. This is not done yet, leading for the moment to low accuracy results but the framework is ready for upcoming implementation.

[1] ''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes, , Bernard et. al., IJNMF 2020''

==== N3 - Conservative mesh-to-mesh interpolation. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====

Mesh to mesh interpolations occur quite often in CFD simulations : in the context of adaptative mesh convergence or in the case of dynamic mesh adaptation for for example.
Quality of the solution on the destination grid will depend on the characteristics of the interpolation method.
In this project, we did not focus on accuracy of the interpolation method but rather on conservativity characteristics.
A conservative interpolation ensures that the integral of the data on the source grid is exactly retrieved on the destination grid.
This property is highly interesting when dealing with scalar quantities or phase indicators, whose values should remained bounded.
In the context of nodes centered Finite Volume schemes, the methodology we used consists in (i) reconstructing element quantity from average nodal quantities on source grid.
Then, for a cell of the destination mesh, (ii) computing the geometrical intersection between cells of source and destination meshes to evaluate to evaluate the rate of quantities they.
Eventually, (iii) redistributing the solution from elements to control volumes of the destination mesh.
The overall process is fully conservative as it is based on geometrical intersection of locally integrated quantities.
The methodology as been implemented and tested on a few basic configurations and the conservativity is retrieved.

==== N4 - Determination of timestep in semi-implicit solver. T. Berthelon (LEGI), G. Balarac (LEGI), H. Lam (LEGI), M. El Moatamid (CORIA) ====
In order to reduce the computation time associated with incompressible LES simulations, an implicit time integration, based on BDF schemes, has been developed within the ICS solver. This integration eliminates the stability constraints associated with explicit schemes, and therefore opens up the question of the appropriate choice of time step.
In parallel, recent work has been carried out on meshing criteria in LES. The strategy in place consists of adapting the mesh by distinguishing two zones:
- "DNS" zones, where the meshing criterion is based on an estimate of the adimensioned spatial error.
- "LES" zones, where the meshing criterion is based on Kolmorogov theory.
During this project, the spatial criteria were extended to include temporal criteria. In the "DNS" zones, the time step is chosen using an estimate of the temporal error of the BDF scheme judiciously scaled to match the spatial error. In the "LES" zones, the time step is chosen using a scaling law associated with fully developed turbulence.
The new time step selection strategy has been tested on the case of a turbulent jet and leads to an accuracy equivalent to the explicit case while reducing the simulation return time by a factor of nearly 3.

Another aspect of this project was to integrate certain implicit temporal schemes (C-N and SDIRK) recently developed by Mr. El Moatamid into the incompressible solver.

==== N5 - Local timestep. T. Berthelon (LEGI), M. Bernard (LEGI), G. Balarac (LEGI) ====
RANS modelling has recently been developed within the YALES2 library. With this modeling strategy, the objective is to reach as quick as possible a steady state.
During this project, we investigate the use of a local time step to reduce the time to solution of steady computation in the incompressible solver.
This implies solving a variable-coefficient Poisson equation. Encouraging results were obtained in the simple case of "Couette plan" flow artificially constrained by a mesh variation. In fact, the use of local time-step reduce drastically the time to solution on this configuration. This method needs to be tested on real RANS case.

==== N6 - Distributed version of DOROTHY ====

==== N7 - Implicit time advancement for low-Reynolds number flows with particles. S. Mendez, C. Raveleau (IMAG), M. El Moatamid, V. Moureau (CORIA) ====
IMAG runs numerous simulations of red blood cells under flow. Those simulations are at low Reynolds number (0.001 to 1.0, typically). Splitting of the time advancement is used to treat the diffusion terms implicitly, albeit with an important numerical cost: implicit diffusion is 50 to 60% of the computational cost. Recently, M. El Moatamid implemented a genral framework to deal with implicit time advancement for scalars. In this project, the general method has been transposed to the advancement of the velocity field in the ICS and RBC solvers of YALES2/YALES2BIO. This enables testing various linear solvers (GMRES based). However, such solvers do not decrease the CPU time compared to the existing implementation. However, while working on this, it was identified that residual recycling was not activated in the current implementation of the implicit diffusion. This sped up the implicit diffusion cost by 35%, for a total gain of 20% for the computation. In addition to this achievement, moving to the framework coded by Moncef will have other beneficial side effects: we anticipate simplifying the implementation, with an easier merging between YALES2BIO and YALES2. The method will also be implemented in the electrosatic solver, for which the Poisson problem should benefit from the new GMRES-based solvers. In addition, this project highlights the importance of improving the treatment of stiff source terms in the red blood cells simulations, to be able to overcome the current limitation in time step due to those term and have a chance to benefit from higher-order time schemes, efficient at high Fourier numbers.

==== N8 - Boundary Element Method in Yales2 ====

=== Turbulence - L. Voivenel, CORIA & P. Bénard, CORIA ===

==== T1 - FSI-1D strategy for internal flows====

Many applications developed at Safran Aerosystem are based on internal turbulent flows coupled to a moving body. 2 cases were studied during this ECFD:

'''Case 1 (Incompressible flow)''': Translation of a piston subjected to a pressure difference in a pipe.

The challenges of this case are twofold: the small gap between the piston and the pipe and the large pressure gradient across the piston (>100bar). During the 1st week of ECFD, the CLIB (Conservative Lagrangian Immersed Boundary) solver was tested on this case. The study showed that the solver was unable to ensure the impermeability of the solid under these pressure conditions. In the rest of the study, a porous medium following Darcy's law will be added to the penalty force of the immersed solid to fully satisfy the impermeability of the piston.

'''Case 2 (Compressible flow)''': Rotation of a butterfly in a discharge vane.

The coupling between the ECS (Explicit Compressible Solver) and ALE (Arbitrary Lagrangian Solver) solvers having recently been implemented, this strategy was tested to model the opening of the valve by rotation of the butterfly. The challenge here lies in the small gap between the bottom of the butterfly and the vane casing. To limit the simulation cost, the gap is meshed with 1 element. In this case, MMG succeeded in adapting the mesh up to a critical angle at which the gap becomes too small (Work In Progress).

==== T2 - Dynamic Smagorinsky in Dorothy ====

==== T3 - Turbulence injection strategy for compressible flows ====

==== T4 - Improve wind farm modeling and simulation workflow ====
The YALES2 library includes an advanced modular implementation of the Actuator Line Method (ALM). This approach remains state-of-the-art when performing an LES-based analysis of a wind turbine wake. The method also provides an accurate assessment of the aerodynamic loads applied on the turbine. Still, applying this method to investigate a wind farm flow can be challenging, both in terms of computational cost and simulation setup. For instance, an inadequate management of the wind turbine individual modeling parts in a HPC context can end up being the main bottleneck of the simulation. From another perspective, a wind farm is usually composed of more than 50 wind turbines. For such a case, setting up all YALES2 required inputs manually can be very tedious and error-prone. This project thus mainly aimed to optimize the YALES2 ALM implementation and the user experience around it. Additionally, a cost-effective alternative to the ALM when modeling wind farm flows, namely the Rotating Actuator Disk Method (ADM-R), has been implemented for further investigations.

'''WP1''': Improve Actuator set rotor modelling
* Parallel processing of the ''actuator sets'' used to model the wind turbines
(Felix)

* Rotating Actuator Disk Model (ADM-R):
According to the usual guidelines, the mesh requirements of the ALM, to profit entirely from its reachable accuracy, can be difficult to achieve or even unaffordable when simulating a wind farm flow, especially from the industrial point of view. Alternatives are available in the literature for this kind of application. Likely, the methods from the Actuator Disk family are the most prominent ones. Several kinds of implementation exist, which mostly differ by their capability to include the wake rotation. During the workshop, a new method from the Rotating Actuator Disk kind has been implemented and underwent an early validation on a single turbine setup. Applications to wind farm flows will follow.

'''WP2''': Improve tools User Experience

Three Python tools have been developed or improved :
*The first tool is the wind farm previsualisation tool, 'y2_wind_previsualization', which is used before the calculation run. This provides an interactive HTML interface for viewing global data for each turbine on the farm (position, hub height, yaw angle, etc.). The tool traces all of these via the parsing of the input file.
* The second tool is for duplicating rotor templates for a wind farm (`y2_wind_duplication`). This tool was developed in the previous ECFD, but this time it has been refactored and incorporated into the y2tools package.
* The third and final tool is a post-processing tool for the temporal processing of global wind turbine simulation metrics (Thrust, Power, etc.), `y2_post_wind`. This tool generates an interactive HTML plot of time-dependent global quantities.

==== T5 - Improve atmospheric inflow turbulence ====
Atmospheric inflow turbulence is generated using the precursor database method. A half-channel flow driven by a pressure gradient is used to obtain the inflow which is used as inlet boundary condition for the wind turbine simulation domain. This project aimed to improve the whole methodology, from generation to injection.

* WP1: Improve inflow generation
Anand: pressure controller

* WP2: Improve injection methodology (method A)
The previous workflow used plane probes in the ASCII format to sample the flow. The COWIT2 toolbox was used to convert the file into turbulence box (.man format). While functioning, this methodology had two major flaws. First the probe files are heavy ~O(10Go). Second, the method requires a lot of human effort, allowing numerous sources of errors.
During this workshop, a new methodology has been developed. First, the probes are generated using the HDF5 format (now available for all probe types), leading to lighter file ~O(1Go). Second, Y2_tools is used to read HDF5 format (working for probes and temporals). HDF5 file is then converted into a Look-up Table. Finally, the Look-up Table is read directly by YALES2 as a boundary conditions.

* WP3: Improve injection methodology (method B)
Even though improvements achieved in WP2 prove to be very handy while removing many potential human errors, injecting a turbulent inflow through wind boxes ('offline' precursor approach) can sometimes remain cumbersome for several reasons: (1) no periodicity is enforced in the streamwise direction of those boxes, (2) potential high memory consumption, and (3) the boxes need to be moved to other cores whenever a mesh adaptation occurs. An alternative consists in co-simulating the precursor flow and the flow of interest (refered as the 'successor' simulation) at the same time ('online' precursor approach). The inlet boundary condition for the successor flow is then obtained by mapping the outflow of the precursor domain. During the workshop, some work has been initiated to implement this kind of coupling using the CWIPI library, for which YALES2 provides already an interface.

==== T6 - FSI model in Dorothy ====

=== Two Phase Flow - J. Leparoux, SAFRAN & J. Carmona, CORIA ===

==== TP1 - Towards very small contact angles in Nucleate boiling ====

Participants: Henri Lam (LEGI), Mohammad Umair (LEGI), Manuel Bernard (LEGI), Robin Barbera (LEGI) and Giovanni Ghigliotti (LPSC)

The solver nucleate boiling had difficulties to run at very small contact angles (with angles below 30°). A modified version of the reinitialization has been implemented during the ECFD. It has been tested successfully on the spray solver where no mass transfer occurs, improving the quality of the interface position without significant increase in the computational time. Then, this new reinitialisation has been tested for nucleate boiling with great improvements. Now simulations of nucleate boiling at very small contact angle (10°) can be performed.

==== TP2 - Modeling spray-film interactions ====

Participants: Nicolas Gasnier (EM2C-SafranTech), Julien Leparoux (SafranTech), Mehdi Helal (CORIA-SafranTech) and Julien Carmona (CORIA)

==== TP3 - High-fidelity two-phase flow simulations of the purge of a fuel feed line ====

Participants: Thomas LAROCHE (Safran HE), Romain JANODET (Safran AE), Julien Leparoux (Safran Tech) and Melody Cailler (Safran Tech)

==== TP4 - Volume of Fluid solver in YALES2 ====

Participants: Léa Voivenel (CORIA), Julien Carmona (CORIA), Mehdi Helal (CORIA), Pierre Portais (CORIA), Julien Leparoux (Safran Tech), Mélody Cailler (Safran Tech) and Nicolas Gasnier (EM2C / Safran Tech)

==== TP5 - Implement a local operator to distribute the solid volume of a particle over multiple cells ====

Participants: Théo Ndereyimana (Université de Sherbrooke), Stéphane Moreau (Université de Sherbrooke)

==== TP6 - Complex thermodynamics in sloshing tanks ====

Participants: C. Merlin (AGS), D. Fouquet (CORIA), V. Moureau (CORIA), J. Carmona (CORIA) and G. Lartigue (CORIA)

=== Combustion - Y. Bechane, CORIA & S. Dillon, SAFRAN & K. Bioche, CORIA ===

==== C1 - LES of the thermal degradation of a composite material ====
Participants: A. Grenouilloux (ONERA), K. Bioche (CORIA), N. Dellinger (ONERA) and R. Letournel (SafranTech)

The FIRE test bed is an experimental air-propane burner operated by ONERA. It is dedicated to the study of the thermal degradation of composite materials. This project concerned the implementation of a three-solver coupling methodology to simulate the dynamics of the impinging flame. The methodology considered is based on the coupling between the variable density solver (VDS) and the radiative solver (RDS) of the massively parallel library YALES2 and the solver dedicated to the degradation of composite materials, MoDeTheC, developed by ONERA. Given the typical test times of the order of tens of seconds, a methodology based on 2D axisymmetric calculations was considered. Various tests were performed to determine the optimal coupling frequency between solvers. Cases dedicated to the injection of pyrolysis gasses were set up, with the aim of simulating the auto-ignition phenomenon. Comparisons with experimental data are presented.

==== C2 - Flame stabilization by NRP plasma discharge ====

==== C3 - Extending and validating a generalized formalism of virtual chemistry ====

==== C4 - Turbulent combustion model for NOx prediction ====

==== C5 - Towards 3D simulation of detonation combustion ====

==== C6 - Flame stabilitity of flame-holders within reheat conditions ====

==== C7 - Thermal radiation in oxyflames ====

==== C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 ====

Participants: M. Laignel (CORIA), G. Lartigue (CORIA), K. Bioche (CORIA) and V. Moureau (CORIA)

In numerical simulations of reacting flows, one of the most computationally intensive tasks is the evaluation of source terms resulting from chemical reactions in the species transport equations. This step can account for up to 90% of the total simulation cost , depending on the complexity of the kinetic mechanism involved. To reduce this cost, various techniques such as mechanism reduction, virtual chemistry, etc. have been explored. However, the emergence of GPUs as powerful accelerators offers a promising alternative by providing massive parallelism. Despite their potential, GPUs often require significant adaptation of CPU-based codes. This project aims to address this challenge by taking a first step towards a hybrid CPU/GPU framework for reactive flow simulations. Specifically, the focus is on coupling Y2 with the updated version of the stiff time integration solver (CVODE), which is compatible with GPU (CUDA, HIP, OpenMP). The ultimate goal is to establish a foundation for hybrid computations by implementing and testing the updated solver on simplified test cases.

==== C9 - Soots numerical modeling ====

==== C10 - TECERACT : Tabulated chemistry generator for aeronautical combustion ====

==== C11 - Exploring efficient tabulation strategies for detailed chemistry ====

==== C12 - Dynamic sub-grid-scale modelling of multi-regime flame wrinkling ====

==== C13 - LES of a semi-industrial burner using a non-adiabatic virtual chemical scheme ====

=== User Experience & Data - L. Korzeczek, GDTECH ===

==== U1 - Low-fidelity (RANS) rotor/stator simulations, application to Kaplan Turbine - Y. Lakrifi, G. Balarac (LEGI), R. Mercier (SAFRAN), V. Moureau (CORIA) ====

==== U2 - Coupling PyTorch/YALES2, combustion cartesian look-up tables - J. Leparoux, N. Treleaven, S. Dillon (SAFRAN), K. Bioche, G. Lartigue (CORIA) ====

==== U3 - Yales2 Trame Editor, toward a fully featured graphical user interface for YALES2 - L. Korzeczek, S. Meynet (GDTECH), J. Leparoux, M. Cailler (SAFRAN) ====

Participants: Julien Leparoux (Safran Tech), Kévin Bioche (CORIA), Ghislain Lartigue (CORIA), Nicholas Treleaven (Safran Tech)

Neural Networks offer a promising alternative to Cartesian look-up tables for combustion simulations, reducing memory footprint. In this project, we investigated how to integrate an NN model for real-time inference in the YALES2 platform, exploring two approaches: a Python interface and a Fortran Torch binding (using FTorch[https://github.com/Cambridge-ICCS/FTorch]). We validated that the model remains accurate when embedded online and identified improvements for robustness. Inference costs were evaluated on a Mac M3 and the Austral cluster, revealing a strong dependency on data volume. To optimize efficiency, we propose grouping cells at the processor level.

Ecfd:ecfd 8th edition

2025-02-10T10:47:13Z

Bioche: /* C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 */

{{DISPLAYTITLE: ECFD workshop, 8th edition, 2025}}

== Description ==

* Event from '''27th of January to 7th of February 2025'''
* Location: [https://www.sport-normandie.fr/le-centre/le-site-de-houlgate Centre Sportif de Normandie], Houlgate, near Caen (14)
* Two types of sessions:
** common technical presentations: roadmaps, specific points
** mini-workshops. Potential workshops are listed below
* Free of charge
* Participants from academics, HPC center/experts and industry are welcome
* The number of participants is limited to 68.

* Objectives
** Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
** Identify the technological barriers of exaflopic CFD via numerical experiments
** Identify industrial needs and challenges in high-performance computing
** Propose action plans to add to the development roadmaps of the CFD codes
* Organizers
** Guillaume Balarac (LEGI), Simon Mendez (IMAG), Pierre Bénard, Vincent Moureau, Léa Voivenel (CORIA).

[[File:ecfd8.png|600px|link=https://ecfd.coria-cfd.fr/index.php/Ecfd:ecfd_8th_edition]]

[[File:Acknowledgments_ecfd8.png|text-bottom|600px]]

== News ==

* 23/10/2024: First announcement of the '''8th Extreme CFD Workshop & Hackathon''' !
* 22/11/2024: Deadline to submit your project

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

To come...

== Projects ==

=== Hackathon GENCI - P. Begou, LEGI ===
This ECFD8 GENCI Hackathon was a rich event, involving 4 differents CFD codes (AVBP, ParaDIGM, SONICS and YALES2) using various paradigms (C++/cuda/hip, Fortran/OpenMP/OpenACC) with several SDKs (AMD, Cray/HPE, Nvidia, Gnu) on a large range of GPU architectures (Nvidia A100, GH100, AMD instinct Mi210, Mi250, Mi300). This two-week event benefited from a high level support from three HPC mentors, two on-site from AMD (J. Noudohouenou and A. Tsetoglou) and one remote from CINES (M. Boudaoud).

==== H1 - ParaDIGM and SONICS on GPU, B. Maugars, G. Staffelbach, R.Cazalbou and B. Michel (ONERA)====

==== H2 - AVBP GPU offloading based on OpenMP, M.Ghenai, L. Legaux and A. Dauptain (CERFACS) ====

==== H3 - YALES2 GPU from OpenACC to OpenMP, P. Bégou (LEGI), V. Moureau, G. Lartigue (CORIA) and R. Dubois (IMAG) ====
This Hackathon focuses on running Yales2 code on AMD Instinct Mi250 and Mi300 GPUs of the Adastra supercomputer (CINES).
Previously, a first solver in the Yales2 CFD code was successfully ported on the GPU accelerators of the Jean-Zay supercomputer (IDRIS) using Nvidia SDK but difficulties remain on Adastra AMD GPUs, mainly related to the available development tools. High compilation time and the impossibility to use debug flags at compile time as soon as OpenACC is enabled are a real challenge when tracking errors. The current project is to evaluate a freshly deployed version (at the begining of the workshop) of the AMD Fortran compiler. This requires moving to OpenMP paradigm, starting from scratch since the OpenACC branch has largely diverged from the master one while tracking spurious remaining bugs.
If the AMD compiler is able to build the cpu version of Yales2 "out of the box" (wich is not the case for Cray Fortran), the compilation time for each file is significantly higher. However, setting up a 2 stages dynamic compilation process allows for high parallelism that is not possible with Cray Fortran 18 and the library build time drops from nearly 2 hours (Cray Fortran 18) to 17 minutes (Amd Fortran compiler).
Large kernels have been ported from OpenACC to OpenMP, raising some difficulties when offloading intrinsics functions or using strutures attributes in kernels loops. These limitations were also known in the previous OpenACC work. The goal was mainly to check the correctness of the results. The offloading of the complex data structure of Yales2 code was then investigated. Here again some limitations of the "young" compiler were discovered and workarounds were implemented. Several reproducers were built during this ECFD8 and provided to developpers by the 2 on-site AMD engineers.
Preliminary tests on micro-applications show good performances of the generated binaries proving that this compiler could be a serious alternative on AMD GPUs and the goal is now to focus on this SDK in an OpenMP strategy while checking the portablility of this new implementation in Nvidia, Cray/HPE (and Gnu ?) environments.

=== Mesh adaptation - A. Grenouilloux, ONERA & G. Balarac, LEGI ===

=== Numerics - M. Bernard, LEGI & G. Lartigue, CORIA ===

==== N2 - Treatment of Inlet conditions in High-Order solver. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====
In the context of node-centered Finite Volumes Method, spacial accuracy of a numerical scheme depends on ability to evaluate accurately fluxes through interface of each control volume (CV). Such accurate evaluation is not straightforward, especially when dealing with distorted grids. This project follows the work of [1] where fluxes use pointwise quantities, which are reconstructed from integrated quantities advanced in time. During the previous edition of the ECFD, a new data structure has been developed to store data at location of the boundary conditions facelets, with application to wall boundary conditions. During this 8th edition of the ECFD, we used the same data structure, but dedicated to the treatment of inlet conditions.
The inlet condition is then either imposed directly at facelets center, or at nodes position them extrapolated to facelets center by use of Taylor expansion. For this later solution, high-order treatment requires the successive derivatives to be computed in the plane of the boundary condition. This is not done yet, leading for the moment to low accuracy results but the framework is ready for upcoming implementation.

[1] ''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes, , Bernard et. al., IJNMF 2020''

==== N3 - Conservative mesh-to-mesh interpolation. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====

Mesh to mesh interpolations occur quite often in CFD simulations : in the context of adaptative mesh convergence or in the case of dynamic mesh adaptation for for example.
Quality of the solution on the destination grid will depend on the characteristics of the interpolation method.
In this project, we did not focus on accuracy of the interpolation method but rather on conservativity characteristics.
A conservative interpolation ensures that the integral of the data on the source grid is exactly retrieved on the destination grid.
This property is highly interesting when dealing with scalar quantities or phase indicators, whose values should remained bounded.
In the context of nodes centered Finite Volume schemes, the methodology we used consists in (i) reconstructing element quantity from average nodal quantities on source grid.
Then, for a cell of the destination mesh, (ii) computing the geometrical intersection between cells of source and destination meshes to evaluate to evaluate the rate of quantities they.
Eventually, (iii) redistributing the solution from elements to control volumes of the destination mesh.
The overall process is fully conservative as it is based on geometrical intersection of locally integrated quantities.
The methodology as been implemented and tested on a few basic configurations and the conservativity is retrieved.

==== N7 - Implicit time advancement for low-Reynolds number flows with particles. S. Mendez, C. Raveleau (IMAG), M. El Moatamid, V. Moureau (CORIA) ====
IMAG runs numerous simulations of red blood cells under flow. Those simulations are at low Reynolds number (0.001 to 1.0, typically). Splitting of the time advancement is used to treat the diffusion terms implicitly, albeit with an important numerical cost: implicit diffusion is 50 to 60% of the computational cost. Recently, M. El Moatamid implemented a genral framework to deal with implicit time advancement for scalars. In this project, the general method has been transposed to the advancement of the velocity field in the ICS and RBC solvers of YALES2/YALES2BIO. This enables testing various linear solvers (GMRES based). However, such solvers do not decrease the CPU time compared to the existing implementation. However, while working on this, it was identified that residual recycling was not activated in the current implementation of the implicit diffusion. This sped up the implicit diffusion cost by 35%, for a total gain of 20% for the computation. In addition to this achievement, moving to the framework coded by Moncef will have other beneficial side effects: we anticipate simplifying the implementation, with an easier merging between YALES2BIO and YALES2. The method will also be implemented in the electrosatic solver, for which the Poisson problem should benefit from the new GMRES-based solvers. In addition, this project highlights the importance of improving the treatment of stiff source terms in the red blood cells simulations, to be able to overcome the current limitation in time step due to those term and have a chance to benefit from higher-order time schemes, efficient at high Fourier numbers.

=== Turbulence - L. Voivenel, CORIA & P. Bénard, CORIA ===

==== T1 - FSI-1D strategy for internal flows====

==== T2 - Dynamic Smagorinsky in Dorothy ====

==== T3 - Turbulence injection strategy for compressible flows ====

==== T4 - Improve wind farm modeling and simulation workflow ====
The YALES2 library includes an advanced modular implementation of the Actuator Line Method (ALM). This approach remains state-of-the-art when performing an LES-based analysis of a wind turbine wake. The method also provides an accurate assessment of the aerodynamic loads applied on the turbine. Still, applying this method to investigate a wind farm flow can be challenging, both in terms of computational cost and simulation setup. For instance, an inadequate management of the wind turbine individual modeling parts in a HPC context can end up being the main bottleneck of the simulation. From another perspective, a wind farm is usually composed of more than 50 wind turbines. For such a case, setting up all YALES2 required inputs manually can be very tedious and error-prone. This project thus mainly aimed to optimize the YALES2 ALM implementation and the user experience around it. Additionally, a cost-effective alternative to the ALM when modeling wind farm flows, namely the Rotating Actuator Disk Method (ADM-R), has been implemented for further investigations.

'''WP1''': Improve Actuator set rotor modelling
* Parallel processing of the ''actuator sets'' used to model the wind turbines
(Felix)

* Rotating Actuator Disk Model (ADM-R):
According to the usual guidelines, the mesh requirements of the ALM, to profit entirely from its reachable accuracy, can be difficult to achieve or even unaffordable when simulating a wind farm flow, especially from the industrial point of view. Alternatives are available in the literature for this kind of application. Likely, the methods from the Actuator Disk family are the most prominent ones. Several kinds of implementation exist, which mostly differ by their capability to include the wake rotation. During the workshop, a new method from the Rotating Actuator Disk kind has been implemented and underwent an early validation on a single turbine setup. Applications to wind farm flows will follow.

'''WP2''': Improve tools User Experience

Three Python tools have been developed or improved :
*The first tool is the wind farm previsualisation tool, 'y2_wind_previsualization', which is used before the calculation run. This provides an interactive HTML interface for viewing global data for each turbine on the farm (position, hub height, yaw angle, etc.). The tool traces all of these via the parsing of the input file.
* The second tool is for duplicating rotor templates for a wind farm (`y2_wind_duplication`). This tool was developed in the previous ECFD, but this time it has been refactored and incorporated into the y2tools package.
* The third and final tool is a post-processing tool for the temporal processing of global wind turbine simulation metrics (Thrust, Power, etc.), `y2_post_wind`. This tool generates an interactive HTML plot of time-dependent global quantities.

==== T5 - Improve atmospheric inflow turbulence ====
Atmospheric inflow turbulence is generated using the precursor database method. A half-channel flow driven by a pressure gradient is used to obtain the inflow which is used as inlet boundary condition for the wind turbine simulation domain. This project aimed to improve the whole methodology, from generation to injection.

* WP1: Improve inflow generation
Anand: pressure controller

* WP2: Improve injection methodology (method A)
The previous workflow used plane probes in the ASCII format to sample the flow. The COWIT2 toolbox was used to convert the file into turbulence box (.man format). While functioning, this methodology had two major flaws. First the probe files are heavy ~O(10Go). Second, the method requires a lot of human effort, allowing numerous sources of errors.
During this workshop, a new methodology has been developed. First, the probes are generated using the HDF5 format (now available for all probe types), leading to lighter file ~O(1Go). Second, Y2_tools is used to read HDF5 format (working for probes and temporals). HDF5 file is then converted into a Look-up Table. Finally, the Look-up Table is read directly by YALES2 as a boundary conditions.

* WP3: Improve injection methodology (method B)
Even though improvements achieved in WP2 prove to be very handy while removing many potential human errors, injecting a turbulent inflow through wind boxes ('offline' precursor approach) can sometimes remain cumbersome for several reasons: (1) no periodicity is enforced in the streamwise direction of those boxes, (2) potential high memory consumption, and (3) the boxes need to be moved to other cores whenever a mesh adaptation occurs. An alternative consists in co-simulating the precursor flow and the flow of interest (refered as the 'successor' simulation) at the same time ('online' precursor approach). The inlet boundary condition for the successor flow is then obtained by mapping the outflow of the precursor domain. During the workshop, some work has been initiated to implement this kind of coupling using the CWIPI library, for which YALES2 provides already an interface.

==== T6 - FSI model in Dorothy ====

=== Two Phase Flow - J. Leparoux, SAFRAN & J. Carmona, CORIA ===

==== TP1 - Towards very small contact angles in Nucleate boiling ====

Participants: Henri Lam (LEGI), Mohammad Umair (LEGI), Manuel Bernard (LEGI), Robin Barbera (LEGI) and Giovanni Ghigliotti (LPSC)

==== TP2 - Modeling spray-film interactions ====

Participants: Nicolas Gasnier (EM2C-SafranTech), Julien Leparoux (SafranTech), Mehdi Helal (CORIA-SafranTech) and Julien Carmona (CORIA)

==== TP3 - High-fidelity two-phase flow simulations of the purge of a fuel feed line ====

Participants: Thomas LAROCHE (Safran HE), Romain JANODET (Safran AE), Julien Leparoux (Safran Tech) and Melody Cailler (Safran Tech)

==== TP4 - Volume of Fluid solver in YALES2 ====

Participants: Léa Voivenel (CORIA), Julien Carmona (CORIA), Mehdi Helal (CORIA), Pierre Portais (CORIA), Julien Leparoux (Safran Tech), Mélody Cailler (Safran Tech) and Nicolas Gasnier (EM2C / Safran Tech)

==== TP5 - Implement a local operator to distribute the solid volume of a particle over multiple cells ====

Participants: Théo Ndereyimana (Université de Sherbrooke), Stéphane Moreau (Université de Sherbrooke)

==== TP6 - Complex thermodynamics in sloshing tanks ====

Participants: C. Merlin (AGS), D. Fouquet (CORIA), V. Moureau (CORIA), J. Carmona (CORIA) and G. Lartigue (CORIA)

=== Combustion - Y. Bechane, CORIA & S. Dillon, SAFRAN & K. Bioche, CORIA ===

==== C1 - LES of the thermal degradation of a composite material ====
Participants: A. Grenouilloux (ONERA), K. Bioche (CORIA), N. Dellinger (ONERA) and R. Letournel (SafranTech)

==== C2 - Flame stabilization by NRP plasma discharge ====

==== C3 - Extending and validating a generalized formalism of virtual chemistry ====

==== C4 - Turbulent combustion model for NOx prediction ====

==== C5 - Towards 3D simulation of detonation combustion ====

==== C6 - Flame stabilitity of flame-holders within reheat conditions ====

==== C7 - Thermal radiation in oxyflames ====

==== C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 ====

Participants: M. Laignel (CORIA), G. Lartigue (CORIA), K. Bioche (CORIA) and V. Moureau (CORIA)

In numerical simulations of reacting flows, one of the most computationally intensive tasks is the evaluation of source terms resulting from chemical reactions in the species transport equations. This step can account for up to 90% of the total simulation cost , depending on the complexity of the kinetic mechanism involved. To reduce this cost, various techniques such as mechanism reduction, virtual chemistry, etc. have been explored. However, the emergence of GPUs as powerful accelerators offers a promising alternative by providing massive parallelism. Despite their potential, GPUs often require significant adaptation of CPU-based codes. This project aims to address this challenge by taking a first step towards a hybrid CPU/GPU framework for reactive flow simulations. Specifically, the focus is on coupling Y2 with the updated version of the stiff time integration solver (CVODE), which is compatible with GPU (CUDA, HIP, OpenMP). The ultimate goal is to establish a foundation for hybrid computations by implementing and testing the updated solver on simplified test cases.

==== C9 - Soots numerical modeling ====

==== C10 - TECERACT : Tabulated chemistry generator for aeronautical combustion ====

==== C11 - Exploring efficient tabulation strategies for detailed chemistry ====

==== C12 - Dynamic sub-grid-scale modelling of multi-regime flame wrinkling ====

==== C13 - LES of a semi-industrial burner using a non-adiabatic virtual chemical scheme ====

=== User Experience & Data - L. Korzeczek, GDTECH ===

==== U1 - Low-fidelity (RANS) rotor/stator simulations, application to Kaplan Turbine - Y. Lakrifi, G. Balarac (LEGI), R. Mercier (SAFRAN), V. Moureau (CORIA) ====

==== U2 - Coupling PyTorch/YALES2, combustion cartesian look-up tables - J. Leparoux, N. Treleaven, S. Dillon (SAFRAN), K. Bioche, G. Lartigue (CORIA) ====

==== U3 - Yales2 Trame Editor, toward a fully featured graphical user interface for YALES2 - L. Korzeczek, S. Meynet (GDTECH), J. Leparoux, M. Cailler (SAFRAN) ====

Ecfd:ecfd 8th edition

2025-02-10T10:47:06Z

Bioche: /* C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 */

{{DISPLAYTITLE: ECFD workshop, 8th edition, 2025}}

== Description ==

* Event from '''27th of January to 7th of February 2025'''
* Location: [https://www.sport-normandie.fr/le-centre/le-site-de-houlgate Centre Sportif de Normandie], Houlgate, near Caen (14)
* Two types of sessions:
** common technical presentations: roadmaps, specific points
** mini-workshops. Potential workshops are listed below
* Free of charge
* Participants from academics, HPC center/experts and industry are welcome
* The number of participants is limited to 68.

* Objectives
** Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
** Identify the technological barriers of exaflopic CFD via numerical experiments
** Identify industrial needs and challenges in high-performance computing
** Propose action plans to add to the development roadmaps of the CFD codes
* Organizers
** Guillaume Balarac (LEGI), Simon Mendez (IMAG), Pierre Bénard, Vincent Moureau, Léa Voivenel (CORIA).

[[File:ecfd8.png|600px|link=https://ecfd.coria-cfd.fr/index.php/Ecfd:ecfd_8th_edition]]

[[File:Acknowledgments_ecfd8.png|text-bottom|600px]]

== News ==

* 23/10/2024: First announcement of the '''8th Extreme CFD Workshop & Hackathon''' !
* 22/11/2024: Deadline to submit your project

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

To come...

== Projects ==

=== Hackathon GENCI - P. Begou, LEGI ===
This ECFD8 GENCI Hackathon was a rich event, involving 4 differents CFD codes (AVBP, ParaDIGM, SONICS and YALES2) using various paradigms (C++/cuda/hip, Fortran/OpenMP/OpenACC) with several SDKs (AMD, Cray/HPE, Nvidia, Gnu) on a large range of GPU architectures (Nvidia A100, GH100, AMD instinct Mi210, Mi250, Mi300). This two-week event benefited from a high level support from three HPC mentors, two on-site from AMD (J. Noudohouenou and A. Tsetoglou) and one remote from CINES (M. Boudaoud).

==== H1 - ParaDIGM and SONICS on GPU, B. Maugars, G. Staffelbach, R.Cazalbou and B. Michel (ONERA)====

==== H2 - AVBP GPU offloading based on OpenMP, M.Ghenai, L. Legaux and A. Dauptain (CERFACS) ====

==== H3 - YALES2 GPU from OpenACC to OpenMP, P. Bégou (LEGI), V. Moureau, G. Lartigue (CORIA) and R. Dubois (IMAG) ====
This Hackathon focuses on running Yales2 code on AMD Instinct Mi250 and Mi300 GPUs of the Adastra supercomputer (CINES).
Previously, a first solver in the Yales2 CFD code was successfully ported on the GPU accelerators of the Jean-Zay supercomputer (IDRIS) using Nvidia SDK but difficulties remain on Adastra AMD GPUs, mainly related to the available development tools. High compilation time and the impossibility to use debug flags at compile time as soon as OpenACC is enabled are a real challenge when tracking errors. The current project is to evaluate a freshly deployed version (at the begining of the workshop) of the AMD Fortran compiler. This requires moving to OpenMP paradigm, starting from scratch since the OpenACC branch has largely diverged from the master one while tracking spurious remaining bugs.
If the AMD compiler is able to build the cpu version of Yales2 "out of the box" (wich is not the case for Cray Fortran), the compilation time for each file is significantly higher. However, setting up a 2 stages dynamic compilation process allows for high parallelism that is not possible with Cray Fortran 18 and the library build time drops from nearly 2 hours (Cray Fortran 18) to 17 minutes (Amd Fortran compiler).
Large kernels have been ported from OpenACC to OpenMP, raising some difficulties when offloading intrinsics functions or using strutures attributes in kernels loops. These limitations were also known in the previous OpenACC work. The goal was mainly to check the correctness of the results. The offloading of the complex data structure of Yales2 code was then investigated. Here again some limitations of the "young" compiler were discovered and workarounds were implemented. Several reproducers were built during this ECFD8 and provided to developpers by the 2 on-site AMD engineers.
Preliminary tests on micro-applications show good performances of the generated binaries proving that this compiler could be a serious alternative on AMD GPUs and the goal is now to focus on this SDK in an OpenMP strategy while checking the portablility of this new implementation in Nvidia, Cray/HPE (and Gnu ?) environments.

=== Mesh adaptation - A. Grenouilloux, ONERA & G. Balarac, LEGI ===

=== Numerics - M. Bernard, LEGI & G. Lartigue, CORIA ===

==== N2 - Treatment of Inlet conditions in High-Order solver. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====
In the context of node-centered Finite Volumes Method, spacial accuracy of a numerical scheme depends on ability to evaluate accurately fluxes through interface of each control volume (CV). Such accurate evaluation is not straightforward, especially when dealing with distorted grids. This project follows the work of [1] where fluxes use pointwise quantities, which are reconstructed from integrated quantities advanced in time. During the previous edition of the ECFD, a new data structure has been developed to store data at location of the boundary conditions facelets, with application to wall boundary conditions. During this 8th edition of the ECFD, we used the same data structure, but dedicated to the treatment of inlet conditions.
The inlet condition is then either imposed directly at facelets center, or at nodes position them extrapolated to facelets center by use of Taylor expansion. For this later solution, high-order treatment requires the successive derivatives to be computed in the plane of the boundary condition. This is not done yet, leading for the moment to low accuracy results but the framework is ready for upcoming implementation.

[1] ''A framework to perform high-order deconvolution for finite-volume method on simplicial meshes, , Bernard et. al., IJNMF 2020''

==== N3 - Conservative mesh-to-mesh interpolation. M. Bernard (LEGI), Ghislain Lartigue (CORIA), Guillaume Balarac (LEGI) ====

Mesh to mesh interpolations occur quite often in CFD simulations : in the context of adaptative mesh convergence or in the case of dynamic mesh adaptation for for example.
Quality of the solution on the destination grid will depend on the characteristics of the interpolation method.
In this project, we did not focus on accuracy of the interpolation method but rather on conservativity characteristics.
A conservative interpolation ensures that the integral of the data on the source grid is exactly retrieved on the destination grid.
This property is highly interesting when dealing with scalar quantities or phase indicators, whose values should remained bounded.
In the context of nodes centered Finite Volume schemes, the methodology we used consists in (i) reconstructing element quantity from average nodal quantities on source grid.
Then, for a cell of the destination mesh, (ii) computing the geometrical intersection between cells of source and destination meshes to evaluate to evaluate the rate of quantities they.
Eventually, (iii) redistributing the solution from elements to control volumes of the destination mesh.
The overall process is fully conservative as it is based on geometrical intersection of locally integrated quantities.
The methodology as been implemented and tested on a few basic configurations and the conservativity is retrieved.

==== N7 - Implicit time advancement for low-Reynolds number flows with particles. S. Mendez, C. Raveleau (IMAG), M. El Moatamid, V. Moureau (CORIA) ====
IMAG runs numerous simulations of red blood cells under flow. Those simulations are at low Reynolds number (0.001 to 1.0, typically). Splitting of the time advancement is used to treat the diffusion terms implicitly, albeit with an important numerical cost: implicit diffusion is 50 to 60% of the computational cost. Recently, M. El Moatamid implemented a genral framework to deal with implicit time advancement for scalars. In this project, the general method has been transposed to the advancement of the velocity field in the ICS and RBC solvers of YALES2/YALES2BIO. This enables testing various linear solvers (GMRES based). However, such solvers do not decrease the CPU time compared to the existing implementation. However, while working on this, it was identified that residual recycling was not activated in the current implementation of the implicit diffusion. This sped up the implicit diffusion cost by 35%, for a total gain of 20% for the computation. In addition to this achievement, moving to the framework coded by Moncef will have other beneficial side effects: we anticipate simplifying the implementation, with an easier merging between YALES2BIO and YALES2. The method will also be implemented in the electrosatic solver, for which the Poisson problem should benefit from the new GMRES-based solvers. In addition, this project highlights the importance of improving the treatment of stiff source terms in the red blood cells simulations, to be able to overcome the current limitation in time step due to those term and have a chance to benefit from higher-order time schemes, efficient at high Fourier numbers.

=== Turbulence - L. Voivenel, CORIA & P. Bénard, CORIA ===

==== T1 - FSI-1D strategy for internal flows====

==== T2 - Dynamic Smagorinsky in Dorothy ====

==== T3 - Turbulence injection strategy for compressible flows ====

==== T4 - Improve wind farm modeling and simulation workflow ====
The YALES2 library includes an advanced modular implementation of the Actuator Line Method (ALM). This approach remains state-of-the-art when performing an LES-based analysis of a wind turbine wake. The method also provides an accurate assessment of the aerodynamic loads applied on the turbine. Still, applying this method to investigate a wind farm flow can be challenging, both in terms of computational cost and simulation setup. For instance, an inadequate management of the wind turbine individual modeling parts in a HPC context can end up being the main bottleneck of the simulation. From another perspective, a wind farm is usually composed of more than 50 wind turbines. For such a case, setting up all YALES2 required inputs manually can be very tedious and error-prone. This project thus mainly aimed to optimize the YALES2 ALM implementation and the user experience around it. Additionally, a cost-effective alternative to the ALM when modeling wind farm flows, namely the Rotating Actuator Disk Method (ADM-R), has been implemented for further investigations.

'''WP1''': Improve Actuator set rotor modelling
* Parallel processing of the ''actuator sets'' used to model the wind turbines
(Felix)

* Rotating Actuator Disk Model (ADM-R):
According to the usual guidelines, the mesh requirements of the ALM, to profit entirely from its reachable accuracy, can be difficult to achieve or even unaffordable when simulating a wind farm flow, especially from the industrial point of view. Alternatives are available in the literature for this kind of application. Likely, the methods from the Actuator Disk family are the most prominent ones. Several kinds of implementation exist, which mostly differ by their capability to include the wake rotation. During the workshop, a new method from the Rotating Actuator Disk kind has been implemented and underwent an early validation on a single turbine setup. Applications to wind farm flows will follow.

'''WP2''': Improve tools User Experience

Three Python tools have been developed or improved :
*The first tool is the wind farm previsualisation tool, 'y2_wind_previsualization', which is used before the calculation run. This provides an interactive HTML interface for viewing global data for each turbine on the farm (position, hub height, yaw angle, etc.). The tool traces all of these via the parsing of the input file.
* The second tool is for duplicating rotor templates for a wind farm (`y2_wind_duplication`). This tool was developed in the previous ECFD, but this time it has been refactored and incorporated into the y2tools package.
* The third and final tool is a post-processing tool for the temporal processing of global wind turbine simulation metrics (Thrust, Power, etc.), `y2_post_wind`. This tool generates an interactive HTML plot of time-dependent global quantities.

==== T5 - Improve atmospheric inflow turbulence ====
Atmospheric inflow turbulence is generated using the precursor database method. A half-channel flow driven by a pressure gradient is used to obtain the inflow which is used as inlet boundary condition for the wind turbine simulation domain. This project aimed to improve the whole methodology, from generation to injection.

* WP1: Improve inflow generation
Anand: pressure controller

* WP2: Improve injection methodology (method A)
The previous workflow used plane probes in the ASCII format to sample the flow. The COWIT2 toolbox was used to convert the file into turbulence box (.man format). While functioning, this methodology had two major flaws. First the probe files are heavy ~O(10Go). Second, the method requires a lot of human effort, allowing numerous sources of errors.
During this workshop, a new methodology has been developed. First, the probes are generated using the HDF5 format (now available for all probe types), leading to lighter file ~O(1Go). Second, Y2_tools is used to read HDF5 format (working for probes and temporals). HDF5 file is then converted into a Look-up Table. Finally, the Look-up Table is read directly by YALES2 as a boundary conditions.

* WP3: Improve injection methodology (method B)
Even though improvements achieved in WP2 prove to be very handy while removing many potential human errors, injecting a turbulent inflow through wind boxes ('offline' precursor approach) can sometimes remain cumbersome for several reasons: (1) no periodicity is enforced in the streamwise direction of those boxes, (2) potential high memory consumption, and (3) the boxes need to be moved to other cores whenever a mesh adaptation occurs. An alternative consists in co-simulating the precursor flow and the flow of interest (refered as the 'successor' simulation) at the same time ('online' precursor approach). The inlet boundary condition for the successor flow is then obtained by mapping the outflow of the precursor domain. During the workshop, some work has been initiated to implement this kind of coupling using the CWIPI library, for which YALES2 provides already an interface.

==== T6 - FSI model in Dorothy ====

=== Two Phase Flow - J. Leparoux, SAFRAN & J. Carmona, CORIA ===

==== TP1 - Towards very small contact angles in Nucleate boiling ====

Participants: Henri Lam (LEGI), Mohammad Umair (LEGI), Manuel Bernard (LEGI), Robin Barbera (LEGI) and Giovanni Ghigliotti (LPSC)

==== TP2 - Modeling spray-film interactions ====

Participants: Nicolas Gasnier (EM2C-SafranTech), Julien Leparoux (SafranTech), Mehdi Helal (CORIA-SafranTech) and Julien Carmona (CORIA)

==== TP3 - High-fidelity two-phase flow simulations of the purge of a fuel feed line ====

Participants: Thomas LAROCHE (Safran HE), Romain JANODET (Safran AE), Julien Leparoux (Safran Tech) and Melody Cailler (Safran Tech)

==== TP4 - Volume of Fluid solver in YALES2 ====

Participants: Léa Voivenel (CORIA), Julien Carmona (CORIA), Mehdi Helal (CORIA), Pierre Portais (CORIA), Julien Leparoux (Safran Tech), Mélody Cailler (Safran Tech) and Nicolas Gasnier (EM2C / Safran Tech)

==== TP5 - Implement a local operator to distribute the solid volume of a particle over multiple cells ====

Participants: Théo Ndereyimana (Université de Sherbrooke), Stéphane Moreau (Université de Sherbrooke)

==== TP6 - Complex thermodynamics in sloshing tanks ====

Participants: C. Merlin (AGS), D. Fouquet (CORIA), V. Moureau (CORIA), J. Carmona (CORIA) and G. Lartigue (CORIA)

=== Combustion - Y. Bechane, CORIA & S. Dillon, SAFRAN & K. Bioche, CORIA ===

==== C1 - LES of the thermal degradation of a composite material ====
Participants: A. Grenouilloux (ONERA), K. Bioche (CORIA), N. Dellinger (ONERA) and R. Letournel (SafranTech)

==== C2 - Flame stabilization by NRP plasma discharge ====

==== C3 - Extending and validating a generalized formalism of virtual chemistry ====

==== C4 - Turbulent combustion model for NOx prediction ====

==== C5 - Towards 3D simulation of detonation combustion ====

==== C6 - Flame stabilitity of flame-holders within reheat conditions ====

==== C7 - Thermal radiation in oxyflames ====

==== C8 - A first step toward hybrid CPU / GPU for reactive flow in YALES2 ====

Participants: M. Laignel (CORIA), G. Lartigue (CORIA), K. Bioche (CORIA) and V. Moureau (CORIA)
In numerical simulations of reacting flows, one of the most computationally intensive tasks is the evaluation of source terms resulting from chemical reactions in the species transport equations. This step can account for up to 90% of the total simulation cost , depending on the complexity of the kinetic mechanism involved. To reduce this cost, various techniques such as mechanism reduction, virtual chemistry, etc. have been explored. However, the emergence of GPUs as powerful accelerators offers a promising alternative by providing massive parallelism. Despite their potential, GPUs often require significant adaptation of CPU-based codes. This project aims to address this challenge by taking a first step towards a hybrid CPU/GPU framework for reactive flow simulations. Specifically, the focus is on coupling Y2 with the updated version of the stiff time integration solver (CVODE), which is compatible with GPU (CUDA, HIP, OpenMP). The ultimate goal is to establish a foundation for hybrid computations by implementing and testing the updated solver on simplified test cases.

==== C9 - Soots numerical modeling ====

==== C10 - TECERACT : Tabulated chemistry generator for aeronautical combustion ====

==== C11 - Exploring efficient tabulation strategies for detailed chemistry ====

==== C12 - Dynamic sub-grid-scale modelling of multi-regime flame wrinkling ====

==== C13 - LES of a semi-industrial burner using a non-adiabatic virtual chemical scheme ====

=== User Experience & Data - L. Korzeczek, GDTECH ===

==== U1 - Low-fidelity (RANS) rotor/stator simulations, application to Kaplan Turbine - Y. Lakrifi, G. Balarac (LEGI), R. Mercier (SAFRAN), V. Moureau (CORIA) ====

==== U2 - Coupling PyTorch/YALES2, combustion cartesian look-up tables - J. Leparoux, N. Treleaven, S. Dillon (SAFRAN), K. Bioche, G. Lartigue (CORIA) ====

==== U3 - Yales2 Trame Editor, toward a fully featured graphical user interface for YALES2 - L. Korzeczek, S. Meynet (GDTECH), J. Leparoux, M. Cailler (SAFRAN) ====

Ecfd:ecfd 8th edition

2025-02-07T17:04:33Z

Bioche:

Ecfd:ecfd 7th edition

2024-02-12T10:41:46Z

2022-01-31T00:32:39Z

Bioche: /* Combustion - K. Bioche, VUB */

{{DISPLAYTITLE: ECFD workshop, 5th edition, 2022}}

== Description ==
{| align="right" style="text-align:center;" cellpadding="2"
| [[File:Logo_ECFD5.png | center | thumb | 350px | ECFD5 workshop logo.]]
|}
* Event from '''23th to 28th of January 2022'''
* Location: [https://www.bonsejour-laplage.com/vacances-tout-compris Centre Bonséjour], Merville-Franceville, near Caen (14)
* Two types of sessions:
** common technical presentations: roadmaps, specific points.
** mini-workshops. Potential workshops are listed below.
* Free of charge
* More than 50 participants from academics (CERFACS, CORIA, IMAG, LEGI, EM2C, UMONS, UVM, VUB, UCL, TUDelft), HPC center/experts (GENCI, AMD, CINES, CRIANN) and industry (Safran, Ariane Group, Siemens-Gamesa).

* Objectives
** Bring together experts in high-performance computing, applied mathematics and multi-physics CFDs
** Identify the technological barriers of exaflopic CFD via numerical experiments
** Identify industrial needs and challenges in high-performance computing
** Propose action plans to add to the development roadmaps of the CFD codes

== News ==

* 03/11/2021: First announcement of the '''5th Extreme CFD Workshop & Hackathon''' !

[[File:Banniere_ECFD5_sponso.png|text-bottom|600px]]

* 13/01/2022: After discussions with the participants, the '''5th Extreme CFD Workshop & Hackathon''' is maintained as an in-person event! It will be also possible to attend to the plenary sessions and participate remotely to the workshop.

* 14/01/2022: The [[#Agenda|ECFD5 program]] is online! The plenary sessions will be announced soon!

* 20/01/2022: The plenary sessions are now defined:
** P1 - 24/01/2022: GPU porting challenges and quantum computing, présentation machine Adastra by G. Staffelbach (CERFACS) + Presentation of the new cluster from CINES called Adastra by C. Andrieu (CINES)
** P2 - 25/01/2022: News, perspectives and future of GPU computing applied to CFD by A. Toure (AMD)
** P3 - 26/01/2022: Theory, applications and perspectives of the Lattice-Boltzmann Method by P. Boivin (M2P2)
** P4 - 27/01/2022: Concepts and notions of mesh adaptation by C. Dapogny (LJK)

* 23/04/2022: '''The ECFD5 event has now started !!''' [https://www.linkedin.com/feed/update/urn:li:activity:6891053385072594944| LinkedIn post]

== Agenda ==

[[File:ECFD5_program.png|text-bottom|600px]]

== Thematics / Mini-workshops ==

These mini-workshops may change and cover more or less topics. This page will be adapted according to your feedback.

=== Combustion - K. Bioche, VUB ===

* '''Sub-project 1: H2/air jet-in-cross-flow numerical simulations (Romain Le Dortz, Eleonore Riber, Quentin Douasbin)'''

The use of hydrogen as a aviation fuel requires new combustion chamber design. Among strategies to prevent flame flashback and low flame residence time, the micromix injection system is further studied. This systems corresponds to a multitude of H2/air jet-in-cross-flow configurations. A 3D numerical simulation with realistic thermodynamics and kinetics is now tractable thanks to massively parralel computing. This week saw the completion of the first steps towards the establishment of a complete simulation. (I) The non-reactive air injection in the combustion chamber. (II) The cross-injection of H2 without ignition. (III) The ignition of this mixture modeled with the skeletal kinetic mechanism of Boivin (H2, H, O2, OH, O, H2O, HO2, H2O2, N2). Further work will be realised concerning mesh refinement, modelling of NOx and porting of the computation on GPU.

* '''Sub-project 3: Convergence of the interface curvature computation (G. Ghigliotti, J. Carmona, G. Balarac, G. Lartigue)'''

=== Static and dynamic mesh adaptation - G. Balarac, LEGI ===

=== Multi-phase flows - M. Cailler, SAFRAN TECH ===

* '''Sub-project 1: Hybrid E-E/E-L two-phase flow method (M. Cailler, F. Pecquery, I. El Yamani, V. Moureau)'''

The High-Fidelity approach based on ACLS & DMA allows a reliable description of interface dynamics. For design exploration, low-CPU methods with controlled level of fidelity are required. An interesting approach to reduce CPU cost relies on an hybrid approach based on an Eulerian representation of the gas & and a Lagrangian description for the liquid. Objective of the ECFD5 was to explore the capability to reconstruct the interface normal on the Eulerian grid. A level-set based strategy relying on Geometric Multiple Markers Projection (Janodet et al., 2022) has been first tested showing good capabilities providing that the iso-surface distance equal 0 is captured. An alternative strategy based on the liquid volume fraction has been tested.

* '''Sub-project 3: Convergence of the interface curvature computation (G. Ghigliotti, J. Carmona, G. Balarac, G. Lartigue)'''

The computation of interface curvature in a level-set framework is based on the classic formula as divergence of the gradient of the levelset function. This function being computed at 2nd order, one obtains a O(0) curvature, meaning that the error does not decrease with mesh refinement.
We have implemented in YALES2 a strategy proposed by Emmanuel Maître and collaborators in a finite element method based on the regularization (filtering) of the level-set gradient and curvature.
This strategy has been tested for the simple test case of a static circular interface.
We used two types of filters (simple gather-scatter or bilaplacian as developed by Lola Guedot (PhD thesis 2015)) on different mesh types (split quadrilaterals, isotropic triangular mesh, unstructured triangular mesh).
The results are encouraging since a O(1) convergence is obtained in all cases.
Further work is needed to tune the filter properties (amplitude and size) for different spatial resolutions and levelset "narrow band" width.

* '''Sub-project 4: Conservative two-fluid momentum transport (F. Pecquery, C. Merlin, M. Cailler, J. Carmona, V. Moureau)'''

=== Numerics - G. Lartigue, CORIA ===

=== Turbulent flows - P. Bénard, CORIA ===
* '''Sub-project 1: Optimization of the actuator set for several wind turbines in YALES2 (F. Houtin Mongrolle, S. Gremmo, E. Muller & B. Duboc)'''

* '''Sub-project 5: TBLE wall model for LES with pressure gradient on a simple turbomachinery geometry (M. Cizeron, N. Odier, R. Vicquelin)'''
Wall modeling is often used in LES to alleviate the computational cost that would be required to resolve all the length scales up to the solid boundaries of the domain. The classical way of doing it is by using an algebraic model to provide the wall friction and heat flux, with a coupling to the LES solver at the first off-wall nodes. The wall model was designed from analyzing RANS equation with strong assumptions such as planar flow, equilibrium and no pressure gradient. These assumptions are often far from true in real applications, such as turbomachinery applications, where the use of a wall model is mandatory due to the size of the calculation. During this workshop, a wall model relying on the resolution of the Thin Boundary Layer Equations (TBLE) was studied, which had been implemented by EM2C. The addition of a pressure gradient to these equations has been conducted and tested, at first only for the 1D wall model solver, then on a 3D turbulent channel. It remains to be tested on a diffuser configuration with a real pressure gradient to quantify the effect of the new wall model. The influence of the point considered to do the coupling between the LES and the wall model (ie. its distance to the wall) has also been tested both for the TBLE and the original algebraic model, showing that coupling farther from the wall yields better results and reduces the so-called log-layer mismatch.

* '''Sub-project 6: Tools for rough wall modelling (A. Barge, S. Meynet)'''

=== Compressible - L. Bricteux, UMONS ===

=== User experience - J. Leparoux, SAFRAN TECH ===

=== Hackathon - G. Staffelbach, CERFACS ===
* '''Participants: I. d'Ast, J. Legaux, G. Staffelbach, P. Begou, G. Lartigue, V. Moureau, A. Toure, C. Laurie, S. Delamare, C. Andrieu, C. Jourdain'''
AMD GPU hardware is still relatively unknown in our CFD community. This hackathon was the opportunity to deep dive into the AMD dev environment to prepare the arrival of AdAstra at CINES.
Both YALES and AVBP have been ported to the AOMP framework using ROCm 4.5 on the GRID5000 Neowise system.
CPU execution posed no issues and we were able to focus on GPU Offloading using OpenMP.
On the YALES2 side, a mini-app encompassing the typical YALES2 structure hierarchy and loop execution was extracted from the code to evaluate different porting strategies and on the AVBP side the current OpenACC GPU offloading was translated to OpenMP focusing on the viscosity computation kernel.
We learnt that the current supported standard of OpenMP in ROCm 4.5 does not allow for direct offloading of reference values inside an derived type structure but is was possible to use aliases such as pointers or flat array copies to do the job. This should be solved with the support of OpenMP 5.0
Another troublesome issues, was the lack of support for offloading of array vector operations (ex : array(:) = 1.0 ) rendering the explicitation of the loops for these manadatory.

Some bugs remain and it is encouraged to use the latest compiler version when working on the porting ( the release of flang 14.0.1 saved us a lot of time as we had started with 14.0.0 ).
Offloading of the miniapp of YALES2 yielded a times 60 acceleration of the kernel whereas the offloading of the viscosity model in a full avbp simulation yielded an 7 times factor in performance when comparing on core to one GPU. These results are to be taken with a grain of salt but are really encouraging.

For the next steps, a porting strategy for both codes will be implemented (depending on the OpenMP 5 support ) and discussions are underway with CINES and other partners so as to offer the best experience to both code's communities on AdAstra at its release.