Latest revision as of 08:11, 3 June 2021

Sponsors

Participants

Flyer

• Flyer

Presentations

• Introduction workshop
• Introduction GENCI
• Roadmap AVBP (HPC)
• Roadmap YALES2

Hackathon white paper

• White paper on AVBP and YALES2 porting on AMD Rome

Booklet

• Template

Project achievements

Project #1: Hackathon GENCI/ATOS/AMD/CERFACS on AVBP

C. Piechurski (GENCI), S. Jauré (ATOS), B. Pajot (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G. Staffelbach (CERFACS), J. Legaux (CERFACS)

We ported the AVBP solver to the AMD Rome system available at GENCI -TGCC ( IRENE Joliot Curie). Characterisation of the application on the architecture showed a 1/3 performance dependency to bandwidth and 2/3 to compute. Strong scaling performance up to 130k cores was measured with openmpi and provided an acceleration of 75% without optimisations. Weak scaling up to 32k MPI ranks suggests that decimation of the processes by a factor 2 improves computational efficiency by up to 30%. This suggests a trade off between mpi imbalance and decimation is possible if imbalance is higher than 30% to improve time to solution.

Currently Openmpi offers the best perfofrmance, intelmpi is still a bit unstable.

During the Hackathon we also introduced colour based cache blocking using ColPack in the code in order to use OpenMP without critical sections. On a 2x18 core Skylake processor the new implementation offered similar speedup using full threading versus full MPI with the best trade off being 4 MPI and 9 threads per MPI. On AMD Rome, Full threading did not offer much acceleration and needs to be inversigated but 8 MPI and 16 threads per MPI seem quite promising.

Final presentation of project #1

Project #2: Hackathon GENCI/ATOS/AMD/CORIA on YALES2

C. Piechurski (GENCI), S. Jauré (ATOS), P.-A. Harraud (AMD), P. Mohanamuraly (CERFACS), G.Lartigue (CORIA), F. Gava (CORIA), K. Bioche (CORIA), P. Begou (LEGI)

During this Hackaton we measured the performances of Yales2 on the AMD Epyc Rome processors.
In particular we focused on scalability and in finding the execution time dependency on memory bandwidth and cache effects.
After removing the MPI contribution, we found that there is a 49% dependency on the bandwidth, a 33% dependency on cache effects, leaving 18% on CPU frequency, prefetcing, etc.
This clearly explains the fact that, with the proper binding, using 2 half filled nodes gives a computation time that is about 50% that of 1 full node.
We also compared the performances on AMD nodes with Intel Skylake nodes on a non-reactive simulation.
Our findings shows that on full nodes the two architectures gives almost equivalent performances. However, AMD seems to perform better when using half-empty nodes.
On the side, we realised that switching from OpenMPI to IntelMPI and setting I_MPI_FABRICS=shm:ofa, whose equivalent is seemingly active by default for OpenMPI, divides almost by a factor 2 the time spent in MPI exchanges on Skylake nodes.

We also developed a simple profiling tool in Yales2, which will allow to measure and analyse more precisely the performances of the code with little intrusion and overhead.

Final presentation of project #2

Project #3: Implementation of a secondary atomization model in YALES2

C. G. Guillamon (Safran Tech), L .Voivenel (Safran Tech), R. Mercier (Safran Tech)

In Lagrangian simulations, droplets are transported following a ballistic motion in an eulerian mesh. For non-reactive environments, droplets might undergo secondary atomization due to the aerodynamic interaction. In this work, we implement in YALES2 a breakup model known as Taylor-Analogy Breakup (TAB). This model is based on the analogy between a droplet and a second-order mechanical system, hence making possible to determine the breakup behaviour by means of Newton's second law.

Another model, the stochastic breakup model by Gorokhovski, is also suggested for future work and will be implemented in YALES2.

Final presentation of project #3

Project #4: Conservative Heat Transfers in the the Accurate Conservative Level-Set framework

François Pecquery (ARIANE GROUP), Mélody Cailler (SAFRAN TECH), Romain Janodet (SAFRAN TECH/CORIA) and Vincent Moureau (CORIA)

Objectives of the project was to introduce conservative heat transfers in the Accurate Conservative Level-Set framework to be able to describe heat transfers and liquid dynamics in an accurate, robust and conservative manner. A Multi-Phase Transport framework relying on the conserving and level-set coherent transport of the temperature is introduced on the Spray solver. The solution is to use the fluxes of a phase indicator that may be sharp, contrarily to the level-set. The new solver was used on a simplified test case where a liquid droplet is transported in a temperature stratified environment. Results show promising capabilities of the new framework. Next work include improvement of the transport equation stability, and of the jump condition at the interface.

Final presentation of project #4

Project #5: Jet-in-crossflow par une méthode d’interface diffuse

T. Laroche, N. Odier, B. Cuenot (CERFACS). In collaboration with M. Pelletier, T. Schmitt, S. Ducruix (EM2C)

In the context of fuel injection in an aircraft engine, liquid fuel is injected through a swirler, and sheared by a high-speed oxyder which destabilizes the liquid interface. This interaction induces liquid ligaments, which break up into large droplets (primary atomization), and then themselves break into small droplets (secondary atomization) This project deals with the implementation of a diffuse-interface method in the massively parallel solver AVBP to represent the liquid interface destabilization during primary atomization for compressible applications. This methodology is found to be very efficient, however a control of the interface diffusion is mandatory as soon as convective effects are added. During this workshop, the methodology proposed by Chiodi and Desjardins ( A reformulation of the conservative level set reinitialization equation for accurate and robust simulation of complex multiphase flows, JCP 2017) to control the interface thickness has been implemented in AVBP, and is currently under validation on a periodic liquid jet with surface tension effects.

Final presentation of project #5

Project #6: Accurate numerical predicti􏴇on of vorti􏴇cal flows using AMR

Laurent Bricteux (U. MONS)

We try to demonstrate that Eulerian method YALES2 using AMR can do a very good job to capture complex vortical flows at moderate Re=10k Here we use an AMR strategy based on vorticity. We investigate the problem of vortex ring collision. We have a gain of 1000 on the numbers of elements compared to a non adaptative approach. We are able to capture the transition from a very simple laminar flow to a complex turbulent flow.

Final presentation of project #6

Project #7: Modélisation de parois pour la simulation des grandes échelles

A. Barge(LEGI), P. Bénard(CORIA) and G. Balarac(LEGI)

We ran simulations to benchmark the Dynamic Slip Wall method from (Bose & Moin, PoF, 2014) that we implemented in YALES2. The performances of the method have been compared with those from the classic Log-Law and the method from (Duprat et al., PoF, 2011) on the standard cases of channel flow and periodic hills. The results from the Dynamic Slip wall showed a satisfying efficiency although its precision stays comparable to the method from Duprat et al. Also, we set up the non-stationary test case of oscillating channel flow to enrich the benchmarking with the above-mentioned methods. Finally, we started to work on the implementation of a new approach proposed by M. Gorokhovski, tests are in progress.

Final presentation of project #7

Project #8: Accurate numerical simulation of contact lines with dynamic mesh adaptation

S. Pertant (LEGI), G. Ghigliotti (LEGI), G. Balarac (LEGI)

The main objective of this project was to develop a methodology to simulate contact lines on unstructured meshes. We especially wanted to get rid of mesh influence on contact line movement when the flow is driven by surface tension and the contact line close to its equilibrium position. A slight modification in the Ghost Fluid Method to apply the pressure jump has been tested and seems promising. The pressure gradient at contact line is indeed less sensitive to mesh elements for high density ratios. Furthermore, dynamic mesh adaptation has been used to simulate a 2D vapour bubble lying on a wall. Due to gravity, the two contact lines are receding until their merging and the bubble departure. The mesh remains fine to capture the contact line dynamics. As a future work, we plan to perform mesh adaptation on 3D contact line cases and to include additional physics such as contact angle imposition (already implemented but not used yet with mesh adaptation).

Final presentation of project #8

Project #9: Remeshed particle method at high Schmidt and Reynolds number

S. Santoso (LJK), J.-B. Lagaert (Math Orsay), G.Balarac (LEGI)

We study the advection of a scalar function in turbulent flows with a multimesh method. The finite volume method is used to solve Navier-Stokes equations on an unstructured mesh (YALES2). The advection equation is solved with remeshed particle method on a cartesian mesh. In the context of parallel computing, we face a very unbalanced problem since a large number of particles are created in a very fine meshed zone. Our strategy to load-balance the problem is to give a weight to every element group which is equal to the density of particle.

Final presentation of project #9

Project #10: Adaptive mesh refinement for turbulent premixed combustion

W. Agostinelli, O. Dounia, , T. Jaravel, O. Vermorel (CERFACS)

The objective of the project was to evaluate the potential of adaptive mesh refinement (AMR) for premixed combustion in unsteady systems. Three target cases were identified: a semi-vented deflagration with laminar to turbulent transition, a planar detonation wave, and a bluff-body stabilized burner subjected to thermoacoustic oscillations. The simulations were performed with AVBP and coupled to the AMR implementation of YALES2. Several metrics and remeshing criterions were developed to identify and correctly resolve both the combustion wave front and the turbulent flow. The comparison of numerical results with reference simulations showed that the main features of the physics could be recovered with a significant speed-up in term of computational cost.

Final presentation of project #10

Project #11: Multiphysics coupling for wind turbine wake modeling

F.Houtin-Mongrolle (CORIA), B. Duboc (Siemens Gamesa Renewable Energy), P. Benard (CORIA)

The goal of this project was to implement the coupling of YALES2 (flow solver, using Actuator Lines) with BHawC(Aero-Servo-Elastic solver developed in SGRE). The objective is to get an accurate prediction of the wake physics with the LES while also modeling the deformations of the wind turbine and its whole control system, which is essential to simulate a realistic behavior of the machine.

The coupling is done by exchanging data with MPI. Several executables of BHawC (1 sequential executable per turbine) are launched together with YALES2. During the temporal loop, BHawC sends the positions and velocities of the structural nodes and impose them on the particles of the actuator lines. YALES2 calculate the forces on those particles and send them back to BHawC, and updates the flow.

So far, the communications between the codes have been implemented. The simulation is correctly initialized with several turbines and the coupled code runs the temporal loop. A small simulation on 3 aligned turbines showed a good computational efficiency of this coupling. However, the timestep of YALES2 is one order of magnitude lower than what is expected by the CFL criterion, because it is now imposed by BHawC. In a future work, the proper sub-stepping will be implemented to decrease the computational cost of the simulations.

Final presentation of project #11

Project #12: Stability of a semi-implicit compressible cavitation solver

H. Garg (LEGI), G. Ghigliotti (LEGI) and G. Balarac (LEGI)

The compressible cavitation solver is used to simulate cavitation inception in an initially liquid flow behind an obstacle. This solver is based on the implicit compressible solver, that has been modified to include a « barotropic » pressure-density relationship playing the role of an equation of state independent from the temperature. While this strategy has proven to be effective for DNS simulations of the implosion of vapour bubbles, the simulation of cavitation inception in an initially liquid flow was leading to strong instabilities in the simulation shortly after the appearence of vapour. The test case chosen is a flow behind a 2D cylinder. The analysis of the results has shown that instabilities were correlated with very low (and even unphysically negative) values of the pressure, that were triggering negative density values leading to code instability. Using limiters to ensure a positive pressure and a density within the range of the equation of state improved the stability and allowed to perform a preliminary simulation of a cavitating flow behind an obstacle. Ultimately instabilities appear anyways, so that the will look to the spatial discretisation schemes.

Final presentation of project #12

Project #13: Validations and comparisons of Diffuse / Sharp interface methods in a structured DNS solver (Titan)

V. Boniou (EM2C), J.M. Dupays (EM2C), M. Pelletier (EM2C), T. Schmitt (EM2C), A. Vié (EM2C)

The project aimed at using academic test cases to compare the sharp (incompressible) and diffuse (compressible) models. In particular, the test case of an inviscid initially elliptical oscillating droplet has been carried out. The solvers features are the following:

- incompressible VOF solver (sharp): Numerical Method: Projection Method, Interface reconstruction: VOF, Surface tension: CSF

- compressible multifluid solver (diffuse): Advection scheme: MUSCL + RK2 + minmod limiter, Surface tension: CSF.

The source term is integrated with operator-splitting, and the curvature computation relies on a 2nd-order differentiation of the liquid volume fraction, which is previously smooth by filtering. This test case showed good agreement on the oscillation period, while exhibiting a slight numerical diffusion in the incompressible case and a strong numerical diffusion in the compressible case. In the compressible case, the use of higher-order splitting (Strang [SIAM Num. An. 1968]) has been tested, yielding no noticeable improvement. Reduction of the number of filtering iterations on the liquid volume fraction provides a slight improvement, which may indicate that a better curvature computation could participate to reduce the numerical diffusion.

Final presentation of project #13

Project #14: High Order Framework

M. Bernard (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)

Aim of this project is to extend the high order framework (HOF) in Yales2. As a reminder, the HOF permits to reconstruct a point-wise quantity from the volume-averaged one, arising from classical Finite-Volume schemes, and thus to improve spatial accuracy of numerical schemes.

During the ECFD workshop #3, a dedicated solver has been created, the high order solver (hos), duplicated from the incompressible solver (ics). We started activating the HOF ingredients previously developed, starting from velocity field advancement. Development is still in progress, but the static Taylor-Green vortices test-case has been investigated in order to see the early improvement.

Final presentation of project #14

Project #15: Validation of a fluid structure interaction case with the coupling ALE/SMS

T. Fabbri (LEGI), G. Lartigue (CORIA), G. Balarac (LEGI), V. Moureau (CORIA)

The objective of this project was the validation of the Turek(2006) benchmark for fluid structure case. The Structural Mechanics Solver (SMS) was already existing before the workshop, as the coupling with the Arbitrary-Lagrangian Eulerian solver. However, the results were not in agreement with the case. The data compared here are the flexible part tip displacement, but also the drag and the lift integrated on the cylinder and the flexible part. The pure structure test cases were validated, but the forces computed for the pure fluid test cases were not satisfying. The work of this week was then to improve the viscous shear computation, which implies the wall normal gradient computation.

Final presentation of project #15

Project #16: Development of a RANS solver in YALES2

G. Sahut (LEGI), G. Balarac (LEGI), V. Moureau (CORIA), G. Lartigue (CORIA), P. Bénard (CORIA), A. Grenouilloux (CORIA)

While the accuracy of LES usually approaches the one of DNS, LES are still too time-consuming for daily use in industrial applications. In this context, we started the development of a RANS solver in YALES2. We are first only interested in the steady state of the solution. In order to remove the CFL constraint, we developed, implemented and validated an implicit projection method for the resolution of the Navier-Stokes equations without turbulence models. The method is based on the implicitation of the velocity predictor ; the Poisson equation and the correction step of the velocity are then solved and applied as in the explicit incompressible solver. We validated the method on a stationary 2D Poiseuille flow with periodic boundary conditions: the simulation runs fine for CFL and Fourier numbers which are inaccessible with the explicit incompressible solver. The advection-diffusion equation for scalars has also been implicited and will be used to add turbulence models to the new implicit incompressible solver developped during this Workshop. More complex boundary conditions will also be addressed in a near future.

Final presentation of project #16

Project #17: IMPLEMENTATION OF A COLD PLASMA MODEL IN YALES2

J.-M. Orlac'h (EM2C), G. Lartigue (CORIA), B. Fiorina (EM2C)

The objective of this project was to further develop the cold plasma solver in YALES2 in order to accurately model silane nanodusty discharges. The electron temperature equation has been implemented successfully and validated against a reference plasma code. In a second step, a detailed electron kinetics has been implemented in YALES2 in order to couple the electron temperature with the charged species mass fractions. The user can now define a list of reactions whose rates depend on the electron temperature. These improvements open the path to the simulation of nanoparticle production in silane discharges using a Lagrangian description for the nanoparticles.

Final presentation of project #17

Project #18: L’Evaporo O Maître

"C. Nguyen Van (EM2C - Safran-Tech), J. Leparoux (Safran AE), H. Musaefendic (Safran AE), R. Mercier (Safran-Tech), B. Fiorina (EM2C)"

This project raises from industrial and academic needs to easily assess fuel evaporation characteristics from the simplest evaporation models to the most complex. The objectives were to :

- Reintegrate recent development related to multi-components evaporation with complex transport properties have been made during the thesis of H.Larabi

- Use the full capacity of the YALES2 solver in order to make a tool addressing evaporation assessment needs

- Encapsulate a test case in a python module

Performing a state of the art of evaporation modelling in YALES2 as well as development realised, combinated with the tool desired showed that the actual framework is not fully adapted to address the objectives. Creation of a new structure ms_droplet is under development.

Final presentation of project #18

Project #19: The Clone Wars

H. Maldonado Colman (EM2C), C. Nguyen Van (EM2C - Safran-Tech), R. Mercier (Safran-Tech), B. Fiorina (EM2C)

The aim of this project was to increase the computation performance using virtual chemistry approach in the YALES2 solver.

In order to reach this goal three test cases where identified:

- 1D laminar premixed flame (methane / air combustion with carbon monoxide prediction)

- 2D laminar premixed bunsen flame (methane / air combustion with carbon monoxide prediction)

- 3D two phase and turbulent flame (nheptane / air combustion with nitrogen monoxide prediction)

Several ways were explored:

- Profiling of reactive simulations when using Virtual Chemistry

- Effect of redundant species transport

- Effect of the size and the numbers of jacobian matrix to compute and solve

- Effect of the correction functions smoothing

Conclusions of the study are:

- String trimming and concatenation heavily affect computing performances

- Redundant species transport and source terms computations has a minor impact on performances

Final presentation of project #19

Project #20: Stiff complex fluid simulation with YALES2

Sam Whitmore, Yves Dubief, M2CE, University of Vermont

The objective was to simulate (1) ionized gases and (2) polymer solutions in flows using YALES2. Both problems are challenging owing to their stiff thermodynamics (1) or polymer dynamics (2). Significant gains were achieved in the implementation of the respective models thanks to the stiff integrator library CVODE. The plasma flow demonstrated an increase in time step of two orders of magnitude compared to previous implementation of the plasma chemistry in the variable density solver. Polymer models are notoriously prone to numerical instability. Again the use of CVODE showed equivalent if not superior stability of the solution at a fraction of the cost of commonly employed algorithms designed to address the stiffness of the problem.

Final presentation of project #20

Project #21: AVBP Dense Gases

Paolo ERRANTE (LMFA), Alexis GIAUQUE (LMFA), Christophe CORRE (LMFA)

The simulation of dense gas flows using AVBP currently relies on the Martin-Hou Equation of State (EoS) to obtain the thermodynamic variables in each grid cell from the local value of density (or specific volume v) and internal energy derived from the conservative variables. The project develops an alternative approach where thermodynamic quantities in each cell are derived from a (given) set of tabulated thermodynamic states (Look-up Table or LuT). In order to preserve flow physics, the interpolation process in the LuT tables must be performed in a consistent way (a simple bilinear interpolation on v and T for each thermodynamic variable is not sufficient). Describing Helmholtz free energy f(v,T) with a bi-quintic Hermitian polynomial function in each cell of the LuT allows to ensure a consistent interpolation process (since all thermodynamic variables are obtained by differentiating the polynomial function). During the workshop the functions needed to perform the consistent interpolation have been implemented in the real gas module. Short-term perspectives are completing the implementation, validating the development on some test-cases previously computed using MAH EoS and optimizing the implementation (in particular the strategy used to identify the position in the LuT of each local grid state).

Final presentation of project #21

Project #22: Numerical prediction of wind turbine wakes using AMR

S. Zeol (UMONS)i, P. Bénard (CORIA), G. Balarac (LEGI), L. Bricteux (UMONS)

The project considered here demonstrates the feasibility of the use of an adaptive mesh refinement method in the Eulerian finite volume code YALES2 for wind turbine wakes prediction. The objective is to determinate the more effective methodology to adapt the mesh based on appropriate criterion. We consider two methods : one for statistically steady flows based and one for purely unsteady flows (e.g. turbine with yaw, wind turbine with strong turbulence, inducing wake meandering) Preliminary results were obtained on a testcase for which wind tunnel data are available: the NTNU blind test 1. The static adaptation method applied on this case produced promising results and should eventually reduce the computational cost of this kind of simulations. The dynamic adaptation method has been elaborated and some tests were performed to find the best adaptivity parameters. The next step is to fully validate the methodology and consider then a more challenging test case with yaw adaptation.

Final presentation of project #22