By Andrew C. Kirby and Dimitri J. Mavriplis
Figure 1. A wind farm in Colorado. The Great Plains states have the best onshore wind resources in the U.S. Image courtesy of
Laura Ockel on Unsplash.
The physics of fluids that span disparate spatial and temporal scales is ubiquitously important. This is particularly true for wind energy and aerospace science applications, which couple interfacial dynamics that involve aerodynamics, aeroelasticity, and atmospheric phenomena. Understanding and ultimately predicting these kinematics is of practical significance for industry and society. However, the direct resolution of all of these scales via numerical simulation is severely intractable; for example, the largest turbulent eddies on a wind farm are characterized by regional weather patterns that cascade down to the smallest eddies, where kinetic energy is dissipated into heat at the level of
Kolmogorov microscales. Researchers generally use highly nonlinear Navier-Stokes equations to describe the governing physics for a complex viscous fluid; no generalized closed-form solutions exist due to this nonlinearity. Yet despite this problem’s intractable nature, the optimal prediction of these chaotic flow fields is of the utmost importance. It is here that the field of computational fluid dynamics (CFD) comes into play.
CFD employs numerical methods—most commonly the finite difference, finite volume, or finite element methods—to discretize the governing equations with prescribed initial and boundary conditions by converting the partial differential equations into a system of algebraic equations. The technique divides the three-dimensional fluid domain of interest (such as the wind farm in Figure 1) into a grid of points or control volumes in which one of the aforementioned numerical methods locally approximates the Navier-Stokes equations.
Figure 2. High-fidelity wind turbine simulations employ body-fitting grids to accurately represent the complex geometry. Simulations use multiple unstructured grids to represent the wind turbine blades and tower, as well as a structured grid for the space around and away from the wind turbine. In each grid box/element, the governing equations of fluid flow are solved numerically. 2a. Overset mesh technology incorporates complex geometry and relative-body motion. 2b. A dynamic mesh adaption captures wakes in a high-fidelity wind turbine simulation. Figure 2a courtesy of [6], Figure 2b courtesy of [7].
If we were to divide the computational domain of a wind farm using a uniform grid to capture all of the scales for typical flows over a wind turbine, the spatial physics alone would require more than \(10^{30}\) discretization points — a quintillion times greater than what is currently possible! Of course, not all areas of the domain necessitate this level of resolution; in fact, much of the flow away from the terrain and wind turbines varies slowly in time and space, since turbulence is often generated and convected in localized regions. We can therefore strategically place grid points to capture the most meaningful fluid dynamics and obtain a reasonable solution. This tactic defines the concept of nonuniform grids. The state of the art in nonuniform grids is called adaptive mesh refinement (AMR), in which the grid changes dynamically in space during a simulation to refine and coarsen the mesh such that it captures the dominant turbulent flow features.
We have developed a computational framework called WAKE3D [6] that simultaneously uses multiple CFD solvers and meshes to support a dynamic overset mesh system. The computational mesh system consists of a collection of overlapping near-body and off-body meshes; the unstructured near-body meshes model the complex geometry and resolve aerodynamic boundary layers, while the off-body meshes comprise a dynamically adaptive structured grid system (see Figure 2). Figure 2a depicts the two meshes: the turquoise grid is a near-body unstructured mesh that models the complex wind turbine tower and nacelle, and the green grid is an off-body structured dynamically-adapting mesh that resolves the incoming atmospheric flow and tracks turbulence from the aerodynamic bodies (see Figure 2b for a more detailed depiction of the latter mesh). In regions of mesh overlap, we utilize an overset grid assembler called TIOGA (Topology Independent Overset Grid Assembler) [1] that automatically determines the best solution amongst the meshes and transfers the information between solvers.
Figure 3. Morton ordering of octree leaves in a forest of octrees. Mesh partitioning of the forest (across message passing interface processes \(\textit{p0}\), \(\textit{p1}\), and \(\textit{p2}\)) is accomplished by assigning user-input integer weights to each octant leaf and distributing loads equally across all processes. Figure courtesy of [2].
To enable dynamic AMR, we leverage an octree-based software known as p4est [2]. The p4est library enables parallel management of a collection of octrees—conveniently named a forest of octrees—that consists of either a set of quads (two dimensions) or hexes (three dimensions). Using the message passing interface (MPI), the library scales to millions of computing cores via its natural partitioning schedule as well as a critical feature known as the Morton space-filling curve for ordering octants (see Figure 3). Users can assign each octant leaf with a weight that represents an approximate computational cost to ultimately create a partition schedule that has an efficient parallel compute scalability for the CFD solver.
Figure 4. Our Lillgrund Wind Farm simulation uses WAKE3D on 22,464 central processing unit cores that are composed of 1.55 billion degrees of freedom. Figure adapted from [6].
The CFD solver in WAKE3D that utilizes
p4est is called dg4est [5] and comprises an
hp-adaptive discontinuous Galerkin finite element method. Here,
h-adaptive corresponds to the nonconforming grid adaption via mesh element subdivision and
p-adaptive corresponds to the variable approximating polynomial order of the discretization within a mesh element. Nonconforming meshes and variable solution order can cause severe workload imbalances across octants, but by providing accurate octant integer loads, we can reduce the wall-clock coefficient of variation (ratio of standard deviation to the mean) across MPI processes to less than five percent. Doing so improves computational throughput by more than 35 percent when compared to uniformly weighted octant loads for problems that are distributed across tens of thousands of central processing unit cores with billions of solution unknowns. This improvement stems from the fact that traditional CFD methods are synchronized at each time step and thus limited by the slowest computing MPI process. Dynamic AMR naturally incurs overhead wall-clock time for a simulation due to mesh element refinement tagging, balancing, and partitioning. These processes, which typically comprise between 20 to 35 percent of the total wall-clock time for other AMR frameworks, amount to less than four percent for our simulations with
p4est! As such, we can obtain more scientific throughput and scale up our computational science, enabling the use of full-scale blade-resolved wind farm simulations — like the Lillgrund Wind Farm off the coast of Sweden, which contains 48 wind turbines (see Figure 4). We can also study the fundamental turbulent dynamics within the wind turbine wake [3, 7] (see Figure 5).
Figure 5. A wind turbine wake simulation that is visualized by tangential flow velocity. These turbulent flow structures propagate downstream, impacting the performance and structural response of other wind turbines. Figure courtesy of [8].
AMR is an essential technology for the future of high-fidelity aerodynamic simulations. In addition to simulating wind energy, we have also leveraged AMR technology to perform high-fidelity simulations of fixed-wing aircraft [9] and rotorcraft [4]. The landscape for AMR development is bright; the p4est framework continues to improve and a p4est spinoff for arbitrary element types is currently under development. Moving forward, we are honing our CFD solvers to extensively leverage these frameworks in tandem with tomorrow’s heterogeneous supercomputers in our quest to unlock the oldest unsolved secret in physics: turbulence.
Andrew Kirby delivered a minisymposium presentation on this research at the 2023 SIAM Conference on Computational Science and Engineering (CSE23), which took place in Amsterdam, the Netherlands, earlier this year. He received funding to attend CSE23 through a SIAM Early Career Travel Award. To learn more about Early Career Travel Awards and submit an application, visit the online page.
References
[1] Brazell, M.J., Sitaraman, J., & Mavriplis, D.J. (2016). An overset mesh approach for 3D mixed element high-order discretizations. J. Comput. Phys., 322, 33-51.
[2] Burstedde, C., Wilcox, L.C., & Ghattas, O. (2011). p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees. SIAM J. Sci. Comput., 33(3), 1103-1133.
[3] Edmonds, A.P., Hassanzadeh, A., Kirby, A.C., Mavriplis, D.J., & Naughton, J.W. (2019). Effects of blade load distributions on wind turbine wake evolution using blade-resolved computational fluid dynamics simulations. In Proceedings of the AIAA Scitech 2019 Forum (AIAA Paper 2019-2081). San Diego, CA: American Institute of Aeronautics and Astronautics.
[4] Kara, K., Brazell, M.J., Kirby, A.C., Mavriplis, D.J., & Duque, E.P. (2020). Hover predictions using a high-order discontinuous Galerkin off-body discretization. In Proceedings of the AIAA Scitech 2020 Forum (AIAA Paper 2020-0771). Orlando, FL.: American Institute of Aeronautics and Astronautics.
[5] Kirby, A.C. (2018). Enabling high-order methods for extreme-scale simulations [Ph.D. thesis, Department of Mechanical Engineering, University of Wyoming].
[6] Kirby, A.C., Brazell, M.J., Yang, Z., Roy, R., Ahrabi, B.R., Stoellinger, M.K., … Mavriplis, D.J. (2019). Wind farm simulations using an overset hp-adaptive approach with blade-resolved turbine models. Int. J. High Perform. Comput. Appl., 33(5), 897-923.
[7] Kirby, A.C., Hassanzadeh, A., Mavriplis, D.J., & Naughton, J.W. (2018). Wind turbine wake dynamics analysis using a high-fidelity simulation framework with blade-resolved turbine models. In Proceedings of the 2018 Wind Energy Symposium (AIAA Paper 2018-0256). Kissimmee, FL: American Institute of Aeronautics and Astronautics.
[8] Kirby, A.C., & Mavriplis, D.J. (2020). GPU-accelerated discontinuous Galerkin methods: 30x speedup on 345 billion unknowns. In Proceedings of the 2020 IEEE High Performance Extreme Computing Conference (HPEC). Institute of Electrical and Electronics Engineers.
[9] Mavriplis, D.J., Bogstad, M., & Kirby, A.C. (2022). RANS and hybrid RANS-LES results for the fourth high-lift prediction workshop using the NSU3D solver. In Proceedings of the AIAA AVIATION 2022 Forum (AIAA Paper 2022-3810). Chicago, IL: American Institute of Aeronautics and Astronautics.
 |
Andrew C. Kirby is an associate research scientist in the School of Computing at the University of Wyoming. His research blends the tools of high-performance computing and computational science to solve challenging problems in wind energy and aerospace applications. Prior to his current position, he was a research scientist with Dimitri Mavriplis' small business Scientific Simulations LLC. He performed postdoctoral research at the Massachusetts Institute of Technology Lincoln Laboratory after completing his doctoral studies in mechanical engineering at the University of Wyoming in 2018. Andrew resides and works hybrid in Boston, MA. |
 |
Dimitri J. Mavriplis is an emeritus professor of mechanical engineering at the University of Wyoming and the CEO and founder of Scientific Simulations LLC. He is a Fellow of the American Institute of Aeronautics and Astronautics and former chair of the NASA CFD 2030 Integration Committee. He worked at the Institute for Applied Mathematics and Computer Science at NASA Langley Research Center for 16 years prior to joining the University of Wyoming in 2003. Mavriplis holds a Ph.D. in mechanical and aerospace engineering from Princeton University. |