In order to execute the individual solver runs, we are going to employ the mini batch processor, for running the calculations on the local machine. We also have to initialize the workflow management system and create a database.
Note:
convStudy.ipynb.
One can directly load this into Jupyter to interactively work with the following code examples.BoSSSpad.dll is required.
You must either set #r "BoSSSpad.dll" to something which is appropirate for your computer
(e.g. C:\Program Files (x86)\FDY\BoSSS\bin\Release\net5.0\BoSSSpad.dll if you installed the binary distribution),
or, if you are working with the source code, you must compile BoSSSpad and put it side-by-side to this worksheet file
(from the original location in the repository, you can use the scripts getbossspad.sh, resp. getbossspad.bat).#r "BoSSSpad.dll"
//#r "C:\Users\flori\Documents\BoSSS-kummer\public\src\L4-application\BoSSSpad\bin\Debug\net6.0\BoSSSpad.dll"
using static BoSSS.Application.BoSSSpad.BoSSSshell;
Init();
using System;
using System.Collections.Generic;
using System.Linq;
using ilPSP;
using ilPSP.Utils;
using BoSSS.Platform;
using BoSSS.Platform.Utils.Geom;
using BoSSS.Foundation;
using BoSSS.Foundation.Grid;
using BoSSS.Foundation.Grid.Classic;
using BoSSS.Foundation.IO;
using BoSSS.Solution;
using BoSSS.Solution.Control;
using BoSSS.Solution.GridImport;
using BoSSS.Solution.Statistic;
using BoSSS.Solution.Utils;
using BoSSS.Solution.Gnuplot;
using BoSSS.Application.BoSSSpad;
using BoSSS.Application.XNSE_Solver;
using BoSSS.Application.GridGen;
BoSSSshell.WorkflowMgm.Init("memprofile");
wmg.SetNameBasedSessionJobControlCorrelation();
Project name is set to 'memprofile'. Default Execution queue is chosen for the database. Creating database '\\fdygitrunner\BoSSStests\memprofile'.
GetDefaultQueue()
| RuntimeLocation | DeploymentBaseDirectory | DeployRuntime | Name | DotnetRuntime | Username | ServerName | ComputeNodes | DefaultJobPriority | SingleNode | AllowedDatabasesPaths |
|---|---|---|---|---|---|---|---|---|---|---|
win\amd64 | \\fdygitrunner\BoSSStests | True | MSHPC-AllNodes-test | dotnet | FDY\jenkinsci | DC2 | [ hpccluster, hpccluster2, hpcluster3, hpccluster4, fdygitrunner ] | Normal | True | [ \\fdygitrunner\BoSSStests ] |
ExecutionQueues
| index | RuntimeLocation | DeploymentBaseDirectory | DeployRuntime | Name | DotnetRuntime | Username | ServerName | ComputeNodes | DefaultJobPriority | SingleNode | AllowedDatabasesPaths |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | win\amd64 | \\fdygitrunner\BoSSStests | True | MSHPC-Gitrunner-HighPrio | dotnet | FDY\jenkinsci | DC2 | [ fdygitrunner ] | Highest | True | [ \\fdygitrunner\BoSSStests, \\fdygitrunner\ValidationTests\databases ] |
| 1 | win\amd64 | \\fdygitrunner\BoSSStests | False | MSHPC-Gitrunner-DefaultTest | dotnet | FDY\jenkinsci | DC2 | [ hpccluster, hpccluster2, hpcluster3, hpccluster4, fdygitrunner ] | Normal | True | [ \\fdygitrunner\BoSSStests, \\fdygitrunner\ValidationTests\databases ] |
| 2 | win\amd64 | \\fdygitrunner\ValidationTests\deploy | True | MSHPC-AllNodes | dotnet | FDY\jenkinsci | DC2 | [ hpccluster, hpccluster2, hpcluster3, hpccluster4, fdygitrunner ] | Normal | True | [ \\fdygitrunner\ValidationTests\databases ] |
| 3 | win\amd64 | \\fdygitrunner\BoSSStests | True | MSHPC-AllNodes-test | dotnet | FDY\jenkinsci | DC2 | [ hpccluster, hpccluster2, hpcluster3, hpccluster4, fdygitrunner ] | Normal | True | [ \\fdygitrunner\BoSSStests ] |
| 4 | win\amd64 | \\fdygitrunner\ValidationTests\deploy | True | MSHPC-FastNodes | dotnet | FDY\jenkinsci | DC2 | [ hpcluster3, hpccluster4 ] | Normal | True | [ \\fdygitrunner\ValidationTests\databases ] |
wmg.DefaultDatabase
{ Session Count = 0; Grid Count = 0; Path = \\fdygitrunner\BoSSStests\memprofile }
int[] Resolutions_3D = new int[] { 256 };
int[] NoOfProcs = new int[] { 16, 32, 64 };
var ggcS = new List<(GridGenControl C, int MPIsize)>();
foreach(int MPIzs in NoOfProcs) {
for(int cnt = 0; cnt < Resolutions_3D.Length; cnt++) {
int Res = Resolutions_3D[cnt];
double[] _xNodes = GenericBlas.Linspace(-1, +1, Res + 1);
double[] _yNodes = GenericBlas.Linspace(-1, +1, Res + 1);
double[] _zNodes = GenericBlas.Linspace(-1, +1, Res + 1);
int J = (_xNodes.Length - 1)*(_yNodes.Length - 1)*(_zNodes.Length - 1);
string GridName = string.Format(wmg.CurrentProject + "-MeshInit_J" + J + "_Sz" + MPIzs);
{
//int NoOfProcs = (int) Math.Min(182, Math.Max(1, Math.Ceiling(J/200000.0)));
Console.WriteLine("Must create: " + GridName + " with " + MPIzs + " processors.");
var C = new GridGenControl();
ggcS.Add((C, MPIzs));
C.SetDatabase(wmg.DefaultDatabase);
C.GridName = GridName;
// ***********************************************************
C.MemoryInstrumentationLevel = ilPSP.Tracing.MemoryInstrumentationLevel.GcAndPrivateMemory;
// ***********************************************************
C.GridBlocks = new GridGenControl.MeshBlock[] {
new GridGenControl.Cartesian3D() {
xNodes = _xNodes,
yNodes = _yNodes,
zNodes = _zNodes
}
};
C.BoundaryRegions.Add((
new BoundingBox(new double[] { -1-1e-8, -2, -2 }, new double[] { -1+1e-8, +2, +2 }),
"wall_left"));
C.BoundaryRegions.Add((
new BoundingBox(new double[] { +1-1e-8, -2, -2 }, new double[] { +1+1e-8, +2, +2 }),
"wall_right"));
C.BoundaryRegions.Add((
new BoundingBox(new double[] { -2, -1-1e-8, -2 }, new double[] { +2, -1+1e-8, +2 }),
"wall_front"));
C.BoundaryRegions.Add((
new BoundingBox(new double[] { -2, +1-1e-8, -2 }, new double[] { +2, +1+1e-8, +2 }),
"wall_back"));
C.BoundaryRegions.Add((
new BoundingBox(new double[] { -2, -2, -1-1e-8 }, new double[] { +2, +2, -1+1e-8 }),
"wall_top"));
C.BoundaryRegions.Add((
new BoundingBox(new double[] { -2, -2, +1-1e-8 }, new double[] { +2, +2, +1+1e-8 }),
"wall_bottom"));
C.SessionName = "GridCreation-" + GridName;
}
}
}
Must create: memprofile-MeshInit_J16777216_Sz16 with 16 processors. Must create: memprofile-MeshInit_J16777216_Sz32 with 32 processors. Must create: memprofile-MeshInit_J16777216_Sz64 with 64 processors.
ggcS
| index | Item1 | Item2 |
|---|---|---|
| 0 | BoSSS.Application.GridGen.GridGenControl | 16 |
| 1 | BoSSS.Application.GridGen.GridGenControl | 32 |
| 2 | BoSSS.Application.GridGen.GridGenControl | 64 |
foreach(var tt in ggcS) {
Console.WriteLine(" Submitting: " + tt.C.SessionName);
var j = tt.C.CreateJob();
j.RetryCount = 1;
j.NumberOfMPIProcs = tt.MPIsize;
j.Activate();
}
Submitting: GridCreation-memprofile-MeshInit_J16777216_Sz16 Deployments so far (0): ; Success: 0 job submit count: 0
unable to determine job status - unknown
Deploying job GridCreation-memprofile-MeshInit_J16777216_Sz16 ...
Warning: no grid seems to be specified for the job to submit.
Deploying executables and additional files ... Deployment directory: \\fdygitrunner\BoSSStests\memprofile-GridGen2023Nov24_191902.959598 copied 29 files. written file: control.obj copied 'win\amd64' runtime. deployment finished. Submitting: GridCreation-memprofile-MeshInit_J16777216_Sz32 Deployments so far (0): ; Success: 0 job submit count: 0
unable to determine job status - unknown
Deploying job GridCreation-memprofile-MeshInit_J16777216_Sz32 ...
Warning: no grid seems to be specified for the job to submit.
Deploying executables and additional files ... Deployment directory: \\fdygitrunner\BoSSStests\memprofile-GridGen2023Nov24_191918.435271 copied 29 files. written file: control.obj copied 'win\amd64' runtime. deployment finished. Submitting: GridCreation-memprofile-MeshInit_J16777216_Sz64 Deployments so far (0): ; Success: 0 job submit count: 0
unable to determine job status - unknown
Deploying job GridCreation-memprofile-MeshInit_J16777216_Sz64 ...
Warning: no grid seems to be specified for the job to submit.
Deploying executables and additional files ... Deployment directory: \\fdygitrunner\BoSSStests\memprofile-GridGen2023Nov24_191935.204039 copied 29 files. written file: control.obj copied 'win\amd64' runtime. deployment finished.
wmg.BlockUntilAllJobsTerminate(7200); // wait at maximum two hours
wmg.AllJobs
All jobs finished.
#0: GridCreation-memprofile-MeshInit_J16777216_Sz16: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests) GridCreation-memprofile-MeshInit_J16777216_Sz16: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests) #1: GridCreation-memprofile-MeshInit_J16777216_Sz32: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests) GridCreation-memprofile-MeshInit_J16777216_Sz32: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests) #2: GridCreation-memprofile-MeshInit_J16777216_Sz64: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests) GridCreation-memprofile-MeshInit_J16777216_Sz64: FinishedSuccessful (MS HPC client MSHPC-AllNodes-test @DC2, @\\fdygitrunner\BoSSStests)
Asserting success:
var NoSuccess = wmg.AllJobs.Values.Where(job => job.Status != JobStatus.FinishedSuccessful).ToArray();
NUnit.Framework.Assert.Zero(NoSuccess.Count(), "Some Jobs Failed");
var plot = wmg.Sessions.GetMPItotalMemory();
We are going to observe that the memory scaling is far from perfect at this point;
plot.PlotNow()
Using gnuplot: C:\Program Files (x86)\FDY\BoSSS\bin\native\win\gnuplot-gp510-20160418-win32-mingw\gnuplot\bin\gnuplot.exe Note: In a Jupyter Worksheet, you must NOT have a trailing semicolon in order to see the plot on screen; otherwise, the output migth be surpressed.!
Maximum of each trace:
var Maxima = plot.dataGroups.Select(grp => (grp.Name, grp.Values.Max()));
Maxima
| index | Item1 | Item2 |
|---|---|---|
| 0 | Tot Mem [MegB] at 16 cores | 42476.7109375 |
| 1 | Tot Mem [MegB] at 32 cores | 52566.33203125 |
| 2 | Tot Mem [MegB] at 64 cores | 117458.30078125 |
NUnit.Framework.Assert.Less(Maxima.ElementAt(0).Item2, 100000.0);
NUnit.Framework.Assert.Less(Maxima.ElementAt(1).Item2, 100000.0);
NUnit.Framework.Assert.Less(Maxima.ElementAt(2).Item2, 200000.0);
wmg.Sessions
#0: memprofile GridCreation-memprofile-MeshInit_J16777216_Sz16 11/24/2023 19:19:24 01a758b9... #1: memprofile GridCreation-memprofile-MeshInit_J16777216_Sz32 11/24/2023 19:19:43 deb366f3... #2: memprofile GridCreation-memprofile-MeshInit_J16777216_Sz64 11/24/2023 19:20:03 eecebaba...
wmg.Sessions.Single(sess => sess.Name.EndsWith("Sz16")).ReportLargestAllocators().Take(5)
| index | Item1 | Item2 | Item3 |
|---|---|---|---|
| 0 | 40 | 19567.65625 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 1 | 24 | 8256.2265625 | <BoSSS.Foundation.Grid.Classic.Grid3D.Cartesian3DGrid>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 2 | 183 | 5983.01171875 | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 3 | 182 | 4987.578125 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 4 | 232 | 4601.44921875 | <BoSSS.Platform.Utils.Geom.PointLocalization..ctor>BoSSS.Foundation.Grid.Classic.GridData+VertexData.CollectVertices>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
wmg.Sessions.Single(sess => sess.Name.EndsWith("Sz32")).ReportLargestAllocators().Take(5)
| index | Item1 | Item2 | Item3 |
|---|---|---|---|
| 0 | 40 | 21424.1953125 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 1 | 41 | 11184.81640625 | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 2 | 167 | 8516.9140625 | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 3 | 24 | 8481.484375 | <BoSSS.Foundation.Grid.Classic.Grid3D.Cartesian3DGrid>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 4 | 166 | 5366.4296875 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
wmg.Sessions.Single(sess => sess.Name.EndsWith("Sz64")).ReportLargestAllocators().Take(5)
| index | Item1 | Item2 | Item3 |
|---|---|---|---|
| 0 | 41 | 33465.02734375 | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 1 | 145 | 33171.57421875 | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 2 | 40 | 28744.703125 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 3 | 144 | 21519.1015625 | <BoSSS.Foundation.Grid.Classic.GridCommons.GetCellNeighbourship>BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 4 | 121 | 13729.6640625 | >BoSSS.Foundation.IO.StandardFsDriver.OpenFileExclusiveBlocking>BoSSS.Foundation.IO.VectorDataSerializer.SaveVector>BoSSS.Foundation.IO.ReflectionVectorDataSerializer.SaveVector_ValueTypeCapable>BoSSS.Foundation.IO.ReflectionVectorDataSerializer.SaveVector>BoSSS.Foundation.IO.GridDatabaseDriver.SaveVectorData>BoSSS.Foundation.IO.GridDatabaseDriver.SaveGrid>SaveGridIfUnique>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
wmg.Sessions.ReportLargestAllocatorImbalance().Take(5)
| index | Item1 | Item2 | Item3 | Item4 |
|---|---|---|---|---|
| 0 | 41 | 32374.18359375 | [ 1090.84375, 11184.81640625, 33465.02734375 ] | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 1 | 127 | 27188.5625 | [ 5983.01171875, 8516.9140625, 33171.57421875 ] | <BoSSS.Foundation.Grid.Classic.GridData.ParallelSetup>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 2 | 163 | 25470.40625 | [ 83.046875, -108.40234375, -25387.359375 ] | >BoSSS.Foundation.Comm.VectorTransceiver`2.TransceiveFinish>BoSSS.Foundation.Grid.Classic.GridData+CellData.InitializeCk>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 3 | 169 | 25221.6328125 | [ 120.95703125, -221.9453125, -25100.67578125 ] | >BoSSS.Foundation.Comm.VectorTransceiver`2.TransceiveFinish>BoSSS.Foundation.Grid.Classic.GridData+CellData.Initialize_h>BoSSS.Foundation.Grid.Classic.GridData..ctor>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |
| 4 | 110 | 18451.46484375 | [ 931.05859375, -1284.1953125, 17167.26953125 ] | <SaveGridIfUnique>BoSSS.Solution.Application`1.SetUpEnvironment>root_frame |