Spin Dynamics Forum

SpinDynamica support => Bug reports => Topic started by: MichaelTayler on March 27, 2013, 01:28:26 PM

Title: Parallel computing using SpinDynamica 2.6.0
Post by: MichaelTayler on March 27, 2013, 01:28:26 PM
I am having some difficulty using "parallelization" in v. 2.6.0, which I think is due to an incorrect loading of the packages.  I am loading SpinDynamica in the standard way via

Code: [Select]
AppendTo[$Path, (*insert path to 2.6.0 here*) ]
Needs["SpinDynamica`"]

The error messages below arise when running Signal1D, Trajectory and similar routines: (repeated 8 times, once for each of 8 Kernels launched)

Code: [Select]
Get::noopen: Cannot open SpinDynamica`.
Needs::nocont: Context SpinDynamica` was not created when Needs was evaluated.

Any guidance is much appreciated!  It would be very useful for someone to upload an example file for a "working" parallel calculation
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on March 27, 2013, 04:12:50 PM
I think the path is not introduced to the subkernels. The code below should set the path.

LaunchKernels[];
ParallelEvaluate[AppendTo[$Path, (*insert path to 2.6.0 here*) ]];


Code like below could be added to SpinDynamica.

path=$Path;
DistributeDefinitions[path];
ParallelEvaluate[$Path = path];
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MichaelTayler on March 28, 2013, 12:24:49 AM
Hi Jyrki,
Great.  That seems to do the job.  The routines evaluate in parallel after replacing

Code: [Select]
path = AppendTo[$Path, (*insert path here*)];
Needs["SpinDynamica`"]

with

Code: [Select]
LaunchKernels[];
path = AppendTo[$Path, (*insert path here*)];
DistributeDefinitions[path];
ParallelEvaluate[$Path = path];
Needs["SpinDynamica`"]
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MichaelTayler on March 28, 2013, 12:39:41 AM
Another question:
There is a warning in /NMR/SpinDynamics.nb (see section "Parallel") that basis transformation matrices should be freshly evaluated.  I assume this means that SpinDynamica otherwise calculates the matrices across all kernels.  Is that correct?

On this subject I remember a while ago that we discussed "hard coding" matrices into SpinDynamica for common bases and basis transformations, so that these may be loaded from definitions as opposed to being calculated on-the-fly.  Maybe there is scope for that in future releases.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on March 29, 2013, 01:12:42 PM
Another question:
There is a warning in /NMR/SpinDynamics.nb (see section "Parallel") that basis transformation matrices should be freshly evaluated.  I assume this means that SpinDynamica otherwise calculates the matrices across all kernels.  Is that correct?

On this subject I remember a while ago that we discussed "hard coding" matrices into SpinDynamica for common bases and basis transformations, so that these may be loaded from definitions as opposed to being calculated on-the-fly.  Maybe there is scope for that in future releases.


Thanks Michael and Jyrki,
 Maybe I do not see the $Path errors because I set the path in the init.m file of my Autoload directory. When the new kernels are launched they probably all use the same init.m. In the next release I will modify the PrepareParallel routine to ensure that $Path is set correctly in all the running kernels, even if this procedure is not used.
 Some tests I use for the parallelization are attached.
 SpinDynamica currently does remember and reuse all calculations of basis transformation matrices, so I think your last point is already taken care of.
all the best
malcolm



Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on April 28, 2013, 04:44:10 PM
SpinDynamica 2.6.1 has been released and includes a fix for this bug. Please let me know if the problem persists in your installation.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: Andreas Brinkmann on May 07, 2013, 04:10:01 PM
Dear all,

I had considered a solution similar to the suggestion by Jyrki and Michael, however there is a problem with this solution when changing the number of kernels in a series of calculations. We had an example notebook (see attachments) that would use (for example on a 4 core machine) Parallel->False, Parallel->2, Parallel->True, ... The suggested approach fails when the number of kernels used is increased from one calculation to the next.

The reason is that DistributeDefinitions and ParallelNeeds will automatically be evaluated on a newly launched subkernel, however ParallelNeeds is evaluated before DistributeDefinitions. Hence, in this case the $Path would not be set correctly.

The only solution I could find is to add the $Path to the init.m file (as Malcolm described) and change the launching of the subkernels so that each reads the init.m file.

Please have a look at the attached files and see if you have another general solution. Otherwise I would suggest that the $Path in general should be specified in init.m.

All the best,

Andreas

Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 07, 2013, 08:33:50 PM
Hi Andreas,

When evaluating LaunchKernels after ParallelNeeds["SpinDynamica`"] has been evaluated Mathematica tries to load SpinDynamica to the sub-kernels automatically. This can be fixed by replacing ParallelNeeds["SpinDynamica`"] with ParallelEvaluate[Needs["SpinDynamica`"]].

I would also change ParallelEvaluate[$Path = path] to ParallelEvaluate[AppendTo[$Path , path]]. I think this would allow users to set paths to sub-kernels manually.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: Andreas Brinkmann on May 09, 2013, 07:37:28 PM
Hi Jyrki,

I had done some extensive testing when Malcolm implemented the parallelization into SpinDynamica. Prerequisite were (i) SpinDynamica handles launching and closing of subkernels, not the user, (ii) subsequent calculations may change the number of subkernels used, and (iii) specifying the location of the SpinDynamica package by the user should be done in a single place for simplicity.

To summarize my findings please have a close look at the attached notebooks, I tried to document them well. They cover all the suggestions by you and Michael that I also had tested earlier. None of the suggestions utilizing $Path on the subkernels work in general. As you can see the main problem occurs when new subkernels are added to existing ones (or relaunched after closing). I am not sure why that is exactly, since it should in principle work, as we all agree. There also seems to be a subtle difference between ParallelNeeds[] and ParallelEvaluate[Needs[]] in spite of the documentation saying that they should be identical.

As a result I had to conclude that currently the only straightforward and simple solution for users is to specify the $Path in init.m. This works flawlessly in all cases (see first attached notebook). Therefore, I still think this is so far the only solution that works in general.

Andreas
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 09, 2013, 09:15:55 PM
Thanks  Andreas,

I will have a look at these notebooks and I will try to program  a version which sets the paths to the sub-kernels without init.m file.

I had a quick look at the notebooks and it seems that SpinDynamica does not distribute SpinSystem[] to the new kernels properly. There also seems to be problems with the EventList[] in  the new kernels.
I think these can be fixed but I need  better look at the parallelization code to be sure.

Jyrki
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 10, 2013, 02:38:10 PM
Hi,

I got it working.

After replacing ParalleNeeds["SpinDynamica`"]; with ParallelEvaluate[Needs["SpinDynamica`"]]; there was still problems with the DistributedContexts option of the Parallelize[] routine. It can't be used to distribute stuff to the subkernels. Code like
Code: [Select]
stufftodistribute = {tmin, tmax, \[Delta]t, events, evopts,
   varsubrules, opbasis, basis, varsubrules, opts};
DistributeDefinitions[stufftodistribute];
can be used instead.
 
I think also things like SpinSystem[],Basis[]  and OperatorBasis[] has to be distributed. I think these should be distributed in the PrepareParallel[] routine.

Jyrki
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 11, 2013, 07:12:52 PM
To summarize:

one way to distribute paths to sub-kernels without init.m file would be to replace  ParalleNeeds["SpinDynamica`"] with ParallelEvaluate[Needs["SpinDynamica`"]] and distribute spin-system and basis using code like below
Code: [Select]
spinsystemandbasis={SpinSystem[], Basis[], OperatorBasis[]};
DistributeDefinitions[spinsystemandbasis];
ParallelEvaluate[{SpinSystem[], Basis[], OperatorBasis[]} =spinsystemandbasis];
in the PrepareParallel[] routine.

The DistributedContexts->All option of the Parallelize[] routine would not be used to distribute stuff.   In the case of Singal1D[] code like below
Code: [Select]
stufftodistribute = {tmin, tmax, \[Delta]t, events, evopts,
   varsubrules, opbasis, basis,  opts};
DistributeDefinitions[stufftodistribute];
would be used instead.


Is there distribution code for  $RecursionLimit, $IterationLimit, $ChopThreshold, $Chronology and basis transformation matrices?  I don't think that  DistributedContexts->All  would distribute these.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MichaelTayler on May 12, 2013, 09:28:12 PM
Jyrki, yes, with 2.6.0 I have had difficulties with parallelised calculations involving non-default $Chronology "RightToLeft"; I prefer to specify events with "RightToLeft" chronology but usually find that the subkernel still evaluates with the default chronology!

At present I am happy to launch by opening all available kernels, distributing the path, then loading SpinDynamica with Needs.  It appears that all definitions such as Basis[], $Chronology etc. are good and consistent across the kernel and subkernels, without any need for explicit distribution. 

Here is what I use:
1) Define my path for SpinDynamica
Code: [Select]
sdPath = ToString[$InstallationDirectory <> "\\SpinDynamica\\SpinDynamica260\\"];
SetSharedVariable[sdPath]

2) Add SpinDynamica path to $Path of both kernel and subkernels (BOTH are required)
Code: [Select]
ParallelEvaluate[AppendTo[$Path, sdPath]];
AppendTo[$Path, sdPath]

3) Load SpinDynamica
Code: [Select]
Needs["SpinDynamica`"]
The above all works fine in 2.6.0 and I can see no bugs or misevaluation with $Chronology or similar global options.  The solution is probably not general enough for solving Andreas's problem, however.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 12, 2013, 10:04:53 PM
Hi Michael,

Making the changes to SpinDynamica I described in my previous message will solve Andreas's problem. Basically changin ParallelNeeds to ParallelEvaluate[Needs[]] in PrepareParallel[] will prevent SpinDynamica to load SpinDynamica to the new sub-kernels, Launched for the subsequent calculation, before the path is set to those kernels. Using distribution code like in my previous message for Parallelize[]will prevent things like SpinSystem[] and Basis[] to be distributed before SpinDynamica is loaded to the new sub-kernels. Consequence of these changes is that SpinSystem[],Basis[] and OperatorBasis[] needs to be distributed in PrepareParallel[].

Jyrki
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on May 14, 2013, 02:33:46 PM
Hi all,
 thanks so much for all the blood, sweat and tears!

 I'm just catching up on the discussion. However I am wondering what the state of play is now. I concur with Andreas that the most desirable solution should have the features

Quote
(i) SpinDynamica handles launching and closing of subkernels, not the user, (ii) subsequent calculations may change the number of subkernels used, and (iii) specifying the location of the SpinDynamica package by the user should be done in a single place for simplicity.

I would also very much like to have as much as possible of the parallel code inside a routine of the form PrepareParallel[..] and not in the individual routines. Some of the suggestions above seem to propose more complex solutions to the observed glitches, but I think that making the code for the individual routines even more complicated is asking for trouble when the routines are developed further.

Please work with the new version 2.6.1 which has been released on the SD website.

Can one of you (Andreas?) volunteer to generate a new version 2.6.2b1 by copying 2.6.1 and which includes the proposed Parallel modifications, and distribute it with a test notebook? Or are we not yet at the stage to try that?

I'm off to Brazil today but I hope to have time to help over the next 2 weeks

all the best malcolm

Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 15, 2013, 10:39:24 AM
Hi all,

I think the problem in the 2.6.1 is as Andreas's described
Quote
The reason is that DistributeDefinitions and ParallelNeeds will automatically be evaluated on a newly launched subkernel
In my previous messages I proposed a workaround to this problem.

 ParalleEvaluate@Needs will not evaluate automatically so changing ParallelNeeds to  ParalleEvaluate@Needs  will prevent SpinDynamica to be loaded automatically to the newly launched sub-kernels,. After this change the new sub-kernels started to complain that the SpinSystem and Basis are not set so I decided to distribute them in the PrepareParallel routine.

DistributedContexts->All all option of the individual  Parallelize[]  routines will probably make SpinDynamica distribute stuff automatically to the new sub-kernels. The DistributedContexts->All option will cause error message
Code: [Select]
EventList::cantdo: cannot interpret event specifications: {events:{_Event..},evlistmore:(Except[_Rule]..),evlistopts___Rule} in the subsequent calculations. I decided to try distribution code like
Code: [Select]
stufftodistribute = {tmin, tmax, \[Delta]t, events, evopts,
   varsubrules, opbasis, basis,  opts};
DistributeDefinitions[stufftodistribute];
instead of the  DistributedContexts->All option. This fixed the problem. There probably is a solution to this problem which does not include complex changes to the individual routines. To find this solution we need to investigate better what causes the 
Code: [Select]
EventList::cantdo: cannot interpret event specifications: {events:{_Event..},evlistmore:(Except[_Rule]..),evlistopts___Rule} messages.

Jyrki
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 15, 2013, 02:52:41 PM
Hi all,

DistributedContexts->"Global`" would only distribute definitions on context Global`. Using this in the Parallelize instead of DistributedContexts->All would be another fix to the problem which causes the
Code: [Select]
EventList::cantdo: cannot interpret event specifications: {events:{_Event..},evlistmore:(Except[_Rule]..),evlistopts___Rule} messages.
I think this would be the only change needed in the individual routines. Other things that need to be distributed could be distributed in PrepareParallel routine.


Jyrki
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on May 24, 2013, 06:32:13 PM
Hi all,

It seems that the DistributedContexts->"Global`" option does not distribute anything to the subkernles.  The definitions defined in the EnsembleAverage modules are in the "NMR`SpinDynamics`Private`" context.
Also the definitions that cause the
Code: [Select]
EventList::cantdo: cannot interpret event specifications: {events:{_Event..},evlistmore:(Except[_Rule]..),evlistopts___Rule} messages, in subsequent calculations, seem to be in the "NMR`SpinDynamics`Private`" context.

 However it seems that the newly  launched sub kernels print the above message right after ParallelEvaluate[Needs["SpinDynamica`"]] is evaluated in the PrepareParallel[] routine and all the kernels work normally after the messages.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on June 12, 2013, 03:29:55 PM
Hi Malcolm,

I tested SDv2.6.2b2 with  trajectory simulations similar to what I usually perform and it seems to work fine.

It seems that my previous message to this thread is false.  DistributedContexts->"Global`" does distribute stuff to sub-kernels.

Jyrki.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on June 13, 2013, 02:28:33 PM
OK good, thanks Jyrki. Does anyone else have any problem to report with parallelization in v2.6.2b2?
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on June 13, 2013, 06:46:30 PM
v2.6.2b3 contains Jyrkis recoding of parallel and also addresses the slow execution of Trajectory reported in the other thread. It is available here:

https://www.dropbox.com/sh/x26heihcib5mbqu/1MN3HBANk5
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: Andreas Brinkmann on June 13, 2013, 07:04:15 PM
Dear all,

I tested SDv2.6.2b2 and it works great for me (I am using mainly Signal1D). Thanks Jyrki!

Andreas
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: Andreas Brinkmann on June 13, 2013, 09:00:02 PM
One thing I noticed though:

If I use Signal1D[..., MaxSteps->xxx] the option is not passed on to NDSolve[]. I am using SignalCalculationMethod -> "COMPUTE" in this example.

Andreas
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on June 14, 2013, 10:18:23 AM
Hi Andreas,
 is this problem specific to Parallel->True or does it also arise for Parallel->False?
malcolm
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: Andreas Brinkmann on June 14, 2013, 03:37:26 PM
Hi Malcolm,

this problem occurs both for Parallel->True and Parallel->False. If I specify a Preparation sequence, it does not occur for the calculation of that one, but only for the Compute part. I notice that in the code of Compute the call to NPropagationOperator[] does not pass any additional options? Should I better open a new thread since this is more Compute related than a parallel problem?

Andreas
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: JyrkiRantaharju on June 14, 2013, 04:25:26 PM
Hi Andreas,

it would be great if you have time to test SDv2.6.2b4 which is available here

https://www.dropbox.com/sh/ptfd6fcwedr3lcv/EjpwCr1yJU (https://www.dropbox.com/sh/ptfd6fcwedr3lcv/EjpwCr1yJU)

I made small modifications to Signal1D of it in-order to fix the memory overshoot issue.

Jyrki.
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on July 24, 2013, 11:11:27 PM
Hi all,
 I hope that these problems have been addressed in the new release SDv2.7.1. I have incorporated most of Jyrki's and your suggestions.

 So far, the implementation of Parallel has worked fine in the tests I have done.

Good luck
malcolm
Title: Re: Parallel computing using SpinDynamica 2.6.0
Post by: MalcolmHLevitt on September 13, 2013, 06:04:30 PM
As far as I know, all of these concerns have been fixed in SDv2.8.1. Please report any further problems!
malcolm