Author Topic: Parallel computing using SpinDynamica 2.6.0  (Read 9330 times)

MichaelTayler

  • Member
  • *
  • Posts: 14
    • View Profile
    • Visit my website
Parallel computing using SpinDynamica 2.6.0
« on: March 27, 2013, 01:28:26 PM »
I am having some difficulty using "parallelization" in v. 2.6.0, which I think is due to an incorrect loading of the packages.  I am loading SpinDynamica in the standard way via

Code: [Select]
AppendTo[$Path, (*insert path to 2.6.0 here*) ]
Needs["SpinDynamica`"]

The error messages below arise when running Signal1D, Trajectory and similar routines: (repeated 8 times, once for each of 8 Kernels launched)

Code: [Select]
Get::noopen: Cannot open SpinDynamica`.
Needs::nocont: Context SpinDynamica` was not created when Needs was evaluated.

Any guidance is much appreciated!  It would be very useful for someone to upload an example file for a "working" parallel calculation

JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #1 on: March 27, 2013, 04:12:50 PM »
I think the path is not introduced to the subkernels. The code below should set the path.

LaunchKernels[];
ParallelEvaluate[AppendTo[$Path, (*insert path to 2.6.0 here*) ]];


Code like below could be added to SpinDynamica.

path=$Path;
DistributeDefinitions[path];
ParallelEvaluate[$Path = path];
« Last Edit: March 27, 2013, 04:45:56 PM by Jyrki Rantaharju »

MichaelTayler

  • Member
  • *
  • Posts: 14
    • View Profile
    • Visit my website
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #2 on: March 28, 2013, 12:24:49 AM »
Hi Jyrki,
Great.  That seems to do the job.  The routines evaluate in parallel after replacing

Code: [Select]
path = AppendTo[$Path, (*insert path here*)];
Needs["SpinDynamica`"]

with

Code: [Select]
LaunchKernels[];
path = AppendTo[$Path, (*insert path here*)];
DistributeDefinitions[path];
ParallelEvaluate[$Path = path];
Needs["SpinDynamica`"]

MichaelTayler

  • Member
  • *
  • Posts: 14
    • View Profile
    • Visit my website
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #3 on: March 28, 2013, 12:39:41 AM »
Another question:
There is a warning in /NMR/SpinDynamics.nb (see section "Parallel") that basis transformation matrices should be freshly evaluated.  I assume this means that SpinDynamica otherwise calculates the matrices across all kernels.  Is that correct?

On this subject I remember a while ago that we discussed "hard coding" matrices into SpinDynamica for common bases and basis transformations, so that these may be loaded from definitions as opposed to being calculated on-the-fly.  Maybe there is scope for that in future releases.

MalcolmHLevitt

  • Administrator
  • Member
  • *****
  • Posts: 103
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #4 on: March 29, 2013, 01:12:42 PM »
Another question:
There is a warning in /NMR/SpinDynamics.nb (see section "Parallel") that basis transformation matrices should be freshly evaluated.  I assume this means that SpinDynamica otherwise calculates the matrices across all kernels.  Is that correct?

On this subject I remember a while ago that we discussed "hard coding" matrices into SpinDynamica for common bases and basis transformations, so that these may be loaded from definitions as opposed to being calculated on-the-fly.  Maybe there is scope for that in future releases.


Thanks Michael and Jyrki,
 Maybe I do not see the $Path errors because I set the path in the init.m file of my Autoload directory. When the new kernels are launched they probably all use the same init.m. In the next release I will modify the PrepareParallel routine to ensure that $Path is set correctly in all the running kernels, even if this procedure is not used.
 Some tests I use for the parallelization are attached.
 SpinDynamica currently does remember and reuse all calculations of basis transformation matrices, so I think your last point is already taken care of.
all the best
malcolm




MalcolmHLevitt

  • Administrator
  • Member
  • *****
  • Posts: 103
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #5 on: April 28, 2013, 04:44:10 PM »
SpinDynamica 2.6.1 has been released and includes a fix for this bug. Please let me know if the problem persists in your installation.

Andreas Brinkmann

  • Member
  • *
  • Posts: 12
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #6 on: May 07, 2013, 04:10:01 PM »
Dear all,

I had considered a solution similar to the suggestion by Jyrki and Michael, however there is a problem with this solution when changing the number of kernels in a series of calculations. We had an example notebook (see attachments) that would use (for example on a 4 core machine) Parallel->False, Parallel->2, Parallel->True, ... The suggested approach fails when the number of kernels used is increased from one calculation to the next.

The reason is that DistributeDefinitions and ParallelNeeds will automatically be evaluated on a newly launched subkernel, however ParallelNeeds is evaluated before DistributeDefinitions. Hence, in this case the $Path would not be set correctly.

The only solution I could find is to add the $Path to the init.m file (as Malcolm described) and change the launching of the subkernels so that each reads the init.m file.

Please have a look at the attached files and see if you have another general solution. Otherwise I would suggest that the $Path in general should be specified in init.m.

All the best,

Andreas


JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #7 on: May 07, 2013, 08:33:50 PM »
Hi Andreas,

When evaluating LaunchKernels after ParallelNeeds["SpinDynamica`"] has been evaluated Mathematica tries to load SpinDynamica to the sub-kernels automatically. This can be fixed by replacing ParallelNeeds["SpinDynamica`"] with ParallelEvaluate[Needs["SpinDynamica`"]].

I would also change ParallelEvaluate[$Path = path] to ParallelEvaluate[AppendTo[$Path , path]]. I think this would allow users to set paths to sub-kernels manually.

Andreas Brinkmann

  • Member
  • *
  • Posts: 12
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #8 on: May 09, 2013, 07:37:28 PM »
Hi Jyrki,

I had done some extensive testing when Malcolm implemented the parallelization into SpinDynamica. Prerequisite were (i) SpinDynamica handles launching and closing of subkernels, not the user, (ii) subsequent calculations may change the number of subkernels used, and (iii) specifying the location of the SpinDynamica package by the user should be done in a single place for simplicity.

To summarize my findings please have a close look at the attached notebooks, I tried to document them well. They cover all the suggestions by you and Michael that I also had tested earlier. None of the suggestions utilizing $Path on the subkernels work in general. As you can see the main problem occurs when new subkernels are added to existing ones (or relaunched after closing). I am not sure why that is exactly, since it should in principle work, as we all agree. There also seems to be a subtle difference between ParallelNeeds[] and ParallelEvaluate[Needs[]] in spite of the documentation saying that they should be identical.

As a result I had to conclude that currently the only straightforward and simple solution for users is to specify the $Path in init.m. This works flawlessly in all cases (see first attached notebook). Therefore, I still think this is so far the only solution that works in general.

Andreas

JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #9 on: May 09, 2013, 09:15:55 PM »
Thanks  Andreas,

I will have a look at these notebooks and I will try to program  a version which sets the paths to the sub-kernels without init.m file.

I had a quick look at the notebooks and it seems that SpinDynamica does not distribute SpinSystem[] to the new kernels properly. There also seems to be problems with the EventList[] in  the new kernels.
I think these can be fixed but I need  better look at the parallelization code to be sure.

Jyrki

JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #10 on: May 10, 2013, 02:38:10 PM »
Hi,

I got it working.

After replacing ParalleNeeds["SpinDynamica`"]; with ParallelEvaluate[Needs["SpinDynamica`"]]; there was still problems with the DistributedContexts option of the Parallelize[] routine. It can't be used to distribute stuff to the subkernels. Code like
Code: [Select]
stufftodistribute = {tmin, tmax, \[Delta]t, events, evopts,
   varsubrules, opbasis, basis, varsubrules, opts};
DistributeDefinitions[stufftodistribute];
can be used instead.
 
I think also things like SpinSystem[],Basis[]  and OperatorBasis[] has to be distributed. I think these should be distributed in the PrepareParallel[] routine.

Jyrki

JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #11 on: May 11, 2013, 07:12:52 PM »
To summarize:

one way to distribute paths to sub-kernels without init.m file would be to replace  ParalleNeeds["SpinDynamica`"] with ParallelEvaluate[Needs["SpinDynamica`"]] and distribute spin-system and basis using code like below
Code: [Select]
spinsystemandbasis={SpinSystem[], Basis[], OperatorBasis[]};
DistributeDefinitions[spinsystemandbasis];
ParallelEvaluate[{SpinSystem[], Basis[], OperatorBasis[]} =spinsystemandbasis];
in the PrepareParallel[] routine.

The DistributedContexts->All option of the Parallelize[] routine would not be used to distribute stuff.   In the case of Singal1D[] code like below
Code: [Select]
stufftodistribute = {tmin, tmax, \[Delta]t, events, evopts,
   varsubrules, opbasis, basis,  opts};
DistributeDefinitions[stufftodistribute];
would be used instead.


Is there distribution code for  $RecursionLimit, $IterationLimit, $ChopThreshold, $Chronology and basis transformation matrices?  I don't think that  DistributedContexts->All  would distribute these.

MichaelTayler

  • Member
  • *
  • Posts: 14
    • View Profile
    • Visit my website
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #12 on: May 12, 2013, 09:28:12 PM »
Jyrki, yes, with 2.6.0 I have had difficulties with parallelised calculations involving non-default $Chronology "RightToLeft"; I prefer to specify events with "RightToLeft" chronology but usually find that the subkernel still evaluates with the default chronology!

At present I am happy to launch by opening all available kernels, distributing the path, then loading SpinDynamica with Needs.  It appears that all definitions such as Basis[], $Chronology etc. are good and consistent across the kernel and subkernels, without any need for explicit distribution. 

Here is what I use:
1) Define my path for SpinDynamica
Code: [Select]
sdPath = ToString[$InstallationDirectory <> "\\SpinDynamica\\SpinDynamica260\\"];
SetSharedVariable[sdPath]

2) Add SpinDynamica path to $Path of both kernel and subkernels (BOTH are required)
Code: [Select]
ParallelEvaluate[AppendTo[$Path, sdPath]];
AppendTo[$Path, sdPath]

3) Load SpinDynamica
Code: [Select]
Needs["SpinDynamica`"]
The above all works fine in 2.6.0 and I can see no bugs or misevaluation with $Chronology or similar global options.  The solution is probably not general enough for solving Andreas's problem, however.

JyrkiRantaharju

  • Member
  • *
  • Posts: 36
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #13 on: May 12, 2013, 10:04:53 PM »
Hi Michael,

Making the changes to SpinDynamica I described in my previous message will solve Andreas's problem. Basically changin ParallelNeeds to ParallelEvaluate[Needs[]] in PrepareParallel[] will prevent SpinDynamica to load SpinDynamica to the new sub-kernels, Launched for the subsequent calculation, before the path is set to those kernels. Using distribution code like in my previous message for Parallelize[]will prevent things like SpinSystem[] and Basis[] to be distributed before SpinDynamica is loaded to the new sub-kernels. Consequence of these changes is that SpinSystem[],Basis[] and OperatorBasis[] needs to be distributed in PrepareParallel[].

Jyrki

MalcolmHLevitt

  • Administrator
  • Member
  • *****
  • Posts: 103
    • View Profile
Re: Parallel computing using SpinDynamica 2.6.0
« Reply #14 on: May 14, 2013, 02:33:46 PM »
Hi all,
 thanks so much for all the blood, sweat and tears!

 I'm just catching up on the discussion. However I am wondering what the state of play is now. I concur with Andreas that the most desirable solution should have the features

Quote
(i) SpinDynamica handles launching and closing of subkernels, not the user, (ii) subsequent calculations may change the number of subkernels used, and (iii) specifying the location of the SpinDynamica package by the user should be done in a single place for simplicity.

I would also very much like to have as much as possible of the parallel code inside a routine of the form PrepareParallel[..] and not in the individual routines. Some of the suggestions above seem to propose more complex solutions to the observed glitches, but I think that making the code for the individual routines even more complicated is asking for trouble when the routines are developed further.

Please work with the new version 2.6.1 which has been released on the SD website.

Can one of you (Andreas?) volunteer to generate a new version 2.6.2b1 by copying 2.6.1 and which includes the proposed Parallel modifications, and distribute it with a test notebook? Or are we not yet at the stage to try that?

I'm off to Brazil today but I hope to have time to help over the next 2 weeks

all the best malcolm