Performance with multiple high-spin electrons

Topics related to Spinach package
Post Reply
faq_user
Posts: 49
Joined: Tue Jul 13, 2021 8:33 am

Performance with multiple high-spin electrons

Post by faq_user »

We have started to use spinach in our lab and we are quite pleased with its functionalities and performance. So far we have successfully simulated solid effect DNP of 1 Gd(III) and one I=1/2 nucleus and results are as expected. Also, the simulations can be run with a rather simple desktop computer with one 3.2 GHz 4-core i5 processor with 8 GB RAM w/o dedicated GPU in very reasonable time allowing to investigate the parameter space. Switching to a Xeon 2.5 GHz dual-processor (12 cores in total) system with 48 GB RAM (and presumably no GPU) cut down the simulation time by about a factor 2.

Now we are naturally facing problems with large dimensional Hilbert space problems (128x128) including two Gd(III) octet electron spins and one nuclear spin where we want to investigate potential cross effect at least in a static case. The scaling of computational cost is (as expected) totally prohibitive (increasing dimensionality from 18x18 (2x S=1, 1x I=1/2) to 50x50 (2x S=2, 1x I=1/2) allowed us to extrapolate ~160 years time requirement for the 2x S=7/2, 1x I=1/2 system for a powder average over 200 orientations even on the "high-end" system.

A workaround for us could be, of course, to only use quartet electrons for qualitative purposes but especially the satellite-to-satellite CE transitions are quite intriguing and for that we would need at least sextets which is out of question with our current resources (~10 days for 200 orientations).

So I practically have two questions:

1) Is there a way to *significantly* simplify calculations while still retaining all important interactions? We are currently running under DNP assumptions (electron lab frame, nuclear rotating frame) with T1-T2-Relaxation model, but especially for DNP a more sophisticated relaxation model might be interesting. I guess T1/T2 is the most efficient?

2) What would be a reasonable computer to run these type of calculations on? We are thinking about buying a dedicated system with a budget of <10k€, preferentially <=5k€. Do a CUDA-enabled systems make a large difference? What would be the optimum (reasonable) #of cores and clock considering today's CPU market? Is it better to sacrifice on the cores and getting better clock frequency? I guess with respect to RAM and cache it would be the more the merrier...
kuprov
Posts: 72
Joined: Mon Mar 29, 2021 4:26 pm

Re: Performance with multiple high-spin electrons

Post by kuprov »

Thought number one: on your ZFS distribution. There are more efficient ways of computing that integral – I had written the enclosed for Daniella at some point, that’s a Gauss-Legendre quadrature, that makes sure you need the smallest number of simulations to sample that distribution well.

Questions:

1. Would you be happy with a finite-time (rather than steady-state) cross effect DNP? Or even the initial rate of DNP accumulation? Much of the current computational complexity is due to the backslash operation which is only needed if you insist on infinite-time asymptotics.

2. Can you live with ignoring gadolinium spin correlations across more than one level? My testing shows that the DNP pattern in the solid effect case does not depend much on the presence of states like T3,3 in the basis.
faq_user
Posts: 49
Joined: Tue Jul 13, 2021 8:33 am

Re: Performance with multiple high-spin electrons

Post by faq_user »

Thanks for your ideas. Regarding the finite time approach I don't think there would be a problem. Since we should be operating far from any saturation limit the initial rate of transfer should be closely proportional to the infinite time polarization. Only thing is that oscillation might occur, which could overestimate the infinite time equilibrium?

Yes, I believe we can safely ignore higer-order correlations. The complexes we work with have ZFS of <1% the Zeeman energy.
kuprov
Posts: 72
Joined: Mon Mar 29, 2021 4:26 pm

Re: Performance with multiple high-spin electrons

Post by kuprov »

Ok, I’ve updated that module with every trick in the book. Hit "update" on your SVN menu.

The thing now has an option to use ILU preconditioned GMRES (parameters.method) and to drop high coherences between distant energy levels (bas.coherences). The two Gd calculation runs to completion in about 8 hours on my 32-core machine here. Beyond that, you would have to use supercomputing facilities, this system is too strongly connected, there are no more efficiency shortcuts.

Two notes:

1. The background wobble is not “noise”. These are all the various DNP matching conditions that are possible in a multi-level system with a pair of ZFS tensors. Broaden the sweep and you will see where the true baseline is.

2. You should not assume your ZFS is small. Actually, when I switch on second-order rotating frame corrections, the pattern does change visibly.
Post Reply