Why was GPU not used for this calculation?

Questions
#1 
 Q-Chem begins on Sun Jan 30 17:48:35 2022  

 Host: 
0

     Scratch files written to /work/qscratch/qchem1125849//
 Dec1321 |scratch|qcdevops|jenkins|workspace|build_RNUM 5828   
 Processing $rem in /home/qchem/config/preferences:
 Processing $rem in /root/.qchemrc:

 Checking the input file for inconsistencies... 	...done.
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Begin brianAPIInit
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Detected compatible BrianQC interface version 5
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Reading config from /home/qchem/brianqc_qchem/config.json
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Setting Brian root dir to /home/qchem/brianqc_qchem
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Found kernel DB in config: /home/qchem/brianqc_qchem/integrators/4817813504-1629217630_kernel_g_v1_2_0_be40d12b1d_CUDA_GeForce_RTX_3070_Ti.db
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Found DFT DB in config: /home/qchem/brianqc_qchem/integrators/18677760-1628115362_dft_kernel_v1_2_0_f382e97647_CUDA_GeForce_RTX_3070_Ti.db
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Found MM DB in config: /home/qchem/brianqc_qchem/integrators/1495040-1628116000_mm_kernel_v1_2_0_f382e97647_CUDA_GeForce_RTX_3070_Ti.db
 tid: 0x0 [2022.01.30-17:48:35.268] [INFO] Start BrianQC module initialization
 tid: 0x0 [2022.01.30-17:48:35.296] [INFO] GPU 0# NVIDIA_GeForce_RTX_3090_GPU-76f424f4-1c83-5894-6cbd-7736167324b1
 tid: 0x0 [2022.01.30-17:48:35.296] [INFO] GPU 1# NVIDIA_GeForce_RTX_3090_GPU-ab0b3ebd-ac39-522c-2752-4c2df5c80d3b
 tid: 0x0 [2022.01.30-17:48:35.296] [INFO] GPU 2# NVIDIA_GeForce_RTX_3090_GPU-880937b9-c4ab-301d-0aad-fa761eb3966f
 tid: 0x0 [2022.01.30-17:48:35.296] [INFO] GPU 3# NVIDIA_GeForce_RTX_3090_GPU-f29a9b06-3745-c1a3-d743-b3cba35f586a
 tid: 0x0 [2022.01.30-17:48:35.298] [SYSTEM] BrianQC version: v1_2_1_ca09c8c438 licensed to: James Kress
 tid: 0x0 [2022.01.30-17:48:35.307] [SYSTEM] Visit us at: https://brianqc.com
 tid: 0x0 [2022.01.30-17:48:35.318] [SYSTEM] BrianQC copyright Streamnovation Ltd. 2013-2020
 tid: 0x0 [2022.01.30-17:48:35.328] [SYSTEM] Contributors: Gy. Cserey, B. Czetenyi, A. Jasz, G. Kis, I. Ladjanszki, A. Rak, G. J. Tornai, A. Vas
 tid: 0x0 [2022.01.30-17:48:35.613] [INFO] Detected device: CPU
 tid: 0x0 [2022.01.30-17:48:35.613] [INFO] Detected device: CUDA NVIDIA GeForce RTX 3090#0000:25:00.0
 tid: 0x0 [2022.01.30-17:48:35.613] [INFO] Detected device: CUDA NVIDIA GeForce RTX 3090#0000:41:00.0
 tid: 0x0 [2022.01.30-17:48:35.613] [INFO] Detected device: CUDA NVIDIA GeForce RTX 3090#0000:81:00.0
 tid: 0x0 [2022.01.30-17:48:35.613] [INFO] Detected device: CUDA NVIDIA GeForce RTX 3090#0000:E1:00.0
 tid: 0x0 [2022.01.30-17:48:35.615] [INFO] Allocating 3 threads for GPU: CUDA NVIDIA GeForce RTX 3090#0000:25:00.0 on node 0
 tid: 0x0 [2022.01.30-17:48:35.616] [INFO] Allocating 3 threads for GPU: CUDA NVIDIA GeForce RTX 3090#0000:41:00.0 on node 0
 tid: 0x0 [2022.01.30-17:48:35.617] [INFO] Allocating 3 threads for GPU: CUDA NVIDIA GeForce RTX 3090#0000:81:00.0 on node 0
 tid: 0x0 [2022.01.30-17:48:35.618] [INFO] Allocating 3 threads for GPU: CUDA NVIDIA GeForce RTX 3090#0000:E1:00.0 on node 0
 tid: 0x0 [2022.01.30-17:48:48.413] [INFO] Used GPU count: 4
 tid: 0x0 [2022.01.30-17:48:48.413] [INFO] Signing in to license server: license.brainqc.com 443
 tid: 0x0 [2022.01.30-17:48:49.590] [INFO] Started license watchdog on pid 1126588
 tid: 0x0 [2022.01.30-17:48:50.381] [INFO] Scratch dir is disabled, storing everything in memory
 tid: 0x0 [2022.01.30-17:48:50.381] [INFO] End BrianQC module initialization
 tid: 0x0 [2022.01.30-17:48:50.389] [INFO] End brianAPIInit
 tid: 0x0 [2022.01.30-17:48:50.389] [INFO] Running preliminary BrianQC license check...
 tid: 0x0 [2022.01.30-17:48:50.389] [INFO] Preliminary BrianQC license check passed

--------------------------------------------------------------
User input:
--------------------------------------------------------------
$molecule
0 1
P        16.5406580499999656      20.3203988500000001       6.7697500000000161
O        16.7734405499999824      20.5185626500000104       8.2158712000000094
O        16.7308927499999882      18.9760559500000099       6.1962742999999962
O        15.1077031499999883      20.9080283499999808       6.3879928000000072
C        14.6993297499999684      22.1943925500000141       6.8457176999999945
...
H        -1.1684560462541862      -8.3937371505611846       6.6368554519841316
H       -12.9838637973644389      -2.2771706180613487      -6.3408257641675325
H         5.9030570947387435      25.7429427652984764      -6.7700435847031892
$end

$rem
METHOD PBEh-3c
BASIS def2-mSVP
INCDFT TRUE
INCFOCK 5
VARTHRESH 3
SCF_CONVERGENCE 6
THRESH 9
SYMMETRY false
SYM_IGNORE true
JOBTYPE sp
MAX_SCF_CYCLES 100
MEM_TOTAL 1000000
MEM_STATIC 200000
$end

 Nuclear Repulsion Energy =      396769.84357099 hartrees
 There are     2402 alpha and     2402 beta electrons
 Requested basis set is def2-mSVP
 There are 4983 shells and 9575 basis functions

 Q-Chem warning in module libmdc/get_mega.C, line 171:

 Allowed MEM_STATIC has been capped at MEM_STATIC=8191.
 Reduce the value of MEM_STATIC to silence this warning.


 Total QAlloc Memory Limit 1000000 MB
 Mega-Array Size      8007 MB
 MEM_STATIC part      8191 MB
 A cutoff of  1.0D-09 yielded 387285 shell pairs
 There are   1495731 function pairs (   1642934 Cartesian)
 Smallest overlap matrix eigenvalue = 1.06E-03

 Scale SEOQF with 1.000000e-03/1.000000e-03/1.000000e-03

 Standard Electronic Orientation quadrupole field applied
 Nucleus-field energy     =     0.0000005261 hartrees
 Guess from superposition of atomic densities
 Warning:  Energy on first SCF cycle will be non-variational
 SAD guess density has 4804.000000 electrons
 A restricted hybrid HF-DFT SCF calculation will be
 performed using Pulay DIIS extrapolation
 Exchange:     0.4200 Hartree-Fock + 0.5800 PBEh-3c
 Correlation:  1.0000 PBEh-3c
 Using SG-1 standard quadrature grid
 Dispersion:   Grimme D3
 SCF converges when DIIS error is below 1.0E-06
 using 64 threads for integral computing
 -------------------------------------------------------
 OpenMP Integral computing Module                
 Release: version 2.0, May 2017, Q-Chem Inc. Pleasanton 
 -------------------------------------------------------
 -------------------------------------------------------
 OpenMP BLAS3 based DFT computing Module                
 Release: version 1.0, May 2013, Q-Chem Inc. Pittsburgh 
 -------------------------------------------------------
 ---------------------------------------
  Cycle       Energy         DIIS Error
 ---------------------------------------
    1  -42235.2562681163      8.04E-03
    2  -42112.7646818921      1.22E-03
    3  -41287.4512145841      4.20E-03
    4  -31289.4436489416      1.82E-02
    5  -40215.6237069091      7.48E-03
    6  -42072.7546435545      1.33E-03
    7  -42084.1982076561      1.13E-03
    8  -42102.0370779315      8.96E-04
    9  -42121.0192771069      4.79E-04
   10  -42124.1965793634      3.43E-04
   11  -42120.8699046222      3.88E-04
   12  -42127.0020166169      2.01E-04
   13  -42127.8103598306      1.12E-04
   14  -42126.2941100304      1.62E-04
   15  -42127.5948608636      1.03E-04
   16  -42127.9101243825      8.31E-05
   17  -42128.1257590994      7.02E-05
 tid: 0x1 [2022.01.30-20:18:19.996] [INFO] 5.08313e+06
 tid: 0x1 [2022.01.30-20:18:19.996] [INFO] 0
 tid: 0x1 [2022.01.30-20:18:19.996] [ERROR] Failed to send keep-alive to: license.brainqc.com:443
 tid: 0x1 [2022.01.30-20:18:20.006] [ERROR] Unable to connect to license.brainqc.com:443 error:00000000:lib(0):func(0):reason(0)
   18  -42127.7289229523      7.06E-05
   19  -42128.0793287413      6.55E-05

Thanks.

Jim Kress

#2

Jim,

What you see is expected behavior, and there are a couple of things happening here. The main reason is that the implementation of PBEh-3c in Q-Chem goes through a code that is not interfaced with BrianQC, hence no use of GPU acceleration. The second issue is support for PBEh-3c in BrianQC, which I am verifying right now with the BrianQC team. Even if the functional is not supported, it is reasonable to expect GPU-accelerated evaluation of the HF bit. I am also reporting this issue internally to see if there is anything we can do in the short term.

#3

Sorry, my previous answer is not entirely accurate. The job above will use GPU acceleration if the three rems

INCDFT TRUE
INCFOCK 5
VARTHRESH 3

are removed or commented out.

The statement about functional support in BrianQC still stands.

#4

Thanks for the reply. If I do remove the items you suggest, the GPUs are utilized. So, PBEh-3c does use the GPUs for at least part of the calculation.

Here’s as snapshot of a run:
$rem
METHOD PBEh-3c
BASIS def2-mSVP
XC_GRID 000099000590
INCDFT 0
INCFOCK 0
SCF_CONVERGENCE 6
THRESH 9
SYMMETRY false
SYM_IGNORE true
JOBTYPE sp
MAX_SCF_CYCLES 100
MEM_TOTAL 1000000
MEM_STATIC 200000
$end

…

General SCF calculation program by
Eric Jon Sundstrom, Paul Horn, Yuezhi Mao, Dmitri Zuev, Alec White,
David Stuck, Shaama M.S., Shane Yost, Joonho Lee, David Small,
Daniel Levine, Susi Lehtola, Hugh Burton, Evgeny Epifanovsky,
Bang C. Huynh

Exchange: 0.4200 Hartree-Fock + 0.5800 PBEh-3c
Correlation: 1.0000 PBEh-3c
Using Euler-Maclaurin-Lebedev (99,590) quadrature formula
Dispersion: Grimme D3
---- BrianQC HCore successfully initialized ----
---- BrianQC J/K successfully initialized ----
BrianQC JK build time 304.0000000000 (s)
A restricted SCF calculation will be
performed using DIIS
SCF converges when DIIS error is below 1.0e-06

Cycle Energy DIIS error

BrianQC JK build time 5.9e+02 (s)
1 -42235.2614328216 8.04e-03
BrianQC JK build time 5.81e+02 (s)
2 -42112.7650115860 1.22e-03
BrianQC JK build time 5.80e+02 (s)
3 -41287.3014173275 4.20e-03
BrianQC JK build time 5.81e+02 (s)
4 -31288.8896969505 1.82e-02
BrianQC JK build time 5.79e+02 (s)
5 -40217.1435938980 7.47e-03

…

As you can see, BrianQC is invoked.

Jim

#5

Yes, this is exactly what I meant in my second (corrected) message: BrianQC will be invoked for J and K, but not for the exchange-correlation part of DFT. If you have further questions, please post here or email us at support@q-chem.com