SRP197114
  2021-08-04 22:06:48 [INFO] Processing SRX5807390. (1/6)
  2021-08-04 22:06:48 [INFO] Processing SRX5807390-SRR9030109. (1/6)
  2021-08-04 22:10:49 [INFO] Processing SRX5807392. (2/6)
  2021-08-04 22:10:49 [INFO] Processing SRX5807392-SRR9030111. (2/6)
  2021-08-04 22:25:42 [ERROR] Adding SRR9030111:chr2_vs_chr21 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 924. MiB for an array with shape (242193530,) and data type int32
  
SRP201909
  2021-08-02 17:08:56 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP201909/aggregate.hdf5
  2021-08-02 17:08:56 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
  2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
  2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
  2021-08-02 17:08:56 [INFO] Processing SRX6095624. (1/8)
  2021-08-02 17:08:56 [INFO] Processing SRX6095624-SRR9328809. (1/8)
  2021-08-04 22:06:25 [INFO] Starting.
  2021-08-04 22:06:25 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP201909/
  2021-08-04 22:06:25 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP201909/aggregate.hdf5
  2021-08-04 22:06:25 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
  2021-08-04 22:06:25 [INFO] Building aggregate for 1bp_raw network.
  2021-08-04 22:06:25 [INFO] Writing every 10 Runs.
  2021-08-04 22:06:25 [INFO] Processing SRX6095624. (1/8)
  2021-08-04 22:06:25 [INFO] Processing SRX6095624-SRR9328809. (1/8)
  2021-08-04 22:10:19 [INFO] Processing SRX6095626. (2/8)
  2021-08-04 22:10:19 [INFO] Processing SRX6095626-SRR9328811. (2/8)
SRP168606
  2021-08-04 22:04:57 [INFO] Writing every 10 Runs.
  2021-08-04 22:04:57 [INFO] Processing SRX5007082. (1/4)
  2021-08-04 22:04:57 [INFO] Processing SRX5007082-SRR8187258. (1/15)
  2021-08-04 22:08:57 [INFO] Processing SRX5007082-SRR8187257. (2/15)
  2021-08-04 22:20:39 [ERROR] Adding SRR8187257:chr14_vs_chr16 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 408. MiB for an array with shape (107043719,) and data type int32
  
SRP120957
  2021-08-04 22:05:05 [INFO] Writing every 10 Runs.
  2021-08-04 22:05:05 [INFO] Processing SRX3312499. (1/6)
  2021-08-04 22:05:05 [INFO] Processing SRX3312499-SRR6202586. (1/14)
  2021-08-04 22:08:44 [INFO] Processing SRX3312499-SRR6202588. (2/14)
  2021-08-04 22:18:37 [ERROR] Adding SRR6202588:chr11_vs_chr7 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 515. MiB for an array with shape (135086623,) and data type int32
  
SRP165933
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 924. MiB for an array with shape (242193530,) and data type int32
  
  2021-08-04 22:04:13 [INFO] Starting.
  2021-08-04 22:04:13 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165933/
  2021-08-04 22:04:13 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165933/aggregate.hdf5
  2021-08-04 22:04:13 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
  2021-08-04 22:04:13 [INFO] Building aggregate for 1bp_raw network.
  2021-08-04 22:04:13 [INFO] Writing every 10 Runs.
  2021-08-04 22:04:13 [INFO] Processing SRX6384764. (1/20)
  2021-08-04 22:04:13 [INFO] Processing SRX6384764-SRR9621904. (1/20)
  2021-08-04 22:07:59 [INFO] Processing SRX6384759. (2/20)
  2021-08-04 22:07:59 [INFO] Processing SRX6384759-SRR9621899. (2/20)
SRP090318
  2021-08-04 22:08:20 [INFO] Processing SRX2821752. (1/4)
  2021-08-04 22:08:20 [INFO] Processing SRX2821752-SRR5558587. (1/4)
  2021-08-04 22:14:33 [INFO] Processing SRX2821753. (2/4)
  2021-08-04 22:14:33 [INFO] Processing SRX2821753-SRR5558588. (2/4)
  2021-08-04 22:26:01 [ERROR] Adding SRR5558588:chr14_vs_chr15 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 408. MiB for an array with shape (107043719,) and data type int32
  
SRP261299
  2021-08-04 22:10:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP261299/aggregate.hdf5
  2021-08-04 22:10:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
  2021-08-04 22:10:18 [INFO] Building aggregate for 1bp_raw network.
  2021-08-04 22:10:18 [INFO] Writing every 10 Runs.
  2021-08-04 22:10:18 [INFO] Processing SRX8330828. (1/2)
  2021-08-04 22:10:18 [INFO] Processing SRX8330828-SRR11777960. (1/2)
  2021-08-04 22:14:06 [INFO] Processing SRX8330827. (2/2)
  2021-08-04 22:14:06 [INFO] Processing SRX8330827-SRR11777961. (2/2)
  2021-08-04 22:19:41 [ERROR] Adding SRR11777961:chr18_vs_chr3 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
  MemoryError: Unable to allocate 307. MiB for an array with shape (80373286,) and data type int32
  
SRP118999
  2021-08-05 08:03:54 [INFO] Processing SRX3222724-SRR6107784. (91/501)
  2021-08-05 08:11:19 [INFO] Processing SRX3222724-SRR6107787. (92/501)
  2021-08-05 08:18:46 [INFO] Processing SRX3222724-SRR6107785. (93/501)
  2021-08-05 08:26:16 [INFO] Processing SRX3222724-SRR6107793. (94/501)
  2021-08-05 08:33:46 [INFO] Processing SRX3222724-SRR6107783. (95/501)
  2021-08-05 08:41:18 [INFO] Processing SRX3222724-SRR6107791. (96/501)
  2021-08-05 08:48:50 [INFO] Processing SRX3222724-SRR6107789. (97/501)
  2021-08-05 08:56:23 [INFO] Processing SRX3222729. (14/77)
  2021-08-05 08:56:23 [INFO] Processing SRX3222729-SRR6107837. (98/501)
  2021-08-05 09:03:59 [INFO] Processing SRX3222729-SRR6107841. (99/501)
  2021-08-05 09:11:35 [INFO] Processing SRX3222729-SRR6107836. (100/501)
  2021-08-05 09:19:04 [INFO] Processing SRX3222729-SRR6107834. (101/501)
  2021-08-05 09:26:36 [INFO] Processing SRX3222729-SRR6107839. (102/501)
  2021-08-05 09:34:08 [INFO] Processing SRX3222729-SRR6107835. (103/501)
  2021-08-05 09:41:41 [INFO] Processing SRX3222729-SRR6107840. (104/501)
  2021-08-05 09:49:16 [INFO] Processing SRX3222729-SRR6107838. (105/501)
SRP105082
  2021-08-05 08:18:00 [INFO] Processing SRX4063563-SRR7143374. (77/699)
  2021-08-05 08:26:25 [INFO] Processing SRX4063563-SRR7143373. (78/699)
  2021-08-05 08:34:55 [INFO] Processing SRX4063563-SRR7143375. (79/699)
  2021-08-05 08:43:16 [INFO] Processing SRX4063629. (19/168)
  2021-08-05 08:43:16 [INFO] Processing SRX4063629-SRR7143639. (80/699)
  2021-08-05 08:51:01 [INFO] Writing to disk...
  2021-08-05 08:52:09 [INFO] Processing SRX4063629-SRR7143638. (81/699)
  2021-08-05 08:59:53 [INFO] Processing SRX4063629-SRR7143637. (82/699)
  2021-08-05 09:07:40 [INFO] Processing SRX4063629-SRR7143636. (83/699)
  2021-08-05 09:15:28 [INFO] Processing SRX4063500. (20/168)
  2021-08-05 09:15:28 [INFO] Processing SRX4063500-SRR7143089. (84/699)
  2021-08-05 09:23:49 [INFO] Processing SRX4063500-SRR7143091. (85/699)
  2021-08-05 09:32:09 [INFO] Processing SRX4063500-SRR7143090. (86/699)
  2021-08-05 09:40:32 [INFO] Processing SRX4063500-SRR7143088. (87/699)
  2021-08-05 09:49:00 [INFO] Processing SRX4063632. (21/168)
  2021-08-05 09:49:00 [INFO] Processing SRX4063632-SRR7143649. (88/699)
SRP157799
  2021-08-04 22:04:52 [INFO] Processing SRX4549837. (1/16)
  2021-08-04 22:04:52 [INFO] Processing SRX4549837-SRR7690256. (1/16)
  2021-08-04 22:08:25 [INFO] Processing SRX4549826. (2/16)
  2021-08-04 22:08:25 [INFO] Processing SRX4549826-SRR7690245. (2/16)
  2021-08-04 22:20:20 [ERROR] Adding SRR7690245:chr22_vs_chrM to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 194. MiB for an array with shape (50818469,) and data type int32
  
SRP012412
  2021-08-05 07:39:18 [INFO] Processing SRX3322340-SRR6213303. (334/618)
  2021-08-05 07:48:57 [INFO] Processing SRX3322340-SRR6213282. (335/618)
  2021-08-05 07:58:37 [INFO] Processing SRX3322340-SRR6213304. (336/618)
  2021-08-05 08:08:06 [INFO] Processing SRX3322340-SRR6213286. (337/618)
  2021-08-05 08:17:20 [INFO] Processing SRX3322340-SRR6213298. (338/618)
  2021-08-05 08:27:08 [INFO] Processing SRX3322340-SRR6213309. (339/618)
  2021-08-05 08:36:58 [INFO] Processing SRX3322978. (39/54)
  2021-08-05 08:36:58 [INFO] Processing SRX3322978-SRR6214233. (340/618)
  2021-08-05 08:46:09 [INFO] Processing SRX3322978-SRR6214244. (341/618)
  2021-08-05 08:55:26 [INFO] Processing SRX3322978-SRR6214245. (342/618)
  2021-08-05 09:04:31 [INFO] Processing SRX3322978-SRR6214256. (343/618)
  2021-08-05 09:13:45 [INFO] Processing SRX3322978-SRR6214254. (344/618)
  2021-08-05 09:22:54 [INFO] Processing SRX3322978-SRR6214249. (345/618)
  2021-08-05 09:32:28 [INFO] Processing SRX3322978-SRR6214262. (346/618)
  2021-08-05 09:41:58 [INFO] Processing SRX3322978-SRR6214260. (347/618)
  2021-08-05 09:51:40 [INFO] Processing SRX3322978-SRR6214259. (348/618)
SRP135798
  2021-08-04 22:05:21 [INFO] Processing SRX3798657. (1/10)
  2021-08-04 22:05:21 [INFO] Processing SRX3798657-SRR6843220. (1/10)
  2021-08-04 22:09:43 [INFO] Processing SRX3798661. (2/10)
  2021-08-04 22:09:43 [INFO] Processing SRX3798661-SRR6843224. (2/10)
  2021-08-04 22:18:49 [ERROR] Adding SRR6843224:chr19_vs_chrM to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 224. MiB for an array with shape (58617617,) and data type int32
  
SRP106379
  2021-08-04 22:04:15 [INFO] Writing every 10 Runs.
  2021-08-04 22:04:15 [INFO] Processing SRX2781964. (1/6)
  2021-08-04 22:04:15 [INFO] Processing SRX2781964-SRR5505001. (1/18)
  2021-08-04 22:07:22 [INFO] Processing SRX2781964-SRR5504999. (2/18)
  2021-08-04 22:15:51 [ERROR] Adding SRR5504999:chr3_vs_chrY to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 756. MiB for an array with shape (198295560,) and data type int32
  
SRP152361
  2021-08-04 22:14:58 [INFO] Processing SRX4734155. (1/2)
  2021-08-04 22:14:58 [INFO] Processing SRX4734155-SRR7896721. (1/2)
  2021-08-04 22:18:46 [INFO] Processing SRX4734156. (2/2)
  2021-08-04 22:18:46 [INFO] Processing SRX4734156-SRR7896722. (2/2)
  2021-08-04 22:22:32 [ERROR] Adding SRR7896722:chr20_vs_chr20 to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 246. MiB for an array with shape (64444168,) and data type int32
  
SRP103077
  2021-08-04 22:17:39 [INFO] Processing SRX3482876. (1/2)
  2021-08-04 22:17:39 [INFO] Processing SRX3482876-SRR6389422. (1/2)
  2021-08-04 22:20:58 [INFO] Processing SRX3482877. (2/2)
  2021-08-04 22:20:58 [INFO] Processing SRX3482877-SRR6389423. (2/2)
  2021-08-04 22:24:29 [ERROR] Adding SRR6389423:chr4_vs_chrM to in memory aggregate failed. Error message:
  Traceback (most recent call last):
    File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
      aggregate_data[c] = aggregate_data[c] + net
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
      return self._add_sparse(other)
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
      return self._binopt(other, '_plus_')
    File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
      indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
  MemoryError: Unable to allocate 726. MiB for an array with shape (190214556,) and data type int32
  
SRP239849
  2021-08-02 17:08:56 [INFO] Processing SRX7515929. (1/18)
  2021-08-02 17:08:56 [INFO] Processing SRX7515929-SRR10845303. (1/17)
  2021-08-02 18:02:48 [INFO] Processing SRX7515932. (2/18)
  2021-08-02 18:02:48 [INFO] Processing SRX7515932-SRR10845306. (2/17)
  2021-08-04 22:04:24 [INFO] Starting.
  2021-08-04 22:04:24 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP239849/
  2021-08-04 22:04:24 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP239849/aggregate.hdf5
  2021-08-04 22:04:24 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
  2021-08-04 22:04:24 [INFO] Building aggregate for 1bp_raw network.
  2021-08-04 22:04:24 [INFO] Writing every 10 Runs.
  2021-08-04 22:04:24 [INFO] Processing SRX7515929. (1/18)
  2021-08-04 22:04:24 [INFO] Processing SRX7515929-SRR10845303. (1/17)
  2021-08-04 22:08:08 [INFO] Processing SRX7515932. (2/18)
  2021-08-04 22:08:08 [INFO] Processing SRX7515932-SRR10845306. (2/17)
  2021-08-04 22:15:59 [INFO] Processing SRX7515934. (3/18)
  2021-08-04 22:15:59 [INFO] Processing SRX7515934-SRR10845308. (3/17)