2021-06-10 03:30:48 [INFO] Starting.
2021-06-10 03:30:48 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-06-10 03:30:48 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-06-10 03:30:48 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 03:30:48 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 03:30:49 [INFO] Building aggregate.
2021-06-10 03:30:49 [INFO] Writing every 10 Runs.
2021-06-10 03:30:49 [INFO] Processing SRX3408797. (1/4)
2021-06-10 03:30:49 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-06-10 03:31:24 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-06-10 03:31:59 [INFO] Processing SRX3408799. (2/4)
2021-06-10 03:31:59 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-06-10 03:32:58 [INFO] Processing SRX3408796. (3/4)
2021-06-10 03:32:58 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-06-10 03:33:35 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-06-10 03:34:32 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-06-10 03:35:17 [INFO] Processing SRX3408798. (4/4)
2021-06-10 03:35:17 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-06-10 03:36:20 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-06-10 03:37:10 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-06-10 03:38:04 [INFO] Writing to disk...
2021-06-10 03:38:44 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:56 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-02 17:08:56 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-02 17:08:56 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing SRX3408797. (1/4)
2021-08-02 17:08:56 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-02 18:07:07 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-02 18:08:52 [ERROR] Adding SRR6308411:chr16_vs_chr16 to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 345. MiB for an array with shape (90338346,) and data type int32

2021-08-04 22:06:06 [INFO] Starting.
2021-08-04 22:06:06 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-04 22:06:06 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-04 22:06:06 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:06:06 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:06:06 [INFO] Writing every 10 Runs.
2021-08-04 22:06:06 [INFO] Processing SRX3408797. (1/4)
2021-08-04 22:06:06 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-04 22:09:49 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-04 22:18:18 [INFO] Processing SRX3408799. (2/4)
2021-08-04 22:18:18 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-04 22:26:44 [INFO] Processing SRX3408796. (3/4)
2021-08-04 22:26:44 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-04 22:35:30 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-04 22:44:21 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-04 22:52:55 [INFO] Processing SRX3408798. (4/4)
2021-08-04 22:52:55 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-04 23:01:43 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-04 23:10:38 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-04 23:19:39 [INFO] Writing to disk...
2021-08-04 23:21:19 [INFO] Finished.
2021-08-12 14:31:17 [INFO] Starting.
2021-08-12 14:31:17 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-12 14:31:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-12 14:31:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:31:18 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:31:18 [INFO] Writing every 10 Runs.
2021-08-12 14:31:18 [INFO] Processing SRX3408797. (1/4)
2021-08-12 14:31:18 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-12 14:31:58 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-12 14:32:50 [INFO] Processing SRX3408799. (2/4)
2021-08-12 14:32:50 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-12 14:33:53 [INFO] Processing SRX3408796. (3/4)
2021-08-12 14:33:53 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-12 14:34:46 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-12 14:35:38 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-12 14:36:24 [INFO] Processing SRX3408798. (4/4)
2021-08-12 14:36:24 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-12 14:37:23 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-12 14:38:25 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-12 14:39:26 [INFO] Writing to disk...
2021-08-12 14:39:54 [INFO] Finished.
2021-08-12 15:09:39 [INFO] Starting.
2021-08-12 15:09:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-12 15:09:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-12 15:09:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:09:40 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:09:40 [INFO] Writing every 10 Runs.
2021-08-12 15:09:40 [INFO] Processing SRX3408797. (1/4)
2021-08-12 15:09:40 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-12 15:10:12 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-12 15:10:56 [INFO] Processing SRX3408799. (2/4)
2021-08-12 15:10:56 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-12 15:11:54 [INFO] Processing SRX3408796. (3/4)
2021-08-12 15:11:54 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-12 15:12:40 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-12 15:13:39 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-12 15:14:24 [INFO] Processing SRX3408798. (4/4)
2021-08-12 15:14:24 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-12 15:15:32 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-12 15:16:25 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-12 15:17:33 [INFO] Writing to disk...
2021-08-12 15:18:02 [INFO] Finished.
2021-08-12 15:44:39 [INFO] Starting.
2021-08-12 15:44:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-12 15:44:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-12 15:44:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:44:40 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:44:40 [INFO] Writing every 10 Runs.
2021-08-12 15:44:40 [INFO] Processing SRX3408797. (1/4)
2021-08-12 15:44:40 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-12 15:45:19 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-12 15:46:01 [INFO] Processing SRX3408799. (2/4)
2021-08-12 15:46:01 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-12 15:46:58 [INFO] Processing SRX3408796. (3/4)
2021-08-12 15:46:58 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-12 15:48:01 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-12 15:49:05 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-12 15:49:57 [INFO] Processing SRX3408798. (4/4)
2021-08-12 15:49:57 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-12 15:51:06 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-12 15:52:15 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-12 15:53:19 [INFO] Writing to disk...
2021-08-12 15:53:51 [INFO] Finished.
2021-08-12 16:31:35 [INFO] Starting.
2021-08-12 16:31:35 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-12 16:31:35 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-12 16:31:35 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:31:36 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:31:36 [INFO] Writing every 10 Runs.
2021-08-12 16:31:36 [INFO] Processing SRX3408797. (1/4)
2021-08-12 16:31:36 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-12 16:32:18 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-12 16:33:00 [INFO] Processing SRX3408799. (2/4)
2021-08-12 16:33:00 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-12 16:34:00 [INFO] Processing SRX3408796. (3/4)
2021-08-12 16:34:00 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-12 16:34:58 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-12 16:36:07 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-12 16:37:08 [INFO] Processing SRX3408798. (4/4)
2021-08-12 16:37:08 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-12 16:38:39 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-12 16:39:54 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-12 16:41:27 [INFO] Writing to disk...
2021-08-12 16:42:10 [INFO] Finished.
2021-08-20 17:55:33 [INFO] Starting.
2021-08-20 17:55:33 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-20 17:55:33 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-20 17:55:33 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 17:55:34 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 17:55:34 [INFO] Writing every 10 Runs.
2021-08-20 17:55:34 [INFO] Processing SRX3408797. (1/4)
2021-08-20 17:55:34 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-20 17:56:06 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-20 17:56:42 [INFO] Processing SRX3408799. (2/4)
2021-08-20 17:56:42 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-20 17:57:32 [INFO] Processing SRX3408796. (3/4)
2021-08-20 17:57:32 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-20 17:58:04 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-20 17:58:47 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-20 17:59:23 [INFO] Processing SRX3408798. (4/4)
2021-08-20 17:59:23 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-20 18:00:39 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-20 18:02:43 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-20 18:04:19 [INFO] Writing to disk...
2021-08-20 18:04:33 [INFO] Finished.
2021-08-23 11:28:00 [INFO] Starting.
2021-08-23 11:28:00 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-23 11:28:00 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-23 11:28:00 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:28:01 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:28:01 [INFO] Writing every 10 Runs.
2021-08-23 11:28:01 [INFO] Processing SRX3408797. (1/4)
2021-08-23 11:28:01 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-23 11:28:15 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-23 11:28:35 [INFO] Processing SRX3408799. (2/4)
2021-08-23 11:28:35 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-23 11:28:59 [INFO] Processing SRX3408796. (3/4)
2021-08-23 11:28:59 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-23 11:29:18 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-23 11:29:39 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-23 11:29:56 [INFO] Processing SRX3408798. (4/4)
2021-08-23 11:29:56 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-23 11:30:23 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-23 11:30:42 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-23 11:31:02 [INFO] Writing to disk...
2021-08-23 11:31:05 [INFO] Finished.
2021-08-23 13:15:29 [INFO] Starting.
2021-08-23 13:15:29 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/
2021-08-23 13:15:29 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP125488/aggregate.hdf5
2021-08-23 13:15:29 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:15:30 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:15:30 [INFO] Writing every 10 Runs.
2021-08-23 13:15:30 [INFO] Processing SRX3408797. (1/4)
2021-08-23 13:15:30 [INFO] Processing SRX3408797-SRR6308412. (1/9)
2021-08-23 13:15:40 [INFO] Processing SRX3408797-SRR6308411. (2/9)
2021-08-23 13:15:47 [INFO] Processing SRX3408799. (2/4)
2021-08-23 13:15:47 [INFO] Processing SRX3408799-SRR6308416. (3/9)
2021-08-23 13:15:54 [INFO] Processing SRX3408796. (3/4)
2021-08-23 13:15:54 [INFO] Processing SRX3408796-SRR6308409. (4/9)
2021-08-23 13:16:04 [INFO] Processing SRX3408796-SRR6308410. (5/9)
2021-08-23 13:16:10 [INFO] Processing SRX3408796-SRR6308408. (6/9)
2021-08-23 13:16:19 [INFO] Processing SRX3408798. (4/4)
2021-08-23 13:16:19 [INFO] Processing SRX3408798-SRR6308415. (7/9)
2021-08-23 13:16:26 [INFO] Processing SRX3408798-SRR6308414. (8/9)
2021-08-23 13:16:34 [INFO] Processing SRX3408798-SRR6308413. (9/9)
2021-08-23 13:16:43 [INFO] Writing to disk...
2021-08-23 13:16:44 [INFO] Finished.