2021-06-10 03:34:04 [INFO] Starting.
2021-06-10 03:34:04 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-06-10 03:34:04 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-06-10 03:34:04 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 03:34:04 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 03:34:05 [INFO] Building aggregate.
2021-06-10 03:34:05 [INFO] Writing every 10 Runs.
2021-06-10 03:34:05 [INFO] Processing DRX167729. (1/9)
2021-06-10 03:34:05 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-06-10 03:34:08 [INFO] Processing DRX181478. (2/9)
2021-06-10 03:34:08 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-06-10 03:34:09 [INFO] Processing DRX181480. (3/9)
2021-06-10 03:34:09 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-06-10 03:34:11 [INFO] Processing DRX181476. (4/9)
2021-06-10 03:34:11 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-06-10 03:34:13 [INFO] Processing DRX167730. (5/9)
2021-06-10 03:34:13 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-06-10 03:34:15 [INFO] Processing DRX181477. (6/9)
2021-06-10 03:34:15 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-06-10 03:34:17 [INFO] Processing DRX181475. (7/9)
2021-06-10 03:34:17 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-06-10 03:34:19 [INFO] Processing DRX181479. (8/9)
2021-06-10 03:34:19 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-06-10 03:34:21 [INFO] Processing DRX167737. (9/9)
2021-06-10 03:34:21 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-06-10 03:34:39 [INFO] Writing to disk...
2021-06-10 03:34:43 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:55 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-02 17:08:55 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-02 17:08:55 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing DRX167729. (1/9)
2021-08-02 17:08:56 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-02 17:59:31 [INFO] Processing DRX181478. (2/9)
2021-08-02 17:59:31 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-02 18:13:43 [ERROR] Adding DRR191014:chr16_vs_chr5 to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 345. MiB for an array with shape (90338346,) and data type int32

2021-08-04 22:06:17 [INFO] Starting.
2021-08-04 22:06:17 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-04 22:06:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-04 22:06:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:06:18 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:06:18 [INFO] Writing every 10 Runs.
2021-08-04 22:06:18 [INFO] Processing DRX167729. (1/9)
2021-08-04 22:06:18 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-04 22:09:19 [INFO] Processing DRX181478. (2/9)
2021-08-04 22:09:19 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-04 22:15:58 [INFO] Processing DRX181480. (3/9)
2021-08-04 22:15:58 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-04 22:22:30 [INFO] Processing DRX181476. (4/9)
2021-08-04 22:22:30 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-04 22:28:51 [INFO] Processing DRX167730. (5/9)
2021-08-04 22:28:51 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-04 22:35:06 [INFO] Processing DRX181477. (6/9)
2021-08-04 22:35:06 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-04 22:41:26 [INFO] Processing DRX181475. (7/9)
2021-08-04 22:41:26 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-04 22:47:43 [INFO] Processing DRX181479. (8/9)
2021-08-04 22:47:43 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-04 22:54:03 [INFO] Processing DRX167737. (9/9)
2021-08-04 22:54:03 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-04 23:01:28 [INFO] Writing to disk...
2021-08-04 23:02:20 [INFO] Finished.
2021-08-12 14:31:16 [INFO] Starting.
2021-08-12 14:31:16 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-12 14:31:16 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-12 14:31:16 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:31:17 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:31:17 [INFO] Writing every 10 Runs.
2021-08-12 14:31:17 [INFO] Processing DRX167729. (1/9)
2021-08-12 14:31:17 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-12 14:31:20 [INFO] Processing DRX181478. (2/9)
2021-08-12 14:31:20 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-12 14:31:21 [INFO] Processing DRX181480. (3/9)
2021-08-12 14:31:21 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-12 14:31:24 [INFO] Processing DRX181476. (4/9)
2021-08-12 14:31:24 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-12 14:31:26 [INFO] Processing DRX167730. (5/9)
2021-08-12 14:31:26 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-12 14:31:28 [INFO] Processing DRX181477. (6/9)
2021-08-12 14:31:28 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-12 14:31:30 [INFO] Processing DRX181475. (7/9)
2021-08-12 14:31:30 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-12 14:31:32 [INFO] Processing DRX181479. (8/9)
2021-08-12 14:31:32 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-12 14:31:34 [INFO] Processing DRX167737. (9/9)
2021-08-12 14:31:34 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-12 14:32:01 [INFO] Writing to disk...
2021-08-12 14:32:04 [INFO] Finished.
2021-08-12 15:09:40 [INFO] Starting.
2021-08-12 15:09:40 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-12 15:09:40 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-12 15:09:40 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:09:40 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:09:40 [INFO] Writing every 10 Runs.
2021-08-12 15:09:40 [INFO] Processing DRX167729. (1/9)
2021-08-12 15:09:40 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-12 15:09:43 [INFO] Processing DRX181478. (2/9)
2021-08-12 15:09:43 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-12 15:09:44 [INFO] Processing DRX181480. (3/9)
2021-08-12 15:09:44 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-12 15:09:46 [INFO] Processing DRX181476. (4/9)
2021-08-12 15:09:46 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-12 15:09:48 [INFO] Processing DRX167730. (5/9)
2021-08-12 15:09:48 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-12 15:09:49 [INFO] Processing DRX181477. (6/9)
2021-08-12 15:09:49 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-12 15:09:51 [INFO] Processing DRX181475. (7/9)
2021-08-12 15:09:51 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-12 15:09:53 [INFO] Processing DRX181479. (8/9)
2021-08-12 15:09:53 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-12 15:09:55 [INFO] Processing DRX167737. (9/9)
2021-08-12 15:09:55 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-12 15:10:14 [INFO] Writing to disk...
2021-08-12 15:10:16 [INFO] Finished.
2021-08-12 15:44:39 [INFO] Starting.
2021-08-12 15:44:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-12 15:44:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-12 15:44:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:44:40 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:44:40 [INFO] Writing every 10 Runs.
2021-08-12 15:44:40 [INFO] Processing DRX167729. (1/9)
2021-08-12 15:44:40 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-12 15:44:43 [INFO] Processing DRX181478. (2/9)
2021-08-12 15:44:43 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-12 15:44:45 [INFO] Processing DRX181480. (3/9)
2021-08-12 15:44:45 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-12 15:44:47 [INFO] Processing DRX181476. (4/9)
2021-08-12 15:44:47 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-12 15:44:50 [INFO] Processing DRX167730. (5/9)
2021-08-12 15:44:50 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-12 15:44:51 [INFO] Processing DRX181477. (6/9)
2021-08-12 15:44:51 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-12 15:44:54 [INFO] Processing DRX181475. (7/9)
2021-08-12 15:44:54 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-12 15:44:56 [INFO] Processing DRX181479. (8/9)
2021-08-12 15:44:56 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-12 15:44:58 [INFO] Processing DRX167737. (9/9)
2021-08-12 15:44:58 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-12 15:45:22 [INFO] Writing to disk...
2021-08-12 15:45:25 [INFO] Finished.
2021-08-12 16:31:34 [INFO] Starting.
2021-08-12 16:31:34 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-12 16:31:34 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-12 16:31:34 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:31:35 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:31:35 [INFO] Writing every 10 Runs.
2021-08-12 16:31:35 [INFO] Processing DRX167729. (1/9)
2021-08-12 16:31:35 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-12 16:31:39 [INFO] Processing DRX181478. (2/9)
2021-08-12 16:31:39 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-12 16:31:43 [INFO] Processing DRX181480. (3/9)
2021-08-12 16:31:43 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-12 16:31:46 [INFO] Processing DRX181476. (4/9)
2021-08-12 16:31:46 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-12 16:31:50 [INFO] Processing DRX167730. (5/9)
2021-08-12 16:31:50 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-12 16:31:53 [INFO] Processing DRX181477. (6/9)
2021-08-12 16:31:53 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-12 16:31:56 [INFO] Processing DRX181475. (7/9)
2021-08-12 16:31:56 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-12 16:32:00 [INFO] Processing DRX181479. (8/9)
2021-08-12 16:32:00 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-12 16:32:04 [INFO] Processing DRX167737. (9/9)
2021-08-12 16:32:04 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-12 16:32:30 [INFO] Writing to disk...
2021-08-12 16:32:34 [INFO] Finished.
2021-08-20 17:56:12 [INFO] Starting.
2021-08-20 17:56:12 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-20 17:56:12 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-20 17:56:12 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 17:56:13 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 17:56:13 [INFO] Writing every 10 Runs.
2021-08-20 17:56:13 [INFO] Processing DRX167729. (1/9)
2021-08-20 17:56:13 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-20 17:56:16 [INFO] Processing DRX181478. (2/9)
2021-08-20 17:56:16 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-20 17:56:18 [INFO] Processing DRX181480. (3/9)
2021-08-20 17:56:18 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-20 17:56:19 [INFO] Processing DRX181476. (4/9)
2021-08-20 17:56:19 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-20 17:56:21 [INFO] Processing DRX167730. (5/9)
2021-08-20 17:56:21 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-20 17:56:23 [INFO] Processing DRX181477. (6/9)
2021-08-20 17:56:23 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-20 17:56:24 [INFO] Processing DRX181475. (7/9)
2021-08-20 17:56:24 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-20 17:56:26 [INFO] Processing DRX181479. (8/9)
2021-08-20 17:56:26 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-20 17:56:28 [INFO] Processing DRX167737. (9/9)
2021-08-20 17:56:28 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-20 17:56:46 [INFO] Writing to disk...
2021-08-20 17:56:48 [INFO] Finished.
2021-08-23 11:28:15 [INFO] Starting.
2021-08-23 11:28:15 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-23 11:28:15 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-23 11:28:15 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:28:16 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:28:16 [INFO] Writing every 10 Runs.
2021-08-23 11:28:16 [INFO] Processing DRX167729. (1/9)
2021-08-23 11:28:16 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-23 11:28:17 [INFO] Processing DRX181478. (2/9)
2021-08-23 11:28:17 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-23 11:28:18 [INFO] Processing DRX181480. (3/9)
2021-08-23 11:28:18 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-23 11:28:19 [INFO] Processing DRX181476. (4/9)
2021-08-23 11:28:19 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-23 11:28:21 [INFO] Processing DRX167730. (5/9)
2021-08-23 11:28:21 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-23 11:28:21 [INFO] Processing DRX181477. (6/9)
2021-08-23 11:28:21 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-23 11:28:23 [INFO] Processing DRX181475. (7/9)
2021-08-23 11:28:23 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-23 11:28:24 [INFO] Processing DRX181479. (8/9)
2021-08-23 11:28:24 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-23 11:28:25 [INFO] Processing DRX167737. (9/9)
2021-08-23 11:28:25 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-23 11:28:37 [INFO] Writing to disk...
2021-08-23 11:28:38 [INFO] Finished.
2021-08-23 13:16:02 [INFO] Starting.
2021-08-23 13:16:02 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/
2021-08-23 13:16:02 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/DRP005173/aggregate.hdf5
2021-08-23 13:16:02 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:16:03 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:16:03 [INFO] Writing every 10 Runs.
2021-08-23 13:16:03 [INFO] Processing DRX167729. (1/9)
2021-08-23 13:16:03 [INFO] Processing DRX167729-DRR177165. (1/9)
2021-08-23 13:16:04 [INFO] Processing DRX181478. (2/9)
2021-08-23 13:16:04 [INFO] Processing DRX181478-DRR191014. (2/9)
2021-08-23 13:16:05 [INFO] Processing DRX181480. (3/9)
2021-08-23 13:16:05 [INFO] Processing DRX181480-DRR191016. (3/9)
2021-08-23 13:16:06 [INFO] Processing DRX181476. (4/9)
2021-08-23 13:16:06 [INFO] Processing DRX181476-DRR191012. (4/9)
2021-08-23 13:16:07 [INFO] Processing DRX167730. (5/9)
2021-08-23 13:16:07 [INFO] Processing DRX167730-DRR177166. (5/9)
2021-08-23 13:16:08 [INFO] Processing DRX181477. (6/9)
2021-08-23 13:16:08 [INFO] Processing DRX181477-DRR191013. (6/9)
2021-08-23 13:16:09 [INFO] Processing DRX181475. (7/9)
2021-08-23 13:16:09 [INFO] Processing DRX181475-DRR191011. (7/9)
2021-08-23 13:16:10 [INFO] Processing DRX181479. (8/9)
2021-08-23 13:16:10 [INFO] Processing DRX181479-DRR191015. (8/9)
2021-08-23 13:16:11 [INFO] Processing DRX167737. (9/9)
2021-08-23 13:16:11 [INFO] Processing DRX167737-DRR177173. (9/9)
2021-08-23 13:16:19 [INFO] Writing to disk...
2021-08-23 13:16:20 [INFO] Finished.