2021-06-10 04:10:39 [INFO] Starting.
2021-06-10 04:10:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-06-10 04:10:39 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-06-10 04:10:39 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 04:10:39 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 04:10:40 [INFO] Building aggregate.
2021-06-10 04:10:40 [INFO] Writing every 10 Runs.
2021-06-10 04:10:40 [INFO] Processing SRX8287328. (1/4)
2021-06-10 04:10:40 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-06-10 04:11:05 [INFO] Processing SRX8287327. (2/4)
2021-06-10 04:11:05 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-06-10 04:11:16 [INFO] Processing SRX8287330. (3/4)
2021-06-10 04:11:16 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-06-10 04:11:31 [INFO] Processing SRX8287329. (4/4)
2021-06-10 04:11:31 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-06-10 04:12:07 [INFO] Writing to disk...
2021-06-10 04:12:12 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:56 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-02 17:08:56 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-02 17:08:56 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing SRX8287328. (1/4)
2021-08-02 17:08:56 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-02 17:29:05 [INFO] Processing SRX8287327. (2/4)
2021-08-02 17:29:05 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-02 17:36:24 [ERROR] Adding SRR11728036:chr4_vs_chrM to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 726. MiB for an array with shape (190214556,) and data type int32

2021-08-04 22:08:59 [INFO] Starting.
2021-08-04 22:08:59 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-04 22:08:59 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-04 22:08:59 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:09:00 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:09:00 [INFO] Writing every 10 Runs.
2021-08-04 22:09:00 [INFO] Processing SRX8287328. (1/4)
2021-08-04 22:09:00 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-04 22:12:51 [INFO] Processing SRX8287327. (2/4)
2021-08-04 22:12:51 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-04 22:20:45 [INFO] Processing SRX8287330. (3/4)
2021-08-04 22:20:45 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-04 22:28:30 [INFO] Processing SRX8287329. (4/4)
2021-08-04 22:28:30 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-04 22:36:27 [INFO] Writing to disk...
2021-08-04 22:37:31 [INFO] Finished.
2021-08-12 14:32:41 [INFO] Starting.
2021-08-12 14:32:41 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-12 14:32:41 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-12 14:32:41 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:32:42 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:32:42 [INFO] Writing every 10 Runs.
2021-08-12 14:32:42 [INFO] Processing SRX8287328. (1/4)
2021-08-12 14:32:42 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-12 14:32:57 [INFO] Processing SRX8287327. (2/4)
2021-08-12 14:32:57 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-12 14:33:16 [INFO] Processing SRX8287330. (3/4)
2021-08-12 14:33:16 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-12 14:33:35 [INFO] Processing SRX8287329. (4/4)
2021-08-12 14:33:35 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-12 14:34:14 [INFO] Writing to disk...
2021-08-12 14:34:21 [INFO] Finished.
2021-08-12 15:11:38 [INFO] Starting.
2021-08-12 15:11:38 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-12 15:11:38 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-12 15:11:38 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:11:39 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:11:39 [INFO] Writing every 10 Runs.
2021-08-12 15:11:39 [INFO] Processing SRX8287328. (1/4)
2021-08-12 15:11:39 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-12 15:11:57 [INFO] Processing SRX8287327. (2/4)
2021-08-12 15:11:57 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-12 15:12:19 [INFO] Processing SRX8287330. (3/4)
2021-08-12 15:12:19 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-12 15:12:38 [INFO] Processing SRX8287329. (4/4)
2021-08-12 15:12:38 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-12 15:13:07 [INFO] Writing to disk...
2021-08-12 15:13:14 [INFO] Finished.
2021-08-12 15:46:14 [INFO] Starting.
2021-08-12 15:46:14 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-12 15:46:14 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-12 15:46:14 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:46:15 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:46:15 [INFO] Writing every 10 Runs.
2021-08-12 15:46:15 [INFO] Processing SRX8287328. (1/4)
2021-08-12 15:46:15 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-12 15:46:29 [INFO] Processing SRX8287327. (2/4)
2021-08-12 15:46:29 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-12 15:46:45 [INFO] Processing SRX8287330. (3/4)
2021-08-12 15:46:45 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-12 15:47:04 [INFO] Processing SRX8287329. (4/4)
2021-08-12 15:47:04 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-12 15:47:38 [INFO] Writing to disk...
2021-08-12 15:47:46 [INFO] Finished.
2021-08-12 16:35:20 [INFO] Starting.
2021-08-12 16:35:20 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-12 16:35:20 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-12 16:35:20 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:35:20 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:35:20 [INFO] Writing every 10 Runs.
2021-08-12 16:35:20 [INFO] Processing SRX8287328. (1/4)
2021-08-12 16:35:20 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-12 16:35:37 [INFO] Processing SRX8287327. (2/4)
2021-08-12 16:35:37 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-12 16:36:01 [INFO] Processing SRX8287330. (3/4)
2021-08-12 16:36:01 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-12 16:36:28 [INFO] Processing SRX8287329. (4/4)
2021-08-12 16:36:28 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-12 16:37:10 [INFO] Writing to disk...
2021-08-12 16:37:19 [INFO] Finished.
2021-08-20 18:02:12 [INFO] Starting.
2021-08-20 18:02:12 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-20 18:02:12 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-20 18:02:12 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 18:02:18 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 18:02:18 [INFO] Writing every 10 Runs.
2021-08-20 18:02:18 [INFO] Processing SRX8287328. (1/4)
2021-08-20 18:02:18 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-20 18:02:38 [INFO] Processing SRX8287327. (2/4)
2021-08-20 18:02:38 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-20 18:02:50 [INFO] Processing SRX8287330. (3/4)
2021-08-20 18:02:50 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-20 18:04:03 [INFO] Processing SRX8287329. (4/4)
2021-08-20 18:04:03 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-20 18:04:24 [INFO] Writing to disk...
2021-08-20 18:04:28 [INFO] Finished.
2021-08-23 11:30:53 [INFO] Starting.
2021-08-23 11:30:53 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-23 11:30:53 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-23 11:30:53 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:30:54 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:30:54 [INFO] Writing every 10 Runs.
2021-08-23 11:30:54 [INFO] Processing SRX8287328. (1/4)
2021-08-23 11:30:54 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-23 11:31:04 [INFO] Processing SRX8287327. (2/4)
2021-08-23 11:31:04 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-23 11:31:15 [INFO] Processing SRX8287330. (3/4)
2021-08-23 11:31:15 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-23 11:31:25 [INFO] Processing SRX8287329. (4/4)
2021-08-23 11:31:25 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-23 11:31:40 [INFO] Writing to disk...
2021-08-23 11:31:41 [INFO] Finished.
2021-08-23 13:17:11 [INFO] Starting.
2021-08-23 13:17:11 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/
2021-08-23 13:17:11 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP066852/aggregate.hdf5
2021-08-23 13:17:11 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:17:12 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:17:12 [INFO] Writing every 10 Runs.
2021-08-23 13:17:12 [INFO] Processing SRX8287328. (1/4)
2021-08-23 13:17:12 [INFO] Processing SRX8287328-SRR11728035. (1/4)
2021-08-23 13:17:19 [INFO] Processing SRX8287327. (2/4)
2021-08-23 13:17:19 [INFO] Processing SRX8287327-SRR11728036. (2/4)
2021-08-23 13:17:27 [INFO] Processing SRX8287330. (3/4)
2021-08-23 13:17:27 [INFO] Processing SRX8287330-SRR11728033. (3/4)
2021-08-23 13:17:34 [INFO] Processing SRX8287329. (4/4)
2021-08-23 13:17:34 [INFO] Processing SRX8287329-SRR11728034. (4/4)
2021-08-23 13:17:43 [INFO] Writing to disk...
2021-08-23 13:17:44 [INFO] Finished.