2021-06-10 03:29:07 [INFO] Starting.
2021-06-10 03:29:07 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-06-10 03:29:07 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-06-10 03:29:07 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 03:29:07 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 03:29:08 [INFO] Building aggregate.
2021-06-10 03:29:08 [INFO] Writing every 10 Runs.
2021-06-10 03:29:08 [INFO] Processing SRX4194158. (1/9)
2021-06-10 03:29:08 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-06-10 03:29:26 [INFO] Processing SRX4194163. (2/9)
2021-06-10 03:29:26 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-06-10 03:29:43 [INFO] Processing SRX4194159. (3/9)
2021-06-10 03:29:43 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-06-10 03:30:15 [INFO] Processing SRX4194160. (4/9)
2021-06-10 03:30:15 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-06-10 03:30:35 [INFO] Processing SRX4194156. (5/9)
2021-06-10 03:30:35 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-06-10 03:30:57 [INFO] Processing SRX4194157. (6/9)
2021-06-10 03:30:57 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-06-10 03:31:28 [INFO] Processing SRX4194162. (7/9)
2021-06-10 03:31:28 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-06-10 03:31:46 [INFO] Processing SRX4194161. (8/9)
2021-06-10 03:31:46 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-06-10 03:32:09 [INFO] Processing SRX4194164. (9/9)
2021-06-10 03:32:09 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-06-10 03:32:43 [INFO] Writing to disk...
2021-06-10 03:32:56 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:55 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-02 17:08:55 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-02 17:08:55 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing SRX4194158. (1/9)
2021-08-02 17:08:56 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-02 17:16:40 [INFO] Processing SRX4194163. (2/9)
2021-08-02 17:16:41 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-02 17:22:51 [ERROR] Adding SRR7291411:chr6_vs_chrM to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 652. MiB for an array with shape (170805980,) and data type int32

2021-08-04 22:05:55 [INFO] Starting.
2021-08-04 22:05:55 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-04 22:05:55 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-04 22:05:55 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:05:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:05:56 [INFO] Writing every 10 Runs.
2021-08-04 22:05:56 [INFO] Processing SRX4194158. (1/9)
2021-08-04 22:05:56 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-04 22:09:51 [INFO] Processing SRX4194163. (2/9)
2021-08-04 22:09:51 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-04 22:17:46 [INFO] Processing SRX4194159. (3/9)
2021-08-04 22:17:46 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-04 22:25:32 [INFO] Processing SRX4194160. (4/9)
2021-08-04 22:25:32 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-04 22:33:23 [INFO] Processing SRX4194156. (5/9)
2021-08-04 22:33:23 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-04 22:41:13 [INFO] Processing SRX4194157. (6/9)
2021-08-04 22:41:13 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-04 22:49:04 [INFO] Processing SRX4194162. (7/9)
2021-08-04 22:49:04 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-04 22:56:52 [INFO] Processing SRX4194161. (8/9)
2021-08-04 22:56:52 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-04 23:04:30 [INFO] Processing SRX4194164. (9/9)
2021-08-04 23:04:30 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-04 23:12:11 [INFO] Writing to disk...
2021-08-04 23:13:25 [INFO] Finished.
2021-08-12 14:31:17 [INFO] Starting.
2021-08-12 14:31:17 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-12 14:31:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-12 14:31:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:31:18 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:31:18 [INFO] Writing every 10 Runs.
2021-08-12 14:31:18 [INFO] Processing SRX4194158. (1/9)
2021-08-12 14:31:18 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-12 14:31:52 [INFO] Processing SRX4194163. (2/9)
2021-08-12 14:31:52 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-12 14:32:15 [INFO] Processing SRX4194159. (3/9)
2021-08-12 14:32:15 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-12 14:32:46 [INFO] Processing SRX4194160. (4/9)
2021-08-12 14:32:46 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-12 14:33:19 [INFO] Processing SRX4194156. (5/9)
2021-08-12 14:33:19 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-12 14:33:49 [INFO] Processing SRX4194157. (6/9)
2021-08-12 14:33:49 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-12 14:34:16 [INFO] Processing SRX4194162. (7/9)
2021-08-12 14:34:16 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-12 14:34:42 [INFO] Processing SRX4194161. (8/9)
2021-08-12 14:34:42 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-12 14:35:07 [INFO] Processing SRX4194164. (9/9)
2021-08-12 14:35:07 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-12 14:35:33 [INFO] Writing to disk...
2021-08-12 14:35:50 [INFO] Finished.
2021-08-12 15:09:39 [INFO] Starting.
2021-08-12 15:09:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-12 15:09:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-12 15:09:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:09:40 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:09:40 [INFO] Writing every 10 Runs.
2021-08-12 15:09:40 [INFO] Processing SRX4194158. (1/9)
2021-08-12 15:09:40 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-12 15:10:15 [INFO] Processing SRX4194163. (2/9)
2021-08-12 15:10:15 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-12 15:10:35 [INFO] Processing SRX4194159. (3/9)
2021-08-12 15:10:35 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-12 15:10:58 [INFO] Processing SRX4194160. (4/9)
2021-08-12 15:10:58 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-12 15:11:25 [INFO] Processing SRX4194156. (5/9)
2021-08-12 15:11:25 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-12 15:11:45 [INFO] Processing SRX4194157. (6/9)
2021-08-12 15:11:45 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-12 15:12:12 [INFO] Processing SRX4194162. (7/9)
2021-08-12 15:12:12 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-12 15:12:35 [INFO] Processing SRX4194161. (8/9)
2021-08-12 15:12:35 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-12 15:12:55 [INFO] Processing SRX4194164. (9/9)
2021-08-12 15:12:55 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-12 15:13:16 [INFO] Writing to disk...
2021-08-12 15:13:33 [INFO] Finished.
2021-08-12 15:44:39 [INFO] Starting.
2021-08-12 15:44:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-12 15:44:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-12 15:44:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:44:40 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:44:40 [INFO] Writing every 10 Runs.
2021-08-12 15:44:40 [INFO] Processing SRX4194158. (1/9)
2021-08-12 15:44:40 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-12 15:45:17 [INFO] Processing SRX4194163. (2/9)
2021-08-12 15:45:17 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-12 15:45:37 [INFO] Processing SRX4194159. (3/9)
2021-08-12 15:45:37 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-12 15:46:04 [INFO] Processing SRX4194160. (4/9)
2021-08-12 15:46:04 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-12 15:46:36 [INFO] Processing SRX4194156. (5/9)
2021-08-12 15:46:36 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-12 15:47:05 [INFO] Processing SRX4194157. (6/9)
2021-08-12 15:47:05 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-12 15:47:33 [INFO] Processing SRX4194162. (7/9)
2021-08-12 15:47:33 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-12 15:48:00 [INFO] Processing SRX4194161. (8/9)
2021-08-12 15:48:00 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-12 15:48:23 [INFO] Processing SRX4194164. (9/9)
2021-08-12 15:48:23 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-12 15:48:47 [INFO] Writing to disk...
2021-08-12 15:49:08 [INFO] Finished.
2021-08-12 16:31:34 [INFO] Starting.
2021-08-12 16:31:34 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-12 16:31:34 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-12 16:31:34 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:31:35 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:31:35 [INFO] Writing every 10 Runs.
2021-08-12 16:31:35 [INFO] Processing SRX4194158. (1/9)
2021-08-12 16:31:35 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-12 16:32:08 [INFO] Processing SRX4194163. (2/9)
2021-08-12 16:32:08 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-12 16:32:30 [INFO] Processing SRX4194159. (3/9)
2021-08-12 16:32:30 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-12 16:32:58 [INFO] Processing SRX4194160. (4/9)
2021-08-12 16:32:58 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-12 16:33:29 [INFO] Processing SRX4194156. (5/9)
2021-08-12 16:33:29 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-12 16:34:02 [INFO] Processing SRX4194157. (6/9)
2021-08-12 16:34:02 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-12 16:34:29 [INFO] Processing SRX4194162. (7/9)
2021-08-12 16:34:29 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-12 16:34:57 [INFO] Processing SRX4194161. (8/9)
2021-08-12 16:34:57 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-12 16:35:27 [INFO] Processing SRX4194164. (9/9)
2021-08-12 16:35:27 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-12 16:35:58 [INFO] Writing to disk...
2021-08-12 16:36:23 [INFO] Finished.
2021-08-20 17:55:31 [INFO] Starting.
2021-08-20 17:55:31 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-20 17:55:31 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-20 17:55:31 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 17:55:33 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 17:55:33 [INFO] Writing every 10 Runs.
2021-08-20 17:55:33 [INFO] Processing SRX4194158. (1/9)
2021-08-20 17:55:33 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-20 17:55:59 [INFO] Processing SRX4194163. (2/9)
2021-08-20 17:55:59 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-20 17:56:17 [INFO] Processing SRX4194159. (3/9)
2021-08-20 17:56:17 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-20 17:56:40 [INFO] Processing SRX4194160. (4/9)
2021-08-20 17:56:40 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-20 17:57:05 [INFO] Processing SRX4194156. (5/9)
2021-08-20 17:57:05 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-20 17:57:27 [INFO] Processing SRX4194157. (6/9)
2021-08-20 17:57:27 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-20 17:57:47 [INFO] Processing SRX4194162. (7/9)
2021-08-20 17:57:47 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-20 17:58:06 [INFO] Processing SRX4194161. (8/9)
2021-08-20 17:58:06 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-20 17:58:26 [INFO] Processing SRX4194164. (9/9)
2021-08-20 17:58:26 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-20 17:58:45 [INFO] Writing to disk...
2021-08-20 17:58:54 [INFO] Finished.
2021-08-23 11:28:00 [INFO] Starting.
2021-08-23 11:28:00 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-23 11:28:00 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-23 11:28:00 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:28:01 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:28:01 [INFO] Writing every 10 Runs.
2021-08-23 11:28:01 [INFO] Processing SRX4194158. (1/9)
2021-08-23 11:28:01 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-23 11:28:14 [INFO] Processing SRX4194163. (2/9)
2021-08-23 11:28:14 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-23 11:28:22 [INFO] Processing SRX4194159. (3/9)
2021-08-23 11:28:22 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-23 11:28:30 [INFO] Processing SRX4194160. (4/9)
2021-08-23 11:28:30 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-23 11:28:44 [INFO] Processing SRX4194156. (5/9)
2021-08-23 11:28:44 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-23 11:28:58 [INFO] Processing SRX4194157. (6/9)
2021-08-23 11:28:58 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-23 11:29:09 [INFO] Processing SRX4194162. (7/9)
2021-08-23 11:29:09 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-23 11:29:20 [INFO] Processing SRX4194161. (8/9)
2021-08-23 11:29:20 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-23 11:29:31 [INFO] Processing SRX4194164. (9/9)
2021-08-23 11:29:31 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-23 11:29:42 [INFO] Writing to disk...
2021-08-23 11:29:44 [INFO] Finished.
2021-08-23 13:15:17 [INFO] Starting.
2021-08-23 13:15:17 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/
2021-08-23 13:15:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP150259/aggregate.hdf5
2021-08-23 13:15:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:15:18 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:15:18 [INFO] Writing every 10 Runs.
2021-08-23 13:15:18 [INFO] Processing SRX4194158. (1/9)
2021-08-23 13:15:18 [INFO] Processing SRX4194158-SRR7291406. (1/9)
2021-08-23 13:15:26 [INFO] Processing SRX4194163. (2/9)
2021-08-23 13:15:26 [INFO] Processing SRX4194163-SRR7291411. (2/9)
2021-08-23 13:15:33 [INFO] Processing SRX4194159. (3/9)
2021-08-23 13:15:33 [INFO] Processing SRX4194159-SRR7291407. (3/9)
2021-08-23 13:15:40 [INFO] Processing SRX4194160. (4/9)
2021-08-23 13:15:40 [INFO] Processing SRX4194160-SRR7291408. (4/9)
2021-08-23 13:15:47 [INFO] Processing SRX4194156. (5/9)
2021-08-23 13:15:47 [INFO] Processing SRX4194156-SRR7291404. (5/9)
2021-08-23 13:15:54 [INFO] Processing SRX4194157. (6/9)
2021-08-23 13:15:54 [INFO] Processing SRX4194157-SRR7291405. (6/9)
2021-08-23 13:15:59 [INFO] Processing SRX4194162. (7/9)
2021-08-23 13:15:59 [INFO] Processing SRX4194162-SRR7291410. (7/9)
2021-08-23 13:16:07 [INFO] Processing SRX4194161. (8/9)
2021-08-23 13:16:07 [INFO] Processing SRX4194161-SRR7291409. (8/9)
2021-08-23 13:16:14 [INFO] Processing SRX4194164. (9/9)
2021-08-23 13:16:14 [INFO] Processing SRX4194164-SRR7291412. (9/9)
2021-08-23 13:16:20 [INFO] Writing to disk...
2021-08-23 13:16:21 [INFO] Finished.