2021-06-10 03:44:40 [INFO] Starting.
2021-06-10 03:44:40 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-06-10 03:44:40 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-06-10 03:44:40 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 03:44:40 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 03:44:41 [INFO] Building aggregate.
2021-06-10 03:44:41 [INFO] Writing every 10 Runs.
2021-06-10 03:44:41 [INFO] Processing SRX5388872. (1/4)
2021-06-10 03:44:41 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-06-10 03:45:21 [INFO] Processing SRX5388874. (2/4)
2021-06-10 03:45:21 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-06-10 03:45:42 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-06-10 03:46:03 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-06-10 03:46:35 [INFO] Processing SRX5388873. (3/4)
2021-06-10 03:46:35 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-06-10 03:46:55 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-06-10 03:47:27 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-06-10 03:47:42 [INFO] Processing SRX5388871. (4/4)
2021-06-10 03:47:42 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-06-10 03:48:17 [INFO] Writing to disk...
2021-06-10 03:48:44 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:55 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-02 17:08:55 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-02 17:08:55 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing SRX5388872. (1/4)
2021-08-02 17:08:56 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-02 18:08:09 [INFO] Processing SRX5388874. (2/4)
2021-08-02 18:08:09 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-02 18:17:50 [ERROR] Adding SRR8588410:chr2_vs_chr7 to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 924. MiB for an array with shape (242193530,) and data type int32

2021-08-04 22:06:36 [INFO] Starting.
2021-08-04 22:06:36 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-04 22:06:36 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-04 22:06:36 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:06:37 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:06:37 [INFO] Writing every 10 Runs.
2021-08-04 22:06:37 [INFO] Processing SRX5388872. (1/4)
2021-08-04 22:06:37 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-04 22:10:37 [INFO] Processing SRX5388874. (2/4)
2021-08-04 22:10:37 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-04 22:22:43 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-04 22:30:20 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-04 22:37:41 [INFO] Processing SRX5388873. (3/4)
2021-08-04 22:37:41 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-04 22:44:56 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-04 22:52:10 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-04 22:59:31 [INFO] Processing SRX5388871. (4/4)
2021-08-04 22:59:31 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-04 23:07:13 [INFO] Writing to disk...
2021-08-04 23:08:15 [INFO] Finished.
2021-08-12 14:31:17 [INFO] Starting.
2021-08-12 14:31:17 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-12 14:31:17 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-12 14:31:17 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:31:18 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:31:18 [INFO] Writing every 10 Runs.
2021-08-12 14:31:18 [INFO] Processing SRX5388872. (1/4)
2021-08-12 14:31:18 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-12 14:32:03 [INFO] Processing SRX5388874. (2/4)
2021-08-12 14:32:03 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-12 14:32:34 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-12 14:33:00 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-12 14:33:28 [INFO] Processing SRX5388873. (3/4)
2021-08-12 14:33:28 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-12 14:33:59 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-12 14:34:29 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-12 14:34:57 [INFO] Processing SRX5388871. (4/4)
2021-08-12 14:34:57 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-12 14:35:53 [INFO] Writing to disk...
2021-08-12 14:36:14 [INFO] Finished.
2021-08-12 15:09:39 [INFO] Starting.
2021-08-12 15:09:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-12 15:09:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-12 15:09:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:09:40 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:09:40 [INFO] Writing every 10 Runs.
2021-08-12 15:09:40 [INFO] Processing SRX5388872. (1/4)
2021-08-12 15:09:40 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-12 15:10:15 [INFO] Processing SRX5388874. (2/4)
2021-08-12 15:10:15 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-12 15:10:40 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-12 15:11:01 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-12 15:11:29 [INFO] Processing SRX5388873. (3/4)
2021-08-12 15:11:29 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-12 15:11:59 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-12 15:12:26 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-12 15:12:52 [INFO] Processing SRX5388871. (4/4)
2021-08-12 15:12:52 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-12 15:13:42 [INFO] Writing to disk...
2021-08-12 15:14:01 [INFO] Finished.
2021-08-12 15:44:39 [INFO] Starting.
2021-08-12 15:44:39 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-12 15:44:39 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-12 15:44:39 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:44:40 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:44:40 [INFO] Writing every 10 Runs.
2021-08-12 15:44:40 [INFO] Processing SRX5388872. (1/4)
2021-08-12 15:44:40 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-12 15:45:22 [INFO] Processing SRX5388874. (2/4)
2021-08-12 15:45:22 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-12 15:45:48 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-12 15:46:18 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-12 15:46:48 [INFO] Processing SRX5388873. (3/4)
2021-08-12 15:46:48 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-12 15:47:17 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-12 15:47:43 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-12 15:48:17 [INFO] Processing SRX5388871. (4/4)
2021-08-12 15:48:17 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-12 15:49:15 [INFO] Writing to disk...
2021-08-12 15:49:47 [INFO] Finished.
2021-08-12 16:31:34 [INFO] Starting.
2021-08-12 16:31:34 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-12 16:31:34 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-12 16:31:34 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:31:35 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:31:35 [INFO] Writing every 10 Runs.
2021-08-12 16:31:35 [INFO] Processing SRX5388872. (1/4)
2021-08-12 16:31:35 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-12 16:32:13 [INFO] Processing SRX5388874. (2/4)
2021-08-12 16:32:13 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-12 16:32:40 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-12 16:33:06 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-12 16:33:34 [INFO] Processing SRX5388873. (3/4)
2021-08-12 16:33:34 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-12 16:34:00 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-12 16:34:28 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-12 16:34:58 [INFO] Processing SRX5388871. (4/4)
2021-08-12 16:34:58 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-12 16:35:58 [INFO] Writing to disk...
2021-08-12 16:36:29 [INFO] Finished.
2021-08-20 17:57:45 [INFO] Starting.
2021-08-20 17:57:45 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-20 17:57:45 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-20 17:57:45 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 17:57:46 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 17:57:46 [INFO] Writing every 10 Runs.
2021-08-20 17:57:46 [INFO] Processing SRX5388872. (1/4)
2021-08-20 17:57:46 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-20 17:58:13 [INFO] Processing SRX5388874. (2/4)
2021-08-20 17:58:13 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-20 17:58:38 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-20 17:59:05 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-20 17:59:25 [INFO] Processing SRX5388873. (3/4)
2021-08-20 17:59:25 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-20 17:59:50 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-20 18:00:48 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-20 18:02:13 [INFO] Processing SRX5388871. (4/4)
2021-08-20 18:02:13 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-20 18:04:01 [INFO] Writing to disk...
2021-08-20 18:04:16 [INFO] Finished.
2021-08-23 11:29:33 [INFO] Starting.
2021-08-23 11:29:33 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-23 11:29:33 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-23 11:29:33 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:29:34 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:29:34 [INFO] Writing every 10 Runs.
2021-08-23 11:29:34 [INFO] Processing SRX5388872. (1/4)
2021-08-23 11:29:34 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-23 11:29:48 [INFO] Processing SRX5388874. (2/4)
2021-08-23 11:29:48 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-23 11:30:02 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-23 11:30:18 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-23 11:30:34 [INFO] Processing SRX5388873. (3/4)
2021-08-23 11:30:34 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-23 11:30:49 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-23 11:31:03 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-23 11:31:19 [INFO] Processing SRX5388871. (4/4)
2021-08-23 11:31:19 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-23 11:31:38 [INFO] Writing to disk...
2021-08-23 11:31:40 [INFO] Finished.
2021-08-23 13:16:24 [INFO] Starting.
2021-08-23 13:16:24 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/
2021-08-23 13:16:24 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP186190/aggregate.hdf5
2021-08-23 13:16:24 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:16:24 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:16:24 [INFO] Writing every 10 Runs.
2021-08-23 13:16:24 [INFO] Processing SRX5388872. (1/4)
2021-08-23 13:16:24 [INFO] Processing SRX5388872-SRR8588405. (1/8)
2021-08-23 13:16:33 [INFO] Processing SRX5388874. (2/4)
2021-08-23 13:16:33 [INFO] Processing SRX5388874-SRR8588410. (2/8)
2021-08-23 13:16:40 [INFO] Processing SRX5388874-SRR8588409. (3/8)
2021-08-23 13:16:46 [INFO] Processing SRX5388874-SRR8588411. (4/8)
2021-08-23 13:16:52 [INFO] Processing SRX5388873. (3/4)
2021-08-23 13:16:52 [INFO] Processing SRX5388873-SRR8588407. (5/8)
2021-08-23 13:16:59 [INFO] Processing SRX5388873-SRR8588408. (6/8)
2021-08-23 13:17:06 [INFO] Processing SRX5388873-SRR8588406. (7/8)
2021-08-23 13:17:14 [INFO] Processing SRX5388871. (4/4)
2021-08-23 13:17:14 [INFO] Processing SRX5388871-SRR8588404. (8/8)
2021-08-23 13:17:22 [INFO] Writing to disk...
2021-08-23 13:17:23 [INFO] Finished.