2021-06-10 04:24:10 [INFO] Starting.
2021-06-10 04:24:10 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-06-10 04:24:10 [INFO] Creating new aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-06-10 04:24:10 [INFO] Using chromosome sizes at /grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes
2021-06-10 04:24:10 [INFO] Using only main chromosomes: chr1, chr2, chr3, chr4, chr5, chr6, chr7, chrX, chr8, chr9, chr11, chr10, chr12, chr13, chr14, chr15, chr16, chr17, chr18, chr20, chr19, chrY, chr22, chr21, chrM
2021-06-10 04:24:10 [INFO] Building aggregate.
2021-06-10 04:24:10 [INFO] Writing every 10 Runs.
2021-06-10 04:24:10 [INFO] Processing SRX8330496. (1/2)
2021-06-10 04:24:10 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-06-10 04:24:21 [INFO] Processing SRX8330495. (2/2)
2021-06-10 04:24:21 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-06-10 04:24:35 [INFO] Writing to disk...
2021-06-10 04:24:38 [INFO] Finished.
2021-08-02 17:08:55 [INFO] Starting.
2021-08-02 17:08:55 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-02 17:08:55 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-02 17:08:55 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-02 17:08:56 [INFO] Building aggregate for 1bp_raw network.
2021-08-02 17:08:56 [INFO] Writing every 10 Runs.
2021-08-02 17:08:56 [INFO] Processing SRX8330496. (1/2)
2021-08-02 17:08:56 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-02 17:23:28 [INFO] Processing SRX8330495. (2/2)
2021-08-02 17:23:28 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-02 17:27:25 [ERROR] Adding SRR11777629:chr13_vs_chr2 to in memory aggregate failed. Error message:
Traceback (most recent call last):
  File "/grid/gillis/data/nfox/hi_c_data_processing/software/aggregate.py", line 428, in add_dataset
    aggregate_data[c] = aggregate_data[c] + net
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/base.py", line 415, in __add__
    return self._add_sparse(other)
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 359, in _add_sparse
    return self._binopt(other, '_plus_')
  File "/grid/gillis/home/nfox/software/miniconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 1226, in _binopt
    indptr = np.empty(self.indptr.shape, dtype=idx_dtype)
MemoryError: Unable to allocate 436. MiB for an array with shape (114364329,) and data type int32

2021-08-04 22:13:50 [INFO] Starting.
2021-08-04 22:13:50 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-04 22:13:50 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-04 22:13:50 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-04 22:13:51 [INFO] Building aggregate for 1bp_raw network.
2021-08-04 22:13:51 [INFO] Writing every 10 Runs.
2021-08-04 22:13:51 [INFO] Processing SRX8330496. (1/2)
2021-08-04 22:13:51 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-04 22:18:09 [INFO] Processing SRX8330495. (2/2)
2021-08-04 22:18:09 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-04 22:25:12 [INFO] Writing to disk...
2021-08-04 22:25:58 [INFO] Finished.
2021-08-12 14:34:24 [INFO] Starting.
2021-08-12 14:34:24 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-12 14:34:24 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-12 14:34:24 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 14:34:25 [INFO] Building aggregate for 40kbp_raw network.
2021-08-12 14:34:25 [INFO] Writing every 10 Runs.
2021-08-12 14:34:25 [INFO] Processing SRX8330496. (1/2)
2021-08-12 14:34:25 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-12 14:34:43 [INFO] Processing SRX8330495. (2/2)
2021-08-12 14:34:43 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-12 14:35:00 [INFO] Writing to disk...
2021-08-12 14:35:03 [INFO] Finished.
2021-08-12 15:13:01 [INFO] Starting.
2021-08-12 15:13:01 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-12 15:13:01 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-12 15:13:01 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:13:02 [INFO] Building aggregate for 25kbp_raw network.
2021-08-12 15:13:02 [INFO] Writing every 10 Runs.
2021-08-12 15:13:02 [INFO] Processing SRX8330496. (1/2)
2021-08-12 15:13:02 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-12 15:13:18 [INFO] Processing SRX8330495. (2/2)
2021-08-12 15:13:18 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-12 15:13:35 [INFO] Writing to disk...
2021-08-12 15:13:38 [INFO] Finished.
2021-08-12 15:47:48 [INFO] Starting.
2021-08-12 15:47:48 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-12 15:47:48 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-12 15:47:48 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 15:47:49 [INFO] Building aggregate for 5kbp_raw network.
2021-08-12 15:47:49 [INFO] Writing every 10 Runs.
2021-08-12 15:47:49 [INFO] Processing SRX8330496. (1/2)
2021-08-12 15:47:49 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-12 15:48:05 [INFO] Processing SRX8330495. (2/2)
2021-08-12 15:48:05 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-12 15:48:21 [INFO] Writing to disk...
2021-08-12 15:48:25 [INFO] Finished.
2021-08-12 16:36:37 [INFO] Starting.
2021-08-12 16:36:37 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-12 16:36:37 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-12 16:36:37 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-12 16:36:38 [INFO] Building aggregate for 1kbp_raw network.
2021-08-12 16:36:38 [INFO] Writing every 10 Runs.
2021-08-12 16:36:38 [INFO] Processing SRX8330496. (1/2)
2021-08-12 16:36:38 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-12 16:37:03 [INFO] Processing SRX8330495. (2/2)
2021-08-12 16:37:03 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-12 16:37:26 [INFO] Writing to disk...
2021-08-12 16:37:31 [INFO] Finished.
2021-08-20 18:04:50 [INFO] Starting.
2021-08-20 18:04:50 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-20 18:04:50 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-20 18:04:50 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-20 18:04:51 [INFO] Building aggregate for 100kbp_raw network.
2021-08-20 18:04:51 [INFO] Writing every 10 Runs.
2021-08-20 18:04:51 [INFO] Processing SRX8330496. (1/2)
2021-08-20 18:04:51 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-20 18:05:12 [INFO] Processing SRX8330495. (2/2)
2021-08-20 18:05:12 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-20 18:05:36 [INFO] Writing to disk...
2021-08-20 18:05:37 [INFO] Finished.
2021-08-23 11:31:35 [INFO] Starting.
2021-08-23 11:31:35 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-23 11:31:35 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-23 11:31:35 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 11:31:36 [INFO] Building aggregate for 250kbp_raw network.
2021-08-23 11:31:36 [INFO] Writing every 10 Runs.
2021-08-23 11:31:36 [INFO] Processing SRX8330496. (1/2)
2021-08-23 11:31:36 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-23 11:31:46 [INFO] Processing SRX8330495. (2/2)
2021-08-23 11:31:46 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-23 11:31:56 [INFO] Writing to disk...
2021-08-23 11:31:57 [INFO] Finished.
2021-08-23 13:17:28 [INFO] Starting.
2021-08-23 13:17:28 [INFO] Processing /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/
2021-08-23 13:17:28 [INFO] Appending to existing aggregate : /grid/gillis/data/nfox/hi_c_data_processing/data_human/SRP165232/aggregate.hdf5
2021-08-23 13:17:28 [WARNING] chroms=/grid/gillis/data/nfox/genomes/homo_sapiens_38/hg38.chrom.sizes is ignored due to existing aggregate.
2021-08-23 13:17:29 [INFO] Building aggregate for 500kbp_raw network.
2021-08-23 13:17:29 [INFO] Writing every 10 Runs.
2021-08-23 13:17:29 [INFO] Processing SRX8330496. (1/2)
2021-08-23 13:17:29 [INFO] Processing SRX8330496-SRR11777628. (1/2)
2021-08-23 13:17:32 [INFO] Processing SRX8330495. (2/2)
2021-08-23 13:17:32 [INFO] Processing SRX8330495-SRR11777629. (2/2)
2021-08-23 13:17:35 [INFO] Writing to disk...
2021-08-23 13:17:36 [INFO] Finished.