Repository navigation
polars.exceptions.ComputeError: could not parse - as dtype i64 at column 'score' (column number 6) #35
Copy link
Copy link
Open
Description
Hello! I am running the latest version of the FIRE pipeline with ONT data. I have not had any issues before, but after the update I got the following error when running a new sample:
localrule fdr_track_chromosome:
input: temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz, results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl
output: temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
jobid: 0
reason: Forced execution
wildcards: sm=260Fiber, v=v0.1.1, chrom=GL456354.1
threads: 4
resources: mem_mb=94208, mem_mib=89844, disk_mb=4096, disk_mib=3907, tmpdir=/tmp/jsakr/36408502, runtime=120, slurm_partition=standard, slurm_account=seyedam_lab
Activating conda environment: ../../../../../../../data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_
[INFO][Time elapsed (ms) 874]: Read FDR table:
threshold FDR shuffled_bp real_bp
0 -1.00 1.000000 1.0 1.0
1 4.69 0.996092 192991831.0 193748972.0
2 4.70 0.996064 192981454.0 193744101.0
3 4.71 0.991812 191478713.0 193059441.0
4 4.72 0.991793 191464091.0 193048458.0
... ... ... ... ...
2147 45.11 0.000611 336.0 550139.0
2148 45.76 0.000419 219.0 522586.0
2149 47.61 0.000379 168.0 443844.0
2150 54.92 0.000219 46.0 209682.0
2151 55.13 0.000000 0.0 205122.0
[2152 rows x 4 columns]
Traceback (most recent call last):
File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 333, in <module>
defopt.run(main, show_types=True, version="0.0.1")
File "/data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_/lib/python3.10/site-packages/defopt.py", line 356, in run
return call()
File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 324, in main
apply_fdr_table(infile, outfile, fdr_table, nrows)
File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 205, in apply_fdr_table
pileup = read_pileup_file(infile, nrows)
File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 48, in read_pileup_file
header = my_read_csv(infile, separator="\t", n_rows=1).columns
File "/data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py", line 31, in my_read_csv
result = pl.read_csv(*args, **kwargs)
File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
return function(*args, **kwargs)
File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
return function(*args, **kwargs)
File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/_utils/deprecation.py", line 135, in wrapper
return function(*args, **kwargs)
[Previous line repeated 1 more time]
File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/io/csv/functions.py", line 422, in read_csv
df = _read_csv_impl(
File "/data/homezvol3/jsakr/.local/lib/python3.10/site-packages/polars/io/csv/functions.py", line 568, in _read_csv_impl
pydf = PyDataFrame.read_csv(
polars.exceptions.ComputeError: could not parse `-` as dtype `i64` at column 'score' (column number 6)
The current offset in the file is 8191 bytes.
You might want to try:
- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),
- specifying correct dtype with the `dtypes` argument
- setting `ignore_errors` to `True`,
- adding `-` to the `null_values` list.
Original error: ```remaining bytes non-empty```
[Tue Mar 11 18:22:31 2025]
Error in rule fdr_track_chromosome:
jobid: 0
input: temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz, results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl
output: temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
conda-env: /data/homezvol3/jsakr/snakemake-conda-envs/fc5018ac87025e0ae5ebc0630d3241a7_
shell:
python /data/homezvol3/jsakr/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp5ngbjd72/file/data/homezvol3/jsakr/packages/FIREv0.1.1/workflow/rules/../scripts/fdr-table.py -v 1 --fdr-table results/260Fiber/additional-outputs-v0.1.1/fire-peaks/260Fiber-v0.1.1-fire-score-to-fdr.tbl temp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz temp/260Fiber/fire-peaks/v0.1.1-GL456354.1-FDR.track.bed
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Any suggestions? Thank you!
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
mrvollger commentedon Mar 12, 2025
Odd, Not sure what would cause this. Sorry! Any chance this is data you can share, or a test case you can make with a subset of the data that I can play with?
jazsakr commentedon Mar 13, 2025
Yes! I added reads from the chromosome that threw the error here. Thank you!
mrvollger commentedon Mar 13, 2025
Thanks! Can you also share the reference or relevant contig so the I can run the whole pipeline?
jazsakr commentedon Mar 13, 2025
Okie, I think I extracted that particular chromosome correctly and uploaded it to the same folder. If not, you can download the reference with
wget https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_mouse/release_M36/GRCm39.genome.fa.gzmrvollger commentedon Mar 17, 2025
I am unable to replicate a
-in column 6 oftemp/260Fiber/v0.1.1-GL456354.1.pileup.bed.gz. Can you confirm you see it there in your file?jazsakr commentedon Mar 17, 2025
I don't see it in my file either (I uploaded it into the same folder). Did it not error out for you? What else should I check?
mrvollger commentedon Mar 17, 2025
Without the full dataset I am not able to run the pipeline to completely recreate this so I was checking the pileup tbl generated from the bam file you uploaded. If you also upload the FDR tbl I can try this one python command in isolation and I'll see if that recreates it.
jazsakr commentedon Mar 17, 2025
Uploaded!
mrvollger commentedon Mar 18, 2025
I am not able to recreate this with the single line command. If you remove the inputs to this rule and rerun do you have the same issue?
jazsakr commentedon Mar 18, 2025
I removed
temp/260Fiber/*andresults/260Fiber/additional-outputs-v0.1.1/before rerunning and I still got the same error. I should also mention that this happens to multiple chromosomes, not just this one.mrvollger commentedon Mar 20, 2025
Hmm, I seem unable to replicate this with the smaller fileset. Do you have a way to share the full one?
jazsakr commentedon Mar 21, 2025
We are planning on publicly releasing the data sometime next week. I can share it with you once it is uploaded, I will let you know. Is there anything else I can try in the meantime?
mrvollger commentedon Mar 22, 2025
You could try altering the polars version in the environment yaml in case it is a bug in a particular patch of polars. Another idea would be to arbitrarily rename the contigs and realign incase there is some weird corner case being hit with contig names. Though I have a hard time imagining what that could be.
I'll let you know if I have any other one off ideas.
dverac commentedon May 21, 2025
I am having a similar error, also with a Nanopore data with good coverage