Nextflow with SLURM Tutorial
Contents
Nextflow with SLURM Tutorial#
Let’s run a Nextflow pipeline.
[1]:
module load pcluster-helpers
[2]:
pcluster-helper --help
Usage: pcluster-helper [OPTIONS] COMMAND [ARGS]...
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --install-completion [bash|zsh|fish|powershe Install completion for │
│ ll|pwsh] the specified shell. │
│ [default: None] │
│ --show-completion [bash|zsh|fish|powershe Show completion for the │
│ ll|pwsh] specified shell, to │
│ copy it or customize │
│ the installation. │
│ [default: None] │
│ --help Show this message and │
│ exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ gen-nxf-slurm-config Generate a slurm.config for nextflow that is │
│ compatible with your cluster. │
│ sinfo A more helpful sinfo │
╰──────────────────────────────────────────────────────────────────────────────╯
Generate a Nextflow slurm.config#
We’ll use the pcluster-helper gen-nxf-slurm-config in order to generate a default slurm configuration file.
[3]:
pcluster-helper gen-nxf-slurm-config --help
Usage: pcluster-helper gen-nxf-slurm-config [OPTIONS]
Generate a slurm.config for nextflow that is compatible with your cluster.
You will see a process label for each partition/node type.
Use the configuration in your process by setting the label to match the label
in the config.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --include-memory --no-include-memory Include scheduleable │
│ memory │
│ [default: │
│ no-include-memory] │
│ --scheduleable-memory FLOAT Schedulable amount of │
│ memory. Default is │
│ 95% │
│ [default: 0.95] │
│ --output TEXT Path to nextflow │
│ slurm config file. │
│ [default: None] │
│ --overwrite --no-overwrite Overwrite existing │
│ file. │
│ [default: │
│ no-overwrite] │
│ --stdout --no-stdout Write slurm config to │
│ stdout │
│ [default: no-stdout] │
│ --help Show this message and │
│ exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
One time Nextflow SLURM Setup#
If you want Nextflow to distribute your jobs using the SLURM cluster you’ll need to generate a SLURM executor config that Nextflow understands.
You can generate this once, and continue to use it for all Nextflow pipelines.
pcluster-helper \
gen-nxf-slurm-config \
--include-memory \
--output ~/slurm.config \
--overwrite
We’ll also want to create a default configuration for jobs that don’t have a process tag. I’ll choose a small one for this demonstration, but you should choose which instance is best for your workflows.
[4]:
cat > ./slurm-default.config <<'EOF'
process {
executor='slurm'
queue = 'dev'
cpus = 8
memory = '30GB'
clusterOptions = '--exclusive --constraint m5a2xlarge'
}
EOF
[5]:
# cleanup
rm -rf test.config
rm -rf test.config.*
rm -rf samplesheet_test.csv
rm -rf samplesheet_test.csv.*
#rm -rf .nextflow
#sleep 1m
#rm -rf .nextflow*
#rm -rf work
#rm -rf results
[6]:
wget --quiet https://raw.githubusercontent.com/nf-core/rnaseq/master/conf/test.config
[7]:
wget --quiet https://raw.githubusercontent.com/nf-core/test-datasets/rnaseq/samplesheet/v3.4/samplesheet_test.csv
cat samplesheet_test.csv |wc -l
8
[8]:
head -n 3 samplesheet_test.csv
sample,fastq_1,fastq_2,strandedness
WT_REP1,s3://nf-core-awsmegatests/rnaseq/input_data/minimal/GSE110004/SRR6357070_1.fastq.gz,s3://nf-core-awsmegatests/rnaseq/input_data/minimal/GSE110004/SRR6357070_2.fastq.gz,reverse
WT_REP1,s3://nf-core-awsmegatests/rnaseq/input_data/minimal/GSE110004/SRR6357071_1.fastq.gz,s3://nf-core-awsmegatests/rnaseq/input_data/minimal/GSE110004/SRR6357071_2.fastq.gz,reverse
Reduce the number of samples#
The samplesheet has 8 rows, and I don’t want to actually run 8. I’ll run the first sample.
[9]:
cat samplesheet_test.csv |head -n 2 > samplesheet_test_t.csv ; mv samplesheet_test_t.csv samplesheet_test.csv
[10]:
module load nextflow
[11]:
#nextflow -h
[12]:
#nextflow run -h
[13]:
export NXF_ANSI_LOG="False"
export NXF_CONDA_CACHEDIR="${HOME}/.conda/envs"
export NXF_SINGULARITY_CACHEDIR="${HOME}/.singularity"
module load nextflow
module load singularity-3.8.5-gcc-7.3.1-g2xhg2m
nextflow \
run \
nf-core/rnaseq \
-with-trace \
-with-report \
-with-dag \
-w ./work \
--input ./samplesheet_test.csv \
-resume \
-profile slurm \
-c test.config \
-c slurm-default.config \
-c ~/slurm.config \
--outdir ./results
exit 0
Lmod has detected the following error: The following module(s) are
unknown: "singularity-3.8.5-gcc-7.3.1-g2xhg2m"
Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
$ module --ignore_cache load "singularity-3.8.5-gcc-7.3.1-g2xhg2m"
Also make sure that all modulefiles written in TCL start with the string
#%Module
N E X T F L O W ~ version 22.10.1
WARN: It appears you have never run this project before -- Option `-resume` is ignored
Launching `https://github.com/nf-core/rnaseq` [curious_jennings] DSL2 - revision: e049f51f02 [master]
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/rnaseq v3.9
------------------------------------------------------
Core Nextflow options
revision : master
runName : curious_jennings
containerEngine : singularity
launchDir : /home/jillian/bioanalyze-hpc/docs/workflow-managers/nextflow
workDir : /home/jillian/bioanalyze-hpc/docs/workflow-managers/nextflow/work
projectDir : /home/jillian/.nextflow/assets/nf-core/rnaseq
userName : jillian
profile : slurm
configFiles : /home/jillian/.nextflow/assets/nf-core/rnaseq/nextflow.config, /home/jillian/bioanalyze-hpc/docs/workflow-managers/nextflow/test.config, /home/jillian/bioanalyze-hpc/docs/workflow-managers/nextflow/slurm-default.config, /home/jillian/slurm.config
Input/output options
input : ./samplesheet_test.csv
outdir : ./results
UMI options
umitools_bc_pattern : NNNN
umitools_bc_pattern2 : null
umitools_umi_separator : null
Read filtering options
bbsplit_fasta_list : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/bbsplit_fasta_list.txt
skip_bbsplit : false
Reference genome options
fasta : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/genome.fasta
gtf : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/genes.gtf.gz
gff : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/genes.gff.gz
transcript_fasta : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/transcriptome.fasta
additional_fasta : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/gfp.fa.gz
hisat2_index : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/hisat2.tar.gz
rsem_index : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/rsem.tar.gz
salmon_index : https://github.com/nf-core/test-datasets/raw/rnaseq/reference/salmon.tar.gz
Alignment options
pseudo_aligner : salmon
Institutional config options
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/rnaseq for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.1400710
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/rnaseq/blob/master/CITATIONS.md
------------------------------------------------------
WARN: =============================================================================
Both '--gtf' and '--gff' parameters have been provided.
Using GTF file as priority.
===================================================================================
WARN: =============================================================================
'--transcript_fasta' parameter has been provided.
Make sure transcript names in this file match those in the GFF/GTF file.
Please see:
https://github.com/nf-core/rnaseq/issues/753
===================================================================================
[32/477cdf] Submitted process > NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)
[a1/7b0bb7] Submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)
[db/614a80] Submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)
[55/3843c8] Submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)
[32/477cdf] NOTE: Process `NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)` terminated with an error exit status (126) -- Execution is retried (1)
[a1/7b0bb7] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)` terminated with an error exit status (126) -- Execution is retried (1)
[db/614a80] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)` terminated with an error exit status (126) -- Execution is retried (1)
[55/3843c8] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)` terminated with an error exit status (126) -- Execution is retried (1)
[10/efe200] Re-submitted process > NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)
[2d/71a190] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)
[4f/0f09ae] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)
[ef/0c4203] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)
[10/efe200] NOTE: Process `NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)` terminated with an error exit status (126) -- Execution is retried (2)
[2d/71a190] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)` terminated with an error exit status (126) -- Execution is retried (2)
[4f/0f09ae] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)` terminated with an error exit status (126) -- Execution is retried (2)
[ef/0c4203] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)` terminated with an error exit status (126) -- Execution is retried (2)
[03/0339e6] Re-submitted process > NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)
[13/c2102d] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)
[d6/baf492] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)
[41/169bc1] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)
[03/0339e6] NOTE: Process `NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)` terminated with an error exit status (126) -- Execution is retried (3)
[13/c2102d] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)` terminated with an error exit status (126) -- Execution is retried (3)
[d6/baf492] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)` terminated with an error exit status (126) -- Execution is retried (3)
[41/169bc1] NOTE: Process `NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)` terminated with an error exit status (126) -- Execution is retried (3)
[de/8ae4a4] Re-submitted process > NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)
[ef/287827] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:UNTAR_SALMON_INDEX (salmon.tar.gz)
[b7/64a360] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_ADDITIONAL_FASTA (gfp.fa.gz)
[1b/5fe614] Re-submitted process > NFCORE_RNASEQ:RNASEQ:PREPARE_GENOME:GUNZIP_GTF (genes.gtf.gz)
Error executing process > 'NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)'
Caused by:
Process `NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK (samplesheet_test.csv)` terminated with an error exit status (126)
Command executed:
check_samplesheet.py \
samplesheet_test.csv \
samplesheet.valid.csv
cat <<-END_VERSIONS > versions.yml
"NFCORE_RNASEQ:RNASEQ:INPUT_CHECK:SAMPLESHEET_CHECK":
python: $(python --version | sed 's/Python //g')
END_VERSIONS
Command exit status:
126
Command output:
(empty)
Command error:
env: ‘singularity’: Permission denied
Work dir:
/home/jillian/bioanalyze-hpc/docs/workflow-managers/nextflow/work/de/8ae4a4a5a39eb12e63e180d742e2da
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-[nf-core/rnaseq] Pipeline completed with errors-
WARN: Killing running tasks (2)
exit
Restarting Bash
[14]:
tree results
results
├── bbmap
│ ├── bbsplit
│ │ └── ref
│ │ ├── genome
│ │ │ └── 1
│ │ │ ├── chr1.chrom.gz
│ │ │ ├── info.txt
│ │ │ ├── merged_ref_8374379829187813017.fa.gz
│ │ │ ├── namelist.txt
│ │ │ ├── reflist.txt
│ │ │ ├── scaffolds.txt.gz
│ │ │ └── summary.txt
│ │ └── index
│ │ └── 1
│ │ ├── chr1_index_k13_c12_b1.block
│ │ └── chr1_index_k13_c12_b1.block2.gz
│ ├── WT_REP1_human_1.fastq.gz
│ ├── WT_REP1_human_2.fastq.gz
│ ├── WT_REP1_primary_1.fastq.gz
│ ├── WT_REP1_primary_2.fastq.gz
│ ├── WT_REP1_sarscov2_1.fastq.gz
│ ├── WT_REP1_sarscov2_2.fastq.gz
│ └── WT_REP1.stats.txt
├── deseq2
│ └── deseq2.dds.RData
├── fastqc
│ ├── WT_REP1_1_fastqc.html
│ ├── WT_REP1_1_fastqc.zip
│ ├── WT_REP1_2_fastqc.html
│ └── WT_REP1_2_fastqc.zip
├── multiqc
│ └── star_salmon
│ ├── multiqc_data
│ │ ├── junction_saturation_known.txt
│ │ ├── junction_saturation_novel.txt
│ │ ├── mqc_cutadapt_filtered_reads_plot_1.txt
│ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.txt
│ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.txt
│ │ ├── mqc_fastqc_adapter_content_plot_1.txt
│ │ ├── mqc_fastqc_overrepresented_sequences_plot_1.txt
│ │ ├── mqc_fastqc_overrepresented_sequences_plot-2_1.txt
│ │ ├── mqc_fastqc_per_base_n_content_plot_1.txt
│ │ ├── mqc_fastqc_per_base_n_content_plot-2_1.txt
│ │ ├── mqc_fastqc_per_base_sequence_quality_plot_1.txt
│ │ ├── mqc_fastqc_per_base_sequence_quality_plot-2_1.txt
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Counts.txt
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Percentages.txt
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.txt
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.txt
│ │ ├── mqc_fastqc_per_sequence_quality_scores_plot_1.txt
│ │ ├── mqc_fastqc_per_sequence_quality_scores_plot-2_1.txt
│ │ ├── mqc_fastqc_sequence_counts_plot_1.txt
│ │ ├── mqc_fastqc_sequence_counts_plot-2_1.txt
│ │ ├── mqc_fastqc_sequence_duplication_levels_plot_1.txt
│ │ ├── mqc_fastqc_sequence_duplication_levels_plot-2_1.txt
│ │ ├── mqc_fastqc_sequence_length_distribution_plot_1.txt
│ │ ├── mqc_picard_deduplication_1.txt
│ │ ├── mqc_preseq_plot_1.txt
│ │ ├── mqc_qualimap_gene_coverage_profile_Counts.txt
│ │ ├── mqc_qualimap_gene_coverage_profile_Normalised.txt
│ │ ├── mqc_qualimap_genomic_origin_1.txt
│ │ ├── mqc_rseqc_infer_experiment_plot_1.txt
│ │ ├── mqc_rseqc_inner_distance_plot_Counts.txt
│ │ ├── mqc_rseqc_inner_distance_plot_Percentages.txt
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events.txt
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions.txt
│ │ ├── mqc_rseqc_junction_saturation_plot_All_Junctions.txt
│ │ ├── mqc_rseqc_junction_saturation_plot_Known_Junctions.txt
│ │ ├── mqc_rseqc_junction_saturation_plot_Novel_Junctions.txt
│ │ ├── mqc_rseqc_read_distribution_plot_1.txt
│ │ ├── mqc_rseqc_read_dups_plot_1.txt
│ │ ├── mqc_salmon_plot_1.txt
│ │ ├── mqc_samtools_alignment_plot_1.txt
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Normalised_Counts.txt
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Observed_over_Expected_Counts.txt
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Raw_Counts.txt
│ │ ├── mqc_star_alignment_plot_1.txt
│ │ ├── multiqc_citations.txt
│ │ ├── multiqc_cutadapt.txt
│ │ ├── multiqc_data.json
│ │ ├── multiqc_dupradar-plot.txt
│ │ ├── multiqc_fastqc_1.txt
│ │ ├── multiqc_fastqc.txt
│ │ ├── multiqc_featurecounts_biotype_plot.txt
│ │ ├── multiqc_general_stats.txt
│ │ ├── multiqc.log
│ │ ├── multiqc_picard_dups.txt
│ │ ├── multiqc_rseqc_bam_stat.txt
│ │ ├── multiqc_rseqc_infer_experiment.txt
│ │ ├── multiqc_rseqc_junction_annotation.txt
│ │ ├── multiqc_rseqc_read_distribution.txt
│ │ ├── multiqc_salmon.txt
│ │ ├── multiqc_samtools_flagstat.txt
│ │ ├── multiqc_samtools_idxstats.txt
│ │ ├── multiqc_samtools_stats.txt
│ │ ├── multiqc_sources.txt
│ │ ├── multiqc_star.txt
│ │ ├── picard_histogram_1.txt
│ │ ├── picard_histogram_2.txt
│ │ ├── picard_histogram.txt
│ │ ├── preseq.txt
│ │ ├── qualimap_rnaseq_cov_hist.txt
│ │ ├── qualimap_rnaseq_genome_results.txt
│ │ ├── rseqc_inner_distance.txt
│ │ ├── rseqc_junction_saturation_all.txt
│ │ └── rseqc_read_dups.txt
│ ├── multiqc_plots
│ │ ├── pdf
│ │ │ ├── mqc_cutadapt_filtered_reads_plot_1_pc.pdf
│ │ │ ├── mqc_cutadapt_filtered_reads_plot_1.pdf
│ │ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.pdf
│ │ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.pdf
│ │ │ ├── mqc_dupradar-plot_1.pdf
│ │ │ ├── mqc_fastqc_adapter_content_plot_1.pdf
│ │ │ ├── mqc_fastqc_overrepresented_sequences_plot_1.pdf
│ │ │ ├── mqc_fastqc_overrepresented_sequences_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_per_base_n_content_plot_1.pdf
│ │ │ ├── mqc_fastqc_per_base_n_content_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_per_base_sequence_quality_plot_1.pdf
│ │ │ ├── mqc_fastqc_per_base_sequence_quality_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Counts.pdf
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Percentages.pdf
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.pdf
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.pdf
│ │ │ ├── mqc_fastqc_per_sequence_quality_scores_plot_1.pdf
│ │ │ ├── mqc_fastqc_per_sequence_quality_scores_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_sequence_counts_plot_1_pc.pdf
│ │ │ ├── mqc_fastqc_sequence_counts_plot_1.pdf
│ │ │ ├── mqc_fastqc_sequence_counts_plot-2_1_pc.pdf
│ │ │ ├── mqc_fastqc_sequence_counts_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_sequence_duplication_levels_plot_1.pdf
│ │ │ ├── mqc_fastqc_sequence_duplication_levels_plot-2_1.pdf
│ │ │ ├── mqc_fastqc_sequence_length_distribution_plot_1.pdf
│ │ │ ├── mqc_featurecounts_biotype_plot_1_pc.pdf
│ │ │ ├── mqc_featurecounts_biotype_plot_1.pdf
│ │ │ ├── mqc_picard_deduplication_1_pc.pdf
│ │ │ ├── mqc_picard_deduplication_1.pdf
│ │ │ ├── mqc_preseq_plot_1.pdf
│ │ │ ├── mqc_qualimap_gene_coverage_profile_Counts.pdf
│ │ │ ├── mqc_qualimap_gene_coverage_profile_Normalised.pdf
│ │ │ ├── mqc_qualimap_genomic_origin_1_pc.pdf
│ │ │ ├── mqc_qualimap_genomic_origin_1.pdf
│ │ │ ├── mqc_rseqc_infer_experiment_plot_1.pdf
│ │ │ ├── mqc_rseqc_inner_distance_plot_Counts.pdf
│ │ │ ├── mqc_rseqc_inner_distance_plot_Percentages.pdf
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events_pc.pdf
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events.pdf
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions_pc.pdf
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions.pdf
│ │ │ ├── mqc_rseqc_junction_saturation_plot_All_Junctions.pdf
│ │ │ ├── mqc_rseqc_junction_saturation_plot_Known_Junctions.pdf
│ │ │ ├── mqc_rseqc_junction_saturation_plot_Novel_Junctions.pdf
│ │ │ ├── mqc_rseqc_read_distribution_plot_1_pc.pdf
│ │ │ ├── mqc_rseqc_read_distribution_plot_1.pdf
│ │ │ ├── mqc_rseqc_read_dups_plot_1.pdf
│ │ │ ├── mqc_salmon_plot_1.pdf
│ │ │ ├── mqc_samtools_alignment_plot_1_pc.pdf
│ │ │ ├── mqc_samtools_alignment_plot_1.pdf
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Normalised_Counts.pdf
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Observed_over_Expected_Counts.pdf
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Raw_Counts.pdf
│ │ │ ├── mqc_star_alignment_plot_1_pc.pdf
│ │ │ └── mqc_star_alignment_plot_1.pdf
│ │ ├── png
│ │ │ ├── mqc_cutadapt_filtered_reads_plot_1_pc.png
│ │ │ ├── mqc_cutadapt_filtered_reads_plot_1.png
│ │ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.png
│ │ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.png
│ │ │ ├── mqc_dupradar-plot_1.png
│ │ │ ├── mqc_fastqc_adapter_content_plot_1.png
│ │ │ ├── mqc_fastqc_overrepresented_sequences_plot_1.png
│ │ │ ├── mqc_fastqc_overrepresented_sequences_plot-2_1.png
│ │ │ ├── mqc_fastqc_per_base_n_content_plot_1.png
│ │ │ ├── mqc_fastqc_per_base_n_content_plot-2_1.png
│ │ │ ├── mqc_fastqc_per_base_sequence_quality_plot_1.png
│ │ │ ├── mqc_fastqc_per_base_sequence_quality_plot-2_1.png
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Counts.png
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Percentages.png
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.png
│ │ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.png
│ │ │ ├── mqc_fastqc_per_sequence_quality_scores_plot_1.png
│ │ │ ├── mqc_fastqc_per_sequence_quality_scores_plot-2_1.png
│ │ │ ├── mqc_fastqc_sequence_counts_plot_1_pc.png
│ │ │ ├── mqc_fastqc_sequence_counts_plot_1.png
│ │ │ ├── mqc_fastqc_sequence_counts_plot-2_1_pc.png
│ │ │ ├── mqc_fastqc_sequence_counts_plot-2_1.png
│ │ │ ├── mqc_fastqc_sequence_duplication_levels_plot_1.png
│ │ │ ├── mqc_fastqc_sequence_duplication_levels_plot-2_1.png
│ │ │ ├── mqc_fastqc_sequence_length_distribution_plot_1.png
│ │ │ ├── mqc_featurecounts_biotype_plot_1_pc.png
│ │ │ ├── mqc_featurecounts_biotype_plot_1.png
│ │ │ ├── mqc_picard_deduplication_1_pc.png
│ │ │ ├── mqc_picard_deduplication_1.png
│ │ │ ├── mqc_preseq_plot_1.png
│ │ │ ├── mqc_qualimap_gene_coverage_profile_Counts.png
│ │ │ ├── mqc_qualimap_gene_coverage_profile_Normalised.png
│ │ │ ├── mqc_qualimap_genomic_origin_1_pc.png
│ │ │ ├── mqc_qualimap_genomic_origin_1.png
│ │ │ ├── mqc_rseqc_infer_experiment_plot_1.png
│ │ │ ├── mqc_rseqc_inner_distance_plot_Counts.png
│ │ │ ├── mqc_rseqc_inner_distance_plot_Percentages.png
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events_pc.png
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events.png
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions_pc.png
│ │ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions.png
│ │ │ ├── mqc_rseqc_junction_saturation_plot_All_Junctions.png
│ │ │ ├── mqc_rseqc_junction_saturation_plot_Known_Junctions.png
│ │ │ ├── mqc_rseqc_junction_saturation_plot_Novel_Junctions.png
│ │ │ ├── mqc_rseqc_read_distribution_plot_1_pc.png
│ │ │ ├── mqc_rseqc_read_distribution_plot_1.png
│ │ │ ├── mqc_rseqc_read_dups_plot_1.png
│ │ │ ├── mqc_salmon_plot_1.png
│ │ │ ├── mqc_samtools_alignment_plot_1_pc.png
│ │ │ ├── mqc_samtools_alignment_plot_1.png
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Normalised_Counts.png
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Observed_over_Expected_Counts.png
│ │ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Raw_Counts.png
│ │ │ ├── mqc_star_alignment_plot_1_pc.png
│ │ │ └── mqc_star_alignment_plot_1.png
│ │ └── svg
│ │ ├── mqc_cutadapt_filtered_reads_plot_1_pc.svg
│ │ ├── mqc_cutadapt_filtered_reads_plot_1.svg
│ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.svg
│ │ ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.svg
│ │ ├── mqc_dupradar-plot_1.svg
│ │ ├── mqc_fastqc_adapter_content_plot_1.svg
│ │ ├── mqc_fastqc_overrepresented_sequences_plot_1.svg
│ │ ├── mqc_fastqc_overrepresented_sequences_plot-2_1.svg
│ │ ├── mqc_fastqc_per_base_n_content_plot_1.svg
│ │ ├── mqc_fastqc_per_base_n_content_plot-2_1.svg
│ │ ├── mqc_fastqc_per_base_sequence_quality_plot_1.svg
│ │ ├── mqc_fastqc_per_base_sequence_quality_plot-2_1.svg
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Counts.svg
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot-2_Percentages.svg
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.svg
│ │ ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.svg
│ │ ├── mqc_fastqc_per_sequence_quality_scores_plot_1.svg
│ │ ├── mqc_fastqc_per_sequence_quality_scores_plot-2_1.svg
│ │ ├── mqc_fastqc_sequence_counts_plot_1_pc.svg
│ │ ├── mqc_fastqc_sequence_counts_plot_1.svg
│ │ ├── mqc_fastqc_sequence_counts_plot-2_1_pc.svg
│ │ ├── mqc_fastqc_sequence_counts_plot-2_1.svg
│ │ ├── mqc_fastqc_sequence_duplication_levels_plot_1.svg
│ │ ├── mqc_fastqc_sequence_duplication_levels_plot-2_1.svg
│ │ ├── mqc_fastqc_sequence_length_distribution_plot_1.svg
│ │ ├── mqc_featurecounts_biotype_plot_1_pc.svg
│ │ ├── mqc_featurecounts_biotype_plot_1.svg
│ │ ├── mqc_picard_deduplication_1_pc.svg
│ │ ├── mqc_picard_deduplication_1.svg
│ │ ├── mqc_preseq_plot_1.svg
│ │ ├── mqc_qualimap_gene_coverage_profile_Counts.svg
│ │ ├── mqc_qualimap_gene_coverage_profile_Normalised.svg
│ │ ├── mqc_qualimap_genomic_origin_1_pc.svg
│ │ ├── mqc_qualimap_genomic_origin_1.svg
│ │ ├── mqc_rseqc_infer_experiment_plot_1.svg
│ │ ├── mqc_rseqc_inner_distance_plot_Counts.svg
│ │ ├── mqc_rseqc_inner_distance_plot_Percentages.svg
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events_pc.svg
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Events.svg
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions_pc.svg
│ │ ├── mqc_rseqc_junction_annotation_junctions_plot_Junctions.svg
│ │ ├── mqc_rseqc_junction_saturation_plot_All_Junctions.svg
│ │ ├── mqc_rseqc_junction_saturation_plot_Known_Junctions.svg
│ │ ├── mqc_rseqc_junction_saturation_plot_Novel_Junctions.svg
│ │ ├── mqc_rseqc_read_distribution_plot_1_pc.svg
│ │ ├── mqc_rseqc_read_distribution_plot_1.svg
│ │ ├── mqc_rseqc_read_dups_plot_1.svg
│ │ ├── mqc_salmon_plot_1.svg
│ │ ├── mqc_samtools_alignment_plot_1_pc.svg
│ │ ├── mqc_samtools_alignment_plot_1.svg
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Normalised_Counts.svg
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Observed_over_Expected_Counts.svg
│ │ ├── mqc_samtools-idxstats-mapped-reads-plot_Raw_Counts.svg
│ │ ├── mqc_star_alignment_plot_1_pc.svg
│ │ └── mqc_star_alignment_plot_1.svg
│ └── multiqc_report.html
├── pipeline_info
│ ├── execution_report_2022-11-28_08-50-33.html
│ ├── execution_report_2022-11-28_08-55-06.html
│ ├── execution_report_2022-11-28_09-00-31.html
│ ├── execution_report_2022-11-28_09-11-17.html
│ ├── execution_report_2022-11-28_09-15-59.html
│ ├── execution_report_2022-11-29_12-49-14.html
│ ├── execution_report_2022-11-29_15-36-21.html
│ ├── execution_report_2022-11-29_18-00-43.html
│ ├── execution_report_2022-11-30_13-44-36.html
│ ├── execution_timeline_2022-11-28_08-50-33.html
│ ├── execution_timeline_2022-11-28_08-55-06.html
│ ├── execution_timeline_2022-11-28_09-00-31.html
│ ├── execution_timeline_2022-11-28_09-11-17.html
│ ├── execution_timeline_2022-11-28_09-15-59.html
│ ├── execution_timeline_2022-11-29_12-49-14.html
│ ├── execution_timeline_2022-11-29_15-36-21.html
│ ├── execution_timeline_2022-11-29_18-00-43.html
│ ├── execution_timeline_2022-11-30_13-44-36.html
│ ├── execution_trace_2022-11-28_08-55-06.txt
│ ├── execution_trace_2022-11-28_09-00-31.txt
│ ├── execution_trace_2022-11-28_09-11-17.txt
│ ├── execution_trace_2022-11-28_09-15-59.txt
│ ├── execution_trace_2022-11-29_12-49-14.txt
│ ├── execution_trace_2022-11-29_15-36-21.txt
│ ├── execution_trace_2022-11-29_18-00-43.txt
│ ├── execution_trace_2022-11-30_13-44-36.txt
│ ├── pipeline_dag_2022-11-28_08-50-33.html
│ ├── pipeline_dag_2022-11-28_08-55-06.html
│ ├── pipeline_dag_2022-11-28_09-00-31.html
│ ├── pipeline_dag_2022-11-28_09-11-17.html
│ ├── pipeline_dag_2022-11-28_09-15-59.html
│ ├── pipeline_dag_2022-11-29_12-49-14.html
│ ├── pipeline_dag_2022-11-29_15-36-21.html
│ ├── pipeline_dag_2022-11-29_18-00-43.html
│ ├── pipeline_dag_2022-11-30_13-44-36.html
│ ├── samplesheet.valid.csv
│ └── software_versions.yml
├── salmon
│ ├── salmon.merged.gene_counts_length_scaled.rds
│ ├── salmon.merged.gene_counts_length_scaled.tsv
│ ├── salmon.merged.gene_counts.rds
│ ├── salmon.merged.gene_counts_scaled.rds
│ ├── salmon.merged.gene_counts_scaled.tsv
│ ├── salmon.merged.gene_counts.tsv
│ ├── salmon.merged.gene_tpm.tsv
│ ├── salmon.merged.transcript_counts.rds
│ ├── salmon.merged.transcript_counts.tsv
│ ├── salmon.merged.transcript_tpm.tsv
│ ├── salmon_tx2gene.tsv
│ └── WT_REP1
│ ├── aux_info
│ │ ├── ambig_info.tsv
│ │ ├── expected_bias.gz
│ │ ├── fld.gz
│ │ ├── meta_info.json
│ │ ├── observed_bias_3p.gz
│ │ └── observed_bias.gz
│ ├── cmd_info.json
│ ├── lib_format_counts.json
│ ├── libParams
│ │ └── flenDist.txt
│ ├── logs
│ │ └── salmon_quant.log
│ ├── quant.genes.sf
│ └── quant.sf
├── star_salmon
│ ├── bigwig
│ │ ├── WT_REP1.forward.bigWig
│ │ └── WT_REP1.reverse.bigWig
│ ├── deseq2_qc
│ │ └── deseq2.dds.RData
│ ├── dupradar
│ │ ├── box_plot
│ │ │ └── WT_REP1_duprateExpBoxplot.pdf
│ │ ├── gene_data
│ │ │ └── WT_REP1_dupMatrix.txt
│ │ ├── histogram
│ │ │ └── WT_REP1_expressionHist.pdf
│ │ ├── intercepts_slope
│ │ │ └── WT_REP1_intercept_slope.txt
│ │ └── scatter_plot
│ │ └── WT_REP1_duprateExpDens.pdf
│ ├── featurecounts
│ │ ├── WT_REP1.biotype_counts_mqc.tsv
│ │ ├── WT_REP1.biotype_counts_rrna_mqc.tsv
│ │ ├── WT_REP1.featureCounts.txt
│ │ └── WT_REP1.featureCounts.txt.summary
│ ├── log
│ │ ├── WT_REP1.Log.final.out
│ │ ├── WT_REP1.Log.out
│ │ ├── WT_REP1.Log.progress.out
│ │ └── WT_REP1.SJ.out.tab
│ ├── picard_metrics
│ │ └── WT_REP1.markdup.sorted.MarkDuplicates.metrics.txt
│ ├── preseq
│ │ ├── log
│ │ │ └── WT_REP1.command.log
│ │ └── WT_REP1.lc_extrap.txt
│ ├── qualimap
│ │ └── WT_REP1
│ │ ├── css
│ │ │ ├── agogo.css
│ │ │ ├── ajax-loader.gif
│ │ │ ├── basic.css
│ │ │ ├── bgfooter.png
│ │ │ ├── bgtop.png
│ │ │ ├── comment-bright.png
│ │ │ ├── comment-close.png
│ │ │ ├── comment.png
│ │ │ ├── doctools.js
│ │ │ ├── down.png
│ │ │ ├── down-pressed.png
│ │ │ ├── file.png
│ │ │ ├── jquery.js
│ │ │ ├── minus.png
│ │ │ ├── plus.png
│ │ │ ├── pygments.css
│ │ │ ├── qualimap_logo_small.png
│ │ │ ├── report.css
│ │ │ ├── searchtools.js
│ │ │ ├── underscore.js
│ │ │ ├── up.png
│ │ │ ├── up-pressed.png
│ │ │ └── websupport.js
│ │ ├── images_qualimapReport
│ │ │ ├── Coverage\ Profile\ Along\ Genes\ (High).png
│ │ │ ├── Coverage\ Profile\ Along\ Genes\ (Low).png
│ │ │ ├── Coverage\ Profile\ Along\ Genes\ (Total).png
│ │ │ ├── Junction\ Analysis.png
│ │ │ ├── Reads\ Genomic\ Origin.png
│ │ │ └── Transcript\ coverage\ histogram.png
│ │ ├── qualimapReport.html
│ │ ├── raw_data_qualimapReport
│ │ │ ├── coverage_profile_along_genes_(high).txt
│ │ │ ├── coverage_profile_along_genes_(low).txt
│ │ │ └── coverage_profile_along_genes_(total).txt
│ │ └── rnaseq_qc_results.txt
│ ├── rseqc
│ │ ├── bam_stat
│ │ │ └── WT_REP1.bam_stat.txt
│ │ ├── infer_experiment
│ │ │ └── WT_REP1.infer_experiment.txt
│ │ ├── inner_distance
│ │ │ ├── pdf
│ │ │ │ └── WT_REP1.inner_distance_plot.pdf
│ │ │ ├── rscript
│ │ │ │ └── WT_REP1.inner_distance_plot.r
│ │ │ └── txt
│ │ │ ├── WT_REP1.inner_distance_freq.txt
│ │ │ ├── WT_REP1.inner_distance_mean.txt
│ │ │ └── WT_REP1.inner_distance.txt
│ │ ├── junction_annotation
│ │ │ ├── bed
│ │ │ │ ├── WT_REP1.junction.bed
│ │ │ │ └── WT_REP1.junction.Interact.bed
│ │ │ ├── log
│ │ │ │ └── WT_REP1.junction_annotation.log
│ │ │ ├── pdf
│ │ │ │ ├── WT_REP1.splice_events.pdf
│ │ │ │ └── WT_REP1.splice_junction.pdf
│ │ │ ├── rscript
│ │ │ │ └── WT_REP1.junction_plot.r
│ │ │ └── xls
│ │ │ └── WT_REP1.junction.xls
│ │ ├── junction_saturation
│ │ │ ├── pdf
│ │ │ │ └── WT_REP1.junctionSaturation_plot.pdf
│ │ │ └── rscript
│ │ │ └── WT_REP1.junctionSaturation_plot.r
│ │ ├── read_distribution
│ │ │ └── WT_REP1.read_distribution.txt
│ │ └── read_duplication
│ │ ├── pdf
│ │ │ └── WT_REP1.DupRate_plot.pdf
│ │ ├── rscript
│ │ │ └── WT_REP1.DupRate_plot.r
│ │ └── xls
│ │ ├── WT_REP1.pos.DupRate.xls
│ │ └── WT_REP1.seq.DupRate.xls
│ ├── salmon.merged.gene_counts_length_scaled.rds
│ ├── salmon.merged.gene_counts_length_scaled.tsv
│ ├── salmon.merged.gene_counts.rds
│ ├── salmon.merged.gene_counts_scaled.rds
│ ├── salmon.merged.gene_counts_scaled.tsv
│ ├── salmon.merged.gene_counts.tsv
│ ├── salmon.merged.gene_tpm.tsv
│ ├── salmon.merged.transcript_counts.rds
│ ├── salmon.merged.transcript_counts.tsv
│ ├── salmon.merged.transcript_tpm.tsv
│ ├── salmon_tx2gene.tsv
│ ├── samtools_stats
│ │ ├── WT_REP1.markdup.sorted.bam.flagstat
│ │ ├── WT_REP1.markdup.sorted.bam.idxstats
│ │ ├── WT_REP1.markdup.sorted.bam.stats
│ │ ├── WT_REP1.sorted.bam.flagstat
│ │ ├── WT_REP1.sorted.bam.idxstats
│ │ └── WT_REP1.sorted.bam.stats
│ ├── stringtie
│ │ ├── WT_REP1.ballgown
│ │ │ ├── e2t.ctab
│ │ │ ├── e_data.ctab
│ │ │ ├── i2t.ctab
│ │ │ ├── i_data.ctab
│ │ │ └── t_data.ctab
│ │ ├── WT_REP1.coverage.gtf
│ │ ├── WT_REP1.gene.abundance.txt
│ │ └── WT_REP1.transcripts.gtf
│ ├── WT_REP1
│ │ ├── aux_info
│ │ │ ├── ambig_info.tsv
│ │ │ ├── expected_bias.gz
│ │ │ ├── fld.gz
│ │ │ ├── meta_info.json
│ │ │ ├── observed_bias_3p.gz
│ │ │ └── observed_bias.gz
│ │ ├── cmd_info.json
│ │ ├── libParams
│ │ │ └── flenDist.txt
│ │ ├── logs
│ │ │ └── salmon_quant.log
│ │ ├── quant.genes.sf
│ │ └── quant.sf
│ ├── WT_REP1.markdup.sorted.bam
│ └── WT_REP1.markdup.sorted.bam.bai
└── trimgalore
├── fastqc
│ ├── WT_REP1_1_val_1_fastqc.html
│ ├── WT_REP1_1_val_1_fastqc.zip
│ ├── WT_REP1_2_val_2_fastqc.html
│ └── WT_REP1_2_val_2_fastqc.zip
├── WT_REP1_1.fastq.gz_trimming_report.txt
└── WT_REP1_2.fastq.gz_trimming_report.txt
71 directories, 438 files
[15]:
# cleanup
rm -rf .nextflow
#sleep 1m
#rm -rf .nextflow*