Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: template refresh #24

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 0 additions & 41 deletions .github/workflows/pythonpackage.yml

This file was deleted.

25 changes: 1 addition & 24 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,24 +1 @@
# PyCharm
.idea

# Python compiled & optimized files
*.pyc
*.pyo

# MyPy Cache directory
.mypy_cache

# for develop installs
*.egg-info

# local dirs
envs
environments
.conda
conda-env-builder

# snakemake
.snakemake

# VSCode dir
.vscode
.DS_Store
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
The MIT License

Copyright (c) 2022 Fulcrum Genomics LLC
Copyright (c) 2025 Fulcrum Genomics LLC

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
103 changes: 28 additions & 75 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,84 +1,37 @@
[![build](https://github.com/fulcrumgenomics/python-snakemake-skeleton/actions/workflows/pythonpackage.yml/badge.svg)](https://github.com/fulcrumgenomics/python-snakemake-skeleton/actions/workflows/pythonpackage.yml)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/fulcrumgenomics/fgbio/blob/main/LICENSE)
[![Language](https://img.shields.io/badge/python-3.6.10-brightgreen)](https://www.python.org/downloads/release/python-3610/)
# Snakemake Project Template

A skeleton repository for Snakemake pipepline(s) and a python command-line toolkit.
This repository provides a template for a Fulcrum Genomics [Snakemake](https://snakemake.readthedocs.io/en/stable/) project with optional supporting [Python toolkit](https://github.com/fulcrumgenomics/python-template).

## Why this repo?

This the starting point for [Fulcrum Genomics][fulcrum-genomics-link] projects that contain Snakemake pipelines
and a python toolkit.

This repo contains the following, in no particular order:

- a hello world snakefile in `src/snakemake/hello_world.smk`
- this uses the `onerror` directive to better display rule errors, in particular the last file
lines of the rule's log
- a python toolkit (`client-tools`) in `src/python/pyclient`
- uses `defopt` for arg parsing
- has custom logging in `core/logging.py`
- has utility methods to support the above `onerror` snakemake directive in `pipeline/snakemake_utils.py`
- has a unit test to ensure the above snakefile is runnable and generally executes the expected rules in `tests/test_hello_world.py`.
This also includes a utility method to support running and verifying snakemake in `tests/util.py`
- supports multiple sub-commands in `tools/__main__.py` with some nice logging when a tool fails
- a little hello world tool in `tools/hello_world.py`

## Modifying this repo for a new client

This repo is a skeleton for Snakemake pipelines and a Python toolkit.

- [ ] Modify `setup.py`
- [ ] update `conda-requirements-minimal.txt` with minimal requirements for the `client-tools` toolkit
- [ ] update `conda-requirements-test.txt` with minimal requirements for the `client-tools` unit testing
- [ ] update `pip-requirements.txt` with minimal requirements for the `client-tools` (prefer conda)
- [ ] update `src/python/pyclient` source code (search for terms: `PYCLIENT`, `pyclient`, `client-tools`

## Install client-tools

- [Install conda][conda-link]


- Create the `pyclient` conda environment
## Creating a Snakemake Project

Install [`cookiecutter`](https://cookiecutter.readthedocs.io/en/stable/) and run a command like:

```console
mamba create -n pyclient \
--override-channels -y \
-c bioconda -c conda-forge -c defaults \
--file conda-requirements-minimal.txt \
--file conda-requirements-test.txt
```

- Activate the `pyclient` conda environment

```bash
conda activate pyclient
❯ cookiecutter [email protected]:fulcrumgenomics/python-snakemake-template.git --output-dir ~/git/
[1/5] Repository name for your project (e.g. 'myworkflow'): myworkflow
[2/5] Describe your project with a period-terminated sentence (...):
[3/5] Your GitHub handle (e.g. @example):
[4/5] GitHub organization to host your project (fulcrumgenomics):
[5/5] License. If 'MIT (Open Source) is selected', a LICENSE file will be generated.
1 - Proprietary (no license)
2 - MIT (Open Source)
Choose from [1/2] (1):
```

- Install all non-conda dependencies via pip
You will have a Git-initialized Snakemake project at the following location:

```bash
pip install -r pip-requirements.txt
```

- Install `pyclient` (in developer mode)

```bash
python setup.py develop
```

- Validate the install via the help message

```bash
client-tools -h
```

- Validate the snakemake install

```bash
snakemake --snakefile src/snakemake/hello_world.smk -j 1
```console
❯ > tree -a myworkflow
myworkflow
├── .git # contents omitted for brevity
├── .github
│   └── CODEOWNERS
├── .gitignore
├── README.md
├── config
│   ├── config.yml
│   └── config_schema.yml
├── environment.yml
└── workflow
└── myworkflow.smk
```

[fulcrum-genomics-link]: https://www.fulcrumgenomics.com
[conda-link]: https://docs.conda.io/projects/conda/en/latest/user-guide/install/

6 changes: 0 additions & 6 deletions ci/flake8.cfg

This file was deleted.

6 changes: 0 additions & 6 deletions ci/mypy.ini

This file was deleted.

75 changes: 0 additions & 75 deletions ci/precommit.sh

This file was deleted.

13 changes: 0 additions & 13 deletions conda-requirements-minimal.txt

This file was deleted.

5 changes: 0 additions & 5 deletions conda-requirements-test.txt

This file was deleted.

15 changes: 15 additions & 0 deletions cookiecutter.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"project_slug": null,
"project_short_description": "...",
"your_github_handle": null,
"project_org": "fulcrumgenomics",
"license": ["Proprietary (no license)", "MIT (Open Source)"],
"__license": "{{ cookiecutter.license.split(' ')|first }}",
"__prompts__": {
"project_slug": "Repository name for your project (e.g. 'myworkflow')",
"project_short_description": "Describe your project with a period-terminated sentence",
"your_github_handle": "Your GitHub handle (e.g. @example)",
"project_org": "GitHub organization to host your project",
"license": "License. If 'MIT (Open Source) is selected', a LICENSE file will be generated."
}
}
8 changes: 8 additions & 0 deletions hooks/post_gen_project.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import subprocess

# Projects need an initialized Git repository
subprocess.call(["rm", "-rf", ".git/"])
subprocess.call(["git", "init"])
subprocess.call(["git", "remote", "add", "origin", "[email protected]:{{cookiecutter.project_org}}/{{cookiecutter.project_slug}}.git"])
subprocess.call(["git", "add", "-A"])
subprocess.call(["git", "commit", "-m", "First commit"])
14 changes: 14 additions & 0 deletions hooks/pre_gen_project.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import re
import sys


GITHUB_HANDLE_REGEX = re.compile(r'^\@[a-z\d][_a-zA-Z\d]+$')
"""Regex for valid GitHub handle with the `@` symbol prepended."""

if not GITHUB_HANDLE_REGEX.match("{{cookiecutter.your_github_handle}}"):
print("ERROR: {{cookiecutter.your_github_handle}} is not a GitHub handle!")
print("ERROR: GitHub handles must start with an `@` character.")
print("ERROR: GitHub handles must not have an underscore after the `@` character.")
print("ERROR: GitHub handles must have more than one character after the `@`.")
print("ERROR: GitHub handles must only contain ASCII letters, numbers, and underscores.")
sys.exit(1)
2 changes: 0 additions & 2 deletions pip-requirements.txt

This file was deleted.

30 changes: 0 additions & 30 deletions setup.py

This file was deleted.

3 changes: 0 additions & 3 deletions src/python/pyclient/__init__.py

This file was deleted.

Loading