1. The Dirichlet-Multinomial in PyMC3

   Modeling Overdispersion in Compositional Count Data

   Having just spent a few too many hours working on the Dirichlet-multinomial distribution in PyMC3, I thought I'd convert the demo notebook I also contributed into a blog post.

   This example (exported and minimally edited from a Jupyter Notebook) demonstrates the use of a Dirichlet mixture of multinomials (a.k.a Dirichlet-multinomial or DM) to model categorical count data. Models like this one are important in a variety of areas, including natural language processing, ecology, bioinformatics, and more.

   The Dirichlet-multinomial can be understood as draws from a Multinomial distribution where each sample has a slightly different probability vector, which is itself drawn from a common Dirichlet distribution. This contrasts with the Multinomial distribution, which assumes that all observations arise from a single fixed probability vector. This enables the Dirichlet-multinomial to accommodate more variable (a.k.a, over-dispersed) count data than the Multinomial.

   Other examples of over-dispersed count distributions are the …

   ...More

2. Things I'm Glad I Learned

   Skills, concepts, techniques, and models

   WARNING: This post was written with haste and therefore contains all kinds of typos, spelling errors, grammatical issues, and delusions of grandeur, wisdom, and writing ability.

   This post is intended as a living document—a gratitude journal of sorts—of some things that I'm glad I learned. I expect many of the items on this list will be relevant to computation biology, but that may change in the future.

   The big idea is that for every item on this list I am (A) glad that someone introduced me to it, and (B) think more people should know about it. This post is my chance to "pay it backwards", as it were; maybe someone else will be grateful for something they find for the first time on this list.

   It may also double as an inspiration list for future posts.

   My goal is to write a small blurb for each item …

   ...More

3. Tutorial: Reproducible data analysis pipelines using Snakemake

   In many areas of natural and social science, as well as engineering, data analysis involves a series of transformations: filtering, aggregating, comparing to theoretical models, culminating in the visualization and communication of results. This process is rarely static, however, and components of the analysis pipeline are frequently subject to replacement and refinement, resulting in challenges for reproducing computational results. Describing data analysis as a directed network of transformations has proven useful for translating between human intuition and computer automation. In the past I've evangelized extensively for GNU Make, which takes advantage of this graph representation to enable incremental builds and parallelization.

   Snakemake is a next-generation tool based on this concept and designed specifically for bioinformatics and other complex, computationally challenging analyses. I've started using Snakemake for my own data analysis projects, and I've found it to be a consistent improvement, enabling more complex pipelines with fewer of the "hacks" that …

   ...More

4. Teaching Python by the (Note)Book

   tl;dr: I tried out a modified Python lesson and I think it was successful at balancing learner motivation with teaching foundational (and sometimes boring) concepts.

   In many ways, teaching Python to scientists is easier than just about every other audience. The learning objective is clear: write code to make my science more accurate, more efficient, and more impactful. The motivation is apparent: data is increasingly plentiful and increasingly complex. The learners are both engaged and prepared to put in the effort required to develop new skills.

   But, despite all of the advantages, teaching anybody to program is hard.

   In my experience, one of the most challenging trade-offs for lesson planners is between motivating the material and teaching a mental model for code execution. For example, scientists are easily motivated by simple data munging and plotting using pandas and matplotlib; these are features of the Python ecosystem that can convince …

   ...More

5. Take five minutes to simplify your life with Make

   WARNING: Because of the Markdown rendering of this blog, tab characters have been replaced with 4 spaces in code blocks. For this reason, the makefile code will not work when copied directly from the post. Instead, you must first replace all 4-space indents with a tab character.

   I use GNU Make to automate my data processing pipelines. I've written a tutorial ¹ for novices on the basics of using Make for reproducible analysis and I think that everyone who writes more than one script, or runs more than one shell command to process their data can benefit from automating that process. I'm not alone.

   However, the investment required to learn Make and to convert an entire project can seem daunting to many time-strapped researchers. Even if you aren't living the dream—rebuilding a paper from raw data with a single invocation of make paper—I still think you can benefit …

   ...More

6. Software carpentry instructor training

   A survival analysis in python

   Edit (2016-05-31): Added a hypothesis for why my results differ somewhat from Erin Becker's. Briefly: I removed individuals who taught before they were officially certified.

   A couple weeks ago, Greg Wilson asked the Software Carpentry community for feedback on a collection of data about the organization's instructors, when they were certified, and when they taught. Having dabbled in survival analysis, I was excited to explore the data within that context.

   Survival analysis is focused on time-to-event data, for example time from birth until death, but also time to failure of engineered systems, or in this case, time from instructor certification to first teaching a workshop. The language is somewhat morbid, but helps with talking precisely about models that can easily be applied to a variety of data, only sometimes involving death or failure. The power of modern survival analysis is the ability to include results from subjects who have not …

   ...More

7. First time teaching Python to novices

   This July I co-instructed with Jennifer Shelton a Software Carpentry workshop at Stanford University, targeted to researchers with genomic or evolutionary datasets. Jennifer taught the shell (Bash) and version control with Git, while I taught the general programming language Python. I've been aware of the organization, which teaches software development and computational methods to scientists, since attending a workshop in 2012. Since then I've served as a helper at one workshop (troubleshooting individual learner's problems and helping catch them up with the rest of the class), and gone through the "accelerated", two day, instructor training at Michigan State University. After the Stanford workshop, I took part in new-instructor debriefing on August 4th, during which I mentioned that I had to greatly pare down the community-written lesson plan, python-novice-inflammation, to fit into the two half-day session we allotted it.

   Karin and Tiffany, who were running the debriefing, asked me to send …

   ...More

8. Compiling SciPy on RHEL6

   Within the past two years I've discovered something interesting about myself (...actually really, really boring about myself): I can be happily entertained for hours on end setting up my computational environment just right. I find that it gives me a similar type of satisfaction to cataloguing my music collection. I guess you could call it a hobby.

   Usually this entails installing the usual suspects (NumPy, Pandas, IPython, matplotlib, etc.) in a python virtual environment. When I'm particularly into it (which is always), I'll also compile the python distribution itself. I've had several opportunities to indulge this pasttime, most recently in setting up my research pipeline on the Flux high-performance compute cluster at The University of Michigan.

   Installing NumPy is usually no trouble at all, but for some reason (if you know, please tell me), SciPy has always given me a "BlasNotFoundError" when installing on the Red Hat Enterprise Linux distros …

   ...More

9. PyMake I: Another GNU Make clone

   (Edit 1): This is the first of two posts about my program PyMake. I'll post the link to Part II here when I've written it. While I still agree with some of the many of the views expressed in this piece, I have changed my thinking on Makefiles.

   (Edit 2): I'll post a new post about the topic when I take the time to write it. I've written a tutorial on using _Make for reproducible data analysis_.

   I am an aspiring but unskilled (not yet skilled?) computer geek. You can observe this for yourself by watching me fumble my way through vim configuration, multi-threading/processing in Python, and git merges.

   Rarely do I actually feel like my products are worth sharing with the wider world. The only reason I have a GitHub account is personal convenience and absolute confidence that no one else will ever look at it besides me …

   ...More

Page 1 / 1