Doing a Systematic Review and Not Being Beaten by Piles of Paper

As with most PhDs based in Health Services Research, my project started with a systematic review. This seems to differ hugely from lab-based PhDs which (from my experience anyway) largely begin with traditional literature reviews. Not sure what the difference is between the two types of review? I’ll point you in the direction of this blog post from Students 4 Best Evidence. In short, systematic reviews can take an absolute age and they require a certain level of patience and persistence that I didn’t realise I had.

Last year HealthPsychTam posted two different posts talking about her experience of doing a systematic review. ‘A Confession…’ which was a brutally honest post about the feeling of wanting to drop out, and ‘Conducting a Systematic Review’ with lots of absolutely brilliant tips on getting through the process. I’d recommend you read both. In this post I want to add to Tamsyn’s experiences and give my own thoughts on the process so far.

What do I aim to achieve with this review?
My primary PhD supervisor has a Cochrane review that looks at methods to improve recruitment to randomised controlled trials, and mine is sort of the mirror of that review. It looks at methods to improve recruitment to randomise controlled trials that are evaluated using only non-randomised evaluations. We know there’s a lot of publications that cover this topic, but as yet there has been no systematic review including only data from non-randomised studies.

What stage am I at now?
Currently I’ve published the protocol for my systematic review (huge gold star to my supervisor for encouraging me to do this – it was a massive motivator), I’ve finished data extraction and we’re now tackling the task of data analysis and synthesis. In very simple terms, I have created a large pile of paper that I now need to shape into something useful.

Things I wish I’d known at the start that I know now

  • Search strategies can never ever weed out all the studies you don’t need
    I worked with an Information Specialist to create my search strategy – put bluntly, I am not an expert in search strategy development and the Information Specialist based in our unit is. She was brilliant to work with, and she made the whole process much easier and quicker than if I was going to figure out how to do this whole thing myself. Still, search strategies can never be perfect and you will always end up with a big pile of studies that won’t make it into your review. I began with over 9,500 abstracts, whittled that down to ~270 full texts to assess, and then ended up with 103 studies in the final review.
  • You will never finish a systematic review of this size in a year
    I still haven’t finished the review and I’m entering month 19th of working on it. That’s a really long slog to go through, most of which was spent reading stuff and meticulously tracking where every abstract, full text and included study was in the biggest spreadsheet I’ve ever made. Be realistic, it’s unlikely you’ll be done within a year unless you’ve got a really small amount of included studies (if this is the case well done you, I am very jealous).
  • A review cannot be done by one person – get people involved as soon as you can
    All of my abstract screening, full text assessments and data extraction were done in duplicate; once by me and once by whichever person I managed to sweet talk that week. It took a lot of time and effort to find people willing to help, and then explain tasks to them via telephone/Skype and a lot of Dropbox files. I couldn’t have done the review without them and I’m so grateful that they offered to help (I had no funds to offer them – they were just being top notch humans). I would thoroughly recommend getting other people involved in your review as early as you can; whether they can help with screening/data extraction or just give you a new perspective on how you’re going to analyse your data, it’s all helpful.

This systematic review has been a really brilliant learning process, but it’s been the longest slog I’ve had over the course of my PhD. Two of my desk drawers are now crammed with papers, some scribbled with ‘include’, others ‘exclude’ – the further down the pile the less clear and politically correct they become, my personal favourite being ‘this is crap, total crap, exclude on the basis it’s utter crap’. I’m on the way with it though! I’ve got the big cloud of screening and extraction out of the way, and I’m on to the fun stuff and seeing what the review itself shows! Hoorah!

If you’re thinking of doing a systematic review, please be realistic with your timescales – and make sure you have snacks along the way. It’s a long process but chocolate definitely helps.

5 thoughts on “Doing a Systematic Review and Not Being Beaten by Piles of Paper

Leave a Reply to HealthPsychTam Cancel reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.