I’ve been thinking recently about how best to organize my research data and experimental records. I’m not sure my needs are the same as everyone else’s, but I doubt they are so very different. Mostly I want to record GenBank searches, phylogenetic analyses, little bits of perl code. Although I support open data, I choose not to before publication. To be honest I don’t even really understand the need or desirability of open lab books. Maybe I’ve just come across more unscrupulous people?
Here is what I think I need
- I want it to be electronic (obviously) as its easier to incorporate my results and code that way. Also I have almost forgotten how to write and typing is much easier.
- I want the format to be open, not proprietary. It is equivalent to me if it has good batch export to a common format e.g. RTF. That way I’m not locked in to a particular program or system. I have swapped between several by batch exporting my records as RTF and importing them to another program.
- It should have both a chronological listing format and good search functions
- It should be easy to insert pictures and resize if necessary.
- It should be easy to work with formatted text, headings, tables and lists
- Links to other entries are useful, as are tags and a good folder hierarchy.
So I have been thinking about (a) Wikis, web and blog software (b) MS Word and Google docs (c) Specifically-designed electronic laboratory notebooks (d) Journaling software. Here are some thoughts and where I am now.
A. Wikis, Web and Blog software
There are a lot of different wikis out there, some simple, some very powerful. You can set up a free wiki on a public server. OpenWetWare is a good example. I don’t want to go that route as I want to keep my experiments private until I publish. Some positives of wikis are that they can be published locally on your hard drive, or on a lab server. TiddlyWiki is a favourite of mine. Wikis give the possibility of sharing. If I had a student using this approach I could log in and read her lab book whenever I wanted. I’ve tried a few wikis and my feeling is that they are a little more difficult to use than necessary. I am used to the ease and power of word processors and well made software. The ones I have tried don’t have that sort of polish. I don’t really know enough about wikis. Is it possible to batch export entries? If I kept a wiki lab book, then found something better would it be easy to select all, export as (say) RTF and then import into my new favourite software? I suspect it may not be very straight forward. I don’t see wikis as the way to go, it doesn’t feel right for me.
Blogs are great, organized chronologically (just like a paper lab book), with the ability to insert pictures, tables and formatting. Again I worry about the sophistication of blogs. I suppose there are different varieties. But I have many of the same questions as of wikis. I am not yet convinced the software is good enough. Blogger is a bit horrible and buggy. Inserting pictures can be problematic. It doesn’t seem to be up to the standard I would expect of my main information storage system.
There are some excellent open source content management systems available. Joomla! for example is a free and powerful content management system for creating web sites. I think setting up templates for different experiments could be a useful way to go. But same complaints as above.
B. MS Word and Google Docs
The new version of MS Word (2008 on my Mac) has a notebook layout. This looks like it could be just the thing, but it isn’t. Its not well designed and I gave up on it within half an hour. Standard Word docs are a possibility too. But then each experiment would be in a separate document I guess, quite difficult to browse and organize them.
Google docs are a simple set of office applications that you use online. They are interesting as they are stored on Google’s servers. You can also mark a document as shared and select who has access to it. It might be a useful approach to showing your supervisor your work if you are a student.
C. Specifically-designed electronic laboratory notebooks
These are usually commercial, and concentrated on features like security, validation and audit trails of what changes were made when. There is a list of some software here and the wikipedia entry is useful. I don’t find these really meet my needs as I really don’t care about documenting my edits and validating my electronic lab book any more than I did my paper one. It might be different if you work in industry. I recently saw iPad though which is free and definitely worth a look. After testing it a bit though I don’t like the environment very much.
D. Journaling Software
These are applications for organizing databases of data in the form of notes, and other media. They are often very sophisticated and powerful. They tend to be operating system specific. I use OS X (most of the time) so the examples below relate to Macs, but Microsoft and Linux have similar stuff. I have used Journler until recently. This is a really excellent software package (US$35), but I have recently swapped to use MacJournal which costs the same. This is a similar program, not quite as well-designed as Journler but it does have the advantage of syncing with my .Mac account so I have the same data available at home and in the office. It turns out this is really important for me.
These journals are a database of entries. Entries can be formatted like a word processor document with styles and tables and inserted pictures. Tags and search facilities are good. Different types of media (e.g. PDFs, JPG) can be dragged in. Audio (or video) notes can be recorded and added simply. Chronological listing and folder structures are useful. Export of notes as html, PDF, RTF, text etc should ensure that the information is always accessible even if you stop using the software.
I am very impressed with information journals like Journler (see pic below) and MacJournal as electronic lab books. They fit my needs, are powerful and very polished pieces of software. I’ve realized that working in a nice software environment is more important to me than anything else. They don’t have the automatic publish to web strengths of blogs and wikis which I guess can be very useful to coordinate between lab members, or to allow a supervisor to routinely read a student’s book. They both do export as html though and hopefully this aspect will develop.
EvoPhylo 
Hi DaveThis is a great summary of different approaches. One thing that might actually be particularly useful to you which you don’t consider is a code repository. Google Code for instance might be a good way of handling your perl and database searches (particularly if you are doing your database calls programatically) and also has wiki features as wellas ‘issue tracking’ which might be a way or organising projects. And obviously you can upload the results of searches as files (or write something to do it automatically)You are right that both Wikis and Blogs are both primitive in their user interface and complicated to use to their full potential. This will change gradually but it has a way to go.You emphasise a wish for collaborative systems. I find blogs particularly good for this because I can track my groups work via RSS feeds. You might want to consider a mixed system. A repository for code and results files plus a blog or blogs for handling high level ‘what I am thinking about’ kind of stuff.Please do follow up on what you decide to do and how it works for you. Getting a feel for what works for different people and areas of science is a really important part of developing new and better systems.
LikeLike
Thanks Cameron, very interesting. I had never really thought about code repositories. Google code uses Subversion doesn’t it? I know almost nothing about this sort of approach and should do some work to find out.At the moment its journaling software for me, been using it for a few months now. But I agree blogs and wikis are going to greatly improve and are probably the way to go, at least as part of a mixed strategy. If you have this strategy running in your lab you should do a post on pros and cons- lots of people would like to hear other researchers’ experiences.I’ll post back if anything changes with how I see things.Best, Dave
LikeLike
Dave,I’ve recently given a talk of how I use blogs, wikis, mailing lists and Google docs to keep track of the research from my group. You can view here.For the actual lab book where the experiments are recorded I am a big fan of a simple wiki – for example see a list of experiment pages here.Because I’m doing Open Notebook Science I make the wiki public. But with Wikispaces you can make it private for $5/month.
LikeLike
Hi,Although you stated that you don’t want to keep a public/open online notebook, I must say that going over your list of features that the new OWW lab notebook fits into nearly every point (except for the private issue!)I’d recommend you sign up and give it a look:http://openwetware.org/wiki/Lab_Notebook
LikeLike
Dave, was going to say ‘my whole blog is about this experience’ but then realised I haven’t ever really written about the experience side of things, much more the technical side. I will put it on the list. We use our blog based notebook system plus I use my higher level blog at some level to track the ‘why’ stuff. But Jean-Claude’s use of Wiki and Blog is probably a more coherent example.Overall my attitude is to use whatever service/system works for what you want for some specific aspect of your record keeping. I would say use a code repository for code, use a blog (or a journal – they are basically blogs withouot the webservice side from a functional perspective) for journal-based activities, a wiki where it is appropriate, and simple documents where they perform better.The weakest point in all of this is how to record the connection between things. There aren’t any terribly good tools for that at the moment.
LikeLike
Cameron – to record connections between things of course you can create links from any blog or wiki page to any other blog or wiki page. Those are useful but they are generally uni-directional. We’ve used tagging on experiment pages. For organic chemistry using InChIKeys works pretty well. Another strategy is to summarize experiments on public GoogleDoc spreadsheets. These can be exported to Excel to do more complicated searching and filtering.The reason we’re doing all of these is that there is not a single store and retrieve strategy that works for all purposes. Of course this adds a bit of time to the process of recording experiments but it is necessary to have a truly functional system.
LikeLike
Thanks for the comments everyone. Yes I have used OWW and generally like it, but not quite enough. I don’t want this to turn into a detailed critique of specific softwares. I definitely want to see OWW and others carry on! I have 2 separate issues. Firstly I want a good solution now to my personal research notebook needs. Wikis, blogs and the like just don’t cut it. Too basic, not easy enough to use, not a powerful and sophisticated work environment. Journaling software beats it hands down. I don’t want to be too tough but for me its like the difference between a text editor and Microsoft Word. The other issue is “the future”, technological advances and development of collaborative environments. Here things are not so clear. Wikis and blogs will undoubtedly get better, more user friendly, more powerful, better designed GUI. Then they should have the edge.Journaling software can make automatic wiki-like links between entries and of course html links to outside web resources, but generally have not tackled the problems of web presence nor multi-user environments. Wikis are almost the opposite having good solutions here, but made much less progress with the user environment and work-orientated GUI design. I don’t think either will “win”, both are in many ways quite similar, one will just subsume the other taking on its strengths.A very interesting talk Jean-Claude. I wasn’t sure I really got the big picture until the very last slide and your (I’m paraphrasing) comments about computer to computer information transfer rather than computer to scientist.The idea of open notebook science is one I find very interesting. At the moment I am heading towards an intermediate position of non-live open science. So when the manuscript is accepted publish all the lab notebooks pertaining to that work. I’m going to be posting again on this topic for sure.
LikeLike
Dave,Publishing your parts of your notebook after your papers come out is a reasonable strategy in your situation. I look forward to your reports on what technological tools you find work for you.
LikeLike
Hi Dave,Journaling software is good for personal notes management, but not great in rerms of sharing information with others or making it sufficiently clear to others (i.e., long-term preservation). I agree with you that wikis and blogs “don’t cut it” since, although they allow easy sharing, they are not great for personal notes management or for making information clear. Thanks for mentioning iPad since it was meant to address exactly that issue – to be both sufficiently nice for pesonal notes management AND allow greater clarity and sharing of info with others at the same time. I understand that the environment is not as “perfect” as in Journaling software you mentioned, but still, I am very curious to know what exactly you didn’t like about it. Finally, isn’t minor dislike worth the ability to share the info with others?Best regards,Alexander PolonskyiPad product manager
LikeLike
Hi Alexander- I have realized that my blog is a bit different from most people talking about ELN stuff in that I am just interested in the personal experience rather than the (probably more important) long-term goals of correct data archiving and accessibility. I don’t have to share my experimental data with anyone at the moment. I was looking for a way to make recording and archiving my experiments a better experience. For me then user interface and data management tools are all important. Journaling software is most powerful and has the best user interface by far. So thats what I’ve chosen for now.I would slightly disagree with you about long-term preservation of data from journaling software. It is a topic I see raised quite frequently but I have a different view. Journler (for example) stores all its data as RTF, which is non-proprietary, transparent and widely supported with numerous data-converters available were it to start to go out of fashion. Even if it weren’t I can select all my records and choose export in numerous formats (xhtml, rich text, richtext plus pictures, PDF, Word doc, plain text, web archive). I think this is pretty future proof! By contrast I still don’t really know how to convert wiki or blog entries to other formats in a simple way. I have started lab books in 2 wikis and never managed to satisfactorily recover the small number of entries when I moved on.When I am more comfortable with implementing ELN strategies I am going to start asking my students and postdocs to follow this route too. Then sharing and accessibility (by me) will become an issue and I will re-evaluate my options.I like many of the things in iPad. It seems a very well thought out application and I am reluctant to put minor interface quibbles here in case people see it as a criticism (email me if you want). I would encourage everyone to check it out. But since I spend most of my working day at the computer making experimental notes into my Lab Book its interface is very important indeed. I wasn’t as happy with my experience in iPad as other software and since lab data sharing is an issue for the future, not now, that was how I came to move on.What is clear to me is how fast-moving and dynamic the area of ELN software is. I like this, and am not set in my choice. It is right for me now, 9 months from now who knows.
LikeLike
Hi Dave,Just to clarify, by long-term preservation I refered to the aspect of clarity of information, not so much the format support. Even if you can open someone else’s notes file, chances are you will be quite lost in it. iPad helps to improve this.If you have a chance, I’d appreciate your feedback about iPad interface, either on iPad forum (http://ipadeln.com/forum) or by email (info@ipadeln.com). Finally, although you “don’t have to share your experimental data with anyone”, does it mean that you shouldn’t? :). (this is not meant to moralize or coerce you into anything, but it is my honest opinion :)).Best regards,Alexander
LikeLike
Evernote is a good alternative. It timestamps everything, allows you to drag and drop images (although not as nice as powerpoint for image manipulation), it’s secure, blazing fast search, text recognition, and more.
LikeLike
Thanks for that. Yes I have been checking out Evernote too. Actually I will post again in a couple of months about e lab books (I have realised that I can’t properly evaluate without a few months usage), my views have changed a bit since I wrote this post. My concern with Evernote is how to get info out of it. Their xml is weird, I don’t see how to import that into another application. The export ability of Windows and Mac is different and I suspect that things are changing fast on planet Evernote and improvements seem to be arriving fast at the moment. I am not completely convinced about Evernote’s commitment to open data formats yet. I’m keen to follow their progress.
LikeLike
I posted mine! I'm so excited about this. Thanks for hosting it! Add
LikeLike