As a researcher, it's important to be able to keep up to date with the work that's happening in your field. However, with countless new articles every week, this can be a daunting task. After our work on ICGenealogy, we teamed up with the CNCB again to help them build NTG, an internal tool to allow researchers to explore a massive corpus of articles across many different publications. To do this, we first had to build a system that would be able to represent the many different kinds of underlying relationships between papers.
The biggest challenge we faced was figuring out ways to allow the user to browse through related articles. Articles can be related by author, topic, citations, keywords, or a number of other factors. We created a comprehensive set of tools to allow the CNCB to explore each and every one of these dimensions, while tailoring each tool's interface for optimal usability. With some extremely valuable help from the CNCB's existing work, we built a tool that visualises over 80,000 papers without breaking a sweat.
It's imperative to us that everything we build has impeccable performance. However, when you're working with tens of thousands of articles with hundreds of thousands of relationships between them, this can be a challenge. We developed novel methods to filter this extensive data both on the backend and on the frontend, including precalculating certain computations and filters. In the end, we managed to allow the user to filter through the entire database in a matter of milliseconds — there wasn't even a need for a loading indicator.
Scientific articles are often difficult enough to understand in and of themselves, but we can at least make browsing them intuitive. We strived to develop various interfaces to allow browsing of articles by various metric of relatedness, allowing researchers not only to consolidate their knowledge on a specific topic, but also to explore an ever-expanding corpus of papers on related topics. We think our use of graphical representations made this process much more enjoyable and effortless.
NTG continues to be in development as an internal tool for the CNCB, with whom we've had an amazing partnership. We look forward to doing more work with them in the future.