RESEARCH FINDINGS
Tool development in the human rights documentation space
Below are some key findings from research conducted by The Engine Room on tool development in the human rights documentation space.
For this research, The Engine Room conducted eight interviews with tool developers, and carried out independent research into relevant tools. The full findings have been edited for this site by The Engine Room.
Key Findings
Civil society documenters of human rights violations tend to operate in high risk contexts, and with limited resources. This, combined with the current pace of change in the technological environment more generally, presents certain challenges for those building tools in the space.
In our research, we identified two notable areas in which developers face challenges: sustaining a tool over the long term, and finding the right balance of features.
We also found a current tendency for developers in the space to build smaller, more focused tools, envisaged as a part of a bigger tool ecosystem. Some developers have also collaborated to make their tools interoperable.
Below are some of our key findings.
01.
Resources required for sustaining a tool over the long term tend to be difficult to fund, and underestimated
The often highly resource-constrained contexts in which human rights documenters tend to work places certain limitations on how developers can fund their tools.
Most developers we spoke to reported facing funding challenges, especially when it comes to longer-term sustainability and support. Some also said they underestimated just how many resources would be needed to sustain their tools over the longer term.
What’s involved in sustaining a tool?
Sustaining a tool successfully over the long term means making sure the tool continues to function well and that it keeps up with changing user needs. In terms of what that involves, the following came up in the research:
- Responding to bugs or other issues quickly.
- Responding to changes in tool dependencies and programming languages – particularly when it comes to security components and dependencies.
- Conducting general backend support, including server maintenance and app security.
- Doing regular audits or penetration testing to find security vulnerabilities.
- Providing user support, including responding to queries and developing requested features.
- Conducting user testing, training, workshops and outreach.
- Adapting to changing user norms and expectations around how tools should look, feel, and function
How resources impact approaches to security audits and penetration testing
Though important, robust security audits and penetration testing tend to be difficult for developers to fund, and software-based solutions can have unsatisfactory results.
- Tool developers with more robust resources said they were able to do audits and/or testing multiple times a year, trying to alternate testing companies, and re-testing after changes are made.
- One tool developer we spoke to said that due to resource limitations most audits of their tool had been done not by themselves but by organisations who wanted to use the tool in their documentation work, using standardised software that produced superficial and sometimes “haphazard” results.
Developers of tools that have been around longer have faced particular challenges in bridging the gap between the environment in which the tool was first built and the current environment – both in terms of user expectations and norms and technologically (for example, in the development languages used).
Talking about how much user expectations have changed in recent years, developers reported that tool users are now accustomed to more intuitive functionalities and web-based interfaces, easy cross-device access, and customer support. Developers of data management tools also reported being asked for increased visualisation and analysis capabilities.
When to sunset a tool
In the last few years, a number of more established tools in the human rights documentation space have been retired.
In talking to tool developers who had made the difficult decision to sunset a tool – i.e. to stop maintaining or supporting it – the decision was often made when the tool started to become too resource-heavy for the developer to properly sustain, and when the tool had at the same time become less relevant to current user expectations: at a certain point, it made more sense to retire the tool than to continue.
In some cases, resources were put into a new tool, or redirected into one of the developer’s other tools.
1.1 How developers fund their tools’ sustainability
In talking to developers of human rights documentation tools, we came across a mix of funding models and strategies being used to try and ensure tool sustainability. Some developers rely entirely on grant funding, some rely only on fees-for-services, and some use a mix of these.
Grant funding
- Many tool developers rely on at least some grant funding, but most reported difficulty getting their work funded beyond the initial tool-building stage.
- Specially-funded servers can allow tool developers to offer hosted versions of their tool free of charge: KoBoToolbox, for example, offers free and unlimited hosting to humanitarian organisations on a designated server provided by the UN.
Fees-for-services
- Services that generate revenue include things like setup and hosting, customising tools to fit specific needs and workflows, and providing trainings and ongoing support.
- Providing services for higher-resourced organisations can also make it possible for a tool developer to offer services at a nominal charge for lower-resourced organisations.
- Challenges: Some noted that being more demand-driven meant that they might not have as much capacity to work on features that they see a need for, but they are nonetheless able to feed useful features developed for customised versions back into the core tool.
Membership-based funding
- This model was floated by one tool developer as a future plan, where membership funds would be used to finance maintenance fees, and anything left over would be used for new features, as decided on by the members themselves.
1.2 Open source tool-building as a sustainability strategy
One tool developer mentioned both using and building open source technology as a sustainability strategy for their tool:
“It’s important to build your tool using trusted, well-maintained, widely-used software so that other developers can make changes and improvements if [our own] funding ever runs out. The risk of having only one or two developers who know the code is that it is very difficult to create a community of developers around the tool.”
02.
Developers are building smaller tools as part of a bigger ecosystem
In general, our research showed a growing shift in the human rights documentation tools community away from big “kitchen-sink”** style apps (Martus, now retired, came up as an example here) and towards an ecosystem of smaller apps aiming to respond to a more limited set of needs, but able to be used together across a workflow. Some reasons given for this included:
- A big, “does-everything” tool can ensure a high level of security and/or a range of different functionalities, but it can also be more difficult to use for organisations who might not need all those features. It can also become a heavy burden on a tool developers’ capacity.
- Building a tool as part of a broader tool ecosystem can be more practical for individual organisations to maintain. Some tool developers thought that this approach could also offer some flexibility for the future:
“Combining tools allows a more tactical, agile approach, which more accurately meets the needs of changing contexts. There’s less often a single point of failure.”
In some cases, tool developers have been actively collaborating to make their tools interoperable. For some examples, see the Tools Table.
**From the English phrase ‘everything but the kitchen sink’, i.e. in this case an app that attempts to do a large number of things.
03.
When it comes to finding the right balance of features, tool developers face common trade-offs
Many of the tool developers we spoke to said that the particular tool they ended up building was arrived at in stages, through learning from past challenges, experimenting with features, and listening carefully to user feedback.
In all cases, many key decisions about features had to be made along the way, and these often involved making trade-offs. In our research, two surfaced as central:
- security vs usability, and
- flexibility vs structured workflows
3.1 Security vs. Usability
Security in the context of human rights documentation is complex and can be done in different ways, responding to different scenarios and addressing different types and levels of risk. Tool developers we spoke to emphasised that however security is approached, finding the right balance between security and usability for your tool’s users is key.
One common approach by developers of apps that face significant ‘security vs usability’ challenges has been to put effort into working directly with specific groups and organisations who are interested in using the tool or who fit the ideal use-case of the tool, rather than aiming for widespread individual uptake. This approach led to more successful uptake of these tools, though the number of users were potentially smaller.
The security/usability tradeoff can be a particularly sticky problem for data management, analysis and visualisation tools.
Developers of these types of tools noted that:
- Some high-security features, such as end-to-end encryption, can make accessing, managing and analysing data very difficult, and can lead to irrecoverable data loss if encryption keys are lost.
- Tools that respond to highly complex threat models but that are difficult for the intended user group to use can struggle with adoption. This approach can also result in security being compromised in unintended ways:
“What we’ve seen is that when [security features] make working with the data too hard, people work around it. So you have this beautiful [i.e. extremely secure] system in theory, but people subvert it.” – tool developer
As a result of this tension, and of learnings gained in the field in recent years, tools in the human rights space that have organisation, analysis or visualisation as their primary functionality lean toward workability over trying to respond to highly complex threat models.
3.2 Flexibility vs. Structured Workflows
In general, tool developers noted that a certain level of flexibility within apps can become a win-win: the tool’s users appreciate being able to adapt the tool themselves, and developers appreciate the reduced support burden.
As one tool developer pointed out, however: too much flexibility, and the user has no pathway or guidance through the app.
Flexibility, with “sensible defaults”
One tool developer suggested:
“An ideal future – though not necessarily new – is where you have flexibility, but are presented with sensible defaults. Organisations working within a workflow like it because it moves you through it, and removes some of the complexity.”
‘Sensible defaults’ might include things like:
- pre-loadable templates and forms (for example, a form that covers the minimum information needed for a certain type of submission to a particular justice mechanism).
- default categories (for example, countries or types of violation) and data structures (for example, relating different types of data to each other).
- default interoperability with another app for a different part of the data collection and management workflow.
- options for data visualisations.
Within a tool that maintains some flexibility, these defaults could then ideally be changed, deleted or added to by the organisation setting up the tool.