There have been quite a few updates to how Unredacted works from a technical perspective. This is a guide to the latest developments.
URLs
We have changed the format of our document viewer URL structure to a more permanent system.
Previously, the document viewer incorporated the PDF as hosted by DocumentCloud in the format
https://unredacted.uk/dcviewer.php?dcid={00000000}… where {0000000} corresponded to the ID of the document hosted on DocumentCloud. Here’s an example.
However, this caused issues when we sometimes had to re-upload a PDF. This has happened frequently in the last few weeks as we have been re-indexing documents so that they are more machine-readable using the Textract OCR engine. This is important for documents that are ‘messy’ – with handwritten notes or less-than-perfect scans.
In these instances, the DocumentCloud ID (dcid) has to change – the old document is deleted and a new version is uploaded – which means the parameter in the URL needs to change as the dcid is how the document is found on DocumentCloud.
NEW FORMAT
To get around the problem of dead URLs, we have created a redirect system so that old dcid URLs will load the new document, but we have now created a new URL structure like this:
https://unredacted.uk/documents/{yyyymmdd-ORG-Doc-Type-999}… where yyyymmdd is a numeric version of the document date, ORG is an acronym for the organisation that published the document, Doc-Type is the document type, and 999 is an ID entry. This format is the one we will use from from 27th November onwards.
This format allows the URL to stay the same regardless of which DocumentCloud PDF is loaded – so URLs don’t need updating if there is a new DocumentCloud PDF. This helps create a more permanent archive of links.
If you have any old format URLs, they will continue to work as we have developed a system to map old dcids to new ones. However, if you find any that for some reason don’t, please get in touch to tell us.
UPDATE, December 6 2024: This new format ended “.pdf” in the URL parameter – this has been removed.
OCR / Textract
As mentioned above, we have been using DocumentCloud’s facility to use Textract, an OCR engine, on many of our more ‘messy’ documents. This allows for a far better searchable experience on Unredacted. An example of a file that has been processed with Textract is on the right.
(OCR stands for Optical Character Recognition, which is a system that converts text in images into a machine-readable format.)
To give an example of why this is so valuable to our archive, look at the following example.
This is the document:
This is how the text was initially rendered in the PDF:
And this is how it is read by Textract:
It’s not word-perfect, but the difference is huge.
We have not run the entire Unredacted archive through Textract, as many documents are perfectly readable without it. Plus, it would take a long time. However, we are keen to run key documents that originate from scans or have handwriting through the system to improve our search results.
If you see any documents you think would benefit from this, please let us know.
Thanks
The Unredacted team.