Database Encryption

[matteosecli]

Hello, I'm having troubles with the database after the last Mendeley update. My current situation is

• Mendeley 1.19.1 (latest version) on my personal laptop (MacOS)
• Mendeley 1.17.11 on my office workstation (Linux)

They are both synced to each other; however, Menotexport can read the database on the Linux box but not on my MacOS laptop. The error I get is the following:

# <Menotexport>: Failed to recoganize the given database file.
file is not a database

I'm pretty sure the problem is due to the fact that Mendeley has recently decided to encrypt the user database "to comply with GDPR & keep your data safe [...]" (which I think it's totally bull**it & it's not what GDPR says). I've found a comment on the issue & some instructions to obtain a decrypted database here, but I was not able to test them on my updated Mendeley install on MacOS because it seems that the MacOS version doesn't have a debug mode.

I'm reporting this as a bug even if it's actually Mendeley's fault; however, I think it could be useful to keep track of the issue.

Thanks again for your work! 🙂

[Xunius]

Hi matteosecli,

Many thanks for the report! It's a big thing, if I can't figure out a way to deal with the encryption then this tool will be dead. I'll put a note in the readme.

[williamgunn]

Hey @Xunius, we're really sorry we broke your tool with the encryption! We want to get it working again and provide a way to access the local database that we can officially support so that tools will keep working as we make changes. We're currently collecting use cases and setting up some one-on-ones with the development team. If you're interested, please email me.

[maphouse]

has this been solved? As a mendeley user (version 1.19.2), can i still use this tool? Would appreciate a definite answer before cloning. Thanks @Xunius !

[tuxflo]

I'm also interested since in version 1.19.3 there is no option to export highlights at all (only sticky notes). I just thought I found a great tool to do the research work for my thesis...

[matteosecli]

@maphouse @tuxflo I think you can still download and use either Mendeley 1.18 (Stable) or Mendeley 1.18.1 (Development) as a workaround. I suggest to install the old version on another machine or on a virtual machine in order to avoid potential conflicts with the newer version.

Here are the links (directly from Mendeley's servers) of these old versions:

Mendeley 1.18 (Stable)

• Ubuntu [32 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18-stable_i386.deb
• Ubuntu [64 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18-stable_amd64.deb
• Linux [32 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18-linux-i486.tar.bz2
• Linux [64 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18-linux-x86_64.tar.bz2
• Windows: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18-win32.exe
• MacOS: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18-OSX-Universal.dmg

Mendeley 1.18.1 (Development)

• Ubuntu [32 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18.1-stable_i386.deb
• Ubuntu [64 bit]: https://desktop-download.mendeley.com/download/apt/pool/main/m/mendeleydesktop/mendeleydesktop_1.18.1-stable_amd64.deb
• Linux [32 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18.1-linux-i486.tar.bz2
• Linux [64 bit]: https://desktop-download.mendeley.com/download/linux/mendeleydesktop-1.18.1-linux-x86_64.tar.bz2
• Windows: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18.1-win32.exe
• MacOS: https://desktop-download.mendeley.com/download/Mendeley-Desktop-1.18.1-OSX-Universal.dmg

[tuxflo]

Thanks for the quick reply. I just installed the Windows Version 1.18.1 but it seems that there is no export feature as well.... maybe I'd just look for some alternative to Mendeley although I really like the Annotation & Sync feature... and it seems to be the only "integrated" tool that runs on Linux as well.

[matteosecli]

@tuxflo have you tried to open the PDFs one-by-one and then select "File -> Export PDF with Annotations"? I have Mendeley 1.19.2 and this feature is still present, though Mendeley does not have bulk-exports and the annotations are exported in a non-standard way -- that's also why Menotexport was made.

Anyway I fully understand your point: it's one of the best reference managers out there and I also need a multi-platform solution, too bad they keep "closing" it more and more.

[tuxflo]

You are totally right: it is possible to do that when the document is open in the internal PDF Viewer (I already edited my comment but for some reason it wasn't saved)

Currently I'm facing troubles with installing your tool: do I have to install some python dependencies first? And do I have to clone the repo or is conda doing that as well (sorry I never heard of conda before...)

[matteosecli]

@tuxflo this tool was written by @Xunius (not by myself), so it's actually better to wait for his answer for this sort of stuff! 😃

[tuxflo]

OK Nevermind, I just skipped the conda stuff and did a regular install and now everything is just fine!
I'm using 18.1, it syncs and the (editable) bulk annotation works. Thanks so much for your help @matteosecli and @Xunius for creating this awesome little tool 👍

[matteosecli]

Glad that it works! 😄

[Xunius]

@tuxflo Glad that this little thing helps.

@matteosecli Thanks for helping out. I was away the previous week. And I just noticed that the PDFs we are exporting don't preserve metadata (some pieces of info the original PDF creator inserts into the file, like title, author, and even doi). For ordinary user this won't be noticeable at all, but for others it may be a pity to lose them as it's much easier to recognize a document from its metadata than parsing the file. So I'm planning to add this.
Also I'm "secretly" making a rip off a mendeley (working name "elemendy" XD). Of cause it won't be as powerful but it will be totally transparent. I would let you know if you are interested, and when it gets somewhere (currently it's still preprepre alpha).

[tuxflo]

@Xunius Yes I'm interested! I just had the same idea (build a open source alternative to mendeley) yesterday.
Currently I don't have time to do some coding myself (writing my thesis, thats why I came in touch with mendeley) but here are just some thoughts I had:

• I would not take care of syncing and use external existing tools for that (nextcloud, gdrive, dropbox...)
• It would be a huge improvement if there would be touch support in the pdf viewer for scrolling and for annotating (Currently there is no pdf viewer for linux that can handle annotations on touch devices)
• Highlights made with other tools should be imported and visible/editable in the tool (one huge disadvantage of Mendeley in my opinion)
• Auto save feature would be nice

If you are still working on the project after march next year I'd be happy to help to get this thing done :)
I have some knowledge in different programming languages, especially in C++ and Qt (but python is good as well)

[matteosecli]

Hi @Xunius, sorry for the very long absence! I thought I would have had less work after October, but at this point I think it was just a sci-fi dream and the work kept increasing and increasing...😅

Anyway I really like your idea! I was trying to look into that by myself a couple of years ago, but I really don't know anything about user interfaces (I'm a command-line guy) so I quickly gave up. But I'd be very interested if you let me know when you have something, at least I can help you testing or (if I have time) give some contributions to the backend part.

About the metadata problem...do you already have some hint about the cause and (possibly) the solution? Unfortunately I don't know so much about the PDF library you're using here. Otherwise one could always resort to a brute-force thing like: copy the meta before adding annotations and then paste them back.

[Xunius]

Hi @tuxflo @matteosecli
Glad to hear positive feedbacks from both of you (I just realized that we probably have a Mendeley guy in this thread, but that's ok I guess).

I can totally appreciate that we all have our own priorities, my own progress has been slowed down a lot recently.

I agree with tuxflo that I'm not going to implement any syncing and will let the user use whatever to their liking. Auto-save is on my plan as well. In terms of pdf viewer, I have little idea how difficult it would be to embed a viewer into this, my plan (at least for the 1st stage) is to let the user fire up an external viewer, and move onto an embedded one should the 1st stage goes smoothly.

A few other things on my list:

• I'm using PyQt5 to build the GUI, currently have got the main frame set up, with the basic structure similar to Mendeley's, plus some minor tweakings in the right hand side tab panel. The top menus, option dialogues etc are still empty now.
• For database I'm using sqlite, same as Mendeley as well. The reading part is done. I didn't use Qt's sql interface, but stored the data into dictionaries and displayed them using QAbstractView. Probably won't be as efficient. The saving part hasn't started yet.
• For Bib files I'm planning to use bibtexparser, and doi querying habanero. No idea for RIS yet.
• For duplicate checking I'm using fuzzywuzzy.
• For fetching metadata from PDFs (a real nice feature of Mendeley), I'm still relying on pdfminer and PyPDF2, this is when I noticed that preserving metadata would be beneficial because it's so much easier. For those that don't have metadata, I'm testing parsing only the 1st page and trying to guess the title, authors etc.. But I kind of doubt this would be a good option, because unless I could make it real accurate, it might be better to leave a field empty than filling with wrong info. For some PDFs pdfminer seems to take quite a long time to parse even the 1st page, which is another discouraging factor.
• Another nice feature of Mendeley is full text search, which I really want to have. I've heard about "xapian", but no idea how easy it would be to implement.
• I'm still collecting ideas. For instance I thought about inserting a note pad somewhere so you can jog down some quick thoughts that's not associated with any document like currently in Mendeley's note tab. But then I thought the user might have his favorite text editor at hand so it doesn't make too much sense to have such a thing, unless it can be made into a really good one.

That' about it, still a whole lot to be done. If you get any idea please let me know. Winter holiday is coming so I should be able to make some process soon.

[sojusnik]

Last year, a Mendeley update went live that had the unintentional consequence of hindering interoperability. We made a change to the Mendeley Desktop application that broke some integrations with users’ local Mendeley database, including Zotero’s import tool, which resulted in it not working with Mendeley. We’re really sorry about this — it was never our intention to break these integrations and we should have picked this up in release testing.

We’ve heard a lot from our users about this and our team has been working behind the scenes on two things.

First, we clearly heard from users that they want to be more in control of how and when they can export their PDFs, annotations and highlights directly from Mendeley. Improvements were released in Mendeley Desktop version 1.19.3 and further improvements to the export process will be released as part of Mendeley Desktop version 1.19.4.
source

Let's see what the future will bring.

[Xunius]

Hi @matteosecli @tuxflo,

It has been long time since we talked about the "rip off" project. I haven't been idle, it took me much longer than I expected to come up with the 1st alpha. But here it is finally (https://github.com/Xunius/MeiTingTrunk).

As I mentioned previously, the database is a sqlite, with a structure similar to Mendeley's but simplified. I've managed to implement import/export of bibtex and RIS. DOI query is added, and autosave is also there. But the big thing missing is citation style formatting, and full text searching in PDFs is lacking atm. I also want to implement import from Zotero, but I guess I need to make this more appealing first, otherwise Zotero users won't be attracted in the first place.

I haven't got time to pack it up in an easier-to-install format (I assume that won't be too difficult), so currently one has to install the dependencies manually, But those are pretty standard packages and can be install easily with pip.

Please give it a look if you are still interested.
Btw, have you tried the newer versions of Mendeley, did they remove the encryption?

[matteosecli]

Hi @Xunius, thanks a lot for the update! It already looks like a great work! 😄

I'll test it more thoroughly and come back here asap! 😉

(btw, newer versions of Mendeley still seem to have encryption onboard)

[JoTrab]

I found a solution to circumvent the encryption problem. After a lot of researching around and always coming back to this page I stumbled over the following:
https://eighty-twenty.org/2018/06/13/mendeley-encrypted-db

The author explains a procedure to de-encrypt the sqlite. I am right now testing a direct zotero import from the de-encrypted file as well as a menotexport extraction on the de-encrypted file. So far it works like a charm.

I had to run the x/s $rdi procedure twice due the $1 = 8 instead of $1 = 0 error as suggested by the author.
If you guys could link/add to the manual or if you feel include the procedure to the program I guess a lot of people will be quite happy.
At least I can finally move on to zotero with my very large literature database.
Keep up the good work

[Xunius]

@JoTrab Sorry I totally forgot about your comments. I've added your suggested solution to README. Many thanks.