Fight for the Internet 1!

Monday, June 9, 2014

How to move files for Moon+ Reader while preserving metadata (bookmarks, highlights) or recover lost metadata

Overview of Problem
This is about using the Android electronic book reader app "Moon+ Reader" to move your files around successfully without losing the metadata (bookmarks, notes, highlights, progress, etc.). This technique can also be used to recover those if you lost them.

Problem
My books were stored in "/sdcard/Books/Changeling: The Dreaming" (for this example). I needed to move these PDFs and CBZs to a different location. Using a File Manager I moved the files to "/storage/extSdCard/Changeling The Dreaming". (I had to remove the ':' in the name because of the Fat32 filesystem format on the SDCard. Curse you, Microsoft.)

Then I re-imported the books into Moon+ Reader. The app removed the previous item entries and imported the new ones, but it failed to recognize they were the same PDF and CBZ files. It did not transfer the metadata. When I say metadata I refer to the bookmarks, the highlights, the notes, the reading progress, or even the names I manually assigned to the books, etc. (Some of the PDFs needed better names.)
This bug breaks lot of features every time there is moving or renaming files/folders. I love being able to highlight my books, but Moon Reader needs to be able to handle if I move my files around.

Solution 1 -- If you have not yet lost your metadata

0) In case it needs to be said, always make a backup of your book files and your information. Moon+ Reader lets you store backups of your metadata easily through its menus. Do this before proceeding.
1) In Moon+ Reader, go to My Files. (Not "My Shelf". You need to go to My Files specifically.)
2) Using Moon+ Reader, Select and then Move any existing items.

Note that Moon+ Reader may not be able to actually move the files. The reason for this is in Android version 4.4.2, Google revoked the privilege for any apps to modify files on external SD cards. (Some limited Google specific apps can use your SD card though.)

My files were on my external SD, so Moon+ Reader could not actually move them. If your files are only on the internal SD card, it seems to be able to move them just fine.

(WARNING: Because of the warped hybrid of external SD and internal SD storage, Moon+ Reader actually deleted any duplicate files at the target move location, so watch out! Your experience may be different, but always have a backup.)
3) Remove the selected item entries from within Moon+ Reader itself.
4) Now in your file manager, move the actual file object itself to new target location.
5) Back in Moon+ Reader, now do a normal Import Books.

Your metadata should have been transferred when you re-import the new files.


Solution 2 -- If you have lost your metadata 
This is when you have moved your files, and re-imported them, and your metadata didn't show up. Here is how to fix the problem.

0) In case it needs to be said, always make a backup of your book files and your information. Moon+ Reader lets you store backups of your metadata easily through its menus. Do this before proceeding.
1) In Moon+ Reader, remove any of the new items you just imported.

In my case, I had moved files /sdcard/Books/Changeling: The Dreaming (for this example) to a different location at "/storage/extSdCard/Changeling The Dreaming". I went back through Moon+ Reader and removed the files now at "/storage/extSdCard/Changeling The Dreaming".

2) Now Moon+ Reader should have none of the botched entries stored in it.
3) In your file manager, move the files back to where they were before Moon+ Reader lost the metadata. (If your files are stored on an external SD card, remember that Android 4.4.2 doesn't allow most apps to modify external SD contents, so may need to remove the SD card and do the file management on your desktop.)
4) In Moon+ Reader, re-import the old files from their old location.
5) Now Moon+ Reader should restore all your old metadata (bookmarks, notes, highlights, etc.)

Now you are ready to migrate your files in the special way to preserve your metadata.

6) In Moon+ Reader, go to My Files. (Not "My Shelf". You need to go to My Files specifically.)
7) Using Moon+ Reader, Select and then Move any existing items. Note Moon+ Reader will not actually move the files. (WARNING: My setup with Moon+ Reader actually deleted any duplicate files at the target move location, so watch out! Your experience may be different, but always have a backup. )
8) Remove the selected item entries from within Moon+ Reader itself.
9) Now in your file manager, move the actual file object itself to new target location.
10) Back in Moon+ Reader, now do a normal Import Books.

Troubleshooting
If you get "import: 0", then you did not remove the entries before trying to import. Be sure to check if you double imported them, because they might have nasty duplicates at the new import location where you were trying that do not have the transferred meta data.

Hack Editing the Backup Files Directly
Be careful when editing the Backup files yourself. You'll probably need some really basic knowledge of how SQL commands work, but not much.

About Moon+ Reader's Backup Files
Moon+ Reader's backup files have the file extension .mrpro (on the Pro version anyway). This is actually a Zip file. If you extract it, inside is a folder called "com.flyersoft.moonreaderp". Inside that is a bunch of files. In my case, there is typically about 60 files, most of them numbered files from 1.tag to 57.tag files. The important one you want is one of the high numbers. Look at the file size also to help you. One of them is actually the SQLite3 database. Mine was about 1.1 MB, while the rest were tiny, which tipped me off it was the real contents.

SQL Editing
I used an SQL editing program to view the SQLite3 database file. If I remember, I used "Sqliteman" which was simple enough for my needs. Using some simple find and replace string commands in SQL, I edited the existing entries with my meta data (highlights, notes, bookmarks, etc) changing them from the old filepaths to point to the working ones.

Finishing -- Zipping and Re-Importing and Restoring
After editing the appropriate .tag file (which is the SQLite3 database file actually), I zipped the entire folder back up, and gave it a name with a newer date, then imported that through Moon+ Reader, restoring from there. This fixed my paths.

Though now I just prefer to root my devices which allow me to give admin privileges to certain apps to move files around.
 

Addendum: Proposed Solution
I sent an email to the developer of Moon+ Reader about a proposed solution to fix this. I suggested he use file checksums (such as sha128 or md5sum) on each item that is imported. This way he could match files that move or are duplicates.
Moon+ Reader uses SQLite version 3 underneath and all the highlighting, bookmarks, notes, etc. are stored in these. You can even gain direct access to these by having the program use the "Backup" and "Restore" features. So if you want you can edit the SQLite database manually. I have actually hacked my Moon+ Reader SQLite files quite a bit at this point to clean up some bad stored book-entries.
While the developer did kindly respond, he never said that he would do this or not. I hope he does. But he did give me a hint on how to what had happened.

9 comments:

  1. Hi
    In my case Moon Reader does indeed move the file and delete the metadata with it, so am I doing this wrong?

    ReplyDelete
    Replies
    1. No, that is fine. I recently discovered the problem for why it was not moving them. In Android version 4.4.2, Google revoked the privilege for any application to modify files on external SD cards. (Some limited Google specific apps can use your SD card though.)

      My files were on my external SD, so Moon+ Reader could not actually move them. If your files are only on the internal SD card, it seems to be able to move them just fine.

      Delete
  2. What if I have lost the old phone but I restore the books from back-up to new phone? Is there a way to get the metadata back?

    ReplyDelete
  3. Hi, I was Googling about this subject, i.e. reading progress, metadata, etc. etc. and I came about another solution which is complementary to your solutions, and which I think would benefit users reading your article here. http://www.techlife.net/2014/03/sync-reading-progress-android-devices.html

    In a nutshell, the linked article is about the Moon Reader function that "How to sync your reading progress between Android devices". It allows users to QUOTE ‘Sync reading positions via Dropbox’ (or Google Drive) (even though the ebook is electronically stored locally on your smartphone or tablet.

    A 2nd valuable function mentioned in the linked artcile allows users to QUOTE "You can also copy your preferences between your devices by hitting Backup button, then choosing Dropbox (or Google Drive), then choosing Restore on your another device."

    ReplyDelete
  4. Hi - you said:
    "Moon+ Reader uses SQLite version 3 underneath and all the highlighting, bookmarks, notes, etc. are stored in these. You can even gain direct access to these by having the program use the "Backup" and "Restore" features. So if you want you can edit the SQLite database manually. I have actually hacked my Moon+ Reader SQLite files quite a bit at this point to clean up some bad stored book-entries."

    I've found the backup MRPRO files that Moon Reader Pro creates, but I can't seem to read them - when I try reading them in a SQLlite browser, it tells me it's an invalid file format. Thought I'd check to see if you have any ideas!

    thank you!

    ReplyDelete
    Replies
    1. Sorry for not noticing your comment earlier.

      Moon+ Reader's backup files have the file extension .mrpro (on the Pro version anyway). This is actually a Zip file. If you extract it, inside is a folder called "com.flyersoft.moonreaderp". Inside that is a bunch of files. In my case, there is typically about 60 files, most of them numbered files from 1.tag to 57.tag files. The important one you want is one of the high numbers. Look at the file size also to help yuo. One of them is actually the SQLite3 database. Mine was about 1.1 MB which tipped me off it was the real contents.

      I used an SQL editing program. If I remember, I used "Sqliteman" which was simple enough for my needs. Editing SQL is kind of a pain but I edited the existing entries with my meta data (highlights, notes, bookmarks, etc) changing them from the old filepaths to point to the working ones.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. What did you use to edit the backup file? I change tablets but the file paths are different so non of meta data are transfered over. Bookmarks, reading hours etc,. I want to edit file paths from the backup file like you said. Please and thank you.

    ReplyDelete
    Replies
    1. Sorry for not noticing your comment earlier. I used an SQL editing program. If I remember, I used "Sqliteman" which was simple enough for my needs. Editing SQL is kind of a pain but I edited the existing entries with my meta data (highlights, notes, bookmarks, etc) changing them from the old filepaths to point to the working ones.

      Delete