Using Your Personal Knowledge Base on an Untrusted Computer

Obsidian, Logseq, and similar note taking and Personal Knowledge Base (PKB) tools work on a local folder of files. This is fantastic, but also means you need to handle encryption, backup, and syncing yourself. Below, I share my experience for a very specific use case: running Obsidian on an untrusted laptop, which requires data-at-rest encryption of our PKB (our folder of markdown files), and syncing the PKB to the cloud.

Our Requirements

  • Run Obsidian to maintain a Personal Knowledge Base on an untrusted laptop (eg: corporate work laptop) running Windows 10
    • if motherboard fries for example, the laptop would need to be sent back to the employer with hard drive data intact. This means, data at rest encryption is required so the employer cannot read our personal KB
  • Sync Obsidian data to an untrusted cloud. Data must be encrypted during transit, and must be stored encrypted in the cloud
  • Backup so we never lose more than an hour of work in case the laptop burns up, for example
  • My PKB consists of ~450 files, including ~25 images and non-.md files, at a total of 15MB.

Solutions attempted

Below, I share my experiences with several solutions that I tried. They include file level and container level encryption of the PKB.

What was I doing before Obsidian:

I still had my vast directories of files. I used to use vim’s built-in blowfish encryption on a per-file basis, and sync these to the cloud via Google Drive. Voila: no setup to do, nothing to worry about. However, this became impossible with Obsidian given its need to be able to read all files in the plain.

A. Safe (a Windows EncFS client) + Google Drive = Errors

Idea: Store EncFS encrypted files into a Google Drive folder, mounted using Safe

  • Wasn’t impressed by the client, its setup, interface, lack of clarity in whether and how it had installed onto my computer

  • Copying my PKB over via Windows Explorer showed a transfer rate of 1kB/s. Not exactly usable. Until it got worse and errored out with:

    An unexpected error is keeping you from copying the file. The process cannot access the file because another process has locked a portion of the file.

So this was a no-go.

B. Cryptomator + Google Drive = Poor Obsidian UI experience

Idea: Same as (A) above, except, using Cryptomator instead of Safe. Use Google Drive “backup and sync” client (NOT GDrive for Desktop)

Typing/scrolling in Obsidian became very laggy, to the point of being unusable! Why? See below:

Investigation

  • Same vault on local disk: fine, no lagginess
  • Same vault on Google Drive (no Cryptomator): fine, no lagginess

  • Cryptomator only (GDrive turned off): poor and very laggy

  • So the problem seems to be Cryptomator
    • Obsidian’s auto-save causes Cryptomator to frequently shoot up to 30% CPU usage for several seconds a time, during which Obsidian UI becomes unusably laggy. This is on a 2021 flagship laptop, so hardware resources are not the issue
    • need to examine Cryptomator logs to find out why
    • 2020 cache-related cryptomator thread post seems to indicate this is not a problem any more because caches are not stored in the vault. I haven’t been able to verify this
    • Cryptomator mounts via Dokany/WebDAV both had similar problems
    • As an aside: when using Dokany, the mount can be mounted and accessed in WSL via: sudo mount -t drvfs D: /mnt/d. Howevever, this is not perfect:
      • there are filesystem errors when doing a find on the mount via WSL.
      • opening more than one file at a time doesn’t work. eg: vim -p or cat x >> y # neither work)
      • things work fine when using git-bash which is a MINGW environment, more native to Windows than WSL

C. rclone mount crypt on local disk + GDrive = Copy Errors

Idea: Same as (B) above, except, use rclone mount instead of Cryptomator

This approach uses rclone’s ‘mount’ option, with a ‘crypt’ remote that is pointed at a local disk folder, which is sync’ed the cloud via Google drive. Fail. 1kB transfers, tons of errors

$ ./rclone.exe mount rctest: E:  --vfs-cache-mode=full
The service rclone has been started.
2021/02/26 22:23:40 ERROR : perf/SPEEDTEST.ref/.obsidian/workspace: Failed to copy: corrupted on transfer: MD5 crypted hash differ "1425e09b36ee959dbe81d94798a35d44" vs "be1125c1cb45531653b55d36ae1cf54c"
2021/02/26 22:23:40 ERROR : perf/SPEEDTEST.ref/.obsidian/workspace: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: corrupted on transfer: MD5 crypted hash differ "1425e09b36ee959dbe81d94798a35d44" vs "be1125c1cb45531653b55d36ae1cf54c"
2021/02/26 22:23:46 ERROR : perf/SPEEDTEST.ref/.obsidian/config: Failed to copy: corrupted on transfer: MD5 crypted hash differ "442ec56ce0618324566c534c08bae93c" vs "7671afa4f61d9bb1ae1b34a87d5a736f"
2021/02/26 22:23:46 ERROR : perf/SPEEDTEST.ref/.obsidian/config: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: corrupted on transfer: MD5 crypted hash differ "442ec56ce0618324566c534c08bae93c" vs "7671afa4f61d9bb1ae1b34a87d5a736f"

See [vfs-caching](https://rclone.org/commands/rclone_mount/|vfs-caching)

This was surprising given how well written and tested rclone is. It’s possibly because rclone mount is not designed to handle high frequency updates to small files? I don’t know.

D. Windows encryption + rclone/crypt sync via cron for sync = Not Secure

This solution relies on Windows 10’s inbuilt EFS file-based encryption for local data-at-rest encryption. To sync to the cloud, a Windows Task Scheduler based periodic task calls rclone sync to upload to a crypt remote on Google Drive

Apparently, it is possible for files encrypted via Windows EFS to be read by admins including domain admins, and local admins. So this is a no go.

E. The Winner! Veracrypt + rclone/crypt sync for encryption + GDrive for sync

Store the PKB on a Veracrypt container. This takes care of local data at rest encryption. However, syncing the container to the cloud directly is painful because of block level changes within a large file. Instead, use rclone’s crypt remote to sync.

The actual solution I use writes to a local rclone crypt, which is picked up by the cloud sync software (Google Drive), that runs on my laptop anyway:

This was the solution that worked very well that I now use. Pros:

  • Obsidian UI is as quick as it is with a local disk without encryption. No lagginess
  • rclone sync takes < 0.1s on my PKB
  • have control over when to sync, and hence, can use the cloud for disaster recovery (eg: accidental deletes don’t get synced instantly and propagate the delete to the cloud)

Cons:

  • Need to open WSL (or cmd.exe) and run the rclone sync manually. Could automate this through a cron or Task Scheduler. From within WSL, I do: 
      test -e "/mnt/e/KnowledgeBase" && rclone sync -v /mnt/e kb:
    • An interesting approach would be to write a simple plugin for Obsidian that calls the command above when invoked via a hotkey
  • need to load Veracrypt container whenever I reboot

Alternate solutions

  • git-crypt will encrypt files, but not filenames and directory names
    • A bare repo, which is encrypted at rest using git-crypt, would be stored on the hard drive, and it would be checked out and unlocked upon each boot, on a drive mounted on memory. This way, if the computer powered down and became non-functional, the unencrypted data would be in accessible
  • A vault encrypter built-in to Obsidian would be the ultimate solution. A password would be asked for when opening the vault, using which all reads would be decrypted and writes, encrypted. This means the vault would never write plaintext to disk, meaning no setup is required other than standard cloud software like Google Drive to sync

Things to keep in mind

If using rclone, encrypt the config file, because it contains your password for the crypt remote. If you are calling rclone from within WSL or Cygwin, use pass to store it conveniently, but securely.

Developing a benchmark that works across a folder of hundreds of .md files, writes to several at random, and measures CPU usage might help evaluate these and other solutions.

Conclusion

It is possible to setup one’s Knowledge Base, and use Obsidian (or similar tools) on an untrusted laptop while backing up to an untrusted cloud. However, many well known tools and software surprisingly fail to work under a small, frequent, file update scenario, which is what Obsidian produces. A working solution was built using Veracrypt and rclone.

Notes mentioning this note

There are no notes linking to this note.