CouchDB maintenance & backups

[Alex]

During lunch, the CouchDB database (the thing that holds all of your budget data) was recently updated to the latest version for security and performance improvements.

No downtime was expected & nothing user-facing was expected to change.

In other news, I've made automated backups of the forum post/topic data on a nightly basis.

In more backup related news, while Financier (the app) has always had nightly backups to multiple redundant locations, I may be introducing live backups soon with a completely separate data backup method for more redundancy and data durability in case of corruption of the nightly backup.

Thanks all!

[asromzek]

Can you restore the budget that I accidentally deleted in November 2016?

[jat255]

Actually, in that vein, do you know off hand what the address for a request to get a budget in export/import format would be? I'm thinking of setting up a cron job with a curl request (presumably with an authentication cookie) to auto-backup my budget .json locally every day (now that my wife and I are sharing one account, things could unintentionally go haywire).

[Alex]

Can you restore the budget that I accidentally deleted in November 2016?

Yes, I can. But, it takes a while to restore since I have to mount the backup data to a special invironment, run your account, 'hack' into your account, and export the budget.

Takes a couple hours. Is there a couple months+ of budget data in it that would make it worthwhile to extract?

Actually, in that vein, do you know off hand what the address for a request to get a budget in export/import format would be? I'm thinking of setting up a cron job with a curl request (presumably with an authentication cookie) to auto-backup my budget .json locally every day (now that my wife and I are sharing one account, things could unintentionally go haywire).

Haha The exporting process is currently done in the browser, not through a rest endpoint or anything

That being said, it would be easyish to make a script to export your budget data. You'd have to strip the backup logic out of the frontend code and pass it username+password+budget id => .json backup file.

[asromzek]

... would make it worthwhile to extract?

Not really, I just want to know if you can do it.

[Alex]

... would make it worthwhile to extract?

Not really, I just want to know if you can do it.

What day do you want? Budget name?

Maybe if I'm bored...

[asromzek]

That being said, it would be easyish to make a script to export your budget data. You'd have to strip the backup logic out of the frontend code and pass it username+password+budget id => .json backup file.

Or use the new library that I'm building in a custom Node.js project? For example, you would create an instance of the library, log in, open the budget, finish syncing, export to a file on the hard drive, and done. It's headless library, so as long as I expose the necessary functions it shouldn't be too difficult. I did not replicate budget export in the mobile app, because... it should be obvious. I'll add that to the list for the library.

[Alex]

The nice thing about the backup routine is it has nothing to do with the budget calculation logic, so it'd be pretty easy to extract into a no-dependency function.

https://gitlab.com/financier/financier/blob/master/src/scripts/services/backup.js

^ that file and run the function after a complete pouchdb sync would be all you need @jat255.

https://pouchdb.com/api.html#sync

[jat255]

... run your account, 'hack' into your account, and export the budget. ...

Is this something that can be done at any time? I hadn't gone through it, but I was hoping that the budgets were encrypted somehow on the remote storage. Is this not the case?

[Alex]

... run your account, 'hack' into your account, and export the budget. ...

Is this something that can be done at any time? I hadn't gone through it, but I was hoping that the budgets were encrypted somehow on the remote storage. Is this not the case?

There's a bit more information here:

https://blog.financier.io/security-at-financier-337ee9a64ea1

I also responded to someone asking that question in the comments of the blog post.

Your password is encrypted (and payment details are stored with Stripe, not me), but other than that, raw budget data is not encrypted.

[Alex]

To clarify, info at rest is not encrypted, but over the wire it is encrypted. Nature of CouchDB - see https://stackoverflow.com/questions/44585302/what-encryption-mechanism-is-used-in-couchdb

https://github.com/calvinmetcalf/crypto-pouch is possible, but it would make life difficult for me with support for regular users - if you lost your password, you'd lose everything, period.

[jat255]

Gotcha. crypto-pouch would be a cool option. It could be disabled by default, but users could enable it with a big warning and clear understanding that no password = no data (even for the budget overlord).

[jat255]

... run the function after a complete pouchdb sync would be all you need @jat255.

Sadly, with my total lack of js knowledge, the easiest thing for me to do is write a selenium script with python to go get the backup, lol