Cannot load budgets due to memory limits

[jat255]

I posted a reply to another thread, but this is probably best off as a bug report.

As a regular user of financier for a few years now, my budget file has grown quite large. It takes a long time to sync, but as of the past few days, I actually cannot load my budget at all. During the syncing process, after the network transfers have finished, the "memory footprint" of the Financier tab grows to 2GB+, and then the tab crashes with the "Aw, Snap" message (SIGTRAP). Seemingly no matter how many times I try to load the page, I cannot get it to load. I have a backup of budget json file, but this is really not good.

Any ideas, @Alex ?

[asromzek]

Have you deleted your budget and re-imported it a few (or many) times? I ask because I did that with my main account when developing the mobile app, not realizing that a deleted budget isn't truly deleted in CouchDB. It gets flagged as deleted but is still included when the database is synced with the browser initially so other synced apps can delete it locally. This is especially a problem with the mobile web app since Safari on iOS limits database sizes to 50MB, and none of my budgets would load when the database exceeded that size. I had to coordinate with @Alex to delete my database so I could import a clean backup that significantly reduced the new database size. I'm not sure if that is the source of your problem, but it sounds similar. Although, 2GB seems really excessive for Financier. The app shouldn't be that heavy.

If you're still working with a single original budget and have an up-to-date backup, then I recommend going through the motions of clearing your browser cache and syncing from scratch again. You could also try a different browser to see if the same thing happens. If you have done all of that already, then @Alex will probably have to weigh in.

Out of curiosity, how big is the json backup file?

[jat255]

That could certainly be. What's interesting is the massive memory use doesn't start until after all the network traffic has finished, and the problem occurs even when loading Financier in an incognito tab, so there shouldn't be any cache or anything hanging around. Actually, the only place I can use it is in my daily computer where I have a recent sync. That loads okay, but on any new machine I can't open it.

Our primary budget file's json is 5.8MiB, so fairly large, but I don't understand what's happening that could use 2 GB of RAM...

[jat255]

I have a script that backs up my budget using headless playwright that still works (must not be subject to the same memory limit as desktop Chrome?), so I have recent backups, meaning if I could purge the DB on the remote server and then let my client sync back, that might help, perhaps?

[jat255]

Like @asromzek suggested, I believe this is due to some sort of database pollution/corruption. I set up my own backend following the method of https://gitlab.com/josh-whitney/financier-backend, and after importing my most recent backup, I'm not having any problems with the machines that were previously crashing (and it's substantially faster since it's all on my local network).

[asromzek]

@jat255, I was going to see if you were comfortable with sending me a copy of your budget to do that exact kind of test, and completely forgot (it's been a busy week). It looks like you came to the same conclusion I did. If @Alex can help you out, the process requires signing out and clearing data in all of the browsers/apps where you have your budget loaded, he creates a new database under the hood, and then you import a fresh backup.

I have not deleted/imported my budget since I bloated my last database, and have tried to be as efficient as possible when making changes. For example, If I need to change the flag color and the cleared status on a transaction, I will go into full edit mode so both changes are saved in a single update, instead of two separate revisions for the transaction.

[jat255]

Yeah, I think I'm actually going to continue to self-host the backend, since I've never been super comfortable with all this info on someone else's server, unencrypted. I trust Alex, but I'd rather have all of this stay on my own servers behind my VPN. As a benefit, I'm understanding how things work a little better under the hood, so maybe I'll find some time to do some playing around. I saw there was at one point an outstanding PR for searching and copy/paste from https://gitlab.com/palidanx/financier/-/commit/763fddcaab73489a1527c75afb55195d04c1b809 that I will try to implement on my fork.

[asromzek]

Talking about this stuff is starting to get the creative juices flowing again. I have been kicking around the idea of building a new budget app similar to YNAB4/Financier, using some of the newer C# technologies that have come out over the last few years. I work with that stuff at my job every day, so I would be very comfortable building something from scratch.

I have gone back and forth on whether it would be a simple, standalone desktop app, or a full blown web app (desktop & mobile) with full syncing capability. If I implement syncing, do I build it so it's self hosted for myself, and available for gurus to self host themselves, or set up a subscription service so anyone can sign up and pay for the syncing capability like Financier? If I didn't have to worry about hosting and securing data for other people, that would certainly keep things simple on my end. But if I build something new and useful, making it available and fully functional for everyone instead of just technology enthusiasts also sounds really attractive to me.

CouchDB provides really nice out-of-the-box syncing capability, but it also has some drawbacks when pushing the limits as we have encountered. It also makes it a little difficult to implement some features like scheduled transactions. I actually sat down and sketched out a plan for building a syncing algorithm a couple months ago that would solve some of those issues, but never got around to writing any code to test it out. I'm probably going to pull that sketch out again to see if I can put it into practice.

Financier has worked great for the last few years and I hate to reinvent the wheel, but it might be fun to build something new that solves some of the underlying issues and makes it possible to implement some of the features that have been requested like scheduled transactions, file import, search, etc.

[Alex]

@jat255 sounds like you might've got a solution.

Just curious - oftentimes the huge syncs is do to importing, deletting, importing, deleting budgets. The budget data isn't ever actually "deleted" in CouchDB due to the way CouchDB's sync mechanism works.

If you'd like, I can remove all data on your Financier account after you export your budget data you want to keep to JSON. Then, once you log out (very important!) from ALL devices, you can re-import, and it should work great.

Alternatively, you can always create a brand new financier account - (hello+blah@gmail.com) and import budget data to that.

Should sync much faster.

[jat255]

@alex, I've been running Financier self-hosted now for about a year or so, which has been working well.

I haven't run into the strange memory error again, but my DB is still taking a _long_ time to initially sync. Looking at the network traffic, this is because the app is making calls to https://host/db/userdb-abcxyz/_bulk_get?revs=true&latest=true that take about 3 minutes to load 200 kB, and it keeps repeating that for quite a while.

This definitely isn't a network issue, since it's all self-hosted on my LAN, and the host, while not the most powerful machine in the world, the couchdb processes are using only about 5-6% of the CPU during the initial download, so I'm not sure where the hangup is coming from.

Have you seen anything like this/do you have any suggestions?

EDIT: it finally finished, and took 1.4 hours to download 14.8MB, according to my DevTools network pane. Thankfully I don't have to do this too often, but I'm wondering if there's some way to fix it. I found this: https://github.com/apache/couchdb/issues/4183 which suggests there have been some recent changes to couchDB (not yet released) that may help things. Just wondering if there's anything that can be done on Financier's side