Forums

I am trying to setup a django webapp which queries data from a large (~1.8GB) json file. Reading this file every time a query is performed takes too long so I need to have the contents of the json file available in memory. In my offline version I can just load the file in the apps config ready() function or even as global variable in the views.

On pythonanywhere however, if i try to load the file in the apps config, it is never ran and if I put it in the views file it will hang the first time a view is loaded because it takes 5-10 minutes for the file to be loaded or created via script.

How can i approach this problem so that the file content is available to the django views? One solution I can think of is to load the file when the webapp starts, but from what I can tell pythonanywhere doesn't allow this as it blocks reloading of the webapp if it takes more than 30 or so seconds.

It looks as though the problem is that the workers use too much memory loading the file so it crashes with a 502 error. How can I get around this?

If you have that much data, it's probably not going to work keeping it in memory. Your website has multiple worker processes, so each one would need its own copy, and even if that fitted within the per-worker RAM limit you'd still be using up a huge amount of RAM.

Is there a specific reason that it's in a JSON file? Could you perhaps keep it in a MySQL database and query it from there? That would be a more normal implementation.