Forums

In my current project, I have a Django web app which both receives data from a number of external sensors, and has a function to view said data. I have left it during the holidays to collect data, and I have now tried to access the data, but I have noticed that the website is insanely slower.

In my function, I have a timing function to estimate the time, and the code which reads the data with minimum calculations took 12 seconds of processing time, which is many, many magnitudes higher than it used to be (much less than even 1 second!), and of course, since the program is set to try and load the data every few seconds, the response can take progressively longer (since Javascript was sending requests before it can receive it, but I can work on realizing how to fix that part specifically).

I've been trying to lookup some solutions, I've saw the page at https://help.pythonanywhere.com/pages/MySiteIsSlow and I have tried to test some of the things and see what was the issue.

I am using MySql, and I have tested the database using the console and the response is as quick as it needs to be (basically instant to retrieve a few fields), and I couldn't find any of the clear reasons for the slowdown.

The only thing I can think off is increasing the web workers, but I am not sure if that could help and I don't want to increase it unless its the last option (I am still not convinced that a few sensors sending and one/two users opening data with a few seconds in between should need that many more web workers, but I could be wrong.).

Could lack of web workers slow down the operations themselves? It was my understanding that it could mean that the server might take longer to start processing the data, but I didn't think it also slowed down the processing itself, or does it?

Thank you for your time.

Zaid

Having too few workers will not affect the time that the code takes to run, it will mainly affect when the code is run. That is, if you're seeing that your requests take longer to start their processing, then it's a worker issue. If the time that the code in the view takes to run is longer, then it's not a worker issue.

I can see workers still being an issue, but maybe indirectly due to the run time.

What I am still confused about is why the runtime increased without any changes to the code. The only logical explanation I had was the database, but testing that alone seemed fine. Is it still possible that the issue is somewhere there? Or do you have any suggestions of where I can find the source of the delay?

A quick edit, the performance seems to be 3-4x better today randomly. Could be just the server being a bit off? it is still significantly slower than usual, but at this point its "ok". But I still am trying to figure out the source of the issue. Another interesting thing is that as the same exact call with the same exact database call keeps happening, even without overlap, it seems to be getting longer and longer slowly over time which... is very weird.

If you were able to detect that the slowdown is happening on the db connection with the same db and the same db state, a possible cause of the slowdown could be the fact that your db is on a shared server, so actions of some users may affect temporarily the performance for other users.

I cannot say for certain its the db connection, but its the only explanation I can think of at the moment. As the db itself is running normally from the console with normal speed. But simple operations that use the django models to get some data are taking longer than usual.

Have you added logging to confirm that the database connection is what is taking the time?

I could see the top-level function that's getting the delay being purely database operations. I will try to do further troubleshooting later but I am trying to figure out where I need to start looking first.

One thing to look out for is whether or not your tables have indexes on columns that you're running queries on; if they don't, then you could get wildly different behaviour dependent on the number of rows in the queried tables, whether MySQL currently has them cached in RAM, and other factors.

One test you could do -- start a MySQL console, and a separate Bash console where you can run a Django manage.py shell. In the shell, you can do different queries from the command line, and see if they're running slowly. If they are, running "show processlist" in the MySQL console at the same time will show you what queries are running -- that may help you work out what is taking up time.

(Obviously that only works when queries are running slowly enough that you can switch between consoles while they are running, but if you've got some that are taking 12 seconds, that should be possible.)