Forums

threading problem ?

Hi, I've been working all day trying to fix what seemed to be a threading problem with my django application. As suggested in this post https://www.pythonanywhere.com/forums/topic/14320/#id_post_57733, I have installed Django-background-task and used Always-on-task to monitor tasks added to the database. Worked perfectly locally, but can't manage to make it work on PA... No task is added to the database and the error messages I get from the server.log doesn't help me (or at least I can't manage to make sense of them). Can someone help me to figure out what's the real issue is ?

--Server Log --

2021-01-18 12:01:40 Mon Jan 18 12:01:40 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /rdv/validate/ (ip 10.0.0.61) !!!
2021-01-18 12:01:40 Mon Jan 18 12:01:40 2021 - uwsgi_response_writev_headers_and_body_do(): Broken pipe [core/writer.c line 306] during POST /rdv/validate/ (10.0.0.61)

2021-01-18 18:22:58 *** uWSGI is running in multiple interpreter mode ***
2021-01-18 18:22:58 gracefully (RE)spawned uWSGI master process (pid: 1)
2021-01-18 18:22:58 spawned uWSGI worker 1 (pid: 2, cores: 1)
2021-01-18 18:22:58 spawned 2 offload threads for uWSGI worker 1
2021-01-18 18:22:58 spawned uWSGI worker 2 (pid: 5, cores: 1)
2021-01-18 18:22:58 metrics collector thread started
2021-01-18 18:22:58 spawned 2 offload threads for uWSGI worker 2
2021-01-18 18:25:34 announcing my loyalty to the Emperor...
2021-01-18 18:25:35 Mon Jan 18 18:25:35 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /static/images/logo_k.svg (ip 10.0.0.61) !!!
2021-01-18 18:25:35 Mon Jan 18 18:25:35 2021 - uwsgi_response_write_headers_do(): Broken pipe [core/writer.c line 248] during GET /static/images/logo_k.svg (10.0.0.61)
2021-01-18 18:25:35 announcing my loyalty to the Emperor...

[formatted by admin]

Can you show us the relevant bits of code? (We don't know what you are trying to do exactly and how you're trying to do it.)

Of course. Sorry long post to come.

From the front-end in javascript this function is called to send back informations to be processed when user clicks on "Register".

// send rdv data to back-end
const validateRdv = async ()=>{
    const rdvInfos = sessionStorage.getItem('slot-infos');
    // get csrf token
    const csrfToken = getCSRF();

    return await fetch('/rdv/validate/', {
        method: 'POST',
        mode: 'same-origin',
        headers: {
            'x-CSRFToken': csrfToken
        },
        body: rdvInfos
    });
};

Informations are supposed to be received on /rdv/validate route. I cut some part of the views.py code for clarity.

@require_POST
def rdv_validate_view(request):
    """Validate appointment with a 3 steps process."""

    # Step 0: convert data received from a bytes format to a python dictionary
    raw_data = request.body.decode()
    received_data = literal_eval(raw_data)

    # Step 1: prepare data
           [...]

        # select calendar
            [...]
        # select location
            [...]
    # Step 2: create payload
    rdv_infos = {
        'practitioner': practitioner,
        'patient': patient,
        'email': email,
        'start': {'dateTime': start},
        'end': {'dateTime': end},
        'description': "standard",
        'location': location,
        'calendar_id': calendar
    }

    # Step 3: modify calendars
        [...]

    try:
        task_create_appointment(workflow, rdv_infos, verbose_name='rdv '
                                                                  'creation')
    except Exception as e:
        return HttpResponse(status=500, reason=f"La création du nouveau rdv "
                                                   f"a échoué pour cette raison: "
                                                   f"{e}")
    return HttpResponse(status=200, reason=f'La tâche a été planifiée avec '
                                           f'succès.')

On step #3 this is where I create a task that is supposed to be added to the database using django-background-task.

The task function is as follow in tasks.py

@background(schedule=1)
def task_create_appointment(workflow, rdv_infos):
    """Create a task wrapper around appointment creation process """
    process = CalendarActions(workflow)
    try:
        result = process.validate_rdv(rdv_infos)
    except Exception as e:
        print(f'Erreur lors de la création du rdv {e}')

The CalendarAction class used is the one using threads to retrieve available slots from different calendar (see retrieve_available_slots method):

class CalendarActions(CalendarProcessHelpers):
    """Class containing necessary methods to validate rdv and retrieve
    available slots"""

    def __init__(self, workflow, location=None):
    self.workflow = workflow
    self.location = location
    self.calendars = CalendarAPI()

    self.possible_slots = None
        [...]


    def retrieve_available_slots(self):
        """Get avalaible slots depending on specified workflow"""

        if self.workflow == 'sh':

            with cf.ThreadPoolExecutor(max_workers=1) as manager:
                general_query = manager.submit(self.calendars.get_matching_slot,
                                         "CALENDAR_ID", query="shiatsu")
                free_pack = general_query.result(timeout=2)
                self.possible_slots = free_pack['items']
                return self.possible_slots

        if self.workflow == 'psy' and self.location is None:
            raise ValueError("A location neeed to be specified for 'psy' "
                             "workflow.")

        if self.workflow == 'psy':

            with cf.ThreadPoolExecutor(max_workers=4) as manager:
                # a 'pack' is the raw complete dictionary received from the API.
                general_query = manager.submit(self.calendars.get_matching_slot,
                                         "CALENDAR_ID", query="psy")

                free_pack = general_query.result(timeout=2)
                self.possible_slots = free_pack['items']
                return self.possible_slots

    def validate_rdv(self, rdv_infos):
        """Validate a rdv by creating a rdv in the appropriate
        calendar while deleting the corresponding available slot."""

        calendar_reference = 'CALENDAR_ID'
        validation_status = {}

        # Step 1: create rdv with specialist according to transmitted
        # information
        try:
            receipt = self.calendars.create_rdv(rdv_infos)
        except ValueError:
            raise
        else:
            validation_status.update({"creation": receipt})


        # Step 2: look for free slots in primary calendar corresponding to the
        # same datetime of reservation
        search_results = self.calendars.get_matching_slot(
            calendar_reference, from_datetime=rdv_infos['start']['dateTime'],
            to_datetime=rdv_infos['end']['dateTime'])

        slots_to_delete_id_aggregate = map(lambda x: x['id'],
                                           search_results['items'])

        slots_to_delete_id = list(slots_to_delete_id_aggregate)

        # Step 3: suppress corresponding slots
        for slot_id in slots_to_delete_id:
            confirmation = self.calendars.suppress_slot(slot_id,
                                                        calendar_reference)
            validation_status.update({"deleted": {slot_id: confirmation}})

        return validation_status

You know have a complete view of the process. I'm not even sure anymore if it's a threading problem or somehow miscommunication between front-end and back-end. When I test the appointment registration online in the application I see an error in the browser. Error I don't have while testing locally.

XHRPOSThttps://www.kiasma.fr/rdv/validate/
Request failed: NetworkError when attempting to fetch resource. rdv_pay.js:49:21
    <anonyme> https://www.kiasma.fr/static/js/rdv_pay.js:49
    (Asynchrone : promise callback)
    <anonyme> https://www.kiasma.fr/static/js/rdv_pay.js:48
    (Asynchrone : EventListener.handleEvent)
    <anonyme> https://www.kiasma.fr/static/js/rdv_pay.js:35

What's your opinion on what might cause the problem ?

Where is retrieve_available_slots called?

Maybe add some logging in different stages of the flow to see in the web app and always-on task logs what happens.