WEEK 9

(July 23, 2025)

Attendees:

Lastly I was tasked told to introduce a pickle file into our pipeline system using the available data in the current safaa repository.
This week I started by rewriting how some things like dependency installation are injected into our pipeline
Below are the summary of things i re-wrote for our pipeline;
- Declared global variable path for static path in the repo
- Splitted all pipeline steps into different independent steps for better tracking of processes
- Introduced the use of arguments to run each steps in the pipeline sequentially.
- Removed requirements.txt file and introduced direct dependency installation in the pipeline script
To introduce a pickle file into the pipeline, i introduced a training step into pipeline, by integrating the existing safaa agent training related script.
Also, this week I created a new branch named testing-train on my own end to test out changes we have made so far, and monitor/simulate how the training will run in the pipeline.
- I introduced a new argument named "--train" in the retraining steps for the training process.
- Routed our model to be saved into a new path so we can visualize pickle file if the training was successful and avoid different permissions error.
The image below shows our completed training process

This week, I had discussion with my mentors on what I have done so far regarding training and modifications I made.
We also had hands on session with update on our sql to ignore contents ignored on the server when fetching copyright content.
And lastly i was also told to introduce testing of our trained model in the pipeline