During the last week, we discussed baseline models and project architecture [TS-34]. We collected basic datasets for roles and tools [TS-35]. We wanted to collect a dataset of projects from the Capstone project organization, but the task turned out to be too difficult because it is hard to understand who knows what, who does what, and who is who from the participant table. It is easier to synthesize the data ourselves, but we took project names and descriptions from the Capstone project table and added people to them with roles and skills [TS-56].
This week, the ML team implemented a basic data migration to test the functionality of both the project and individual microservices [TS-57] [PR]. Core classes for interacting with the databases were also developed [TS-55][PR]. We encountered some problems here because there are essential fields which we didn’t have in previous dataset, so we got to lead to a single format. Airflow was integrated for periodic recommendation updates. Additionally, a dummy recommendation service was created [TS-84][PR], and baseline models were added in two parts: Tag-based, including BoW + Cosine similarity [TS-72] [PR], and Description-based, using the AllMini-v6 Transformer and Cosine similarity [TS-73] [PR]. Also we decided to use KeyDB instead of Redis because it works faster and uses the same clients for connection.
This week, the ML team researched which metrics are the most relevant for our project [TS-156]. We chose MAP@10 (easy to interpret and calculate), nDCG@10 (shows how good the ranking in the list is), and MRR@10 (shows when the first relevant item occurs). We also implemented a class for these metrics [TS-146-163]: https://github.com/IU-Capstone-Project-2025/team-sync/pull/51. However, we encountered some problems with automating it.
The first issue is that indexes can change, so we are not sure whether a certain project will still belong to the user. The second issue is that the current dataset will be deleted when we publish our project to production, so we can’t use it in this way right now. So we took 3 different users, and tried to manually predict if project is relevant or not. At the end of this manipulation we got the following results: (MAP): 0.7653439153439153, (NDCG): 0.8729514321708831, (MRR): 0.8333333333333334.
Nevertheless, the ML team slightly refactored the tag-based part of the recommender system [TS-111]. We also researched algorithms for it [TS-165]. We considered different models: neural networks are not suitable at this point since skills are addable and it is very expensive to retrain them every time; boosting algorithms could be used as predictors for what skill will be taken next given the current state, but they also require retraining, so they are still a bad option.
The best option for now is clustering + TF-IDF: we plan to treat clusters as documents. Our hypothesis is the following: every cluster corresponds to one role - e.g., the first cluster for ML engineers, the second for backend developers, etc.
We added new part of RecSys that depends on the roles of users and projects. This helps us to recommend more relevant projects to users. As an improvements: we still have to implement researched tag-based algorithm and we could somehow combine tag-based and role-based parts to find complex connections among them and improve recsys more. For description-based we could use another embedder: much powerful and which takes more tokens, but for now we can’t afford it because of poor system requirements.
During this week, we realized that we need to make our ML services lighter, since their dependencies took up around 30GB [TS-179] [PR]. To address this, we found an ONNX model in open repositories and removed heavy libraries such as sentence-transformers and PyTorch. This also solved another problem: previously, model loading relied on an ethernet connection and always took too long to download, but ONNX resolves this [TS-110].
Additionally, we isolated the embedder into a separate service, so we can use one embedder for different services: Recsys and the future search system [TS-193].
Moreover, we designed the first prototype of the search system [TS-164]. It consists of two algorithmic parts: BM25 and Semantic embeddings, with reciprocal rank fusion and a reranker. We are also planning to implement smart pagination for it. We have already created a Qdrant image, but we still need to develop an interface to communicate with it [TS-191] [PR].
On the other side, we create a new module in the recommendation system for role-based matching. Role-based model includes BoW + Cosine similarity [TS-206]. [PR] We have developed a recommendation algorithm for new users who have incomplete data (cold start problem) [TS-74], so we could improve the users experience on the next week. We also faced the problem of a lack of necessary methods in the database class, so we added them [TS-210].