Database engineer to set up living cloud database and api for (near-)real time data delivery

Requirements - Communicative: This is a (partly) collaborative project. At the end of the project, I need to have a decent understanding of how everything works. Communication and documentation will thus be important. - Terraform, SQL and Python programming. - Google Cloud Platform skills: it seems like google is the best option to host the databases on so you would need to have a firm understanding of the google stack (preferably with gcp data engineering certification) - (Optional: React front-end experience) What I am trying to build (NOTE that this is the entire vision, not all of this must be delivered in this project. This project is just to set up the foundation to enable me to build on): I have been trying to build the infrastructure to do academic research on a large volume of economic time-series data. The research would be both nowcasting of economic health and analyses of historic data. So for this, we would need to have a database that is constantly kept up-to-date from different data-sources and that can serve data to researchers. Additionally, we want to be able to integrate the analytics and models that we develop into the API to serve up the results of the models in a timely manner. I am currently setting up this research stream with a very small team but we would already like to make it scalable, and affordable for the long-term. Eventually (NOT IN THIS PROJECT) I would like to add a front-end so that I can provide access to the data and our model-outputs to less technical researchers. This does not need to happen in this project but the foundations should be built with this in mind. High-level idea of the needed elements (not rigid on this and open to suggestions): - a BigQuery database that would hold the full archive of data - cloud-based jobs that constantly pull new data from the source APIs and push it to the BigQuery - a python-based API framework built to expose the data and model outputs through api end-points. - cloud-based jobs to run multiple models on the data in the background and generate and save their outputs to a table in the bigquery Work that has already been completed: - Data extraction scripts in Python for retrieval of the data from the target sources - scripts to clean and transform the raw API data to the format I want in the BigQuery - 800+gb of raw data retrieved in Parquet format - SQL database schema draft (in SQLAlchemy) - First draft of the API written in Fast API Work that needs to be completed: Phase 1 - Tell me your hobby so I know you read this ad completely. - Review SQL database schema to identify where improvements can be made to the tables for the envisioned use-cases - Creation of BigQuery table to store historic data - partitioned by day - Terraform (or some other language) file to manage infrastructure changes and service accounts / permissions - Github repositories to manage source code for each of - images that get built (e.g source code in src folder and docker file) - terraform file(s) - Containerise (docker) the data extraction scripts to be stored in good cloud container repository - Containerise (docker) the Fast API app to be stored again in GCP container repository - Deploy an instance of the Fast API app on cloud run - Deploy appropriately sized VMs to run the containers with the data extraction scripts Phase 2 - upon successful completion of Phase 1 - Add front-end to enable others to explore the data interactively (basic graphs and tables) As said in the beginning phase 2 is not necessary to be completed in this project but if you can help with this as well, or can involve someone who you can collaborate with to do so, the budget can reflect this. Send your resume and envisioned approach and I hope we can work together. (I plan on reaching out to plan the interviews the week of the 23th of May) Kind regards, Milan