Best set up for scarping project
I was talking to a colleague about my scraping project and my desire to deploy it to aws and he suggested I use a raspberry pi. After looking into it, my understanding is that building a server using miniPCs. However, I am not well versed in the subject.
My goal is to have a kubernetes cluster, that hosts airflow which will schedule jobs (data scraping and processing pipelines) using the kubernetes operator. I also want to host a postgres database (maybe not within the kubernetes cluster) but on the server as well. I am not closed at the idea of having multiple miniPCs.
What would be a good setup to do this? Also do you have resources or tutorial on how to set up that?
One reason I want to go through kubernetes is to learn it too. Thanks