Small Scale NLP Pretraining Experiment

I'm seeking a freelancer to assist with the implementation of a small scale natural language processing (NLP) pretraining experiment. We want to experiment with pretraining SLMs on clustered data. Required skills: - Transformers and language models - MOE architecture, including upcycling from dense models - SLM pretraining, using commmon frameworks - large text dataset clustering - Huggingface Your task will be to guide and implement the chosen experiment professionally and effectively. I look forward to your bids.