I am looking for an Elasticsearch Expert having many years industry work experience for a long-term consulting.
Our Elasticsearch System requires the following functionality,
1. Heavy search query from client side, i.e full-text search, keyword search, Regex search etc.
2. The search latency should not be more than 1 sec for full-text and keyword search, but for regex it can be a bit slow but not more than 3-4 sec.
3. We will ingest data to the server 24/7 i.e huge amount of write per second from many servers.
4. We need the search to be as real-time as possible.
5. The nature of data we want to ingest into Elasticsearch are of text-type. There will be files too, that would need to be indexed in some reliable/fast way for search ( maybe some external CDN for fetching )
6. The No. of Documents to be indexed in Elasticsearch is ever-growing, hence the architecture should be flexible. We are not going to delete the indexes i.e the data is historical e.g weather data for instance. An example be, we are expecting at least 20 Billions rows to be indexed in upcoming months.
The consultant will help with the following things,
1. scalable indexing strategy
2. scalable server architecture
3. server hardware capacity planning
4. indexing methods (data ingestion techniques)
5. Re-Indexing strategy
6. Data redundancy avoidance strategy
7. operational considerations
8. security considerations
9. maintenance strategy
The requirements might change as we start working with our system, so we need someone who has excellent work history in some corporate environment.