Don't make the same mistake as me. Scala used to be the best language for big data and data engineering stuff. Spark api for scala were always a step ahead of the python api. But most projects have now realized that python is the better alternative for data engineering stuff so they are providing better support for python api of their libraries. Scala 2 to Scala 3 migration has also been a cluster fuck.
I was curious mostly for the sake of personal enrichment rather than projects. But it does sound ilke Python has become the target for a lot of these things