Manage your and your employer’s expectation as a future data scientist
If you are following your passion and/or chasing your dream to become a data scientist, I want to congratulate you first since that is awesome. Data scientist is often a well-paid position so it means one thing less to worry in your life. And it is also a meaningful career in every single fields. More importantly, you can realise your dream and work around the clock with passion. At least I thought about it this way.
But the reality unveils true herself without asking your preference of her appearance. Here is one thing or two I learnt as a data scientist in a freshly established data team in the last 3 years. Hope this can help you in a way to manage expectations and prevent you from getting surprised.
The Gap
In the last 5 years, data scientist has been one of the hottest professions all over the world. It makes perfect sense in a world of digital transformation. Many talents are fascinated by the capabilities of commercialised technologies like machine learning, and are paving their ways to become a data scientist through systematic educational programs. I was in the same party too. Such a heat wave helps the recruiting departments of many companies to meet the demand of data scientists in term of quantity. Ironically, the gap between data science and business has never been bridged easily with education.
Ok, I can already imagine there are question marks on your face, which I take as a sign of intelligence. Before I go on and answer these “imaginary” questions, lets be “real”. Without knowing any of the answers to the questions in the following, you can still be a capable data scientist after stepping into this field, fit in well with the working style of your company in time, and will figure out all of these by yourself in the end. Instead of judging, I will simply tell my experience and thoughts.
The Facts
Data science in business is more than just data science.
Though this is so obvious when I put it on this page, no one ever told me before I secured my first job as a data scientist in a consulting firm.
Nowadays, everyone with higher education is trained to be a researcher, specialist and expert, etc. For us, we are “meant” to be data scientists after the bachelor, master or Phd study in the field of data science. I deeply agreed since I never had doubt in my passion and capabilities.
Only after starting my data scientist career, I found out what I believed so deeply was only conditionally true. Why is that? I think it is because that data science educations are driven by knowledge while real world businesses are chasing values. To put it simple for you, our future data scientists, the education and the business evaluate data science on different metrics.
In most companies, Data science knowledge is valuable, but not directly. The actions/responsibilities of a data scientist in business are defined most directly by business values, while the data scientists we are train as is defined by knowledge. I think it is a solid assumption that you can find the condition under which the definition of a scholar data scientist is the closest to that of a commercial data scientist.
With that being said, why it matters after all if every data scientist can figure this out after a while in their career as a data scientist? Please hear me out next.
The failure of realising the existence of such a gap triggers my deep confusion and doubts about my decision and future as a data scientist. From all the sources, I know, this is not uncommon.
Since I didn’t know there can be such difference between the expectation from my data science education and from data science business, I felt confused and hopeless after my data science project proposals are rejected again and again in the first half year. I learnt and persisted. As the return, about two years later at this moment, I know how to sell a data science project anf what to sell to my manager and different clients. But some colleagues and friends left their companies. They started new challenges somewhere else they think they can be a data scientist as expected. On average in my “sample”, they needed to hop in and out 3 times before they could really settle down.
So what is happening here? Confusion and self-doubt are definitely not healthy to us and to our career. As reactions, some people choose to avoid the cause of these negative psychological activities, which in this case is leaving the company. Dealing with the cause is another possible reaction for some others. I don’t think one way is better than the other as long as your expectation is met. But in either way, we have to understand the root cause of the pain. Proposals being rejected is just one of the complications. The gap is the root cause.
The gap is due to the mismatch of the key points in data science and business. The most crucial ones are feasibility and viability, policy and budgets, explainability and trust.
I don’t like failures. However i will never give up one single opportunity to learn lessons from them, especially free ones. I still remember what my manager always tell us: “The solution doesn’t have to be perfect. But it must be viable and fit in the budget”. The phrase will be different and key words probably be reorded else where. But the frequency won’t differ too much. That is the business world. Moreover, you have to make them understand you/your solution, which is not as easy as explaining a predictive model to your classmates. So instead telling “the add to cart behaviour of the visitors from these two groups follow the same distribution”, it is better to tell that “this promotion/marketing activity doesn’t drive more visitors to add products to cart more often”. I learnt this one sentence lesson in two years. It is expensive for me but free for you. Learn it and hopefully, you don’t have to go through any of this as a future data scientist.
The Actions
Manage your expectations from your employer and what will be expected from you by choosing the right type of organisations wisely.
If you don’t want to waste one to two good years in your life as a data scientist like me, think before you leap.
Type of the organisation:
It shouldn’t surprise you that a Phd of data science in a research institution or big tech company is working on cutting edge algorithms or ML models. On the contrary, it is common that client-facing and pitching is the daily routine of a data scientist in a consulting firm.
Size of the data (science) team:
The size of your data team matters. You probably won’t have to working on proposals day in day out since many others contribute to business development as well. This means you can expect some qualitative time to work on what we are trained for.
Other aspects:
You should be well paid in your local labor markets. Although this is a personal thing, it also says how the business composition of your potential employer values data science work and probably how the business development for data science team will go in the coming years, which definitely should draw your attention if your want to grow and develop yourself fast.
The End
Read my story. Find out what suits you best and make your decision towards it. Hope your story is different than mine. If you take anything away from my story, I will be honoured to be part of your story.
Anyway, good luck to you all as future data scientists!
