Mark Scarr, Senior Director Of Information Science At Atlassian

Mark Scarr is the Senior Director of Information Science at Atlassian, the place he heads up the Core Machine Learning Group. We talked to him about what the group is engaged on, Atlassian’s use instances, cloud migration, and what’s in retailer for the longer term.

Please briefly introduce your self and description your function at Atlassian

I lead the Core Machine Learning Group at Atlassian. It’s a cross-functional agile group, and we have now fairly a broad remit. We work with varied different groups throughout the group from advertising to development/product on numerous completely different initiatives. Only for just a little little bit of further context, stepping again, I’ve been at Atlassian about 4 years. And I joined and have grown the group from a tiny seed. Previous to Atlassian, I labored for a few years nonetheless within the B2B house at Field. After which previous to that I labored at PayPal again even additional, Yahoo. I’ve been within the tech house for numerous years on the B2C facet but additionally on the B2B facet. So I’ve form of seen a 360 diploma view of how issues have advanced in these completely different areas.

What number of fashions do you have got in manufacturing and the way massive is the ML group at Atlassian?

We’re a comparatively small ML group. And as I mentioned, we have now a really cross-functional remit. So a few of the tasks we work with span the entire gamut of the group throughout advertising and development. We have now labored on the ML facet with Trello, which is considered one of our merchandise, with Confluence, with Jira. By way of the sorts of fashions that we’ve been concerned with, advice engines are one use case. Propensity modeling after which beginning to discover extra just lately the generative AI house–it’s one thing that we contemplate is definite in Atlassian’s future in numerous completely different capacities.

What are Atlassian’s main machine studying use instances throughout the Jira household (Software program, Service Administration, and many others.), Confluence, Trello, and the remainder of your product portfolio?

We work very intently with our efficiency advertising group on constructing out a framework for harvesting key phrases for efficiency advertising and search key phrases and bidding optimization–we just lately filed some patents in that house as properly. That framework permits us to take a pool of key phrases after which increase it by in search of different key phrases that we predict can be appropriate and fall inside the similar bucket. So you possibly can apply plenty of attention-grabbing NLP-type approaches to discover that house, and clustering strategies to extend that pool. In the event you like, you need to use semantic similarity, and there’s a complete bunch of different methods underneath the NLP umbrella that we may additionally make use of. That’s one space inside advertising the place we proceed to do numerous work, and construct out that performance. That’s an ongoing undertaking and that’s form of the place I believe the last word imaginative and prescient can be to have the system absolutely dynamic and absolutely automated end-to-end.

We’ve additionally carried out numerous work round modeling the shopper lifetime worth. Buyer lifetime worth within the B2B house is much more nuanced than within the B2C house. There have been numerous modeling and information challenges in arising with an honest technique to really compute that, so we’ve carried out some attention-grabbing modeling work there. And that truly ties into the applying I used to be simply mentioning, as a result of clearly buyer lifetime worth–or a by-product of that–is definitely used as a part of the enter into making these bidding predictions for what the downstream worth can be of a specific buyer.

Are you able to speak just a little about how Atlassian’s migration to the cloud and the way that has impacted the enterprise and ML group particularly?

Cloud migration has been a recreation changer for our enterprise alternatives. Since we’re speaking about ML, the largest influence is absolutely the prevalence of extra information. In cloud-based purposes, you have got a a lot richer information set accessible to you, which lets you prepare fashions much more effectively and successfully in comparison with a hosted resolution. So the largest recreation changer actually from the cloud migration, from an ML perspective, is the extra richness of information that we have now obtainable for mannequin coaching. Cloud migration is ongoing, however we’re dedicated to cloud options throughout our product portfolio as a result of it’s advantageous from an ML and information richness perspective. One other key related subject with that’s clearly instrumentation. In the event you’re going to be counting on or having numerous this extra information obtainable to you, then you definitely actually need to ensure that your instrumentation is there to have the ability to seize all these new information factors. So, that’s one other space we’ve explored and been investing in as properly.

One of many advantages of cloud migration is the power to instrument issues rather a lot simpler, rather a lot quicker, in a cookie cutter vogue throughout all various kinds of use instances. Are you making use of enterprise metrics or is that one thing that’s nonetheless just a little bit down the road?

No, it’s important that you would be able to really measure the elevate or the change in these enterprise metrics. So the enterprise metric could or might not be your goal operate relying on the issue. But it surely’s key that you would be able to really measure, you possibly can really show a change in these metrics. So we work very intently with that and clearly with the experimentation group, if we’re gonna run an experiment, these are the metrics we need to transfer the needle on, quite than our customary, you already know, your inside mannequin metrics, whether or not you’re taking a look at, you already know, MRR or precision/recall X or regardless of the metric occurs to be. Internally, clearly we monitor these, nevertheless it’s actually the exterior enterprise metrics, which is why having the enterprise stakeholder as a part of that partnership. And the analytics individual as properly, as a result of they’ve a deal with on that, and sometimes can assist set up what that baseline is that we need to transfer the needle from. So I believe that’s essential as properly.

My tutorial background is statistics, so I’m a giant fan of a gentleman by the title of William Edwards Deming–He was a statistician and he coined the Plan, Do, Examine, Act cycle. Which has many, many variants on the market. And it’s very a lot this sort of cyclical, iterative method to constructing and creating fashions the place you possibly can fail quick and transfer on. I discover {that a} very illuminating technique to body the engagement and form of what that course of seems to be like. It’s been adopted by, in varied guises over time, in numerous completely different disciplines, however I discover that very helpful as a framework.

How do you collaborate with enterprise and product leads and tie mannequin metrics to enterprise outcomes?

We undertake a really versatile engagement mannequin that varies from undertaking to undertaking, relying on the group we’re working with. However sometimes we companion very intently with our analytics compatriots, and any undertaking we work on is pushed by a enterprise stakeholder. So if there isn’t a enterprise stakeholder concerned, then we wouldn’t work on it. We’re not within the enterprise of constructing one thing we predict is “cool” after which purchasing it round–that’s a recipe for failure. I’ve been by that and the ML scientists working in that house are tremendous excited as a result of they’re constructing one thing cool, however the enterprise of us are like, really we requested X and also you’ve answered Y. So it’s a lose all spherical. In the event you flip it round and construct one thing the enterprise stakeholders need, they’re completely satisfied that they’re getting an answer to their enterprise, the ML scientists are completely satisfied that they’re making an influence, so it’s a win-win. So we positively undertake that method.

It takes a village because the previous saying goes to construct, prepare and develop a mannequin, and put it into manufacturing, so we ensure that all of these completely different events are within the loop and they also every convey one thing to the desk. The analysts are normally the specialists in that individual space and know the metrics and the information. Clearly the enterprise stakeholder retains everyone trustworthy, so be certain we’re answering the appropriate query. After which the ML scientists convey within the modeling experience. After which, you already know, the engineering design and all the opposite items play their roles as properly. However sometimes that’s how we take into consideration engagement, and it might be roughly embedded relying on the undertaking. It’s fairly versatile and fluid in that sense.

How do you view the evolving MLOps and ML infrastructure house?

It’s a really wealthy dynamic house. There are numerous firms working in that space, producing plenty of instruments and software program for post-production mannequin monitoring, bias, and have drift, so there are many instruments on the market to assist with that. However what’s much less clear to me is how that tooling suits in organizationally. And so I believe that’s the place the MLOps group is available in as form of analogous to DevOps, the place they’ll really personal the fashions post-production, and be liable for the monitoring and the refinement of deployment–if there’s further fashions that you simply need to deploy as soon as that mannequin’s in manufacturing.

So I believe they’re positively what I’d contemplate as a separate group as a result of it’s nearly a separate skillset to your customary engineering groups or your customary platform groups who would work with them. I believe that’s one thing that as you scale, you positively need to have devoted sources. I see an ML scientist as very completely different from an ML engineer who’s very completely different from an information engineer who’s very completely different from an ML platform engineer and so forth and so forth. These are all fairly distinct disciplines inside the house and there’s clearly overlap, however I believe a few of these disciplines not solely require completely different expertise, however various kinds of individuals. There are particular individuals who gravitate extra in the direction of the R&D elements of ML science, and individuals who gravitate to the engineering facet of issues. I believe it’s good for us to acknowledge that these are these variations and it’s not a one dimension suits all.

Is there something your group is engaged on that you simply’re enthusiastic about? Something within the generative house you’re beginning to discover?

We’ve been engaged on some advice engine purposes for our Atlassian apps and one of many aspirations there’s to really construct it out right into a full ecosystem. That’s an space we’ve been creating and constructing out a advice engine in that house. After which within the product house we’ve additionally carried out some work with the Confluence group on suggestions. Within the generative AI house you possibly can think about with Confluence, there’s plenty of concepts round NLP performance like auto completion, textual content summarization are simply two that come to thoughts. Within the Jira house, we’re excited about routinely producing tickets for instance. These are some concepts that we’re exploring amongst others.

And in order that brings us into the specifics round generative AI. There are purposes inside Atlassian we’ve recognized the place we’re taking a look at producing textual content, video, picture, and voice as properly. So, possibly quite than interacting by a keyboard, you really converse to Confluence or Jira. I’m positively enthusiastic about this house.

And I believe from an organization like Atlassian, these publicly obtainable APIs and providers aren’t essentially a recreation changer or give us a aggressive benefit in and of themselves as a result of they’re publicly obtainable. You may get a short-term first motion benefit, however you’re not going to get the long-term aggressive benefit crucial. However I believe the place firms like Atlassian actually get the benefit is in pre-trained LLMs coupled with what I’d name a secret sauce. It’s going to incorporate clearly the richness of information in your area particular space and what you do with that information, together with the enterprise data, characteristic engineering that you simply apply on high of that information, and any augmented fashions that you simply prepare on high of the LLMs. I believe that’s the place the aggressive benefit is, and we’ve seen just a little little bit of that in a few of the exploration work that we’ve carried out.

Are you able to speak just a little bit about Atlassian’s current releases?

The massive information was the announcement at Group ‘23 of Atlassian Intelligence on the corporate’s dedication to infusing AI performance into our portfolio of merchandise. That takes many various kinds, and I don’t essentially need to go into any nice element, however definitely all of the press releases round Group ‘23 have extra details about that. However numerous it really focuses round generative AI purposes and leveraging LLMs to make the tip consumer’s life rather a lot rather a lot higher. It’s very thrilling that we’ve made that dedication to AI.

What are you most enthusiastic about over the following 3 to six months both in Group ‘23 or in simply the generative house?

What I discover most enjoyable is the eagerness and the engagement round LLMs, and the truth that there’s been a extremely unprecedented degree of democratization round ML, which I believe is general unbelievable. There’s a darker facet to that too, with well-documented shortcomings of LLMs. Hallucination is one instance, and we may go on and possibly spend an hour speaking about different issues as properly, and the restrictions. However I believe simply specializing in the optimistic and what’s thrilling is the adoption and the truth that there are all these purposes on the market and we’re beginning to discover them, and embody them in our product portfolio.

What’s attention-grabbing from a machine studying perspective is how can we begin excited about measuring the efficacy of those fashions? If I’ve two or three completely different fashions that I need to strive for a special software, how do I determine which is the perfect? There are some standardized frameworks on the market for the time being. The CRFM of us at Stanford have produced a framework known as HELM and Open AI has the Evals framework, for instance.These are very a lot customary generic frameworks for mannequin comparability. However if you get right down to your particular person purposes, like if i’m offering, say, some textual content summarization capabilities in Confluence. How do I do know mannequin A is healthier than mannequin B? How do I measure that? And so I believe that turns into a way more attention-grabbing drawback from an information science perspective. There’s plenty of inside dialogue round that, and I believe that’s the place science can add worth, nevertheless it requires much more thought, and it’s depending on the use case. There are many other ways we will method that however arising with a framework to permit us to try this goes to be important to measuring the efficacy of those fashions.

How do you see office interactions with generative AI shifting?

Work will evolve, and so will how we work and the way we work together. If you consider how we work together with our computer systems by a keyboard–it’s very inefficient. And even how the keyboard is laid out is well-known to be an inefficient keyboard structure. So I believe there’s room for enchancment. And as soon as we form of get extra comfy with these digital assistants–which can take many kinds–there’s going to be numerous privateness challenges there with avoiding information leakage. However I believe transferring past that, how we work together with machines will evolve. The logical subsequent step is speech to textual content. There’s loads of speech to textual content software program on the market already that works very properly, so that you’re already beginning to see a few of that. I’ve just lately been within the ER at my native hospital, and there was a complete wall of desks, and the medical doctors had been all sitting there with some form of recorder. They’re all talking into these gadgets, and it’s simply transcribing it straight on the display. And that’s a way more environment friendly method in a excessive strain atmosphere like an emergency room the place they’ll really get their ideas down into their experiences. And so you already know that’s most likely forward of the curve, however I can see you already know the pure evolution of labor trending in that course. After which there’s additionally the augmented actuality monitor the place I believe that there’s going to be some advances in that house as properly.

How do you see information scientists and machine studying engineers evolving with these adjustments?

The majority of the ML house remains to be unsupervised studying. In the event you had been to attract a pie chart or some form of graph, the chunk that’s associated to generative AI relative to those supervised studying items is far smaller. And I don’t see that altering within the quick to medium time period. There’s nonetheless going to be a necessity for what you may name these days conventional supervised fashions, and I believe that house will evolve. However you already know, anybody who’s ever constructed any form of supervised mannequin is aware of that we’re already at some extent the place the mannequin is commoditized. You construct your coaching information, set your pipelines, and also you mainly take an off the shelf mannequin, whether or not it’s a typical logistic or a linear regression mannequin, or perhaps a deep studying mannequin and plug it into your pipeline. However localized, specialised data remains to be important to that course of, and that’s one thing that I don’t see altering that dramatically. However essentially the most thrilling potential piece of the present LLMs is with the notion of plugins, or what LangChain calls brokers, the place you mainly have an LLM that acts like a conductor in an orchestra and farms out providers. Now, that’s the place it may get actually attention-grabbing, you may really assemble a complete ML pipeline by the pure language if you happen to had the appropriate items, and I’m certain persons are beginning to consider it from that perspective, however I haven’t seemed into it in any nice element.

You co-authored a paper known as “Measuring Productiveness and User Engagement from Workspace and Community Interactions.” How did you method finding out this and what are a few of the attention-grabbing takeaways?

That was an attention-grabbing undertaking that grew out of a few of the extra properly established, well-known social skilled community graphs. Firms like Linkedin have an expert community graph, and clearly Fb has their social graph, so the concept was that we may construct an identical community graph to mannequin office interactions or collaboration. In gentle of the truth that numerous software program these days is geared in the direction of of us who can collaborate digitally, how do you measure productiveness? And the concept was that if you happen to may measure productiveness for a given product, then you possibly can clearly connect worth to that. And if you happen to can present that there’s a altering productiveness or an enchancment because of that individual product, and clearly there’s worth to the tip consumer. In order that was the motivation behind the paper and measuring productiveness, particularly white collar productiveness is sort of arduous to do. So the concept right here was to make use of the collaboration graph method utilizing some form of community principle to mainly say, can we measure productiveness when it comes to how individuals really collaborate on tasks? The concept with this firm I labored for–this was previous to Atlassian–had a product just like Confluence known as Field Notes, and mainly individuals may produce content material after which share it with different individuals. After which people may edit it, touch upon it prefer it, and so forth, and so forth. So this creates a community of knowledge sharing proper? After which you possibly can create a community graph based mostly off of that, see how data disseminates over time, and the way it expands and contracts. Then you definitely apply classical community principle metrics and strategies, and search for communities and influences within the graph. That data can then be used to make strategic enterprise selections and supply perception again to how persons are really utilizing the product. It’s very helpful for focusing on from a advertising perspective.

What are some ideas you’d give to ML leaders and even anybody getting began within the house?

I believe that most likely the important thing one is flexibility–the power to be versatile with how you consider issues. With this inflow of generative AI fashions, how can we take into consideration them in a optimistic gentle? With the ability to say: Okay, generative AI is empowering for these causes, and now I have to focus extra on some issues I wouldn’t have earlier than. That might be the one factor I’d say is that this notion of getting a really versatile thought course of and mindset, and with the ability to adapt. As a result of it’s by no means going to return to the way it was earlier than. And I believe it’s about acknowledging that, and being comfy with that, and with the ability to adapt and embrace this new know-how. In the end, I believe that’s one of many the reason why we’re in house within the first place, and it’s such a privilege to be on the forefront of it. It’s a good time for machine studying and AI, as a result of we’ve opened up a complete new dimension right here. It’s going to be thrilling over the following 6 to 12 months to see the way it evolves.

Are you hiring proper now?

There are at all times alternatives for the appropriate candidates. Anybody with a sturdy background inside completely different realms of machine studying are at all times in demand. So, we’re at all times open to these sorts of candidates.