This episode is brought to you by Maplytics by Inogic. Today we are joined by return guest Sean McNellis, Principal PFE with Microsoft. 

“When those service protection throttles went in, we saw a significant increase in service availability and decrease in service outage calls.”

Sean McNellis, CRM Audio

Topics in this episode:

  • Changes to the pfe PowerShell libraries and .Net Core
  • Service protection throttles and the reason for them
  • Tips for building future friendly code and limiting API calls
  • Covid-19 driving digital transformation and decoupling things that have always been combined and driving low-code for business agility.
  • Changes in the maker experience

Watch on Youtube:

Links:

Snap Camera https://snapcamera.snapchat.com/

CRM Audio with Sean McNellis transcript powered by Sonix—easily convert your audio to text with Sonix.

CRM Audio with Sean McNellis was automatically transcribed by Sonix with the latest audio-to-text algorithms. This transcript may contain errors. Sonix is the best audio automated transcription service in 2020. Our automated transcription algorithms works with many of the popular audio file formats.

Joel Lindstrom:
Welcome to Power Platform, Quarantine audio and I don’t I don’t remember where I am or what day it is anymore, but I’m glad to be here. And joining me is George Doubinski. Shawn Tabor and another Sean, Sean McNellis from Microsoft PFE extraordinaire. I didn’t know. That’s your real backyard, isn’t it?

Sean McNellis:
Yeah, it is beautiful, isn’t it? Minneapolis. Pretty good for Minneapolis, Minnesota. Exactly. It’s pretty. Where? Yeah. Testing out some of the new preview features for the team’s Team. How about that?

Joel Lindstrom:
Nice and Shawn Tabor, that’s that’s not a fake background there.

Shawn Tabor:
That’s you know, this is my my resplendent nerd nerd cave. Oh, my toys.

Joel Lindstrom:
All right. So the big story that everybody is living through right now is the Covid 19 shut down worldwide. So, McNellis, how is this impacting what you’re doing from the PFE and the broader Microsoft team? I know there’s a lot of things going on there.

Sean McNellis:
Yeah. Absolutely. So it’s been really interesting as a PFE in the PFE role, generally speaking, is, you know, generally speaking, we have most people either entirely remote that go on site periodically or we do have a certain number of PFE fees that go on site every single day like it’s their their day job with the customer. That’s where they work from. For them, there’s a big change for those who travel on a weekly basis. Huge change, a lot less, miles. But a lot less pollution than I would argue, a whole ton of extra productivity from a company perspective. It’s really interesting. People who work from our our offices in Redmond now are working at home. So the the first two weeks were very novel. You know, like, hey, I’m at home. Isn’t this cool? And so people who are used to working in the field, it’s like, OK, yeah, that’s neat. That’s cool. So I think there’s an interesting amount of maybe empathy that comes out of this. It both ways where people can say, hey, I know what it’s like to work from home every day of the week or, you know, you’re going. My favorite observation is I just go from call to call to call. It’s crazy. I don’t know. I can’t even play table tennis. So, you know, that’s something that I’ve never had. Well, I’ve had that experience kind of for a few years in Fargo. But anyway, so I think it’s a good level setting of everybody. And for customers, too, there’s a lot of customers that default to have an office. I have a desk. We have a contract. We have a PFG. Come on in. Every day or every week. This is really a forced scenario. And so it’s been a little hard at times, but I think a lot of customers have really rolled with it. And it’s sort of jumbled up the typical maybe I don’t like how this works or I don’t want to join a web conference. Everybody has to do it. And so now we’re all used to it. And so I think it will drive some interesting new behaviors as we we do our service delivery in the future.

George Doubinski:
It’s interesting that I wanted to ask about the perception of people working remotely, because one of the things I noticed that one of the push backs against working remotely always be from bosses kind of. I used the term generously is pushback, saying, oh, people would mow the lawn and then play with the kids and do some of this stuff. And I don’t have control over that. And the productivity will drop. And now what we see is that generally people are responsible adults and when they paid money for doing their work, they actually tend to do their work. The other aspect obviously working from home. And I know that because I’ve been working from kind of remotely, not not from home, but as far as customers. I’ve been working remotely for the past few years is that you tend to when you work from home. One of the dangers is you actually work longer, much longer house than you would do otherwise. You don’t have a clear distinction. Like right now we’re sharing the office with my wife and she would stand up at 6 p.m. and say, okay, I’m leaving the office.

Sean McNellis:
It’s very true. I moved to a home office and working remotely for almost 10. It’ll be 10 years ago in August, completely remote. My wife and I and it was a big it was a big change. At first, I think I was better at first at breaking away and like, this is my lunch. This is my time. Go on a walk. You know, it is an interesting you know, there’s pluses and minuses on both sides. So I’m not super great at it. You know, hey, we’re not much going on. You get stacked up with stuff. And next thing you know, it’s 6:30-7:00. You take a quick break for dinner. And, you know, now it’s I get my son on the virtual classroom for Boy Scouts and then I turn around and do conference calls because there’s another conference call going on talking about whatever. So let’s jump in. And everybody kind of knows that. Well, frankly, nobody has a life right now. The secret, I didn’t have one before. So this isn’t much of a change. But anyway, it yeah, it’s really interesting. There’ll be as you know, there will be books written for decades on this period of time. And I’m sure people are collecting call data and collaboration data and we’ll be looking at that I think for really good reasons on what really works, what didn’t work and what can we do that makes sense this way. I think it sort of underscores the importance of physical connections. Hopefully there’s this. Weird mental component, biological component, we’re being in the same space with others as absolutely as you guys know, completely different. You your brain and they’ve proven it. You know, they put people under scanners and they watch how brains function differently. And I think this underscores the importance of those things. Where is maybe before it’s like we had video chat. People use that, right? I don’t use it.

George Doubinski:
You could I think if you just feel guilty about wanting to strangle someone close to you as opposed to your co-worker.

Sean McNellis:
Yeah, that could be, too. Yeah.

Joel Lindstrom:
George, you had some things around this topic that were more developer. And, you know, we’re not we’re not. This isn’t a developer episode, but you now have your developer moment here to ask you developer questions about Github.

George Doubinski:
We know that. Well, don’t know that Dot Net Core is common then. Dotnet 5 is common. And so we’ve been poking around with Dot Net core alpha release, sdk libraries. But you you own kind of on GitHub set of tools. PFE Library and the like PowerShell library with Ken. So I just wonder, like, did you have a chance to play with it? And this morning I read about oData and .net 5, which is kind of a unifying theme coming up which he is currently alpha I think. Anyway, what’s what’s your perspective? You have a chance in this lock down to play with it. And what’s the future for the libraries and how it’s going to change things?

Sean McNellis:
Yes. So a little bit of a chance. I have a coworker, Bob Guidinger her who’s also on Twitter and has actually just released a ALM tool for Power BI that we’re using on a project we needed to think.

Joel Lindstrom:
George has a Bob Guidinger shirt, don’t you?

George Doubinski:
I’m a big fan. We pimped Bob for MVP for quite some time and then he turned around and swallowed the pill. So I didn’t know you worked with him.

Sean McNellis:
Yep. Yeah. No, we work together. We share a customer together. So we’re working on a health and human service project. And so we’ve done some experimenting with things like mini custom portals. And he’s spent way more time at the controls of the Alpha SDK than I have. I speny a little bit time looking at it with Matt in Bellevue prior to the whole lockdown, but Bob spent some actual time coding on it, which is pretty cool. One of the really interesting things is you can put it into containers like a docker container, you can run it under Linux app service and ship these things completely. And it is really, really cool. And for things like you need antivirus, Clam AV has a docker container, can run it. And before it would be like, well, how do we broker the connection between here and here in the vnet? And now it’s two container or a container. You put it, you package it up, you deploy it. The two things can talk together and it really opens things up. So I know he’s really excited about it. I’m speaking for myself. I’m excited about it. I think having it capable of running on any platform is really cool, obviously. And I think that unification piece of having all the libraries in there so people aren’t going, wow, there’s this one thing. There’s two big things that are that are dealbreakers. And I can’t do core, now we can with this library. Now you’re talking V2 functions in Azure and relieved from a developer perspective, there’s very little change from my point of view. At least it’s still code. It looks the same. It builds on.

George Doubinski:
Tell it to Colin, who’s been busy writing the dependency injections left and right and encapsulated connections and things like that. Colin Vermander. He’s been doing some open source development on .net core. As far as CDS and things like that.

Sean McNellis:
So it’s really cool when it when it’s more available and we kind of get the namespaces ironed out and some of the details, finer details ironed out. Matt Barbour has been just I think putting a lot of him in the team more than him, I’m sure are putting a lot of work into it. And so it’s pretty cool as far as, you know, we’ll kind of wait to see what happens with PowerShell. I would not be surprised if we start working towards a first party option there. If not, we’ll look to convert it over and make it core capable. I mean, it’s used a lot. I think I just saw the other day it’s pretty close to 300000 installations from PowerShell Gallery. So it’s used, I think a lot. Pipelines. I know I use it for some of my projects and we’re converting over to the CDS build actions. But anyway, it’s yeah, it’s really cool to see us go in there finally and getting to that direction and seeing the authentication libraries mature a little bit more into MSA, all where hopefully things kind of get vetted out a little bit better. It’s a smoother process to get everybody on board and under one authentication library. So yeah, it it’s.

George Doubinski:
Last question, I suppose, with your PFE library. Are you looking for the moment where you can just hit compile and it just works? Or are you actually rewriting parts of the library to take advantage of perhaps different things in the core for the core library?

Sean McNellis:
Yeah. The parallelism stuff. Yeah. You know. Good question. I don’t know. But I think back to that, when we architected it myself and a gentleman by the name Austin Jones and Austin’s in the product team now working on some really cool projects as well. And it was really around simplifying the connections, simplifying retrieve,abstracting away the stuff that is not super complex, but just there’s a lot of moving pieces. So it’s easier for people just to say, I’ve got one of these created. Now I want to have them create them or two million or whatever. I think right now I moved over to service client instead of org service. So there’s a v9 branch on github now with a slightly different naming because Microsoft owns the Microsoft namespace and I’m not the company. So I think it’s I can’t remember the name offhand, but we’ll have to see how people are continuing to use it or if there’s still a need for it. And then we convert it over, you know. And as far as I know, one of the things we want to talk about a little bit today was throttles. One of the things you get with service client is there is some at least really rudimentary basic throttle handling within the service client. So it has the ability to say, oh, that was an exception because the throttling you had an API limit, service protection limit. I’m gonna go ahead and pause on that operation for I think the default is five or ten seconds and then I’ll retry up to five times 10 seconds apart. And these are policies you can change. So that was one of the big reasons I wanted to get that working in the core library, because you’re always brushing up against the throttles, if not, you know, slamming into them headfirst. And I know we wanted to talk a little bit about throttles. There are two different types of throttles going on. One is an entitlement throttle. Those aren’t fully wired in quite yet. And then there’s this idea of a service protection throttle. Protection for the when you get back, which says, hey, you’ve done more than any number of seconds within a minute.You get back some detail about, hey, you’re overloading the service. And so those are obviously really important. Fun fact about those. When those service protection throttles went in, we saw a significant increase in service availability and decrease in service outage calls. So there it was too easy for a select set of customers that were very busy to negatively impact a skilled group or a group of servers that that serve up our service. So service protection throttles, which a lot of services have by us putting that in. I know Charles and the ops team have seen a. Like huge improvement in how the service continues to be resilient. Stay on line.

Joel Lindstrom:
I think that’s an important message that a lot of customers don’t understand. They think of throttles as people being limited and they are. But there’s good reason for them. So what you just mentioned is if we had no throttles, the service would grind to a halt for everybody. Basically is people with you. So you have to have a reasonable amount and working with your team. You only had a customer that has to do a large migration in a short time window or something. They can be raised, right. You can you can temporarily raise those throttles.

Sean McNellis:
We have been able to manage through exceptions before. And I I believe I’m not speaking too far out of school to say that there’s a couple things to keep in mind. And actually this dovetails into one concept I want to make sure we talk about, which is if you’re writing an integration, if you’re using KingswaySoft, if you’re using the core library, using an application user is important. Eventually the goal is that we’ll have some sort of overcommit or, you know, this fuzzy area of, oh, hey, you’ve been you know, the way Azure does it a burstable workloads are burnable V M’s, maybe this concept of, you know, you’ve been good here or you have a very smooth pattern of usage. Now you’re gonna spike up for a few minutes. Let’s allow that.

Joel Lindstrom:
That’s a good transition point, because that is something that was announced in October, I believe it was, or September, actually. But it didn’t really sink in because it hasn’t been enforced yet. Now it’s starting to be enforced and people like FastTrack are bringing it up on projects. And we still don’t have really granular reporting to tell you how many API calls you’re using in a day. But it’s coming. And once that comes, the expectation is Microsoft is going to start sending people bills when they exceed that, exceed that limit in some way. That hasn’t been clearly communicated yet. But the limits as see if I get this right. Test me here. It’s twenty thousand per day per log in password user and then a pool of 100 thousand per day for all your application users is kind of a shared pool because you can create as many applications you usually want. So they have this one pool. All right. Right. So and again, I’ve looked and I’ve seen some big customers in at least working with support. We’ve figured out they’re within the limit. But there some of them that have, you know, like once a month, they had a million or more API calls then the rest of the month they’re under that. So I think that’s where some clarification is needed. You know, how is Microsoft monitoring that? That is going to be you go over by 10 percent one day every six months, you’re going to get a wammy or what?

Sean McNellis:
Ok, it’s a really good question. Again, I just want to underscore to two different types of throttles. One being service protection, one being entitlement entitlement, meaning tokens that you put in the machine or tokens you’re expiring when you’re doing it because I will tell you the difference between an entitlement like a resource limit and a throttle. It’s it makes a lot of sense that that’s confusing, right. Because they’ll both stop you potentially.

Joel Lindstrom:
That was a great sentence. It makes sense that it’s confusing. I think we have an episode title.

George Doubinski:
But steping just a little bit about the throttle limits, which has nothing to do with licensing and users nothing like that. Right. So one thing that didn’t stop to amaze me is people saying because it was free for all right. So people were abusing then abusing probably in some instances not even intentional just simply because lazy programming, so to speak. Right. So I could do it. So I don’t care about the quality of my code. I just need to get things on bum-bum done. And abusing basically bashing the service as hard as they could. And people when Microsoft started to introduce these limits, suddenly introducing this limits people complains, oh, Microsoft, you know, again for money grab and say, wait a minute. You tell me show me a single public service, right. That takes customers money. And his well-known service that doesn’t have those limits, like every single service I know has some kind of throttling going on.

Joel Lindstrom:
You can’t say I’m going to go to Salesforce’s because they’re their API limits are are higher or less expensive. Right.

George Doubinski:
So Salesforce had all the time. So I. Just amazed that people you take this free thing they took for granted, you take it away and say, hey, look, guys, we really need to maintain quality of service and people stop worrying. Now, I have to think when I write code damn.

Sean McNellis:
I see both sides of this one really well. I believe at least the part we’re missing. And Joel hit on it. I think we’ve all hit on it. The lack of the ability to have insight here can’t if you can’t measure it, you can’t prove it.

George Doubinski:
He’s right–every API call I write should return how close am I to the threshold?

Sean McNellis:
Well, and they do for the throttle for the service protection throttles. There’s a response in web API and if you hit for certain throttles, not every single one of them. On the entitlement throttle, though, actually for both not being able to measure it is killer. And so there’s a lot of work being done right now by the PPAC team, the admin center team to add reporting in. And it’s. Unfortunately, not super trivial. So it’s taking longer than than maybe we would want it to. But I mean, that’s a that’s a mantra from our customers. It’s a mantra from our internal field people. And I think they know people on our product group know that you need to you can’t measure it. You can’t improve it.

Joel Lindstrom:
I think the lesson for the team member license is if you don’t really give people clear direction on how to comply with a rule, you can’t blame them for not complying with it. Same thing like Fblow, for example, Power Automate. When we create a power automate flow using the c._d.’s connector, it should tell us this will use X number of API calls and then show you like if you put in a changeset then it will use this many per run or something like that. That’s the kind of kind of thing they’ll be really helpful because I think it’s this unknown quantity. Business likes dependability, especially these days when everything’s so undependable and having the perspective that I could without trying get a big bill that I haven’t budgeted for scares people, especially with the messaging around citizen developers spin up as many CDS environments as you want. Have individuals writing flows and power apps and things like that. Businesses want to be able to predict or come close to predicting what this is going to cost them and they don’t want to get a surprise, bill.

Sean McNellis:
Budgeting is a reality. Go on.

Shawn Tabor:
I had a customer who was really interested in using Power Automate and canvas apps and they had over twenty thousand team licenses. So the licensing impact caused us to deviate completely from our design and it just changed the plan altogether. And it wasn’t. It had an impact on a couple sides, not only on the you know, we had already started going down a path of explaining how these cool new features are going to help them in their business process, which they got excited about. And then we had a rip it away and say, okay, we’re going to do something different. That still met the objective, but not in the same cool way. Right.

Joel Lindstrom:
Part of that falls on sales and we get embarrassed the salespeople now. But OK, great. Now, I think we’ve all seen the case where either a partner, salesperson or a Microsoft salesperson at the end of the fiscal year over promises, something to get a deal and to meet their number. And, you know, I haven’t seen that happen a lot, but I have examples of where a sales person has pushed team members within the last six months. Yeah, I know that these enforcement were coming. So in that in that case, that’s a hard hurdle to get over. If somebody was told, for example, hey, you can do back in the Dynamics Portals days. Yeah. Yeah. You could do a portal for all your people and then they find out, no, we can’t have licenses. So. So. Yeah. So that’s that’s not I was a sales person so I can commiserate with salespeople, but I think it’s it’s the reality that it is coming. And I think the biggest point is just we need more information before we can do that. But with that in mind, we’ve talked about the throttles. Let’s throttle back and throttle conversation and talk a little bit what the API is. So what are the things you would say or a deployment? What are the biggest mistakes you see people make like developers or integrators or people configure or build apps that set them behind the eight ball, four for the throttles or the limits or what should people be doing differently than they generally are now?

Sean McNellis:
So yeah, and maybe even just beyond throttles and limits. But one of the big things so designing beginning with the end in mind as far as design goes with deployment, the integrating it into something like Azure dev ops, making a deployable, you know, one click, build and deploy seems like it’s easy to do but I don’t have time right now. It’s a little too hard on the hours. I can go into a story about directly related to some of the COVID stuff which I’m sure everybody is inundated with. But why that matters in terms of API throttling, I think we look at code the the big things to do to prepare for the future are to try and optimize what you’re connecting for and what you’re doing. So time and number of of API calls, that’s a lot of analysis can go into that. But kind of know what you’re getting into. Adequate logging is huge because I think you can tell pretty quickly and I think once we get our PPAC report for API calls, that will help a lot. You know, here’s the source along those lines. You know, app user, app user, app used to create an app, give it credentials. Better yet, use manage identity. If you’re using Azure, smash these concepts together and use an application user. So using modern authentication from a security perspective. It’s much better you can rotate passwords. That will help.

Joel Lindstrom:
Is there a point of minimal return? So you reach a certain point because if you have a shared API pool of 100 thousand per day and you’ve got busy integrations, then you’ve got flows using service principles. This all can add up, and if you’ve got this limited pool of one hundred thousand per day, you reach a point where it makes sense to start using some regular user accounts in there as well.

Sean McNellis:
So I kind of stick with them. We had some one of the guys is no longer at Microsoft who moved on to a different company, but his mantra around throttles were always throttles and like using multiple users and that sort of thing is if it doesn’t make sense, it doesn’t make sense. I don’t because not and not everything is fully implemented and we don’t have the reporting. That’s a really tough question to answer because I’m not a fan of the named user versus app user work around or multiple users a sort of bypass the throttle because eventually we sell a license and API calls out at a cost. The better and faster we can make the service, the higher density we can get, the better experience, the more people, the more scale, you know, margins come into place or probably more accurately, it reduces our costs to get those things out to customers and you get more for your dollar.

Joel Lindstrom:
Well, I want to verify another thing, just to make sure my understand is correct. My understanding is system actions don’t count against the API call, and that includes both the Athena export to Data Lake and the Data Export Service, is that correct?

Sean McNellis:
Correct. Yep. That’s my understanding.

Joel Lindstrom:
Is doing things, reporting or integrations that are read heavy. One thing I’ve started recommending is don’t read from the live database. That’s probably better for performance reasons anyway, but the export either to sequel does or were to the to link. And then read from there and then only connect to basically update or write data.

Sean McNellis:
Yeah. And that’s a really good pattern to use. I mean working on a project now where initially the the design was get an integration out to a data lake. You know if Athena lands in that particular environment then we can use Athena if not want to do something else, but get it into Data Lake and then puts an app. So we’ve been working a little bit with the Synapse Product Group on On-Demand queries as well as SQL pool. If you’re familiar with SQL Data Warehouse and Azure, that Synapse pools is just data warehouses rename. But the on-demand stuff is really cool. There should be some more features hopefully coming around that to make it easier. But doing your BI like high scale BI on, you know, very complicated aggregations or hopefully we get time series data. So append only immutable storage in Data Lake where at every time you update it lays a new version of the record down and then you can reduce it down for a time series. There’s some great BI capabilities that are sort of right under the surface that we’re scratching right now, but it helps with API calls. But if you’re looking to high-scale reporting using Athena into a data lake and figuring that out early is it helps with the API calls, but it will also help in terms of what you can do with Power BI or whatever reporting solution you want, be it snowflake or anything else.

Joel Lindstrom:
How Dual Write impact this? Now that that’s currently available for people that have F&O and what we used to call CE which we don’t know what to call it now. And how does Deuel right. If I had my data going near-real-time from F&O into CDS, is that count as a systme action or does that kind of gets my API elements?

Sean McNellis:
That is a fantastic question and I don’t know the answer to it because that I literally had not thought of that scenario. I thought I had most of the first party scenarios, but I that is one that I don’t know.

Joel Lindstrom:
So I imagine if hopefully it set up in a way because it’s more of a system platform function. Right. You have an F&O integration that’s using SSIS or something that if you can move that to there, but also virtual entities is another thing that are getting more powerful now as well. I can imagine that’s also good, a good way to limit the number of reads and writes directly to CPS as well.

Sean McNellis:
It could be. Yeah. Virtual entities. Yeah. I’m trying to think of a scenario where that would work real well, I mean, it’s scenario specific. And I you know, I’m not trying to speak for anybody on the COG side, cost good side or on Charles’s team. But I think what’s happening is we get the reporting up and we discover more and more scenarios. We are working with that team. Might the team I work on with the FastTrack team and others to try and make sure we’re surfacing as many common customer scenarios? At the end of the day, I think what’s really important to know is that over 95 percent of the customers that the throttles weren’t just randomly picked out of thin air. They did a lot of discovery on the majority. And I mean the large, not just your 80/20 majority like ninety five plus percent don’t even come work anywhere near. That’s great news. But I know all three of us on the phone call work with a lot of customers that are in that top 1 percent, if not the top, certainly the top five.

Joel Lindstrom:
I’ve heard that some of the people that are the biggest offenders are some of the smaller customers. I’ve heard stories of five user deployments with petabytes of data and things like that.

Sean McNellis:
I have I have heard similar things where there are a couple or where it was actually really kind of unclear. Was it a runaway situation that just left running? Yeah. There have been I have heard the same thing. I’ve heard there’s been a couple of those out there and I don’t think are like on the big customers. I don’t think anybody’s abusing. It’s you know, I think George made a really good point. You just don’t know. And then again, it comes back to reporting. But it also comes back to our API’s. Maybe there’s a more efficient way for us. And that’s what I think this will drive long term is putting some limits out there, will probably adjust those limits like we have the throttles, like we have other things. We’ll probably find that right balance where it makes sense. And then the other thing is too, if everybody has a scenario where I’m making this thing, this call, this update, or I’m downloading data from my instance and that takes a ton API calls and it’s bad for you. Microsoft is bad for me. It’s expensive. You know, that’s a really in my opinion, it comes down to a pretty easy feature to go. This is worth it. Maybe over the next semester we look at offering a new API call. For this, we offer a new service and we kind of use that as a mechanism to say this is this is worth it for us. It’s worth it for the customer. You know, that kind of discussion happens today already. Look, everybody’s doing it this way. Can’t be wrong. So there’s bad guidance. Did do we give new guidance? Do we offer a new API? Do we know? Which is awesome? You know, I think that people are shedding that thing of we said it this way six years ago. So that’s the way we have to do it. And now it’s. Wait a minute. Why is it really doing it this way? Should we? How do we address this in a way where we’re hopefully we can improve the service, improve what people are doing. Lower the API calls because that you know, and again, I mentioned it briefly before. But if it costs our company less to host it, you know, less resources, that means we get more density. We can give you more performance, better speed, more API calls for the same amount of money, hypothetically. Right. Or offer different features so that, you know, that isn’t just. This should happen, that those conversations do happen. And so this is just a thinking, the evolution of that. And it is really interesting to see now that Charles has spent. I mean, I don’t know what has been two years. Maybe a little more. And fundamentals, fundamentals, fundamentals. You guys heard it right. There’s been a ton of fundamentals work. And you know, I know I just emailed a product team guy the other night and said, I know I don’t say this enough, but thank you so much, because they are really doing a lot of great work. And nobody can do everything 100 percent perfect. But man is the the culture and the approach has changed significantly. You know, I showed a customer once the number of updates that go through in an environment. This is the type of updating we’re doing is like what? We didn’t even know that. You mean we’re getting an update at least once a week? Absolutely. And they were the customer that used to call me on every update.

Joel Lindstrom:
Look at the Solution manager and see all the patch solutions and all that. You don’t know. Just go in there. That’s amazing. But with along with the limits, there are three kind of big things kind of going down now or will be by the end of the year. Most of those been delayed because Microsoft decided that this isn’t the time to spring changes on people. But you talk about shutting off the classic UI, you talk about enforcing the team member licenses and the capacity storage change everybody is seeing. Those are a lot of kind of fundamental big changes the people are are seeing this year. How is what’s your perspective on that from the PFG side? What conversations are you having with customers? What are you hearing from them?

Sean McNellis:
Yeah. Change is hard. It’s hard.

Shawn Tabor:
We had we had a customer who hasn’t moved a unified interface yet. This was several months ago and we reminded them that it was coming in October at the time. And and they they said, oh, no, we got we have it scheduled. We have it scheduled when we’re moving. And they said, OK. When when are you looking? It’s October. It’s getting a little close.

George Doubinski:
But I was I was wondering the one thing I wanted to to ask about all this changes and kind of about all these throughout limits they affect to a certain degree uptime. Right. And I remember a few years back, we’ve been in the war room where all the uptime is monitored. And I’m sure you’re a frequent visitor to that room. And I’ve been wondering in the current climate, how’s that going and how people actually provide support? First line of support because they do need these humongous monitor on the wall. Right. So how’s that work?

Sean McNellis:
So you mean especially with people working from home? Right. So a lot of that work has now gone to transition into the development teams, quite frankly. Meaning they are responsible for their own queues as calls come in. There is some some service operators that work just on, you know, having access to services and an elevated access. But the team has gone from when you guys saw the operations center. The team has changed significantly since then in terms of really adopting true to heart, the DevOps model. And, you know, there are some really good things with that and there are some negatives. But because of that, a lot of the devs are on call and they trade you know, they trade shifts. And a lot of times they’re working from home. So I have not noticed from my perspective or customers perspective a difference in their ability to respond or or handle this, which is just really thinking about it right now is pretty remarkable. In general, things have gone very smooth from that perspective. I haven’t noticed much of a difference. Pivoting a little bit on that, though. I mean, a couple of things I want to mention and we didn’t mention at the top, but Microsoft, for anybody that’s interested, if you just search Microsoft and COVID 19, there is a news article page that kind of lists Microsoft’s response and what what we are offering with gratuitous licensing or services and that sort of thing, depending on what kind of company you are, depending if you want to use Teams or not. There’s different entitlements that go for a certain number of months or what have you. And if you’re a larger customer working with your account team on, hey, what can you do for me? There are things like even if teams is completely free for you, maybe you want FastTrack assistance and you’ve got you got to get on, you know, let’s say a thousand users in classrooms on board of with Teams I work with. Other PFE’s in Texas, a couple of them. And they were literally onboarding thousands of people. Single day of the week. Before it was I mean, sometimes from nothing to teams because they didn’t have my kids. A school luckily had a very good distance learning program already in place and they just activated it. So we didn’t need a lot of the extra time, which is really cool. The other thing is though, too, I mentioned that with Bob and I working on this Health and Human Services customer, they have a clinic based business. You have to go in and see them to interact with them in these several states, in a couple of Indian nations and. I tell you what, we’ve been working with them nonstop the past four weeks now. I said three before, but it’s been four, I guess. And it has been amazing and a lot of work, right? Adopting hundreds of thousands of text messages to blast out to people to tell them new processes. And we have people that work in clinics that are now working from home or they go into the clinic and then they’re converting into drive up because you still you you can do things online. But taking that that maybe 20 minutes or 30 minutes or even an hour sometimes of a clinic filling out paperwork and stuff. A lot of that we’ve been able to last minute low turnaround to put it out digitally so someone can get call up the clinic. Hey, you know, can you help me out? They shoot a text over to that person. They can sign up. They can enter in their information, take a picture, sign on their phone. These are all kind of table stakes with most companies. But where that hasn’t been a need because you have this physical presence. It’s those types of businesses and companies where a lot is happening. It takes a lot of effort. But I want to work back into the ALM suggestion. When you said, what can you do today to set yourself up better? We’ve gone from a release maybe once every three to four months with this group. It’s a lot to move into, sometimes releasing twice in a week. And luckily, we’ve got some people involved that are really big on let’s make sure we’ve got ALM going. Let’s make sure we’re doing things repeatably all of that. All of those seeds that are planted are coming back and we’re able to harvest that now. And anyway, it’s been you know, it’s been really interesting because when you’re in a room of people that say or a virtual room of people that say we only release every three to four months because we need six weeks, UAT and blah, blah, blah, blah, blah. And you go, can we get weekly or daily releases? You know, and everybody puts on mute and laughs like, are you kidding me? That’s impossible. And all the sudden, if something like this happens and it’s like. We can’t let people in the clinic. It might be we don’t even know it might be a health hazard, but we need to help them. They have to eat or they have to. And it’s like, guess what, we’re doing daily releases, you know. And that can be a 24 hour day or a five hour day.

Joel Lindstrom:
We’ve seen on twitter or a bunch of times, what drives your digital transformation, your CEO or Covid? All right. Is this is pushing a lot of things. I’ve been wondering with call centers and customer service. There were a lot of customers that had on premise call centers and things like that. And I wonder, how are they all transitioning? I know we’ve done some work with that. And but imagine if you had had 100 people in a call center and now they can’t come in. But you need to keep taking calls. That’s an immediate need to change to a different approach.

Shawn Tabor:
Yeah. And if you don’t have an investment in soft phones and things like that, it’s it’s a tough transition. Tough transition.

Sean McNellis:
The same customer. They have a large call, larger, not huge, but largest call center large enough. They have access to the the system we’re hosting. But, you know, the phone bit was like there was a huge scramble. What are we going to do? We looked at teams. We looked at every I mean, everything is on the table. Right. That’s the other thing. The culture of the company being different, it’s like, OK, if Team’s not going to work because it’s not in the zone, what will? And as it turns out, they were able to work with their telephony provider. And I think we’ve got a solution for them. But I just I the acrobatics that are going in to make this to decouple things that have always been tightly coupled, not because they need to be, but just because it’s easier to leave it that way is really interesting. And I I often wonder, you know, like with GDPR it forced changes within Microsoft that we didn’t even know we could make in such short periods of time stuff. Tons of amazing like stuff that blow your mind in terms of, you know, this doesn’t you can’t delete out of that. It’s not possible. Well, guess what is it?

Joel Lindstrom:
Delete audit trails and change disabled user records.

Sean McNellis:
Yeah. Or think bigger services within Microsoft. Like telemetry services and stuff where it’s not because of negligence, it just was never designed for that. And what was interesting about that, anyway, getting away from those specific stories is that was done in an incredibly short period of time. And I think it reset the bar in terms of, OK, we can do this if we need to. And so when things like JEDI comes up, we could do this if we need to. You know, things like Covid those trials that we’ve been through have really pushed, you know, teams being there and being more de-coupled between the service layer so that we can scale, you know, tens of times higher than what we were, you know, scaling. And so it it’s just really unbelievable. But I wonder in the marketplace if this sets a new bar in terms of we need to get going quickly. And let me take you back to a power platform. We need to get going quickly before you go. Let’s do low code, no code, flow, blah, blah, blah. We’ll put this together and go, well, we don’t have to do that. I wonder if this maybe changes the attitude a little bit and you go, well, we can decouple away from this here and let’s do something quick. Let’s roll it out there, because we know that we have a more resilient business that’ll be incredibly interesting over the next six months, but even the next year, year and a half to see how this impacts customer thinking and what they’re willing to do, not because they’re pushed, but maybe it doesn’t matter as much anymore and maybe they’re less tied into it has to be this way because it’s always been this way. And, you know, this gives us a new thing to really think about. How do we be nimble and that could be standing up a new call center in a new location or that could be, you know, a lockdown like this or you know to me. It’s a terrible thing, but it hopefully one of the outcomes we get is a really good way to architecturally approach these things and talk to customers more honestly about what can happen and how to enable them, because this stuff does matter. Companies that had that VoIP platform, even if they weren’t using it. Turn it on. Bang. Now you’re using it. If you didn’t have that before, that could set you back three, four weeks. With contracts. So, yeah, and this is more of the same. I’m sure everybody who’s listen to this or you guys have been through this, but it’s fascinating for me to sit and think about.

Joel Lindstrom:
Well, it seems that to me is with all the colleges closed and high schools, kids is going home. They’re all using teams or Zoom or some kind of some kind of meeting like this. This is giving them exposure to this technology. So they’re going to be the people coming to the workforce next. And this is now part of their lives. I mean, my fifth grade son is getting on Zoom meetings every week and it’s like he would have never done that before for any reason. Yes. Interesting. So one thing that I always used to ask you or Shawn Dieken when I would see it, Convergence or whatever. I’d love to get your take on now. I would always ask you, what’s the top things that support is getting calls about. And it was always outlook client or Dynamics mobile. So I’m curious, how does that change? Like what what are what are you some of the top things that support tickets are logged about these days.

Joel Lindstrom:
You know, that’s a really good question. So being more decoupled from the direct phone support team, I don’t have a great sense of it. I probably should’ve looked into that. It’s been kind of a busy few weeks, but I don’t know, like the top call drivers per say. You know, I will say the server side sync. Outlook Add-In configuration always as a call driver. And, you know, the beauty is it’s not because the .Com add-in crashed didn’t load. Yeah, it’s it’s other stuff now like how do we can figure the right sync? You know, what’s the best way to roll this out? So it’s changed the conversation there a little bit.

Joel Lindstrom:
Or if you stop syncing a contact, then you want to sink it again. That’s always the tough one.

Sean McNellis:
Yeah, that. And then, you know, certain things with appointments and delegates and, you know. And I will say I can foresee that being a little bit more of a call driver and on to because we have an announced deprecation of the .com add-in. And now people are also moving to that actively where maybe they hadn’t had that before. In some cases. So I think there’s some more questions around that.

Joel Lindstrom:
Well, if if the if the if the power apps, people ever fully wake up that this capability is there and start using it, that would really blow that up, too. Because I think generally people I see with Power Appss are building. Having emails sent via the Outlook Connector or something haven’t really woken up that, hey, we got this email activity inside cds.

Sean McNellis:
Right. Yeah. So I think that’s one. The other one is I know for our team talking about UCI, the UCI scenarios covering the gaps if there are any or not, this is one of those valid. It might not do that one thing, but it’s still you still get the outcome as a different way to do it. There are still in my opinion, there’s still a couple of gaps there that all I’m sure will be out before October.

Joel Lindstrom:
I’m not seeing showstopper stuff with UCI anymore. I am seeing little quirkss where why don’t we have the view related records dropped down on this form, but we have it over here.

Sean McNellis:
Yeah. And the biggest headache is, in my opinion, just. These more complicated customers getting them moved over it. You know that it’s just a hurdle and there are sins of the past that have to be dealt with now. But, working with a very large customer, we’re going to UCI, we’re going before we’re setting our sights on a September, even with the delay, I think we can make it work. But now we’re moving from legacy to UCI and they were still using IE and there was a couple of services they had bought that required that and hadn’t been updated. And so it it’s like just going to UCI isn’t that big of a deal, but dealing with, you know, issues or getting these services updated. And it’s not necessarily a huge, huge, huge project in many cases.

Joel Lindstrom:
IE isn’t the browser that gets the bulk of the testing now too. So if you’re going to have an issue, it’s probably going to be an IE user, not a Chrome user.

Sean McNellis:
Yeah. So what we’re doing right now is we’re decoupling from IE in the legacy web client. And then as soon as we get that nailed down and we have people onboarded we’re rolling out edge, the new Eedge for everybody. Once that happens, then it’s UCI time and then you move from there just to get off IE.

Joel Lindstrom:
Are the bigger ones going to UCI sticking with one main miles driven app or people adopting the micro app strategy.

Sean McNellis:
You know, it’s kind of a mixed bag. The. I believe in the case of this one customer and the one who’s on IE and their minority, not a lot of customers are still on it. But that will probably go with the app strategy because it makes a lot of sense up. And as you folks know, in moving, if you’re moving horizontally across what you’re doing, having one larger app makes a lot more sense. If you have a X department and Y department, a Z department and they don’t really cross that much, then you get a lot more flexibility with multiple apps.

Joel Lindstrom:
If it is one main process users don’t like to switch between different apps. But could you could have one main app and I’ve seen people do help poor apps who are scenario specific apps, sometimes a canvas, sometimes model. But like if you have this, one thing you do every three months have a unique app for it. But if it’s sales and customer service, they cross-pollinate each other, then then one app still makes sense, I think.

Sean McNellis:
Similar. Yeah. I mean it’s it’s not much different than how you look at things like multiple forms versus single forms. It’s it’s it is different, but it’s the same type of thing where if you have all this crossover you kind of need things in one spot. If it has discrete boundaries then it makes more sense where where things are really interesting, I think is retail and front line workers. If you have like three different concepts and maybe they do all three, you might have like a landing page that they go to and you have or else to each app. I have seen that like imagine if you’re a gym or something and you might have personal training and you’ve got like scheduling and new memberships. It would make sense, I think from a business point of view to say we’ve got three discrete apps and when a person logs in, they see that home home page and they click on that, whether it’s our home page or it’s one that they do. But that way it’s like I’ve got I’m in personal training, boom. I’m in there, I’m doing person up. I need to sell a membership, but go home, go back here and I have a different experience. That kind of thing is pretty interesting and I think apps actually work pretty well that way.

Joel Lindstrom:
So are you recommending to people– this is some we’ve had a debate kind of internally about some people– Are you recommending people build on the hubs like sales hub or customer service hub or are they setting up custom moderated apps? The reason I ask that is the product teams are building the settings for those into the hubs now. So like the sales features to convert were documents to PDF or the SLA and queue settings. Those are in the settings of those hubs. And so there’s also they’ve added some functionality only works in the hub and doesn’t work in custom model driven apps. So I’m kind of torn because Old Joel would say you should create your own Model driven app because Microsoft is going to pass an update and screw your app up or something. But we don’t do that for field servers or marketing. So I’m kind of turning around to say, let’s use the hubs. I was curious what your thought was.

Sean McNellis:
If it’s pretty cut and dried, I go the hub route, a lot of the I mean, a lot of the customers I’ve been working with are really a mixed bag and mash up of various entities and the legacy app. They will go into a custom app initially. What I’ve seen some or heard some customers talking about is they’re going to bring forward their deployment, but then they’re going to stand up. Let’s say they’re very service oriented or it’s for a business group and then they want to roll it out for global sales. They’ll use the sales hub to roll in the new users, because there’s this really interesting thing and a lot of companies are finally well, they’re getting on board with that. I guess they have been on board in the past, but it doesn’t come out that way, which is let’s stay as out of the box as we can. And the UCI upgrade is a firm reminder of that. Now, we all know being around the product for so long like these, you had to do something. You can just say, well, we can’t do it, but people are more apt and businesses are more apt to say it sales Let’s just start with the base. What does that look like? And yet the smaller the customer or the less complicated lot of times ago go, just show me what the regular sales thing is. Can we tweak this? Can we tweak that? How well does it fit if it fits pretty darn good. And we can just layer a little bit on top of it seems like a slam dunk if it is totally different or they’re coming from legacy where it’s all mashed into one and there isn’t a good fit, to me it seems to make sense to to go through custom. And that’s the thing like on net new customers very I would think very differently for a brand new customer to the platform than I would someone who is upgrading. And that’s as you guys know, that’s kind of tough on the head, right. Because you’re constantly thinking in one mode and then something goes, can we do? You’re starting new. You just let’s go here first.

Joel Lindstrom:
You know, it’s a challenge for somebody who’s been doing it. They had this big monolithic app that they have to move away from. And then that through their team members over here now. So my biggest wish is in this in this area is the ability to “save as” a model or an app, because everything else, including canvas apps, including forms, including business process flows, including all the stuff you can take, what Microsoft gives you, save a copy, edit copy and then you’re safe. And we really need that. I think with apps to give me the ability to license sales can be the ability to save the sales job to make Joel’s super duper sales hub changed the way I want leave the original one kind of pristine as it is as a reference. And then and then good to go.

Sean McNellis:
That’s I mean that’s a really good idea. And it’s interesting how this bumps up awfully close to solutions versus apps. Right. And what is the difference? What you know, how does this work? And you the you know, there there is a lot of effort between now and this fall and probably this time next year of let’s make sure the experience of putting an app together is to get the best of the old thing, because there still are some really great things about the old app and solution editor that need to come forward. And they will let people worry less about reasons. I’m thinking more like the navigation piece. But you know, there’s get those components that are really making the old thing, which is not optimal, making it sticky. I don’t like that, but I like going to the old one better. Me specifically because of the navigation, not because I know what that to me just makes sense. Like here’s a bucket, expand the bucket and that now I don’t have to type what I’m looking for. So that’s you know, that’s something that’ll be coming forward. And then when you’re in make, you know, I want to make stuff if we’re gonna put all of it into the portal. So unification of things like, you know, is this form bound to an entity or not, you know, canvas versus model rather than starting off I know what I need to do when I open it up, let people start building it for prototyping and really setting down that base layer, figuring things out. That’s going to help remove some of those concerns that are not on purpose. But they are artificial ish right there. They’re there and they could be gotten rid of. And the team is working really hard to get rid of those things and those boundaries. So, yeah, it’ll be I’m really interested to see all that work and effort in the fundamentals category. Now that those things are really stitched up much nicer and there’s still work to be done, but it’s getting done. You have this monumentous amount of effort that gets put on things like dev teams handling those, whether they’re bugs or design changes that come in. But then also now being more nimble and having a better strategy of how we do release and and servicing the stuff that we’ll be able to get out over the next year, I am really excited to see. And, you know, we do work as closely as we can with those teams. And so it’s great when they reach out and we get to share things like I’m sure you guys are all in contact to between the DL and other connections. You know, the response there is much better than it was, say, three, four years ago or five years ago. And more work is getting done faster. And I think at a way higher quality and in a lot of it has to do with you putting out smaller things faster. And I think the fundamentals being better really drives to a better quality bar. So there’s always work to be done. But I am just hugely impressed. I’m I’m really happy with how things are turned out.

Shawn Tabor:
What do you think, Tabor’s should we let George ask another developer questions?

George Doubinski:
I think I think we can. He’s jonesing for a dove. He’s been quiet. Please.

George Doubinski:
Hi. I don’t have any developer questions, but I have a suggestion for teams for the backgrounds. Can we have Sean McNellis’ hair as an option? I can just put it right on there. So, yeah.

Sean McNellis:
Anybody listening out there, You can download a desktop Snapchat camera and it layers over the top of your camera and you can put images and hair and glasses and that can be your your camera through teams.

George Doubinski:
So I don’t have it on this computer, but apparently blow c_p_u_ out of the water completely.

Sean McNellis:
Oh, it’s killer plug in because that sucker will rip your battery apart.

Joel Lindstrom:
I was the robot. It’s pretty cool.

George Doubinski:
What do you do for good heair?

Sean McNellis:
Yeah, well, that’s my favorite thing to do is jump on with the state director somewhere, you know, as my head expressed as a roll of toilet paper, you know.

Joel Lindstrom:
Yeah. There’s there’s one of them that will show like double view than four of you. Than eight of you. And you go pretty high like the lowest third of you. I’ve been using that sometimes. And just during the call incrementing it there’s two of me and then there’s four of me. It’s like I’m multiplying during the call. It’s awesome. I know. I love I love people turning on the web cam and seeing that they’re in there, you know, ratty T-shirts and stuff like that. That’s been the that’s been the biggest thing to me to be on calls where the vice president from the company is on. And he’s wearing he’s wearing a t shirt that obviously has been around for many years.

Shawn Tabor:
Oh, yeah, yeah. As a whole, you make more money that you can you can afford to at least go with the polo.

Joel Lindstrom:
And I want to I want to see how long the beard is. I hear Purvin Patel is giving Santa Claus a run for his money these days.

Sean McNellis:
Is that his thing? He’s not going to shave until he’s out of quarantine.

George Doubinski:
I just want the guys you see people using current quarantine as an excuse to be sloppy in their in their work because now we will have excuse. I mean, given people are more forgiving, like my customers definitely leave a bit more room for these sleep deadlines and things like that. And we try and not to abuse these good moyal. But I wonder if you’ve seen the cases where people kind of start abusing this goodwills as of right now.

Joel Lindstrom:
If I’m a salesguy, right now, if I lose any deals, I’m blaming covid it for it. Whether, oh, you know that you aren’t going to do it because it covid.

Shawn Tabor:
You know, I I would say from it from a delivery standpoint, I’m not saying that if anything, it’s it’s it’s the customer that gets the concept of, well, you’re you’re not traveling, you’re available. So you should be able to do what they need you to do. And for all intents and purposes, I mean, that’s that’s true. So we’re we’re actually getting more work with my projects. I’m getting more work because of it. Yeah.

Sean McNellis:
And I think, you know, my guess my case might be a little bit different. There’s there’s several of us that work with government. Customers are working on task forces for like, you know, test drive up testing and stuff. And so this is the busiest by far. I have been in 16 years at the company. And we’ve been through some really busy. We’ve been through some slow times and we’ve been through some really busy times. And this beats it at one hundred times. It seems like. And it yeah, it’s. And I you know, thank goodness I my wife as much with her job able to decouple a little bit more from calls and help coach the kids and stuff. And so she’s been able to kind of be teacher and. Time remote worker and. Yeah, so it ‘s. But by far the customers have been very forgiving and I do think in some cases we’re happy to overdeliver. If you want to call it that sounds weird, but I do.

George Doubinski:
Look, I experienced it firsthand because before when customer calls, I’d be like, I’m sorry, I already left the office. So can we do it tomorrow? And customer Yeah, OK. Yeah. Fair enough. And now they more like I know you there. I know. You’re welcome. I called you. You have something to do, huh?

Joel Lindstrom:
Well, thanks Sean McNellis and Shawn Tabor. But thanks, Sean, for for joining us again. And you’re always welcome back here. It’s always informative to talk to you and find out what’s really going on with the PFE team. And yeah.

Sean McNellis:
And next time we can bring Dieken with and we can really confuse people. And Sean Noll is also a PFG now.

Shawn Tabor:
So get as many Shawn as we can. I think that’d be fantastic..

Joel Lindstrom:
And maybe Shan MacArthur for there is give it just to screw it all up.

Joel Lindstrom:
My current project.The CSP is named Sean. There’s a consultant named Sean and I’m the tech. So it’s a good time.

Joel Lindstrom:
All the groomsmen my wedding were named John. That was really easy cause I said hey John and they all turn around.

Sean McNellis:
My pleasure to be here. Thank you. Thanks for having me. I appreciate it. Stay safe. And hopefully we’ll kind of work our way through this here in short order. And everything will be better than the worst case scenario. So we can hope so.

George Doubinski:
Join us next time

Quickly and accurately automatically transcribe your audio audio files with Sonix, the best speech-to-text transcription service.

Sonix uses cutting-edge artificial intelligence to convert your mp3 files to text.

Create better transcripts with online automated transcription. Sonix converts audio to text in minutes, not hours. Rapid advancements in speech-to-text technology has made transcription a whole lot easier. Do you have a podcast? Here’s how to automatically transcribe your podcasts with Sonix. Get the most out of your audio content with Sonix. Are you a podcaster looking for automated transcription? Sonix can help you better transcribe your podcast episodes.

Sonix uses cutting-edge artificial intelligence to convert your mp3 files to text.

Sonix is the best online audio transcription software in 2020—it’s fast, easy, and affordable.

If you are looking for a great way to convert your audio to text, try Sonix today.

This episode is a production of Dynamic Podcasts LLC.