How Google IT Manages Enterprise Extensions (Cloud Next ’19)


[ELECTRONIC MUSIC] FLETCHER OLIVER:
Hello, everybody. My name is Fletcher. And I work on the
Chrome Browser team, as a Chrome browser specialists. I’m here with my
associate, Nick. Nick, give yourself
a quick intro. NICK PETERSON: Hi, everyone. I’m Nick. I work on Google security
in the IT department. FLETCHER OLIVER:
Great, so what we’re going to be covering
today in this session– is we’ve gotten a lot of
feedback from our customers that managing
extensions at scale can tend to be a
painful process. It can be a little cumbersome. And we hear you. And obviously, there’s
people in the seats today. So most likely, you
guys may feel similarly. So what we’re going
to be covering is we’ll talk about, at a high
level, what extensions are, why they provide value
in an enterprise, as well as some
inherent risks that would be included with
having those present on your end-users’ machines. And then I’ll softball
it over to my pal Nick. And he’ll talk a little bit
more about how Google manages extensions internally
and how we moved from a traditional way
of managing extensions, through whitelists
and blacklists, to a more enhanced methodology
that will detail there. And then we’ll
wrap everything up. And we’ll talk about how you,
yourselves as administrators, can be able to roll this
out in your enterprise. So let’s get started. So why extensions? What are extensions? Obviously– hopefully– people
are moderately aware of what that is, if you’re here today. But if you’re not,
really, it’s just a software program that’s
going to customize your Chrome browser experience. So users may use it to add
something as simple as a theme, have a change in how Chrome
browser works from a behavior perspective– so every
time I open up a new tab, I get greeted with a beautiful
kitten picture or something along those lines– as well as productivity
tools, where I need to join a web
meeting or record my screen. So, really, those are
the values that it adds from a customization
and productivity side. But Nick, I’ll throw
it over to you. Tell me a little
bit more about what risks come with
extensions, especially at scale in an enterprise. NICK PETERSON: Yeah
so while extensions have a lot of
productivity benefits, there are some issues. Because really, you’re
extending the browser and all of its capabilities. So extensions can
introduce vulnerabilities if they’re not properly coded
or in case issues come up later with third-party libraries. Additionally, depending
on the permissions, they can access your
cookies, which is not good, because that’s how you
authenticate to your websites. They can make requests
as you, as well as inject JavaScript,
which can leak data if it’s not properly coded. FLETCHER OLIVER:
Great– thanks, Nick. So some challenges that exist
with managing extensions– there’s three things that really
make it a painful process. First is limited visibility into
what extensions are installed– so how they’ve been installed,
where they’re installed, as well as what
rights or permissions they require in order to run. And then, being able to have
a formal vetting process– I’ll allow this extension. I won’t allow this one. It can tend to be
confusing– as well as being able to have a
management process around that. Especially if you don’t
have much visibility into what extensions
are present, it’s hard to determine
what you can or can’t run, as they might be
crucial to the end users in order to get their job done. So we have a dual
approach to protection. The first side is
securing the source. And securing the source– that’s on us. That’s on Google to
be able to make sure that there is a safe and
secure mechanism for your users or your administrators to be
able to download and install extensions. On the other hand,
securing the endpoint– that’s on the administrator,
through different policies. And we’ll talk more about
not only the traditional ways of managing extensions but
this enhanced methodology that you’ll hear us pretty
much blathering around through the whole session. So let’s talk about our
side, the Google side, which is securing the source. So we’ve had some recent
developments in the Chrome Web Store on enhancing security. So we’ve removed
in-line installations. So extensions either need
to be installed directly from the Chrome Web Store or
pushed out via Force Install by an administrator. As well as– we’ve tightened
the review process. So as developers submit their
extensions for approval, our algorithms have been
tightened to prevent malicious extensions being even
present on the web machines before they get live or present
on your end-users’ machines. As well as– we’ve developed
and enhanced our platform through Manifest Version 3
at a super, super high level. We make it easier for
developers to develop extensions in a more secure way and
harder for developers to create it in a
less secure way. So policy managements–
this is, as I said, securing the
end point, which is up to the
administrator, in order to roll out these policies. Traditionally, this is all
done through whitelists. These extensions are
fine to be installed. Blacklists– these
extensions, I never want to be installed
on my end-points. The problem with
traditional management– as we’ll talk
throughout the session– is those whitelists
and blacklists tend to get pretty lengthy. And it can be a
full-time job sometimes in order to maintain those. And actually, as Nick
will highlight later on in the session,
it’s something that Google
traditionally, internally, used to manage through
those traditional whitelists and blacklists. But we actually moved to
enhanced management style, which we’ll highlight
and provide you guys some tips and tricks on how
you guys can do it yourselves. But also, it’s really
managing by the behavior of the extension itself. So the permissions
or what rights those extensions need
in order to run– managing just by that, instead
of by the extension itself, as well as blocking the action
of the extension on your most sensitive websites. So users are able to get the
extensions that they want. But you’re able to protect
your most sensitive sites from extensions reading
information for them. So want I foreshadowed,
I guess, how Google moved from a
traditional whitelist and blacklist method to
an enhanced methodology. So I’m actually going
to turn it over to Nick. And he’s going to talk about
that journey for Google. So go ahead, Nick. NICK PETERSON: Thanks, Fletcher. So just a little FYI– like a lot of other companies,
we have a lot of machines. I see a lot of machines. It’s in the hundreds
of thousands. So trying to make this
work is something– we need to make sure it scales. And rather than whitelisting
one or two extensions, we have about 60,000 extensions
that our employees actively use. So going through and
individually looking at each one of these, isn’t
really something we can do. Additionally, our
employees really like developing new extensions
for their productivity. We have around 7,000 extensions. And that’s not just that
people have made at some point. But these are actively used and
installed on people’s machines. So internally, we wanted
to make sure there was something that worked for us. So the whitelist and
blacklist just wasn’t working. So our philosophy– we
could have just said, let’s just block everything or
let’s just allow everything. But we wanted to make sure
we allowed users to access the extensions they need. But we also don’t want that
additional risk around it. So we wanted to have something
that still let people be productive and didn’t
have a risk to our users, because we have the
responsibility of protecting the data that our
users entrust with us. So as Fletcher
mentioned earlier, when we’re using the traditional
blacklist and whitelist, it was missing the mark. There was limitations. Our blacklists we’re getting
into the tens of thousands of extensions long. And that’s just not
something that’s doable. That also means that reviewing
these tens of thousands of extensions on the whitelist
isn’t really something we can do in depth. And it really doesn’t
help the situation when an extension
can go and pull code from a different website
the moment it loads, because there’s really
no way to review that. Additionally, reviewing
these means you need someone to be able to go
through the JavaScript, figure out what it’s doing. And when it’s
minified and things like, that it makes
it really hard to do. So it wasn’t really giving
us the security we needed. So we needed to come up
with something different. We needed something that
would balance the productivity of the users– so they got to
use extensions that wanted– so they weren’t waiting
on to review an extension and whitelist it– along with the
needs of security, because we need to protect our
employees as well as the data that our users entrust with us. So we have a new approach that
Fletcher talked about earlier. It’s enhanced
management capability. So we needed something to scale,
to fit those tens of thousands of extensions, that increased
user productivity, so they’re not waiting on us to
whitelist extensions, as well as mitigating the
risk that came along with it. And we did this through
the introduction of managed by permission
and the runtime host controls, which we’ll talk
about in a little bit. Before we talk about
managing by permission, we need to understand what
permissions really are. So to limit the risk
to users, extensions must declare which
sites they’re going to interact with as well as the
risky actions they may take. And they do this by
requiring developers, when they’re making
the extension, to declare permissions. And depending on what
the extension does, some sites have say permissions,
some have device permissions. Some have both. And it’s the
combination of these two that cover most of
the risky actions that an extension can take. So the site permissions
are the list of sites that an extension
may interact with. And the device permissions
cover the risky actions specific to the device. And the convergence of these
two cover most of the risk. So when we talk about
manage by permission, what this is actually
allowing you to do is block the installation
of an extension, specifically by the
permissions it declares. So if you don’t
want an extension to be able to view the
screen, you could say, the Desktop Capture
permission is blocked. And then you
wouldn’t have to look at the code of each extension. Chrome would know,
at installation time, that extension has a
Desktop Capture permission. Additionally, you can
configure extensions. So if there is a extension
that you want to allow, even though it has pretty good
permission, you can do that. This also allows you to– what
we call “permission pinning.” So if you have an
extension you like, you can list the
permissions it can use. And if in the future it updates
and adds a new permission, it’s instantly disabled
on all of your users, until you modify it the policy. So you don’t have to
worry about extensions updating new permissions
without you knowing. And, additionally, you can list
a custom installation error message. So when users are
blocked, you can tell them where to go to get help. So this new method is
allowing you to be safer from compromised extensions. Because rather than
worrying about what happens if this thing
updates or making sure that you’ve reviewed
the extension properly, you’re looking at its behavior. And you don’t have these long
whitelists and blacklists anymore. You don’t have to
worry about updates. And you don’t have to vet
each extension individually. So let’s take a look at
how this looks in action. FLETCHER OLIVER:
Great– all right. So thanks, Nick. Yeah– so we talked about
block by permission. We talked about how it works
and the value that it offers. But let’s see it in action, OK? So, in this scenario,
I have a user. And that user has customized
their browser experience. They’ve installed this
extension that, every time they open a new tab,
they get presented with a super, awesome,
anime-style studio Ghibli screenshot. It’s dynamic. It looks great. Myself, as an administrator– really don’t care
if they have this installed on their machine. If they want to anime it up,
I’m totally cool with it. However– and unlucky
for this user– this particular extension
uses a permission or has a specific right that
I have determined is no bueno. So this is a screenshot of the
manifest for this extension. And I’ve highlighted,
in my research, that there’s a certain
permission that I say, no way is this going to
run on any of my devices. And that permission is
the Storage permission. Quick side note– there’s
not really much wrong with the Storage permission. It’s not nefarious or
anything like that. Just for demonstration
purposes– I picked it. So unfortunately
for my user, they’re going to lose their super,
awesome, studio Ghibli anime theme. But lucky for me,
we’re going to be able to secure the end-point,
which is really the goal. So there’s actually
three ways that you can be able to do this via policy. So the first way
is Group Policy. And this is the way
that most enterprises are managing Chrome Browser. So if you’re not
managing via Group Policy and you have it in
your environment, I highly recommend
that you take a look and download the Chrome
Enterprise bundle. It includes some
handy-dandy MSIs that you can use to deploy
out to your end-users through your application
deployment method of choice, as well as 300-plus
policies that you can use to maintain
the look and feel, manage bookmarks, security
settings– all that stuff. So if you have the
super, super eagle eyes, you can see that I’m in the
Extension Management Settings policy. But I’ll zoom in for you. So I put a JSON string in there. And if you take a
look at the left– or stage left, right-left– I don’t know. There’s a star. And in the star, that denotes
that this is a default policy. And what that means
is that this policy will apply to all extensions. So extensions that are already
installed will be disabled. And any extensions that
are going to be installed will be blocked. Just note that, if the
extension’s already present, it’ll just disable it. It will not uninstall it. And then I’ve declared
that the actual blocked permission– and then
Storage is the permission that I’m blocking. So I talked about–
there’s three ways. First way– Group Policy. Second– Windows Registry. So, if you don’t necessarily
like to write JSON, which maybe there’s some
JSON gurus in the audience. I’m not sure. I’m sure you can get your
kicks on the Group Policy part. But here is the location where
you can create a sub-key under HKEY_LOCAL_MACHINE SOFTWARE
Policy Google Chrome– yeah, I got them all. And then the Extension Settings. And then there’s a
sub-key under there. See that star again? So this is a default policy. And before you start
pulling out your phones and taking photos or– no, you’re totally fine. I was going to say that we
actually have a white paper that’s coming out– it’s
called “Managing Extensions in your Enterprise”– that
has step-by-step instructions. But you can feel free to take
your photo whenever you want. It’s not NDA or anything. Otherwise, there would have
been a million slides that say, do not take photos! Anyway, we have a white
paper that’s coming out, which has step-by-step
processes not only to do this enhanced method
but also the traditional way. So maybe the enhanced
method isn’t right for you– people like options. So if you want traditional
whitelists and blacklist, it has step-by-step instructions
in order to do that. And that’s actually was
published yesterday. So at the end, we’ll
have some resource links. And in a couple of
days or so, it’ll get pushed to your next app. You can go ahead
and click on that. So I’ve created this key. It actually converts it to JSON
on the back end, which is nice. And I talked about three ways– Group Policy, Windows Registry. And the last one is the
Google Admin Console or the new Chrome Browser
Cloud Management feature that was just released– Phil, was it today? PHIL: Yes. FLETCHER OLIVER: It was
today, so hurray for you guys and for us. So Chrome Browser
Cloud Management– if you guys haven’t heard of it,
it’s an exciting new feature. It’s a free feature. And it’s a great way to not
only be able to push policies from the cloud but, also,
gives you really enhanced view into the extensions that
are installed and present on your users’ machines. As I talked way, way long
ago, in the beginning of the presentation,
that’s the pain, right?– being able to find out what
extensions are installed, how they were installed, and
what permissions or rights they require in order to run. So you can actually get all that
information from Chrome Browser Cloud Management. So I highly recommend
you take a look at it. And if you wanted to do
this blocked by permission, you check a box. So like I said, not throwing
anything up on the JSON people that like to write JSON,
but checking a box, to me, seems a little bit easier. At least, it was easy
to get the screenshot. So I’ve rolled this out through
one of my three policies– Group Policy, Windows Registry,
and the Google Admin Console or the new Chrome
Browser Cloud Management. And let’s see what happens. So sad face for my user– I’ve updated the policy. And they get a message. And it says, this
has been blocked. As Nick highlighted before,
you can customize this message to say, Contact IT, File a
ticket through your ITSM tool– whatever you guys like. So you can see, once it
blocks, I open up new tabs. And unfortunately for them,
the anime wallpaper is gone. So poor one out for this user. So all right– NICK PETERSON:
And it also blocks the installation
of new extensions, not just the currently
installed extensions as well. FLETCHER OLIVER: Exactly. So Nick, tell us
a little bit more about runtime host controls. NICK PETERSON: Sure– so blocked
by permission is helpful. But it’s a rather
heavy-handed way of going about things
sometimes, because that means the user can’t
actually use the extension. So another option is
Runtime Host Controls. And what this feature
does is it lets you define a list of websites
that you consider sensitive. And it lets the extension
work on all of the sites except for the ones you
define as sensitive. So when they start
to do anything like reading cookies or
injecting JavaScripts or modifying web requests
about a particular site, the browser will say,
what site are you attempting to do this on? And if it’s one of the
listed sensitive sites, then it will be blocked. So the users still get to
install their extensions. And it’ll work on all the
other websites, just not your websites which have
your sensitive customer data on them. So the benefits– the users
can use the more extensions. And you also get to protect
your most sensitive data. You still don’t
have to do anyway whitelists or blacklists. You don’t have to
review any updates. And you don’t have to look at
the code of each extension. You simply defined the list
of sites you want to protect. FLETCHER OLIVER: Great. NICK PETERSON: And the
combination of these two is really what we
rely on, instead of the whitelist and blacklist. FLETCHER OLIVER: So that’s
what Google IT uses internally. NICK PETERSON: Exactly. FLETCHER OLIVER: Great. All right– so let’s
see it in action. So I have a user. They have an
extension installed. This extension toggles a
super cool and edgy dark mode. They can toggle it on. They can toggle it off. And as I felt the same way
about my anime wallpaper user, I feel the exact same way
about my dark mode user. I do not care if you have
this extension installed. I have other things
to worry about. So yes, if they want to ruin
their eyes in their parents’ basement– totally fine with me. However, I’ve denoted that
Google Arts and Culture is my sensitive site. And I don’t want any
extension to read any data off of my sensitive site,
because it could have my IP. It could have PII or Personal
Identifying Information for my users or my customers. So no extensions can
run on that site. So let’s talk about
how you can do it. So we talked about three ways. As you guessed, the first
one is Group Policy. I’m back in my friendly
Nest, which is the extension management settings policy. And I have some JSON. I’m using the star for
the default policy. So this is going to
say, no extensions can run on my super
secret public site ArtsandCulture.Google.com. As you could see, there’s
some stars also in the URL. So there’s some syntax that you
can put in here to make sure that you can– this will
cover, like HTTP, HTTPS– as well as you can put
some stars in there. So that way, you
cover your sub pages. You’re not just covering
your landing page. So first way– Group Policy. Second way– Windows Registry. So you can see, I’m kind
of a broken record here. So it’s the same sub-key. And I’ve just added
a string there for runtime_blocked_hosts. And that value is the same way. So Group Policy– Windows Registry. Big surprise– last one is the
Admin Console or Chrome Browser Cloud Management. So I’m actually in
the same setting where I did the
block by permission. Just right under there where
I checked storage before, I have a blocked URL section. So I just put that
syntax in there. And if I want to
block multiple sites, I just hit Enter, put
one under the other. And then I can block my
most sensitive sites. So Group Policy, Windows
Registry, Google Admin Console, Chrome Browser
Cloud Management– your flavor of choice. And let’s see the after effects. So the user goes. The policy is applied. He goes to– he or she or they– click on the Toggle. And it does not affect
my hypersensitive site of Google Arts and Culture. However, it’s able to work
just fine on CNN.travel. So this is really what we’re
trying to get to– what every IT administrator’s
trying to get to– that nirvana state that
people don’t think exists. But it’s the balance
between users actually getting what
they want and being secure on the endpoint. So users– they can still
install these extensions. That can still run them and
get the customized experience or those productivity tools. But also, we have a
dual layer of security to make sure that users– we’re protecting our
most sensitive sites and our endpoints
at the same time. So Nick– NICK PETERSON: Yeah, so when
we went about rolling this out, the first step was
really discovering what extensions are out there. And we needed to be
able to determine what permissions they use. So we didn’t simply
push out a change and affect everyone and
remove all their extensions. We needed to know
what the impact was. So this is something that
external customers can also use CBCM for, the Chrome
Browser Cloud Management, to be able to determine
what extensions you have, as well as what
permissions are on those. And from there,
you can figure out which permissions you’re
going to want to block, as well as which extensions
need an exception. So the next step
was the planning. That was the grandfathering
in the existing extinctions. So our users didn’t notice. So we grandfathered in all
of our existing extensions to just stop the bleeding. And then we’d go from there and
whittle that whitelist down. So we audited that whitelist and
removed the unnecessary entries as time went on. After that, we did that
permission-based blocking. Then we went back and we did
the runtime host policies. And we added those
in one by one, just slowly making sure
that we didn’t affect users. And from then on, for
ongoing maintenance, we created,
essentially, a flowchart that we handed to our Help
Desk that they can simply follow– whether or not to add
an extension to the whitelist, based on what it’s doing,
what existing things are whitelisted. And from that, we actually
have a fairly low queue. It’s about, I think, about a
1/4 of an FTE currently that is required to run that– or less– for a company our size. So it’s definitely something
they can scale pretty well. FLETCHER OLIVER: Great,
what was the before state? Obviously, the blacklists and
whitelist was like “rrrr.” NICK PETERSON: Yeah–
many tens of thousands of extensions on our blacklist. And even then, it
wasn’t really giving us the security benefits we needed. FLETCHER OLIVER: OK, and I would
assume that you’re this laissez faire person, like leaning
back and just kicking it. NICK PETERSON: Oh, it’s
certainly a lot better now. FLETCHER OLIVER: You
seem relaxed today. So that’s good. NICK PETERSON: Yeah, yeah– this has helped. FLETCHER OLIVER: Look
at us with banter. This is great. OK, so how do you guys apply
this in your enterprise? Pretty similar– right? And gosh!– I’m just
going to plug it again– Chrome Browser
Cloud Management– being able to get visibility
into the extensions that are installed– it’s
really the first step. You need to find
out where they’re installed, who installed them,
what permissions they require. And you can really get all this
visibility in Chrome Browser Cloud Management. And it is the
first step in order to get to enhanced
extension management– is auditing the extensions. Because if you don’t
know what’s out there and what different departments
or geolocations are using, there could be some
issues when you just start rolling out policies that
may block extensions that are crucial to core applications. So whether you use Chrome
Browser Cloud Management– which obviously, I’m biased to
recommend, but I recommend– you can also survey
your end users. So call, email–
however you do it– pester them, until they find
those gold users that represent each individual
department or geolocation and find out what extensions
that have installed. And then take that information
to determine and build a master list. So this would be– these are the hosts that
I know must be secure– that I need to protect. So those would be the runtime
host ones that you would block. Yeah– exactly. I’m agreeing with myself. Sorry. And then these are the
permissions to restrict. These are the
permissions I never want to run on any of my endpoints. And then review any
existing process. And as Nick
highlighted, you might find that you’re
whitelist and blacklist might be able to be shortened. And you might be able to
let some extensions run and be installed that you
normally wouldn’t have been able to let go in the wild. And then the rest of
these steps are really just traditional
change management, IT project management. You’re going to want
to talk to the people– the department heads. Because you’re going to
be affecting their users. And you’re going to want to
test it in a lab and a pilot. I know, nobody in this room
tests this in production, right? “Rrr”– all right– that
wasn’t really reassuring there. Establish an
operational process. New extensions are going
to be rolling in every day. So you’re going to need
find a way to vet in and vet out, usually by
looking at the manifest and seeing what
permissions are present. Or since you have
your baseline of what permissions you
will or won’t allow, you can just have the
user install them. And if it gets blocked, then
they can submit a ticket. And then you can take a
look at it at that point. And then review results,
survey your end users, roll up production in phases
to limit possible collateral damage. And here’s how you can
get started right away. So sign up for Chrome
Browser Cloud Management. Take that first step to be
able to see what extensions are present on your end
users’ machines, so you can start building
that master list. It’s a really crucial
first step to make sure that you’re going to be
effective with this management style. Test out the extension
management style that we’re talking about today. So block by permission,
blocked by runtime host. And also, take a look at some
of the latest documentation. Like I said, we have a
white paper that’s live now. It’s actually linked. It’s the second bullet
point, which is the extension management white paper. So I think that this will be
pushed to your guys’ next app, via PDF, in a day or so. Otherwise, I tried it
today just to see how well the keywords was doing. And just type in Extension
Management in Enterprise. And it should come up. Otherwise, I need to argue
with my documentation person. So the Help Center– feel
free to take a look there, as well as at the very bottom– the Browser Specialist. So that’s the team
that I work on. So if you guys want to get
some more information about how you can manage Chrome
Browser in your environment, you want to talk about making
Chrome a default browser– I recommend it– feel free to
email us at Chrome enterprise browser [email protected] It just rolls off the tongue. It’s really easy to remember. But like I said, it’ll be
pushed to your next app. [ELECTRONIC MUSIC]

Add a Comment

Your email address will not be published. Required fields are marked *