Google Chrome Keystone is Removing /var symlink on non SIP Macs Causing Boot Issues

72 thoughts on “Google Chrome Keystone is Removing /var symlink on non SIP Macs Causing Boot Issues”

1. Pingback: Michael Tsai - Blog - Chrome Updater Bug Prevents Macs From Booting
2. Pingback: Episode 139: So, Anything Happening? – Mac Admins Podcast
3. Pingback: Calm down – Chrome probably isn’t slowing down your Mac’s performance
4. Pingback: Google says its update software is crashing some Macs - 9to5Google
5. Pingback: Mac Pros running Avid suffering 'massive failures' in Hollywood-9to5Mac
6. Pingback: Cyberattack or Human Error, You Need the Best Tool to Manage the Unexpected | Groupdolists

7. Just an update, I really didn’t have too much to sweat as I did have a very recent bootable backup, so I just restored that without having to reinstall. Another reminder that we should all religiously maintain bootable backups of our internal drive with life-saving utilities such as CarbonCopyCloner.

   I can’t even imagine how long it would have taken me to get two machines back by reinstalling the OS.

8. Hi guys,

   I have a MacBook 12″ that died during my vacation …it gave me a Google Chrome crash after Chrome update this Tuesday, then I had few moments unstable system and then MacOS crash, the one with press any key to restart. After the restart I saw apple logo then the forbidden logo …it barely boot twice, one boot Tuesday and one boot Wednesday but the MacBook crashed after few minutes of usage …now no sign of boot or startup at all. The MacBook has the latest MacOS update.

   Any ideas? Thanks!

   1. Hello Valif,

      It is recommended that you reinstall macOS over the top of your old OS. This will preserve the data yet repair the /var symlink that is causing problems.

      Thanks

9. Hello,

   I did the step by step instruction on a 10.10.5 machine. But here what happen.
   -It rebooted fine but if you restart the Mac the issue comes back.
   -All the admin account is gone from the Users and Group. But the standard user account are still there.
   -The Mac cannot access the internet or any network resource. Looking at the network the IP, gateway and DNS is correct, but it needed an admin username and password to make some changes. The problem is there are no admin account listed.

   Any further help would be greatly appreciated.

   https://derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/#more-10541

10. Hello all,

    10.10 or older:
    only do symblink and the hidden chflag and
    remove launch agents from
    /Users/[affected user]/Library/LaunchAgents/
    OR
    /Library/LaunchAgents/
    – com.google.keystone.agent.plist
    – com.google.keystone.xpcservice.plist
    # exit
    (recommended) # csrutil enable (10.11+)
    8. Reboot

    Could someone please help translate this into what command are need. Like the fix for 10.11 and later
    Thank you in advance!!!

11. Two of my Macs were affected by this, a 5,1 Mac Pro running High Sierra and a Mac Mini running El Capitan and Server.app. Strangely enough, while the Mac Pro had that issue, it wasn’t even showing all my boot disks. Thank God, I had plenty of different date backups, the one from September 21 saved my posterior.

12. This didn’t work on my infected machines.
    `/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle` doesn’t exists because is in `~/Library/Google/` (under the home/user directory, not disk root)
    so `/Users//Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle`

    Because of that, what worked instead was:

    “`
    chroot /Volumes/Macintosh\ HD # “Macintosh HD” is the default
    rm -rf /Users//Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
    mv var var_back # var may not exist, but this is fine
    ln -sh private/var var
    chflags -h restricted /var
    chflags -h hidden /var
    xattr -sw com.apple.rootless “” /var
    “`

    Where username is the user with the Google services installed

    1. /Users/USERNAME/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle

       “`
       chroot /Volumes/Macintosh\ HD # “Macintosh HD” is the default
       rm -rf /Users/USERNAME/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
       mv var var_back # var may not exist, but this is fine
       ln -sh private/var var
       chflags -h restricted /var
       chflags -h hidden /var
       xattr -sw com.apple.rootless “” /var
       “`

       The text validator has stripped “USERNAME” tags from the pathss

       1. Stockmind, thanks for the tip! I added it to the fix section.

          1. @admin —

             I suggest that anywhere `/Users/USERNAME/` is in the path, it be changed to `/Users/*/` because this really should be done for all users on all affected Macs.

             I also add my voice to the wish that the instructions for 10.9 and 10.10 were an explicit action sequence with explicit commands, just like the instructions for 10.11+.

             1. TallTed,

                Take a peek at Rich Trouton’s Post. He has step by step instructions for 10.9 and 10.10! https://derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/#more-10541

13. Hi guys can you please help me to fix this problem in my mac os 10.10.5
    I’ve already try this one but nothing work for me

    1. Boot into Recovery
    2. Launch Terminal
    (only if you’re on 10.11+ with SIP)
    # chroot /Volumes/[affected install]
    # mv var vv
    # ln -s private/var var
    # chflags -h restricted /var
    # chflags -h hidden /var
    # xattr -sw com.apple.rootless “” /var

    1. Hello Sam,

       You are on 10.10 The instructions you tried are for 10.11 and above Macs. I have a section at the bottom of #7 For 10.9 and 10.10 systems. Thanks!

       1. Hi admin
          Did you talk about this one in 10.10.5

          10.10 or older:
          only do symblink and the hidden chflag and
          remove launch agents from
          /Users/[affected user]/Library/LaunchAgents/
          OR
          /Library/LaunchAgents/
          – com.google.keystone.agent.plist
          – com.google.keystone.xpcservice.plist
          # exit
          (recommended) # csrutil enable (10.11+)
          8. Reboot

          Can you please give me the commands that should I use because i didn’t know what i write in terminal

14. Reinstalled 10.10.5 yesterday from recovery mode after crashing during reboot and haven’t turned off my MacBook Pro since. I am completely lost with issues like this, will just upgrading to a newer OSX from where I am now fix it?

    1. Hello Johan,

       If you upgrade to at least 10.11 El Capitan you should be protected because it includes SIP. MacOS (SIP) System Integrity Protection would have prevented Google Chrome Keystone Updater from modifying the /var Directory.

15. So is there a fix to networking and DNS being disabled, or is it implied that we have to do a complete OS re-install in order to remedy this?

    1. Zuki: networking and DNS are not affected, unless your Mac is managed by an IT department eg using Jamf. Contact them if so.

16. Hey! I ran the commands in the Hot To Fix section, was able to access all my data, but not connect to the internet. I tried to open Safari (avoiding Chrome), and the internet wouldn’t connect. I thought hmm, might just to a restart. After the restart, the same crash happened.

    Ran the recovery mode once more thinking it would fix again. Input the cmds, and the following:
    Input: chroot /Volumes/Macintosh\ HD
    O: #nothing
    I: -rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
    O: #nothing
    I: mv var var_back
    O: mv: rename var to va_back/var: Directory not empty
    I: ln -sh private/var var
    O: #nothing
    I: chflags -h restricted /var
    O: #nothing
    I: chflags -h hidden /var
    O: #nothing
    I: xattr -sw com.apple.rootless “” /var
    I: reboot

    Upon rebooting, crash at roughly ~25% of the loading bar.
    Rebooting (again) with SHIFT; CMD + V, I see the following (final few lines):
    BootCacheControl: Unable to open /var/db/BootCache.playlist: 2 No such file or directory
    BootCacheControl: Unable to open /var/db/BootCaches/PreheatedUser/Merged.playlist: 2 No such file or directory
    BootCache Control: Unable to open /var/db/BootCaches/PreheatedUser/Login.playlist: 2 No such file or directory
    BootCacheControl: Added 32-bit shared cache to the low priority batch
    bash: /etc/rc.server : No such file or directory
    OSX FIPS Integrity Test: The /var/db/FIPS/fips_data file does not exist (2)
    FIPS USER Space POST: Integrity test failed!
    FIPS_POST failed!

    And then the shutting down jargon. For reference, I’m on 10.10.5, Macbook Pro 2011. Any insight?

    1. Martt,

       I updated the article to more clearly separate the 10.9-10.10 and the 10.11-10.14 fix.

       If you are still having problems you can reinstall 10.10 over your current installation. All data will be preserved and that should get you up and running.

       1. Thanks. Will try to run the cmds once more. If no avail, will re-install.
          On a side note, I imagine this must affecting thousands (hundreds of thousands?) of users. Why is there not more mainstream traction of the issue? I can’t imagine the number of calls the genius bar is getting.

          1. Martt, after it was found that the issue was directly related to non or disabled SIP (System Integrity Protection) systems the answer became clear. Most Macs used today have SIP enabled so they were not affected by this issue. The /var file was protected and could not be modified. So it ended up that 10.9 and 10.10 systems were affected and anyone who had a newer mac but disabled SIP. The number of users in this situation was not a big % of the total Mac population, but ended up being thousands of users! I am glad Apple implemented SIP as it protects critical system files from being modified.

17. Thanks for the info! There’s a typo in the commands listed as the Google provided fix: “choot” should be “chroot”.

    1. Kimmo, Great catch! Fixed.
       Thanks

18. Ran the fix posted by Google on several machines running 10.10.5 that were all disabled, managed to get them to boot again – also reporting that the ‘users’ and ‘sharing’ preferences all seem to have been disabled in the System Preferences/Sharing panel. No DNS or browsing Internet, however ssh does work through the command line.

    Hoping that fixes will be posted soon to acknowledge this fairly major screw-up.

19. Oh man !
    My Hackintosh crashed yesterday morning with all this symptômes : some apps could not reach DNS, I restarted and was in infinite boot loop.
    I recovered from a version from last Saturday and I will remove Chrome completely to be sure…hoping he has not already updated to the faulty version again.
    Good to have all the infos on your page… I was about to buy a new motherboard thinking it may had a hardware problem…
    thanks a lot !

20. We have had this happen to two of our Mac Pro’s in the last hour, one a laptop that we had fixed just hours previously – the article says the update had been disabled, so how was it still able to do this? We do not have SIP on either computer, the only real fix we know for now is to not use Chrome.

21. I’ve had the issue under 10.10.5 yesterday, had to reinstall “manually” Yosemite (using another Mac in Target mode) ’cause recovery mode gives “service unavailable” message….
    I’ve just the “ls -ldO /var” command in terminal and I get this (the Mac is on since the reinstallation, thus the login date):
    Last login: Tue Sep 24 17:48:03 on console
    mac-pro-de-user:~ user$ ls -ldO /var
    lrwxr-xr-x@ 1 root wheel hidden 11 24 sep 17:30 /var -> private/var
    mac-pro-de-user:~ user$
    Is it normal?

    I’ve deleted both Launch Agent which, indeed, showed the exact date and hour of the crash…

    Thanks a lot for all your work!

    Mathieu
    I don’t see the “retsricted” thing as described in the article.

    1. Mathieu, I added an extra note in the fix part to mention 10.10 does not support SIP. You have to run a different set of commands.

       Thanks!

22. FYI – from Hacker News

    https://news.ycombinator.com/item?id=21065504

    —–
    Hey. Google Keystone tech lead here. We are aware of the issue, and we’ve stopped the release. We’re building a replacement that fixes the problem. In the meantime, to fix affected machines:

    sudo rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
    sudo ln -F /private/var /var

    This deletes the affected version of Keystone and reinstates the damaged /var symlink.

    The version of Keystone packaged with Chrome is not affected by this bug, so allowing it to reinstall Keystone will not recreate the issue.

23. After using this fix, the computer booted normally, but nothing has network access except chrome.
    Also the administrator account doesn’t work, after rebooting, once again the computer fails to boot.
    What could I have done wrong, what else would I need to do after booting into the OS normally?

    1. I’m having the same problem. Mac os 10.10.5. I followed google terminal commands for fix and afterwards noticed this article and the difference fixes based on OS version. Did we screw up the filesystem by entering the 10.11+ commands into terminal instead of 10.10 or below?

       1. Bizz_gg,

          What happened after you ran those commands? Did you get an error? Is the system ok now?

24. I have no problem with calling Google’s software a Virus ;-P.

25. Trying to run the fix code that starts with xattr -sw com.apple.rootless and thereafter doesn’t work.

    1. Derek,

       What error are you getting? Are you booted to recovery? What OS version are you trying this on?

26. how can you fix this if you can’t boot into recovery or single user mode? Using a Hackintosh.

    1. Tim,

       I updated the post and added #8 for you! Let me know if it works.

27. may want to note that if you are on OS 10.9 or 10.10 then the output would be

    lrwxr-xr-x@ 1 root wheel hidden 11 Apr 1 2018 /var -> private/var

    1. Thanks Steve! I will add that in.

28. How to disable Google Chrome auto updates, preventatively:

    launchctl unload ~/Library/LaunchAgents/com.google.keystone.agent.plist
    launchctl unload ~/Library/LaunchAgents/com.google.keystone.xpcservice.plist
    launchctl remove com.google.keystone.xpcservice.plist
    launchctl remove com.google.keystone.agent.plist

29. Are all affected Macs managed by Jamf? We went through a similar nightmare a year ago that affected Macs across our organization. We tracked it down to a faulty Mac OS install policy.

    1. Concerned Mac, this issue happened to some machines that were not managed by Jamf.

30. just to verify. the fix can also be done booting into single user mode?

    1. Bbunn,

       Yes, you should be able use the fix in single user mode. The fix was only tested in recovery mode though. If you try it and it works can you report back ?

       Thanks!

       1. yes it works. you only need to “mount -uw /” to make the file system writable.

          Thank you for the good work!

31. There is a lot of technical jargon in the crucial part of this post that tells people how to check and/or fix the issue. Could you translate for the layman?

    1. Alex,

       Agreed. I will make a section that explains the issue better so more people can understand it now!

       1. Thank you.

          So I’ve checked my systems by following Step 6 of the article. All appears to be well despite having SIP disabled on one of the machines. I’m still afraid to reboot the computers! Is there anything we should be doing in order to actually prevent the issue from occurring at all? Without taking action I feel like a sitting duck!

          Should I follow step 7 as a preventative measure? Should I disable auto-updates to Chrome?

32. Would you mind if I linked to your explanation and solution on my Avid-based blog?

    1. Approved!!!

       Thanks

       1. Thanks. Here’s the blog if you’re at all interested:
          https://avidhatesyou.blogspot.com/2019/09/avid-behaving-exceptionally-badly-macs.html

          1. Man, your article crucified Avid. I get that the site is meant to do that, but Avid really isn’t at fault here.

             1. Michael, are you talking about my article or someone else’s? Mine was one of the first to say it was NOT AVID.

             2. No, Judd’s. Yours is spot on.

                (And I cant seem to reply directly to your comment)

                1. Got you Michael! I agree the Reply system is not the best.

33. I ran the script, and get “symlink” as the output. Does that mean I’m safe?

    1. Stevens,

       You can run ls -ldO /var to see. It should say “restricted” and point to private/var like below

       lrwxr-xr-x@ 1 root wheel restricted,hidden 11 Apr 1 2018 /var -> private/var

       If the link is broken it will show something like this

       drwxr-xr-x 5 503 wheel – 170 Sep 24 14:37 /var

    2. I confirmed that “symlink” means you’re ok.

34. The Jamf extension attribute won’t work if you are using a FQDN to connect your macs to your JAMF Pro server. /var/resolv.conf is needed for DNS to work properly and since the system is creating a bogus /var folder, and the file in /private/var is inaccessible, DNS breaks. Macs with this problem can’t check in, get policy, or update inventory with extension attribute data. We are reverting to Apple Remote Desktop to send UNIX commands to determine which machines have broken symlinks.

35. Strange thing is… We are affected by this and all our Mac’s have SIP enabled…. Which is even more strange…

    1. Yes, me too – 2018 MacBook Pro 13″ macOS 10.14.6 up to date, SIP enabled. Got stuck in a Boot screen asking to choose a language, looked a lot like the Recovery Boot screen… Force rebooting with Command-Option-P-R a few times worked for me (so maybe a different issue?)

    2. me too (Mac Pro late 2013, latest OSX)

    3. Me too. 10.13.6 with SIP enabled. Multiple MacBook Pro models. Frustrating as hell since none of the crash reports were helpful.

       1. Forgot to mention that all was fixed with Google’s instructions (as posted in the article above).
          chroot /Volumes/Macintosh HD
          rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
          mv var var_back
          ln -sh private/var
          chflags -h restricted /var
          chflags -h hidden /var
          xattr -sw com.apple.rootless “” /var

    4. Looks like Google has found a reproducible case of this when SIP is enabled. It’s mentioned on the following page:

       https://bugs.chromium.org/p/chromium/issues/detail?id=1007358

Comments are closed.