UPDATE 04/10/20:I have noticed increased traffic to my Mac BridgeOS DFU Restore Article. I looked around a bit found a small number of reports that the 10.15.4 Supplemental Update is Bricking T2 Macs.To the user the Mac seems dead. I had 3 different reports sent to me and found 3otherreportposts. The good news is, a BridgeOS restore was able to bring some of the Macs back to life.
In a surprise today, Apple released a new Supplemental update for macOS Catalina 10.15.4. This update weighs in at 1.01gb and is available now for all 10.15.4 Catalina users. This update looks to fix 4 different issues. Apple also updated BridgeOS and re released a new installer.app, combo update and delta update.
The Catalina 10.15.4 Supplemental Update Includes the following fixes.
macOS Catalina 10.15.4 supplemental update improves the stability and security of your Mac.
Fixes an issue where Mac computers running macOS Catalina 10.15.4 could not participate in FaceTime calls with devices running iOS 9.3.6 and earlier or OS X El Capitan 10.11.6 and earlier
Resolves an issue where you may repeatedly receive a password prompt for an Office 365 account
Fixes an issue where MacBook Air (Retina, 13-inch, 2020) may hang in Setup Assistant or when disconnecting and reconnecting a 4K or 5K external display
Resolves an issue where a USB-C port in your Mac may become unresponsive
BridgeOS update
The T2 BridgeOS was updated in this macOS Catalina Supplemental Update.
T2 BridgeOS Version = 17.16.14281
Current and Previous Build Versions of 10.15
10.15.4 = (19E287) April 8th, 2020 = Current Release
If I deployed/cached the old 10.15.4 Installer.app for OS Upgrades, do I need to redeploy?
Do I have to replace my deployable 10.15 Installer.app?– Yes
If you deployed the old version of the 10.15.4 (19E266) Installer, you should update it to the new (19E287). If you don’t your users will have to install the New Catalina Supplemental Update after installing or upgrading.
Security Content
The Supplemental Update does not list any Published CVE entries.
Some Intel GPU Only Based MacBooks Airs are Freezing / Crashing while using Video Conf apps like Zoom After installing the 2020-002 Security updates.
After Apple Released the 2020-002 Security Updates, some users started to report that their Mac would would freeze up when using GPU Hardware Accelerated apps or video.
UPDATE 05/26/20 – Apple has just released the 2020-003 Security Update for macOS Mojave 10.14 and High Sierra 10.13. Please let me know if the new update fixes the issue for you!!!
UPDATE 05/18/20 – On Friday I tested Zoom client v4.6.8 on a 2017 MacBook Air with 10.13.6. I experienced a freeze & 5 different app crashes on a multi user meeting. Today I installed 2020-003 Beta and did not see a single crash for over 6 hours on the same meeting. The 2020-003 Security Update is looking really good, but I still would like to see more confirmations. If you installed 2020-003 Beta and it fixed the issue for you , please let me know. I am hoping the update is released tomorrow or sometime this week!
UPDATE 4/29/20 – Today Apple released the Developer Beta version of 2020-003 Security Update for Mojave and High Sierra. I am trying to find out if it includes a fix for this issue. I will update you as soon as I have more information.
UPDATE 4/28/20: The consensus amongst users is upgrading to Catalina fixes the issue. The only problem with this is, some users have reported other GPU related weird issues in 10.15.4. Sometimes the system will freeze for a few seconds in Finder, Safari or performing other tasks. The good news is, even if the Mac does freeze (only for a few seconds) it will not require a hard power down. With that said, you could upgrade to Catalina and not have any of the above issues! If you are cautious, it might be better to wait for an update from Apple.
UPDATE 4/08/20: As the update is installed on more Macs, reports continue to come in. Apps like Illustrator and Animate from the Adobe Creative Cloud Suite are now causing freezing A MacAdmins User who has a ticket in, says Apple is aware of the issue and is actively working on a fix.
UPDATE 4/03/20: MacAdmins User Bollman decided to test the latest Zoom installer (4.6.9) and has not had any crashes for 6 hours. I loaded up (4.6.8) 3 times to confirm the crashes and gather additional logs. Then I updated to Zoom 4.6.9 and have not had any crashes for over an hour. I added this new information to the work around section below.I can’t explain this as the Zoom update patch notes only mention updates to fix the installer issues brought up by Security Researchers.
UPDATE 4/02/20: New reports are still rolling in. As each new report rolls in with a confirmed .gpuRestart log I will add that application to affected list below. The issue might not be only related to Video Conference Apps. Some users are seeing the issue with anything that is related to Hardware Accelerated Video.Full screen video, video in Safari, or YouTube.
I reported on a similar issue in August of 2019 when the macOS Mojave 10.14.6 Update started to cause Kernel Panics if you used the Built In FaceTime Camera.
This article has seen a big uptick in traffic as of a few days ago. Then I started to receive emails from users who were having their Mac freeze up after installing the Security Updates. After that the reports started to come in on MacAdmins Chat.
In this article, I will give you you an overview of the issue. In the end, I will show you a few workarounds that might work until Apple releases a fix.
Let’s dive right in and see what’s going on here.
Table of Contents
1. Affected macOS Build Versions
2. Affected Mac Hardware & Intel GPU Versions
3. User Reports
4. What is the Issue? Mac will Freeze up requiring a hard shutdown
5. .gpuRestart Freeze Log Report
6. Software that can cause the FreezingIssue
7. This time around the issue CAN be reproduced
8. Why rolling back with Automatic Update Snapshots will NOT work.
9. Workarounds
10. If you are seeing this issue, please let Apple know.
11. Conference Software Freezing Issue Links
12. Hat Tip/Credits
1. Affected macOS Build Versions
This issue affects the following macOS Build Versions.
Catalina 10.15.4 Update(19E266)March 24th, 2020
Mojave 10.14.6 Security Update 2020-002(18G4032) March 24th, 2020
High Sierra 10.13.6 Security Update 2020-002 (17G12034) March 24th, 2020
2. Affected Mac Hardware & Intel GPU Versions
I have looked over a bunch of MacAdmin and User reports reports. It looks like the affected machines are 5th Generation Intel HD Graphics GPU only based Macs.
This is the Hardware that we think is affected so far.
1. 2015 MacBook Air
2. 2017 MacBook Air
3. 2015 12″ MacBook
4. 2015 13″ MacBook Pro
5. 2015 21.5″ iMac
Intel only GPU Versions
1. Intel HD Graphics 6000
2. Intel HD Graphics HD 5300
3. Intel Iris 6100
4. Intel Iris Pro 6200
If you have the issue on other Macs like the Mac Mini or older Macs, please do not hesitate to Contact Me.
3. User Reports.
The first report came in just two days after Apple released the Security Updates.
Anyone have issue with Zoom 4.6.7 for the Mac running on 10.14.6 where the use of the internal camera causes it to crash.
The next day more detailed reports started to roll in.
We’ve seen hard crashes on macs running 10.13.6 with latest security update (17G12034) and latest zoom version 4.6.8 (19178.0323). So far, 4 out of 80 machines with this combination of OS and zoom. What’s more in common with these machines is that they are MBA 2015 (non-retina). Anyone else seeing problems with latest security update on 10.13.6?
After doing this job for many years, I get an sense when things are starting to become an issue. It was not until this post came in on the following Monday.
For those running 10.15.4 (or latest 10.14/10.13 Security Update 2020-002 update) on the following hardware, can you try starting a zoom video conference (possibly may happen with other video conference software)? do you experience a hard crash?
After Balmes posted this, it was enough for me to take a closer look. Sure enough, users have started to report the same issues.
4. What is the Issue? The Mac will Freeze up requiring a hard shutdown.
UPDATE 04/01/20: After posting the article, I am getting a ton of reports that this issue is not just Video Conference Apps. Users are saying the Freeze / Lockup issue happens when using GPU Hardware Accelerated Video. This could be full screen video based activity.
How does the issue start? All you need to do is use some type of Video Conference Software that has multiple users with video enabled.
Once in the meeting the affected Mac can freeze up within one minute!
After the Mac Freezes, it will become 100% non responsive. The screen will freeze up and you will not be able to force quit. The only thing you can do is force power down the Mac.
5. .gpuRestart Freeze / Crashing Log Report
After you power up the Mac again, macOS will say that it was shut down due to a problem. At this point you need to look at the log to find the .gpuRestart log file.
UPDATE 04/02/20: To get the .gpuRestart log to show up, you have to let the Mac say on the frozen screen for at least a few minutes.
/Library/Logs/DiagnosticReports
You can also do a quick search by running this command
sudo ls -lah /Library/Logs/DiagnosticReports | grep .gpuRestart
Or use the Console.app, select “System Reports”.
A look at the console app and a few 2017 MacBook Air .gpuRestart error log files.
Application exampleGoogle Chrome He - Slack Helper (GP - zoom.us
Graphics Hardware exampleIntel HD Graphics 6000 - Iris Pro 6100
Signature example 803 - 802 - 806
6. Software that can cause the Freezing Issue
The following software can cause your affected Mac to freeze up. Below is a list of confirmed applications with a confirmed .gpuRestart freeze.
Zoom.us
Slack
Webex
Teams
Skype
BlueJeans
FaceTime
Sublime Text
Google Meet
Google Hangouts
Adobe Creative Cloud apps
Illustrator & Animate
VMWare Fusion
Spotify Helper
AnyDesk
ScreenConnect
Visual Studio Code
NOTE: Some of the new reports say that this happens when running the Video Conference Software from a Chrome Browser.
7. This time around the issue CAN be reproduced
I am able to reproduce this issue. If you would like to see what happens all you need to do is setup your Mac with the following.
2015-2017 MacBook Air
10.13.6 High Sierra with the 2020-002 Update installed.
Install zoom.us
Join any zoom meeting with multiple active users with their camera activated.
Join with Computer Audio. You can activate your own video or not does not matter.
Wait
Within about 1-5 min the MacBook Air screen will completely freeze and become unresponsive. You will need to hard power it off.
UPDATE 04/01/20: I was able to ssh into one of the MacBook Airs that was frozen. You can run commands like top and others. I attempted to force quit zoom.app and that did not change anything. I also tried to kill the loginwindow no go. Finally I attempted to restart the device with sudo reboot , I got the message that the ssh connection was closed like it was going to reboot but it didn’t.
8. Why rolling back with Automatic Update Snapshots will NOT work.
You might think, what if I roll back to a previous version of Mojave before the Security Update? In the past, this might have worked as the Update or security update is supposed to take an automatic tmutil localsnapshot before installing the update. If something went wrong you could boot to recovery and restore from that snapshot taken just before the update.
In this case that will not work because Update Snapshots are no longer working since 10.15.3!
Most issues like this have some type of workaround. Sometimes a workaround is found by accident or after hours of testing. This time around a few users on MacAdmins Slack have reported the following workarounds.
UPDATE 4/02/20: We are now hearing that Apple Support is recommending that users upgrade to macOS Catalina 10.15.4 to fix the issue. I can’t confirm if this fixes the issue but after looking at a large amount of .gpuRestart logs, I have not seen one from 10.15.4 yet. Many users are writing to me that after updating to 10.15.4, they are not having the freezing issue anymore.
UPDATE 4/03/20: For users who are having the freezing issue when using the Zoom.app, update to the latest version (4.6.9) and you should not see any more crashes.
Disable Zoom’s “Enable hardware acceleration for receiving video” option in the application video preferences. Scroll down and then hit the advanced button.
If the issue is happening in Chrome, some users found success with turning off Use hardware acceleration when available in Preferences > Advanced> System .
Use Firefox instead of Chrome when joining a browser based conference meeting.
Use conferencing in a browser instead of the application. An example of this is zoom. If you cancel out of the constant prompts to download the zoom.app, you will finally get an option to “Join Meeting with your Browser”
If you find any other workarounds please Contact Me
10. If you are seeing this issue, please let Apple know.
The only way to let Apple know that this is a big issue is to file a FeedBack Report. AppleCare Call or an Apple Enterprise Support Ticket.
This will help Apple Prioritize the issue.
11. Conference Software Freezing Issue Links
I created a MacAdmins Chat Channel to disccus the issue.
Bollman – MacAdmins Slack User who did a ton of testing. He also spun up a zoom meeting room where we could all test.
bp – MacAdmins Slack User who’s post get me to take a closer look.
vplc – MacAdmins Slack User who was able to get me info and logs
Georgia – MrMacintosh Reader who was able to quicly answer a bunch of questions along with logs and a sysdiagnose.
Apple Engineer – Who jumped on the issue almost immediately after being invited to the #conf-freezing-issue chat. Gathered logs and FB and Enterprise Support Tickets to help get attention on the issue.
Everyone who emailed me, shared information in Slack, DM’d me or shared my article. Without your help I wouldn’t have been able to put all this information together.
The 10.15.3 Update Erases almost all of the log files in /var/log – System, Custom & Vendor Logs
The 10.15.3-10.15.6 Update Erases Almost all /var/log files
UPDATE: 09/01/20 – The problem is still in the latest build of 10.15.6.
Think about the last issue time that you had an issue and needed to troubleshoot. Right off the bat, you would start looking over the logs to pinpoint the exact point of failure. After installing the Catalina 10.15.3 Update, it’s going to be a little harder to do that. Almost all the /var/log files have been erased and start over the minute after the 10.15.3 update finished installing.
10.15.3 Update Problems
This is my 4th article on 10.15.3 Combo Update issues. If you have not seen them yet, you can view them below.
All log files have the exact “Created” date and time when the 10.15.3 combo update was installing.
Almost all /var/log files erased and have the same “Created time”. The install.log was spared.
What can I do about this?
Let Apple know about this! Hopefully this can be fixed in 10.15.4!
Until then you probably want your log files. The best thing you can do for now is to run a Jamf policy that will backup your /var/log files. We install our updates with a Jamf policy.
Right before we kick off softareupdate -iaR we backup all /var/log files to a temporary directory. We put them back with a LaunchDaemon that kicks off after the combo update reboot.
I hope that these articles have helped you! If you have any questions, leave a comment below or Contact Me.
Boot to Internet Recovery, Recovery Partition or Diagnostics from macOS using nvram or Recovery Selector.app
How to Boot to Internet Recovery, Recovery Partition or Diagnostics from inside macOS.
UPDATE 01/25/21 – Martin Nobel @martinnobel_ – If you want to make an Intel Mac boot into the Startup Manager automatically, type into terminal: “Sudo nvram manufacturing-enter-picker=true”
This means that we can boot to almost every single recovery mode EXCEPT for Internet Recovery!
When an undocumented macOS command or option is discovered, the MacAdmin community gets pretty excited. This is one of those times, as a new nvram key and value was uncovered over the weekend.
If you need to boot to Internet Recovery, you first need to remember the Mac Boot Up Keyboard Combination. Can you remember all of them? I can’t and I work on this stuff every day! Below is the complete list Mac Startup Key Combinations. The second article adds two additional keyboard combinations bringing the total to twelve! The first one will “Reinstall the macOS that came with your Mac, or the closest version still available.” The second command will “Upgrade to the latest macOS that is compatible with your Mac.”
Tim, announcing the first version of Recovery Selector.app that could boot to the Recovery Partition from macOS.
Tim found an undocumented nvram command that you could use to boot your Mac to the Recovery Partition from macOS! From there, he had an idea to create an open source app that would allow you to boot to the Recovery Partition without knowing the exact command.
Someone found additional nvram keys and values!
I replied to Tim’s tweet letting him know that I put in an Apple Enterprise Support ticket to see if we could uncover if an Internet Recovery key existed. Before I could hear back from support, someone found and sent the new key internet-recovery-mode over to Tim. He then improved the app allowing you to boot into four different modes!
Nvram Keys and Values
The commands that we needed are set with two different keys and four different values.
The new values and keys are
recovery-boot-mode = Local Recovery Partition Value
unused Boot to Recovery Partition
internet-recovery-mode = Internet Recovery Value
RecoveryModeNetwork Internet Recovery (Shift-Option-⌘-R)
RecoveryModeDisk Recovery Partition (⌘- R)
DiagsModeDisk Boot to Local Apple Hardware Diagnostic (D)
DiagsModeNetwork Boot to Internet/AST Diagnostic (Option-D)
Let’s put it all together, keep in mind you must run the nvram as an administrator.
All four different nvram internet-recovery-mode values and keys
Your Mac will immediately reboot and start up in Internet Recovery Mode.
Compatibility, Caveats and Requirements
UPDATE 01/27/20
I tested the above commands with the following hardware and OS versions.
2018 T2 15″ MacBook Pro = 10.15.3 Beta 2
2016 13″ MacBook Pro = 10.14.6
2010 13″ MacBook Air = 10.13.6
This should confirm that the command works from 10.13.6-10.15.3 and on Mac Hardware from 2010-2019.
Network Requirements (For Internet Boot Options)
Wired Ethernet Connection
Wifi = Any WPA2 saved connection.
WPA2 Enterprise WIFI is NOT Supported
The WiFi network Internet Recovery will the Top “Preferred Network” listed to boot to Internet Recovery. If for some reason the Mac can’t connect to that network you will be prompted in firmware to connect to a different WiFi network or ethernet network.
Boot Security Requirements
Firmware Password Protection – can be ON or OFF. If ON then you will be required to enter in the firmware password.
Secure Boot – can enabled , the commands works fine.
FileVault – can be enabled, you do not have to enter in your FV2 password.
Restart Requirements
You do not have to reset the boot disk, clear out nvram commands or reset anything. When you restart the Mac, it will boot right back into macOS.
Twocanoes Recovery Selector.app = Easy Mode!!!
Are you going to remember all of the different nvram keys and values?
Probably not
Why not use one simple open source application to do this for you?
Recovery Selector.app menu = easy mode for your test Mac or VM!
Tim had the great idea to take all the above command options and put them into one application. Two clicks gets your Mac rebooted to the Recovery Partition, Internet Recovery, Local HW Diagnostics or Internet HW Diagnostics.
Note: Admin access to reboot is not required, the app uses a LaunchDaemon.
You can download and try Recovery Selector.App below.
We are only missing one critical nvram Internet Recovery Value.
We are only one nvram value away from perfect.
Option-⌘-R
“Upgrade to the latest macOS that is compatible with your Mac.“
I am going to change my Apple Enterprise Support ticket into an Enhancement Request. If this final value is added we will have all 5 boot modes available in macOS.
If you have any questions please comment below or Contact Me!
Here we go again, the Update Keychain Password Function is broken again in 10.15.1.
The AD “Update Keychain Bug” was fixed in 10.15.0, only to be Broken again in 10.15.1.
UPDATE: 03/26/20 – The bug is fixed after installing the Catalina 10.15.4 Combo Update!
UPDATE: 02/03/20 – This bug is still not fixed in 10.15.3! Please contact Apple about this if you haven’t already.
When the issue was first reported to me, I really didn’t believe the bug could be back right after it was fixed. You have to understand my frustration here, I first reported this bug back in 10.14.4!!!
I was disappointed that Apple didn’t fix this bug before the final release of Mojave. Near the end of Mojave, Apple did confirm the issue was fixed in a Beta Build of 10.15.
For Mojave users the fix for the issue would be
Upgrade to macOS Catalina
The bug is back.
During the 10.15 Beta period, I confirmed the bug was fixed in and figured that would be the end of it.
Yesterday, I confirmed the bug is back in 10.15.1.
The Bug is Exactly the same as the 10.14.4 bug.
The 10.15.1 Update Keychain Password bug is the same exact problem as the 10.14.4 issue.
If you change your Active Directory Password off the Mac, you will see the Update Keychain Password Dialog. If you click the 2nd button to UPDATE your login keychain password, the dialog box disappears and a new keychain is created for you. The old Login Keychain is still there but is renamed!
This is what you SHOULD see happen. Once you click the Update Keychain Password button a password box shows up. From here you need to type in your OLD keychain password. Once you do this, your Login Keychain Password is synced up and you are good to go.
Workaround
The good news is, you can remove the “New” keychain and rename your previous login keychain so you can access it again. You can follow the same instructions listed in my 10.14.4 article.
I will submit an Enterprise Support ticket tomorrow morning. If you use Mobile Accounts, I would ask that you do the same to build an impact statement. Please reach out to your SE or if you are a regular user support.apple.com/
Credits
I’d like to give special thanks toMr. Macintosh reader Cesar who first reported this issue.
Mojave 2019-001 Security Update Causing Data Loss if Interrupted – 2019-001 FileVault can’t login
UPDATE: 01/22/20 – This problem was reported as FIXED in the latest Mojave 10.14.6 Security Update 2019-002(18G2022) &Catalina 10.15.2 Update. The fixed T2 BridgeOS version is 17.16.12551. Apple did not list the fix in the 10.15.2 or 2019-002 Security Update notes but DID put them in the AppleSeed 10.15.2 Beta 2 Update Notes. I can’t post them here, but you can check the AppleSeed Patch Notes portal or you can contact Apple Support to confirm if you need further information.
Final Verdict – After further investigation, the problem was NOT the user’s fault. After more users reported what happened during the update I found that the update stalled out during the BridgeOS update phase. The black screen is only supposed to last 2-5 minutes. In reality, the update stopped and the Mac would be on the same black screen for up to 1 hour. The user had NO CHOICE but to shut down the Mac. By that time it was already too late.
Users are Unable to Unlock FileVault 2 if the Mojave 2019-001 Security Update is Interrupted.
This issue was first reported about one month ago on the MacAdmins Slack. It was reported that after some users installed the Mojave 10.14.6 Supplemental Update #3, they were unable unlock their Mac with the FV2 Password or PRK. The issue was not widely reported though so it was thought to be a fluke.
Mojave 2019-001 Security Update
All this changed when the Mojave 2019-001 Security Update was released. MacAdmins started to report the problem again.
I have a Mac that just finished installing the 2019-001 Security Update. I can’t get past the FileVault 2 screen with the password or Personal Recovery Key.
MacAdmin User Report
More and more MacAdmins are starting to report this devastating 2019-001 FileVault can’t login issue.
Who, What, When, Where & Why Index
1. Affected Mac Hardware = T2 Machines
2. Affected macOS Build Versions UPDATE!
3. FileVault 2 Encrypted Machines Only. UPDATE!
4. Evidence? Reports of a Black Screen Followed by User Power Off
(18G1012) Mojave Security Update Released on 10/29/19
(17G9016) High Sierra Security Update 2019-006 – 10/29/19
(17G9016) High Sierra Security Update 2019-005 – 9/26/19
3. FileVault 2 Encrypted Machines Only. UPDATE!*
UPDATE! – We now have two separate reports of this happening when the Mac is NOT FV2 Encrypted.
If your Mac is unencrypted you should be fine.
* I have not seen any reports as of 11/09/19, that include a T2 Mac that was not encrypted.
4. Evidence? Reports of a Black Screen Followed by User Power Off
After the reports started to roll in, we started to investigate. One of the common threads is that users reported a problem with the update during the install.
Black Screen – Users reported that the Mac looked like it powered down. They would try to power it back on, interrupting the install process.
Black Screen with Apple Logo & Progress Bar Stuck – While the Update was installing, some users have reported that the update hung or stalled out. This was followed by a power down.
5. Can’t login with FV2 Password or PRK?
After the user powered down the Mac, they reported the following.
Can’t login past FileVault 2 with my Password.
Can’t boot the Mac up with the PRK.
In this situation the Mac is unable to boot up at all. The only thing that the user can do is boot to the Recovery Partition or Internet Recovery.
After booting to the Recovery Partition we tried to first mount the disk.
This did NOT work!
You can confirm the issue by typing in diskutil ap list
Volume disk4s1
| ---------------------------------------------------
| APFS Volume Disk (Role): disk4s1 (No specific role)
| Name: Macintosh HD (Case-insensitive)
| Mount Point: Not Mounted
| Capacity Consumed: 171872342016 B (171.9 GB)
| Encrypted: ERROR -69808
We would expect that the Encrypted status line should be:
FileVault: Yes (Unlocked)
or
FileVault: Yes (locked)
Note the Encrypted line. It should say LOCKED or UNLOCKED. Instead you get ERROR -69808
xartutil CLI Binary
You can also use the xartutil binary to check for the Encryption Keys.
xartutil --list
You should see 2 entries listed = This is a normal output
Total Session Count: 2
If you see
xartutil: ERROR: No supported link to the SEP Present = Not a T2 Mac
If you see
Total session count: 0 = The Encryption Keys are Lost.
7. Workarounds?
Currently no known workaround is available.
We have tried multiple things.
Mounting the disk in the Recovery Terminal
Mounting the disk in Disk utility
Target Disk Mode
8. Should I block this update?
After Reviewing Multiple Reports, the issue only looks to have affected a small number of users. One MacAdmin said out of 150 machines, the issue only affected 2 of them.
I tested this issue out on a 2018 MacBook Pro.
Powered Off during the first Black Screen.
Powered Off during the second Black Screen
Powered Off during the first Security Update Progress Bar
Powered Off during the second Security Update Progress Bar
The BridgeOS Update and the 2019-001 Security Update installed successfully!
If you move forward with the update you can ask users not to interrupt the install process.
9. Will Apple fix this issue?
The black screen BridgeOS update process has been around since 2018. Something must have changed in the 2019-001 Security Update.
If you have this issue please report it to Apple ASAP.
If you are here for the 10.15.1 issue, you can follow the same 10.14.4 workaround instructions below.
UPDATE: 07/31/19
This will probably be the final update. Sadly the issue is NOT fixed in 10.146. Even worse, this will be the final update and the issue will not be fixed in Mojave. I submitted this issue right after it was found in 10.14.4 and it’s just a bummer that this will never be fixed in Mojave. The only good news I can give you is that this is fixed in macOS Catalina 10.15.
UPDATE: 07/09/19
The AD Mobile Account option to “Update Keychain Password” when resting your password outside the Mac is still broken in macOS Mojave 10.14.5. This issue is still not fixed in current 10.14.6 Beta! Be sure to contact Apple if you haven’t already done so!
10.14.4 Update password fixes/problems
I really like the 10.14.4 update, trust me I do! It arrived with so many fixes that have really helped MacAdmins. The problem is, it also broke a few things. Just when I thought we found all the fixes/problems a new one pops up. If you have been following along, this is now my 4th article on password fixes/problems in the 10.14.4 update. Lets quickly review
10.14.4 Update breaks “Update Keychain Password” process for Ad Mobile Accounts.
This issue affects Active Directory Mobile Account users. If you use Mobile Accounts you have seen this message before.
Which option should I pick? In 10.14.4 it doesn’t matter they both create a new keychain!
You will only see this message if you change your Active Directory Password outside the Mac. An example of this would be if you changed your AD password on a 2nd Mac, Windows PC or Web Portal. Logging in with the new password will sync that new password down to the Macs local cache but can NOT change the keychain password without the OLD password. You can click “Create New Keychain” and brand new login keychain will be created. But what if you have Xcode Developer Certs and Private keys or Wifi certs? In this case you need your old keychain intact.
Clicking “Update Keychain Password” just creates a new login keychain.
If you click “Update Keychain Password” you should see this. (10.14.0-10.14.3)
What you should see in 10.14.0-10.14.3
Instead, after clicking the update button you will not see this message and you are now at the desktop. If you open up keychain access you will see that your login keychain was wiped out.
Workaround – Find renamed keychain, change password and restore.
Good news, I have a workaround for you. The old login Kkeychain luckily still remains in ~/Library/Keychains
We will have to perform a few steps to restore your old login keychain
1. Find renamed keychain – located in ~/Library/Keychains and called login_renamed_1.keychain-db
2. Change password of login_renamed_1.keychain-db from old to new
3. Remove login.keychain-db
4. Rename login_renamed_1.keychain-db to login.keychain-db
5. Restore login.keychain-db to Keychain Access.app
6. Log out and back in.
1. Find renamed keychain
The old keychain is located in ~/Library/Keychains and called login_renamed_1.keychain-db
Your old login keychain and the new one that was just created.
2. Change login_renamed_1.keychain-db password
You used to be able to change the login keychain password through Keychain Access. This is no longer possible.
What if we clicked “Add Keychain” and tried to add the renamed keychain then try to change the password?
This looks promising but after clicking “change password for keychain login_renamed” nothing happens. I then tried to unlock it with the old password.
After unlocking I attempted to change the password again.
Still no go! After clicking change password nothing happened. At this point, I thought I was out of luck.
Enter CLI command security
Never give up a fight without visiting the Command Line Interface! The CLI can be your best friend. Let’s take a look at the security man page and see if anything will help us. Open terminal and type in man security
set-keychain-password Set password for a keychain.
Oh ya, now we are talking! Let’s take a look at the options.
Perfect, just what we are looking for. Lets try it out.
You will be prompted for your old and new password. Now that the old keychain has the same password as your AD account we can move it back into Keychain Access.app.
3. Remove login.keychain-db
Now we can just delete the empty login keychain.
Right click on login and select Delete Keychain “Login” then click “Delete References & Files”. You should now only have Local Items, System and System Roots.
4. Rename login_renamed_1.keychain-db to login.keychain-db
We now need to rename login_renamed_1.keychain-db to login.keychain-db. You can either do this in keychain access or in the finder. Let’s rename in the finder. Click once on login_renamed_1.keychain-db and change it to login.keychain-db.
5. Restore login.keychain-db to Keychain Access.app
Now all we need to do is add our old keychain back to Keychain Access.app. Right click in the keychain section and select “Add Keychain”.
Navigate to ~/Library/Keychains/login.keychain-db and select it. You will now see login in the keychain box! At this time it will be locked. You can test unlocking it now. Right click on login and select “Unlock Keychain login”
You will now be prompted to enter in your current password.
6. Log out and back in to confirm
You have now restored your old keychain. Log out and then back in to confirm. You are now good to go!
As always, we need to submit a bug report to Apple.
I can not stress how important this is. The more reports we put in the higher priority the issue gets. We are also running out time and only have about 3 weeks before 10.14.5 is released.
Thanks to hawkzhang45 from JAMF Nation forum for calling this issue out. Also to m.entholzner for conformation and submitting an Apple Enterprise Ticket. You can read the original thread here.
10.14.4 Update breaks local account password reset when using FileVault Recovery Key.
UPDATE: 05/16/19 – 10.14.5 Update fixes this issue
The macOS 10.14.5 update fixes this issue.
As noted above this issue is now fixed in macOS 10.14.5. You can read on if you are interested in how this all went down.
I have been testing the new password fixes/changes in macOS Mojave 10.14.4. You can see the changes in the “What’s new in the updates for macOS Mojave” support document. What I found was, the 10.14.4 Update breaks local account password reset when using the FileVault Recovery Key.
Enterprise Content section from the what’s new in 10.14.4 update document.
I wrote about how Apple fixed mobile password syncing issues on how 10.14.4 fixes Mobile Account Password syncing issues in 10.14.0-10.14.3. This was a huge win for Active Directory Users. We finally have a functioning password change system in place. I found this problem while testing these new fixes. Instructions for this procedure are listed in this Apple Support Document.
Let’s confirm this on 10.14.3 and 10.14.4
I setup a fresh 10.14.4 (18E226) system, created a local account and then enabled FileVault. I then performed the following test.
Boot system – Select user
Click the ? Button so I can enter the recovery key.
The system will now boot to the login window
You will see the username filled in with your username with the password reset window.
Type in a brand new password and then hit “Reset Password”
The window thinks for a second then shakes you off.
The password is not changed.
Reset password window after entering in the recovery key.
Performing the same test on 10.14.3 (18D109) worked as designed. After clicking “Reset Password” the system accepts the new password then logs you in.
Workaround: resetpassword in Recovery
Good thing is, the resetpassword application in the recovery partition still works.
Trusty ole resetpassword still works.
1st way to reset your password. Boot to Recovery
Boot your Mac holding Command R to boot the Mac into the Recovery Partition. Once in click Utilities from the Menu Bar then select Terminal. Once in type in resetpassword, then follow the instructions.
Note: If you have a T2 Mac, this option requires that you have a SecureToken Admin on the system to access the Terminal.app.
2nd way to reset your password, the FV2 Screen.
You can trigger the 2nd way at the FV2 login window.
Wait up to a minute at the login screen, until you see a message saying that you can use the power button on your Mac to shut down and start up again in Recovery OS. If you don’t see this message, FileVault isn’t on.
Press and hold the power button until your Mac turns off.
Press the power button again to turn on your Mac.
When the Reset Password window appears, follow the onscreen instructions to create a new password.
If you would like to follow Apple’s instructions on how to reset local account passwords you can visit this Apple Support Article.
“Radar or it didn’t happen”
This was a really great quote from Jason Broccardo @zoocoup. Filing bugs and tickets is a really important task for MacAdmins. Apple rates issues by the number of reports/tickets they get for each issue. If this feature is important to you please do the following.
Then open up an Open Radar on openradar.appspot.com. This will help with tracking and you can let others know about the issue. (This site is not affiliated with Apple Inc.)
In this article, I will talk a little bit about the current state of Apple’s Documentation. After that, I will show you 3 Undocumented 10.14 Mojave fixes that can help you as a MacAdmin.
Documentation, Documentation, and Documentation. Say it three times fast! MacAdmins just want Apple to provide proper documentation for features, controls and security settings and Enterprise Fixes. In some cases, Apple provides excellent documentation. An example of this would be the T2 Security Chip Security Overview released in October of last year. In other cases when it comes to binaries like sysadminctl not so much.
I tried to searching for something that specifically mentions SecureToken or sysadminctl and came up empty.
The best that I could find was a document called “If you see authentication server errors when turning FileVault on in macOS High Sierra“. This article does not even mention SecureToken. You can get a few nuggets of information by checking the sysadminctl binary options but sysadminctl doesn’t have a man page. I even performed a search on developer.apple.com/documentation as you can see in the picture above. I will be writing about sysadminctl next week. Maybe I can create a MacAdmins version of a sysadmincatl man page! Yet when I search for “SmartCard” three documents show up. SmartCard support is a small piece in the overall macOS pie, yet has multiple documents! Side Note: Shout out to all my peeps in the MacAdmins.slack.com #SmartCard channel (about 5 people) 🙂
Documentation is getting better.
If you have been keeping track, Apple documentation is getting better. If you look at the “What’s new in the updates for macOS Mojave” page you will see a large number of fixes. Eagle eye MacAdmins will be first to spot “Enterprise Content”, this is the stuff MacAdmins are interested in.
10.14.2
10.14.3
10.14.4
Check out that first one under 10.14.4! As noted in my previous article, I fought to get that one fixed since 10.14.0. It’s really great to see that fix get mentioned in the Enterprise Content area.
What do you mean undocumented fixes ?
Apple is constantly fixing things behind the scenes. MacAdmins continue to file radars, call Apple Care, test beta releases, submit feedback and submit Apple Enterprise Support tickets. Defects and bugs ARE getting fixed but are not listed in Apple’s Enterprise Content listing. I am not totally sure why certain fixes do not make the list.
Maybe Apple wants to keep the list short while focusing on the major fixes. I wish Apple would list more of them, even if they posted them in an enterprise only area. An example of this would be AppleSeed for IT. If you are part of an Enterprise or School you can be selected to join the program. I highly recommend joining if you are not a member already. You can read the FAQ about joining eligibility here. Inside you will find links to macOS beta downloads and beta documentation. Each beta release (Sometimes up to 6 releases per combo update) will show what has been fixed between updates. This is great information for any MacAdmin to have so you can stay on top of what’s going on.
3 Apple Enterprise fixes included in 10.14.0 – 10.14.4
1. macOS 10.14 Mojave can now provide FV2 Authenticated Restarts for Combo and startOSinstalls.
In 10.14 macOS Updates and Upgrades are now able to perform Authorized Restarts. This feature was not an option in previous releases. This is a pretty big deal, especially for #MacEDU and Enterprise customers who have computer labs.
Previously if you installed a macOS update and the system was FV2 encrypted it would restart but STOP at the FV2 unlock screen. If you performed this update remotely you would lose control of the machine. Things get worse at FV2 login window because firmware will shut the Mac down after 5 minutes of inactivity. The same problem will happen when you start a macOS Upgrade. You will be disappointed after returning from lunch thinking the update is complete only to find the Mac turned OFF. You then power the Mac back on only to find the installer has just started with 40 minutes remaining. With 10.14 if you kick off a combo update or macOS upgrade the installer will perform an Authorized Restart and you will never get stuck at the FV2 prompt again!
For startosinstall you just have to store the mojave.app in a folder like /Users/Shared. Then kick it off with this command – sudo /Users/Shared/Install\ macOS\ Mojave.app/Contents/Resources/startosinstall –nointeraction The –nointeraction option will prevent license agreement message.
2. Installing software updates using the -R restart option at the login window now properly restarts the Mac to the installer. (10.14.4)
When Apple released the T2 security chip they also added additional options to the softwareudpate binary so it could handle BridgeOS updates. Installing a combo update on a T2 Mac is now a multi-step process. Using softareupdate step one remains unchanged, it will download the combo update from Apple which in turn stores in /Library/Updates. For step two, the Mac reaches out to Apple’s personalization service (gs.apple.com) verify the BridgeOS and combo update. When the verification is complete you will have a new folder in /Library/Updates called PersonalizedManifest.
You are automate the entire process by using sudo softwareupdate -iaR. Options -i will install the update, -a will download all updates and -R will perform an automated restart. The process works just fine if you are the logged in user. If the system needs to update the BridgeOS the Mac will shutdown and then will power back on with the T2 Chip to install the BridgeOS update. If the system does not require a BrigeOS update the system will restart to the update installer. The problem comes in if you try to automate the install from the login window using the softwareupate -R or –restart option. Softwareudpate will run run through the process listed out above only to stop at the very end and be unable to restart.
Looks great until the very end, when at the login window the system will NOT restart!
Once all your Macs are updated to 10.14.4, you can now use the -R restart for all situations. Softwareupdate can now restart the Mac if it’s at loginwindow.
3. 10.14 FV2 Authorized restarts can use the PRK (Personal Recovery Key) again.
When 10.13 arrived you could no longer perform FV2 Authenticated restarts using the PRK (Personal Recovery Key). This feature was just flat out broken. This previously worked in 10.12 Sierra and below. NOTE: You could still perform an Authorized restart with your FV2 name and password. An example of a PRK Authorized restart would be if you are a JAMF Pro customer and had a policy that installed a package but it also required a restart. You could select the option “Perform Authenticated Restart” Jamf would then send a fdesetup authrestart using the PRK. The package would install and then the system would perform an FV2 authorized reboot so the user did not have to enter in the password at the FV2 unlock screen.
10.12, 10.11 & 10.10 – Works!
sudo fdesetup authrestart = Enter a password for ‘/’, or the recovery key:
10.13 – Doesn’t work
sudo fdesetup authrestart = Enter the user name: ( hit the enter key to toggle Recovery Key Entry) = Error: Missing user name. Error: Unable to restart (error = -54).
10.14 – Works again!
sudo fdesetup authrestart = Enter the Username: (again hit the enter key to toggle Recovery Key Entry) Enter the current recovery key:
I hope that at least one of the fixes I mentioned in this article helps you. In the future I would love to see more documented Enterprise fixes listed in the combo update patch notes. Until then though, I will continue to document said fixes and let you know about them when I can.
Before we get started I’m am going to talk a little bit about how macOS and Active Directory work together. I will also go into some history behind the built-in AD Connector. In the end, I will explain the current problems we are having with Active Directory Mobile Account password syncing and how Apple fixed the issue.
If your company or school uses Active Directory, you most likely use Mobile Accounts. To get Mobile Accounts to work you first have to bind the Mac to Active Directory, once bound the Mac is now trusted. You can now log in with any Active Directory user and access to Global Groups, Kerberos and Directory Contacts.
Sounds great right? It was!
Once you log in, the system caches your AD account to the local directory. If you then disconnect the Mac from the network you can still log and continue to work. When the time comes to change your AD password you could change it on a 2nd Mac, a Windows device or even a Web Portal.
How did the AD password change work?
If you changed your password on the Mac it would first check if any password requirements are set at the domain level. If you passed the requirements the password would be immediately changed in Active Directory. The password would then change at the local offline level of your Mac. If you changed your password outside the Mac (Web Portal etc..) the system would receive the password change the next time you connected the Mac to the network and logged in. You would then be promoted to Update or create a new Login Keychain.
Active Directory & FileVault 2
The AD password change system changed in 10.7 with the addition of FileVault 2. Now when you changed your password an extra step had to be performed. Once the password was changed in AD it would then change the locally cached password and then had to sync that password down to your FV2 account. When you turned on your Mac, you could then use the same password as your AD account to unlock the volume and start booting the system. The AD password sync system worked pretty well from 10.7 all the way up to 10.12 Sierra. Users would sometimes have issues here and there when the Mac dropped off the domain but usually a rebind and would save the day.
10.13, APFS and SecureToken
Apple introduced the next-generation file system called APFS (Apple File System). We first got to test it out in 10.12 Sierra in beta form. APFS was standard for all SSD drive installations on 10.13 High Sierra installs. You could still opt out with commands and spinning hard drives would still use HFS. When 10.14 arrived APFS was standard across all hard drives. The introduction of APFS brought an added undocumented security feature called SecureToken. If you wanted to enable FileVault 2 you had to have SecureToken enabled for said account. You could no longer you use the PRK (Personal Recovery Key) or even a local admin to add extra users to unlock the volume like you could with HFS. You have to grant the 2nd user a SecureToken before they could become an authorized CryptoUser. To get the token in the first place you had to be the first user logging into macOS at the SetupAssistant.
10.13 was the start of syncing issues for Mobile Accounts.
The main problem with this new system was that the SecureToken system was not tested. Mobile account users, in particular, had nothing but problems with this new system. From 10.13.0-10.13.3 AD Mobile Account password syncing to FV2 flat out did not work. During this time frame we had multiple high priority tickets in with Apple Enterprise Support. When 10.13.4 hit Apple finally fixed the issue and password syncing started to somewhat work again. I say somewhat because everyone still reported issues but at least it worked SOME of the time now. 10.13 still had problems when you changed the password off the Mac.
Enough of the history lesson, get to the 10.14 problem!
Once 10.14 hit we were hoping that the problems we had on 10.13 Mobile Accounts would fixed. Unfortunately, we were wrong, way wrong. The problem worse when 10.14.0 released. How could it be worse than 10.13 and how did miss the problem in 10.14 beta? On 10.13 as I mentioned above we had to deal with of Mobile Account syncing of FV2 passwords. The user would change the password outside the system and it would not make it down to FV2. The good thing is we actually have fix for this!
The following instructions will change or Sync the password for the CryptoUser account that belongs to the AD user. This will only work if the user KNOWS the old password since the command will prompt for the current password.
1. diskutil apfs list (Grab the disk label for Volume Macintosh HD usually disk1s1)
2. sudo fdesetup list -extended (grab the UUID of the OS USER) You can also use diskutil apfs listCryptoUsers /
3. diskutil apfs changePassphrase disk1s1 -user UUIDhere (add in the UUID of the OS USER from the pevious command and put it in UUIDHere.
That’s cool, but you still haven’t told us why is the situation worse in 10.14?
The local cached offline password is never changed!
The problem is the issue is undetectable UNTIL the user attempts to authenticate OFF the network. When they try the current password it will NOT work. They will then have to call the helpdesk, who then changes the AD password making the situation even worse. This is the situation for 10.14.3 and below.
When connected to the network = Current AD Password works!
When Disconnected from the network = ONLY previous password works.
FV2 password = Previous Password.
Things get even more annoying is if the user actually uses the old password to authentiate the Screen Saver while offline. The system will accept the password but then immediately prompt the user to unlock the Login Keychain. This is due to the keychain being set to the current AD password. You would be in a never-ending keychain password cycle.
Do you have a fix for the offline sync issue?
The good news is we do, as long as you have a SecureToken enabled Admin user. All you need to do is turn off SecureToken and then turn it back on. Something in this system will then sync the offline cached password. Shout out to @annemacro on MacAdmins Slack for figuring this out!
Now that 10.14.4 is out the password sync mechanism now working. As long as you update systems to 10.14.4 before users change their AD password they will not have this issue going forward. I have not had the chance to actually test the 10.14.4 update on a system that is already out of sync. The good news is that even if it doesn’t fix the issue when it is already happening you now have the tools to fix it yourself. The next time the user changes their password they will not experience the issue.
Why did this take so long to fix?
The answer to this question is pretty simple. Everyone missed the bug from beta 1 all the way through into 10.14.0. I performed hours and hours of testing in beta. I was so concerned about that the FV2 password did not sync that I did not even think to test the offline password. Even worse, neither did anyone else including Apple. It was not until around 10.14.2 when I had an enterprise ticket in with Apple and finally got a response.
“This bug will not be fixed until the next release of macOS”
When I read the reply to this support ticket I was in complete shock. Are you going to tell me that we are going to flat out have a NON FUNCTIONING MOBILE ACCOUNT SYSTEM FOR THE ENTIRE 10.14 RELEASE? I could not believe it. The Apple Enterprise Support Engineer I was working with also agreed and he was fantastic to work with and helped work through the issue with me. At this point what else would a #MacAdmin do but RANT in MacAdmins Slack!!! The best way to do this is to document and explain the issue to others. You can then rally other MacAdmins to file Enterprise tickets or Radars. This will draw attention to the issue and generate heat inside Apple Enterprise Support. In the end that’s exactly what happened. We were not the only company that Used Mobile accounts. Those same companies let Apple know that we needed a fix ASAP. Apple realized this was important and fixed this issue for us in 10.14.4. (Thank you Apple!)
Where does that leave Mobile Accounts?
Before 10.13 Mobile Accounts worked very well. We had thousands of Macs connected to AD utilizing Mobile accounts and did not have any issues. Once 10.13 hit things started to go downhill. The problem is, it seems like Apple is not spending enough time on Mobile Accounts. The MacAdmins community has started to realize this and starting at the end of 2017 and into 2018. This began what I call “The great Mobile Account exodus to Local Accounts”. NoMAD and Enterprise Connect make using local accounts while still having the ability to use AD resources easy. Mobile Accounts still serves it’s purpose but it seems the writing is on the wall.
Thanks
If you stuck around to read the entire article I really do appreciate it. If you are at WWDC 2019 or JNUC 2019 I will buy you a beer or non-alcoholic beverage. Just mention coupon code #ISURVIVEDMOBILEACCOUNS.
I hope to write many more articles like this in the future. Over the past 15 years MacAdmins have helped me get to where I am today. I hope I can give back to the community and help the next generation of MacAdmins rise through the ranks! Drop me a note at com gmail mrmacintoshblog
