Rule 3: Patch Management is mandatory!
What can you expect from this document:
This document covers some basic security best practices that, if combined with other security measures, will help to increase the overall security of your system.
What is the threat?
If security patches are not applied regularly and in time, attackers can leverage known vulnerabilities present in your software to compromise your system. Therefore it is highly recommended to enable automatic updates or manually apply security patches as soon as possible to protect your server from well-known attacks.
10 Best Practice Rules for Patch Management
Patch management is a critical part of maintaining the security of your systems and network. The patch management system that you build and maintain is, among other things, the channel through which you deploy security updates from Microsoft and other vendors. Although patch management is sometimes viewed as a systems management discipline, rather than a security discipline, its role in addressing vulnerabilities through the deployment of updates makes it a vital component in an organization’s security operations. Because the timely application of security updates is one of the most important and effective things you can do to protect your systems and network, your patch management system must be as efficient as possible.
To help customers develop and maintain efficient patch management strategies, Microsoft provides information about tools and strategies on our patch management page on the TechNet Security site (http://www.microsoft.com/technet/security/guidance/patchmanagement.mspx). There, you will find a wealth of important information on the nuts-and-bolts aspects of building and maintaining a patch management system to support Microsoft products. This site is an excellent and valuable resource. In helping customers with questions and concerns around security updates for some years, I have found that they don’t always make clear why we make particular recommendations. We have provided good resources concerning the practice of Microsoft patch management, but we haven’t provided in-depth information about Microsoft patch management.
In this article, ten principles of Microsoft patch management will be outlined. With a better understanding of these principles, you can better align your patch management strategy with Microsoft, and thus improve the efficiency of your patch management system. You can also prevent unpleasant surprises that can result from pursuing a strategy or tactics that Microsoft doesn’t recommend or support.
1. Service packs should form the foundation of your patch management strategy Microsoft recommends viewing the service packs as the primary vehicle for security maintenance and to look at security updates as something that augments service packs.
Security updates are comprehensive for the vulnerabilities they address, and they are only released when they reach an appropriate level of quality. But service packs are a broader vehicle, both in the scope of the updates they contain and the testing process they undergo. A service pack includes all the security updates made available for that product before its release. The service pack also contains other updates and improvements from the ongoing work of code maintenance that individual security updates may not contain, so it always offers the overall protections you may need for you environment.
Your patch management strategy should focus first on service packs and then on security updates, rather than the other way around.
2. Make Product Support Lifecycle a key element in your strategy Directly related to the central role of service packs should be the role of the Microsoft Product Support Lifecycle in your patch management strategy.
Like all technology companies, Microsoft products follow a timeline of lifecycle support. For several years now, Microsoft has worked to make this process as predictable and transparent as possible by developing and posting information about our Product Support Lifecycle at this site: www.microsoft.com/lifecycle.
From a security point of view, one of the most important things Product Support Lifecycle governs, is the timeline for how long security updates for a particular product will be made publicly available. When a product is no longer publicly supported under the Product Support Lifecycle, Microsoft no longer publicly provides security updates for that product.
The Product Support Lifecycle is also relevant to our first principle regarding service packs. Security update support for products is specific to particular service packs for that product. This means that for as long as a product is publicly supported, security updates are made available only for those specific service packs of that product; updates are not made available for service packs of products that are no longer publicly supported. The Product Support Lifecycle Web site provides timelines for specific service pack support for publicly supported products.
By keeping up to date with service packs, you ensure that your environment is always in conformity with the Product Support Lifecycle. Doing so means that you are never in a situation where security updates are released and you have no information about the vulnerable state of your environment in the security bulletin because you’re on an unsupported version.
To make your patch management strategy most effective, then, you should integrate the timelines from the Product Support Lifecycle into your patch management strategy.
3. Perform risk assessment using the Severity Rating System as a starting point Microsoft Security Bulletins serve to make you aware that security updates are available for specific Microsoft products. Another goal of the bulletins is to help you with understanding issues in the security bulletin so that you can perform a risk assessment of the issues in your environment in accordance with your organization’s security policies. Risk assessment is an important step in the practice of patch management because it helps to answer questions relevant to the prioritization, testing, and deployment of security updates.
To help customers with risk assessment, Microsoft Security Bulletins use a Severity Rating System. With it, we evaluate each issue and quantify the issue’s impact objectively on a technical level using criteria that we have publicly posted on this site: http://www.microsoft.com/technet/security/bulletin/rating.mspx.
In the “Executive Summary” of each Security Bulletin, you will find a table that lists each vulnerability addressed in the bulletin. It also lists a severity rating for that vulnerability for each product listed as affected. For bulletins that address multiple vulnerabilities, a maximum severity for all vulnerabilities is provided for each product listed as affected in the bulletin. Finally, for ease of reference, a maximum severity for all vulnerabilities for all products is provided in the Summary at the top.
In addition, more technical information about the vulnerabilities addressed in the bulletin is provided in the Vulnerability Details section.
This information in the security bulletins is intended to support the risk assessment processes and procedures that customers have implemented as part of their patch management strategy. Microsoft provides assessment and guidance regarding the issue that you can use as a starting point to make your own risk assessment of the issue in your individual environment. Because the Severity Rating System evaluates the issue solely on technical grounds, it cannot account for specific aspects in customer environments. Therefore, it’s important to use the Severity Rating System as a starting point for your own risk assessment that can evaluate those elements specific to your environment.
4. Use mitigating factors to determine applicability and priority Provided for each specific vulnerability in the “Vulnerability Details” section of the bulletin, the information about mitigating factors explains how the impact of the vulnerability may be lessened or mitigated and is important for your own risk assessment in understanding the applicability, scope, and threat of the issue in your environment.
Mitigating factors are used in the Security Bulletins as part of the criteria in determining the severity under the Severity Rating System. For example, a mitigating factor where a particular vulnerable component is disabled by default will result in a lesser severity rating than if that component were enabled by default. One of the goals in explicitly calling out these mitigating factors is to help you understand why an issue has been rated with the severity that it has to help you with your own risk assessment process. Continuing the example, if in your environment you’ve chosen to enable the particular component by default, you will want your risk assessment to reflect that fact.
Ultimately, you should use information about mitigating factors first to determine the applicability of an issue for your specific environment. If it is applicable, you then should incorporate the information about mitigating factors into your risk assessment for prioritization of the deployment of security updates.
It is important to point out that information about mitigating factors is never meant to justify neglecting a security update. Microsoft recommends that you always apply all relevant security updates. You should look at mitigating factors to determine when to apply the security update,not if you should apply the security update.
5. Only use workarounds in conjunction with deployment As part of the process of investigating reports of vulnerabilities, Microsoft tries to identify workarounds that are applicable to help protect against attempts to exploit the vulnerabilities that are being addressed by the security update. If viable workarounds can be identified, they are published in the Vulnerability Details section of the bulletin. In case that a viable workaround cannot be identified, this information is also published in the Vulnerability Details section.
The goal in providing workaround information for specific vulnerabilities is to give you an option that you can use to protect your environment right away, while you put the security updates through the appropriate testing before deploying them broadly. Just as mitigating factor information is never intended to justify neglecting security updates, workaround information is provided as an interim measure until you apply the relevant security updates. In your patch management strategy, workarounds should be viewed as being closely tied both to your risk assessment and your deployment processes and procedures. You should consider implementing workarounds immediately for issues that you identify as a high risk to your environment in order to provide protections while you are deploying the updates. Issues for which there are no workarounds available may merit increased priority for deployment.
ssues with Security Updates are documented in the Security Bulletin Master Knowledge Base Article A very common question from customers after the release of security updates is, “Are there any known issues with the security update?” All security updates go through an extensive testing process and are only released when they meet an appropriate level of quality, but as part of the risk assessment process administrators often want to identify any known issues.
On those rare occasions when an issue occurs after the application of a security update, that issue is documented in a Knowledge Base article by our Product Support Services. Any Knowledge Base article about a security update is also documented in the “Master Knowledge Base Article” for that Security Bulletin. Each bulletin has a Master Knowledge Base Article associated with it that is referenced in parenthesis after the bulletin’s title. Also, the “Caveats” section in the Summary is updated to call attention to the inclusion of information in the Master Knowledge Base Article.
To see whether there are any known issues with a security update, look at the “Caveats” section of the bulletin. If there is no listing, take the Master Knowledge Base Article number from the security bulletin and view the Master Knowledge Base Article on the Microsoft Product Support Site (http://support.microsoft.com). If there is no information there about known issues, it means that no issues have been identified with that security update.
7. Test updates before deployment Testing security updates before deployment is an industry-recognized best practice. Proper testing of security updates allows you to understand the possible impact of the security update on your specific environment. You can factor that understanding into your risk assessment to prioritize the deployment of the security update.
Just as your testing process should influence your risk assessment, your risk assessment should influence your testing process. Security updates that you identify as being particularly critical to your environment may merit abbreviated or expedited testing. A security update addressing an issue that you assess as low risk, but that updates critical systems instead, should be tested extensively to allow broader testing.
What should testing entail? Unfortunately, the correct answer is always going to be specific to your circumstances. The appropriate testing environment and procedures will depend on the production applications that you run, the type and mixture of system types, and the resources that you can afford to dedicate to testing. In general, though, you want to balance your costs and resources devoted to testing against the costs and resources potentially incurred by deploying an untested update in your environment.
8. Contact Microsoft Product Support Services if you encounter problems in testing or deployment While testing, it is important to know what to do when you identify a possible security update issue in your environment. Microsoft strongly recommends to contact the Product Support Services before you do anything.This is the most efficient way toidentify real security issues.It is possible to gather the required technical details, only if you work directly with the Microsoft Product Support Services. Microsoft can use this information to help you address any real issues in your environment.
The Product Support Services is only able to identify and document the known issues in the Master Knowledge Base Article, if the customer contacts them. . If a customer tries to solve the problem withough calling Support, this will likely delay the resolution.An important thing to remember is that Microsoft provides no-charge support for issues related to security updates. You can get in touch with Microsoft for security bulletin support through the Security Support Site at https://support.microsoft.com/securityitpro.
9. Use only methods and information recommended for detection and deployment Detection and deployment of security updates is a critical part of the patch management process. It’s the application of the security update, that addresses the vulnerability discussed in the Security Bulletin.So, it should always be the final step in your patch management process. The detection and deployment of security updates are sometimes handled by a group that does not read Microsoft Security Bulletins and is not aware of the specific information provided there. This can lead to the use of methods that have not been recommended in the Microsoft Security Bulletin.
Part of the process of building and releasing security updates is developing and verifying information for each security bulletin around detection and deployment. With that information, customers can identify systems that the security updates apply to and verify that they have been installed properly. This information around detection and deployment is included in the Security Bulletin under the “Frequently asked questions (FAQ) related to this security update” and “Security Update Information” sections. This critically important material represents the methods and information around detection and deployment that Microsoft has tested and verified to be accurate. Because any other methods or information have not been tested, Microsoft cannot guarantee that they will be accurate or reliable.
Problems concerning detection and deployment directly affect your ability to successfully apply and manage security updates. So you may only use methods and information recommended in a bulletin for detection and deployment. If you are not currently using information from the security bulletins, you should immediately investigate implanting something like Microsoft Security Baseline Analyzer 2.0, Windows Software Update Services (WSUS) or Systems Management Server 2003 to support detection and deployment. If you are using another patch management system, ensure that it uses methods and information around detection and deployment included in the Security Bulletin.
10. The Security Bulletin is always authoritative Any time there is any question about information related to a Microsoft Security Update, the place you should go for an answer is the Microsoft Security Bulletin that accompanies that security update. Information is entered into the bulletin after it has been verified for accuracy, so you know that information in a security bulletin is authoritative. In addition, Microsoft will always put important information that is relevant to a security update in the Microsoft Security Bulletin, so you know that if you need to find information that is important about a security update, it will always be present in the security bulletin.
Sometimes Microsoft gets new information that is relevant to a security update after it has been released. When Microsoft gets new information that is relevant to a security update, the information is added to the bulletin. All changes to the security bulletins are documented at the bottom of the bulletin and include the date and the nature of the change. Anytime a bulletin is updated, the change is announced to the Security Notification Service, Comprehensive Edition (SNSCE). This list will notify you of all changes to security bulletins. If you want to keep up to date on changes to Microsoft Security Bulletins, you can register for SNSCE on the notification page: http://www.microsoft.com/technet/security/bulletin/notify.mspx.
Microsoft Security Bulletins are the single, authoritative source for the important information you need about Microsoft Security Updates.
Guidelines for critical Servers Software system security has come to depend on customer information technology (IT) organizations closely monitoring patches for vulnerabilities, and on the ability of those organizations to test and deploy the patches before they can be exploited. While, on one hand, many IT organizations are measured, formally or informally, on the uptime of the systems they maintain, many software patches are not effective until the affected computer has been rebooted. Rebooting a business-critical computer decreases the system's uptime and introduces the potential for longer service outages if incompatibilities are introduced. This article provides three rules on how to manage patches to reduce downtime while still maintaining the security of software services through the proactive reduction of dependencies and the use of workaround solutions.
Before any patch can be evaluated, you must recognize the existence of a relevant patch. That can be a problem for organizations with many software packages on a large number of computers. So, it is critical to have a list of installed software and the computer systems where they are. Patches can cause two types of disruptions. First, applying a patch can potentially cause servers to reboot. Second, the patched software might have different behavior that causes application incompatibility problems. Because you cannot know of either of these conditions in advance, it is necessary to use a test server, configured exactly like a production server, to test the impact of the patch.
Rule 1: Check patches. You can use the following process to analyze a patch that is recommended by a security bulletin:
- First, determine whether the patch applies to code that could be executed on the server. Note that any code installed on the server could be exploited if it ever executes, even if it is not essential to the operation of the services. For security reasons, if the code is installed on the box, it must be patched.
- Next, identify whether the patch requires a reboot. Typically the vendor's documentation for the patch will specify that a reboot is required if the software engineers cannot guarantee that a reboot will not be required.
- Finally, track the suggested workaround solutions to see if they will safely mitigate the vulnerability without compromising business-critical services.
With this process in mind, try the patch on a test server. If the patch can be installed without a reboot, chances are good that it will install on running servers with no reboot as well. It is still a best practice to take one production server out of rotation and then test the patch to be sure that the patch is applied properly. Once you have verified the patch, you can apply it to all servers.
Practices to Avoid Downtime ou can avoid downtime by limiting the code that actually runs on business-critical machines during normal operation. No production technology has yet been able to reliably patch executable code that is loaded into computer memory and running. That implies that any patch to running code will require the code to be stopped and restarted. In the case of the operating system, that means a reboot.
To understand the impact of patching on server downtime, analyze all affected Microsoft Security Bulletins for Windows Server 2012 (R2). The bulletins were separated into those that did not require rebooting the operating system to complete the patch, and those that might require reboots specifically for server installations. It was apparent that only a small number could be installed with no fear of a reboot. For the remainder of the bulletins, there are methods that can limit the number of required reboots.
It's a useful best practice to think of reboot reduction using the 10-80-10 rule. The data from Windows Server 2012 showed that patches not requiring a reboot are about 10 percent of the total. Evidence shows that another 10 percent of patches apply to the Windows Server kernel and thus require a reboot unless there is a workaround. The remaining 80 percent will only require reboots if the specific executable code being patched is running when the patch is applied. Paying attention to this last class of patches can help increase the time between reboots and even improve the overall security of the server.
Rule 2: Wherever possible, ensure that only essential code is running during normal operation. When building the server operating system image, do not include any role that is not absolutely essential. If possible, configure the system to avoid starting services automatically unless their use during normal operation is certain. The less code is running, the smaller is the attack surface exposed, and the less likely a patched executable will force a reboot.
How to Extend Operating Time Without Reboots
Rule 3: Look for a workaround. If the patch requires a reboot, review the patch documentation for a workaround that can be safely used, for the near-term, in lieu of the patch. For longer-term maintainability, it is recommended toinclude the deferred patch in the next patch cycle for which you determine that a reboot is required.As an alternative, you can wait to apply the patch until it can be installed together with a new operating system image (e.g. with a new service pack). This brings us to the final topic: You can prevent a reboot:
- By reducing the number of running code dependencies, and
- by using workaround solutions instead of the patch.
As stated above, this two-prong strategy applies to a high percentage of released patches. There are two additional specific steps you can take to reduce the number of running code dependencies on a server:
Try to run the workload on the Server Core installation option of Windows Server 2012. The Server Core installation option omits the Windows graphical shell, as well as several other less obvious components, but still supports many common workloads. This step limits the number of code modules installed by default.
lassify the server by role. For example, the Active Directory, Domain Name System (DNS), and print server roles have a large surface area and are frequently affected by patches. However, a web server shouldn't have any of those roles installed.
Configuration of Automatic Updates
- Click Control Panel -> System and Security -> Windows Updates
- Click “Turn on automatic updates”
- You can control the time at which updates are applied. Bear in mind that some updates require a system restart.
By incorporating the ten principles of Microsoft patch management into your patch management strategy you can help ensure that your processes and procedures are working in harmony with Microsoft. That way, you help ensure that we are all working together efficiently as team with one simple and important goal in mind: helping you to better protect your systems.
For critical systems, where availability matters, there are steps that any IT professional can take to maintain the security and availability of a business-critical infrastructure. One key to success could be the number of days between the reboots. The rules below can double the time between 2 reboots.
- Check for patches for all code that is loaded on all computers.
- Remove all roles or services that are not essential for the task at hand.
- Try to find a workaround for reported vulnerabilities to delay reboots.
- Creating a Patch and Vulnerability Management Program (National Institute of Standards and Technology, Special Publication SP 800-40): http://csrc.nist.gov/publications/nistpubs/800-40-Ver2/SP800-40v2.pdf Microsoft Safety and Security Center – Website that provides additional information and links to security tools and updates: https://www.microsoft.com/security/pc-security/default.aspx#Safety-products-and-scans
- CERT Coordination Center – Run for the U.S. Government by Carnegie Mellon: http://www.cert.org/certcc.html
- Microsoft 10 rules of Patch Management: https://technet.microsoft.com/en-us/library/cc512589.aspx
The contribution provided by Microsoft is intended to serve general information purposes and the content is AS IS without any express or implied warranties of any kind with respect to the accuracy, correctness or reliability. The information is provided without any warranty of fitness for a particular purpose. The information is compiled with the necessary care, however no liability is assumed in this respect, in particular with regard to the absence of errors, topicality with regard to the specific state of knowledge or use as the basis for the responsible decisions of the user.
Content provided by 1&1