Maester – Swiss army knife for M365 security testing

Introduction

In today’s digital environment, security and compliance are vital for organizations to protect their assets and meet regulatory standards. While traditional frameworks like CIS Controls offer a solid foundation, they often lack the flexibility and specificity needed for dynamic, modern threats.

Maester is an open-source PowerShell-based test automation framework designed to help monitor and maintain the security configuration of Microsoft 365 environments. It provides automated testing, customizable tests, and integration with DevOps pipelines for continuous monitoring.

This guide will cover high over aspects from installation and configuration to integration with DevOps workflows, custom test creation, alerting options, and a breakdown of its policy tests. By the end, you’ll see why Maester is an invaluable tool for security professionals seeking to elevate their organization’s security posture.

What is Maester and why use it?

Maester is a versatile, automated security assessment and compliance tool for Microsoft 365 environments. Built on PowerShell with the Pester testing framework, enables organizations to enforce security policies and can continuously monitor compliance. With comprehensive reporting, DevOps integrations, and customizable tests, Maester ensures that your organization maintains a proactive stance on security.

How it works

Maester follows a structured process to gather, analyze, and report security data:

1. Data Collection: Gathers information from your Microsoft 365 environment, such as configurations, policies, and user settings.
2. Policy verification: Runs extensive tests against best practices and predefined security standards.
3. Detailed reporting: Generates comprehensive reports with insights into security posture, categorized by severity.
4. Continuous monitoring: Offers automated scans on a scheduled basis via DevOps integration.
5. Real-time alerts: Provides configurable alerts via email, Slack, or Teams to notify your team of critical findings instantly.

Prerequisites and requirements

Before deploying Maester, ensure that your environment meets the following requirements:

These prerequisites ensure that Maester can operate effectively within your environment.

Installation process

To install Maester and its dependencies, follow these steps:

1. Install the Pester Module:

  Install-Module Pester -SkipPublisherCheck -Force -Scope CurrentUser

Note: The -SkipPublisherCheck parameter is used to bypass the publisher check.

2. Install the Maester Module:

   Install-Module Maester -Scope CurrentUser

3. Set up the test directory and install Maester Tests:

   md maester-tests
   cd maester-tests
   Install-MaesterTests

This creates a directory named maester-tests, navigates into it, and installs the Maester test suite.

4. Connect to your Microsoft 365 Tenant:

   Connect-Maester

This command initiates a sign-in process to your Microsoft 365 tenant.

5. Run the Maester tests:

   Invoke-Maester

This executes the installed tests against your tenant.

Optional Modules for additional tests:

Maester includes optional tests, such as those from the Cybersecurity and Infrastructure Security Agency (CISA), which require additional modules:

• Azure Module:

   Install-Module Az -Scope CurrentUser

• Exchange Online Management Module:

   Install-Module ExchangeOnlineManagement -Scope CurrentUser

After installing these modules, connect to the respective services:

Connect-Maester -Service All

This command prompts you to sign in to Azure, Exchange Online, and other services as needed.

For detailed information, refer to the Maester Installation Guide and/or see below key commands.

Key commands

Below is a summary of the most commonly used commands in Maester for installation, configuration, test execution, and troubleshooting:

Examples of using commands

• Run all tests with default configuration:

  Invoke-Maester

• Generate a detailed HTML report:

 $results = Invoke-Maester
 $results | Export-MaesterReport -Format HTML -Path .\TestReport.html

• Update Maester Tests:

  Update-MaesterTests

Tips for using commands

• Filtering tests: Use -Tags or -Test parameters with Invoke-Maester to focus on specific areas like compliance or security.
• Exporting results: Always use Export-MaesterReport to generate shareable reports in formats like HTML or JSON.
• Staying up to date: Regularly run Update-Maester and Update-MaesterTests to ensure you’re using the latest features and test suites.

Overview of the core tests and policies

Maester’s extensive suite of tests covers essential security areas. Here’s a breakdown of key tests and policies:

Authentication methods

Authentication checks focus on ensuring secure user authentication mechanisms are in place to prevent unauthorized access.

• FIDO2 security keys: Verifies if FIDO2 keys are enforced for login, ensuring passwordless authentication methods are in use. It checks if self-service key registration is restricted to prevent unauthorized devices from being added and ensures proper attestation is required during registration to validate device authenticity.
• Microsoft Authenticator: Examines settings to confirm advanced security features are enabled. This includes:
• Number matching to mitigate push notification attacks.
• Application context to help users recognize legitimate login prompts.
• Displaying geographic information to provide additional context during authentication attempts.
• Temporary access pass: Ensures temporary access passes are configured for one-time, time-limited use. This is useful for onboarding users without a traditional password.

Authorization and permissions

These checks validate the organization’s default permissions, ensuring guests and users are granted minimal access by default.

• Default guest permissions: Assesses whether guest users are restricted from accessing sensitive resources, ensuring compliance with the principle of least privilege.
• Risk-based admin consent: Ensures policies are in place to require step-up consent for high-risk scenarios, adding an extra layer of protection when users approve application permissions.
• Self-service policies: Verifies configurations that allow or restrict users from performing self-service operations, such as registering security keys or managing roles.

Password rules

Checks here ensure that strong password policies are enforced to protect against credential compromise.

• Password protection:
• Validates the use of custom or default banned password lists to prevent weak or predictable passwords.
• Checks smart lockout thresholds to reduce brute-force attempts.
• Expiration policies:
• Verifies whether passwords have a defined expiration interval to limit exposure to compromised credentials.
• Ensures policies require users to change passwords upon first use, enhancing security during onboarding.

Conditional access policies

Conditional access checks enforce secure access to resources by applying contextual rules.

• Device compliance: Verifies whether policies enforce that only compliant devices (e.g., with encryption, antivirus, and updated OS) can access organizational resources.
• Legacy authentication: Ensures that legacy protocols (like POP and IMAP) are blocked, as these do not support modern authentication methods, making them more vulnerable.
• MFA enforcement:
• Confirms that multi-factor authentication (MFA) is mandatory for all users, including guests.
• Checks if MFA is required for risky sign-ins, such as those originating from untrusted locations or devices.
• Named location policies: Validates restrictions on access based on geographic locations, ensuring that only users from approved regions can log in.

Privileged access management

These checks ensure that administrative roles and permissions follow best practices.

• Role assignments:
• Confirms that least-privilege principles are enforced, ensuring roles are assigned only as needed.
• Validates that privileged roles are not permanently active, reducing the risk of privilege escalation.
• Alerts for privileged role activation: Verifies whether notifications are configured for privileged role activations to alert administrators of potential misuse.
• Privileged identity management (PIM): Ensures that PIM is configured and active, providing just-in-time access and additional auditing capabilities for sensitive roles.

Email security (Exchange Online)

Email-related checks focus on securing communications and preventing spoofing or unauthorized access.

• SPF, DKIM, and DMARC:
• Validates configurations for Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication, Reporting, and Conformance (DMARC) to protect against email spoofing and phishing.
• External sender warnings: Ensures warning banners are displayed for emails originating from external domains, helping users identify potential phishing attempts.
• Email forwarding: Verifies that automatic forwarding to external domains is blocked to prevent sensitive data from being inadvertently shared.

Consent and app management

These checks assess how permissions and consent workflows are managed for applications in the tenant.

• Admin consent policies: Ensures policies require admin approval for third-party applications requesting sensitive permissions, reducing the risk of unauthorized app access.
• App consent reviews: Validates whether periodic reviews of app consent workflows are performed to detect and revoke unnecessary or risky permissions.

Risk management

Risk-based checks evaluate the organization’s ability to detect and respond to suspicious activities.

• Sign-in risk policies: Validates whether conditional access policies address risky user sign-ins, such as those from untrusted locations or devices.
• User risk policies: Checks if thresholds are set for triggering risk-based actions, such as password resets or account lockouts, to mitigate threats.

Microsoft Entra security

These checks focus on enforcing security best practices within Microsoft Entra (formerly Azure AD).

• Blocking legacy protocols: Ensures legacy authentication protocols are blocked, as these are common attack vectors.
• Identity protection policies: Verifies the implementation of Identity Protection policies to manage and mitigate risks for users and roles with elevated privileges.

Interpreting Maester’s output

Maester generates comprehensive reports, categorizing issues by severity:

Pass: Configuration meets the requirements

Fail: Configuration violates the defined policy

Reports are available in JSON and CSV formats, making it easy to export.

Real-Time alerting with Email, Slack, and Microsoft Teams

Maester offers flexible alerting options, allowing security teams to stay informed of critical issues as soon as they’re detected. Configuring alerts ensures that any significant findings are communicated instantly, facilitating rapid response.

• Email alerts: Set up email notifications for critical findings, delivered directly to your inbox.
• Slack integration: Configure Slack alerts to notify a specific channel, enabling collaborative responses.
• Microsoft Teams: For Teams users, configure alerts to deliver notifications to a designated security channel.

These alerting options help you keep your team in the loop, enabling real-time, collaborative incident response.

Integrating with DevOps pipelines

Maester integrates with DevOps tools to make security an integral part of the CI/CD pipeline.

Azure DevOps integration

1. Create an Azure DevOps project: Set up a dedicated project for Maester tests.
2. Import Maester Tests: Clone the Maester Tests repository into Azure DevOps.
3. Configure pipelines: Set up an Azure pipeline to automate Maester tests on a schedule or based on specific triggers (e.g., configuration changes).

Refer to the Azure DevOps integration documentation for detailed setup instructions.

GitHub Actions integration

1. Create a GitHub repository: Host Maester tests in a repository.
2. Clone the Maester Tests: Copy the Maester Tests repository to GitHub.
3. Set up workflows: Configure GitHub Actions to run Maester tests on schedule or in response to events (e.g., pushes, pull requests).

For setup instructions, see the GitHub Actions integration documentation.

Writing custom tests in Maester

Custom tests extend Maester’s functionality, allowing you to enforce unique security policies.

1. Create a custom test file: Add a new .Tests.ps1 file in the Custom directory.
2. Write the test using pester: Define tests using the Pester framework. For example, to check if a specific security group exists:

   Describe "Security Group Validation" {
       It "Verifies the 'Admins' group exists" {
           Get-AzureADGroup -Filter "DisplayName eq 'Admins'" | Should -Not -BeNullOrEmpty
       }
   }

These custom tests provide flexibility to adapt Maester to your organization’s specific security requirements.

Conclusion

Maester is a comprehensive tool for security compliance and vulnerability management in Microsoft 365 environments. Its extensive test library, customization, seamless DevOps integration, and flexible alerting options make it an essential part of any modern security strategy.

With real-time alerts delivered via email, Slack, or Microsoft Teams, Maester keeps security teams informed, enabling rapid response to critical issues. Whether for routine audits, real-time vulnerability management, or custom policy enforcement, Maester empowers your organization to stay proactive and resilient in today’s complex cybersecurity landscape.