I have been upgrading my RTM Exchange 2013 to CU1.  I have 2 multirole servers in DAG. I have started to install CU1 on the node hosting only passive copies of databases. In step 1 of 18. Organization preparation from GUI setup it generated error as it can be seen in the following Picture.

Recommended workaround from Microsoft is to delete the following object from AD configuration partition using AdsiEdit

CN=ClassificationDefinitions,CN=Rules,CN=Transport Settings,CN=<Your organization name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<domain>i,DC=<com>

The object is also shown in the Picture

After deletion setup can be restarted. Object is then re-created and setup can continue.

I needed to use DC as File Share Witness for my LAB and in one small environment. Microsoft doesn´t recommend that, but if you have only limited number of servers, you dont have a choice.

Theory:

Really basics, full article can be found on MS Technet)

• FSW must be configured every time you create Database Availability Group for Exchange 2010 and Exchange 2013 (if you do not specify, Exchange will configure FSW on first CAS server without mailbox role installed). Besides the other parameters you should specify the WitnessServer and WitnessDirectory parameters

New-DatabaseAvailabilityGroup -Name DAG1 -WitnessServer CAS1 -WitnessDirectory D:\DAG1_FSW

• FSW is used to maintain quorum (node majority for DAG application) when even number of nodes in the DAG
• FSW is only actively used, when there is even number of servers in the DAG. One case is that you have configured even number of servers by design or you have conffigured odd number of servers by design and one of those is broken. Otherwise Witness directory on Witness server is empty
• Alternate FSW must be also configured, if you enable Datacentre Activation Coordination

Set-DatabaseAvailabilityGroup <identity> -DatacenterActivationMode DAGOnly -AlternateWitnessServer <FQDN or NetBIOS name of the server> -AlternateWitnessDirectory <Path>

• More then one DAG can have FSW on the same server, but directory must be unique

Solution:

To configure FSW on DC there are more steps to perform before configuration of FSW:

• Add domain controller to Exchange Trusted Subsystem security group
• Add Exchange Trusted Subsystem to Buildin\Administrators
• Create Directory on the DC and share the directory with the share name of the DAG
• Set sharing permissions so that virtual account for DAG will have Full Control

If set some of the point incorrectly, you will get the result, that DAG cannot access FSW and availability of DAG is limited

Get-DatabaseAvailabilityGroup -Status

Result is shown in Picture:

If this happens to you, fix incorrectly set steps and re-enable FSW:

Set-DatabaseAvailabilityGroup <identity> -WitnessServer <FQDN or NetBios name of the server>

Let me make a short look at the following error:

Log Name: Application
Source: MSExchangeIS Mailbox Store
Date: 3/9/2013 7:21:21 PM
Event ID: 10001
Task Category: Background Cleanup
Level: Warning
Keywords: Classic
User: N/A
Computer: ExCas32.lifa.intra
Description:
Mailbox cleanup could not completely remove the mailbox for user 9e7c5f65-9cdd-44d9-af51-d2b16b786157 (Root Fid 6fa-4FD75D76E9) following mailbox removal. Encountered error 0xfffffae8. Should this message continue to persist for the same mailbox, it may be indicative of a problem that requires further investigation.

This event usually indicates failed cleanup process which cannot clean corrupted fields in the mailbox database. It is mainly related to mailbox moving (either for the source database with SoftDeleted mailbox, the destination database or both of them). It can occur also for the moving which was completed successfully.

How to stop this event?
In many cases this problem has been resolved by dismounting and mounting the database. If there are many databases having the problem restart of IS service or reboot the computer could help also.

How to find the affected database?
It could be achieved through ExchangeGuid from the event 018e1b17-524c-4ee6-958c-2d8721d3714d.

Get-Recipient -Identity "9e7c5f65-9cdd-44d9-af51-d2b16b786157"

If you do not have a success, you can try to use LDAP Query:

Get-ADObject -LDAPFilter "(&(objectclass=*)(msExchMailboxGuid=\65\5f\7c\9e\dd\9c\d9\44\af\51\d2\b1\6b\78\61\57))"

How to specify ExchangeGuid for LDAP Query:

1. Original GUID from the Event Log: 9e7c5f65-9cdd-44d9-af51-d2b16b786157
2. Divide it with backslashes: \40\34\19\4C\21\C1\4D\C1\95\1D\65\13\20\57\7A\N0
3. Invert the first 3 columns (9e7c5f65-9cdd-44d9): \4C\19\34\40\C1\21\C1\4D\95\1D\65\13\20\57\7A\N0

Source: How to find the Object that belongs to a GUID

If you do not have the success again, you can try to re-mount all active/mounted mailbox databases on reported server from the event:

$dbs = Get-MailboxDatabase | where {$_.server -like "ExCas32.lifa.intra"} 
$dbs | Dismount-Database
$dbs | Mount-Database

Theory

By default OAB is a point in time snapshot of global address list and it is used as cached source of information about Exchange recipients properties. OAB is stored on Exchange servers (see my previous article http://ficility.net/2013/03/04/oab-differences-between-exchange-2010-and-exchange-2013-in-brief/) and downloaded to client once Outlook is configured in Cached mode. I would like to test modes of using address book,
while Outlook is in cached mode. There are several methods to download OAB. These methods depends on registry settings of Outlook (full article here:http://support.microsoft.com/kb/823580)
shortly:

If the following registry key is present (XX.0 means office version – 15.0 for Office 2013), Outlook behaves upon the DWORD value inside:

HKEY_CURRENT_USER\Software\Microsoft\Office\XX.0\Outlook 
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\XX.0\Outlook

DownloadOAB DWORD supported values:
  0 = The Offline Address Book does not download automatically.
  1 = The Offline Address Book uses the Download Full Items download mode. This is the default setting.
  2 = Download the Offline Address Book in any download mode, but download a differential update in the Header only download mode.
  3 = Always download the Offline Address Book and a differential update in one of the following download modes:
Download Headers and then Full Items
Download Full Items
Download Headers

The goal of this article is to test differences in OAB behavior between modes 0 and 1. I want to see how it looks when:

1) Mode 1 – The Offline Address Book uses the Download Full Items download mode.
a)Download OAB and check
b)Change GAL and test downloading OAB instantly
c)Update OAB and check

2) Mode 0 – The Offline Address Book does not download automatically.
a) Test behavior once mode 0 is configured while old OAB files are still on the client
b) Test behavior once mode 0 is configured while old OAB files are removed from client

Test scenarios

1) Mode 1 – The Offline Address Book uses the Download Full Items download mode.

By default OAB is downloaded from server hosting Active mailbox database with Organizational mailbox. (or in 2010 from CAS server distribution point or in 2007 and 2010 from Public Folders)

a) Download OAB and check

I have opened OAB from my mailbox and result is in the Picture

b) Change GAL and test downloading OAB instantly

Creation of a mailbox doesn´t updtate OAB itsetf. To create mailbox use command

New-Mailbox OAB_Test_o1 -UserPrincipalName OABTestO1@salonovi.cz
WARNING: A script or application on the FRONTEND1.SALONOVI.CZ remote computer is sending a prompt request. When
prompted, enter sensitive information such as credentials or password only if you trust the remote computer and the
application or script requesting it.

cmdlet New-Mailbox at command pipeline position 1
Supply values for the following parameters:
Password: ********

Name                      Alias                ServerName       ProhibitSendQuota
----                      -----                ----------       -----------------
OAB_Test_o1               OABTestO1            backend1         Unlimited

c) Update OAB and check

To update OAB use command

Get-OfflineAddressBook | Update-OfflineAddressBook

Now I have tested if OAB change is reflected in client computer (should not be)

And now I have downloaded new OAB to client and tested again

2) Mode 0 – The Offline Address Book does not download automatically

One registry key setting and Outlook client will work online from Addres book point of view, BUT! This setting also requires to clean up OAB files from client computer to behave correctly. I will test both possibilities and try to find differneces in behavior.

First I will set up registry key and restart Outlook

In my lab I dont use any special GPO so yhe setting is done via registry key HKEY_CURRENT_USER\Software\Microsoft\Office\XX.0\Outlook

OAB files are left on the client

a) Test behavior once mode 0 is configured while old OAB files are still on the client

New mailbox should be immediately visible for client without need to download OAB since information should be available online.

b) Change GAL and test if it appears immediately to client

Change in OAB is not visible immediately, because we have OAB files on the client computer and Outlook use those!

New-Mailbox OAB_Test_o2 -UserPrincipalName OABTestO2@salonovi.cz

c) Update OAB automatically by restarting Outlook

If Outlook is restarted, OAB version is checked / downloaded from Exchange server.

AND NOW THE NAUGHTY stuff! OAB is not updated anymore even you download files successfully from Exchange. In this stage Outlook is stuck somewhere between mode 0 and 1 and updates are not received by client.

d) Update OAB and download change to client manually even mode 0 is used

In this scenario I would like to prove, that setting mode 0 is not the only thing to consider to have Outlook work correctly.

• To update OAB use command

Get-OfflineAddressBook | Update-OfflineAddressBook

• Download OAB to client and check if changes are displayed to client.

 Changes are only reflected if I try to manually download full or incremental copy of address book. OAB is not downloaded during the client startup!

OAB files are removed from client

a) Test behavior once mode 0 is configured while old OAB files are removed from the client

I added new mailbox again, updated OAB, but for now I have removed all OAB-Related files from client. Changes should appear to client immediately.

To remove OAB files

• locate:
  c:\Users\<USERNAME>\AppData\Local\Microsoft\Outlook\Offline Address Books\
• Delete folders
  Example: (c:\Users\lelicek\AppData\Local\Microsoft\Outlook\Offline Address Books\6a285982-48d6-43ee-979b-f84dd5b7d989\)
• Start Outlook
• After Outlook startup, folders will be re-creaded, but will be empty. It proves we have mode 0 set to Outlook client.
• open Offline Address Book
• Newly created mailbox should be here

 d) Do change in GAL and test if it appears immediately to client

• Create new mailbox

New-Mailbox OAB_Test_o5 -UserPrincipalName OABTesto5@salonovi.cz

• Open Offline Address Book and new mailbox should be again there

e) Download OAB manually and test if the changes made to GAL after OAB download will be imediatelly visible to client

Conclusion

Mode 1

• Conclusion is , that mode 1 works fine and as it should and user will get updated OAB after regularOAB update schedule or after manual run of Update-OfflineAddressBook command.
• Note that in Exchange 2010 you must restart File distribution service, to distribute updated OAB to WEB distribution points. In Exchange 2013 it is not needed anymore.

Mode 0

• Use mode 0 only in case, that you do a lot of changes in GAL and you need clients to see changes immediatelly while taking advantages of Outlook Cached connection.
• Once mode 0 is used, administrator has to make sure, that OAB files will be removed from client computer (for example by logon script / GPO), otherwise user must use manual OAB update via Send/Receive -> Send/receive groups / Download Address Book.
• If user tries to manual download OAB while mode 0 is used, Online functionality will STOP working from that time until OAB files are deleted again!

My friend´s Exchange 2013 implementation was experiencing problem with e-mail delivery. E-mails were returned from inside as well as outside of organization with NDR pointing to DNS failure.

I was investigating this issue, but not all e-mails were lost. Only some of those. I knew, that there was a change recently, that new network interface for VPN connection has been added to Exchange server.

Symptoms:

DNS for next hop is not resolvable returning DNS error

“554 5.4.4 SMTPSEND.DNS.NonExistentDomain; nonexistent domain”.

Investigation:

I started to investigate what could cause the problem. It was DNS error, so I started to use NSLOOKUP tool and tried to resolve DNS name for next hop domain. The answer was different, then I thought it would be. I have received Time Out answer from VPN DNS server, even there is another DNS server set in production environment. I realized from my previous Exchange 2010 projects, that priority of network interfaces is added from Highest to lowest based on sequence interfaces are added to system. So for example if you configure Production network interface before Replication, Replication interface will get higher priority and if you use DNS servers in your replication network (for whatever reason), wrong DNS server will answer to your request and this might lead to unexpected failures.

Now the naughty stuff:

If you use IPv6, you are still OK, since IPv6 DNS servers will answer even though Replication interface has higher priority then production one.

If you disable IPv6 on your interfaces, you have troubles. Once interface priority is changed to incorrect order, you will not be albe to at least resolve DNS queries correctly. Incorrect priority might also lead to packet loss.

Solution:

Very simple:

• Right click network icon in notification area (right low corner) of your server and click Open Network And Sharing Center
• Click on Change Adapter Settings
• Press ALT key and select Advanced -> Advanced Settings
•  In the Advanced Settings window select interface with incorrectly set priority and press arrows to move interface to correct place

Results before priority change

Results after priority change

Why not mention Exchange 2010 bug – The ActiveSyncDevice identity cannot be found.

Symptoms

• The user has a Microsoft Exchange ActiveSync partnership that works as expected.
• You move the user to a new organizational unit (OU) or rename a user account in Active Directory Domain Services (AD DS).
• You try to perform a remote wipe operation for the device in the Exchange Management Console (EMC).

KB

http://support.microsoft.com/kb/2721428

Reporting

Get-ActiveSyncDevice -Mailbox 00164 | select UserDisplayName,Identity 

UserDisplayName : liintra.intra/Users/00164
Identity : liintra.intra/Users2/00164/ExchangeActiveSyncDevices/NokiaEmail§IMiEI284675044284679

Affected objects can be found and reported (csv):

[PS] C:\>Get-ActiveSyncDevice -ResultSize unlimited | sort -Property Identity -Unique | select Identity,UserDisplayName | ? {$_.Identity -notmatch $_.UserDisplayName} | select UserDisplayName,Identity  | Export-Csv -Delimiter "," -Encoding unicode -Path "C:\Users\filip\Desktop\Report170413.txt"

Solution

• Remove-ActiveSyncDevice –Identity “new path = Identity from Get-ActiveSyncDevice“

Remove-ActiveSyncDevice -Identity "liintra.intra/Users2/00164/ExchangeActiveSyncDevices/NokiaEmail§IMiEI284675044284679"

• During the next mail sync user’s device will perform full sync automatically, but the sync will take longer than usually.

If you have configured OWA redirect in my case through TMG (Setting up an OWA Redirect in Forefront TMG 2010), you will see the following notification for a while:

Your browser will be automatically redirected. 
If your browser does not support automatic redirection, browse to the original URL again.

If your browser does not support automatic redirection, you will see ONLY the notification despite of successful authentication.

How to enable automatic redirection:

1. Open Internet Explorer or Control Panel
2. Internet Options
3. Click the “Security” tab and click the “Custom Level” button
4. Click the “Enable” radio button underneath the “Allow META REFRESH” header.
5. Close and reopen browser to finish enabling automatic redirect.

Hi everyone!

Thanks to Filip’s & Zbynek’s invitation I also have the privilege to be part of this excellent FICILITY.net blog. So here we go. This is my first post

If you happen to have an Application log on your Mailbox server full of eventIDs 10033, Source: “MSExchangeIS Mailbox Store” with the following information:

A folder is being scoped by a search which no longer exists. This may result in failures or errors when attempting to update the folder contents. If this message continues to persist for this mailbox, moving the mailbox to a different database may resolve the issue.

As instructed this issue can probably be resolved by moving the affected mailbox to a different mailbox database but I’ve found out that you can fix this issue by using cmdlet “New-MailboxRepairRequest” on affected mailbox.

New-MailboxRepairRequest -Mailbox [mailbox name] -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView

After that you’ll see that mailbox-repair request was created. Also check Application log on mailbox server where mailbox is homed for event with ID 10047:

Mailbox level online integrity check for request 7cef315f-60d7-4b8d-995f-32c22c68f4de started:Database=OPEXDAG1-MB1

Mailbox=DF49FD31-D429-423A-BECB-3594C19255FE

Flags=Detect, Fix

Tasks=SearchFolder, View, AggregateCount, ProvisionedFid

After a while you should see EventID 10062 that corruptions were detected & fixed:

Corruptions detected during online integrity check for request 7cef315f-60d7-4b8d-995f-32c22c68f4de

Mailbox:DF49FD31-D429-423A-BECB-3594C19255FE (MAILBOX NAME)

Database:OPEXDAG1-MB1

Corruption        Is Fixed        FID        Property        Resolution

“Folder Backlinks”, Yes, “1-38645 (Synkronointiongelmia1)”, 0×67870102, “Remove unfound folder(s) from the backlink list”

“Folder Backlinks”, Yes, “1-38649 (Ristiriidat)”, 0×67870102, “Remove unfound folder(s) from the backlink list”

“Folder Backlinks”, Yes, “1-3864D (Paikallisia virheitä)”, 0×67870102, “Remove unfound folder(s) from the backlink list”

“Folder Backlinks”, Yes, “1-38651 (Palvelinvirheitä)”, 0×67870102, “Remove unfound folder(s) from the backlink list”

“Folder Backlinks”, Yes, “1-12829678 (Digium)”, 0×67870102, “Remove unfound folder(s) from the backlink list”

….

 And when Repair-request is completed you will find EventID 10048:

 Online integrity check for request 7cef315f-60d7-4b8d-995f-32c22c68f4de completed successfully.

 After that original EventID 10033 disappeared

One of our customers has Exchange 2010 (currently SP3). When I was installing latest build (SP3 from SP4 RU 5v2), I experienced problems that server was unaccessible, Exchange web services were not able to serve clients, Powershell was not working and Forefront was crashing taking down MS Exchange Transport service. I spent several hours troubleshooting and here is the overview and solution.
Description:

Configuration:

Lets say that my server was in Windows NLB cluster having:

LB IP: 192.168.1.120 /24
 Production IP:192.168.1.100 /24
 Production IP2 192.168.1.99 /24

From the start server has been installed and configured with the following  IP´s:

LB IP: 192.168.1.120 /24
Production IP:192.168.1.100 /24

and second Production IP address has been added later as requirement for one business Exchange Web Services-related application.

From IIS point of view:

From the begining there was only one application pool with all virtual directories as shown in the following picture (NOTE binding settings).

Site bindings for Default application web site application pool:

After business application has been added new application pool has been added too(NOTE binding settings).

Site Bindings for business application Application pool

Process of upgrade:

The process of upgrade was normal and well known from MS Technet, however after reboot of server I lost connectivity to Powershell and to EWS on the server. Therefore I expected problems in IIS, but how to find it?

Errors in IIS:

Errors in Event log:

Solution:

I checked IIS:

• Application pools up and running
• IIS up and running
• Server giving error 404 page cannot be displayed if I tried https://localhost/OWA url
• I checked virtual directory paths and all needed files were in place
• I checked bindings and voila! Bidning of localhost has been moved from Default Web Site to Application Web Site

Binding on Bussiness application web site has now localhost:

Bindings on Default Web Site is missing localhost record:

Move Localhost binding to correct place and restart IIS!

Explanation:

There is a logic how IP addresses are set in IIS. One basic logic says, that WEB server can have as many IP addresses as needed, BUT the default one must be the one with lowest number. For example if I

have 3 IP addresses:

192.168.10.10
192.168.10.11
192.168.10.12

IP address with highest priority is 192.168.10.10 and in some circumstances localhost binding can be moved under application pool, which is listening under IP address with highest priority without notice. This is quite logical, that Powershell and OWA virtual directories stopped to work against localhost binding, because there was no virtual path inside newly created application pool for bussiness application.

Conclusion:

Plan the possibility to extend server with some more application pools and listeners already in design phase and reserve some IP addresses with higher numbers for future needs.

Finally here is the continuation of previous article about Exchange federation trust. So we have established the trust between Microsoft Federation Gateway and our organizations. Next step is to configure inter-organizational behavior. It is a mesh-like net, where 1:1 organization relationship is established.

Prerequisites

• Autodiscover service must be accessible to at least one CAS server from the internet
• EWS should be accessible to at least one server and External URL should match the name accessible from  internet and 3rd party certificate SN or SAN name

Organization Relationship

Once we have configured our organizations to trust MS Federation Gateway, we can use it to create organization relationship. We will use

command Get-FederationInformation about opposite organization and pipe it to create new organization relastionship. Access level on both side of relationship should be the same.

In our organization:

Get-FederationInformation -DomainName metrosys.cz | New-OrganizationRelationship -Name "Metrosys" -FreeBusyAccessEnabled $true 
-FreeBusyAccessLevel -LimitedDetails

Or directly:

New-OrganizationRelationship -Name <foreignorganizationname>  -FreeBusyAccessEnabled $True -FreeBusyAccessLeve LimitedDetails -
Enabled $true -PhotosEnabled $true -TargetAutodiscoverEpr https://email.foreigndomain.cz/autodiscover/autodiscover.svc/wssecurity -
DomainNames .cz -TargetApplicationURI http://fydibohf25spdlt.foreigndomain.cz/ -TargetSharingEpr 
https://email.foreigndomain.cz/EWS/Exchange.asmx

Note: Domain names are CASE SENSITIVE!
Result of creation test:

Test-OrganizationRelationship -identity <ForeignOrganizationname> -UserIdentity primarysmtpaddress@salonovi.cz -Verbose

In foreign organization:

Get-FederationInformation -DomainName salonovi.cz | New-OrganizationRelationship -Name "Salonovi" -FreeBusyAccessEnabled $true -
FreeBusyAccessLevel LimitedDetails

Or directly:

New-OrganizationRelationship -Name  -FreeBusyAccessEnabled $True -FreeBusyAccessLeve LimitedDetails -Enabled $true -PhotosEnabled 
$true -TargetAutodiscoverEpr https://mail.salonovi.cz/autodiscover/autodiscover.svc/wssecurity -DomainNames salonovi.cz -TargetApplicationURI 
http://fydibohf25spdlt.salonovi.cz/ -TargetSharingEpr https://mail.salonovi.cz/EWS/Exchange.asmx

Note: Domain names are CASE SENSITIVE!

Finally result of proper configuration is, that you can see Free/Busy limited details of users in foreign organization

Errors you might face

Index error is cause by Case sensitive domain name inserted (in my case Metrosys.cz instead of metrosys.cz or wrong URLs for EWS or Autodiscover.

Errors from the following picture are caused by wrongly or misspelled URLs (Self explaining)

Usually autodiscover URL is created in format https://autodisvocer.domianname.cz/autodiscover/autodiscover.xml, however Federation trust use autodiscover service, which is created as URL: https://autodisvocer.domianname.cz/autodiscover/autodiscover.svc/WSSecurity where WSSecurity is authentication used by federeation trust:

Links:

• Configuring Federation Trust http://technet.microsoft.com/en-us/library/jj657462(v=exchg.150).aspx
• Configure OrganizationRelationship http://technet.microsoft.com/en-us/library/jj657451(v=exchg.150).aspx

During a restore I was facing the following error so why not to mention it here.

Couldn't mount the database that you specified. Specified database: RDB2; Error code: An Active Manager operation failed. Error The database action f
ailed. Error: Operation failed with message: MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)

I already had an one recovery database mounted on server and it is reason for the error. Why? Because each Exchange 2010 or Exchange 2013 server allows for only one recovery database to be mounted at a time. The server can contain as many recovered databases as disk space allows, but only one can be mounted as the recovery database. The database mounted as the recovery database is counted in the maximum number of databases that can be mounted at a time. A recovered database mounted as a server’s recovery database is not associated with the original mailbox in any way (more).

Additionally, here is my procedure of the restore:

1. New-MailboxDatabase -Recovery -Name RDB2 -Server EX2010 -EDBFilePath R:\RDB2\DB\RDB2.edb -LogFolderPath R:\RDB2\Logs
2. Get-MailboxDatabase -Status| ft name, mounted, Recovery -a
3. Backup Exec restore process (data restored into RDB, the database is mounted automatically)
4. Get-MailboxDatabase RDB | Get-MailboxStatistics | ? {$_.DisplayName -like “*Pepa Novak*”}|fl
5. New-MailboxRestoreRequest -name “Restore210613″ -TargetMailbox kasajfilip -SourceDatabase “RDB2″ -SourceStoreMailbox “Personal Archive – Pepa Novak” -BadItemLimit 100 -AllowLegacyDNMismatch -AcceptLargeDataLoss -targetrootfolder “Restore210613″
6. Get-MailboxRestoreRequest -name Restore210613 | Get-MailboxRestoreRequestStatistics | fl
7. New-MailboxExportRequest -name “Export210613″ -Mailbox kasajfil -Include FoldersRestore210613/* -FilePath \\EX2010A\PSTfiles\PepaNovak(arch).pst
8. Remove-MailboxDatabase RDB2
9. Manually removing the database file

One of our customers has ExRAAS ( Exchange health and remediation check service) every year to audit their environment for health, performance and MS best practices implementation. ExRAAS tools are developed every year and this years tool discovered very interesting issue about DAG networks.

Description:

Our customers DAG has 3 networks:

• Production – meant to be client network, where only client traffic is enabled, replication traffic is disabled
• Replication – not routable to MAPI network – custom 5Gbit bandwidth only for log replication
• Backup – only for VSS backups, no MAPI nor replication traffic should flow there

Problem:

By design DAG is set, that Backup network should be ignored, however if I give Get-DatabaseAvailabilityGroupNetwork command, I can see MapiAccessEnabled parameter in $True, even though this network doesn´t have Clients for Windows Networks feature enabled and according to MS it is not supported network for clients. The magic starts when I set IgnoreNetwork to $false. Right after the change MapiAccessEnabled parameter is in correct value.

Get-DatabaseAvailabilityGroupNetwork DAG1\BACKUP | Set-DatabaseAvailabilityGroupNetwork -IgnoreNetwork $false
Get-DatabaseAvailabilityGroupNetwork | fl

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : BACKUP
Description        : VSS BACKUP Backup subnet - Ignored
Subnets            : {{172.24.188.0/24,Up}, {172.29.99.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,172.24.188.108}, {DC1MBX2,Up,172.24.188.110}, {DC1MBX3,Up,172.24
                     .188.112}, {DC1PF1,Up,172.24.188.104}, {DC2MBX1,Up,172.29.99.109}, {DC2MBX2,U
                     p,172.29.99.111}, {DC2MBX3,Up,172.29.99.113}, {DC2PF1,Up,172.29.99.105}}
MapiAccessEnabled  : False
ReplicationEnabled : False
IgnoreNetwork      : False
Identity           : DAG1\BACKUP
IsValid            : True

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : MAPI
Description        : Production and possible replication
Subnets            : {{192.168.0.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,192.168.0.108}, {DC1MBX2,Up,192.168.0.110}, {DC1MBX3,Up,192.168
                     .0.112}, {DC1PF1,Up,192.168.0.104}, {DC2MBX1,Up,192.168.0.109}, {DC2MBX2,
                     Up,192.168.0.111}, {DC2MBX3,Up,192.168.0.113}, {DC2PF1,Up,192.168.0.105}}
MapiAccessEnabled  : True
ReplicationEnabled : False
IgnoreNetwork      : False
Identity           : DAG1\MAPI
IsValid            : True

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : REPLICATION
Description        : Only replication
Subnets            : {{10.146.231.0/27,Up}}
Interfaces         : {{DC1MBX1,Up,10.146.231.24}, {DC1MBX2,Up,10.146.231.26}, {DC1MBX3,Up,10.146.2
                     31.28}, {DC1PF1,Up,10.146.231.20}, {DC2MBX1,Up,10.146.231.25}, {DC2MBX2,Up,10
                     .147.231.27}, {DC2MBX3,Up,10.146.231.29}, {DC2PF1,Up,10.146.231.21}}
MapiAccessEnabled  : False
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAG1\REPLICATION
IsValid            : True

When I change the Ignorenetwork back to $true, MapiAccessEnabled is set to $True as well.

Get-DatabaseAvailabilityGroupNetwork DAG1\BACKUP | Set-DatabaseAvailabilityGroupNetwork -IgnoreNetwork $true
Get-DatabaseAvailabilityGroupNetwork | fl

RunspaceId         : 7d204cce-1dde-4e6f-9d52-cde8b238d2a9
Name               : BACKUP
Description        : VSS BACKUP Backup subnet - Ignored
Subnets            : {{172.24.188.0/24,Up}, {172.29.99.0/24,Up}}
Interfaces         : {{DC1MBX1,Up,172.24.188.108}, {DC1MBX2,Up,172.24.188.110}, {DC1MBX3,Up,172.24
                     .188.112}, {DC1PF1,Up,172.24.188.104}, {DC2MBX1,Up,172.29.99.109}, {DC2MBX2,U
                     p,172.29.99.111}, {DC2MBX3,Up,172.29.99.113}, {DC2PF1,Up,172.29.99.105}}
MapiAccessEnabled  : True
ReplicationEnabled : False
IgnoreNetwork      : True
Identity           : DAG1\BACKUP
IsValid            : True

Conclusion:

This lead to errors in ExRAAS report and to question what is the right way. How should I behave to the network configuration? Better way is to set IgnorenNetwork parameter to $True and just ignore MapiAccessEnabled in $True. This article will be updated after I get info from MS for the resolution. It is also worth to mention, that last best practice says, that compression and encryption should be ENABLED on DAG replication network!

Links:

http://blogs.technet.com/b/schadinio/archive/2010/12/08/exchange-2010-mailbox-dag-based-practice-network-configurations.aspx

I upgraded Exchange Management Tools by SP3 and RU10 (Exchange 2007) on a backup server and after server reboot I could not be able to start Exchange Management Shell. The shortcut did not refer to valid object. It seemed to be disk related issue and when I used Disk Management and saw the following:

The operation failed to complete because the Disk Management console view is not up-to-date. Refresh the view by using the refresh task. If the problem persists close the Disk Management console, then restart Disk Management or restart the computer.

It could be probably solved by:

• Uninstalling the unknown devices - http://laptop-driver.blogspot.fi/2010/12/disk-management-console-view-is-not-up.html
• Using Acronis Disk Director Suite - http://blog.amastaneh.com/2010/12/operation-failed-to-complete-because.html
• Using Diskpart - http://answers.microsoft.com/en-us/windows/forum/windows_vista-performance/disk-management-console-view-is-not-up-to-date/42b1ebc3-edaf-4c3f-93cd-beda4bab2671

But in my case the problem was solved by disabling and enabling of affected disks via Device Manager:

I came accross one very odd error, while I was installing CU2 for Exchange 2013 on WS2012. The problem was, that I used command line to install Exchange to different directory C:\EXCHANGE2013.

Command used:

.\setup.exe /IacceptExchangeServerLicenseTerms /roles:mailbox,ClientAccess /MDBName:DB01 /m:install /DBFilePath:E:\DB01\DB01.edb /t:c:\EXCHANGE2013 /installWindowsComponents

Error during setup states, that FMS service cannot be started. Correct. It can be seen for all Exchange-related services. I tried to start the service manually as well:

No success. I checked via GUI and found the root cause. All Exchange related services are set to DEFAULT install path instead of the one I provided to Exchange setup (Sorry for Czech language).

I also tried to install Exchange using GUI with the same results. The only solution for my lab was to use default installation directory.
Does someone have the same problem?

I have faced another kind of Error ID: 8224, which was, however on virtual machine, not caused by TCP chimney issue, so here is the solution.

Problem

When running command

.\setup /PrepareSchema

I have received error 8224 as shown in picture:

In my previous article the problem was caused by incorrect settings of TCP interface (http://ficility.net/2012/09/06/extending-schema-for-exchange-2013-error-id-8224/), which caused delays in replication and change actually solved the problem. In this case TCP interface was set correctly and I still experienced delays in replication. Here is how to deal with this kind of problem.

Solution

Before extending schema domain controllers hosting FSMO roles (at least Schema master) must be accessible and replication should work correctly. In my case there was a problem replicating Schema and Forest zones to secondary DC, which was connected via VPN. I have resolved VPN connection problems and after that I forced replication between DCs. Note, that MS have incorrect syntax on their technet site. You have to use quotations to run command successfully.
Replication errors:

To force replication run the following command from elevated command prompt. http://technet.microsoft.com/cs-cz/library/cc742152(v=ws.10).aspx

repadmin /replicate <SOURCE_DC> <DESTINATION_DC> "<DC=domain,DC=local>"

Result:

After successful replication it is possible to extend schema.

Let me provide you fresh experience with Exchange 2010 SP3 upgrade.

In the first place, thank you Zbynek, because final solution was his idea!

Problem

Exchange 2010 SP3 upgrade unexpectedly failed for 2 servers from 9. Those servers had separated Exchange roles. So the following error occurred for MBX as well as HUB role.

[10/05/2013 18:58:41.0984] [2] Saving object "EXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" and state "New".
[10/05/2013 18:58:42.0015] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0481] [2] Searching objects "DEXMBX02\PowerShell-Proxy (Default Web Site)" of type "ADPowerShellVirtualDirectory" under the root "$null".
[10/05/2013 18:58:43.0497] [2] Previous operation run on domain controller 'DC03.contoso.local'.
[10/05/2013 18:58:43.0497] [2] Ending processing new-PowerShellVirtualDirectory
[10/05/2013 18:58:43.0497] [1] The following 1 error(s) occurred during task execution:
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: A failure occurred while trying to update metabase properties.
[10/05/2013 18:58:43.0497] [1] 0.  ErrorRecord: Microsoft.Exchange.Data.Common.LocalizedException: A failure occurred while trying to update metabase properties. ---> System.Runtime.InteropServices.COMException (0x80070003): The system cannot find the path specified.

As can be seen it was IIS related problem (a failure occurred while trying to update metabase properties) especially with PowerShellVirtualDirectory.

Root Cause

Only suspicion:

• firewall or application (e.g. an anti-virus) was cutting the connection
• an application was locking the IIS metabase (e.g. a backup solution)
• not sufficient permissions

Solution

This solution is intended for separated (CAS, MBX, HUB) as well as multi roles.

1.  Remove corrupted PowerShellVirtualDirectory:

* remove all virtual directories whether CAS role

Get-PowerShellVirtualDirectory EXMBX02\* | Remove-PowerShellVirtualDirectory

2.  Recover Exchange server:

Setup /m:RecoverServer

Appendix

• Exchange SP3 fails upgrade on Hub Transport Role

  http://social.technet.microsoft.com/Forums/exchange/en-US/74c7de22-da60-42b6-948e-24c75aad3808/exchange-sp3-fails-upgrade-on-hub-transport-role?forum=exchange2010

• Exchange 2010 SP1 or SP2 trouble with: The virtual directory ‘PowerShell’ already exists

  http://www.arconi.com/index.php/blog/item/exchange-2010-sp1-or-sp2-trouble-with-the-virtual-directory-powershell-already-exists

• Recover an Exchange Server

  http://technet.microsoft.com/en-us/library/dd876880(v=exchg.141).aspx

• Recover a Database Availability Group Member Server

  http://technet.microsoft.com/en-us/library/dd638206(v=exchg.141).aspx

• Call-ClusterExe: cluster.exe did not succeed, but 5058 was not a retry-able error code

  http://clintboessen.blogspot.fi/2012/05/call-clusterexe-clusterexe-did-not.html

  http://www.cdolar.com/2013/01/call-clusterexe-cluster-exe-did-not-succeed-but-5058-was-not-a-retry-able-error-code/

My friend came with problem to add second copy of databese in Exchange 2013 DAG, because replication service failed to perform initial seed due to different configuration between source (first copy)  and target (second copy)disks. I wanted to test it, so here is case study:

LAB:

• 2x DC, 2x Exchange 2013 - CAS and Mailbox role in DAG, HAProxy load balancer, Exchange 2010 SP3 multirole server
• All running under VMWare player
• TESTDB: First copy on one DAG member on mountpoint F: (GPT, SCSI, NTFS , 8kB / sector)

Getting info about NTFS:

I used the Powershell function presented in this article http://cornasdf.blogspot.cz/2010/03/using-powershell-to-get-ntfs-info-such.html?showComment=1362006532506#c4110837443209977959

Thanks to Conrad then gathering NTFS info is as easy as:

Get-NTFSInfo f

and result

Drive                           : f
NTFS_Volume_Serial_Number       : 0xfaf6f756f6f7121d
Version                         : 3.1
Number_Sectors                  : 20901887 (0x00000000013eefff)
Total_Clusters                  : 1306367 (0x000000000013eeff)
Free_Clusters                   : 1234271 (0x000000000012d55f)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : <Not Supported>
Bytes_Per_Cluster               : 8192
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 524288 (0x0000000000080000)
Mft_Start_Lcn                   : 393216 (0x0000000000060000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 393248 (0x0000000000060020)
Mft_Zone_End                    : 418848 (0x0000000000066420)
RM_Identifier                   : 0E320AB6-7A27-11E2-B355-000C2940FA17

Plan:

Test replication issues while using the following setup:

1. Create second copy of the database using SCSI, GPT, NTFS but larger size of the block (65kB)
2. Create second copy of the database using SCSI, GPT, NTFS but lower size of the block (2kB)
3. Create second copy of the database using IDE, MBR disk with the same NTFS config as the first copy of the database
4. Create second copy of the database using IDE, MBR disk with the different NTFS config as the first copy of the database (65kB block)

Ad 1) Create second copy of the database using SCSI, GPT, NTFS but different size of the block (65kB)

I have preconfigured the disk with the same drive letter F: , GPT and now I will format the NTFS to 65kB block size. The info from NTFS:

[PS] C:\Windows\system32>get-ntfsinfo f

Drive                           : f
NTFS_Volume_Serial_Number       : 0x5ed0732ad0730793
Version                         : 3.1
Number_Sectors                  : 20901887 (0x00000000013eefff)
Total_Clusters                  : 163295 (0x0000000000027ddf)
Free_Clusters                   : 162095 (0x000000000002792f)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 
Bytes_Per_Cluster               : 65536
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 65536 (0x0000000000010000)
Mft_Start_Lcn                   : 49152 (0x000000000000c000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 49152 (0x000000000000c000)
Mft_Zone_End                    : 52384 (0x000000000000cca0)
RM_Identifier                   : 1749F9E1-7CB8-11E2-B6CB-000C29ECA938

Adding DB copy:

Add-MailboxDatabaseCopy -Identity TestDB -MailboxServer FrontEnd1 -ActivationPreference 2

Result:

Works normally. Database is seeded and also incremental seed works. Before point 2 I removed the mailbox database copy and formatted NTFS with lower block size.

[PS] C:\Windows\system32>Remove-MailboxDatabaseCopy testdb\frontend1

Confirm
Are you sure you want to perform this action?
Removing database copy for database "TESTDB" on server "FRONTEND1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The copy of mailbox database "TESTDB" on server "FRONTEND1" has been removed. If necessary, manually delete
the database copy's files located at "f:\TESTDB" and "F:\TESTDB\Testdb.edb" on that server.
[PS] C:\Windows\system32>

AD 2) Create second copy of the database using SCSI, GPT, NTFS but lower size of the block (2kB)

I have preconfigured the disk with the same drive letter F: , GPT and now I will format the NTFS to 2kB block size.

Works normally. Database is seeded and also incremental seed works. Before point 2 I removed the mailbox database copy and formatted NTFS with lower block size.

AD 3)Create second copy of the database using IDE, MBR disk with the same NTFS config as the first copy of the database

Seeding works normally as in other configurations.

PS C:\Users\administrator.SALONOVI> get-ntfsinfo f

Drive                           : f
NTFS_Volume_Serial_Number       : 0x42f4703af47031f1
Version                         : 3.1
Number_Sectors                  : 20965375 (0x00000000013fe7ff)
Total_Clusters                  : 1310335 (0x000000000013fe7f)
Free_Clusters                   : 1300725 (0x000000000013d8f5)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 512
Bytes_Per_Cluster               : 8192
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 0
Mft_Valid_Data_Length           : 262144 (0x0000000000040000)
Mft_Start_Lcn                   : 393216 (0x0000000000060000)
Mft2_Start_Lcn                  : 1 (0x0000000000000001)
Mft_Zone_Start                  : 393216 (0x0000000000060000)
Mft_Zone_End                    : 418848 (0x0000000000066420)
RM_Identifier                   : 700D6323-8444-11E2-9E80-000C29ECA938

Ad 4) Create second copy of the database using IDE, MBR disk with the different NTFS config as the first copy of the database (4kB block)

Works as other configurations.

Drive                           : f
NTFS_Volume_Serial_Number       : 0x0a262c1b262c0a71
Version                         : 3.1
Number_Sectors                  : 20965375 (0x00000000013fe7ff)
Total_Clusters                  : 10482687 (0x00000000009ff3ff)
Free_Clusters                   : 9682285 (0x000000000093bd6d)
Total_Reserved                  : 0 (0x0000000000000000)
Bytes_Per_Sector                : 512
Bytes_Per_Physical_Sector       : 512
Bytes_Per_Cluster               : 1024
Bytes_Per_FileRecord_Segment    : 1024
Clusters_Per_FileRecord_Segment : 1
Mft_Valid_Data_Length           : 524288 (0x0000000000080000)
Mft_Start_Lcn                   : 3145728 (0x0000000000300000)
Mft2_Start_Lcn                  : 8 (0x0000000000000008)
Mft_Zone_Start                  : 3145728 (0x0000000000300000)
Mft_Zone_End                    : 3350560 (0x0000000000332020)
RM_Identifier                   : 700D6344-8444-11E2-9E80-000C29ECA938

Result:

I haven´t found error or problematic configuration, however, there might be some stuff useful for others. It took so much energy to test, that I would still like to post this article for future refference.

This article is continuation of the http://ficility.net/2013/10/16/exchange-2010-sp3-hybrid-server-in-organization-which-have-had-exchange-200x-before-freebusy-issues/ . After installation of RU2 for Exchange Server 2010 SP3 we have had problem to intefrate ForeFront to its new (mailbox role). The MSExchangeIS service stucked and started to Start/Stop in the loop. Here are the corrective actions (actions will remove ForeFront and its settings, so if you are not sure how to configure your ForeFront, don´t use it):

Problem:

The following Event IDs are circulating: 1007 (FSC moniror Initialize) -> 1008 (FSC moniror Termination) -> 9581 (MSExchangeIS – Virus scan cannot be loaded) -> 9564 (Cannost start Information Store because Virus Scan failed to load)

Solution:

• Stop services

Stop-service MSExchangeIS
Stop-service MSExchangeTransport
Stop-Service FSCController -force

• Remove ForeFront Integration (I have used commands to integrate ForeFront previously here: http://ficility.net/2013/01/26/exchange-2010-msexchangetransport-service-crashed-forefront-mail-pickup-service-error-event-id4999-event-id10003-%ef%bb%bfevent-id5167-event-id17007/)
  From FF protection powershell console run

FSCutility.exe /Disable

• Uninstall ForeFront (From Programs and features menu)
• Restart server
• Install ForeFront from media again and it will be automatically integrated to MSExchangeIS again.
• Restart server
• Configure ForeFront again, because re-installation removed your settings.
• Test server
  From FF management console run

   FSCUtility.exe /status

• If ForeFront is integrated, run the following to test services on the server and to check if DB is mounted.

  Test-ServiceHealthGet-MailboxDatabaseCopyStatus
  Get-MailboxDatabaseCopyStatus

I have problems in some environments, where these SChannel errors are generated. Well. It took me several days to find reasonable “why” it is logged.

Problem:

The event ID from the picture can be seen from time to time:

Solution:

Based on several articles I have read and some discussions. First you have to make sure, that the process causing this error is LSASS.exe, which is by the way local security authentication server (authenticating users to winlogon service, using authentication such as msgina.dll and so on). To make sure it is LSASS.EXE. Open Event ID and check the Event ID details, Click on Details tab -> Expand System while friendly view is selected. Check Process ID.

Then use powershell and run:

Get-Process | select name,id | sort id

Result should give you the name of the processes. It will be lsass.exe.

Why:

Reason is simple. Not standard or corrupted behavior of web browsers or users. The problem behind SChannel and Exchange 2012 is, that sometimes users use HTTP protocol, but on port 443, which expects certificates exchange rather than GET command.

How to test:

Option 1#:

Test is easy. For example you can input URL to your browser address bar, which is obviously wrong and see the results: HTTP://MAIL.DOMAIN.LOCAL:443/OWA – It says to use HTTP protocol (not HTTPS) on the 443 port and it generates errors immediately.

Option 2#:

Run Telnet and test command:

Telnet localhost 443 (to connect to HTTPS)

In Telnet window:

Get /index.htm (on HTTPS SSL must be established first so it will generate errors immediately. Result will not be seen in telnet window)

What is the solution?

Solution #1:

Some IT guys recommend to disable SCHannel logging to get rid of these events, but I cannot recommend that. To be honest. It is better to see, that somebody is trying to connect using HTTP on HTTPS port, because this might be some attempt to DoS attack or info, that users don´t know how to type OWA URL correctly. Shortly it is better to know something is wrong than disable logging.

Solution #2:

I suspect wrong redirect configuration for the websites from HTTP to HTTPS. I would check IIS if redirect is set correctly. For those having this issue without redirect I would suspect problem in web browser area.

Links:

To test SSL via command line:

http://www.bearfruit.org/2008/04/17/telnet-for-testing-ssl-https-websites/

LSASS description:

http://www.neuber.com/taskmanager/process/lsass.exe.html

I came into problem recently. After upgrade of Exchange 2013 CU2 to CU3 my ECP stopped to work. I tried to:

• re-create ECP virtual directory
• renew certificate (which expired recently)

none of those helped.

Error:

Then I checked authentication methods for my virtual directories and found, that Exchange 2013 CU3 setup messed up my OWA authentication methods. The goal here is to have both virtual directories set with the same authentication methods. In my case FBA and Basic.

Solution:

Set correct authentication methods and run IISreset on problematic backend exchange servers.

Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -FormsAuthentication $true -BasicAuthentication $true