Maintenance Troubleshooting

1. Troubleshooting

This document will guide you through common failures for both working with the 12d Synergy Master Server and the 12d Synergy File Replication Servers. This will cover: 

1. Connectivity Errors
2. Authentication Errors
3. General Database Errors

This is intended to be an exhaustive list, but is a good first port of call for general and common errors.

2. Master Server Failures

Service Starts and then Stops

Usually, an automatic stop is caused by:

1. Credential issues for the service
2. The Microsoft .NET Framework being out of date
3. A Corrupted installation
4. Incorrect settings after a migration / moving folders on disk

If the service is immediately stops, there are two locations to check for more information.

If the service is unable to communicate with the database to work out where it should log to, you should check the Windows Event Viewer for any errors. The application log should contain at least one error regarding 12d Synergy.

If there are no errors, you should check the 12d Synergy File Log for any errors. If you are unable to connect to the server, see the appendix on how to find the file logs.

Server Starts but LAN and WAN Users Can't Connect

3. Warm Up Failures

When the service starts, it will attempt to:

1. Connect to the database
2. Connect to Active Directory, if active directory is being used

As those operations may take time, the service itself may start even if these processes fail.

As some of these services may be slower to start than the 12d Synergy Server, any attempts to connect that are met with failure will cause the service to sleep and try again after about thirty seconds.

This may give the appearance of the service starting, but it still may be in an initialization or warm up phrase.

Any errors during this phase will be visible in the Windows Application Event Viewer.

Once the server has started, you should expect to see the following in the logs:

12d Synergy Server is now operational! 

4. Firewall Restrictions

If the 12d Synergy Server is operational, as per the file log, but users cannot connect, then make sure firewall exception for the installed ports are in place.

By default, 12d Synergy uses port 1212 and 1250 for communication.

You should test only using the LAN until you can determine if users are able to connect again.

Note: LAN Users Can Connect but WAN Users Cannot

4.1. External Firewall / Router Restrictions

If Users on the LAN can connect but users outside the LAN cannot, then it is likely that a hardware or external firewall is the culprit.

Check any relevant external firewalls / routers to ensure the installed ports (default to 1212 and 1250) are allowed through, and are being forwarded to the correct machine.

5. Authentication / Login Issues

You may receive the following errors:

1. The username and password combination you supplied was not correct or not valid for this server.
2. You are not authorized to access this 12d Synergy Server.

12d Synergy has three modes of authentication, which will be tried in the following order:

1. Windows Authentication against AD
2. Windows Authentication against Local User Accounts on the server.
3. Custom login Authentication, if set up

Windows Authentication will user Active Directory first if your server is on a domain. The Active Directory setting in 12d Synergy refers only to Group Management. If the server is attached to a domain, AD will be used for authentication as a first attempt.

Next, 12d Synergy will try look at locate Windows Logins.

At this point, if Windows Single Sign On is not successful, the user will be prompted for a user name or password.

This may occur if they are logging in from a computer whose credentials do not match the credentials for the server, or if there are custom logins.

If Custom Logins are not setup, and the user cannot login, this is a problem with the windows environment and their credentials should be checked.

Turning on security audit logging on the Windows Server may yield more information.

Receiving the error message "The username and password combination you supplied was not valid for the server" typically means Windows has failed to authenticate you.

If you receive a message indicating "You are not authorized to access this 12d Synergy server", it means that Windows has authenticated you but 12d Synergy cannot find information about your User Identity.

Your Administration should check your identity settings for your account to make sure they match, 12d Synergy Identities should not contain domain information.

6. Certificate Problems

If you are using custom logins, you may receive this message:

Future versions will remove this error message as it is needlessly technical.

This problem arises because Windows has failed to authenticate the user, and is now attempting to use custom logins. However, custom logins require the use of an x509 / SSL Certificate.

Certificates have two purposes:

1. Encryption
2. Identity

In this case, the client cannot confirm the identity of the server. This has probably occurred because the address they have entered does not match the identity encoded in the certificate.

The certificate subject or address must match the address the user enters to verify identity.

It is possible to turn off identity checked on a per client basis, by setting a Windows Environment Variable named Skip12dSynergyIdentityCheck to 1

See the Appendix for how to set a Windows Environment Variable.

7. Timeout Errors or Event Cancellations

From time to time, users may receive errors about timeouts or their events may be cancelled, with accompanying errors in the logs about timeouts. If this takes place, this means there is a problem with the database server.

There are several things to look at.

Latency

Every time 12d Synergy needs to know something from the database, or to write something to the database it sends a query and retrieves a result. The time it takes is known as the round tripping time. If there is significant latency between the 12d Synergy Service and the SQL Server service, this may result in timeouts. Ensure that latency between the two systems is <10 ms, but <1 ms is optimal. Latency is not an issue if the 12d Synergy Server Service and the SQL Server Service are on the same physical machine.

8. Available Memory

SQL Server is very sensitive to memory. You must meet our minimum requirements, but more so, you must ensure there is ample memory available to SQL Server on the server. There must be ample room for SQL Server to grow its memory as required - SQL Server is very good at managing its memory, assuming there are no other greedy services (such as Domain Controllers, Microsoft Exchange or even other SQL Servers) on the same machine.

Once SQL Server runs out of available memory, it will start a process known as paging to disk, which can slow down queries significantly.

If you have over 75% of available memory being used, and you are experiencing timeouts, you may wish to: 

1. Increase the RAM on your server
2. Investigate other services that may be consuming your resources.

9. Index Fragmentation

Databases uses Indexes to speed up how they look for information. However, on each write to a table, indexes must be adjusted. Over time, these can become significantly fragmented which will slow down queries to the table and updates/inserts to the data.

It is the responsibility of the SQL Server Administrators offers an option to 'tune' the database. This should be done out of hours when usage is low, but will attempt to defragment and rebuild your indexes.

Should you need more finely tuned fragmentation, please see your Database Administrator.

A query for checking fragmentation and for recalculating has been included in the appendix for your reference.

10. Transaction Log Truncation

SQL Server operates with two recovery modes: Simple or Full.

Simple Recovery mode means that you are responsible for every backup - if there is a failure, you may use only the last full or differential backup. This requires less maintenance, but has greater risk.

Full Recovery mode uses a transaction log that can give you to the second recovery if required. However this requires more maintenance.

The Transaction Log must be backed up and truncated regularly - as the log fills and grows, subsequent writes will slow down and can lead to timeouts. A large transaction log file is not a problem if it is mostly empty, but a large and full file is a problem.

See the appendix for more information on how to backup and truncate your transaction log properly.

11. File Replication Server Failures

Service Starts and Then Stops

Usually, an automatic stop is caused by:

1. Credential issues for the service
2. The Microsoft .NET Framework being out of date
3. A corrupted installation
4. Incorrect settings after a migration / moving folders on disk

If the service immediately stops, there are two locations to check for more information.

If the service is unable to communicate with the database to work out where it should log to, you should check the Windows Event Viewer for any errors. The application log should contain at least one error regarding 12d Synergy.

If there are no errors, you should check the 12d Synergy File Log for any errors. If you are unable to connect to the server, see the appendix on how to find the file logs.

Server Starts but LAN and WAN Users Can't Connect

Warm up Failures

When the server starts, it will attempt to:

1. Connect to the database
2. Connect to the Master Server for settings and configuration

As these operations may take time, the service itself may start even if these processes fail.

As some of these services may be slower to start than the FRS, any attempts to connect that are met with failure will cause the service to sleep and try again after about thirty seconds.

This may give the appearance of the service starting, but it still may be in an initialisation or warm up phrase.

Any errors during this phase will be visible in the Windows Application Event Viewer.

Once the server has started, you should expect to see the following in the logs:

File replication server is now ready for connections

12. Firewall Restrictions

If the File Replication Server is operational, as per the file log, but users cannot connect, then make sure firewall exception for the installed ports are in place.

By default, the 12d Synergy FRS uses port 1212 for inbound connections.

However, the 12d Synergy FRS uses a network of servers - it will communicate with the Master Server over its default transfer port (1250), and to any other FRS' in your network.

You need to add the relevant exceptions to any software or hardware firewalls.

13. Common Failures

13.1. Running Out of Disk Space

While users may only see that their event has been cancelled, server logs may reveal errors indicating that there is not enough disk space.

This can be an issue in two separate places:

File Stores

File Stores are where 12d Synergy Stores its files. Jobs use rules that dictate which files get stored in which File Store. Consider using expandable storage, or add new file stores as necessary.

13.2. Temporary Storage

When files are uploaded, these are stored in a temporary location until they can be moved into the relevant file store. By default, this is the windows Temporary Directory. 12d Synergy's temp space is cleaned up regularly, including after a restart, but in some cases, files may exceed the available space.

You can set a new path by setting the windows environment variable, 12dSynergyTempPath, to a more appropriate path with more space.

See the Appendix for details on how to set a windows environment variable. Setting this will require a restart of the relevant service.