NFS storage for Atlassian Server applications
Atlassian applications have the tendency to grow on you and your organization. What often starts as a tool for a single team, will soon pick up momentum and more and more teams will follow. To a point that management will realize that it is becoming an important tool within the business. Data stored in the application is business critical and needs to be safeguarded. To realize this, the so called under-the-desk-server will be handed-over to the IT department and there it will be properly nurtured by the IT staff. But what does that even mean? Naturally you have an excellent operating IT department which:
- will upgrade the applications to a stable version on regular intervals
- patch quickly security vulnerabilities, when Atlassian releases one
- will patch the OS when needed
- monitor the servers
- create backups
- etc
When more and more users of your organization are using the applications, they also add more content to it. The content ranges from office documents, to pictures, to git repositories, etc. Over time all these bits and bytes will add up and you find yourself expanding the local disks of your server to the point that it makes sense to rethink the choices you made in the passed. You might even consider to go to a high availability solution in the form of an Atlassian Data Center solution. To prepare for this transition you can start storing the data on a NFS file server.
So what do you do?
Well while you’re at it, you probably want to move the complete home directory of your application to the NFS server, because why not? Looking at the data center setup of an Atlassian application you can easily make the decision to move the home directory of your Atlassian server application to NFS.
Well this is where I have to stop you! The applications cannot handle a home directory which is located on NFS. This has multiple reasons:
- Jira needs to index the issues, over NFS this will be very time consuming. Storing the index on your local SSD will improve index time drastically.As an example: A re-index of a single project with 2400 issues where the home directory is located on NFS took 3 hours and 14 minutes. On the same instance, with only a part of the home directory on NFS, it took 35 minutes to complete a re-index of the system (total of 250k+ issues).
- A similar story goes for Confluence. Having the complete home directory on NFS will result in a very slow indexing of the pages.
- For Bitbucket there is an even more compelling reason to only store a part of the home directory on NFS. The main reason is the speed of git operations.What we’ve observed is that when you start creating a clone for a repository, git will start to create a git-pack-file on the server. During the creation of this git-pack-file the download speed is up to 60 times slower then when the creation of the git-pack-file is finished. This behavior especially comes to light when you clone very large repositories (>1 GB). A second observation was that when multiple people will start the same clone operation (same repository, same commit) they all have to wait on the same git-pack-file operation. Once finished, speeds raised again for all clones. Finally we saw that once you’ve started a clone operation, but decided to break it off at the client site, the git-pack operation continued on the server.
So what needs to be done is to only allocate certain directories on the NFS server to store data on. All other data in the home directory should be stored on storage which is local to your server, preferably an SSD drive.
Split your home!
And this is how you do it. First of all do not change the wheel while driving, so shutdown your application before starting the data migration.
For Jira you need to copy the current data of the home folders ‘data’, ‘plugins’, ‘logos’, ‘import’ and ‘export’ to the NFS server. Then export these on the NFS server, create mount over directories in the Jira home directory, mount them and Bob is your uncle!
For Confluence you do the same, but now with the folders ‘attachments’, ‘backups’, ‘export’, ‘recovery’, ‘restore’, ‘shared-home’ and ’thumbnails’. Note that on your server some of these directories might not exist. If you didn’t use certain functionality, like restoring a space, the directory is not created.
And finally for Bitbucket you need to move the ‘shared’ to the NFS server.
After you have split the home directory this way, you will continue your journey with a blistering speed and huge growth possibility!
If you are interested and want to learn more about NFS feel free to contact our team. Contact detail are on this part of our website (https://www.tmcalm.nl/).
