How to Back Up Your AI Models, Datasets and Training Checkpoints

The expensive part of an AI project is rarely the hardware. It is the work stored on it: datasets someone spent weeks cleaning, a fine-tuned model that took days of GPU time, the prompts, evaluation sets and configuration that make the system behave. All of it usually sits on one internal SSD with no redundancy. Backing it up well starts with sorting files into what can be downloaded again and what cannot be recreated, protecting the second group with versioned copies on a NAS and a further copy elsewhere, and setting a rule for training checkpoints so they do not swallow the storage. A NAS costing a couple of thousand dollars is a reasonable way to protect a workstation costing ten times that, and the work on it, which is worth more than either.

What needs backing up, and how carefully?

The small items are the ones most often forgotten and the hardest to recreate. A lost base model is an inconvenience. A lost system prompt, evaluation set or labelled dataset can set a project back by months.

How should checkpoints be handled?

Checkpoints are saved copies of a model partway through training, so a run can resume after an interruption. They are large, they multiply, and most are worthless once the run finishes.

CALC. Fully fine-tuning a model whose checkpoint is about 15GB, saving every 500 steps for 10,000 steps, produces 10,000 / 500 = 20 checkpoints, or 20 x 15 = 300GB for a single run. Keeping only the last three and the best one stores 4 x 15 = 60GB. Five such runs differ by 1,500GB against 300GB. The sizes are illustrative; parameter-efficient methods such as LoRA save adapters measured in megabytes, which changes the picture entirely.

A workable rule:

• During a run, keep the latest few checkpoints on the workstation and copy them to the NAS on a schedule.
• When the run ends, keep the final model, the best checkpoint by evaluation score, and the exact configuration and data version that produced them.
• Delete the rest. Record what was deleted and why.

What does a sound backup plan look like?

The long-standing 3-2-1 rule fits AI work well: three copies of anything irreplaceable, on two different devices or media, with one copy off site.

1. Copy 1: the workstation. The working copy on fast internal storage.
2. Copy 2: the NAS. A scheduled sync from the workstation, onto redundant drives, with snapshots so earlier versions can be recovered. Snapshots protect against the common accidents: an overwritten adapter, a deleted folder, a script that modified the dataset in place.
3. Copy 3: somewhere else. A second NAS or backup appliance in another location, or a cloud backup target, holding at least the irreplaceable items. This copy is what survives fire, theft or ransomware that reaches the NAS.

Two refinements matter. Make at least one copy that cannot be altered or deleted for a set period, where your equipment supports it, since ransomware targets backups first. And test a restore: a backup that has never been restored is an assumption. For the wider recovery picture see the single point of failure problem.

Ask of every file: if this vanished tonight, could I get it back by downloading, by waiting, or not at all? The third group is your backup priority, and it is usually the smallest.

What about versions?

A backup tells you a file exists. It does not tell you which dataset trained which model. Keep a simple record alongside each trained model: the base model and its version, the dataset version, the training configuration and the evaluation result. With that, a model can be reproduced; without it, a restored file is a mystery. Dataset versioning also guards against a quieter loss, where data is changed in place and the earlier state is gone. See the AI data pipeline.

What are the common mistakes?

• Relying on RAID. RAID keeps a system running through a drive failure. It copies deletions and corruption faithfully.
• Backing up the big files and missing the small ones. Prompts, configs and evaluation sets.
• Keeping every checkpoint until the volume is full and backups fail silently.
• A backup share the workstation can freely delete from. Malware on the workstation can then remove the backup too.
• No restore test.
• No power protection. A power cut mid-write can corrupt a checkpoint or a volume.

Buying in Singapore

Primary backup NAS: the Synology DS925+ (S$1,260) or Synology DS1825+ (S$2,148), with Seagate IronWolf Pro drives from S$390. Dedicated second copy with protected backups: the Synology ActiveProtect DP340 (S$6,526.80). What is being protected: the NVIDIA DGX Spark 4TB at S$9,450 or an RTX PRO 6000 Blackwell workstation card at S$29,100 (listed 11 Nov 2026; prices change; ask for a current quotation). To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

What should be backed up in an AI project? Prioritise what cannot be recreated: fine-tuned models and adapters, datasets you built or labelled, code, prompts, configuration and evaluation sets. Public base models are lower priority because they can usually be downloaded again.

Do I need to keep every training checkpoint? No. Keep the latest few during a run, then retain the final model and the best checkpoint with the configuration that produced them, and delete the rest.

Is RAID a backup for AI data? No. RAID protects against a drive failing but copies deletions, overwrites and ransomware damage across all drives. Versioned backups and a separate copy are still needed.

How many copies of AI data should there be? Follow the 3-2-1 rule for irreplaceable data: three copies, on two different devices, with one off site, and test that a restore works.